Softwaremaker

Hi-Def AVCHD Video Format Processing for mere mortals ...

William Tay — Fri, 13 Jun 2008 04:13:25 GMT

I had recently purchased a Canon Hi-Def Flash Camcorder HF100 at wholesale price (please dont ask me how much and where I got it from). There was a long thought process before this high-end purchase. I knew I wanted a camcorder to record in Hi-Def (HD) format. The question I had was the recorded video format. I did some research and poking around and there were some pros and cons that I was seriously considering such as:

What is the recording storage medium ?
What is the recording format ? If answer to [1] was a DV Tape or sort, then the answer would probably be HDV/MPEG-2 format.
Do I have enought processing power / software infrastructure to deal with the answer to [2] ?

In the end, I decided that I would not want to do the route of using a DV Tape. Tape is proven, tested, good, mature and cheap but has its limitations. The fact that it is a sequential access medium puts me off. Even newER backup solutions of today seems to provide disk storage, whose prices have dropped in recent years, as an alternative to tape. Usually, the restoration granularity and the time it takes to restore is the deciding factor for customers to champion disk over tape. Moreover, if I record on tape and then later edit on disk, it does somehow seem that I am going backwards.

So, instead of carrying bulkIER tapes around with me (and I do a lot of random recording), not forgetting that the housing for these tapes in the camcorders itself does take up some bulk and effectively limits the handling of the camera at hand, I dumped the idea of either the Canon HV20 or HV30. Mind you - their dual recording format in a choice of either Standard-Definition (SD)or HD is really attractive but I doubt that I would want to record in SD in a couple of years down the road where computing power, screens, bandwidth are all commodities.

That left me with Question [3] above. What does it take to process/edit those videos ? From searches of many forums and reviews, a lot of people buy a AVCHD Camcorder (such as the Canon HF100) without realizing that they dont have the infrastructure to process and edit the recorded HD clips. I guess a lot of peple dont realize that there is not much choice of video-editing software that can process a AVCHD video clip today. So. what most of them did was

Pay X Dollar for the camcorder and then 2X Dollar for a brand new Mac - Holy Smokes. Since when does money grow on trees ?
Pay a couple of hundred dollars more to buy a decent video editing software such as the Pinnacle Studio or the Sony Vegas

I didnt like both options. First or all, I edit video clips - Yes - but I dont consider myself to be a "pro-consumer" of sorts that would want to fork out much money just to have 3000 over video transitions up my sleeve ... and ... I am not a MAC fan. Yes, I admit. Crucify me. I am just not genetically engineered to use a MAC or any of Apples' products. Yes, I love my ZUNE and its marketplace very much. Thank you.

Therefore, I had to look for an intermediate solution since my old, trusted and most FREE Windows Movie Maker and Media Player cannot handle AVCHD video files natively and I am not willing to fork out anything more than SGD100.00

Luckily, my prayers are answered and my search leads me to media\video developer ShedWorx who has the VoltaicHD for both the PC and the MAC. Bascially, VoltaicHD transforms your AVCHD High-Def video clips to WMV-HD, which both Windows Media Player and Microsoft Movie Maker can handle. FAQ here. This little known shareware (just USD30.00) has gotten some great independent reviews so I went for a trial, downloaded a sample AVCHD .MTS file and it worked like a charm.

[Note to ShedWorx]: Now if you could make a command-prompt version of your awesome tool, that would be a great addition as it would complete a workflow scenario of an "unattended" conversion process of the captured AVCHD .MTS files to WMV-HD.

With that, I bought it and went broke but GOSH - what a camcorder !!! Its light, intuitive, great handling and churn out great looking HD video clips. I guess the reviews out there in the wild will do it better justice than me writing about it here.

Yes, the computing power and storage resources are high. At the best quality mode, the HF100 records at 17Mbps and my usual mode would be to record at a compromised (between storage and battery power) bitrate of 7Mbps. Even with a decent Core2 Duo Processor T7200 2.0 GHZ (highly-rated) Merom chip and 2Gs of RAM that I have, editing a WMV-HD 7Mbps video clip does require some patience. And the file recordings are huge - as a rough gauge - AVCHD are abt 120Mb (15MB) /min of footage and becomes 500Mb (wmv) after decompression !!!

Luckily, I delegated the conversion of AVCHD -> WMV-HD files to one of my servers, running a Dual Core XEON Pro 5140 2.33GHZ 4MB L2 cache 1333MHz FSB - Woodcrest Chip and this was much faster, comparatively. In any case, this can be done unattended, and this would also give me a good excuse to plug in another same processor on this 2-way box in the near future. .

All in all, this is a great buy at near-wholesale price and I already had quite a lot of fun doing roving and recording real 1080p high-definition videos and enjoying the processed WMV-HD clips on my wide-screen LCD monitor, my HDTV as well as my Rapsody N35 media center (which plays WMV-HD High-Definition videos).

Below is a "short" clip I took with the above Canon HF100, with the sarcastic emphasis on "short". I took this 50-second clip in full 1080/9Mbps HD glory. Uncompressed file size is 210MB. In order to "dumb" it down so that it can squeeze and play better over the HTTP ravine, I had to re-encode it to a smaller scale/Mbps at 856x480/3Mbps. Even then, this same 50-second 856x480/3Mbps clip's file size is still at a large 19MB !!! If you blow up the player to your full-screen, you can see that it maintains a clear and good quality at full-screen even at 3Mbps. Mind you, the source look great on my local playback at 1080/9Mbps/25fps.

Windows Cardspace Breach ? I think NOT

William Tay — Sat, 31 May 2008 08:45:51 GMT

I was recently pointed to this post that highlights a "successful attempt" by some students in Germany to crack Microsoft Cardspace.After reading through the post several times, I became convinced that it is NOT what it seems it is and that if the "breach" is what it says it is, there must be some pre-conditions that must be satisfied before it can happen and these criteria are not going to be easy...

Just as I was putting some of my thoughts down that relates to why I think the attempt is somehow "inappropriately glorified":

If an end-user would be stupid enough to put and store his/her passwords, credit card information on his PC
There must be some sort of DNS compromise on the end-user side, which also means successfully hacking into his/her router
There must be some sort of Digital Certificate Store compromise on the end-user side, which also means successfully hacking into his machine with highly-elevated priviledges or saying, the user's machine password has been stolen

Points [2] and [3] relates to the statements from the attempt and I quote from the above post:

To reproduce the demonstration, you should change your own DNS settings and install an untrusted certificate

If I can do both those points sucessfully, to be honest, I already have control over what the user does on his machine, stealing his Infocard is probably of low priority at that point in time.

Then, the brains behind Cardspace, Kim Cameron, himself, wrote a comprehensive reply, which basically was a detailed answer to my brief thoughts above, to counter the students' attempt and should really put any doubts in anyone's mind to rest.

[Added 02 June 2008]: In this video on his blog, Kim demonstrates how YOU, the end-user, must FIRST POISON your own machine first before the attack can happen: http://www.identityblog.com/wp-content/images/2008/05/Students/Students.html

Some comments standout and I quote:

The demonstrator shows that if you are willing to compromise enough parts of your system using elevated access, you can render your system attackable. This aspect of the students’ attack is not noteworthy.

There is, however, one interesting aspect to their attack. It doesn’t concern CardSpace, but rather the way intermittent web site behavior can be combined with DNS to confuse the browser. The student’s paper proposes implementing a stronger “Same Origin Policy” to deal with this (and other) possible attacks. I wish they had concentrated on this positive contribution rather than making claims that require suspension of disbelief.

However, the students propose equipping browsers with end user certificates so the browsers would be authenticated, rather than the sites they are visiting. This represents a significant privacy problem in that a single tracking key would be used at all the sites the user visits. It also doesn’t solve the problem of knowning whether I am at a “good” site or not. The problem here is that if duped, I might provide an illegitimate site with information which seriously damages me.

While I know the ignorant media will find some ways to sensationalize this unworthy episode, especially when Microsoft is such a big target, this brings to mind a popular joke which I think can be used as an anology:

Q: How do you make 1 million dollars ?
A: Start with 2.

Too many books - too little time and space

William Tay — Tue, 29 Apr 2008 11:08:53 GMT

Gosh, I think I am in desparate need for some new empty bookshelves ...

<-- Click this pic to see a higher resolution for even more details.

...and you havent even seen my other bookshelves containing my other interest, which I wont share for now ...

This is simply amazing ...

William Tay — Wed, 16 Apr 2008 06:26:49 GMT

I am left speechless at this. Really. Simply astounding ...

You know you are amongst geeks when ...

William Tay — Tue, 08 Apr 2008 07:22:57 GMT

One cannot run away from understanding infrastructure needs when one is pitching or designing software solutions in the enterprise (which I do a lot of) and it is sometimes strange (in a pleasant way) when the conversation goes like this:

Please make sure you have failover expertise in your next meeting. I recommend getting Steve to proxy in for William, even though I dont think anyone can impersonate him. At least, I have been able to ascertain that Steven can mirror William quite well and will be able to backup William in the event of a failure

Windows Workflow Foundation: MultiThreaded Parallelism

William Tay — Mon, 25 Feb 2008 08:44:37 GMT

I remember back in 2005/2006 when I was still touring the APAC circuits such as Sydney (Australia) and Kuala Lumpur (Malaysia) doing training and consulting gigs for customers, partners about Windows Workflow Foundation (WF) and Windows Communication Foundation (WCF, previously - Indigo) and some of the initial Windows Workflow questions came up regarding the use of Parallel Activities. It came as a surprise to many people that parallel activities are not independently asynchronous.

I explained that a WF instance gets only one instance from the runtime. There are reasons for this single-threaded execution model so each activity have to work with this single thread efficiently. There are ways to spin off differents thread when real parallelism activities are reqquired but because documentation was scare at that time, I had some trouble articulating how to do so.

I just read "Multithreaded Parallelism in Windows Workflow Foundation" on MSDN and while it is a definite deep technical article, if you can grok it, you will understand how "MultiThreaded Parallelism" can be done in WF using both the (rather hard-to-use) "Call External Method Activity (CEMA)" and the "Handle External Event Activity (HEMA)". Not only that, the authors (whom actually implemented such a system for their own use) also shared how to pair those 2 activities up using correlation and how to create wrappers aoround them so that it can be reused and therefore "not require talented software developer use of call-external-method and handle-external-event activities along with the CLR thread-pool"

A gem of a read.

BizTalk 2006: BizTalk Server Operations Guide is available live

William Tay — Sat, 16 Feb 2008 22:25:50 GMT

If you make your living installing, deploying, managing and operating Microsoft BizTalk Server 2006. I guarantee that you will be excited and be thankful for the availability of the BizTalk Server Operations Guide. So, what's in it ?

Guidance based on real-world experience. The idea for the guide originated with Microsoft field representatives, partner organizations, and customers who plan, deploy, and maintain BizTalk Server installations. This group of IT professionals has accumulated extensive hands-on experience with a diverse range of BizTalk solutions. As they gained experience they created checklists, best practices, and presentations to guide future BizTalk Server operations. We collected and organized this information to create the guide.
Key portions of this guide are new; however, a considerable portion consists of documentation taken from BizTalk Server 2006 R2 Help, white papers, Knowledge Base articles, and other sources. It has been carefully reviewed and vetted by experts from the community of BizTalk Server IT professionals and members of the product development team, whom we gratefully acknowledge at the end of this topic. We believe that the information presented here will help BizTalk Server users solve, and above all, avoid many of the common problems that can occur while deploying and maintaining a BizTalk Server installation.

Enjoy.

The BizTalk Server Operations guide is now available for download in DOCX, CHM, and PDF file formats.

BizTalk 2006: Business Activity Monitoring Portal Enabled But Not Configurable

William Tay — Thu, 14 Feb 2008 23:07:47 GMT

Problem: If you are in the configuration screen of Microsoft BizTalk Server 2006 and you are configuring Business Activity Monitoring (BAM) Portal, you checked 'Enable BAM Portal" and you noticed that the Account and Windows Group options are still grayed out.

Cause: I suspect that if you have configured BAM Portal once from any server in the BizTalk Server group, the user account fields are disabled on the rest of the servers even though Enable BAM Portal option is still available.

Resolution: Reset BAMVRoot in the BAM config file: BAMConfig.xml and then run BM update-config

Used bm.exe to get the BAM configuration XML:

Open a command window
Go to C:\Program Files\Microsoft BizTalk Server 2006\Tracking
Type: bm.exe get-config -FileName:BAMConfig.xml

It specifies BAMVRoot so it appears someone configured BAMPortal before. Deleted this line:

<GlobalProperty Name="BAMVRoot">http://FOOBAR:80/BAM</GlobalProperty>

Saved the file and updated the BAM configuration:

Open a command window
Go to C:\Program Files\Microsoft BizTalk Server 2006\Tracking
Type: bm.exe update-config -FileName:NewBAMConfig.xml

The options will no longer grayed out. You will probably get an error that the "BAMAppPool already exists". That is fine. Delete it, restarted IIS and it will be configured successfully.

As far as I can tell - this is not documentated very widely and this has helped me so I hope it helps someone out there as well.

MOSS 2007: Index and Web-Front-End Server Topology

William Tay — Thu, 14 Feb 2008 13:48:19 GMT

Microsoft Office SharePoint Server (MOSS) has seen tremendous take-up rates, not just here in Singapore, but all over the world. I came across many kinds of architectural topology designs during field work with my customers.

One of the most common confusions I have come across arises from the topology design when it comes to the deployment of the MOSS Farms. Some people advocate that the Index server be placed in the same box as the Web-Front-End (WFE) servers with the view that idle power is wasted resources.

There is logic behind this:- Indexing/Crawling not just takes up processing power, it places a certain load on bandwidth as well. The typical organization will set it to crawl at night when online transactions are fairly low. Of course, this is very subjective and differs from environments to environments. A huge load doesnt mean just crawling thousands of websites and fileshares but sets it at a very frequent and short period for incremental crawling. Most of my customers set it to crawl 2-3 times a day (morning, lunch-time, wee-hours at night). That is really not a huge load.

If those are your requirements, you may think having a dediated Indexing Server may be an overkill. If you can bulk-up one of the WFE servers on roids (using a combination of RAM and CPUs), that particular WFE may be able to double-up as a Index server. This will save some costs as well as processing power on that one machine that is just expected to work 3 times a day. Mind you, that machine is expected to be of some decent build as well.

An indexing server can neither be load-balanced nor clustered. What most people want is availabilty at the Query servers, not the Index servers. The built indices are propagated to the Query/Search servers.

There are certain things you need to be aware of, especially when it comes to high-availability in your environment. As said in the referenced link, if an Index server doubles up as a Query service as well, it will not propogate its indices to the other Query servers - that is the gotcha. So, if you have already scoped out 2 WFEs (with the Query service ON) and 1 Index server together with a pair of clustered SQL boxes (one of the most common 5-server MOSS setups), you should try to see if you can run WFE on that Index server and turn the Query service of that particular server OFF. If you can successfully do that, you would have a 3 WFE, 2 Query and 1 Index logical server deployment. You should have the best of both worlds this time around. Hardware resources on that Index server will be more efficiently utilized and you can take one of the WFE servers offline and still enjoy redundancy.

Do take note that this is my own personal advice only. No two customers have exactly the same functional and non-functional requirements and a few of the cases I have seen actually run the crawlers on hundreds of sites, thousands of documents and fileshares and build up the indices up to 10 times a day. In those cases, you should have a dedicated Index server and not have any resource processing contention issues with it. Follow the principles to make sure you max out the hardware resources and costs.