Blog Home  Sign In RSS 2.0 Atom 1.0 CDF  

  def Softwaremaker() :
         return "William Tay", "<Challenging Conventions />"

  knownType_Serialize, about = Softwaremaker()
 

 Friday, January 23, 2009

Yes, really. No fluff. C'mon - has there ever been on this blog.

SQL Server 2008 really does go well with our Windows Server 2008, especially when you are replications across Wide Area Networks (WAN). Microsoft's internal IT, whom is managing global MSDN sites, did this project and concluded with this study: Geo-Replication Performance Gains with Microsoft SQL Server 2008 Running on Windows Server 2008.

This is a highly-used reference since being published a couple of months back. Note the case study numbers shows at least 100x improvement in a few cases. I quote some key content from the source.

openquotes.png

The Publisher and Distributor databases for the MSDN applications are located in a data center in Tukwila, Washington. The Subscriber databases are hosted in separate facilities in Washington and California. As shown in Table 1, the team calculated baseline latency of four milliseconds between the MSCOM Ops data centers in Washington; a maximum of 23 milliseconds between facilities in Washington and California; and 150 milliseconds from Tukwila, Washington to Dublin, Ireland.

Data Center

Tukwila

Quincy

Santa Clara

Blueridge

Dublin

Tukwila, Washington

1 ms

4 ms

19 ms

77 ms

150 ms

Quincy, Washington

4 ms

1 ms

23 ms

68 ms

146 ms

Santa Clara, California

19 ms

23 ms

1 ms

79 ms

156 ms

Blueridge, Virginia

77 ms

68 ms

79 ms

1 ms

84 ms

Dublin, Ireland

150 ms

146 ms

156 ms

84 ms

1 ms

SQL Server 2005 Running on Windows Server 2003
The team’s previous attempts to replicate data between Microsoft data centers located in Redmond, Washington, and Virginia by using SQL Server 2005 running on Windows Server 2003 helped reduce latency to a certain extent. But, to account for the real-world demands of the data-centric applications targeted for geographic redundancy, the team decided to conduct further tests on this platform by using the same data center locations.

The following describes the methodology that the MSCOM Ops team used to evaluate replication performance of SQL Server 2005 running on Windows Server 2003.

  • Tests were conducted between data centers located in Redmond and Virginia—a distance of approximately 3,000 miles.
  • Baseline testing was conducted for transactional replication of both push and pull subscriptions.
  • Push subscription model was used for initial test and pull subscription model was used for all subsequent testing.
  • Rows in database tables were 1K in length.
  • Rows were inserted in increments of 1,000, 10,000, and 100,000.
  • Each test pass was run three times, and run times were averaged.
  • The distribution database was housed on the Publisher in the replication topology for all tests.
  • Tests included “live” data from the MSDN database, including two tables that had schemas identical to those used in production. 
  • Based on a scenario in which the database system was separated into reads and writes, latency was determined during testing to be as high as four seconds.

Windows Server 2008 and SQL Server 2008
In late summer of 2008, the MSCOM Ops team initiated testing of the updates to the Database Engine in SQL Server 2008 alongside the improved TCP/IP stack in Windows Server 2008.

The following describes the methodology that the MSCOM Ops team used to evaluate replication performance of SQL Server 2008 on Windows Server 2008.

  • Tests were conducted between data centers located in Redmond and Virginia—a distance of approximately 3,000 miles.
  • Baseline testing was conducted for both push and pull subscriptions with transactional replication.
  • Push subscription model was used for initial test and pull subscription model was used for all subsequent testing.
  • Rows in database tables were 1K in length.
  • Rows were inserted in increments of 1,000, 10,000, 100,000 and 1,000,000.
  • Each test pass was run three times, and run times were averaged together.
  • The distribution database was housed on the Publisher in the replication topology for all tests.
  • Live data from the MSDN database, including two tables with identical schemas used in production, were included in tests. 
  • The agent profile changes for testing on SQL Server 2008 and Windows Server 2008 were made by using 1,000,000 records, equal to 15.5 gigabytes (GB) of data transfer.

Test Results Comparison
Testing showed that using transactional replication with SQL Server 2008 running on Windows Server 2008 dramatically outperformed SQL Server 2005 running on Windows Server 2003. As illustrated in Table 2, the most substantial performance gains occurred when the Publisher and Subscriber were both running SQL Server 2008 on Windows Server 2008. 

Testing also showed that the scope of the performance gains correlated with the type of replication and the type of data. Push subscription replication of character data with SQL Server 2008 running on Windows Server 2008 yielded a 104 percent increase over SQL Server 2005 running on Windows Server 2003, and pull subscription replication of the same data yielded a 1,298 percent gain. The team noted that changing the PacketSize or ReadBatchSize parameters in the Replication Distribution Agent (http://msdn.microsoft.com/en-us/library/ms147328.aspx) profile during testing did not significantly affect performance; these changes resulted in a savings of less than 60 seconds for replicating 1,000,000 rows of varbinary (max) data equal to 15.5 GB of data moved.

It should be noted that not all of the disk performance counters were relevant during testing, as the partitioning of the disks to support dual boot and two instances of SQL Server on each partition rendered the disk performance counters questionable. The key take-away from the disk counters is that the distribution database files are “hot spots” during replication, and that the process is input and output intensive. Disk-queue length averaged 150 for the Publisher with the test hardware.

MSCOM Ops is continuing to drill down into various parameters and variables in subsequent testing to further develop guidance and best practices. However, based on the substantial performance gains witnessed in the initial round of testing, the team believes it is possible to build a geographically redundant, fault-tolerant database system, including a high read and write version.

Performance Indicators

Test Scenarios

SQL Server 2005 on Windows Server 2003 (A)

SQL Server 2008 on Windows Server 2008 (B)

Performance Gains or Losses

[(A-B)/B]*100

CPU Utilization (%)

All

15%

15%

0%

Memory

All

99%

99%

0%

Push Replication

1-GB

1,000,000 1k character records

226.12 (minutes)

110.42 (minutes)

104.78%

Pull Replication

1-GB

1,000,000 1k character records

174.87 (minutes)

12.5 (minutes)

1298.96%

Linked Server

10-MB

10,000 1k character records

107.6 (minutes)

113.6 (minutes)

–5.28%

Push Replication

112-MB

100,000 varbinary (max) records

247.07 (minutes)

59.13 (minutes)

317.84%

Pull Replication Records

112-MB

100,000 varbinary (max) records

223.18 (minutes)

1.95 (minutes)

11345.13%

Snapshot Replication

11.3-GB 10,100,000 1k records

Not tested

22.75 (minutes)

Comparison not available

Friday, January 23, 2009 12:59:40 AM (Malay Peninsula Standard Time, UTC+08:00)  #    Disclaimer 
  • Blog reactions

  •  Thursday, January 08, 2009

    A short but note(pun-intended)-worthy blog entry. Microsoft releases SongSmith: Karaoke in reverse.

    openquotes.png Microsoft Research on Thursday is releasing software that gives musicians, both casual and professional, a new way to speed up song development. Called SongSmith, the $29.99 application creates musical accompaniment based on whatever is sung into the computer's microphone.

    In order to do this, the software processes the pitch and tone of what's recorded and lets users hear how it might sound if they had a little backup in the form of a virtual piano, drums, and keyboard. Microsoft is expecting them to use the new track either as inspiration for further song development or as a simple way to create karayoke-quality recordings for friends and family members.

    The software lets users change the feel of a song completely using various sliders that adjust mood, volume levels, tempo and what instruments are being used. Users are also able to purchase additional instruments from Garritan for a small fee that can drastically change the way a track sounds. Each purchased instrument comes wrapped in a special installer that automatically adds it to SongSmith. Dan Morris of Microsoft Research tells me there may eventually be a marketplace for other sample providers, although for now the software is using it exclusively because of its the only compatible format.


    SongSmith lets you simply sing into your computer's microphone to hear what it would sound like if you had a back-up band.

    (Credit: CNET Networks)
    SongSmith is starting out as a digital download only, and will be available from Microsoft's recently launched digital downloads store front. Morris says there are no current plans to make the software part of a larger suite of music oriented products from Microsoft. Competitor Apple has offered a slightly similar feature in its Garageband software that gives you virtual band mates that can accompany you as you record music with an in-line microphone, however each of the instruments must be programmed by the user.

    One interesting thing to note is that the technology is fully capable of providing automated accompaniment in near real-time. Morris says the only hurdle there is that the programming does all its magic by seeing where users are going with a melody and compensating accordingly. Morris also says a Web based version of the software could be possible later on down the line, although development in that area has been slowed down due to latency and recording quality bottlenecks.

    Embedded below are before and after clips of what SongSmith is capable of. As mentioned before, to change the sound of this song users simply need to adjust a slider or two.
    closequotes.png

    Thursday, January 08, 2009 3:27:01 PM (Malay Peninsula Standard Time, UTC+08:00)  #    Disclaimer 
  • Blog reactions

  •  Friday, October 24, 2008

    Ahhh ... it has been a while, hasnt it ?

    My life is just torn between working with bits of 2, beats of 4 and nucleotides of 4. But while challenging, it has been really fun. As spoken to a friend today, my passions in life seeks out to expand the comfort boundaries of gray matter, which we called the mind and to constantly challenge and stimulate the brain to learn and absorb new things that one would never think of learning if one boxed themself in a virtual space, which techies like me would call "typecast".

    One example that I highlighted to my friend today, which I respectfully pointed out to them that he falls under, is when he said: "But we tech people are not good at talking to people and engaging them in meaningful conversations ..."

    Typecast alert !

    I ended up talking with him (not to him) for a good 20 minutes and told him we just had a meaningful conversation and that he could hold one really well. I told him that he himself set up this virtual boundary to box himself in. No one did and that he could easily remove this barrier and elevate himself to do and more importantly, to learn new things and behaviors. Instead of having new curiousities about old things, have new questions, passions and interests towards new things.

    Anyways, I wont be talking about my new-found passions here but I will be briefly touching on a topic that many people knew I have passions for (and I still do) - and that is the innards and the plumbings of software technologies.

    I came across types of this type of questions a lot in emails, forum questions and usergroup events:

    openquotes.png I have this WSDL file that looks something like this:

    <?xml version='1.0' encoding='UTF-8'?>

    <definitions name="someCustomer" targetNamespace="urn:someCustomer" xmlns:typens="urn:someCustomer" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns="http://schemas.xmlsoap.org/wsdl/">
          <message name="add_someCustomer">
                <part name="resId" type="xsd:string"/>
                <part name="cPortable" type="xsd:string"/>
          </message>
          <message name="add_someCustomerResponse">
                <part name="add_someCustomerReturn" type="xsd:string"/>
          </message>
          <portType name="someCustomerPortType">
                <operation name="add_someCustomer">
                      <input message="typens:add_someCustomer"/>
                      <output message="typens:add_someCustomerResponse"/>
                </operation>
          </portType>
          <binding name="someCustomerBinding" type="typens:someCustomerPortType">
                <soap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http"/>
                <operation name="add_someCustomer">
                      <soap:operation soapAction="urn:someCustomerAction"/>
                      <input>
                            <soap:body namespace="urn:someCustomer" use="encoded" encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
                      </input>
                      <output>
                            <soap:body namespace="urn:someCustomer" use="encoded" encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/>
                      </output>
                </operation>
          </binding>
          <service name="someCustomerService">
                <port name="someCustomerPort" binding="typens:someCustomerBinding">
                      <soap:address location="http://foo/bar/someCustomer.php"/>
                </port>
          </service>
    </definitions>

    However, I need to change the add_someCustomerReturn  type from xsd:string to a complex type.

    I’ve made several tests variants around trying to add a complex type, like the following:

          <message name="add_someCustomerResponse">
                <xsd:complexType name="respType" >
                      <xsd:sequence>
                            <xsd:element name="someStatus" type="xsd:boolean" />
                            <xsd:element name="someResult" type="xsd:boolean" />
                      </xsd:sequence>
                </xsd:complexType>
                <part name="add_someCustomerReturn" type="typens:respType"/>
          </message>

    However I always end up having an error like:

    Custom tool error: Unable to import WebService/Schema. Unable to import binding 'customerBinding' from namespace 'urn:customer'. Unable to import operation 'add_customer'. The datatype 'urn:customer:respType' is missing. closequotes.png


    One thing to note is the above "web service" is using: soap:binding style="rpc". While I am not advocating one over another (document/literal), I personally think that if you stripped the religious and philisophical debates, one can certainly build a RPC-style web service using doc/literal encoding.

    The above exceptions funs afoul of what many techies called: Section 5 Encoding

    For the above to be resolved, you need to define a complexType reference by wsdl:part “add_someCustomerReturn” in the schema.
    To do this, you MUST define wsdl:types and add the schema to the WSDL that defines the complex and change the type=”xsd:string” (of the wsdl:part) to the identifying complexType in the schema (encoded in wsdl:types)

    While this is an old article written by Tim, the same principles apply. Do check it out of you need to stimulate your brain: The Argument against SOAP Encoding

    Friday, October 24, 2008 12:58:46 AM (Malay Peninsula Standard Time, UTC+08:00)  #    Disclaimer 
  • Blog reactions

  •  Friday, June 13, 2008

    I had recently purchased a Canon Hi-Def Flash Camcorder HF100 at wholesale price (please dont ask me how much and where I got it from). There was a long thought process before this high-end purchase. I knew I wanted a camcorder to record in Hi-Def (HD) format. The question I had was the recorded video format. I did some research and poking around and there were some pros and cons that I was seriously considering such as:

    1. What is the recording storage medium ?
    2. What is the recording format ? If answer to [1] was a DV Tape or sort, then the answer would probably be HDV/MPEG-2 format.
    3. Do I have enought processing power / software infrastructure to deal with the answer to [2] ?

    In the end, I decided that I would not want to do the route of using a DV Tape. Tape is proven, tested, good, mature and cheap but has its limitations. The fact that it is a sequential access medium puts me off. Even newER backup solutions of today seems to provide disk storage, whose prices have dropped in recent years, as an alternative to tape. Usually, the restoration granularity and the time it takes to restore is the deciding factor for customers to champion disk over tape. Moreover, if I record on tape and then later edit on disk, it does somehow seem that I am going backwards.

    So, instead of carrying bulkIER tapes around with me (and I do a lot of random recording), not forgetting that the housing for these tapes in the camcorders itself does take up some bulk and effectively limits the handling of the camera at hand, I dumped the idea of either the Canon HV20 or HV30. Mind you - their dual recording format in a choice of either Standard-Definition (SD)or HD is really attractive but I doubt that I would want to record in SD in a couple of years down the road where computing power, screens, bandwidth are all commodities.

    That left me with Question [3] above. What does it take to process/edit those videos ? From searches of many forums and reviews, a lot of people buy a AVCHD Camcorder (such as the Canon HF100) without realizing that they dont have the infrastructure to process and edit the recorded HD clips. I guess a lot of peple dont realize that there is not much choice of video-editing software that can process a AVCHD video clip today. So. what most of them did was

    • Pay X Dollar for the camcorder and then 2X Dollar for a brand new Mac - Holy Smokes. Since when does money grow on trees ?
    • Pay a couple of hundred dollars more to buy a decent video editing software such as the Pinnacle Studio or the Sony Vegas

    I didnt like both options. First or all, I edit video clips - Yes - but I dont consider myself to be a "pro-consumer" of sorts that would want to fork out much money just to have 3000 over video transitions up my sleeve ... and ... I am not a MAC fan. Yes, I admit. Crucify me. I am just not genetically engineered to use a MAC or any of Apples' products. Yes, I love my ZUNE and its marketplace very much. Thank you.

    Therefore, I had to look for an intermediate solution since my old, trusted and most FREE Windows Movie Maker and Media Player cannot handle AVCHD video files natively and I am not willing to fork out anything more than SGD100.00

    Luckily, my prayers are answered and my search leads me to media\video developer ShedWorx who has the VoltaicHD for both the PC and the MAC. Bascially, VoltaicHD transforms your AVCHD High-Def video clips to WMV-HD, which both Windows Media Player and Microsoft Movie Maker can handle. FAQ here. This little known shareware (just USD30.00) has gotten some great independent reviews so I went for a trial, downloaded a sample AVCHD .MTS file and it worked like a charm.

    [Note to ShedWorx]: Now if you could make a command-prompt version of your awesome tool, that would be a great addition as it would complete a workflow scenario of an "unattended" conversion process of the captured AVCHD .MTS files to WMV-HD.

    With that, I bought it and went broke but GOSH - what a camcorder !!! Its light, intuitive, great handling and churn out great looking HD video clips. I guess the reviews out there in the wild will do it better justice than me writing about it here.

    Yes, the computing power and storage resources are high. At the best quality mode, the HF100 records at 17Mbps and my usual mode would be to record at a compromised (between storage and battery power) bitrate of 7Mbps. Even with a decent Core2 Duo Processor T7200 2.0 GHZ (highly-rated) Merom chip and 2Gs of RAM that I have, editing a WMV-HD 7Mbps video clip does require some patience. And the file recordings are huge - as a rough gauge - AVCHD are abt 120Mb (15MB) /min of footage and becomes 500Mb (wmv) after decompression !!!

    Luckily, I delegated the conversion of AVCHD -> WMV-HD files to one of my servers, running a Dual Core XEON Pro 5140 2.33GHZ 4MB L2 cache 1333MHz FSB - Woodcrest Chip and this was much faster, comparatively. In any case, this can be done unattended, and this would also give me a good excuse to plug in another same processor on this 2-way box in the near future. .

    All in all, this is a great buy at near-wholesale price and I already had quite a lot of fun doing roving and recording real 1080p high-definition videos and enjoying the processed WMV-HD clips on my wide-screen LCD monitor, my HDTV as well as my Rapsody N35 media center (which plays WMV-HD High-Definition videos).

    Below is a "short" clip I took with the above Canon HF100, with the sarcastic emphasis on "short". I took this 50-second clip in full 1080/9Mbps HD glory. Uncompressed file size is 210MB. In order to "dumb" it down so that it can squeeze and play better over the HTTP ravine, I had to re-encode it to a smaller scale/Mbps at 856x480/3Mbps. Even then, this same 50-second 856x480/3Mbps clip's file size is still at a large 19MB !!! If you blow up the player to your full-screen, you can see that it maintains a clear and good quality at full-screen even at 3Mbps. Mind you, the source look great on my local playback at 1080/9Mbps/25fps.

    Friday, June 13, 2008 4:13:25 AM (Malay Peninsula Standard Time, UTC+08:00)  #    Disclaimer 
  • Blog reactions

  •  Saturday, May 31, 2008

    I was recently pointed to this post that highlights a "successful attempt" by some students in Germany to crack Microsoft Cardspace.After reading through the post several times, I became convinced that it is NOT what it seems it is and that if the "breach" is what it says it is, there must be some pre-conditions that must be satisfied before it can happen and these criteria are not going to be easy...

    Just as I was putting some of my thoughts down that relates to why I think the attempt is somehow "inappropriately glorified":

    1. If an end-user would be stupid enough to put and store his/her passwords, credit card information on his PC
    2. There must be some sort of DNS compromise on the end-user side, which also means successfully hacking into his/her router
    3. There must be some sort of Digital Certificate Store compromise on the end-user side, which also means successfully hacking into his machine with highly-elevated priviledges or saying, the user's machine password has been stolen

    Points [2] and [3] relates to the statements from the attempt and I quote from the above post:

    openquotes.png To reproduce the demonstration, you should change your own DNS settings and install an untrusted certificate closequotes.png

    If I can do both those points sucessfully, to be honest, I already have control over what the user does on his machine, stealing his Infocard is probably of low priority at that point in time.

    Then, the brains behind Cardspace, Kim Cameron, himself, wrote a comprehensive reply, which basically was a detailed answer to my brief thoughts above, to counter the students' attempt and should really put any doubts in anyone's mind to rest.

    [Added 02 June 2008]: In this video on his blog, Kim demonstrates how YOU, the end-user, must FIRST POISON your own machine first before the attack can happen: http://www.identityblog.com/wp-content/images/2008/05/Students/Students.html

    Some comments standout and I quote:

    openquotes.png The demonstrator shows that if you are willing to compromise enough parts of your system using elevated access, you can render your system attackable. This aspect of the students’ attack is not noteworthy.

     closequotes.png

    openquotes.png There is, however, one interesting aspect to their attack.  It doesn’t concern CardSpace, but rather the way intermittent web site behavior can be combined with DNS to confuse the browser.  The student’s paper proposes implementing a stronger “Same Origin Policy” to deal with this (and other) possible attacks.  I wish they had concentrated on this positive contribution rather than making claims that require suspension of disbelief. closequotes.png

    openquotes.png However, the students propose equipping browsers with end user certificates so the browsers would be authenticated, rather than the sites they are visiting.  This represents a significant privacy problem in that a single tracking key would be used at all the sites the user visits.  It also doesn’t solve the problem of knowning whether I am at a “good” site or not.  The problem here is that if duped, I might provide an illegitimate site with information which seriously damages me.

     closequotes.png

    While I know the ignorant media will find some ways to sensationalize this unworthy episode, especially when Microsoft is such a big target, this brings to mind a popular joke which I think can be used as an anology:

    Q: How do you make 1 million dollars ?
    A: Start with 2.

    Saturday, May 31, 2008 8:45:51 AM (Malay Peninsula Standard Time, UTC+08:00)  #    Disclaimer 
  • Blog reactions

  •  Tuesday, April 29, 2008

    Gosh, I think I am in desparate need for some new empty bookshelves ...

       <-- Click this pic to see a higher resolution for even more details.

    ...and you havent even seen my other bookshelves containing my other interest, which I wont share for now ...

    Tuesday, April 29, 2008 11:08:53 AM (Malay Peninsula Standard Time, UTC+08:00)  #    Disclaimer 
  • Blog reactions

  •  Wednesday, April 16, 2008
    Wednesday, April 16, 2008 6:26:49 AM (Malay Peninsula Standard Time, UTC+08:00)  #    Disclaimer 
  • Blog reactions

  •  Tuesday, April 08, 2008

    One cannot run away from understanding infrastructure needs when one is pitching or designing software solutions in the enterprise (which I do a lot of) and it is sometimes strange (in a pleasant way) when the conversation goes like this:

    openquotes.png Please make sure you have failover expertise in your next meeting. I recommend getting Steve to proxy in for William, even though I dont think anyone can impersonate him. At least, I have been able to ascertain that Steven can mirror William quite well and will be able to backup William in the event of a failure closequotes.png

    Tuesday, April 08, 2008 7:22:57 AM (Malay Peninsula Standard Time, UTC+08:00)  #    Disclaimer 
  • Blog reactions