It is amazing to note the reach to the masses of a single MSDN Webcast. I had conducted a MSDN Webcast titled: "Acks and NAcks: Why We Need the Principles of TCP/IP Reliability in SOAP" in mid-2006 and till this day, I am still receiving questions and feedback about that topic I presented in the webcast. (Incidentally, you can view the on-demand webcast here.)
For those of you who do not want to go through this webcast again, I am going to break that presentation into the WHYs, WHATs and HOWs on this blog. This will make for easy-reading and straight-to-the-point as well.
WHY ?
This is rather straightforward. Why is it important? Why does it matter whether messages can be sent and received reliably? Web services is based on XML messages being sent and received. The most useful Web services will often be those that are the most complex and that rely on many messages being sent and received in a very specific order. That's the way that complex transactions can be built, after all.
Let us look at this problem in context before figuring out where the pain-points are. Most complex transactions in enterprises require far more messages to be sent and received. The messages will be much more intricate and may have very complex dependencies on one another, so that certain processes can't be kicked off until certain previous messages have been sent and acknowledged. Entire complex transactions can fail because a simple message didn't get through. And while it may sound simple to guarantee reliability, it's much more difficult than you might expect.
To add to the complexity, the main characteristics of Web services is communication over unreliable communication channels such as the Internet employing unreliable data transfer protocols such as HTTP, SMTP and FTP. These are considered unreliable, as they do not offer the reliable messaging services such as guaranteed delivery. Accordingly reliable messaging becomes one of the first problems that need to be addressed for Web services to become truly enterprise-capable technologies.
...there is nothing in SOAP or even HTTP that guarantees anyone that a message is delivered, or that allows a someone to tell the sender he/she has received the message...
Making reliable messaging more difficult is that messages are sent not just over the notoriously unreliable Internet, but also between partners that use entirely different networking infrastructures.
What many companies did in the past was to rely on proprietary messaging infrastructures to connect applications with one another. In these days of "open-standards", the industry would much prefer what Web services promises -- a messaging paradigm NOT based on any proprietary implementations."
To put it simply - If that reliability can't be guaranteed, Web services simply won't be used in the enterprise. The benefits of Web Services cannot be realized if it is not being used for complex scenarios in the enterprise.
Let us look at some of the inherent problems in which cross-boundaries communication happen today, especially over the cloud:
- Re-ordering of Messages in Multi-Path Routing Scenario
This is one problem that not many people realize because not many people would know the entire infrastructure physical topology of a end-to-end communication network channel. If you have 2 or more intermediaries that act as routers over a load-balanced network for example, messages being sent out may be routed differently over the network. In other words - The first Message ONE may be routed via Router 2 while the second Message TWO routed over Router 1. The complexity arises when the routers have their own routing table and paths and due to congestion of Router 2 over Router 1, the second Message TWO may arrive first, faster than Message ONE, at the final destination.
- Intermediary Reliability
What is not so self-explanatory is this - What happens if/when the intermediary drops my message ? How do I know if the other party has received it ? If I should send the same message again (just to be sure), how do I know if the serivce is designed to be "idempotent" ? This point has very explicit relations with the Quality-of-Service (QoS) assertions of a service. Each message sent must be received exactly once (once and only once). Failure to deliver a message be made known to both the sender and receiver.
- Connection Management
- “… my connection with the service drops frequently within this Wi-Fi network, but I don’t want to lose messages or the underlying session …”
- “Bursty Connections - I may go idle for long periods of time and I don’t want to hold transport connections open …”
- “… my service is using resources for each session and I want to release those resources if the client is not there anymore …”
- Need for Transport Independence
As I have mentioned earlier, different partners have different networking infrastructures and the need to rely on "open-standards" means that there is a need to decouple that reliablity element away from the transport. This ensures that different parties can choose to use whichever transport that suits them best but yet have a common accepted understanding on the reliability protocol that is independent of the chosen mode of transport.
To sum it up simply, we live in the real world. One that is not so perfect. The one where
- Servers fail
- Systems get out of synch
- Messages get lost
- Messages get re-ordered
- Messages cannot be safely re-tried
and the same one that explains why we need reliable messaging in Web Services.
NEXT UP: WS-ReliableMessaging Simplified - WHAT and HOW ?