Event Stream Processing & Event Driven Architecture - Microsoft Technology Stack?

Context:

The recent rise in the adoption of service oriented architecture (SOA) continues to alter the landscape of modern IT integration infrastructure and methodology. Technology stovepipes of the past are now being connected by enterprise service bus (ESB) technology. An ESB provides a backbone of networking, communication, mediation, and service container management. With an ESB it is easier, faster, and cheaper to connect applications than was possible in the past.

Almost every integration software vendor provides some form of ESB in their products. ESBs have risen to the status of a de facto standard for service-oriented application integration. But with applications more easily integrated, what is the next step in the evolution of enterprise computing infrastructure?

Event-Driven Architectures (EDA) is being hailed as the next major piece of the integration fabric by leading vendors and analysts. There are few mature products already available in the market which can process event streams from different sources in real-time.


"By 2011, a new generation of application platforms stemming from the convergence of extreme transaction processing (XTP) technologies will supersede Java EE and .NET as the platform of choice for large-scale, business-critical operational applications." , Gartner, July 2007


Real Problem & Solution:

In fact I am currently working to re architect a critical application which need to process (in real-time) around 5000/messages per sec from different railway signalling system in order to depict current state of railway network. This mean we need to process large amount of data, correlate & find patterns on that data then match it with assets (Train, Signals, Tracks etc) so that we can show each train, signal (green or red) on computer screen as it is in live.

The existing system which does all that is written in .Net but it is tumbling as it is custom developed software in a traditional way for this specific application.

The main point here is Event Stream Processing solutions are required in most of the industry:

· Transportation (Railways, Buses & Airports)
· Banking (Fraud Detection, Algorithmic Trading etc.)
· Supply Chain & Manufacturing (RFID processing)

After doing bit of research I found out list of enterprise class commercial product available out of shelf which can provide most of the functionality which I just mentioned above. In fact all big vendors like Progress (Sonic), IBM, Oracle have a CEP/ESP product available in market today. No surprises Microsoft is missing from this list.

Being worked on Microsoft technology from last 8 years I would be interested to see how Microsoft can provide an enterprise class CEP/ESP product in future. Can they leverage BizTalk, WCF etc & package them as a product?

BizTalk Best Practices (Handling Large Projects)

1. Use common .snk (key file) on all development machines so that you can uniquely qualify an assembly across all machines.

2. If you have used any helper utility or external assembly then make a shared network drive and put the build of your assembly (.dll) in that. Other people can use them without making a build on their local machine.

3. Avoid use of parallel shapes wherever possible. Its helps you in fast debugging of orchestrations and also gives best performance (scenario based).

4. If your development is going on multiple machines then use one master development server where you can configure and run full solution. Always use binding file of this master server to deploy orchestrations on test server.

Handling Different Encodings in BizTalk Server 2004

BizTalk Server 2004 supports many different encodings for XML and flat file documents, for example UTF-16, UTF-8, Simplified Chinese GBK, Simplified Chinese GB18030, and so on.

For inbound documents, BizTalk Server can recognize the encoding declaration in XML documents, such as "". The flat file schema has a Code Page property to indicate the encoding of the inbound flat file documents.

For outbound documents, XML and flat file assemblers use the Target charset property. If this property is specified, BizTalk Server converts the outbound documents to the specified character set regardless of the original one. If no Target charset property is set, XML uses the UTF-8 protocol and flat files use the code page specified in the flat file schema.

­Example

Let say we have “£” in our incoming source.

This symbol you can define using ISO-8859-1 encoding.

So make your inbound document like this and as said BizTalk server recognize all standard encodings so the schema that you have created in BizTalk would successfully recognize this xml as valid.

?xml version="1.0" encoding="ISO-8859-1"?
ns0:Root xmlns:ns0="http://MapTrial.Incoming"
Name>transfering £ 500/ns0:Root>