StreamInsight

StreamInsight is a platform developed by Microsoft which allows developers to create and deploy complex event processing (CEP) applications. This platform is based on the existing .NET Microsoft platform and it enables developers to implement robust and highly efficient CEP applications. There are a lot of possible event sources, some of the most relevant are:

• Financial trading applications
• Web analytics
• Manufacturing applications
• Server monitoring applications

One can use this platform to easily create tools to monitor data from multiple sources for meaningful patterns, trends, exceptions and opportunities. Analyzing and correlation can be done incrementally while data is produced (in real time) without storing it first, which translates in to having a low latency application. As a source of events historical data can also be used.

Key Benefits

In the following I will try to talk about the most important features and advantages offered by this platform.

Highly optimized performance and data throughput

StreamInsight supports highly parallel execution of continuous queries over high-speed data because it implements a lightweight streaming architecture. The use of in-memory cache and result computation done incrementally provide an excellent performance with high data throughout and low latency. In StreamInsight all processing is automatically triggered by incoming events based on defined queries. Also the platform provides the functionality for handling out-of-order events and in addition static reference or historical data can be accessed and included in the low-latency analysis.

.NET development environment

Microsoft created the .NET development environment in which programming languages like C#, tools like Visual Studio and services like SQL Server can be easily integrated and used for applications development while still keeping the loose coupling between them. StreamInsight is included in this environment in which one can easily develop fast and robust applications. Developers can write their CEP applications using C#, leveraging the advanced language platform LINQ (Language Integrated Query) to create queries.

Given the fact that there is a large community of the developers already familiar with these Microsoft technologies the cost and time of the development of a CEP application is significantly reduced.

Flexible deployment capability

StreamInsight platform provides two ways of deployment scenarios. First is a fully integrated into the developed application as a hosted (embedded) DLL. The second way is deploying StreamInsight as a stand-alone server with multiple applications and users sharing the server. This means that one can develop multiple, independent, applications which use the same StreamInsight instance. The CEP server runs in a wrapper such as an executable or the server could be packaged as a Windows Service.

Extensibility

StreamInsight allows developers to extend its functionality by giving them the possibility to define their own operators, functions and aggregates to be used in queries and define specific event types against which to run the defined queries.

One of the great things about StreamInsight is that it was designed to seamlessly integrate with any domain specific business logic. This means that the platform does not come with any implemented functionality for specific business sectors but it allows developers to plugin any specific business logic.

CEP Query Visualization and Analysis

Microsoft StreamInsight provides a stand-alone Event Flow Debugger which is a powerful GUI tool that enables visual inspection of a continuous query. One can use this graphical tool to quickly inspect the query tree, replay data processing and perform analysis.

Latest version

Currently the latest version of StreamInsight is 2.3, this was released together with SQL Server 2014 on the first of April 2014. Release 2.3 contains only a licensing update, so any code written against the previous version, 2.1, will still work.


In the future posts about StreamInsight I will present some of its most important components.

Robert Rusu