![Page 1: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/1.jpg)
Advanced Microsoft SQL Server 2008 R2 StreamInsightBeysim Sezgin (Architect)Roman Schindlauer (Program Manager)Microsoft Corporation
PDC09-SVR08
![Page 2: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/2.jpg)
Agenda> Use Cases & Challenges> Formulating Declarative Queries> Windows in Time> Implementing Adapters> Architecture> Extensibility> Event Flow Debugging
![Page 3: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/3.jpg)
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
What is CEP?
![Page 4: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/4.jpg)
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
request
response
Event output
streaminput stream
What is CEP?
![Page 5: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/5.jpg)
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
request
response
Event output
streaminput stream
What is CEP?
![Page 6: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/6.jpg)
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or lessData Rate Hundreds of
events/sec> Tens of thousands of events/sec
request
response
Event output
streaminput stream
What is CEP?
![Page 7: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/7.jpg)
Standing Queries
Query Logic
Event sources Event targets
`
Devices, Sensors
Web servers
Event stores & Databases
Stock ticker, news feeds
Event stores & Databases
Pagers &Monitoring devices
KPI Dashboards, SharePoint UI
Trading stations
StreamInsight Application at Runtime
.NETC#
LINQStreamInsight Application Development
InputAdapter
s
OutputAdapter
sStreamInsight
Engine
Query Logic
Query Logic
StreamInsight Platform
12 3
4
![Page 8: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/8.jpg)
![Page 9: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/9.jpg)
Use Case: NBC Sunday Night Football
Telemetry Receiver
StreamInsight
Listener Adapter
GeoTag and group by region
Count total eventsCount session
startsCount active
sessions
SQL Adapter
PerfCounter Adapter
1
2
3
4
![Page 10: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/10.jpg)
Use Case: Data Center
QueryCentral
time seriesarchive
Power Consumption
Process Information
Visualize
Complex Aggregations
/Correlations
Power MeterInput
Adapter
ETWInput
Adapter
Query
Query1 23
![Page 11: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/11.jpg)
ChallengesHow do I …
> detect interesting patterns?> reason about temporal semantics?> correlate data?> aggregate data?> avoid writing custom imperative code?> create a runtime environment for
continuous and event-driven processing?
As a developer, I need a platform!
![Page 12: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/12.jpg)
Query Expressiveness> Selection of events (filter)> Calculations on the payload
(project)> Correlation of streams (join)> Stream partitioning (group and
apply)> Aggregation (sum, count, …) over
event windows> Ranking over event windows
(topK)
![Page 13: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/13.jpg)
Query Expressiveness
var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 };
> Projection
![Page 14: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/14.jpg)
Query Expressiveness
var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 };
> Projection> Filter
![Page 15: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/15.jpg)
Query Expressiveness
var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w };
> Projection> Filter> Correlation (Join)
![Page 16: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/16.jpg)
Query Expressiveness
var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) };
> Projection> Filter> Correlation (Join)> Aggregation over windows
![Page 17: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/17.jpg)
> Projection> Filter> Correlation (Join)> Aggregation over windows> Group and Aggregate
Query Expressiveness
var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) };
![Page 18: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/18.jpg)
Time Windows
Time
Tumbling Window
![Page 19: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/19.jpg)
Time Windows
Time
Hopping Window
![Page 20: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/20.jpg)
demo LINQ Queries
![Page 21: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/21.jpg)
> Import data into StreamInsight platform
> Adapter characteristics:> Pull vs Push> Out-of-order vs. In-Order Events
> Adapter API> Observable model
> wraps adapter framework
Adapters
![Page 22: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/22.jpg)
Adapter Factory
public class MyFactory : ITypedInputAdapterFactory<MyConfig>{ public InputAdapterBase Create<Payload>(MyConfig conf, EventShape ev) { return new MyAdapter(conf); }
public void Dispose() { ... }}
![Page 23: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/23.jpg)
Adapter API
public class MyAdapter : TypedPointInputAdapter<MyPayload>{ public MyAdapter(MyConfig conf) { ... } public override void Start() { ... } public override void Resume() { ... }}
![Page 24: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/24.jpg)
Stream Liveliness
??
Time
![Page 25: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/25.jpg)
.Net Development
User App
Engine
Development Steps
LINQ
StreamInsight IL
Adapter Runtime query
1
Adapter
3
2
4
![Page 26: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/26.jpg)
CompilationParser Algebrizer
Compiler
Type SystemMetadata
Expression Service
Runtime
Stream Manager
EventsStreams
Execution Operators Scheduler
Event Manager
Stream OS Resource Governor
Synch. Primitives
Buffer Manager CLR Hosting
StreamInsight Architecture
![Page 27: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/27.jpg)
Query Compilation> Multiple Syntactic surfaces
over a single algebra, e.g., LINQ, CQL, etc.
> Compile time type checking and type safe code generation for minimal runtime impact.
> Leverage CLR > Code generator, JIT
support > Type System> Tools and Libraries (LINQ
Expressions, IDE, etc.)> Multiple Deployment
Environment with different form factors, e.g. SmartPhone, PDA, PCs.
CompilationParser Algebrizer
Compiler
Type SystemMetadata
Expression Service
![Page 28: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/28.jpg)
Runtime – Events, Streams, Scheduler> JIT code generation for field
references, expression evaluation
> Event manager is implemented as a combination of managed and native code in order to minimize overhead and ensure predictable performance.
Runtime
Stream Manager
EventsStreams
Execution Operators Scheduler
Event Manager
> Cooperative Scheduler (less context switch, better SLA)> Operators are affinitized to a Scheduler. Periodic checks
and migration for load balancing> DataFlow Architecture: Reduced coupling and pipeline
parallelism
![Page 29: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/29.jpg)
Runtime – Execution Operators> Efficient implementation of
operators that perform incremental evaluation as each event is processed.
> GroupAndApply Operator:> Enables parallelism for both
scale-up (multi-core) and scale-out(cluster).
> Groups are dynamically instantiated and torn down based upon the data. Large numbers of groups can be simultaneously active.
Machine2Machine1
~ MAP
ApplyApplyApply
Union X,Y,Z
ABC
Machine1Group A,B,C
BBBAA CC
XX ZZYYY
XYZ
~ REDUCE
![Page 30: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/30.jpg)
Extensibility
> Need for domain-specific extensions> Integrate with functionality available in existing libraries
> User-defined operators, functions, aggregates
> Code written in .NET, deployed as .NET assembly
> Window-based semantics for UDOs, UDAs> Receive a set of events> Produce a value / set of events
![Page 31: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/31.jpg)
Supportability> Diagnostic interface
> Retrieve statistical information at runtime
> Part of the object model API> Manage through Powershell> Debugger tool
> Retrieve live diagnostics> Record and replay queries> Analyze event processing along
operators
![Page 32: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/32.jpg)
Debugger Tool
demo
![Page 33: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/33.jpg)
Conclusion> CEP Platform & API> Event-triggered, fast Computation> API for Adapters, Queries, Applications> Declarative LINQ> Flexible Adapter API> Extensible> Supportability
![Page 34: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/34.jpg)
Please Evaluate!> Sign in to microsoftpdc.com> Follow the evaluation link for the
session> PDC will donate $1 for each person
who completes an evaluation to the local Boys and Girls Club
![Page 35: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/35.jpg)
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
![Page 36: Advanced Microsoft SQL Server 2008 R2 StreamInsight](https://reader036.vdocument.in/reader036/viewer/2022062302/5681650d550346895dd78530/html5/thumbnails/36.jpg)