curing your event processing blues with reactive extensions (rx) bart j.f. de smet senior software...

Curing Your Event Processing Blues with Reactive Extensions (Rx)

Bart J.F. De SmetSenior Software Development EngineerMicrosoft Corporation

The Event Processing Landscape

Socialmedia

Stock tickers

RSS feeds

GPS

Server managementUI eve

nts

Reactive Extensions Architecture

Concurrency

IScheduler TimeThreads Cloud Dispatchers

Event Streams

IObservable<T>

IObserver<T>ISubject<T>

LINQ to Events

from quote in stockwhere quote.Symbol == “MSFT”select quote.ValueProjectionFi

lteri

ng

AggregatingG

rou

pin

g

Join

s

WindowingSh

arin

g

SamplingTh

rottlin

g

Timeout

Merg

ing

Recovery


Concurrency


LINQ to Events


lteri

ng

AggregatingG

rou

pin

g

Join

s

WindowingSh

arin

g

SamplingTh

rottlin

g

Timeout

Merg

ing

Recovery

Event Streams

Towards a unified programming modelProducers are observable sequences

.NET events, WinRT events, sensor APIs, APM methods, tasks, etc.

Consumers are observersHooking up “continuations” or handlers

Observable

Subscribe

Observer

Essential Interfacesnamespace System{ public interface IObservable<out T> { IDisposable Subscribe(IObserver<T> observer); }

public interface IObserver<in T> { void OnNext(T value); void OnError(Exception error); void OnCompleted(); }}

Notification Grammar

OnNext(42)

source1

OnNext(43)OnComplete

d

OnNext(“Hello”)

source2

OnError(error)

OnNext* (OnError | OnCompleted)?

Limitations of .NET Events

exchange.StockTick += (sender, args) =>{ if (args.Quote.Symbol == “MSFT”) { // Imperative code }};

exchange.StockTick -= /* what goes here? */;

Can’t pass around Hidden data source

Hard resource maintenance

Lack of composition

Observable Sequences to the Rescue

IObservable<Quote> stockQuotes = …;

var msft = stockQuotes

.Where(quote => quote.Symbol == “MSFT”);

var subscription = msft.Subscribe(quote => /* … */);

subscription.Dispose();

Objects can be passed

Source of Quotes

Easy resource maintenance

Can define query operators

Are .NET Events Obsolete?

.NET Events

Code centric

Design-time experience

Not first class

Non-compositional

Lightweight

Rigid execution model (IL)

Observables

Data centric

No design-time experience

First class objects

Rich composition

Slightly more cost

Translatable with expression trees

demo

Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team

Events versus Observables


Concurrency


Event Streams

IObservable<T>


Dictionary Suggest

ReactDictionary web

service

Asynchronous request

ReactionReactiveReactor

IObservable<string>

IObservable<DictionaryWord[]>Data binding

on UI thread

“React”

Converting Events and Asynchronous Methods// Convert the TextChanged event to IObservable<string>var input = (from evt in Observable.FromEventPattern(txt, “TextChanged”) select ((TextBox)evt.Sender).Text) .Throttle(TimeSpan.FromSeconds(0.5)) .DistinctUntilChanged();

// Convert asynchronous proxy methods to Func<string, IObservable<string[]>>var lookup = Observable.FromAsyncPattern<string, string[]>(svc.BeginLookup, svc.EndLookup);

// Compose both sources using a queryvar res = from term in input from words in lookup(term).TakeUntil(input) select words;

No longer needed in Rx v2.0 for .NET 4.5, using

Task<T>

demo


Querying Event Streams

Stock Trade AnalysisMSFT27.01

ticks

INTC21.75

MSFT27.96

MSFT31.21

INTC22.54

INTC20.98

MSFT30.73

from tick in ticks


ticks

INTC21.75

MSFT27.96

MSFT31.21

INTC22.54

INTC20.98

MSFT30.73

27.01

27.96

31.21

30.73

MSFT

21.75

22.54

20.98

INTC

from tick in ticks

group tick by tick.Symbol


ticks

INTC21.75

MSFT27.96

MSFT31.21

INTC22.54

INTC20.98

MSFT30.73

MSFT

INTC

from tick in ticks

group tick by tick.Symbol into companyfrom openClose in company.Buffer(2, 1)

[27.01, 27.96]

[27.96, 31.21]

[31.21, 30.73]

[21.75, 22.54]

[22.54, 20.98]


ticks

INTC21.75

MSFT27.96

MSFT31.21

INTC22.54

INTC20.98

MSFT30.73

MSFT

INTC

from tick in ticksgroup tick by tick.Symbol into companyfrom openClose in company.Buffer(2, 1)

let diff = (openClose[1] – openClose[0]) / openClose[0]

0.034 0.104-

0.015

0.036-

0.069


ticks

INTC21.75

MSFT27.96

MSFT31.21

INTC22.54

INTC20.98

MSFT30.73

MSFT

INTC

from tick in ticksgroup tick by tick.Symbol into companyfrom openClose in company.Buffer(2, 1)let diff = (openClose[1] – openClose[0]) / openClose[0]

where diff > 0.1

0.034 0.104-

0.015

0.036-

0.069


ticks

INTC21.75

MSFT27.96

MSFT31.21

INTC22.54

INTC20.98

MSFT30.73

from tick in ticksgroup tick by tick.Symbol into companyfrom openClose in company.Buffer(2, 1)let diff = (openClose[1] – openClose[0]) / openClose[0]where diff > 0.1

select new { Company = company.Key, Increase = diff }

res

Company = MSFT

Increase = 0.104

demo


Complex Event Processing


Event Streams

IObservable<T>


LINQ to Events


lteri

ng

AggregatingG

rou

pin

g

Join

s

WindowingSh

arin

g

SamplingTh

rottlin

g

Timeout

Merg

ing

Recovery

The Role of SchedulersParameterize Concurrency

//// Runs a timer on the default scheduler//IObservable<long> Timer(TimeSpan dueTime);

//// Every operator that introduces concurrency// has an overload with an IScheduler//IObservable<long> Timer(TimeSpan dueTime, IScheduler scheduler);

The Role of SchedulersSynchronization

var xs = Observable.Return(42, Scheduler.ThreadPool);xs.Subscribe(x => lbl.Text = "Answer = " + x);

xs.ObserveOn(new ControlScheduler(frm)) .Subscribe(x => lbl.Text = "Answer = " + x);xs.ObserveOn( frm ) .Subscribe(x => lbl.Text = "Answer = " + x);

The IScheduler Interfacepublic interface IScheduler{ DateTimeOffset Now { get; }

IDisposable Schedule<TState>( TState state, Func<IScheduler, TState, IDisposable> action);

IDisposable Schedule<TState>( TimeSpan dueTime, TState state, Func<IScheduler, TState, IDisposable> action);

IDisposable Schedule<TState>( DateTimeOffset dueTime, TState state, Func<IScheduler, TState, IDisposable> action);}

Operational Layering of Rxpublic static IObservable<T> Return<T>(T value, IScheduler scheduler){ return Observable.Create<T>(observer => { // Serialize state to scheduler; return ability to cancel return scheduler.Schedule(new { value, observer }, (_, x) => { x.observer.OnNext(x.value); x.observer.OnCompleted(); return Disposable.Empty; // No recursive work }); });}

Virtualizing Time for Testingvar scheduler = new TestScheduler();

var input = scheduler.CreateHotObservable( OnNext(300, “Bart De Smet”), OnNext(400, “Erik Meijer”), OnCompleted<string>(500));

var results = scheduler.Start(() => from name in input select name.Length);

results.Messages.AssertEqual( OnNext(300, 12), OnNext(400, 11), OnCompleted<int>(500));

demo


Testing in Virtual Time

Expression Tree Representation of Queries

Runtime translation of reactive queriesLike IQueryable<T> for classic LINQ

Examples:LINQ to WMI Events

Translates LINQ queries into WQL statements

LINQ to PowerShellTranslates LINQ queries into PowerShell pipelines

LINQ to TwitterTranslates LINQ queries into Twitter queries

Etc.

The IQbservable<T> Interface

public interface IQbservable<out T> : IObservable<T>{ Type ElementType { get; } Expression Expression { get; } IQbservableProvider Provider { get; }}

public interface IQbservableProvider{ IQbservable<T> CreateQuery<T>(Expression expression);}

The Query Processing Landscape

Pull-based Push-based

Local execu

tion

(in

-pro

cess I

L)

Qu

ery

lan

gu

ag

etr

an

sla

tion

IEnumerable<T> IObservable<T>

IQueryable<T> IQbservable<T>

Iterators (yield)LINQ to Objects

Observable.Create

LINQ to Events (Rx)

Expression trees

LINQ to SQL (T-SQL)

Expression trees

LINQ to WMI (WQL)

LINQ

demo


LINQ to WMI Events

announcing

Now with support for hosting Rx v1.0 queries using IQbservable<T>

StreamInsight v2.1

The Future of RxSupport for .NET Framework 4.5

Synergy with asynchronous programming features

Prefer using Task<T> for single-value asynchronyDeprecation of FromAsyncPatternConvert to IObservable<T> for more power

Use IObservable<T> for event streamsAwait support returns last value

Reduce blocking operationsDeprecation of First, Last, Single, ForEach

Leveraging new platform capabilitiesE.g. ExceptionDispatchInfo

The Asynchronous Programming Landscape

Synchronous Asynchronous

Sin

gle

valu

e (

1)

Mu

ltip

le v

alu

es (

*)

Func<T> Task<T>

IEnumerable<T> IObservable<T>

var y = f(x);var z = g(y);

var y = await fAsync(x);var z = await gAsync(y);

var res = from p in products where p.Name == “Rx” select p.Price;

foreach (var x in res) …

var res = from s in stocks where s.Symbol == “MSFT” select q.Quote

res.Subscribe(x => …

Async without Rx…

async Task<string> GetHtmlAsync(Uri url){ var client = new WebClient();

var download = client.DownloadStringAsync(url); var timeout = Task.Delay(TimeSpan.FromSeconds(30)); if (await Task.WhenAny(download, timeout) == timeout) throw new TimeoutException();

var html = await download; return html;}

Imperative glue~

WaitForMultipleObjectsEx

Async with Rx… Better together!

async Task<string> GetHtmlAsync(Uri url){ var client = new WebClient();

var download = client.DownloadStringAsync(url) .ToObservable() .Timeout(TimeSpan.FromSeconds(30));

var html = await download; return html;}

Compositionof query operators

Await supportfor IObservable<T>

The Future of RxInteroperability with WinRT

Use WinRT-style events with FromEventPatternDifferent method signatures in IL

New IScheduler implementationsFor WinRT ThreadPool and CoreDispatcherObserveOnDispatcher and SubscribeOnDispatcher support

Conversions of IAsync* typesSmooth interop with WinRT asynchronous operationsSupport for progress tracking using IObservable<T>

The Future of RxTowards Portable Library Support

System.Reactive.InterfacesStable interfaces for forward compatibility

System.Reactive.CoreBase classes, core schedulers, extensions methods

System.Reactive.LinqQuery operators

Syste

m.R

eactiv

e.

Win

dow

s.Thre

adi

ng

System.Reactive.Providers

Expression tree support

Syste

m.R

eactiv

e.

Win

dow

s.Form

s

Syste

m.R

eactiv

e.

Win

dow

sRuntim

e

Syste

m.R

eactiv

e.

Rem

otin

g

Syste

m.R

eactiv

e.P

latfo

rmS

erv

ices

Pla

tform

Enlig

hte

nm

ents

The Future of RxPerformance

Reducing locking in subjects [3x]Faster producers [100-1000x]

New ISchedulerLongRunning interfacePipeline throughput [4x]

One call frame per operatorReduced wrapping of observersLess operators defined in terms of others

Eliminating redundant locksEradicating time skew due to timing issuesPlay nicer with GC

demo


Preview of Rx v2.0

Summary

Tame your event streams using Rx and LINQ!

Download Rx today!Through http://www.microsoft.com/download (search for Rx SDK)Using NuGet @ www.nuget.org (search for Rx-Main)Now available: Rx v2.0 RC

Watch videos at http://channel9.msdn.com/tags/Rx

Visit our forums on MSDN

http://www.microsoft.com/download

http://www.nuget.org/

http://channel9.msdn.com/tags/Rx

Related Content

DEV414 – LINQ, Take Two – Realizing the LINQ to Everything Dream

Find Me Later This Week In The Ask The Experts Area

Resources

Connect. Share. Discuss.

http://europe.msteched.com

Learning

Microsoft Certification & Training Resources

www.microsoft.com/learning

TechNet

Resources for IT Professionals

http://microsoft.com/technet

Resources for Developers

http://microsoft.com/msdn

http://europe.msteched.com/

http://europe.msteched.com/

http://www.microsoft.com/learning

http://microsoft.com/technet

http://microsoft.com/msdn

Evaluations

http://europe.msteched.com/sessions

Submit your evals online

http://europe.msteched.com/sessions

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS

PRESENTATION.

curing your event processing blues with reactive extensions (rx) bart j.f. de smet senior software...

Documents