Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1NAoRFg. Bart de Smet discusses how the reactive programming paradigm fits the requirements of rich querying capabilities for event streams and can be applied from small devices all the way to cloud-scale infrastructures. Filmed at qconnewyork.com. Bart de Smet is a Principal Software Development Engineer working on large scale stream processing systems in the Applications and Services Division at Microsoft, a course and book author. In his current role at Microsoft, he’s overseeing the architecture, design, and implementation of event processing systems that run at massive scale in data centers, as well as on cloud-connected devices.

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/reactive-cloud-scale

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

5. 
 Tesla

 mid 2000s





 LINQ to Events
Nikola Tesla
www.knowledgeoftoday.org

6. 
 Tier-splitting
 metadata







 asynchronous
www.Wikipedia.org

7. 


 Events are not first-class objects

[RunOnClient]
public event EventHandler<MouseEventArgs> MouseMoved;
// Runs in cloud
public void CloudCanvas()
{
MouseMoved += (o, e) => { /* do stuff */ };
}
An electric eel…

8. 
 delegates

Action a = new Action(Foo); // explicit creation of delegate instance
Action b = Foo; // method group conversion
Action c = () => { … }; // creates anonymous method
void Foo() { … }
event Action Bar // metadata that refers to …
{
add { … } // add accessor
remove { … } // remove accessor
}
http://en.wikipedia.org/wiki/First-class_citizen

9. 
 duality enumerator

interface IObservable<out T>
{
IDisposable Subscribe(IObserver<T> observer);
}
interface IObserver<in T>
{
void OnNext(T value);
void OnError(Exception error);
void OnCompleted();
}
Notification grammar
OnNext* (OnError | OnCompleted)?
“Gang of four” book
Addison-Wesley

10. 
 Composition
 Composition
interface IScheduler
{
IDisposable Schedule(Action work);
…
}
static class Observable
{
static IObservable<T> Where<T>(this IObservable<T> source, Func<T, bool> f);
static IObservable<R> Select<T, R>(this IObservable<T> source, Func<T, R> p);
…
}
Function composition
www.Wikipedia.org

11.  Function composition
www.Wikipedia.org
public static class Observable
{
public static IObservable<T> Merge<T>(this IObservable<T> xs, IObservable<T> ys)
{
return Create<T>(observer =>
{
var gate = new object();
return new CompositeDisposable
{
xs.Subscribe(x => { lock (gate) { observer.OnNext(x); } }, …),
ys.Subscribe(y => { lock (gate) { observer.OnNext(y); } }, …),
};
});
}
}
First-class: can build
extension methods
Composition of
resource management

12.  layering


 asynchrony and time


 virtual time historical time
public static IObservable<T> Return<T>(T value, IScheduler scheduler)
{
return Create<T>(obs => scheduler.Schedule(() => { obs.OnNext(value);
obs.OnCompleted(); }));
}
Space-time
www.Wikipedia.org

13. 
 push pull

Category theory
www.Wikipedia.org
interface IObservable<out T>
{
IDisposable Subscribe(IObserver<T> observer);
}
interface IObserver<in T>
{
void OnNext(T value);
void OnError(Exception error);
void OnCompleted();
}
interface IEnumerable<out T>
{
IEnumerator<T> GetEnumerator();
}
interface IEnumerator<out T> : IDisposable
{
bool MoveNext() throws Exception;
T Current { get; }
void Reset();
}

14. 

 Await Stephen Kleene
Kleene star (closure)
www.Wikipedia.org
Func<T> f
var x = wait f();
IEnumerable<T> xs
foreach (var x in xs) {
f(x);
}
Task<T> t
var x = await t;
IObservable<T> xs
xs.Subscribe(x => {
f(x);
});
OneMany
Synchronous Asynchronous

15. 
 Code-as-data


Alan Kay
Homoiconicity, 1969
www.Wikipedia.org
IEnumerable<T> xs
from x in xs where f(x) …
IQueryable<T> xs
from x in xs where f(x) …
IObservable<T> xs
from x in xs where f(x) …
IQbservable<T> xs
from x in xs where f(x) …
CodeData
Pull Push

18.  Events




 scalable abstraction



Satya Nadella
“Cloud-first, mobile-first”
www.businessinsider.com

19. 
 Bing


 Cortana


 Cloud and device



20.  Optimization



 Reliability






Richard Feynman
Westview Press

21. from w in weather
where w.City == “Seattle”
select w
flights
.Where(f => f.Code == “BA49”)
.DelaySubscription(departure – 24 * H)
.TakeUntil(arrival + 24 * H)
.Select(f => f.Status)
userLocation
.Sample(1 * M)
.DistinctUntilChanged()
.Select(here =>
traffic(here, meetingLocation)
.TakeUntil(meeting – minimumTime)
.Select(t => Timer(meeting – t.EstimatedTime))
.StartWith(Timer(meeting – estimatedTime))
.Switch())
.Switch()
.Select(_ => “Leave now for ” + subject)
.DelaySubscription(meeting – estimatedTime)
Temporal query
operators
Device-side
event stream
// Insert cloud-side observable sequence
// of time-to-leave timer here
Cloud-side
event stream
Higher-order
query operators
Remember
tier-splitting?

22.  IReactiveProcessing



Bing cloud
IRP
Partner1 cloud
IRP
Partner2 cloud
IRP
Tablet
IRP
Phone
IRP
Node
IRP
Node
IRP Node
IRP

23.  Rx



 distributed







25. 
 Proxies
 Definitions
 Metadata


 Hyper-cube





Sync Async
IntrinsicExtrinsic
Higher dimensions
Calabi-Yau manifold
www.Wikipedia.org

26. 
 Proxies
 Definitions
 Metadata


 Hyper-cube
 Synchronous versus asynchronous
 Extrinsic versus intrinsic identities
 Code-as-data versus code
 Reliable versus non-reliable
 Etc.
Higher dimensions
Calabi-Yau manifold
www.Wikipedia.org
Async
Extrinsic
E.g. this is the
client-side API
Need string theory


27.  context




 proxies

var conn = new ReactiveServiceConnection(endpoint);
var ctx = new ClientContext(conn);
var traffic = ctx.GetObservable<TrafficInfo>(trafficUri);
var http = ctx.GetObserver<HttpData>(httpUri);
Explicit identifiers
provided for artifacts

28. 

 extrinsic identifiers
 Async


var traffic = ctx.GetObservable<TrafficInfo>(trafficUri);
var http = ctx.GetObserver<HttpData>(httpUri);
var subscription = await traffic.Where(t => t.Road == “I-90”)
.Select(t => new HttpData { Uri = myService,
Body = t.ToString() })
.SubscribeAsync(http, mySubUri);
Explicit identifiers
provided for artifacts

29.  SubscribeAsync
 expression tree



 serialized



 IRP

30. var traffic = ctx.GetObservable<TrafficInfo>(trafficUri);
var http = ctx.GetObserver<HttpData>(httpUri);
var subscription = await traffic.Where(t => t.Road == “I-90”)
.Select(t => new HttpData { Uri = …, Body = … })
.SubscribeAsync(http, mySubUri);
Invoke
rx://subscribe Invoke
rx://select Invoke
rx://where trafficUri
httpUri
λ
t => new { bing://http/uri = …,
bing://http/body = … }
λ t => t.Road == “I-90”
Unbound parameters
processed by binder
Structural typing to
reduce coupling

31.  definition of artifacts



var traffic = ctx.GetObservable<TrafficInfo>(trafficUri);
// Define
await ctx.DefineObservableAsync<string, TrafficInfo>(
trafficByRoadUri,
road => traffic.Where(t => t.Road == road));
// Proxies
var trafficByRoad = ctx.GetObservable<string, TrafficInfo>(trafficByRoadUri);
trafficByRoad(“I-90”).…

32. 
 parameterized observables



var xs = ctx.GetObservable<int>(xsUri);
var iv = ctx.GetObserver<int>(ivUri);
await xs.Where(x => x > 0).Select(x => x * x).SubscribeAsync(iv, …);
var xs = ctx.GetObservable<int>(xsUri);
var iv = ctx.GetObserver<int>(ivUri);
var where = ctx.GetObservable<IARQ<int>, Expr<Func<int, bool>>, int>(whereUri);
var select = ctx.GetObservable<IARQ<int>, Expr<Func<int, int>>, int>(selectUri);
await select(where(xs, x => x > 0), x => x * x).SubscribeAsync(iv, …);

33.  [KnownResource] attributes can be used everywhere
 Provide shorthand syntax for Get* operations
 Enable code generation using the Metadata facilities
 What’s IAsyncReactiveQbservable<T>?
 Async because it’s client-side (cf. SubscribeAsync)
 Reactive because we had to disambiguate with Rx concepts
 Qbservable because it’s expression tree based
static class AsyncReactiveQbservable
{
[KnownResource(whereUri)]
static IAsyncReactiveQbservable<T> Where<T>(this IAsyncReactiveQbservable<T> xs,
Expression<Func<T, bool>> filter) {…}
}

34.  properties


 tooling


 delegation


37. 


 recovery
 density
 reliability
 Scalable

 checkpointing
 Acknowledge / replay



38.  Coordinator engines

Event Processing
Execution Engine Node1
Rx
IRP
Event Processing
Execution Engine Node2
Rx
IRP
Checkpoint
storage
Save
Load
Checkpoint
storage
Save
Load
Reliable
messaging
OnNext
Replay
Ack

39.  traverse the operator tree






 ISubscribable<T>

interface ISubscribable<out T>
{
ISubscription Subscribe(ISubscriber<T> subscriber);
}
interface ISubscription : IDisposable
{
void Accept(ISubscriptionVisitor v);
}

40. 



 Physical schedulers


 logical schedulers

 Unit of pausing



42.  onion




 GitHub




 bartde@microsoft.com
Service
Host
Data model
Query engine
Operators

43. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/reactive-
cloud-scale