![Page 1: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/1.jpg)
Chaos Engineering at Jet.com
Rachel Reese | @rachelreese | rachelree.seJet Technology | @JetTechnology | tech.jet.com
![Page 2: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/2.jpg)
Why do you need chaos testing?
![Page 3: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/3.jpg)
The world is naturally chaotic
![Page 4: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/4.jpg)
But do we need more testing?
Unit Sanity Random Continuous
UsabilityA/BLocalizationAcceptance
Regression Performance Integration Security
![Page 5: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/5.jpg)
You’ve already tested all your components in multiple ways.
![Page 6: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/6.jpg)
![Page 7: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/7.jpg)
It’s super important to test the interactions in your environment
![Page 8: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/8.jpg)
Jet? Jet who?
![Page 9: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/9.jpg)
We plan to be the new Amazon.com
Launched July 22, 2015• Both Apple & Android named
our app as one of their top 5 for 2015
• Over 20k orders per day• Over 10.5 million SKUs• #4 marketplace worldwide• 700 microservices
We’re hiring!http://jet.com/about-us/working-at-jet
![Page 10: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/10.jpg)
Azure
Web sites
Cloud services VMs Service
bus queues
Services bus topics
Blob storage
Table storage Queues Hadoop DNS Active
directory SQL Azure RF# Pake
t FSharp.Data Chessie Unquote SQLProvider Python
Deedle FAKE FSharp.Async React Node Angular SAS
Storm
Elastic Search Xamarin Microservices Consul Kafka PDW
Splunk Redis SQL Puppet Jenkins Apache Hive
Apache Tez
![Page 11: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/11.jpg)
What is chaos engineering?
![Page 12: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/12.jpg)
It’s just wreaking havoc with your code for fun, right?
![Page 13: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/13.jpg)
![Page 14: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/14.jpg)
Chaos Engineering is…
Controlled experiments on a distributed system that help you build confidence in the system’s ability to tolerate the inevitable failures.
![Page 15: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/15.jpg)
![Page 16: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/16.jpg)
Principles of Chaos Engineering
1. Define “normal”2. Assume ”normal” will continue in both a control
group and an experimental group. 3. Introduce chaos: servers that crash, hard drives
that malfunction, network connections that are severed, etc.
4. Look for a difference in behavior between the control group and the experimental group.
![Page 17: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/17.jpg)
Going farther
Build a Hypothesis around Normal BehaviorVary Real-world EventsRun Experiments in ProductionAutomate Experiments to Run Continuously
From http://principlesofchaos.org/
![Page 18: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/18.jpg)
Benefits of chaos engineering
![Page 19: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/19.jpg)
Benefits of chaos engineeringYou're awake
Design for failure
Healthy systems Self service
![Page 20: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/20.jpg)
Current examples of chaos engineering
![Page 21: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/21.jpg)
Maybe you meant Netflix’s Chaos Monkey?
![Page 22: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/22.jpg)
How is Jet different?
![Page 23: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/23.jpg)
We’re not testing in prod (yet).
![Page 24: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/24.jpg)
SQL restarts & geo-replication
Start- Checks the source db for write access- Renames db on destination server (to create a new one)- Creates a geo-replication in the destination region
Stop- Shuts down cloud services writing to source db- Sets source db as read-only- Ends continuous copy- Allows writes to secondary db
![Page 25: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/25.jpg)
Azure & F#
![Page 26: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/26.jpg)
Why F#?
![Page 27: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/27.jpg)
![Page 28: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/28.jpg)
I have now delivered three business critical projects written in F#. I am still waiting for the first bug to come in.
Simon CousinsUK Power company
““ “
![Page 29: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/29.jpg)
Concise and powerful code
public abstract class Transport{ }
public abstract class Car : Transport { public string Make { get; private set; } public string Model { get; private set; } public Car (string make, string model) {
this.Make = make; this.Model = model;
} } public abstract class Bus : Transport {
public int Route { get; private set; } public Bus (int route) {
this.Route = route; }
}
public class Bicycle: Transport { public Bicycle() {}
}
type Transport = | Car of Make:string * Model:string | Bus of Route:int | Bicycle
C# F#
Same info as C#!
Trivial to pattern match on!
![Page 30: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/30.jpg)
F# p
atte
rn
mat
chin
g
C#
![Page 31: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/31.jpg)
Concise and powerful code
public abstract class Transport{ }
public abstract class Car : Transport { public string Make { get; private set; } public string Model { get; private set; } public Car (string make, string model) {
this.Make = make; this.Model = model;
} } public abstract class Bus : Transport {
public int Route { get; private set; } public Bus (int route) {
this.Route = route; }
}
public class Bicycle: Transport { public Bicycle() {}
}
type Transport = | Car of Make:string * Model:string | Bus of Route:int | Bicycle | Train of Line:int
let getThereVia (transport:Transport) = match transport with | Car (make,model) -> ... | Bus route -> ... | Bicycle -> ...
Warning FS0025: Incomplete pattern matches on this expression. For example, the value ’Train' may indicate a case not covered by the pattern(s)
C# F#
![Page 32: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/32.jpg)
Units of Measure
![Page 33: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/33.jpg)
Expanded feature set: Type providers
920 pages!
31 lines!
![Page 34: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/34.jpg)
TickSpec – an F# project
Thanks to Scott Wlaschin for his post, Cycles and modularity in the wild
![Page 35: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/35.jpg)
SpecFlow– a comparable C# project
Thanks to Scott Wlaschin for his post, Cycles and modularity in the wild
![Page 36: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/36.jpg)
Entity Framework (C#)
Thanks to Scott Wlaschin for his post, Cycles and modularity in the wild
![Page 37: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/37.jpg)
Chaos code!
![Page 38: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/38.jpg)
Our code!
let selectRandomInstance compute hostedService = async { try let! details = getHostedServiceDetails compute hostedService.ServiceName let deployment = getProductionDeployment details
let instance = deployment.RoleInstances |> Seq.toArray |> randomPick
return details.ServiceName, deployment.Name, instance with e -> log.error "Failed selecting random instance\n%A" e reraise e}
![Page 39: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/39.jpg)
Our code!
let restartRandomInstance compute hostedService = async { try let! serviceName, deploymentId, roleInstance = selectRandomInstance compute hostedService match roleInstance.PowerState with | RoleInstancePowerState.Stopped -> log.info "Service=%s Instance=%s is stopped...ignoring...” serviceName roleInstance.InstanceName
| _ -> do! restartInstance compute serviceName deploymentId roleInstance.InstanceName with e -> log.error "%s" e.Message}
![Page 40: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/40.jpg)
Our code!
compute|> getHostedServices|> Seq.filter ignoreList|> knuthShuffle|> Seq.distinctBy (fun a -> a.ServiceName) |> Seq.map (fun hostedService -> async { try return! restartRandomInstance compute hostedService with e -> log.warn "failed: service=%s . %A" hostedService.ServiceName e return () })|> Async.ParallelIgnore 1 |> Async.RunSynchronously
![Page 41: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/41.jpg)
Has it helped?
![Page 42: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/42.jpg)
Elasticsearch restart
![Page 43: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/43.jpg)
Additional chaos finds
- Redis- Checkpointing
![Page 44: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/44.jpg)
![Page 45: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/45.jpg)
If availability matters, you should be testing for it.
![Page 46: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/46.jpg)
Azure + F# + Chaos = <3
![Page 47: Chaos Testing with F# and Azure by Rachel Reese at Codemotion Dubai](https://reader036.vdocument.in/reader036/viewer/2022070510/58a9f3731a28ab36018b68ff/html5/thumbnails/47.jpg)
Chaos Engineering at Jet.com
Rachel Reese | @rachelreese | rachelree.seJet Technology | @JetTechnology | tech.jet.com