microservices in clojure
TRANSCRIPT
Lucas Cavalcanti@lucascs
Microservices in Clojure
ContextMicroservices
~80 Clojure services ~60 engineers
~10 teams 3.5 years old
OOP Objects, the mainstream abstraction
Image @ http://www.eduardopires.net.br/2015/01/solid-teoria-e-pratica/
What about Functional Programming?
SÃO PAULO, BRASIL
TABLE OF CONTENTS
Immutability Components Pure Functions Schemas Ports and Adapters
SÃO PAULO, BRASIL
Immutability
SOUTHEAST BRAZIL REGION FROM SPACE
Immutability Definition
“If I’m given a value, it’s guaranteed that it won’t ever change”
Technology choices Immutability
Clojure Immutability
All default data structures are immutable: -Maps, Lists, Sets -Records
Mutability is explicit: atoms/refs @, dynamic vars *…*
Datomic Immutability
Datomic stores the changes/transactions, not just the data -append only -db as a value -everything is data (transaction, schema, entities)
Kafka Immutability
Persistent Queues/Topics -each consumer has its offset -ability to replay messages
AWS + Docker Immutability
Ability to spin machines with a given image/configuration -Each build generates a docker image -Each deploy spins a new machine with the new
version -As soon as the new version is healthy, old version is
killed. (blue-green deployment)
Components
SOUTHEAST BRAZIL REGION FROM SPACE
Components https://github.com/stuartsierra/component
(defprotocol Database (query [this query-str]))(defrecord SomeDatabase [config-1 config-2 other-components] component/Lifecycle (start [this] (assoc this :connection (connect! config-1 config-2 other-components))) (stop [this] (release! (:connection this)) (dissoc this :connection)) Database (query [this query-str] (do-query! (:connection this) query-str)))
System map Components
{:database #SomeDatabase{...} :http-client #HttpClient{...} :kafka #Kafka{...} :auth #AuthCredentials{...} ...}
-Created at startup -Entrypoints (e.g http server or kafka consumers) have access to all
components the business flows need -dependencies of a given flow are threaded from the entry point until
the end, one by one if possible -Thus no static access to system map! (e.g via a global atom) -Any resemblance to objects and classes is just coincidence ;)
Pure functions
SOUTHEAST BRAZIL REGION FROM SPACE
Pure functions Definition
"Given the same inputs, it will always produce the same output"
Simplicity Pure functions
-easier to reason about, fewer moving pieces -easier to test, less need for mocking values -parallelizable by default, no need for locks or STMs
Datomic Pure functions
-Datomic’s db as a value allows us to consider a function that queries the database as a pure function -db is a snapshot of the database at a certain point in time. -So, querying the same db instance will always produce the same result
Impure functions Pure functions
-functions that produce side effects should be marked as such. We use `!` at the end. -split code which handles and transforms data from code that handles side effects -should be moved to the borders of the flow, if possible -Consider returning a future/promise like value, so side effect results can be composed (e.g with manifold or finagle)
https://github.com/ztellman/manifoldhttps://github.com/twitter/finagle
Schema/Spec
SOUTHEAST BRAZIL REGION FROM SPACE
Schema Legacy
Majority of our code base was written before clojure.spec existed, so I’ll be talking about the Schema library instead. Most principles apply to clojure.spec as well.
Schema/Spec Documentation
-Clojure doesn’t force you to write types -parameter names are not enough -declaring types helps a lot when glancing at the function -values can be verified against a schema
Function declaration Schema/spec
-All pure functions declare schemas for parameters and return value -All impure functions declare for parameters and don’t declare output type if it’s not relevant. -Validated at runtime in dev/test environments, on every function call -Validation is off on production.
Wire formats Schema/Spec
-Internal schemas are your domain models -Wire schemas are how you expose data to other services/clients -If they are different, you can evolve internal schemas without breaking clients -Need an adapter layer -wire schemas are always validated on entry/exit points, specially in production -single repository for all wire schemas (for all 60+ services) -caveat: this repository has a really high churn. Beware
Growing Schemas Spec-ulation
Please watch Rich Hickey’s talk at Clojure Conj 2016 Spec-ulation:
https://www.youtube.com/watch?v=oyLBGkS5ICk
Ports and Adapters (a.k.a Hexagonal Architecture)
SOUTHEAST BRAZIL REGION FROM SPACE
Ports and Adapters Definition
Core logic is independent to how we can call it (yellow) A port is an entry-point of the application (blue) An adapter is the bridge between a port and the core logic (red)
http://www.dossier-andreas.net/software_architecture/ports_and_adapters.htmlhttp://alistair.cockburn.us/Hexagonal+architecture
Ports and Adapters (Nubank version) Extended Definition
Pure business logic (green) Controller logic wires the flow between the ports (yellow) A port is an entry-point of the application (blue) An adapter is the bridge between a port and the core logic (red)
Ports (Components) Ports and Adapters
-Ports are initialised at startup -Each port has a corresponding component -Serializes data to a transport format (e.g JSON, Transit) -Usually library code shared by all services -Tested via integration tests
HTTP
Kafka
Datomic
File Storage
Metrics
Adapters (Diplomat) Ports and Adapters
-Adapters are the interface to ports
-Contain HTTP and Kafka consumer handlers
-Adapt wire schema to internal schema
-Calls and is called by controller functions
-Tested with fake versions of the port components, or mocks
HTTP
Kafka
Datomic
File Storage
Metrics
Controllers Ports and Adapters
-Controllers wires the flow between entry-point and the side effects
-Only deals with internal schemas
-Delegates business logic to pure functions
-Composes side effect results -Tested mostly with mocks
HTTP
Kafka
Datomic
File Storage
Metrics
Business Logic Ports and Adapters
-Handles and transforms immutable data
-Pure functions -Best place to enforce invariants and type checks (e.g using clojure.spec)
-Can be tested using generative testing
-Should be the largest part of the application
HTTP
Kafka
Datomic
File Storage
Metrics
Microservices Ports and Adapters
-Each service follows about the same design
-Services communicate with each other using one of the ports (e.g HTTP or Kafka)
-Services DON’T share databases
-HTTP responses contain hypermedia, so we can replace a service without having to change clients
-Tested with end to end tests, with all services deployed
Clojure is simple
Keep your design simple
Keep your architecture simple
SÃO PAULO, BRASIL
Lucas Cavalcanti@lucascs
Thank you