clojure@nuday
TRANSCRIPT
Why Clojure?
● See “Hackers and Painters”● Small dev team can do a lot quickly● Easier to reason about concurrency● Engineers attracted to Clojure tend to be
good● Fits in well with all the battle-tested Java
machinery for service at scale
Users
DNS round robin
nginx nginx
Elastic Beanstalk
ELB
Tomcat
...
...
SNS
S3
Elastic Beanstalk
Worker
Tomcat
SQS
RDS DynamoDB
Routes
(ns insurrection.routes.core (:require [insurrection.resource.get :as get] [insurrection.resource.put :as put]))
(defroutes rockscience-routes (GET "/health" [] get/health) (GET "/player/:playerId/profile" [] get/profile) (GET "/player/:playerId/games" [] get/games) (GET "/player/:playerId/events" [] get/events) (PUT "/player/:playerId/pushToken" [] put/push-token))
Resources(ns insurrection.resource.get)
(def events (get-resource :events view/events [:playerId :timestamp] :transform-fn {:playerId to-int :timestamp to-long} :validation-fn {:playerId (validate EntityId) :timestamp (validate Timestamp)}))
Resources under the hood(ns insurrection.routes.resources)
(defn get-resource [resource resource-fn args & {:as opts}] (-> (lib/resource :allowed-methods [:get] :available-media-types [json-media-type] :available-charsets ["UTF-8"] :authorized? authorized? :exists? (make-exists? resource resource-fn args opts) :handle-ok (make-response-handler resource :ok) :handle-not-found (make-not-found-handler resource args opts)) wrap-keyword-params wrap-json-params wrap-json-response))
http://clojure-liberator.github.io/liberator/assets/img/decision-graph.svg
Handling a GET
(defn make-exists? [resource-name resource-fn args & [opts]] (fn [ctx] (let [params-str (params->str ctx args opts)] (log/info (log-str "Getting %s" resource-name params-str)) (try (when-let [resource (apply resource-fn (params->values ctx args opts))] {resource-name resource}) (catch Exception e (log-error e (log-str "Failed to get %s" resource-name params-str)) {:exception e})))))
Response(defn make-response-handler [resource-name & [status]] {:pre [(s/validate s/Keyword resource-name) (or-validate HttpStatus status)]}
(let [status (or status :ok)] (fn [ctx] (let [exception (:exception ctx) resource (if exception {:status false, :message (.getMessage exception)} (resource-name ctx)) resp (make-response resource) http-status (if exception 500 (http/status status))]
(ring-response (-> resp (assoc :status http-status)))))))
Types, what are they good for?
● Dynamic typing is wonderful, but…● Prismatic Schema gives you validation and
documentation without fighting the compiler
Schemas(ns rockscience.models.entity (:require [rockscience.rules :as rockscience] [schema.core :as s])
(def EntityId (s/both s/Int (s/pred pos? 'pos?)))
(def Timestamp (s/both s/Int (s/pred #(>= % rockscience/epoch) 'ts?)))
Event sourcinghttp://martinfowler.com/eaaDev/EventSourcing.html
The fundamental idea of Event Sourcing is that of ensuring every change to the state of an application is captured in an event object, and that these event objects are themselves stored in the sequence they were applied for the same lifetime as the application state itself.
Querying event logGET /player/:playerId/events?timestamp=1413801431000
{ "timestamp": "2014-10-20T15:40:47Z", "events": [ { "eventType": "ChallengeReceived", "playerId": "123", "timestamp": "2014-10-20T11:05:40Z", "entityId": "912" }, { "eventType": "RoundAnswered", "playerId": "123", "timestamp":"2014-10-20T11:06:12Z", "entityId":"912" }, { "eventType":"GameEnded", "playerId":"123", "timestamp":"2014-10-20T11:09:05Z", "entityId":"912" } ]}
Event log view
(defn events [player-id timestamp] {:pre [(validate EntityId player-id) (validate Int timestamp)] :post [(validate ApiPlayerNotificationEvents %)]}
{:timestamp (->api-timestamp (to-long (now))) :events (->> (player/get-events player-id timestamp) (map ->api-player-notification-event))})
More interesting schemas(def ApiPlayerNotificationEvents {:timestamp ApiTimestamp :events [ApiPlayerNotificationEvent]})
(def ApiTimestamp (s/both s/Str (s/pred api-timestamp? 'api-timestamp?)))
(defn api-timestamp? [s] (try (parse (formatters :date-time-no-ms) s) (catch Exception _ false)))
Schemas built of schemas(def ApiPlayerNotificationEvent (->> (merge PlayerNotificationEvent {:event-type ApiPlayerNotificationEventType :timestamp ApiTimestamp}) (mmap #(vector (->camelCase %1) %2))))
(def ApiPlayerNotificationEventType (->> event-types (map ->api-event-type) (apply s/enum)))
(def event-types #{:challenge-received :round-completed :game-ended})
It’s turtles all the way down
(def PlayerNotificationEvent {:player-id s/Str :event-type PlayerNotificationEventType :entity-id s/Str :timestamp s/Int})
(def PlayerNotificationEventType (apply s/enum event-types))
Transmogrify turtles to elephants(defn ->api-player-notification-event [ev] {:pre [(s/validate PlayerNotificationEvent ev)] :post [(s/validate ApiPlayerNotificationEvent %)]}
(->> ev (mmap (fn [k v] [(->camelCase k) (case k :event-type (->api-event-type v) :timestamp (->api-timestamp v) v)]))))
(defn ->api-timestamp [ts] {:pre [(s/validate s/Int ts)] :post [(s/validate ApiTimestamp %)]} (unparse ts-formatter (from-long ts)))
Persisting events
● Events created on every mutation, so write fast
● Not really relational● Good fit for NoSQL!
Specifically, DynamoDB● Dynamo is scaled for
reads/sec and writes/sec per table
Events table(require '[amazonica.aws.dynamodbv2 :as dynamo])
(-> (dynamo/describe-table "events") :table pprint)
;=> {:table-size-bytes 28296, :item-count 418, :table-status "ACTIVE", :creation-date-time #<DateTime 2014-09-17T11:41:47.000+02:00>, :table-name "events", :attribute-definitions [{:attribute-type "S", :attribute-name "player-id"} {:attribute-type "N", :attribute-name "timestamp"}], :key-schema [{:attribute-name "player-id", :key-type "HASH"} {:attribute-name "timestamp", :key-type "RANGE"}], :provisioned-throughput {:number-of-decreases-today 0, :read-capacity-units 1, :write-capacity-units 1}}
Querying events(->> (dynamo/query :table-name "events" :select "ALL_ATTRIBUTES" :key-conditions {:player-id {:attribute-value-list ["123"] :comparison-operator "EQ"} :timestamp {:attribute-value-list [1413801431000] :comparison-operator "GE"}}) :items first pprint)
{:event-type "challenge-received", :player-id "123", :entity-id "912", :timestamp 1413803140000}
Writing events
(dynamo/put-item :table-name "events", :item {:event-type "challenge-received", :player-id "123", :entity-id "912", :timestamp 1413803140000}))
Replaying events
● Our events report on mutation, but don’t cause it
● Not enough data for replay :(
CQRS?http://martinfowler.com/bliki/CQRS.html
Ideal mutation flow?HTTP POST, PUT, or DELETE
Create mutation commandLog mutation command to DynamoDB, return ID
Publish mutation ID to SQS
Receive mutation ID from SQSApply mutation command
Update DynamoDB with mutation outcome
HTTP GET /mutation/:idUpdate client state
App
REST service
Worker
App
Challenges with this approach
● Our app assumes synchronous requests● Extra SQS puts and gets introduce more
latency● Implementing long polling after the fact is a
PITA● Web sockets to the rescue?
Compromise mutation flow
HTTP POST, PUT, or DELETE
Apply mutation commandLog mutation command to DynamoDB
Return result
Update client state
App
REST service
App
Replaying commands
Read command from DynamoDBPublish command to SQS
Receive mutation command from SQSApply mutation command
If result is different than it was the first time, fail?
Loader
Worker
No CQRS
HTTP POST, PUT, or DELETE
Log mutation model to DynamoDBPerform mutation
Return result
Update client state
App
REST service
App
Replaying without CQRS
Read mutation model from DynamoDBHTTP POST, PUT, or DELETE
Loader
Perform mutationReturn result
REST service
If result is different, fail? Loader