the curious clojurist - neal ford (thoughtworks)

The Curious Clojure-ist

1

Agenda Data

Data as Code

Destructuring

Macros

Protocols

The Expression Problem

Concurrency

2

Data

3

datahttps://github.com/edn-format/edn

Extensible Data Notationednedn

4

edn Person5

edn ⊇ Clojure syntax

used by Datomic and others as data transfer format

language/implementation neutral

edn is a system for the conveyance of values.

characteristicsedn


6

a type system

schema based

a system for representing objects


NOT:

7

Scalars

nil nil, null, or nothing

booleans true or false

stringsenclosed in “double quotes”

may span multiple lines\t \r \n supported

characters\c

\newline, \return, \space and \tab

8

Scalars

integers0-9

negative

floating point 64-bit (double) precision is expected.

9

Names

symbols

used to represent identifiersshould map to something other than strings

may include namespace prefixs:my-namespace/foo

keywords

identifiers that designate themselvessemantically akin to enumeration values

symbols that must start with ::fred or :my/fred

10

Collections

listsa sequence of values

zero or more elements within ()(a b 42)

vectors

a sequence of values……that supports random access

zero or more elements within [][a b 42]

11

Collections

maps

collection of key/value associationsevery key should appear only once

unorderedzero or more elements within {}

{:a 1, "foo" :bar, [1 2 3] four}

sets

collection of unique valuesunordered

heterogeneouszero or more elements within #{}

#{a b [1 2 3]}

12

Data as Code

13

Clojure Syntax

edn + …

14

Functions

fn callarg

semantics:

structure:stringsymbol

list

15

Operators (No Different than Functions)

fn call args

list

16

Defining Functions17

defn Semanticsdefine a

fn fn namedocstring

arguments

fn body

18

defn Structure

symbol symbolstring

vector

list

19

Multi-arity

functionmeta-data

20

Control Flow

21

Decisions

true branch

false branch

22

Decisions

23

Refactor

More Arities

24

Don’t Forget…

25

(source …)

26

Namespaces

27

Namespace Declaration

(ns com.example.foo)

names correspond to Java packages,imply same directory structure

28


(ns com.example.foo (:require clojure.data.generators clojure.test.generative))

load some libs

29


(ns com.example.foo (:require [clojure.data.generators :as gen] [clojure.test.generative :as test]))

provide short aliasesfor other libs

30


(ns ^{:author "Stuart Halloway" :doc "Data generators for Clojure."} clojure.data.generators (:refer-clojure :exclude [byte char long ...]) (:require [clojure.core :as core]))

namespace metadata

31

Don’t Do This

(ns com.example.foo (:use clojure.test.generative))

“:use” makes all names unqualified

32

Seqs

33

Sequences Abstraction of traditional Lisp lists

(seq coll)

if collection is non-empty, return seq object on it, else nil

(first seq)

returns the first element

(rest seq)

returns a sequence of the rest of the elements

34

Laziness Most of the core library functions that produce

sequences do so lazily

e.g. map, filter etc

And thus if they consume sequences, do so lazily as well

Avoids creating full intermediate results

Create only as much as you consume

Work with infinite sequences, datasets larger than memory

35

Sequences(drop 2 [1 2 3 4 5]) -> (3 4 5)

(take 9 (cycle [1 2 3 4]))-> (1 2 3 4 1 2 3 4 1)

(interleave [:a :b :c :d :e] [1 2 3 4 5])-> (:a 1 :b 2 :c 3 :d 4 :e 5)

(partition 3 [1 2 3 4 5 6 7 8 9])-> ((1 2 3) (4 5 6) (7 8 9))

(map vector [:a :b :c :d :e] [1 2 3 4 5])-> ([:a 1] [:b 2] [:c 3] [:d 4] [:e 5])

(apply str (interpose \, "asdf"))-> "a,s,d,f"

(reduce + (range 100)) -> 495036

Seq Cheat Sheet

clojure.org/cheatsheet

37

Vectors

38

(def v [42 :rabbit [1 2 3]])

(v 1) -> :rabbit

(peek v) -> [1 2 3]

(pop v) -> [42 :rabbit]

(subvec v 1) -> [:rabbit [1 2 3]]

(contains? v 0) -> true ; subtle

(contains? v 42) -> false ; subtle

Vectors

39

Maps

40

(def m {:a 1 :b 2 :c 3})

(m :b) -> 2 ;also (:b m)

(keys m) -> (:a :b :c)

(assoc m :d 4 :c 42) -> {:d 4, :a 1, :b 2, :c 42}

(dissoc m :d) -> {:a 1, :b 2, :c 3}

(merge-with + m {:a 2 :b 3}) -> {:a 3, :b 5, :c 3}

Maps

41

Nested Structures(def jdoe {:name "John Doe", :address {:zip 27705, ...}})

(get-in jdoe [:address :zip])-> 27705

(assoc-in jdoe [:address :zip] 27514)-> {:name "John Doe", :address {:zip 27514}}

(update-in jdoe [:address :zip] inc) -> {:name "John Doe", :address {:zip 27706}}

42

Sets(use clojure.set)(def colors #{"red" "green" "blue"})(def moods #{"happy" "blue"})

(disj colors "red")-> #{"green" "blue"}

(difference colors moods)-> #{"green" "red"}

(intersection colors moods)-> #{"blue"}

(union colors moods)-> #{"happy" "green" "red" "blue"}

bonus: all relational algebra primitives

supported forsets-of-maps

43

Destructuring

44

Pervasive Destructuring

DSL for binding names

Works with abstract structure

Available wherever names are made*

Vector binding forms destructure sequential things

Map binding forms destructure associative things

45

Why Destructure?

(defn next-fib-pair [pair] [(second pair) (+ (first pair) (second pair))])

(iterate next-fib-pair [0 1])-> ([0 1] [1 1] [1 2] [2 3] [3 5] [5 8] [8 13]...)

without destructuring, next-fib-pair is dominated by code to “pick apart” pair

destructure it yourself…

46

Sequential Destructure

(defn next-fib-pair [[a b]] [b (+ a b)])

(iterate next-fib-pair [0 1])-> ([0 1] [1 1] [1 2] [2 3] [3 5] [5 8] [8 13] ...)

…or you can do the samething with a simple []

47

Simple Things Inline

(defn fibs [] (map first (iterate (fn [[a b]] [b (+ a b)]) [0 1])))

which makes next-fib-pairso simple that you willprobably inline it away!

48

Associative Data

(defn format-name [person] (str/join " " [(:salutation person) (:first-name person) (:last-name person)]))

(format-name {:salutation "Mr." :first-name "John" :last-name "Doe"})-> "Mr. John Doe"

same problem as before:code dominated bypicking apart person

49

Associative Destructure

(defn format-name [name] (let [{salutation :salutation first-name :first-name last-name :last-name} name] (str/join " " [salutation first-name last-name]))


pick apart name

50

The :keys Option

(defn format-name [{:keys [salutation first-name last-name]}] (str/join " " [salutation first-name last-name]))


a common scenario:parameter names and key names are the same, so say

them only once

51

Optional Keyword Args

(defn game [planet & {:keys [human-players computer-players]}] (println "Total players: " (+ human-players computer-players))) (game "Mars” :human-players 1 :computer-players 2)Total players: 3

not a language feature, simply a consequence of variable arity fns

plus map destructuring

52

Platform Interop

53

Java new

java new Widget("foo")

clojure sugar (Widget. "red")

54

Access Static Members

java Math.PI

clojure sugar Math/PI

55

Access Instance Members

java rnd.nextInt()

clojure sugar (.nextInt rnd)

56

Chaining Access

java person.getAddress().getZipCode()

clojure sugar (.. person getAddress getZipCode)

57

Parenthesis Count

java ()()()()

clojure ()()()

58

all forms are created equal !

interpretation is everything

59

form syntax example

function list (println "hello")operator list (+ 1 2)

method call list (.trim " hello ")import list (require 'mylib)

metadata list (with-meta obj m)control flow list (when valid? (proceed))

scope list (dosync (alter ...))

all forms are created equal !

60

Special Forms

61

Special Forms (def symbol init?)

(if test then else?)

(do exprs*)

(quote form)

(fn name? [params*] exprs*)

(fn name? ([params*] exprs*)+)

(let [bindings*] exprs*)

(loop [bindings*] exprs*)

(recur exprs*)

(throw expr)

(try expr* catch-clause* finally-clause?)

62

Macros

63

Programs writing Programs

Reader

evaluator/compiler

Effect

data structures

Code

Text

bytecode

You

JVM

characters

characters

Program

data structures

Reader

evaluator/compiler

Effect

data structures

Code

Text

bytecode

You

JVM

characters

characters

Program

data structures

Program(macro)

data structures

64

Inside Out?

{:name "Jonathan"}

(assoc {:name "Jonathan"} :nickname "Jon")

(dissoc (assoc {:name "Jonathan" :password "secret"} :nickname "Jon") :password)

65

Thread First ->

(-> {:name "Jonathan" :password "secret"} (assoc :nickname "Jon") (dissoc :password))

(dissoc (assoc {:name "Jonathan" :password "secret"} :nickname "Jon") :password)

66

Syntactic Abstraction

Reader

evaluator/compiler

Effect

data structures

Code

Text

bytecode

You

JVM

characters

characters

Program

data structures

Program(macro)

data structures

67

Seq Ops Inside Out

(range 10)

(map inc (range 10))

(filter odd? (map inc (range 10)))

(reduce + (filter odd? (map inc (range 10))))

68

Thread Last ->>

(->> (range 10) (map inc) (filter odd?) (reduce +))

(reduce + (filter odd? (map inc (range 10))))

69

defrecord

70

AFn

IFn

ifn?

AFunction

APersistentVector

APersistentMap

APersistentSet

Keyword

MultiFn

Ref

RestFn

Symbol

Var Callability

Fn

fn?

71

From Maps...(def stu {:fname "Stu" :lname "Halloway" :address {:street "200 N Mangum" :city "Durham" :state "NC" :zip 27701}})

(:lname stu)=> "Halloway"

keyword access

(-> stu :address :city)=> "Durham"

nested access

(assoc stu :fname "Stuart")=> {:fname "Stuart", :lname "Halloway", :address ...}

update

(update-in stu [:address :zip] inc)=> {:address {:street "200 N Mangum", :zip 27702 ...} ...}

nestedupdate

data oriented

72

...to Records!(defrecord Person [fname lname address])(defrecord Address [street city state zip])(def stu (Person. "Stu" "Halloway" (Address. "200 N Mangum" "Durham" "NC" 27701)))

(:lname stu)=> "Halloway"

(-> stu :address :city)=> "Durham"

(assoc stu :fname "Stuart")=> :user.Person{:fname "Stuart", :lname"Halloway", :address ...}

(update-in stu [:address :zip] inc)=> :user.Person{:address {:street "200 N Mangum", :zip 27702 ...} ...}

still data-oriented:everything works

as beforetype is therewhen you

care

object oriented

73

defrecord(defrecord Foo [a b c])-> user.Foo

named typewith slots

(def f (Foo. 1 2 3))-> #'user/f positional

constructor(:b f)-> 2 keyword

access(class f)-> user.Foo plain ol'

class(supers (class f))-> #{clojure.lang.IObj clojure.lang.IKeywordLookup java.util.Map clojure.lang.IPersistentMap clojure.lang.IMeta java.lang.Object java.lang.Iterable clojure.lang.ILookup clojure.lang.Seqable clojure.lang.Counted clojure.lang.IPersistentCollection clojure.lang.Associative}

casydht*

*Clojure abstracts so you don't have to

74

Protocols

75

(defprotocol AProtocol "A doc string for AProtocol abstraction" (bar [a b] "bar docs") (baz [a] "baz docs"))

Named set of generic functions

Polymorphic on type of first argument

No implementation

Define fns in the same namespaces as protocols

Protocols

76

Extending Protocols

77

Extend Protocols Inline

(defrecord Bar [a b c] AProtocol (bar [this b] "Bar bar") (baz [this] (str "Bar baz " c)))

(def b (Bar. 5 6 7))

(baz b)

=> "Bar baz 7"

78

Extend Protocols Inlinefrom ClojureScript

browser.clj

79

Extending to a Type(baz "a")

java.lang.IllegalArgumentException: No implementation of method: :baz of protocol: #'user/AProtocol found for class: java.lang.String

(extend-type String AProtocol (bar [s s2] (str s s2)) (baz [s] (str "baz " s)))

(baz "a")

=> "baz a"

80

Extending to Many Types

from Clojurereducers.clj

note extendto nil

81

Extending to Many Protocols

from ClojureScriptcore.cljs

82

Composition with Extend

from Clojure java/io.clj

the “DSL” for advanced reuse is maps and assoc

83

Reify

(let [x 42 r (reify AProtocol (bar [this b] "reify bar") (baz [this ] (str "reify baz " x)))] (baz r))

=> "reify baz 42"

instantiate an unnamed type implement 0

or more protocols

or interfaces

closes overenvironment

like fn

84

Code Structurepackage com.acme.employees;

Employee

raise()

roles()

updatePersonalInfo()

Manager

roles()

approvalProfile()

interface Employee {}

(namespace com.acme.employees)

(raise )

(updatePersonalInfo )

(roles )

(approvalProfile )

(defprotocol Employee )

85


86


abstraction

concretion

A B

A should be able to work with B's abstractions, and vice versa,

without modification of the original code

87

Is This Really a Problem?

abstraction

concretion

just use interfaces for abstraction (??)

BA

88

Example: ArrayList vs.the Abstractions

java.util.List

ArrayList

clojure.lang.Counted

clojure.lang.Seqable

?89

Example: String vs.the Abstractions

java.util.List

String clojure.lang.Counted

clojure.lang.Seqable

?

90

A Can't Inherit from B

B is newer than A

A is hard to change

We don’t control A

happens even within a single library!

91

Some Approachesto the Expression

Problem

92

1. Roll-your-own

if/then instanceof? logic

closed

93

A Closed World

94

so make a NiftyString

that is

2. Wrappers

NiftyString

java.util.List

String

java.util.Collectionstrings are

not collections

95

Wrappers = Complexity

Ruin identity

Ruin Equality

Cause nonlocal defects

Don’t compose: AB + AC ≠ ABC

Have bad names

96

3. Monkey Patching

common in e.g. rubynot possible in java

String

java.util.List

java.util.Collectionsneak in

and change them!

strings are not

collections

97

Monkey Patching = Complexity

Preserves identity (mostly)

Ruins namespacing

Causes nonlocal defects

Forbidden in some languages

98

4. Generic Functions (CLOS)

don't touch existing implementation,

just use it

String

map

reduce

count

polymorphism lives in the

fns

99

Generic Functions

Decouple polymorphism & types

Polymorphism in the fns, not the types

no “isa” requirement

no type intrusion necessary

100

protocols = generic functions - arbitrary dispatch + speed + grouping

(and still powerful enough tosolve the expression problem!)

101

Concurrency

102

concurrency, coincidence of events or space

parallelism, the execution of operations concurrently by separate parts of a computer

103

Our Tools

threads

104

Our Tools

42

places

105

critical sections

Our Tools

42

42 42

106

memory, the capacity ... for returning to a previous state when the cause of the transition from that state is removed

record, the fact or condition of having been written down as evidence...... an authentic or official report

107

Memory, Records = Places?

Memory is small and expensive

Storage is small and expensive

Machines are precious, dedicated resources

Applications are control centers

108

A Different Approach

New memories use new places

New records use new places

New moments use new places

“In-place” changes encapsulated by constructors

109

Values

110

Values

Immutable

Maybe lazy

Cacheable (forever!)

Can be arbitrarily large

Share structure

111

What Can Be a Value?

42

112


{:first-name "Stu", :last-name "Halloway"}

42

113


42{:first-name "Stu", :last-name "Halloway"}

114



115



Anything?

116

References

Refer to values (or other references)

Permit atomic, functional succession

Model time and identity

Compatible with a wide variety of update semantics

117

Epochal Time Model

v1 v2 v3

values

118

Epochal Time Model

v1 v2 v3

f1 f2

functions

119

Epochal Time Model

v1 v2 v3

f1 f2

atomic succession

120

Epochal Time Model

v1 v2 v3

f1 f2

reference

121

Epochal Time Model

v1 v2 v3

f1 f2

observers perceive identity, can remember and record

122

Epochal Time Model

v1 v2 v3

f1 f2

observers do notcoordinate

123

Epochal Time Model

v1 v2 v3

f1 f2

124

Atoms

125

Atoms

(def a (atom 0))

(swap! a inc)=> 1

(compare-and-set! a 0 42)=> false

(compare-and-set! a 1 7)=> true

126

Atoms

(def a (atom 0))

(swap! a inc)=> 1



functional succession

127

Atoms

(def a (atom 0))

(swap! a inc)=> 1



optimistic concurrency

128

Software Transactional Memory

129

Software Transactional Memory

Refs can change only within a transaction

Provides the ACI in ACID

Transactions are speculative, will be retried

130

F

v1 v2 v3 v4

v1 v2 v3 v4

v1 v2 v3 v4

v1 v2 v3 v4

F

F

F

F

F

F

F F F

F

F

Transactions131

Transactions

(defn transfer [from to amount] (dosync (alter from - amount) (alter to + amount)))

(alter from - 1)=> IllegalStateException No transaction running

132



Transactions

scope transaction

133



Transactions

functional succession

134

Transactions



coordination guaranteed!

135

STM Details

Uses locks, latches internally to avoid churn

Deadlock detection and barging

No read tracking

Readers never impede writers

Nobody impedes readers

136

Summary

137

Summary

Serious Lisp on the JVM

Built as a destination

Advanced language features

Advanced implementation

Secret weapon?

138

Clojure in the wild?“We re-coded our flagship application XXXXXX from Java to Clojure about a year ago. NEVER looked back. Why?

Reduced our lines of code down by at least half.

Support and bugs have likewise been cut by about 65-70%.

We have large enterprise clients. How did we get Clojure into these very old guard environments????

139

Between you and me....

We don't talk about Clojure. We talk about "Java Extensions" or "the Clojure Java Extension". No one is the wiser.Clients LOVE us for our blistering fast turn around. We present ourselves as a larger company with fake Linkedin employees. We actually only have 4 real employees. But with Clojure we do the same work as if we had 20.”

we lie.

140

?’sThe preceding work is licensed under the Creative Commons Attribution-Share Alike 3.0 License.

http://creativecommons.org/licenses/by-sa/3.0/us/

Presentation PatternsNeal Ford, Matthew McCullough, Nathaniel Schuttahttp://presentationpatterns.com

Functional Thinking bit.ly/nf_ftvideo

Clojure (inside out)Neal Ford, Stuart Hallowaybit.ly/clojureinsideout

141

the curious clojurist - neal ford (thoughtworks)

Technology