functional operations - susan potter

Functional Operations

#dmconf1521 November 2015

Susan PotterLookout

twitter: @SusanPottergithub: mbbx6spp

% whoami

Figure: From developer to (dev)ops engineer

Agenda

1 Motivation for reasoning

2 Review functional programming 101

3 Illustrate forms of reasoning

4 Case study: Reimagine package management

Edsger Dijkstra

Reliability

“Those who want really reliable software will discoverthat they must find means of avoiding the majority ofbugs to start with, and as a result the programmingprocess will become cheaper. If you want more effectiveprogrammers, you will discover that they should notwaste their time debugging, they should not introducethe bugs to start with.”[?]

Why care now?

1 Economic factorsnecessity of distributed systems

2 Human factorshigh churn/turnover, low quality of ops life

3 Technological factorsprogrammable infrastructure & FP no longer just for academics

The Problem. . .

Application Delivery

1 Provision infrastructure

2 Configure nodes

3 Orchestrate services

Need to support more

1 Application services

2 Environments

3 Data services

4 Distributed services

Optimize for

1 Scalability solved by on-demand ”cloud”

2 Reliability solved by . . .

So what yields reliability?

Reason

The required techniques of effective reasoning are prettyformal, but as long as programming is done by peoplethat don’t master them, the software crisis will remainwith us and will be considered an incurable disease. [?]

Functions 101

Functions have inputs (Ruby)

1 # Two input arguments here

2 def add(x, y)

3 x + y

4 end

5

6 # One input argument here

7 def len(s)

8 s.size

9 end

Functions have inputs (Scala)

1 object Functions {

2 // Two typed input arguments here

3 def add(x: Int , y: Int) = x + y

4

5 // One typed input argument here

6 def len(s: String) = s.size

7 }

Functions return a result

1 scala > add(5, 6)

2 res0: Int = 11

3

4 scala > len("Hello ,␣Barcelona")

5 res1: Int = 16

Only depend on inputs 1/2

1 scala > val defaultTimeout: Int = 30

2 scala > val timeout1: Option[Int] = Some (15)

3 scala > val timeout2: Option[Int] = None

4 scala > :paste

5 def defaulter[A](a: => A, ma: Option[A]) =

6 ma match {

7 case Some(x) => x

8 case None => a

9 }

Only depend on inputs 2/2

1 scala > timeout1

2 timeout1: Option[Int] = Some (15)

3

4 scala > timeout2

5 timeout2: Option[Int] = None

6

7 scala > defaulter(defaultTimeout , timeout1)

8 res0: Int = 15

9

10 scala > defaulter(defaultTimeout , timeout2)

11 res1: Int = 30

Return same result given same inputs


2 res0: Int = 16

3


5 res1: Int = 16

6 ...


8 res3333333333: Int = 16

9

10 scala > // Always!

The Big idea

Referential TransparencyGiven same inputs, return same result. Always.

Functions can use other values

1 // type aliasing a function

2 type Pred[A] = A => Boolean

3

4 // Passing a function as an input argument

5 def is[A](p: Pred[A])(a: A) = p(a)

6

7 // This uses already defined function +is+

8 def not[A](p: Pred[A])(a: A) = !is(p)(a)

Values can be functions

1 // Returning a function as a value

2 def lessThanN(n: Int): Pred[Int] = _ < n

3

4 // Using two in scope functions

5 def islessThanN(n: Int)(x: Int) =

6 is(ltN(n))(x)

7

8 // Those values can be functions :)

Another important idea

Higher Order FunctionsBuild useful functions from simpler functions!

Questions so far?

Figure: Awake?

Function Composition

UNIX Pipes 1/4

1 sh> echo -n "Hello ~~~" \

2 | sed ’s/~//g’ \

3 | tr ’[:lower:]’ ’[:upper:]’ \

4 | wc -c

5 5

UNIX Pipes 2/4

1 sh> alias sanitize=’sed "s/~//g"’

2 sh> alias toUpper=’tr "[: lower :]" "[: upper:]"’

3 sh> alias len=’wc -c’

4

5 sh> alias myfun0=’sanitize | toUpper ’

6 sh> alias myfun1=’myfun0 | wc -c’

UNIX Pipes 3/4

1 sh> echo -n "Hello ~~~" \

2 | sed ’s/~//g’ \

3 | tr ’[:lower:]’ ’[:upper:]’ \

4 | wc -c

5 5

6

7 sh> echo -n "Hello ~~~" | myfun1

8 5

UNIX Pipes 4/4

• Character-based

• File descriptors

Function Composition 1/3

1 def toUpper = (s: String) => s.toUpperCase

2 def len = (s: String) => s.size

3 def sanitize = "~".r replaceAllIn (_, "")


1 scala > def myfun0 = sanitize andThen toUpper

2 scala > def myfun1 = myfun0 andThen len

3

4 scala > myfun0 "Hello ~~~"

5 res0 = HELLO

6

7 scala > myfun1 "Hello ~~~"

8 res1: Int = 5


• Value-based

• Functions

Questions so far?

Figure: Awake?

Functions as building blocks

Figure: Build solid walls/foundations from simple, regular building blocks

Reasoning

Equational Reasoning

• Simpler testing; reduce test suite complexity

• Substitute repeated expression with name

• Refactoring is trivial; not error prone

Testing non-RT code

Figure: Side effecting, non-RT code requires complex testing setup.


1 scala > f(v1, v2, otherfun) == expected

2 res0: Boolean = true

3

• no setup/teardown complexity to test assertions

• no mutation of SUT just for testing


1 scala > val x = "YOLO"

2 x: java.lang.String = YOLO

3

4 scala > val r1 = x.reverse

5 r1: String = OLOY

6

7 scala > "YOLO".reverse

8 res0: String = OLOY

9


1 scala > var age = 27

2 scala > def incr = { age += 1; age }

3

4 scala > incr

5 res0: Int = 28

6

7 scala > incr

8 res1: Int = 29

9

Axiomatic Reasoning

• Axioms are basic assumptions

• Allows us to specify properties

• Infer potential problems due to properties

• What can we guarantee? What can we not?

Axiomatic Reasoning

1 class Vpc < AwsResource

2 def initialize(conf)

3 @cidr = conf[’vpc.cidr’]

4 end

5 end

6 # Is the Vpc instance created with an

7 # empty conf Hash guaranteeing the

8 # premise of the initialize contract?

9

Axiomatic Reasoning

1 // validate inputs before instantiating

2 case class Vpc(cidr: Cidr)

3

4 def createVpc(ipStr: String , pfx: String) =

5 for {

6 ip <- IpAddress.toIpAddress(ipStr)

7 prefix <- Cidr.toPrefix(pfx)

8 } yield Vpc(Cidr(ip , prefix ))

9

Axiomatic Reasoning

1 // if fun is 1-to -1 or bijective

2 // we know inverse must exist

3 def toLower(u: UpperCaseChar ): LowerCaseChar = ???

4 def toUpper(l: LowerCaseChar ): UpperCaseChar = ???

5

6 // idempotency typically important in ops

7 f(f(x)) == f(x)

8

Axiomatic Reasoning

1 // commutativity important in dist sys

2 // f, binary operator over type A

3 // x and y are of type A then

4 // x @ y = y @ x

5 f(x, y) == f(y, x)

6

7 // associativity important in concurrency

8 // f, binary operator over type A

9 // x, y, and z are of type A then

10 // x @ (y @ z) = (x @ y) @ z

11 f(x, f(y, z)) == f(f(x, y), z)

12

Axiomatic Reasoning - DistributedSystems

• Simple causal consistency (partial ordering)

• CRDTs (semilattice algebra)

• Invariant confluence (coordination freeexecution)

Axiomatic Reasoning - Concurrency

• MVars, TVars, LVars have algebraic properties

• Design for liveness via axiomatic reasoning

Axiomatic Reasoning - More ExhaustiveTesting

• Property-based testing

1 // untested - uses scalacheck

2 def inverse[A]( implicit a: Action[A]) =

3 Prop.forAll { (x: A) =>

4 x === a.inverse(a.inverse(x))

5 }

6

7 implicit def actionArbitrary[A]

8 (implicit a: Arbitrary[A]) = Arbitrary { /* TODO */ }

9

Generic Reasoning

• Reasoning on generic type functions

• Less specific the types the more we know aboutthe function

• Find design-time bugs

Generic Reasoning

1 def f0[A](a: A): A = ???

2

Generic Reasoning

1 // Only definition

2 def f0[A](a: A): A = a

3

Generic Reasoning

1 def f1[A](a: A)(ma: Option[A]): A = ???

2

Generic Reasoning

1 def f1[A](a: A)(ma: Option[A]): A = a

2

1 def f1[A](a: A)(ma: Option[A]): A = ma match {

2 case Some(x) => x

3 case None => a

4 }

5

Generic Reasoning

1 def f2[A](as: List[A]): A = ???

2

Generic Reasoning

1 // what about an empty list? Yikes!

2 def f2[A](as: List[A]): A = ???

3

Generic Reasoning

1 // Assuming we want the head element!

2 // A more sensible type signature design

3 def f3[A](as: List[A]): Option[A] = as match {

4 case Nil => None

5 case a :: rest => a

6 }

7

Generic Reasoning

1 def f4[A](ma: Option[A]): A = ???

2

Review

Reasoning Process

• Assume as little as possible

• Define all inputs

• Build only on top of RT

• Derive properties from domain

• Infer properties from generic types

Methods / Techniques

• Encode normal error cases in results

• Strive for termination

• Strive for RT where ever possible

Iteration / Improvement

• So much to learn

• Start out informal; add formal reasoning asneeded/learned

• The more we do the easier it gets

Example in the wild

Mainstream Package Management

Based on shared + mutable state (filesystem)

Violates RT

Alternatives

• shared + immutable

• private + mutable

• expensive coarse grained locks

• hybrid without the expense

Define all inputs

• Force clean build env (chroot)

• Requires explicit inputs

• Full dependency definition

Ensure RT

• Use private mutable space

• Different inputs, different result

• Symlink unique results (atomic op)

Different inputs give unique results

• Shared + immutable provides RT

• New inputs result in different path

• Install many versions/configurations

• Means we can do binary substitution

RT foundation

• Update existing node more reliably

• Supports immutable infrastructure

• Supports congruent configuration

Recap

1 Referential Transparency allows us to reason

2 Functions are simple, regular building blocks

3 HOFs & Composition is the motar

4 Logical reasoning provides solid foundation

Don’t fear the maths!

Figure: Domain specific language of the sciences and engineering

Questions

Figure: Heckle me @SusanPotter later too.

functional operations - susan potter

Data & Analytics