go’ing to the next level, lessons from go at couchbase and introducing the go sdk: couchbase...

GO’ING TO THE NEXT LEVEL, LESSONS FROM GO AT COUCHBASE AND INTRODUCING THE GO SDK

Marty Schoch and Matt Ingenthron

Lessons from Goat Couchbase

Lessons from Go at Couchbase

©2015 Couchbase Inc. 4

Couchbase Labs

2011 2 engineers 0 rules

©2015 Couchbase Inc. 5

What if we used Go?

Easy concurrency Easy HTTP Server Easy HTTP Client Easy JSON Rich Standard Library

©2015 Couchbase Inc. 6

Timeline

January 2012go-couchbase

2012

2014

September 2012sync getaway

September 2012cbfs

October 2012

tuq -> tuqtng -> n1ql -> sql for documents

November 2012cbgb

January 2013cbugg

February 2013

Godu, vbmap, and other utils

August 2013

Secondary indexing April 2014

bleve

August 2014goxdcr

October 2014cbft

December 2014gocb

©2015 Couchbase Inc. 7

Why do developers like Go?

Excellent tooling CPU/memory profilers Race detector Integrated Unit Tests

Channels for synchronization as well as communication

Satisfy interface without importing its package

©2015 Couchbase Inc. 8

What don’t developers like about Go?

No good debugger gdb works in some cases not others Others still immature

error as interface is powerful construct … but too often its fmt.Errorf(“…”) … which leads to string matching later Define constants so that errors can be differentiated and

handled accordingly

Garbage Collection

©2015 Couchbase Inc. 10

Don’t Make Garbage

©2015 Couchbase Inc. 11

Sounds Simple

Be mindful of allocations Be mindful of operations that copy data

string <-> []byte conversions Many standard library functions operating on []byte But, once you avoid making copies you need clear notion

of ownership Re-use already allocated objects when possible

Include Reset() methods in your API sync.Pool offers solution when objects are

reusable and have clear start/end lifecycle

©2015 Couchbase Inc. 12

Do I really have a GC problem?

Review CPU Profiles

©2015 Couchbase Inc. 13

Do I really have a GC problem?

Graph GC Pause Time (exposed by expvar)

©2015 Couchbase Inc. 14

Do I really have a GC problem?

Visualize Garbage Collector Details Using gcvis by Dave Cheney

©2015 Couchbase Inc. 15

Is this variable on the stack or the heap?

Have the compiler tell you. Build with: go build -gcflags=-m

./main.go:54: map[string]interface {} literal escapes to heap./main.go:112: (*matchPhraseQuery).Validate q does not escape

©2015 Couchbase Inc. 16

Solutions for Advanced Users

Slab allocation See github.com/couchbase/go-slab

Off-heap allocation Lots of pointers, even if largely static, may cause

unreasonable GC Large maps, tree structures with child pointers, etc GC has to check pointers in map, even though you never

plan on freeing them Several new projects in this area…

©2015 Couchbase Inc. 17

Other Memory Surprises?

©2015 Couchbase Inc. 18

Sync Gateway – Many Concurrent Users

Observation: often 20GB+ memory usage Action: Review heap profile

(pprof) top25Total: 3972.5 MB 3875.0  97.5%  97.5%   3875.0  97.5% compress/flate.(*compressor).initDeflate   19.0   0.5%  98.0%     19.5   0.5% github.com/couchbaselabs/sync_gateway/db.(*Database).MultiChangesFeed   12.5   0.3%  98.3%     12.5   0.3% net/http.newBufioWriterSize   10.5   0.3%  98.6%     10.5   0.3% bufio.NewReaderSize   10.0   0.3%  98.9%     10.0   0.3% compress/flate.newHuffmanBitWriter    7.0   0.2%  99.0%      7.0   0.2% newdefer    4.5   0.1%  99.1%      4.5   0.1% github.com/couchbaselabs/sync_gateway/channels.readString    3.0   0.1%  99.2%      3.0   0.1% net/textproto.(*Reader).ReadMIMEHeader    2.0   0.1%  99.3%      2.0   0.1% github.com/couchbaselabs/sync_gateway/auth.func·001    2.0   0.1%  99.3%      2.0   0.1% reflect.mapassign    1.5   0.0%  99.4%      1.5   0.0% github.com/couchbaselabs/sync_gateway/channels.TimedSet.Copy    1.5   0.0%  99.4%      1.5   0.0% net/http.Header.clone    1.5   0.0%  99.4%      1.5   0.0% net/textproto.MIMEHeader.Set    1.5   0.0%  99.5%      1.5   0.0% net/url.parseQuery    1.5   0.0%  99.5%      1.5   0.0% runtime.malg    1.0   0.0%  99.5%      1.0   0.0% compress/flate.(*huffmanEncoder).generate    1.0   0.0%  99.6%      1.0   0.0% concatstring    1.0   0.0%  99.6%      1.0   0.0% encoding/json.(*decodeState).objectInterface    1.0   0.0%  99.6%      1.0   0.0% github.com/couchbaselabs/sync_gateway/channels.(*ChangeLog).TruncateTo    1.0   0.0%  99.6%      5.5   0.1% github.com/couchbaselabs/sync_gateway/channels.decodeChangeLog    1.0   0.0%  99.7%      1.0   0.0% github.com/gorilla/context.Set    1.0   0.0%  99.7%      1.0   0.0% github.com/gorilla/mux.(*routeRegexpGroup).setMatch    1.0   0.0%  99.7%      4.5   0.1% net/http.ReadRequest    0.5   0.0%  99.7%   3885.5  97.8% compress/flate.NewWriter    0.5   0.0%  99.7%      0.5   0.0% compress/gzip.NewWriterLevel

©2015 Couchbase Inc. 19

Gzipped Responses

Sync Gateway clients are mobile, gzipped response makes a lot of sense.

Initial implementation, invoked compress/flate NewWriter on each response

Each flate.Writer consumed 1.4MB Rewritten to use sync.Pool

Acquire flate.Writer instances from pool Call Reset() and return to pool when done

©2015 Couchbase Inc. 20

Slices are Great… But…

Read 1MB of bytes off the wire into a buffer Parse the bytes, find the data you’re interested

in Return []byte with the key (say 16 bytes) The ENTIRE 1MB buffer is held in memory while

the slice is referenced. If these slices are long-lived and small relative to

their parent array, may be cheaper to copy them.

Go Routines / Channels

©2015 Couchbase Inc. 22

Too many channels?

Channels are synchronized

Synchronization has a cost

Use wisely, not haphazardly

©2015 Couchbase Inc. 22

Repeatable Builds

©2015 Couchbase Inc. 24

New/Prototype Projects

Bleeding Edge ‘go get’ everything Low friction Things break, and we fix them

But no one pretends this works for production!

24

©2015 Couchbase Inc. 25

Sync Gateway Builds

First Go project to need a solution to the problem Decided on a custom solution

Dependencies are vendored using git submodules Use wrapper scripts around native tools, go.sh, build.sh,

run.sh Scripts setup custom $GOPATH containing only

sync_gateway and its dependencies Works well for sync_gateway team

Separate $GOPATH can make for more complex integrations with IDE’s and tools like Go oracle

©2015 Couchbase Inc. 26

Projects integrating with Couchbase Server

Couchbase Server already has a well established process for repeatable builds using ‘repo’ All external dependencies are cloned into a separate

github organization “couchbasedeps” In our repo manifests, we get sources from couchbase

deps, but install them into $GOPATH using their canonical package name

<project name="protobuf" remote="couchbasedeps"        revision="655cdfa588ea190e901bc5590e65d5621688847c"        path="godeps/src/github.com/golang/protobuf"/>

©2015 Couchbase Inc. 27

Set $GOPATH and all Go tools are happy!

©2015 Couchbase Inc. 27

Using cgo

©2015 Couchbase Inc. 29

cgo on Windows Depends on gcc

No option to use MSVC We recently went to great pains to remove dependency

on GCC and now we had to add it back Considered dynamically loading DLL

High maintenance for wrappers when APIs change Current solution is to install gcc on Windows

build boxes

©2015 Couchbase Inc. 30

Reliable Core Dumps with crash inside cgo

With gdb attached, we see what we expect However, if no debugger is attached, resulting

core file does not have correct stack trace Using gccgo does work, but not comfortable shipping this

to customers

Thoughts…

©2015 Couchbase Inc. 32

Go at Couchbase, at a Crossroads

©2015 Couchbase Inc. 32

©2015 Couchbase Inc. 33

Real Problems, Real Solutions

Solve the problems Contribute back to the community

Solve the problems Don’t contribute them back

Don’t solve them Move away from Go in the future

The Couchbase Go SDKgocb

©2015 Couchbase Inc. 35

Couchbase SDKs

What does it mean to be a Couchbase SDK?

Cluster

Bucket

CRUDView

QueryN1QL Query

FunctionalManage connections to the bucket within the cluster for different services.Provide a core layer where IO can be managed and optimized.Provide a way to manage buckets.

APIinsertDesignDocument()flush()listDesignDocuments()

FunctionalHold on to cluster information such as topology.

APIReference Cluster ManagementopenBucket()info()disconnect()

FunctionalGive the application developer a concurrent API for basic (k-v) or document management

APIget()insert()upsert()remove()

FunctionalAllow for querying, execution of other directives such as defining indexes and checking on index state.

APIclient.NewN1QLQuery( “SELECT * FROM default LIMIT 5” ) .Consistency(gocouchbase.RequestPlus);

FunctionalAllow for view querying, building of queries and reasonable error handling from the cluster.

APIabucket.NewViewQuery().Limit().Stale()

©2015 Couchbase Inc. 36

Being True to Go Philosophy

Easy to read, write, understand with high-level concurrency (goroutines), type safety. Low level access if needed.

gocbcore

gocb

See more in a gist from Brett Lawson.

var user interface{}

bucket.Get("user_1", &user)fmt.Printf("Got user_1: %v\n", user)

bucket.Get("user_2", &user)fmt.Printf("Got user_2: %v\n", user)

bucket.Get("user_3", &user)fmt.Printf("Got user_3: %v\n", user)

signal := make(chan int)go func() {

var user interface{}bucket.Get("user_1",

&user)fmt.Printf("Got user_1:

%v\n", user)signal <- 1

}()go func() {

var user interface{}bucket.Get("user_2",

&user)fmt.Printf("Got user_2:

%v\n", user)signal <- 1

}()go func() {

var user interface{}bucket.Get("user_3",

&user)fmt.Printf("Got user_3:

%v\n", user)signal <- 1

}()for i := 0; i < 3; i++ {

<-signal}

// Note that this does not perform transcoding (ie: JSON marshalling)agent := bucket.IoRouter()signal := make(chan int)

_, err = agent.Get([]byte("user_1"), func(value []byte, flags uint32, cas uint64, err error) {

fmt.Printf("Got user_1: %v\n", string(value))

signal <- 1})

_, err = agent.Get([]byte("user_2"), func(value []byte, flags uint32, cas uint64, err error) {

fmt.Printf("Got user_2: %v\n", string(value))

signal <- 1})

_, err = agent.Get([]byte("user_3"), func(value []byte, flags uint32, cas uint64, err error) {

fmt.Printf("Got user_3: %v\n", string(value))

signal <- 1})

for i := 0; i < 3; i++ {<-signal

}

gocb is iterating toward interface stability

gocbcore will be a volatile interface for some time

©2015 Couchbase Inc. 37

JSON and Other Data Types

cbgo uses Go’s support for JSON internally

type BeerFull struct {Type string `json:"type"`Id string

`json:"id,omitempty"`BreweryId string

`json:"brewery_id"`Name string `json:"name"`Description string

`json:"description"`Style string `json:"style"`Category string

`json:"category"`Abv float64 `json:"abv"`Ibu float64 `json:"ibu"`Srm float64 `json:"srm"`Upc float64 `json:"upc"`

}

var beer BeerFull_, _, err := bucket.Get(id, &beer)if err != nil {

fmt.Fprintf(w, "Get Error: %v\n", err)return

}

// reference things like thisbeer.Name

Given this struct

You can unmarshal with a Get()

©2015 Couchbase Inc. 38

gocbcore – Where all of the Performance Tricks Live

High level concurrency in Go is great, but… Better still (for go) is avoiding the use of the

primitives

gocbcore is (mostly) lockless Has a concept of a “pipeline” and only two locks

within it Coarse grained “lock” as an RWMutex used only in the

case of death for cleanup Fine grain mutex on the operation list (memdOpMap) when

adding and removing items to the “pipeline”== minimal blocking of your goroutines

DemoTour of a Web Application

©2015 Couchbase Inc. 40

Current Status and Roadmap

Status Current version: 0.2.0 Release early and often to get user feedback, iterate

Roadmap Continue to iterate on 0.x releases adding features… N1QL Support Durability Requirements Complete test coverage && Fully Review Couchbase QE

tests Release 1.0!

Q&A

Thanks!Marty Schoch – @mschoch

Matt Ingenthron – @ingenthr

Get Started with Couchbase Server 4.0: www.couchbase.com/beta

Get Trained on Couchbase: training.couchbase.com