writing stream processors in kotlin...•more than 80% of our kafka related source code is kotlin...

Writing Stream Processors in Kotlin

KotlinConf 2018 - Amsterdam

Who of you uses ?

A quick look over the shoulder

3

Kafka in a Nutshell

Sou

rce: W

ikip

edia

5

What is so special about Kafka?

• Persistent

• Time and Log compaction

•Guaranteed order per partition

Messages

•Have a byte[] key

•Have a byte[] value

• Contain Metadata like a timestamp and the partition

Image from http://kafka.apache.org/documentation/

Event Streams

• null-key and a bare or serialized value

• Compaction of the topic by time

• Examples: Clickstreams, Logs, IoT events

Image from http://cloudurable.com/blog/kafka-architecture-log-compaction/index.html

Log Compaction

Stream Processors

• Imagine you could do Kotlin Sequences on an infinite list of messages

• Stateful

• Parallel

• And the output is one or more Kafka topic(s)

What now?

• I do home automation, a lot

• I use homee, which aggregates multiple protocols

(Z-Wave, ZigBee, etc)

• It has no official API, yet

• But … They (unintentionally) made one of their repos public on Github

• And now I have an API Blueprint file ☺

• Guess what? It’s a websocket!

Let’s get started

• The websocket issues text messages when something happens

• Attribute Changes of any connected device

(motion, temperature, light, door opens, etc.)

• System Events (user created, system update, etc.)

• We use Kafka Connect to build what is called a “source” and pipe these events

into Kafka.

Let’s look at some code(no live demo, sorry)

Now, why Kotlin?The Java code did not look that bad…

An engineering view

• Less verbose, better to read

• Extension functions enable composable, high level abstractions

• Extensions can be called from the Java folks

• One can write libraries with business abstractions, reusable within

the company, that blend into the original API

A team leads view

• Onboarding people coming from non-JVM languages to Kotlin is easier than

Java

• The education material available is very rich, based on current teaching

standards

• People love the language and the feeling of productivity

• Unmodifiable / Immutable helps a lot of people to write more stable code,

• resulting in better outcome and higher motivation to go on with learning

• because “it just works es expected”

• You can write good code even without knowing the language in-depth

Kotlin at trivago

• More than 80% of our Kafka related source code is Kotlin

• Kafka Connect Sinks, Transforms, Converters

• Stream Processors

• Custom Solutions, based on Spring Boot 2, Spring Kafka, Spring Integration

• My current team writes client facing REST and GRPC services based on Spring Boot 2

entirely in Kotlin

• Our recently relaunched Android App is written entirely in Kotlin

• Some of our B2B focused teams started to rewrite their PHP applications in Kotlin

Curious about how we use Apache Kafka?

Kafka and Debezium at trivago (code.talks 2017)

https://speakerdeck.com/xenji/kafka-and-debezium-at-trivago-code-dot-talks-2017-edition