Writing Stream Processors in Kotlin
KotlinConf 2018 - Amsterdam
Who of you uses ?
A quick look over the shoulder
3
Kafka in a Nutshell
Sou
rce: W
ikip
edia
5
What is so special about Kafka?
• Persistent
• Time and Log compaction
•Guaranteed order per partition
Messages
•Have a byte[] key
•Have a byte[] value
• Contain Metadata like a timestamp and the partition
Image from http://kafka.apache.org/documentation/
Event Streams
• null-key and a bare or serialized value
• Compaction of the topic by time
• Examples: Clickstreams, Logs, IoT events
Image from http://cloudurable.com/blog/kafka-architecture-log-compaction/index.html
Log Compaction
Stream Processors
• Imagine you could do Kotlin Sequences on an infinite list of messages
• Stateful
• Parallel
• And the output is one or more Kafka topic(s)
What now?
• I do home automation, a lot
• I use homee, which aggregates multiple protocols
(Z-Wave, ZigBee, etc)
• It has no official API, yet
• But … They (unintentionally) made one of their repos public on Github
• And now I have an API Blueprint file ☺
• Guess what? It’s a websocket!
Let’s get started
• The websocket issues text messages when something happens
• Attribute Changes of any connected device
(motion, temperature, light, door opens, etc.)
• System Events (user created, system update, etc.)
• We use Kafka Connect to build what is called a “source” and pipe these events
into Kafka.
Let’s look at some code(no live demo, sorry)
Now, why Kotlin?The Java code did not look that bad…
An engineering view
• Less verbose, better to read
• Extension functions enable composable, high level abstractions
• Extensions can be called from the Java folks
• One can write libraries with business abstractions, reusable within
the company, that blend into the original API
A team leads view
• Onboarding people coming from non-JVM languages to Kotlin is easier than
Java
• The education material available is very rich, based on current teaching
standards
• People love the language and the feeling of productivity
• Unmodifiable / Immutable helps a lot of people to write more stable code,
• resulting in better outcome and higher motivation to go on with learning
• because “it just works es expected”
• You can write good code even without knowing the language in-depth
Kotlin at trivago
• More than 80% of our Kafka related source code is Kotlin
• Kafka Connect Sinks, Transforms, Converters
• Stream Processors
• Custom Solutions, based on Spring Boot 2, Spring Kafka, Spring Integration
• My current team writes client facing REST and GRPC services based on Spring Boot 2
entirely in Kotlin
• Our recently relaunched Android App is written entirely in Kotlin
• Some of our B2B focused teams started to rewrite their PHP applications in Kotlin
Curious about how we use Apache Kafka?
Kafka and Debezium at trivago (code.talks 2017)
https://speakerdeck.com/xenji/kafka-and-debezium-at-trivago-code-dot-talks-2017-edition