expanding beyond spl -- more language support in ibm streams v4.1
TRANSCRIPT
Multiple Language Support
1
IBM Streams Enterprise Ready
Distributed Platform
IBM Streams Application Features
SPL Java Scala Python(experimental) …
Integration with Publish & Subscribe
2
SPL Java Scala
SPL Job
Java Job
Scala JobJava Job with
SPL & Scala Elements
SPL Job
Agenda
• Streamsx.topology project
• Java Applications
• Scala Applications
• Data Interchange
• Streams Features
• SPL integration
Streamsx.topology
• A project that supports building streaming topologies (applications) for IBM Streams in different programming languages, such as Java and Scala.
https://github.com/IBMStreams/streamsx.topology
• Open Source
• Agile Development
Frequent releases
• Not tied to IBM Streams product release cycle
• 1.1 release included in IBM Streams 4.1 – November 20th
Java Application API
• Build streaming applications for IBM Streams with Java
Natural for Java developer
Make the common case simple
Supports Java 8
Supports IBM Streams 4.0+
No SPL knowledge required
Streams of Java objects
7
TStream<String>
“#stayshere” “#vegas” “#ibminsight”
TStream<HashTag>
{“#stayshere”, 4567, 2015/10/26 03:42}
{“#ibminsight”, 8364, 2015/10/26 08:23}
{“#pbr”, 1324, 2015/10/25 16:23}
• Streams are sourced, transformed and sinked by functions implemented in Java
• Represented as:
A lambda expression (Java 8)
A method reference (Java 8)
An instance of an interface
• Anonymous class
• Class that implements the interface
Functional Programming
8
ƒ
Functional Filter
9
test( “Goal! #3lions” )= false
tuple discarded
(not present on resultant stream)
Functional Transformation (10/N)
11
apply(“It is not in the stars to hold our
destiny but in ourselves”
“It”, “is ”, “not”, “in”, “the”,
“stars”, “to”, “hold”, “our”,
“destiny”, “but”, “in”,
“ourselves”
)=
Each word will appear on
words stream as a tuple
Scala is a JVM language
• Very good interoperability between the two languages
http://www.codecommit.com/blog/java/interop-between-java-and-scala
• Java can use Scala classes/methods
• Scala can use Java classes/methods
16
IBM Streams Scala Application support
• Scala support is therefore through the Java Application API
• API also provides implicit conversions to allow Scala functions to be used as functional elements
17
https://github.com/IBMStreams/streamsx.topology/wiki/Scala-Support
• Streams instances typically contain multiple applications
Exchanging data using export/import streams
Multiple Applications
20
Ingest
App
Analytic
App
Analytic
App
Analytic
App
Decision
App
Decision
AppIngest
App
• Publish a stream by
Type – Type of the stream
Topic – A string – Recommendation to use MQTT style names
• vehicle/auto/sensor/engine/temp
• Subscribe to a stream by
Type
Topic – Single topic, no MQTT wildcard support
Publish/Subscribe
21
Interchangeable Data Types
22
SPL Java Scala
SPL Tuple
JSON
XML
String
Binary
Java Class
Scala Class
IBM Streams Features
• Windowing
Partitioned
• Parallel Regions
Round-robin, Key or Hash Partitioned
• Submission time values
• Fan-in, Fan-out of streams
• Checkpoint of operator state
• Resource Placement by tags
• Control over mapping of functions to processes (PEs)
• EMBEDDED – Application must only contain Java code
• DISTRIBUTED
• ANALYTICS_SERVICE – Direct submission from Java application to
• STANDALONE
• BUNDLE
• STANDALONE_BUNDLE
• TOOLKIT
• EMBEDDED_TESTER
• STANDALONE_TESTER
• DISTRIBUTED_TESTER
Execution Contexts
26
Creation of a bundle
for later submission
Testing of applications
using JUnit or similar
SPL Streams within the Java Application
• Re-use of the IBM Streams Java Operator API
• Created by:
Invocations of SPL operators
Subscribing to an stream of SPL tuples
• Re-use existing SPL toolkits
C++ primitive, Java primitive, SPL composite
• Create SPL composites to wrap SPL sub-graphs
That can then be invoked from a Java/Scala application
SPL Operator Invocation
29
IBM Streamsexpanded to a wider developer pool
31
IBM Streams Enterprise Ready
Distributed Platform
IBM Streams Application Features
SPL Java Scala …
Integration across languages
• Make the simple - simple
• Re-use of existing SPL toolkits
• Variety of execution mechanisms
Python(experimental)