flinkspector - forward copy · alexander kolb - 2016 - otto group agenda • introduction •...
TRANSCRIPT
FlinkspectorTaming the Squirrel
Alexander Kolb otto group BI
@lofifnc
Alexander Kolb Otto Group BI
@lofifnc
Alexander Kolb - 2016 - otto group
Agenda
• Introduction
• Testing Apache Flink Applications
• Flinkspector • Concept • Input • Expectations • Execution
• Conclusion
• Outlook
Alexander Kolb - 2016 - otto group
Introduction
Alexander Kolb - 2016 - otto group
• Multichannel Retail
• Financial Services
• Services
• 30 countries
• 123 companies
Alexander Kolb - 2016 - otto group
Problem
• Transparency
• Traceability
• Reproducibility
Alexander Kolb - 2016 - otto group
Solutions
• Visualisation
• Datalineage graph / Message tracing
• Unit Testing
Alexander Kolb - 2016 - otto group
Testing Apache Flink Applications
Alexander Kolb - 2016 - otto group
Apache Flink Applicationfinal StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream<String> text = env.socketTextStream("localhost", 9999); DataStream<Tuple2<String, Integer>> words = text.flatMap(new WordCount.LineSplitter()); DataStream<Tuple2<String, Integer>> aWords = words.filter(new StartsWithAFilter()); DataStream<Tuple2<String, Integer>> counts = aWords.keyBy(0).timeWindow(Time.of(2, TimeUnit.MINUTES)).sum(1); counts.print(); env.execute("Wordcount Example");
Alexander Kolb - 2016 - otto group
User Defined Functions
public final class StartsWithAFilter implements FilterFunction<Tuple2<String,Integer>> { @Override public boolean filter(Tuple2<String, Integer> t) throws Exception { return t.f0.startsWith("a"); } }
Alexander Kolb - 2016 - otto group
User Defined Functions
@Testpublic void testAFilter() throws Exception { StartsWithAFilter filter = new StartsWithAFilter(); Assert.assertEquals(true, filter.filter(Tuple2.of("aTest",1))); Assert.assertEquals(false, filter.filter(Tuple2.of("bTest",1))); }
Alexander Kolb - 2016 - otto group
Stream Transformations
public static DataStream<Tuple2<String,Integer>>countAWords(DataStream<Tuple2<String,Integer>> aWords) { return aWords .keyBy(0) .timeWindow(Time.of(2, TimeUnit.MINUTES)) .sum(1); }
Alexander Kolb - 2016 - otto group
Stream Transformations@Testpublic void aWordCountTest() { StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(); env.setParallelism(1); DataStream<Tuple2<String, Integer>> words = env.fromCollection(Arrays.asList( Tuple2.of("a1", 1), Tuple2.of("a2", 1), Tuple2.of("a1", 1) )); DataStream<Tuple2<String, Integer>> results = SocketTextStreamWordCount.countAWords(words); Iterator<Tuple2<String, Integer>> output = DataStreamUtils.collect(results); Assert.assertEquals(output.next(), Tuple2.of("a1", 1)); Assert.assertEquals(output.next(), Tuple2.of("a2", 2)); Assert.assertEquals(output.hasNext(), false); }
Alexander Kolb - 2016 - otto group
Flinkspector
Alexander Kolb - 2016 - otto group
Concept
1. Specify input.
2. Aquire data stream from the input specification.
3. Define expectations for the resulting data stream.
4. Apply the expectations to the produced data stream.
Alexander Kolb - 2016 - otto group
TestStreamEnvironment
Runtime
Subscriber
Test Source Input
Filter
KeyedWindowReducer
Concept
Verifier Test Sink Publisher
Trigger(default)
paired
Alexander Kolb - 2016 - otto group
Expectations
Alexander Kolb - 2016 - otto group
Input
EventTimeInput<Tuple2<String,Integer>> input = EventTimeInputBuilder.startWith(Tuple2.of("a1", 1)) .emit(Tuple2.of("a2", 1), after(1, minutes)) .emit(Tuple2.of("a3", 1), before(1, seconds), times(2)) .emit(Tuple3.of(“a4”, 1), intoWindow(4, seconds)) .repeatAll(after(1, minutes), times(3));
Alexander Kolb - 2016 - otto group
List Based
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4)) .refine().only();
ExpectedRecords.create(asList(1,2,3,4)) .refine().sameFrequency();
(4,1,2,3,3,5)
(4,1,2,3,3,5)
(4,1,2,3,3,5)
Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3,5)
(4,1,2,3,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4)) .refine().only();
ExpectedRecords.create(asList(1,2,3,4)) .refine().sameFrequency();
Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3,5)
(4,1,2,3,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4)) .refine().only();
ExpectedRecords.create(asList(1,2,3,4)) .refine().sameFrequency();
Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3)
(4,1,2,3,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4)) .refine().only();
ExpectedRecords.create(asList(1,2,3,4)) .refine().sameFrequency();
Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3)
(4,1,2,3,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4)) .refine().only();
ExpectedRecords.create(asList(1,2,3,4)) .refine().sameFrequency();
Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3)
(4,1,2,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4)) .refine().only();
ExpectedRecords.create(asList(1,2,3,4)) .refine().sameFrequency();
Alexander Kolb - 2016 - otto group
List Based
ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(notStrict).all();
ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(strict).from(1);
List(4,1,2,3,3,5)
List(5,2,3,4,1)
Alexander Kolb - 2016 - otto group
List Based
List(4,1,2,3,3,5)
List(5,2,3,4,1)
ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(notStrict).all();
ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(strict).from(1);
Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,3,5,4)
List(5,2,3,4,1)
ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(notStrict).all();
ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(strict).from(1);
Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,3,5,4)
List(5,2,3,4,1)
ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(notStrict).all();
ExpectedRecords.create(asList(1,2,3,4)) .refine().inOrder(strict).from(1);
Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4)) .refine() .only() .sameFrequency() .inOrder(strict).from(1);
Alexander Kolb - 2016 - otto group
List Based
ExpectedRecords.create(asList(1,2,3,4)) .refine() .only() .sameFrequency() .inOrder(strict).from(1);
List(1,2,3,4,1)
MatcherBuilder
only
sameFrequency
order
Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4)) .refine() .only() .sameFrequency() .inOrder(strict).from(1);
Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4)) .refine() .only() .sameFrequency() .inOrder(strict).from(1);
Alexander Kolb - 2016 - otto group
List Based
List(2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4)) .refine() .only() .sameFrequency() .inOrder(strict).from(1);
Alexander Kolb - 2016 - otto group
List Based
List(2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4)) .refine() .only() .sameFrequency() .inOrder(strict).from(1);
Alexander Kolb - 2016 - otto group
Assertion Based
new MatchTuples<Tuple2<String,Integer>>("word","count") .assertThat("word", startsWith("a")) .assertThat("count", greaterThan(3)) .onEachRecord();
Alexander Kolb - 2016 - otto group
Assertion Based
new MatchTuples<Tuple2<String,Integer>>("word","count") .assertThat("word", startsWith("a")) .assertThat("count", greaterThan(3)) .oneOfThem() .onEachRecord();
Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(B,…)
.assertThat(C,…)
.eachOfThem()
.onAnyRecord()
A B C
Alexander Kolb - 2016 - otto group
Assertion Based
A B C
.assertThat(A,…)
.assertThat(B,…)
.assertThat(C,…)
.eachOfThem()
.onAnyRecord()
Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(B,…)
.eachOfThem()
.onAnyRecord()
A B C
Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(B,…)
.assertThat(C,…)
.eachOfThem()
.onAnyRecord()
A B C
Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(A,…)
.assertThat(A,…)
.atLeastNOfThem(2)
.onEach()
A B C
Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(B,…)
.assertThat(C,…)
.atExactlyNOfThem(2)
.onEach()
A B C
Alexander Kolb - 2016 - otto group
Assertion Basedpublic class Each<T> extends UntilCombineMatcher<T> { public Each(Iterable<Matcher<? super T>> matchers) { super(matchers); } @Override public String prefix() { return "each of"; } @Override public boolean validWhen(int matches, int possibleMatches) { return matches == possibleMatches; } @Factory public static <T> Each<T> each(Iterable<Matcher<? super T>> matchers) { return new Each<T>(matchers); } }
Alexander Kolb - 2016 - otto group
TestStreamEnvironment
Runtime
Subscriber
Test Source Input
Filter
KeyedWindowReducer
Execution
Verifier Test Sink Publisher
Trigger(default)
paired
Alexander Kolb - 2016 - otto group
Execution
Test SourceInput
Filter
KeyedWindowReducer
Test Sink Publisher
Test Source Input
Filter
KeyedWindowReducer
Test Sink Publisher
1
1
1
1
2
2
2
2
Loca
l Clu
ster
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger(default)
Alexander Kolb - 2016 - otto group
Execution
Loca
l Clu
ster
TS I1
F
KWR
TSi P1
1
1
TS I22
F2
KWR1
1
2
TSi P2
2
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger(default)
Timeout
Alexander Kolb - 2016 - otto group
ExecutionTestStreamEnvironment
Runtime
Subscriber
Verifier Trigger(default)
Loca
l Clu
ster
TS I1
F
KWR
TSi P1
1
1
TS I22
F2
KWR1
1
2
TSi P2
2
Timeout
Alexander Kolb - 2016 - otto group
ExecutionTestStreamEnvironment
Runtime
Subscriber
Verifier Trigger(default)
Loca
l Clu
ster
TS I1
F
KWR
TSi P1
1
1
TS I22
F2
KWR1
1
2
TSi P2
2
Timeout
Alexander Kolb - 2016 - otto group
Loca
l Clu
ster
TSI1
F
KWR
TSiP1
1
1
TSI22
F2
KWR1
1
2
TSiP2
2
ExecutionTestStreamEnvironment
Runtime
Subscriber
Verifier Trigger(default)
Timeout
CLOSE 1
CLOSE 2
Alexander Kolb - 2016 - otto group
TSI1
F
KWR
TSiP1
1
1
TSI22
F2
KWR1
1
2
TSiP2
2
ExecutionTestStreamEnvironment
Runtime
Subscriber
Verifier Trigger(default)
Loca
l Clu
ster
Alexander Kolb - 2016 - otto group
ExecutionTestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
Loca
l Clu
ster
TS I1
F
KWR
TSi P1
1
1
TS I22
F2
KWR1
1
2
TSi P2
2
TimeoutTimeout
Alexander Kolb - 2016 - otto group
ExecutionTestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
Loca
l Clu
ster
TS I1
F
KWR
TSi P1
1
1
TS I22
F2
KWR1
1
2
TSi P2
2
Alexander Kolb - 2016 - otto group
ExecutionTestStreamEnvironment
Runtime
Subscriber
Verifier Trigger(default)
Loca
l Clu
ster
TS I1
F
KWR
TSi P1
1
1
TS I22
F2
KWR1
1
2
TSi P2
2
TimeoutTimeout
Alexander Kolb - 2016 - otto group
ExecutionTestStreamEnvironment
Runtime
Subscriber
Verifier Trigger(default)
Loca
l Clu
ster
TS I1
F
KWR
TSi P1
1
1
TS I22
F2
KWR1
1
2
TSi P2
2
TimeoutTimeout
Alexander Kolb - 2016 - otto group
Conclusion
Alexander Kolb - 2016 - otto group
Conclusion
• Small and concise test cases.
• Extensible dedicated runtime.
• Integration into existing test platforms.
• ProcessingTimeWindows.
• Input specification too expansive.
Alexander Kolb - 2016 - otto group
Outlook
Alexander Kolb - 2016 - otto group
Outlook
• Scala (Test) support.
• Symbolic testing integration.
• System tests on the actual cluster.
Alexander Kolb - 2016 - otto group
https://github.com/ottogroup/flink-spector
Alexander Kolb - 2016 - otto group
ottogroup.comWE ARE HIRING!