cbor - the better json

CBORThe Better JSON

www.hazelcast.com@noctarius2k

Who’s that dude?• Chris Engelbert• Manager of Developer Relations @Hazelcast

• Java-Passionate (10+ years)

• Performance• Garbage Collection• JVM / Benchmark Fairytales


Brief look at the history


In the beginning there was darkness…


…and binary data


Debugging binary data is hard


Binary data is just a big blob


Possible with custom tooling


Possible with custom tooling

• unique to this specific use case• involves model classes to be known• utilizes workarounds• generally feels weird :-)

Often:


“I need to understand it!”


Human Readability, Expressiveness



This is not a flame graph!



This is not a flame graph!

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>${maven.failsafe.plugin.version}</version> <configuration combine.self="override"> <redirectTestOutputToFile>true</redirectTestOutputToFile> <argLine> ${jacoco.agent.argLine} -Xms129m -Xmx1G -XX:MaxPermSize=129M -Dxyz.feature1.enabled=false -Dxyz.feature2.enabled=false -Dxyz.feature3.type=none -Dxyz.feature4=true </argLine> <includes> <include>**/TestCast*.java</include> <include>**/IntegrationTestCase*.java</include> </includes> <groups> com.xyz.test.annotation.QuickTest, com.xyz.test.annotation.SlowTest, com.xyz.test.annotation.NightlyTest </groups> </configuration> <executions> <execution> <goals> <goal>integration-test</goal> </goals> </execution> </executions> </plugin>


Too verbose!


Too understandable?!


Just gross?


Grossbusters!


There is no Dana, only Zuul!JSON


Human Readability, Concise


Human Readability, Concisebase64 encoded image


Human Readability, Concise

Does not play “well”with binary data :-(

base64 encoded image


But let’s be honest…


…do we need to store human readable?


Binary but understandable!


So what’s the next evolution step?


C(oncise) B(inary) O(bject) R(epresentation)


Features• Stores JSON-alike content• Extremely concise• Binary• Streamable• Type safe• Schemaless• Highly Extensible• Well defined specification• Standard (by the IETF, RFC7049)


Real World (Size) Example

www.pathway-game.com

• Stores Geometry Information• Mainly Float Data• Often floatToIntBits for Speed• Lots of Sequences• Only a few Dictionaries

Really simple data structure

http://www.pathway-game.com


Real World (Size) Example

2250000

4500000

6750000

9000000

XML JSON BSON CBOR MSGPACK

MSGPACK CBORBSONJSONXML

1.408.737 bytes1.415.215 bytes5.575.783 bytes5.755.188 bytes8.592.541 bytes

1.0x1.0x3.9x4.0x6.0x


Debugging capability built-inDictionary [ ByteString{ ANI0 }=Sequence [ Sequence [ UInt{ 7264 }, UInt{ 4 }, UInt{ 2724 }, UInt{ 37 }, Sequence [ NInt{ -1053818880 }, UInt{ 1077214225 }, UInt{ 0 }, UInt{ 0 }, UInt{ 1068825617 }, ] ] ] ]

Since items carry their type information,writing a quick debugger or using theprovided to-string capabilities of libs


Implementations

• JavaScript• browser• node.js

• Julia• C#• Java• Lua• PHP• Python• Go

• C• C++• Rust• Perl• Ruby• D• Swift• Erlang• Elixir


Implementations

• JavaScript• browser• node.js

• Julia• C#• Java• Lua• PHP• Python• Go

• C• C++• Rust• Perl• Ruby• D• Swift• Erlang• Elixir

CBOR-java:https://github.com/peteroupc/CBOR-Java

Jackson Dataformat:https://github.com/FasterXML/jackson-dataformat-cbor

cbor-java:https://github.com/c-rack/cbor-java

JACOB:https://github.com/jawi/jacob

borabora:https://github.com/noctarius/borabora

https://github.com/peteroupc/CBOR-Java

https://github.com/FasterXML/jackson-dataformat-cbor

https://github.com/c-rack/cbor-java

https://github.com/jawi/jacob

https://github.com/noctarius/borabora


borabora

•Skip-scan parser•Simple, fluent API•Supports object queries•Reading + writing•Supports mutating (hacky prototype)•Extended datatype support


borabora


Not just a blur of color


borabora


but the island ;-)


Time for some demos, isn’t it?


Comparisons


Comparisons

“Alternative Facts”

in the comparisons, are all fake news!


MessagePack• Extremely concise• Available in almost every language• UTF-8 and binary strings• No map keys datatype restriction• Schemaless• Binary

• 32bit size limitation for certain data types• Specification is concise too (~450 lines)• No official standard


MessagePack• Extremely concise• Available in almost every language• UTF-8 and binary strings• No map keys datatype restriction• Schemaless• Binary

• 32bit size limitation for certain data types• Specification is concise too (~450 lines)• No official standard

1:02:0DrawDrawDrawDraw

2:12:22:3


JSON• Human readable• Official standard (ECMA-404)• Subset of JavaScript (strong support)• Schemaless• Kind of type safe

• Human readable• Text-based• Map keys can be strings only• Large bytesize


• Human readable• Official standard (ECMA-404)• Subset of JavaScript (strong support)• Schemaless• Kind of type safe

• Human readable• Text-based• Map keys can be strings only• Large bytesize

1:0Draw2:0DrawDraw

2:12:22:32:4

JSON


• Binary• Available in a lot of languages• Derived from JSON, interoperability 1+• Schemaless• Type safe• Supports in-place updates

• No official standard• Extremely concise specification• Map keys can be strings only• Large bytesize

BSON


• Binary• Available in a lot of languages• Derived from JSON, interoperability 1+• Schemaless• Type safe• Supports in-place updates

• No official standard• Extremely concise specification• Map keys can be strings only• Large bytesize

Draw1:02:0DrawDraw3:0

3:13:23:33:4

BSON


• Human readable• Official standard (W3C REC-xml)• Strong commercial support (e.g. IBM)• Supported everywhere

• Type safe or schemaless• Human readable• Text-based• Very verbose• Large bytesize

XML


• Human readable• Official standard (W3C REC-xml)• Strong commercial support (e.g. IBM)• Supported everywhere

• Type safe or schemaless• Human readable• Text-based• Very verbose• Large bytesize

1:0Draw2:03:0

3:13:23:33:43:5

XML


More alternativesText-based:

• bencode• YAML• …

Binary:• Amazon ION• Smile• Apache Thrift• Apache Avro• Protobuf• …

ASCII based encoding, unaffected by endianess, type safecommonly for configuration, superset of JSON, type safe

skip-scan parsable, type safe, commercial support (Amazon)based on JSON, property-name back-referencesDSL, schema-bound, developed by Facebook, RPC servicesDSL, schema-bound, primarily used by HadoopDSL, schema-bound, developed by Google, RPC services


More information:• http://cbor.io/• http://bsonspec.org/• http://www.json.org/• https://www.w3.org/TR/REC-xml/• http://msgpack.org/ • https://github.com/noctarius/borabora

• https://en.wikipedia.org/wiki/Bencode• https://en.wikipedia.org/wiki/YAML• https://amznlabs.github.io/ion-docs/• https://en.wikipedia.org/wiki/Apache_Thrift• https://en.wikipedia.org/wiki/Apache_Avro• https://en.wikipedia.org/wiki/Protocol_Buffers• https://en.wikipedia.org/wiki/Smile_(data_interchange_format)

Thank You!

http://cbor.io/

http://bsonspec.org/

http://www.json.org/

https://www.w3.org/TR/REC-xml/

http://msgpack.org/

https://github.com/noctarius/borabora

https://en.wikipedia.org/wiki/Bencode

https://en.wikipedia.org/wiki/YAML

https://amznlabs.github.io/ion-docs/

https://en.wikipedia.org/wiki/Apache_Thrift

https://en.wikipedia.org/wiki/Apache_Avro

https://en.wikipedia.org/wiki/Protocol_Buffers

https://en.wikipedia.org/wiki/Smile_(data_interchange_format)


Thank You!Any Questions?

@noctarius2k http://www.sourceprojects.org

http://github.com/noctarius

@hazelcast http://www.hazelcast.com http://www.hazelcast.org

http://github.com/hazelcast

cbor - the better json

Technology