Err....

QuarksPreon

Preon

Adeclarativedatabindingframeworkforbinaryencodeddata

Binary Encoded Data?

Anyfileforwhich'file‐bi{filename}'doesnotreturnamimetypestartingwith'text/'

Non‐binary:text/plain(*.txt)text/html(*.html)text/xml(*.xml)

Binary:application/pdf(*.pdf)application/octet‐stream(*.mp4)image/png(*.png)

Not only byte stream

Alwaysoctects(8bitvalues)

8bits8bits

But also bit stream

Alwaysoctects(8bitvalues)

5bits 5bits

Compressed Data

2kg

1021pages

Network Traffic

TomTom Map Files

Approx.300pagesofC++sourcecode,justfordecodingonly...

Why?

Binary45.71%

Non‐binary54.29%

Binaryvs.non‐binarydistributiononarandomdirectoryonmysystem

Why?

...andonmywife'ssystem

Binaryfilesonly100%

Challenges

● Decodingfrombitstreamnotforthefaint‐hearted

● Encodingtobitstreamnotforthefaint‐hearted

● Hardtomaintain

● Hardtoextend

● Javadoesn'thelp(Bug4504839)

● Decoderandencodereasilygooutofsync

● Documentationandsoftwareeasilygooutofsync

What if....

..this could all be solved easily?

Preon Ambitions

● Declarativelymapdatastructuretoencodingformat

● Getthedecoder/encoder/documentation,freeofcharge

5-Second User Guide

01Filefile=newFile(...);02Codec<BitMap>codec=Codecs.create(BitMap.class);03BitMapbitmap=Codecs.decode(codec,file);

DecodingaBitMap:

What just happened?

01Filefile=newFile(...);02Codec<BitMap>codec=Codecs.create(BitMap.class);03BitMapbitmap=Codecs.decode(codec,file);

CreateaCodecforinstancesofBitMap

...anduseittodecodeaBitMap.

Find the specification

ThedatastructureISthespecification:classBitMap{@Boundintwidth;@Boundintheight;@BoundintnrColors;@BoundList(size=”nrColors”)Color[];@BoundList(size=”width*height”)byte[]pixels;}

classColor{@Boundintred;@Boundintgreen;@Boundintblue;}

Demo Decoding

But what is actually happening?

CodecisnothingbutafacacetoachainofCodecs.

EachCodeconlyunderstandsitsowntask.

CodecswilldelegatetootherCodecs

Codecs

Encapsulateeverythingthereistoknowaboutthemappingbetweenin‐memoryrepresentationandencodedrepresentation

Demo Documentation

So, now....

ForgeteverythingIjusttoldyou

Preon just works

Annotations

● @Bound● @BoundList● @BoundString● @BoundNumber● @BoundObject● @BoundExplicitly● @If● @LazyLoading● @LengthPrefix● @TypePrefix● @ByteAlign● @Slice

Types

int,byte,short,long,Integer,Byte,Short,Long,boolean,Boolean,String,List<T>,T[],type‐safeenums,Object

Expressions

attributes:.{name}items:[{number}],or[{expr}]arithmetic:+,‐,/,*,^combinatorial:&&,||relational:>=,<=,==,<,>literals:'foobar',0x0f,0b01101

Convention over Configuration

//DefaultsettingsforintCodec://32‐bits,littleendian@Boundintfoo;

//Readanint,butconstructtheint//from5bitsonly@BoundNumber(size=”5”)intfoo;

Expressions (1)

/***Aniconofmaximal15*15pixels.Eachcolor*hasacolorintherange0‐255.*/publicclassIcon{@BoundNumber(size=”4”)intheight;@BoundNumber(size=”4”)intwidth;@BoundList(size=”height*width”)byte[]pixels;}

ByacceptingStringsinsteadofbooleansandintegers,Preonallowsyoutopassinexpressions.

Expressions (2)

@BoundString(size=”{expr}”,...)

@BoundList(size=”{expr}”,offset=”{expr}”,...)

@BoundNumber(size=”{expr}”,...)

@Slice(“{expr}”)

...

Boolean expressions

@BoundprivateintmapVersion;

@If(“mapVersion>=700”)@BoundprivateMapFlagsflags;

//Butalso://mapVersion+1>=700//mapVersion>3&&mapVersion<300//etc.

References

Backwardreferencesonly

Referencethe“outer”object

addresses[1].streetouter.driversLicenses[1].statedriveresLicenses[nrDriverLicenses‐1].state

Variable Introductions

@Boundint[]offsets;@BoundList(offset=”offsets[index]”,...)List<Node>nodes;

● PreonwillinjectListimplementation

● ListimplementationwillloadNodeslazily,ondemand

● Callingnodes.get(3)willcausetheListimplementationto

– Calculatethenode'sposition:offsets[index=3]=120

– CallCodec<Node>tostartdecodingfromthatpoint

Introduction

Inheritance

JavaClasses PreonPerspective

Codecs class is your friend

static <T> T decode(Codec<T> codec, byte[] buffer)static <T> T decode(Codec<T> codec, ByteBuffer buffer)static <T> T decode(Codec<T> codec, File file)

static <T> Codec<T> create(Class<T> type) static <T> Codec<T> create(Class<T> type, CodecFactory...

factories)static <T> Codec<T> create(Class<T> type, CodecDecorator...

decorators)

static <T> void document(Codec<T> codec, ArticleDocument document)

static <T> void document(Codec<T> codec, DocumentType type, OutputStream out)

static <T> void document(Codec<T> codec, DocumentType type, File file)

Preon Layers

Anexpressionlanguagecapableofrenderingitselftohumanreadabletext.(limbo.sourceforge.net)

Afluentinterfaceforgeneratingdocuments.(pecia.sourceforget.net)

BitBufferabstractions

Databinding

Preon License

Apache2.0

Apache2.0

GPL+ClasspathException

GPL+ClasspathException

If not Preon, then what else?

Declarativeapproachisnotnew:– BSDL(BitstreamSyntaxDescriptionLanguage)– Flavor(http://flavor.sourceforge.net/)– XFlavor– BFlavor(http://multimedialab.elis.ugent.be/bflavor/)

Noneofthemwouldworkedinourcase:– Overlycomplicated– Nosolidimplementation– Assumptionsaresurreal:

● “everythingwilljustfitinmemory”● “therewillonlybeasinglethread”● “ourcurrentfeaturesetisallyoueverneed”

The Preon Answer

“everythingwilljustfitinmemory”

Preoniscapableofusingmemory‐mappeddata.(DefaultBitBufferimplementationwrapsaroundByteBuffer,andhencealsoMappedByteBuffer.)

“therewillonlybeasinglethread”

InPreon,everythreadcanhaveitsownreferencetothecurrentpositionintheBitBuffer.

“ourcurrentfeaturesetisallyoueverneed”

Preonhasincrediblyopen:implementCodecFactoryorCodecDecoratortocreateyourowncodecsbasedontypeinformationorannotations.

CodecFactory

interface CodecFactory { <T> Codec<T> create(AnnotatedElement metadata, Class<T> type, ResolverContext context);}

AnnotationsontheobjectexpectingdatatobedecodedbytheCodec.

Thetypeofobjecttobedecoded.

Theobjectforconstructingreferences.

returnsnullifitdoesnothaveawaytoconstructaCodec

CodecFactory Usage

● CodecFactoriescanbepassedtoCodecs.create(...)

● ...whichwillcausetheCodecsclasstoconsiderthesefactorieswhenconstructingthevariousCodecs.

● Internally,abigchainofcommands

Current Status

● http://preon.sourceforge.net/

● Interfacesfairlystable

● Currentversion1.0‐SNAPSHOT

● CompleteJavaclassexamplebeforefirstreleasecandidate

● Bugs...

● Iwantyou

Future work

● Theencodeoperation(after1.0)

● Betterdebugging

● Annotationdrivensupportforothercompressiontechniques

● Morehyperlinkinginthedocumentation

● Betteralgebraicsimplificationofexpressions

● DescriptionsfromJavaDoc

If there is only a couple of things...

● XMLisjustalamewayofbinaryencoding

● PreoncutsouttheInfosetmodel,preventingunnecessarytransformations

● Preonmakesbinaryencodingeasy

● Preonisextensible

● Preon(probably)scalesquitewell

● PreonisfriendlytoJavadevelopers

Confused?