encoding data as text - the conference for java & software ...€¦ · peter lawrey . ceo,...
TRANSCRIPT
Encoding Data as Text JAX London 2019 Peter Lawrey CEO, Chronicle Software
Peter Lawrey
18m downloads 2018
Self funded $2.1m/y revenue
Several Tier 1 banking clients
Low latency FIX Engine
< 20 us 99.9%
Low latency Replication
< 30 us
Low latency Persisted messaging
< 1 us 99%
Custom FX Trading With 3 month ROI
Turning numbers into text Unsigned or Signed? Integer, decimal or floating point? Letters: upper, lower or both? Symbols? Built-in formats or additional?
Methods of encoding
Puzzler
When is x == -x ?
Puzzler
When is x == -x ? (byte) 0, (short) 0, ‘\0’, 0, 0L 0.0f, 0.0L
Puzzler
When is x == -x ? (byte) 0, (short) 0, ‘\0’, 0, 0L 0.0f, 0.0L Integer.MIN_VALUE, Long.MAX_VALUE -0.0f, -0.0
Built in integer conversions
String u2 = Long.toString(1000000, 2);
long l2 = Long.parseUnsignedLong(u2, 2);
String u8 = Long.toString(1000000, 8);
long l8 = Long.parseUnsignedLong(u8, 8);
String s10 = Long.toString(1000000);
long l10 = Long.parseLong(s10);
String u10 = Long.toUnsignedString(1000);
long l10b = Long.parseUnsignedLong(u10);
String u16 = Long.toString(1000000, 16);
long l16 = Long.parseLong(u16, 16);
String u36 = Long.toString(1000000, 36);
long l36 = Long.parseLong(u36, 36);
11110100001001000000
3641100
1000000
1000000
f4240
lfls
Bad Joke
Why did the programmer confuse Halloween and Christmas?
Bad Joke
Why did the programmer confuse Halloween and Christmas? Because Oct 31 == Dec 25
Puzzler
Long.MIN_VALUE Double.MIN_VALUE Long.MAX_VALUE Double.MAX_VALUE
Double.NaN 0.0
-0.0 0.0
017.0 017
0x17.0p0 0x17
Puzzler
Long.MIN_VALUE < Double.MIN_VALUE Long.MAX_VALUE < Double.MAX_VALUE
Double.NaN != 0.0
-0.0 == 0.0
017.0 > 017
0x17.0p0 == 0x17
Built in floating point conversions
String pi = Double.toString(Math.PI);
double pid = Double.parseDouble(pi);
String min = Double.toString(MIN_NORMAL);
double mind = Double.parseDouble(min);
String min16 = Double.toHexString(MIN_NORMAL);
double mind16 = Double.parseDouble(min16);
String max = Double.toString(-MAX_VALUE);
double maxd = Double.parseDouble(max);
String max16 = Double.toHexString(-MAX_VALUE);
double maxd16 = Double.parseDouble(max16);
3.141592653589793
2.2250738585072014E-30
0x1.0p-1022
-1.7976931348623157E308
-0x1.fffffffffffffp1023
Built in floating point conversions
String nan = Double.toString(NaN);
double nand = Double.parseDouble(nan);
String neg = Double.toString(NEGATIVE_INFINITY);
double negd = Double.parseDouble(neg);
String pos = Double.toHexString(POSITIVE_INFINITY);
double posd = Double.parseDouble(pos);
String nzero = Double.toHexString(-0.0);
double nzerod = Double.parseDouble(nzero);
NaN
-Infinity
Infinity
-0x0.0p0
A set of doubles
Set<Double> nums = new TreeSet<>();
nums.add(0.0);
nums.add(-0.0);
nums.add(0 / 0.0);
nums.add(1 / 0.0);
nums.add(1 / -0.0);
System.out.println(nums);
A set of doubles
Set<Double> nums = new TreeSet<>();
nums.add(0.0);
nums.add(-0.0);
nums.add(0 / 0.0);
nums.add(1 / 0.0);
nums.add(1 / -0.0);
System.out.println(nums);
[-Infinity, -0.0, 0.0, Infinity, NaN]
Floating point in Java code
// From java.lang.Double
public static final double POSITIVE_INFINITY = 1.0 / 0.0;
public static final double NEGATIVE_INFINITY = -1.0 / 0.0;
public static final double NaN = 0.0d / 0.0;
// 1.7976931348623157e+308
public static final double MAX_VALUE = 0x1.fffffffffffffP+1023;
// 2.2250738585072014E-308
public static final double MIN_NORMAL = 0x1.0p-1022;
// 4.9e-324
public static final double MIN_VALUE = 0x0.0000000000001P-1022;
Puzzler
char ch = '0';
ch /= 0.9;
System.out.println("ch: "+ ch);
prints
ch: 5
Base 64
String base64 =
Base64.getEncoder()
.encodeToString("1000000".getBytes(UTF_8));
String decode = new String(Base64.getDecoder()
.decode(base64),
UTF_8);
MTAwMDAwMA==
1000000
Base 64
String base64 =
Base64.getEncoder()
.encodeToString("00?00>".getBytes(UTF_8));
String decode = new String(Base64.getDecoder()
.decode(base64), UTF_8);
MDA/MDA+
00?00>
Base 64
Base 64
String base64 =
Base64.getUrlEncoder()
.encodeToString("00?00>".getBytes(UTF_8));
String decode = new String(Base64.getUrlDecoder()
.decode(base64), UTF_8);
MDA_MDA-
00?00>
Base 85 – Chronicle Wire
// from Base85LongConverter
private static final String CHARS =
"0123456789" +
":;<=>?@" +
"ABCDEFGHIJKLMNOPQRSTUVWXYZ_" +
"abcdefghijklmnopqrstuvwxyz" +
"\"#$%&'()*+,-./ ";
Base 85 – Chronicle Wire
LongConverter c = Base85LongConverter.INSTANCE;
for (String s : (",a,ab,abc,abcd,ab.de,123=56,1234567," +
"12345678,zzzzzzzzzz,+ko2&)z.H0").split(",")) {
long v = c.parse(s);
assertEquals(s, c.asString(v));
}
Base 85 and Microsecond timestamps
class Data extends AbstractMarshallable {
@LongConversion(Base85LongConverter.class)
long symbol, name;
@LongConversion(MicroTimestampLongConverter.class)
long start, end;
}
Data data = Marshallable.fromString("!Data {\n" +
" symbol: 1st.symbol,\n" +
" name: the.name,\n" +
" start: 2019-10-08T11:49:15.54854,\n" +
" end: 2019-10-08T11:49:15.548557\n" +
"}\n");
System.out.println(data);
Base 85 and Microsecond timestamps
!Data {
symbol: 1st.symbol,
name: the.name,
start: 2019-10-08T11:49:15.54854,
end: 2019-10-08T11:49:15.548557
}
class Data extends AbstractMarshallable {
@LongConversion(Base85LongConverter.class)
long symbol, name;
@LongConversion(MicroTimestampLongConverter.class)
long start, end;
}
Word encoding, Base2K
LongConverter bic = new WordsLongConverter();
for (long i = 1; i <= 1e12; i *= 10)
System.out.println(i + ": " + bic.asString(i));
1: aid
10: ask
100: you
1000: afford
10000: arrive.all
100000: taxi.is
1000000: net.keep
10000000: staying.spot.aim
100000000: cost.trance.due
1000000000: lend.thief.clerk
10000000000: amongst.fort.faith.aid
100000000000: act.urgent.hockey.at
1000000000000: act.dolls.than.argue
Word encoding, Base2K
Uses 2048 common short words without
homophones
Questions
https://chronicle.software Linkedin: Chronicle Performance Engineers
@ChronicleSoftw [email protected]