Download - BINARY DATA ADVENTURES IN BROWSER JAVASCRIPT
INTRO• Background • Evolution in the browser • The Binary Arsenal in JS (type system, inputs, outputs) • Solving Cool, Practical Problems
001101100111001001010101001000011
101010111010000010101001010101001
010100101110101010010101010100101
0101010111110010101000101011101001
01010101111010000001010101000011
010110010101010101111000010101010
TRADITIONALLY• Outside of JS:
Low level stuff: Compression (gzip/deflate), Bitfields (ACLs), Networks (CRC32), Graphics (algorithms, positioning), ...
• Inside of JS: Solving weird issues
$http.post('/someUrl', {score:42}). success(function(data, status, headers, config) { // not today :-( }). error(function(data, status, headers, config) { // data=Object {‘error’: ‘Float64 is not int’} }
$http.post('/someUrl', {score:42}). success(function(data, status, headers, config) { // not today :-( }). error(function(data, status, headers, config) { // data=Object {‘error’: ‘Float64 is not int’} }
In JS, all numbers are of type Float64 But all bitwise ops are Int32!
Bitwise OR: number | 0 Bitwise NOT: ~~number Shift: number >> 0
parseInt: parseInt(number, 10) floor: Math.floor(number) modulo: number - number % 1
BROWSERS, EVOLVED• Type system: Blob, ArrayBuffer, TypedArray, DataView • Inputs:
Input methods: XHR 2, File API, Canvas, WebSockets, WebRTC, ... • Outputs:
MSE, Canvas, WebSockets, WebRTC, WebGL, ...
WHAT MAKES TYPED ARRAYS FAST
• Passed to native interfaces completely AS IS (direct memory access) • Native “endianness” - watch out! • DataView adapter
Byte order 0x12345678 Hex inside a Uint32Array of 4
Little-endian: 78 56 34 12
12 34 56 78Big-endian:
STREAMING VIDEO WITH JS
• No plugins (Netflix: silverlight, Facebook: flash) • MSE: Pure JS DASH/HLS (YouTube) • DRM Support (EME) • MSE + WebRTC: Bittorrent, in the browser! (WebTorrent)
‘USE ASM’sys/libkern/strlen.c size_t strlen(const char *str) { const char *s; for (s = str; *s; ++s); return(s - str); }
asm.js function strlen(ptr) { ptr = ptr|0; var curr = 0; curr = ptr; while (MEM8[curr]|0 != 0) { curr = (curr + 1)|0; } return (curr - ptr)|0; }
a subset of JS for handling numbers faster
• OdinMonkey - AOT in FF (fallback to IonMonkey JIT) • Support by all major browsers (to an extent) • Typed Array used as “memory” • All add/sub are 32 bit (number | 0) • DI: function(stdlib, foreign, heap)
‘USE ASM’
KRIPKEN TO THE RESCUEC/C++ > LLVM > Emscripten > asm.js!
• Huge code bases (SQLite, BananaBread,J2ME VM, …)- close to impossible by hand
• Enjoy Clang/LLVM optimizations- decades of work done to optimize compiled code
• OpenGL > WebGL “for free” - directly map a lot of OpenGL ES 2.0 command to WebGL - not only for graphics rendering, also for GPU offloading
• Use Docker to run Emscripten - lots of software (emscripten, emscripten-fastcomp, emscripten-
fastcomp-clang, llvm clang) - build your own or use one from DockerHub
KRIPKEN TO THE RESCUE
• Obtain bytes of video (and audio) by downloading them
• Decode the video (and audio) using native decoders in the browser/OS
• Animate the frames to create the video experience
VIDEO TAG EMULATION
• Obtain bytes of video (and audio) by downloading them > XHR 2.0, response type “arraybuffer”
• Decode the video (and audio) using native decoders in the browser/OS > Cross compile a native decoder to JS using Emscripten
• Animate the frames to create the video experience > For each decoded pixel, create a 2D Texture, transform it to RGBA using a fragment shader, and render it with WebGL using canvas (“experimental-webgl”).
VIDEO TAG EMULATION
YUV -> RGB• Luma + (2 * Chroma) vs
3 * (Chroma + Luma)
• Different needs, different inputs
• Color space conversion heavily computational (floating point coefficients matrix multiplication)
function yuv2canvas(buffer, width, height) { var lumaSize = width * height; var chromaSize = lumaSize >> 2;
webGLCanvas.YTexture .fill(buffer.subarray(0, lumaSize));
webGLCanvas.UTexture .fill(buffer.subarray(lumaSize, lumaSize + chromaSize));
webGLCanvas.VTexture .fill(buffer.subarray(lumaSize + chromaSize, lumaSize + 2 * chromaSize));
webGLCanvas.drawScene(); }
4:2:0
=
+
precision highp float; varying highp vec2 vTextureCoord; uniform sampler2D YTexture; uniform sampler2D UTexture; uniform sampler2D VTexture; const mat4 YUV2RGB = mat4 ( 1.1643828125, 0 , 1.59602734375, -.87078515625, 1.1643828125, -0.39176171875, -0.81296875 , .52959375, 1.1643828125, 2.017234375 , 0 , -1.081390625, 0 , 0 , 0 , 1 );
void main(void) { gl_FragColor = vec4(texture2D(YTexture, vTextureCoord).x, texture2D(UTexture, vTextureCoord).x, texture2D(VTexture, vTextureCoord).x,
1) * YUV2RGB; }
<SCRIPT TYPE="X-SHADER/X-FRAGMENT">