streams, pipes and mega pipes - amazon web...
TRANSCRIPT
![Page 1: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/1.jpg)
Streams, Pipes and Mega Pipes
11.06.2011 (v1)Felix Geisendörfer
![Page 2: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/2.jpg)
@felixge
Twitter / GitHub / IRC
Co-founder transloadit.com
![Page 3: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/3.jpg)
Core Developer
&
Module Author
node-mysql node-formidable
![Page 4: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/4.jpg)
Streams
![Page 5: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/5.jpg)
Image Resizing Server
var http = require('http');var spawn = require('child_process').spawn;
http.createServer(function(req, res) { var params = req.url.split('/'); var path = __dirname + '/' + params[1]; var size = params[2];
var convert = spawn('convert', [path, '-resize', size, '-']); res.writeHead(200); convert.stdout.pipe(res);}).listen(8080);
![Page 6: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/6.jpg)
On Github
felixge/node-convert-example
![Page 7: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/7.jpg)
“Streams are to time as arrays are to space.”-- Jed Schmidt @ JSConf.eu
![Page 8: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/8.jpg)
var fs = require('fs');var stream = fs.createReadStream('/dev/random');
stream.on('data', function(buffer) { console.log(buffer);});
$ node read.js<Buffer 30 85 85 f1 33 f3 4e b2 24 fa f7 dc cf ... ><Buffer 02 36 4e a1 f1 96 2b 3e 0f 2e 26 2e 74 ... >...
Readable Streams
![Page 9: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/9.jpg)
Readable Streams
• Inherits from require(‘stream’).Stream
• Events: ‘data’, ‘end’, ‘close’, ‘error’
• Methods: pause(), resume(), end(), destroy()
• Property: readable (bool)
![Page 10: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/10.jpg)
var fs = require('fs');var stream = fs.createWriteStream('/tmp/test.dat');
stream.write(new Buffer('Hello World\n'));stream.write('How are you?');stream.end();
$ node write.js$ cat /tmp/test.datHello WorldHow are you?
Writable Streams
![Page 11: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/11.jpg)
Writable Streams
• Inherits from EventEmitter
• Events: ‘drain’, ‘error’, ‘close’, ‘pause’, ‘resume’
• Methods: write(), end(), destroy()
• Property: writable (boolean)
![Page 12: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/12.jpg)
Buffers
• Default data type for Streams
• Array-like representation of bytes
• Fixed length
![Page 13: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/13.jpg)
Buffers
var buffer = new Buffer([1, 10, 255]);// <Buffer 01 0a ff>buffer.write('abc');// <Buffer 61 62 63>
![Page 14: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/14.jpg)
Strings
• Ascii, utf-8, binary
• Require copy to send to a socket (slow)
![Page 15: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/15.jpg)
UTF-8
![Page 16: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/16.jpg)
UTF-8var http = require('http');
http.createServer(function(req, res) { var data = ''; req .on('data', function(buffer) { data += buffer; }) .on('end', function() { console.log('received: ', data); });}).listen(8080);
NOT UTF-8 SAFE
![Page 17: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/17.jpg)
UTF-8
• Each character is represented by 1-4 bytes
• Can’t split characters
• Hard to Stream
![Page 18: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/18.jpg)
UTF-8
req.setEncoding('utf-8');
![Page 19: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/19.jpg)
UTF-8var http = require('http');
http.createServer(function(req, res) { var data = ''; req.setEncoding('utf-8'); req .on('data', function(buffer) { data += buffer; }) .on('end', function() { console.log('received: ', data); });}).listen(8080);
![Page 20: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/20.jpg)
UTF-8
• 0xxxxxxx (1 byte)
• 110xxxxx (2 bytes)
• 1110xxxx (3 bytes)
• 11110xxx (4 bytes)
![Page 21: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/21.jpg)
UTF-8: String Decoder
• Scans last 3 bytes of incoming data
• Buffers 1-3 bytes if incomplete character detected
• Only converts/emits the safe part of the string
![Page 22: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/22.jpg)
UTF-8
• UTF-8 is the correct spelling
• utf-8 (lowercase) is ok too (supported by IANA)
• utf8 (no hyphen) is wrong, but supported by node and many other things (browsers)
![Page 23: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/23.jpg)
Pipes
![Page 24: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/24.jpg)
Pipes
Readable WritablePipe
File Upload File
![Page 25: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/25.jpg)
pipe()
• Reads ‘data’ events on source, calls write() on destination
• Also calls end() on destination once source closes
• and more ...
![Page 26: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/26.jpg)
Back Pressure
![Page 27: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/27.jpg)
Back Pressure
• Problem: Writable stream that is slower than the readable stream
• Pipe solves this by calling pause() on the readable stream (if supported)
![Page 28: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/28.jpg)
Beware
var http = require('http');
http.createServer(function(req, res) { req.pause(); setTimeout(function() { req.on('data', function() { ... }); req.resume(); }, 1000);}).listen(8080);
![Page 29: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/29.jpg)
Mega Pipes
![Page 30: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/30.jpg)
Mega Pipes
Readable WritablePipeFull
DuplexPipe
File Gzip Socket
![Page 31: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/31.jpg)
Example: Realtime Encoding
![Page 32: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/32.jpg)
Realtime Encoding
Upload Pipe FFMPEG
Naive Implementation
![Page 33: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/33.jpg)
Realtime Encoding
Upload
FFMPEG
Pipe File (w)
PipeFile (r)
![Page 34: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/34.jpg)
transloadit.com
![Page 35: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/35.jpg)
T-Pipes
![Page 36: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/36.jpg)
T-Pipes
Readable
Writable
Pipe
PipeWritable
Upload
S3
File
![Page 37: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/37.jpg)
T-Pipes
• One readable stream into 2+ writable streams
• Back-pressure will cause slowest writable stream to determine throughput
• Also possible to buffer data to avoid back pressure
![Page 38: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/38.jpg)
EPIPE
![Page 39: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/39.jpg)
EPIPE
Readable NotWritable
Pipe
![Page 40: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/40.jpg)
Writing your own streams
![Page 41: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/41.jpg)
Passthrough Streamvar Stream = require('stream').Stream;var util = require('util');
module.exports = PassthroughStream;function PassthroughStream() { this.writable = true; this.readable = true;}util.inherits(PassthroughStream, Stream);
PassthroughStream.prototype.write = function(data) { this.emit('data', data);};
PassthroughStream.prototype.end = function() { this.emit('end');};
PassthroughStream.prototype.destroy = function() { this.emit('close');};
![Page 42: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/42.jpg)
Passthrough Stream
var Stream = require('stream').Stream;var util = require('util');
module.exports = PassthroughStream;function PassthroughStream() { this.writable = true; this.readable = true;}util.inherits(PassthroughStream, Stream);
![Page 43: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/43.jpg)
Passthrough Stream
PassthroughStream.prototype.write = function(data) { this.emit('data', data);};
PassthroughStream.prototype.end = function() { this.emit('end');};
PassthroughStream.prototype.destroy = function() { this.emit('close');};
![Page 44: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/44.jpg)
Passthrough Streamvar PassthroughStream = require('passthrough_stream');var fs = require('fs');
var source = fs.createReadStream('source.txt');var dest = fs.createWriteStream('dest.txt');var passthrough = new PassthroughStream();
source.pipe(passthrough);passthrough.pipe(dest);
dest.on('close', function() { console.log('done!');});
![Page 45: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/45.jpg)
On Github
felixge/node-passthrough-stream
![Page 46: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/46.jpg)
Examples
• delayed stream
• form-data
• growing-file
![Page 47: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/47.jpg)
Questions?
☝@felixge
![Page 48: Streams, Pipes and Mega Pipes - Amazon Web Servicesfelixge.s3.amazonaws.com/11/nodejs-streams.pdfReadable Streams. Readable Streams ... stream.end(); $ node write.js $ cat /tmp/test.dat](https://reader036.vdocument.in/reader036/viewer/2022081505/5f02aba37e708231d4056c9a/html5/thumbnails/48.jpg)