Streams Intro
Streams process large files piece by piece instead of all at once.
Syntax
fs.createReadStream(path)Reading a whole file into memory is fine for small files, but wasteful for large ones. Streams let you process data in small chunks as it flows, using far less memory.
When to use a stream
- Large files (videos, logs, big datasets).
- Data that arrives over time, like a network response.
fs.createReadStream gives you a readable stream that emits 'data' events as chunks arrive.
Example
const fs = require('fs');
const stream = fs.createReadStream('big-log.txt', 'utf8');
let chunks = 0;
stream.on('data', (chunk) => {
chunks++;
console.log(`Chunk ${chunks}: ${chunk.length} chars`);
});
stream.on('end', () => console.log('Finished reading.'));When to use it
- A video streaming server pipes a large MP4 file through a readable stream to the HTTP response so the file is never fully loaded into memory.
- An ETL pipeline reads a 2 GB CSV file as a readable stream, transforms each chunk, and writes results to another stream without memory exhaustion.
- A file-upload handler uses a writable stream to save incoming multipart data to disk chunk by chunk as it arrives from the network.
More examples
Read a large file with streams
Creates a readable stream from a file and prints each chunk to stdout without loading the whole file into memory.
const fs = require('fs');
const readable = fs.createReadStream('big-file.log', { encoding: 'utf8' });
readable.on('data', chunk => process.stdout.write(chunk));
readable.on('end', () => console.log('\nDone'));Write data with a writable stream
Writes 1 000 lines to a file using a writable stream -- more memory-efficient than `writeFile` for large outputs.
const fs = require('fs');
const writable = fs.createWriteStream('output.log');
for (let i = 0; i < 1000; i++) {
writable.write(`Line ${i}\n`);
}
writable.end(() => console.log('Done writing'));Pipe a readable to a writable
Uses `.pipe()` to copy a file by connecting a readable stream to a writable stream with automatic backpressure handling.
const fs = require('fs');
const src = fs.createReadStream('source.txt');
const dest = fs.createWriteStream('copy.txt');
src.pipe(dest);
dest.on('finish', () => console.log('File copied'));
Discussion