Working With Large Data Efficiently
The read-it-all-into-memory antipattern, and streaming JSON/CSV at constant memory.
The instinct to fs.readFile a file, JSON.parse it, and map over the array works right up until the file is bigger than your heap — then the process OOMs in production while it was fine on your laptop. The senior habit is: if the data is unbounded, stream it.
The numbers that matter
JSON.parseneeds the entire string in memory, plus the resulting object graph — often 2–3x the file size in RAM.- V8 strings cap out around ~512MB; a big file will throw before you even parse it.
- A streaming parser holds only the current record, so a 50GB file costs megabytes, not gigabytes.
// Antipattern: dies on a large file.
const all = JSON.parse(await fs.readFile('huge.json', 'utf8'));
// Better: process line-delimited JSON one record at a time.
for await (const line of readLines('huge.ndjson')) {
handle(JSON.parse(line));
}Prefer line-delimited formats
For pipelines, NDJSON (one JSON object per line) and CSV stream trivially — you never hold more than one record. If you are the producer, emitting NDJSON instead of one giant array is a gift to every downstream consumer.
Example
// Aggregate a multi-gigabyte NDJSON log to per-user totals at flat memory,
// using readline over a read stream. Only the running totals are retained.
const fs = require('node:fs');
const readline = require('node:readline');
async function sumByUser(path) {
const rl = readline.createInterface({
input: fs.createReadStream(path),
crlfDelay: Infinity, // treat \r\n as one line break
});
const totals = new Map();
let lineNo = 0;
for await (const line of rl) {
lineNo++;
if (!line) continue;
let rec;
try { rec = JSON.parse(line); }
catch { console.warn('skipping bad line', lineNo); continue; }
totals.set(rec.userId, (totals.get(rec.userId) ?? 0) + rec.amount);
}
return totals;
}
const totals = await sumByUser('events.ndjson');
for (const [user, amount] of totals) console.log(user, amount);
// Heap stays constant whether the file is 10MB or 100GB.When to use it
- An ETL job processes a 5 GB CSV export by reading it as a stream, transforming each line, and writing results to the database incrementally without exhausting RAM.
- A media transcoder reads video frames as Buffer chunks from a Readable stream and passes them to ffmpeg via stdin rather than loading the whole video.
- A log-analytics service uses `readline` to stream-process a 10 GB access log file line-by-line on a server with only 512 MB of RAM.
More examples
Line-by-line processing with readline
Uses `readline` to process a log file one line at a time -- memory usage stays constant regardless of file size.
const fs = require('fs');
const readline = require('readline');
const rl = readline.createInterface({
input: fs.createReadStream('access.log'),
});
let count = 0;
rl.on('line', line => {
if (line.includes('ERROR')) count++;
});
rl.on('close', () => console.log('Errors found:', count));Streaming large JSON with JSONStream
Uses `JSONStream` to parse a large JSON array element-by-element in a streaming pipeline without loading the full array.
const fs = require('fs');
const JSONStream = require('JSONStream');
const { pipeline } = require('stream/promises');
await pipeline(
fs.createReadStream('large.json'),
JSONStream.parse('items.*'), // emit each item in the array
new (require('stream').Writable)({
objectMode: true,
write(item, enc, cb) { console.log(item.id); cb(); },
})
);Chunk-based file hashing
Hashes a large file by feeding it to `crypto.createHash` chunk by chunk via a stream, using O(1) memory.
const fs = require('fs');
const crypto = require('crypto');
async function hashFile(path) {
const hash = crypto.createHash('sha256');
const stream = fs.createReadStream(path);
for await (const chunk of stream) hash.update(chunk);
return hash.digest('hex');
}
hashFile('huge.iso').then(console.log);
Discussion