Solved

Process runs out of memory reading a 2 GB CSV

Asked by Samuel Okafor · 557 views · 2 replies

Samuel Okafor8 messages

Works on the sample file, dies on the real one:

const data = await fs.promises.readFile('export.csv', 'utf8');
for (const line of data.split('\n')) {
  await handle(line);
}

Error is JavaScript heap out of memory. I can raise --max-old-space-size and it gets further, but that feels like the wrong answer.

2 replies

Tomasz WiśniewskiSolution

It is the wrong answer — you would be buying headroom for a file that will be 3 GB next quarter.

readFile holds the whole file in memory, and split('\n') then allocates a second copy as an array of strings. So you need roughly twice the file size before any work starts. Read it as a stream instead, a line at a time:

import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

const rl = createInterface({
  input: createReadStream('export.csv'),
  crlfDelay: Infinity,          // treat \r\n as one break
});

for await (const line of rl) {
  await handle(line);
}

Memory is now a buffer and one line, whatever the file size. The for await loop also applies backpressure for free: the stream pauses while handle is awaited, so a slow database write cannot let the reader run ahead and queue the file in memory.

Link
Hannah Lindqvist

If the CSV has quoted fields containing newlines, do not split on lines at all — a quoted value can legally span them and you will silently corrupt those rows. A streaming CSV parser handles the quoting and gives you records instead of lines. Same memory profile, correct output.

Link