Memory and Large Data

memory_limit, reading a large file one line at a time, generators that yield instead of returning a list, and unset as a last resort.

Syntaxfunction rows(string $path): Generator { /* ... */ yield $row; }

memory_limit caps how much memory a single PHP request may use. The bundled php.ini sets 128M, and crossing it is a fatal error that names the number of bytes the failing allocation wanted. Almost every breach has the same shape: the whole thing was loaded into an array first. file(), file_get_contents() and PDOStatement::fetchAll() each build the complete list in memory before your code sees any of it.

Read a stream, not a file

Open the file with fopen() and read one line at a time with fgets(), or one record at a time with fgetcsv(). Memory then holds one line whatever the size of the file. The same idea applies to a result set: fetch() in a loop instead of fetchAll().

Generators

A function containing yield is a generator, available since PHP 5.5. Calling it runs none of the body; it returns a Generator object you can foreach. Each yield hands back one value and pauses there until the loop asks for the next. That keeps the reading code and the consuming code separate without the full list ever existing. yield from, added in PHP 7.0, forwards the values of another generator, which is how reader, filter and limit become three small functions.

unset is not a strategy

Dropping a reference with unset() inside a long loop helps when you are holding a large value you have finished with. Check it with memory_get_peak_usage(true) rather than assuming. If you need it everywhere, the loop is accumulating something it should be streaming.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Importing a CSV far larger than memory_limit without the script being killed.
  • Rewriting a report that calls fetchAll() on a million rows into a loop that holds one row.
  • Streaming log lines through a filter without building an intermediate array at each step.
  • Finding which part of a batch job pushes peak memory past the limit.

More examples

Importing a large CSV in batches

A 2 GB import that never holds more than one row. The generator yields associative records, so the loop reads $record['sensor'] rather than $row[3]. The row length is checked because array_combine throws a ValueError when the two arrays differ in size, and one ragged line should not stop an import. Committing every thousand rows keeps the transaction, and therefore the database's own undo log, from growing as large as the file.

Example · php
<?php

declare(strict_types=1);

function records(string $path): Generator
{
    $handle = fopen($path, 'rb');

    if ($handle === false)
    {
        throw new RuntimeException("cannot open {$path}");
    }

    try
    {
        $header = fgetcsv($handle, escape: '');

        if ($header === false)
        {
            return;
        }

        while (($row = fgetcsv($handle, escape: '')) !== false)
        {
            if (count($row) !== count($header))
            {
                continue;
            }

            yield array_combine($header, $row);
        }
    }
    finally
    {
        fclose($handle);
    }
}

$insert = $pdo->prepare('INSERT INTO readings (sensor, value) VALUES (?, ?)');
$pdo->beginTransaction();
$done = 0;

foreach (records('/var/imports/readings.csv') as $record)
{
    $insert->execute([$record['sensor'], $record['value']]);

    if (++$done % 1000 === 0)
    {
        $pdo->commit();
        $pdo->beginTransaction();
    }
}

$pdo->commit();

A pipeline of generators

Three functions, each holding one line, composed like a shell pipeline. Nothing is read until the foreach asks, and because firstN returns after 20 items the reader stops there as well: a 5 GB log file is read only as far as the twentieth error. Typing the parameters as iterable rather than Generator lets these stages also take a plain array, which makes them easy to test. yield from would be the shorthand if a stage wanted to forward every value of another generator unchanged.

Example · php
<?php

declare(strict_types=1);

function onlyMatching(iterable $lines, string $needle): Generator
{
    foreach ($lines as $line)
    {
        if (str_contains($line, $needle))
        {
            yield $line;
        }
    }
}

function firstN(iterable $items, int $limit): Generator
{
    $seen = 0;

    foreach ($items as $item)
    {
        if ($seen++ >= $limit)
        {
            return;
        }

        yield $item;
    }
}

// lines() is the reader from the main example.
foreach (firstN(onlyMatching(lines('/var/log/app.log'), ' ERROR '), 20) as $line)
{
    echo $line, "\n";
}

Measuring, and the one place unset earns its keep

The unset here is not superstition. Without it, the next iteration builds the new report while the old string is still referenced by $report, so the peak holds two reports rather than one. That is the shape worth the call: a large value, finished with, still referenced until reassignment. memory_get_peak_usage(true) reports what PHP actually took from the system, which is the figure to compare against memory_limit; passing false reports what the allocator handed out, which is always lower. Raising the limit is reasonable for a CLI job and a warning sign in a request handler.

Example · php
<?php

declare(strict_types=1);

// A one-off job may raise the ceiling; a web request should not need to.
ini_set('memory_limit', '512M');

foreach ($batches as $batch)
{
    $report = buildReport($batch);          // several MB of HTML
    file_put_contents("/var/reports/{$batch['id']}.html", $report);

    unset($report);                         // see explanation
}

printf("peak: %.1f MB\n", memory_get_peak_usage(true) / 1048576);

Related

php adv generatorsphp profilingphp cachingmemory_get_peak_usagefgets

Discussion

  • Be the first to comment on this lesson.