Patches in a Real Deploy

A million-row patch holds the deploy window open; batch it, mark it non-transactional, and make it resumable.

Syntax

Generator::generate($rangeField, Select $select, $batchSize = 100, $batchStrategy = BatchIteratorInterface::UNIQUE_FIELD_ITERATOR)

In a deploy, data patches run during bin/magento setup:upgrade, or during the setup:db-data:upgrade step if the pipeline splits schema from data. That step runs while the storefront is in maintenance mode. setup:upgrade does not enable maintenance mode for you; the deploy script does. So the time your patch takes is time the shop is down.

Why a long patch is worse than it looks

Every data patch is wrapped in a single database transaction unless it implements Magento\Framework\Setup\Patch\NonTransactionableInterface. One transaction over a million rows holds locks and grows the undo log for the whole duration, and other writers wait behind it. The row in patch_list is written only at the end, so a killed deploy leaves you with nothing done and nothing recorded.

Batching

Inject Magento\Framework\DB\Query\Generator and split one select into ranges over an indexed column. Add NonTransactionableInterface so each batch commits as it goes. That removes the long transaction, and it removes the rollback with it: the patch must now be resumable. Make the select self-limiting, for example WHERE tier_code IS NULL, so a re-run continues instead of starting over.

The better answer for very large tables

Keep the deploy short and move the work out of it. A patch can set up the minimum and hand the backfill to a console command, or to a queue message consumed after the site is back up. The loop then becomes: build, setup:di:compile, upgrade, bring the site up, then backfill while customers shop.

Example

<?php
// app/code/Acme/Loyalty/Setup/Patch/Data/BackfillTierCode.php
declare(strict_types=1);

namespace Acme\Loyalty\Setup\Patch\Data;

use Magento\Framework\DB\Query\Generator;
use Magento\Framework\Setup\ModuleDataSetupInterface;
use Magento\Framework\Setup\Patch\DataPatchInterface;
use Magento\Framework\Setup\Patch\NonTransactionableInterface;

class BackfillTierCode implements DataPatchInterface, NonTransactionableInterface
{
    private const BATCH_SIZE = 5000;

    public function __construct(
        private readonly ModuleDataSetupInterface $moduleDataSetup,
        private readonly Generator $queryGenerator
    ) {
    }

    public function apply(): self
    {
        $connection = $this->moduleDataSetup->getConnection();
        $table = $this->moduleDataSetup->getTable('acme_loyalty_member');

        $select = $connection->select()
            ->from($table, ['member_id', 'points'])
            ->where('tier_code IS NULL');

        foreach ($this->queryGenerator->generate('member_id', $select, self::BATCH_SIZE) as $batch) {
            $ids = [];

            foreach ($connection->fetchAll($batch) as $row) {
                $code = match (true) {
                    $row['points'] >= 2000 => 'gold',
                    $row['points'] >= 500 => 'silver',
                    default => 'bronze',
                };
                $ids[$code][] = (int) $row['member_id'];
            }

            foreach ($ids as $code => $memberIds) {
                $connection->update($table, ['tier_code' => $code], ['member_id IN (?)' => $memberIds]);
            }
        }

        return $this;
    }

    public static function getDependencies(): array
    {
        return [];
    }

    public function getAliases(): array
    {
        return [];
    }
}

// Deploy step:
//   bin/magento maintenance:enable
//   bin/magento setup:upgrade --keep-generated
//   bin/magento maintenance:disable
// Each batch commits on its own. Interrupted, the patch re-runs and
// picks up only the rows still NULL, because of the WHERE clause.

When to use it

Backfill a column on a members or quote table with millions of rows without a long maintenance window.
Explain why a release took forty minutes and which patch held the transaction open.
Convert an existing slow patch into a batched, resumable one before it reaches a large store.
Split a deploy into a short upgrade plus a post-release console backfill.

More examples

match and the if-else it replaces

Identical results, so pick by context. match is an expression, which means it can be assigned, passed as an argument or returned directly, and it refuses to fall through: with no matching arm and no default it throws UnhandledMatchError instead of leaving the variable unset. The if-else chain is what you will find in Magento core, which still targets older standards in places. In a batching patch the match keeps the inner loop short enough to read.

<?php
// Shorthand: match, an expression, added in PHP 8.0. Comparisons are
// evaluated top to bottom and the first true arm wins.
$code = match (true) {
    $points >= 2000 => 'gold',
    $points >= 500 => 'silver',
    default => 'bronze',
};

// Longhand: the same thing as statements.
if ($points >= 2000) {
    $code = 'gold';
} elseif ($points >= 500) {
    $code = 'silver';
} else {
    $code = 'bronze';
}

Moving the backfill out of the deploy

The shape to use when the backfill is too slow to sit inside a maintenance window. The batching logic moves into a plain class, TierBackfill, which both this command and a small patch can call; the patch then handles new installs, where the table is empty and finishes instantly, while existing stores run the command after the deploy. The command needs an entry in the module's etc/di.xml under Magento\Framework\Console\CommandList, and bin/magento setup:di:compile before it appears in production mode.

<?php
// app/code/Acme/Loyalty/Console/Command/BackfillTiers.php
declare(strict_types=1);

namespace Acme\Loyalty\Console\Command;

use Acme\Loyalty\Model\TierBackfill;
use Magento\Framework\Console\Cli;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class BackfillTiers extends Command
{
    public function __construct(
        private readonly TierBackfill $backfill,
        ?string $name = null
    ) {
        parent::__construct($name);
    }

    protected function configure(): void
    {
        $this->setName('acme:loyalty:backfill-tiers')
            ->setDescription('Fills tier_code for members that do not have one');
    }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $output->writeln(sprintf('<info>Updated %d members</info>', $this->backfill->run()));

        return Cli::RETURN_SUCCESS;
    }
}

// $ bin/magento acme:loyalty:backfill-tiers
// Updated 1043219 members

Watching progress from another shell

A non-transactional patch commits as it goes, so the first query actually decreases while the deploy runs; with the default transactional patch it would stay flat until the very end, because nothing is visible to another session until the commit. The second query returns nothing until the patch completes. Together they tell you whether a deploy that looks stuck is working or hung, which is the question someone will ask you in the middle of a release.

-- How much is left, while the patch or command is running.
SELECT COUNT(*) AS remaining
FROM acme_loyalty_member
WHERE tier_code IS NULL;

-- Has the patch been recorded yet?
SELECT patch_name FROM patch_list
WHERE patch_name = 'Acme\\Loyalty\\Setup\\Patch\\Data\\BackfillTierCode';

Discussion

Be the first to comment on this lesson.

Why a long patch is worse than it looks

Batching

The better answer for very large tables

Example

When to use it

More examples

match and the if-else it replaces

Moving the backfill out of the deploy

Watching progress from another shell

Related

Discussion