Testing an Upgrade

Rehearsing an upgrade on a staging copy of production data, the journeys to walk, and the tests that make it repeatable.

Syntaxvendor/bin/mftf run:group upgrade-smoke

An upgrade is tested by doing it twice: once on a copy of production, then once for real. The copy is the point. A clean install with sample data proves nothing about a catalogue with 40,000 products, eight years of orders and twelve extensions.

The staging copy

Restore a recent production database dump and the pub/media directory onto a host running the target PHP version. Change the base URLs, switch payment and shipping integrations to test credentials or disable them, and point mail at a catcher. Then run the upgrade sequence and time it: that number is the maintenance window you ask for.

The journeys to walk

Four by hand, every time: a guest order through to a confirmation email; a logged-in customer reordering; search and layered navigation on a real category; and in the admin, finding that order, invoicing it and shipping it. Then one journey per custom feature you own. Watch var/log/exception.log and var/report/ while you click.

Making it affordable

Hand-walking does not scale to the third rehearsal. The Magento Functional Testing Framework (MFTF) drives a real browser from XML test files and ships with the core, so a smoke group can be run on demand. Integration tests cover your own modules against a real database.

Finish with bin/magento setup:db:status, which exits 2 if schema or data patches still need to run.

Example

Example · xml
<!-- app/code/Vendor/Checkout/Test/Mftf/Test/VendorUpgradeSmokeTest.xml -->
<tests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="urn:magento:mftf:Test/etc/testSchema.xsd">
    <test name="VendorStorefrontSearchSmokeTest">
        <annotations>
            <features value="Upgrade"/>
            <stories value="Post-upgrade smoke"/>
            <title value="Storefront home page and search still answer"/>
            <description value="Walks the two pages that break first after an upgrade."/>
            <severity value="CRITICAL"/>
            <group value="upgrade-smoke"/>
        </annotations>

        <amOnPage url="/" stepKey="openHome"/>
        <waitForPageLoad stepKey="waitForHome"/>
        <seeElement selector="input#search" stepKey="seeSearchField"/>

        <fillField selector="input#search" userInput="shirt" stepKey="fillSearch"/>
        <click selector="button.action.search" stepKey="submitSearch"/>
        <waitForPageLoad stepKey="waitForResults"/>
        <seeInCurrentUrl url="catalogsearch/result" stepKey="seeResultsUrl"/>
        <dontSee userInput="There has been an error processing your request"
                 stepKey="seeNoErrorPage"/>
    </test>
</tests>

<!--
The file is XML, not PHP: MFTF generates the PHP test from it.
After adding or editing a test:
    vendor/bin/mftf build:project
    vendor/bin/mftf run:group upgrade-smoke
A passing run means a real browser loaded the storefront, searched,
and landed on a results page with no error template.
-->

When to use it

  • Sizing the maintenance window from a timed rehearsal on production-sized data.
  • Proving a custom checkout step still works on the target version before the real upgrade.
  • Giving a merchant a list of journeys to sign off after the upgrade window.
  • Setting up a CI job that runs the store's smoke group against the next Magento version.

More examples

Building the staging copy the rehearsal runs on

Do this before every rehearsal, not once, because a stale copy stops representing production. Two steps are not optional: rewriting the base URLs, or the storefront redirects visitors to the live site, and replacing live integration credentials, or the rehearsal sends real emails and real payment calls. Timing setup:upgrade on production-sized data is the only honest way to size the window, since data patches run row by row.

Example · bash
# On the staging host, before the upgrade sequence.

# Production data, restored
mysql -u magento -p magento_staging < prod-dump.sql
rsync -a prod:/var/www/magento/pub/media/ pub/media/

# Stop it behaving like production
bin/magento config:set web/unsecure/base_url https://staging.example.com/
bin/magento config:set web/secure/base_url https://staging.example.com/
bin/magento config:set admin/security/session_lifetime 28800
bin/magento cache:flush
bin/magento indexer:reindex

# A real admin account, since the production ones may use SSO
bin/magento admin:user:create \
    --admin-user=rehearsal --admin-firstname=Up --admin-lastname=Grade \
    [email protected] --admin-password='<password>'

# Now time the upgrade. This number is the maintenance window.
time bin/magento setup:upgrade

An integration test, in attribute and docblock form

Reach for integration tests for the code you own, where a browser test would be slow and vague about the cause. The framework installs a test database and rolls each test back, so assertions can touch real repositories and real configuration. Recognise both annotation forms: core and older extensions are full of the docblock version, while new tests use attributes, which the IDE can check. These are the tests worth running first after a version bump, because a failure points at a class rather than at a page.

Example · php
<?php
// app/code/Vendor/Theme/Test/Integration/StoreNoticeTest.php

declare(strict_types=1);

namespace Vendor\Theme\Test\Integration;

use Magento\TestFramework\Fixture\AppArea;
use Magento\TestFramework\Fixture\DbIsolation;
use Magento\TestFramework\Helper\Bootstrap;
use PHPUnit\Framework\TestCase;
use Vendor\Theme\ViewModel\StoreNotice;

class StoreNoticeTest extends TestCase
{
    #[DbIsolation(true)]
    #[AppArea('frontend')]
    public function testNoticeIsEmptyWhenUnset(): void
    {
        // Integration tests are the one place the object manager is used
        // directly: there is no constructor to inject into.
        $viewModel = Bootstrap::getObjectManager()->create(StoreNotice::class);

        $this->assertSame('', $viewModel->getNotice());
    }
}

// The same two instructions in the older docblock form. Identical meaning;
// attributes are the form to use for new tests.
//
//     /**
//      * @magentoDbIsolation enabled
//      * @magentoAppArea frontend
//      */
//     public function testNoticeIsEmptyWhenUnset(): void
//
// Run from dev/tests/integration, after copying phpunit.xml.dist to
// phpunit.xml and etc/install-config-mysql.php.dist to install-config-mysql.php:
//     ../../../vendor/bin/phpunit --filter StoreNoticeTest

The automated pass, in the order to run it

This order is deliberate: each step costs more than the one before it, so failing early saves the expensive step. Run the same sequence on the staging rehearsal and in CI against the target version, so the upgrade is tested continuously rather than in one window. MFTF needs a reachable store and admin credentials in dev/tests/acceptance/.env, which is why it never belongs in a pipeline pointed at production.

Example · bash
# 1. Your own units: fastest, no database
vendor/bin/phpunit -c dev/tests/unit/phpunit.xml.dist \
    app/code/Vendor/Theme/Test/Unit

# 2. Integration: real database, your modules only
cd dev/tests/integration && ../../../vendor/bin/phpunit --filter Vendor

# 3. Browser journeys: slowest, closest to the merchant's experience
vendor/bin/mftf doctor           # check the MFTF environment first
vendor/bin/mftf build:project
vendor/bin/mftf run:group upgrade-smoke

# A single named test while fixing one failure
vendor/bin/mftf run:test VendorStorefrontSearchSmokeTest

# 4. Last: has every schema and data patch run?
bin/magento setup:db:status; echo $?   # 0 up to date, 2 setup:upgrade needed

Related

m2 mftfm2 integration testsm2 test cim2 upgrade runm2 upgrade conflicts

Discussion

  • Be the first to comment on this lesson.