Scaling and Availability

What has to be shared once a store runs on more than one web node, and why the database stays a single writer.

Syntaxbin/magento setup:config:set --session-save=redis --session-save-redis-host=redis.internal --session-save-redis-db=2

Scaling Magento is mostly a question of what is shared. Every web node must be interchangeable, so anything one node writes and another reads has to live in a service both can reach.

Sessions, caches, media

Sessions go to Redis, or to Valkey, which speaks the same protocol. That removes the need for sticky sessions at the balancer. The default cache and the full page cache go there too, each in its own database number, by convention 0 and 1 with sessions in 2, because flushing one must not empty the others. pub/media moves to object storage, or to a shared filesystem: an image uploaded through the admin on one node is requested from another seconds later.

Static content and locks

Build static content once and ship the same build to every node, with the same --content-version. If each node generates its own, the versioned asset path differs per node and the browser 404s half the page. A lock provider is how Magento stops two processes doing the same job at once; the default provider is the database, and Adobe recommends ZooKeeper (--lock-provider=zookeeper) once there is more than one node.

The single writer

There is one primary database. Reads can be sent to a replica declared under db/slave_connection in app/etc/env.php, named after its master connection; the indexer connection cannot have one. Splitting the schema across three databases was an Adobe Commerce feature only, and Adobe deprecated it in 2.4.2, so plan for one writer and reduce writes instead. Any change to env.php takes effect after bin/magento cache:flush.

Example

Example · php
<?php
// app/etc/env.php - only the parts that make a store horizontally scalable
declare(strict_types=1);

return [
    'db' => [
        'table_prefix' => '',
        'connection' => [
            'default' => [
                'host' => 'mysql-primary.internal',
                'dbname' => 'magento',
                'username' => 'magento',
                'password' => getenv('MAGENTO_DB_PASSWORD'),
                'active' => '1',
            ],
        ],
        // One replica per master connection, same key name. Reads only.
        'slave_connection' => [
            'default' => [
                'host' => 'mysql-replica.internal',
                'dbname' => 'magento',
                'username' => 'magento_ro',
                'password' => getenv('MAGENTO_DB_REPLICA_PASSWORD'),
                'active' => '1',
            ],
        ],
    ],
    'session' => [
        'save' => 'redis',
        'redis' => ['host' => 'redis.internal', 'port' => '6379', 'database' => '2'],
    ],
    'cache' => [
        'frontend' => [
            'default' => [
                'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                'backend_options' => ['server' => 'redis.internal', 'database' => '0'],
            ],
            'page_cache' => [
                'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                'backend_options' => ['server' => 'redis.internal', 'database' => '1'],
            ],
        ],
    ],
    'lock' => [
        'provider' => 'zookeeper',
        'config' => ['host' => 'zookeeper.internal:2181', 'path' => '/magento/locks'],
    ],
];

When to use it

  • Add a second web node to a store that has outgrown one machine.
  • Stop losing sessions and media every time a node is replaced.
  • Explain to a client why more web nodes will not fix a slow checkout.
  • Give read-heavy reporting traffic its own database replica.

More examples

Writing that configuration with the CLI instead of by hand

setup:config:set edits env.php for you, which matters when several nodes must end up identical. remote-storage:sync copies existing media into the bucket; skip it and the storefront loses every image already uploaded. Run this on one node, then distribute env.php, because it contains no node-specific values.

Example · bash
# Shorthand and longhand of the same change: these commands write exactly the
# array shown above into app/etc/env.php. Editing the file by hand is the
# longhand; setup:config:set is the shorthand, and nothing else differs.
bin/magento setup:config:set \
  --cache-backend=redis --cache-backend-redis-server=redis.internal \
  --cache-backend-redis-db=0 \
  --page-cache=redis --page-cache-redis-server=redis.internal \
  --page-cache-redis-db=1 \
  --session-save=redis --session-save-redis-host=redis.internal \
  --session-save-redis-db=2 \
  --lock-provider=zookeeper --lock-zookeeper-host=zookeeper.internal:2181

# Media off the local disk and onto S3, then copy what is already there.
bin/magento config:set system/media_storage_configuration/media_storage 0
bin/magento setup:config:set --remote-storage-driver=aws-s3 \
  --remote-storage-bucket=store-media --remote-storage-region=eu-west-1
bin/magento remote-storage:sync

bin/magento cache:flush

One static build, deployed to every node

--content-version fixes the /static/version<n>/ segment in every asset URL. Without it each node stamps its own timestamp, and a balancer that sends the HTML and the CSS to different nodes produces a page with no styling for a random share of visitors. Use this whenever static content is generated per node rather than in CI.

Example · bash
# On the build machine, once per release.
VERSION=1.42
bin/magento setup:di:compile
bin/magento setup:static-content:deploy -f en_US de_DE \
  --jobs=4 --content-version="$VERSION"
tar -czf release-$VERSION.tgz vendor generated pub/static app/etc/config.php

# On each web node: same artefact, same content version, same asset URLs.
for node in web-1 web-2 web-3; do
  scp release-$VERSION.tgz "$node:/tmp/"
  ssh "$node" 'cd /var/www/magento && tar -xzf /tmp/release-1.42.tgz \
    && bin/magento cache:flush && systemctl reload php8.4-fpm'
done

The balancer in front of the web nodes

least_conn rather than ip_hash, because sessions are in Redis and no request needs to return to the node that served the last one. X-Forwarded-Proto is not decoration: without it Magento builds http:// URLs behind a TLS terminator and the storefront redirect-loops. Reload with nginx -t && systemctl reload nginx; note that forwarding the header does not by itself make Magento log the real client IP.

Example · nginx
# /etc/nginx/conf.d/balancer.conf on the load balancer
upstream magento_web {
    least_conn;
    server web-1.internal:80 max_fails=2 fail_timeout=10s;
    server web-2.internal:80 max_fails=2 fail_timeout=10s;
    server web-3.internal:80 max_fails=2 fail_timeout=10s;
}

server {
    listen 443 ssl;
    server_name store.example.com;

    location / {
        proxy_pass http://magento_web;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Related

m2 stackm2 cache redism2 varnishm2 perf scalem2 kubernetes m2

Discussion

  • Be the first to comment on this lesson.