Import and Export

The CSV importer, what it validates, how to add your own entity, and why a 100,000-row file needs a different plan.

Syntax<entity name="vendor_supplier_stock" label="Supplier Stock" model="Vendor\Supplier\Model\Import\Stock" behaviorModel="Magento\ImportExport\Model\Source\Import\Behavior\Basic"/>

Import and export move CSV files in and out of Magento's tables. In the admin they live under System > Data Transfer. An import chooses an entity (products, customers, advanced pricing) and a behaviour: Add/Update (add_update), Replace (replace) or Delete (delete). Files are read from var/import, and product images from var/import/images.

Validation

Check Data validates every row before anything is written: unknown columns, empty SKU, values an attribute does not allow. Two strategies decide what happens next - validation-stop-on-errors with an allowed error count, or validation-skip-errors, which imports the good rows and reports the rest.

Your own entity

A module declares an import entity in etc/import.xml and supplies a model extending Magento\ImportExport\Model\Import\AbstractEntity. Three members are required: getEntityTypeCode(), validateRow() and _importData(). Rows arrive in bunches, 100 at a time by default (import/format_v2/bunch_size), so write in batches rather than row by row.

Large files

A 100,000-row import is a different exercise. Put indexers in Update by Schedule first (bin/magento indexer:set-mode schedule), split the file, and expect to run bin/magento indexer:reindex and bin/magento cache:flush afterwards. Scheduled, repeating imports are an Adobe Commerce feature; Magento Open Source has no import scheduler and no import CLI command, so a recurring load needs your own cron job or the asynchronous bulk API.

Export

Export is queued, not immediate: the admin publishes to the topic import_export.export and the consumer exportProcessor writes the file. No consumer, no file.

Example

Example · xml
<!-- app/code/Vendor/Supplier/etc/import.xml -->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_ImportExport:etc/import.xsd">
    <entity name="vendor_supplier_stock"
            label="Supplier Stock"
            model="Vendor\Supplier\Model\Import\Stock"
            behaviorModel="Magento\ImportExport\Model\Source\Import\Behavior\Basic"/>
    <relatedIndexer entity="vendor_supplier_stock" name="catalog_product_price"/>
</config>

When to use it

  • Load a supplier's price list every morning without a third-party extension.
  • Add a custom CSV import for a table your module owns.
  • Work out why an admin export never produced a downloadable file.
  • Plan a 100,000-row product import on a live store without taking the catalogue offline.

More examples

The import entity model

getEntityTypeCode must return the name used in import.xml, or the admin cannot find the model. validateRow records errors through addRowError and returns whether the row survived; the error codes become message keys, and addMessageTemplate gives each one readable text. _importData is called once and loops over bunches, which is why the whole file never has to fit in memory. Values go in through insertOnDuplicate, which binds them, rather than into a hand-built SQL string. Run bin/magento setup:upgrade, cache:flush and setup:di:compile, then the entity appears in the Import dropdown.

Example · php
<?php
// app/code/Vendor/Supplier/Model/Import/Stock.php

declare(strict_types=1);

namespace Vendor\Supplier\Model\Import;

use Magento\Framework\App\Config\ScopeConfigInterface;
use Magento\Framework\App\ResourceConnection;
use Magento\Framework\Stdlib\StringUtils;
use Magento\ImportExport\Model\Import\AbstractEntity;
use Magento\ImportExport\Model\Import\ErrorProcessing\ProcessingErrorAggregatorInterface;
use Magento\ImportExport\Model\ImportFactory;
use Magento\ImportExport\Model\ResourceModel\Helper;

class Stock extends AbstractEntity
{
    private ResourceConnection $resource;

    public function __construct(
        StringUtils $string,
        ScopeConfigInterface $scopeConfig,
        ImportFactory $importFactory,
        Helper $resourceHelper,
        ResourceConnection $resource,
        ProcessingErrorAggregatorInterface $errorAggregator,
        array $data = []
    ) {
        $this->resource = $resource;
        parent::__construct(
            $string,
            $scopeConfig,
            $importFactory,
            $resourceHelper,
            $resource,
            $errorAggregator,
            $data
        );
    }

    public function getEntityTypeCode(): string
    {
        return 'vendor_supplier_stock';
    }

    public function validateRow(array $rowData, $rowNumber): bool
    {
        if (empty($rowData['sku'])) {
            $this->addRowError('skuIsEmpty', $rowNumber, 'sku');
        }

        if (!isset($rowData['qty']) || !is_numeric($rowData['qty'])) {
            $this->addRowError('qtyIsNotNumeric', $rowNumber, 'qty');
        }

        return !$this->getErrorAggregator()->isRowInvalid($rowNumber);
    }

    protected function _importData(): bool
    {
        $table = $this->resource->getTableName('vendor_supplier_stock');

        while ($bunch = $this->getDataSourceModel()->getNextBunch()) {
            $rows = [];
            foreach ($bunch as $row) {
                $rows[] = ['sku' => $row['sku'], 'qty' => (float)$row['qty']];
            }
            $this->_connection->insertOnDuplicate($table, $rows, ['qty']);
        }

        return true;
    }
}

Running a large import safely

Importing with indexers in Realtime mode reindexes after each bunch, which is where the hours go; Update by Schedule writes changelog rows instead and the indexers catch up in one pass. Splitting also gives you a restart point: if part three fails you re-run part three. Check the import history under System > Data Transfer > Import History for the error report of each run, and bring the indexers back to the mode the site normally uses once you are done.

Example · bash
# 1. stop every save from reindexing inline
bin/magento indexer:set-mode schedule

# 2. split the file; 100k rows in one upload will hit the request limits
split -l 20000 --additional-suffix=.csv stock.csv var/import/stock_part_

# 3. import the parts from the admin, Add/Update, skip errors

# 4. catch up afterwards
bin/magento indexer:reindex
bin/magento cache:flush

Export, and the consumer it needs

Since the export was moved onto the queue, the admin no longer streams a file back to the browser: it queues a message on import_export.export, and the exportProcessor consumer writes the CSV into var/export. On a fresh install with no cron and no consumers this is the single most reported import/export problem - the form says success and no file ever appears. Either leave the consumers cron runner on, or supervise exportProcessor with the rest of your consumers.

Example · bash
# the admin Export form publishes a message, then shows:
# "Message is added to queue, wait to get your file soon.
#  Make sure your cron job is running to export the file"

bin/magento queue:consumers:list | grep exportProcessor
# exportProcessor

bin/magento queue:consumers:start exportProcessor --max-messages=10 --single-thread

ls -l var/export/

Related

m2 message queuesm2 consumersm2 cronm2 indexer modesm2 async bulk api

Discussion

  • Be the first to comment on this lesson.