Streaming Large Files

Stream an upload straight into storage with FileStreamInterceptor or a custom storage engine instead of buffering it in memory.

SyntaxFileStreamInterceptor(fieldName: string, localOptions?: Omit<MultipartOptions, 'dest' | 'storage'>): Type<NestInterceptor>

Memory storage holds the entire file in the heap. Ten concurrent 500 MB uploads is 5 GB of resident memory, and the process dies before the tenth finishes. Disk storage keeps it off the heap, but if the file is bound for S3 anyway you have written it twice: once to disk, once back out.

Streaming in

On the Express adapter the handler can never stream the upload, because multer has already consumed the request inside the interceptor. To touch the bytes as they arrive you go one level down and write a multer storage engine: an object with _handleFile(req, file, callback), where file.stream is the only place the bytes exist, plus _removeFile for cleanup. Whatever you pass to the callback is merged onto the file object the handler receives.

From NestJS 12.1 the Fastify adapter has a direct route. FileStreamInterceptor from @nestjs/platform-fastify/multipart hands the handler file.stream unread. It accepts one file, and a file in any other field fails the request with a 400. The size is not known up front, so MaxFileSizeValidator and the magic-number check do not apply; instead limits.fileSize errors the stream mid-transfer with a PayloadTooLargeException.

Streaming out

Sending a file back is StreamableFile from @nestjs/common, built from a Readable or a Uint8Array.

Shorthand and longhand

These produce the same response: return new StreamableFile(stream, { type, disposition }), or the long form with @Res() res, res.set(...) and stream.pipe(res). The short form keeps the handler testable and adds setErrorHandler() on Express.

Example

Example · typescript
// src/uploads/uploads.controller.ts  (Fastify adapter, NestJS 12.1+)
import { randomUUID } from 'node:crypto';
import { Controller, Post, UploadedFile, UseInterceptors } from '@nestjs/common';
import {
  FileStreamInterceptor,
  MultipartFileStream,
} from '@nestjs/platform-fastify/multipart';
import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';

const s3 = new S3Client({});

@Controller('uploads')
export class UploadsController {
  @Post()
  @UseInterceptors(
    // Hands the handler file.stream unread: nothing is buffered into the
    // heap and nothing is written to disk first. dest and storage are not
    // accepted here, because this interceptor does not store anything.
    FileStreamInterceptor('file', {
      limits: { fileSize: 2 * 1024 * 1024 * 1024 }, // 2 GiB
    }),
  )
  async upload(@UploadedFile() file: MultipartFileStream) {
    const key = `uploads/${randomUUID()}`;

    const upload = new Upload({
      client: s3,
      params: {
        Bucket: process.env.UPLOAD_BUCKET,
        Key: key,
        Body: file.stream, // a Readable, so this becomes a multipart upload
        ContentType: file.mimetype, // still the client's claim
      },
      partSize: 8 * 1024 * 1024, // 5 MiB is the S3 minimum part size
      queueSize: 4, // parts in flight at once
    });

    // Memory used is roughly partSize * queueSize, whatever the file size.
    // If limits.fileSize is exceeded the stream errors here with
    // PayloadTooLargeException, and lib-storage aborts the parts already
    // sent because leavePartsOnError defaults to false.
    await upload.done();

    return { key, originalName: file.originalname };
  }
}

When to use it

  • Accept a multi-gigabyte video upload on a container with 512 MB of memory.
  • Push an upload straight into S3 without touching the local filesystem.
  • Write an incoming CSV to disk for a background import job to pick up.
  • Serve a generated PDF as a download with the right filename and a 404 if it is missing.

More examples

A streaming storage engine for the Express adapter

This is the only way to stream an upload on Express, because FileInterceptor finishes multer before your handler is called. The engine runs inside the interceptor step, so by the time the handler executes the object is already in the bucket and file.path holds the key. Reach for it when a single node must accept uploads larger than you are willing to hold in memory, and keep _removeFile honest: it is what stops a multi-file request from leaving half its files behind.

Example · typescript
// src/uploads/s3.storage.ts
import { randomUUID } from 'node:crypto';
import type { Request } from 'express';
import type { StorageEngine } from 'multer';
import { DeleteObjectCommand, S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';

export function s3Storage(client: S3Client, bucket: string): StorageEngine {
  return {
    // multer calls this once per file, while the request is still arriving.
    // file.stream is the only place the bytes exist: read it or lose them.
    _handleFile(req: Request, file, callback) {
      const key = `uploads/${randomUUID()}`;

      new Upload({
        client,
        params: { Bucket: bucket, Key: key, Body: file.stream },
      })
        .done()
        // Properties passed here are merged onto the file object the
        // handler later receives, so file.path becomes the S3 key.
        .then(() => callback(null, { path: key }))
        .catch(callback);
    },

    // multer calls this to undo a stored file when a later part of the
    // same request fails, so a rejected request leaves no orphans.
    _removeFile(req: Request, file, callback) {
      client
        .send(new DeleteObjectCommand({ Bucket: bucket, Key: file.path }))
        .then(() => callback(null))
        .catch(callback);
    },
  };
}

// Wired in like any other engine:
//   MulterModule.register({
//     storage: s3Storage(new S3Client({}), process.env.UPLOAD_BUCKET!),
//     limits: { fileSize: 1024 * 1024 * 1024 },
//   })

Streaming an upload to disk with pipeline

The simplest streaming sink, and the one to use when the file is processed on the same machine: a CSV import, a video to transcode, a backup to verify. Memory stays at one buffer regardless of file size. Always use pipeline from node:stream/promises rather than .pipe() here, because .pipe() leaves the write stream open and the partial file on disk when the source errors. Delete the partial file in a catch block if a half-written upload would ever be mistaken for a complete one.

Example · typescript
// src/uploads/archive.controller.ts  (Fastify adapter)
import { createWriteStream } from 'node:fs';
import { mkdir } from 'node:fs/promises';
import { randomUUID } from 'node:crypto';
import { join } from 'node:path';
import { pipeline } from 'node:stream/promises';
import { Controller, Post, UploadedFile, UseInterceptors } from '@nestjs/common';
import {
  FileStreamInterceptor,
  MultipartFileStream,
} from '@nestjs/platform-fastify/multipart';

@Controller('archives')
export class ArchiveController {
  @Post()
  @UseInterceptors(
    FileStreamInterceptor('archive', {
      limits: { fileSize: 500 * 1024 * 1024 },
    }),
  )
  async upload(@UploadedFile() file: MultipartFileStream) {
    const dir = join(process.cwd(), 'storage', 'incoming');
    await mkdir(dir, { recursive: true });

    const path = join(dir, randomUUID());

    // pipeline() propagates errors in either direction and destroys both
    // streams, which a bare file.stream.pipe(out) does not. If the client
    // aborts, or fileSize is exceeded, this rejects and nothing is left open.
    await pipeline(file.stream, createWriteStream(path));

    return { path, originalName: file.originalname };
  }
}

Sending a file back: short form and long form

Both routes send an identical response, so you need to recognise either in a codebase. Prefer the StreamableFile form: the handler returns a value, which means interceptors and unit tests still work on it, and setErrorHandler covers the case where the read fails after the headers have gone out. The @Res() form is worth keeping only when you need a header computed at the last moment, and @Res({ passthrough: true }) gives you that without opting out of Nest's response handling.

Example · typescript
// src/reports/reports.controller.ts
import { createReadStream } from 'node:fs';
import { join } from 'node:path';
import type { Response } from 'express';
import {
  Controller,
  Get,
  Param,
  Res,
  StreamableFile,
} from '@nestjs/common';

@Controller('reports')
export class ReportsController {
  // Shorthand. Nest sets the headers and pipes the stream for you. type
  // defaults to application/octet-stream when you leave it out.
  @Get(':id/short')
  short(@Param('id') id: string): StreamableFile {
    const stream = createReadStream(join(process.cwd(), 'storage', `${id}.pdf`));

    return new StreamableFile(stream, {
      type: 'application/pdf',
      disposition: `attachment; filename="report-${id}.pdf"`,
    }).setErrorHandler((err, res) => {
      // Express only. On Fastify, stream errors go to the exceptions layer
      // and become a 500 by default.
      if (res.headersSent) {
        res.end();
        return;
      }
      res.statusCode = 404;
      res.send('Report not found');
    });
  }

  // Longhand. The same bytes and the same headers, done by hand. Taking
  // @Res() without passthrough means you own the response: nothing you
  // return is sent, and interceptors can no longer modify the body.
  @Get(':id/long')
  long(@Param('id') id: string, @Res() res: Response): void {
    const stream = createReadStream(join(process.cwd(), 'storage', `${id}.pdf`));

    res.set({
      'Content-Type': 'application/pdf',
      'Content-Disposition': `attachment; filename="report-${id}.pdf"`,
    });
    stream.pipe(res);
  }
}

Related

nest upload basicsnest upload storagenest cloud storagenest interceptorsnest queues bullmq

Discussion

  • Be the first to comment on this lesson.