Storing Files

Local disk, object storage or a BLOB column: what each costs, and why the metadata row belongs in the database but the bytes do not.

SyntaxMulterModule.register({ storage: diskStorage({ destination, filename }) })

Once a file is validated you have to put it somewhere. There are three real options and they are not equivalent.

Local disk

The simplest: set dest, or pass diskStorage, and the file becomes a path. It also belongs to exactly one machine. A second replica cannot read it, a container restart loses it, and backups, cleanup and serving are all yours to build. Reasonable for a single box, or as scratch space before the real upload.

Object storage

S3, GCS, Azure Blob. Durable, visible to every replica, and able to serve the file directly through a signed URL or a CDN so the bytes never pass through Node a second time. The cost is credentials to manage, a network hop on write, and the fact that the object and your database row are now two separate writes.

The database

A bytea or BLOB column is rarely the answer. Every read and write streams through the application while holding a connection from the pool, every dump and every replica carries the bytes, and you lose range requests and HTTP caching. It is defensible when files are small, few, and must commit in the same transaction as their row.

The usual shape

Bytes in object storage under an opaque key; a row in the database with the key, size, detected type, owner and display name. The row decides who may download, and the key is never shown to clients. Write the object first and the row second: an orphaned object costs pennies, while a row pointing at nothing is a 500 on the next download.

Example

Example · typescript
// src/uploads/uploads.service.ts
import { randomUUID } from 'node:crypto';
import { Injectable } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { PutObjectCommand, S3Client } from '@aws-sdk/client-s3';
import { StoredFile } from './stored-file.entity';

@Injectable()
export class UploadsService {
  private readonly s3: S3Client;
  private readonly bucket: string;

  constructor(
    config: ConfigService,
    @InjectRepository(StoredFile)
    private readonly files: Repository<StoredFile>,
  ) {
    // Credentials come from the environment or the instance role; only the
    // region and bucket are ours to configure.
    this.s3 = new S3Client({ region: config.getOrThrow<string>('AWS_REGION') });
    this.bucket = config.getOrThrow<string>('UPLOAD_BUCKET');
  }

  async store(file: Express.Multer.File, ownerId: string): Promise<StoredFile> {
    // The key is ours, opaque, and never built from originalname.
    const key = `uploads/${ownerId}/${randomUUID()}`;

    // 1. Bytes first. PutObjectCommand takes the whole body at once, which
    //    is fine for a buffered file of a few MB and wrong for a large one.
    await this.s3.send(
      new PutObjectCommand({
        Bucket: this.bucket,
        Key: key,
        Body: file.buffer,
        ContentType: file.mimetype,
        ContentLength: file.size,
      }),
    );

    // 2. Then the row. This is what owns permissions and the display name.
    return this.files.save(
      this.files.create({
        key,
        ownerId,
        bytes: String(file.size),
        mimeType: file.mimetype,
        displayName: file.originalname,
      }),
    );
  }
}

When to use it

  • Move an existing disk-based upload route to S3 before scaling past one replica.
  • Give a browser a 5-minute download link instead of streaming the file through your API.
  • Store a tenant's documents under a key prefix per tenant.
  • Find and delete bucket objects that no longer have a row pointing at them.

More examples

dest and the diskStorage it desugars to

dest is not a different mechanism from storage, it is a shortcut that constructs diskStorage for you; set both and storage wins. Move to the long form as soon as you care about the filename, want the directory chosen per request, or need a custom engine. The destination function form is useful for sharding by date or tenant, but unlike the string form it must return a directory that already exists.

Example · typescript
// src/uploads/uploads.module.ts
import { randomUUID } from 'node:crypto';
import { extname } from 'node:path';
import { Module } from '@nestjs/common';
import { MulterModule } from '@nestjs/platform-express';
import { diskStorage } from 'multer';

// Shorthand. multer builds a disk storage engine for you and names each
// file with 32 random hex characters, no extension.
const shorthand = MulterModule.register({ dest: './upload' });

// Longhand. The same engine, written out, so the name is yours to choose.
const longhand = MulterModule.register({
  storage: diskStorage({
    // A string directory is created recursively if it does not exist.
    // Omit it entirely and files land in os.tmpdir().
    destination: './upload',
    filename: (_req, file, callback) => {
      // extname() of a client-supplied name. Safe here only because the
      // random prefix means the result is never an existing path, and the
      // extension is cosmetic: the real type lives in your own table.
      callback(null, `${randomUUID()}${extname(file.originalname)}`);
    },
  }),
});

@Module({ imports: [longhand] })
export class UploadsModule {}

// `shorthand` is exported only to be read next to `longhand`.
export { shorthand };

Serving a download without proxying the bytes

The client gets a short-lived URL and fetches the file from the storage provider directly, so a 2 GB download never occupies a Node process or a database connection. This is the main reason object storage beats a BLOB column at scale. Keep the expiry short, authorise on your side before signing, and remember the signed URL is a bearer credential: anyone holding it can read the object until it expires.

Example · typescript
// src/uploads/downloads.service.ts
import { Injectable, NotFoundException } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { GetObjectCommand, S3Client } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { StoredFile } from './stored-file.entity';

@Injectable()
export class DownloadsService {
  constructor(
    private readonly s3: S3Client,
    @InjectRepository(StoredFile)
    private readonly files: Repository<StoredFile>,
  ) {}

  async urlFor(id: string, viewerId: string, bucket: string): Promise<string> {
    // The database row, not the bucket, decides who may read the object.
    const row = await this.files.findOneBy({ id });
    if (!row || row.ownerId !== viewerId) {
      throw new NotFoundException();
    }

    const command = new GetObjectCommand({
      Bucket: bucket,
      Key: row.key,
      // Let S3 send the headers a browser needs for a save dialog.
      ResponseContentType: row.mimeType,
      ResponseContentDisposition:
        `attachment; filename="${encodeURIComponent(row.displayName)}"`,
    });

    // expiresIn is in seconds and defaults to 900 if omitted.
    return getSignedUrl(this.s3, command, { expiresIn: 300 });
  }
}

The metadata row

This table is the part of an upload that belongs in the database. It gives you ownership checks, soft deletes, usage totals per account and a list of keys to reconcile against the bucket, all without reading a byte of content. Add a checksum column if you need to deduplicate or detect corruption. The commented bytea column is what the third option costs you, and it is the easiest line in the file to regret.

Example · typescript
// src/uploads/stored-file.entity.ts
import {
  Column,
  CreateDateColumn,
  Entity,
  Index,
  PrimaryGeneratedColumn,
} from 'typeorm';

@Entity('stored_files')
export class StoredFile {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  // The object key. Generated by us, unique, never shown to a client.
  @Column({ unique: true })
  key: string;

  @Index()
  @Column('uuid')
  ownerId: string;

  // bigint is mapped to string by TypeORM, because a file can be larger
  // than Number.MAX_SAFE_INTEGER allows you to count exactly.
  @Column('bigint')
  bytes: string;

  // The type we detected, not the one the client declared.
  @Column()
  mimeType: string;

  // The client's original filename, kept to show back in the UI.
  @Column()
  displayName: string;

  @CreateDateColumn()
  createdAt: Date;

  // The database-as-storage option, left here as a counterexample. Every
  // SELECT *, every pg_dump and every replica would carry the bytes.
  // @Column({ type: 'bytea', nullable: true })
  // data?: Buffer;
}

Related

nest cloud storagenest upload streamingnest entitiesnest config modulenest upload basics

Discussion

  • Be the first to comment on this lesson.