Object Storage

Put uploads, downloads and presigned URLs behind one injectable service so bytes never live in your process or your database.

SyntaxgetSignedUrl(client, new GetObjectCommand({ Bucket, Key }), { expiresIn: 300 }): Promise<string>

Object storage keeps files by key inside a bucket: S3, Google Cloud Storage, Cloudflare R2. Your database keeps the row that describes the file; the bucket keeps the bytes. A container's disk vanishes when it restarts and its memory is not shared with the next replica, so anything a user uploads has to leave the process.

Three operations, one service

Put bytes, read bytes back, and presign. A presigned URL is an ordinary HTTPS URL with a signature in the query string: it authorises one operation on one key for a fixed number of seconds. Hand it to the browser and the browser talks to the bucket directly, so a 2 GB upload never passes through your API at all.

Where this sits in a request

For a direct upload, FileInterceptor runs at the interceptor step — after guards, before pipes. It parses the multipart body and attaches the file to the request. ParseFilePipe then validates it at the pipe step, and the handler hands the result to your storage service. For a download, return a StreamableFile and Nest pipes it to the response after the handler returns.

What trips people up

new Upload({ client, params }).done() is shorthand: below one part size it issues a single PutObjectCommand, above it the multipart sequence. Both forms are shown below. And a signature covers the headers you signed — presign a PUT with a ContentType and the browser must send that exact value, or S3 answers 403.

Example

Example · typescript
// src/storage/storage.service.ts
import { Inject, Injectable, NotFoundException } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import {
  DeleteObjectCommand,
  GetObjectCommand,
  PutObjectCommand,
  S3Client,
} from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { Upload } from '@aws-sdk/lib-storage';
import { Readable } from 'node:stream';
import { S3_CLIENT } from '../aws/aws.module';

@Injectable()
export class StorageService {
  private readonly bucket: string;

  constructor(
    @Inject(S3_CLIENT) private readonly s3: S3Client,
    config: ConfigService,
  ) {
    this.bucket = config.getOrThrow<string>('S3_BUCKET');
  }

  // Upload accepts a stream and splits large bodies into parts, so a big
  // file is never held in memory in one piece.
  async put(key: string, body: Readable | Buffer, contentType: string) {
    const upload = new Upload({
      client: this.s3,
      params: { Bucket: this.bucket, Key: key, Body: body, ContentType: contentType },
      partSize: 5 * 1024 * 1024, // 5 MiB: S3's minimum for any part but the last
      queueSize: 4,              // parts uploaded concurrently
    });
    const result = await upload.done();
    return { key, etag: result.ETag };
  }

  // expiresIn is in seconds. Keep it short: the URL is a bearer token and
  // anyone who gets the link can use it until it expires.
  presignDownload(key: string, expiresIn = 300): Promise<string> {
    return getSignedUrl(
      this.s3,
      new GetObjectCommand({ Bucket: this.bucket, Key: key }),
      { expiresIn },
    );
  }

  // Whatever is signed here the client must send back byte for byte.
  presignUpload(key: string, contentType: string, expiresIn = 300): Promise<string> {
    return getSignedUrl(
      this.s3,
      new PutObjectCommand({ Bucket: this.bucket, Key: key, ContentType: contentType }),
      { expiresIn },
    );
  }

  async read(key: string): Promise<Readable> {
    const out = await this.s3.send(
      new GetObjectCommand({ Bucket: this.bucket, Key: key }),
    );
    if (!out.Body) throw new NotFoundException(`No object at ${key}`);
    return out.Body as Readable; // on Node the SDK returns a Readable stream
  }

  async remove(key: string): Promise<void> {
    await this.s3.send(new DeleteObjectCommand({ Bucket: this.bucket, Key: key }));
  }
}

When to use it

  • Store user avatars in S3 and keep only the key in the users table.
  • Let the browser upload a 2 GB video straight to the bucket with a presigned PUT.
  • Serve a private report as a download without making the bucket public.
  • Hand a background worker the same storage service the HTTP layer uses.

More examples

Upload through the API, download as a stream

The route-level half of the same feature, and a clear view of the ordering: guards, then FileInterceptor, then ParseFilePipe, then the handler. Use this shape for small files where it is fine for the bytes to travel through your API. Returning StreamableFile rather than piping to the raw response means Nest still applies your interceptors and exception filters to the call.

Example · typescript
// src/files/files.controller.ts
import {
  Controller,
  Get,
  Param,
  ParseFilePipeBuilder,
  Post,
  StreamableFile,
  UploadedFile,
  UseInterceptors,
} from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { randomUUID } from 'node:crypto';
import { StorageService } from '../storage/storage.service';

@Controller('files')
export class FilesController {
  constructor(private readonly storage: StorageService) {}

  @Post()
  // Interceptor step: parses the multipart body, exposes file.buffer.
  @UseInterceptors(FileInterceptor('file'))
  async upload(
    // Pipe step: runs after the interceptor, before the method body.
    @UploadedFile(
      new ParseFilePipeBuilder()
        .addFileTypeValidator({ fileType: /^image\/(png|jpeg)$/ })
        .addMaxSizeValidator({ maxSize: 5 * 1024 * 1024 })
        .build({ fileIsRequired: true }),
    )
    file: Express.Multer.File,
  ) {
    const key = `avatars/${randomUUID()}`;
    return this.storage.put(key, file.buffer, file.mimetype);
  }

  @Get(':key')
  async download(@Param('key') key: string): Promise<StreamableFile> {
    const body = await this.storage.read(`avatars/${key}`);
    // Nest pipes the stream to the response and sets the headers from options.
    return new StreamableFile(body, {
      type: 'image/png',
      disposition: `attachment; filename="${key}.png"`,
    });
  }
}

The presign flow: browser uploads straight to the bucket

The pattern for anything large or slow. Your API spends milliseconds issuing a signature instead of minutes proxying bytes, and no request body limit applies. The cost is that the object appears in the bucket without your code seeing it, so confirm the key afterwards — a HeadObject call or an S3 event notification — before you treat the upload as real.

Example · typescript
// src/files/direct-upload.controller.ts
import { Body, Controller, Post, UseGuards } from '@nestjs/common';
import { IsIn, IsString, MaxLength } from 'class-validator';
import { randomUUID } from 'node:crypto';
import { JwtAuthGuard } from '../auth/jwt-auth.guard';
import { StorageService } from '../storage/storage.service';

class CreateUploadDto {
  @IsString()
  @MaxLength(200)
  filename!: string;

  // Pin the type here: it is signed, so the client cannot change it later.
  @IsIn(['image/png', 'image/jpeg', 'application/pdf'])
  contentType!: string;
}

@Controller('uploads')
@UseGuards(JwtAuthGuard) // the API authorises; the signed URL carries that decision
export class DirectUploadController {
  constructor(private readonly storage: StorageService) {}

  @Post()
  async create(@Body() dto: CreateUploadDto) {
    const key = `uploads/${randomUUID()}/${dto.filename}`;
    const url = await this.storage.presignUpload(key, dto.contentType, 120);

    // The browser then does:
    //   fetch(url, { method: 'PUT', body: file,
    //                headers: { 'Content-Type': dto.contentType } })
    // The Content-Type must match what was signed or S3 returns 403.
    return { key, url, expiresInSeconds: 120 };
  }
}

The multipart helper, and the longhand it replaces

The same three uploads written both ways. Upload is the form to use, but reading the longhand explains its behaviour: why parts have a 5 MiB floor, why an abandoned upload leaves billable parts behind until you abort it, and why the ETag of a multipart object is not the MD5 of the file. You will meet the longhand in older code and in anything that resumes an interrupted upload.

Example · typescript
// src/storage/storage.multipart.ts
import {
  CompleteMultipartUploadCommand,
  CreateMultipartUploadCommand,
  PutObjectCommand,
  S3Client,
  UploadPartCommand,
} from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';

// Shorthand. One call, whatever the size.
export async function putShorthand(s3: S3Client, bucket: string, key: string, body: Buffer) {
  const upload = new Upload({ client: s3, params: { Bucket: bucket, Key: key, Body: body } });
  upload.on('httpUploadProgress', (p) => console.log(p.loaded, '/', p.total));
  return upload.done();
}

// Longhand A: what Upload does when the body fits in a single part.
export async function putSmallLonghand(s3: S3Client, bucket: string, key: string, body: Buffer) {
  return s3.send(new PutObjectCommand({ Bucket: bucket, Key: key, Body: body }));
}

// Longhand B: what Upload does above the part size.
export async function putLargeLonghand(
  s3: S3Client,
  bucket: string,
  key: string,
  parts: Buffer[], // each at least 5 MiB except the last
) {
  const { UploadId } = await s3.send(
    new CreateMultipartUploadCommand({ Bucket: bucket, Key: key }),
  );

  const completed: { ETag: string; PartNumber: number }[] = [];
  for (const [index, part] of parts.entries()) {
    const PartNumber = index + 1; // parts are 1-based
    const { ETag } = await s3.send(
      new UploadPartCommand({ Bucket: bucket, Key: key, UploadId, PartNumber, Body: part }),
    );
    completed.push({ ETag: ETag!, PartNumber });
  }

  return s3.send(
    new CompleteMultipartUploadCommand({
      Bucket: bucket,
      Key: key,
      UploadId,
      MultipartUpload: { Parts: completed },
    }),
  );
}

Related

nest upload storagenest upload streamingnest upload validationnest cloud sdknest cloud config

Discussion

  • Be the first to comment on this lesson.