Blob and Object Storage
Why large media never belongs in the database, how presigned uploads keep bytes out of your app tier, and serving through a CDN.
getSignedUrl(s3Client, new PutObjectCommand({ Bucket, Key }), { expiresIn: 900 })Object storage holds opaque blobs under keys, with HTTP as the API: S3, GCS, Azure Blob, R2. It is cheap per gigabyte, effectively unbounded, and durable without your involvement. It is the right home for anything large and immutable — uploads, originals, derived images, video segments, exports, backups.
Why not the database
A blob column drags megabytes through every layer tuned for rows: it inflates backups and restore time, evicts useful pages from the buffer pool, makes replication lag, and ties file reads to database connections, the scarcest resource you have. Store the key and the metadata in the database, and the bytes in the bucket. The row is the record; the object is the payload.
Presigned uploads
A presigned URL is a signed, expiring grant to perform one operation on one key. The client uploads straight to the bucket, so no request body passes through your servers and no bucket credentials reach the browser. Two forms exist: a presigned PUT URL, and a presigned POST policy that can additionally constrain size and content type. With SigV4, an expiry of up to seven days is allowed; credentials from an assumed role cut that to the role's session length.
Serving
Serve through a CDN with the bucket private, reached by origin access control, not by making the bucket public. Reading through your application turns every image into a request that occupies a worker and pays egress twice.
Example
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { createPresignedPost } from "@aws-sdk/s3-presigned-post";
import crypto from "node:crypto";
const s3 = new S3Client({ region: "eu-west-1" });
export async function uploadTarget(userId, contentType) {
// Key chosen by the server, never by the client: a client-supplied key is
// how one user overwrites another's object.
const Key = `uploads/${userId}/${crypto.randomUUID()}`;
// SHORT FORM - one signed URL. The client does: PUT <url> with the bytes.
// expiresIn is seconds; SigV4 permits at most 604800 (7 days).
const putUrl = await getSignedUrl(
s3,
new PutObjectCommand({ Bucket: "app-media", Key, ContentType: contentType }),
{ expiresIn: 900 }
);
// LONG FORM - a signed POST policy. The same upload, expressed as a form
// post, with conditions S3 itself enforces. Expires defaults to 3600.
const post = await createPresignedPost(s3, {
Bucket: "app-media",
Key,
Expires: 900,
Fields: { "Content-Type": contentType },
Conditions: [
["content-length-range", 1, 10 * 1024 * 1024], // 1 byte .. 10 MiB
["eq", "$Content-Type", contentType], // must match Fields
],
});
// The database gets the key and the state machine, not the bytes.
await db.media.insert({ key: Key, userId, status: "pending" });
// post is { url, fields } - every field goes into the multipart/form-data
// body before the file part.
return { putUrl, post };
}When to use it
- Moving user avatars and attachments out of a Postgres bytea column that has made backups unmanageable.
- Letting a browser upload a 2 GB video without the bytes crossing the application tier.
- Bounding the size of an anonymous upload so a presigned URL cannot be abused.
- Serving user media through a CDN while keeping the bucket entirely private.
More examples
Large files: presigned multipart parts
Use this above roughly 100 MB, and always for video. The operational detail people miss: an abandoned multipart upload keeps its parts and keeps billing for them, so the bucket needs a lifecycle rule to abort incomplete uploads after a few days.
import {
CreateMultipartUploadCommand, UploadPartCommand,
CompleteMultipartUploadCommand,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
// A single PUT tops out at 5 GB. Multipart takes parts of 5 MiB to 5 GiB,
// up to 10,000 of them, uploaded in parallel and retried individually.
export async function startBigUpload(Bucket, Key, partCount) {
const { UploadId } = await s3.send(
new CreateMultipartUploadCommand({ Bucket, Key })
);
// One presigned URL per part; the browser uploads them concurrently.
const urls = await Promise.all(
Array.from({ length: partCount }, (_, i) =>
getSignedUrl(
s3,
new UploadPartCommand({ Bucket, Key, UploadId, PartNumber: i + 1 }),
{ expiresIn: 3600 }
)
)
);
return { UploadId, urls };
}
// The client returns each part's ETag; the server completes the upload.
export const finish = (Bucket, Key, UploadId, Parts) =>
s3.send(new CompleteMultipartUploadCommand({
Bucket, Key, UploadId, MultipartUpload: { Parts }, // [{ ETag, PartNumber }]
}));Private bucket, public CDN
This is the default shape for serving user media: nothing is publicly listable, nothing is readable except through the CDN, and your application never proxies an image. For media that must stay private per user, keep this policy and add CloudFront signed URLs or signed cookies at the edge rather than streaming the object through your app.
// The bucket stays private. CloudFront reaches it with Origin Access Control,
// and the bucket policy trusts only that distribution.
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "AllowCloudFrontOACOnly",
"Effect": "Allow",
"Principal": { "Service": "cloudfront.amazonaws.com" },
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::app-media/*",
"Condition": { "StringEquals": {
"AWS:SourceArn": "arn:aws:cloudfront::111122223333:distribution/E1EXAMPLE"
}}
}]
}The row and the object, kept consistent
Reach for this whenever an upload has to appear in a listing or a feed. It is the standard answer to the two-system consistency problem here: the bucket is the source of truth for bytes, the bucket's own event is the signal, and the database row never claims an object exists before S3 says it does.
// Uploads fail halfway, so the database row is a small state machine rather
// than a fact. The client confirms; a bucket event, not the client, promotes.
// media(id, key, user_id, status, bytes, content_type, created_at)
// status: pending -> uploaded -> processed
// S3 -> SQS notification on s3:ObjectCreated:* for the uploads/ prefix.
export async function onObjectCreated(record) {
const key = decodeURIComponent(record.s3.object.key.replace(/\+/g, " "));
await db.media.update(
{ key, status: "pending" },
{ status: "uploaded", bytes: record.s3.object.size }
);
await queue.send({ type: "derive-thumbnails", key }); // worker writes /derived
}
// Rows left pending past the presigned URL's expiry are dead: sweep them, and
// let a bucket lifecycle rule delete orphaned objects with no row at all.
Discussion