Design an Image Feed

Presigned uploads, a transcode pipeline that survives duplicate deliveries, immutable CDN URLs, and where ranking stops.

SyntaxPOST /uploads -> {uploadId, presignedUrl} -> client PUT -> object-created event -> transcode worker

An image feed is a timeline problem with a large binary payload attached. The timeline half is the same fan-out question as any feed; the new work is getting bytes in, turning them into the handful of renditions clients actually request, and serving those renditions without paying origin bandwidth for them.

Upload

Do not proxy image bytes through the application tier. The client asks for an upload slot, gets a presigned URL, and PUTs directly to object storage; the app tier handles only small JSON. An object-created event then puts a job on a queue, and a worker produces a fixed ladder of renditions — a thumbnail, a feed-width image, a full-size one — in a modern format such as WebP or AVIF. Only when the renditions exist does the post become feed-visible and fan-out run. A post whose derivatives are still processing renders as a broken image on every client.

Delivery

Make every rendition URL immutable: media id, rendition name, format. Then it can carry Cache-Control: public, max-age=31536000, immutable and you never purge the CDN — a re-render publishes a new key instead. This is the single biggest cost lever in the design, because the CDN absorbs essentially all image traffic while your origin serves JSON.

The ranking boundary

The feed service returns ids and URLs, never bytes, and ranking sits behind it with a deadline. Keeping that boundary means a slow ranker degrades ordering, not availability.

What trips people up

Queues deliver at least once. A transcode handler without an idempotency key will happily create the same post twice.

Example

Example · javascript
// Transcode worker. The queue is at-least-once, so this handler has to be
// safe to run twice for the same upload.
import sharp from "sharp";

const LADDER = [
  { name: "thumb", width: 320 },
  { name: "feed",  width: 1080 },
  { name: "full",  width: 1440 },
];

async function handleUpload(message) {
  const { uploadId, originalKey, authorId } = JSON.parse(message.Body);

  // Claim the work with a conditional insert. uploadId is the idempotency
  // key: a duplicate delivery loses the race and returns instead of
  // creating a second post.
  const claimed = await db.oneOrNone(
    `INSERT INTO media (upload_id, author_id, state)
     VALUES ($1, $2, 'processing')
     ON CONFLICT (upload_id) DO NOTHING
     RETURNING media_id`,
    [uploadId, authorId]
  );
  if (!claimed) return;                 // already claimed; nothing to do

  const original = await getObject("uploads-prod", originalKey);

  for (const step of LADDER) {
    const buf = await sharp(original)
      .rotate()                         // applies the EXIF orientation tag
      .resize({ width: step.width, withoutEnlargement: true })
      .toFormat("webp", { quality: 82 })
      .toBuffer();

    // Keys are immutable: a re-render gets a new mediaId, never the same
    // path with different bytes, so the CDN never needs purging.
    await putObject(
      "media-prod",
      `m/${claimed.media_id}/${step.name}.webp`,
      buf,
      { contentType: "image/webp", cacheControl: "public, max-age=31536000, immutable" }
    );
  }

  // Visible only once every rendition exists.
  await db.none("UPDATE media SET state = 'ready' WHERE upload_id = $1", [uploadId]);
  await fanoutQueue.send({ mediaId: claimed.media_id, authorId });
}

When to use it

  • Accepting a 12 MP phone photo without sizing the application tier around upload bandwidth.
  • Generating thumbnail, feed and full renditions once instead of resizing on every request.
  • Serving a viral post's images from the CDN so origin egress stays flat.
  • Keeping the feed up when the ranking service is redeploying.

More examples

Presigned upload slot

The app tier never touches image bytes: it issues a short-lived, size-constrained URL and records the pending upload. Reach for this whenever uploads are large enough that proxying them would size your application tier around bandwidth rather than requests.

Example · javascript
import { PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

app.post("/uploads", async (req, res) => {
  const uploadId = crypto.randomUUID();
  const key = `incoming/${req.userId}/${uploadId}`;

  // The signature pins bucket, key and content type. It does not cap size -
  // for that, use a presigned POST policy with content-length-range, or cap
  // it at the CDN/WAF in front of the bucket.
  const url = await getSignedUrl(
    s3,
    new PutObjectCommand({
      Bucket: "uploads-prod",
      Key: key,
      ContentType: "image/jpeg",
    }),
    { expiresIn: 900 }   // 15 minutes: long enough for a bad mobile connection
  );

  // Record the intent so an upload that never completes can be reaped, and
  // so the object-created event can be matched to a user.
  await db.none(
    `INSERT INTO pending_uploads (upload_id, author_id, object_key)
     VALUES ($1, $2, $3)`,
    [uploadId, req.userId, key]
  );

  res.status(201).json({ uploadId, url });
});

Client-aware rendition selection

One upload produces several renditions; the client picks one. Emitting a srcset lets the browser choose by device pixel ratio and viewport, and the fallback <img> keeps clients without WebP working. Use this when image bytes dominate your egress bill.

Example · javascript
// The feed API returns ids and URLs only. Choosing a rendition is the
// client's job, so the response carries all of them.
//
// Every URL here has to correspond to an object the transcode ladder actually
// wrote. The ladder emits WebP only, so there is no .jpg rendition to offer:
// add a JPEG step to the ladder first if you need to serve a client that
// cannot decode WebP.
function mediaUrls(mediaId) {
  const base = `https://cdn.example.com/m/${mediaId}`;
  return {
    thumb: `${base}/thumb.webp`,
    feed: `${base}/feed.webp`,
    full: `${base}/full.webp`,
  };
}

// Rendered markup. The browser picks by viewport and device pixel ratio;
// sizes tells it how wide the image will actually be laid out.
function imgTag({ mediaId, alt, width, height }) {
  const u = mediaUrls(mediaId);
  return `<img src="${u.feed}"
    srcset="${u.thumb} 320w, ${u.feed} 1080w, ${u.full} 1440w"
    sizes="(max-width: 700px) 100vw, 700px"
    width="${width}" height="${height}"
    loading="lazy" decoding="async" alt="${alt}">`;
  // width/height are stored at upload time; without them the feed reflows
  // as each image arrives.
}

The feed boundary, with ranking optional

Shows exactly where ranking is allowed to influence the response and what happens when it does not answer in time. Reach for this before ranking ships, so the fallback path is the one that has been running in production all along.

Example · javascript
const RANK_BUDGET_MS = 60;

async function feed(userId, limit = 20) {
  // Timeline retrieval is the availability-critical part: ids only.
  const ids = await timeline.page(userId, { limit: limit * 3 });
  if (ids.length === 0) return [];

  let ordered = ids;                       // reverse-chronological default
  try {
    // Ranking is a bounded, optional improvement on that order.
    ordered = await withTimeout(ranker.order(userId, ids), RANK_BUDGET_MS);
  } catch {
    metrics.increment("feed.rank.fallback");
  }

  const media = await mediaStore.mget(ordered.slice(0, limit));
  return media
    .filter(m => m && m.state === "ready")  // never show a half-processed post
    .map(m => ({
      id: m.media_id,
      authorId: m.author_id,
      urls: mediaUrls(m.media_id),          // URLs, not bytes
      width: m.width,
      height: m.height,
    }));
}

Related

sd object storagesd queuessd reverse proxy cdnsd design twittersd idempotency

Discussion

  • Be the first to comment on this lesson.