Working with S3

Put and get objects in S3, stream large ones through lib-storage, and hand out presigned URLs that need no AWS credentials.

Syntaxconst url = await getSignedUrl(s3, new GetObjectCommand({ Bucket, Key }), { expiresIn: 900 });

Amazon S3 stores files. A bucket is a named container; an object is one file inside it, addressed by a key such as invoices/2026/q1.pdf. There are no real directories, only keys that contain slashes.

Putting and getting

PutObjectCommand takes Bucket, Key and Body. The body can be a string, a Buffer, or a readable stream.

await s3.send(new PutObjectCommand({
  Bucket: "invoices", Key: "q1.pdf", Body: bytes,
  ContentType: "application/pdf",
}));

GetObjectCommand does not return the file contents. It returns a response whose Body is a readable stream, so a large object never has to fit in memory. For something small, await out.Body.transformToString() reads it to the end and decodes it; transformToByteArray() gives you the bytes.

Large uploads

A single PUT is limited, and a failure halfway through means starting over. The Upload class in @aws-sdk/lib-storage is the shorthand for the longhand sequence of CreateMultipartUpload, many UploadPart calls and CompleteMultipartUpload. It splits the body into parts, uploads several at once and reassembles them. partSize defaults to 5 MB and may not be smaller; queueSize, the number of parts in flight, defaults to 4.

Presigned URLs

A presigned URL is a normal HTTPS link that carries a signature in its query string. Whoever holds it can perform that one operation on that one key until it expires, with no AWS credentials of their own. getSignedUrl(client, command, { expiresIn }) builds one; expiresIn is in seconds. Sign a PutObjectCommand and a browser can upload straight to S3, so the bytes never pass through your server.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Store a user's uploaded avatar and serve it back through a link that expires in an hour.
  • Move a multi-gigabyte export to S3 without holding it in memory.
  • Let a browser upload a large video directly to a bucket so your service never carries the bytes.
  • Stream a stored CSV through a parser line by line instead of loading the whole file.

More examples

Uploading a large object with progress

Upload wraps multipart upload so you do not track part numbers or ETags yourself. Pass it a stream and it reads as it uploads, so memory stays near partSize times queueSize rather than the size of the file. leavePartsOnError false matters on a long upload: without it a failure leaves paid-for incomplete parts in the bucket.

Example · javascript
import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { createReadStream } from "node:fs";

const client = new S3Client({});

const upload = new Upload({
  client,
  params: {
    Bucket: "exports",
    Key: "2026/orders.csv",
    Body: createReadStream("/var/tmp/orders.csv"),
    ContentType: "text/csv",
  },
  partSize: 10 * 1024 * 1024, // 10 MB; 5 MB is the minimum
  queueSize: 4,               // parts uploaded at the same time
  leavePartsOnError: false,   // abort and clean up if a part fails
});

// The only event you normally listen for.
upload.on("httpUploadProgress", (p) => {
  console.log(`${p.loaded} of ${p.total} bytes`);
});

await upload.done();

A presigned URL for a browser upload

Use this when the file is large or the traffic is heavy, so your service is never a pipe for the bytes. The signature covers the method, the bucket, the key and the signed headers, so the holder cannot change any of them. Because the URL grants access on its own, treat it as a secret and keep expiresIn as short as the upload needs.

Example · javascript
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const s3 = new S3Client({});

// Your API returns the URL; the browser does the PUT.
export async function uploadTarget(userId, contentType) {
  const command = new PutObjectCommand({
    Bucket: "user-uploads",
    // The server chooses the key, so a client cannot overwrite someone else's.
    Key: `avatars/${userId}.jpg`,
    ContentType: contentType,
  });
  // expiresIn is seconds. Keep it short: the link is the permission.
  return getSignedUrl(s3, command, { expiresIn: 300 });
}

// In the browser:
// await fetch(url, { method: "PUT", body: file,
//                    headers: { "Content-Type": file.type } });

Streaming an object to disk

The right pattern for an object too big to hold in memory. pipeline is what makes it safe: on an error anywhere in the chain it destroys every stream, so the S3 socket is released instead of being held by a half-read body. Reach for transformToString only when you know the object is small.

Example · javascript
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";

const s3 = new S3Client({});

const out = await s3.send(new GetObjectCommand({
  Bucket: "exports",
  Key: "2026/orders.csv",
}));

// out.Body is a readable stream. pipeline handles backpressure and destroys
// both streams if either one fails, which is why the body never leaks.
await pipeline(out.Body, createWriteStream("/tmp/orders.csv"));

console.log("content length:", out.ContentLength);

Related

node aws sdknode streamsnode pipenode adv large datanode aws retries

Discussion

  • Be the first to comment on this lesson.