Working with S3

Put and get S3 objects, stream large ones with the upload manager, and hand out presigned URLs.

Syntaxout, err := client.GetObject(ctx, &s3.GetObjectInput{Bucket: aws.String(b), Key: aws.String(k)})

S3 stores objects. An object is a run of bytes plus a name, called the key, inside a bucket. A bucket is a named space with a region and permissions. There are no real folders: a key such as 2026/q1.csv simply contains slashes.

Put and get

PutObject writes one object. Its Body field is an io.Reader, the interface for anything you can read bytes from, so an open file or an incoming request body can be uploaded without first loading it all into memory. GetObject returns a value whose Body is an io.ReadCloser: read from it, then close it.

Bucket and Key are *string, a pointer to a string, so the SDK can tell "not set" from "set to empty". aws.String(k) is shorthand for taking the address of a variable, as in v := k; &v, and aws.ToString(p) is the reverse, returning "" when the pointer is nil rather than panicking.

Large objects

One PutObject sends the whole body in a single request. That caps the object at 5 GB and starts again from nothing if the connection drops. For larger or unknown-length data use manager.NewUploader, which splits the body into parts of 5 MB by default, sends 5 at a time, and retries a failed part by itself. AWS marks that package superseded by feature/s3/transfermanager, but it is still what most code uses.

Presigned URLs

A presigned URL is an ordinary URL carrying a signature and an expiry time. Whoever holds it may perform that one operation with no credentials of their own. Presigning is local arithmetic with no network call, and it lets a browser upload straight to S3 instead of through your service.

Example

Example · go
package main

import (
	"bytes"
	"context"
	"fmt"
	"io"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

// uploader names the one method of *s3.Client that saveReport needs.
type uploader interface {
	PutObject(context.Context, *s3.PutObjectInput, ...func(*s3.Options)) (*s3.PutObjectOutput, error)
}

// saveReport writes one object. The real client and the fake below both
// satisfy uploader, so this function never has to know which it got.
func saveReport(ctx context.Context, c uploader, bucket, key string, body io.Reader) error {
	_, err := c.PutObject(ctx, &s3.PutObjectInput{
		Bucket:      aws.String(bucket),
		Key:         aws.String(key),
		Body:        body,
		ContentType: aws.String("text/csv"),
	})
	if err != nil {
		return fmt.Errorf("put %s: %w", key, err)
	}
	return nil
}

// fakeS3 records the call instead of sending it, so this program needs no
// network and no AWS account.
type fakeS3 struct{}

func (fakeS3) PutObject(_ context.Context, in *s3.PutObjectInput, _ ...func(*s3.Options)) (*s3.PutObjectOutput, error) {
	b, err := io.ReadAll(in.Body)
	if err != nil {
		return nil, err
	}
	fmt.Printf("bucket=%s key=%s bytes=%d\n", aws.ToString(in.Bucket), aws.ToString(in.Key), len(b))
	return &s3.PutObjectOutput{}, nil
}

func main() {
	body := bytes.NewReader([]byte("id,total\n1,42\n"))
	if err := saveReport(context.Background(), fakeS3{}, "reports", "2026/q1.csv", body); err != nil {
		log.Fatal(err)
	}
}

// Output:
// bucket=reports key=2026/q1.csv bytes=14

When to use it

  • Store a generated PDF or CSV report and serve it from the bucket afterwards.
  • Let a browser upload a large video straight to S3 with a presigned PUT, keeping it off your server.
  • Stream a nightly database dump into S3 while the dump is still being written.
  • Read a seed or configuration file out of S3 when a service starts.

More examples

Downloading to a file without filling memory

This is the download you will write most often: a restore, a report fetch, a cached asset. Two things earn their lines. The deferred Close on out.Body runs on every exit path, including the error returns below it. The final return f.Close() is deliberate: writes are buffered, and Close is where a disk-full error finally appears, so a function that only defers Close can report success for a truncated file. Calling Close twice is harmless; the deferred one just returns an error nobody reads.

Example · go
// download copies one object to a local file in small chunks.
func download(ctx context.Context, c *s3.Client, bucket, key, dest string) error {
	out, err := c.GetObject(ctx, &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	})
	if err != nil {
		return fmt.Errorf("get %s: %w", key, err)
	}
	defer out.Body.Close()

	f, err := os.Create(dest)
	if err != nil {
		return err
	}
	defer f.Close()

	// io.Copy moves bytes through a small buffer, so a 4 GB object still
	// uses only a few kilobytes of memory here.
	if _, err := io.Copy(f, out.Body); err != nil {
		return fmt.Errorf("copy %s: %w", key, err)
	}
	return f.Close()
}

Streaming a backup with the upload manager

Reach for the manager whenever the data is big or its size is not known in advance. It performs a multipart upload: the object is sent as numbered parts and assembled by S3 at the end, so one failed part is retried on its own instead of restarting the whole transfer. PartSize and Concurrency trade memory for speed, because roughly PartSize times Concurrency bytes are buffered at once. The package is github.com/aws/aws-sdk-go-v2/feature/s3/manager, a separate module from service/s3.

Example · go
uploader := manager.NewUploader(client, func(u *manager.Uploader) {
	u.PartSize = 10 * 1024 * 1024 // 10 MB parts instead of the 5 MB default
	u.Concurrency = 4             // four parts in flight at once
})

// Body is a pipe that pg_dump is still writing into: the total length is
// unknown, which PutObject could not handle but the manager can.
res, err := uploader.Upload(ctx, &s3.PutObjectInput{
	Bucket: aws.String("backups"),
	Key:    aws.String("db/2026-01-01.sql.gz"),
	Body:   pipeReader,
})
if err != nil {
	return fmt.Errorf("upload backup: %w", err)
}
log.Println("stored at", res.Location)

A presigned PUT so the browser uploads directly

Use this when a user uploads something large and your service has no reason to touch the bytes. Your API returns the URL; the browser PUTs the file to S3; your bandwidth and request time are untouched. The signature covers the bucket, the key, the method and the signed headers, so the holder cannot rewrite the key or upload somewhere else, and it stops working when the expiry passes. Keep the window short, minutes rather than days, because anyone who copies the URL can use it until then.

Example · go
presign := s3.NewPresignClient(client)

req, err := presign.PresignPutObject(ctx, &s3.PutObjectInput{
	Bucket:      aws.String("uploads"),
	Key:         aws.String("avatars/42.png"),
	ContentType: aws.String("image/png"),
}, s3.WithPresignExpires(15*time.Minute))
if err != nil {
	return fmt.Errorf("presign: %w", err)
}

// req.URL is the address to PUT the bytes to. req.SignedHeader lists the
// headers the uploader must send unchanged, here Content-Type.
fmt.Println(req.Method, req.URL)
for name := range req.SignedHeader {
	fmt.Println("must send:", name)
}

Related

go io interfacesgo aws sdkgo forms uploadsgo retriesgo context timeout

Discussion

  • Be the first to comment on this lesson.