Incident Response for APIs

# runbooks/api-credential-compromise.md
# A runbook is only useful if it contains commands you can paste at 3am.

## Trigger
#   - a canary credential was used
#   - a key appeared in a public repository
#   - anomalous volume from one credential

## 0. Declare (1 minute)
#   Open an incident channel. Assign: lead, comms, scribe.
#   Everything below goes in the channel as it happens.

## 1. CONTAIN (target: under 5 minutes)
export KEY_ID="..."
export ACTOR_ID="..."

# Revoke the specific credential
curl -X POST "https://admin.dfg.com/api/keys/$KEY_ID/revoke" -H "$ADMIN_AUTH"

# Revoke every session and refresh family for the account
curl -X POST "https://admin.dfg.com/api/users/$ACTOR_ID/revoke-all" -H "$ADMIN_AUTH"

# If the SIGNING KEY may be compromised, rotate it. This invalidates every
# outstanding token globally — disruptive, and correct if there is any doubt.
#   aws kms create-key ... && update JWKS && switch ACTIVE_KID

# If a specific endpoint is being abused, disable that route only.
curl -X POST "https://admin.dfg.com/api/flags/$ENDPOINT/disable" -H "$ADMIN_AUTH"

## 2. PRESERVE (before restarting or rotating anything else)
mkdir -p incident/INC-$(date +%F)
aws logs create-export-task --log-group-name /aws/api/prod \
  --from $(date -d '30 days ago' +%s000) --to $(date +%s000) \
  --destination incident-evidence --destination-prefix "INC-$(date +%F)"

psql -c "\copy (SELECT * FROM audit_log WHERE actor_id='$ACTOR_ID'
                 AND at > now() - interval '30 days' ORDER BY at)
         TO 'incident/actor-timeline.csv' CSV HEADER"

## 3. ASSESS
# What did they access?
psql -c "SELECT target_type, count(*), min(at), max(at)
           FROM audit_log WHERE actor_id='$ACTOR_ID'
            AND at > now() - interval '30 days'
          GROUP BY target_type ORDER BY 2 DESC"

# How much data left?
psql -c "SELECT sum((metadata->>'rowCount')::int) FROM audit_log
          WHERE actor_id='$ACTOR_ID' AND event='data.exported'"

# Which tenants are affected? (this drives the disclosure decision)
psql -c "SELECT DISTINCT tenant_id FROM audit_log WHERE actor_id='$ACTOR_ID'"

# When did it start? (the answer is usually earlier than the alert)
psql -c "SELECT min(at) FROM audit_log WHERE actor_id='$ACTOR_ID'
          AND ip NOT IN (SELECT ip FROM known_ips WHERE user_id='$ACTOR_ID')"

## 4. ERADICATE
#   - fix the vulnerability, not the symptom
#   - rotate every credential that MAY have been exposed, not only the known one
#   - add a detection rule for this technique

## 5. RECOVER
#   - re-enable the endpoint behind a tighter limit
#   - watch the new detection for 48 hours

## 6. REVIEW (within 5 working days)
#   Timeline. What worked. What did not. Actions with owners and dates.
#   Blameless: the process failed, not a person.

## Contacts
#   Security lead: ...    Legal/DPO: ...    Comms: ...    Cloud support: ...

// Every containment action should be one API call, available to on-call,
// audited, and tested. Building it during an incident does not work.

// ── 1. Kill switch per endpoint ──────────────────────────────────────
app.use(async (req, res, next) => {
  const route = `${req.method} ${req.route?.path ?? req.path}`;
  if (await flags.isDisabled(route)) {
    return res.status(503).json({
      error: 'temporarily_unavailable',
      detail: 'This endpoint is temporarily disabled for maintenance.',
    });
  }
  next();
});

adminRouter.post('/flags/:route/disable', requireRole('security'), async (req, res) => {
  await flags.disable(req.params.route, {
    by: req.user.id, reason: req.body.reason, incident: req.body.incidentId,
  });
  await audit.record({ event: 'admin.action', actor: req.user,
    metadata: { action: 'endpoint_disabled', route: req.params.route } });
  res.sendStatus(204);
});

// ── 2. Revoke everything for an actor, in one call ───────────────────
adminRouter.post('/users/:id/revoke-all', requireRole('security'), async (req, res) => {
  const userId = req.params.id;
  await Promise.all([
    destroyAllSessionsFor(userId),
    db('refresh_tokens').where({ user_id: userId }).update({ revoked_at: new Date() }),
    db('api_keys').where({ user_id: userId }).update({ revoked_at: new Date() }),
    db('users').where({ id: userId }).increment('token_version', 1),  // kills live JWTs
  ]);
  await audit.record({ event: 'admin.action', actor: req.user,
    target: { type: 'user', id: userId },
    metadata: { action: 'revoke_all', incident: req.body.incidentId } });
  res.sendStatus(204);
});

// ── 3. Emergency global read-only mode ───────────────────────────────
// When you are unsure of the scope, stopping writes preserves state without
// taking the product down.
app.use((req, res, next) => {
  if (readOnlyMode.enabled && !['GET', 'HEAD', 'OPTIONS'].includes(req.method)) {
    return res.status(503).json({
      error: 'read_only_mode',
      detail: 'The service is temporarily read-only.',
    });
  }
  next();
});

// ── 4. Block an actor without deleting evidence ──────────────────────
adminRouter.post('/block/:actorId', requireRole('security'), async (req, res) => {
  await redis.set(`blocked:${req.params.actorId}`, JSON.stringify({
    by: req.user.id, reason: req.body.reason, at: new Date(),
  }), 'EX', 86400);
  res.sendStatus(204);
});
// Note it BLOCKS rather than deletes — the account and its history remain for
// the investigation.

// ── 5. And TEST these, quarterly ─────────────────────────────────────
// A kill switch nobody has exercised does not work. Run a game day:
//   - disable an endpoint in staging, confirm the 503 and the audit entry
//   - revoke-all a test user, confirm every credential type stops working
//   - time it. If containment takes more than five minutes, fix the tooling.

# Decide the criteria before an incident. Deciding under pressure, with legal
# and comms in the room, is how the 72-hour clock gets missed.

## Is this a notifiable personal data breach? (GDPR Art. 33)
#
#  1. Was personal data involved?
#       NO  → not a personal data breach. Still handle it as an incident.
#       YES → continue
#
#  2. Was it accessed, disclosed, altered, lost or destroyed without authority?
#       NO  → an attempt, not a breach. Document it.
#       YES → continue
#
#  3. Is a risk to individuals' rights and freedoms UNLIKELY?
#       Consider: sensitivity, volume, identifiability, whether the data was
#       encrypted with keys the attacker does not hold, and whether it was
#       exfiltrated or merely accessible.
#       UNLIKELY → document the reasoning. No notification. Keep the record.
#       OTHERWISE → NOTIFY THE SUPERVISORY AUTHORITY WITHIN 72 HOURS
#                   of becoming AWARE (not of the breach occurring)
#
#  4. Is the risk to individuals HIGH?
#       YES → also notify the AFFECTED INDIVIDUALS, without undue delay
#       Exception: if the data was strongly encrypted and the keys are safe

## What the notification must contain
#   - the nature of the breach, categories and approximate numbers affected
#   - the DPO or contact point
#   - the likely consequences
#   - the measures taken or proposed
#
#   You do NOT need complete information to notify. Notify within 72 hours
#   with what you have, and supplement afterwards. Late is worse than partial.

## Customer notification, separately from regulatory
#   Contracts frequently require notification faster than regulation does.
#   Check the DPA terms BEFORE an incident, not during one.

## Prepare in advance
#   - a template holding statement
#   - the supervisory authority's contact and portal, tested
#   - the DPO's out-of-hours number
#   - a customer notification template per contract tier
#   - a status page you can update without a deploy

## And what NOT to do
#   - do not speculate publicly before the assessment is complete
#   - do not say "no data was accessed" unless the logs actually prove it
#   - do not delete anything, including the attacker's account
#   - do not brief customers before the regulator, if the timeline requires it
#
#   The most damaging statements are the confident ones made in hour two and
#   contradicted in day three.