# Staging cutover runbook

Single-pass checklist to stand up `staging.scopeforged.com` per [Plan 281](../../docs/plans/281-staging-instance.md). All in-repo code is shipped; this file is the sudo / dashboard work the script can't do for you.

Each step is gated on either:
- **YOU on EC2** — SSH to the server, run as `ubuntu` with sudo.
- **YOU on the cloud dashboard** — Route53 (or AWS CLI) / Stripe / Sentry / UptimeRobot / 1Password.
- **YOU locally** — your dev box, in the repo.

Tick boxes as you go; if you have to break the sequence (e.g. a CIDR isn't published yet), record what was deferred at the bottom.

---

## Phase 1 — Host & DNS Foundation

### 1.1  DNS (Route53)

DNS for `scopeforged.com` is in Route53 hosted zone `Z0035399YKCMZK4BAHN8`. The EC2 host's IAM role has Route53 write access — the bulk-swap script at `/home/ubuntu/.scripts/route53-swap-ip.sh` proves this end-to-end.

- [ ] Add **ALIAS A** records (subdomains → apex A) for the four subdomains so they inherit from `staging.scopeforged.com`:
  - `staging.admin.scopeforged.com`
  - `staging.portal.scopeforged.com`
  - `staging.websites.scopeforged.com`
  - `staging.api.scopeforged.com`
- [ ] Add the apex A record `staging.scopeforged.com → <EIP>` (use the EIP `35.80.110.71` if standing this up post-2026-06-24 resize).
- [ ] Verify each resolves: `dig +short staging.scopeforged.com` (and the other four).

### 1.2  MySQL DB + user with connection cap (YOU on EC2)

```sql
-- mysql -u root -p
CREATE DATABASE client_portal_staging CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'cp_staging'@'localhost' IDENTIFIED BY '<generated-strong-pw>';
GRANT ALL PRIVILEGES ON client_portal_staging.* TO 'cp_staging'@'localhost';
ALTER USER 'cp_staging'@'localhost' WITH
  MAX_USER_CONNECTIONS 20
  MAX_QUERIES_PER_HOUR 0
  MAX_UPDATES_PER_HOUR 0;
FLUSH PRIVILEGES;

-- Sanity: cp_staging must NOT have grants on client_portal
SHOW GRANTS FOR 'cp_staging'@'localhost';

-- Record current prod peak before the cap goes live (re-tune if needed)
SHOW STATUS LIKE 'Max_used_connections';
```

- [ ] Save the password to `/home/ubuntu/.secrets/staging-db.txt` — `chmod 600`, owner `ubuntu`.

### 1.3  Directory skeleton (YOU on EC2)

- [ ] Decide dedicated EBS vs shared volume (record in [STAGING.md](../../docs/guides/STAGING.md)).

```bash
sudo mkdir -p /var/www/client-portal-laravel-staging/{releases,shared/storage/{app/public,framework/{cache/data,sessions,views},logs}}
# User created in 2.0; assigning here is fine because chown is re-issued by the deploy script anyway.
sudo chmod -R 775 /var/www/client-portal-laravel-staging/shared/storage
```

If you took the shared-volume path, also configure an XFS project quota with a 10GB initial hard cap.

---

## Phase 2 — Host isolation, Apache & TLS

### 2.0  PHP-FPM pool (YOU on EC2)

- [ ] Create the OS user:
  ```bash
  sudo useradd --system --no-create-home --shell /usr/sbin/nologin www-staging
  sudo usermod -aG www-data www-staging
  sudo chown -R www-staging:www-data /var/www/client-portal-laravel-staging
  ```
- [ ] **Audit prod first** — pull `request_terminate_timeout` and `memory_limit` from `/etc/php/8.x/fpm/pool.d/www.conf`. Replace `<MATCH_PROD>` in [`server-configs/php-fpm-pool-staging.conf`](server-configs/php-fpm-pool-staging.conf) before installing.
- [ ] Install:
  ```bash
  PHP_VERSION=$(php -r 'echo PHP_MAJOR_VERSION.".".PHP_MINOR_VERSION;')   # e.g. 8.3
  sudo install -m 0644 -o root scripts/staging/server-configs/php-fpm-pool-staging.conf \
    /etc/php/${PHP_VERSION}/fpm/pool.d/staging.conf
  sudo systemctl reload php${PHP_VERSION}-fpm
  sudo systemctl status  php${PHP_VERSION}-fpm   # should show both `www` and `staging` pools
  ls -l /run/php/php-staging.sock                # socket should exist
  ```

### 2.1  Apache vhosts (YOU on EC2)

- [ ] Clone the five prod vhost pairs as the templates, change only `ServerName` and `DocumentRoot`:
  ```bash
  cd /etc/apache2/sites-available
  for sub in '' admin. portal. websites. api.; do
    sudo cp ${sub}scopeforged.com.conf       staging.${sub}scopeforged.com.conf
    sudo cp ${sub}scopeforged.com-le-ssl.conf staging.${sub}scopeforged.com-le-ssl.conf
    sudo sed -i \
      -e "s|ServerName ${sub}scopeforged.com|ServerName staging.${sub}scopeforged.com|g" \
      -e 's|/var/www/client-portal-laravel/current/public|/var/www/client-portal-laravel-staging/current/public|g' \
      staging.${sub}scopeforged.com.conf staging.${sub}scopeforged.com-le-ssl.conf
  done
  ```
- [ ] Paste the snippets from [`server-configs/apache-staging-vhost-snippets.conf`](server-configs/apache-staging-vhost-snippets.conf) into each `-le-ssl.conf`:
  - **BLOCK A** (X-Robots-Tag) — all five
  - **BLOCK B** (FPM handler at `/run/php/php-staging.sock`) — all five
  - **BLOCK C** (basic auth) — all five
  - **BLOCK D** (webhook bypass) — `api` only (others can have it; harmless)
  - **BLOCK E** (Stripe IP allowlist) — `api` only. Pull current CIDRs from https://stripe.com/docs/ips and replace `<STRIPE_CIDR_N>` placeholders. **Document the review date** in [STAGING.md](../../docs/guides/STAGING.md) quarterly cadence section.
  - **BLOCK F** (`/up` bypass) — all five
  - **BLOCK G** (uptime-token marker) — all five; replace `<UPTIME_TOKEN_HERE>` with the value you store in 1Password
  - **BLOCK H** (Reverb proxy) — marketing `-le-ssl.conf` only

- [ ] Enable + reload:
  ```bash
  sudo a2enmod proxy_wstunnel
  for vh in staging.scopeforged.com staging.admin.scopeforged.com staging.portal.scopeforged.com staging.websites.scopeforged.com staging.api.scopeforged.com; do
    sudo a2ensite ${vh}.conf ${vh}-le-ssl.conf
  done
  sudo apache2ctl configtest && sudo systemctl reload apache2
  ```
- [ ] Confirm a staging request never reaches the prod FPM pool: `sudo tail -f /var/log/php-fpm-error.log` — should stay silent during a staging hit.

### 2.2  Let's Encrypt (YOU on EC2)

```bash
sudo certbot certonly --webroot \
  -w /var/www/client-portal-laravel-staging/current/public \
  -d staging.scopeforged.com \
  -d staging.admin.scopeforged.com \
  -d staging.portal.scopeforged.com \
  -d staging.websites.scopeforged.com \
  -d staging.api.scopeforged.com \
  --deploy-hook "systemctl reload apache2"
```

- [ ] Hand-add `SSLCertificateFile` / `SSLCertificateKeyFile` directives to each `*-le-ssl.conf` (paths shown by `certbot`).
- [ ] `sudo systemctl list-timers | grep certbot` — confirm the renewal timer exists.
- [ ] `sudo certbot renew --dry-run` — passes.
- [ ] **Acceptance test:** `sudo certbot renew --force-renewal --cert-name staging.scopeforged.com`, then re-run the §2.3 acceptance curls below. Both basic-auth challenge AND webhook bypass must still behave correctly afterward — this confirms the `certonly` choice prevented vhost clobbering.

### 2.3  Access guard (YOU on EC2)

- [ ] Create the htpasswd file:
  ```bash
  sudo htpasswd -c -B /etc/apache2/.htpasswd-staging scopeforged-staging
  sudo chown root:www-data /etc/apache2/.htpasswd-staging
  sudo chmod 0640 /etc/apache2/.htpasswd-staging
  ```
- [ ] Store the password in the 1Password vault item. Update [STAGING.md](../../docs/guides/STAGING.md) §7 with the item ID (NOT the password).
- [ ] **Acceptance curl tests:**
  ```bash
  # webhook from non-Stripe IP → expect 403 from Apache (IP allowlist)
  curl -X POST https://staging.api.scopeforged.com/webhooks/stripe -d '{}'
  # /up bypassed → 200 no challenge
  curl -i https://staging.scopeforged.com/up
  # root → 401 basic-auth challenge
  curl -i https://staging.scopeforged.com/
  ```

---

## Phase 3 — Application bootstrap

### 3.1  First release directory (YOU on EC2 — once)

```bash
sudo -u www-staging git clone https://github.com/<your-org>/client-portal-laravel.git \
  /var/www/client-portal-laravel-staging/releases/$(date +%Y%m%d%H%M%S)/
# Switch into the release; the deploy script takes over from the next deploy onward.
```

Then from your dev box, run the deploy:

```bash
node scripts/deploy/deploy.cjs --env=staging --migrate
```

The first deploy will fail the post-deploy health check (no `/up` content yet) and auto-rollback — that's expected; it's just confirming the wiring. After the first successful deploy, run the seeder:

```bash
ssh ec2 'cd /var/www/client-portal-laravel-staging/current && sudo -u www-staging php artisan db:seed --class=StagingSeeder'
```

### 3.2  Staging .env (YOU on EC2)

- [ ] Copy [`env.staging.example`](env.staging.example) to `/var/www/client-portal-laravel-staging/shared/.env`.
- [ ] Replace every `AUDIT-FIRST` placeholder per the inline comments. The audit list is:
  - `APP_KEY` (run `php artisan key:generate` after first install; verify it differs from prod)
  - `APP_TIMEZONE` (must match prod)
  - `DB_PASSWORD` (from `/home/ubuntu/.secrets/staging-db.txt`)
  - `REVERB_APP_KEY`, `REVERB_APP_SECRET` (fresh, differ from prod)
  - `STRIPE_KEY`, `STRIPE_SECRET`, `STRIPE_WEBHOOK_SECRET`, `STRIPE_API_VERSION` (from Phase 6.1)
  - `SENTRY_DSN` (from Phase 9.1)
  - `UPTIME_TOKEN`, `PURGE_HC_URL`, `SCHEDULER_HC_URL` (from Phase 9.2 / 9.3)
- [ ] Set permissions:
  ```bash
  sudo chmod 600 /var/www/client-portal-laravel-staging/shared/.env
  sudo chown www-staging:www-data /var/www/client-portal-laravel-staging/shared/.env
  ```
- [ ] **Backup-on-edit policy:** every future edit must `cp .env .env.$(date +%s).bak` first.

---

## Phase 5 — Supervisor / cron / logrotate (YOU on EC2)

```bash
sudo install -m 0644 -o root scripts/staging/server-configs/supervisor-staging-reverb.conf  /etc/supervisor/conf.d/client-portal-staging-reverb.conf
sudo install -m 0644 -o root scripts/staging/server-configs/supervisor-staging-worker.conf  /etc/supervisor/conf.d/client-portal-staging-worker.conf
sudo install -m 0644 -o root scripts/staging/server-configs/cron-staging-scheduler          /etc/cron.d/client-portal-staging-scheduler
sudo install -m 0644 -o root scripts/staging/server-configs/logrotate-staging               /etc/logrotate.d/client-portal-staging
sudo install -m 0644 -o root scripts/staging/server-configs/cron-staging-schema-drift       /etc/cron.d/client-portal-staging-schema-drift
sudo install -m 0755 -o root scripts/staging/schema-drift-check.sh                          /usr/local/bin/scopeforged-schema-drift-check.sh
sudo touch /var/log/client-portal-staging-schedule.log /var/log/client-portal-staging-schema-drift.log
sudo chown www-staging:www-data /var/log/client-portal-staging-schedule.log

sudo supervisorctl reread && sudo supervisorctl update
sudo supervisorctl status client-portal-staging-reverb client-portal-staging-worker

# Validate logrotate
sudo logrotate -d /etc/logrotate.d/client-portal-staging   # dry run
sudo logrotate -f /etc/logrotate.d/client-portal-staging   # forced real run
```

- [ ] Tail scheduler log for 70s to confirm it fires: `sudo tail -f /var/log/client-portal-staging-schedule.log`
- [ ] CloudWatch alarm on `/var/www` utilization ≥ 75% (warning) / ≥ 90% (page). Route to the same channel as existing prod alarms.

---

## Phase 6 — Third-party integrations (YOU on Stripe dashboard)

- [ ] Stripe Dashboard → Test mode → Developers → Webhooks → **Add endpoint**:
  - URL: `https://staging.api.scopeforged.com/webhooks/stripe`
  - **Pin API version** to match production's pinned version.
  - Subscribe to the same event list as production (audit `app/Services/Websites/Webhooks/WebhookDispatcher::HANDLERS`).
- [ ] Copy the signing secret into staging `.env` `STRIPE_WEBHOOK_SECRET`.
- [ ] Trigger a `customer.created` test event from Stripe CLI: `stripe trigger customer.created --api-key=sk_test_...` — confirm it lands in the staging queue.
- [ ] Test-mode hygiene rule (record in [STAGING.md](../../docs/guides/STAGING.md) §6): only use `*@example.com` or `*+stagingtest@scopeforged.com` addresses.
- [ ] Document the integration matrix in [STAGING.md](../../docs/guides/STAGING.md) §5 (Slack/ClickUp/Postmark — default disabled).

---

## Phase 9 — Observability (YOU on Sentry / Healthchecks / UptimeRobot)

### 9.1  Sentry

- [ ] Create a separate Sentry project `scopeforged-staging` (or use `environment=staging` tag on the existing project — decide based on cost).
- [ ] Set `SENTRY_DSN=` and `SENTRY_ENVIRONMENT=staging` in staging `.env`.
- [ ] Trigger a forced exception on staging (e.g. visit a debug route) and confirm it lands in the staging project, NOT prod's.
- [ ] Sentry alert rules: new issue type → dev channel; volume spikes do NOT page.

### 9.2  Uptime

- [ ] Configure UptimeRobot / BetterStack to monitor `https://staging.<each-of-the-five>.scopeforged.com/up`.
- [ ] Send `X-Uptime-Token: <secret>` header (value from §2.3 / staging `.env` `UPTIME_TOKEN`).
- [ ] Cadence: every 5 minutes. Alerts route to dev channel only, not on-call.

### 9.3  Scheduler heartbeat

- [ ] Create a Healthchecks.io check named `scopeforged-staging-scheduler`, grace 30 min, expected period 10 min.
- [ ] Add a `Schedule::call(fn () => Http::get('<HC_URL>'))->everyTenMinutes()` block in `routes/console.php` inside the existing `if (app()->environment('staging')) { ... }` block — this is a follow-up code change that lives with the rest of the plan's deferred items.
- [ ] Create a second check `scopeforged-staging-purge` keyed to the daily `staging:purge-old-data` run; put the URL in staging `.env` `PURGE_HC_URL`.
- [ ] Create a third check `scopeforged-staging-scheduler` (period 10 min, grace 30 min) keyed to the 10-min heartbeat; put the URL in staging `.env` `SCHEDULER_HC_URL`.

---

## Phase 8 — Validation

After every Phase 1–7 + 9 box is ticked:

```bash
ssh ec2 'cd /var/www/client-portal-laravel-staging/current && bash scripts/staging/validate.sh'
```

- [ ] Every check passes (no FAIL).
- [ ] Negative deploy test: intentionally break `/up` (e.g. set a bogus `DB_HOST` in staging `.env`), run a staging deploy, confirm the health check fails AND the auto-rollback fires. Restore the correct env afterward.

---

## Deferred / partial

Record anything that couldn't be done in this pass so the next operator has a clean handover:

- _e.g. "Stripe IP allowlist: deferred — CIDR list at https://stripe.com/docs/ips returned 404; revisit YYYY-MM-DD"_