# WorkOnClock Production Runbook

This runbook covers production migration, backup, restore, and schema-contract practices for WorkOnClock.

## 1. Deployment Roles

- Product owner: approves functional release scope.
- Technical owner: approves migrations, rollback plan, and infrastructure readiness.
- Privacy/security owner: approves monitoring, retention, and access-control posture.
- Operator: executes deployment and records timestamps, commit SHA, and verification results.

## 2. Environment Inventory

Required production environment variables:

- `DATABASE_URL`: MySQL connection string for the web/admin Prisma app.
- `DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASSWORD`, `DB_NAME`: NestJS runtime API DB settings, unless `DATABASE_URL` is also used there.
- `NEXTAUTH_URL`: public web/admin URL.
- `NEXTAUTH_SECRET`: strong random secret.
- `JWT_SECRET`: strong random secret for NestJS API tokens.
- `RESEND_API_KEY`: production email provider key.
- `EMAIL_FROM`: verified sender address.
- `CONTACT_TO`: destination for public contact form submissions.
- `EMAIL_REQUIRE_PROVIDER`: set to `true` in staging/production so missing provider credentials fail loudly.
- `CRON_SECRET`: secret for scheduled screenshot retention.
- `API_BASE_URL`: public API base URL used for uploaded screenshot preview URLs.
- `WOC_API_URL`: desktop API URL for packaged desktop builds.

Production infrastructure should include:

- Managed MySQL with automated backups.
- Private object storage for screenshots.
- Centralized logs.
- Error reporting.
- TLS on all public endpoints.
- Secrets managed outside source control.

### Resend Email Setup

1. Verify the sending domain in Resend.
2. Configure DNS records from Resend for SPF/DKIM and wait for verification.
3. Create a production API key with email send permission.
4. Set:
   - `RESEND_API_KEY`
   - `EMAIL_FROM`, for example `WorkOnClock <no-reply@your-domain.com>`
   - `CONTACT_TO`
   - `EMAIL_REQUIRE_PROVIDER=true`
5. Restart the web/admin service.
6. Send a smoke email:
   - `EMAIL_SMOKE_TO=you@your-domain.com npm run test:email -w @workonclock/web`
7. Verify these flows deliver:
   - registration email verification,
   - invite email,
   - password reset email,
   - contact form notification,
   - adhoc approval/rejection notification.

## 3. Migration Runbook

### Pre-Migration Checklist

1. Confirm the target commit SHA and release notes.
2. Confirm no unreviewed Prisma migrations exist.
3. Run locally or in staging:
   - `npm run build -w @workonclock/web`
   - `npm run build -w @workonclock/api`
   - `npm run test:tenant -w @workonclock/web`
   - `npm run test -w @workonclock/api`
4. Confirm current production schema version:
   - `npx prisma migrate status --schema apps/web/prisma/schema.prisma`
5. Take a pre-migration database backup.
6. Record backup location and checksum.
7. Confirm rollback owner and decision deadline.

### Migration Execution

1. Put the system in maintenance mode if the migration is destructive or long-running.
2. Run Prisma migrations:
   - `npx prisma migrate deploy --schema apps/web/prisma/schema.prisma`
3. Restart web/admin app.
4. Restart NestJS runtime API if shared tables or runtime queries changed.
5. Run post-migration smoke checks:
   - Web login.
   - Admin dashboard.
   - User list.
   - Project list.
   - Settings read/write.
   - Desktop login.
   - Tracking start/stop.
   - Screenshot upload.
6. Remove maintenance mode.
7. Record completion timestamp and verification results.

### Rollback Procedure

Prefer forward fixes for additive migrations. Use restore only if data integrity or core access is broken.

1. Stop web/admin and runtime API writers.
2. Restore the latest pre-migration database backup.
3. Deploy previous known-good application commit.
4. Restart services.
5. Verify login, tenant scoping, tracking, and screenshots.
6. Record incident notes and root cause.

## 4. Backup Procedure

### Database Backup

Recommended frequency:

- Automated daily full backup.
- Point-in-time recovery enabled if the managed provider supports it.
- Manual backup before every production migration.

Manual backup example:

```bash
mysqldump \
  --single-transaction \
  --routines \
  --triggers \
  --set-gtid-purged=OFF \
  "$DB_NAME" > "workonclock-$(date +%Y%m%d-%H%M%S).sql"
```

Verify backup:

```bash
mysql --database=workonclock_restore_check < workonclock-backup.sql
```

Keep backup metadata:

- Timestamp.
- Environment.
- Database host.
- Application commit SHA.
- Operator.
- File size.
- Checksum.
- Restore test result.

### Screenshot/Object Backup

Current local development stores screenshots under `apps/api/storage/uploads/screenshots/`.

Production should store screenshots in private object storage with:

- Bucket versioning where available.
- Lifecycle rules aligned with screenshot retention settings.
- Access logs.
- Encryption at rest.
- Disaster recovery replication if required by customer commitments.

## 5. Restore Procedure

### Database Restore

1. Stop application writers.
2. Create a fresh restore database or clear the target database.
3. Restore dump:

```bash
mysql "$DB_NAME" < workonclock-backup.sql
```

4. Run validation queries:
   - Count organizations, users, roles, projects, time entries, screenshots.
   - Confirm `woc_organization_settings` rows exist for active organizations.
   - Confirm no old non-prefixed tables are active.
5. Start services.
6. Run smoke checks.

### Object Restore

1. Restore screenshot objects to the configured private bucket/prefix.
2. Confirm object keys match `woc_screenshots.objectKey` or stored preview metadata.
3. Regenerate signed URLs if the access pattern requires it.
4. Verify admin screenshot detail pages.

## 6. Schema Contract Direction

Current state:

- Web/admin uses Prisma models.
- NestJS runtime API uses direct SQL.
- `packages/shared` contains hand-maintained domain and permission types.

Recommended direction:

1. Keep Prisma as the source of truth for database shape.
2. Add generated or validated DTO contracts for API payloads before expanding external integrations.
3. Keep permission keys and defaults in `packages/shared`.
4. Add contract tests that verify:
   - Shared `TrackingPolicy` matches web settings and NestJS policy output.
   - Shared `ScreenshotEvent` matches NestJS screenshot responses.
   - Shared role permission defaults match seeded Prisma permission records.
5. Evaluate one of these options:
   - Generate shared types from Prisma for model-like data.
   - Generate OpenAPI/Zod schemas for API request/response contracts.
   - Keep hand-written shared types but enforce them with contract tests.

Short-term recommendation: keep hand-written shared types, add contract tests, and avoid generating client-facing types directly from persistence models until API response envelopes are normalized.

## 7. Production Release Gate

Do not externally release screenshot monitoring until all are complete:

- Migration runbook tested.
- Backup restore tested.
- Private screenshot storage configured.
- `CRON_SECRET` configured.
- Email provider configured.
- Tenant isolation smoke test passing.
- Desktop runtime smoke test passing.
- Legal/privacy launch gate completed.
- Signed desktop installers available for target platforms.