# Cron Jobs Inventory of scheduled tasks, how they're configured, and what to do at 2am when one is stuck or silently failing. See also: - [ADMIN_IMPERSONATION.md](./ADMIN_IMPERSONATION.md) - impersonation is stateless; there is no cleanup cron to run - [DATABASE_BACKUP_RESTORE.md](./DATABASE_BACKUP_RESTORE.md) - backup cron runbook ## Cron Inventory | Name | Schedule | Endpoint / Script | Scheduled via | Owner | Notes | | ------------------------------ | ----------------------------- | -------------------------------------------------- | ------------------------------ | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | Cleanup expired registrations | Not currently scheduled (gap) | `POST/GET /api/cron/cleanup-expired-registrations` | External (Bearer token auth) | Ops | Route exists at `app/api/cron/cleanup-expired-registrations/route.ts`. Intended daily. Deletes `PendingRegistration` rows where `expiresAt < now`. | | GSC drain queue | `*/5 * * * *` (every 5 min) | `POST /api/cron/gsc-drain` | External (Bearer token auth) | Ops | Drains `GscJob` queue, respects Indexing API quota and preflight checks. Route at `app/api/cron/gsc-drain/route.ts`. | | GSC daily audit | `0 3 * * *` (03:00 daily) | `POST /api/cron/gsc-audit` | External (Bearer token auth) | Ops | Enqueues daily `inspect_url` jobs for active dealer homepages. Sends anomaly summary if issues detected. Route at `app/api/cron/gsc-audit/route.ts`. | | Daily database backup | `0 2 * * *` (America/Toronto) | `scripts/server/backup-to-remote.sh` | Server crontab (postgres user) | Ops | Defined at `scripts/server/backup-to-remote.sh:13`. Dumps prod + staging DBs, copies via rclone, emails alerts. | No Vercel cron config (`vercel.json` absent). No GitHub Actions cron (`.github/workflows/` contains only `claude-code-review.yml`, `claude.yml`, `test.yml` - none are scheduled). ## How Scheduling Is Set Up The platform does not run on Vercel. Production and staging are hosted on a self-managed server (see `PRODUCTION_SERVER_SETUP.md`), with Next.js under PM2. Scheduling options in active use: ### Server crontab The database backup runs from a system crontab owned by the `postgres` user: ```cron # /etc/crontab or postgres' crontab 0 2 * * * bash /home/amsoildlp/scripts/backup-to-remote.sh ``` Defined in the script's usage comment at `scripts/server/backup-to-remote.sh:12-13`. The script must `cd /` before running because `postgres` cannot read the app directory. Lock file at `/var/lock/amsoil-backup.lock` prevents concurrent runs (`scripts/server/backup-to-remote.sh:123-128`). ### HTTP cron (Bearer-token auth) `/api/cron/cleanup-expired-registrations` is designed to be called by an external scheduler that can send `Authorization: Bearer `. It accepts both `GET` (for Vercel-style schedulers) and `POST` (`app/api/cron/cleanup-expired-registrations/route.ts:27,67-69`). Auth uses a timing-safe comparison (`app/api/cron/cleanup-expired-registrations/route.ts:17-23`). To schedule it from the server, add to root crontab: ```cron # Hourly cleanup of expired pending registrations 0 * * * * curl -fsS -X POST -H "Authorization: Bearer $(cat /etc/amsoil/cron-secret)" https://amsoil.aimclear.com/api/cron/cleanup-expired-registrations ``` ### GSC drain queue (`/api/cron/gsc-drain`) **Schedule:** every 5 minutes (`*/5 * * * *`) **Route:** `app/api/cron/gsc-drain/route.ts` **Auth:** Bearer token (`CRON_SECRET`) Processes the `GscJob` queue, respecting Google Indexing API quota and preflight checks. Each execution drains pending jobs up to the configured Indexing API rate limit and validates against GSC preflight rules before sending indexing requests. **Crontab entry:** ```cron # Drain GSC job queue every 5 minutes */5 * * * * curl -fsS -X POST -H "Authorization: Bearer $(cat /etc/amsoil/cron-secret)" https://amsoil.aimclear.com/api/cron/gsc-drain ``` ### GSC daily audit (`/api/cron/gsc-audit`) **Schedule:** daily at 03:00 UTC (`0 3 * * *`) **Route:** `app/api/cron/gsc-audit/route.ts` **Auth:** Bearer token (`CRON_SECRET`) Enqueues daily `inspect_url` jobs for all active dealer homepages and monitors for indexing anomalies. If anomalies are detected (indexing failures, quota exhaustion, or policy violations), sends a summary email to `GSC_ALERT_EMAIL`. **Crontab entry:** ```cron # Daily GSC audit at 03:00 0 3 * * * curl -fsS -X POST -H "Authorization: Bearer $(cat /etc/amsoil/cron-secret)" https://amsoil.aimclear.com/api/cron/gsc-audit ``` ## Impersonation: no cleanup cron needed Impersonation is **stateless** - a support session lives entirely in the signed JWT cookie and auto-expires after 1 hour (JWT `exp` + cookie `maxAge`). There is no server-side session state to clean up. (An earlier DB-backed design tracked sessions in `User.impersonatedById/At/Expiry` and needed a `clearExpiredImpersonations()` cron, but that design was removed; those columns are vestigial and unused.) See [ADMIN_IMPERSONATION.md](./ADMIN_IMPERSONATION.md). ## Failure Alerting ### Current state - **Backup script:** sends success _and_ failure emails via Resend (`scripts/server/backup-to-remote.sh:58-94`, `107-114`, `218-225`). Subject lines `[BACKUP OK] ...` and `[BACKUP FAILED] ...` go to `team-amsoil-dlp@aimclear.com`. This pattern was added in commit 56161bb. - **Cron routes (`/api/cron/*`):** log to the server logger on failure (`app/api/cron/cleanup-expired-registrations/route.ts:57-59`). **No email is sent.** If the external cron caller gets a 500 back, there's no pager. - **External cron caller itself:** no heartbeat. If cron doesn't fire at all (crontab unloaded, network dead), nothing notices. ### Proposal Extend the backup-email pattern (56161bb) to HTTP cron routes and external callers: 1. **Server-side:** in each `/api/cron/*` route's error path, send a Resend email to `team-amsoil-dlp@aimclear.com` with subject `[CRON FAILED] - `. Use the existing email service (`lib/email-service.ts` or similar - cross-reference `EMAIL_SERVICE.md`). 2. **Heartbeat:** on success, optionally PUT to a healthcheck service (Healthchecks.io, BetterStack) that alerts if no ping arrives within the expected window. Cheaper than running our own. 3. **Curl wrapper:** the crontab command should check the HTTP status and alert on non-2xx - `curl -fsS` already exits non-zero on HTTP errors; pipe into a send-failure-email helper. ## Incident Runbook ### "Broken cron" at 2am Symptom: pager fires `[BACKUP FAILED]` or you notice `pending_registrations` count climbing. 1. **Identify which cron.** - If backup: SSH to server, `tail /var/log/amsoil-backup.log`, look for the most recent error line. The script's `on_error` trap logs the failing line number (`scripts/server/backup-to-remote.sh:99-116`). - If HTTP cron: check server logs for the `[Cleanup]` prefix or equivalent. Check `systemd` or `syslog` for the crontab execution entry. 2. **Run manually.** - Backup: `sudo bash /home/amsoildlp/scripts/backup-to-remote.sh` - the lock file (`/var/lock/amsoil-backup.lock`) will prevent overlap with a scheduled run. - HTTP cron: `curl -X POST -H "Authorization: Bearer $CRON_SECRET" https://amsoil.aimclear.com/api/cron/`. 3. **Look for root cause.** Common: expired `CRON_SECRET`, missing env var, rclone remote unreachable, Postgres connection pool exhausted. Fix in place, don't just rerun. 4. **Post-incident.** If the failure went undetected (no alert fired), add or extend the Resend alert path for that cron before closing the incident. ## Related Files | File | Purpose | | ----------------------------------------------------- | ------------------------------------------------- | | `app/api/cron/cleanup-expired-registrations/route.ts` | Existing HTTP cron route | | `scripts/server/backup-to-remote.sh` | Daily DB backup + Resend alerts (pattern 56161bb) | | `.github/workflows/` | CI workflows - no scheduled jobs today |