# Turnstile Widget Pool Management This document covers the Turnstile **widget pool** - the set of Cloudflare Turnstile widgets we maintain, the database records that shadow them, and the assignment logic that hands a widget to a newly activated custom domain. It is distinct from `docs/TURNSTILE_SETUP.md`, which covers how Turnstile is wired into public forms (site keys, client hook, verification). If you are adding Turnstile to a new form, start there. If you are working on provisioning, assignment, or exhaustion runbooks, you are in the right place. Read this before modifying `lib/turnstile-pool-management.ts`, `lib/turnstile-pool.ts`, `lib/turnstile.ts`, or `scripts/provision-turnstile-pool.ts`. --- ## 1. Why a widget pool exists Cloudflare Turnstile enforces a hard limit on the number of allowed hostnames per widget. The limit depends on plan tier; the free tier allows 10 per widget, enterprise allows 200 (`lib/turnstile.ts:9`). The default in our code is 200 (`lib/turnstile.ts:9,15`), overridable via `TURNSTILE_HOSTNAME_LIMIT`. Every dealer custom domain (e.g. `bobsoil.com`) must be an allowed hostname on some widget, because forms on that dealer's site submit tokens that are verified against that widget's secret. Our base domains (`myamsoil.com`, `myamsoil.ca`, `shopamsoil.com`, `shopamsoil.ca`, `localhost`) live on a single **primary** widget (`lib/turnstile-pool.ts:17-23`, `scripts/provision-turnstile-pool.ts:49-55`). For custom domains we maintain a **pool** of additional widgets. Each pool widget holds up to `maxHostnames` custom domains. When a dealer activates a custom domain we assign them to a pool widget with available capacity. The default provisioned pool is 20 widgets × 10 hostnames = 200 custom domains (`scripts/provision-turnstile-pool.ts:65`). Lead forms only exist on Growth and higher tiers, so starter-tier dealers don't consume pool capacity (`lib/turnstile-pool-management.ts:24-35`). --- ## 2. Pool architecture **Model:** `TurnstileWidget` (`prisma/schema.prisma:378-403`). Relevant fields: - `cloudflareWidgetId` - widget ID/sitekey at Cloudflare. Used for all Cloudflare API calls (`prisma/schema.prisma:381`). - `sitekey` - public sitekey served to the browser. - `secretKeyEncrypted` - AES-256-GCM ciphertext of the verification secret. Requires `TURNSTILE_ENCRYPTION_KEY` to decrypt (`lib/turnstile-secrets.ts`). - `isPrimary` - true for the single base-domains widget; false for pool widgets (`prisma/schema.prisma:388`). - `hostnameCount` - **denormalized** counter of currently assigned hostnames. This is the field the assignment loop reads to pick a widget and conditionally increments (`prisma/schema.prisma:389`). - `maxHostnames` - per-widget cap. Default 10; comment notes Cloudflare's API enforces 10 even though public docs say 15 (`prisma/schema.prisma:390`). - `isActive` - soft flag; inactive widgets are skipped by both the assigner and the lookup (`prisma/schema.prisma:393`). **Dealer → widget link:** `Dealer.turnstileWidgetId` (FK, see `prisma/schema.prisma:396` for the back-reference `dealers`). A dealer has at most one assigned pool widget; its custom domain is a hostname on that widget at Cloudflare. **Why denormalize `hostnameCount`?** Because we need an indexable, cheaply-updatable number for the race-safe conditional update in assignment. The source of truth is Cloudflare's widget config (the `domains` array on the widget resource), but reading that on every assignment would require a round-trip and wouldn't support atomic conditional updates. The counter must stay in sync with Cloudflare - see the rollback section below. **Domain → widget lookup at request time:** `getWidgetForDomain(domain)` (`lib/turnstile-pool.ts:73-111`) returns the primary widget for base domains and the assigned pool widget for an active custom domain. This is what `verifyTurnstile` consults to pick the correct secret when a form is submitted (`lib/turnstile.ts:94-103`). --- ## 3. Optimistic locking and retry Two concurrent custom-domain activations could both read "widget X has 9/10 used" and both try to claim the last slot. The assignment loop in `assignDomainToWidget` (`lib/turnstile-pool-management.ts:22-141`) handles this with optimistic locking: 1. **Find candidate.** Query all active, non-primary widgets and pick the first with `hostnameCount < maxHostnames` (`lib/turnstile-pool-management.ts:47-58`). We sort `desc` by `hostnameCount` to fill existing widgets before starting a new one. Filtering happens in-memory because Prisma can't compare two columns in a `where` clause - this is called out in `lib/turnstile-pool.ts:37-41`. 2. **Conditional update.** Attempt to increment `hostnameCount` with a guard: `where: { id, hostnameCount: { lt: maxHostnames } }` (`lib/turnstile-pool-management.ts:65-71`). If another request got there first, `updateResult.count === 0`. 3. **Retry with a different widget.** On a 0-count update, add the widget id to `triedWidgetIds` and loop. `MAX_ASSIGNMENT_RETRIES = 3` (`lib/turnstile-pool-management.ts:6`). 4. **Bind dealer → widget.** On successful reservation, set `dealer.turnstileWidgetId` (`lib/turnstile-pool-management.ts:85-88`). 5. **Add hostname at Cloudflare.** Call `addTurnstileHostname(domain, cloudflareWidgetId)` (`lib/turnstile.ts:324-396`) to update the widget's allowed domains via the Cloudflare API. 6. **Rollback on Cloudflare failure.** If the API call throws, run a transaction to unset `dealer.turnstileWidgetId` and decrement `hostnameCount` (`lib/turnstile-pool-management.ts:101-130`), then re-throw. Without this the counter drifts above the real Cloudflare state, leaking a slot every time the API fails. Removal (`removeDomainFromWidget`, `lib/turnstile-pool-management.ts:150-187`) is the mirror operation: unset the FK, decrement the counter, and remove the hostname at Cloudflare - all in a single `$transaction`. **Implication.** The counter is correct as long as every assign/unassign path goes through these two functions. Do not write new code that pokes `hostnameCount` directly or mutates widget hostnames at Cloudflare from anywhere else. If you do, you will break future assignments and the pool-exhaustion runbook below will give misleading readings. --- ## 4. Provisioning capacity The provisioning script is `scripts/provision-turnstile-pool.ts`. It creates widgets at Cloudflare and registers them in the database. **Required env vars** (`scripts/provision-turnstile-pool.ts:235-250`): - `CLOUDFLARE_API_TOKEN` - must have Turnstile write permission. Not required in `--import-bootstrap` mode. - `CLOUDFLARE_ACCOUNT_ID` - same; not required for import mode. - `TURNSTILE_ENCRYPTION_KEY` - 32-byte hex, generated via `openssl rand -hex 32`. Used by `lib/turnstile-secrets.ts` to encrypt the Cloudflare-issued secret before storing it in `secretKeyEncrypted`. - `DATABASE_URL` - the target environment's Postgres. **Common invocations** (`scripts/provision-turnstile-pool.ts:14-31`): ```bash # Create the primary widget (base domains only) — once per Cloudflare account npx tsx scripts/provision-turnstile-pool.ts --primary # Create N pool widgets (default 20) npx tsx scripts/provision-turnstile-pool.ts --count=20 # Preview without writing npx tsx scripts/provision-turnstile-pool.ts --count=20 --dry-run ``` **Cross-environment bootstrapping.** When dev, staging, and prod share a Cloudflare account, widgets only need to be created at Cloudflare once. For subsequent environments, export a bootstrap file and import it into the target DB (`scripts/provision-turnstile-pool.ts:19-23,258-327`): ```bash # Source environment — create + export npx tsx scripts/provision-turnstile-pool.ts --count=20 \ --export-bootstrap=turnstile-bootstrap.json # Target environments — register rows without calling Cloudflare npx tsx scripts/provision-turnstile-pool.ts \ --import-bootstrap=turnstile-bootstrap.json ``` The bootstrap file contains sitekeys, encrypted secrets, and widget IDs. It is safe to transfer only because the secrets are encrypted under `TURNSTILE_ENCRYPTION_KEY`, which the file does **not** contain. Target environments need the same encryption key in their env to decrypt at runtime. The file is gitignored by default. **Adding capacity.** When utilization is high (see the runbook below), run the script with `--count=N` where N is the number of new widgets. Existing widgets are detected by name and skipped (`scripts/provision-turnstile-pool.ts:340-376`). Cloudflare API calls are rate-limited with a 300ms delay between creates (`scripts/provision-turnstile-pool.ts:434-436`). --- ## 5. Pool exhaustion runbook **Symptom.** A dealer activating a custom domain sees an error referencing "Turnstile widget pool exhausted." The error surfaces from `lib/turnstile-pool-management.ts:59-61`. **Diagnose.** 1. Check pool utilization via `getPoolStatus()` (`lib/turnstile-pool.ts:116-149`). This returns every widget's `hostnameCount / maxHostnames`, total used, total capacity, available slots, and utilization percent. ```ts import { getPoolStatus } from '@/lib/turnstile-pool'; console.log(await getPoolStatus()); ``` 2. Confirm the DB counters match Cloudflare. Fetch each widget's `domains` array via `getTurnstileWidget(cloudflareWidgetId)` (`lib/turnstile.ts:215-242`) and compare `domains.length` to `hostnameCount` in the DB. Drift means an assignment or removal ran partially - see section 3. 3. Verify `isActive = true` on the widgets you expect to be available. The assignment query filters on this (`lib/turnstile-pool-management.ts:48-54`). **Recover.** - **Fastest:** add more widgets. ```bash npx tsx scripts/provision-turnstile-pool.ts --count=10 ``` New widgets are immediately available to the assignment loop. - **If counters have drifted** (DB says capacity free but Cloudflare is full, or vice versa): run `npm run sync-turnstile` (see `docs/TURNSTILE_SETUP.md`) to reconcile hostnames between DB-tracked custom domains and Cloudflare widget config. After reconciliation, update `hostnameCount` from the ground truth (`widget.domains.length` at Cloudflare). A one-off script or manual `prisma studio` fix is appropriate here - this is a correctness fix, not a hot path. - **If a widget is stuck inactive:** flip `isActive = true` once you've confirmed the widget still exists at Cloudflare. **Prevent.** - Monitor `getPoolStatus().utilizationPercent`. The existing guidance (`docs/TURNSTILE_SETUP.md` section "Capacity Planning") is to add widgets before 80%. - A near-exhaustion alert should fire well ahead of the first failing activation - three retries is enough to absorb small races, not a capacity shortfall. --- ## 6. Cross-references - `docs/TURNSTILE_SETUP.md` - how Turnstile is used in forms, env vars, manual setup, sync script, and graceful degradation. - `lib/turnstile.ts` - low-level Cloudflare API helpers (`addTurnstileHostname`, `removeTurnstileHostname`, `getTurnstileWidget`, `verifyTurnstile`). - `lib/turnstile-secrets.ts` - AES-256-GCM helpers used by provisioning and pool lookup.