# DeployHQ Rollout Automation **Date:** 2026-07-09 **Status:** Proposed (scripts committed; root setup pending) ## Problem DeployHQ uploads files and runs "After Changes" SSH commands as the deploy user (`acdev3` on dev, `amsoildlp` on staging/production). That user can run `npm install`, `prisma generate`, `prisma migrate deploy`, and `npm run build`, but the PM2 daemon that serves the apps belongs to **root**, so the final `pm2 reload` has always required a manual root SSH session. Deploys therefore "succeed" in DeployHQ while the old process keeps serving until someone logs in and reloads. A user group cannot fix this: PM2 has no group-access concept. Its IPC socket lives under `/root/.pm2/` and answers only to root. The two workable mechanisms are a narrow sudoers rule on a root-owned wrapper (this plan), or migrating to per-user PM2 daemons (see Future Work). ## Design One committed script, `scripts/server/rollout.sh`, runs the whole pipeline as the deploy user. The single privileged step escalates through a fixed-argument sudoers rule to a root-owned wrapper that reloads PM2 via each app directory's own `ecosystem.config.js`: ``` DeployHQ (After Changes, as deploy user) └── bash scripts/server/rollout.sh ├── npm install ├── npx prisma generate ├── npx prisma migrate deploy ├── npm run build (staging/production only) └── sudo -n /usr/local/sbin/amsoil-pm2-reload (root wrapper) ├── cd ├── pm2 startOrReload ecosystem.config.js --update-env └── pm2 save ``` Design points: - **`scripts/server/` is the only scripts directory whitelisted in `.gitignore`** (`scripts/*` is otherwise ignored), so the script actually ships with deploys. A `scripts/rollout.sh` would be silently untracked. - **The wrapper is installed outside the deploy tree.** Everything under the app directory is writable by the deploy user, so sudoers must never reference a repo path. The repo copy (`scripts/server/pm2-reload-wrapper.sh`) is a template; the live copy is root's. - **Sudoers entries are exact-argument matches** (no wildcards), one per environment valid on that box. - **`pm2 startOrReload` instead of `reload`** so the command also recovers from a stopped/crashed app (e.g. after a reboot) instead of erroring. - **`sudo -n`** makes a missing/typoed sudoers rule fail loudly in the DeployHQ log instead of hanging on a password prompt. - **A per-user `flock`** serializes rollouts on the shared staging/production box so two `next build` runs never compete for memory. - **Build out of place, swap atomically**: `next build` used to write into the live `.next`, deleting chunks the running server still needed - every deploy errored SSR (dealer page) requests for the whole ~6 minute build, and a build killed mid-write left a landmine that broke the next reload (both observed on production 2026-07-13). The build cannot simply use a different distDir and be renamed afterwards: Next inlines `config.distDir` into every compiled route module at build time (`__NEXT_RELATIVE_DIST_DIR` in `next/dist/build/define-env.js`), so the built output only works from the exact distDir name it was built with. The pipeline therefore builds with the default `.next` inside a hardlink-mirrored sibling workspace (`.rollout-build-`, created with `rsync --link-dest` in seconds, no file data copied), verifies completeness (`BUILD_ID` + `prerender-manifest.json`, the latter written near the end of the build), then swaps: live `.next` -> `.next-prev`, workspace `.next` -> live `.next`, and reloads immediately. The served build is never touched until the new one is complete; a failed or killed build leaves the site serving the old build untouched. **Rollback one generation:** `mv .next .next-bad && mv .next-prev .next`, then `sudo -n /usr/local/sbin/amsoil-pm2-reload ` (files only - migrations are not rolled back). - **Migrate before build**: `next build` prerenders force-static pages that query the database (the directory hub calls `getDealersForApex`, which selects dealer columns through Prisma), so building against the old schema would fail any deploy whose build-time queries use new columns or tables. The old process keeps serving while `migrate deploy` runs, so migrations must stay backward compatible (expand/contract). Failure mode: if the build or a later step fails after a successful migrate, the old code keeps serving against the already-migrated database, which the backward-compatibility rule makes safe, but the box needs manual follow-up before redeploying (applied migration statements are not rolled back). - **`npm install`, not `npm ci`**: matches the existing DeployHQ pipelines and stays incremental on the shared host. `npm ci` would delete `node_modules` on every deploy; switch to it only if lockfile drift ever becomes a real problem. - **Dev gets a stop, clear `.next`, start cycle instead of a live reload.** The dev server runs `next dev` (Turbopack), whose persistent cache in `.next` corrupts when the server dies mid-compile - observed 2026-07-13 when a deploy rewrote `next.config.mjs` (triggering next dev's own restart) and the pipeline's reload killed that restart 18 seconds later; every subsequent compile panicked (`TurbopackInternalError: Failed to write app endpoint`) until the cache was cleared. The wrapper therefore stops the dev app, removes `.next` while nothing is running, and starts fresh; `next dev` recompiles on demand, so the first request after a deploy takes a few seconds. Two invariants remain: never delete `.next` under a live `next dev` (500s until rebuild), and never clear it on staging/production, where `.next` is the served `next build` output. **After changing the wrapper, re-install it on the affected box** (the live copy in `/usr/local/sbin` does not update from git): ```bash install -o root -g root -m 0755 \ /home/acdev3/amsoil-dlp.acdev3.com/scripts/server/pm2-reload-wrapper.sh \ /usr/local/sbin/amsoil-pm2-reload ``` ## One-Time Root Setup ### Dev box (acdev3) ```bash sudo install -o root -g root -m 0755 \ /home/acdev3/amsoil-dlp.acdev3.com/scripts/server/pm2-reload-wrapper.sh \ /usr/local/sbin/amsoil-pm2-reload sudo visudo -f /etc/sudoers.d/amsoil-deploy # add exactly: # acdev3 ALL=(root) NOPASSWD: /usr/local/sbin/amsoil-pm2-reload dev ``` ### Staging/production box (amsoildlp) ```bash sudo install -o root -g root -m 0755 \ /home/amsoildlp/public_html/aimclear.biz/scripts/server/pm2-reload-wrapper.sh \ /usr/local/sbin/amsoil-pm2-reload sudo visudo -f /etc/sudoers.d/amsoil-deploy # add exactly: # amsoildlp ALL=(root) NOPASSWD: /usr/local/sbin/amsoil-pm2-reload staging, /usr/local/sbin/amsoil-pm2-reload production ``` **Keeping paths in sync:** the environment-to-directory mappings are intentionally duplicated in `rollout.sh` and the wrapper (the wrapper cannot reference anything under the deploy-user-writable tree). If an app directory ever moves, update both scripts and deliberately re-install the wrapper to `/usr/local/sbin`; the sudoers lines are path-independent and only change if the wrapper's install path or argument set changes. ### Verify (per box, before wiring DeployHQ) ```bash sudo -l -U acdev3 # or amsoildlp; must list the wrapper entries su - acdev3 -c 'sudo -n /usr/local/sbin/amsoil-pm2-reload dev' # reloads, no password ``` ## DeployHQ Configuration Replace the existing After Changes SSH commands with one command per server: | Server | Command | | --- | --- | | acdev3 (Development) | `bash /home/acdev3/amsoil-dlp.acdev3.com/scripts/server/rollout.sh dev` | | amsoildlp (Staging) | `bash /home/amsoildlp/public_html/aimclear.biz/scripts/server/rollout.sh staging` | | amsoildlp (Production) | `bash /home/amsoildlp/public_html/amsoil.aimclear.com/scripts/server/rollout.sh production` | Production deploys stay **manually queued** in DeployHQ; the command only makes the queued deploy complete hands-off (no root SSH afterwards). Invoking via `bash ` sidesteps any executable-bit loss in DeployHQ's file sync. The `.gitattributes` rule (`*.sh text eol=lf`) guarantees the script arrives LF; CRLF would fail with `$'\r': command not found`. ## Pre-Flight Checks 1. **Ecosystem file scoping.** ✅ Confirmed 2026-07-09: each app directory has its own `ecosystem.config.js` declaring only that environment's app, and reloads are always run from inside the matching directory. The wrapper's `cd`-then-reload mirrors that practice, so a staging deploy cannot bounce production. 2. **Process owner.** ✅ Confirmed 2026-07-09: apps run as root under root's PM2 on both servers. This is the accepted current topology and this plan does not change it (the wrapper reloads as root, same as the manual command it replaces). Moving apps off root is deliberately out of scope; see Future Work. 3. **Reboot persistence.** `pm2 startup` + `pm2 save` should already be in place for root's daemon; `pm2 save` in the wrapper keeps the resurrect list current. ## Security Notes This automation adds no new trust boundary. Root already reloads the deploy-user-writable `ecosystem.config.js` by hand on every deploy today; the wrapper automates exactly that action with a fixed argument list. The standing topology (root's PM2 executing apps from user-owned directories) is the thing to revisit separately; see Future Work. Rollback is two file deletions per box (`/etc/sudoers.d/amsoil-deploy` and `/usr/local/sbin/amsoil-pm2-reload`) plus restoring the previous DeployHQ SSH commands. ## Future Work: Per-User PM2 Daemons The structurally cleaner endgame is one PM2 daemon per app user (`sudo pm2 startup systemd -u amsoildlp --hp /home/amsoildlp`, then apps started as that user). The deploy user then owns its own reload, the sudoers rule and wrapper get deleted, and apps stop having any relationship to root's process manager. It requires a one-time migration of live processes (delete from root's PM2, start in the user's, re-run `pm2 save` on both), so schedule it as deliberate maintenance rather than bundling it with this change. Note the MCP server (`amsoil-dlp-mcp`) also lives in root's PM2 on production and would need the same decision.