# Production Server Setup Guide

This guide documents the changes needed on the staging VPS server to add the production deployment. Both staging and production will run on the same InterServer VPS.

**Production URL**: `amsoil.aimclear.com`
**Staging URL**: `aimclear.biz` (existing)
**Server**: vps3222305.trouble-free.net (InterServer VPS with cPanel/WHM)

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                        InterServer VPS                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Cloudflare ──► Apache ──► PM2 (Port 3000) ──► Staging App    │
│   (aimclear.biz)          (amsoil-dlp-staging)                 │
│                                                                 │
│   Cloudflare ──► Apache ──► PM2 (Port 3001) ──► Production App │
│   (amsoil.aimclear.com)   (amsoil-dlp-production)              │
│                                                                 │
│   PostgreSQL                                                    │
│   ├── amsoil_dlp_staging (staging database)                    │
│   └── amsoil_dlp_production (production database)              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

## 1. Database Setup

Create a separate production database and user.

```bash
# Generate a secure password
openssl rand -base64 24
# Save this - you'll need it for .env.local

# Access PostgreSQL
sudo -u postgres psql
```

In PostgreSQL shell:

```sql
-- Create production user
CREATE USER amsoil_production WITH PASSWORD 'YOUR_GENERATED_PASSWORD';

-- Create production database
CREATE DATABASE amsoil_dlp_production OWNER amsoil_production;

-- Grant privileges
GRANT ALL PRIVILEGES ON DATABASE amsoil_dlp_production TO amsoil_production;

\q
```

Test the connection:

```bash
psql -h localhost -U amsoil_production -d amsoil_dlp_production
```

## 2. Application Directory Setup

### Create Production Directory

```bash
mkdir -p /home/amsoildlp/public_html/amsoil.aimclear.com
cd /home/amsoildlp/public_html/amsoil.aimclear.com
```

### Deploy Application Code

Option A - Copy from staging and update:

```bash
cp -rp /home/amsoildlp/public_html/aimclear.biz/* /home/amsoildlp/public_html/amsoil.aimclear.com/
rm -rf /home/amsoildlp/public_html/amsoil.aimclear.com/.next
rm -rf /home/amsoildlp/public_html/amsoil.aimclear.com/node_modules
```

Option B - Fresh deployment via DeployHQ or git (preferred).

### Fix Ownership

```bash
chown -R amsoildlp:amsoildlp /home/amsoildlp/public_html/amsoil.aimclear.com
```

## 3. Environment Configuration

Create `/home/amsoildlp/public_html/amsoil.aimclear.com/.env.local`:

```env
# ===========================================
# DATABASE
# ===========================================
DATABASE_URL="postgresql://amsoil_production:PASSWORD@localhost:5432/amsoil_dlp_production?schema=public"

# ===========================================
# NEXTAUTH
# ===========================================
NEXTAUTH_URL="https://amsoil.aimclear.com"
NEXTAUTH_SECRET="<generate with: openssl rand -base64 32>"

# IMPORTANT: Set this to match NEXTAUTH_URL for ISR revalidation and internal API calls
NEXT_PUBLIC_BASE_URL="https://amsoil.aimclear.com"

# ===========================================
# OAUTH - Use PRODUCTION credentials
# ===========================================
# Google OAuth (configure in Google Cloud Console for production domain)
GOOGLE_CLIENT_ID="<production-google-client-id>"
GOOGLE_CLIENT_SECRET="<production-google-client-secret>"

# Apple OAuth (configure in Apple Developer Console for production domain)
APPLE_CLIENT_ID="<production-apple-service-id>"
APPLE_CLIENT_SECRET="<production-apple-private-key>"
APPLE_TEAM_ID="<your-apple-team-id>"
APPLE_KEY_ID="<your-apple-key-id>"

# ===========================================
# STRIPE - LIVE KEYS (not test keys!)
# ===========================================
STRIPE_SECRET_KEY="sk_live_..."
STRIPE_WEBHOOK_SECRET="whsec_..."
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY="pk_live_..."

# Stripe Price IDs - REQUIRED for checkout to work
# Get these from Stripe Dashboard > Products > [Product] > Pricing
STRIPE_PRICE_STARTER="price_..."
STRIPE_PRICE_GROWTH="price_..."
STRIPE_PRICE_ENHANCED="price_..."
STRIPE_PRICE_PROFESSIONAL="price_..."

# ===========================================
# CLOUDFLARE - For automatic subdomain DNS
# ===========================================
CLOUDFLARE_API_TOKEN="<your-cloudflare-api-token>"
CLOUDFLARE_ZONE_ID="<zone-id-for-aimclear-com>"
SERVER_IP_ADDRESS="69.169.111.27"

# ===========================================
# EMAIL - Choose ONE provider
# ===========================================
# Option A: Resend (recommended)
EMAIL_PROVIDER="resend"
RESEND_API_KEY="re_..."
EMAIL_FROM="noreply@amsoil.aimclear.com"

# Option B: SMTP (uncomment if using SMTP instead)
# EMAIL_PROVIDER="smtp"
# SMTP_HOST="smtp.example.com"
# SMTP_PORT="587"
# SMTP_USER="<smtp-username>"
# SMTP_PASS="<smtp-password>"
# SMTP_SECURE="true"
# EMAIL_FROM="noreply@amsoil.aimclear.com"

# ===========================================
# RATE LIMITING - Upstash Redis (optional but recommended)
# ===========================================
# Without Redis, rate limiting is per-instance only (in-memory)
# Get credentials from https://console.upstash.com/
UPSTASH_REDIS_REST_URL="https://...-us1.upstash.io"
UPSTASH_REDIS_REST_TOKEN="<your-upstash-token>"

# ===========================================
# APPLICATION
# ===========================================
NODE_ENV="production"
PORT="3001"

# Server Actions encryption key - REQUIRED for self-hosted deployments
# Without this, Next.js generates a new key on every build, causing
# "Failed to find Server Action" errors for users who loaded pages pre-deploy
NEXT_SERVER_ACTIONS_ENCRYPTION_KEY="<generate ONCE with: openssl rand -base64 32>"
```

Create symlink for Prisma:

```bash
cd /home/amsoildlp/public_html/amsoil.aimclear.com
ln -s .env.local .env
```

**Important differences from staging:**

- `PORT=3001` (staging uses 3000)
- `NEXTAUTH_URL` and `NEXT_PUBLIC_BASE_URL` point to production domain
- Use LIVE Stripe keys and price IDs, not test keys
- Use production OAuth credentials (must be configured in Google/Apple consoles for new domain)
- Separate database
- Separate Stripe webhook secret (configured in Stripe Dashboard)
- Email and Redis configs can be shared or separate depending on your needs

## 4. Install Dependencies and Build

```bash
cd /home/amsoildlp/public_html/amsoil.aimclear.com

# Run install, generate, and build as amsoildlp (not root)
# The prebuild script will block root builds to prevent .next ownership issues
sudo -u amsoildlp bash -c 'npm ci && npx prisma generate && npm run build'

# Run migrations (requires database access)
sudo -u amsoildlp npx prisma migrate deploy
```

> **Note:** Requires passwordless sudo to amsoildlp. If not configured: `echo 'root ALL=(amsoildlp) NOPASSWD: ALL' > /etc/sudoers.d/amsoildlp`

## 5. PM2 Configuration

Create `/home/amsoildlp/public_html/amsoil.aimclear.com/ecosystem.config.js`:

```javascript
module.exports = {
  apps: [
    {
      name: 'amsoil-dlp-production',
      script: 'node_modules/next/dist/bin/next',
      args: 'start --keepAliveTimeout 60000',
      cwd: '/home/amsoildlp/public_html/amsoil.aimclear.com',
      instances: 1,
      autorestart: true,
      watch: false,
      max_memory_restart: '1G',
      env: {
        NODE_ENV: 'production',
        PORT: 3001,
      },
    },
  ],
};
```

The `--keepAliveTimeout 60000` sets Node.js HTTP keep-alive to 60 seconds, ensuring the timeout chain is: Cloudflare (120s) > Next.js (60s) > Apache pool ttl (15s). This prevents Apache from sending requests on connections that Next.js has already closed.

Start the production app:

```bash
cd /home/amsoildlp/public_html/amsoil.aimclear.com
pm2 start ecosystem.config.js
pm2 save
```

Verify both apps are running:

```bash
pm2 list
# Should show:
# amsoil-dlp-staging    (port 3000)
# amsoil-dlp-production (port 3001)
```

## 6. Apache Proxy Configuration

### Add Domain in cPanel

1. Log into **cPanel** for the amsoildlp account
2. Go to **Domains** > **Domains**
3. Add `amsoil.aimclear.com` pointing to `/home/amsoildlp/public_html/amsoil.aimclear.com`

### Create SSL Proxy Configuration

```bash
mkdir -p /etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com
```

Create `/etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/proxy.conf`:

```apache
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:3001/
ProxyPassReverse / http://127.0.0.1:3001/

RewriteEngine On
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule ^/?(.*) ws://127.0.0.1:3001/$1 [P,L]
```

Create `/etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/proxy-headers.conf`:

```bash
echo 'RequestHeader set X-Forwarded-Proto "https"' | tee /etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/proxy-headers.conf
```

### Configure Dealer Domain ServerAliases

Dealer landing pages are served from separate domains (myamsoil.com, shopamsoil.com, etc.). Apache needs ServerAlias directives to route these requests to the production Next.js app.

Create `/etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/serveralias.conf`:

```apache
# Dealer subdomain routing - all dealer domains proxy to this Next.js app
# The proxy.ts middleware extracts the subdomain and rewrites to /dealers/[subdomain]
ServerAlias *.myamsoil.com myamsoil.com
ServerAlias *.myamsoil.ca myamsoil.ca
ServerAlias *.shopamsoil.com shopamsoil.com
ServerAlias *.shopamsoil.ca shopamsoil.ca
```

**Important:** Without these ServerAlias entries, requests to dealer subdomains (e.g., `dealername.myamsoil.com`) will not reach the Next.js app, resulting in 404 errors.

### Create HTTP Redirect Configuration

```bash
mkdir -p /etc/apache2/conf.d/userdata/std/2_4/amsoildlp/amsoil.aimclear.com
```

Create `/etc/apache2/conf.d/userdata/std/2_4/amsoildlp/amsoil.aimclear.com/proxy.conf`:

```apache
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
```

### Apply Apache Configuration

```bash
/scripts/rebuildhttpdconf
systemctl restart httpd
```

## 6b. Apache Proxy Connection Pool (Shared Balancer)

All VirtualHosts share a single connection pool to Next.js via `mod_proxy_balancer`. This replaced the previous per-VirtualHost pooling where each of the 100+ custom domain VHosts had its own isolated pool, causing CLOSE-WAIT connections to accumulate below cleanup thresholds and eventually freeze the server.

**Required modules** (installed via WHM > EasyApache 4 > Apache Modules):
- `mod_proxy_balancer`
- `mod_lbmethod_byrequests`

### Global Balancer Definition

**File:** `/etc/apache2/conf.d/includes/pre_virtualhost_global.conf` (at the top, before the `IncludeOptional` line):

```apache
# Shared proxy pool for Next.js backend
# All VirtualHosts share this single connection pool
<Proxy balancer://nextjs>
    BalancerMember http://127.0.0.1:3001 \
        connectiontimeout=3 \
        timeout=30 \
        retry=0 \
        keepalive=On \
        max=150 \
        smax=25 \
        ttl=15
</Proxy>
```

| Setting             | Value | Purpose                                                  |
| ------------------- | ----- | -------------------------------------------------------- |
| `max`               | 150   | Matches `MaxRequestWorkers 150` - pool can't bottleneck  |
| `smax`              | 25    | Cleanup fires when idle connections exceed 25             |
| `ttl`               | 15s   | Idle connections older than 15s get reaped during cleanup |
| `connectiontimeout` | 3s    | Fail fast on dead connections                             |
| `timeout`           | 30s   | Max time to wait for a response from Next.js              |
| `retry`             | 0     | Retry immediately on failure (don't mark backend as down) |
| `keepalive`         | On    | HTTP keep-alive, synced with Next.js keep-alive           |

**Why a shared balancer matters:** Apache prepends a VirtualHost-specific UUID to each worker's hash key. `ProxyPass / http://127.0.0.1:3001/` inside VHost A and VHost B creates two completely separate workers with separate pools. A `balancer://` URL is registered globally, so all VirtualHosts referencing it share one pool. One pool means `smax`/`ttl` cleanup actually has a meaningful pool to manage.

**Timeout chain:** Cloudflare (120s) > Next.js keepAliveTimeout (60s) > Apache timeout (30s) > Apache ttl (15s) > Apache connectiontimeout (3s)

### Main SSL Proxy Config

**File:** `/etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/proxy.conf`:

```apache
ProxyPreserveHost On

# Exclude error pages and cPanel paths from proxying to Next.js
# Without these, a 502 from Next.js triggers ErrorDocument 502 /502.shtml,
# which gets caught by the catch-all ProxyPass and proxied back to Next.js,
# creating a cascade of timeout errors.
ProxyPass /400.shtml !
ProxyPass /502.shtml !
ProxyPass /503.shtml !
ProxyPass /504.shtml !
ProxyPass /cgi-sys/ !

ProxyPass / balancer://nextjs/
ProxyPassReverse / http://127.0.0.1:3001/

# WebSocket for Next.js
RewriteEngine On
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule ^/?(.*) ws://127.0.0.1:3001/$1 [P,L]
```

### Custom Domain Configs

All custom domain VHosts in `/etc/apache2/conf.d/custom-domains/` need the same error page exclusions as the main VHost, plus the balancer proxy:

```apache
ProxyPreserveHost On

# Exclude error pages from proxying (prevents cascade when Next.js is down)
ProxyPass /400.shtml !
ProxyPass /502.shtml !
ProxyPass /503.shtml !
ProxyPass /504.shtml !

ProxyPass / balancer://nextjs/
ProxyPassReverse / http://127.0.0.1:3001/
```

New custom domains provisioned by `ssl-manager.sh` automatically get these exclusions.

To batch-add error page exclusions to existing custom domain configs:

```bash
# Backup first
cp -r /etc/apache2/conf.d/custom-domains /etc/apache2/conf.d/custom-domains.bak

# Add error page exclusions before the catch-all ProxyPass in each config
for conf in /etc/apache2/conf.d/custom-domains/*.conf; do
  # Skip if exclusions already present
  if grep -q "ProxyPass /400.shtml" "$conf" 2>/dev/null; then continue; fi
  sed -i '/ProxyPass \/ balancer:\/\/nextjs\//i\    # Exclude error pages from proxying (prevents cascade when Next.js is down)\n    ProxyPass /400.shtml !\n    ProxyPass /502.shtml !\n    ProxyPass /503.shtml !\n    ProxyPass /504.shtml !\n' "$conf"
done

# Verify
grep -A1 "ProxyPass.*shtml" /etc/apache2/conf.d/custom-domains/*.conf | head -20
```

### Apply Changes

```bash
apachectl configtest && /scripts/rebuildhttpdconf && systemctl restart httpd
```

## 7. Cloudflare DNS Configuration

In the Cloudflare dashboard for `aimclear.com`:

### A Record for Production

```
Type: A
Name: amsoil
Content: 69.169.111.27
Proxied: Yes (orange cloud)
TTL: Auto
```

### Wildcard for Dealer Subdomains

```
Type: A
Name: *.amsoil
Content: 69.169.111.27
Proxied: Yes (orange cloud)
TTL: Auto
```

This enables `dealername.amsoil.aimclear.com` for dealer pages.

### SSL Settings

1. Go to **SSL/TLS** > **Overview**
2. Set mode to **Full** (not Full strict)

## 8. SSL Certificate

### Option A: Use Cloudflare (Recommended)

With Cloudflare proxying enabled (orange cloud), Cloudflare handles SSL automatically. The origin server can use any certificate.

### Option B: Install Cloudflare Origin Certificate

1. **Cloudflare** > **SSL/TLS** > **Origin Server** > **Create Certificate**
2. Include hostnames: `amsoil.aimclear.com`, `*.amsoil.aimclear.com`
3. Download certificate and key
4. Install via **cPanel** > **SSL/TLS** > **Install and Manage SSL**

### Disable AutoSSL for Production Domain

To prevent conflicts:

1. **cPanel** > **SSL/TLS Status**
2. Find `amsoil.aimclear.com`
3. Click **Exclude from AutoSSL**

## 9. Stripe Webhook Configuration

**Important**: Stripe webhooks must be manually configured in the Stripe Dashboard for the production domain.

### Create Production Webhook Endpoint

1. Go to **Stripe Dashboard** > **Developers** > **Webhooks**
2. Click **Add endpoint**
3. Enter endpoint URL: `https://amsoil.aimclear.com/api/webhooks/stripe`
4. Select events to listen for:
   - `checkout.session.completed`
   - `customer.subscription.created`
   - `customer.subscription.updated`
   - `customer.subscription.deleted`
   - `invoice.paid`
   - `invoice.payment_failed`
5. Click **Add endpoint**
6. Copy the **Signing secret** (starts with `whsec_`)
7. Add to `.env.local` as `STRIPE_WEBHOOK_SECRET`

### Verify Webhook is Working

After setup, test by creating a checkout session. Check:

```bash
pm2 logs amsoil-dlp-production --lines 50 | grep -i webhook
```

You should see webhook events being received and processed.

## 10. Email Configuration

The application sends emails for various events (welcome emails, password resets, etc.).

### Option A: Resend (Recommended)

1. Create account at [resend.com](https://resend.com)
2. Add and verify your domain (`aimclear.com`)
3. Create an API key
4. Add to `.env.local`:
   ```env
   EMAIL_PROVIDER="resend"
   RESEND_API_KEY="re_xxxxxxxxxxxx"
   EMAIL_FROM="noreply@amsoil.aimclear.com"
   ```

### Option B: SMTP

If using an existing SMTP server:

```env
EMAIL_PROVIDER="smtp"
SMTP_HOST="smtp.yourprovider.com"
SMTP_PORT="587"
SMTP_USER="your-smtp-username"
SMTP_PASS="your-smtp-password"
SMTP_SECURE="true"
EMAIL_FROM="noreply@amsoil.aimclear.com"
```

### Verify Email is Working

After configuration, test by triggering an email (e.g., registration flow). Check logs:

```bash
pm2 logs amsoil-dlp-production --lines 50 | grep -i email
```

## 11. Rate Limiting (Redis)

The application uses rate limiting to prevent abuse. By default, it uses in-memory storage, but for production with multiple instances, Redis is recommended.

### Why Redis for Production?

- **In-memory (default)**: Rate limits are per-process. Each PM2 instance has its own counters.
- **Redis**: Rate limits are shared across all instances. More accurate protection.

Since staging and production are separate PM2 processes, without Redis each has independent rate limit buckets.

### Setup Upstash Redis

1. Create account at [console.upstash.com](https://console.upstash.com)
2. Create a new Redis database (choose region closest to your server)
3. Copy the REST URL and token
4. Add to `.env.local`:
   ```env
   UPSTASH_REDIS_REST_URL="https://your-db-us1.upstash.io"
   UPSTASH_REDIS_REST_TOKEN="your-token"
   ```

### Shared vs Separate Redis

- **Shared Redis**: Both staging and production share rate limit state. A user hitting limits on staging affects production.
- **Separate Redis**: Create two Upstash databases. More isolation but more cost.

For most cases, **separate Redis databases** are recommended for true production isolation.

## 12. Verification

### Test Application Locally

```bash
curl -I http://localhost:3001
# Should return HTTP 307 or 200
```

### Test Through Apache

```bash
curl -I http://127.0.0.1:80 -H "Host: amsoil.aimclear.com"
# Should return redirect to HTTPS
```

### Test Full Chain

```bash
curl -I https://amsoil.aimclear.com 2>/dev/null | head -20
# Should show:
# - server: cloudflare
# - x-powered-by: Next.js
# - location: /api/auth/signin (or appropriate response)
```

### Verify PM2 Processes

```bash
pm2 list
pm2 logs amsoil-dlp-production --lines 20
```

## 13. Post-Setup Checklist

### Infrastructure

- [ ] Production database created and accessible
- [ ] .env.local configured with ALL required variables (see Section 3)
- [ ] `NEXT_SERVER_ACTIONS_ENCRYPTION_KEY` set (prevents "Failed to find Server Action" errors after deploys)
- [ ] Application built successfully (`npm run build`)
- [ ] PM2 running production app on port 3001
- [ ] Apache proxy configured for amsoil.aimclear.com
- [ ] Apache ServerAlias configured for dealer domains (_.myamsoil.com, _.myamsoil.ca, _.shopamsoil.com, _.shopamsoil.ca)
- [ ] Cloudflare DNS pointing to server (A record + wildcard)
- [ ] SSL working (Cloudflare Full mode)
- [ ] AutoSSL disabled for production domain

### Authentication & Payments

- [ ] OAuth credentials configured in Google Console for production domain
- [ ] OAuth credentials configured in Apple Developer Console for production domain
- [ ] Stripe webhook endpoint created in Stripe Dashboard
- [ ] Stripe webhook secret added to `.env.local`
- [ ] Stripe LIVE price IDs configured (`STRIPE_PRICE_*`)

### Email & Services

- [ ] Email provider configured (Resend or SMTP)
- [ ] Test email sending works
- [ ] Upstash Redis configured (optional but recommended)

### Functional Testing

- [ ] Test dealer registration flow end-to-end
- [ ] Test Stripe checkout with LIVE mode (use a real card, refund after)
- [ ] Test dealer publishing creates Cloudflare DNS record
- [ ] Test dealer subdomain is accessible
- [ ] Monitor logs for errors after each test

### Security

- [ ] `NODE_ENV` is set to `production`
- [ ] No test/development secrets in production `.env.local`
- [ ] Dev auth endpoint returns 403 (not accessible in production)

## 14. Ongoing Operations

### Deployments

When deploying updates, deploy to both environments.

> **WARNING:** Always run `npm run build` as the `amsoildlp` user, NOT as root. Building as root creates `.next` files owned by root, which prevents ISR cache updates at runtime and can exhaust DB connections. A `prebuild` script guard will block root builds, but if you've accidentally built as root, fix with: `chown -R amsoildlp:amsoildlp .next/`

```bash
# Staging
cd /home/amsoildlp/public_html/aimclear.biz
sudo -u amsoildlp bash -c 'git pull && npm ci && npm run build'
pm2 restart amsoil-dlp-staging

# Production
cd /home/amsoildlp/public_html/amsoil.aimclear.com
sudo -u amsoildlp bash -c 'git pull && npm ci && npm run build'
pm2 restart amsoil-dlp-production
```

### Database Migrations

Run migrations on both databases:

```bash
# Staging
cd /home/amsoildlp/public_html/aimclear.biz
npx prisma migrate deploy

# Production
cd /home/amsoildlp/public_html/amsoil.aimclear.com
npx prisma migrate deploy
```

### Monitoring

Check both apps:

```bash
pm2 list
pm2 logs --lines 50
```

## Quick Reference

| Item              | Staging                                  | Production                                      |
| ----------------- | ---------------------------------------- | ----------------------------------------------- |
| Domain            | aimclear.biz                             | amsoil.aimclear.com                             |
| Dealer Subdomains | \*.aimclear.biz                          | \*.amsoil.aimclear.com                          |
| Port              | 3000                                     | 3001                                            |
| PM2 Name          | amsoil-dlp-staging                       | amsoil-dlp-production                           |
| Database          | amsoil_dlp_staging                       | amsoil_dlp_production                           |
| DB User           | amsoil_staging                           | amsoil_production                               |
| Directory         | /home/amsoildlp/public_html/aimclear.biz | /home/amsoildlp/public_html/amsoil.aimclear.com |
| Stripe Mode       | Test keys (`sk_test_`, `pk_test_`)       | Live keys (`sk_live_`, `pk_live_`)              |
| Stripe Webhook    | Configured for staging URL               | Configured for production URL                   |
| Redis             | Optional (shared OK)                     | Recommended (separate)                          |

## Troubleshooting

### Production app not responding

```bash
pm2 logs amsoil-dlp-production --lines 50
curl -I http://localhost:3001
```

### Apache not proxying

```bash
apachectl -S | grep amsoil.aimclear.com
cat /etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/*.conf
/scripts/rebuildhttpdconf && systemctl restart httpd
```

### Database connection issues

```bash
psql -h localhost -U amsoil_production -d amsoil_dlp_production -c "SELECT 1"
```

### SSL issues

```bash
openssl s_client -connect 127.0.0.1:443 -servername amsoil.aimclear.com 2>/dev/null | openssl x509 -noout -subject -issuer
```

### Application Freeze Debugging

The app may occasionally freeze where requests hang until PM2 or Apache is restarted. PostgreSQL is configured to log diagnostic information when this happens.

**PostgreSQL logging enabled (as of 2025-12-22):**

| Setting                      | Value | Purpose                              |
| ---------------------------- | ----- | ------------------------------------ |
| `log_lock_waits`             | on    | Logs when queries wait on locks > 1s |
| `deadlock_timeout`           | 1s    | Threshold for lock wait logging      |
| `log_min_duration_statement` | 5000  | Logs queries taking > 5 seconds      |

**During a freeze, run these diagnostics:**

```bash
# 1. Connection overview - check for idle_in_tx or high waiting count
PGPASSWORD='<PRODUCTION_DB_PASSWORD>' psql -h 127.0.0.1 -U amsoil_dlp_production -d amsoil_dlp_production -c "
SELECT count(*) AS total,
       count(*) FILTER (WHERE state = 'active') AS active,
       count(*) FILTER (WHERE state = 'idle') AS idle,
       count(*) FILTER (WHERE state = 'idle in transaction') AS idle_in_tx,
       count(*) FILTER (WHERE wait_event_type IS NOT NULL) AS waiting
FROM pg_stat_activity WHERE datname = current_database();"

# 2. Who is blocking whom (the smoking gun)
PGPASSWORD='<PRODUCTION_DB_PASSWORD>' psql -h 127.0.0.1 -U amsoil_dlp_production -d amsoil_dlp_production -c "
SELECT blocked.pid AS blocked_pid,
       age(now(), blocked.query_start) AS blocked_for,
       LEFT(blocked.query, 80) AS blocked_query,
       blocker.pid AS blocker_pid,
       LEFT(blocker.query, 80) AS blocker_query
FROM pg_stat_activity blocked
JOIN pg_stat_activity blocker ON blocker.pid = ANY(pg_blocking_pids(blocked.pid))
WHERE blocked.wait_event_type = 'Lock' LIMIT 10;"

# 3. Longest running queries
PGPASSWORD='<PRODUCTION_DB_PASSWORD>' psql -h 127.0.0.1 -U amsoil_dlp_production -d amsoil_dlp_production -c "
SELECT pid, state, age(now(), query_start) AS runtime, wait_event_type, LEFT(query, 100) AS query
FROM pg_stat_activity WHERE datname = current_database() AND query_start IS NOT NULL
ORDER BY query_start LIMIT 15;"

# 4. Check connection limits
PGPASSWORD='<PRODUCTION_DB_PASSWORD>' psql -h 127.0.0.1 -U amsoil_dlp_production -d amsoil_dlp_production -c "SHOW max_connections;"
```

**After a freeze (or anytime), check PostgreSQL logs:**

```bash
tail -500 /var/lib/pgsql/data/log/postgresql-*.log | grep -iE "(lock|duration|waiting|deadlock|ERROR)"
```

**What to look for:**

| Symptom                                  | Likely Cause                           |
| ---------------------------------------- | -------------------------------------- |
| `idle_in_tx > 0`                         | Sessions holding locks without working |
| Many blocked queries behind one blocker  | Lock contention (freeze cause)         |
| `wait_event_type = 'Lock'`               | Transactions waiting on each other     |
| Connections near `max_connections` (100) | Pool exhaustion                        |
| `duration: Xms` in logs                  | Slow queries identified                |

### Dealer subdomain 404 errors

If dealer subdomains (e.g., `dealername.myamsoil.com`) return 404 with no logs in the Next.js app:

1. **Check Apache ServerAlias configuration:**

   ```bash
   apachectl -S | grep -i myamsoil
   cat /etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/serveralias.conf
   ```

2. **If ServerAlias is missing**, create it:

   ```bash
   cat > /etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/serveralias.conf << 'EOF'
   ServerAlias *.myamsoil.com myamsoil.com
   ServerAlias *.myamsoil.ca myamsoil.ca
   ServerAlias *.shopamsoil.com shopamsoil.com
   ServerAlias *.shopamsoil.ca shopamsoil.ca
   EOF
   ```

3. **Rebuild and restart Apache:**

   ```bash
   /scripts/rebuildhttpdconf
   systemctl restart httpd
   ```

4. **Verify the fix:**
   ```bash
   curl -I http://127.0.0.1:80 -H "Host: dealername.myamsoil.com"
   ```

**Root cause:** Apache VirtualHost matching uses `ServerName`/`ServerAlias` directives, not DNS. Without ServerAlias entries for dealer domains, requests don't reach the Next.js app even if DNS resolves correctly.

### Apache Proxy & Connection Pool Debugging

**Check balancer pool health:**

```bash
# Connection state summary (CLOSE-WAIT should stay below smax=25)
ss -tanp | grep ":3001" | awk '{print $1}' | sort | uniq -c | sort -rn

# Watch CLOSE-WAIT trend over time
watch -n 10 'ss -tanp | grep ":3001" | grep CLOSE-WAIT | wc -l'

# Bypass Apache — check if Next.js is responding directly
time curl -sI http://127.0.0.1:3001

# Check for proxy-related errors
tail -100 /etc/apache2/logs/error_log | grep -iE "proxy|timeout|connection"

# Count Apache processes
pgrep -c httpd
```

**Connection state reference:**

| State      | Meaning                                      | Concern Level |
| ---------- | -------------------------------------------- | ------------- |
| ESTAB      | Active connections carrying requests          | Normal        |
| TIME-WAIT  | Properly closed, kernel holding for 60s       | Normal        |
| CLOSE-WAIT | Next.js closed, Apache hasn't closed its side | Watch if >25  |
| LISTEN     | Next.js listening for connections (always 1)  | Normal        |

If CLOSE-WAIT exceeds `smax=25`, the balancer's maintain function should be reaping them. If it consistently stays above 25 and climbing, the shared pool cleanup may not be keeping up. See Section 6b for pool settings.

## Quick Reference Commands

### Server Status Check

```bash
# PM2 process status
pm2 list
pm2 show 13

# Apache status
systemctl status httpd
apachectl -S | grep amsoil

# Database connections
PGPASSWORD='<PASSWORD>' psql -h 127.0.0.1 -U amsoil_dlp_production -d amsoil_dlp_production -c "
SELECT count(*) as total, state FROM pg_stat_activity WHERE datname = current_database() GROUP BY state;"

# Quick health check
curl -sI http://127.0.0.1:3001/ | head -3
```

### Log Locations

| Log               | Location                                                 |
| ----------------- | -------------------------------------------------------- |
| PM2 app logs      | `pm2 logs 13` or `/home/amsoildlp/logs/amsoil-dlp-*.log` |
| PM2 error logs    | `pm2 logs 13 --err`                                      |
| Apache error log  | `/etc/apache2/logs/error_log`                            |
| Apache access log | `/etc/apache2/logs/domlogs/amsoildlp/*`                  |
| PostgreSQL logs   | `/var/lib/pgsql/data/log/postgresql-*.log`               |

### Common Operations

```bash
# Graceful reload (zero-downtime)
pm2 reload 13

# Hard restart (causes brief downtime)
pm2 restart 13

# View recent errors
pm2 logs 13 --err --lines 100

# Rebuild Next.js (zero-downtime)
cd /home/amsoildlp/public_html/amsoil.aimclear.com
npm run build && pm2 reload 13

# Restart Apache
systemctl restart httpd

# Rebuild Apache config (cPanel)
/scripts/rebuildhttpdconf && systemctl restart httpd
```

### Config File Locations

| Config                | Location                                                                              |
| --------------------- | ------------------------------------------------------------------------------------- |
| App environment       | `/home/amsoildlp/public_html/amsoil.aimclear.com/.env.local`                          |
| PM2 ecosystem         | `/home/amsoildlp/public_html/amsoil.aimclear.com/ecosystem.config.js`                 |
| Apache SSL proxy      | `/etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/proxy.conf`       |
| Apache ServerAlias    | `/etc/apache2/conf.d/userdata/ssl/2_4/amsoildlp/amsoil.aimclear.com/serveralias.conf` |
| Custom domain configs | `/etc/apache2/conf.d/custom-domains/custom-*.conf`                                    |
| Global Apache include | `/etc/apache2/conf.d/includes/post_virtualhost_global.conf`                           |
| Apache main config    | `/etc/apache2/conf/httpd.conf`                                                        |

### Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────────────┐
│                           Request Flow                                   │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│   User Request                                                          │
│        ↓                                                                │
│   Cloudflare (CDN, WAF, SSL termination)                               │
│        ↓                                                                │
│   Apache (port 443)                                                     │
│   ├── VirtualHost matching (ServerName/ServerAlias)                    │
│   ├── mod_proxy → http://127.0.0.1:3001                                │
│   └── Potential freeze point if proxy connections get stuck            │
│        ↓                                                                │
│   PM2 → Next.js (port 3001)                                            │
│   ├── proxy.ts (subdomain routing, auth checks)                        │
│   ├── App Router (pages, API routes)                                   │
│   └── Prisma → PostgreSQL                                              │
│        ↓                                                                │
│   Response flows back up the chain                                      │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
```

### Cloudflare Analytics

To check traffic and error rates:

1. Log into Cloudflare Dashboard
2. Select the domain (e.g., aimclear.com)
3. Go to **Analytics & Logs** → **Traffic**
4. Key metrics to watch:
   - **4xx error rate** > 30% = Heavy bot traffic
   - **5xx error rate** > 1% = Server issues
   - **Cached rate** < 20% = CDN not helping much

### Bot Protection

If seeing high 4xx errors from bots:

1. **Cloudflare Dashboard** → **Security** → **Bots** → Enable **Bot Fight Mode**
2. **Security** → **WAF** → Add rule to block `.php`, `wp-admin`, etc.