# AMSOIL Dealer Pages - Operations Checklist ## Purpose This checklist helps operations teams maintain the AMSOIL Dealer Pages platform. Regular completion of these tasks ensures system reliability, security, and optimal performance. --- ## Daily Checks ### System Health - [ ] Review application error logs for critical issues - [ ] Check uptime monitoring (no extended downtime) - [ ] Verify database connection pool is healthy - [ ] Review Stripe webhook delivery success rate ### User Experience - [ ] Monitor user sign-in success rate - [ ] Check for payment processing errors - [ ] Review support ticket queue --- ## Weekly Checks ### Authentication & Security - [ ] Review failed sign-in attempts for patterns - [ ] Check for unusual OAuth activity - [ ] Verify rate limiting is functioning - [ ] Review security logs for suspicious activity ### Subscriptions & Billing - [ ] Review payment_failed statuses in dealer database - [ ] Check for suspended accounts needing follow-up - [ ] Verify Stripe webhook processing (no backlog) - [ ] Review subscription upgrades/downgrades ### Performance - [ ] Check API response times - [ ] Review database query performance - [ ] Monitor memory usage trends - [ ] Check disk space usage --- ## Monthly Checks ### Platform Maintenance - [ ] Review and update dependencies (security patches) - [ ] Test disaster recovery procedures - [ ] Verify backup integrity - [ ] Review and archive old logs ### Business Metrics - [ ] Generate monthly subscription report - [ ] Calculate churn rate - [ ] Review dealer signup conversion rate - [ ] Analyze OAuth provider usage (Google vs Apple) ### Documentation - [ ] Update runbook for any new procedures - [ ] Review and update operational documentation - [ ] Document any incidents or outages --- ## Quarterly Checks (Every 90 Days) ### Apple OAuth Maintenance ⚠️ CRITICAL - [ ] **Check Apple JWT expiration date** (expires every 180 days) - [ ] Verify calendar reminders are set (150 and 175 days) - [ ] Test Apple sign-in on production - [ ] Review application logs for Apple OAuth errors - [ ] **Document current JWT generation date** > **Reference:** See [APPLE_OAUTH_MAINTENANCE.md](./APPLE_OAUTH_MAINTENANCE.md) for detailed procedures ### Security Review - [ ] Review access control policies - [ ] Audit admin user accounts - [ ] Review API key rotation schedule - [ ] Check SSL/TLS certificate expiration dates ### Capacity Planning - [ ] Review database growth trends - [ ] Assess server capacity needs - [ ] Plan for scaling if needed - [ ] Review CDN usage and costs --- ## Semi-Annual Checks (Every 6 Months) ### Critical: Apple JWT Regeneration ⚠️ REQUIRED - [ ] **Schedule server restart before JWT expiration (Day 180)** - [ ] Notify team of maintenance window - [ ] Execute restart procedure (see [APPLE_OAUTH_MAINTENANCE.md](./APPLE_OAUTH_MAINTENANCE.md)) - [ ] Verify Apple sign-in functionality post-restart - [ ] Update calendar reminders for next 180-day cycle - [ ] **Document new JWT generation date in operations log** > **Critical:** Missing this check will cause Apple sign-in to fail silently! ### Infrastructure Review - [ ] Review cloud infrastructure costs - [ ] Assess need for architectural changes - [ ] Review disaster recovery plan - [ ] Test failover procedures ### Compliance - [ ] Review data retention policies - [ ] Audit GDPR compliance - [ ] Review PCI compliance (Stripe handles payment data) - [ ] Update privacy policy if needed --- ## Annual Checks ### Strategic Planning - [ ] Annual security audit - [ ] Review SLA commitments - [ ] Assess technology stack (migrations needed?) - [ ] Plan major feature releases ### Team Training - [ ] Review operations procedures with team - [ ] Update on-call rotation and escalation - [ ] Conduct incident response drills - [ ] Document lessons learned from past year --- ## Event-Driven Checks ### After Each Deployment - [ ] **Verify build was NOT run as root** - `npm run build` must run as the `amsoildlp` user. Running as root causes `.next` file ownership issues that break ISR cache updates and can exhaust DB connections. A `prebuild` guard will block this, but verify with: `ls -la .next/server/app/ | head` (should show `amsoildlp amsoildlp`) - [ ] Verify deployment success (all services running) - [ ] Check for new errors in logs (especially `EACCES` permission errors in the error log) - [ ] Verify no "Failed to find Server Action" errors (`NEXT_SERVER_ACTIONS_ENCRYPTION_KEY` must be set) - [ ] Test critical user flows (sign-in, payment) - [ ] Monitor for 30 minutes post-deployment - [ ] **If Apple OAuth credentials changed:** Verify Apple sign-in works - [ ] **Document deployment date** (for Apple JWT tracking) ### After Production Incident - [ ] Document incident timeline - [ ] Conduct post-mortem meeting - [ ] Update runbook with lessons learned - [ ] Implement preventive measures - [ ] Communicate with stakeholders ### When Support Tickets Increase - [ ] Identify common issues - [ ] Check for recent code changes - [ ] Review error logs for patterns - [ ] Escalate to engineering if systemic issue --- ## Critical Alerts Requiring Immediate Action ### 🔴 P0 - Critical (Respond within 15 minutes) - Application completely down - Database connection failure - All sign-ins failing - Payment processing broken - Data breach detected ### 🟠 P1 - High (Respond within 1 hour) - Apple OR Google OAuth completely failing - Stripe webhooks failing consistently - Database performance severely degraded - Rate limiting not functioning ### 🟡 P2 - Medium (Respond within 4 hours) - Intermittent sign-in failures - Payment processing delays - Individual dealer reports site down - Elevated error rates (but not critical) ### 🟢 P3 - Low (Respond within 1 business day) - Single feature not working - UI/UX issues - Non-critical performance degradation - Documentation updates needed --- ## Apple OAuth JWT Expiration Tracking **Current Production Deployment:** - **Deployment Date:** _[TO BE FILLED ON DEPLOYMENT]_ - **Apple JWT Generated:** _[TO BE FILLED ON DEPLOYMENT]_ - **Apple JWT Expires:** _[DEPLOYMENT DATE + 180 days]_ - **First Reminder Date:** _[DEPLOYMENT DATE + 150 days]_ - **Urgent Reminder Date:** _[DEPLOYMENT DATE + 175 days]_ **History:** | Deployment Date | JWT Generated | JWT Expires | Restart Performed | Notes | | --------------- | ------------- | ----------- | ----------------- | ----------------------------- | | 2025-01-15 | 2025-01-15 | 2025-07-14 | TBD | Initial production deployment | | | | | | | **Instructions:** 1. After each production deployment, fill in the dates above 2. Set calendar reminders for the reminder dates 3. Before expiration (day 175), schedule maintenance restart 4. After restart, document in history table and update current dates --- ## Maintenance Windows ### Standard Maintenance Window - **Frequency:** As needed (typically monthly) - **Duration:** 30 minutes - **Time:** Tuesday 2:00 AM - 2:30 AM CST (low traffic) - **Notification:** 48 hours advance notice to dealers ### Apple JWT Regeneration Window (Semi-Annual) - **Frequency:** Every ~175 days (before 180-day expiration) - **Duration:** 10 minutes (server restart) - **Time:** Tuesday 2:00 AM - 2:10 AM CST - **Notification:** 1 week advance notice - **Critical:** Cannot be skipped or delayed ### Emergency Maintenance - **Trigger:** Critical security patch or system failure - **Duration:** Variable (document in incident log) - **Notification:** As soon as possible, may be post-facto - **Follow-up:** Post-mortem required --- ## Contact Information ### Escalation Path **Level 1: Operations Team** - **Scope:** Routine checks, monitoring, basic troubleshooting - **Contact:** ops@amsoil.com **Level 2: Engineering Team** - **Scope:** Code issues, complex troubleshooting, deployments - **Contact:** engineering@amsoil.com **Level 3: On-Call Engineer** - **Scope:** Critical production issues (P0/P1) - **Contact:** [PagerDuty/on-call phone] **Level 4: CTO/VP Engineering** - **Scope:** Major incidents, business-impacting issues - **Contact:** [Leadership contact] ### External Vendors **Stripe Support** - **Issues:** Payment processing, webhook failures - **Contact:** https://support.stripe.com **Apple Developer Support** - **Issues:** Apple OAuth, JWT, Sign In with Apple - **Contact:** https://developer.apple.com/support/ **Hosting Provider (Vercel/AWS/etc.)** - **Issues:** Infrastructure, networking, deployments - **Contact:** [Provider support portal] --- ## Documentation References - **[APPLE_OAUTH_MAINTENANCE.md](./APPLE_OAUTH_MAINTENANCE.md)** - Apple JWT regeneration procedures - **[CLAUDE.md](../CLAUDE.md)** - Project overview and architectural decisions - **[REGISTRATION_ARCHITECTURE.md](./REGISTRATION_ARCHITECTURE.md)** - Registration flow details - **[STYLING_GUIDE.md](./STYLING_GUIDE.md)** - Frontend styling system --- ## Revision History | Date | Version | Changes | Author | | ---------- | ------- | ------------------------------------ | ----------- | | 2025-01-11 | 1.0 | Initial operations checklist created | Claude Code | --- **Last Updated:** 2025-01-11 **Next Review:** 2025-04-11 (Quarterly review)