# Rate Limiting Monitoring Guide

This guide covers how rate limiting works in the AMSOIL Dealer Pages platform and how to monitor, configure, and troubleshoot rate-limited requests.

## Architecture Overview

Rate limiting is handled entirely at the Cloudflare edge, not within the application. This approach was adopted in PR #272 to:

- **Reduce latency**: Requests are blocked before reaching the application server
- **Improve efficiency**: No application resources consumed for blocked requests
- **Simplify infrastructure**: No Redis/Upstash dependency for rate limiting
- **Leverage Cloudflare's scale**: Distributed rate limiting across Cloudflare's global network

### What This Means for Operators

The application **cannot see rate-limited requests** in its own logs. All monitoring must be done through the Cloudflare dashboard. Blocked requests never reach the application server.

## Cloudflare Security Events Dashboard

### Accessing the Dashboard

1. Log into [Cloudflare Dashboard](https://dash.cloudflare.com)
2. Select your zone (e.g., `myamsoil.com`)
3. Navigate to **Security > Events** (old dashboard) or **Analytics > Events tab** (new dashboard)
4. Enterprise customers can access account-level views under **Security Center > Security Events**

### Dashboard Components

| Section              | What It Shows                                                           |
| -------------------- | ----------------------------------------------------------------------- |
| Events Summary       | Total security events grouped by action, host, country, or ASN          |
| Events by Service    | Activity per security feature (Managed rules, Rate limiting, etc.)      |
| Top Events by Source | Details on flagged traffic: IP addresses, user agents, paths, countries |
| Sampled Logs         | Individual security events with timestamps and applied rules            |

### Filtering Rate Limiting Events

To view only rate limiting activity:

1. In the **Events by service** section, locate "Rate limiting rules"
2. Click **Filter** to narrow the dashboard to rate limiting events only
3. Or manually add filter: `Service equals Rate limiting`

### Exporting Data

- Export up to 500 raw events in JSON format from Sampled Logs
- Use the Export button to download for SIEM integration
- Share filtered views by copying the URL with filter parameters

## Monitoring Rate-Limited Requests

### Daily Monitoring Checklist

- [ ] Review Security Events dashboard for unusual spikes
- [ ] Check rate limiting rule hit counts
- [ ] Identify top rate-limited IP addresses
- [ ] Look for geographic patterns in blocked requests

### Key Metrics to Track

| Metric                  | Where to Find                     | Healthy Range                    |
| ----------------------- | --------------------------------- | -------------------------------- |
| Rate limit events/hour  | Events by Service > Rate limiting | < 100 for normal traffic         |
| Unique IPs blocked      | Top Events by Source              | Varies; watch for sudden spikes  |
| Top paths targeted      | Sampled Logs                      | Should match sensitive endpoints |
| Geographic distribution | Events Summary by Country         | Should match user base           |

### Warning Signs

- **Sudden spike in rate-limited requests**: May indicate DDoS attempt or misconfigured client
- **Many unique IPs from one country**: Could be coordinated attack
- **High rate limits on non-sensitive paths**: May indicate bot scraping
- **Repeated rate limits from same IP**: Single aggressive client or attack

## Setting Up Alerts

Cloudflare offers automated security alerting based on traffic spikes.

### Alert Types

| Alert Type                     | Plan Required          | Description                                     |
| ------------------------------ | ---------------------- | ----------------------------------------------- |
| Security Events Alert          | Business or Enterprise | Monitors spikes across all security services    |
| Advanced Security Events Alert | Enterprise only        | Granular filtering by service, zone, and action |

### Configuring Alerts

1. Navigate to **Notifications** in Cloudflare Dashboard
2. Click **Create** and select alert type
3. Configure delivery method:
   - Email
   - PagerDuty (for on-call integration)
   - Webhook (for custom integrations/Slack)
4. Select zones to monitor
5. (Enterprise) Filter by services including "Rate Limiting"

### Alert Behavior

- Alerts trigger on **traffic spikes**, not individual events
- Uses z-score analysis over the last 6 hours
- Alert threshold: z-score > 3.5 AND 200+ events in spike
- Mean detection time: ~5 minutes
- Duplicate alerts suppressed within 2-hour windows

### Recommended Alert Configuration

**For Business Plans:**

```
Type: Security Events Alert
Zones: myamsoil.com (all subdomains)
Delivery: Email to ops@yourcompany.com
```

**For Enterprise Plans:**

```
Type: Advanced Security Events Alert
Zones: myamsoil.com
Services: Rate Limiting (specifically)
Actions: Block, Challenge
Delivery: PagerDuty (P1) + Email + Slack Webhook
```

## Rate Limiting Rules Configuration

### Accessing Rules

1. Cloudflare Dashboard > Security > WAF > Rate limiting rules
2. View existing rules and their hit counts
3. Create or modify rules as needed

### Recommended Rules for AMSOIL DLP

| Endpoint Pattern         | Rate Limit | Period | Action    | Rationale                     |
| ------------------------ | ---------- | ------ | --------- | ----------------------------- |
| `/api/auth/*`            | 30 req     | 1 min  | Block     | Prevent brute force login     |
| `/api/register/*`        | 10 req     | 1 min  | Block     | Prevent registration abuse    |
| `/api/check-subdomain`   | 20 req     | 1 min  | Block     | Prevent subdomain enumeration |
| `/api/dealers/*/publish` | 5 req      | 1 min  | Block     | Prevent publish flooding      |
| `/api/contact`           | 10 req     | 1 min  | Block     | Prevent spam submissions      |
| `/api/support/clickup`   | 10 req     | 1 min  | Block     | Prevent support form spam     |
| `/api/checkout/*`        | 20 req     | 1 min  | Block     | Prevent checkout abuse        |
| `*` (global)             | 600 req    | 1 min  | Challenge | General protection            |

### Adjusting Rate Limits

**To increase limits:**

1. Navigate to WAF > Rate limiting rules
2. Edit the specific rule
3. Increase "Requests per period" value
4. Save and deploy

**To decrease limits (more restrictive):**

1. Monitor current traffic patterns in Analytics
2. Ensure new limit won't affect legitimate users
3. Edit rule with lower threshold
4. Monitor for false positives after deployment

### Testing Rate Limits

Use Cloudflare's "Request rate analysis" to find appropriate thresholds:

1. Go to Analytics > Request rate analysis tab
2. View distribution of request rates by client
3. Set limits above normal traffic but below attack patterns

## Troubleshooting

### Issue: Legitimate Users Being Rate Limited

**Symptoms:**

- Support tickets about "too many requests" errors
- Specific users/IPs blocked unexpectedly

**Diagnosis:**

1. Get the user's IP address
2. Search Security Events for that IP
3. Check which rule triggered and how many times

**Resolution:**

- If limit too low: Increase threshold for that rule
- If shared IP (office/VPN): Consider IP allowlist or higher limits
- If user behavior unusual: Investigate client-side issues

### Issue: No Rate Limiting Events Visible

**Possible causes:**

- Rules not configured in Cloudflare
- Rules in "Log" mode instead of "Block"
- Traffic not routed through Cloudflare (direct origin access)

**Resolution:**

1. Verify rate limiting rules exist and are enabled
2. Check rule actions are set to Block or Challenge
3. Ensure DNS is proxied (orange cloud in Cloudflare DNS)

### Issue: High Rate Limit Events but No Attacks

**Possible causes:**

- Overly aggressive rate limits
- Legitimate automation/integrations hitting limits
- CDN or proxy aggregating IPs

**Resolution:**

1. Review which paths trigger limits
2. Adjust thresholds for affected endpoints
3. Add allowlists for known good IPs (internal systems, partners)

### Issue: Attack Bypassing Rate Limits

**Possible causes:**

- Attack using distributed IPs
- Attack targeting unprotected endpoints
- Rate limit period too long

**Resolution:**

1. Add more specific rules for targeted paths
2. Reduce period window (e.g., 10 seconds instead of 1 minute)
3. Enable "Counting characteristics" by additional fields (headers, cookies)
4. Consider enabling Under Attack Mode temporarily

## Integration with Application Logs

Although the application cannot see rate-limited requests, it can detect when requests succeed after rate limiting:

### Application-Side Indicators

Look for patterns in application logs:

- Sudden drop in requests from specific IPs
- Increased 429 responses from downstream services (Stripe API, etc.)
- Gaps in expected traffic patterns

### Correlating Events

To correlate Cloudflare events with application logs:

1. Export Security Events from Cloudflare (JSON)
2. Match timestamps with application logs
3. Look for IP addresses appearing in both
4. Identify if rate-limited IPs later succeeded or gave up

## Maintenance

### Weekly Review

- [ ] Check rate limiting rule hit counts
- [ ] Review any new attack patterns
- [ ] Verify no false positives reported
- [ ] Adjust thresholds if traffic patterns changed

### Monthly Review

- [ ] Audit all rate limiting rules for relevance
- [ ] Compare traffic patterns month-over-month
- [ ] Review geographic distribution of blocks
- [ ] Update alert thresholds if needed

### After Major Releases

- [ ] Check for new endpoints needing rate limits
- [ ] Verify existing limits still appropriate
- [ ] Monitor for unusual traffic to new features

## Plan Requirements Summary

| Feature                  | Free     | Pro      | Business | Enterprise |
| ------------------------ | -------- | -------- | -------- | ---------- |
| Rate limiting rules      | Limited  | Yes      | Yes      | Yes        |
| Security Events log      | 24 hours | 72 hours | 7 days   | 30 days    |
| Security Events Alert    | No       | No       | Yes      | Yes        |
| Advanced Security Alerts | No       | No       | No       | Yes        |
| Account-level dashboard  | No       | No       | No       | Yes        |

## References

- [Cloudflare Rate Limiting Rules](https://developers.cloudflare.com/waf/rate-limiting-rules/)
- [Cloudflare Security Events](https://developers.cloudflare.com/waf/analytics/security-events/)
- [Cloudflare Security Event Alerts](https://developers.cloudflare.com/waf/reference/alerts/)
- [Finding Appropriate Rate Limits](https://developers.cloudflare.com/waf/rate-limiting-rules/find-rate-limit/)
- [Rate Limiting Best Practices](https://developers.cloudflare.com/waf/rate-limiting-rules/best-practices/)
- [Available Cloudflare Notifications](https://developers.cloudflare.com/notifications/notification-available/)
- [CLOUDFLARE_PRODUCTION_DEPLOYMENT.md](./CLOUDFLARE_PRODUCTION_DEPLOYMENT.md) - Production deployment guide
- [OPERATIONS_CHECKLIST.md](./OPERATIONS_CHECKLIST.md) - Operations procedures