# SEO and Search Indexing Short doc for the support team and anyone fielding "my site isn't on Google" questions. ## When does a dealer site become indexable? A dealer's site is indexable starting when their **subdomain is first set** - i.e., when they pick `name.myamsoil.com` (or one of the other three valid suffixes) and the site goes live at that URL. **This is not the same as account creation**; a dealer can sign up in January, set their subdomain in May, and their indexing clock starts in May. Look at `dealer.lastPublishedAt` (or, when richer activity is exposed via MCP, the activity log) to determine when "going live" actually happened. ## How long does indexing take? Typically **about 7 days** from the moment the subdomain is first set. Sometimes faster, sometimes a couple of weeks for less-trafficked domains. If a dealer reports their site isn't on Google: | Time since subdomain set | Action | | ------------------------ | ------------------------------------------------------------------------- | | < 7 days | "Google takes about a week to index new sites - sit tight." | | 7–14 days | "Should be soon. If you don't see it by day 14, let us know." | | > 14 days | Escalate - there may be a real issue (sitemap, robots.txt, redirect bug). | ## Subdomain changes reset the clock If a dealer changes their subdomain (admin override - subdomains are read-only after creation, see `CLAUDE.md`), the new URL is effectively a new site as far as Google is concerned, and indexing starts over. ## Sitemap and robots The sitemap auto-includes the dealer at publish time. If a dealer's subdomain is set and their site is live, they should be in `/sitemap.xml`. To verify, fetch `https://./sitemap.xml` - the dealer's URLs should appear. The sitemap/robots are host-aware (`app/sitemap.ts`, `app/robots.ts`). Each **parent domain** (`myamsoil.com`, `shopamsoil.com`, `myamsoil.ca`, `shopamsoil.ca`) lists **only its own** active dealers - scoped by `domainPrefix`+`domain`, rooted at the request host, and excluding dealers on an active custom domain (those canonicalize to their own apex). The **platform domain** (`amsoil.aimclear.com`) lists static pages only. `robots.txt` advertises the request host's own sitemap. So a parent-domain sitemap never lists cross-host URLs. For active-status guarding and custom-domain sitemap behaviour, see commit `34277cc` (sitemap: active-status guard) and `4b4015b` (subdomain/custom-domain aware sitemap). For per-parent-domain scoping (each domain lists only its own dealers, rooted at the request host), see PR #890. The sitemap also lists the public dealer **directory** on each apex: the hub at `/dealers` plus each qualifying region page `/dealers/` (`app/sitemap.ts:256-281`). These reuse the directory data layer (`getDealersForApex` / `groupDealersByRegion`), so the same grouping/threshold/exclusion rules that drive the directory pages drive their sitemap entries. The platform domain resolves to no apex, so it gets no directory entries. ### What robots.txt does and doesn't block (PR #908) Disallowed on **every** host: `/api/`, `/dev-tools/`. Main domains additionally disallow `/dashboard/`, `/auth/`, `/admin/`, `/dev/`, `/dev-tools/`, `/sites/`, `/home/`, and `/directory/` (`app/robots.ts:71-82`). Notes: - `/dev/` does **not** prefix-match `/dev-tools/`; both are listed deliberately. - `/sites/` is the internal dealer-site route (renamed from `/dealers/`); it must stay blocked so the internal path never competes with the dealer's canonical host. - `/home/` (host-aware apex stub homepages) and `/directory/` (the apex `/dealers` directory implementation) are internal route prefixes; the public-facing apex `/` and `/dealers` URLs are served via rewrites and remain crawlable. Two paths are **deliberately left crawlable**: - **`/_next/`** - Googlebot renders pages in a real browser and obeys robots.txt for subresources. Blocking `/_next/` makes Google render every page with no CSS/JS and blocks `/_next/image` from image search. Do not re-add it. - **`/og`** - the OG image endpoints are extensionless PNG responses that Google would otherwise index as "pages". They are de-indexed via an `X-Robots-Tag: noindex` response header instead (all three `/og` routes, plus a `next.config.mjs` header rule for the raw `/og-cache/` static files). A robots.txt block would prevent crawlers from ever fetching the URL and seeing that header, stranding "indexed, though blocked by robots.txt" zombie entries. Social scrapers ignore `noindex`, so link previews are unaffected. ### Outbound link rel policy (PR #908) - **ZO-attributed AMSOIL links are compensated** → `rel="sponsored"` (Google's required label for affiliate links). Applied in `DealerTemplate`, the publisher's static header/footer and dealer-nav builder, `MobileNav`, and `QuickLinks`. - **Dealer-authored external links in CMS content** → `rel="nofollow ugc"` so one dealer's outbound links can't damage the shared domain's link reputation. AMSOIL links in content get `sponsored` instead. - **Internal links are never nofollowed** - that would discard the dealer's own link equity. The CMS policy lives in `lib/cms/rel-utils.ts` (`relForCmsLink`), shared by both sanitizers so the editor preview matches published output. Protocol-relative hrefs (`//host/path`) count as external. ## Canonical URL emission All SEO URLs are built from two helpers in `lib/seo-utils.ts`: - **`getDealerOrigin(dealer)`** (`lib/seo-utils.ts:52`) - the bare origin, scheme + host, **no path, no trailing slash** (e.g. `https://bobsoil.myamsoil.com` or `https://bobsoil.com`). This is the single source of truth for dealer host resolution. Use it for any **non-document** URL: a GSC API target, an `/og` image path, or a JSON-LD `@id` fragment (`#localbusiness`). - **`getCanonicalUrl(dealer, pagePath)`** (`lib/seo-utils.ts:83`) - the canonical **document** URL: `getDealerOrigin` plus the normalized path. For a dealer with an active custom domain, every emitted canonical URL (``, `og:url`, JSON-LD `url`, sitemap entries, GSC submissions) uses the **bare apex, lowercase** - e.g. `https://example.com/services`. The subdomain form, `www.example.com`, and `EXAMPLE.COM` all canonicalize to the same string so Google consolidates indexing and link equity onto one hostname. Two layers cooperate: 1. **Storage:** `normalizeDomainInput` (`lib/custom-domain-validation.ts`) strips `www.` and lowercases at write time, so `dealer.customDomain` is always stored as the bare apex. 2. **Emission:** `stripWwwPrefix` (`lib/host-utils.ts:6`) is applied defensively at output time in `getDealerOrigin` (`lib/seo-utils.ts:59`) and `lookupDealerCanonicalHost`, so any legacy or hand-edited row that slipped through still produces a clean canonical. A trailing slash on a stored host is also stripped to avoid a double slash downstream. (PR #859) Dealers without an active custom domain canonicalize to their subdomain (`bobsoil.myamsoil.com`) - that path is untouched by the www-strip. ### No-trailing-slash homepage form (PR #903) The homepage canonicalizes to the **bare origin with NO trailing slash** - `https://host`, not `https://host/`. `getCanonicalUrl(dealer, '')` and `getCanonicalUrl(dealer, '/')` both return the bare origin (`lib/seo-utils.ts:90-92`). The reason is alignment with what Next.js actually renders. Next's Metadata API renders `` and `og:url` with `trailingSlash: false` (our `next.config` default) and strips any trailing slash, so emitting `/` here would be silently discarded on the page and only desync the surfaces we render by hand (JSON-LD `url` fields and the sitemap ``). Every surface is therefore aligned on the no-slash form: - Canonical tag / `og:url` - Next renders no-slash. - Sitemap `` - `app/sitemap.ts` passes `''` for the homepage (`buildDealerSitemap` / `buildMainSitemap`). - JSON-LD `url` fields (`AutoPartsStore`, `WebSite`, `WebPage`) use `pageUrl` (the bare origin) directly; only the `@id` **fragment** identifiers keep the `${pageUrl}/#localbusiness` form (`components/DealerTemplate.tsx`, `generateDealerSchema`). - GSC submissions - the indexing worker submits the bare origin too (`homepageUrl` in `lib/gsc-jobs.ts:20-26`, `getDealerOrigin` in `lib/gsc-preflight.ts:48-51`). A CI test guards `next.config`'s `trailingSlash` staying `false` (`lib/__tests__/seo-utils.test.ts`); flipping it to `true` would silently turn this fix into a mismatch. An earlier change (PR #900) had added the trailing slash; PR #903 reverted to the no-slash form so canonical, og:url, JSON-LD, and sitemap all agree. ## Apex domain canonical identity (PRs #900, #901) The four dealer **parent apex** domains - `myamsoil.com`, `shopamsoil.com`, `myamsoil.ca`, `shopamsoil.ca` - serve host-aware stub homepages with genuinely distinct copy and DOM layout per domain (Google doorway-page policy: no near-duplicate doorway sites). The per-apex identity, positioning copy, layout variant, and metadata live in `lib/apex-config.ts`; routing of an apex `/` to the internal stub at `/home/{slug}` happens in `proxy.ts` via `getApexFromHost()`. The apex **set** is **derived**, not hand-maintained. `lib/production-domains.ts` is the single source of truth: `PRODUCTION_DOMAINS` minus `PLATFORM_DOMAIN` yields `AMSOIL_APEX_DOMAINS` (`lib/production-domains.ts:45-47`), so the set can never drift from the registry. The platform domain `amsoil.aimclear.com` is intentionally excluded - it keeps its AIMCLEAR (dealer-acquisition) branding via `public/index.html` and is never rewritten to a stub. `AMSOIL_APEX_DOMAINS` is a `Set` for O(1) **exact**-match lookups: a dealer subdomain like `bobsoil.myamsoil.com` must NOT match, so the subdomain-aware `matchesDomainList()` is deliberately not used for it. The derived set is not the whole story for a **brand-new** apex, though. The host-aware stub identity, copy, layout variant, metadata, **and routing** all key off `APEX_CONFIGS` in `lib/apex-config.ts`: `getApexFromHost()` iterates `APEX_CONFIGS` with exact host matching (`lib/apex-config.ts:374-381`), and the `/home` and `/directory` route params are generated from `Object.keys(APEX_CONFIGS)` (`app/home/[domainSlug]/page.tsx:34-35`, `app/directory/[domainSlug]/page.tsx:26-27`). So adding a fifth apex means **both** a `PRODUCTION_DOMAINS` registry entry **and** an `APEX_CONFIGS` entry - PR #901's derivation only keeps the apex set in sync, it does not synthesize the per-apex config or stub/directory pages. ### Dealer JSON-LD cleanup and enrichment (PR #900) `generateDealerSchema` (`components/DealerTemplate.tsx`) emits the homepage structured-data graph. PR #900: - Removed invalid AMSOIL **Product** nodes that misrepresented the dealer as the manufacturer. - Enriched the `AutoPartsStore` node with `image`, `logo`, `hasMap`, `sameAs`, and `priceRange`. - Made the schema respect `hideAddress` - `streetAddress` and `hasMap` are omitted when the dealer has hidden their address. - Used `dealer.domainPrefix` (not a hardcoded `myamsoil`) for the schema `@id`/`url`, so `shopamsoil` dealers no longer point at a sibling's site in knowledge graphs. ## Per-page JSON-LD on dealer CMS pages (PR #910) Until PR #910 only the dealer **homepage** emitted structured data (the `AutoPartsStore` graph). Dealer blog posts and CMS pages now also emit page-level JSON-LD, built by pure functions in `lib/cms/article-jsonld.ts` (same pattern as `lib/directory-jsonld.ts` / `generateDealerSchema` - plain objects, optional fields omitted so no nulls serialize). Two graphs: - **`BlogPosting`** (blog posts only): `headline`, `description` (`metaDescription` → `excerpt`), `datePublished` (`publishedAt` → `createdAt`), `dateModified` (`updatedAt`), `mainEntityOfPage` (custom-domain-aware canonical), `image` (absolute `featuredImage` → cached OG image), `author` (a `Person` from the post author / dealer contact name, else an `Organization`), and `publisher` (`Organization` + logo). - **`BreadcrumbList`**: `Home → Blog → {post}` on blog posts, `Home → Blog` on the blog index, `Home → {page}` on regular CMS pages. All URLs are absolute, from `getCanonicalUrl` / `getDealerOrigin`, so they follow the no-trailing-slash convention above. User content is never interpolated into the JSON string; `renderJsonLdScript` serializes via `JSON.stringify` and escapes `<`. ## Apex directory JSON-LD (PRs #909, #918) The public apex dealer directory (`{apex}/dealers` hub and `{apex}/dealers/` region pages) emits its own structured data, built by pure functions in `lib/directory-jsonld.ts` keyed off `apex.hostname` (not a dealer host). Both surfaces return a `@graph`: - **Hub** (`buildHubJsonLd`): a `WebSite` node, a `CollectionPage` (dealer-count `description`, `isPartOf` the apex `#website`), and a **`BreadcrumbList`** `{siteName} → Dealers` (added in PR #918 - the region pages had a breadcrumb since #909, the hub did not). - **Region** (`buildRegionJsonLd`): a `WebSite` node, a `CollectionPage` (`isPartOf` the apex `#website`), a **`BreadcrumbList`** `{siteName} → Dealers → {region}`, and an `ItemList` naming each dealer with their canonical site URL. **Self-contained `WebSite` / `isPartOf` (PR #918).** Both pages emit the apex's `WebSite` node - the same `@id` (`${origin}/#website`), `name` (`apex.siteName`), and `description` the apex stub homepage emits (`app/home/[domainSlug]/page.tsx`), built by the shared `buildWebSiteNode` helper. Each `CollectionPage.isPartOf` then resolves to a node defined in the **same** `@graph` rather than dangling at a `#website` `@id` that only exists on the homepage. This keeps the page self-contained for parsers and reinforces the per-apex site name that the `#website` machinery exists to fix (see "Apex domain canonical identity" above). The region `CollectionPage` previously had no `isPartOf` at all; PR #918 made the two pages symmetric. Both breadcrumbs are produced by the shared `buildBreadcrumbList` helper, so positions, the absolute `item` URL on every crumb (including the last/current one, per Google's guidance), and structure stay identical across the two pages. URLs follow the no-trailing-slash convention above; crumb 1 uses `apex.siteName` (not a literal "Home") to match the apex stub homepage identity. These are the indexable local-discovery surfaces (`/directory/` is the internal route prefix; `/dealers` is the public rewrite), so the breadcrumb can earn the Google breadcrumb rich result. This builder is **not** shared with the dealer-host `buildBreadcrumbJsonLd` in `lib/cms/article-jsonld.ts` (PR #910): that one resolves dealer-host canonicals and would be wrong for cross-host apex pages. ## Display-name privacy gating (PR #911) `getDealerDisplayName` (`lib/dealer-display-name.ts`) resolves the name shown on public surfaces with the chain `businessName → contactName → fallback`. The `contactName` step is **personal data**, gated by the dealer's `showContactName` toggle (Prisma default `false`). Previously the `contactName` fallback was ungated: a dealer with no `businessName` and `showContactName=false` leaked their personal name into the homepage title/description/`og:site_name`, the `AutoPartsStore` JSON-LD `name`, the H1, CMS/blog titles, publisher header/footer, and article JSON-LD author/publisher. The fix moves the gate **into the helper** (`lib/dealer-display-name.ts:37`): `contactName` participates only when `context === 'dealer'` (dashboard/editor - the dealer sees their own name) or `showContactName === true`. A public surface for an un-opted-in dealer with no business name renders the generic `AMSOIL Independent Dealer`. `DealerNameFields.showContactName` is a **required** field (`boolean | null`) so the compiler forces every caller to wire the toggle; the same requirement is propagated to `DealerData` (publisher) and `JsonLdDealer` (article JSON-LD). The OG-image generator dropped its now-redundant local gating wrapper and passes the toggle through. ## Dealer affiliation modeled as a credential, not brand ownership (PR #915) `generateDealerSchema` (`components/DealerTemplate.tsx`) previously declared a `LocalBusiness.brand` → AMSOIL `Brand` edge and typed the AMSOIL node as `['Organization', 'Brand']` with `@id` `/#brand`. That **falsely implied the independent dealer owns the AMSOIL brand**, which Google's structured-data policy treats as an affiliation/ownership misrepresentation. PR #915 corrects the affiliation modeling: - **Dropped the `brand` edge.** The AutoPartsStore no longer claims `brand: AMSOIL`. - **Re-typed the AMSOIL node.** It is now `@type: 'Organization'` with `@id` `/#organization` (was `@type: ['Organization', 'Brand']`, `@id` `/#brand`). - **Added `hasCredential`** on the `AutoPartsStore` - an `EducationalOccupationalCredential` with `name` "Authorized AMSOIL Dealer", `credentialCategory` "Authorized Dealer", and `recognizedBy` pointing at the AMSOIL `#organization` node. This models the relationship accurately: AMSOIL **recognizes** the dealer as authorized, rather than the dealer owning the brand. - **Added `email`** on the `AutoPartsStore`, emitted only when the dealer has an email (optional field omitted otherwise, like the rest of the graph). - **Added `knowsAbout`** - a static list of topics the dealer deals in (synthetic oil, motor oil, lubricant, grease, oil/air/fuel filters), each emitted as a `Thing` with a `sameAs` link to its canonical Wikipedia article for entity grounding. - **Added `disambiguatingDescription`** - `Independent, authorized AMSOIL dealer based in {city}, {state}.` - a one-line entity-disambiguation string that keeps the independent dealer distinct from AMSOIL INC. in the Knowledge Graph. The conditional `founder` → `Person` node (the dealer's named contact with `jobTitle` "Authorized AMSOIL Dealer", gated on `showContactName`, added in PR #914) is unchanged by this PR. ## GSC indexing pipeline (PRs #849, #861, #864, #867, #896) Behind the support-facing "indexing takes ~7 days" guidance is an automated **Google Search Console** pipeline that submits dealer URLs and sitemaps to Google. It is a durable, queue-driven worker - not a fire-and-forget call - so transient Google API failures retry and resume across deploys. ### Job queue and worker Work is modeled as `GscJob` rows (Prisma). A cron-driven worker drains them: - **Drain endpoint:** `app/api/cron/gsc-drain/route.ts` - POST-only, Bearer-authenticated with `CRON_SECRET` (timing-safe compare). No GET alias on purpose: an authenticated GET could let a link-prefetch or log replay silently re-drain the queue. The server crontab calls it with `curl -X POST`. - **Worker:** `runWorker` in `lib/gsc-worker.ts` claims a batch (default 50), processes each job sequentially, and writes a terminal status (`completed` / `failed` / `wont_fix`) or defers/retries. - **Job builders:** `lib/gsc-jobs.ts` enqueues jobs on dealer lifecycle events (activation, publish, admin re-index, cancellation, custom-domain activation/deactivation). **Job types** (`lib/gsc-worker.ts`, `executeJob`): `submit_sitemap`, `inspect_url`, `notify_indexing`, `verify_property`, `register_property`, `delete_sitemap`, `delete_property`. The first five are "indexing-class" (`INDEXING_JOB_TYPES`); the two `delete_*` types are cleanup and always run. **Reliability invariants** (see comments in `lib/gsc-worker.ts`): - **Stale-`running` recovery:** rows stuck `running` longer than `STALE_RUNNING_MINUTES` (15) are reset to `pending` before each claim, with a per-job heartbeat re-stamp of `startedAt` so a still-live worker is never mistaken for a dead one. Every job body is idempotent, so re-execution is safe. - **Retry/backoff:** `BACKOFF_MS` escalates 1m → 5m → 15m → 1h → 6h. 403 is non-retryable (fails loudly once); 429/5xx are retryable. Exhausted jobs dead-letter (`prisma/migrations/...gsc_dead_letter`). - **Quota:** the Indexing API has a hard daily cap of 200 (`INDEXING_API_DAILY_CAP`); the worker enforces a soft cap of 180 (`INDEXING_API_SOFT_CAP`) - a 10% reserve - and the anomaly report's "quota high" alert fires at the same 0.9 threshold (`lib/gsc-constants.ts`). ### Custom-domain verification & registration (heal chain) A dealer on an **active custom domain** needs its own GSC URL-prefix property, which requires proving ownership. The chain is `verify_property → register_property → submit_sitemap` (`lib/gsc-worker.ts`): 1. **`verify_property`** fetches the Google site-verification META token, stores it on `dealer.gscVerificationToken`, then **revalidates the dealer's ISR pages** so the live page actually carries the `` tag. If revalidation fails, the step throws a **retryable** error - the `register_property` follow-up is NOT enqueued until the tag is provably live (token write + getVerificationToken are idempotent/deterministic). 2. **`register_property`** gates on the GSC property cap (`GSC_PROPERTY_CAP` = 1000, minus `FIXED_GSC_PROPERTY_COUNT` = 5 fixed properties; over-cap fails **non-retryably** with `GSC_PROPERTY_CAP_ERROR_PREFIX`, surfaced as a `property_cap` anomaly), then `verifyOwnership` + `addProperty`, stamps `gscPropertyRegisteredAt`, and enqueues the sitemap. 3. **`delete_property`** (on custom-domain deactivation) deletes the property + verification, clears `gscPropertyRegisteredAt`/`gscVerificationToken`, and revalidates to purge the stale meta tag. Only a 404 from Google is swallowed (idempotent); 403 propagates (non-retryable, human-worth), 429/5xx retry. For an active custom-domain dealer whose property is not yet registered (`gscPropertyRegisteredAt == null`), activation/admin-reindex route through `verify_property` (the full chain) instead of bare indexing rows that would 403 on an unverified property (`lib/gsc-jobs.ts`). ### META verification token extraction + auto-heal (PR #861) Google's `siteVerification/v1/token` endpoint with `verificationMethod=META` returns the **full HTML meta tag string** in the `token` field, not just the bare verification code. `extractVerificationCode` (`lib/google/site-verification.ts:38`) pulls the `content="…"` value out via a single anchored regex; `getVerificationToken` extracts at write time, and the dealer page also extracts defensively at read time (`app/sites/[subdomain]/[domainSlug]/page.tsx`) so any legacy malformed value still renders a valid tag. A data-heal migration (`prisma/migrations/20260522200156_heal_malformed_gsc_verification_tokens`) repairs dealers whose token was persisted as the full `` string by the pre-fix code: it NULLs the malformed token, deletes the failed `register_property` terminal rows that the bad token caused, and enqueues a fresh `verify_property` to restart the chain. Idempotent - a re-run with no malformed tokens left is a no-op. ### Preflight HEAD→GET fallback (PR #864) Before burning Indexing-API quota on a `submit_sitemap` / `notify_indexing` job, the worker preflights the dealer's homepage (`lib/gsc-preflight.ts`). It probes the **bare origin** (`getDealerOrigin`, no trailing slash) - exactly what the pipeline submits to Google, not the on-page document URL. It issues a `HEAD` first; if the server answers **405 Method Not Allowed** (some Apache/nginx configs reject HEAD via `limit_except`), it **falls back to GET** so a server-config quirk isn't misread as "unreachable" and doesn't block indexing. A healthy result is cached for 6h (`gscPreflightHealthyAt`); a non-2xx/3xx result defers the job 1h without consuming a retry attempt. ### Eligibility gating + self-heal (PR #896) Only dealers whose sites are actually published and serving 200s participate. `GSC_ELIGIBLE_STATUSES = ['active', 'cancelled_pending']` (`lib/gsc-constants.ts`); `cancelled_pending` stays eligible because the dealer is still serving during a paid grace period and de-indexing a live, still-paying site would be wrong. Eligibility is checked at three layers: 1. **Enqueue gates** - every `enqueue*` in `lib/gsc-jobs.ts` short-circuits for ineligible dealers (first line of defense). 2. **Before preflight** (the PR #896 fix) - for `submit_sitemap` / `notify_indexing` the worker checks `isGscEligibleStatus(dealer.status)` **before** calling `preflight()`. A suspended/cancelled dealer's site usually returns 404, which would otherwise make preflight defer the job for 1h indefinitely (an infinite defer loop). Checking first no-op-completes the job (`skipped: dealer no longer GSC-eligible`) without a network round-trip. `status` was added to `preflightSelect` for this. 3. **Defense-in-depth in `executeJob`** - indexing-class jobs throw a `{ skip: true }` sentinel for dealers who became ineligible between enqueue and execution. On cancellation, `enqueueDealerCancellation` (`lib/gsc-jobs.ts`) marks stale pending/running indexing jobs `wont_fix` (not `failed`, to keep the anomaly report accurate) and enqueues a `delete_sitemap`. ### Admin UX & 403 handling (PR #867) The `/admin/gsc` dashboard and per-dealer re-index controls drive and observe the queue: a summary API (`app/api/admin/gsc/summary`), jobs list (`app/api/admin/gsc/jobs`), dead-letter release/wontfix endpoints (`app/api/admin/gsc/dead-letter/[id]/...`), backfill, and a per-dealer re-index (`app/api/admin/dealers/[id]/reindex-gsc`). PR #867 overhauled this UX and made it resilient to Google 403s (treated as non-retryable rather than infinitely looping).