CRCrawlRanker
Back
How CrawlRanker scores a site

Methodology

Two scores from a single URL: an SEO audit graded against best practice, and an AI-automation score that reads how much of the site looks AI-generated. Here's exactly how each one is calculated, what we trust it for, and where we know it falls short.

The pipeline at a glance

Every scan goes through four stages. We crawl the URL, extract structured signals from the HTML, run those signals through a deterministic SEO rubric, and ask an LLM to read the same page and produce an AI-automation score with a confidence percentage on every finding.

  1. Fetch. A single GET request with a custom user agent, 10-second timeout, 5 MB cap. We follow up to two redirects. Internal, local, and private hosts are blocked at the URL parser.
  2. Parse. The page is parsed with linkedom, a server-side DOM that mirrors browser behavior. We extract metadata (title, description, canonical, robots), heading structure, link inventory, image inventory, schema markup, and the visible text body.
  3. SEO score. A fixed rubric of seven checks runs against the parsed signals. Each check returns pass / warn / fail and contributes a weighted share of the 0–100 score. No LLM in this path. The same page produces the same SEO score every time.
  4. AI score. The body text and a structural summary go to Claude with a cached scoring prompt. The model returns an AI-automation score, a confidence percentage, a breakdown across content / structure / imagery / tone, and up to three short findings. If confidence is below 0.6, or the first pass cannot evaluate the page, the request is escalated to a stronger model automatically.

The SEO audit

The audit grades pages against established SEO conventions. Nothing exotic. These are the same things any reviewer would look at on a first pass, encoded so the result is repeatable.

The seven checks

  • Meta tags. Presence of a non-empty <title>, meta description and canonical link. A page with more than one description tag, or with a description Google ignores because an element that doesn’t belong in the head (such as an image) sits above it, gets a warning. Title and description lengths are shown in the report’s search preview, not scored.
  • Heading structure. Exactly one h1, and at least one h2 to give the page sections. No h1 fails; several, or no h2, is a warning.
  • Mobile readiness. A viewport meta tag set to width=device-width. The scan reads the HTML and doesn’t render the page, so it doesn’t measure touch targets or font sizes.
  • Page speed signals. What one fetch of the page can show: how long the server takes to return the HTML, how large the HTML is, how many images the page has, and how many images have no width and height set, which makes the layout jump as they load. These are signals, not a lab measurement like PageSpeed Insights, and unsized images only ever produce a warning, since a stylesheet can size images in ways the HTML doesn’t show.
  • Schema markup. At least one JSON-LD block. The check doesn’t validate the markup or its types; use Google’s Rich Results Test for that.
  • Broken links. A sample of internal links is checked with HEAD requests (GET when a server refuses HEAD), following redirects. Error responses, timeouts and network failures count as broken.
  • Image alt text. The share of <img> elements with non-empty alt text. An empty alt="" counts as missing, even on a decorative image.

The score

Every check counts equally: a pass earns full credit, a warning half and a fail none, and the score is the share earned across the seven checks. The final 0–100 maps to four verdicts: Excellent 90+, Good 75–89, Fair 60–74, Needs work below 60.

The AI-automation score

The AI score answers one question: how much of this site reads as machine-generated, top to bottom? It is a fingerprint of the site, not a verdict on any single sentence.

The four signal categories

  • Content. Lexical patterns, sentence-shape variance, hedged phrasing, factual specificity, and the markers an LLM tends to leave when it writes long prose.
  • Structure. Heading regularity, paragraph length distribution, list cadence, and the kind of templated section ordering that auto-generated pages produce.
  • Imagery. File-name patterns, dimensions, repeated alt text, and visible artifacts characteristic of generative image models.
  • Tone. Voice consistency across sections, register shifts, and the emotional flatness LLMs default to when there is no editorial pass.

Confidence

Every score ships with a confidence percentage. Confidence is the model's own self-reported certainty, not a statistical measurement. Pages with mixed signals (a templated header, a clearly human blog post, a generated FAQ) deliberately produce lower confidence. The model says “I see both,” and the verdict surface reads Mixed signals rather than committing to a side. When confidence drops below 0.6 we re-run the analysis on a stronger model to resolve the uncertainty or confirm it.

Verdict bands

  • Mostly AI-generated. Score above 65, confidence above 0.7. Multiple categories agree.
  • Mostly human-written. Score below 35, confidence above 0.7.
  • Mixed signals. Anything else. Either the score is ambiguous or confidence is low. Both lead to the same honest answer.

What the AI score is not

It is not a plagiarism detector. It does not look at any other page, it does not check for word reuse, and it does not produce a “copied from” output. A site can be entirely original and still score high on AI-automation if a human used a model to draft the prose.

It is not a quality judgment. AI-generated content can be excellent and human-written content can be terrible. The score reads automation, not value.

It is not a moderation tool. We do not flag content for hosting providers or report it anywhere. The score exists on the public report URL and nowhere else.

Limitations we know about

  • Single-page only. Phase 1 scans one URL, not a site crawl. A landing page that looks AI-generated may sit in front of a human-written blog the scanner never sees. We will add multi-page passes when the volume warrants it.
  • Language coverage. Findings are produced in English regardless of the scanned site's language. Scoring works on non-English pages, but the rendered explanation is always English in v1.
  • Auth walls and JS-only content. If the page renders most content client-side without server-rendered fallback, the audit sees only the shell. SEO checks reflect that honestly; AI scoring degrades gracefully but produces lower confidence.
  • Adversarial sites. A site engineered to evade AI detection will eventually evade ours too. We do not claim a hard ceiling on either side.

How findings are written

Every finding has three fields: a severity (Strong / Clear / Worth noting), a category, and a one-sentence headline. Headlines describe what we saw, not what to do about it. Findings are deliberately short. They exist to anchor the score, not prescribe edits.

The model returns at most three findings per scan, even when more signals are present. Forcing the model to pick its top three keeps the report scannable. The full signal set sits behind the score; it is not surfaced in v1.

Versioning

The scoring rubric and the AI prompt are versioned together. When either changes (a new check, a reweight, a prompt revision), the change is recorded in the methodology changelog with the date and the rationale. Past scans are not re-scored. A scan's scored-at timestamp implicitly fixes which methodology version it ran under.

Questions or disputes

If a scan looks wrong to you, send the report URL and what you expected to hello@crawlranker.com. Disputes that surface a real methodology issue land in the changelog.