Back to Blog

Can AI Find Your Website? Inside Our Technical Readiness Scan

Brightwill Team·2026-03-26

AI engines like ChatGPT, Claude, and Gemini don't just know things - they learn from web content that their crawlers collect. If those crawlers can't access your website, or if your content isn't structured in a way AI can parse, you're invisible to them regardless of how good your business is.

Our technical readiness scan checks whether your website is set up to be found, read, and understood by AI engines. It runs in parallel with the main audit (when you provide a website URL) and contributes 20% of your overall GEO Score.

brightwill.ai/report
Brightwill AI engine analysis showing per-provider visibility breakdown
Per-engine analysis shows visibility, coverage, sentiment, and top blockers/strengths for each AI platform.

What We Scan

The scan fetches three resources from your website in parallel: your robots.txt file, your homepage HTML, and your llms.txt file. From these three requests, we evaluate four areas:

AI Crawler Access
0-8 points
Source: robots.txt
Which AI crawlers are allowed to access your site
Schema.org Markup
0-6 points
Source: Homepage HTML
Structured data that helps AI understand your business
Meta Directives
0-3 points
Source: Homepage HTML
Whether meta tags block AI from using your content
llms.txt
0-3 points
Source: /llms.txt
The emerging standard for AI-specific site content

AI Crawler Access: Who Gets In?

Your robots.txt file tells web crawlers which parts of your site they can access. Traditionally, this was about search engine bots like Googlebot. Now, AI companies have their own crawlers, and blocking them means AI engines can't learn from your content.

We check for six AI crawlers:

GPTBot
OpenAImajor
Powers ChatGPT's knowledge base
ClaudeBot
Anthropicmajor
Powers Claude's knowledge base
Google-Extended
Googlemajor
Powers Gemini's training data
anthropic-ai
Anthropicmajor
Secondary Anthropic crawler
PerplexityBot
Perplexity
Powers Perplexity AI search
CCBot
Common Crawl
Open dataset used by many AI systems

Four of these are classified as “major” crawlers: GPTBot, ClaudeBot, Google-Extended, and anthropic-ai. Each unblocked major crawler contributes 2 points to your score (maximum 8 points from this section).

We check three possible statuses for each crawler: allowed (explicitly permitted or no rule blocking it), blocked (explicitly disallowed in robots.txt), or no rule (no robots.txt found, so crawlers default to allowed). If your site has no robots.txt at all, all crawlers are effectively allowed.

Schema.org Markup: Can AI Understand Your Data?

JSON-LD Schema.org markup is structured data embedded in your HTML that tells machines what your page is about. For AI engines, this is one of the most reliable ways to extract facts about your business - your name, address, hours, services, reviews.

Business Schema Types

We look for schema types that identify your business:

LocalBusinessOrganizationRestaurantHotelStoreProductServiceProfessionalService

Rich Schema Types

We also check for “rich” schema types that provide additional context AI engines can leverage:

FAQPageHowToReviewAggregateRatingBreadcrumbListArticle

The scoring is cumulative: 2 points for having any schema at all, 2 more for having a business schema type, and 2 more for having a rich schema type. A site with a well-structured LocalBusiness schema and an FAQPage earns the full 6 points.

Meta Directives: Are You Blocking AI Without Knowing It?

Some websites include meta tags that specifically tell AI systems not to use their content. The two directives we check for are:

noai - Tells AI systems not to use your content for training or responses
noimageai - Tells AI systems not to use your images

If neither directive is present, you earn 3 points. Some content management systems add these directives by default, so it's worth checking even if you didn't intentionally set them. Blocking AI from your content means AI engines have less information to work with when deciding whether to recommend you.

llms.txt: The Emerging Standard

llms.txt is a new standard (similar to robots.txt) that provides AI-specific information about your website. It's a plain text file at /llms.txt that gives AI engines a structured summary of your site - who you are, what you do, and what content is available.

If we find an llms.txt file on your site, you earn 3 points. This is still a rare feature - most websites don't have one yet. But as AI engines increasingly look for structured ways to understand websites, having an llms.txt file will become a competitive advantage.

The Scoring System

The technical readiness scan produces a score from 0 to 20, broken down across the four areas:

AI Crawler Access
8 pts
2 points per unblocked major crawler (4 major crawlers)
Schema.org Markup
6 pts
2 for any schema + 2 for business schema + 2 for rich schema
Meta Directives
3 pts
3 points if no noai/noimageai directives found
llms.txt
3 pts
3 points if llms.txt file exists

Letter Grades

The 0–20 score maps to a letter grade:

A
18-20
B
14-17
C
10-13
D
6-9
F
0-5

An A grade means your site is well-optimized for AI discovery: crawlers can access it, your data is structured, and you're not blocking AI systems. An F means significant barriers exist between AI engines and your content.

How It Affects Your GEO Score

The technical readiness score contributes 20% of your overall AI Visibility Score (0–100). This is the “Technical AI Readiness” component in the scoring formula. A perfect 20/20 technical score adds 20 points to your GEO Score, while a 0/20 adds nothing.

If no website URL is provided (for businesses without a website, or for the free audit), the 20% weight is redistributed proportionally across the other scoring components. This means the technical scan doesn't penalize businesses that don't have a website - it only rewards those that have their technical setup right.

Issues found in the technical scan also feed directly into your action plan. Blocked crawlers become “technical” action items. Missing schema becomes a “knowledge gap” action. Each issue gets a specific, prioritized recommendation for how to fix it.

Check Your Technical Readiness

The technical readiness scan runs as part of every full audit ($19). When you provide your website URL during checkout, the scan runs in parallel with the AI conversation audit - no extra time required.

Start with a free audit to see your ChatGPT recommendation probability, then upgrade to include the full technical scan plus your personalized action plan.

Continue Reading

Methodology9 min read

From Data to Action: How We Build Your Personalized GEO Action Plan

How we turn audit data into 15-30 specific optimization actions — from pre-computed gap analysis to opportunity scoring and the earned vs owned media split.

Read more
Methodology6 min read

AI Visibility Score: How We Measure If AI Recommends Your Business

A transparent look at our methodology — how we track visibility, position, and sentiment across 150+ real AI conversations to calculate your AI Visibility Score (0-100), using organic-only scoring and source attribution.

Read more
Methodology8 min read

How We Analyze Every AI Response: Our Multi-Layer Extraction Pipeline

How we turn raw AI conversations into structured data — extracting mentions, sentiment, competitor rankings, and source citations from every response.

Read more