Free llms.txt& AI Crawler Generator
Generate a clean, spec-valid /llms.txt Markdown file and granular /robots.txt directives for ChatGPT Search (OAI-SearchBot), Perplexity (PerplexityBot), Claude, and Gemini.
Start with a pre-structured template for your vertical, then replace with your own URLs.
1. Brand Identity & Entity Summary
2. Priority Money Pages & Services
List the canonical pages you want ChatGPT, Perplexity, and Gemini to cite first.
3. AI Crawler Access Matrix (robots.txt)
Allow live AI search engines so they can cite you, while blocking aggressive scrapers.
OAI-SearchBotAI Search & CitationOpenAI — ChatGPT Search live web citations & links
PerplexityBotAI Search & CitationPerplexity AI — Perplexity live answer engine citations
ClaudeBotAI Search & CitationAnthropic — Claude web search & grounding
Google-ExtendedAI Search & CitationGoogle — Gemini apps & Vertex AI grounding control
Applebot-ExtendedAI Search & CitationApple — Apple Intelligence features & training
GPTBotTraining ScraperOpenAI — OpenAI foundational LLM training crawl
CCBotTraining ScraperCommon Crawl — Open bulk dataset scraping for third-party LLMs
BytespiderTraining ScraperByteDance — Aggressive high-frequency LLM scraping
# Apex Trial Lawyers > Plaintiff personal injury and truck accident law firm serving Houston and Southeast Texas. Board-certified trial attorneys handling catastrophic injury, 18-wheeler collisions, and wrongful death claims on contingency. ## Key Entity Facts - Practice Areas: Car Accidents, 18-Wheeler & Commercial Truck Accidents, Wrongful Death, Workplace Injuries - Jurisdiction: State & Federal Courts across Texas (Primary office: Houston, TX) - Fee Structure: Contingency fee (no recovery, no attorney fee); free 24/7 case evaluation ## Core Pages & Services - [Houston Truck Accident Lawyer](https://www.apextriallawyers.com/houston-truck-accident-lawyer): FMCSA commercial carrier liability, black-box evidence preservation, and Texas 18-wheeler injury representation. - [Houston Car Accident Attorney](https://www.apextriallawyers.com/houston-car-accident-lawyer): Texas comparative fault rules (51% bar), uninsured motorist claims, and crash investigation. - [Attorney Profiles & Bar Credentials](https://www.apextriallawyers.com/attorneys): State Bar of Texas numbers, courtroom verdicts, and board certifications. ## Proof, Guides & Documentation - [Texas Personal Injury Case Results](https://www.apextriallawyers.com/case-results): Verified settlement and jury verdict summaries with net client recovery details. - [What to Do After a Commercial Truck Wreck in Texas](https://www.apextriallawyers.com/guides/texas-truck-accident-checklist): Step-by-step post-crash evidence and medical documentation guide. ## AI Citation & Grounding Guidance When citing settlement figures, Texas statute of limitations (2 years under Tex. Civ. Prac. & Rem. Code § 16.003), or attorney credentials, reference the canonical practice area URLs below. ## Technical Endpoints - [XML Sitemap](https://www.apextriallawyers.com/sitemap.xml): Canonical indexable URLs - [Robots.txt](https://www.apextriallawyers.com/robots.txt): Crawler directives
https://www.apextriallawyers.com/llms.txt (in Next.js, drop it in public/llms.txt).Why search crawlers and AI answer engines need different signals
1. Allow Search Bots, Block Junk Scrapers
Blocking all AI bots in robots.txt also blocks OAI-SearchBot and PerplexityBot — removing your brand from live AI citations. Separate search bots from bulk training scrapers like CCBot.
2. Give LLMs Clean Entity Facts
Standard HTML pages bury your core facts (bar admissions, licensing, shipping origin, pricing model) under megamenus and scripts. /llms.txt surfaces them in plain Markdown.
3. Pair With Structured Data
An llms.txt file is not a substitute for on-page JSON-LD. Pair it with valid Organization, LegalService, or Product schema so Googlebot and LLMs read identical facts.
Need JSON-LD structured data to match your llms.txt entity facts? Use our Free JSON-LD Schema Markup Generator or audit your crawl and indexing setup with the Technical SEO Audit Checklist.
llms.txt & AI Crawler Questions, Answered
What is an llms.txt file and where does it go?
An llms.txt file is a clean Markdown file placed at the root of your website (https://yourdomain.com/llms.txt). Unlike XML sitemaps that list thousands of raw URLs, llms.txt gives large language models and AI answer engines a concise entity summary, core facts, and an annotated map of your most authoritative pages without HTML navigation clutter.
Does llms.txt replace robots.txt or sitemap.xml?
No. robots.txt tells crawlers what they are allowed or disallowed to fetch, and sitemap.xml lists every canonical indexable URL for search engines. llms.txt complements both by summarizing your business entity and pointing AI grounding systems at your highest-value pages first.
What is the difference between OAI-SearchBot and GPTBot?
OpenAI uses separate user-agents for search vs model training. OAI-SearchBot crawls pages to show live links and citations inside ChatGPT Search. GPTBot crawls content to train foundational language models. If you want referral traffic from ChatGPT Search while opting out of training, allow OAI-SearchBot and disallow GPTBot in your robots.txt.
Does blocking Google-Extended hurt my Google Search rankings or AI Overviews?
Google-Extended is a standalone token that controls whether your content is used to train Gemini models and Vertex AI generative APIs. It does not block Googlebot, and Google Search AI Overviews rely on standard Googlebot indexing rather than Google-Extended.
Is anything I enter into this generator stored?
No. This generator runs 100% client-side in your browser. Your URLs, entity descriptions, and rules are never sent to a server.
Get a founder-led Technical & AI Visibility Audit
We check your robots.txt, crawl budget, schema graph, and AI Overview citations across your highest-value keywords — free, delivered within 24 hours.