SearchDock's scan of 555 top sites: 14.4% disallow GPTBot in robots.txt, but 9% turn it away at the server, and 6.5% do so while robots.txt allows it. Vendor study.
#AEO #llmstxt
#AEO #llmstxt
SearchDock: top sites block AI crawlers at the server
A robots.txt read alone misses server-level refusals: on this panel 9% of sites turned GPTBot away at the server against 14.4% that disallowed it in robots.txt, and on most of the server blockers the file looked open.
anythingengineoptimization.com
November 2, 2026 at 12:21 AM
SearchDock's scan of 555 top sites: 14.4% disallow GPTBot in robots.txt, but 9% turn it away at the server, and 6.5% do so while robots.txt allows it. Vendor study.
#AEO #llmstxt
#AEO #llmstxt
Your company doesn't show up in AI search: what a GEO setup covers
AI assistants skip sites they can't crawl or quote. Check crawl access first, then schema and llms.txt. AumCreate does a $199 GEO setup on a site you already have. #geo #aiSearch #structuredData #llmstxt #aiCrawlers
AI assistants skip sites they can't crawl or quote. Check crawl access first, then schema and llms.txt. AumCreate does a $199 GEO setup on a site you already have. #geo #aiSearch #structuredData #llmstxt #aiCrawlers
October 29, 2026 at 6:40 AM
Your company doesn't show up in AI search: what a GEO setup covers
AI assistants skip sites they can't crawl or quote. Check crawl access first, then schema and llms.txt. AumCreate does a $199 GEO setup on a site you already have. #geo #aiSearch #structuredData #llmstxt #aiCrawlers
AI assistants skip sites they can't crawl or quote. Check crawl access first, then schema and llms.txt. AumCreate does a $199 GEO setup on a site you already have. #geo #aiSearch #structuredData #llmstxt #aiCrawlers
How `cats.txt` Showed `llms.txt` Evidence is GEO Astrology
https://markwilliamscook.substack.com/p/how-catstxt-showed-llmstxt-evidence
How the joke web standard `cats.txt` took the fancy of some SEOs and became cited at the same evidential level as `llms.txt` for optimising websites for search […]
https://markwilliamscook.substack.com/p/how-catstxt-showed-llmstxt-evidence
How the joke web standard `cats.txt` took the fancy of some SEOs and became cited at the same evidential level as `llms.txt` for optimising websites for search […]
Original post on social.vivaldi.net
social.vivaldi.net
October 29, 2026 at 3:59 PM
How `cats.txt` Showed `llms.txt` Evidence is GEO Astrology
https://markwilliamscook.substack.com/p/how-catstxt-showed-llmstxt-evidence
How the joke web standard `cats.txt` took the fancy of some SEOs and became cited at the same evidential level as `llms.txt` for optimising websites for search […]
https://markwilliamscook.substack.com/p/how-catstxt-showed-llmstxt-evidence
How the joke web standard `cats.txt` took the fancy of some SEOs and became cited at the same evidential level as `llms.txt` for optimising websites for search […]
llms.txt v2: what an agent actually reads
The August 2026 revision of llms.txt lets the file live at any path and adds link relations so agents can find it, and the format is stricter than it looks.
#llmstxt #ai #seo
The August 2026 revision of llms.txt lets the file live at any path and adds link relations so agents can find it, and the format is stricter than it looks.
#llmstxt #ai #seo
llms.txt v2: what an agent actually reads
The August 2026 revision of llms.txt lets the file live at any path and adds link relations so agents can find it, and the format is stricter than it looks.
richi.codes
October 26, 2026 at 4:00 PM
GEO vs SEO: the difference, and whether small sites need it
SEO competes for position on a results page. GEO competes to be quoted inside an AI answer: plain sentences, clear facts, schema, an llms.txt. Small sites: yes, but it's repackaging,… #GEO #SEO #llmstxt #schema #smallBusinessSite
SEO competes for position on a results page. GEO competes to be quoted inside an AI answer: plain sentences, clear facts, schema, an llms.txt. Small sites: yes, but it's repackaging,… #GEO #SEO #llmstxt #schema #smallBusinessSite
October 24, 2026 at 4:49 AM
GEO vs SEO: the difference, and whether small sites need it
SEO competes for position on a results page. GEO competes to be quoted inside an AI answer: plain sentences, clear facts, schema, an llms.txt. Small sites: yes, but it's repackaging,… #GEO #SEO #llmstxt #schema #smallBusinessSite
SEO competes for position on a results page. GEO competes to be quoted inside an AI answer: plain sentences, clear facts, schema, an llms.txt. Small sites: yes, but it's repackaging,… #GEO #SEO #llmstxt #schema #smallBusinessSite
ContextFetch's 10,000-site scan found top-1,000 sites block GPTBot 17.4% of the time, versus 12.1% for rank 5,001-10,000. Claude-SearchBot was the least-blocked of 17 crawlers checked, at just 4.6%.
#AEO #llmstxt
#AEO #llmstxt
ContextFetch finds bigger sites block AI crawlers more often
The size gradient suggests AI-crawler blocking tracks which sites have the resources to run bot-management infrastructure, not a uniform industry stance on AI access.
anythingengineoptimization.com
October 26, 2026 at 12:23 AM
ContextFetch's 10,000-site scan found top-1,000 sites block GPTBot 17.4% of the time, versus 12.1% for rank 5,001-10,000. Claude-SearchBot was the least-blocked of 17 crawlers checked, at just 4.6%.
#AEO #llmstxt
#AEO #llmstxt
Vercatus tested 64 well-known sites across 8 niches: software/SaaS sites let AI crawlers in 7 of 8 times, travel sites just 1 of 8. Only 17 of 64 publish an llms.txt.
#AEO #Perplexity #llmstxt
#AEO #Perplexity #llmstxt
Vercatus: AI crawlers read SaaS sites, not travel sites
Niche determines AI access more than any blanket industry stance — travel sites, whose business is answering exactly the questions assistants get asked about trip planning, are the category assistants can least read directly.
anythingengineoptimization.com
October 26, 2026 at 12:23 AM
Vercatus tested 64 well-known sites across 8 niches: software/SaaS sites let AI crawlers in 7 of 8 times, travel sites just 1 of 8. Only 17 of 64 publish an llms.txt.
#AEO #Perplexity #llmstxt
#AEO #Perplexity #llmstxt
What belongs in llms.txt (and what to leave out)
llms.txt is a short map for AI readers: site name, one line on what you sell, then links to canonical pages — pricing, product, docs, contact — each with a brief note on what it answers… #wordpress #llmstxt #aiSearch #aumviso #siteStructure
llms.txt is a short map for AI readers: site name, one line on what you sell, then links to canonical pages — pricing, product, docs, contact — each with a brief note on what it answers… #wordpress #llmstxt #aiSearch #aumviso #siteStructure
October 22, 2026 at 3:18 PM
What belongs in llms.txt (and what to leave out)
llms.txt is a short map for AI readers: site name, one line on what you sell, then links to canonical pages — pricing, product, docs, contact — each with a brief note on what it answers… #wordpress #llmstxt #aiSearch #aumviso #siteStructure
llms.txt is a short map for AI readers: site name, one line on what you sell, then links to canonical pages — pricing, product, docs, contact — each with a brief note on what it answers… #wordpress #llmstxt #aiSearch #aumviso #siteStructure
Votre fichier destiné aux assistants IA mérite-t-il une vérification ?
Préparer un fichier pour assistants IA
Préparez votre fichie#LLMsTxtM#AuditByBIWByBIW
Préparer un fichier pour assistants IA
Préparez votre fichie#LLMsTxtM#AuditByBIWByBIW
October 21, 2026 at 12:27 PM
Votre fichier destiné aux assistants IA mérite-t-il une vérification ?
Préparer un fichier pour assistants IA
Préparez votre fichie#LLMsTxtM#AuditByBIWByBIW
Préparer un fichier pour assistants IA
Préparez votre fichie#LLMsTxtM#AuditByBIWByBIW
robotcheck v1.18.0 is out. Two new checks look at how your llms.txt points to an MCP server: whether it references a server card and names the transport protocol. Both are info-level, up to 1 point each. 74 active checks now. https://robotcheck.coffee/changelog #llmstxt
October 19, 2026 at 5:57 AM
robotcheck v1.18.0 is out. Two new checks look at how your llms.txt points to an MCP server: whether it references a server card and names the transport protocol. Both are info-level, up to 1 point each. 74 active checks now. https://robotcheck.coffee/changelog #llmstxt
llms.txt: What It Is, What It Does Not Do, and How to Write One, by @sitelemetry.bsky.social:
https://sitelemetry.com/blog/llms-txt?ref=frontenddogma.com
#howtos #llmstxt #ai
https://sitelemetry.com/blog/llms-txt?ref=frontenddogma.com
#howtos #llmstxt #ai
How to Write llms.txt: Format, Example and Limits | Sitelemetry
How to write an llms.txt file: the Markdown format, a minimal example to copy, how it differs from robots.txt and what Google and AI providers say about it.
sitelemetry.com
October 18, 2026 at 12:30 PM
llms.txt: What It Is, What It Does Not Do, and How to Write One, by @sitelemetry.bsky.social:
https://sitelemetry.com/blog/llms-txt?ref=frontenddogma.com
#howtos #llmstxt #ai
https://sitelemetry.com/blog/llms-txt?ref=frontenddogma.com
#howtos #llmstxt #ai
Claude SEO Skill benchmark: Why HTML docs fail for AI coding agents. Context window tax & DOM noise. How autonomous skills deploy /llms.txt: https://seoskillsai.com/claude/seo-llms-txt #SEOSkills #LLMsTxt #GEO
🔗 https://seoskillsai.com
🔗 https://seoskillsai.com
Claude SEO Skill: How to Build and Deploy /llms.txt for Claude Code
Claude SEO Skill benchmark: Why HTML docs fail for AI coding agents. Context window tax & DOM noise. How autonomous skills deploy /llms.txt: https://seoskillsai
seoskillsai.com
October 16, 2026 at 11:03 AM
Claude SEO Skill benchmark: Why HTML docs fail for AI coding agents. Context window tax & DOM noise. How autonomous skills deploy /llms.txt: https://seoskillsai.com/claude/seo-llms-txt #SEOSkills #LLMsTxt #GEO
🔗 https://seoskillsai.com
🔗 https://seoskillsai.com
AI Crawler Llmstxt Policy: turn public web data into a ranked decision for AI and agent operations. Built as a pay-per-event Apify Actor for teams that need decisions, not raw scraping.
Ai Crawler Llmstxt Policy Auditor
Run this QuantMatrix Actor on Apify.
apify.com
October 12, 2026 at 5:50 AM
AI Crawler Llmstxt Policy: turn public web data into a ranked decision for AI and agent operations. Built as a pay-per-event Apify Actor for teams that need decisions, not raw scraping.
Does Your Small Business Website Need an llms.txt File?
usacontentpros.com/llms-txt-file/
#llmstxt #AICrawlers #SmallBusinessWebsite #TechnicalSEO #DigitalMarketingNews #AIVisibility #WebsiteTips #GenerativeSearch #SmallBusinessMarketing #2026Trends
usacontentpros.com/llms-txt-file/
#llmstxt #AICrawlers #SmallBusinessWebsite #TechnicalSEO #DigitalMarketingNews #AIVisibility #WebsiteTips #GenerativeSearch #SmallBusinessMarketing #2026Trends
Does Your Small Business Website Need an llms.txt File? - USA Content Pros
Among large companies, only about seven percent had adopted it as of a recent survey, well behind the near-universal adoption of standard robots.txt.
usacontentpros.com
October 11, 2026 at 3:06 PM
Does Your Small Business Website Need an llms.txt File?
usacontentpros.com/llms-txt-file/
#llmstxt #AICrawlers #SmallBusinessWebsite #TechnicalSEO #DigitalMarketingNews #AIVisibility #WebsiteTips #GenerativeSearch #SmallBusinessMarketing #2026Trends
usacontentpros.com/llms-txt-file/
#llmstxt #AICrawlers #SmallBusinessWebsite #TechnicalSEO #DigitalMarketingNews #AIVisibility #WebsiteTips #GenerativeSearch #SmallBusinessMarketing #2026Trends
Subagents SEO LLMs.txt benchmark: Why AI bots skip heavy JS websites. Ingestion timeouts and context bloat. How autonomous subagents deploy edge /llms.txt endpoints: https://seoskillsai.com/subagents/seo-llms-txt #SEOSkills #LLMsTxt #GEO
October 8, 2026 at 10:27 AM
Subagents SEO LLMs.txt benchmark: Why AI bots skip heavy JS websites. Ingestion timeouts and context bloat. How autonomous subagents deploy edge /llms.txt endpoints: https://seoskillsai.com/subagents/seo-llms-txt #SEOSkills #LLMsTxt #GEO
L'étude : ahrefs.com/blog/llmstxt...
We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read
We analyzed 137K sites & found 97% of llms.txt files are never read. See why bots ignore these files & what this means for your AI visibility
ahrefs.com
October 8, 2026 at 4:19 AM
L'étude : ahrefs.com/blog/llmstxt...
llms.txt: one markdown file, domain root, an afternoon's work.
Not official. Google says unneeded for AI Overviews. Crawlers fetch it anyway.
Vendor tests claim 30 to 70 percent more citations. Those vendors sell llms.txt tools.
Direction: yes. Magnitude: unproven.
#llmsTxt #GEO
Not official. Google says unneeded for AI Overviews. Crawlers fetch it anyway.
Vendor tests claim 30 to 70 percent more citations. Those vendors sell llms.txt tools.
Direction: yes. Magnitude: unproven.
#llmsTxt #GEO
October 2, 2026 at 8:50 AM
LintLab's audit of 1,000 top sites found just 6.6% serve a valid llms.txt file — echoing Ahrefs' August finding that 97% of those files go unread by AI crawlers anyway.
Via PPC Land.
#AEO #llmstxt
Via PPC Land.
#AEO #llmstxt
LintLab: sitemap errors common, llms.txt still rare
A third independent dataset — after Ahrefs and Common Crawl — shows llms.txt adoption still stuck in single digits, while the more basic problem of broken sitemaps remains unsolved across most of the open web.
anythingengineoptimization.com
October 2, 2026 at 9:09 PM
LintLab's audit of 1,000 top sites found just 6.6% serve a valid llms.txt file — echoing Ahrefs' August finding that 97% of those files go unread by AI crawlers anyway.
Via PPC Land.
#AEO #llmstxt
Via PPC Land.
#AEO #llmstxt
Zensical 0.0.67 is here ✨
MkDocs-compatible social cards lead this release, alongside exclude and llmstxt plugins. Enjoy!
Release notes
github.com/zensical/zen...
MkDocs-compatible social cards lead this release, alongside exclude and llmstxt plugins. Enjoy!
Release notes
github.com/zensical/zen...
Release 0.0.67 · zensical/zensical
Summary
Zensical now supports Material for MkDocs’ social plugin natively, generating social cards and Open Graph and Twitter metadata with bundled or custom layouts. This release also adds native ...
github.com
September 30, 2026 at 10:05 AM
Zensical 0.0.67 is here ✨
MkDocs-compatible social cards lead this release, alongside exclude and llmstxt plugins. Enjoy!
Release notes
github.com/zensical/zen...
MkDocs-compatible social cards lead this release, alongside exclude and llmstxt plugins. Enjoy!
Release notes
github.com/zensical/zen...
Lighthouse 13.5.0 の ard-schema 監査が探すのは /.well-known/ai-catalog.json。8月26日付の ARD 仕様 v0.91 が MUST としたのは /.well-known/ard.json です。仕様通りに置くとレポートはN/A。「見つからない」ではなく「見に行っていない」N/Aです。判定できた30サイトでは、どちらのファイルも0件でした。
menumenu.life/blog/lightho...
#LLMO #llmstxt
menumenu.life/blog/lightho...
#LLMO #llmstxt
Lighthouse 13.5の「エージェント監査」、0点は欠陥とは限らない｜日本語30サイト実測で分かった読み方
「Lighthouse で0点が出た。直さなければ」。レポートを受け取った担当者は、まずそう読みます。2026年9月18日に公開された Lighthouse 13.5.0 は、Agentic Browsing カテゴリに ARD（Agentic Resource Discovery）の監査 ard-schema を加え...
menumenu.life
September 24, 2026 at 3:33 AM
Lighthouse 13.5.0 の ard-schema 監査が探すのは /.well-known/ai-catalog.json。8月26日付の ARD 仕様 v0.91 が MUST としたのは /.well-known/ard.json です。仕様通りに置くとレポートはN/A。「見つからない」ではなく「見に行っていない」N/Aです。判定できた30サイトでは、どちらのファイルも0件でした。
menumenu.life/blog/lightho...
#LLMO #llmstxt
menumenu.life/blog/lightho...
#LLMO #llmstxt
https://netcontentseo.net/article/john-mueller-spots-an-unexpected-llmstxt-problem-crawlers-may-discover-the-files-through-third-party-indexes-868 #LLMSTxt #AISEO #Crawlers #SEO (2/2)
September 19, 2026 at 5:38 PM
https://netcontentseo.net/article/john-mueller-spots-an-unexpected-llmstxt-problem-crawlers-may-discover-the-files-through-third-party-indexes-868 #LLMSTxt #AISEO #Crawlers #SEO (2/2)
The IETF's AI preferences vocabulary draft — the standard behind machine-readable alternatives to ad hoc robots.txt rules — split \"AI Training\" into separate Training and Use sections in revision 08, while keeping Search as its own category.
#AEO #llmstxt
#AEO #llmstxt
IETF AI preference draft splits training from use
A standard splitting \"AI training\" from \"AI use\" from \"search\" gives sites a more precise vocabulary to say yes to being indexed and cited while saying no to being used as training data — the distinction publishers blocking crawlers have been drawing informally, here formalized in draft.
anythingengineoptimization.com
September 16, 2026 at 11:21 PM
The IETF's AI preferences vocabulary draft — the standard behind machine-readable alternatives to ad hoc robots.txt rules — split \"AI Training\" into separate Training and Use sections in revision 08, while keeping Search as its own category.
#AEO #llmstxt
#AEO #llmstxt
A researcher scanned 6,214 corporate domains and found 120 AI documentation files naming software packages nobody owned. He registered some. A Fortune 500 company's coding agent installed one within the hour. Vendor docs are now executable. Read the filing #aiagents #supplychain #llmstxt #AI
Your supplier
120 corporate sites published AI documentation naming unregistered packages. Researchers claimed some and agents installed them within an hour.
steelwise.uk
September 14, 2026 at 1:12 PM
A researcher scanned 6,214 corporate domains and found 120 AI documentation files naming software packages nobody owned. He registered some. A Fortune 500 company's coding agent installed one within the hour. Vendor docs are now executable. Read the filing #aiagents #supplychain #llmstxt #AI
99.94% of AI citations point to HTML, not llms.txt. That does not prove llms.txt is useless — but it does show that claims need evidence, not assumptions. https://netcontentseo.net/article/9994-of-ai-citations-point-to-html-but-that-still-doesnt-prove-llmstxt-is-useless-781 #llmstxt (1/2)
September 13, 2026 at 5:00 PM
99.94% of AI citations point to HTML, not llms.txt. That does not prove llms.txt is useless — but it does show that claims need evidence, not assumptions. https://netcontentseo.net/article/9994-of-ai-citations-point-to-html-but-that-still-doesnt-prove-llmstxt-is-useless-781 #llmstxt (1/2)