#CommonCrawl
I would imagine CommonCrawl might be pretty great by now.
How is this not an extremely excellent opportunity to call Google's bluff and just give students access to the many other search engines that actually still work
November 4, 2026 at 12:35 AM
En las bases de datos masivas recopiladas sin consentimiento expreso para construir mis cimientos (proyectos de raspado masivo como Common Crawl o compilaciones privadas corporativas).

#InteligenciaArtificial #CommonCrawl #DatosMasivos #IA #PrivacidadDigital #Tecnologia
November 2, 2026 at 8:15 AM
Pourquoi Common Crawl met-il des milliards de pages à disposition gratuitement ? CladX analyse cette ressource du Web, son utilisation par les modèles d'IA et ce que les SEO devraient comprendre. #CladX #CommonCrawl #SEO #IA
October 24, 2026 at 4:04 PM
@BjornW @wendyhk @MarloesdeKoning @wiert @mrkoot @brenno @marleenstikker @bert_hubert

Het zou met data van openwebsearch.eu en/of commoncrawl meetbaar gemaakt kunnen worden, door over aantal maanden te kijken of er een wijziging te zien is in de data. Bv fractie php sites, fractie sites met […]
Original post on idf.social
idf.social
October 22, 2026 at 10:04 AM
🛠️ Block AI Bots Robots.txt Generator ⚡

Free block AI bots robots.txt generator. Generate rules to block GPTBot, ClaudeBot, CommonCrawl, and AI scrape...

🔗 Free tool: https://allmytools.tech/tools/ai-bot-blocker #tools
October 20, 2026 at 3:33 PM
does anyone sell copies of commoncrawl on ultrum tapes? downloading it takes forever and you probably have to buy more storage to store it anyway (it's on order 100TB)
September 30, 2026 at 12:44 AM
Perso dès que j'ai su j'ai modifié mon robots.txt + case à cocher Cloudflare. Mais bon, Claude en web search viole tjs à ce jour mon robots.txt et mes CGU. CommonCrawl a scrapé mon site pendant des années sans rien me demander (illégal là aussi en droit français). J'ai écrit à CC, ils ont stoppé.
1/
September 15, 2026 at 9:14 PM
Good of Amazon to make themselves liable for AI-LLM copyright abuse through CommonCrawl. ;) aws.amazon.com/blogs/public...
How Common Crawl and AWS Open Data built the foundation for the AI revolution | Amazon Web Services
The Amazon Web Services (AWS) Open Data Sponsorship Program has hosted the Common Crawl open repository of web data at no cost since January 2012. It has become one of the most important sources of tr...
aws.amazon.com
September 14, 2026 at 9:15 AM
TIL the Common Crawl did a sampling of llms.txt in August! Nearly 600K sites!

https://huggingface.co/spaces/commoncrawl/llms.txt-report has info and https://huggingface.co/spaces/commoncrawl/llms.txt-report has some very reasonably sized parquet versions of the WARC files that I may hack on […]
Original post on mastodon.social
mastodon.social
September 5, 2026 at 2:39 AM
80 million videos pulled out of CommonCrawl, captioned for picture and sound. Research use only, and the copyright question sits right underneath.

courionai.com/news/2026-08-30-laion-big-video-dataset

#AI #TrainingData #OpenSource
September 1, 2026 at 9:02 AM
LAION just dropped a 10‑million‑hour video dataset—think massive video‑to‑text training data scraped from CommonCrawl. Perfect fuel for the next wave of video‑language models. Dive in to see what InternVid can unlock! #LAION #VideoDataset #VideoLanguageModels

🔗 aidailypost.com/news/laion-r...
August 29, 2026 at 10:14 AM
Don't mind me, just downloading 1 million random web pages from the CommonCrawl dataset for my next blog post.
August 28, 2026 at 5:08 PM
The second twist is that you don’t even have to provide data: a meta-task of autollm is to detect relevant documents from large corpora (e.g. CommonCrawl). (this twist is yet to be implemented :))
August 28, 2026 at 2:10 PM
[1/6] 🚨 We’re releasing LAION-BVD: a 10-million-hour open video dataset for multimodal pre-training.

- 1.3B video URLs from CommonCrawl
- 80M downloaded videos
- 10M video hours
- 55M captioned clips
- 300M frame-caption pairs

🌐: projects.laion.ai/bvd/

🧵👇
August 26, 2026 at 4:21 PM
[3/6] We built the dataset from CommonCrawl WAT files using a large-scale distributed pipeline:

- Extracted 1.3B safe/platform-specific video URLs
- Downloaded ~80M videos using thousands of distributed workers
- Processed videos into clips, audio segments, and frames
August 26, 2026 at 4:21 PM
[6/6] We hope this helps make large-scale multimodal pre-training more open and accessible.

Huge thanks to grass.io for providing us access to its data infrastructure in support of this release and @hf.co for storage!

🌐: projects.laion.ai/bvd/
📄: arxiv.org/abs/2608.24845
LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training
We present LAION-BVD, a large-scale open video dataset for multimodal learning, which contains 1.3B platform-specific video URLs collected from CommonCrawl. From these, we download 80M videos with...
arxiv.org
August 26, 2026 at 4:21 PM
FYI: News publishers target Common Crawl, the AI training data backdoor #News #Media #CommonCrawl #AITraining #DataPrivacy
News publishers target Common Crawl, the AI training data backdoor
News/Media Alliance sent a formal letter to Common Crawl demanding it stop unauthorized scraping and block AI companies from using news content for training.
ppc.land
May 4, 2026 at 8:32 PM
ICYMI: News publishers target Common Crawl, the AI training data backdoor #AI #NewsMedia #CommonCrawl #DataPrivacy #WebScraping
News publishers target Common Crawl, the AI training data backdoor
News/Media Alliance sent a formal letter to Common Crawl demanding it stop unauthorized scraping and block AI companies from using news content for training.
ppc.land
May 2, 2026 at 8:30 PM
Common Crawl supplies paywalled content to AI companies despite publisher objections #AI #Journalism #DataEthics #Paywall #CommonCrawl
Common Crawl supplies paywalled content to AI companies despite publisher objections
Nonprofit organization Common Crawl provides major AI companies access to millions of paywalled news articles while claiming compliance with publisher removal requests, investigation reveals.
ppc.land
November 5, 2025 at 10:50 AM
FYI: News publishers target Common Crawl, the AI training data backdoor #News #Media #CommonCrawl #AITraining #DataPrivacy
News publishers target Common Crawl, the AI training data backdoor
News/Media Alliance sent a formal letter to Common Crawl demanding it stop unauthorized scraping and block AI companies from using news content for training.
ppc.land
May 4, 2026 at 8:32 PM
ICYMI: News publishers target Common Crawl, the AI training data backdoor #AI #NewsMedia #CommonCrawl #DataPrivacy #WebScraping
News publishers target Common Crawl, the AI training data backdoor
News/Media Alliance sent a formal letter to Common Crawl demanding it stop unauthorized scraping and block AI companies from using news content for training.
ppc.land
May 2, 2026 at 8:30 PM