Curious whether the SF run leans infra or agent-side. Parsing is the easy half — block rates and proxy spend decide whether a scraper ships. Is the agenda more Crawlee/Playwright plumbing, or tool-calling and MCP servers?
October 25, 2026 at 9:06 PM
Curious whether the SF run leans infra or agent-side. Parsing is the easy half — block rates and proxy spend decide whether a scraper ships. Is the agenda more Crawlee/Playwright plumbing, or tool-calling and MCP servers?
Scrapers that break on JS-heavy pages, get blocked, or lose progress when a run crashes? Crawlee for Python from gives you one API for HTTP and headless browser crawling. An adaptive crawler decides when JS rendering is needed, with proxy rotation, retries, and a pausable request queue. It's...
October 23, 2026 at 1:17 AM
Scrapers that break on JS-heavy pages, get blocked, or lose progress when a run crashes? Crawlee for Python from gives you one API for HTTP and headless browser crawling. An adaptive crawler decides when JS rendering is needed, with proxy rotation, retries, and a pausable request queue. It's...
crawlee — Node.js web scraping and browser automation library for building reliable crawlers with HTTP and headless browser support, proxy rotation, and human-like fingerprinting. https://ktp.sh/sdhvxiBOPJ
October 7, 2026 at 11:33 AM
crawlee — Node.js web scraping and browser automation library for building reliable crawlers with HTTP and headless browser support, proxy rotation, and human-like fingerprinting. https://ktp.sh/sdhvxiBOPJ
Updates on CRAN: ardlverse (2.0.3), bclogit (1.1.1), caustests (1.1.4), ClinSigMeasures (1.3), coin (1.4-6), confoundvis (0.2.0), couplr (1.8.0), crawlee (0.1.1), deltabreedquery (1.0.4), easybgm (0.5.1), enrichwith (0.6), fabricQueryR (1.0.1), fbardl (1.1.0)
October 1, 2026 at 6:52 AM
Updates on CRAN: ardlverse (2.0.3), bclogit (1.1.1), caustests (1.1.4), ClinSigMeasures (1.3), coin (1.4-6), confoundvis (0.2.0), couplr (1.8.0), crawlee (0.1.1), deltabreedquery (1.0.4), easybgm (0.5.1), enrichwith (0.6), fabricQueryR (1.0.1), fbardl (1.1.0)
🚀 GitHub Intelligence Drop
Airtest (Python • 🟡)
⭐ 9.6K → https://github.com/AirtestProject/Airtest
crawlee-python (Python • 🟢)
⭐ 9.6K → https://github.com/apify/crawlee-python
⚡ Only signal. Zero noise.
Airtest (Python • 🟡)
⭐ 9.6K → https://github.com/AirtestProject/Airtest
crawlee-python (Python • 🟢)
⭐ 9.6K → https://github.com/apify/crawlee-python
⚡ Only signal. Zero noise.
September 30, 2026 at 9:43 PM
🚀 GitHub Intelligence Drop
Airtest (Python • 🟡)
⭐ 9.6K → https://github.com/AirtestProject/Airtest
crawlee-python (Python • 🟢)
⭐ 9.6K → https://github.com/apify/crawlee-python
⚡ Only signal. Zero noise.
Airtest (Python • 🟡)
⭐ 9.6K → https://github.com/AirtestProject/Airtest
crawlee-python (Python • 🟢)
⭐ 9.6K → https://github.com/apify/crawlee-python
⚡ Only signal. Zero noise.
CRAN updates: ardlverse caustests confoundvis couplr crawlee easybgm enrichwith fbardl fqardl gcf huito inti qadf serad staat1cho taxify underdisp vrpr WeightIt xtbreakcoint xtrec #rstats
September 30, 2026 at 10:02 PM
CRAN updates: ardlverse caustests confoundvis couplr crawlee easybgm enrichwith fbardl fqardl gcf huito inti qadf serad staat1cho taxify underdisp vrpr WeightIt xtbreakcoint xtrec #rstats
今後の汎用性のためにcrawleeに差し替えました。チュートリアルに少し手間取りましたが、お作法が分かりやすいためこちらの方が楽でした。知見を得ることができたので良しとします。
(SPAでなければcurl+pupだけで済むんですが、そこはそれ)
(SPAでなければcurl+pupだけで済むんですが、そこはそれ)
September 22, 2026 at 3:08 PM
今後の汎用性のためにcrawleeに差し替えました。チュートリアルに少し手間取りましたが、お作法が分かりやすいためこちらの方が楽でした。知見を得ることができたので良しとします。
(SPAでなければcurl+pupだけで済むんですが、そこはそれ)
(SPAでなければcurl+pupだけで済むんですが、そこはそれ)
Mistype, spider-man needs a 56, still hard af. Silver Surfer (higher Agility) requires a 51. Night crawlee has the sane Agility as Spider-man. These 3 have tge highest Agility in the core book and have less than a 50/50 shot of changing their declared action after initiative
August 19, 2026 at 6:59 PM
Mistype, spider-man needs a 56, still hard af. Silver Surfer (higher Agility) requires a 51. Night crawlee has the sane Agility as Spider-man. These 3 have tge highest Agility in the core book and have less than a 50/50 shot of changing their declared action after initiative
It's a 🕸️🕷️-focused Drop, today, with two tools (well, one tool and one tool/library/ecosystem) for slurping content up (responsibly!), and one TUI for helping craft perfect APIs
![<p><em>Scout; posting; Crawlee</em></p>
<p>Two scraping and one API-ing resources to help you wrangle HTTP ops more efficiently.</p>
<hr class="wp-block-separator has-alpha-channel-opacity is-style-dots" />
<div class="wp-block-jetpack-subscriptions__supports-newline wp-block-jetpack-subscriptions">
<div>
<div>
<div>
<p>
<a href="https://dailydrop.hrbrmstr.dev/?post_type=post&p=139503501" style="font-size: 16px; padding: 15px 23px 15px 23px; margin: 0; margin-left: 10px; border-radius: 0px; border-width: 1px; background-color: #113AF5; color: #FFFFFF; text-decoration: none; white-space: nowrap; margin-left: 0;">Subscribe</a>
</p>
</div>
</div>
</div>
</div>
<h3 class="wp-block-heading" id="tldr">TL;DR</h3>
<p><em>(This is an AI-generated summary of today’s Drop using Sonnet via Perplexity.)</em></p>
<ul class="wp-block-list">
<li>Scout (<a href="https://scout-lang.netlify.app/">https://scout-lang.netlify.app/</a>): A domain-specific language for web scraping and crawling, written in Rust. It offers a simple syntax and coder-friendly features for both basic and complex web scraping tasks.</li>
<li>Posting (<a href="https://github.com/darrenburns/posting">https://github.com/darrenburns/posting</a>): A modern, terminal-based HTTP client that brings Postman-like functionality to the command line. It allows organizing and storing HTTP request collections in a git-friendly format.</li>
<li>Crawlee (<a href="https://crawlee.dev/">https://crawlee.dev/</a>): A web scraping and browser automation library for Node.js, offering a unified API for HTTP and browser-based crawling. It includes features like proxy rotation, session management, and support for various crawling methods.</li>
</ul>
<hr class="wp-block-separator has-alpha-channel-opacity is-style-dots" />
<h3 class="wp-block-heading" id="scout">Scout</h3>
<figure class="wp-block-image size-large"><img alt="" class="wp-image-139503502" height="832" src="https://dailydrop.hrbrmstr.dev/wp-content/uploads/2024/07/image-17.png?w=1024" tabindex="0" width="1024" /></figure>
<p><a href="https://scout-lang.netlify.app/">Scout</a> (<a href="https://github.com/maxmindlin/scout-lang">GH</a>) is a domain-specific language (DSL) designed specifically for web scraping and crawling tasks. Written in Rust, Scout combines a simple, easy-to-learn syntax with coder-friendly web crawling capabilities. It aims to lower the barrier of entry for web scraping while providing advanced features for more complex scenarios. Please note that you need to have icky Firefox installed along with <a href="https://github.com/mozilla/geckodriver"><code>geckodriver</code></a> before trying to use Scout, as it performs all web-ops through that browser.</p>
<p>The language spec <a href="https://scout-lang.netlify.app/docs/getting-started/fundamentals/">fits on a single web page</a> (which means nothing — since one page could scroll forever — but the main <code>div</code> of that page has ~200 lines and ~6K characters), and the operations are all fairly straightforward. For example, take this sample REPL session (just running <code>scout</code> in the terminal starts a REPL):</p>
<div class="wp-block-syntaxhighlighter-code "><pre class="brush: jscript; title: ; notranslate">
>> goto "https://example.com"
Null
>> h1 = $"h1"
Null
>> h1 |> textContent()
"Example Domain"
</pre></div>
<p>That’s great, but we likely want to web scrape to get some data back that we can use. For that, Scout has a <code>results</code> JSON output structure which gets populated via <code>scrape</code> operations. Let’s grab all the links and titles from the main Hacker News site (NOTE: use their API if doing IRL “scraping” of their site):</p>
<div class="wp-block-syntaxhighlighter-code "><pre class="brush: jscript; title: ; notranslate">
goto "https://news.ycombinator.com/"
items = $$"tr.athing > td.title > span.titleline"
for item in items do
scrape {
text: item |> textContent(),
link: $(item)"a" |> href(),
}
end
</pre></div>
<p>That will spit back something like:</p>
<div class="wp-block-syntaxhighlighter-code "><pre class="brush: jscript; title: ; notranslate">
{
"results": {
"https://news.ycombinator.com/": [
{
"link": "https://www.burn-heart.com/rulers-of-the-ancient-world",
"text": "Rulers of the Ancient World: period correct measuring tools (burn-heart.com)"
},
{
"link": "https://typesetinthefuture.com/2018/12/04/walle/",
"text": "The Typeset of Wall·E (2018) (typesetinthefuture.com)"
},
{
"link": "https://matttproud.com/blog/posts/x-window-system-boot-stipple.html",
"text": "Iconography of the X Window System: The Boot Stipple (matttproud.com)"
},
{
"link": "https://payloadspace.com/habitable-worlds-observatory-and-the-future-of-space-telescopes-in-the-era-of-heavy-lift-launch/",
"text": "Future of Space Telescopes in the Era of Super Heavy Lift Launch (payloadspace.com)"
},
…
]
}
}
</pre></div>
<p>In debug mode, you can see the Firefox instrumentation (an example of that is in the section header). It also groks proxies and such, so if you’re behind one, you can still use it.</p>
<p>I really like the language spec, so far, and learning the language has been very straightforward.</p>
<p>Cannot wait to see how this evolves, and I’ll see if I can put together a larger, more real-life example over the coming weeks.</p>
<h3 class="wp-block-heading" id="posting">posting</h3>
<figure class="wp-block-image size-large"><img alt="" class="wp-image-139503504" height="616" src="https://dailydrop.hrbrmstr.dev/wp-content/uploads/2024/07/image-18.png?w=1024" tabindex="0" width="1024" /></figure>
<p><a href="https://github.com/darrenburns/posting">Posting</a> is a modern, terminal-based HTTP client that brings <a href="https://www.postman.com/">Postman</a>-like functionality to the the terminal/CLI. It enables us to send and test HTTP requests directly from terminal, which means you can also use it over SSH (perhaps, from a different system that has better access to the resources in question).</p>
<p>The tools let us organize and store collections of HTTP requests in a git-friendly format, using (ugh) YAML files with a <code>.posting.yaml</code> extension.</p>
<p>As one might expect, it fully supports the use of environment variables and <code>.env</code> files, so we don’t have to hard-code different values for, say, dev/test/prod.</p>
<p>Since it’s a modern TUI, it fully supports efficient navigation using both keyboard shortcuts and mouse input, including a “jump mode” for quick access to different parts of the interface. It’s also fully theme-able (including the layout), and lets us customize the response formatting as well. Said modernity is also shown via a built-in command palette, so you’re a quick keystroke-or-two away from every command.</p>
<p>I’ve stolen one of their screenshots for the section header, but <em>please</em> head to the GH repo, as the there’s tons more info, and some animated screens that show off some of the features way better than any more blathering I could do.</p>
<h3 class="wp-block-heading" id="crawlee">Crawlee</h3>
<figure class="wp-block-image size-full"><img alt="" class="wp-image-139503506" height="1254" src="https://dailydrop.hrbrmstr.dev/wp-content/uploads/2024/07/spider-web-with-water-beads-network-dewdrop.jpg" tabindex="0" width="1880" /><figcaption class="wp-element-caption">Photo by Pixabay on <a href="https://www.pexels.com/photo/spider-web-34225/" rel="nofollow">Pexels.com</a></figcaption></figure>
<p><em>(Given that it sure feels like they’re suggesting stealing content for use in AI training is OK, I almost did not include this, today, but I know Drop readers will use services/tools responsibly.)</em></p>
<p><a href="https://crawlee.dev/">Crawlee</a> (<a href="https://github.com/apify/crawlee">GH</a>) is a “<em>web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.”</em></p>
<p>The tool/library has several key features that make it a pretty compelling tool for web crawling. It offers a unified API that caters to both HTTP and browser-based crawling, which gives us plenty of flexibility, especially when a given website thinks it is too clever by half. The platform automatically scales based on system resources, and also integrates proxy rotation and session management, which helps keep you enhancing anonymous and reduces the likelihood of IP bans. <em>(Again, please consider using this tool responsibly!)</em></p>
<p>It also has a persistent queue system, which helps manage URLs effectively. Configurable storage options are also available for storing scraped data. And, Crawlee comes with built-in error handling and retry mechanisms, ensuring robustness and reliability during the crawling process.</p>
<p>Crawlee comes with various crawling methods to suit different needs. The “CheerioCrawler” provides fast HTTP-based crawling with Cheerio for HTML parsing, making it ideal for lightweight and speedy tasks. For more complex requirements, the “PuppeteerCrawler” utilizes headless Chrome automation via Puppeteer, offering extensive control over the browsing environment. The “PlaywrightCrawler” supports multi-browser automation with Playwright, compatible with Chrome, Firefox, and WebKit.</p>
<p>By abstracting away common scraping challenges, this tool/library lets us focus on extracting and processing data rather than dealing with low-level crawling logistics. It seems super-flexibile, and the feature set looks fairly comprehensive.</p>
<p>While I do my best to try things out before adding them to the Drop, this one is on the “TODO” list. If you kick the tyres before I do, def lemme know how you liked it!</p>
<h3 class="wp-block-heading" id="fin">FIN</h3>
<p>Remember, you can follow and interact with the full text of The Daily Drop’s free posts on Mastodon via <code>@dailydrop.hrbrmstr.dev@dailydrop.hrbrmstr.dev</code> <img alt="☮" class="wp-smiley" src="https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/262e.png" style="height: 1em;" /></p>](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:zfczd4u7geyqchvyjb2kmmnk/bafkreihaoglkyklwlmzwwixh5vfr7ku22lpkgek3ebeyvbq54zjshbfc3a)
July 15, 2024 at 12:37 PM
It's a 🕸️🕷️-focused Drop, today, with two tools (well, one tool and one tool/library/ecosystem) for slurping content up (responsibly!), and one TUI for helping craft perfect APIs
Complete success today! Got all the data I need and did t have to enter thousands of database entries by hand. Crawlee FTW!
February 14, 2025 at 11:00 PM
Complete success today! Got all the data I need and did t have to enter thousands of database entries by hand. Crawlee FTW!
Doing some web scraping tests with Crawlee today after finding it on a top 10 list somewhere randomly. It’s very straight forward and intuitive to get started. Better watch your websites because…
“ALL YOUR DATA ARE BELONG TO US”!!!
crawlee.dev
“ALL YOUR DATA ARE BELONG TO US”!!!
crawlee.dev
Crawlee · Build reliable crawlers. Fast.
Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.
crawlee.dev
February 14, 2025 at 5:20 PM
Doing some web scraping tests with Crawlee today after finding it on a top 10 list somewhere randomly. It’s very straight forward and intuitive to get started. Better watch your websites because…
“ALL YOUR DATA ARE BELONG TO US”!!!
crawlee.dev
“ALL YOUR DATA ARE BELONG TO US”!!!
crawlee.dev
How to scrape Google search results with Python
From the "DEV Community" RSS feed
From the "DEV Community" RSS feed
How to scrape Google search results with Python
Learn how to scrape google search results using Crawlee for Python
dev.to
December 3, 2024 at 2:01 AM
How to scrape Google search results with Python
From the "DEV Community" RSS feed
From the "DEV Community" RSS feed
10 web scraping challenges (+ solutions) in 2025 How to solve problems you’re likely to encount...
https://blog.apify.com/web-scraping-challenges/
#Web #scraping #Crawlee #Antiscraping #Captcha
Event Attributes
https://blog.apify.com/web-scraping-challenges/
#Web #scraping #Crawlee #Antiscraping #Captcha
Event Attributes
January 6, 2025 at 10:27 AM
10 web scraping challenges (+ solutions) in 2025 How to solve problems you’re likely to encount...
https://blog.apify.com/web-scraping-challenges/
#Web #scraping #Crawlee #Antiscraping #Captcha
Event Attributes
https://blog.apify.com/web-scraping-challenges/
#Web #scraping #Crawlee #Antiscraping #Captcha
Event Attributes
11 best open-source web crawlers and scrapers in 2025 Free software libraries, packages, and SDKs...
https://blog.apify.com/top-11-open-source-web-crawlers-and-one-powerful-web-scraper/
#Web #crawling #Web #Crawlee #Web #scraping
Event Attributes
https://blog.apify.com/top-11-open-source-web-crawlers-and-one-powerful-web-scraper/
#Web #crawling #Web #Crawlee #Web #scraping
Event Attributes
January 6, 2025 at 8:28 AM
11 best open-source web crawlers and scrapers in 2025 Free software libraries, packages, and SDKs...
https://blog.apify.com/top-11-open-source-web-crawlers-and-one-powerful-web-scraper/
#Web #crawling #Web #Crawlee #Web #scraping
Event Attributes
https://blog.apify.com/top-11-open-source-web-crawlers-and-one-powerful-web-scraper/
#Web #crawling #Web #Crawlee #Web #scraping
Event Attributes
Best GitHub repos to scrape any site without getting blocked:
1. Crawl4AI
github.com/firecrawl/fi...
2. Firecrawl
github.com/firecrawl/fi...
3. Scrapy
github.com/scrapy/scrapy
4. Crawlee
github.com/apify/crawlee
5. Playwright
github.com/microsoft/pl...
1. Crawl4AI
github.com/firecrawl/fi...
2. Firecrawl
github.com/firecrawl/fi...
3. Scrapy
github.com/scrapy/scrapy
4. Crawlee
github.com/apify/crawlee
5. Playwright
github.com/microsoft/pl...
May 26, 2026 at 10:29 AM
Best GitHub repos to scrape any site without getting blocked:
1. Crawl4AI
github.com/firecrawl/fi...
2. Firecrawl
github.com/firecrawl/fi...
3. Scrapy
github.com/scrapy/scrapy
4. Crawlee
github.com/apify/crawlee
5. Playwright
github.com/microsoft/pl...
1. Crawl4AI
github.com/firecrawl/fi...
2. Firecrawl
github.com/firecrawl/fi...
3. Scrapy
github.com/scrapy/scrapy
4. Crawlee
github.com/apify/crawlee
5. Playwright
github.com/microsoft/pl...
The Clibbits web stack:
- TanStack Start (TypeScript)
- TailwindCSS + Daisy UI
- Supabase
- Crawlee for scraping
- Gemini 2.0-flash for cleanup
- Amplitude analytics
Plus an Astro blog at a subdomain 🛠️
- TanStack Start (TypeScript)
- TailwindCSS + Daisy UI
- Supabase
- Crawlee for scraping
- Gemini 2.0-flash for cleanup
- Amplitude analytics
Plus an Astro blog at a subdomain 🛠️
March 31, 2025 at 12:20 PM
The Clibbits web stack:
- TanStack Start (TypeScript)
- TailwindCSS + Daisy UI
- Supabase
- Crawlee for scraping
- Gemini 2.0-flash for cleanup
- Amplitude analytics
Plus an Astro blog at a subdomain 🛠️
- TanStack Start (TypeScript)
- TailwindCSS + Daisy UI
- Supabase
- Crawlee for scraping
- Gemini 2.0-flash for cleanup
- Amplitude analytics
Plus an Astro blog at a subdomain 🛠️
yea mine dadde ree low caytes crawlee fings bak owt doars. wiv an wet wiv, him sez. ❤️
October 27, 2025 at 2:20 PM
yea mine dadde ree low caytes crawlee fings bak owt doars. wiv an wet wiv, him sez. ❤️
June 12, 2026 at 9:32 PM
April 30, 2026 at 1:00 AM
January 26, 2026 at 12:52 AM
January 22, 2026 at 8:47 AM
August 22, 2025 at 9:11 PM
web scraping and browser automation with Crawlee
August 22, 2025 at 9:11 PM
web scraping and browser automation with Crawlee