Blog tag

#web crawling

2 posts tagged with web crawling.

← Back to all posts
4 min read

How We Scaled Our Web Crawler to 50 Instances Using Camoufox and Fleet Orchestration

We replaced Chrome with Camoufox, added IPv6 rotation, and scaled to 50 Vultr instances—here’s how we stayed stealthy and stable.

web crawlingCamoufoxIPv6 rotationbrowser automationVultrscaling
Read more
4 min read

Building a Smarter Web Crawler: How We Implemented Two-Phase Intelligent Exploration in Vultr Crawler

We rebuilt our web crawler to move beyond brute-force scraping—now it learns patterns and adapts in real time.

web crawlingPlaywrightRedispattern recognitiondistributed systems
Read more