We rebuilt our web crawler to move beyond brute-force scraping—now it learns patterns and adapts in real time.
How atomic Redis operations fixed state corruption during worker shutdowns in our distributed Vultr Crawler.
We moved job claiming in the Vultr Crawler from Redis to Postgres for better consistency, auditability, and operational simplicity.
We replaced ARQ with a lightweight Redis Streams polling worker—cutting 6k+ lines and improving reliability across our scraping fleet.
We replaced a tangled mess of Python workers with ARQ and Redis, cutting complexity and boosting reliability in our scraping pipeline.