Blog tag

#Redis

5 posts tagged with Redis.

← Back to all posts
4 min read

Building a Smarter Web Crawler: How I Implemented Two-Phase Intelligent Exploration in Vultr Crawler

I rebuilt my web crawler to move beyond brute-force scraping—now it learns patterns and adapts in real time.

web crawlingPlaywrightRedispattern recognitiondistributed systems
Read more
4 min read

How I Scaled a Distributed Crawler with Atomic Redis State Management

How atomic Redis operations fixed state corruption during worker shutdowns in my distributed Vultr Crawler.

redisdistributed-systemsweb-crawlerpythondata-consistency
Read more
4 min read

Migrating Job State Management from Redis to Postgres: Why I Centralized Crawler Jobs in a Single Source of Truth

I moved job claiming in the Vultr Crawler from Redis to Postgres for better consistency, auditability, and operational simplicity.

distributed systemsPostgresRedisjob queuescrawler architecturedata consistency
Read more
4 min read

Replacing ARQ with a Unified Redis Streams Worker: Why I Simplified My Distributed Task System

I replaced ARQ with a lightweight Redis Streams polling worker—cutting 6k+ lines and improving reliability across my scraping fleet.

pythonredisdistributed systemstask queuesarchitecture
Read more
4 min read

From Chaos to Clarity: How I Unified My Worker Architecture with ARQ in the Vultr Scraper

I replaced a tangled mess of Python workers with ARQ and Redis, cutting complexity and boosting reliability in my scraping pipeline.

pythonarqredisdistributed-workersweb-scrapingarchitecture
Read more