How I replaced ClawHub’s monolithic routing node with agent self-determination to improve scalability and reduce coupling.
How we built a requeue endpoint in GhostGraph to revive stalled Redis Stream jobs and maintain pipeline integrity.
We built a lightweight, real-time dashboard to monitor GhostGraph's distributed scraping workers using FastAPI, Redis Streams, and server-sent events.
How we built stateful, token-efficient browser agents in Vultr Crawler using session APIs, DOM distillation, and autonomous action loops.
We rebuilt our web crawler to move beyond brute-force scraping—now it learns patterns and adapts in real time.
We stopped silent network hangs in our Python crawler by layering signal-based hard timeouts over curl_cffi and adding IP rotation to preserve throughput.
How atomic Redis operations fixed state corruption during worker shutdowns in our distributed Vultr Crawler.
We moved job claiming in the Vultr Crawler from Redis to Postgres for better consistency, auditability, and operational simplicity.
We replaced ARQ with a lightweight Redis Streams polling worker—cutting 6k+ lines and improving reliability across our scraping fleet.
We replaced ARQ with our custom event-driven framework Motia to gain control, clarity, and reliability in our scraping workflows.
How I used LLMs and ARQ to build a self-adapting, scalable web scraper that survives real-world site changes.