How I replaced ClawHub’s monolithic routing node with agent self-determination to improve scalability and reduce coupling.
How I built a requeue endpoint in GhostGraph to revive stalled Redis Stream jobs and maintain pipeline integrity.
I built a lightweight, real-time dashboard to monitor GhostGraph's distributed scraping workers using FastAPI, Redis Streams, and server-sent events.
How I built stateful, token-efficient browser agents in Vultr Crawler using session APIs, DOM distillation, and autonomous action loops.
I rebuilt my web crawler to move beyond brute-force scraping—now it learns patterns and adapts in real time.
I stopped silent network hangs in my Python crawler by layering signal-based hard timeouts over curl_cffi and adding IP rotation to preserve throughput.
How atomic Redis operations fixed state corruption during worker shutdowns in my distributed Vultr Crawler.
I moved job claiming in the Vultr Crawler from Redis to Postgres for better consistency, auditability, and operational simplicity.
I replaced ARQ with a lightweight Redis Streams polling worker—cutting 6k+ lines and improving reliability across my scraping fleet.
I replaced ARQ with my custom event-driven framework Motia to gain control, clarity, and reliability in my scraping workflows.
How I used LLMs and ARQ to build a self-adapting, scalable web scraper that survives real-world site changes.