We built a lightweight, real-time dashboard to monitor GhostGraph's distributed scraping workers using FastAPI, Redis Streams, and server-sent events.
We replaced sequential HTTP fetching with asyncio-powered concurrency—5 requests at a time—and slashed our crawl times by 70%.
How we built stateful, token-efficient browser agents in Vultr Crawler using session APIs, DOM distillation, and autonomous action loops.
We stopped silent network hangs in our Python crawler by layering signal-based hard timeouts over curl_cffi and adding IP rotation to preserve throughput.
How I replaced raw Postgres queries with a type-safe repository pattern in a production scraper—improving testability and long-term maintainability.
We replaced ARQ with our custom event-driven framework Motia to gain control, clarity, and reliability in our scraping workflows.
We replaced a tangled mess of Python workers with ARQ and Redis, cutting complexity and boosting reliability in our scraping pipeline.
How I used LLMs and ARQ to build a self-adapting, scalable web scraper that survives real-world site changes.
We architected a real-time site discovery engine for the Vultr Scraper in under 24 hours—here's how modular design and smart routing made it possible.
We turned a script-driven scraper into a fully observable web interface using FastAPI and server-rendered templates—no frontend framework needed.