Blog tag

#python

18 posts tagged with python.

February 15, 20264 min read

From ClawHub to Loom: How We Executed a Seamless Project-Wide Rename in Python

Renaming ClawHub to Loom taught us how to refactor a mature Python codebase without breaking imports, tests, or team momentum.

pythonrefactoringcodebasetestingdeveloper-experience

February 10, 20264 min read

How We Built a Real-Time Fleet Dashboard for Distributed Scraping Workers in GhostGraph

We built a lightweight, real-time dashboard to monitor GhostGraph's distributed scraping workers using FastAPI, Redis Streams, and server-sent events.

FastAPIRedis Streamsreal-time monitoringdistributed systemsweb scrapingPython

February 8, 20263 min read

From Sequential to Parallel: How We Scaled URL Fetching in Our LLM-Powered Crawler

We replaced sequential HTTP fetching with asyncio-powered concurrency—5 requests at a time—and slashed our crawl times by 70%.

asyncioweb scrapingconcurrencyPythonVultr Crawlerperformance

January 31, 20264 min read

How We Fixed Hung Connections in Our Distributed Crawler with Hard Timeout Enforcement

We stopped silent network hangs in our Python crawler by layering signal-based hard timeouts over curl_cffi and adding IP rotation to preserve throughput.

PythonWeb ScrapingDistributed Systemscurl_cffiTimeoutsDebugging

January 30, 20264 min read

How We Scaled a Distributed Crawler with Atomic Redis State Management

How atomic Redis operations fixed state corruption during worker shutdowns in our distributed Vultr Crawler.

redisdistributed-systemsweb-crawlerpythondata-consistency

January 28, 20264 min read

Replacing ARQ with a Unified Redis Streams Worker: Why We Simplified Our Distributed Task System

We replaced ARQ with a lightweight Redis Streams polling worker—cutting 6k+ lines and improving reliability across our scraping fleet.

pythonredisdistributed systemstask queuesarchitecture

January 27, 20263 min read

From Direct Queries to Clean Repositories: Refactoring a Python Scraper’s Database Layer

How I replaced raw Postgres queries with a type-safe repository pattern in a production scraper—improving testability and long-term maintainability.

pythondatabase designweb scrapingclean architecturetesting

January 26, 20264 min read

Decoupling from Legacy Services: Removing Motia Integration in the Vultr Scraper

How we ripped out a brittle third-party integration and replaced it with a unified, maintainable worker model.

pythonrefactoringlegacy-codesystem-designbackend-engineering

January 25, 20264 min read

Migrating from ARQ to Motia: Building a Lightweight, Event-Driven Worker Framework for Scalable Scraping

We replaced ARQ with our custom event-driven framework Motia to gain control, clarity, and reliability in our scraping workflows.

PythonBackground JobsDistributed SystemsWeb ScrapingSystem Design

January 24, 20264 min read

From Chaos to Clarity: How We Unified Our Worker Architecture with ARQ in the Vultr Scraper

We replaced a tangled mess of Python workers with ARQ and Redis, cutting complexity and boosting reliability in our scraping pipeline.

pythonarqredisdistributed-workersweb-scrapingarchitecture

January 23, 20264 min read

Building a Smart Crawler with LLM-Powered Extraction and ARQ Task Orchestration

How I used LLMs and ARQ to build a self-adapting, scalable web scraper that survives real-world site changes.

web scrapingARQLLMdistributed systemsPythondata pipelines

January 21, 20264 min read

Building a Web UI for a Headless Scraping Engine: How We Brought the Vultr Scraper to Life with FastAPI and HTML Templates

We turned a script-driven scraper into a fully observable web interface using FastAPI and server-rendered templates—no frontend framework needed.

FastAPIweb scrapingPythonobservabilitydeveloper tooling