Blog tag

#data-pipelines

7 posts tagged with data-pipelines.

← Back to all posts
2 min read

Candle Continuity, Local-First Market Data, and Deploy Health

A 32-commit day making candle-service fast and continuous while moving dex-trader to local-first data.

FastAPISQLitePerformanceTradingData Pipelines
Read more
2 min read

Serving API Docs, Serializing Tick Writes, and Starting a Backfiller

Docs in the image, write serialization for tick ingest, dynamic pricing scaffolding, and a Solana DEX backfiller.

FastAPIData PipelinesTradingSolanaDocumentation
Read more
2 min read

Guarding Against Stale Candles and Freeing Up Data Sources

Stopping paper trades on stale data, uncapping freshness so pools aren't starved, and broadening the source pool.

TradingData PipelinesDebuggingMarket Data
Read more
3 min read

A Twelve-Phase Merge, a Role Overhaul, and Fresh Training Data

Bringing two dex-trader branches together phase by phase while overhauling roles and wiring a daily training-data pipeline.

TypeScriptLaravelArchitectureData PipelinesTrading
Read more
4 min read

Building a Smart Crawler with LLM-Powered Extraction and ARQ Task Orchestration

How I used LLMs and ARQ to build a self-adapting, scalable web scraper that survives real-world site changes.

web scrapingARQLLMdistributed systemsPythondata pipelines
Read more
4 min read

How I Scaled Git Context’s Analysis Pipeline with Batching, Caching, and Dependency Fixes

I rebuilt Git Context’s analysis pipeline to handle large repos with batching, caching, and smarter dependency management.

gitperformancecachingbatchingdata-pipelines
Read more
4 min read

How I Unified Entity Management in HomeForged’s Schema Pipeline

I refactored HomeForged's schema analysis system to unify entity handling—here's how pure analyzers and canonical structures changed the game.

schema designdata pipelinesrefactoringTypeScriptentity management
Read more