Zero to Pipeline

Situation

Data engineers spend hours writing the same boilerplate for every new API source. Reading docs, implementing auth, handling pagination, writing retry logic. All of that before a single byte of real data moves. The problem is not skill; the work is repetitive enough to be automatable but granular enough that most teams never prioritise it.

Task

Build a framework that removes connector-writing from the data pipeline setup process entirely, using an LLM to do the discovery work that engineers currently do by hand. Publish it so anyone can use it with a single pip install.

Action

• pipeline source add <provider> sends the provider name to an LLM (OpenAI, Anthropic, Ollama, Azure, or Groq) which probes endpoints, infers auth type (OAuth, API key, none), identifies pagination strategy (cursor, offset, page_token), and writes the config.
• pipeline sync run <provider> extracts data with automatic cursor checkpointing, self-healing on auth failures, and parallelisable DAG execution. The second sync fetches only new records using the saved cursor.
• Credentials stored securely via keyring. Works with any LLM provider, swappable via env var.
• Demonstrated at EuroPython 2026 with a live fraud-detection DAG: 15,000 synthetic transactions, Feast feature views, MLflow, Prometheus, and a Linear issue, orchestrated end-to-end in 4.4 seconds.

Result

• Published on PyPI. Used by EuroPython 2026 conference attendees as a starting point for production data pipelines.
• Reduces API connector setup from hours or days to under 5 minutes for any REST or GraphQL API.
• Talk delivered at Europe's largest Python conference (EuroPython 2026, Krakow) and PyData Helsinki.