Overview
BelegPilot is a production API that turns uploaded receipts and invoices into structured JSON that downstream accounting systems can process reliably. Document extraction with multimodal models is easy to demo and hard to operate — the engineering emphasis here was on the operating part: observability, deployment repeatability, and independence from any single model provider.
Extraction Pipeline
flowchart LR up["File upload"] --> api["FastAPI service"] api --> vlm["Multimodal extraction<br/>(VLM via LiteLLM)"] vlm --> val["Validation +<br/>normalization"] val --> out["Structured JSON"]
An uploaded document flows through a vision-language model that extracts the fields, followed by validation and normalization so that what leaves the API conforms to a stable schema — regardless of which model produced the raw extraction.
Provider Abstraction
Model providers change pricing, deprecate models, and have outages. BelegPilot routes all model calls through LiteLLM, so the extraction layer is written once against a single interface and the underlying provider can be switched by configuration — a resilience and cost lever, not a rewrite.
Observability
Every request is traced end to end with OpenTelemetry — from upload through model call to structured response. When an extraction goes wrong in production, the trace shows where: the model call, the validation step, or the input itself. This is the difference between debugging production and guessing at it.
Deployment
The service ships as a Docker container through an AWS deployment pipeline, keeping delivery repeatable from local development to production.