The challenge
An ingestion endpoint is only one part of the system. Applications also need clear event contracts, usable clients, and a path from accepted events to database writes.
I built IngestKit to connect those pieces. The schema should carry through the pipeline, so the API, generated clients, and storage models share an explicit contract.
The approach
Define events in a schema, generate models and clients, buffer the stream through Redpanda, and let consumers write batches through PostgreSQL COPY.
The streaming buffer separates accepting events from writing them to storage. Batch writes reduce the overhead of individual inserts, while consumers own their processing and retry behavior.
Event schema
→ generated Python / TypeScript clients
→ Go ingestion API
→ Redpanda
→ batch consumer
→ PostgreSQL COPYDeveloper experience
The schema becomes the common contract across the pipeline. Python and TypeScript clients make the application-facing workflow explicit, while generated storage code carries the same event definitions into the database.
For me, developer experience is part of the system design: an ingestion pipeline is easier to use when the event contract and the path to storage are understandable together.
Performance in context
The repository includes load-testing documentation. HTTP request throughput and consumer processing throughput measure different parts of the pipeline, and results depend on the workload, environment, batching, and acknowledgement boundary.
Read the methodology alongside any benchmark number. The results have not been independently reproduced for this portfolio, so I keep the project focused on the architecture and working source.