Build a real-time event pipeline for millions of events
![[object Object]](https://assets.ovhcloud.com/1fidFy3Lrybpn2pjt5ktm6LF6_sMDEyMjYxNDEzMDQzNmI5/e9e1a795-d727-4c58-8eb2-9574a5bceb70_ima_161ad419.png)
The two bottlenecks of a real-time event pipeline
Without a durable, streaming backbone, distributed data producers and data consumers become tightly coupled. Producers are the applications that generate and send events into Kafka topics, while consumers are the services that read those events. Producers write directly to queues, databases, or custom services, and when a consumer goes down or lags, the producer either blocks or starts dropping events.
Furthermore, PostgreSQL and similar OLTP Public Cloud databases are designed for row-level operations: inserts, updates, deletes. They are not designed to scan billions of rows, aggregate across multiple dimensions, or update dashboards every minute. As event volume grows, queries that used to return in 200 ms start taking seconds, then tens of seconds, then minutes. The system becomes a bottleneck for product decisions.
![[object Object]](https://assets.ovhcloud.com/1fiz7C746p58ztknphdL8LvwA_sMzA4YzI3MGZjZTg2ZGI0/c896bdad-54c9-4280-bac0-2f723311d064_ima_fb6e21f3.png)
Operate with confidence at enterprise scale
A reference architecture: Apache Kafka + ClickHouse
Apache Kafka as the streaming backbone
Kafka acts as the central nervous system for events. Producers write events to topics, which are logically ordered and partitioned across brokers. Each partition can be consumed independently, which is how Kafka scales throughput horizontally. Consumer groups read from these partitions, and each consumer in a group is responsible for a subset of the partitions.
Partitions are the key to throughput. A topic with more partitions can handle more writes and more consumers in parallel, up to the number of partitions. Consumer groups allow replay. Schemas for applications are typically managed separately, often with a Schema Registry, so producers and consumers can evolve without breaking each other. Events are kept for a configurable retention period, gives you a window to replay data without having to reprocess from the source. This durability and continuous decoupling is what makes Kafka a reliable backbone for high-volume event streams.
ClickHouse as the real-time OLAP layer
ClickHouse is a columnar, distributed database designed for analytical workloads. It stores data by column rather than by row, which means it can skip irrelevant columns during queries and compress data heavily. Combined with its storage engine, which is optimized for sequential reads and predicate pushdown, this allows ClickHouse to scan billions of rows quickly and return aggregated results in milliseconds. New events can be ingested in real time, and ClickHouse can pre-aggregate them using materialized views that update as data arrives. This means continuous dashboards and alerts, without waiting for a batch job or ETL window. The database is designed to scale horizontally, adding nodes to increase storage and compute capacity as the event volume grows.
Object Storage as a tiered cold-data tier for old events
Not all data needs to live in hot storage keeping every event in ClickHouse on local disks becomes expensive and unnecessary. The common pattern is to tier older data into OVHcloud Object Storage (S3-compatible)* storage, while keeping recent data in ClickHouse for fast queries.Events older than a configured threshold and queried on demand when deeper historical analysis is needed. This approach reduces the cost of long-term retention while preserving the ability to run queries across the full dataset.
For many real processing use cases, the hot layer (ClickHouse) holds days or weeks of data for dashboards and real-time alerts, while the cold layer (OVHcloud Object Storage) holds months or years of history for compliance, audits, and periodic deep dives. This separation keeps the operational cost predictable and allows the pipeline to scale without hitting a hard app storage ceiling.
The native Kafka engine integration in ClickHouse
ClickHouse has a built-in Kafka table engine that lets you subscribe directly to Kafka topics. You define a table with the Kafka engine, specify the topic name, broker list, and consumer group, and ClickHouse starts consuming events from Kafka as if they were rows in a regular table. This is a direct, native integration that does not require additional middleware.The Kafka engine in ClickHouse is designed for streaming ingestion. It reads records from the topic, parses them according to the format you specify (JSON, Avro, Protobuf, etc.), and surfaces them as rows that can be queried or inserted into other tables.

Shape incoming events for faster decisions
Pre-aggregate recurring measures
Materialized views continuously calculate totals, rates and grouped metrics as events arrive, reducing repeated work at query time.
Align tables with access patterns
Order data by time and commonly filtered dimensions so investigations read fewer irrelevant records.
Read only useful fields
Column selection limits scanning to the attributes required by each dashboard, alert or exploratory query.
Design retention around demand
Separate frequently queried records from less active data. Retention-aware tables help sustain sub-second analytics for operational views while keeping investigations practical as histories grow.
The cost of waiting
![[object Object]](https://assets.ovhcloud.com/1fitwjqvnd7nXjkFXSbtuwmqp_sYmE2ZDhhMjUyMWIzOWIw/fed77cc2-9906-43a4-a0c9-caeed38870f7_ima_cc88d033.png)
Assign each layer a clear responsibility
Streaming backbone
Buffer, order and replay high-volume event flows so producers remain independent from processing and analytical workloads.
Analytical serving
Expose fresh events to rapid operational queries, dashboards and investigations while maintaining a dedicated environment for analytical demand.
Cold-data tier
Retain older history outside performance-focused storage. This separation supports long-term access without keeping every event in the hot analytical path. The common pattern is to tier older data into OVHcloud Object Storage
Managed operations & delivery
OVHcloud manages replication, backups and monitoring across the service architecture. Teams can deploy resources through the OVHcloud Control Panel or define repeatable environments with Terraform.
Deploy a real-time event pipeline in minutes
For teams that want help sizing the stack, OVHcloud offers direct access to data infrastructure specialists, including for Managed Databases for PostgreSQL , who can assess your current event volumes and query patterns and provide a cost comparison against your existing stack.Whether you are a Data Engineer who needs billion-row real-time results with minimal infrastructure time, or a Head of Data who wants reduced ops load and predictable cloud costs, the managed stack removes the operational overhead while keeping the architecture portable and open-source.
*S3 is a registered trademark of Amazon Technologies, Inc. OVHcloud services are not sponsored or approved by, nor affiliated with Amazon Technologies, Inc. in any way.

Explore our streaming, analytical and storage services, then turn high-volume events into dependable, queryable insights while reducing the operational work required to run the underlying infrastructure.