Build a real-time event pipeline for millions of events

Modern applications generate a constant stream of clicks, transactions, sensor readings, API calls, and logs.  Turn continuous, high-volume streams into reliable analytical insight. This enterprise architecture unifies collection, processing and analysis, combining durable ingestion with rapid queries for AdTech, FinTech, IoT and observability teams that need to act on current data.
[object Object]

The two bottlenecks of a real-time event pipeline

Real-time event pipelines break down in one of two places: events don't make it through the ingestion layer reliably, or they never become fast queries in the platform. These are separate problems that require different tools, and solving only one of them is how teams end up with expensive, fragile stacks.

Without a durable, streaming backbone, distributed data producers and data consumers become tightly coupled. Producers are the applications that generate and send events into Kafka topics, while consumers are the services that read those events. Producers write directly to queues, databases, or custom services, and when a consumer goes down or lags, the producer either blocks or starts dropping events.

Furthermore, PostgreSQL and similar OLTP Public Cloud databases are designed for row-level operations: inserts, updates, deletes. They are not designed to scan billions of rows, aggregate across multiple dimensions, or update dashboards every minute. As event volume grows, queries that used to return in 200 ms start taking seconds, then tens of seconds, then minutes. The system becomes a bottleneck for product decisions.
[object Object]

Operate with confidence at enterprise scale

The real-time event pipeline includes replication, backups and monitoring
[object Object]
Kafka enables proper streaming
[object Object]
ClickHouse supports fast analysis
[object Object]
Replication, backups and monitoring
[object Object]
European regions and data-sovereignty controls

A reference architecture: Apache Kafka + ClickHouse

Apache Kafka as the streaming backbone

Kafka acts as the central nervous system for events. Producers write events to topics, which are logically ordered and partitioned across brokers. Each partition can be consumed independently, which is how Kafka scales throughput horizontally. Consumer groups read from these partitions, and each consumer in a group is responsible for a subset of the partitions.

Partitions are the key to throughput. A topic with more partitions can handle more writes and more consumers in parallel, up to the number of partitions. Consumer groups allow replay. Schemas for applications are typically managed separately, often with a Schema Registry, so producers and consumers can evolve without breaking each other. Events are kept for a configurable retention period, gives you a window to replay data without having to reprocess from the source. This durability and continuous decoupling is what makes Kafka a reliable backbone for high-volume event streams.

ClickHouse as the real-time OLAP layer

ClickHouse is a columnar, distributed database designed for analytical workloads. It stores data by column rather than by row, which means it can skip irrelevant columns during queries and compress data heavily. Combined with its storage engine, which is optimized for sequential reads and predicate pushdown, this allows ClickHouse to scan billions of rows quickly and return aggregated results in milliseconds. New events can be ingested in real time, and ClickHouse can pre-aggregate them using materialized views that update as data arrives. This means continuous dashboards and alerts, without waiting for a batch job or ETL window. The database is designed to scale horizontally, adding nodes to increase storage and compute capacity as the event volume grows.

Object Storage as a tiered cold-data tier for old events

Not all data needs to live in hot storage keeping every event in ClickHouse on local disks becomes expensive and unnecessary. The common pattern is to tier older data into OVHcloud Object Storage (S3-compatible)* storage, while keeping recent data in ClickHouse for fast queries.Events older than a configured threshold and queried on demand when deeper historical analysis is needed. This approach reduces the cost of long-term retention while preserving the ability to run queries across the full dataset. 

For many real processing use cases, the hot layer (ClickHouse) holds days or weeks of data for dashboards and real-time alerts, while the cold layer (OVHcloud Object Storage) holds months or years of history for compliance, audits, and periodic deep dives. This separation keeps the operational cost predictable and allows the pipeline to scale without hitting a hard app storage ceiling.


The native Kafka engine integration in ClickHouse

Apache KafkaKafka ConnectClickHouse
There are three main approaches: the native Kafka table engine in ClickHouse, materialised views built on top of that engine, and Kafka Connect for plugging into external systems.

ClickHouse has a built-in Kafka table engine that lets you subscribe directly to Kafka topics. You define a table with the Kafka engine, specify the topic name, broker list, and consumer group, and ClickHouse starts consuming events from Kafka as if they were rows in a regular table. This is a direct, native integration that does not require additional middleware.The Kafka engine in ClickHouse is designed for streaming ingestion. It reads records from the topic, parses them according to the format you specify (JSON, Avro, Protobuf, etc.), and surfaces them as rows that can be queried or inserted into other tables. 
Technical diagram of a real-time event pipeline with Apache Kafka at the centre connecting applications, databases, downstream platforms and
The real-time event pipeline collects impression, click and conversion streams as they occur. Campaign teams can monitor pacing, update attribution signals and identify unusual traffic patterns before delayed batch reports expose them. Independent event types remain available for investigation and recalculation.

Transaction events feed live risk indicators, behavioural checks and operational alerts. Analysts can examine account, payment and device activity together, helping investigation teams recognise suspicious sequences and respond using current information rather than waiting for scheduled processing.

Sensors, connected devices and vehicles produce continuous telemetry. The pipeline captures measurements, status changes and fault signals, enabling teams to assess device health, track fleets and detect operating conditions that require intervention.

Logs, metrics and traces flow into ClickHouse for rapid filtering and correlation across services. The Kafka table engine provides the streaming bridge into analytical tables, where platform teams can explore incidents, compare system behaviour and support dashboards without running separate ingestion paths for each signal type.
The real-time event pipeline collects impression, click and conversion streams as they occur. Campaign teams can monitor pacing, update attribution signals and identify unusual traffic patterns before delayed batch reports expose them. Independent event types remain available for investigation and recalculation.

Shape incoming events for faster decisions

Pre-aggregate recurring measures

Materialized views continuously calculate totals, rates and grouped metrics as events arrive, reducing repeated work at query time.

Align tables with access patterns

Order data by time and commonly filtered dimensions so investigations read fewer irrelevant records.

Read only useful fields

Column selection limits scanning to the attributes required by each dashboard, alert or exploratory query.

Design retention around demand

Separate frequently queried records from less active data. Retention-aware tables help sustain sub-second analytics for operational views while keeping investigations practical as histories grow.

The cost of waiting

LatencyETLTerraform
When the app pipeline is slow or unreliable, the consequences are measurable. Fraud detection response times stretch from seconds to minutes, which in FinTech translates directly into financial loss and reputational exposure. Slow reporting cycles block product decisions and force teams to rely on snapshots that are already stale by the time they are reviewed. To compensate, teams start patching the stack. Custom ETL jobs, hand-assembled message queues, and reporting views accumulate into an ad-hoc pipeline that must be maintained manually. The real cost is not just the immediate latency. It is the operational drag, the technical debt, and the fact that the team cannot iterate on real-time features because they are busy keeping the existing pipeline alive.
[object Object]

Assign each layer a clear responsibility

Streaming backbone

Buffer, order and replay high-volume event flows so producers remain independent from processing and analytical workloads.

Analytical serving

Expose fresh events to rapid operational queries, dashboards and investigations while maintaining a dedicated environment for analytical demand.

Cold-data tier

Retain older history outside performance-focused storage. This separation supports long-term access without keeping every event in the hot analytical path. The common pattern is to tier older data into OVHcloud Object Storage

Managed operations & delivery

OVHcloud manages replication, backups and monitoring across the service architecture. Teams can deploy resources through the OVHcloud Control Panel or define repeatable environments with Terraform. 

Deploy a real-time event pipeline in minutes

Provisioning a real-time event pipeline on OVHcloud takes minutes. You deploy a Managed Apache Kafka cluster and a Managed ClickHouse instance through the OVHcloud Control Panel or Terraform, with replication, backups, and monitoring enabled by default.

For teams that want help sizing the stack, OVHcloud offers direct access to data infrastructure specialists, including for Managed Databases for PostgreSQL , who can assess your current event volumes and query patterns and provide a cost comparison against your existing stack.Whether you are a Data Engineer who needs billion-row real-time results with minimal infrastructure time, or a Head of Data who wants reduced ops load and predictable cloud costs, the managed stack removes the operational overhead while keeping the architecture portable and open-source.

*S3 is a registered trademark of Amazon Technologies, Inc. OVHcloud services are not sponsored or approved by, nor affiliated with Amazon Technologies, Inc. in any way.
OVHcloud_Cloud_Database_Main_600x400.png
Start building dependable real-time insight
Running Kafka and ClickHouse yourself is possible, but it is expensive in engineering time and operational risk. OVHcloud offers both Managed Apache Kafka and Managed ClickHouse as fully managed services with a focus on European sovereignty, predictable pricing, and high availability. 
Explore our streaming, analytical and storage services, then turn high-volume events into dependable, queryable insights while reducing the operational work required to run the underlying infrastructure.
Start building