But for many teams, the “controlled environment” is a luxury they don’t have. Once your data touches the edge, mobile devices, or third-party webhooks, watermarks transition from a strict guarantee to a useful—but incomplete—heuristic.

Here is why watermarks eventually hit a wall, and the architectural patterns used to fill the gap.

I’m Subhan Hagverdiyev and welcome to Dataheimer - where we explore the atomic impact of data.

We have already reached 100+ subscribers. Thanks for all supporting. If you would like to receive more of this type of deep dives subscribe to the newsletter and I will send one each week

Subscribe now

First: What is a Watermark, anyway?

Before we break them, we have to understand their purpose. In stream processing, there is a fundamental tension between Event Time (when something happened) and Processing Time (when your engine sees it).

Because data can arrive out of order, a streaming engine needs a way to know when to stop waiting for data and actually perform a calculation (like a sum or average for a specific minute).

A watermark is a “completeness” signal. It is a timestamp flowing through the stream that effectively says: “I am reasonably confident that no more events with a timestamp earlier than X are going to arrive.”

• If a watermark for 10:05 AM arrives, the engine “closes” the 10:00–10:05 window and emits the result.

• The “Allowed Lateness” is the buffer you add—telling the engine to wait an extra 10 or 60 seconds before moving that watermark forward.

Red dots (above the watermark) are late events — they carry an old event time, but by the time they showed up, the watermark had already moved beyond them. For example, an event that happened at 10:01 but only arrived when the watermark was already at 10:03.

Windows close when the watermark crosses their boundary. Once Window 1’s time range ends and the watermark passes 10:02, the system emits results for that window. Any event for Window 1 arriving after that is considered late and may be dropped or handled separately.

The “Controlled Environment” Illusion

Watermarks rely on a fundamental assumption: that processing time and event time are loosely coupled and somewhat predictable. This works perfectly when:

• All producers are on synchronized cloud clocks.

• The network is stable.

• There is a single, linear ingestion point (like a single Kafka topic).

In these scenarios, a watermark is a reliable signal of “completeness.” But the real world is rarely this polite.

Filling the Gap: From Heuristics to Layered Accuracy

In reality there are a lot of cases where the heuristic breaks:

1. The mobile/edge reality

Mobile apps are the number one watermark killer.

Let’s say you set an allowed lateness of 30 seconds. Sounds generous right? But then look at what actually happens when you check your event arrival distribution:

• p90 of events arrive within 2–3 seconds. Fine.

• p95 stretches to about 15 seconds — still within buffer.

• p99 hits 4–8 minutes.

• And then there’s a long tail — maybe 2–5% of total events — arriving hours late.

Why? Users go into subway tunnels. They board flights. Their phone OS backgrounds the app. The device buffers events locally and dumps them all at once when it reconnects.

The problem is that by the time these events arrive, the watermark has already moved on. The window closed. The aggregate was emitted. Your real-time dashboard is now showing a number that’s wrong, and nobody knows it until someone compares streaming totals with the batch report and asks “why don’t these match?”

I think this is important to understand: in any company with a mobile user base, late data is not an edge case. It’s a structural property of the data. You can’t just set a bigger allowed lateness buffer and hope for the best — you need an architecture that handles it

2. Multi-Region Clock Skew

This one is more subtle and it specifically hurts when you’re doing stream-to-stream joins.

Let’s say you have two services. Service A runs in US-East, Service B runs in EU-West. Both emit events with event_time stamped from their local system clock, and you’re joining them on a shared key like session_id.

In theory, Network Time Protocol keeps cloud instances synchronized to within milliseconds. In practice, you’ll see clock drift anywhere from 50ms to 500ms across regions. During instance migrations or high CPU load it can be worse.

Here’s a concrete example. You have 1-minute tumbling windows. A user action triggers an event in Service A at 10:04:59.800 (US-East clock) and a corresponding event in Service B at 10:05:00.200 (EU-West clock). These events are about the same thing — they happened 400ms apart in reality. But because they straddle the minute boundary, Service A’s event goes into the 10:04–10:05 window and Service B’s event goes into the 10:05–10:06 window.

Your join now produces a null on both sides. The 10:04 window has an A event with no matching B. The 10:05 window has a B event with no matching A. Neither event is “late” from the watermark’s perspective — they’re just in the wrong windows.

Unfortunately this won’t show up in your late-event metrics. Everything looks healthy. You will only notice when join hit rates drop a few percentage points and you can’t explain why.

The usual fixes are either widening your windows (increases latency), using session windows with a generous gap (adds complexity), or pre-aligning timestamps to a shared reference clock before windowing. None of them are free.

3. The “Completeness” Join

This one is probably the least talked about but I think it causes the most confusion in microservice architectures.

In many systems, a single “event” doesn’t come from one place. Its full context depends on joining multiple streams. Let’s say you have an e-commerce platform where Service A emits order_placed events and Service B emits payment_confirmed events. You need to join them on order_id inside a time window to produce a complete order_completed record.

Service A is high-throughput — it processes thousands of events per second and they land in your stream almost instantly. Service B depends on a third-party payment gateway, so events arrive anywhere from 500ms to 30 seconds later, sometimes more.

I guess now you see the problem? If not don’t worry, I am going to explain :) Your watermark is advancing based on the fastest stream. Service A’s events are flowing in steadily, so the watermark moves forward at near real-time speed. But this watermark tells you nothing about whether the matching Service B event has arrived yet.

So what happens? The window closes because Stream A pushed the watermark past the boundary. The payment_confirmed event from Service B shows up 15 seconds later — technically on time by its own event timestamp, but the window is already gone. Your join emits a partial record or no record at all.

The result: you get order_placed events with no matching payment, which downstream looks like failed transactions. In a reports you will see the orders are dropping and when you debug and deep dive you will figure it out that it was just payments were slow.

I think this is the scenario where watermarks are most misleading. They give you a completeness signal for a single stream, but in a multi-stream join the completeness of one stream is irrelevant without the other. You need to either set your watermark strategy per-source (which most engines support but few teams configure properly), use event-time based session windows with wide gaps, or move the join out of the streaming layer entirely and do it in a reconciliation step.

Before jumping into fixes — how do you even know your watermarks are wrong? The two metrics to watch are late-event ratio per window (what percentage of events arrive after the window closes) and watermark lag (the gap between your watermark timestamp and wall-clock time). If your late-event ratio keeps climbing or your watermark lag is consistently high, that's your signal.

The Patterns That Fill the Gap

When watermarks alone can't guarantee accuracy, you need a layered approach. Here are the four main patterns teams use — and more importantly, when to pick which one.

Late-Data Side Outputs

The first thing to do is stop throwing away late data. Instead of dropping events that miss the watermark, you route them to a side output — basically a secondary stream that catches everything the main window missed.

In Flink this looks something like:

OutputTag<Event> lateTag = new OutputTag<Event>("late-data"){};

SingleOutputStreamOperator<Result> result = stream
    .keyBy(e -> e.getKey())
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .allowedLateness(Time.seconds(30))
    .sideOutputLateData(lateTag)
    .aggregate(new MyAggregateFunction());

DataStream<Event> lateStream = result.getSideOutput(lateTag);

You write these late events to a separate table (or topic) and let your query layer merge them with the main results. Your real-time view stays fast, and the late data doesn't disappear.

The “Close and Reopen” Strategy

We often think of a window as a one-time thing. It fires, emits a result, done. But it doesn’t have to be.

Many teams use a multi-pass approach:

The first pass closes the window based on the watermark and powers the real-time dashboard. It’s maybe 90-95% accurate and available in seconds.

The second pass runs hours later. It reopens the same window logic, incorporates the side output data, and emits a corrected version of the aggregate. Your dashboard number from 2 PM gets quietly updated by 5 PM.

I think this pattern works really well for analytics dashboards where “close enough now, exact later” is acceptable. If you’re building something where users see totals and trends — daily active users, revenue dashboards, funnel metrics — this is probably what you want.

Reconciliation Batch Jobs (Lambda Redux)

I know we are talking about streaming jobs but in real life most of the time they complement to each other. What does it mean for watermarks?

The streaming job handles the “now” — low-latency aggregates for dashboards and alerts. But alongside it, a nightly batch job (running in Snowflake, BigQuery, Spark, whatever you have) scans the raw event logs in S3 or GCS and recomputes everything from scratch. These raw logs eventually contain everything, no matter how late it was. The batch results overwrite the streaming aggregates.

Now I know what you’re thinking — this sounds like Lambda Architecture. And yes, conceptually it is. But the reason the original Lambda got so much hate was the “two codebases” problem: you had to maintain completely separate logic for the batch layer and the speed layer, and keeping them in sync was painful.

The modern version avoids this. You can use the same dbt models or the same SQL for both your streaming and batch paths. Flink SQL and your batch SQL can share the same transformation logic. The architecture is similar but the operational pain is significantly lower.

I think this pattern is non-negotiable for anything financial. Your dashboard can be slightly off at 2 PM, but your revenue reports need to be 100% accurate by next morning.

Designing for Idempotency

This is the one I think more teams should adopt. The idea is simple: instead of trying to make your windows perfect, make the downstream consumers not care if they receive updates multiple times.

If your writes are idempotent, you don’t need to “close” a window at all. Every time a late event arrives, you just re-emit the updated aggregate. The database overwrites the previous value.

In practice this means using MERGE or upsert semantics instead of INSERT:

-- Instead of this (breaks on late data):
INSERT INTO hourly_metrics (window_start, dimension, total)
VALUES ('2026-01-15 10:00', 'checkout', 542);

-- Do this (handles late data automatically):
MERGE INTO hourly_metrics AS target
USING (SELECT '2026-01-15 10:00' AS window_start, 
              'checkout' AS dimension, 
              547 AS total) AS source
ON target.window_start = source.window_start 
   AND target.dimension = source.dimension
WHEN MATCHED THEN UPDATE SET total = source.total
WHEN NOT MATCHED THEN INSERT (window_start, dimension, total) 
                      VALUES (source.window_start, source.dimension, source.total);

Every late event just updates the row. Your downstream consumers always see the latest truth. No side outputs, no reprocessing, no reconciliation needed.

The catch is that this only works when your consumers can handle updates. If something downstream already read the old value and acted on it (sent a notification, charged a customer), an update won’t undo that. So this pattern fits best for read-heavy analytical systems, not transactional workflows.

Conclusion

Watermarks are an optimization for latency, not a guarantee of accuracy. I think this is the most important thing to understand before you build any streaming pipeline.

In a controlled environment — synchronized clocks, stable network, single ingestion point — watermarks work great. But once your data touches mobile devices, crosses regions, or depends on joining multiple services, your streaming windows will be wrong sometimes. That’s not a bug. That’s just how distributed systems work.

The goal is not to make the stream perfect. The goal is to build a system that handles late arrivals without losing data. Use watermarks to get your low-latency metrics out the door. Use side outputs and idempotent writes to catch what the watermark missed. And use batch reconciliation to make sure the final numbers are actually correct.

"Thank you for reading this far. See you in my next articles. Don't forget to subscribe to get more of this interesting data engineering content!"

I’m Subhan Hagverdiyev and welcome to Dataheimer - where we explore the atomic impact of data.

Just like splitting an atom releases enormous energy, the right data engineering decisions can transform entire organizations.

This is where I break down complex concepts and share all the fascinating discoveries from my journey.

Want to join the adventure? Here you go:

We hear these words quite often these days: RAG, LLM, vector databases and of course MCP servers(though we don’t need it for this article). If you are planning to take a path of AI software engineer or simply don’t want to fall behind in AI development you need to understand what is RAG and how it helps LLMs.

In this article we will build a functional RAG system from scratch using Python and Ollama to run high-performance models locally on your machine.

What is RAG?

I think we already know what is LLM. A chatbot (like ChatGPT or Claude) that understands and generates human language.An LLM is like a pre-compiled binary with no internet access. RAG(Retriaval Augmented Generation) is a process where you find a specific piece of text from your own data and "paste" it into the prompt you send to the LLM. This allows the LLM to answer questions using information it wasn't originally trained on.

A real-world example would be asking Claude “How much debt I have to the bank?”. Claude cannot answer this question because it doesn’t have access to external knowledge, such as your financial information.

To address this limitation, we need to provide external knowledge to the model (in this example, a financial record)

A RAG system consists of two key components:

• A retrieval model that fetches relevant information from an external knowledge source, which could be a database, search engine, or any other information repository.

• A language model that generates responses based on the retrieved knowledge

Let’s create a simple RAG system that retrieves information from a predefined dataset and generates responses based on the retrieved knowledge. The system will comprise the following components:

1. Embedding model: A pre-trained language model that converts input text into embeddings - vector representations that capture semantic meaning. These vectors will be used to search for relevant information in the dataset.

2. Vector database: A storage system for knowledge and its corresponding embedding vectors. While there are many vector database technologies like Qdrant, Pinecone, and pgvector, we’ll implement a simple in-memory database from scratch.

3. Chatbot: A language model that generates responses based on retrieved knowledge. This can be any language model, such as Llama, Gemma, or GPT.

Indexing phase

Before we can ask questions, we must "index" our data. This involves breaking documents into chunks and converting them into vectors.

The embedding vectors can be later used to retrieve relevant information based on a given query. Think of it as SQL WHERE clause, but instead of querying by exact text matching, we can now query a set of chunks based on their vector representations.

Implementation

We will use Python to implement the RAG with Ollama because it allows us to run these models locally without API fees or privacy concerns.

Models we will use

• Embedding model: https://ollama.com/library/nomic-embed-text

• Language model: https://ollama.com/library/llama4:scout

And for the dataset, we’ll use a simple list of facts about cats. Each fact will be treated as a chunk during the indexing phase.

Install Ollama and pull models

Start by downloading Ollama from ollama.com.

ollama pull nomic-embed-text
ollama pull llama4:scout

```
If everything goes well, you'll see output like this:
```

pulling manifest
...
verifying sha256 digest
writing manifest
success

Next, install the Ollama Python package:

pip install ollama

Load the dataset

Create a new Python script and load the dataset. The dataset is a plain text file where each line is a cat fact — each line becomes a chunk for indexing.

You can download the example dataset from here. Here’s how to load it:

dataset = []
with open('cat-facts.txt', 'r') as file:
    dataset = file.readlines()
    print(f'Loaded {len(dataset)} entries')

Build the vector database

import ollama

EMBEDDING_MODEL = 'nomic-embed-text'
LANGUAGE_MODEL = 'llama4:scout'

# Each element in VECTOR_DB is a tuple: (chunk, embedding)
# An embedding is a list of floats, e.g. [0.1, 0.04, -0.34, 0.21, ...]
VECTOR_DB = []

def add_chunk_to_database(chunk):
    embedding = ollama.embed(model=EMBEDDING_MODEL, input=chunk)['embeddings'][0]
    VECTOR_DB.append((chunk, embedding))

For simplicity, each line in the dataset is treated as its own chunk:

for i, chunk in enumerate(dataset):
    add_chunk_to_database(chunk)
    print(f'Added chunk {i+1}/{len(dataset)} to the database')

Implement the retrieval function

Next, we need a way to find the most relevant chunks for a given query. We’ll compute the cosine similarity between the query’s embedding and every chunk embedding in our database, then return the top matches.

Cosine similarity measures how “close” two vectors are in the embedding space — a higher value means the texts are more semantically similar.

def cosine_similarity(a, b):
    dot_product = sum([x * y for x, y in zip(a, b)])
    norm_a = sum([x ** 2 for x in a]) ** 0.5
    norm_b = sum([x ** 2 for x in b]) ** 0.5
    return dot_product / (norm_a * norm_b)

And the retrieval function:

def retrieve(query, top_n=3):
    query_embedding = ollama.embed(model=EMBEDDING_MODEL, input=query)['embeddings'][0]
    similarities = []
    for chunk, embedding in VECTOR_DB:
        similarity = cosine_similarity(query_embedding, embedding)
        similarities.append((chunk, similarity))
    # Sort descending — higher similarity = more relevant
    similarities.sort(key=lambda x: x[1], reverse=True)
    return similarities[:top_n]

Generate a response

Now comes the generation phase. We take the retrieved chunks, inject them into a prompt as context, and let the language model produce an answer grounded in that context.

input_query = input('Ask me a question: ')
retrieved_knowledge = retrieve(input_query)

print('Retrieved knowledge:')
for chunk, similarity in retrieved_knowledge:
    print(f' - (similarity: {similarity:.2f}) {chunk}')

instruction_prompt = f'''You are a helpful chatbot.
Use only the following pieces of context to answer the question. Don't make up any new information:
{'\n'.join([f' - {chunk}' for chunk, similarity in retrieved_knowledge])}
'''

Then pass it to Ollama for generation:

stream = ollama.chat(
    model=LANGUAGE_MODEL,
    messages=[
        {'role': 'system', 'content': instruction_prompt},
        {'role': 'user', 'content': input_query},
    ],
    stream=True,
)

print('Chatbot response:')
for chunk in stream:
    print(chunk['message']['content'], end='', flush=True)

Putting it all together

Save the complete code to a file called demo.py and run it using following command:

python demo.py

Ask me a question: tell me about cat speed
Retrieved chunks: ...
Chatbot response:
According to the given context, cats can travel at approximately 31 mph (49 km) over a short distance. This is their top speed.

Improvements

This implementation is intentionally minimal. Here are some meaningful ways to improve it:

Smarter query construction. If the user’s question spans multiple topics, a single retrieval pass may miss important context. One approach is to have the language model rewrite or decompose the user’s question into multiple targeted queries before retrieval.

Reranking. The top-N results from cosine similarity aren’t always the most useful. A dedicated reranking model can re-score the retrieved chunks based on deeper relevance to the query, improving answer quality significantly.

Use a proper vector database. Our in-memory list won’t scale. For real applications, consider a purpose-built vector store like Qdrant, Pinecone, pgvector, or ChromaDB. These offer fast approximate nearest-neighbor search, persistence, and filtering.

Better chunking strategies. We’re treating each line as a chunk, which is simplistic. For longer documents, you’ll want to experiment with overlapping chunks, semantic chunking, or recursive splitting to capture more context per chunk.

Upgrade the language model. We used a relatively small model here for speed and simplicity. Larger models like Llama 4 Maverick, Qwen3, or DeepSeek V3 will produce more coherent and accurate responses, especially for complex questions.

Conclusion

RAG remains one of the most practical techniques for making language models useful with your own data. By building a simple RAG from scratch, we’ve walked through the core concepts: embedding text into vectors, retrieving relevant context via similarity search, and grounding generation in that context.

The ecosystem has matured substantially — running high-quality open models locally is now straightforward with tools like Ollama, and the range of available models and vector databases keeps growing. Whether you’re building a quick prototype or a production system, the fundamentals covered here are the foundation everything else builds on.

Thank you for reading this far. See you in my next articles. Don’t forget to subscribe to get more of this interesting data engineering content!”

I‘m Subhan Hagverdiyev and welcome to Dataheimer - where we explore the atomic impact of data.

Just like splitting an atom releases enormous energy, the right data engineering decisions can transform entire organizations.

This is where I break down complex concepts and share all the fascinating discoveries from my journey.

Want to join the adventure? Here you go:

Subscribe now

A few years ago, most conversations about data systems revolved around scale: more rows, faster queries, cheaper storage. Lately, the tone has shifted. Teams are still chasing performance, but they are also wrestling with a subtler problem—how systems understand the situation they’re operating in.

That’s where context engineering comes in.

The Tweet That Started It All

In June 2025, Shopify CEO Tobi Lütke posted something that made waves in AI commnity.Andrej Karpathy—yes, that Karpathy—immediately co-signed it. His take was more technical but hit the same nerve:

"People associate prompts with short task descriptions you'd give an LLM in your day-to-day use. When in every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window with just the right information for the next step."

So What Is Context Engineering, Really?

Here’s my working definition: Context engineering is the discipline of designing and optimizing the entire information environment that an LLM operates in—not just the prompt, but everything the model sees at inference time.

That includes:

• System instructions (the static stuff that tells the model how to behave)

• Retrieved knowledge (documents, database records, whatever you pulled from your RAG pipeline)

• Tool definitions (what functions the model can call)

• Conversation history (what’s been said before)

• User metadata (who’s asking, what permissions they have, what timezone they’re in)

• State (what step are we on, what’s already been tried)

How context shows up inside data engineering

For data engineering teams, context engineering is not merely an extension of existing ETL (Extract, Transform, Load) processes but a fundamental revaluation of what constitutes "data readiness". Traditionally, data engineering focused on moving raw digital signals into passive storage for human consumption. In the context-engineered enterprise, pipelines are redesigned to deliver "executable understanding"—data that is enriched with the semantic, structural, and operational metadata required for machines to act autonomously

Semantic Context and the Unified Meaning Layer

The first pillar of data-centric context engineering is semantic context, which ensures that machines and humans share a canonical understanding of business concepts. In complex enterprise environments, metric logic—such as the definition of "revenue" or "customer churn"—often varies across departments. Context engineering mandates that these definitions be centralized, versioned, and programmatically discoverable. Without this semantic grounding, AI agents frequently suffer from reasoning failures caused by mismatches in metric logic rather than model flaws.

Structural Context: Lineage as a Reasoning Graph

Lineage has transitioned from a passive compliance requirement to an active reasoning backbone. Agents require structural context to navigate the interconnected landscape of enterprise data, utilizing lineage graphs to trace anomalies upstream, estimate the “blast radius” of potential actions, and choose alternative data paths when a primary source is unavailable. This transformation often involves the use of knowledge graphs that link physical data assets (tables and columns) to business metrics and transformations.

Operational Context and Probabilistic Trust

Unlike traditional data quality monitoring, which treats trust as a binary (certified or uncertified), context engineering views trust as a dynamic, use-case dependent signal. Operational context includes real-time telemetry on data freshness, distribution shifts, and historical reliability patterns. For an agentic system, a dataset may be considered “good enough” for an internal forecast but “unsafe” for regulatory reporting; context engineering encodes this nuance, allowing agents to explain why a decision is safe or unsafe based on the current state of the underlying data.

Policy Context and Enforceable Constraints

The final data pillar is policy context, which ensures that agents operate within the legal, ethical, and regulatory boundaries of the organization. This involves embedding sensitivity classifications, regional constraints, and purpose limitations directly into the data context. These constraints must be machine-readable and enforced at decision time, providing an auditable trail of how data was used and for what purpose.

The realization of context-aware systems requires a significant upgrade to the traditional data stack, moving from simple relational tables to multidimensional vector databases and semantic graphs. These technologies provide the foundational "RAM" required for contextual understanding at scale. Although this is also quite important concept this will be discussed later in another article.

Wrapping Up

For data engineers, this is both a challenge and an opportunity. A challenge because the expectations are higher than ever. An opportunity because the skills you've built over years of pipeline work translate directly to this new domain.

Have thoughts on context engineering in your data stack? I'd love to hear what patterns you're seeing. The field is moving fast and none of us have all the answers.

There's a moment when you realize most data engineering courses teach you tools, not thinking.

You memorize Spark syntax. You follow Airflow tutorials. You build toy pipelines that would crumble under real load.

But then you find the real teachers. The ones who've processed billions of records at 3am when everything's on fire. The ones who explain why your pipeline failed, not just how to restart it.

Here are the resources that rewired my brain. They didn't just teach me data engineering. They taught me how to think like someone who builds systems that don't break.

Andreas Kretz - Learn Data Engineering

I found Andreas's channel while debugging a Kafka issue at my work.

No fancy animations. No "DESTROYING DATA PIPELINES WITH THIS ONE TRICK." Just a guy who's been doing this for 10+ years showing you what actually works.

Andreas doesn't teach like other YouTubers. He tears down real architectures—like how Nielsen processes 55TB daily. Real systems with real constraints and real failures.

His videos changed how I approach problems. ETL vs ELT wasn't abstract anymore. I understood why you'd pick Kafka over Kinesis, when batch beats streaming, why some architectures cost millions to fix later.

Every video made me sharper. Made me question my designs. And most importantly: made me less terrified of building at scale.

Seattle Data Guy - Ben Rogojan

Ben's content hit different because he's been on both sides—building at Meta and consulting for Fortune 500s.

There's a gap most channels never bridge: the line between pipelines that run... and pipelines that make money. That's where Ben shines.

His channel shows what actually happens in big companies. Not the sanitized conference talks. The real mistakes that cost millions. The architectural decisions that haunt you for years. The politics behind technical choices.

He covers the modern stack—dbt, Trino, Airflow—but always ties it back to business impact. That's what makes him essential.

After watching Ben, I started seeing the bigger picture. The organizational debt. The resume-driven development. The difference between being right and being effective.

Data with Zach (Zack Wilson)

Zach's channel found me when I needed it most—transitioning from software engineering to data engineering.

Most data content feels like it's made by people who've never written production code. Zach's different. He brings the software engineering lens to data problems. And that perspective changes everything.

He doesn't just show you how to build pipelines. He shows you how to build pipelines that other engineers won't curse you for. How to write tests that actually catch failures. How to structure code that humans can maintain.

His deep dives make you think. Why do we accept 99% accuracy? When is "good enough" actually good enough? How do you balance perfection with shipping?

After watching Zach, I started treating data code like real code. Version control became sacred. Tests became non-negotiable. Documentation became an act of kindness to future me.

The Books That Actually Matter

1. Fundamentals of Data Engineering (Reis & Housley)

You could read 20 tool-specific books and still not understand data engineering like you will after reading this one. Reis and Housley don't teach you Spark or Airflow. They teach you how to think.

They introduce this concept called "undercurrents"—security, DataOps, orchestration, architecture—the invisible forces that affect everything you build. Once you see them, you can't unsee them.

But what really got me was their approach to the data engineering lifecycle. They map out every stage from generation to serving. Not as a checklist. As a way of thinking about how data flows through systems.

After reading it, I stopped chasing shiny tools. Started asking better questions. What problem does this solve? What happens at 3am when this breaks? What are we optimizing for—cost, speed, or simplicity?

This book doesn't age. The tools it mentions might become obsolete. The thinking never will.

2. Designing Data-Intensive Applications (Kleppmann)

Everyone in data engineering tells you to read this book. And they're right.

Kleppmann breaks down how Kafka, Cassandra, and all those distributed systems actually work under the hood. Not the marketing version. The real version—with all the ways they can fail.

The book's famous for its consistency models chapter. Apparently it's the moment when everything clicks. When you finally understand why distributed systems are hard. Why your data disappears. Why "it works on my machine" means nothing at scale.

What people love: Kleppmann explains the why behind every design choice. No tool worship. Just trade-offs. He shows you when MongoDB makes sense and when it'll ruin your life.

I would just add a small warning that it's dense. Like textbook dense. Multiple engineers told me they had to read chapters twice. But they all said it was worth it.

The consensus? This book turns you from someone who uses distributed systems to someone who understands them. You'll start seeing problems before they happen. You'll make better choices. You'll finally get why senior engineers are so paranoid about network partitions.

To sum it up briefly

We're drowning in content but starving for wisdom. Everyone's selling courses. Everyone's promising to make you job-ready in 12 weeks

These resources are different. They come from people who've built things that matter. Who've felt the weight of bad decisions. Who care more about making you think better than making you code faster.

Let the learning compound. Let it change how you see systems. Let it make you dangerous in the best way.

The tools will change. The platforms will evolve. But the ability to think clearly about data at scale? That's yours forever.

Thank you for reading this far. See you in my next articles. Don't forget to subscribe to get more of this interesting data engineering content!"

I'm Subhan Hagverdiyev and welcome to Dataheimer - where we explore the atomic impact of data.

Just like splitting an atom releases enormous energy, the right data engineering decisions can transform entire organizations.

This is where I break down complex concepts and share all the fascinating discoveries from my journey.

Want to join the adventure? Here you go:

Subscribe now

In the world of big data analytics, counting distinct elements efficiently is a fundamental challenge.

Traditional methods like COUNT(DISTINCT) in SQL can be computationally expensive and memory-intensive when dealing with massive datasets. This is where HyperLogLog (HLL) comes to the rescue—a probabilistic data structure that can estimate the cardinality (number of unique elements) of large datasets with remarkable accuracy while using minimal memory.

The algorithm has been widely adopted across the database ecosystem, from traditional SQL databases to NoSQL systems, distributed query engines, and cloud data warehouses. This widespread adoption demonstrates its universal value for cardinality estimation in modern data processing.

Origins and History

The algorithm has its roots in academic research dating back to the 1980s. Philippe Flajolet and G. Nigel Martin introduced the foundational concepts in their 1984 paper "Probabilistic Counting Algorithms for Data Base Applications", which led to the Flajolet-Martin algorithm.

The evolution continued with Flajolet developing the LogLog algorithm in 2003, followed by SuperLogLog. Finally, HyperLogLog emerged as an extension of the earlier LogLog algorithm, offering improved accuracy and efficiency.

How HyperLogLog Works

HyperLogLog operates on a clever mathematical principle: it analyzes the patterns in hashed values to estimate cardinality. The algorithm works by:

1. Hashing:

Each input element is hashed to produce a uniform bit pattern

2. Leading Zero Analysis

Here's where HyperLogLog gets sophisticated. Instead of using the entire hash, we split it into two parts:

Part 1: Bucket Selection (first b bits)

• Use the first b bits to determine which bucket this element belongs to

• This creates 2^b buckets (typically b=4 to b=16)

• Each bucket will maintain its own statistics

Part 2: Rank Calculation (remaining bits)

• Use the remaining (64-b) bits to calculate the rank

For each hash, count the number of leading zeros in its binary representation. This becomes your "rank" for that element.

The mathematical insight: In a uniform random distribution, the probability of seeing exactly k leading zeros is 2^(-k-1). So if you observe k leading zeros, you've likely seen about 2^k unique elements.

Each bucket keeps track of the maximum rank it has seen:

Bucket 0: max_rank = 3
Bucket 1: max_rank = 7  
Bucket 2: max_rank = 2
...
Bucket 15: max_rank = 5

3. Final Cardinality Estimate

The final estimate uses the harmonic mean of all bucket estimates to reduce variance:

Why harmonic mean? Regular averaging would be dominated by buckets with very high ranks (outliers). The harmonic mean is more robust to outliers and provides better estimates

Key Advantages

Memory Efficiency

The Redis HyperLogLog implementation uses up to 12 KB and provides a standard error of 0.81%. This means you can estimate the cardinality of billions of unique elements using just 12 KB of memory—a remarkable feat compared to traditional exact counting methods.

Scalability

Traditional exact unique counting in SQL uses O(unique elements) additional memory, making it impractical for large datasets. HyperLogLog uses constant memory regardless of the dataset size.

Accuracy

Despite being probabilistic, HyperLogLog provides highly accurate estimates. The error for very small cardinalities tends to be very small, making it suitable for real-world applications.

HyperLogLog Across Database Systems

PostgreSQL: One of the first major SQL databases to embrace HyperLogLog through extensions like postgresql-hll, enabling efficient cardinality estimation for analytical workloads.

Microsoft SQL Server: Implements approximate distinct counting capabilities that leverage HyperLogLog principles in functions like APPROX_COUNT_DISTINCT.

Oracle Database: Provides HyperLogLog functionality through various approximate query processing features for large-scale analytics.

Distributed Query Engines

Presto/Trino: Facebook implemented HyperLogLog in Presto with the APPROX_DISTINCT function, reducing query times from days to hours for large-scale cardinality estimation.

Apache Spark: Integrates HyperLogLog through various libraries and built-in functions for distributed data processing.

Apache Drill: Supports HyperLogLog for efficient approximate distinct counting in distributed environments.

Cloud Data Warehouses

Google BigQuery: Provides comprehensive HLL functions (HLL_COUNT.EXTRACT, HLL_COUNT.MERGE) for large-scale analytics workloads.

Amazon Redshift: Implements approximate distinct counting using HyperLogLog-based algorithms.

Snowflake: Offers HLL functions for efficient cardinality estimation in cloud data warehousing scenarios.

Azure Synapse Analytics: Includes HyperLogLog-based approximate functions for big data analytics.

NoSQL and Specialized Systems

Redis: Popularized HyperLogLog with simple PFADD, PFCOUNT, and PFMERGE commands, making it accessible to application developers.

Apache Druid: Uses HyperLogLog for real-time analytics and approximate cardinality queries.

ClickHouse: Implements various HyperLogLog variants optimized for analytical workloads.

Elasticsearch: Provides cardinality aggregations based on HyperLogLog for search analytics.

SQL Usage Examples

Here are typical SQL pattern that could be used in real-world in Snowflake. Let’s say we are calculating user retention:

CREATE OR REPLACE TABLE user_activity AS
SELECT 
    DATEADD(day, UNIFORM(0, 29, RANDOM()), '2024-01-01')::DATE as activity_date,
    UNIFORM(1000, 9999, RANDOM()) as user_id,
    UNIFORM(1, 10, RANDOM()) as page_id
FROM TABLE(GENERATOR(ROWCOUNT => 100000));

We will calculate daily active users using HLL:

CREATE OR REPLACE TABLE daily_active_users AS
SELECT 
    activity_date,
    HLL_BUILD_AGG(user_id) as user_hll_sketch
FROM user_activity 
GROUP BY activity_date
ORDER BY activity_date;

For weekly active users calculation using HLL union:

SELECT 
    DATE_TRUNC('week', activity_date) as week,
    HLL_ESTIMATE(HLL_COMBINE(user_hll_sketch)) as estimated_wau
FROM user_activity ua 
     WHERE DATE_TRUNC('week', ua.activity_date) = DATE_TRUNC('week', dau.activity_date)) as exact_wau
FROM daily_active_users dau
GROUP BY DATE_TRUNC('week', activity_date)
ORDER BY week;

Limitations and Considerations

Approximation Nature

HyperLogLog provides estimates, not exact counts. While the error rate is typically low (around 0.81%), applications requiring exact counts should use traditional methods.

Small Dataset Accuracy

For very small datasets (fewer than 100 unique elements), exact counting might be more appropriate as the memory savings of HyperLogLog are minimal.

Hash Function Dependency

The accuracy of HyperLogLog depends on the quality of the hash function used. Poor hash functions can lead to biased estimates.

Use Cases and Applications

Web Analytics

• Counting unique visitors to websites

• Tracking distinct page views

• Measuring user engagement across different time periods

Database Analytics

• Estimating table cardinalities for query optimization

• Monitoring data distribution in large datasets

• Real-time analytics dashboards

Stream Processing

• Counting unique events in real-time data streams

• Monitoring distinct users in live applications

• Fraud detection systems

Data Warehousing

• ETL pipeline optimization

• Data quality assessment

• Historical trend analysis

Conclusion

Depending on your use case (some of them can't stand approximation!), HyperLogLog can be an incredibly useful thing in your toolbox!

HyperLogLog is particularly efficient at estimating the cardinality of large datasets:

• with relatively high accuracy (typically ~0.81% error rate)

• while using a small amount of memory and CPU (~12KB memory vs gigabytes for exact counting)

• enabling real-time analytics that would otherwise be impossible (queries finishing in minutes instead of hours)

Whether you're tracking daily active users, analyzing click streams, or optimizing database queries, HyperLogLog offers that sweet spot between accuracy and performance that makes big data analytics actually feasible. Just remember: when you absolutely need exact counts, stick with traditional methods—but for everything else, HLL is your friend!

Thank you for reading this far. See you in my next articles. Don't forget to subscribe to get more of this interesting data engineering content!"

I'm Subhan Hagverdiyev and welcome to Dataheimer - where we explore the atomic impact of data.

Just like splitting an atom releases enormous energy, the right data engineering decisions can transform entire organizations.

This is where I break down complex concepts and share all the fascinating discoveries from my journey.

Want to join the adventure? Here you go:

Subscribe now

At first glance, hash tables seem like the obvious winner when choosing a data structure for database indexing. After all, they offer O(1) lookup time for exact matches—how can anything compete with that? B+ trees, by contrast, offer O(log n) performance, which feels sluggish in comparison.

But here's what's puzzling: PostgreSQL, MySQL, Oracle, and virtually every major database system defaults to B+ trees for their primary indexes. Are these billion-dollar companies making a fundamental performance mistake?

Spoiler alert: They're not.

Intro

Despite their O(1) lookup time , hashmaps are not preferred in database indexing. Because in real world , databases are asked to do more than just find a single record by its exact key.

Hash tables excel at one thing: fast, exact key lookups. But they fall flat when we need:

• Range queries (e.g., "find all users registered between January and March")

• Sorting (e.g., "order by last name")

• Prefix or partial matches (e.g., "name starts with 'Al'")

B+ trees, on the other hand, gracefully support all of the above. Their structure maintains data in sorted order, enabling efficient in-order traversal, range filtering, and even multi-level indexing with minimal overhead.

We will now explore their strength and internal structure

Disclaimer: We won’t go into too much detail while explaining each of them since our goal is to find out why databases prefer one over another for indexing. If their will be interest I will write about each of them in depth on next articles.

Hash table

Let's start with what everyone learns in algorithms class:

A hash table is a data structure that maps keys to values using a hash function to compute an index into an array of buckets or slots.

Let’s say we have 2 key-value pairs (Banana, 1) and (Apple, 2). We pass each key to the hash function and the hash function returns a numeric hash code( for example "Banana" → 100, "Apple" → 150). Then the hash code is converted to a valid array index by taking it modulo the array size. In our case the array size is 6:

• 100 % 6 = 0 → index for "Banana"

• 150 % 6 = 5 → index for "Apple"

Each index stores key-value pairs:

• Index 0 → ("Banana", 1)

• Index 5 → ("Apple", 2)

Since the hash codes map to different indices (0 and 5), there is no collision.

However collisions happen when two keys hash to the same index:

We are now adding (Mango, 3) pair and both hash codes(100, 200) points to index 0. There are several different ways of addressing collisions. The most common one is simply to store all the objects that get assigned to the same index in a linked list. In this scenario, instead of simply storing the value at that index, the linked list must contain both the entire key and the value in pairs instead of just the value, so that the values can be uniquely tied to a key.

Performance:

• Average Case Lookup: O(1)

  • With a good hash function and low collision rate, lookup, insert, and delete operations take constant time.

• Best Case: Lightning fast

  • If there are no collisions, the operation is almost instantaneous.

• Worst Case: O(n)

  • If the hash function causes many collisions or the table is poorly sized, performance can degrade to linear time.

B+ Trees:

A B+ tree is a type of self-balancing tree data structure. Unlike a regular B-tree, a B+ tree separates its “table of contents” from its actual “content pages”

In B+ trees we keep all our actual data at the bottom level (leaf nodes) and use the upper levels purely for navigation

[7][17][25] represent our root nodes. These are pure navigation aids – they contain key values that guide our search but don't store actual data. Think of them as highway signs pointing us in the right direction.

[1][4] → [7][10] → [17][19][21] → [25][31] are our leaf nodes. These contain the actual data (or pointers to the actual data).

The Magic of B-Tree Search

Here's where it gets interesting. When we search for a value, the B-tree doesn't check every single piece of data. Instead, it uses those key values as a roadmap:

Looking for the number 19?

1. Start at the root: [7][17][25]

2. Since 19 > 17 but < 25, follow the path between 17 and 25

3. Land directly at the leaf node [17][19][21]

4. Found it! Only 2 steps instead of potentially searching through dozens of records.

These linked leaf nodes in the diagram are the reason B+ tree can have super fast range queries.If we need all records between 10 and 21? We just need to find the starting point and follow the links. No more tree traversal needed!

Performance:

• All Operations (Search, Insert, Delete): O(log n)

  • Each operation takes logarithmic time because the tree remains balanced, and each node can have multiple children, reducing the tree height.

What Real Applications Actually Need

Let's look at what a typical e-commerce application needs:

-- Exact lookup (Hash tables excel here)
SELECT * FROM users WHERE user_id = 12345;

-- Range query (B+ tree is faster)
SELECT * FROM orders WHERE order_date >= '2025-01-01' AND order_date < '2025-02-01';

-- Sorting (B+ tree is faster)
SELECT * FROM products ORDER BY price ASC;

-- Partial matching (B+ tree is faster)
SELECT * FROM users WHERE email LIKE 'john%';

-- Top-N queries (B+ tree is faster)
SELECT * FROM products ORDER BY rating DESC LIMIT 10;

Out of these five common query patterns, hash tables only help with one. B+ trees handle all of them efficiently.

1. Range Queries

B+ trees maintain sorted order, making range scans efficient. Need all orders from January? The B+ tree can find the first January record and scan sequentially to the end.

Hash tables? We'd have to check every single day in January individually.

2. Sorting

Since B+ tree leaves are already sorted, getting ordered results is essentially free. The data is already in the order we need.

Hash tables provide no ordering guarantees. To sort hash table results, we need a separate sorting step.

3. Partial Matches

In the query example we are looking for all emails starting with "john". B+ trees can find the first match and scan forward. Hash tables require scanning the entire table.

4. Memory Efficiency

B+ trees have excellent cache locality. Related data is stored together, reducing disk I/O and improving cache performance.

Hash tables can have poor cache performance due to random access patterns.

Major Databases usage of B+ tree

PostgreSQL's Approach

PostgreSQL offers both hash and B+ tree indexes but defaults to B+ trees for good reason:

-- PostgreSQL's default
CREATE INDEX idx_user_id ON users(user_id);  -- Uses B+ tree

-- You can force hash indexes, but rarely should
CREATE INDEX idx_user_id_hash ON users USING HASH(user_id);

The hash index is only useful for equality comparisons and doesn't support WAL logging (making it unsuitable for replication).

MySQL's InnoDB Strategy

MySQL's InnoDB engine uses B+ trees exclusively for all indexes. They decided that the consistency and versatility of B+ trees outweighed the potential O(1) benefits of hash tables.

Oracle's Balanced Approach

Oracle uses B+ trees for most indexes but implements hash clusters for specific use cases where we genuinely only need exact lookups.

When Hash Tables Actually Make Sense

So until here we covered for indexing B+ tree definitely wins. However hash tables aren't useless—they excel in specific scenarios:

1. Redis and In-Memory Stores

Redis uses hash tables for O(1) key-value lookups because:

• Data fits in memory

• Access patterns are primarily exact lookups

• Range queries are handled by other data structures (sorted sets, lists)

2. Database Hash Joins

During query execution, databases often build temporary hash tables for join operations:

-- The query optimizer might build a hash table 
-- for the smaller table to speed up this join
SELECT * FROM small_table s
JOIN large_table l ON s.id = l.small_id;

3. Distributed Systems

Systems like Cassandra use hash-based partitioning to distribute data across nodes, but still use other structures for local storage.

Conclusion

The choice between hash tables and B+ trees in database indexing isn't really about which data structure is "better" in isolation—it's about which one better serves the diverse needs of real-world applications.

While hash tables offer O(1) lookup time, databases aren't just glorified key-value stores. They're complex systems that need to handle everything from exact matches to range queries, from sorting operations to partial text searches. In this broader context, B+ trees emerge as the clear winner for general-purpose indexing.

The logarithmic O(log n) performance of B+ trees might seem inferior to hash tables' constant time operations, but remember: in a well-balanced B+ tree with thousands or even millions of records, we’re typically looking at just 3-4 disk reads to find any piece of data

This is why PostgreSQL, MySQL, Oracle, and other major database systems have converged on B+ trees as their default indexing strategy. They've chosen versatility and consistency over the narrow performance advantage that hash tables provide for exact lookups.

Hash tables still have their place in computing, particularly in in-memory systems, caching layers, and specialized use cases. But for the foundational task of database indexing, B+ trees reign supreme—and now you know why.

Thank you for reading this far. See you in my next articles. Don't forget to subscribe to get more of this interesting data engineering content!"

I have seen quite a lot of AI papers flooding around internet, and curious why data engineering papers was not shared enough. Today I am going to share top Data Engineering papers that are must read.

1. MapReduce

This paper from Google introduced a programming model and execution framework for processing large-scale data across distributed systems. By abstracting parallelization, fault-tolerance, and load balancing, MapReduce made distributed computing accessible to engineers and sparked the big data revolution.It is probably fair to say that half of the academia are now working on problems heavily influenced by MapReduce.

2. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

   

   This is the research paper behind the Spark cluster computing project at Berkeley. It represents RDDs as a key abstraction enabling fast, in-memory computations while maintaining fault tolerance. It revolutionized batch and iterative data processing by offering better performance and easier programming models than MapReduce.

3. What Goes Around Comes Around

   This paper explores the design and performance of columnar storage formats, specifically how past ideas in database architecture are being revived and optimized for modern analytical workloads. It emphasizes the importance of vectorized execution and columnar encoding for big data systems.

4. The Google File System

   

   A foundational work that details a scalable, distributed file system built to handle Google’s internal data needs. GFS is designed for fault tolerance, high throughput, and large files, and it laid the groundwork for systems like HDFS and Bigtable.

5. Shark: SQL and Rich Analytics at Scale

   

   Shark extended Apache Spark to support SQL queries alongside complex analytics. It bridged the gap between fast in-memory computation and declarative query languages, making it possible to unify ETL, analytics, and machine learning workloads.More importantly, the paper discusses why previous SQL on Hadoop/MapReduce query engines were slow.

6. CAP Twelve years later: How the "Rules" have Changed

   

   The CAP theorem, proposed by Eric Brewer, asserts that any net­worked shared-data system can have only two of three desirable properties: Consistency, Availability, and Partition-Tolerance. A number of NoSQL stores reference CAP to justify their decision to sacrifice consistency. A reflective piece that revisits the CAP theorem (Consistency, Availability, Partition Tolerance) a decade after its original formulation. It clarifies misconceptions, explores real-world implications, and offers a more nuanced understanding of trade-offs in distributed systems.

7. Architecture of a Database System

   

   This survey paper dissects the major components and design principles of modern database systems, from storage and indexing to query processing and transaction management. It's a comprehensive blueprint for understanding how databases work under the hood.

8. Kafka: a Distributed Messaging System for Log Processing

   

   Kafka was introduced as a high-throughput, distributed messaging system designed for log data by Linkedin. This paper outlines its architecture and how it decouples producers and consumers while providing fault tolerance and scalability, forming the backbone of many real-time data pipelines.

9. Bigtable: A Distributed Storage System for Structured Data

   Google’s Bigtable paper describes a distributed, sparse, and scalable data storage system that supports structured data. It underpins services like Google Analytics and Search, and inspired open-source projects such as HBase and Cassandra

10. Dynamo: Amazon’s highly available key-value store

    

    Amazon’s Dynamo paper presents a distributed key-value storage system optimized for high availability and scalability. It introduced techniques like eventual consistency, consistent hashing, and vector clocks, influencing systems like Cassandra, Riak, and DynamoDB.

    
    If you would like to read more this type of papers you can check the Git repository which is created by Reynold Xin- cofounder of Databricks: https://github.com/rxin/db-readings?tab=readme-ov-file

    
    If you enjoyed it, consider subscribing to this newsletter grow!

    Subscribe now

The CDC Revolution

Traditional data integration relied on batch processing – ETL jobs that ran nightly or hourly, leaving businesses operating on yesterday's data. CDC changed everything by monitoring database transaction logs and streaming changes as they happen. Instead of asking "What changed since last night?", CDC answers "What just changed?" in real-time.

So when you make a change in the database let’s say :

You have a customer database table:

ID    Name    Email 
1     Alice   alice@email.com 
2     Bob     bob@email.com

Changes:

1. Bob updates his email.

2. A new customer, Carol, is added.

This output can be sent to analytics systems, caches, or data lakes for real-time updates.

Why Low Latency Matters for Streaming

Modern dashboards need live metrics, not hour-old snapshots. A 5-second delay in detecting a system outage could mean millions in lost revenue. Organizations using real-time CDC report:

• 60% faster decision-making with live dashboards

• 40% reduction in customer service issues due to data consistency

• 25% improvement in fraud detection accuracy

• Significant cost savings from preventing data-driven errors

But with great power comes great complexity. The CDC tool you choose will determine whether you achieve these benefits or struggle with operational overhead. Let's explore your options.

🔧 Open Source Champions

Debezium + Kafka The streaming powerhouse

Debezium transforms your database transaction logs into Kafka streams, offering true real-time CDC with sub-second latency. Built on Kafka Connect, it supports virtually every major database and integrates seamlessly with the Kafka ecosystem.

Best for: Teams already invested in Kafka infrastructure
Strengths: Zero licensing costs, highly flexible, mature community
Watch out for: Operational complexity, requires Kafka expertise, at-least-once delivery only
Latency: ~0.5 – 2 seconds
Cost: Free (infrastructure costs apply)

Architecture: Debezium runs as Kafka Connect source connectors on a Kafka Connect cluster. Each connector (MySQL, Postgres, Mongo, etc.) connects to its source DB’s log (binlog or logical stream) and writes change events to Kafka topics. Kafka Connect offset topics (or an external store) track connector state. Debezium also supports a standalone Debezium Server (no Kafka) to send events to Kinesis, Pub/Sub, Pulsar, etc. Users often build full pipelines by pairing Debezium (CDC) sources with Kafka Connect sinks.

Apache Flink CDC The exactly-once guarantee

Flink CDC combines the power of Apache Flink's stream processing with robust CDC capabilities. It's the only open-source solution offering true exactly-once semantics out of the box.

Best for: Organizations needing complex real-time transformations
Strengths: Exactly-once processing, automatic schema evolution, extremely low latency
Watch out for: Steep learning curve, requires Flink cluster management
Latency: Sub-second (often <100ms)
Cost: Free (infrastructure costs apply)

Architecture: A Flink cluster executes Flink CDC jobs defined via YAML or code. Each job uses Flink’s DataStream API (or Table API) with a CDC source connector. The job can incorporate transformations and connect to sinks (e.g. Kafka, JDBC, Iceberg). Flink CDC can run in streaming mode for continuous flow. It also supports batch operation by scanning static data or performing periodic snapshots. State (offsets, transformation state) is managed by Flink’s checkpointing and state backends (e.g. RocksDB).

Airbyte The connector ecosystem

With 400+ connectors, Airbyte democratizes data integration. While not purely streaming (it runs frequent batch jobs), it's user-friendly and cost-effective for many use cases. However I think for pure streaming this is not a good solution because it is very slow.

Best for: Startups and teams wanting low-cost, easy setup
Strengths: Massive connector library, user-friendly UI, open-source with cloud option Watch out for: Batch-oriented approach, 5+ minute minimum intervals
Latency: 5+ minutes
Cost: Free self-hosted, usage-based cloud pricing

Architecture: A central Airbyte server (cloud service or self-hosted) orchestrates jobs. Connectors run in Docker containers. Users configure sources and destinations in the UI (or via API). For CDC sources, Airbyte schedules frequent sync jobs (it treats CDC sources as regular syncs at high frequency). There’s no persistent streaming – each sync job pulls changes since the last sync.

🏢 Enterprise Solutions

Fivetran (Commercial SAAS)

Architecture: Fully-managed SaaS. Fivetran separates the user’s environment, Fivetran cloud, and customer cloud for security and performance. When a connector sync is due, Fivetran’s backend spins up transient worker processes (in Fivetran-managed infrastructure) to extract and load data. In practice, users log in to Fivetran’s dashboard to create connectors. The Fivetran control plane stores configuration, and its execution engine (workers) handles the actual data movement. Fivetran uses parallelization on-demand to maximize throughput.

Best for: Organizations prioritizing reliability and support over cost
Strengths: Zero maintenance, automatic schema handling, comprehensive connector catalog
Watch out for: High costs at scale, vendor lock-in, black-box operations
Latency: Sub-5 minutes
Cost: Usage-based (can be expensive)

I think for very low-latency (sub-1-minute) or custom pipelines, Fivetran may not be ideal.

Qlik Replicate (Attunity)

A mature, GUI-driven solution designed for mission-critical enterprise replication. Offers exactly-once delivery and supports even mainframe systems.

Best for: Large enterprises with complex, heterogeneous environments
Strengths: Exactly-once guarantees, minimal source impact, comprehensive enterprise features
Watch out for: High licensing costs, Windows-centric, heavyweight for small projects Latency: Sub-second to seconds
Cost: High (perpetual licenses)

Architecture: Typically installed as a server with agent components. Qlik has a listener that taps DB transaction logs (or triggers) and an in-memory engine that stages changes. It features two data paths: an ongoing CDC stream for updates and a batch loader for full loads/backfills. A central checkpoint store tracks progress for exactly-once delivery. Configuration is done via a GUI or CLI.

☁️ Cloud-Native Options

AWS DMS

AWS DMS is a managed cloud service for migrating and replicating databases. It supports both full load and ongoing CDC for many sources and targets.

Best for: AWS migrations and simple replications
Strengths: Fully managed, good for AWS ecosystems, handles legacy sources
Watch out for: Limited to AWS targets, higher latency, frequent task failures reported Latency: Not sub-second; typical latency might be seconds to minutes
Cost: Per-hour instance pricing

Architecture: AWS-managed. You create “tasks” that run on DMS replication instances (AWS EC2 under the hood). The instance connects to a source DB and target. DMS handles log-reading (CDC) internally. Architecture is serverless from the user’s perspective (AWS scales the replication instance).

AWS itself notes that DMS “is not a streaming system” and that latency can vary and is hard to monitor

🎉 BONUS

Estuary Flow The exactly-once innovator

Estuary Flow is a unified streaming/batch data platform. It uses a custom DSL (Flow) to define pipelines that capture, transform, and load data. Its core is Gazette (open-source) which manages the data streams (Collections). Flow can do CDC, batch, and real-time analytics in one model.

Best for: Organizations needing guaranteed exactly-once delivery with multiple targets
Strengths: True exactly-once end-to-end, automatic schema evolution, unified real-time/batch
Watch out for: Newer ecosystem, primarily commercial offering
Latency: <100ms
Cost: Usage-based with free tier

Architecture: Flow pipelines run on Estuary’s control plane (SaaS) and data plane (can be public or private cloud, or on-prem). A pipeline attaches to source systems (e.g. database tables or logs) and continuously reads changes. Internally, every source stream is stored in Collections (append-only log). One collection per source/table, which can then be read by multiple sinks. A Flow app can write to multiple targets from the same source collection. Underneath, Estuary provides connectors for many sources/targets, and a “Kafka emulation” (Dekaf) if needed.

I think this one is truly designed for real-time streaming. It captures every change exactly once as it happens (via DB WAL or log streaming). It also supports batch backfills: you can initiate a snapshot that will emit historical data into the collection. Crucially, real-time and historical data are unified in the same pipeline.

Conclusion

Choosing the right CDC solution is all about balancing your organization’s priorities—latency, complexity, cost, and ecosystem fit. Ultimately, real-time CDC is no longer a luxury but a necessity for data-driven businesses. By aligning your CDC tool with your team’s skills, infrastructure, and performance targets, you’ll unlock faster insights, more dependable reporting, and a decisive edge in a world where “what just changed?” matters more than ever.

Hey data folks,

Remember when Databricks dropped $800 million on Neon back in May and we were all scratching our heads wondering "why the hell does a lakehouse company need a Postgres startup?"

Well, plot twist: They weren't just buying Neon for their cute serverless database tech. They were playing 4D chess, and we just got to see their master move.

Lakebase is Databricks' new fully-managed Postgres database built specifically for AI applications and integrated with their lakehouse platform Databricks. It was announced at the Data + AI Summit on June 11, 2025 Databricks Launches Lakebase: a New Class of Operational Database for AI Apps and Agents - Databricks and is now in public preview.

The "Aha!" Moment We've All Been Waiting For

Here's what actually happened: Databricks saw the future of AI applications and realized there was a massive gap in their platform. Sure, you could run your fancy ML models and analytics on their lakehouse, but what about all those AI apps and agents that need to interact with operational data in real-time?

You know the drill – your AI chatbot needs to query customer data, your recommendation engine needs to hit product catalogs, your intelligent automation needs to read from and write to transactional systems. All of that lives in operational databases, usually Postgres, and it was living completely separate from your analytical lakehouse environment.

Before Lakebase, if you wanted to build an AI app on Databricks, you had to maintain separate infrastructure, deal with multiple security models, juggle different connection patterns, and probably write a bunch of custom ETL to keep everything in sync. That's not just technical debt – that's technical bankruptcy.

The Bigger Picture: Databricks' Master Plan

Step back and look at what Databricks is building here. They started with the lakehouse – unified analytics and ML on one platform. Then they added generative AI capabilities, vector databases, and model serving. Now they're completing the circle with operational databases.

This isn't just about having more products. This is about owning the entire AI application lifecycle:

1. Data ingestion and preparation (existing lakehouse)

2. Model training and fine-tuning (existing ML platform)

3. Model deployment and serving (existing model serving)

4. AI application runtime (new Lakebase)

When your AI application needs to serve a recommendation, it can query operational data in Lakebase, invoke a model served on Databricks, and write the results back to Lakebase – all within the same platform, same security model, same billing account.

Why Millisecond Response Times Matter (And Why Delta Tables Can't Deliver Them)

Here's the fundamental problem that's been haunting data teams: your beautiful lakehouse is fantastic for analytics, but it's terrible for serving live applications. When your customer-facing API needs to return personalized recommendations in under 50ms, querying a Delta table just isn't going to cut it.

The Physics Problem:

• Delta tables are optimized for throughput, not latency

• Parquet files require scanning and decompression

• Even with Z-ordering and liquid clustering, you're still talking hundreds of milliseconds for simple lookups

• Connection overhead to Spark clusters adds another layer of latency

The Real-World Impact:

Typical Delta Lake query: 200-500ms
API SLA requirement: <50ms
Customer patience: ~3 seconds before abandoning

This mismatch has forced teams into complex, expensive workarounds that Lakebase finally eliminates.

Lakebase's Dual-Mode Architecture

Lakebase solves this with two distinct operational modes, each optimized for different use cases:

Mode 1: Delta-Postgres Sync (The Game Changer)

This is where the magic happens. Lakebase maintains real-time synchronization between your Delta tables and corresponding Postgres tables:

How It Works:

1. Change Data Capture (CDC): Lakebase monitors your Delta table's transaction log

2. Incremental Sync: Only changed rows are propagated to Postgres

3. Schema Evolution: DDL changes in Delta automatically update Postgres schema

4. Conflict Resolution: Built-in handling for concurrent updates

Technical Implementation:

-- Create a synced table
CREATE TABLE customer_features 
SYNC WITH DELTA 'dbfs:/mnt/lakehouse/customer_features'
REFRESH EVERY 30 SECONDS;

-- Now query with sub-10ms response times
SELECT recommendation_score, last_purchase_date 
FROM customer_features 
WHERE customer_id = 12345;

Mode 2: Native Postgres Tables (Traditional OLTP)

For pure transactional workloads that don't need lakehouse integration:

CREATE TABLE user_sessions (
    session_id UUID PRIMARY KEY,
    user_id BIGINT,
    created_at TIMESTAMP,
    last_activity TIMESTAMP,
    session_data JSONB
);

CREATE INDEX idx_user_sessions_user_id ON user_sessions(user_id);
CREATE INDEX idx_user_sessions_activity ON user_sessions(last_activity);

The Bottom Line: Engineering Impact

Lakebase solves a fundamental architecture problem that's been forcing teams into complex, expensive workarounds. By providing sub-10ms query performance on lakehouse data, it eliminates the need for:

• Complex caching layers

• Custom ETL pipelines for operational data

• Separate operational databases

• Multiple security and governance systems

• Custom sync mechanisms

For engineering teams, this means faster development cycles, fewer systems to maintain, and the ability to build AI applications that were previously architecturally impossible or prohibitively expensive.

The real test will be how well it handles the edge cases and scale requirements of production workloads, but the fundamental approach is sound and addresses a genuine gap in the modern data stack.

Subscribe now