There are 4 trends that have contributed to the transformation in the enterprise software industry, and have impacted how engineering organizations approach the software development life cycle (SDLC). From monoliths to micro-services, Kubernetes, and the rise of Site Reliability Engineering (SRE) to AI-assisted development, these trends have set the stage for shifting critical functions — and the ownership of those functions — to the left.

Before I dive in, I’d like to share my background. I started my software career in 2013 as a backend engineer after completing a bootcamp at Flatiron School. I spent several years at Percolate, where I witnessed the rise of micro-services and Kubernetes firsthand, and followed the team through its acquisition by Seismic in 2019. Today, I’m part of Seismic’s SRE team, where one of my responsibilities has been evaluating AI-assisted development tools and rolling them out across the engineering organization.

Monoliths to Micro-services

The shift from monoliths to micro-services is largely a by-product of the rise of cloud computing. Monolithic architectures provide distinct advantages over micro-services if the following bullet points describe your organization:

• Small engineering team / engineering organizations

• Simple applications without complex logic / no need for breaking out specialized domains to meet scaling needs

• Focus on product-market fit over revenue generation

I’ve had a hand in several monolith-to-micro-service transition projects. While there were different reasons used to justify each of the projects, scaling was a common thread. Performance scaling is a major driver, driven by the rise of cloud computing and a corresponding decrease in compute costs. This driver allows engineering teams to have full control of horizontal and vertical scaling isolated to a specific domain.

For example, I worked on a Django monolith application, and during the course of trying to achieve product-market fit (PMF), the domain logic for managing content had become bloated. This meant that were no clearly defined boundaries between the content domain and other domains in the monolith application, and resulted in performance implications that ultimately impacted customers. When a monolith starts to encounter scaling challenges and low-cost optimization solutions have been exhausted, spinning out the problematic domain with scaling challenges into a micro-service becomes a realistic option.

Organizational scaling is another driver. As an organization starts to scale beyond an extent where there is more collision of development changes in a monolith application code base, the organization typically will break up the engineering group into multiple teams where each team oversees a particular domain. This allows for teams to have more autonomy and oversight of their domain from a code management perspective.

The diagram below illustrates the main differences between monoliths and micro-services, and how they ultimately shape an engineering organization. The bottom half illustrates how this architectural shift mirrors organizational structure — a startup with 5-20 people sharing a monolithic codebase versus an enterprise with dedicated teams owning their own domains.

The diagram informs us of several key insights into organizational structure, and how they manifest downstream.

Monoliths have characteristics that are more favorable for startups:

• Tight coupling of business logic, data access layer, and database(s)

• Engineering teams “share” the monolith application domains

• Supporting infrastructure (CICD, observability, etc) coalesced around monolith

• Fast feature development that requires less cross-functional coordination

Enterprises, on the other hand, benefit from micro-services:

• Loose coupling of business logic, data access layer, and database(s)

• Engineering teams oversee their own application domains

• Supporting infrastructure has to scale to support increasing complexity of services

• Slower feature development that requires cross-functional coordination

Note that these are just a few of the many differences between monoliths and micro-services. An organization’s structure ultimately informs the path that the organization will take in the monolith-to-micro-service journey, and that has absolutely been true in my case. As organizations scale their teams and transition their services from monoliths to micro-services, critical functions such as service compute scaling, unit testing, and optimization of business logic within a domain get shifted left to a team that directly oversees that domain.

Kubernetes

Kubernetes (K8s) is the technology trend that has had the most significant impact on my career over the last 12 years. There was a Cloud Native Computing Foundation (CNCF) survey that said “96% of organizations [surveyed] are either using or evaluating Kubernetes”. The survey data is broken down below:

K8s introduced an orchestration layer that provided engineering organizations with 3 key benefits:

1. Resource management

2. Operational efficiency

3. Engineering productivity

K8s’ killer feature is a “scheduling” mechanism that promoted more efficient use of compute resources. The feature allows an engineering team to coordinate the deployment of services in such a way that the team’s services only utilize what they need by specifying the resources that they need at startup (“requests”) and the maximum that they can scale up to (“limits”). By configuring K8s features such as HorizontalPodAutoscaler resources or PodDisruptionBudget resources, engineering teams have a lot of capabilities they can leverage for resource management. In short, K8s makes up for its steep learning curve by providing a lot of features that are necessary for managing compute infrastructure in a cost-effective manner.

Operational efficiency has improved for engineering teams due to the leap in resource management capabilities that K8s introduced. The most significant impact has been in the resource optimization capabilities made possible by the K8s Metrics API. The Metrics API provides an endpoint for CPU and memory resource usage at both the pod level and the node level. This makes it quite powerful for teams to understand how services are utilizing resources, frees up teams from building out bespoke monitoring solutions, and allows teams to execute on strategic priorities such as cost optimization and efficient resource utilization.

That visibility also enables automation. K8s makes it possible to leverage the Metrics API and dynamically adjust resource allocations through scaling operations. For instance, if a service is experiencing 80% average CPU utilization across all of its pods, the HorizontalPodAutoscaler can automatically provision new pods to scale up the service and meet additional resource needs. This combination of observability and automated response is what makes K8s’ operational efficiency gains so durable — teams aren’t just seeing problems faster, they’re resolving them without manual intervention.

Engineering productivity has also skyrocketed for engineering teams that leverage K8s largely due to the ecosystem that K8s introduced. From ArgoCD to Harness and other GitOps-related tools for CICD pipelines, K8s has helped provide a platform for shifting what has historically been DevOps functions to the left. With enterprise organizations adopting K8s, there are 3 critical benefits:

• ease of integration with CI/CD pipelines

• faster time-to-market development time for new services

• ability to provide multi-cloud strategies

Prior to the introduction of K8s, lots of different engineering organizations used a variety of tools or internally developed patterns for provisioning infrastructure and deploying services. From bespoke deployment scripting tools to other virtualization technologies like VMWare, the ecosystem for service orchestration was quite fragmented, and grew even more so as products started to transition from monolith services to micro-services. In my personal experience, K8s helped to shift left the management of resources at the service level from specialized DevOps teams to backend engineering teams.

Site Reliability Engineering

The Site Reliability Engineering (“SRE”) function is where I’ve also seen a significant amount of change, particularly as engineering organizations shift performance and reliability initiatives away from core engineering teams. This is largely thanks to Google’s influence on engineering organizations via Google SRE, which took traditional DevOps / operations teams, applied software approaches and best practices, and created the SRE discipline. The intent behind the SRE discipline was to bridge the gap between engineering development and operations teams, and it has since become a critical role within engineering teams. Full disclosure: SRE is also where my focus has been lately as part of the SRE team at Seismic.

SRE also has seen a profound impact on enterprise engineering organizations. In the DevOps Institute’s Global SRE Pulse 2022 report, 62% of organizations have adopted SRE in some shape or form. Within those organizations, 19% leveraged SRE across the entire organization, and 55% implemented SRE practices within specific teams, products, or services. Given that SRE is a relatively nascent discipline, this is a significant shift where engineering leaders see integrating SRE practices as a way to maintain competitive advantage.

At Seismic, the SRE team was built from scratch in response to scaling and reliability pain. As the platform grew — both organically and through several acquisitions — the operational burden on engineering teams became noticeable. There was no dedicated function focused on reliability as a discipline, and incidents were handled reactively rather than systematically. Standing up a dedicated SRE team was the answer to a question that had been building for a while: who owns reliability?

The answer, it turns out, is everyone — but SRE’s role is to make that ownership practical. At Seismic, we’ve shifted reliability left in three meaningful ways.

First, engineering teams now define and own their own Service Level Objectives (SLOs) and Service Level Indicators (SLIs). Rather than SRE dictating what “reliable” means for a given service, the team that builds and operates the service sets those targets based on their understanding of customer expectations. Error budgets provide a shared language for balancing reliability investment against feature velocity — when you’re burning through your error budget, reliability work takes priority; when you have budget to spare, you ship features with confidence.

Second, on-call ownership has shifted to the teams closest to the code. This was a cultural shift as much as an operational one. Engineers who build a service are best positioned to diagnose and resolve issues with it, and owning on-call creates a natural feedback loop — if your service pages you at 2 AM, you’re motivated to fix the underlying issue rather than apply a band-aid.

Third, we invested in self-service observability. SRE built out shared platforms and tooling that engineering teams use to instrument, monitor, and alert on their own services. Instead of filing a ticket with an ops team to get a dashboard, engineers have the autonomy to build the visibility they need. This democratization of observability has been one of the highest-leverage investments we’ve made — it gives teams the data they need to make informed reliability decisions without creating a bottleneck.

— Observability Gap

This investment was far from unique — according to the Catchpoint SRE Report 2025, over half of organizations still report not having enough observability.

The common thread across these three shifts is a move from centralized gatekeeping to distributed ownership, enabled by shared platforms and practices. SRE didn’t shift responsibility onto engineering teams and walk away — it shifted capability left by building the tooling, frameworks, and culture that make reliability a first-class concern for every team. That’s a fundamentally different model from traditional operations, and it’s one that scales with the organization rather than against it.

AI-Assisted Development

AI-assisted development is the most recent, and arguably the most disruptive, trend I’ve witnessed. As part of Seismic’s SRE team, one of my responsibilities has been evaluating AI development tools and identifying use cases where they can meaningfully improve engineering workflows.

We took a structured approach to the evaluation, looking at multiple tools including GitHub Copilot, Claude, and ChatGPT across different use cases. Two criteria rose to the top: workflow integration and measurable ROI. A tool that produces impressive demos but doesn’t fit into an engineer’s existing IDE, CI/CD pipeline, or review process won’t see sustained adoption. Similarly, engineering leadership needed to see concrete productivity gains, and not just anecdotal enthusiasm, to justify the investment.

The initial reception among engineers was skeptical. Many viewed AI code assistants as glorified autocomplete, and there were legitimate concerns about code quality, security implications, and whether these tools would generate more technical debt than they resolved. Winning people over required a mix of approaches: data-driven metrics for leadership, live demos of real workflows, and enabling a handful of early adopters whose results could speak for themselves. Attitudes shifted once engineers saw where AI genuinely helped rather than just impressed.

At the individual developer level, the impact has been most visible in four areas: generating boilerplate and scaffolding code that nobody wants to write by hand, navigating unfamiliar codebases where the AI provides context-aware suggestions faster than reading through documentation, generating unit tests and edge cases that improve coverage, and producing documentation and inline comments from existing code. None of these are flashy use cases, but that’s the point. The cumulative time saved on routine work compounds into meaningful productivity gains across an engineering organization. The chart below from Catchpoint illustrates the impact of various AI use cases.

Where things get more interesting, and more relevant to the “shifting left” theme of this post, is in applying AI to SRE-specific workflows. I’ve been experimenting with connecting AI tools to observability and incident management platforms through integration protocols like MCP (Model Context Protocol). To appreciate why this matters, consider what incident triage typically looks like without it: an engineer gets paged, then scrambles across four or five tools — the alerting platform, observability dashboards, log search, deployment history, and a runbook — context-switching between tabs and mentally correlating signals. How long it takes to form a hypothesis depends on the engineer’s familiarity with the service, the clarity of the alert, and sometimes just luck. MCP collapses that workflow: rather than an on-call engineer manually jumping between dashboards, log viewers, and run-book documentation during an incident, the AI can pull from those sources and help synthesize what’s happening. The biggest win so far has been in the first few minutes of an incident - the triage phase where identifying the blast radius and narrowing down the likely root cause has the highest leverage. Having an AI assistant that can query your observability platform, cross-reference recent deployments, and surface relevant historical incidents in a single conversation meaningfully compresses the time from “something is broken” to “here’s where we should look.”

Outside of work, I’ve been using Claude and Claude Code for personal projects, and the experience has shifted how I think about AI tools as an engineer. Claude has become more of an architectural thinking partner — something I use to talk through design decisions, debug tricky problems interactively, and explore unfamiliar technologies before committing to an approach. The distinction matters because it reveals that “AI-assisted development” isn’t a single capability — it’s a spectrum that ranges from autocomplete to autonomous agents, and different tools occupy different positions on that spectrum depending on the use case.

The “shifting left” dimension of AI-assisted development is perhaps the most profound of the four trends in this post. AI tools are compressing the feedback loop between writing code and understanding its implications for testing, security, performance, and documentation. Tasks that previously required specialized knowledge or a separate review cycle can increasingly happen in the moment, at the point of development. When an engineer gets real-time feedback on a potential security vulnerability or a performance anti-pattern as they write the code, that’s shifting left in its purest form. And when an on-call engineer can triage an incident in minutes instead of the time it used to take to manually correlate data across half a dozen tools, that’s shifting left applied to operations, which brings the entire arc of this post full circle.

Takeaways

Looking back at the last 12 years, these four forces collectively shifted software engineering from monolithic craftsmanship toward distributed, automated, reliability-engineered systems. Infrastructure became code. Services became composable units. Operations became a discipline. And artificial intelligence became a collaborator. The cumulative effect has been to fundamentally raise the ceiling of what small teams can build and operate at scale by shifting complex organizational functions to the left.

The practical lesson I’d share with engineers navigating these shifts is that each trend didn’t eliminate complexity, but rather redistributed it. Micro-services gave teams autonomy but introduced coordination overhead. Kubernetes gave teams power over their infrastructure but demanded a steep investment in learning. SRE gave teams ownership of reliability but required cultural change around on-call and error budgets. AI tools are giving teams speed and rapid iteration but demand new judgment about when to trust the output and when to question it.

The ones who thrive through these transitions aren’t the ones who master every new tool first. They’re the ones who understand why the shift is happening and adapt their mental models accordingly. “Shifting left” isn’t just about moving tasks earlier in the development lifecycle - it’s about moving ownership, capability, and accountability closer to the people who are best positioned to act upon them.

There are a handful of tasks that do a great job of instilling dread in a software engineer. You know the feeling - you're not quite sure how to approach it, or what the potential ramifications could be should the task implementation go wrong. Nothing does a better job of dredging up those feelings than Kafka migrations.

In the first Technical Tuesdays post, I'll discuss how to migrate Kafka topics as well as consumer group offsets from a source cluster to a destination cluster using Strimzi Operator and MirrorMaker 2 on Kubernetes.

Generally, a simple producer-consumer cutover process is more than enough for most Kafka migration use cases, but this approach results in delayed message delivery during the cutover. There are some cases where delays are not an option during the cutover. This is where MirrorMaker 2 comes in - MirrorMaker 2 allows for a seamless migration and cutover process that mitigates the downtime risk.

Let's dive in.

What’s the goal?

The goal is to migrate two topics from one cluster to another. A migration of Kafka topics across clusters is a unique challenge in that there are a lot of moving parts involved. For example, in a simple migration scenario we have to consider the migration of the following:

• Topics

• Consumer group offsets

The following diagram represents a high-level migration of 2 simple topics from an "origin cluster" to a "destination cluster":

Using the above diagram as an example, we'll leverage MirrorMaker 2 to sync the 2 topics and their consumer group offsets.

The rest of the post will explore the "MirrorMaker2" component in the middle of the diagram.

Time Out: What's MirrorMaker?

MirrorMaker 2 (MM2) is a tool provided by Apache Kafka for copying data between two Kafka clusters. MM2 supports functionality for replicating "topics, topic configurations, consumer groups and their offsets, and ACLs". MM2 leverages Kafka Connectors to consume data from a source cluster, and produce that data to the destination cluster.

From here on forth, I'll refer to MM2's replication functionality as mirroring. Apache makes that distinction because replication is a core feature set of Kafka, and the "mirroring" term is a way to distinguish that data is being replicated from one Kafka cluster to the other.

You can read more about MirrorMaker and its implementation at the official apache/kafka repo.

Let's set up MirrorMaker 2

This implementation will leverage Strimzi Operator's MM2 configuration in order to allow for data mirroring between 2 clusters. We'll set up an active/passive MM2 mirror, which will allow for syncing data in a uni-directional fashion. It's also possible to set up an active/active mirror, but that's outside of the scope of this post.

Strimzi Operator allows for deploying Kafka infrastructure via Kubernetes. It also allows for deploying other Kafka-related technologies, such as Kafka Connect, MM2,  and Kafka Bridge. Strimzi Operator provides Custom Resource Definitions (CRDs)  that make it easier to manage and operate Kafka infrastructure as a first-class resource in Kubernetes. In order to get started with Strimzi Operator in minikube (or any K8S cluster), you can get started with the Strimzi Operator's bootstrap template. An example implementation is available at irmiller22/mirrormaker2-poc.

Now, on to the MM2 configuration.

The MM2 configuration is based off of the KafkaMirrorMaker2 spec of the Strimzi Kafka Operator API, and is implemented in Kubernetes as a KafkaMirrorMaker2 CRD. Looking at the spec, there are a couple of key properties:

• connectCluster

• clusters

• mirrors

Let's explore these by looking at an example YAML KafkaMirrorMaker2 resource configuration, and we'll discuss the key properties further.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: kafka-mirror
spec:
  version: 3.4.0
  replicas: 1
  connectCluster: "destination-cluster"
  clusters:
  - alias: "origin-cluster"
    bootstrapServers: origin-cluster-kafka-bootstrap:9093
    tls: 
      trustedCertificates:
      - secretName: origin-cluster-ca-cert
        certificate: ca.crt
    authentication: 
      type: tls
      certificateAndKey:
        secretName: admin-user-1
        certificate: user.crt
        key: user.key
    config:
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1       
  - alias: "destination-cluster"
    bootstrapServers: destination-cluster-kafka-bootstrap:9093
    tls: 
      trustedCertificates:
      - secretName: destination-cluster-ca-cert
        certificate: ca.crt
    authentication: 
      type: tls
      certificateAndKey:
        secretName: admin-user-2
        certificate: user.crt
        key: user.key
    config:
      config.storage.replication.factor: 1
      config.storage.topic: mirrormaker2-cluster-configs
      offset.storage.replication.factor: 1
      offset.storage.topic: mirrormaker2-cluster-offsets
      status.storage.replication.factor: 1
      status.storage.topic: mirrormaker2-cluster-status
  mirrors:
  - sourceCluster: "origin-cluster"
    targetCluster: "destination-cluster"
    sourceConnector:
      tasksMax: 3
      config:
        replication.factor: 3
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
        replication.policy.separator: "."
        replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
        refresh.topics.interval.seconds: 60
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 3
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 3      
        sync.group.offsets.enabled: true
        refresh.groups.interval.seconds: 60
    topicsPattern: ".*"
    groupsPattern: ".*"
  logging:
    type: inline
    loggers:
      connect.root.logger.level: "INFO"
  readinessProbe:
    initialDelaySeconds: 45
    timeoutSeconds: 5
  livenessProbe:
    initialDelaySeconds: 45
    timeoutSeconds: 5

Let's break each key property down step by step.

The connectCluster property should be set to the alias that's specified for the destination cluster. In other words, it should be defined as the cluster that data is being mirrored to. In the example above, connectCluster would be set to destination-cluster:

spec:
  ...
  connectCluster: "destination-cluster"
  ...

The clusters property represents a list of Kafka clusters that will be used in the replication process, one of which is the source cluster and one of which is the destination cluster. The KafkaMirrorMaker2ClusterSpec describes the properties that make up the spec. Let's look at the the spec specified in the example above:

spec:
  ...
  clusters:
  - alias: "origin-cluster"
    bootstrapServers: origin-cluster-kafka-bootstrap:9093
    tls: 
      trustedCertificates:
      - secretName: origin-cluster-ca-cert
        certificate: ca.crt
    authentication: 
      type: tls
      certificateAndKey:
        secretName: admin-user-1
        certificate: user.crt
        key: user.key
    config:
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  - alias: "destination-cluster"
    bootstrapServers: destination-cluster-kafka-bootstrap:9093
    tls: 
      trustedCertificates:
      - secretName: destination-cluster-ca-cert
        certificate: ca.crt
    authentication: 
      type: tls
      certificateAndKey:
        secretName: admin-user-2
        certificate: user.crt
        key: user.key
    config:
      config.storage.replication.factor: 1
      config.storage.topic: mirrormaker2-cluster-configs
      offset.storage.replication.factor: 1
      offset.storage.topic: mirrormaker2-cluster-offsets
      status.storage.replication.factor: 1
      status.storage.topic: mirrormaker2-cluster-status

For each cluster definition, there are several keys that need to be defined:

• alias

• bootstrapServers

• tls (only if TLS is enabled)

• authentication (only if authentication is required for the respective cluster)

• config (only if there are additional Kafka configs that are required)

These properties will look familiar to you if you've worked with Kafka before. These keys represent the configuration details needed in order for MirrorMaker2 to connect to each cluster.

The mirrors property represents the KakfaMirrorMaker2 mirroring configuration, and is a list of mirroring configurations. In other words, this property specifies how the data should flow in between the clusters configured in the clusters property. The KafkaMirrorMaker2MirrorSpec describes the properties that make up the spec. Let's look at the mirrors property that we defined earlier:

spec:
  ...
  mirrors:
  - sourceCluster: "origin-cluster"
    targetCluster: "destination-cluster"
    sourceConnector:
      tasksMax: 3
      config:
        replication.factor: 3
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
        replication.policy.separator: "."
        replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
        refresh.topics.interval.seconds: 60
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 3
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 3
        sync.group.offsets.enabled: true
        refresh.groups.interval.seconds: 60
    topicsPattern: ".*"
    groupsPattern: ".*"
  ...

There are a couple of key properties for each mirroring configuration:

• sourceCluster

• targetCluster

• sourceConnector

• heartbeatConnector

• checkpointConnector

• topicsPattern

• groupsPattern

The sourceCluster and targetCluster properties are self-explanatory, but let's dig into the other properties one-by-one.

The sourceConnector represents the configuration for the Kafka Connector belonging to the source cluster. The source cluster's Kafka Connector is responsible for replicating topics, consumer groups, ACLs, and any other cluster-specific configurations from the source cluster to the destination cluster. Additionally, it's also responsible for emitting messages to the offset-syncs topic that keeps track of the latest consumer group offsets for the respective topics. In the configuration example above, the sourceConnector configuration has the following properties:

spec:
  ...
  mirrors:
  - ...
    sourceConnector:
      tasksMax: 3
      config:
        replication.factor: 3
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
        replication.policy.separator: "."
        replication.policy.class: "io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy"
        refresh.topics.interval.seconds: 60
  ...

For the above configuration, the number of tasks that the connector will run is set to 3 via the tasksMax setting. The replication.factor setting for replicated topics is 3. This means that the respective topic data will have three copies for each partition of the topic, and each copy will live on a different Kafka broker. For the offset-syncs topic, the replication.factor setting is 1, meaning that there are no copies of partitions in the offset-syncs topic. The sync.topic.acls.enabled settings is set to false, meaning that no ACLs are mirrored across clusters. The replication.policy.class setting configures how mirrored topics are named in the destination cluster. By default, mirrored topic names are prepended with the name of the destination cluster. However, the io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy setting ensures that the topic retains the original name of the topic. More information is available via the Strimzi documentation on cluster configurations.

The heartbeatConnector represents the configuration for the Kafka Connector that manages heartbeats. The heartbeats Kafka Connector emits heartbeat messages to the heartbeats topic, and allows for monitoring of the mirroring process between clusters.

The checkpointConnector represents the configuration for the Kafka Connector that manages the checkpoints. The checkpoints Kafka Connector consumes messages from the offset-syncs topic, and creates checkpoints for the mirroring process to allow MM2 to continue the mirroring process where it left off in the event of a failure. In the example above, the checkpointConnector has the following properties:

spec:
  ...
  mirrors:
  - ...
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 3
        sync.group.offsets.enabled: true
        refresh.groups.interval.seconds: 60
  ...

For the above configuration, the replication factor for the checkpoints topic is 3. This means that there are three copies of each partition in the checkpoints topic, and each copy lives on a different broker. The sync.group.offsets.enabled setting is set to true, meaning that consumer group offsets are mirrored from the source cluster to the destination cluster.

Finally, the topicsPattern and groupsPattern specify which topics and consumer groups are mirrored, respectively. Both of these parameters take a regex pattern. The .* value specified for both of these parameters means that all topics and all consumer groups are mirrored from the source cluster to the destination cluster.

Let's visualize the MirrorMaker2 configuration

Let's start with the Kafka Connector for the source cluster.

Above, we see the Kafka Connector for the source cluster and the data flows associated with it. The connector for the source cluster has 2 main responsibilities:

• replicate topics and consumer group data from the source cluster to the destination cluster

• produce consumer group offset mappings to the offset-syncs topic

MM2 leverages the offset-syncs topic so that the connector tasks can pick up where they left off in the event of a failure or a restart. In the diagram above, the dotted line represents MM2 keeping track of the consumer group offset mapping so that it can keep the consumer groups in sync across both the origin and destination clusters.

Let's take a look at the Kafka Connector for heartbeats:

The connector for heartbeats has 1 primary responsibility:

• produces a message from the origin cluster to the heartbeats topic on a specified interval (default is 1 second) to indicate liveness

The heartbeats connector is intended for monitoring purposes to indicate that both the source and destination clusters are available and healthy. If there's no "heartbeat", then MirrorMaker2 takes that as a signal to interrupt the replication process.

Finally, there's the Kafka Connector for checkpoints:

The connector for checkpoints has one primary responsibility:

• produces messages containing consumer group offsets on the origin cluster to the checkpoints topic on the destination cluster

MM2 uses the checkpoints topic to keep consumer groups that have been replicated to the destination cluster up to date. MM2 will periodically check the checkpoints topic, compare the messages in the topic to the specified offsets for the respective consumer group, and will update the consumer group offset if necessary.

Migration Gotchas

MM2 does an exceptional job of simplifying migrations between Kafka clusters, but there are two gotchas to account for and keep in mind.

Offset Translation. MM2 by default does a great job with offset translation in cases where Kafka topics have a retention period defined. However, if there are consumer applications that store offsets in a database, then those applications will need to account for the change in offsets.

Let's illustrate this with an example. Let's say that topic animals has 1,000 messages on the topic, and the latest offset is equal to 1,000. The retention period, defined via retention.ms, is set to 604,800,000 ms, which is the equivalent of 1 week. Of the 1,000 messages, let's say that 250 of those were created within the last week. So that means that 250 messages are still retained on the topic, and 750 messages have been cleaned up due to the topic retention policy. This means that when initiate the migration, only 250 messages for topic animals will get replicated from the origin cluster to the destination cluster, and the latest offset for the respective consumer group will be 1000 on the origin cluster and 250 on the destination cluster.

Size of Consumer Groups / Topics. Be mindful of how many consumer groups and topics there are on the source cluster. If there's a significant number of consumer groups, you may need to adjust the number of tasks running on behalf of the source connector. Monitor the replication status during the migration, and adjust the number of tasks via the tasks parameter accordingly.

Schema Registry Topic Schemas. When working with topics that leverage a schema, make sure that the destination cluster is working with the same schema definitions that the source cluster is utilizing. Schemas leverage something called a "magic byte", which is a unique identifier that a schema uses when serializing / deserializing messages. For example, with Confluent Schema Registry, I was bitten by the "magic byte" during the migration. The "magic byte" is a unique identifier that is randomly generated during the creation of a schema, and is unique per schema.  It's not enough to recreate the schemas and utilize them with the new cluster, even if the schemas have the same schema IDs as the origin cluster. This is because while the schema IDs might match, the magic bytes won't have the same values. Fortunately, Confluent allows for exporting schemas across schema registries, which preserves the <magic_byte><schema_id><schema_bytes> format for each schema.

Fine-tune with Minikube. I can't emphasize this one enough. MM2 is a wonderful technology, but it does take some time to get used to and understand, especially if you haven't worked with Kafka much. For example, I built out a POC at irmiller22/mirrormaker2-poc with Minikube to better understand Strimzi Operator,  MM2 configuration, and the data replication flow between clusters.

Closing Thoughts

The Strimzi team has done an exceptional job building out their Kafka operator for Kubernetes, and has one of the more in-depth documentation resources that I've seen out there for open-source technology.

Even though I used Strimzi Kafka operator explicitly for the MM2 functionality, I'll consider using the operator for projects where a managed solution is prohibitively expensive.

Please reach out at ian@ltce.dev or leave a message in the comments if there's more that you'd like to learn about MM2 or other topics. Thanks for reading!

Leave a comment