Getting Started with Apache Kafka in .NET (2026): Producers, Consumers & Patterns
Apache Kafka is the standard choice when a .NET system needs to publish and process high volumes of events reliably — order events, sensor data, audit logs, or any scenario where multiple services need to react to the same stream of activity. This guide covers a working producer and consumer using Confluent.Kafka, the officially maintained .NET client.
Step 1: Run a local Kafka broker
The quickest way to get a broker running locally is Docker Compose, using Kafka's built-in KRaft mode (no separate ZooKeeper needed):
kafka:
image:confluentinc/cp-kafka:latest
ports:
- "9092:9092"
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
Step 2: Install the NuGet package
Step 3: Build a producer
This publishes an order-created event to a topic:
varconfig=newProducerConfig
{
BootstrapServers="localhost:9092",
Acks=Acks.All,
EnableIdempotence=true
};
usingvarproducer=newProducerBuilder<string,string>(config).Build();
varorderEvent= JsonSerializer.Serialize(new
{
OrderId="ORD-1024",
Total=249.99,
CreatedAt= DateTime.UtcNow
});
varresult= awaitproducer.ProduceAsync("orders",
newMessage<string,string> { Key = "ORD-1024", Value = orderEvent});
Console.WriteLine($"Delivered to {result.TopicPartitionOffset}");
EnableIdempotence = true and Acks.All together prevent duplicate messages on producer retries and ensure the broker confirms the write to all in-sync replicas before returning success.
Step 4: Build a consumer
varconfig=newConsumerConfig
{
BootstrapServers="localhost:9092",
GroupId="order-processing-service",
AutoOffsetReset=AutoOffsetReset.Earliest,
EnableAutoCommit=false
};
usingvarconsumer=newConsumerBuilder<string,string>(config).Build();
consumer.Subscribe("orders");
while(true)
{
varresult=consumer.Consume();
Console.WriteLine($"Processing: {result.Message.Value}");
// ...handle the event (idempotently)...
consumer.Commit(result);
}
EnableAutoCommit = false plus a manual Commit() after successful processing avoids the common bug where a message is marked as consumed before your handler actually finishes — if the process crashes mid-handling, the message gets redelivered instead of silently lost.
A pattern worth adopting: register in DI as a hosted service
In a real ASP.NET Core app, wrap the consumer loop in a BackgroundService so it starts and stops cleanly with the host:
{
protected override asyncTaskExecuteAsync(CancellationTokenstoppingToken)
{
using varconsumer= BuildConsumer();
consumer.Subscribe("orders");
while(!stoppingToken.IsCancellationRequested)
{
varresult=consumer.Consume(stoppingToken);
awaitProcessAsync(result.Message.Value);
consumer.Commit(result);
}
}
}
Register it in Program.cs with builder.Services.AddHostedService<OrderConsumerService>() and it'll run for the lifetime of your app, shutting down gracefully with the rest of the host.
Designing an event-driven architecture for a real system, or need help deciding whether Kafka is the right fit for your workload? This is the kind of architecture decision I work through as part of custom software development engagements.