02.3-keys-partitions.md

<< back to main index

Lab 2.3: Producing with Keys

Overview

Understanding key mappings to partitions

Run time

10 mins

Step-1: Create Topic kv

$   ~/apps/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
    --create --topic kv \
    --replication-factor 1 --partitions 10

Step-2: Listen on topic

We will use Kafkacat on terminal-1

$   kafkacat -q -C -b localhost:9092 -t kv -f 'Partition %t[%p], offset: %o, key: %k, value: %s\n'

Step-3: Produce Some key-value pairs

We will use kafkacat on terminal-2

$   kafkacat -P -b localhost:9092 -t kv -K :

And paste the following values on kcat producer terminal-2

A:a1
B:b1
A:a2
C:c1
B:b2
C:c2
D:d1
D:d2
E:e1
F:f1
G:g1
F:f2
A:a3

Step-4: Observe the Consumer Output

A sample output from consumer terminal-1 may look like this:

Partition kv[2], offset: 2, key: D, value: d1
Partition kv[2], offset: 3, key: D, value: d2
Partition kv[2], offset: 4, key: F, value: f1
Partition kv[2], offset: 5, key: F, value: f2
Partition kv[3], offset: 8, key: B, value: b1
Partition kv[3], offset: 9, key: C, value: c1
Partition kv[3], offset: 10, key: B, value: b2
Partition kv[3], offset: 11, key: C, value: c2
Partition kv[5], offset: 6, key: A, value: a1
Partition kv[5], offset: 7, key: A, value: a2
Partition kv[8], offset: 1, key: E, value: e1
Partition kv[8], offset: 2, key: G, value: g1
Partition kv[5], offset: 8, key: A, value: a3

A few things to note:

Inspect partition for key A. It will always be the same (here it is kv[5]; yours might be different partition number)

You can see one key will always map to ONE partition

But you will also see that more than one key can map to same partition. Here keys D and F are mapping to partition 2

Step-5: Try Sending in More Data

Go and type in some data in key-value format, like this. Feel free to improvise.

Observe how the keys are mapping to which partitions

customer-1:hi
customer-2:hello
customer-1:bye
customer-2:ciao

and some random text

zzz:fjsa
yyy:kdfja