Apache Kafka Basics: Topics, Partitions and Offsets

Our shop's website produces thousands of events every minute: Leyla opened the catalog, Aysel put an item into the cart, Murad reached the payment page. Analytics counts the popular pages, and tomorrow the team will need a recommendation service that needs all page views of the week. A RabbitMQ queue will not help: a message disappears after ack and cannot be read again. Apache Kafka was created for this. We will compare a log with a queue, learn what a topic, a partition and an offset are, run Kafka in Docker and write our first producer and consumer in Node.js.

A queue or a log

A RabbitMQ queue is like a pile of order slips: the cook takes a slip, cooks and throws it away. Kafka is a log: a notebook where records are only added at the end and nothing is erased. Every reader has their own bookmark and reads from the place they choose. A record that has been read stays where it is: another reader, or the same one tomorrow, can open the same page.

Topic, partition, offset

All of Kafka stands on four concepts:

• Event (also record) — an entry of the log: a key, a value (usually JSON), a timestamp and optional headers. For Kafka it is just bytes.
• Topic — a named log of events of one kind, for example page-views.
• Partition — a part of a topic: an independent log where events are only appended. It is the unit of parallelism and the unit of order.
• Offset — the number of an event inside its partition: 0, 1, 2… Numbers are never reused. The address of an event is "topic + partition + offset".

An offset is counted in each partition separately. The log model is a few lines of JavaScript:

// one partition = an array that only grows
const partition = [];

function append(event) {
  partition.push(event);
  return partition.length - 1; // the offset of the new event
}

append({ user: 'Leyla', page: '/home' });
append({ user: 'Aysel', page: '/cart' });
append({ user: 'Leyla', page: '/cart' });

// a reader remembers only a bookmark: the next offset to read
let offset = 0;
while (offset < partition.length) {
  const { user, page } = partition[offset];
  console.log(`offset ${offset}: ${user} opened ${page}`);
  offset++;
}

// reading removed nothing: move the bookmark back and read again
offset = 1;
console.log(`replay from ${offset}:`, partition[offset].user);

Program output:

offset 0: Leyla opened /home
offset 1: Aysel opened /cart
offset 2: Leyla opened /cart
replay from 1: Aysel

The array only grows (push), and the reader keeps a single number, offset. Reading removes nothing, and reading again is just moving the bookmark back. Real Kafka does the same on disk and on several servers.