Apache Kafka Чтение из топиков

Fri, 14 Feb 2025 09:19:41 +0000

Простое чтение топика с нулевой позиции:

./kafka-console-consumer.sh --bootstrap-server $HOST:$PORT --topic $TOPIC_NAME --from-beginning

Для чтения с определённой позиции нужно знать максимальный оффсет:

./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $HOST:$PORT --topic $TOPIC_NAME

Цифра, которую выдаст команда, это максимальное смещение.

./kafka-console-consumer.sh --bootstrap-server $HOST:$PORT --topic $TOPIC_NAME --offset $OFFSET_START --partition $PARTITION_NUMBER

$OFFSET_START — это максимальное смещение минус некая величина до позиции, с которой мы хотим вычитать топик.

Old Apache Kafka + ZooKeeper + Clickhouse in Docker

Thu, 02 Nov 2023 07:52:07 +0000

docker-compose.yml

version: '3.5'
services:
  kafka:
    image: bitnami/kafka:2.6.0
    ports:
      - "9092:9092"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.23.0.13:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper
    networks:
      clickhouse-network:
        ipv4_address: 172.23.0.13
  zookeeper:
    image: zookeeper:3.5.8
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - "2181:2181"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    networks:
      clickhouse-network:
        ipv4_address: 172.23.0.10
  clickhouse01:
    image: yandex/clickhouse-server:20.8.12.2
    container_name: clickhouse01
    hostname: clickhouse01
    networks:
      clickhouse-network:
        ipv4_address: 172.23.0.11
    ports:
      - "127.0.0.1:8123:8123"
      - "127.0.0.1:9000:9000"
    volumes:
      - ${PWD}/clickhouse01:/etc/clickhouse-server
      - /mnt/volume/clickhouse01/data:/var/lib/clickhouse/
      - /mnt/volume/clickhouse01/logs:/var/log/clickhouse-server/
    depends_on:
      - zookeeper
  clickhouse02:
    image: yandex/clickhouse-server:20.8.12.2
    container_name: clickhouse02
    hostname: clickhouse02
    networks:
      clickhouse-network:
        ipv4_address: 172.23.0.12
    volumes:
      - ${PWD}/clickhouse02:/etc/clickhouse-server
      - /mnt/volume/clickhouse02/data:/var/lib/clickhouse/
      - /mnt/volume/clickhouse02/logs:/var/log/clickhouse-server/
    depends_on:
      - zookeeper

networks:
  clickhouse-network:
    name: clickhouse-network
    ipam:
      config:
        - subnet: 172.23.0.0/24

M9K blog: заметки с тегом apache kafka

Apache Kafka Чтение из топиков

Old Apache Kafka + ZooKeeper + Clickhouse in Docker