「Apache Kafka」:修訂間差異

出自Gea-Suan Lin's Wiki
跳至導覽 跳至搜尋
本頁面具有訪問限制。如果您看見此訊息,這代表您沒有訪問本頁面的權限。
行 77: 行 77:
== 測試 ==
== 測試 ==


  沒有TCP load balancer時可以直接指定所有的ZooKeeper主機:
  可以這樣測試:
 
<syntaxhighlight lang="bash">
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com:2181
/opt/kafka/bin/kafka-topics.sh --create --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com:2181 --replication-factor 2 --partitions 1 --topic my-replicated-topic
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com:2181
</syntaxhighlight>
 
如果ZooKeeper不是使用TCP load balancer時 可以直接指定所有的ZooKeeper主機:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
行 86: 行 94:


 其中所有的<code>:2181</code>都可以簡化省略,因為ZooKeeper的預設Port就是Port 2181:
 其中所有的<code>:2181</code>都可以簡化省略,因為ZooKeeper的預設Port就是Port 2181:
<syntaxhighlight lang="bash">
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com
/opt/kafka/bin/kafka-topics.sh --create --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com --replication-factor 2 --partitions 1 --topic my-replicated-topic
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com
</syntaxhighlight>


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
行 91: 行 105:
/opt/kafka/bin/kafka-topics.sh --create --zookeeper 1.2.3.4,5.6.7.8,9.10.11.12 --replication-factor 2 --partitions 1 --topic my-replicated-topic
/opt/kafka/bin/kafka-topics.sh --create --zookeeper 1.2.3.4,5.6.7.8,9.10.11.12 --replication-factor 2 --partitions 1 --topic my-replicated-topic
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper 1.2.3.4,5.6.7.8,9.10.11.12
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper 1.2.3.4,5.6.7.8,9.10.11.12
</syntaxhighlight>
有TCP load balancer時:
<syntaxhighlight lang="bash">
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com
/opt/kafka/bin/kafka-topics.sh --create --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com --replication-factor 2 --partitions 1 --topic my-replicated-topic
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com
</syntaxhighlight>
</syntaxhighlight>



於 2019年12月31日 (二) 09:19 的修訂

Apache Kafka是一套基於JavaScala,提供Streaming架構的軟體。

介紹

Kafka本身不過多處理多節點之間的效能分配問題,而是透過ZooKeeper處理。可以從「#測試」的章節看到操作時都是指定ZooKeeper當作API Endpoint。在Production環境上,一般我們會用TCP load balancer放在前面,讓用戶端架構比較簡單(僅需設定單一名稱)。

安裝

目前在Ubuntu上沒有系統套件或是PPA可以安裝,一般有兩種安裝方式:

  • 自行下載後裝到/opt下。
  • 使用Confluent的版本安裝。(這也是一般社群建議的方式)

這篇將會使用自行下載的方式安裝,其中最新版的資訊可以在Kafka的Download頁面得到,這邊使用2.1.1版:

cd /tmp
wget https://ftp.jaist.ac.jp/pub/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz
cd /opt
sudo tar zxvf /tmp/kafka_2.12-2.1.1.tgz
sudo chown -R nobody:nogroup kafka_2.12-2.1.1/
sudo ln -fs kafka_2.12-2.1.1 kafka

設定

先修改/opt/kafka/config/server.properties內的一些設定:

# This needs to be different in different broker.
broker.id=1

#
log.dir=/var/lib/kafka-logs

# w/o TCP load balancer (ELB in this case) and w/.  ":2181" can be implicit due to default port.
zookeeper.connect=1.2.3.4:2181,5.6.7.8:2181,9.10.11.12:2181
#zookeeper.connect=1.2.3.4,5.6.7.8,9.10.11.12
#zookeeper.connect=test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com:2181
#zookeeper.connect=test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com

建立對應的log目錄與權限:

sudo install -d -o nobody -g nogroup -m 0700 /var/lib/kafka-logs

安裝systemd設定(/lib/systemd/system/kafka.service):

[Unit]
Description=Apache Kafka server (broker)
Documentation=https://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=nobody
Group=nogroup
Environment=JAVA_HOME=/usr
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

使用下面指令設定在開機時啟動,並且馬上啟動:

sudo systemctl daemon-reload
sudo systemctl enable kafka
sudo service kafka start

測試

可以這樣測試:

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com:2181
/opt/kafka/bin/kafka-topics.sh --create --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com:2181 --replication-factor 2 --partitions 1 --topic my-replicated-topic
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com:2181

如果ZooKeeper不是使用TCP load balancer時,可以直接指定所有的ZooKeeper主機:

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper 1.2.3.4:2181,5.6.7.8:2181,9.10.11.12:2181
/opt/kafka/bin/kafka-topics.sh --create --zookeeper 1.2.3.4:2181,5.6.7.8:2181,9.10.11.12:2181 --replication-factor 2 --partitions 1 --topic my-replicated-topic
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper 1.2.3.4:2181,5.6.7.8:2181,9.10.11.12:2181

其中所有的:2181都可以簡化省略,因為ZooKeeper的預設Port就是Port 2181:

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com
/opt/kafka/bin/kafka-topics.sh --create --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com --replication-factor 2 --partitions 1 --topic my-replicated-topic
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper test-gslin-zookeeper-xxxxxxxxx.us-east-1.elb.amazonaws.com
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper 1.2.3.4,5.6.7.8,9.10.11.12
/opt/kafka/bin/kafka-topics.sh --create --zookeeper 1.2.3.4,5.6.7.8,9.10.11.12 --replication-factor 2 --partitions 1 --topic my-replicated-topic
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper 1.2.3.4,5.6.7.8,9.10.11.12

相關連結

外部連結