如何使用Docker Swarm部署Elasticsearch？

Question

如何使用Docker Swarm部署Elasticsearch？

3

我使用docker-machine创建了三个虚拟机，它们分别是：

NAME       ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER     ERRORS
cluster    -        virtualbox   Running   tcp://192.168.99.101:2376           v18.09.5   
cluster2   -        virtualbox   Running   tcp://192.168.99.102:2376           v18.09.5   
master     -        virtualbox   Running   tcp://192.168.99.100:2376           v18.09.5

然后我在 master 机器上创建了一个 Docker Swarm:

 docker-machine ssh master "docker swarm init ----advertise-addr 192.168.99.100"

在 cluster 和 cluster2 中加入 master：

docker-machine ssh cluster "docker swarm join --advertise-addr 192.168.99.101 --token xxxx 192.168.99.100:2377"

docker-machine ssh cluster2 "docker swarm join --advertise-addr 192.168.99.102 --token xxxx 192.168.99.100:2377"

docker node ls 命令输出：

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
r4a6y9wie4zp3pl4wi4e6wqp8     cluster             Ready               Active                                  18.09.5
sg9gq6s3k6vty7qap7co6eppn     cluster2            Ready               Active                                  18.09.5
xb6telu8cn3bfmume1kcektkt *   master              Ready               Active              Leader              18.09.5

有一个部署配置文件swarm.yml:

version: "3.3"

services:
  elasticsearch:
    image: elasticsearch:7.0.0
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      - cluster.name=elk
      - network.host=_eth1:ipv4_
      - network.bind_host=_eth1:ipv4_
      - network.publish_host=_eth1:ipv4_
      - discovery.seed_hosts=192.168.99.100,192.168.99.101
      - cluster.initial_master_nodes=192.168.99.100,192.168.99.101
      - bootstrap.memory_lock=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    networks:
      - backend
    deploy:
      mode: replicated
      replicas: 3
      #endpoint_mode: dnsrr
      restart_policy:
        condition: none
      resources:
        limits:
          cpus: "1.0"
          memory: "1024M"
        reservations:
          memory: 20M
networks:
  backend:
    # driver: overlay
    # attachable: true

我将Elasticsearch镜像拉到虚拟机上：

docker-machine ssh master "docker image pull elasticsearch:7.0.0"
docker-machine ssh cluster "docker image pull elasticsearch:7.0.0"
docker-machine ssh cluster2 "docker image pull elasticsearch:7.0.0"

在运行之前，我运行了以下命令以修复一些 ElasticSearch 引导错误：

docker-machine ssh master "sudo sysctl -w vm.max_map_count=262144"
docker-machine ssh cluster "sudo sysctl -w vm.max_map_count=262144"
docker-machine ssh cluster2 "sudo sysctl -w vm.max_map_count=262144"

然后我运行了`docker stack deploy -c swarm.yml es`，但Elasticsearch集群无法工作。

docker-machine ssh master
docker service logs es_elasticsearch -f

展示：

es_elasticsearch.1.uh1x0s9qr7mb@cluster    | {"type": "server", "timestamp": "2019-04-25T16:28:47,143+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk", "node.name": "e8dba5562417",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [192.168.99.100, 192.168.99.101] to bootstrap a cluster: have discovered []; discovery will continue using [192.168.99.100:9300, 192.168.99.101:9300] from hosts providers and [{e8dba5562417}{Jy3t0AAkSW-jY-IygOCjOQ}{z7MYIf5wTfOhCX1r25wNPg}{10.255.0.46}{10.255.0.46:9300}{ml.machine_memory=1037410304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
es_elasticsearch.2.swswlwmle9e9@cluster2    | {"type": "server", "timestamp": "2019-04-25T16:28:47,389+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk", "node.name": "af5d88a04b42",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [192.168.99.100, 192.168.99.101] to bootstrap a cluster: have discovered []; discovery will continue using [192.168.99.100:9300, 192.168.99.101:9300] from hosts providers and [{af5d88a04b42}{zhxMeNMAQN2evKDlsA33qA}{fpYPTvJ6STmyqrgxlMkD_w}{10.255.0.47}{10.255.0.47:9300}{ml.machine_memory=1037410304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }
es_elasticsearch.3.x8ouukovhh80@master    | {"type": "server", "timestamp": "2019-04-25T16:28:48,818+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elk", "node.name": "0e7e4d96b31a",  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [192.168.99.100, 192.168.99.101] to bootstrap a cluster: have discovered []; discovery will continue using [192.168.99.100:9300, 192.168.99.101:9300] from hosts providers and [{0e7e4d96b31a}{Xs9966RjTEWvEbuj4-ySYA}{-eV4lvavSHq6JhoW0qWu6A}{10.255.0.48}{10.255.0.48:9300}{ml.machine_memory=1037410304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }

我猜测集群形成失败可能是由于网络配置错误引起的。我不知道如何修复它，尝试多次修改配置，但仍然失败。

- L.T

为什么不创建一个docker网络并将其分配给容器，这样更容易;) - Anuga

否则，这是一个很好的模板：https://github.com/jakubhajek/elasticsearch-docker-swarm - Anuga

6个回答

2

尝试手动指定所有特定的IP和绑定是棘手的，因为Swarm重叠网络。相反地，只需使您的ES节点可发现并让Swarm负责节点发现和通信即可。要使它们可发现，我们可以使用可预测的名称，例如Swarm节点主机名。

请尝试按照以下方式更改swarm.yml文件中的环境设置:

    environment:
    - network.host=0.0.0.0
    - discovery.seed_hosts=elasticsearch #Service name, to let Swarm handle discovery
    - cluster.initial_master_nodes=master,cluster,cluster2 #Swarm nodes host names
    - node.name={{.Node.Hostname}} #To create a predictable node name

当然，前提是我们已经知道了集群节点的主机名，就像你在上面的截图中指出的那样。如果不知道这些值，我们就没有办法得到一个可以预测的节点名称集合。在这种情况下，您可以创建一个ES节点条目，并使用特定的节点名称，然后再创建另一个条目，引用第一个条目的节点名称作为cluster.initial_master_nodes。 "Original Answer"的翻译是"最初的回答"。

- Andrew Cachia

1

使用dnsrr模式而不需要端口。使用nginx暴露elasticsearch；）请查看我的docker-compose.yml。

- Ahmet Vehbi Olgaç

1

这太棒了，非常感谢！不过，作为 StackOverflow 的答案，如果相关信息被复制到这里会更好。 - Ross Presser

1

根据我的经验，https://github.com/shazChaudhry/docker-elastic 运行得非常完美，只需从整个 repo 下载一个文件即可。我下载了 https://github.com/shazChaudhry/docker-elastic/blob/master/docker-compose.yml 并删除了 logstash 部分，因为我不需要。然后将以下内容添加到 .bashrc 中。

export ELASTICSEARCH_HOST=$(hostname)
export ELASTICSEARCH_PASSWORD=foobar
export ELASTICSEARCH_USERNAME=elastic
export ELASTIC_VERSION=7.4.2
export INITIAL_MASTER_NODES=$ELASTICSEARCH_HOST

执行 docker stack deploy --compose-file docker-compose.yml elastic 命令可以生效。

- chx

0

Elasticsearch 8.5.0 答案。

根据我的需求，我不想添加反向代理/负载均衡器，但是我希望在运行 Elasticsearch 副本的 swarm 节点上公开端口 9200（仅使用 swarm），以便外部客户端可以访问 Elasticsearch REST API。因此，我使用了 endpoint 模式 dnsrr (ref)，并在副本运行的主机上公开了端口 9200。

如果您不需要公开端口 9200（即，在 swarm 外部将不会连接到 elasticsearch 副本），请从 elasticsearch 服务的 ports: 配置中删除它。

我还只希望 elasticsearch 副本在我的 swarm 节点子集（其中的 3 个）上运行。我在这三个节点上创建了 docker 节点标签 elasticsearch。然后，mode: global 和约束条件 node.labels.elasticsearch==True 将确保每个节点上运行 1 个副本。

我也在这 3 个节点之一上运行 kibana：由于端口 5601 在 swarm 的入口覆盖网络上公开，因此 swarm 可以选择其中一个。

你很可能需要编辑的行用 ###### 标记。

# docker network create -d overlay --attachable elastic-net

# cat elastic-stack-env
#!/bin/bash
export STACK_VERSION=8.5.0   # Elasticsearch and Kibana version
export ES_PORT=9200          # port to expose Elasticsearch HTTP API to the host
export KIBANA_PORT=5601      # port to expose Kibana to the host

read -p "Enter elastic user password: " ELASTIC_PASSWORD
read -p "Enter kibana_system user password: " KIBANA_PASSWORD
export KIBANA_URL=https://kibana.my-domain.com:$KIBANA_PORT   #######
export SHARED_DIR=/some/nfs/or/shared/storage/elastic         #######
export KIBANA_SSL_KEY_PATH=config/certs/kibana.key
export KIBANA_SSL_CERT_PATH=config/certs/kibana.crt
export ELASTIC_NODES=swarm_node1,swarm_node2,swarm_node3      #######
  # ELASTIC_NODES must match what docker reports from {{.Node.Hostname}}
export KIBANA_SSL_CERT_AUTH_PATH=config/certs/My_Root_CA.crt  #######
export CLUSTER_NAME=docker-cluster
export MEM_LIMIT=4294967296  # 4 GB; increase or decrease based on the available host memory (in bytes)

# cat elastic-stack.yml
version: "3.8"
services:
  elasticsearch:
    image: localhost:5000/elasticsearch:${STACK_VERSION:?}  ####### I have a local registry
    deploy:
      endpoint_mode: dnsrr
      mode: global  # but note constraints below
      placement:
        constraints:
          - node.labels.elasticsearch==True
      resources:
        limits:
          memory:
            ${MEM_LIMIT}
    dns: 127.0.0.11  # use docker DNS only (may not be required)
    networks:
      - elastic-net
    volumes:
      - ${SHARED_DIR:?}/certs:/usr/share/elasticsearch/config/certs
      - /path/to/some/local/storage/elasticsearch:/usr/share/elasticsearch/data
    ports:  ##### remove if nothing outside of swarm needs to access port 9200
      - target: 9200
        published: ${ES_PORT}  # we publish this port so that external clients can access the ES REST API
        protocol: tcp
        mode: host  # required when using dnsrr
    environment:    # https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
                    # https://www.elastic.co/guide/en/elasticsearch/reference/master/docker.html#docker-configuration-methods
      - node.name={{.Node.Hostname}}  # see Andrew Cachia's answer
      - cluster.name=${CLUSTER_NAME}
      - discovery.seed_hosts=elasticsearch             # use service name here, since (docker's) DNS is used:
        # https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#unicast.hosts
      - cluster.initial_master_nodes=${ELASTIC_NODES}  # use node.names here
        # https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#initial_master_nodes
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
      - xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=basic
    healthcheck:
      test:
        [ "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
    logging:        # we use rsyslog
      driver: syslog
      options:
        syslog-facility: "local2"

  kibana:
    # this service depends on the setup service (defined below), but docker stack has no
    # way to specify dependencies, but more importantly, there's been a move away from this:
    # https://dev59.com/qFYN5IYBdhLWcg3wQWTJ#47714157
    image: localhost:5000/kibana:${STACK_VERSION:?}    ######
    hostname: kibana
    deploy:
      placement:
        constraints:
          - node.labels.elasticsearch==True  # run KB on any one of the ES nodes
      resources:
         limits:
          memory:
            ${MEM_LIMIT}
    dns: 127.0.0.11  # use docker DNS only (may not be required)
    networks:
      - elastic-net
    volumes:
      - ${SHARED_DIR:?}/kibana:/usr/share/kibana/data
      - ${SHARED_DIR:?}/certs:/usr/share/kibana/config/certs
    ports:
      - ${KIBANA_PORT}:5601
    environment:  # https://www.elastic.co/guide/en/kibana/master/settings.html
                  # https://www.elastic.co/guide/en/kibana/master/docker.html#environment-variable-config
                  # CAPS_WITH_UNDERSCORES must be used with Kibana
      - SERVER_NAME=kibana
      - ELASTICSEARCH_HOSTS=["https://elasticsearch:9200"]
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - SERVER_PUBLICBASEURL=${KIBANA_URL}
      # if you don't want to use https/TLS with Kibana, comment-out
      # the next four lines
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_KEY=${KIBANA_SSL_KEY_PATH}
      - SERVER_SSL_CERTIFICATE=${KIBANA_SSL_CERT_PATH}
      - SERVER_SSL_CERTIFICATEAUTHORITIES=${KIBANA_SSL_CERT_AUTH_PATH}
      - TELEMETRY_OPTIN=false
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -sIk https://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
    logging:
      driver: syslog
      options:
        syslog-facility: "local2"

  setup:
    image: localhost:5000/elasticsearch:${STACK_VERSION:?}   #######
    deploy:
      placement:
        constraints:
          - node.labels.elasticsearch==True
      restart_policy:  # https://docs.docker.com/compose/compose-file/compose-file-v3/#restart_policy
        condition: none
    volumes:
      - ${SHARED_DIR:?}/certs:/usr/share/elasticsearch/config/certs
    dns: 127.0.0.11  # use docker DNS only (may not be required)
    networks:
      - elastic-net
    command: >
      bash -c '
        until curl -s --cacert config/certs/ca/ca.crt https://elasticsearch:9200 | grep -q "missing authentication credentials"
        do
          echo "waiting 30 secs for Elasticsearch availability..."
          sleep 30
        done
        echo "setting kibana_system password"
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://elasticsearch:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"
        do
          echo "waiting 10 secs before trying to set password again..."
          sleep 10
        done
        echo "done"
      '
    logging:
      driver: syslog
      options:
        syslog-facility: "local2"

networks:
  elastic-net:
    external: true

部署：

# . ./elastic-stack-env
# docker stack deploy -c elastic-stack.yml elastic
# # ... after Kibana comes up, you can remove the setup service if you want:
# docker service rm elastic_setup

以下是我创建 Elasticsearch CA 和证书的步骤：

# cat elastic-certs.yml
version: "3.8"
services:
  setup:
    image: localhost:5000/elasticsearch:${STACK_VERSION:?}   #######
    volumes:
      - ${SHARED_DIR:?}/certs:/usr/share/elasticsearch/config/certs
    user: "0:0"
    command: >
      bash -c '
        if [ ! -f certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: elasticsearch\n"\
          "    dns:\n"\
          "      - elasticsearch\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
          echo "Setting file permissions"
          chown -R root:root config/certs;
          find . -type d -exec chmod 750 \{\} \;;
          find . -type f -exec chmod 640 \{\} \;;
        fi;
        sleep infinity
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/elasticsearch/elasticsearch.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

# . ./elastic-stack-env
# docker stack deploy -c elastic-certs.yml elastic-certs
# # ... ensure files are created under $SHARED_DIR/certs, then
# docker stack rm elastic-certs

如何创建Kibana证书超出了本问题的范围。

我运行一个Fluent Bit swarm服务（mode：global，docker网络elastic-net），将日志发送到elasticsearch服务。虽然超出了本问题的范围，但以下是关键配置：

[OUTPUT]
    name          es
    match         <whatever is appropriate for you here>
    host          elasticsearch
    port          9200
    index         my-index-default
    http_user     fluentbit
    http_passwd   ${FLUENTBIT_PASSWORD}
    tls           on
    tls.ca_file   /certs/ca/ca.crt
    tls.crt_file  /certs/elasticsearch/elasticsearch.crt
    tls.key_file  /certs/elasticsearch/elasticsearch.key
    retry_limit   false
    suppress_type_name on
#   trace_output  on

主机elasticsearch将由Docker的DNS服务器解析为elasticsearch副本的三个IP地址，因此不存在单点故障。

- Mark Rajcok

0

我从Ahmet Vehbi Olgaç的docker-compose.yml中获得的想法，对我很有用：

使用 deployment / mode: global。这将导致Swarm在每个配置为此的节点上为每个Swarm工作器部署一个副本。
使用 deployment / endpoint_mode: dnsrr。这将允许Swarm中的所有容器通过服务名称访问节点。
使用 hostname: {{.Node.Hostname}} 或类似的基于模板的表达式。这确保了每个部署的容器都有唯一的名称。
使用 environment / node.name={{.Node.Hostname}}。同样，您可以变化模式。重点是每个ES节点都应该获得唯一的名称。
使用 cluster.initial_master_nodes=*hostname1*,*hostname2*,...。假设您知道Docker工作机器的主机名。使用您在步骤3中使用的任何模式，但替换整个主机名，并包括所有主机名。

如果您不知道自己的主机名，可以像Andrew Cachia的答案建议的那样：设置一个容器（不要复制它）仅充当主种子，并给它一个可预测的主机名，然后让所有其他节点将其称为主种子。但是，这会引入单点故障。

- Ross Presser

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- Ahmet Vehbi Olgaç · Accepted Answer

试一试，这个可以工作 :) docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    hostname: "{{.Node.Hostname}}"
    environment:
      - node.name={{.Node.Hostname}}
      - cluster.name=my-cluster
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=node1,node2,node3
      - node.ml=false
      - xpack.ml.enabled=false
      - xpack.monitoring.enabled=false
      - xpack.security.enabled=false
      - xpack.watcher.enabled=false
      - bootstrap.memory_lock=false
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    deploy:
      mode: global
      endpoint_mode: dnsrr
      resources:
        limits:
          memory: 4G
  nginx:
    image: nginx:1.17.1-alpine
    ports:
      - 9200:9200
    deploy:
      mode: global
    command: |
      /bin/sh -c "echo '
      user nobody nogroup;
      worker_processes auto;
      events {
        worker_connections 1024;
      }
      http {
        client_max_body_size 4g;
        resolver 127.0.0.11 ipv6=off;
        server {
          listen *:9200;
          location / {
            proxy_set_header Connection keep-alive;
            set $$url http://elasticsearch:9200;
            proxy_pass $$url;
            proxy_set_header  Host $$http_host;
            proxy_set_header  X-Real-IP $$remote_addr;
            proxy_set_header  X-Forwarded-For $$proxy_add_x_forwarded_for;
          }
        }
      }' | tee /etc/nginx/nginx.conf && nginx -t && nginx -g 'daemon off;'"

volumes:
  elasticsearch-data: