Hashicorp Vault em Alta Disponibilidade com Consul

Consul

O HashiCorp Consul foi criado em 2014 como um DNS-SD (DNS based Service Discovery) para resolver o problema de comunicação entre microsserviços, mas ele também possui outras funcionalidades como health checking e armazenamento de chave-valor (KV Store).

Cada nó possui um agente de monitoramento instalado que é responsável por enviar, de forma segura, informações do nó a um ou mais servidores.

Os servidores mantêm um catálogo com as informações dos nós, como quais serviços estão disponíveis em cada um e seus status.
Para alta disponibilidade, é recomendado ter de 3 a 5 servidores. Um deles será eleito como o líder.

Configuração manual

Para entender a arquitetura em detalhes, é interessante realizar a instalação passo a passo. Mais adiante será mostrado como realizar as configurações usando Docker-compose.

O processo de instalação do Consul e do Vault está bem detalhado na documentação oficial.

De forma resumida o passo a passo é o seguinte:

  1. Inicie 5 máquinas com Linux

    Pode ser em alguma cloud ou localmente. Três serão para o servidor Consul e duas para o Vault.
    Na versão gratuita, apenas um nó do Vault fica ativo; os outros nós ficam em standby redirecionando os requests para o nó ativo. Se o nó que estava ativo fica indisponível, outro nó é eleito como o nó ativo.
    O Consul também será instalado nas máquinas do Vault, porém em modo cliente.

  2. Defina os hostnames

    (vault-01)# echo "vault-01" > /etc/hostname
    (vault-02)# echo "vault-02" > /etc/hostname
    (consul-01)# echo "consul-01" > /etc/hostname
    (consul-02)# echo "consul-02" > /etc/hostname
    (consul-03)# echo "consul-03" > /etc/hostname
    
  3. Instale o Consul em todas as máquinas

    O site oficial possui instruções de instalação nas distribuições mais populares.

    No Arch Linux seria o seguinte:

    # pacman -S consul
    
  4. Edite o arquivo de configuração dos servidores

    # cat /etc/consul.d/consul_s1.json
    
    {
      "server": true,
      "node_name": "$NODE_NAME",
      "datacenter": "dc1",
      "data_dir": "$CONSUL_DATA_PATH",
      "bind_addr": "0.0.0.0",
      "client_addr": "0.0.0.0",
      "advertise_addr": "$ADVERTISE_ADDR",
      "bootstrap_expect": 3,
      "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"],
      "ui": true,
      "log_level": "DEBUG",
      "enable_syslog": true,
      "acl_enforce_version_8": false
    }
    
  5. Edite o arquivo de configuração dos clientes

    # cat /etc/consul.d/consul_c1.json
    
    {
      "server": false,
      "datacenter": "dc1",
      "node_name": "$NODE_NAME",
      "data_dir": "$CONSUL_DATA_PATH",
      "bind_addr": "$BIND_ADDR",
      "client_addr": "127.0.0.1",
      "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"],
      "log_level": "DEBUG",
      "enable_syslog": true,
      "acl_enforce_version_8": false
    }
    
  6. Edite o systemd unit file dos servidores

    # cat /etc/systemd/system/consul.service
    ### BEGIN INIT INFO
    # Provides:          consul
    # Required-Start:    $local_fs $remote_fs
    # Required-Stop:     $local_fs $remote_fs
    # Default-Start:     2 3 4 5
    # Default-Stop:      0 1 6
    # Short-Description: Consul agent
    # Description:       Consul service discovery framework
    ### END INIT INFO
    
    [Unit]
    Description=Consul server agent
    Requires=network-online.target
    After=network-online.target
    
    [Service]
    User=consul
    Group=consul
    PIDFile=/var/run/consul/consul.pid
    PermissionsStartOnly=true
    ExecStartPre=-/bin/mkdir -p /var/run/consul
    ExecStartPre=/bin/chown -R consul:consul /var/run/consul
    ExecStart=/usr/local/bin/consul agent \
        -config-file=/usr/local/etc/consul/consul_s1.json \
        -pid-file=/var/run/consul/consul.pid
    ExecReload=/bin/kill -HUP $MAINPID
    KillMode=process
    KillSignal=SIGTERM
    Restart=on-failure
    RestartSec=42s
    
    [Install]
    WantedBy=multi-user.target
    
  7. Edite o systemd unit file dos clientes

    ### BEGIN INIT INFO
    # Provides:          consul
    # Required-Start:    $local_fs $remote_fs
    # Required-Stop:     $local_fs $remote_fs
    # Default-Start:     2 3 4 5
    # Default-Stop:      0 1 6
    # Short-Description: Consul agent
    # Description:       Consul service discovery framework
    ### END INIT INFO
    
    [Unit]
    Description=Consul client agent
    Requires=network-online.target
    After=network-online.target
    
    [Service]
    User=consul
    Group=consul
    PIDFile=/var/run/consul/consul.pid
    PermissionsStartOnly=true
    ExecStartPre=-/bin/mkdir -p /var/run/consul
    ExecStartPre=/bin/chown -R consul:consul /var/run/consul
    ExecStart=/usr/local/bin/consul agent \
        -config-file=/usr/local/etc/consul/consul_c1.json \
        -pid-file=/var/run/consul/consul.pid
    ExecReload=/bin/kill -HUP $MAINPID
    KillMode=process
    KillSignal=SIGTERM
    Restart=on-failure
    RestartSec=42s
    
    [Install]
    WantedBy=multi-user.target
    
  8. Verifique se todos os serviços estão rodando

    # systemctl daemon-reload
    # systemctl start consul
    # systemctl status consul
    
    $ consul members
    $ consul operator raft list-peers
    

    Um dos servidores deve estar como líder.

  9. Instale o Vault nas duas VMs

    # pacman -S vault
    
  10. Edite o arquivo de configuração do Vault

    # cat /etc/vault/vault_s1.hcl
    
    listener "tcp" {
      address          = "127.0.0.1:8200"
      cluster_address  = "127.0.0.1:8201"
      tls_disable      = "true"
    }
    
    storage "consul" {
      address = "127.0.0.1:8500"
      path    = "vault/"
    }
    
    api_addr =  "$API_ADDR"
    cluster_addr = "$CLUSTER_ADDR"
    
  11. Edite o systemd unit file dos Vaults

    # cat /etc/systemd/system/vault.service
    
    ### BEGIN INIT INFO
    # Provides:          consul
    # Required-Start:    $local_fs $remote_fs
    # Required-Stop:     $local_fs $remote_fs
    # Default-Start:     2 3 4 5
    # Default-Stop:      0 1 6
    # Short-Description: Consul agent
    # Description:       Consul service discovery framework
    ### END INIT INFO
    
    [Unit]
    Description=Consul client agent
    Requires=network-online.target
    After=network-online.target
    
    [Service]
    User=consul
    Group=consul
    PIDFile=/var/run/consul/consul.pid
    PermissionsStartOnly=true
    ExecStartPre=-/bin/mkdir -p /var/run/consul
    ExecStartPre=/bin/chown -R consul:consul /var/run/consul
    ExecStart=/usr/local/bin/consul agent \
        -config-file=/usr/local/etc/consul/consul_c1.json \
        -pid-file=/var/run/consul/consul.pid
    ExecReload=/bin/kill -HUP $MAINPID
    KillMode=process
    KillSignal=SIGTERM
    Restart=on-failure
    RestartSec=42s
    
    [Install]
    WantedBy=multi-user.target
    
  12. Verifique se todos os serviços estão rodando

    # systemctl daemon-reload
    # systemctl start vault
    # systemctl status vault
    
  13. Inicialize o Vault

    $ vault operator init
    

    Salve as chaves e o root token em um local seguro.

  14. Unseal

    $ vault operator unseal <unseal_key_1>
    $ vault operator unseal <unseal_key_2>
    $ vault operator unseal <unseal_key_3>
    

    Daqui em diante, podemos seguir como detalhado em outro artigo.

Configuração via docker-compose

$ consul keygen
LvqkL8MCukmz4/Z6n7v/38XSVh9jjbNoDXx5dsFd6Nw=
$ consul tls ca create
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem
$ consul tls cert create -server -dc dc1    ### -additional-dnsname=consul.server1
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem
$ consul tls cert create -server -dc dc1
==> Saved dc1-server-consul-1.pem
==> Saved dc1-server-consul-1-key.pem
$ consul tls cert create -server -dc dc1
==> Saved dc1-server-consul-2.pem
==> Saved dc1-server-consul-2-key.pem
$ consul tls cert create -client -dc dc1 -additional-dnsname=consul-c1
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem
$ consul tls cert create -client -dc dc1 -additional-dnsname=consul-c2
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-1.pem
==> Saved dc1-client-consul-1-key.pem
$ mv dc1-server-consul-0.pem Pasta/consul/config/server1/certs/server.pem
$ mv dc1-server-consul-1.pem Pasta/consul/config/server2/certs/server.pem
$ mv dc1-server-consul-2.pem Pasta/consul/config/server3/certs/server.pem
$ cp dc1-client-consul-0.pem Pasta/consul/config/client1/certs/client.pem
$ cp dc1-client-consul-1.pem Pasta/consul/config/client2/certs/client.pem
$ mv dc1-client-consul-0.pem Pasta/vault/config/server1/consul_tls/certs/client.pem
$ mv dc1-client-consul-1.pem Pasta/vault/config/server2/consul_tls/certs/client.pem
$ mv dc1-server-consul-0-key.pem Pasta/consul/config/server1/private/server-key.pem
$ mv dc1-server-consul-1-key.pem Pasta/consul/config/server2/private/server-key.pem
$ mv dc1-server-consul-2-key.pem Pasta/consul/config/server3/private/server-key.pem
$ cp dc1-client-consul-0-key.pem Pasta/consul/config/client1/private/client-key.pem
$ cp dc1-client-consul-1-key.pem Pasta/consul/config/client2/private/client-key.pem
$ mv dc1-client-consul-0-key.pem Pasta/vault/config/server1/consul_tls/private/client-key.pem
$ mv dc1-client-consul-1-key.pem Pasta/vault/config/server2/consul_tls/private/client-key.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server1/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server2/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server3/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/client1/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/client2/certs/root.pem
$ cp consul-agent-ca.pem Pasta/vault/config/server1/consul_tls/certs/root.pem
$ cp consul-agent-ca.pem Pasta/vault/config/server2/consul_tls/certs/root.pem

Certificado auto assinado para o Vault:

$ cat selfsign_s1.cfr
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = BR
ST = state
L =  city
O = company
CN = *

[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
DNS.2 = s1.vault
DNS.3 = host.docker.internal
IP.1 = 172.16.238.151
IP.3 = 127.0.0.1
$ cat selfsign_s2.cfr
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = BR
ST = state
L =  city
O = company
CN = *

[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
DNS.2 = s2.vault
DNS.3 = host.docker.internal
IP.1 = 172.16.238.152
IP.2 = 127.0.0.1
$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout server-key.pem -out server.pem -config selfsign_s1.cfr -days 9999
$ mv server.pem Pasta/vault/config/server1/tls/certs/server.pem
$ mv server-key.pem Pasta/vault/config/server1/tls/private/server-key.pem
$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout server-key.pem -out server.pem -config selfsign_s2.cfr -days 9999
$ mv server.pem Pasta/vault/config/server2/tls/certs/server.pem
$ mv server-key.pem Pasta/vault/config/server2/tls/private/server-key.pem
$ cat docker-compose.yaml
version: "3.8"

services:
  consul_s1:
    container_name: consul.server1
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/server_agent.json"]
    volumes:
      - ./consul/config/server1:/consul/config
      - ./consul/data/data_s1:/consul/data
    restart: unless-stopped
    ports:
      - "8090:8080"
      - "8300:8300"
      - "8500:8500"
      - "8600:8600/udp"
    networks:
      internal_net:
        ipv4_address: 172.16.238.53

  consul_s2:
    container_name: consul.server2
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/server_agent.json"]
    volumes:
      - ./consul/config/server2:/consul/config
      - ./consul/data/data_s2:/consul/data
    restart: unless-stopped
    ports:
      - "8091:8080"
      - "8310:8300"
      - "8510:8500"
      - "8610:8600/udp"
    networks:
      internal_net:
        ipv4_address: 172.16.238.54

  consul_s3:
    container_name: consul.server3
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/server_agent.json"]
    volumes:
      - ./consul/config/server3:/consul/config
      - ./consul/data/data_s3:/consul/data
    restart: unless-stopped
    ports:
      - "8092:8080"
      - "8320:8300"
      - "8520:8500"
      - "8620:8600/udp"
    networks:
      internal_net:
        ipv4_address: 172.16.238.55

  consul_c1:
    container_name: consul.client1
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/client_agent.json"]
    volumes:
      - ./consul/config/client1:/consul/config
      - ./consul/data/data_c1:/consul/data
    restart: unless-stopped
    ports:
      - "8093:8080"
      - "8330:8300"
      - "8530:8500"
      - "8630:8600/udp"
    depends_on:
      - consul_s1
    networks:
      internal_net:
        ipv4_address: 172.16.238.201
        aliases:
          - consul-c1

  consul_c2:
    container_name: consul.client2
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/client_agent.json"]
    volumes:
      - ./consul/config/client2:/consul/config
      - ./consul/data/data_c2:/consul/data
    restart: unless-stopped
    ports:
      - "8094:8080"
      - "8340:8300"
      - "8540:8500"
      - "8640:8600/udp"
    depends_on:
      - consul_s2
    networks:
      internal_net:
        ipv4_address: 172.16.238.202
        aliases:
          - consul-c2

  vault_s1:
    container_name: vault.server1
    image: vault:latest
    ports:
      - "9200:8200"
    expose:
      - "8500"
    volumes:
      - ./vault/config/server1:/vault/config
      - ./ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
      - ./vault/logs_s1:/vault/logs
    cap_add:
      - IPC_LOCK
    command: ["server", "-log-level=info"]
    environment:
      - VAULT_LOCAL_CONFIG={"ui":"true", "backend":{"consul":{"address":"consul-c1:8080",
        "path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/consul_tls/certs/root.pem",
        "tls_cert_file":"/vault/config/consul_tls/certs/client.pem",
        "tls_key_file":"/vault/config/consul_tls/private/client-key.pem", "tls_skip_verify":0}},
        "listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201",
        "tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem",
        "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}},
        "api_addr":"http://s1.vault:8200", "cluster_addr":"https://s1.vault:8201"}
      - VAULT_SKIP_VERIFY=false
    depends_on:
      - consul_c1
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      internal_net:
        ipv4_address: 172.16.238.151
        aliases:
          - s1.vault
    restart: unless-stopped

  vault_s2:
    container_name: vault.server2
    image: vault:latest
    ports:
      - "9210:8200"
    expose:
      - "8500"
    volumes:
      - ./vault/config/server2:/vault/config
      - ./ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
      - ./vault/logs_s2:/vault/logs
    cap_add:
      - IPC_LOCK
    command: ["server", "-log-level=info"]
    environment:
      - VAULT_LOCAL_CONFIG={"ui":"true", "backend":{"consul":{"address":"consul-c2:8080",
        "path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/consul_tls/certs/root.pem",
        "tls_cert_file":"/vault/config/consul_tls/certs/client.pem",
        "tls_key_file":"/vault/config/consul_tls/private/client-key.pem", "tls_skip_verify":0}},
        "listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201",
        "tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem",
        "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}},
        "api_addr":"http://s2.vault:8200", "cluster_addr":"https://s2.vault:8201"}
      - VAULT_SKIP_VERIFY=false
    depends_on:
      - consul_c2
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      internal_net:
        ipv4_address: 172.16.238.152
        aliases:
          - s2.vault
    restart: unless-stopped

networks:
  internal_net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.16.238.0/24
$ docker-compose -f docker-compose.yaml up -d --remove-orphans

Endereços das UIs:


Julio Batista Silva
Julio Batista Silva
Engenheiro de Dados

Eu sou um engenheiro de computação apaixonado por ciência, tecnologia, fotografia e idiomas. Atualmente trabalhando como Engenheiro de Dados na Alemanha.

comments powered by Disqus