Hashicorp Vault em Alta Disponibilidade com Consul

Julio Batista Silva
Apr 19, 2021 7 min read

Consul

O HashiCorp Consul foi criado em 2014 como um DNS-SD (DNS based Service Discovery) para resolver o problema de comunicação entre microsserviços, mas ele também possui outras funcionalidades como health checking e armazenamento de chave-valor (KV Store).

Cada nó possui um agente de monitoramento instalado que é responsável por enviar, de forma segura, informações do nó a um ou mais servidores.

Os servidores mantêm um catálogo com as informações dos nós, como quais serviços estão disponíveis em cada um e seus status.
Para alta disponibilidade, é recomendado ter de 3 a 5 servidores. Um deles será eleito como o líder.

Configuração manual

Para entender a arquitetura em detalhes, é interessante realizar a instalação passo a passo. Mais adiante será mostrado como realizar as configurações usando Docker-compose.

O processo de instalação do Consul e do Vault está bem detalhado na documentação oficial.

De forma resumida o passo a passo é o seguinte:

1. Inicie 5 máquinas com Linux

Pode ser em alguma cloud ou localmente. Três serão para o servidor Consul e duas para o Vault.
Na versão gratuita, apenas um nó do Vault fica ativo; os outros nós ficam em standby redirecionando os requests para o nó ativo. Se o nó que estava ativo fica indisponível, outro nó é eleito como o nó ativo.
O Consul também será instalado nas máquinas do Vault, porém em modo cliente.

2. Defina os hostnames

(vault-01)# echo "vault-01" > /etc/hostname
(vault-02)# echo "vault-02" > /etc/hostname
(consul-01)# echo "consul-01" > /etc/hostname
(consul-02)# echo "consul-02" > /etc/hostname
(consul-03)# echo "consul-03" > /etc/hostname

3. Instale o Consul em todas as máquinas

O site oficial possui instruções de instalação nas distribuições mais populares.
No Arch Linux seria o seguinte:

# pacman -S consul

4. Edite o arquivo de configuração dos servidores

# cat /etc/consul.d/consul_s1.json

{
  "server": true,
  "node_name": "$NODE_NAME",
  "datacenter": "dc1",
  "data_dir": "$CONSUL_DATA_PATH",
  "bind_addr": "0.0.0.0",
  "client_addr": "0.0.0.0",
  "advertise_addr": "$ADVERTISE_ADDR",
  "bootstrap_expect": 3,
  "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"],
  "ui": true,
  "log_level": "DEBUG",
  "enable_syslog": true,
  "acl_enforce_version_8": false
}

5. Edite o arquivo de configuração dos clientes

# cat /etc/consul.d/consul_c1.json

{
  "server": false,
  "datacenter": "dc1",
  "node_name": "$NODE_NAME",
  "data_dir": "$CONSUL_DATA_PATH",
  "bind_addr": "$BIND_ADDR",
  "client_addr": "127.0.0.1",
  "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"],
  "log_level": "DEBUG",
  "enable_syslog": true,
  "acl_enforce_version_8": false
}

6. Edite o systemd unit file dos servidores

# cat /etc/systemd/system/consul.service
### BEGIN INIT INFO
# Provides:          consul
# Required-Start:    $local_fs $remote_fs
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Consul agent
# Description:       Consul service discovery framework
### END INIT INFO

[Unit]
Description=Consul server agent
Requires=network-online.target
After=network-online.target

[Service]
User=consul
Group=consul
PIDFile=/var/run/consul/consul.pid
PermissionsStartOnly=true
ExecStartPre=-/bin/mkdir -p /var/run/consul
ExecStartPre=/bin/chown -R consul:consul /var/run/consul
ExecStart=/usr/local/bin/consul agent \
    -config-file=/usr/local/etc/consul/consul_s1.json \
    -pid-file=/var/run/consul/consul.pid
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

7. Edite o systemd unit file dos clientes

### BEGIN INIT INFO
# Provides:          consul
# Required-Start:    $local_fs $remote_fs
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Consul agent
# Description:       Consul service discovery framework
### END INIT INFO

[Unit]
Description=Consul client agent
Requires=network-online.target
After=network-online.target

[Service]
User=consul
Group=consul
PIDFile=/var/run/consul/consul.pid
PermissionsStartOnly=true
ExecStartPre=-/bin/mkdir -p /var/run/consul
ExecStartPre=/bin/chown -R consul:consul /var/run/consul
ExecStart=/usr/local/bin/consul agent \
    -config-file=/usr/local/etc/consul/consul_c1.json \
    -pid-file=/var/run/consul/consul.pid
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

8. Verifique se todos os serviços estão rodando

# systemctl daemon-reload
# systemctl start consul
# systemctl status consul

$ consul members
$ consul operator raft list-peers

Um dos servidores deve estar como líder.

9. Instale o Vault nas duas VMs

# pacman -S vault

10. Edite o arquivo de configuração do Vault

# cat /etc/vault/vault_s1.hcl

listener "tcp" {
  address          = "127.0.0.1:8200"
  cluster_address  = "127.0.0.1:8201"
  tls_disable      = "true"
}

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

api_addr =  "$API_ADDR"
cluster_addr = "$CLUSTER_ADDR"

11. Edite o systemd unit file dos Vaults

# cat /etc/systemd/system/vault.service

### BEGIN INIT INFO
# Provides:          consul
# Required-Start:    $local_fs $remote_fs
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Consul agent
# Description:       Consul service discovery framework
### END INIT INFO

[Unit]
Description=Consul client agent
Requires=network-online.target
After=network-online.target

[Service]
User=consul
Group=consul
PIDFile=/var/run/consul/consul.pid
PermissionsStartOnly=true
ExecStartPre=-/bin/mkdir -p /var/run/consul
ExecStartPre=/bin/chown -R consul:consul /var/run/consul
ExecStart=/usr/local/bin/consul agent \
    -config-file=/usr/local/etc/consul/consul_c1.json \
    -pid-file=/var/run/consul/consul.pid
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

12. Verifique se todos os serviços estão rodando

# systemctl daemon-reload
# systemctl start vault
# systemctl status vault

13. Inicialize o Vault

$ vault operator init

Salve as chaves e o root token em um local seguro.

14. Unseal

$ vault operator unseal <unseal_key_1>
$ vault operator unseal <unseal_key_2>
$ vault operator unseal <unseal_key_3>

Daqui em diante, podemos seguir como detalhado em outro artigo.

Configuração via docker-compose

$ consul keygen
LvqkL8MCukmz4/Z6n7v/38XSVh9jjbNoDXx5dsFd6Nw=

$ consul tls ca create
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem

$ consul tls cert create -server -dc dc1    ### -additional-dnsname=consul.server1
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem

$ consul tls cert create -server -dc dc1
==> Saved dc1-server-consul-1.pem
==> Saved dc1-server-consul-1-key.pem

$ consul tls cert create -server -dc dc1
==> Saved dc1-server-consul-2.pem
==> Saved dc1-server-consul-2-key.pem

$ consul tls cert create -client -dc dc1 -additional-dnsname=consul-c1
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem

$ consul tls cert create -client -dc dc1 -additional-dnsname=consul-c2
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-1.pem
==> Saved dc1-client-consul-1-key.pem

$ mv dc1-server-consul-0.pem Pasta/consul/config/server1/certs/server.pem
$ mv dc1-server-consul-1.pem Pasta/consul/config/server2/certs/server.pem
$ mv dc1-server-consul-2.pem Pasta/consul/config/server3/certs/server.pem

$ cp dc1-client-consul-0.pem Pasta/consul/config/client1/certs/client.pem
$ cp dc1-client-consul-1.pem Pasta/consul/config/client2/certs/client.pem
$ mv dc1-client-consul-0.pem Pasta/vault/config/server1/consul_tls/certs/client.pem
$ mv dc1-client-consul-1.pem Pasta/vault/config/server2/consul_tls/certs/client.pem

$ mv dc1-server-consul-0-key.pem Pasta/consul/config/server1/private/server-key.pem
$ mv dc1-server-consul-1-key.pem Pasta/consul/config/server2/private/server-key.pem
$ mv dc1-server-consul-2-key.pem Pasta/consul/config/server3/private/server-key.pem

$ cp dc1-client-consul-0-key.pem Pasta/consul/config/client1/private/client-key.pem
$ cp dc1-client-consul-1-key.pem Pasta/consul/config/client2/private/client-key.pem
$ mv dc1-client-consul-0-key.pem Pasta/vault/config/server1/consul_tls/private/client-key.pem
$ mv dc1-client-consul-1-key.pem Pasta/vault/config/server2/consul_tls/private/client-key.pem

$ cp consul-agent-ca.pem Pasta/consul/config/server1/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server2/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server3/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/client1/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/client2/certs/root.pem

$ cp consul-agent-ca.pem Pasta/vault/config/server1/consul_tls/certs/root.pem
$ cp consul-agent-ca.pem Pasta/vault/config/server2/consul_tls/certs/root.pem

Certificado auto assinado para o Vault:

$ cat selfsign_s1.cfr

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = BR
ST = state
L =  city
O = company
CN = *

[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
DNS.2 = s1.vault
DNS.3 = host.docker.internal
IP.1 = 172.16.238.151
IP.3 = 127.0.0.1

$ cat selfsign_s2.cfr

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = BR
ST = state
L =  city
O = company
CN = *

[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
DNS.2 = s2.vault
DNS.3 = host.docker.internal
IP.1 = 172.16.238.152
IP.2 = 127.0.0.1

$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout server-key.pem -out server.pem -config selfsign_s1.cfr -days 9999
$ mv server.pem Pasta/vault/config/server1/tls/certs/server.pem 
$ mv server-key.pem Pasta/vault/config/server1/tls/private/server-key.pem

$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout server-key.pem -out server.pem -config selfsign_s2.cfr -days 9999
$ mv server.pem Pasta/vault/config/server2/tls/certs/server.pem 
$ mv server-key.pem Pasta/vault/config/server2/tls/private/server-key.pem

$ cat docker-compose.yaml

version: '3.8'

services:
  consul_s1:
    container_name: consul.server1
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/server_agent.json"]
    volumes:
      - ./consul/config/server1:/consul/config
      - ./consul/data/data_s1:/consul/data
    restart: unless-stopped
    ports:
      - "8090:8080"
      - "8300:8300"
      - "8500:8500"
      - "8600:8600/udp"
    networks:
      internal_net:
        ipv4_address: 172.16.238.53

  consul_s2:
    container_name: consul.server2
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/server_agent.json"]
    volumes:
      - ./consul/config/server2:/consul/config
      - ./consul/data/data_s2:/consul/data
    restart: unless-stopped
    ports:
      - "8091:8080"
      - "8310:8300"
      - "8510:8500"
      - "8610:8600/udp"
    networks:
      internal_net:
        ipv4_address: 172.16.238.54

  consul_s3:
    container_name: consul.server3
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/server_agent.json"]
    volumes:
      - ./consul/config/server3:/consul/config
      - ./consul/data/data_s3:/consul/data
    restart: unless-stopped
    ports:
      - "8092:8080"
      - "8320:8300"
      - "8520:8500"
      - "8620:8600/udp"
    networks:
      internal_net:
        ipv4_address: 172.16.238.55

  consul_c1:
    container_name: consul.client1
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/client_agent.json"]
    volumes:
      - ./consul/config/client1:/consul/config
      - ./consul/data/data_c1:/consul/data
    restart: unless-stopped
    ports:
      - "8093:8080"
      - "8330:8300"
      - "8530:8500"
      - "8630:8600/udp"
    depends_on:
      - consul_s1
    networks:
      internal_net:
        ipv4_address: 172.16.238.201
        aliases:
          - consul-c1

  consul_c2:
    container_name: consul.client2
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/client_agent.json"]
    volumes:
      - ./consul/config/client2:/consul/config
      - ./consul/data/data_c2:/consul/data
    restart: unless-stopped
    ports:
      - "8094:8080"
      - "8340:8300"
      - "8540:8500"
      - "8640:8600/udp"
    depends_on:
      - consul_s2
    networks:
      internal_net:
        ipv4_address: 172.16.238.202
        aliases:
          - consul-c2

  vault_s1:
    container_name: vault.server1
    image: vault:latest
    ports:
      - "9200:8200"
    expose:
      - "8500"
    volumes:
      - ./vault/config/server1:/vault/config
      - ./ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
      - ./vault/logs_s1:/vault/logs
    cap_add:
      - IPC_LOCK
    command: ["server", "-log-level=info"]
    environment:
      - VAULT_LOCAL_CONFIG={"ui":"true", "backend":{"consul":{"address":"consul-c1:8080", "path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/consul_tls/certs/root.pem", "tls_cert_file":"/vault/config/consul_tls/certs/client.pem", "tls_key_file":"/vault/config/consul_tls/private/client-key.pem", "tls_skip_verify":0}}, "listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201", "tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem", "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}}, "api_addr":"http://s1.vault:8200", "cluster_addr":"https://s1.vault:8201"}
      - VAULT_SKIP_VERIFY=false
    depends_on:
      - consul_c1
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      internal_net:
        ipv4_address: 172.16.238.151
        aliases:
          - s1.vault
    restart: unless-stopped

  vault_s2:
    container_name: vault.server2
    image: vault:latest
    ports:
      - "9210:8200"
    expose:
      - "8500"
    volumes:
      - ./vault/config/server2:/vault/config
      - ./ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
      - ./vault/logs_s2:/vault/logs
    cap_add:
      - IPC_LOCK
    command: ["server", "-log-level=info"]
    environment:
      - VAULT_LOCAL_CONFIG={"ui":"true", "backend":{"consul":{"address":"consul-c2:8080", "path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/consul_tls/certs/root.pem", "tls_cert_file":"/vault/config/consul_tls/certs/client.pem", "tls_key_file":"/vault/config/consul_tls/private/client-key.pem", "tls_skip_verify":0}}, "listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201", "tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem", "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}}, "api_addr":"http://s2.vault:8200", "cluster_addr":"https://s2.vault:8201"}
      - VAULT_SKIP_VERIFY=false
    depends_on:
      - consul_c2
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      internal_net:
        ipv4_address: 172.16.238.152
        aliases:
          - s2.vault
    restart: unless-stopped

networks:
  internal_net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.16.238.0/24

$ docker-compose -f docker-compose.yaml up -d --remove-orphans

Endereços das UIs:

  • Vault 1: https://localhost:9200/ui
  • Vault 2: https://localhost:9210/ui
  • Consul 1: https://localhost:8090/ui
  • Consul 2: https://localhost:8091/ui
  • Consul 3: https://localhost:8092/ui
Julio Batista Silva
Julio Batista Silva
Data Engineer

I’m a computer engineer passionate about science, technology, photography, and languages. Currently working as a Data Engineer in Germany.

comments powered by Disqus