Hashicorp Vault em Alta Disponibilidade com Consul

April 19, 2021   

Consul

O HashiCorp Consul foi criado em 2014 como um DNS-SD (DNS based Service Discovery) para resolver o problema de comunicação entre microsserviços, mas ele também possui outras funcionalidades como health checking e armazenamento de chave-valor (KV Store).

Cada nó possui um agente de monitoramento instalado que é responsável por enviar, de forma segura, informações do nó a um ou mais servidores.

Os servidores mantêm um catálogo com as informações dos nós, como quais serviços estão disponíveis em cada um e seus status.
Para alta disponibilidade, é recomendado ter de 3 a 5 servidores. Um deles será eleito como o líder.

Configuração manual

Para entender a arquitetura em detalhes, é interessante realizar a instalação passo a passo. Mais adiante será mostrado como realizar as configurações usando Docker-compose.

O processo de instalação do Consul e do Vault está bem detalhado na documentação oficial.

De forma resumida o passo a passo é o seguinte:

1. Inicie 5 máquinas com Linux

Pode ser em alguma cloud ou localmente. Três serão para o servidor Consul e duas para o Vault.
Na versão gratuita, apenas um nó do Vault fica ativo; os outros nós ficam em standby redirecionando os requests para o nó ativo. Se o nó que estava ativo fica indisponível, outro nó é eleito como o nó ativo.
O Consul também será instalado nas máquinas do Vault, porém em modo cliente.

2. Defina os hostnames

(vault-01)# echo "vault-01" > /etc/hostname
(vault-02)# echo "vault-02" > /etc/hostname
(consul-01)# echo "consul-01" > /etc/hostname
(consul-02)# echo "consul-02" > /etc/hostname
(consul-03)# echo "consul-03" > /etc/hostname

3. Instale o Consul em todas as máquinas

O site oficial possui instruções de instalação nas distribuições mais populares.
No Arch Linux seria o seguinte:

# pacman -S consul

4. Edite o arquivo de configuração dos servidores

# cat /etc/consul.d/consul_s1.json
 1{
 2  "server": true,
 3  "node_name": "$NODE_NAME",
 4  "datacenter": "dc1",
 5  "data_dir": "$CONSUL_DATA_PATH",
 6  "bind_addr": "0.0.0.0",
 7  "client_addr": "0.0.0.0",
 8  "advertise_addr": "$ADVERTISE_ADDR",
 9  "bootstrap_expect": 3,
10  "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"],
11  "ui": true,
12  "log_level": "DEBUG",
13  "enable_syslog": true,
14  "acl_enforce_version_8": false
15}

5. Edite o arquivo de configuração dos clientes

# cat /etc/consul.d/consul_c1.json
 1{
 2  "server": false,
 3  "datacenter": "dc1",
 4  "node_name": "$NODE_NAME",
 5  "data_dir": "$CONSUL_DATA_PATH",
 6  "bind_addr": "$BIND_ADDR",
 7  "client_addr": "127.0.0.1",
 8  "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"],
 9  "log_level": "DEBUG",
10  "enable_syslog": true,
11  "acl_enforce_version_8": false
12}

6. Edite o systemd unit file dos servidores

 1# cat /etc/systemd/system/consul.service
 2### BEGIN INIT INFO
 3# Provides:          consul
 4# Required-Start:    $local_fs $remote_fs
 5# Required-Stop:     $local_fs $remote_fs
 6# Default-Start:     2 3 4 5
 7# Default-Stop:      0 1 6
 8# Short-Description: Consul agent
 9# Description:       Consul service discovery framework
10### END INIT INFO
11
12[Unit]
13Description=Consul server agent
14Requires=network-online.target
15After=network-online.target
16
17[Service]
18User=consul
19Group=consul
20PIDFile=/var/run/consul/consul.pid
21PermissionsStartOnly=true
22ExecStartPre=-/bin/mkdir -p /var/run/consul
23ExecStartPre=/bin/chown -R consul:consul /var/run/consul
24ExecStart=/usr/local/bin/consul agent \
25    -config-file=/usr/local/etc/consul/consul_s1.json \
26    -pid-file=/var/run/consul/consul.pid
27ExecReload=/bin/kill -HUP $MAINPID
28KillMode=process
29KillSignal=SIGTERM
30Restart=on-failure
31RestartSec=42s
32
33[Install]
34WantedBy=multi-user.target

7. Edite o systemd unit file dos clientes

 1### BEGIN INIT INFO
 2# Provides:          consul
 3# Required-Start:    $local_fs $remote_fs
 4# Required-Stop:     $local_fs $remote_fs
 5# Default-Start:     2 3 4 5
 6# Default-Stop:      0 1 6
 7# Short-Description: Consul agent
 8# Description:       Consul service discovery framework
 9### END INIT INFO
10
11[Unit]
12Description=Consul client agent
13Requires=network-online.target
14After=network-online.target
15
16[Service]
17User=consul
18Group=consul
19PIDFile=/var/run/consul/consul.pid
20PermissionsStartOnly=true
21ExecStartPre=-/bin/mkdir -p /var/run/consul
22ExecStartPre=/bin/chown -R consul:consul /var/run/consul
23ExecStart=/usr/local/bin/consul agent \
24    -config-file=/usr/local/etc/consul/consul_c1.json \
25    -pid-file=/var/run/consul/consul.pid
26ExecReload=/bin/kill -HUP $MAINPID
27KillMode=process
28KillSignal=SIGTERM
29Restart=on-failure
30RestartSec=42s
31
32[Install]
33WantedBy=multi-user.target

8. Verifique se todos os serviços estão rodando

# systemctl daemon-reload
# systemctl start consul
# systemctl status consul
$ consul members
$ consul operator raft list-peers

Um dos servidores deve estar como líder.

9. Instale o Vault nas duas VMs

# pacman -S vault

10. Edite o arquivo de configuração do Vault

# cat /etc/vault/vault_s1.hcl
 1listener "tcp" {
 2  address          = "127.0.0.1:8200"
 3  cluster_address  = "127.0.0.1:8201"
 4  tls_disable      = "true"
 5}
 6
 7storage "consul" {
 8  address = "127.0.0.1:8500"
 9  path    = "vault/"
10}
11
12api_addr =  "$API_ADDR"
13cluster_addr = "$CLUSTER_ADDR"

11. Edite o systemd unit file dos Vaults

# cat /etc/systemd/system/vault.service
 1### BEGIN INIT INFO
 2# Provides:          consul
 3# Required-Start:    $local_fs $remote_fs
 4# Required-Stop:     $local_fs $remote_fs
 5# Default-Start:     2 3 4 5
 6# Default-Stop:      0 1 6
 7# Short-Description: Consul agent
 8# Description:       Consul service discovery framework
 9### END INIT INFO
10
11[Unit]
12Description=Consul client agent
13Requires=network-online.target
14After=network-online.target
15
16[Service]
17User=consul
18Group=consul
19PIDFile=/var/run/consul/consul.pid
20PermissionsStartOnly=true
21ExecStartPre=-/bin/mkdir -p /var/run/consul
22ExecStartPre=/bin/chown -R consul:consul /var/run/consul
23ExecStart=/usr/local/bin/consul agent \
24    -config-file=/usr/local/etc/consul/consul_c1.json \
25    -pid-file=/var/run/consul/consul.pid
26ExecReload=/bin/kill -HUP $MAINPID
27KillMode=process
28KillSignal=SIGTERM
29Restart=on-failure
30RestartSec=42s
31
32[Install]
33WantedBy=multi-user.target

12. Verifique se todos os serviços estão rodando

# systemctl daemon-reload
# systemctl start vault
# systemctl status vault

13. Inicialize o Vault

$ vault operator init

Salve as chaves e o root token em um local seguro.

14. Unseal

$ vault operator unseal <unseal_key_1>
$ vault operator unseal <unseal_key_2>
$ vault operator unseal <unseal_key_3>

Daqui em diante, podemos seguir como detalhado em outro artigo.

Configuração via docker-compose

$ consul keygen
LvqkL8MCukmz4/Z6n7v/38XSVh9jjbNoDXx5dsFd6Nw=
$ consul tls ca create
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem
$ consul tls cert create -server -dc dc1    ### -additional-dnsname=consul.server1
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem
$ consul tls cert create -server -dc dc1
==> Saved dc1-server-consul-1.pem
==> Saved dc1-server-consul-1-key.pem
$ consul tls cert create -server -dc dc1
==> Saved dc1-server-consul-2.pem
==> Saved dc1-server-consul-2-key.pem
$ consul tls cert create -client -dc dc1 -additional-dnsname=consul-c1
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem
$ consul tls cert create -client -dc dc1 -additional-dnsname=consul-c2
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-1.pem
==> Saved dc1-client-consul-1-key.pem
$ mv dc1-server-consul-0.pem Pasta/consul/config/server1/certs/server.pem
$ mv dc1-server-consul-1.pem Pasta/consul/config/server2/certs/server.pem
$ mv dc1-server-consul-2.pem Pasta/consul/config/server3/certs/server.pem
$ cp dc1-client-consul-0.pem Pasta/consul/config/client1/certs/client.pem
$ cp dc1-client-consul-1.pem Pasta/consul/config/client2/certs/client.pem
$ mv dc1-client-consul-0.pem Pasta/vault/config/server1/consul_tls/certs/client.pem
$ mv dc1-client-consul-1.pem Pasta/vault/config/server2/consul_tls/certs/client.pem
$ mv dc1-server-consul-0-key.pem Pasta/consul/config/server1/private/server-key.pem
$ mv dc1-server-consul-1-key.pem Pasta/consul/config/server2/private/server-key.pem
$ mv dc1-server-consul-2-key.pem Pasta/consul/config/server3/private/server-key.pem
$ cp dc1-client-consul-0-key.pem Pasta/consul/config/client1/private/client-key.pem
$ cp dc1-client-consul-1-key.pem Pasta/consul/config/client2/private/client-key.pem
$ mv dc1-client-consul-0-key.pem Pasta/vault/config/server1/consul_tls/private/client-key.pem
$ mv dc1-client-consul-1-key.pem Pasta/vault/config/server2/consul_tls/private/client-key.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server1/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server2/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server3/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/client1/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/client2/certs/root.pem
$ cp consul-agent-ca.pem Pasta/vault/config/server1/consul_tls/certs/root.pem
$ cp consul-agent-ca.pem Pasta/vault/config/server2/consul_tls/certs/root.pem

Certificado auto assinado para o Vault:

$ cat selfsign_s1.cfr
 1[req]
 2distinguished_name = req_distinguished_name
 3x509_extensions = v3_req
 4prompt = no
 5
 6[req_distinguished_name]
 7C = BR
 8ST = state
 9L =  city
10O = company
11CN = *
12
13[v3_req]
14subjectKeyIdentifier = hash
15authorityKeyIdentifier = keyid,issuer
16basicConstraints = CA:TRUE
17subjectAltName = @alt_names
18
19[alt_names]
20DNS.1 = localhost
21DNS.2 = s1.vault
22DNS.3 = host.docker.internal
23IP.1 = 172.16.238.151
24IP.3 = 127.0.0.1
$ cat selfsign_s2.cfr
 1[req]
 2distinguished_name = req_distinguished_name
 3x509_extensions = v3_req
 4prompt = no
 5
 6[req_distinguished_name]
 7C = BR
 8ST = state
 9L =  city
10O = company
11CN = *
12
13[v3_req]
14subjectKeyIdentifier = hash
15authorityKeyIdentifier = keyid,issuer
16basicConstraints = CA:TRUE
17subjectAltName = @alt_names
18
19[alt_names]
20DNS.1 = localhost
21DNS.2 = s2.vault
22DNS.3 = host.docker.internal
23IP.1 = 172.16.238.152
24IP.2 = 127.0.0.1
$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout server-key.pem -out server.pem -config selfsign_s1.cfr -days 9999
$ mv server.pem Pasta/vault/config/server1/tls/certs/server.pem 
$ mv server-key.pem Pasta/vault/config/server1/tls/private/server-key.pem
$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout server-key.pem -out server.pem -config selfsign_s2.cfr -days 9999
$ mv server.pem Pasta/vault/config/server2/tls/certs/server.pem 
$ mv server-key.pem Pasta/vault/config/server2/tls/private/server-key.pem
$ cat docker-compose.yaml
  1version: '3.8'
  2
  3services:
  4  consul_s1:
  5    container_name: consul.server1
  6    image: consul:latest
  7    command: ["agent", "-config-file=/consul/config/server_agent.json"]
  8    volumes:
  9      - ./consul/config/server1:/consul/config
 10      - ./consul/data/data_s1:/consul/data
 11    restart: unless-stopped
 12    ports:
 13      - "8090:8080"
 14      - "8300:8300"
 15      - "8500:8500"
 16      - "8600:8600/udp"
 17    networks:
 18      internal_net:
 19        ipv4_address: 172.16.238.53
 20
 21  consul_s2:
 22    container_name: consul.server2
 23    image: consul:latest
 24    command: ["agent", "-config-file=/consul/config/server_agent.json"]
 25    volumes:
 26      - ./consul/config/server2:/consul/config
 27      - ./consul/data/data_s2:/consul/data
 28    restart: unless-stopped
 29    ports:
 30      - "8091:8080"
 31      - "8310:8300"
 32      - "8510:8500"
 33      - "8610:8600/udp"
 34    networks:
 35      internal_net:
 36        ipv4_address: 172.16.238.54
 37
 38  consul_s3:
 39    container_name: consul.server3
 40    image: consul:latest
 41    command: ["agent", "-config-file=/consul/config/server_agent.json"]
 42    volumes:
 43      - ./consul/config/server3:/consul/config
 44      - ./consul/data/data_s3:/consul/data
 45    restart: unless-stopped
 46    ports:
 47      - "8092:8080"
 48      - "8320:8300"
 49      - "8520:8500"
 50      - "8620:8600/udp"
 51    networks:
 52      internal_net:
 53        ipv4_address: 172.16.238.55
 54
 55  consul_c1:
 56    container_name: consul.client1
 57    image: consul:latest
 58    command: ["agent", "-config-file=/consul/config/client_agent.json"]
 59    volumes:
 60      - ./consul/config/client1:/consul/config
 61      - ./consul/data/data_c1:/consul/data
 62    restart: unless-stopped
 63    ports:
 64      - "8093:8080"
 65      - "8330:8300"
 66      - "8530:8500"
 67      - "8630:8600/udp"
 68    depends_on:
 69      - consul_s1
 70    networks:
 71      internal_net:
 72        ipv4_address: 172.16.238.201
 73        aliases:
 74          - consul-c1
 75
 76  consul_c2:
 77    container_name: consul.client2
 78    image: consul:latest
 79    command: ["agent", "-config-file=/consul/config/client_agent.json"]
 80    volumes:
 81      - ./consul/config/client2:/consul/config
 82      - ./consul/data/data_c2:/consul/data
 83    restart: unless-stopped
 84    ports:
 85      - "8094:8080"
 86      - "8340:8300"
 87      - "8540:8500"
 88      - "8640:8600/udp"
 89    depends_on:
 90      - consul_s2
 91    networks:
 92      internal_net:
 93        ipv4_address: 172.16.238.202
 94        aliases:
 95          - consul-c2
 96
 97  vault_s1:
 98    container_name: vault.server1
 99    image: vault:latest
100    ports:
101      - "9200:8200"
102    expose:
103      - "8500"
104    volumes:
105      - ./vault/config/server1:/vault/config
106      - ./ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
107      - ./vault/logs_s1:/vault/logs
108    cap_add:
109      - IPC_LOCK
110    command: ["server", "-log-level=info"]
111    environment:
112      - VAULT_LOCAL_CONFIG={"ui":"true", "backend":{"consul":{"address":"consul-c1:8080", "path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/consul_tls/certs/root.pem", "tls_cert_file":"/vault/config/consul_tls/certs/client.pem", "tls_key_file":"/vault/config/consul_tls/private/client-key.pem", "tls_skip_verify":0}}, "listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201", "tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem", "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}}, "api_addr":"http://s1.vault:8200", "cluster_addr":"https://s1.vault:8201"}
113      - VAULT_SKIP_VERIFY=false
114    depends_on:
115      - consul_c1
116    extra_hosts:
117      - "host.docker.internal:host-gateway"
118    networks:
119      internal_net:
120        ipv4_address: 172.16.238.151
121        aliases:
122          - s1.vault
123    restart: unless-stopped
124
125  vault_s2:
126    container_name: vault.server2
127    image: vault:latest
128    ports:
129      - "9210:8200"
130    expose:
131      - "8500"
132    volumes:
133      - ./vault/config/server2:/vault/config
134      - ./ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
135      - ./vault/logs_s2:/vault/logs
136    cap_add:
137      - IPC_LOCK
138    command: ["server", "-log-level=info"]
139    environment:
140      - VAULT_LOCAL_CONFIG={"ui":"true", "backend":{"consul":{"address":"consul-c2:8080", "path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/consul_tls/certs/root.pem", "tls_cert_file":"/vault/config/consul_tls/certs/client.pem", "tls_key_file":"/vault/config/consul_tls/private/client-key.pem", "tls_skip_verify":0}}, "listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201", "tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem", "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}}, "api_addr":"http://s2.vault:8200", "cluster_addr":"https://s2.vault:8201"}
141      - VAULT_SKIP_VERIFY=false
142    depends_on:
143      - consul_c2
144    extra_hosts:
145      - "host.docker.internal:host-gateway"
146    networks:
147      internal_net:
148        ipv4_address: 172.16.238.152
149        aliases:
150          - s2.vault
151    restart: unless-stopped
152
153
154networks:
155  internal_net:
156    driver: bridge
157    ipam:
158      driver: default
159      config:
160        - subnet: 172.16.238.0/24
$ docker-compose -f docker-compose.yaml up -d --remove-orphans

Endereços das UIs:

  • Vault 1: https://localhost:9200/ui
  • Vault 2: https://localhost:9210/ui
  • Consul 1: https://localhost:8090/ui
  • Consul 2: https://localhost:8091/ui
  • Consul 3: https://localhost:8092/ui



comments powered by Disqus