Hashicorp Vault em Alta Disponibilidade com Consul
Consul
O HashiCorp Consul foi criado em 2014 como um DNS-SD (DNS based Service Discovery) para resolver o problema de comunicação entre microsserviços, mas ele também possui outras funcionalidades como health checking e armazenamento de chave-valor (KV Store).
Cada nó possui um agente de monitoramento instalado que é responsável por enviar, de forma segura, informações do nó a um ou mais servidores.
Os servidores mantêm um catálogo com as informações dos nós, como quais serviços estão disponíveis
em cada um e seus status.
Para alta disponibilidade, é recomendado ter de 3 a 5 servidores. Um deles será eleito como o líder.
Configuração manual
Para entender a arquitetura em detalhes, é interessante realizar a instalação passo a passo. Mais adiante será mostrado como realizar as configurações usando Docker-compose.
O processo de instalação do Consul e do Vault está bem detalhado na documentação oficial.
De forma resumida o passo a passo é o seguinte:
-
Inicie 5 máquinas com Linux
Pode ser em alguma cloud ou localmente. Três serão para o servidor Consul e duas para o Vault.
Na versão gratuita, apenas um nó do Vault fica ativo; os outros nós ficam em standby redirecionando os requests para o nó ativo. Se o nó que estava ativo fica indisponível, outro nó é eleito como o nó ativo.
O Consul também será instalado nas máquinas do Vault, porém em modo cliente. -
Defina os hostnames
(vault-01)# echo "vault-01" > /etc/hostname (vault-02)# echo "vault-02" > /etc/hostname (consul-01)# echo "consul-01" > /etc/hostname (consul-02)# echo "consul-02" > /etc/hostname (consul-03)# echo "consul-03" > /etc/hostname
-
Instale o Consul em todas as máquinas
O site oficial possui instruções de instalação nas distribuições mais populares.
No Arch Linux seria o seguinte:
# pacman -S consul
-
Edite o arquivo de configuração dos servidores
# cat /etc/consul.d/consul_s1.json
{ "server": true, "node_name": "$NODE_NAME", "datacenter": "dc1", "data_dir": "$CONSUL_DATA_PATH", "bind_addr": "0.0.0.0", "client_addr": "0.0.0.0", "advertise_addr": "$ADVERTISE_ADDR", "bootstrap_expect": 3, "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"], "ui": true, "log_level": "DEBUG", "enable_syslog": true, "acl_enforce_version_8": false }
-
Edite o arquivo de configuração dos clientes
# cat /etc/consul.d/consul_c1.json
{ "server": false, "datacenter": "dc1", "node_name": "$NODE_NAME", "data_dir": "$CONSUL_DATA_PATH", "bind_addr": "$BIND_ADDR", "client_addr": "127.0.0.1", "retry_join": ["$JOIN1", "$JOIN2", "$JOIN3"], "log_level": "DEBUG", "enable_syslog": true, "acl_enforce_version_8": false }
-
Edite o systemd unit file dos servidores
# cat /etc/systemd/system/consul.service ### BEGIN INIT INFO # Provides: consul # Required-Start: $local_fs $remote_fs # Required-Stop: $local_fs $remote_fs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Consul agent # Description: Consul service discovery framework ### END INIT INFO [Unit] Description=Consul server agent Requires=network-online.target After=network-online.target [Service] User=consul Group=consul PIDFile=/var/run/consul/consul.pid PermissionsStartOnly=true ExecStartPre=-/bin/mkdir -p /var/run/consul ExecStartPre=/bin/chown -R consul:consul /var/run/consul ExecStart=/usr/local/bin/consul agent \ -config-file=/usr/local/etc/consul/consul_s1.json \ -pid-file=/var/run/consul/consul.pid ExecReload=/bin/kill -HUP $MAINPID KillMode=process KillSignal=SIGTERM Restart=on-failure RestartSec=42s [Install] WantedBy=multi-user.target
-
Edite o systemd unit file dos clientes
### BEGIN INIT INFO # Provides: consul # Required-Start: $local_fs $remote_fs # Required-Stop: $local_fs $remote_fs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Consul agent # Description: Consul service discovery framework ### END INIT INFO [Unit] Description=Consul client agent Requires=network-online.target After=network-online.target [Service] User=consul Group=consul PIDFile=/var/run/consul/consul.pid PermissionsStartOnly=true ExecStartPre=-/bin/mkdir -p /var/run/consul ExecStartPre=/bin/chown -R consul:consul /var/run/consul ExecStart=/usr/local/bin/consul agent \ -config-file=/usr/local/etc/consul/consul_c1.json \ -pid-file=/var/run/consul/consul.pid ExecReload=/bin/kill -HUP $MAINPID KillMode=process KillSignal=SIGTERM Restart=on-failure RestartSec=42s [Install] WantedBy=multi-user.target
-
Verifique se todos os serviços estão rodando
# systemctl daemon-reload # systemctl start consul # systemctl status consul
$ consul members $ consul operator raft list-peers
Um dos servidores deve estar como líder.
-
Instale o Vault nas duas VMs
# pacman -S vault
-
Edite o arquivo de configuração do Vault
# cat /etc/vault/vault_s1.hcl
listener "tcp" { address = "127.0.0.1:8200" cluster_address = "127.0.0.1:8201" tls_disable = "true" } storage "consul" { address = "127.0.0.1:8500" path = "vault/" } api_addr = "$API_ADDR" cluster_addr = "$CLUSTER_ADDR"
-
Edite o systemd unit file dos Vaults
# cat /etc/systemd/system/vault.service
### BEGIN INIT INFO # Provides: consul # Required-Start: $local_fs $remote_fs # Required-Stop: $local_fs $remote_fs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Consul agent # Description: Consul service discovery framework ### END INIT INFO [Unit] Description=Consul client agent Requires=network-online.target After=network-online.target [Service] User=consul Group=consul PIDFile=/var/run/consul/consul.pid PermissionsStartOnly=true ExecStartPre=-/bin/mkdir -p /var/run/consul ExecStartPre=/bin/chown -R consul:consul /var/run/consul ExecStart=/usr/local/bin/consul agent \ -config-file=/usr/local/etc/consul/consul_c1.json \ -pid-file=/var/run/consul/consul.pid ExecReload=/bin/kill -HUP $MAINPID KillMode=process KillSignal=SIGTERM Restart=on-failure RestartSec=42s [Install] WantedBy=multi-user.target
-
Verifique se todos os serviços estão rodando
# systemctl daemon-reload # systemctl start vault # systemctl status vault
-
Inicialize o Vault
$ vault operator init
Salve as chaves e o root token em um local seguro.
-
Unseal
$ vault operator unseal <unseal_key_1> $ vault operator unseal <unseal_key_2> $ vault operator unseal <unseal_key_3>
Daqui em diante, podemos seguir como detalhado em outro artigo.
Configuração via docker-compose
$ consul keygen
LvqkL8MCukmz4/Z6n7v/38XSVh9jjbNoDXx5dsFd6Nw=
$ consul tls ca create
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem
$ consul tls cert create -server -dc dc1 ### -additional-dnsname=consul.server1
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem
$ consul tls cert create -server -dc dc1
==> Saved dc1-server-consul-1.pem
==> Saved dc1-server-consul-1-key.pem
$ consul tls cert create -server -dc dc1
==> Saved dc1-server-consul-2.pem
==> Saved dc1-server-consul-2-key.pem
$ consul tls cert create -client -dc dc1 -additional-dnsname=consul-c1
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem
$ consul tls cert create -client -dc dc1 -additional-dnsname=consul-c2
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-1.pem
==> Saved dc1-client-consul-1-key.pem
$ mv dc1-server-consul-0.pem Pasta/consul/config/server1/certs/server.pem
$ mv dc1-server-consul-1.pem Pasta/consul/config/server2/certs/server.pem
$ mv dc1-server-consul-2.pem Pasta/consul/config/server3/certs/server.pem
$ cp dc1-client-consul-0.pem Pasta/consul/config/client1/certs/client.pem
$ cp dc1-client-consul-1.pem Pasta/consul/config/client2/certs/client.pem
$ mv dc1-client-consul-0.pem Pasta/vault/config/server1/consul_tls/certs/client.pem
$ mv dc1-client-consul-1.pem Pasta/vault/config/server2/consul_tls/certs/client.pem
$ mv dc1-server-consul-0-key.pem Pasta/consul/config/server1/private/server-key.pem
$ mv dc1-server-consul-1-key.pem Pasta/consul/config/server2/private/server-key.pem
$ mv dc1-server-consul-2-key.pem Pasta/consul/config/server3/private/server-key.pem
$ cp dc1-client-consul-0-key.pem Pasta/consul/config/client1/private/client-key.pem
$ cp dc1-client-consul-1-key.pem Pasta/consul/config/client2/private/client-key.pem
$ mv dc1-client-consul-0-key.pem Pasta/vault/config/server1/consul_tls/private/client-key.pem
$ mv dc1-client-consul-1-key.pem Pasta/vault/config/server2/consul_tls/private/client-key.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server1/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server2/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/server3/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/client1/certs/root.pem
$ cp consul-agent-ca.pem Pasta/consul/config/client2/certs/root.pem
$ cp consul-agent-ca.pem Pasta/vault/config/server1/consul_tls/certs/root.pem
$ cp consul-agent-ca.pem Pasta/vault/config/server2/consul_tls/certs/root.pem
Certificado auto assinado para o Vault:
$ cat selfsign_s1.cfr
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = BR
ST = state
L = city
O = company
CN = *
[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
DNS.2 = s1.vault
DNS.3 = host.docker.internal
IP.1 = 172.16.238.151
IP.3 = 127.0.0.1
$ cat selfsign_s2.cfr
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = BR
ST = state
L = city
O = company
CN = *
[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
DNS.2 = s2.vault
DNS.3 = host.docker.internal
IP.1 = 172.16.238.152
IP.2 = 127.0.0.1
$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout server-key.pem -out server.pem -config selfsign_s1.cfr -days 9999
$ mv server.pem Pasta/vault/config/server1/tls/certs/server.pem
$ mv server-key.pem Pasta/vault/config/server1/tls/private/server-key.pem
$ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout server-key.pem -out server.pem -config selfsign_s2.cfr -days 9999
$ mv server.pem Pasta/vault/config/server2/tls/certs/server.pem
$ mv server-key.pem Pasta/vault/config/server2/tls/private/server-key.pem
$ cat docker-compose.yaml
version: "3.8"
services:
consul_s1:
container_name: consul.server1
image: consul:latest
command: ["agent", "-config-file=/consul/config/server_agent.json"]
volumes:
- ./consul/config/server1:/consul/config
- ./consul/data/data_s1:/consul/data
restart: unless-stopped
ports:
- "8090:8080"
- "8300:8300"
- "8500:8500"
- "8600:8600/udp"
networks:
internal_net:
ipv4_address: 172.16.238.53
consul_s2:
container_name: consul.server2
image: consul:latest
command: ["agent", "-config-file=/consul/config/server_agent.json"]
volumes:
- ./consul/config/server2:/consul/config
- ./consul/data/data_s2:/consul/data
restart: unless-stopped
ports:
- "8091:8080"
- "8310:8300"
- "8510:8500"
- "8610:8600/udp"
networks:
internal_net:
ipv4_address: 172.16.238.54
consul_s3:
container_name: consul.server3
image: consul:latest
command: ["agent", "-config-file=/consul/config/server_agent.json"]
volumes:
- ./consul/config/server3:/consul/config
- ./consul/data/data_s3:/consul/data
restart: unless-stopped
ports:
- "8092:8080"
- "8320:8300"
- "8520:8500"
- "8620:8600/udp"
networks:
internal_net:
ipv4_address: 172.16.238.55
consul_c1:
container_name: consul.client1
image: consul:latest
command: ["agent", "-config-file=/consul/config/client_agent.json"]
volumes:
- ./consul/config/client1:/consul/config
- ./consul/data/data_c1:/consul/data
restart: unless-stopped
ports:
- "8093:8080"
- "8330:8300"
- "8530:8500"
- "8630:8600/udp"
depends_on:
- consul_s1
networks:
internal_net:
ipv4_address: 172.16.238.201
aliases:
- consul-c1
consul_c2:
container_name: consul.client2
image: consul:latest
command: ["agent", "-config-file=/consul/config/client_agent.json"]
volumes:
- ./consul/config/client2:/consul/config
- ./consul/data/data_c2:/consul/data
restart: unless-stopped
ports:
- "8094:8080"
- "8340:8300"
- "8540:8500"
- "8640:8600/udp"
depends_on:
- consul_s2
networks:
internal_net:
ipv4_address: 172.16.238.202
aliases:
- consul-c2
vault_s1:
container_name: vault.server1
image: vault:latest
ports:
- "9200:8200"
expose:
- "8500"
volumes:
- ./vault/config/server1:/vault/config
- ./ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
- ./vault/logs_s1:/vault/logs
cap_add:
- IPC_LOCK
command: ["server", "-log-level=info"]
environment:
- VAULT_LOCAL_CONFIG={"ui":"true", "backend":{"consul":{"address":"consul-c1:8080",
"path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/consul_tls/certs/root.pem",
"tls_cert_file":"/vault/config/consul_tls/certs/client.pem",
"tls_key_file":"/vault/config/consul_tls/private/client-key.pem", "tls_skip_verify":0}},
"listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201",
"tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem",
"tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}},
"api_addr":"http://s1.vault:8200", "cluster_addr":"https://s1.vault:8201"}
- VAULT_SKIP_VERIFY=false
depends_on:
- consul_c1
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
internal_net:
ipv4_address: 172.16.238.151
aliases:
- s1.vault
restart: unless-stopped
vault_s2:
container_name: vault.server2
image: vault:latest
ports:
- "9210:8200"
expose:
- "8500"
volumes:
- ./vault/config/server2:/vault/config
- ./ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt
- ./vault/logs_s2:/vault/logs
cap_add:
- IPC_LOCK
command: ["server", "-log-level=info"]
environment:
- VAULT_LOCAL_CONFIG={"ui":"true", "backend":{"consul":{"address":"consul-c2:8080",
"path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/consul_tls/certs/root.pem",
"tls_cert_file":"/vault/config/consul_tls/certs/client.pem",
"tls_key_file":"/vault/config/consul_tls/private/client-key.pem", "tls_skip_verify":0}},
"listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201",
"tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem",
"tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}},
"api_addr":"http://s2.vault:8200", "cluster_addr":"https://s2.vault:8201"}
- VAULT_SKIP_VERIFY=false
depends_on:
- consul_c2
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
internal_net:
ipv4_address: 172.16.238.152
aliases:
- s2.vault
restart: unless-stopped
networks:
internal_net:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.16.238.0/24
$ docker-compose -f docker-compose.yaml up -d --remove-orphans
Endereços das UIs:
- Vault 1: https://localhost:9200/ui
- Vault 2: https://localhost:9210/ui
- Consul 1: https://localhost:8090/ui
- Consul 2: https://localhost:8091/ui
- Consul 3: https://localhost:8092/ui