Hashicorp Vault

Introduction to Hashicorp Vault

Julio Batista Silva
Apr 19, 2021 12 min read
Consider Infisical as an alternative to Hashicorp Vault.

HashiCorp Vault is a tool for securely accessing secrets (passwords, API keys, certificates, etc.) through a common interface with detailed logs for auditing.

Some of the Vault’s features include:

  • Encryption
  • Dynamic secret generation with expiration time
  • Secret renewal and revocation

Many of these features are also available in services like Azure Key Vault and AWS Key Management Service, but Vault has the advantage of being open source and not tied to a specific cloud (cloud agnostic).

Secrets are stored in a storage backend that can be memory, disk, database, Azure Blob, AWS S3, among others. The Vault serves as an interface. In another article, I showed how to configure Vault to use Consul’s key-value store.

Vault was written in Go and is distributed as a single binary. It can be interacted with through the command line, HTTP REST, or the web interface.

Secrets Engines

Secret engines are responsible for storing, generating, and encrypting data.

Some secret engines only store data, while others connect to services and generate dynamic credentials on demand. Others provide encryption as a service, generation of TOTP, and certificates.

The Vault site lists nearly 50 secret engines. Among them are AWS, Azure, MySQL, PostgreSQL, Active Directory, PKI, and SSH.

The engines are activated in a path. The vault path-help command can be used to obtain a description of the engine and discover which paths it responds to.

Lifecycle of a secret engine:

  • Enable: vault secrets enable
  • Disable: vault secrets disable
  • Move: vault secrets move
  • Tune: vault secrets tune

KV Secrets Engine

The kv secret engine is used to store key-value type secrets.

There are two versions:

  • KV version 1:
    • No versioning
    • A deleted secret cannot be recovered
    • Better performance
  • KV version 2:
    • Has versioning (10 versions by default)
    • Has soft delete
    • Lower performance

Secrets Lifecycle

  • Create: kv put
  • Read: kv get
  • Update: kv put
  • Delete/Undelete: kv delete / kv undelete
  • Destroy: kv destroy
  • Metadata Destroy: kv metadata delete

Authentication Methods

Vault supports multiple authentication methods, such as LDAP, Azure Active Directory, and AWS Identity & Access Management.

  • List active methods: vault auth list
  • Enable a method: vault auth enable [method]
  • Configure an authentication method: vault write auth/[method]/config
  • Add a role: vault write auth/[method]/role/[role_name]
  • Disable a method: vault auth disable [method]

Access Policies

A Vault access policy (Vault Policy) is a document in HCL (HashiCorp Configuration Language) or JSON that specifies who can do what in what way.

It is possible to have fine-grained control over what each user can or cannot access, including which commands they can/should use in each path. The official documentation has several useful examples.

  • List policies: vault policy list
  • Create a policy: vault policy write [policy] [policy_file.hcl]
  • Update a policy: vault write sys/policy/[policy]/ policy=[policy_file.hcl]
  • Delete a policy: vault delete sys/policy/[policy]

Dynamic Secrets

Dynamic Secrets are secrets created on-demand with a lease period. If not renewed, the secret is revoked.

Cubbyhole Response Wrapping

Instead of providing a token directly to the user, we can place that token in a cubbyhole and generate a token for one-time use with a short duration that has access to the cubbyhole. This is called Cubbyhole Response Wrapping.

Advantages:

  • The secret is exposed for a short time (lease TTL)
  • If the token leaks, it cannot be used again
  • The user does not have direct access to the secret, but rather a reference

Audit

Vault logs all requests made to the server and all responses sent, including errors.

The logs are in JSON format and can be saved in more than one location (audit device: file, syslog, or socket) at the same time.

If the log_raw=true option is not passed during activation, only the hashes of the secrets will be saved in the log.

For security, Vault stops accepting requests if it cannot access any of the activated audit devices.

  • Enable an audit device: vault audit enable -path=vault_path [type]
  • Disable an audit device: vault audit disable [path]
  • List audit devices: vault audit list --detailed

Practical Step-by-Step

Examples involving everything explained above.

  • Install and start Vault

    Using the package manager of your distribution or as detailed in the other article.

    # pacman -S vault

    # cat /etc/vault.hcl
backend "file" {
    path = "/var/lib/vault"
}

listener "tcp" {
    tls_disable = 1
}

    # Start the vault service

    $ vault operator init
  Unseal Key 1: p/rCDw28akkqMBogtkpuH7OoB0IbwNK6fOPE1/K56mCw
  Unseal Key 2: 4Qy/bymi4BShb0JaoMnQm8Qry/MrrM3pP3CQDeJu93s+
  Unseal Key 3: +Qdh8mBl1yzqi5V2u1tKG1u3pWtRfXEQ1cwMjoYpuSAy
  Unseal Key 4: lG6Vl8iiJhUpvmmWF5S8EgKd89wMUHsL4PNHuT4xGZ82
  Unseal Key 5: rsufBM3HiobRE39El8ErIkzx+qkxz9unzIrMb/w5gZTB

  Initial Root Token: s.3iZvNY9lhNdeJVH5fIHU1RDv

  • Export the environment variables with the address and token

    $ export VAULT_ADDR='http://127.0.0.1:8200'
$ export VAULT_TOKEN='s.3iZvNY9lhNdeJVH5fIHU1RDv'

    If you are running Vault in a Docker container, it’s worth creating the following alias:

    $ alias vault='docker exec -it -e VAULT_TOKEN=$VAULT_TOKEN vault.server1 vault "$@"'

  • Unseal

    $ vault operator unseal p/rCDw28akkqMBogtkpuH7OoB0IbwNK6fOPE1/K56mCw
  …
  Sealed             true
  Unseal Progress    1/3
  …

    $ vault operator unseal +Qdh8mBl1yzqi5V2u1tKG1u3pWtRfXEQ1cwMjoYpuSAy
  …
  Sealed             true
  Unseal Progress    2/3
  …

    $ vault operator unseal rsufBM3HiobRE39El8ErIkzx+qkxz9unzIrMb/w5gZTB
Sealed          false

  • Login

    $ vault login

  • Enable the secrets backend

    $ vault secrets enable -path=kv -version=2 kv
Success! Enabled the kv secrets engine at: kv/

  • Configure to keep a maximum of 2 versions

    $ vault write kv/config max_versions=2
Success! Data written to: kv/config

  • Write a secret

    $ vault kv put kv/meu_app senha=123456
  Key              Value
  ---              -----
  created_time     2021-04-19T04:11:34.948088026Z
  deletion_time    n/a
  destroyed        false
  version          1

  • Create a second version of the secret

    $ vault kv put kv/meu_app senha=123456 usuario=julio
  Key              Value
  ---              -----
  created_time     2021-04-19T04:11:44.839504165Z
  deletion_time    n/a
  destroyed        false
  version          2

  • Create a third version of the secret

    $ vault kv put kv/meu_app senha=654321 usuario=julio
  Key              Value
  ---              -----
  created_time     2021-04-19T04:11:53.768980035Z
  deletion_time    n/a
  destroyed        false
  version          3

  • Read the value of the latest version

    $ vault kv get kv/meu_app
  ====== Metadata ======
  Key              Value
  ---              -----
  created_time     2021-04-19T04:11:53.768980035Z
  deletion_time    n/a
  destroyed        false
  version          3

  ===== Data =====
  Key        Value
  ---        -----
  senha      654321
  usuario    julio

  • Read the value of the second version

    $ vault kv get -version=2 kv/meu_app
  ====== Metadata ======
  Key              Value
  ---              -----
  created_time     2021-04-19T04:11:44.839504165Z
  deletion_time    n/a
  destroyed        false
  version          2

  ===== Data =====
  Key        Value
  ---        -----
  senha      123456
  usuario    julio

  • Read the value of the first version

This version was destroyed because we set a limit to save only the last two.

$ vault kv get -version=1 kv/meu_app
No value found at kv/data/meu_app
  • Get the value of the second version in JSON
$ vault kv get -format=json -version=2 kv/meu_app
  {
    "request_id": "379bbb04-a21b-cebe-9e99-392b85372592",
    "lease_id": "",
    "lease_duration": 0,
    "renewable": false,
    "data": {
      "data": {
        "senha": "123456",
        "usuario": "julio"
      },
      "metadata": {
        "created_time": "2021-04-19T04:11:44.839504165Z",
        "deletion_time": "",
        "destroyed": false,
        "version": 2
      }
    },
    "warnings": null
  }

  • Filter with jq

    $ vault kv get -format=json -version=2 kv/meu_app | jq -r .data.data.senha
123456

  • Get the value of the second version using the API

    $ curl -k --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/kv/data/meu_app?version=2 | jq .data.data
  {
    "senha": "123456",
    "usuario": "julio"
  }

  • Delete versions 2 and 3 of the secret

    If you do not specify the version, the last one is deleted.

    $ vault kv delete -versions="2,3" kv/meu_app
Success! Data deleted (if it existed) at: kv/meu_app

    $ vault kv get -version=2 kv/meu_app
  ====== Metadata ======
  Key              Value
  ---              -----
  created_time     2021-04-19T04:11:44.839504165Z
  deletion_time    2021-04-19T04:14:27.514456987Z
  destroyed        false
  version          2

    $ vault kv get -version=3 kv/meu_app
  ====== Metadata ======
  Key              Value
  ---              -----
  created_time     2021-04-19T04:11:53.768980035Z
  deletion_time    2021-04-19T04:14:27.514457229Z
  destroyed        false
  version          3
    • Retrieve version 2
    $ vault kv undelete -versions=2 kv/meu_app
Success! Data written to: kv/undelete/meu_app

  • Destroy version 2 of the secret

    $ vault kv destroy -versions=2 kv/meu_app
Success! Data written to: kv/destroy/meu_app

    $ vault kv get -version=2 kv/meu_app
  ====== Metadata ======
  Key              Value
  ---              -----
  created_time     2021-04-19T04:11:44.839504165Z
  deletion_time    n/a
  destroyed        true
  version          2

  • Delete all metadata

    $ vault kv list kv/
  Keys
  ----
  meu_app

    $ vault kv metadata delete kv/meu_app
Success! Data deleted (if it existed) at: kv/metadata/meu_app

    $ vault kv list kv/
No value found at kv/metadata

  • Create the admin policy

$ cat policy_admin.hcl
  # Mount secrets engines
  path "sys/mounts/*" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

  # Configure the database secrets engine and create roles
  path "database/*" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

  # Manage the leases
  path "sys/leases/+/database/creds/readonly/*" {
    capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]
  }

  path "sys/leases/+/database/creds/readonly" {
    capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]
  }

  # Write ACL policies
  path "sys/policies/acl/*" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

  # Manage tokens for verification
  path "auth/token/create" {
    capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]
  }

$ vault policy write admin policy_admin.hcl
Success! Uploaded policy: admin
  • Create the apps policy
$ cat policy_apps.hcl
  # Get credentials from the database secrets engine 'readonly' role.
  path "database/creds/readonly" {
    capabilities = [ "read" ]
  }

$ vault policy write apps policy_apps.hcl
Success! Uploaded policy: apps

  • List the policies

    $ vault policy list
  admin
  apps
  default
  root

  • Start Docker and pull the PostgreSQL image

    # systemctl start docker.service
$ docker pull postgres:13

  • Create a database

    $ docker run \
        --name postgres \
        --env POSTGRES_USER=root \
        --env POSTGRES_PASSWORD=rootpassword \
        --detach  \
        --publish 5432:5432 \
        postgres:13

  • Connect to the database and create a role named ro with read permission on all tables

    $ docker exec -it postgres psql
  root=# CREATE ROLE ro NOINHERIT;
  CREATE ROLE
  root=# GRANT SELECT ON ALL TABLES IN SCHEMA public TO "ro";
  GRANT
  root=# \q

  • Enable the database secrets engine

    $ vault secrets enable database
Success! Enabled the database secrets engine at: database/

  • Configure the engine to work with Postgres

    $ vault write database/config/postgresql \
       plugin_name=postgresql-database-plugin \
       connection_url="postgresql://{{username}}:{{password}}@localhost:5432/postgres?sslmode=disable" \
       allowed_roles=readonly \
       username="root" \
       password="rootpassword"

  • SQL used to create credentials

    $ cat readonly.sql
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' INHERIT;
GRANT ro TO "{{name}}";

  • Create the readonly role

     vault write database/roles/readonly \
        db_name=postgresql \
        creation_statements=@readonly.sql \
        default_ttl=1h \
        max_ttl=24h
  Success! Data written to: database/roles/readonly

  • Generate a credential in the DB

    $ vault read database/creds/readonly
  Key                Value
  ---                -----
  lease_id           database/creds/readonly/hTFHqyO2AVcCBQRFmRTv8bGV
  lease_duration     1h
  lease_renewable    true
  password           NMVS-H2YO5BNMAkzP6Nl
  username           v-root-readonly-sKjgJBT41RYrrLtB1MtS-1618807755

  • List the users of Postgres

    $ docker exec -it postgres psql
  root=# \du
                                               List of roles
              Role name       |                   Attributes              | Member of
  ----------------------------+-------------------------------------------+-----------
   ro                         | No inheritance, Cannot login              | {}
   root                       | Superuser, Create…Replication, Bypass RLS | {}
   v-root-readonly-sKj…7755 | Password valid until 2021-04-19 05:49:20+00 | {ro}

  root=# \q

  • List leases

    $ vault list sys/leases/lookup/database/creds/readonly
Keys
----
hTFHqyO2AVcCBQRFmRTv8bGV

  • Store the ID in a variable

    $ LEASE_ID=$(vault list -format=json sys/leases/lookup/database/creds/readonly | jq -r ".[0]")

    $ echo $LEASE_ID
hTFHqyO2AVcCBQRFmRTv8bGV

  • Lease renew

    If the requested value exceeds the limit, it will stay at the limit.

    $ vault lease renew -increment=3600 database/creds/readonly/$LEASE_ID

    $ vault lease renew -increment=96400 database/creds/readonly/$LEASE_ID
WARNING! The following warnings were returned from Vault:

* TTL of "26h46m40s" exceeded the effective max_ttl of "23h33m18s"; TTL
value is capped accordingly

Key                Value
---                -----
lease_id           database/creds/readonly/hTFHqyO2AVcCBQRFmRTv8bGV
lease_duration     23h33m18s
lease_renewable    true

    root=# \du
                                             List of roles
            Role name       |                   Attributes              | Member of
----------------------------+-------------------------------------------+-----------
 ro                         | No inheritance, Cannot login              | {}
 root                       | Superuser, Create…Replication, Bypass RLS | {}
 v-root-readonly-sKj…7755 | Password valid until 2021-04-20 04:49:20+00 | {ro}

root=# \q

  • Lease revoke

    $ vault lease revoke database/creds/readonly/$LEASE_ID
All revocation operations queued successfully!

    $ docker exec -it postgres psql
root=# \du
                                   List of roles
 Role name |                         Attributes                 | Member of
-----------+----------------------------------------------------+-----------
 ro        | No inheritance, Cannot login                       | {}
 root      | Superuser, Create role, …, Replication, Bypass RLS | {}

root=# \q

    $ vault list sys/leases/lookup/database/creds/readonly
No value found at sys/leases/lookup/database/creds/readonly

  • List active authentication methods:

    $ vault auth list
Path      Type     Accessor               Description
----      ----     --------               -----------
token/    token    auth_token_5e067aef    token based credentials

  • Enable userpass:

    $ vault auth enable userpass
Success! Enabled userpass auth method at: userpass/

  • View available options

    $ vault path-help auth/userpass

  • Add a user

    $ vault write auth/userpass/users/julio \
    password=minha_senha \
    policies=admin
Success! Data written to: auth/userpass/users/julio

  • List users

    $ vault list auth/userpass/users
Keys
----
julio

  • Open a new terminal and log in with the created user:

    $ export VAULT_ADDR='http://127.0.0.1:8200'
$ vault login -method=userpass username=julio
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.8vRGmBVEQ1indCPtGL796od3
token_accessor         CmTWhmRHFq4RMLb6wccFgOFT
token_duration         768h
token_renewable        true
token_policies         ["admin" "default"]
identity_policies      []
policies               ["admin" "default"]
token_meta_username    julio

  • Delete the user:

    $ vault delete auth/userpass/users/julio
Success! Data deleted (if it existed) at: auth/userpass/users/julio

  • Save a secret in Vault:

    $ vault kv put kv/app-server api-key=123456
Key              Value
---              -----
created_time     2021-04-19T06:02:13.217806976Z
deletion_time    n/a
destroyed        false
version          1

  • Create a wrapping token valid for 5 minutes:

    $ vault kv get -wrap-ttl=300 kv/app-server
Key                              Value
---                              -----
wrapping_token:                  s.wqyqcD8lPcaOGznSg4ZNVFPC
wrapping_accessor:               sy6OAxfoKcmPLbUufzGhxkFY
wrapping_token_ttl:              5m
wrapping_token_creation_time:    2021-04-19 13:08:13.210327049 -0300 -03
wrapping_token_creation_path:    kv/data/app-server

  • Access the secret using this token

    $ curl --header "X-Vault-Token: s.wqyqcD8lPcaOGznSg4ZNVFPC" --request POST \
  $VAULT_ADDR/v1/sys/wrapping/unwrap | jq
{
  "request_id": "69598fb3-0f90-a1c8-22d8-387f5c79564e",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "data": {
      "api-key": "123456"
    },
    "metadata": {
      "created_time": "2021-04-19T06:02:13.217806976Z",
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

  • Create a kv store at the webkv path

    $ vault secrets enable -path=webkv kv
Success! Enabled the kv secrets engine at: webkv/

  • Add a secret to webkv

    $ vault kv put webkv/app-server api-key=123456
Success! Data written to: webkv/app-server

  • Create a web policy

    $ cat webpol.hcl
path "webkv/*" {
  capabilities = ["read", "list"]
}

    $ vault policy write web webpol.hcl
Success! Uploaded policy: web

  • Create a token valid for 5 minutes using the web policy

    $ vault token create -policy=web -wrap-ttl=300
Key                              Value
---                              -----
wrapping_token:                  s.FPSHG7GJ3pWWliA0zSDEb77w
wrapping_accessor:               oLR1tRBpyQACPICrMIWrtfDm
wrapping_token_ttl:              5m
wrapping_token_creation_time:    2021-04-19 14:58:31.807307058 -0300 -03
wrapping_token_creation_path:    auth/token/create
wrapped_accessor:                eP85Zj7EmwrDgE1GoagsNJ6p

  • Retrieve the token

    $ curl --header "X-Vault-Token: s.FPSHG7GJ3pWWliA0zSDEb77w" --request POST \
    $VAULT_ADDR/v1/sys/wrapping/unwrap | jq
{
  "request_id": "4cfe3fd2-942a-d7e3-34e5-2b7d644b3e28",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": null,
  "wrap_info": null,
  "warnings": null,
  "auth": {
    "client_token": "s.lDqthe94B6bFVyXxNqLMuwYO",
    "accessor": "eP85Zj7EmwrDgE1GoagsNJ6p",
    "policies": [
      "default",
      "web"
    ],
    "token_policies": [
      "default",
      "web"
    ],
    "metadata": null,
    "lease_duration": 2764800,
    "renewable": true,
    "entity_id": "",
    "token_type": "service",
    "orphan": false
  }
}

  • Retrieve the secret using the token received in the response of the previous command

    $ curl --header "X-Vault-Token: s.lDqthe94B6bFVyXxNqLMuwYO" $VAULT_ADDR/v1/webkv/app-server | jq
{
  "request_id": "f63dcc67-b9f6-43f6-dbb8-756c709e6a2f",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 2764800,
  "data": {
    "api-key": "123456"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}
Hashicorp
Julio Batista Silva
Julio Batista Silva
Data Engineer

I’m a computer engineer passionate about science, technology, photography, and languages. Currently working as a Data Engineer in Germany.

comments powered by Disqus