seatgeek open sourced seatgeek/hashi-helper

Disaster Recovery and Configuration Management for Consul and Vault

This post is the first of a bonus series on tooling for the Hashi-stack - Consul, Nomad, Vault. We also recommend our previous series on using Vault in production.

Configuration Management for your Configuration

In our initial Vault rollout, one pain-point we quickly came across was managing Consul and Vault configuration. We use both Hashicorp tools for managing secrets and access control across our entire infrastructure, and knowing what configuration was setup where in each cluster is quite critical. On top of this, disaster recovery quickly became an issue we knew we needed to tackle before a more broad roll-out.

One of our Systems Engineers started looking at what our needs were in regards to properly managing Consul and Vault configuration, and came up with a wonderful workflow through the use of a tool we like to call hashi-helper. We use hashi-helper internally to manage multiple clusters in different environments via a git repository that contains our canonical configuration. It is now pretty trivial for us to:

• Standup a vault cluster using our normal provisioning toolchain.
• Unseal vault via gpg key via the normal vault tooling, or keybase via hashi-helper vault-unseal-keybase.
• Provision mounts, policies, and secrets using hashi-helper vault-push-all.
• Provision custom registered consul services via hashi-helper consul-push-services.
• Manage all of our configuration via either blackbox gpg-encrypted files or AWS KMS encryption through tooling such as sm.

Example Workflow

For those curious about what a typical workflow might look like, the following directory structure may be suitable for a typical organization with a Consul/Vault cluster per-environment:

• /${env}/apps/${app}.hcl (encrypted) Vault secrets for an application in a specific environment.
• /${env}/auth/${name}.hcl (encrypted) Vault auth backends for an specific environment ${env}.
• /${env}/consul_services/${type}.hcl (cleartext) List of static Consul services that should be made available in an specific environment ${env}.
• /${env}/databases/${name}/_mount.hcl (encrypted) Vault secret backend configuration for an specific mount ${name} in ${env}.
• /${env}/databases/${name}/*.hcl (cleartext) Vault secret backend configuration for a specific Vault role belonging to mount ${name} in ${env}.

Here is an example for managing Vault Secrets:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# environment name must match the directory name
environment "production" {

  # application name must match the file name
  application "XXXX" {

    # Vault policy granting any user with policy XXXX-read-only read+list access to all secrets
    policy "XXXX-read-only" {
      path "secret/XXXX/*" {
        capabilities = ["read", "list"]
      }
    }

    # an sample secret, will be written to secrets/XXXX/API_URL in Vault
    secret "API_URL" {
      value = "http://localhost:8181"
    }
  }
}

hashi-helper is available now on GitHub under the BSD 3-Clause License.

If you think these kinds of things are interesting, consider working with us as a Software Engineer at SeatGeek. Or, if backend development isn’t your thing, we have other openings in engineering and beyond!