This post is the second of a two-part series on using Vault in production. Both posts are slightly redacted forms of internal documentation. This post covers day-to-day usage of Vault, while the previous post covered our specific workflow.

Please note that some of the referenced tooling is not currently publicly available.

Not all of our practices will apply to your situation, and there are cases certainly where our setup may be suboptimal for your environment.

Using Vault

This is the tl;dr you were looking for…

For Developers

Developers should expect credentials to live in environment variables that can be loaded into an app when needed. These credential names are specified within an app.json manifest file, and Ops should be contacted to place their values in Vault. The deploy process currently uses the vault api to retrieve appropriate credentials for a service, transparent to the development staff.

The basic flow for reading or writing credentials is the following:

1. Login to Vault and receive a token
2. Make a request with the token to read or write

Schema

Secrets accordingly to the following convention:

1

secret/ENVIRONMENT/APP/KEY value=VALUE

Login

Note: Developer Vault logins require a Github Access Token (https://github.com/blog/1509-personal-api-tokens)

CLI:

1

vault auth -method=github token=GITHUB_ACCESS_TOKEN

Upon success, a Vault token will be stored at $HOME/.vault-token.

HTTP_API:

1
2
3

curl \
  -L http://vault.service.consul:8200/v1/auth/github/login \
  -d '{ "token": "GITHUB_ACCESS_TOKEN" }'

List

CLI:

1

vault list secret/path/to/bucket

This will use the token at $HOME/.vault-token if it exists. Otherwise, you will need to login via vault auth first.

HTTP API:

1
2
3
4

curl \
    -H "X-Vault-Token: VAULT_TOKEN" \
    -X GET \
    -L http://vault.service.consul:8200/v1/secret/path/to/bucket?list=true

Read

CLI:

1

vault read secret/path/to/key

This will use the token at $HOME/.vault-token if it exists. Otherwise, you will need to login via vault auth first.

HTTP API:

1
2
3
4

curl \
    -H "X-Vault-Token: VAULT_TOKEN" \
    -X GET \
    -L http://vault.service.consul:8200/v1/secret/path/to/key

Write

CLI:

1
2

vault write secret/path/to/key \
    value=THISISASECRET

This will use the token at $HOME/.vault-token if it exists. Otherwise, you will need to login via vault auth first.

HTTP API:

1
2
3
4
5
6

curl \
    -H "X-Vault-Token: VAULT_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{"value":"THISISASECRET"}' \
    -L http://vault.service.consul:8200/v1/secret/path/to/key

For Admins

Initialization

Required reading: https://www.vaultproject.io/intro/getting-started/deploy.html

vault init is used to bootstrap a new Vault cluster. This generates a number of keys and requires a majority threshold of these keys in order to unseal Vault (more on unsealing below).

For SeatGeek’s Vault cluster, vault init has been run as follows:

1
2
3
4

vault init \
    -key-shares=5
    -key-threshold=3
    -pgp-keys = "keybase:OPS1,keybase:OPS2,keybase:OPS3,keybase:OPS4,keybase:OPS5"

Note: Local public key files can also submitted for the pgp-keys option

Initializing Vault this way leverages its support for authorizing users to be able to unseal Vault via their private GPG keys. This method was chosen as we already using blackbox to encrypt secrets within certain repositories.

When vault is initialized, an unseal tokens are printed out for each pgp key specified. The order of these keys matches the order in which the pgp keys were specified, and each can only be decrypted by the corresponding pgp key. Those unseal tokens should be securely distributed to the corresponding operations engineer and stored in a secure fashion. Loss of keys exceeding the threshold will result in a loss of ability to unseal the cluster.

Vault should only need to be reinitialized if all of the data in lost, which for SeatGeek would be in a loss of the Consul Cluster.

Unsealing

Vault boots up in a sealed state, and in this state no requests are answered. Each machine within a Vault cluster can be in a sealed or active state, and all must be unsealed before answering any requests. There is also a standby state, in which the machine is unsealed but not primary, and is ready for failover if the currently active primary dies.

A Vault machine can be unsealed via the following command:

1
2

export VAULT_ADDR="http://ip.for.vault.instance:8200"
echo "$VAULT_UNSEAL_KEY" | base64 -D | keybase pgp decrypt | xargs vault unseal

The VAULT_UNSEAL_KEY is specific to each user who was specified in the vault init command. All unseal keys were distributed at the time of initialization.

Note: This requires having the vault binary installed locally.

Root Token

An initial root token is created when the Vault Cluster is initialized. Other root tokens are created from this token, and as such, if a root token is needed it must be created by an existing holder of a root token.

A new root token can be created from an existing root token via the following command:

1

vault token-create -metadata "name=ADMIN_NAME" -display-name="ADMIN_USER_NAME" -orphan -no-default-policy

Note: You must be logged in with a root token in order to run this command

In the emergency case that a new root token needs to be created, the following command can be run:

1

vault generate-root

This operation requires a majority of unseal key holders to execute.

Note: At this time of writing, Vault 0.6.2 has deprecated this workflow surrounding root tokens and our usage is subject to change in the future.

Provisiong a New Service

When provisioning a new service, secrets can simply be written to the appropriate bucket (secret/ENVIRONMENT/APP_NAME/KEY value=VALUE). Everything under secret/ is a “key” and all necessary paths will be created.

Whitelisting a New Service

On first boot of infrastructure, the Jenkins Whitelist deploy job, a dependency of the Jenkins Configure job, will be run. If not already, the new application will be created and its corresponding IAM Role will be whitelisted. This job also ensures all other IAM Role whitelists are up to date in Vault.

To ensure that Jenkins has the correct permissions, a special role allowing it access to write auth and policy documents should be written to Vault. The following can be used to create the policy, which is stored with all other custom vault policies:

1

vault policy-write ENV-jenkins data/vault/policies/env-jenkins-policy.hcl

Policies must be audited on a regular basis, consistent with all other internal auditing processes.

When used in contexts that do not easily support passing roles, you can create a vault token for this or any policy. The following creates a renewable token that is valid for 60 seconds:

1
2

TOKEN_ID="$(uuidgen)"
vault token-create -policy="ENV-jenkins" -display-name="ENV-jenkins" -id="$TOKEN_ID" -ttl=60s

Rekey and Key Rotation

Vault’s usage of unseal keys is based on Shamir’s secret sharing algorithm.

https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing

The vault rekey command allows for the recreation of unseal keys as well as changing the number of key shares and key threshold. This is useful for adding or removing Vault admins.

The vault rotate command is used to change the encryption key used by Vault. This does not require anything other than a root token. Vault will continue to stay online and responsive during a rotate operation.

Disaster Response

In the case of an emergency, Vault should be sealed immediately via:

1

vault seal

This will prevent any actions or requests to be performed against the Vault server, and gives time to investigate the cause of the issue and an appropriate solution.

1. A secret stored in Vault is leaked A new secret should be generated and replaced in Vault, with a key rotation following.
2. Vault user credentials are leaked The user credentials should be revoked and a key rotation should be performed.
3. Vault unseal keys are leaked A rekey should be performed.

If you think these kinds of things are interesting, consider working with us as a Software Engineer at SeatGeek. Or, if backend development isn’t your thing, we have other openings in engineering and beyond!