Sources of Secrets for HashiCorp Vault

Secrets Distribution

The problem domain of secrets distribution is a common one. You have a secret, you want to distribute it to a service, and you want to implement this action in a secure way. The secret could be a password, an API key, a TLS certificate, or anything else that you want to keep secret.

Also, a common problem within secrets distribution is the need to rotate secrets. This is a security best practice, and it's often required by compliance standards. Rotating secrets can be a complex process, especially when you must do it across multiple services.

And finally, you might also be aware of the problem that distributed secrets can be hard to manage. You might have secrets stored in multiple places, and it can be hard to keep track of where they are and who has access to them.

Sources of Secrets

HashiCorp Vault is a secrets management tool that can help you solve these problems. Vault can store secrets, distribute them to services, and rotate them. It can also help you manage access to secrets, so you can manage access permissions effectively.

But of course, you need to get the secrets into Vault first. This is where sources of secrets come into play. Sources of secrets are the places where you can get secrets from and import them into Vault. There are different sources of secrets, and each source has its own way of getting secrets into Vault.

Sample Use Case - 1Password

As an example, let's look at 1Password. 1Password is a password manager that allows you to store and manage your passwords securely by UI or browser extension - but you might also want to use these secrets in your applications. And for this use case, you can setup a source of secrets, to synchronise these secrets from 1Password to your HashiCorp Vault.

 1locals {
 2  vault_name = "Infralovers Coders"
 3  vault_items = {
 4    "Docker Hub Service Account"            = "docker/service_account",
 5    "Quay Service Account"                  = "quay/service_account",
 6    "Gitlab Service Account"                = "gitlab/service_account",
 7    "Terraform HCP Service Account"         = "terraform_hcp/service_account",
 8    "HCP Packer Servie Account"             = "hcp_packer/service_account",
 9    "Mondoo Service Account"                = "mondoo/service_account"
10  }
11}

As first step we need to define which secrets from 1Password should be transfered/synchronized to HashiCorp Vault. In this example we have defined a local variable vault_items with the secrets we want to transfer. The key of the key-value-pair inside the map is the name of the secret in 1Password, the value is the path in HashiCorp Vaults secret engine.

 1resource "vault_mount" "kv" {
 2  path        = "kv"
 3  type        = "kv"
 4  description = "Generic Key/Value secrets engine"
 5}
 6
 7module "vault_1password" {
 8  for_each   = local.vault_items
 9  source     = "./1password"
10  vault_name = local.vault_name
11  vault_item = each.key
12}
13
14resource "vault_generic_secret" "1password_item" {
15  for_each  = local.vault_items
16  path      = "${vault_mount.kv.path}/op/${each.value}"
17  data_json = module.vault_1password[each.key].json_data
18}

The next step is to create a secret engine in Vault, where we can store the information from 1Password. In this example we are using the kv engine, which is a generic key/value secrets engine to store abritrary data. Finally, we are using a module to fetch the secrets from 1Password and store them in a local variable. Finally we are using the vault_generic_secret resource to store the secrets in Vault. The path of the secret is a combination of the path of the kv engine and the path of the secret in the vault_items map.

The interesting part is the ./1password module. This module is responsible for fetching the secrets from 1Password and returning them in a format that can be used by the vault_generic_secret resource. This logic is separated into a module to make this funcitonality reusable. The module could look like this:

 1variable "vault_name" {
 2  type = string
 3  description = "1password vault name"
 4}
 5
 6variable "vault_item" {
 7  type    = string
 8  description = "1password item title"
 9}
10
11data "onepassword_item" "secure_info_title" {
12  vault = var.vault_name
13  title = var.vault_item
14}
15
16locals {
17  core_data = {
18    username = data.onepassword_item.secure_info_title.username
19    password = data.onepassword_item.secure_info_title.password
20    url      = data.onepassword_item.secure_info_title.url
21    hostname = data.onepassword_item.secure_info_title.hostname
22    database = data.onepassword_item.secure_info_title.database
23    category = data.onepassword_item.secure_info_title.category
24    note     = data.onepassword_item.secure_info_title.note_value
25  }
26
27  fields = flatten(flatten([for s in data.onepassword_item.secure_info_title.section :
28    [for f in s.field : {
29      "${replace(f.label, " ", "_")}" = f.value
30    }]
31  ]))
32  json_data = merge(local.core_data, local.fields...)
33}
34
35output "json_data" {
36  value     = jsonencode(local.json_data)
37  sensitive = true
38  description = "data from 1password as json (object)"
39}

Additionally, the module is using the onepassword_item data source to fetch the information from 1Password. The data source requires the vault and title parameters to fetch the information, which are passed as variable to the module. The data source returns a lot of information, which is then transformed into a JSON object that can be used by the vault_generic_secret resource. The JSON object contains the core data like username, password, and so on. Additionally the module is flattening the fields of the 1Password sections and merging them into the JSON object.

Conclusion

The integration of 1Password with HashiCorp Vault is just one example of how you can use sources of secrets to get secrets into Vault. There are many other sources of secrets, like AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or even your own custom sources. The important thing is that you can use sources of secrets to get secrets into Vault in a secure and automated way.

If you are using that code in combination with Terraform HCP you are forced to run your own customized Agent which can interact with the 1Password API, because the Terraform provider of 1Password requires the CLI to be available. But this is a topic for another blog post.

Have fun trying it yourself!