Introduction

If you use Kolla-Ansible, you probably know the standard procedure to generate passwords with kolla-genpwd. All the passwords are stored in plaintext in the passwords.yml file, which is not ideal in terms of security.

There's another way to handle that: generate the passwords, store them in Vault, then remove all traces of these passwords in plaintext and use the Vault secrets. With that, no more worries, you can push everything as-is into your versioning system.

You can grab the scripts here: https://github.com/ab-a/kolla-vault.

Note: in this repository, the passwordsFile is set to etc/kolla/passwords.yml to work with the CI. If you use the script manually, you probably need to replace it with your specific path.

Part II is available here: Hashicorp Vault and Kolla Ansible, Part II: integration with GitLab CI.

There are no password leaks in these blog posts or in the GitHub repository—all the passwords you see have been specifically generated for this content.

Why not use what’s already available?

There is already an official way with Kolla to push and retrieve passwords in HashiCorp Vault, using kolla-writepwd and kolla-readpwd commands. But there is a significant problem with this solution: it either push secrets from an existing passwords.yml file or recreate a new passwords.yml using Vault secrets, which means you’re still stuck with plaintext passwords at some point in your workflow.

My idea was to store the secrets in HashiCorp Vault and dynamically retrieve them, without having plaintext passwords, which is not possible with the existing solution.

Technically, the store_kolla_passwords script does almost exactly the same thing as kolla-writepwd, but I still decided to recreate it, mainly because it was technically interesting and challenging, but also fun to do.

Prerequisite

• A working Kolla-Ansible environment
• A working golang environment
• Hashicorp Vault installed

I'm mostly using this script as part of a quick way to bootstrap a development environment.

Since I have more than 1 OpenStack cluster, I implemented a way to use the same Vault for multiple deployments.

Variables and parameters

• VAULT_PATH
  The VAULT_PATH variable is the subpath where the passwords will be stored in Vault. If the variable is not set, all the passwords will be stored under secret/kolla/default. For example, setting the variable to dev1 will store the passwords under secret/kolla/dev1.

• VAULT_TOKEN
  Be sure to also set the VAULT_TOKEN environment variable.

• basePath / defaultPath
  You can also change the basePath and defaultPath in the scripts if you want to use a different path or mountpoint.

• vault_url
  Last step, add this in your globals.yml:

vault_url: "http://127.0.0.1:8200"

This value is used for retrieving the secret in the lookup:

'{{ lookup('community.general.hashi_vault', 'secret/data/kolla/default/example', 'url={{ vault_url }}', token=lookup('env', 'VAULT_TOKEN')) }}'

Generate the passwords

Here we’ll do the usual stuff. Run the command:

kolla-genpwd

The passwords will be generated by default in /etc/kolla/passwords.yml, unless you specify another path with kolla-genpwd --passwords custom/path:

You can see the generated passwords here:

head -5 /path/to/passwords.yml

Output:

aodh_database_password: zIPQbYpZN5yVer7PHCvdBO9KfELGlQvWLM2AOHhT
aodh_keystone_password: k3yA7hNGyVsjbbfBggriIJP4UVF9HIx0dgcNi18U
barbican_crypto_key: LL3Vg1l6DxurrOF2BuPOLcJ_sAehQs-UmqppqYKpdQg=
barbican_database_password: ibZQonH0IktmIaC9boAekiANOwvqjPF2zXrUOXDQ
barbican_keystone_password: U1qW19z4CuU7e5Cc1mWWpxoA749kfvDXivXJJRxR

Push the passwords in Vault

First, set the VAULT_TOKEN environment variable:

export VAULT_TOKEN=$(vault print token)

Create this little Go script that handles the creation of all secrets in Vault. I named it store_kolla_passwords.go:

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/hashicorp/vault/api"
	"gopkg.in/yaml.v2"
	"io/ioutil"
)

const (
	passwordsFile = "/etc/kolla/passwords.yml" // Path to the passwords.yml file
	vaultAddress  = "http://127.0.0.1:8200"    // Address of the Vault server
	basePath      = "secret/data/kolla"        // Base Vault path, includes the 'data' section
	defaultPath   = "default"                  // Default subdirectory to use in Vault path
)

func main() {
	// Determine the Vault subdirectory from the environment variable or use the default
	vaultSubDir := getEnv("VAULT_PATH", defaultPath)
	vaultPath := fmt.Sprintf("%s/%s", basePath, vaultSubDir)
	vaultToken := os.Getenv("VAULT_TOKEN")

	// Ensure VAULT_TOKEN is set, or exit
	if vaultToken == "" {
		log.Println("VAULT_TOKEN is not set. Exiting.")
		return
	}

	// Initialize the Vault client
	client, err := initializeVaultClient(vaultAddress, vaultToken)
	if err != nil {
		log.Fatalf("Failed to create Vault client: %v", err)
	}

	// Load the passwords.yml file into memory
	data, err := ioutil.ReadFile(passwordsFile)
	if err != nil {
		log.Fatalf("Failed to read passwords.yml: %v", err)
	}

	// Parse the YAML content into a map
	passwords := make(map[string]interface{})
	err = yaml.Unmarshal(data, &passwords)
	if err != nil {
		log.Fatalf("Failed to parse YAML: %v", err)
	}

	// Store the parsed passwords in Vault
	if err := storePasswordsInVault(client, vaultPath, passwords); err != nil {
		log.Fatalf("Failed to store passwords: %v", err)
	}

	log.Println("All passwords have been stored in Vault.")
}

// getEnv retrieves the value of an environment variable or returns a fallback value if not set
func getEnv(key, fallback string) string {
	if value := os.Getenv(key); value != "" {
		fmt.Printf("Using Vault path '%s/%s'.\n", basePath, value)
		return value
	}
	fmt.Printf("VAULT_PATH is not set. Using default path '%s/%s'.\n", basePath, fallback)
	return fallback
}

// initializeVaultClient creates a new Vault client and sets the authentication token
func initializeVaultClient(address, token string) (*api.Client, error) {
	client, err := api.NewClient(&api.Config{Address: address})
	if err != nil {
		return nil, err
	}
	client.SetToken(token)
	return client, nil
}

// storePasswordsInVault stores the passwords recursively in Vault
func storePasswordsInVault(client *api.Client, vaultPath string, data map[string]interface{}) error {
	// Delegate the storage process to storeNestedMap to handle both simple and nested values
	return storeNestedMap(client, vaultPath, data)
}

// storeNestedMap recursively processes and stores nested structures in Vault
func storeNestedMap(client *api.Client, path string, data map[string]interface{}) error {
	for key, value := range data {
		switch v := value.(type) {
		case map[interface{}]interface{}:
			// Convert map[interface{}]interface{} to map[string]interface{} for compatibility with Vault
			nestedMap := make(map[string]interface{})
			for k, val := range v {
				strKey := fmt.Sprintf("%v", k)
				nestedMap[strKey] = val
			}
			// Recursively store nested maps
			if err := storeNestedMap(client, fmt.Sprintf("%s/%s", path, key), nestedMap); err != nil {
				return err
			}
		case string:
			// Store simple string values as secrets in Vault
			if err := storeSecret(client, path, key, v); err != nil {
				return err
			}
		case nil:
			// Handle nil values (currently skipping them)
			log.Printf("Skipping nil value for key %s", key)
		default:
			// Skip unsupported types
			log.Printf("Skipping unsupported type for key %s: %T", key, v)
		}
	}
	return nil
}

// storeSecret stores a single key-value pair in Vault at the specified path
func storeSecret(client *api.Client, vaultPath, key, value string) error {
	// Write the secret to Vault in the appropriate path
	_, err := client.Logical().Write(
		fmt.Sprintf("%s/%s", vaultPath, key),
		map[string]interface{}{"data": map[string]interface{}{"value": value}},
	)
	if err != nil {
		log.Printf("Error storing secret %s at path %s/%s: %v", key, vaultPath, key, err)
	}
	return err
}

Prepare the environment and run the script:

go mod init kolla-export-vault
go get github.com/hashicorp/vault/api
go run store_kolla_passwords.go

Output:

$ go mod init import-kolla-passwords
go: creating new go.mod: module import-kolla-passwords
go: to add module requirements and sums:
	go mod tidy
$ go get github.com/hashicorp/vault/api
go: downloading github.com/hashicorp/vault v1.17.3
go: downloading github.com/hashicorp/vault/api v1.14.0
[...]
$ go run store_kolla_passwords.go
Using Vault path 'secret/data/kolla/default'.
2024/08/21 17:35:13 Skipping nil value for key docker_registry_password
2024/08/21 17:35:13 All passwords have been stored in Vault.

Now you can confirm that everything was created properly in Vault:

$ vault kv list -mount=secret kolla/default

All the secret keys should be listed:

Keys
----
aodh_database_password
aodh_keystone_password
barbican_crypto_key
barbican_database_password
barbican_keystone_password
barbican_p11_password
[...]

You can also check if the secret is properly set:

$ vault kv get -mount=secret kolla/default/database_password

You can see the secret on the last line,value:

=========== Secret Path ===========
secret/data/kolla/default/database_password

======= Metadata =======
Key                Value
---                -----
created_time       2024-08-09T15:06:55.641708Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

==== Data ====
Key      Value
---      -----
value    axHOMIvWxdMqqROaJJRETPgjQJyZ7GfIAYOiIshb

Nested secrets

The script also works with nested secrets, to store SSH keys. You can see there's 2 secrets under kolla_ssh_key: private_key and public_key:

vault kv list -mount=secret kolla/default/kolla_ssh_key/
Keys
----
private_key
public_key

And each secret store the proper values:

================== Secret Path ==================
secret/data/kolla/default/kolla_ssh_key/public_key

======= Metadata =======
Key                Value
---                -----
created_time       2024-08-20T13:37:14.288326Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

==== Data ====
Key      Value
---      -----
value    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC6FrO6EykrtXnyfjEAMccwIW1Zs4O8lA8tWnaZgUd+8KDb5tFANpPms6ce8UZ7a3QXn9y/37GeSvtnB7bxbs4iciQyOjOC3pRaLRuykkG4D+NPTOslw/cM+56vrflHGTpdv2Ydcjmo8tJ2B1ZSt2tf/01NQnfXk7cbbSuAnLiKzQ/3042oRXYX5YRotf05R1vBlPLbTt9HtCRHx8q9CyZMcjQol/5RuAIRrVhJyfseT5pQGD8e90NdhzUfVSb+hVmGW+w9q0K6LvpXcYrK5mN+wAwGVhHGUE9kAJsZQucFBNOSQfrCDQVoPipKDhD1zMXWF/4zX7kGfOPzne/+3p4oIeNQn9UTVw7Nh04dOMioZugdv1w/diORCxFZOV/s1lYpnX8dFuHCk9McWVqLu/FWt1TOIN+qTaC7/5ofW/tlYbwzHBe18LTp4hbUOBmSsANzv+laEEA0wZO5YcxI04fxfQmYc4IaWD3AGVx57DXoqLSyJJjDsH7Qq1xWjqCR43ZDpi9x+bMfV7Bzlo43/UfVbN6WikXhFHRDqHP2l+sBNdeqMmQJRjzRryycc9ycsc4UIA8JbROdekJpoZD3VGaQLCatKVet70jUWkx20lbMU91a/F5FRzoa6VmT9XlLOwwO8c9fpibjBucN+8ckq3X2ixbEWJ/o4HenlBywN3j3LQ==

Update Kolla-Ansible

Now it's time to update the playbook to use the Vault secrets.

Here's another script to do that:

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"strings"

	"gopkg.in/yaml.v2"
)

const (
	passwordsFile = "/etc/kolla/passwords.yml"  // Path to your passwords.yml file
	vaultURLVar   = "http://127.0.0.1:8200"     // Placeholder for Vault URL variable
	basePath      = "secret/data/kolla"         // Base Vault path including '/data'
	defaultPath   = "default"                   // Default subdirectory for Vault path
)

func main() {
	// Determine the Vault path from the environment variable, or use the default if not set
	vaultSubDir := getEnv("VAULT_PATH", defaultPath)
	vaultPath := fmt.Sprintf("%s/%s", basePath, vaultSubDir)

	// Load the passwords.yml file from disk
	data, err := ioutil.ReadFile(passwordsFile)
	if err != nil {
		log.Fatalf("Failed to read passwords.yml: %v", err)
	}

	// Parse the YAML content into a map
	passwords := make(map[string]interface{})
	err = yaml.Unmarshal(data, &passwords)
	if err != nil {
		log.Fatalf("Failed to parse YAML: %v", err)
	}

	// Iterate over the parsed passwords map and replace each password with a Vault lookup
	for key, value := range passwords {
		switch v := value.(type) {
		case string:
			// If the value is a string and doesn't already contain a Vault lookup, update it
			if v == "" || !containsVaultLookup(v) {
				log.Printf("Updating key %s with Vault lookup.", key)
				passwords[key] = generateVaultLookup(vaultPath, key)
			}
		case map[interface{}]interface{}:
			// If the value is a nested map, process the nested structure
			log.Printf("Processing nested map for key %s", key)
			nestedVaultPath := fmt.Sprintf("%s/%s", vaultPath, key)
			passwords[key] = processNestedMap(v, nestedVaultPath)
		default:
			// Log unsupported types that aren't handled
			log.Printf("Unhandled type for key %s: %T", key, v)
		}
	}

	// Serialize the updated passwords map back to YAML with proper single-quote handling
	updatedData := manualYAMLSerialization(passwords)

	// Write the updated data back to passwords.yml file
	err = ioutil.WriteFile(passwordsFile, []byte(updatedData), 0644)
	if err != nil {
		log.Fatalf("Failed to write updated passwords.yml: %v", err)
	}

	fmt.Println("passwords.yml has been updated with Vault secret references.")
}

// getEnv retrieves the value of an environment variable or returns a fallback value if not set
func getEnv(key, fallback string) string {
	if value := os.Getenv(key); value != "" {
		return value
	}
	return fallback
}

// generateVaultLookup generates a Vault lookup string for a given variable name and path
func generateVaultLookup(vaultPath, variableName string) string {
	return fmt.Sprintf("{{ lookup('community.general.hashi_vault', '%s/%s', 'url=%s', token=lookup('env', 'VAULT_TOKEN')) }}",
		vaultPath, variableName, vaultURLVar)
}

// processNestedMap processes nested maps by generating Vault lookups for each nested key
func processNestedMap(nestedMap map[interface{}]interface{}, vaultPath string) map[string]string {
	updatedMap := make(map[string]string)
	for subKey, subValue := range nestedMap {
		subKeyStr := fmt.Sprintf("%v", subKey) // Convert key to string
		switch v := subValue.(type) {
		case string:
			// If the value is a string and doesn't already contain a Vault lookup, update it
			if v == "" || !containsVaultLookup(v) {
				updatedMap[subKeyStr] = generateVaultLookup(vaultPath, subKeyStr)
			}
		default:
			// Log unsupported nested types that aren't handled
			log.Printf("Skipping unsupported nested type for key %s: %T", subKeyStr, v)
		}
	}
	return updatedMap
}

// containsVaultLookup checks if the string already contains a Vault lookup
func containsVaultLookup(value string) bool {
	return strings.Contains(value, "hashi_vault")
}

// manualYAMLSerialization serializes the map back to YAML format with proper single-quote handling
func manualYAMLSerialization(data map[string]interface{}) string {
	var result strings.Builder
	for key, value := range data {
		// Convert each key-value pair to a YAML-formatted string
		result.WriteString(fmt.Sprintf("%s: '%v'\n", key, value))
	}
	return result.String()
}

I named it replace_kolla_passwords.go. Prepare the environment and run it:

go get gopkg.in/yaml.v2
go run replace_kolla_passwords.go

The output should be:

2024/08/21 17:36:10 Updating key designate_pool_id with Vault lookup.
2024/08/21 17:36:10 Updating key ironic_inspector_keystone_password with Vault lookup.
2024/08/21 17:36:10 Updating key magnum_keystone_password with Vault lookup.
[...]
passwords.yml has been updated with Vault secret references.

Everything should be updated, included nested secret:

2024/08/21 17:36:10 Processing nested map for key haproxy_ssh_key
2024/08/21 17:36:10 Updating subKey private_key under haproxy_ssh_key with Vault lookup.
2024/08/21 17:36:10 Updating subKey public_key under haproxy_ssh_key with Vault lookup.

Validation

Now you should be able to see that there are no plaintext passwords anymore, and all the passwords are referenced to their own Vault secret:

$ grep neutron passwords.yml
neutron_keystone_password: '{{ lookup('community.general.hashi_vault', 'secret/data/kolla/default/neutron_keystone_password', 'url={{ vault_url }}', token=lookup('env', 'VAULT_TOKEN')) }}'
neutron_ssh_key: 'map[private_key:{{ lookup('community.general.hashi_vault', 'secret/data/kolla/default/neutron_ssh_key/private_key', 'url={{ vault_url }}', token=lookup('env', 'VAULT_TOKEN')) }} public_key:{{ lookup('community.general.hashi_vault', 'secret/data/kolla/default/neutron_ssh_key/public_key', 'url={{ vault_url }}', token=lookup('env', 'VAULT_TOKEN')) }}]'
neutron_database_password: '{{ lookup('community.general.hashi_vault', 'secret/data/kolla/default/neutron_database_password', 'url={{ vault_url }}', token=lookup('env', 'VAULT_TOKEN')) }}'

Conclusion

That’s it! Now you have a solid and robust way to use secrets from Vault in your Kolla-Ansible deployment. You can use the same idea for anything that needs to be redacted, in Kolla or any Ansible playbook, of course.

You can check the part II, explaining how to automate all of that: Hashicorp Vault and Kolla Ansible, Part II: integration with GitLab CI.