Hashicorp Vault and Kolla Ansible, Part II: integration with GitLab CI

Introduction

I like using GitLab CI to automate tasks. In the first part, I explained how to generate passwords, store them in Vault, remove all traces of plaintext passwords, and use Vault secrets.

Now, let's integrate that process into GitLab CI!

Part I is available here: Hashicorp Vault and Kolla Ansible, Part I: Integrate Vault secrets in your playbook.

You can grab the scripts here: https://github.com/ab-a/kolla-vault.

There are no password leaks in these blog posts or in the GitHub repository—all the passwords you see have been specifically generated for this content.

Why not use what’s already available?

As explained in the first post, there is already an official way with Kolla to push and retrieve passwords in HashiCorp Vault using the kolla-writepwd and kolla-readpwd commands. These commands either push secrets from an existing passwords.yml file or recreate a new passwords.yml using Vault secrets.

I wanted to dynamically retrieve the secrets from Vault and stop relying on a file with plaintext passwords, which the current implementation does not allow, so I decided to create my own.

The store_kolla_passwords script does almost exactly the same thing as kolla-writepwd, but I still decided to rewrite it for the challenge and to unify everything. And a bonus with golang: the ability to ship a binary that can run virtually anywhere, which is nice in a CI/CD pipeline context.

Should I use it in production

I implemented several mechanisms in the pipeline to minimize mistakes in more sensitive environments:

1. The changes are pushed into a separate and timestamped branch.
2. The CI can only run on a branch with the KOLLA_BOOTSTRAP tag.
3. The CI will not run if VAULT_PATH is set to production.
4. The CI pipeline can only be triggered manually, except for the setup stage, which is run automatically.
5. The CI will never run if the two conditions above are not met.

In simpler terms:

• sensitive jobs can only be triggered manually, regardless of the rules, and require specific variables to run.
• even if the conditions are met, the jobs will never run automatically in any context.

Even if you accidentally push a new set of passwords, you can still roll back the Vault secret to a previous version.

Here’s the disclaimer: For now, I’ve only used this automation in my lab to speed up deployment bootstraps. I'm not 100% comfortable using it in production yet, but it should be "probably" safe.

Create a Project Access Token

For the push job, you’ll need a Project Access Token.

Under "Settings", "Access Tokens", create a new token with the role Developer, and the scopes api, read_repository, write_repository.

Copy the token.

Create secrets in GitLab

Create a variable PROJECT_ACCESS_TOKEN where you will put the token created on the previous step. Create the VAULT_PATH variable if you want to use another path than secret/kolla/default.

These variables should be expanded and masked.

More informations: GitLab CI/CD Variables.

Push the scripts in your repo

For the GitLab pipeline to work, you need to upload the scripts to store the passwords in Vault and update the playbook.

You can grab the scripts and the CI file from my GitHub repo:

curl -O https://raw.githubusercontent.com/ab-a/kolla-vault/main/store_kolla_passwords.go
curl -O https://raw.githubusercontent.com/ab-a/kolla-vault/main/replace_kolla_passwords.go
curl -O https://raw.githubusercontent.com/ab-a/kolla-vault/main/.gitlab-ci.yml

How to setup your repo and set the CI tag

The pipeline will only trigger if the $CI_COMMIT_TAG is KOLLA_BOOTSTRAP and the VAULT_PATH is not production.

To ensure you’re pushing the tag, you just need to run:

git tag KOLLA_BOOTSTRAP
git push origin KOLLA_BOOTSTRAP

Or set the tag when you run the pipeline:

About the Vault path

If you don't specify the VAULT_PATH variable, the passwords secret will be created in secret/kolla/default. Set this variable if you plan to have multiple deployments.

You can also change the basePath and defaultPath in the scripts if you want to use a different path or mountpoint.

GitLab CI file

Here's the .gitlab-ci.yml file that you need to push:

image: ubuntu:latest

stages:
  - setup
  - push_in_vault
  - add_vault_lookups
  - push_changes

variables:
  VAULT_TOKEN: "$VAULT_TOKEN"
  VAULT_URL: "http://127.0.0.1:8200"
  KOLLA_CONFIG_PATH: "etc/kolla"
  GIT_USER_NAME: "GitLab CI"
  GIT_USER_EMAIL: "[email protected]"

compile:
  stage: setup
  before_script:
    - apt-get update -qq && apt-get install -y -qq python3-pip golang
    - go mod init kolla-export-vault || true
    - go get github.com/hashicorp/vault/api
    - go get gopkg.in/yaml.v2
  script:
    - go build -o store_kolla_passwords store_kolla_passwords.go
    - go build -o replace_kolla_passwords replace_kolla_passwords.go
  artifacts:
    paths:
      - store_kolla_passwords
      - replace_kolla_passwords

kolla_genpwd:
  stage: setup
  before_script:
    - apt-get update -qq && apt-get install -y -qq python3-pip
    - pip3 install kolla-ansible ansible --break-system-packages
    - cp /usr/local/share/kolla-ansible/etc_examples/kolla/passwords.yml $KOLLA_CONFIG_PATH/passwords.yml
  script:
    - kolla-genpwd
  artifacts:
    paths:
      - $KOLLA_CONFIG_PATH/passwords.yml

push_in_vault:
  stage: push_in_vault
  script:
    - ./store_kolla_passwords
  when: manual
  rules:
    - if: '$CI_COMMIT_TAG == "KOLLA_BOOTSTRAP"'
    - if: '$VAULT_PATH != "production"'
    - when: never
  needs: ["compile", "kolla_genpwd"]

add_vault_lookups:
  stage: add_vault_lookups
  script:
    - ./replace_kolla_passwords
  when: manual
  rules:
    - if: '$CI_COMMIT_TAG == "KOLLA_BOOTSTRAP"'
    - if: '$VAULT_PATH != "production"'
    - when: never
  artifacts:
    paths:
      - $KOLLA_CONFIG_PATH/passwords.yml
  needs: ["compile", "kolla_genpwd", "push_in_vault"]

push_changes:
  stage: push_changes
  before_script:
    - apt-get update -qq && apt-get install -y -qq git
  script:
    - git config --global user.email "$GIT_USER_EMAIL"
    - git config --global user.name "$GIT_USER_NAME"
    - git reset --hard origin/main
    - BRANCH_NAME="update-passwords-$(date +'%Y%m%d')"
    - git checkout -b "$BRANCH_NAME"
    - git add $KOLLA_CONFIG_PATH/passwords.yml
    - git commit -m "Add protected passwords.yml"
    - git push -u https://gitlab-ci-token:${PROJECT_ACCESS_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git  $BRANCH_NAME
  when: manual
  rules:
    - if: '$CI_COMMIT_TAG == "KOLLA_BOOTSTRAP"'
    - if: '$VAULT_PATH != "production"'
    - when: never
  needs: ["add_vault_lookups"]

In this CI, there are 4 stages:

1. setup: In this stage, dependencies and go packages are installed and the Go scripts are compiled. The kolla passwords are generated and the passwords.yml file is populated
2. push_in_vault: In this stage, the script that store all the passwords as Vault secrets is executed.
3. add_vault_lookups: In this stage, the passwords.yml plaintext passwords are replaced with Vault lookups.
4. push_changes: In this stage, the new passwords.yml file is pushed to a separate branch, with a timestamp.

The when: never is to ensure this job will never run if it's not triggered manually and with the KOLLA_BOOTSTRAP tag and if the VAULT_PATH is not production. Only the first two steps will run automatically.

As explained in the introduction, this is to prevent pushing a new set of passwords to Vault by mistake.

Artifacts

The push_in_vault and add_vault_lookups jobs use the artifacts from the compile and kolla-genpwd jobs. The generated artifacts are:

• a store_kolla_passwords binary (compile job)
• a replace_kolla_passwords binary (compile job)
• a populated passwords.yml file (kolla-genpwd job)

Demo of the pipeline

Dependencies

In this screenshot, you can see the dependencies: some jobs depend on other jobs to run. The two jobs in the setup stage run in parallel:

The push_in_vault stage needs the compile and kolla_genpwd jobs to finish:

The add_vault_lookup job also needs the compile and kolla_genpwd jobs, and will only run if the push_in_vault stage is completed:

Finally, all the stages must be completed successfully before being able to push the passwords.yml file to the new branch:

Setup

The first two steps are for building the Go binaries and generating the passwords:

Pushing to Vault and updating the artifact

The next two jobs are manually triggered:

It's safe to allow the automatic execution of the add_vault_lookups stage, as it only modifies the artifact, but I have decided to keep it manually triggered for now.

Push the changes

The last job to manually trigger is to get the new generated artifact, passwords.yml and push it in the repo.

Validate

The pipeline should be successful:

You should have a new branch, timestamped, with the updated passwords.yml file:

 (main)$ git branch -r
  origin/HEAD -> origin/main
  origin/update-passwords-20240822

And the new passwords.yml pushed in this new branch, under etc/kolla:

Now you are ready to create your merge request!

Conclusion

With all of that, we automated the entire password generation, export to Vault, and adding Vault lookups in the playbook through this pipeline.

This is just a small example of what you can do with GitLab CI. It’s incredibly powerful: you can handle virtually everything, from bootstrapping and deploying to upgrading and testing your infrastructure.

If you want more informations about the scripts, check the part I here: Hashicorp Vault and Kolla Ansible, Part I: Integrate Vault secrets in your playbook.