Context

After having successfully run Ansible on Windows using Docker, as documented inside my previous post, I thought about documenting how to use Ansible Vault on Windows.
This tool was included in Ansible since version 1.5 and its purpose is to ensure sensitive data like credentials, private keys, certificates, etc., used by Ansible playbooks, are stored encrypted.
This post will present my approach for running Ansible Vault on Windows using Docker, along with the issues I have encountered and their fixes.

As a real life example of when to use Ansible Vault, I have chosen the task of running a Docker container inside a virtual machine:

  • Create the VM
    • I’ll use Docker Machine to create a VM using Hyper-V driver; this approach has the added benefit of creating a VM which already has Docker installed
    • Beside having access to a Docker host with minimum medium effort, I ended up tinkering with a new Linux distro, other than what I’m usually exposed to (Ubuntu and CentOS)
  • Setup the VM to be managed by Ansible
    • Provide SSH access - already done, since Docker Machine will handle it while creating the VM
    • Provide a working Python version - as you’ll see below, this step is not difficult at all
  • Clone a git repository from GitHub containing the Ansible playbook used for running the Docker container based on the hello-world image
  • Add the Docker Hub credentials ued for pulling the image inside the appropriate Ansible variable YAML file
  • Run Ansible Vault from a Docker container to encrypt these credentials
  • Run the Ansible playbook used for pulling the Docker image, run the container, then remove them both

I will use satrapu/ansible-alpine-apk Docker image for running both Ansible and Ansible Vault on Windows.
All the Docker and Docker Machine related commands below must be executed inside a Powershell console run as admin (use Git Bash as a backup for some commands - e.g. “docker-machine ssh”).

Prerequisites

All versions below are the latest at the time of writing this particular section (March 26th, 2018).

  • Windows 10 Professional Edition (v1709)
  • Hyper-V
  • Docker for Windows - I’ve recently upgraded to v18.03.0-ce, but older versions should be good enough
  • Docker Machine - v0.13.0 or older, since v0.14.0 (coming with Docker for Windows v18.0.3.0-ce) is unable to create VMs using hyperv driver - see more details here
    • Right after I’ve upgraded Docker for Windows from 17.12.1-ce to v18.03.0-ce, I was no longer able to create VMs using Docker Machine and hyperv driver; this issue did not occur when using v0.13.0!
    • Download Docker Machine v0.13.0 from GitHub, rename it to docker-machine.exe and then move it inside %DOCKER_HOME%\resources\bin to overwrite the existing docker-machine.exe (v0.14.0)
  • Visual Studio Code - v1.21.1
    • Any other editor capable of switching line endings between CRLF and LF should be fine too - see below for the actual motivation behind this prerequisite ;)
  • Git - v2.16.2
    • The version is not that important, but installing Git Bash along with Git is!

Setup Ansible managed node using Docker Machine

  • Create a virtual network switch named ansible, as described here
  • Create a Hyper-V virtual machine named ansible-vault having 2 CPUs, 2048 MB RAM, 10 GB disk and attached to the previously created external virtual switch
    • The boot2docker ISO URL is explicitly set to fixate the Docker version (v18.03.0-ce) for repeatability purposes
    • Prepare to wait for a rather long period of time (10 minutes or more) for the VM to be created
    • Ignore the SSH reported error
docker-machine create `
               --driver hyperv `
               --hyperv-cpu-count 2 `
               --hyperv-memory 2048 `
               --hyperv-disk-size 10240 `
               --hyperv-virtual-switch "ansible" `
               --hyperv-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v18.03.0-ce/boot2docker.iso `
               ansible-vault
# Running pre-create checks...
# (ansible-vault) Boot2Docker URL was explicitly set to "https://github.com/boot2docker/boot2docker/releases/download/v18.03.0-ce/boot2docker.iso" at create time, so Docker Machine cannot upgrade this machine to the latest version.
# Creating machine...
# (ansible-vault) Boot2Docker URL was explicitly set to "https://github.com/boot2docker/boot2docker/releases/download/v18.03.0-ce/boot2docker.iso" at create time, so Docker Machine cannot upgrade this machine to the latest version.
# (ansible-vault) Downloading C:\Users\admin\.docker\machine\cache\boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v18.03.0-ce/boot2docker.iso...
# (ansible-vault) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
# (ansible-vault) Creating SSH key...
# (ansible-vault) Creating VM...
# (ansible-vault) Using switch "ansible"
# (ansible-vault) Creating VHD
# (ansible-vault) Starting VM...
# (ansible-vault) Waiting for host to start...
# Waiting for machine to be running, this may take a few minutes...
# Detecting operating system of created instance...
# Waiting for SSH to be available...
# Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded
  • Check that the VM is running (look for “STATE Running”):
 docker-machine ls
# NAME            ACTIVE   DRIVER   STATE     URL                        SWARM   DOCKER    ERRORS
# ansible-vault   -        hyperv   Running   tcp://192.168.1.168:2376           Unknown   Unable to query docker version: Get https://192.168.1.168:2376/v1.15/version: x509: certificate signed by unknown authority
  • Get the IPv4 address of the VM, since you’ll needed it inside the Ansible inventory file:
docker-machine ip ansible-vault
# 192.168.1.168
  • Connect to the VM using SSH (see more here)
    • In case you’re unable to enter the VM via SSH from a Powershell terminal, try using Git Bash run as admin - welcome to Windows!
docker-machine ssh ansible-vault
#                         ##         .
#                   ## ## ##        ==
#                ## ## ## ## ##    ===
#            /"""""""""""""""""\___/ ===
#       ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
#            \______ o           __/
#              \    \         __/
#               \____\_______/
#  _                 _   ____     _            _
# | |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
# | '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
# | |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
# |_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
# Boot2Docker version 18.03.0-ce, build HEAD : 404ee40 - Thu Mar 22 17:12:23 UTC 2018
# Docker version 18.03.0-ce, build 0520e24
  • Install Python and Python setup tools on the VM, as they are needed by Ansible - check this StackOverflow article for instructions.
    Keep in mind that all changes done to this machine will be lost after a restart, as documented here!
tce-load -wi python python-setuptools
# python.tcz.dep OK
# tk.tcz.dep OK
# readline.tcz.dep OK
# Downloading: libffi.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# python-setuptools.tcz.dep OK
# libffi.tcz           100% |*************************************************************************************************| 16384   0:00:00 ETA
# libffi.tcz: OK
# Downloading: expat2.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# expat2.tcz           100% |*************************************************************************************************| 73728   0:00:00 ETA
# expat2.tcz: OK
# Downloading: ncurses.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# ncurses.tcz          100% |*************************************************************************************************|   196k  0:00:00 ETA
# ncurses.tcz: OK
# Downloading: readline.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# readline.tcz         100% |*************************************************************************************************|   144k  0:00:00 ETA
# readline.tcz: OK
# Downloading: gdbm.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# gdbm.tcz             100% |*************************************************************************************************| 73728   0:00:00 ETA
# gdbm.tcz: OK
# Downloading: tcl.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# tcl.tcz              100% |*************************************************************************************************|  1128k  0:00:00 ETA
# tcl.tcz: OK
# Downloading: tk.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# tk.tcz               100% |*********************************************************************************************************************************************|   916k  0:00:00 ETA
# tk.tcz: OK
# Downloading: openssl.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# openssl.tcz          100% |*********************************************************************************************************************************************|  1500k  0:00:00 ETA
# openssl.tcz: OK
# Downloading: bzip2-lib.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# bzip2-lib.tcz        100% |*********************************************************************************************************************************************| 28672   0:00:00 ETA
# bzip2-lib.tcz: OK
# Downloading: sqlite3.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# sqlite3.tcz          100% |*********************************************************************************************************************************************|   388k  0:00:00 ETA
# sqlite3.tcz: OK
# Downloading: python.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# python.tcz           100% |*********************************************************************************************************************************************| 11820k  0:00:00 ETA
# python.tcz: OK
# Downloading: python-setuptools.tcz
# Connecting to repo.tinycorelinux.net (89.22.99.37:80)
# python-setuptools.tc 100% |*********************************************************************************************************************************************|   236k  0:00:00 ETA
# python-setuptools.tcz: OK

In case you forgot this step, when running Ansible playbook you’ll see something like this:

# PLAY [docker_hosts] ************************************************************

# TASK [Gathering Facts] *********************************************************
# fatal: [ansible_vault_example]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "/bin/sh: /usr/local/bin/python: not found\r\n", "msg": "MODULE FAILURE", "rc": 0}
#         to retry, use: --limit @/opt/ansible-playbooks/hello-world.retry

# PLAY RECAP *********************************************************************
# ansible_vault_example      : ok=0    changed=0    unreachable=0    failed=1
  • Display Python version
python --version
# Python 2.7.14
  • Exit the VM:
exit

Clone Ansible Vault example

  • Clone the following git repository hosted on GitHub somewhere on your Windows machine (e.g. E:\Satrapu\Programming\Ansible\ansible-vault-on-windows):
cd E:/Satrapu/Programming/Ansible
git clone https://github.com/satrapu/ansible-vault-on-windows.git

This git repo is based on the classic Ansible folder structure, as documented here.

  • Change the Ansible inventory file named local
    • Set the value of the ansible_host property to the IP address of the ansible-vault VM (e.g. ansible_host=192.168.1.168)
    • Please note property ansible_ssh_private_key_file has been set to “/opt/docker-machine/ansible-vault/id_rsa” value - the id_rsa represents a private key generated by Docker Machine while creating ansible-vault VM and which will be made available inside the Ansible Docker container via a Docker volume; this property should not be changed without fully understanding what else needs to be changed (see below)
  • Create a file named vault_password under ../ansible-vault-password folder (outside Git repo!) and add a password (one line, no line ending)
    • Since this file contains a password, it must not be put under source control, that’s why it should be created outside the Git repo
    • To make it available inside Ansible Docker container, we’ll mount the containing folder as a Docker volume under path “/opt/ansible-vault-password”
      • Example: “-v E:/Satrapu/Programming/Ansible/ansible-vault-password:/opt/ansible-vault-password”
    • I have used https://strongpasswordgenerator.com to generate such password
      • Click “Show Options” panel under the “Generate password” big green button to fine tune your password
  • Replace the TBD placeholders from the /ansible-vault-on-windows/group_vars/docker_hosts/vault.yml file:
vault_docker_registry_url: TBD
vault_docker_registry_auth_username: TBD
vault_docker_registry_auth_password: TBD
vault_docker_registry_auth_email: TBD

with the appropriate values, like this:

vault_docker_registry_url:  https://index.docker.io/v1/
vault_docker_registry_auth_username: some_user_name
vault_docker_registry_auth_password: P@zZwWwooRdddd
vault_docker_registry_auth_email: some_user_name@server.ro

This file should be put under source control once it has been encrypted.
For instance, the Docker Hub registry URL can be found via this command:

docker info | findstr Registry
# Registry: https://index.docker.io/v1/

In case you forgot to correctly update vault.yml file, when running Ansible playbook you should see something like this:

# PLAY [docker_hosts] ************************************************************

# TASK [Gathering Facts] *********************************************************
# ok: [ansible_vault_example]

# TASK [run_hello_world_container : Install pip] *********************************
# changed: [ansible_vault_example]

# TASK [run_hello_world_container : Install docker-py] ***************************
# changed: [ansible_vault_example]

# TASK [run_hello_world_container : Login into Docker registry TBD] **************
# fatal: [ansible_vault_example]: FAILED! => {"changed": false, "failed": true, "msg": "Parameter error: the email address appears to be incorrect. Expecting it to match /[^@]+@[^@]+\\.[^@]+/"}
#         to retry, use: --limit @/opt/ansible-playbooks/hello-world.retry

# PLAY RECAP *********************************************************************
# ansible_vault_example      : ok=3    changed=2    unreachable=0    failed=1
  • You’ll see a vars.yml file under the same folder, /ansible-vault-on-windows/group_vars/docker_hosts:
docker_registry_url: "{{ vault_docker_registry_url }}"
docker_registry_auth_username: "{{ vault_docker_registry_auth_username }}"
docker_registry_auth_password: "{{ vault_docker_registry_auth_password }}"
docker_registry_auth_email: "{{ vault_docker_registry_auth_email }}"

Ansible will use the password residing inside the one-line file passed as the value of the –vault-password-file argument (e.g. –vault-password-file=/opt/ansible-vault-password/vault_password) to automatically decrypt the vault.yml file and will populate the above variables with the correct sensitive data, e.g. the user name and password used for pulling images from Docker Hub.

  • After applying the aforementioned changes, the local git repo should look like this:
# Change drive letters and paths according to your local setup
E:; cd E:/Satrapu/Programming/Ansible/ansible-vault-on-windows; tree /F
# E:\SATRAPU\PROGRAMMING\ANSIBLE\ANSIBLE-VAULT-ON-WINDOWS
# │   .gitattributes
# │   .gitignore
# │   ansible.cfg
# │   hello-world.yml
# │   LICENSE
# │   local
# │   README.md
# │   vault_password_provider.py
# │
# ├───group_vars
# │   └───docker_hosts
# │           vars.yml
# │           vault.yml
# │
# └───roles
#     └───run_hello_world_container
#         ├───defaults
#         │       main.yml
#         │
#         └───tasks
#                 main.yml

Encountered issues

Issue #1: Executable bit

Running Ansible Vault from a Docker container will fail since I’m trying to mount a Windows folder in a Linux container and all of its files will be mounted with all Linux permissions (read, write and execute):

docker container run `
                 --rm `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-on-windows:/opt/ansible-playbooks `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-password:/opt/ansible-vault-password `
                 satrapu/ansible-alpine-apk:2.4.1.0-r0 `
                 ansible-vault encrypt `
                    --vault-password-file=/opt/ansible-vault-password/vault_password `
                    ./group_vars/docker_hosts/vault.ym
#  [WARNING]: Error in vault password file loading (default): Problem running
# vault password script /opt/ansible-vault-password/vault_password ([Errno 8]
# Exec format error). If this is not a script, remove the executable bit from the
# file.
# ERROR! Problem running vault password script /opt/ansible-vault-password/vault_password ([Errno 8] Exec format error). If this is not a script, remove the executable bit from the file.

Here are the permissions found inside the Docker container:

docker container run `
                 --rm `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-password:/opt/ansible-vault-password `
                 satrapu/ansible-alpine-apk:2.4.1.0-r0 `
                 ls -al /opt/ansible-vault-password
# total 5
# drwxr-xr-x    2 root     root             0 Mar 29 19:56 .
# drwxr-xr-x    1 root     root          4096 Mar 29 20:03 ..
# -rwxr-xr-x    1 root     root           100 Mar 24 19:20 vault_password

The above executable bit related error message is pretty clear, unfortunately, at the moment there is no easy way of mounting files without the execute bit, as stated here.

On the other hand, Ansible knows how to process a file with executable bit containing a Vault password if it is a Python script, as documented here, so the idea is to load the password via a Python script, which will be passed as the value of the –vault-password-file argument - see an example here.
At this moment I’m able to bypass the pesky Windows-Docker-folder-mounting issue, but this has lead me to the 2nd issue :)

Issue #2: Line endings

Ansible Vault being able to run a Python script which returns the password is great news, but keep in mind we’re still editing files on Windows, which uses CRLF as line ending, which, of course, will not work on Linux:

docker container run `
                 --rm `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-on-windows:/opt/ansible-playbooks `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-password:/opt/ansible-vault-password `
                 satrapu/ansible-alpine-apk:2.4.1.0-r0 `
                 ansible-vault encrypt `
                    --vault-password-file=./vault_password_provider.py `
                    ./group_vars/docker_hosts/vault.yml
#  [WARNING]: Error in vault password file loading (default): Problem running
# vault password script /opt/ansible-playbooks/vault_password_provider.py ([Errno
# 2] No such file or directory). If this is not a script, remove the executable
# bit from the file.
# ERROR! Problem running vault password script /opt/ansible-playbooks/vault_password_provider.py ([Errno 2] No such file or directory). If this is not a script, remove the executable bit from the file.

The fix is to edit vault_password_provider.py with an editor having line endings set for this file to “LF” instead of “CRLF” - see such setup for Visual Studio Code.

Ansible Vault commands

Having fixed the above 2 issues, the following Ansible Vault commands will work like a charm:

  • Encrypt vault.yml:
docker container run `
                 --rm `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-on-windows:/opt/ansible-playbooks `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-password:/opt/ansible-vault-password `
                 satrapu/ansible-alpine-apk:2.4.1.0-r0 `
                 ansible-vault encrypt `
                    --vault-password-file=./vault_password_provider.py `
                    ./group_vars/docker_hosts/vault.yml
  • Decrypt vault.yml:
docker container run `
                 --rm `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-on-windows:/opt/ansible-playbooks `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-password:/opt/ansible-vault-password `
                 satrapu/ansible-alpine-apk:2.4.1.0-r0 `
                 ansible-vault decrypt `
                    --vault-password-file=./vault_password_provider.py `
                    ./group_vars/docker_hosts/vault.yml
  • View the decrypted vault.yml:
docker container run `
                 --rm `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-on-windows:/opt/ansible-playbooks `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-password:/opt/ansible-vault-password `
                 satrapu/ansible-alpine-apk:2.4.1.0-r0 `
                 ansible-vault view `
                    --vault-password-file=./vault_password_provider.py `
                    ./group_vars/docker_hosts/vault.yml
# vault_docker_registry_url: https://index.docker.io/v1/
# vault_docker_registry_auth_username: xxxxxxx
# vault_docker_registry_auth_password: xxxxxxx
# vault_docker_registry_auth_email: xxxxxxx

Run Ansible via Docker container

  • Run Ansible playbook:
# Replace <YOUR_ADMIN_USERS> placeholder with the Windows user name used for creating ansible-vault VM.
# Tip: Increase the verbosity of the ansible-playbook output by adding "-vvv" option at the end of the below line
docker container run `
                 --rm `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-on-windows:/opt/ansible-playbooks `
                 -v E:/Satrapu/Programming/Ansible/ansible-vault-password:/opt/ansible-vault-password `
                 -v C:/Users/<YOUR_ADMIN_USERS>/.docker/machine/machines/ansible-vault:/opt/docker-machine/ansible-vault `
                 satrapu/ansible-alpine-apk:2.4.1.0-r0 `
                 ansible-playbook `
                    --inventory-file=local `
                    --vault-password-file=./vault_password_provider.py `
                    hello-world.yml

# PLAY [docker_hosts] ************************************************************

# TASK [Gathering Facts] *********************************************************
# ok: [ansible_vault_example]

# TASK [run_hello_world_container : Install pip] *********************************
# ok: [ansible_vault_example]

# TASK [run_hello_world_container : Install docker-py] ***************************
# ok: [ansible_vault_example]

# TASK [run_hello_world_container : Login into Docker registry https://index.docker.io/v1/] ***
# changed: [ansible_vault_example]

# TASK [run_hello_world_container : Pull Docker image hello-world:linux] *********
# changed: [ansible_vault_example]

# TASK [run_hello_world_container : Logout from Docker registry https://index.docker.io/v1/] ***
# ok: [ansible_vault_example]

# TASK [run_hello_world_container : Run Docker container hello-world-from-satrapu] ***
# changed: [ansible_vault_example]

# TASK [run_hello_world_container : Remove Docker container hello-world-from-satrapu] ***
# changed: [ansible_vault_example]

# TASK [run_hello_world_container : Remove Docker image hello-world:linux] *******
# changed: [ansible_vault_example]

# PLAY RECAP *********************************************************************
# ansible_vault_example      : ok=9    changed=5    unreachable=0    failed=0

Resources

ansible   ansible-vault   boot2docker   docker   docker-machine   hyper-v   programming   windows