Add option to use vault to manage key used for encryption

Using a local PGP-key for encryption of the secrets in the configuration
is not very secure and makes it hard to rotate and distribute the
key. Sops provides the option to use managed services for this
purpose, e.g. HashiCorp Vault.

This change adds the option to use HashiCorp Vault, when using the
provided python scripts to encrypt the config file.

Change-Id: I7683fbfdbed00506c3bca264ac8565f48bc5ea73

documentation/config-management.md

@@ -3,9 +3,8 @@
 The configuration in the `config.yaml` contains secrets and should not be openly
 accessible. To secure the data contained within it, the values can be encrypted
 using a tool called [`sops`](https://github.com/mozilla/sops). This tool will use
-a GPG-key to encrypt the values of the yaml file. Having the PGP-key also allows
+a key to encrypt the values of the yaml file. Access to the key allows decryption of the values.
-to decrypt the values and work with the file. As long as the key is not compromised,
+As long as the key is not compromised, the encrypted file can be shared securely between collaborators.
-the encrypted file can be shared securly between collaborators.
 
 The process of using `sops` is described below.
 
@@ -17,6 +16,10 @@ On OSX, `sops` can be installed using brew:
 brew install sops
 ```
 
+## Using a local PGP key
+
+### Install GPG
+
 Install `gpg`:
 
 ```sh
@@ -31,7 +34,7 @@ GPG_TTY=$(tty)
 export GPG_TTY
 ```
 
-## Create GPG-key (first time only)
+### Create GPG-key (first time only)
 
 Create a key by running the following command and following the instructions on
 the screen:
@@ -40,7 +43,7 @@ the screen:
 gpg --gen-key
 ```
 
-## Encrypt the config-file
+### Encrypt the config-file
 
 Run the following command to encode the file:
 
@@ -66,22 +69,15 @@ the file:
 pipenv run python ./gerrit-monitoring.py \
   --config config.yaml \
   encrypt \
-  --pgp "abcde1234"
+  --enc-method "pgp" \
+  --pgp-id "abcde1234"
 ```
 
 The gpg-key used to encrypt the file can be selected by giving the fingerprint,
-key ID or part of the unique ID to the `--pgp`-argument. This identifier has to
+key ID or part of the unique ID to the `--pgp-id`-argument. This identifier has to
 be unique among the keys in the GPG keystore.
 
-## Decrypt file
+### Export GPG-key
-
-To decrypt the file, run:
-
-```sh
-sops --in-place -d $FILE_TO_DECODE
-```
-
-## Export GPG-key
 
 For other developers or build servers to be able to decrypt the configuration,
 the key has to be exported:
@@ -98,6 +94,68 @@ gpg --import public.key
 gpg --allow-secret-key-import --import private.key
 ```
 
+## Encrypt using HashiCorp Vault
+
+### Install `vault` CLI tool
+
+On OSX, `vault` can be installed using brew:
+
+```sh
+brew install vault
+```
+
+### Log into vault
+
+Use the CLI to log into your vault instance:
+
+```sh
+vault login -method=<auth-method> -address=https://vault.example.com
+```
+
+### Create a key to use for encryption (first time only)
+
+To use sops with HashiCorp Vault, a secret engine of type transit containing
+at least one key has to be created:
+
+```sh
+vault secrets enable -path=some-engine transit
+vault write sops/keys/some-key type=rsa-4096
+```
+
+### Encrypt the config-file
+
+Run the following command to encode the file:
+
+```sh
+sops \
+  --encrypt \
+  --in-place \
+  --encrypted-regex '(password|htpasswd|cert|key|apiUrl|caCert|secret|accessToken)$' \
+  --hc-vault-transit https://vault.example.com/v1/some-engine/keys/some-key \
+  $FILE_TO_ENCODE
+```
+
+Alternatively, the `gerrit-monitoring.py encrypt`-script can be used to encrypt
+the file:
+
+```sh
+pipenv run python ./gerrit-monitoring.py \
+  --config config.yaml \
+  encrypt \
+  --enc-method "vault" \
+  --vault-url https://vault.example.com \
+  --vault-engine some-engine \
+  --vault-key some-key
+```
+
+## Decrypt file
+
+To decrypt the file, run:
+
+```sh
+sops --in-place -d $FILE_TO_DECODE
+```
+
 ## Links
 
 [1] https://github.com/mozilla/sops/issues/304

gerrit_monitoring.py

@@ -20,7 +20,14 @@ from subcommands import encrypt, install, uninstall
 
 
 def _run_encrypt(args):
-    encrypt(args.pgp_identifier, os.path.abspath(args.config))
+    encrypt(
+        args.enc_method,
+        os.path.abspath(args.config),
+        pgp_identifier=args.pgp_identifier,
+        vault_address=args.vault_address,
+        engine=args.vault_engine,
+        key=args.vault_key,
+    )
 
 
 def _run_install(args):
@@ -87,12 +94,41 @@ def main():
     parser_encrypt.set_defaults(func=_run_encrypt)
 
     parser_encrypt.add_argument(
-        "-p",
+        "-m",
-        "--pgp",
+        "--method",
+        help="Encryption method used by SOPS.",
+        dest="enc_method",
+        action="store",
+        choices=["pgp", "vault"],
+        required=True,
+    )
+
+    parser_encrypt.add_argument(
+        "--pgp-id",
         help="PGP fingerpint or associated email.",
         dest="pgp_identifier",
         action="store",
-        required=True,
+    )
+
+    parser_encrypt.add_argument(
+        "--vault-url",
+        help="URL of vault instance (incl. protocol).",
+        dest="vault_address",
+        action="store",
+    )
+
+    parser_encrypt.add_argument(
+        "--vault-engine",
+        help="Secret engine managing the key used for encryption.",
+        dest="vault_engine",
+        action="store",
+    )
+
+    parser_encrypt.add_argument(
+        "--vault-key",
+        help="Name of the key used for encryption.",
+        dest="vault_key",
+        action="store",
     )
 
     args = parser.parse_args()

subcommands/encrypt.py

@@ -29,18 +29,7 @@ ENCRYPTED_KEYS = [
 ]
 
 
-def encrypt(pgp_identifier, config_path):
+def _pgp(pgp_identifier, config_path):
-    """Encrypt the config file using sops and a PGP key.
-
-    Arguments:
-        pgp_identifier {string} -- A unique identifier of the PGP key to be used.
-            This can be the fingerprint, keyid or part of the uid (e.g. the email
-            address)
-        config_path {string} -- The path to the config file to be encrypted
-
-    Raises:
-        ValueError: Error, if no (unique) PGP key could be found
-    """
     gpg = gnupg.GPG()
     gpg_keys = gpg.list_keys()
     selected_keys = list(
@@ -69,3 +58,50 @@ def encrypt(pgp_identifier, config_path):
         config_path,
     ]
     subprocess.check_output(command)
+
+
+def _vault(vault_address, engine, key, config_path):
+    url = f"{vault_address}/v1/{engine}/keys/{key}"
+
+    command = [
+        "sops",
+        "--encrypt",
+        "--in-place",
+        "--encrypted-regex",
+        f"({'|'.join(ENCRYPTED_KEYS)})",
+        "--hc-vault-transit",
+        url,
+        config_path,
+    ]
+    subprocess.check_output(command)
+
+
+def encrypt(
+    method, config_path, pgp_identifier=None, vault_address=None, engine=None, key=None
+):
+    """Encrypt the config file
+
+    Args:
+        method {string}: The method of receiving the encryption key.
+            (options: 'pgp', 'vault')
+        config_path {string}: The path to the config file to be encrypted
+        pgp_identifier ({string}, optional): A unique identifier of the PGP key
+            to be used. This can be the fingerprint, keyid or part of the uid
+            (e.g. the email address). Required for method 'pgp'. Defaults to None.
+        vault_address ({string}, optional): Base URL of Vault incl. protocol.
+            Required for method 'vault'. Defaults to None.
+        engine ({string}, optional): Name of the secret engine. Required for
+            method 'vault'. Defaults to None.
+        key ({string}, optional): Name of the key. Required for method 'vault'.
+            Defaults to None.
+    """
+    if method == "pgp":
+        if pgp_identifier is None:
+            raise ValueError("Missing PGP identifier.")
+        _pgp(pgp_identifier, config_path)
+    elif method == "vault":
+        if None in [vault_address, engine, key]:
+            raise ValueError("Missing required metadata for accessing vault.")
+        _vault(vault_address, engine, key, config_path)
+    else:
+        raise ValueError("Unknown method to retrieve key for encryption.")