Modern datacenter security best-practices call for us to use TLS within our infrastructure, as a "defense in depth" approach to reducing the impact of intrusions. But managing TLS certificates for this usually requires running an in-house certificate authority, which can be difficult to set up and tedious to run.
This article shows how Hashicorp Terraform, a tool normally associated with infrastructure provisioning, can be used to create and manage a small in-house certificate authority with minimal hassle. It also shows how such an approach might be used with Hashicorp Vault to establish a CA with which to configure its TLS certificates and PKI backend.
Why use Terraform?
Those running a private CA will usually use the openssl command line tool or some wrapper around it such as easyrsa. When running a CA in this manner there are many different (and often cryptic) commands to learn and many small files to keep track of, which creates a steep learning curve and requires complex procedures to keep track of the CA state in a secure manner.
Terraform has built into it a TLS provider that contains the TLS primitives necessary to run a simple certificate authority. Terraform's TLS support is in turn based on the crypto libraries that come with the Go programming language, which are also used by Hashicorp Vault for much of its cryptography work.
Terraform has two characteristics that make it more convenient for this purpose than typical CLI-based tooling: its declarative configuration language provides a straightforward way to describe the certificates and other resources required, and its "state" concept gives us a single artifact that retains all of the necessary state for the CA, allowing us to more easily establish processes for securely storing this data.
The configuration can safely be stored in a version control repository for easy collaboration.
It is important to handle the state file with care: an organization following the process described in the following sections will create a state file which, if obtained by an attacker, would undermine the entire CA by giving that attacker the ability to arbitarily issue trusted certificates. Those who run the CA must define processes for how and where the state file will be stored, how it can be obtained by CA operators in order to issue new certificates, etc. It may be desirable to run Terraform only on a specific trusted, hardened host when interacting with the CA, to prevent remnants of the state file from being left on-disk on various different computer systems.
This article presumes some familiarity with Terraform, and in particular familiarity with its general workflow.
Establishing the Root Certificate
A root certificate is one that stands on its own and is not vouched for by any other certificate. Unless your CA is subordinate to another (an idea we'll explore more later), your CA will be built around a root certificate that must be explicitly trusted by any systems that will accept the certificates issued by your CA.
Another way to refer to a certificate that is not vouched for by another is the idea of a self-signed certificate. This is what it sounds like: the certificate "vouches for" itself, claiming both to own and to be verified by the same private key.
To produce a self-signed certificate for our CA we must first generate our CA's private key, which will then be used to sign the certificate. A Terraform config file root.tf can do this as follows:
We first generate a private key, and then use that key to produce a self-signed certificate. The content of subject doesn't matter that much for an internal CA, but it's important to specify is_ca_certificate and the cert_signing allowed use as shown here, or else clients will not accept certificates that descend from this one.
The validity_period_hours argument defines when this certificate will expire. In this example we set this to three years, but you should consider your own context when choosing an appropriate value to use. Any child certificates must expire before the root expires.
Terraform's early_renewal_hours attribute will cause Terraform to produce a new root certificate at some point before the current one has expired. Here we have set this to one year. Terraform's built-in dependency management will cause all issued certificates to be re-created automatically once a replacement root is established, allowing the CA administrators to re-issue them and get all systems updated before the original certificates become invalid.
Running terraform apply against this configuration will cause Terraform to generate the private key and the certificate and write both of them into the state file. terraform show will display all of the attributes of these resources, including the cert_pem attribute of the self-signed certificate, whose value can be installed on other systems to establish trust of the CA.
Issuing a Certificate
The usual workflow for a CA consists of individuals or departments outside of the CA team requesting certificates using certificate signing requests, or CSRs.
A CSR is a machine-readable description of the desired certificate, signed by the private key held by the party that will use the certificate. The job of the CA is to verify that the CSR is trustworthy and correct, and then issue a certificate vouching for the given information.
The teams requesting certificates would likely not be using Terraform, and will probably generate a CSR using some other workflow, such as the following openssl commands:
$ openssl genrsa -out example.com.key 2048 Generating RSA private key, 2048 bit long modulus .........+++ .........................................................................+++ e is 65537 (0x10001) $ openssl openssl req -new -sha256 -key example.com.key -out example.com.csr You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:CA Locality Name (eg, city) []:Pirate Harbor Organization Name (eg, company) [Internet Widgits Pty Ltd]:Example, Inc. Organizational Unit Name (eg, section) []:IT Department Common Name (e.g. server FQDN or YOUR name) []:intranet.example.com Email Address []:it@example.com
Our hypothetical IT department would send example.com.csr (but not example.com.key) to the CA operators. Those on the CA team would then inspect the CSR and see if the details within appear correct and compliant with organizational standards:
$ openssl req -text -noout -verify -in csrs/example.com.pem
verify OK
Certificate Request:
Data:
Version: 0 (0x0)
Subject: C=US, ST=CA, L=Pirate Harbor, O=Example, Inc., OU=IT Department, CN=intranet.example.com/emailAddress=it@example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:a7:b6:65:d6:c7:e0:02:2e:6a:d2:d4:a3:3f:fc:
bd:1f:85:85:af:8b:1e:35:f7:82:bd:83:b5:31:6b:
4b:62:4d:64:f4:07:cc:2f:c1:fd:c3:8b:47:41:22:
a6:f1:bf:2a:4f:8a:f8:6f:c7:7c:55:b6:d3:03:5d:
e6:1a:25:f2:64:bb:b2:30:d2:cd:ab:d0:4d:be:17:
d6:41:86:20:1c:bf:70:c2:f3:5f:58:fe:d4:49:85:
c7:ae:b3:29:6c:04:1c:44:80:9b:fc:19:24:b1:2d:
ab:6a:9e:d8:99:aa:a9:51:ee:29:02:95:da:b7:38:
98:78:73:37:19:28:e6:dc:e0:94:6f:b9:75:3c:d5:
ee:45:b7:87:b9:39:b8:62:86:0c:fa:71:e2:71:49:
62:04:a5:a6:0b:da:6e:99:90:c4:e3:2b:cb:23:90:
43:5b:d4:f3:fc:3d:66:b8:3f:18:d0:bd:ed:d0:dd:
7d:6f:6e:4f:a6:da:3f:64:a8:5b:fd:18:06:90:34:
96:a4:6c:70:b8:a7:99:e2:be:58:4f:57:8d:11:d8:
f5:1c:0a:9d:17:be:04:d1:33:ed:26:26:b4:e2:c0:
5e:e5:67:91:8c:9f:07:7f:f5:99:71:08:e6:f8:15:
5e:4d:74:6f:9f:c1:92:25:87:fa:44:d5:cb:b2:19:
2d:53
Exponent: 65537 (0x10001)
Attributes:
a0:00
Signature Algorithm: sha256WithRSAEncryption
11:78:db:dd:da:14:e6:bf:f8:85:d8:d8:5f:7d:e5:69:38:f4:
33:fb:d4:d5:09:a6:68:c2:a7:e0:03:56:93:0b:29:78:61:95:
76:09:aa:82:6d:5e:44:21:de:57:ce:1f:32:1e:4a:6c:30:0f:
18:ac:7b:22:04:e6:a0:55:da:a0:e8:98:9b:d0:62:c1:56:9f:
04:1d:6e:e0:e5:11:2d:3f:4e:a5:79:08:60:ee:a7:ba:ed:91:
a9:96:6d:23:93:bb:09:d1:01:5b:9e:cd:9e:93:9b:8e:3a:16:
a8:7c:f8:e4:36:e9:7f:60:ca:0d:38:df:89:a9:b3:ce:6a:8e:
05:95:31:53:69:c7:d6:8c:f9:b4:8a:c1:bc:70:2d:15:93:ce:
2d:e4:09:a0:55:30:23:d1:06:23:37:bf:5c:f4:49:f6:dc:44:
da:5b:fe:4d:54:16:55:80:d2:c0:65:f6:b1:0e:06:ee:30:4a:
e9:48:31:28:13:7f:8d:02:a0:ff:99:48:68:23:5a:68:fd:cf:
35:a4:a9:fb:99:28:18:d1:c8:87:79:44:08:a2:07:19:83:f7:
a8:e5:1b:62:fe:a7:4e:89:d7:57:79:cb:c1:05:e7:ea:86:28:
50:a4:b2:38:0c:97:f1:a0:99:03:3e:dc:12:d2:97:b2:df:e5:
3b:d4:45:9bAssuming everything looks good, the CSR file can be placed in a file within the configuration called (for example) csrs/example.com.pem and the certificate itself can be issued through another Terraform resource block, in intranet.tf:
This time we use the tls_locally_signed_cert resource, which combines a certificate request with a CA certificate (and its associated key) in order to produce a descendent certificate.
The allowed_uses keyword server_auth means that this server can be presented by a server to a client, and used by the client to verify the server. This is the appropriate setting for a certificate that will be configured for a TLS server.
We set the validity period of the certificate to two years. As noted earlier, this is required to be earlier than the CA certificate expiration, and so one must be careful to set this appropriately if a new certificate is issued toward the end of the life of the root certificate.
Now we can apply to issue the certificate:
$ terraform apply tls_private_key.root: Refreshing state... (ID: ...) tls_self_signed_cert.root: Refreshing state... (ID: ...) tls_locally_signed_cert.intranet: Creating... allowed_uses.#: "" => "1" allowed_uses.0: "" => "server_auth" ca_cert_pem: "" => "444386791920640878cc460b7aeaddcb715c831f" ca_key_algorithm: "" => "ECDSA" ca_private_key_pem: "" => "5038895240aab6f181a150d1fae9b57689b1483e" cert_pem: "" => "<computed>" cert_request_pem: "" => "b051ed1d744c164461d860e74ff320b1aaf49e87" early_renewal_hours: "" => "8760" validity_end_time: "" => "<computed>" validity_period_hours: "" => "17520" validity_start_time: "" => "<computed>" tls_locally_signed_cert.website: Creation complete Apply complete! Resources: 1 added, 0 changed, 1 destroyed. $ terraform show ... tls_locally_signed_cert.website: id = 135470627697209310048898171923231502097 allowed_uses.# = 1 allowed_uses.0 = server_auth ca_cert_pem = 444386791920640878cc460b7aeaddcb715c831f ca_key_algorithm = ECDSA ca_private_key_pem = 5038895240aab6f181a150d1fae9b57689b1483e cert_pem = -----BEGIN CERTIFICATE----- MIIDmjCCAvugAwIBAgIQZeqxn+/wwQfrlNBA0PFvETAKBggqhkjOPQQDBDCBwzEL MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1QaXJhdGUgSGFyYm9y MRkwFwYDVQQJExA1ODc5IENvdHRvbiBMaW5rMRMwEQYDVQQREwo5NTU1OS0xMjI3 MRUwEwYDVQQKEwxFeGFtcGxlLCBJbmMxLDAqBgNVBAsTI0RlcGFydG1lbnQgb2Yg Q2VydGlmaWNhdGUgQXV0aG9yaXR5MRowGAYDVQQDExFFeGFtcGxlIEluYy4gUm9v dDAeFw0xNjA5MTIwMDIxMzdaFw0xODA5MTIwMDIxMzdaMIGBMQswCQYDVQQGEwJV UzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDVBpcmF0ZSBIYXJib3IxFjAUBgNVBAoT DUV4YW1wbGUsIEluYy4xFjAUBgNVBAsTDUlUIERlcGFydG1lbnQxHTAbBgNVBAMT FGludHJhbmV0LmV4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEAp7Zl1sfgAi5q0tSjP/y9H4WFr4seNfeCvYO1MWtLYk1k9AfML8H9w4tH QSKm8b8qT4r4b8d8VbbTA13mGiXyZLuyMNLNq9BNvhfWQYYgHL9wwvNfWP7USYXH rrMpbAQcRICb/BkksS2rap7YmaqpUe4pApXatziYeHM3GSjm3OCUb7l1PNXuRbeH uTm4YoYM+nHicUliBKWmC9pumZDE4yvLI5BDW9Tz/D1muD8Y0L3t0N19b25Ppto/ ZKhb/RgGkDSWpGxwuKeZ4r5YT1eNEdj1HAqdF74E0TPtJia04sBe5WeRjJ8Hf/WZ cQjm+BVeTXRvn8GSJYf6RNXLshktUwIDAQABo0YwRDATBgNVHSUEDDAKBggrBgEF BQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFCve2v+DpZZ3Sgrn/pbH8xhi N5DcMAoGCCqGSM49BAMEA4GMADCBiAJCAfBsc8zW82bzpPJac934PpxB+1axwMAF 1B8x3ojSePhW6RD4ukld/73bAHmdWRZOlIezqf2XeTQQM9tZZq739MimAkIAq50y xJRdSNXGDbl/aq/ltWwfsv+m/hj16Npi48wAHqefi2VuS6ALjRzgCkH1OwWqxbi4 tjNXCtrWsvmiDgz9sI4= -----END CERTIFICATE----- cert_request_pem = b051ed1d744c164461d860e74ff320b1aaf49e87 early_renewal_hours = 8760 validity_end_time = 2018-09-11T17:21:37.024909783-07:00 validity_period_hours = 17520 validity_start_time = 2016-09-11T17:21:37.024909783-07:00
The cert_pem value here is what the CA operator would provide to the IT department, along with the CA certificate created earlier, so that they can configure this hypothetical Intranet server.
Requesting Certificates from within Terraform
The previous section assumed that the requester of the certificate was distinct from the CA operations team, and presented a workflow supporting that situation. In a smaller team, it's very possible that the CA will be run by the same individuals that are configuring the rest of the infrastructure, and in that situation it might be appropriate to request and issue the certificates entirely within Terraform.
We can use the tls_cert_request resource, along with some other resources we've already seen, to make Terraform orchestrate the request/issue process we described above, in a new file infrastructure.tf:
Where before the private key and CSR were created using subcommands, this time we use Terraform resources to achieve the same result. After a single terraform apply, all of these resources will be created and the certificate's PEM serialization can be obtained from terraform show as before.
Extracting Certificates to Standalone Files
In the previous sections we saw how we can locate the cert_pem attribute on our generated certificates in order to obtain the PEM-encoded certificate contents.
Most software expects to be provided keys and certificates each in their own separate on-disk file. It would be convenient to automatically extract these values into such files, and that is relatively easy to achieve since the Terraform state file is JSON-encoded and easy to consume from scripts. As part of the prototyping for this article, I wrote a Python script to extract the certificates, creating a directory for each distinct certificate name and placing files in here for the issued cert and the CA cert respectively.
Setting up a Vault server with a TLS certificate
Hashicorp Vault has been generally praised for striking a good compromise between security and usability. For many situations, it will do the right thing "out of the box", lowering the barrier to having reasonably-secure handling of secrets within network applications.
However, one big upset to this ease of getting started is the chicken-and-egg problem of needing to establish enough certificate infrastructure to issue a certificate for the Vault server itself to use, before there's a Vault server in which to store or generate the necessary secrets.
Through a Terraform config like our infrastructure.tf example above, we can quickly produce and issue a server certificate and associated private key for Vault to use, and configure Vault clients to trust our CA certificate. The common name of this certificate must be the hostname at which clients will access the Vault server.
The listener section of a Vault configuration file is where we will specify the paths to files containing the Vault server's private key and certificate:
Clients connecting to the Vault server will also need to trust the root CA certificate. How this is done unfortunately varies depending on the operating system; on Linux systems one of the following files is consulted for lists of trusted root CA certificates in PEM format:
/etc/ssl/certs/ca-certificates.crt
/etc/pki/tls/certs/ca-bundle.crt
/etc/ssl/ca-bundle.pem
/etc/pki/tls/cacert.pem
Alternatively, the VAULT_CACERT environment variable can be set on the Vault process to explicitly specify the CA certificate that Vault should expect.
Issuing Certificates Automatically with Vault
Alongside the use of TLS for client-to-server communication, Vault also has a secret backend that allows it to automatically issue TLS certificates using the same underlying cryptography code that Terraform uses to implement the resources we've seen so far.
This Vault feature can be used to automate the issuing of certificates to TLS clients and servers, so it can be set up once and then run largely without operator intervention as new applications join and leave the environment. This can be an effective way to enable mutual authentication between TLS clients and servers within a datacenter, which can be a step towards a "zero trust" architecture.
The recommended way to use this feature is to establish a root CA, as we've done earlier in this article, and then use it to establish an intermediate CA within Vault. This means that Vault maintains its own CA that is subordinate to our root CA.
We can adapt our earlier infrastructure.tf configuration to create a new signing certificate for Vault to use, but first we need to ask Vault to generate a certificate signing request. At this step, Vault generates a private key but does not expose it to the operator:
$ vault mount pki Successfully mounted 'pki' at 'pki'! $ vault write /pki/intermediate/generate/internal common_name=vault-ca.example.com exclude_ca_from_sans=true Key Value csr -----BEGIN CERTIFICATE REQUEST----- MIICljCCAX4CAQAwHzEdMBsGA1UEAxMUdmF1bHQtY2EuZXhhbXBsZS5jb20wggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC7OjTpRfItq455zCKit3F98Wra vtSq0LjpQWMOBHQzG3OMrqdgAPlBLl38+8ey1j1f+GI15B5NeQhOwe2WqjlwFpzN xUcZPq40hghDMHycDAuClkDgv5J8vnTmA/mc8OdrwAlU9lHz0YR1hkq2jsgTRW1G 9b3G3D0JvhVpCtLUj7CU6i8IiMeBWt/xI2uh4SNQL44jB5mVxV5mEBZ/ldhnC4kU hFJoIdVO/AGzKQrm1horssditPVH0KaMia72LGeb/OEe93b46AjBy2PX2mFQJssV pWvAvvo7XaJJ/Ox/mA717j4blTbx0Hkp6j2g7xGJJXQOmbWgoJ8OsUylSArVAgMB AAGgMjAwBgkqhkiG9w0BCQ4xIzAhMB8GA1UdEQQYMBaCFHZhdWx0LWNhLmV4YW1w bGUuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQCvhAgfm9MlYFRlm/9FcWRitgxvOgkJ 95qNByFshsrhRHoyWWpZjNPIqkzWEFTAMsTVV18TkEOSmZZgcmqi3TWg61EKLoTm LRLv1bAa+xUv6i55LZOkllgCKfWGxOynFBHpn2KvwfR9A7N5lEM8FPOhY+WtxpWE G7FUhZN1wyqpfdWIxdWgyi9djpuBBEGAsXWKKvMX7U3MDKYoBJ+PXzTtPnXChL8Y CbsrURvpKLsnM+zKPl991P9d9ubXsXl0yeOSoOglNd8lN/CeHMueXAqyyQS+lLV+ drde2OSezqo/bBwFwtm8sw+kwZWuc+uLHm7He1kBfYxg2/Bpq38b6NBo -----END CERTIFICATE REQUEST-----
This CSR block (with the csr key name removed from the first line) can be placed at csrs/vault.pem and then we can create vault.tf to issue the certificate:
This time we again set is_ca_certificate and allow cert_signing, since this certificate will be used by Vault to issue further certificates.
After terraform apply, we can extract the generated certificate, place it in a file called vault.crt and complete setup by loading it into Vault:
$ vault write /pki/intermediate/set-signed -certificate=@vault.crt
See the Vault documentation on the PKI auth backend for more information on how this can be used to issue short-lived server and client certificates (and associated private keys) for applications to use.
Terraform CA Caveats
In addition to the earlier warning about the need to securely store the Terraform state containing the CA secrets, there are some other caveats to keep in mind when using Terraform as the basis of a certificate authority.
Certificates issued by Terraform will not specify the location of a certificate revocation list, meaning that there is no ready mechanism to cancel any issued certificates in the event that they are compromised. It is therefore ideal to use Terraform only for the most foundational parts of the CA, which can be protected most carefully, and then use it to provision a system like Vault's PKI secret backend described above in order to delegate the more routine issuing of certificates to a more specialized tool. In this case, Terraform is just used to solve the chicken-and-egg problem in its initial configuration.
Additionally, a Terraform configuration with hundreds or thousands of resources is currently not especially performant, and will result in an oversize state file that may be hard to transmit and store securely. This is a further reason to limit Terraform's role to the initial setup, and delegate broader management tasks to a more appropriate system.
Conclusion
In this article we've seen how Terraform can be used to establish and operate a small certificate authority within an organization, and explored some practical situations where such a solution may be helpful.
As always with security and cryptography concerns, context is important and there is no "one size fits all" solution. Where possible I have tried to make my assumptions explicit, but when considering the techniques within this article, be sure to consider any regulatory or organizational constraints that may affect the applicability of this technique within your particular environment.