Friday, October 30, 2015

Consul: Adding TLS to Consul using Self Signed Certificates

I'm currently working on setting up TLS for Consul. As of this writing, I'm still in the experimentation/set-up phase, but we plan to roll consul out into production with TLS support. So, this document may get updated but I wanted to capture what I had to do while it's fresh in my mind.

Consul's documents are a little light on specifics, which made this endeavor more difficult than I anticipated. I will post links at the bottom of this article. The following steps were used to create a self signed certificate on Centos 6.6. 

Make a directory to hold our files, create the certificate authority (ca) conf file, seed our index and create a cert index file:

> mkdir -p /opt/consul/ssl
> cat << EOF > /opt/consul/ssl/demo.conf
[ ca ]
default_ca = demo

[ crl_ext ]
# issuerAltName=issuer:copy  #this would copy the issuer name to altname

[ demo ]
new_certs_dir = /tmp
unique_subject = no
certificate = /opt/consul/ssl/demo-root.cer
database = /opt/consul/ssl/certindex
private_key = /opt/consul/ssl/privkey.pem
serial = /opt/consul/ssl/serial
default_days = 365
default_md = sha1
policy = demo_policy
x509_extensions = demo_extensions

[ demo_policy ]
commonName = supplied
stateOrProvinceName = supplied
countryName = supplied
emailAddress = optional
organizationName = supplied
organizationalUnitName = optional

[ demo_extensions ]
basicConstraints = CA:false
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
keyUsage = digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth
crlDistributionPoints = URI:

> touch /opt/consul/ssl/certindex
> echo 000a > /opt/consul/ssl/serial
> cd /opt/consul/ssl

NOTE: You may need this step to ensure the certs work with Consul. Edit the /etc/pki/tls/openssl.cnf file and add the following:


Below I have one way to make the change. Read more about extended key usage here:

> cp /etc/pki/tls/openssl.cnf  /etc/pki/tls/openssl.cnf.bak
> sed -i"" 's|# extendedKeyUsage = critical,timeStamping|extendedKeyUsage=serverAuth,clientAuth|' /etc/pki/tls/openssl.cnf

Generate the root certificate:

> openssl req -newkey rsa:2048 -days 3650 -x509 -nodes -out /opt/consul/ssl/demo-root.cer -keyout /opt/consul/ssl/private.pem 
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:New York
Locality Name (eg, city) [Default City]:New York
Organization Name (eg, company) [Default Company Ltd]:Demo Company
Organizational Unit Name (eg, section) []:Demo
Common Name (eg, your name or your server's hostname) []:
Email Address []:

Consul want's the certs and the servers to be server.<data center>.consul - just adjust the request below as I used dc1 as my datacenter. Generate a certificate signer request (csr):

> openssl req -newkey rsa:1024 -nodes -out /opt/consul/ssl/server.csr -keyout /opt/consul/ssl/server.key
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New-York
Locality Name (eg, city) []:New York
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Demo Company
Organizational Unit Name (eg, section) []:Demo
Common Name (e.g. server FQDN or YOUR name) []:server.dc1.consul
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Generate the self signed cert:

> openssl ca -batch -config /opt/consul/ssl/demo.conf -notext -in /opt/consul/ssl/server.csr -out /opt/consul/ssl/server.cer

To verify your certificate use the following command and make sure the "X509v3 Extended Key Usage" matches:

> openssl x509 -noout -text -in /opt/consul/ssl/server.cer
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication

Now you can configure your consul server to use the self signed certs. These lines were take out of my consul.json file:

    "ca_file": "/opt/consul/ssl/demo-root.cer",
    "cert_file": "/opt/consul/ssl/server.cer",
    "key_file": "/opt/consul/ssl/server.key",

On your agents, you're going to need to specify the "ca_file" and set "verify_outgoing":true in your consul configs. 

If you get errors about trusting the signing authority, you will need to trust the demo-root.cer. To trust the root certificate on your server(s) do the following:

1. Install the ca-certificates package
2. Enable the dynamic CA configuration feature
3. Add it as a new file to /etc/pki/ca-trust/source/anchors/:
4. Use command:

> yum install -y ca-certificates
> update-ca-trust enable
> cp /opt/consul/ssl/demo-root.cer /etc/pki/ca-trust/source/anchors/
> update-ca-trust extract

Here are the documents I had to read:


Unknown said...
This comment has been removed by the author.
Unknown said...

Hi Russell Simpkins
I am trying to enable tls to the consul by following above steps, but its still showing me the same error
X509 certificate signed by unknown authority
can help me with this !!!!

Russell Simpkins said...


Did you execute the last 4 steps? Those last 4 steps need to be done on each server that intends to connect to the server with the self-signed certificate.

Unknown said...

HI Russell
I solved it by adding the CA certificate to the TLS trust store That is "/etc/pki/tls/certs/ca-bundle.crt" for centos 7
Thanks for your reply ... :)

Russell Simpkins said...

Glad you got it working. Step 4 is an alternative, suggested way to trust the certificate. I thought maybe I missed a step, but if you look at this post you'l see that you did the older way of trusting the certificate.

Unknown said...
This comment has been removed by the author.
Kunal said...

Hi Russell

In the doc above the command:
openssl req -newkey rsa:2048 -days 3650 -x509 -nodes -out /opt/consul/ssl/demo-root.cer -keyout /opt/consul/ssl/private.pem
is using private.pem key but we are using key name private_key = /opt/consul/ssl/privkey.pem in demo.conf. I hope it's a typo.

Russell Simpkins said...


The openssl program has some variations. If yours works with -private_key instead of -keyout, that's not a big deal. I often find myself going back to openssl --help or openssl req --help to verify the parameters.

Unknown said...

Hi Russel

I have Consul as a service/discovery utility for the spring boot micro-services. I was able to enable TLS 1-way, i.e. consul -> microservice.
For that, my consul.json read:
"ca_file": "C:\\ca.cer",
"verify_outgoing": true

Spring boot configuration read:
# HTTP Server (Tomcat) Port
port: 443
key-store: C:\spring.p12
key-store-password: changeit
key-store-type: JKS

The above works well.
But the moment I try the 2-way, it gives me bad cert on consul logs, empty chain at spring boot application logs. Here is the configuration:

"ca_file": "C:\\ca.cer",
"verify_outgoing": true,
"cert_file": "C:\\consul.cer",
"key_file": "C:\\consul.key",
"verify_incoming": true

application.yml on spring boot app.
# HTTP Server (Tomcat) Port
port: 443
key-store: C:\spring.p12
key-store-password: changeit
key-store-type: JKS
trust-store: C:\spring-ts.p12
trust-store-password: changeit
trust-store-type: JKS
client-auth: NEED
enabled: true

Any reason why the consul agent is not able to pass the certs to the server? Any help will be greatly appreciated.


Russell Simpkins said...

Sandeep, make sure you've executed the last 4 steps for every node. When you use self signed certificates you need to update the certificate authority (CA) on each node.