Friday, October 30, 2015

Consul: Adding TLS to Consul using Self Signed Certificates

I'm currently working on setting up TLS for Consul. As of this writing, I'm still in the experimentation/set-up phase, but we plan to roll consul out into production with TLS support. So, this document may get updated but I wanted to capture what I had to do while it's fresh in my mind.

Consul's documents are a little light on specifics, which made this endeavor more difficult than I anticipated. I will post links at the bottom of this article. The following steps were used to create a self signed certificate on Centos 6.6. 

Make a directory to hold our files, create the certificate authority (ca) conf file, seed our index and create a cert index file:

> mkdir -p /opt/consul/ssl
> cat << EOF > /opt/consul/ssl/demo.conf
[ ca ]
default_ca = demo

[ crl_ext ]
# issuerAltName=issuer:copy  #this would copy the issuer name to altname

[ demo ]
new_certs_dir = /tmp
unique_subject = no
certificate = /opt/consul/ssl/demo-root.cer
database = /opt/consul/ssl/certindex
private_key = /opt/consul/ssl/privkey.pem
serial = /opt/consul/ssl/serial
default_days = 365
default_md = sha1
policy = demo_policy
x509_extensions = demo_extensions

[ demo_policy ]
commonName = supplied
stateOrProvinceName = supplied
countryName = supplied
emailAddress = optional
organizationName = supplied
organizationalUnitName = optional

[ demo_extensions ]
basicConstraints = CA:false
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
keyUsage = digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth
crlDistributionPoints = URI:

> touch /opt/consul/ssl/certindex
> echo 000a > /opt/consul/ssl/serial
> cd /opt/consul/ssl

NOTE: You may need this step to ensure the certs work with Consul. Edit the /etc/pki/tls/openssl.cnf file and add the following:


Below I have one way to make the change. Read more about extended key usage here:

> cp /etc/pki/tls/openssl.cnf  /etc/pki/tls/openssl.cnf.bak
> sed -i"" 's|# extendedKeyUsage = critical,timeStamping|extendedKeyUsage=serverAuth,clientAuth|' /etc/pki/tls/openssl.cnf

Generate the root certificate:

> openssl req -newkey rsa:2048 -days 3650 -x509 -nodes -out /opt/consul/ssl/demo-root.cer -keyout /opt/consul/ssl/private.pem 
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:New York
Locality Name (eg, city) [Default City]:New York
Organization Name (eg, company) [Default Company Ltd]:Demo Company
Organizational Unit Name (eg, section) []:Demo
Common Name (eg, your name or your server's hostname) []:
Email Address []:

Consul want's the certs and the servers to be server.<data center>.consul - just adjust the request below as I used dc1 as my datacenter. Generate a certificate signer request (csr):

> openssl req -newkey rsa:1024 -nodes -out /opt/consul/ssl/server.csr -keyout /opt/consul/ssl/server.key
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New-York
Locality Name (eg, city) []:New York
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Demo Company
Organizational Unit Name (eg, section) []:Demo
Common Name (e.g. server FQDN or YOUR name) []:server.dc1.consul
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Generate the self signed cert:

> openssl ca -batch -config /opt/consul/ssl/demo.conf -notext -in /opt/consul/ssl/server.csr -out /opt/consul/ssl/server.cer

To verify your certificate use the following command and make sure the "X509v3 Extended Key Usage" matches:

> openssl x509 -noout -text -in /opt/consul/ssl/server.cer
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication

Now you can configure your consul server to use the self signed certs. These lines were take out of my consul.json file:

    "ca_file": "/opt/consul/ssl/demo-root.cer",
    "cert_file": "/opt/consul/ssl/server.cer",
    "key_file": "/opt/consul/ssl/server.key",

On your agents, you're going to need to specify the "ca_file" and set "verify_outgoing":true in your consul configs. 

If you get errors about trusting the signing authority, you will need to trust the demo-root.cer. To trust the root certificate on your server(s) do the following:

1. Install the ca-certificates package
2. Enable the dynamic CA configuration feature
3. Add it as a new file to /etc/pki/ca-trust/source/anchors/:
4. Use command:

> yum install -y ca-certificates
> update-ca-trust enable
> cp /opt/consul/ssl/demo-root.cer /etc/pki/ca-trust/source/anchors/
> update-ca-trust extract

Here are the documents I had to read:


brijan elwadhi said...
This comment has been removed by the author.
brijan elwadhi said...

Hi Russell Simpkins
I am trying to enable tls to the consul by following above steps, but its still showing me the same error
X509 certificate signed by unknown authority
can help me with this !!!!

Russell Simpkins said...


Did you execute the last 4 steps? Those last 4 steps need to be done on each server that intends to connect to the server with the self-signed certificate.

brijan elwadhi said...

HI Russell
I solved it by adding the CA certificate to the TLS trust store That is "/etc/pki/tls/certs/ca-bundle.crt" for centos 7
Thanks for your reply ... :)

Russell Simpkins said...

Glad you got it working. Step 4 is an alternative, suggested way to trust the certificate. I thought maybe I missed a step, but if you look at this post you'l see that you did the older way of trusting the certificate.