Consul: Adding TLS to Consul using Self Signed Certificates

I'm currently working on setting up TLS for Consul. As of this writing, I'm still in the experimentation/set-up phase, but we plan to roll consul out into production with TLS support. So, this document may get updated but I wanted to capture what I had to do while it's fresh in my mind.

Consul's documents are a little light on specifics, which made this endeavor more difficult than I anticipated. I will post links at the bottom of this article. The following steps were used to create a self signed certificate on Centos 6.6. 

Make a directory to hold our files, create the certificate authority (ca) conf file, seed our index and create a cert index file:

> mkdir -p /opt/consul/ssl
> cat << EOF > /opt/consul/ssl/demo.conf
[ ca ]
default_ca = demo

[ crl_ext ]
# issuerAltName=issuer:copy  #this would copy the issuer name to altname
authorityKeyIdentifier=keyid:always

[ demo ]
new_certs_dir = /tmp
unique_subject = no
certificate = /opt/consul/ssl/demo-root.cer
database = /opt/consul/ssl/certindex
private_key = /opt/consul/ssl/privkey.pem
serial = /opt/consul/ssl/serial
default_days = 365
default_md = sha1
policy = demo_policy
x509_extensions = demo_extensions

[ demo_policy ]
commonName = supplied
stateOrProvinceName = supplied
countryName = supplied
emailAddress = optional
organizationName = supplied
organizationalUnitName = optional

[ demo_extensions ]
basicConstraints = CA:false
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
keyUsage = digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth
crlDistributionPoints = URI:http://path.to.crl/demo.crl
EOF

> touch /opt/consul/ssl/certindex
> echo 000a > /opt/consul/ssl/serial

> cd /opt/consul/ssl

NOTE: You may need this step to ensure the certs work with Consul. Edit the /etc/pki/tls/openssl.cnf file and add the following:

extendedKeyUsage=serverAuth,clientAuth

Below I have one way to make the change. Read more about extended key usage here: https://www.openssl.org/docs/manmaster/apps/x509v3_config.html#extended_key_usage_

> cp /etc/pki/tls/openssl.cnf  /etc/pki/tls/openssl.cnf.bak
> sed -i"" 's|# extendedKeyUsage = critical,timeStamping|extendedKeyUsage=serverAuth,clientAuth|' /etc/pki/tls/openssl.cnf

Generate the root certificate:

> openssl req -newkey rsa:2048 -days 3650 -x509 -nodes -out /opt/consul/ssl/demo-root.cer -keyout /opt/consul/ssl/private.pem 

Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:New York
Locality Name (eg, city) [Default City]:New York
Organization Name (eg, company) [Default Company Ltd]:Demo Company
Organizational Unit Name (eg, section) []:Demo
Common Name (eg, your name or your server's hostname) []:
Email Address []:

Consul want's the certs and the servers to be server.<data center>.consul - just adjust the request below as I used dc1 as my datacenter. Generate a certificate signer request (csr):

> openssl req -newkey rsa:1024 -nodes -out /opt/consul/ssl/server.csr -keyout /opt/consul/ssl/server.key
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New-York
Locality Name (eg, city) []:New York
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Demo Company
Organizational Unit Name (eg, section) []:Demo
Common Name (e.g. server FQDN or YOUR name) []:server.dc1.consul
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Generate the self signed cert:

> openssl ca -batch -config /opt/consul/ssl/demo.conf -notext -in /opt/consul/ssl/server.csr -out /opt/consul/ssl/server.cer

To verify your certificate use the following command and make sure the "X509v3 Extended Key Usage" matches:

> openssl x509 -noout -text -in /opt/consul/ssl/server.cer
.....
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
.....

Now you can configure your consul server to use the self signed certs. These lines were take out of my consul.json file:

    "ca_file": "/opt/consul/ssl/demo-root.cer",
    "cert_file": "/opt/consul/ssl/server.cer",
    "key_file": "/opt/consul/ssl/server.key",

On your agents, you're going to need to specify the "ca_file" and set "verify_outgoing":true in your consul configs. 

If you get errors about trusting the signing authority, you will need to trust the demo-root.cer. To trust the root certificate on your server(s) do the following:

1. Install the ca-certificates package
2. Enable the dynamic CA configuration feature
3. Add it as a new file to /etc/pki/ca-trust/source/anchors/:
4. Use command:

> yum install -y ca-certificates
> update-ca-trust enable
> cp /opt/consul/ssl/demo-root.cer /etc/pki/ca-trust/source/anchors/
> update-ca-trust extract

Here are the documents I had to read:

• Consul Encryption
• Self Signed CA
• Securing Consul

Generating a public key for SSH using your private RSA key

In order to ssh onto a server using public private key pairs, you need a specific type of public key. If you have the private key you can generate the public to install into the ~/.ssh/authorized_keys file with the following

echo "actual private key data" > private
chmod 600 private
ssh-keygen -y -f private

You can generate a public key with the following

 openssl rsa -in private.pem -pubout > public

But it won't work for using ssh e.g. ssh -i private

Changing commit logs in git

It sounds like something you shouldn't do, but sometimes you may want to adjust who made a commit. Maybe you did a commit on a vagrant box or maybe you fat fingered your name or email address while typing too fast. To change the committer I found this handy

#!/bin/sh
 
git filter-branch --env-filter '

OLD_EMAIL="bad@emailaddress"
CORRECT_NAME="Russell Simpkins"
CORRECT_EMAIL="russellsimpkins@real-domain"

if [ "$GIT_COMMITTER_EMAIL" = "$OLD_EMAIL" ]
then
    export GIT_COMMITTER_NAME="$CORRECT_NAME"
    export GIT_COMMITTER_EMAIL="$CORRECT_EMAIL"
fi
if [ "$GIT_AUTHOR_EMAIL" = "$OLD_EMAIL" ]
then
    export GIT_AUTHOR_NAME="$CORRECT_NAME"
    export GIT_AUTHOR_EMAIL="$CORRECT_EMAIL"
fi
' --tag-name-filter cat -- --branches --tags

Simply adjust the OLD_EMAIL, CORRECT_NAME and CORRECT_EMAIL and stuff that into a bash script. Then issue a git push --force

Resizing an ext4, ebs volume

You use lsblk and see your EBS volume is the right size, but running df -h shows the device is smaller. To fix this, the command to use is resize2fs 

resize2fs /dev/xvdf

While you can do it on a mounted device, these things are often better done when it's unmounted, just to be safe.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ext4grow.html

Fun with GPG

I once had a desire to create a team GPG key that I could use for signing RPMs. I've moved in a different direction, but I want to capture the steps in case I decide to use this again in the future.

You can import a private GPG with the following:

gpg --allow-secret-key-import --import private.key.file
gpg --list-keys
gpg --edit-key <ID> 

Once you run --edit-key you're able to trust the key. Execute **trust** and choose level **5**

With that done, you can decrypt using the key - assuming you know the password.

gpg -d -u "name <email>" encrypted.file.gpg > outputfile


To encrypt for the team key to unlock:

gpg -se -r "name <email>" -u "name <email>" encrypted.file

Port Forwarding on Mac OSX

If your running vagrant and you're forwarding traffic to vagrant over 8080, but you really prefer to hit port 80, you can use Mac's pfctl function. 

Here's a couple of links that you might find helpful
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man8/pfctl.8.html
http://krypted.com/mac-os-x/a-cheat-sheet-for-using-pf-in-os-x-lion-and-up/

I was reading this article http://salvatore.garbesi.com/vagrant-port-forwarding-on-mac/ and it suggested adding a vagrant plugin, but it's a ruby gem. Hard to imagine, but the gem failed to install.

You can still implement port forwarding. Create a pfctl.conf file in your vagrant folder:

echo "rdr pass on lo0 inet proto tcp from any to 127.0.0.1 port 80 -> 127.0.0.1 port 8080
rdr pass on lo0 inet proto tcp from any to 127.0.0.1 port 443 -> 127.0.0.1 port 8443" > pfctl.conf

To run this, you first need to enable pfctl

pfctl -e -f pfctl.conf

Once you enable the firewall rules, you can hit your vagrant box by going against localhost. 

To disable pfctl: 

pfctl -d

Searching Log files with AWK

I was digging through my notes and came across this little nugget. Say you have a log file and that log file entry has URL encoded values that you would prefer to see decoded. Here's the AWK I used to URL decode:

awk -F ^C '$4 ~ /SearchingFor/ {print $4}' access_log | awk '
{
    str = $0
    while (match(str,/%/)) {
      L = substr(str,1,RSTART-1) # chars to left of "%"
      M = substr(str,RSTART+1,2) # 2 chars to right of "%"
      R = substr(str,RSTART+3)   # chars to right of "%xx"
      str = sprintf("%s%c%s",L,hex2dec(M),R)
    }
    printf("%s\n",str)
    
}
function hex2dec(s,  num) {
    num = index("0123456789ABCDEF",toupper(substr(s,length(s)))) - 1
    sub(/.$/,"",s)
    return num + (length(s) ? 16*hex2dec(s) : 0)
}'

I found this on the web and I post it here so I don't loose it. I would love to give credit if I could remember where I got it.

How to merge in specific files from one git branch to another

I found this write up very helpful:

http://jasonrudolph.com/blog/2009/02/25/git-tip-how-to-merge-specific-files-from-another-branch/

git checkout <branch> <file>

So, say you have a feature branch foo and your master branch and you want to get one file, src/bar.c into master

git checkout master
git checkout foo src/bar.c
git commit -m "merging bar.c from foo branch" src/bar.c

That's it.