Wednesday, March 23, 2022

App Scripts are pretty neat

I was doing some analysis and looking to create a graph in my Google Spreadsheet. The data had gaps and I was looking for a quick/easy way to set all empty cells to zero. I found an example after searching that I turned into the following:

function fillBlanks() {
var sheet = SpreadsheetApp.getActiveSheet();
var sheetLR = sheet.getLastRow();
var sheetLC = sheet.getLastColumn();
var range = sheet.getRange(1, 1, sheetLR, sheetLC);
var values = range.getValues();
for (var r = 1; r < sheetLR; r++) {
for (var c = 0; c < sheetLC; c++) {
if(String(values[r][c]).trim() == "") {
values[r][c] = 0;
}
}
}
range.setValues(values);
}
fillBlanks()

Have your spreadsheet open, then run it from App Scripts. Feel free to take, modify and re-use as you see fit.

Sunday, March 14, 2021

gRPC SSL certs on Windows with WSL2

The past few days has been one hell of a nightmare. I have been creating gRPC gateway services and we wanted to implement server side HTTP2 push. Turns out HTTP2 & gRPC requires you to implement SSL to take advantage of HTTP2 push in the browser. Implementing SSL is a pain in the a$$. You need to make sure everything is running with the correct hostnames and almost all of the documentation out there is for folks who have said the heck with it and opted for self signed certificates. I couldn't find any good tutorials for setting up a valid SSL cert. 

I ran into too many issues. The first one that took many hours to identify, if you are doing your development on Windows 10 + WSL2, it's not straight forward how best to update /etc/hosts. Recall that for HTTPS to work, you need the hostname to match which means you need DNS to resolve. Here's how to fix this. Open up a terminal on your linux (I use Ubuntu) and run ifconfig. Take the result of that and update your /etc/hosts file (this requires root/sudo.) Then open up notepad as an Administrator and update your c:\Windows\System32\drives\etc\hosts and save that. 

If you do this, you can successfully open up your web browser and use  your custom domain name AND use curl to test your custom domain name. 

I lost countless hours on this one part - trying like hell to figure out why my custom domain wasn't resolving to 127.0.0.1. I think Windows/WSL2 have something about 127.0.0.1 and the secret was getting the IP for the WSL2 Linux distro.

Once you've got your hostnames configured in /etc/hosts, you can move on to securing your golang services. One thing you may need to do is to concatenate your SSL certificate with the providers intermediary certificate. This isn't terribly difficult. Simple create a new text file, have your valid cert in PEM format then a new line and the intermediary. If done correctly, it will look like this: (these are not real - this is just an example)

-----BEGIN CERTIFICATE-----
t1befnVnPZlIlyEMDw/WPyR4Bfi1cemqjaKSxzR+lEtCTQC7xKM8678RMHBtZ7/v
NsfRUitk6otcQnBX/sErZmFxqdPyl7aOpifkj+pQV0mfn3bGTPhSPfc6BM84lZo8
W2HRMbWsw5Z5ZIwEfMEbaCtbtw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ
RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD
-----END CERTIFICATE-----


You can start your gRPC service over TLS like this:

serverOptions := []grpc.ServerOption{
    grpc.UnaryInterceptor(interceptor.Unary()),
    grpc.StreamInterceptor(interceptor.Stream()),
}
if enableTLS {
    tlsCredentials, err := credentials.NewServerTLSFromFile(serverCertFile, serverKeyFile)
    if err != nil {
        return fmt.Errorf("cannot load TLS credentials: %w", err)
    }
    serverOptions = append(serverOptions, grpc.Creds(tlsCredentials))
}
s := grpc.NewServer(serverOptions...)

That will load the gRPC service over TLS in manner that any client can validate the SSL certificate. It's important to note that you may not need to add the intermediary certificate. Try starting your gRPC service and then connecting to it. I found this handy curl command for testing gRPC HTTP2 POSTs to get Streaming responses.

curl -i --http2 -H "Content-Type: application/grpc" -H "TE: trailers" -X POST https://youhost.com/grpc.Service/Message

Feel free to add -vvv to that if you're having issues. Now that you have your golang gRPC service running under TLS, you just need to adjust your gateway client to handle HTTPS connections. Here's a snippet:

config := &tls.Config{
    InsecureSkipVerify: false,
}
return grpc.DialContext(ctx, addr, grpc.WithTransportCredentials(credentials.NewTLS(config)))


With that, your gateway can connect to the gRPC service using TLS. If you've got Nginx in front of everything, you may find this blog helpful. I found that it was 90% correct. They share some configuration that mostly works, but my version of Nginx didn't like putting the locations outside of the server entry. 

Wednesday, December 30, 2020

Managing passwords with GPG

I recently moved back to working on Windows. I've been really enjoying the development setup of Windows 10 with WDL 2 which allows me to have Ubuntu right at hand. The one downside is that I can't open my Apple encrypted .dmg file. I have been in the habit of keeping my passwords in a clear text file, but storing it on an encrypted disk image that I keep on my cloud back up drive. I know that these companies claim to not check the contents, but this makes it nearly impossible. Now that I'm on Windows, I needed another solution and decided to get back to gpg. I created the following bash functions to let me view my password file quickly and easily.

Here's the code in a gist. I put the logic in my ~/.bashrc so that I can access the file by typing "pw". When I'm done getting information or adding a new password, the code re-encrypts the file, cleans up the temporary file and maintains a minimum backup of previous encrypted files.


Monday, December 21, 2020

Fun things to do with Docker

I recently got back to doing some dev work and I'm using docker. Docker is nice in that you can make build boxes and run them which can be really handy if you're developing on a mac but your code is destined for some linux variant.

One thing I found helpful today was to mount my local project so that I could run inside my Docker centos box. I did it with something like this:

$> docker run -di --mount type=bind,source="/path/to/project/folder",target=/var/opt/project"image:tag"

When run in that manner, I can have a container running that has my local project in /var/opt/project. I can edit the code and run it after shelling onto the box. Other common commands I use a lot.

$> docker ps
$> docker exec -it <id> /bin/bash
$> docker stop <id>


Saturday, October 14, 2017

Starting out right with Python

If you're new to python or you don't program in python as often as you'd like to, it's key to start off by setting up your environment. Make sure you install a virtual environment tool. I like to use virtualenvwrapper. Virtualenvwrapper helps keep your pip installs clean and separate. It's super easy to set up and use.

$ pip install virtualenvwrapper

That's all you need to do for installation, simple. Once you have it installed you need to configure it. I work on a mac and on the terminal in linux environments. I normally use a projects folder and under that a python-projects folder. If you don't have a preference, go ahead and follow suit.

mkdir -p ~/projects/python-projects
Now you can go ahead and update your bashrc or your bash_profile depending on your preferences.

$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bash_profile
$ echo "export WORKON_HOME=$HOME/projects/python-projects" >> ~/.bash_profile 
$ source ~/.bash_profile


Once that's done you can start using virtualenvwrapper.


$ mkvirtualenv sample
$ workon sample 


The first command will create a new virtual environment directory e.g. $HOME/projects/python-projects/sample. Now if you run any pip installs, the libraries and binaries will be in that directory and will not pollute any of your core system libraries. This will help you in the future, especially if you decide to share your code because all of your pip dependencies can now be easily listed e.g.

pip list --format columns

That's it. Happy python coding.

Thursday, September 22, 2016

Packing perl modules into RPMs

Yesterday I was tasked with packing up some perl code into RPM. That was the easy part. The hard part was the dependency graph I created for myself. Maybe there's an easy way to tell an RPM that it can get it's files from /usr/share/RPM, but that part eluded me. Instead I decided I should package up my modules into RPMs. Here's a few things that made this task easier. For starters, I had some perl on the machine, so I didn't have to configure cpan. 

1. Install cpanm 

$> perl -MCPAN -e shell
cpan[1]> install App::cpanminus
cpan[1]> exit 

2. Install cpan2rpm. There was a bug in the source. So, I had to fix the file before it would install. I got this <file> 

http://search.cpan.org/CPAN/authors/id/E/EC/ECALDER/cpan2rpm-2.028.tar.gz

Here's the steps:

$> cd /usr/local/src
$> wget <file>
$> tar -zxf cpan2rpm-2.028.tar.gz
$> cd cpan2rpm-2.028
$> sed -i"" "s|Pod::Text|Pod::PlainText|g" cpan2rpm
$> cd ../ && tar -czf cpan2rpm-2.028.tar.gz cpan2rpm-2.028
$> cpanm ./cpan2rpm-2.028.tar.gz

Cpan2prm is a pretty nice utility. It will run through the paces, creating an rpm and a source rpm. I ran into one issue while building the namespace::clean module. For some reason, the RPM added a _Util.pm file, that it provided, but then created a "Requires: perl(namespace::clean::_Util)" dependency. The solution I used was to extract the source from the source RPM and then modify the spec file to manually add a "Provides" e.g.

provides: perl(ExtUtils::HasCompiler) = 0.014 
provides: perl(namespace::clean) = 0.27

provides: perl(namespace::clean::_Util) = 0.27

Here's how you can extract a source rpm, edit the file and then build using the spec file:

$> cd ~/rpmbuild/SRPMS
$> mkdir perl-namespace-clean && cd perl-namespace-clean
$> rpm2cpio ../perl-namespace-clean-0.27-1.src.rpm|cpio -idmv
$> emacs namespace-clean.spec
$> cp namespace-clean-0.27.tar.gz ~/rpmbuild/SOURCES
$> rpmbuild -ba namespace-clean.spec

Friday, July 22, 2016

Making selinux work for you

I'm no selinux expert, and I got asked to figure out why all these audit logs were showing up in /var/log/messages that looked like this:

Jul 22 21:21:46 du-proc01 kernel: type=1400 audit(1469222506.799:118232): avc:  denied  { read } for  pid=25450 comm="httpd" name="feed_status.json" dev=xvdj ino=4325992 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=file

It turns out that my auditd daemon was dead and selinux was set to permissive mode. When selinux is in permissive mode, it writes permission failures, like the one above, to /var/log/audit/audit.log, or if auditd is dead to /var/log/messages. 

After some interesting back and forth on slack, I wanted to know if there was an easy way to enable selinux, without causing a bunch of headaches for my colleagues. Yes, there is.

Selinux will run in permissive mode. When in permissive mode, selinux will log all access violations. Put selinux into permissive mode with the following:

setenforce 0

Then let your system run for a while. Or, if you have integration or acceptance tests, go ahead and run them. Try to execute all of the possible operations that might be blocked by selinux. You can generate a rule to fix all of your broken permissions with following command:

cat  /var/log/audit/audit.log|audit2allow -m

That command will generate the rule in human readable form, so you can verify what rules would need to be added. To generate the module, run:

cat  /var/log/audit/audit.log|audit2allow -M <module_name>

Where <module_name> makes sense to you. Once that is run, you can turn it on with 

semodule -i <module_name>

If you generate the module with -M, there will be a file you can copy onto other machines. The .pp should be installed here:

/etc/selinux/targeted/modules/active/modules

Once you're satisfied with your module and your system isn't generating any selinux access denied messages you can start enforcing, but don't just re-enable selinux. You will need to relabel the file system

touch /.autorelabel
shutdown -r now

Then you can start enforcing:

setenforce 1

It's just that easy to take an existing system and get it working with selinux.