Packing perl modules into RPMs

Yesterday I was tasked with packing up some perl code into RPM. That was the easy part. The hard part was the dependency graph I created for myself. Maybe there's an easy way to tell an RPM that it can get it's files from /usr/share/RPM, but that part eluded me. Instead I decided I should package up my modules into RPMs. Here's a few things that made this task easier. For starters, I had some perl on the machine, so I didn't have to configure cpan. 

1. Install cpanm 

$> perl -MCPAN -e shell
cpan[1]> install App::cpanminus
cpan[1]> exit 

2. Install cpan2rpm. There was a bug in the source. So, I had to fix the file before it would install. I got this <file> 

http://search.cpan.org/CPAN/authors/id/E/EC/ECALDER/cpan2rpm-2.028.tar.gz

Here's the steps:

$> cd /usr/local/src
$> wget <file>
$> tar -zxf cpan2rpm-2.028.tar.gz
$> cd cpan2rpm-2.028
$> sed -i"" "s|Pod::Text|Pod::PlainText|g" cpan2rpm
$> cd ../ && tar -czf cpan2rpm-2.028.tar.gz cpan2rpm-2.028
$> cpanm ./cpan2rpm-2.028.tar.gz

Cpan2prm is a pretty nice utility. It will run through the paces, creating an rpm and a source rpm. I ran into one issue while building the namespace::clean module. For some reason, the RPM added a _Util.pm file, that it provided, but then created a "Requires: perl(namespace::clean::_Util)" dependency. The solution I used was to extract the source from the source RPM and then modify the spec file to manually add a "Provides" e.g.

provides: perl(ExtUtils::HasCompiler) = 0.014 
provides: perl(namespace::clean) = 0.27

provides: perl(namespace::clean::_Util) = 0.27

Here's how you can extract a source rpm, edit the file and then build using the spec file:

$> cd ~/rpmbuild/SRPMS
$> mkdir perl-namespace-clean && cd perl-namespace-clean
$> rpm2cpio ../perl-namespace-clean-0.27-1.src.rpm|cpio -idmv
$> emacs namespace-clean.spec
$> cp namespace-clean-0.27.tar.gz ~/rpmbuild/SOURCES
$> rpmbuild -ba namespace-clean.spec

Making selinux work for you

I'm no selinux expert, and I got asked to figure out why all these audit logs were showing up in /var/log/messages that looked like this:

Jul 22 21:21:46 du-proc01 kernel: type=1400 audit(1469222506.799:118232): avc:  denied  { read } for  pid=25450 comm="httpd" name="feed_status.json" dev=xvdj ino=4325992 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=file

It turns out that my auditd daemon was dead and selinux was set to permissive mode. When selinux is in permissive mode, it writes permission failures, like the one above, to /var/log/audit/audit.log, or if auditd is dead to /var/log/messages. 

After some interesting back and forth on slack, I wanted to know if there was an easy way to enable selinux, without causing a bunch of headaches for my colleagues. Yes, there is.

Selinux will run in permissive mode. When in permissive mode, selinux will log all access violations. Put selinux into permissive mode with the following:

setenforce 0

Then let your system run for a while. Or, if you have integration or acceptance tests, go ahead and run them. Try to execute all of the possible operations that might be blocked by selinux. You can generate a rule to fix all of your broken permissions with following command:

cat  /var/log/audit/audit.log|audit2allow -m

That command will generate the rule in human readable form, so you can verify what rules would need to be added. To generate the module, run:

cat  /var/log/audit/audit.log|audit2allow -M <module_name>

Where <module_name> makes sense to you. Once that is run, you can turn it on with 

semodule -i <module_name>

If you generate the module with -M, there will be a file you can copy onto other machines. The .pp should be installed here:

/etc/selinux/targeted/modules/active/modules

Once you're satisfied with your module and your system isn't generating any selinux access denied messages you can start enforcing, but don't just re-enable selinux. You will need to relabel the file system

touch /.autorelabel
shutdown -r now

Then you can start enforcing:

setenforce 1

It's just that easy to take an existing system and get it working with selinux. 

Testing Varnish-Cache

Originally posted here.

Our team has many RestFul APIs written in PHP that serve information stored in a MySQL database. The content we serve supports many sections of the paper and that means we get a lot of traffic. The majority of our content is not personalized and we control how often that content is updated, which makes it attractive to cache. Before adding Varnish to our stack, we spent significant effort dealing with load spikes, especially at the database layer. The applications that were causing the most pain had a very large, complicated code base and fixing that code or even replacing that code was estimated at six or more months. We wanted a solution that we could implement quickly and didn't require a major code rewrite. We wanted a solution that would take the pressure off of our application servers and our database servers to remove the distractions of dealing with production load alerts. Varnish was the perfect solution to our problem. By adding Varnish in front of our API servers, we took a huge load off of our API and database servers.

At first we added basic caching, but as we used the software and got more experience we found different things we could do in the varnish layer. Varnish has a lot of features and capabilities. Taking advantage of the software inevitably results in a lot of configuration and tweaking. Some functionality can be achieved by editing the varnish configuration language (VCL) and sometimes you need to add a module or create your own module. Writing VCL or working with varnish modules (vmods) leads to testing, which is what I'm writing about today; how to test Varnish. This post will focus on how to test VCL and if I have time, I will add another post to talk about how to test varnish modules.

Let’s imagine a real-life feature and how we can go about testing the feature. Our javascript programmers use JQuery and sometimes turn on JQuery cache busting. When enabled, JQuery adds a timestamp e.g. “_1331829184859=” to the query string in an attempt to bust the cache. So, if I strip the query string parameter, I can prevent JQuery from busting our cache. Here's one way I could clean our URL using VCL:

The first “if” statement strips the “_1331829184859=” and the rest of the lines are there to clean the url so it isn't left with unneeded characters. The URL is reset on line 11.

How can I test this code? I could start a varnish server with a backend apache or nginx instance that logs requests, issue a variety of curl requests and then manually verify the logs, but there’s an easier way. Varnish ships with the ability to test using the testing tool varnishtest. Varnishtest gives you the ability to write VCL tests you can run on the command line or as part of your build process. Here's an example:

Line 1 is just for documentation purposes. Lines 2 - 6 define a server that will accept a request and issue a response. Since I intend to clean the URL before it get’s passed to a backend server, I added a test to verify the backend URL sent to the server with the “expect” syntax on line 4. If the value doesn’t match, the test stops and you get lots of debugging style output.

Line 7 is the syntax to add a backend definition and VCL. Line 8 is optional. I added it to illustrate you're allowed to import vmods. At line 9 I define the vcl_recv subroutine and lines 10 – 16 has the logic I need to remove the cache busting parameter.

Lines 19 – 21 define vcl_deliver to set a response header. I did this to illustrate one way to validate logic during a client request. Line 23 is where I define a client. The client's are where you issue requests with the txreq (transmit request) and receive the response with rxresp (receive response.) At line 26, I use expect to verify the test variable is as expected. Adding the "expect" in the client logic is a better way to test multiple inputs. I kept the example short, but it's trivial to add multiple txreq -url, rxresp, expect lines to test different inputs. You can also copy and paste the client c1 to create c2, c3...cN clients.

Line 28 is where I run the client logic.

Our tests are saved in files e.g. test01.vtc. Assuming you compiled and installed varnish in the standard locations, running varnishtest is this easy:

When the test fails, you get a lot of output to look at. Normally, your good tests produce very little output. However, you can run your tests in “verbose” mode (-v) to get full output. For the simple test above, a passing test in verbose mode produces 253 lines of output. A failing tests produces less lines of output, but only a few less: 189.

Varnishtest allows you some flexibility as well. If you’re building and testing tweaks to varnish, you can specify what varnishd to use with –D e.g.

You also use the -D option to your advantage and pass variables to your test, since ${varnishd} (or anything else defined with –D) will be available to your VCL when the test is compiled and run. For example, when building and compiling custmo varnish modules, you can import the library from the build directory e.g.

Conclusion

Testing with varnishtest made a huge impact when working with my VCL. I can quickly and easily add logic and test theories on my local vagrant box. There is a bit of a learning curve to get started with varnishtest, but the documentation is getting better and there are good examples to follow on the internet.

crontab reminder

# For details see man 4 crontabs
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed

Getting Started with RPMs

Red Hat Package Manager (RPM) is a way to package and deploy software on Red Hat variants that’s similar to Advanced Packaging Tool (APT) on Ubuntu variants. To create RPM files, you can use third party tools like Ant or FPM, but with a little bit of effort you can quickly write your own spec files. This paper isn’t intended to be a complete reference on RPM files. There are a couple of great resources out there to get the full scoop on RPMs and I’ve added them at the end of this paper.

To make an RPM you need two things, a spec file and the rpmbuild program. You can get rpm-build easily enough e.g.

$> yum install –y rpm-build rpmdevtools
$> rpmdev-setuptree

The last command will create your rpm build tree e.g.

/home/user/rpmbuild
/home/user/rpmbuild/RPMS
/home/user/rpmbuild/SPECS
/home/user/rpmbuild/BUILD
/home/user/rpmbuild/SOURCES
/home/user/rpmbuild/SRPMS

If you put your tree somewhere else or if you want to be explicit, add an .rpmmacros file to your home directory.

$> echo “%packager First Last 
%vendor Your Company
%_topdir ${HOME}/rpmbuild” > ~/.rpmmacros

Now you have the basic requirements and we can start writing spec files. However, before we dive into the spec file it helps to understand that spec files have macros and that macros can be a single value or a multi-line value. Here is how we define a variable in a spec file:

%define name value

You can conditionally set the variable like so:

%{!?_version: %define _version 1.0.0}

When you use the variable you use you use the %{} syntax e.g.

Version: %{_version}

You can conditionally use a variable e.g.

Release: 1%{?dist}

The %{?dist} syntax means, output the value of “dist” if it’s set, otherwise output nothing. rpmbuild will fail if you reference macros that are not defined. If you want to see what the current value is of a predefined variable you can run the following command:

$> rpm --eval '%{_rpmdir}'

To get a list of all pre-defined variables, see the links at the end. If you want to see everything you can use either of these two commands:

$> rpm --showrc | less
$> rpmbuild --showrc | less

I piped them to less in my example because there’s a lot of information when you run either of those commands. Since we can define variables and show them, if you wanted to play around you can:

$> rpm --define '_some value' --eval '%{_some}'
value

You can also include variables listed in a separate file

%include some.spec.file

When you define variables in a separate file, you can define macros that span multiple lines. Let’s assume you have a set of application specific preparation steps. Create your steps in a separate file e.g. mysteps.spec Once you do that you can define your macro e.g. Now you have a macro “prepsteps” that can be used in any step after it’s defined. Here’s how you can use it in the “prep” phase e.g.

%prep
%prepsteps

Let’s pull some of this together with a simple example spec file:

$> rpmdev-newspec ex.spec
$> cat ex.spec
Name:           ex
Version:      
Release:        1%{?dist}
Summary:      
Group:        
License:      
URL:          
Source0:      
BuildRequires:
Requires:     
%description
 
%prep
%setup -q
 
%build
%configure
make %{?_smp_mflags}
 
%install
rm -rf $RPM_BUILD_ROOT
make install DESTDIR=$RPM_BUILD_ROOT
 
%clean
rm -rf $RPM_BUILD_ROOT
 
%files
%defattr(-,root,root,-)
%doc
%changelog

If you run that with rpmbuild you’re going to get your first taste of errors. Since we’re playing around, let’s just try to run the prep phase.

$> rpmbuild -bp ex.spec
error: line 2: Empty tag: Version:

Unfortunately, rpmbuild only reports the first error and not the rest. Here is what it looks like if we fill it out some more:

Name:           ex
Version:        1.0.0
Release:        1%{?dist}
Summary:        This is an example
Group:          Example group
License:        GNU
URL:            http://example
Source0:        ex-%{version}.tar.gz
%description
 
%prep
%setup -q
 
%build
%configure
make %{?_smp_mflags}
 
%install
rm -rf $RPM_BUILD_ROOT
make install DESTDIR=$RPM_BUILD_ROOT
 
%clean
rm -rf $RPM_BUILD_ROOT
 
%files
%defattr(-,root,root,-)
%doc
%changelog

Building RPMS: http://www.rpm.org/max-rpm/index.html 
Building RPMS: https://fedoraproject.org/wiki/How_to_create_an_RPM_package 
Naming guidelines: http://fedoraproject.org/wiki/Packaging:NamingGuidelines