Parsing HTTP::Request->content with CGI.pm

Back in the mists of time, when the Web was young and unconquered, Lincoln Stein wrote a module for Perl that would allow people to easily deal with parameters handed to a CGI program and to generate HTML. This module eventually grew to include not one but several kitchen sinks. It includes its own autoload mechanism, it's own file handle class and more. It Just Works when called FastCGI, Perlex, mod_perl and others.

While CGIs have all but disappeared, this module is still very useful for handling all the finicky edge cases for dealing with HTTP request content. But if you write your own web server environment, using CGI.pm to parse the HTTP content can get be hard. You basically have to fake it out.

This is how you get the params from a GET request.

# $req is a HTTP::Request object
local $ENV{REQUEST_METHOD} = 'GET';
local $CGI::PERLEX = $CGI::PERLEX = "CGI-PerlEx/Fake";
local $ENV{CONTENT_TYPE} = $req->header( 'content-type' );
local $ENV{'QUERY_STRING'} = $req->uri->query;
my $cgi = CGI->new();

# Now use $cgi as you wish

And here we parse the params from a POST request. Note that POST request can be big. Very big. If you aren't careful, they will fill up your memory. Always check Content-Length before reading in a POST request. In the following code, all the content was written to a file.

# $req is a HTTP::Request object
# $file is a filename that contains the unparsed request content
local $ENV{REQUEST_METHOD} = 'POST';
local $CGI::PERLEX = $CGI::PERLEX = "CGI-PerlEx/Fake";
local $ENV{CONTENT_TYPE} = $req->header( 'content-type' );
local $ENV{CONTENT_LENGTH} = $req->header( 'content-length' );
local $CGITempFile::TMPDIRECTORY = "/YOUR/TEMP/DIR/HERE";
# CGI->read_from_client reads from STDIN
my $keep = IO::File->new( "<&STDIN" ) or die "Unable to reopen STDIN: $!";
open STDIN, "<$file" or die "Reopening STDIN failed: $!";
my $cgi = CGI->new();
open STDIN, "<&".$keep->fileno or die "Unable to reopen $keep: $!";
undef $keep;
unlink $file

# Now use $cgi as you wish

The fun is that CGI will only read POST data from STDIN, so we have to redirect that to our file, saving and restoring the previous STDIN.

The above code also works when you are uploading a file with multipart/form-data which is how I got caught up in all this kerfuffle.

It's really to bad that one can't just do

my $cgi = CGI->new( $req );

Fun with perl

Want to help me out? Run the following code:

#!/usr/bin/perl

use strict;
use warnings;

use Data::Dump qw( ddx );
BEGIN { ddx "DIE=", $SIG{__DIE__}; };
use Test::More tests => 1;
BEGIN { ddx "DIE=", $SIG{__DIE__}; };

diag "Perl: ", sprintf "%vd", $^V;
diag "Test::More: $Test::More::VERSION";
pass "one test";

What do you expect $SIG{__DIE__} to contain? Acceptable answers are undef, "" and "DEFAULT". And the answer varies according to what version of perl and what version of Test::More you are running.

Example output:

# DIE=""
1..1
# DIE=sub { "???" }
# Perl: 5.004_05
# Test::More: 0.47
ok 1 - one test

# die:7: ("DIE=", undef)
1..1
# die:9: ("DIE=", "")
# Perl: 5.8.8
# Test::More: 1.001002
ok 1 - one test

# die:7: ("DIE=", undef)
1..1
# die:9: ("DIE=", undef)
# Perl: 5.10.1
# Test::More: 1.001002
ok 1 - one test

1..1
# -:9: ("DIE=", undef)
# Perl: 5.16.2
# Test::More: 1.001002
ok 1 - one test

And yes, I know 5.004 dates from the previous century.

Why I hate the CPAN

(OK, actually I love the CPAN, but that now that I have your attention.)

Every time I write and upload a new distro, I dream of how it's going to make some toiling programmers life wonderful, the way POE, LWP and other bits of the CPAN give my life meaning. IKC and then POE::Component::Generic where my bids at taking over the world. Or at least a certain part of it.

However uploading a distro to the CPAN is something of a commitment. My code is going to live for several years/decades, if not forever. I also do not want to be like the CPAN authors that annoy me, with inadequate test coverage, poor doco and annoying holes in their feature set.

These three points intertwine.

I'll be revising the doco and come to a point where I write "doing X is not advised/not yet implemented." Or I'll write "doing X and Y together has not been tested."

But then I kick myself with "Why isn't it implemented? Why don't I write a unit test for that edge case?" The implementation would take 20 minutes, the test cases for that new feature 30 minutes. A new paragraph of doco and now I've lost my place in my original task of revising the documentation.

What's more, I won't do it just once, but 4 times. 8 times. Before you know it, I've spent 2 days improving a module that originally took 2 hours to write.

These aren't just random numbers. The initial draft of POEx::HTTP::Server took 2 hours Monday morning, of which 30 minutes were wasted because I didn't adequately remember how POE::Session::Multiplex works. I spent the rest of the day improving on that start, then 2 more days adding features, tweaking POE::Session::Multiplex and POE::Component::Deamon because of misfeatures I'd discovered.

But backing up further, I wrote POEx::HTTP::Server because last Friday I thought to myself "Now that Sphinx integration is nearly complete, what will I work on next." And the answer was of course SPEED SPEED SPEED! So I went to look at lighttpd as a front end, with a FastCGI or SCGI call to the app server for dynamic content.

So while my real goal was to get version 2.5 of the Document Warehouse newly known as Quaero ready as soon as possible, I ended up spending a week implementing some technology from 1990. Or 1997 depending on how you look at it. And it's not over; today I realised there was a race condition in my code. A race that IKC probably shares. So a day or three tracking that down.

Still, Firebug shows 19ms response times from POEx::HTTP::Server. Of which only 5ms are waiting aka POEx::HTTP::Server generating the content the other 12 being DNS and network overhead.

Which is damn sweet.

lighttpd and Thawte

Thawte have done it again: mucked about with their root cert. This did this once, years ago. You'd think they'd learn.

So, after much grief I found out how to set up a new SSL certificate in lighttpd:

domain.key is the key you signed your CSR with.
domain.cert is what you have just "Picked up" from Thawte. You want the X.509 one.

cat domain.key domain.cert >domain.pem

wget https://search.thawte.com/library/VERISIGN/ALL_OTHER/thawte%20ca/SSL123_CA_Bundle.pem

Note, please change domain to whatever the FQDN your certificate is for.

You then need to following two lines in your lighttpd config file

ssl.pemfile     = "/etc/lighttpd/domain.pem"

ssl.ca-file     = "/etc/lighttpd/SSL123_CA_Bundle.pem"

The CA Bundle is a chain of certificates. Normaly, an SSL cert is signed directly by a root certificate installed with the browser. But Thawte likes doing things the hard way. So they signed a certificate with their root and now sign all new SSL certs with that intermediate certificate. So the web server has supply both the SSL cert and the intermediate certificate to the browser. That's what SSL123_CA_Bundle.pem is. If you bought one of the more expensive options, you should download another bundle.