General

It seems like I've been working on so many random things the past few weeks. And all on different platforms: C (UNIX/POSIX), Python, Cocoa, iOS.

Anyhow, the only things that are almost ready for public consumption are the security-related libraries/modules. (Each succeeding project builds on the previous.) All are BSD licensed.

sha-asaddi
Everything SHA, from SHA-1 to SHA-2 (SHA-224, SHA-256, SHA-384, SHA-512) to the newest truncated editions (SHA-512/224, SHA-512/256). Also includes HMAC wrappers for each. This is a cleaned up and refactored version of my previous command-line sha project. Still WIP though.

pbkdf2
Implementations of PBKDF2 in C and Python. The C version depends on sha-asaddi, though I suppose it can easily be swapped to use any other SHA/HMAC implementations with similar signatures. The Python version has no dependencies outside of the standard library.

scrypt
Implementations of Colin Percival's scrypt in C and pure Python. This was more of an academic exercise since Mr. Percival's version is far more optimized and is already available under a BSD license. (At least, I'm assuming it's more optimized — I haven't actually looked at his code yet. But he did seem to have assembly/SSE optimized versions.) The Python version is probably too slow to be practical. (A Python wrapper for the original scrypt implementation is already available.) Will gladly take optimization suggestions for the Python version!

Oh, and these three projects represent a general shift in my packaging/building ideals. For now, I've decided to ditch GNU's autotools in favor of CMake for my projects. CMake just seems like a more coherent tool for configuring/building in a platform-independent manner. But, that just, like, my opinion, man...

It's strange, but my most ubiquitous work is probably sha (or rather, its individual hash implementations). I lost track of the number of random products I've run across that have a copyright/license attribution to me. SHA implementations are a dime a dozen though, and I'm sure the versions in OpenSSL (libcrypto) draw the most users.

But I have to wonder, what draws developers to mine? The random portability/optimization knobs? The BSD license?

Anyway, I reworked my SHA implementations today. Brought the API to my naming/style standards for this decade. Added SHA-512/224 and SHA-512/256. And, from the work on pam_totp, added HMAC wrappers.

Of course, with the addition of HMAC, I had to work it into sha (the utility) as well, which I did.

So after a little over 7 years, here's a new sha release...

sha 1.1 tarball

I've been wanting to install some sort of two-factor authentication scheme on my server for a while now. There's Google Authenticator, but unfortunately, it appears to be written for Linux-PAM and is rife with Linuxisms. But all was not lost, however, as it lead me to OATH and its related specs, HOTP and TOTP authentication.

It turns out that HOTP/TOTP is relatively simple — solely based on HMAC-SHA1. Great, I thought. I just needed an HMAC implementation... and I also needed to learn how to write a PAM module (specifically, an OpenPAM module, which is what FreeBSD uses). And yes, I know that "PAM module" is technically redundant, no one needs to point that out.

So I studied RFC 4226, RFC 2104 and this useful article about OpenPAM. I've been doing that in my spare time for a few weeks now. It wasn't until this morning that I decided to start writing some code.

And in a few hours, I had HMAC-SHA1 (built on top of my SHA1 implementation... I wanted to avoid libcrypto to keep things lightweight), HOTP, and finally a working pam_totp. I went with TOTP-only for now as that's what I wanted and I didn't really fancy keeping state for each user (aside from their keys). (But as an aside, it looks like I'll need to keep state anyway if I want to avoid replay attacks and have some clock drift tracking.)

Anyway, what I have is in an extreme alpha state, but needless to say, I've already installed it into my sshd PAM auth chain. As for a token generator, I use a nice, free iOS app called OATH Token.

I won't bother releasing anything, as the intended audience is rather small (FreeBSD admins who want TOTP auth). Maybe I'll work on it more someday, add event-based HOTP support, develop it into a true pam_oath (which I couldn't find anywhere, strangely). But at least that itch has been scratched...

I didn't realize this, but all my SHA implementations were still living in my Subversion repository. Well, after some meticulous converting from both my archived CVS and SVN repositories, I've managed to finally move them over to Mercurial.

Anyway, I'm not sure what else in Subversion I'd like to move over. Certainly, there's nothing left in there that's all that interesting to the public. So I will take down the public-facing SVN repo... someday.

Searching Google, I see nothing links directly svn.saddi.com. However, there's quite a lot of documentation that references flup and fcgi.py within it. Well, if anyone's been checking out that flup, it's 2 years out of date. And the standalone fcgi.py module... I'd rather just forget about that since it's hopelessly out of sync from the flup version. (It's basically the flup threaded fcgi server without thread pools.)

Just a few updates on some non-happenings. I recently discovered hg convert and hg transplant. I decided to re-do my Mercurial repositories (most originated in Subversion, while others came from CVS). I was entirely obsessed with converting the entire history to HG. (Projects like yafic and newsfish had the most history, being converted from CVS to SVN, then SVN to HG. Unfortunately, I didn't preserve history when going to SVN.) Anyhow, I'm pretty happy with the results.

On another note, I went ahead and upgraded Trac to 10.4. I also installed the Mercurial plugin and switched to that. Unfortunately, Trac doesn't currently support multiple repositories (see #2086), so my catch-all Trac page, http://trac.saddi.com/projects, only tracks a single project. (And that happens to be yafic since it's had recent activity...) I don't really feel like switching to one Trac environment per project, so I guess I'll wait out that feature request.

I'm currently trying out Mercurial, a distributed SCM that's 99% pure Python. I was able to migrate some of my recent projects (including full history) over to it. My repository is http://hg.saddi.com/.

I haven't fully committed yet, but it's very tempting since it's far easier to install (it's literally a "python setup.py install"). Setting up the web interface was a breeze too (it's built on top of WSGI!)

Oh well, I mention this now because I've finally committed something to my Hg repository. In other words, my SVN and Hg repositories are now out of sync...

Anyway, I know there's a Trac plugin for Hg which I will have to try out as well. Not sure if and when I'll fully commit. Only time will tell and I really haven't been working on any personal projects lately...

There don't seem to be many AJP C libraries. In fact, there don't seem to be any (according to Google, at least). There's at least one FastCGI C library, which is unsurprising given the ubiquity of FastCGI. So yesterday afternoon, I decided to "read spec, write code" yet again and began a C implementation of the "container" (app server) side of AJP. After not having touched C for over 2-3 years, it was a good feeling to muck around with C and BSD sockets again. (Procedural programming, how I missed you!)

I finished it up in a few hours and it is now fairly complete. It's actually a pretty simple protocol, I've realized. All the complexity comes from the way requests/responses are encoded and decoded. (Otherwise, it's a fairly straightforward 1:1 mapping.)

Of course there were the 3 undocumented spec additions, the first two I had to figure out through experimentation so long ago and the last was conveyed to me by someone who actually looked at the source mod_jk/mod_proxy_ajp source. (As much as I believe in the whole "the best documentation is the source" thing, I don't really like looking at similar/related source code when implementing something.)

Anyhow, I don't think those 3 undocumented additions are documented anywhere (hah!) besides my source (ajp_base.py and ajp.c). So:

• When decoding strings, a string with the length of 0xffff is the same as the empty string. However, its trailing NUL is not in the stream.
• When sending SEND_BODY_CHUNK packets, the packets must be NUL terminated. However, this NUL must not be included in the SEND_BODY_CHUNK's length (but must be included in the packet's length).
• The value following an SSL_KEY_SIZE attribute is not an encoded string, but rather an AJP integer.

Anyway, the ultimate goal of this project is to create a Python WSGI AJP server that is implemented in C as much as possible. At this point, I have a simple proof-of-concept working that makes calls into an embedded Python interpreter. It doesn't implement WSGI at all though.

As far as request/response throughput is concerned, it looks promising. While the threaded pure-Python AJP server could only handle ~86 requests per second (with 100 parallel clients), the threaded C/Python hybrid version was nearly pushing 1000 requests per second. Of course it doesn't include WSGI overhead yet. But we shall see.

All this effort is, of course, inspired by the WSGI servers built upon the FastCGI C library: fcgiapp and python-fastcgi. If I actually used FastCGI, I'd probably be using one of those servers rather than flup's.

I guess.

I've worked on my Python "blog" project for a number of years now. If I remember correctly, under the hood, it started out using flat files for storage and CGI. Eventually I moved it to mod_python+Publisher, then to FastCGI+my own version of Publisher. Sometime before that, I switched to using PostGreSQL... eventually transitioning the site content and then the session data to the database. (The templating engine changed at least once. And of course eventually, came the switch of the whole webserver interface to WSGI.)

Anyhow, what I was aiming for was a user-customizable blog/forum site. Multi-column, a selection of premade components (such as polls and HTML boxes), user profiles, personal bookmarks, etc. It was pretty ambitious and it met most of those goals. Though little did I realize that if I just generalized my goals a bit, I would actually be creating a Content Management System.

I didn't come to that realization until I started playing around with CMS's recently. At first, being in my Java mood, I sought open source Java CMS's. I only stumbled across Lenya. I ran into some difficulty, and promptly tabled it. I figured there would be more Java CMS's out there. But I really didn't have much luck finding anymore... (ok, so I didn't try that hard, either)

I briefly looked at Plone (based on Zope). It seemed a bit overkill for what I wanted... and overtly complex. So I moved on the PHP-based projects. (Note that I eventually did try Plone. But my gut feeling was correct, especially compared to the PHP CMS's.)

So I decided to look at Joomla! and Drupal, based on all the hype around those projects. (Some of it good, some of it bad... i.e. security vulnerabilities that made the front page of Slashdot. )

I tried Drupal first and immediately fell in love. It was utterly simple to use and maintain. It was a breeze to set up multiple instances using the same installation. It had a whole array of 3rd party modules that did what I wanted. (And I found, it could do most of what I wanted out of the box already - I just had to enable the module.) Anyhow, I switched 3 of my (static) sites to Drupal to try it out.

Unfortunately, at this point, I never gave Joomla! a chance. I hear many good things about Joomla! pertaining to the ease of use (and the eye candy). But it had one fatal flaw in my eyes: it only supported MySQL. Hence the reason that I tried Drupal first...

As an aside, it's probably obvious, but I'm not a big fan of the L-M-P in LAMP. 1) I am and have been an avid FreeBSD user for the past decade. 2) I prefer PostGreSQL over MySQL, speed be damned. I like my transactions, thank you. 3) Whether the P stands for Perl or PHP, I'd rather have my apps written in Python and to some extent, Java.

Anyway, the only thing really missing (so far) from Drupal is a good CAPTCHA module. Solving a simple addition problem seems a bit simplistic. I'd be surprised if there weren't bots that could answer those CAPTCHAs already. And the image CAPTCHA support seems a bit lame, especially compared to the last CAPTCHA library I worked with, JCaptcha. Anyhow, I bring this up because even though I normally turn off user registration, I did forget to turn it off for one of my sites. Overnight, some 3-4 "people" (all using rather fake looking yahoo addresses) registered for my site. I'm sure if I had some actual content that you could post comments to, I'd also have a bunch of spam to take care of.

Anyway, one of the sites that was converted happens to be saddi.com. If I get more familiar/comfortable with Drupal, I'll probably be subsuming this blog into its Drupal instance.

I decided to put on my sysadmin hat and do a little work on kalahari (my co-located server). I didn't really have a plan. And I'd been upgrading things here and there over the past few weeks. (Like bringing Subversion up to 1.3.1 from 1.1.x. Thankfully that went painlessly.)

I decided to set up Trac, just to give it a try. Rather than install it through the FreeBSD ports system, I installed most of the dependencies and did a manual install. It was surprisingly painless. The Trac files and environment live in the trac user's home directory... and I run Trac using the WSGI patch + flup's AJP server. Quite simple.

As a digression, sometimes I wonder if people wonder about my affinity for AJP. My webserver has both mod_fastcgi and mod_jk. (Alas, I'm not brave enough to upgrade to httpd 2.2.x yet, for the built-in mod_proxy_xxx modules.) But I prefer to run web applications/services as a different user. CGI/FastCGI would require me to set up the application to run as www:www... and just setting up the permissions alone would be a nightmare. (Not to mention the insecurity of running as the generic www user... not that I don't trust the other users of my server. ) I can't use SCGI because mod_scgi doesn't coexist peacefully with mod_fastcgi under httpd 2.x. And I realize FastCGI apps can be run in an "app-server" mode for lack of a better term (as a manually spawned process), but configuration of AJP seems so much simpler.

Well, that and I run Tomcat as well. I would say the web applications/services on kalahari are now evenly split between Python-based and J2EE-based platforms. (Although one internal app still runs on WebObjects...)

Anyhow, I also decided to revive the old wiki, the main use of which was to serve the help pages for blog.flup.org. I did the same as I did with Trac. Manual installation, WSGI backend + AJP server from flup.

Lastly, I revived flup.org/u (my simple URL shortener). I took it down some time ago because of abuse, but it was always my intent to restore it once I had some basic authentication/authorization built in. Well, with the CAS server now long in place, it was pretty easy. As an aside, I've been itching to write some sort of CAS WSGI middleware, but I see Paste already has something. Still, it would be nice to have some sort of generic filter/interceptor framework. If not for WSGI, then for flup's Publisher.

I have to admit, I've barely done any Python programming since I started my new job. Though, oddly enough, my Java skills and J2EE knowledge have been steadily growing. In fact, lately, I can honestly say I've been enjoying doing Java/J2EE stuff. I can probably thank Eclipse and Spring.

I've had a few personal forays into Java web programming-land in the past (namely Velocity-Struts-Hibernate, and WebObjects). WebObjects I gave up because of the uncertain future and the lagging standards compliance. (I haven't tried WO 5.3 yet though.) Velocity-Struts-Hibernate was just a pain to set up. Plus I didn't like the fact that I had to edit a number of XML files just to add a new action/form...

But apparently, I'm a glutton for punishment because I just recently assembled a skeleton web application consisting of Hibernate (for the persistence layer), Spring (for management of the business layer), Struts (form validator and controller), and Velocity (template engine). Now, either I've become tolerant of editing lots of silly XML files (Spring adds at least one more file to edit), or Spring actually makes it easier to integrate everything.

I'm thinking it's a combination of both... and the fact that I just love Eclipse (which I was introduced to at work). It just makes editing Java so easy. After trying out the refactor feature, I don't think I can go back to using "just" emacs.

Anyhow, I recently deployed CAS as my single sign-on service on kalahari. It currently uses the blog.flup.org user database to authenticate against. (And the development version of blog.flup.org has already been modified to use CAS.) But, as long as I keep the momentum, the first Velocity-Struts-Spring-Hibernate application I deploy will be an account manager. (Providing registration, password changing, & password reset services.)

Hopefully with an SSO service in place, I can deploy more services that I developed but never finished a long time ago. Poor kalahari. It's running 3 different types of application servers, sometimes multiple instances of a particular type...