Allan Saddi's projects blog

A very minor update to flup. If you aren't having any problems now, then there's probably no need to upgrade.

This update concerns certain web servers & configurations that don't correctly supply PATH_INFO and/or QUERY_STRING to flup. When missing these environment entries, the flup servers will now attempt to deduce PATH_INFO and/or QUERY_STRING from REQUEST_URI (if it is present) before defaulting to the empty string (as it did before).

Thanks to Richard Davies for the patch.

Just a few updates on some non-happenings. I recently discovered hg convert and hg transplant. I decided to re-do my Mercurial repositories (most originated in Subversion, while others came from CVS). I was entirely obsessed with converting the entire history to HG. (Projects like yafic and newsfish had the most history, being converted from CVS to SVN, then SVN to HG. Unfortunately, I didn't preserve history when going to SVN.) Anyhow, I'm pretty happy with the results.

On another note, I went ahead and upgraded Trac to 10.4. I also installed the Mercurial plugin and switched to that. Unfortunately, Trac doesn't currently support multiple repositories (see #2086), so my catch-all Trac page, http://trac.saddi.com/projects, only tracks a single project. (And that happens to be yafic since it's had recent activity...) I don't really feel like switching to one Trac environment per project, so I guess I'll wait out that feature request.

With the advent of flup 1.0 and the disappearance of the middleware modules (I don't think anyone really cares about Publisher ), I've been getting a few queries here and there about alternatives.

Rather than continue to answer people individually, here are some alternatives to flup's middleware and publisher modules. However, they represent alternatives that I've personally chosen. (And considering how far removed I am nowadays from the world of Python web development, take my choices with a grain of salt.) Suggestions for other alternatives (alternative alternatives?!?) are welcome.

For users of flup's SessionMiddleware, I recommend Beaker. Switching over is a simple matter of changing the environ key from 'com.saddi.service.session' to 'beaker.session' and ensuring that session.save() is called before the WSGI application returns. (After you've appropriately swapped out flup's SessionMiddleware with Beaker's in the WSGI stack, of course.)

GzipMiddleware and ErrorMiddleware have analogues in Python Paste.

flup's Publisher is probably most like Colubrid, though I haven't personally tried it. (Also remember that flup's Publisher was inspired by mod_python's Publisher, which was apparently inspired by Zope's ZPublisher ).

Anyway, I updated the old flup 0.5 branch and uploaded the source distribution to the Cheese Shop. I really don't want to maintain two branches, so consider this 0.5 release final.

I've been wanting to do this for a while now, especially after hearing that SHA-1 was broken. (Ok, I guess that's rather old news.) Anyhow, I added SHA-256 to yafic. Adding support for future hashes should be fairly trivial now as well.

It can be found at http://www.saddi.com/software/yafic/dist/

I've done very basic testing, so use at your own risk. I plan to start using it regularly to test it a bit more.

I plan to add another feature or two (like XML output) before making an official 1.3 release.

And as usual, it may be found in my Mercurial repository: http://hg.saddi.com/yafic-sha256
(I don't plan to merge it into the main yafic repository until it's undergone more testing.)

I resolved a few more outstanding issues with flup and decided to make a release. I went ahead and called it 1.0. And as I previously mentioned, I went ahead and removed the publisher and middleware modules.

If you've been following the Subversion repository for flup (which has been quiet due to my switching to Mercurial), there have only been 1 or 2 new fixes.

I also uploaded the tarball and 2.5 egg to the Cheese Shop. Further updates will also go to the Cheese Shop.

And on the subject of further updates, consider flup to be in maintenance mode - there won't be any more new development. Bug fixes and bug reports are still welcome, of course. (Though this has been the way I've run flup for quite a while now, so I'm just making a formal announcement. )

And as a bit of editorializing, did I ever mention how much I hate FastCGI? Around the time I first released flup, I had transitioned my apps from FastCGI to AJP, and I never missed FastCGI since. But much to my disappointment, the FastCGI module was the most popular... most likely due to ubiquitous & cheap shared hosting services (e.g. Dreamhost).

Oh well, I don't even use flup myself anymore. I've switched to ajp-wsgi...

I'm currently trying out Mercurial, a distributed SCM that's 99% pure Python. I was able to migrate some of my recent projects (including full history) over to it. My repository is http://hg.saddi.com/.

I haven't fully committed yet, but it's very tempting since it's far easier to install (it's literally a "python setup.py install"). Setting up the web interface was a breeze too (it's built on top of WSGI!)

Oh well, I mention this now because I've finally committed something to my Hg repository. In other words, my SVN and Hg repositories are now out of sync...

Anyway, I know there's a Trac plugin for Hg which I will have to try out as well. Not sure if and when I'll fully commit. Only time will tell and I really haven't been working on any personal projects lately...

As far as Java development goes, for personal projects and for work, I find myself in that odd place between needing only a servlet container (e.g. Tomcat) and needing a full JEE container. I'm sure this situation is quite common. The one thing that I almost always end up needing beyond plain servlets is JTA. As I've mentioned in the past, I've tried various solutions from plugging in JOTM, which I'm not at all too happy with, to using stripped-down versions of full application servers.

Well, the problem for me is still unresolved, but I did try out Jencks over the weekend. It was fairly easy to set up, and it worked as expected. (Just have to remember to set the connection manager's transaction type to XA to actually get XA transactions!) And rather than using DataSourceMCF, one would probably want to use a real MCF for your database for XA functionality.

Which brings me to the point of this post. In the past I've blogged about problems with PostgreSQL/XA/TranQL (here and here). Although the problem was resolved in the driver long ago, I don't believe I ever published the actual PostgreSQL TranQL connector I made. (And believe me, I would rather fold it back into the original project... but I'm having a hard time finding contact information... or a working mailing list... anything.)

Anyway, the connectors: http://maven.saddi.com/org/tranql/

I've tested it with Jencks and Geronimo. (And by "test", I mean I just verified that the PostgreSQL driver is indeed using 2PC.) The source can still be found in the same place: http://www.saddi.com/software/tranql/. You'll probably need to use the latest 8.2 JDBC driver (which includes the above-mentioned fix).

Hopefully anyone searching for TranQL/PostgreSQL will eventually stumble upon it. I mean, I can't be the only person interested in using XA transactions with PostgreSQL in Geronimo, right???

ajp-wsgi 1.0 has been released. It may be downloaded here (ChangeLog). It's been sitting around for a few months now, in heavy use at my server for both "production" and development WSGI apps. I think it's pretty safe to say it's at least on par with flup's ajp WSGIServer now.

The only lingering issue is a core dump with Trac and only Trac, and only when it is killed or SIGHUP'd. Unfortunately, with so many variables (Trac imports quite a few C extension modules), I haven't been able to isolate problem. (And maybe it's just my environment... sadly, I don't have the resources to try elsewhere.)

Anyway, 1.0... so what now?

My C WSGI implementation is relatively decoupled from the transport code (AJP). It's conceivable that replacing the transport layer with something else is entirely possible. Like say... a full HTTP 1.1 server. I guess there could be two directions to take that:

http-wsgi
I suppose one could take an embeddable HTTP server (or write one from scratch, if so inclined) and glue the WSGI code into the request handling pipeline. Personally, it's not a project that interests me much. Leave web server writing to the web server experts, I say. However, suppose one wrote it as an extension module for one of the popular web servers...

mod_wsgi
How about an extension module for say, Apache HTTPD or lighttpd? The problem I see with this is that, functionally, it would just be equivalent to mod_python. (After all, you're just embedding Python into the web server and patching into its request processing.) I think the only difference would be that the WSGI-adapter code (the layer that sits just above Python) would be written in C. Additionally, you inherit some of the more interesting problems of mod_python... namely conforming Python's process/threading model to that of the web server's. In all likelihood, you would be running multi-process, not multi-threaded. I guess if you were clever, you could run the Python interpreter in only a single process, similar to how CGI works in modern versions of Apache HTTPD.

But then, if you do that, all it buys you is automatic process management versus using an external server model like AJP/SCGI. So why bother with yet another mod_somethingsomething?

Anyway, ajp-wsgi will remain ajp-wsgi for the foreseeable future. (If I'm sufficiently bored and curious, I may try creating an scgi-wsgi someday.) Though http-wsgi does pique my interest somewhat... I just have to find a suitably feature-laden embeddable C web server.

Well, not much has been happening. ajp-wsgi has been humming along, stable as a rock. It's been running all of my Python-based websites for a little over a month now (longer, really, since I had to reboot my server early January to update FreeBSD). Once I make a few documentation updates, I think it would be safe to declare a 1.0 release. (Seemingly a big milestone nowadays...)

I'd also like to document a news site/comment moderation system that I've been thinking about for a very long time. I'll only paint the broad picture since really, I haven't given it much thought. I credit this discussion from long ago on kuro5hin for inspiration. (Some of "my" ideas may be blatant rips too... so I give credit where credit is due. )

Since moderation systems tend to get gamed... why not just turn it into a game? The site would revolve around fictitious money. I was thinking of having certain actions (like submitting a story or posting a comment) cost money. Not only that, but it would be like an investment. Highly rated comments or active stories would yield profit for the originator. Comments that are universally rated down or stories marked as spam would end up with all or most of the initial investment lost.

It works for people doing moderation/rating as well. For example, it would cost a small amount of money to rate a comment. When you rate a comment, you are basically saying: "I am betting that the average rating of this comment will be X" After some amount of time, an amount of money would be returned to you based on how close you were to the "actual" rating.

It would be a dynamic system. The cost of submitting a story, leaving a comment, rating, etc. would all be variables. As would things such as the return factor for positively-rated stories/comments. It would be adjusted (preferably automatically) to keep the economy adequately sized vs. the number of active users as well as encourage users to say, post more stories under a specific topic.

"New user inflation" (where the economy explodes due to a large influx of new users who get a starting balance) can be combated by transforming the site into something akin to a social network or an invite-only service. Basically, new users start out with a 0 balance and they are given loans, with interest, from existing users. The system enforces the loans by automatically paying it back in installments. (Though what to do with a deadbeat user...?)

Anyhow, the goal of the entire system, ultimately, would be to deter spamming and gaming by the use of multiple accounts. The system can be controlled, albeit indirectly, by setting various prices and rates.

Well, after a week of coding, I decided to "release" ajp-wsgi (i.e. make its existence known beyond my web sites and blog). It's a rather niche project, but if one other person finds it useful, then hey, cool. Releasing it costs me nothing.

I found it highly educational to create, illuminating the mysteries of the Python C API. Plus I'm using it everywhere now. And after having closed ticket #4, the solution of which seems to have been a panacea to all current issues, it's pretty much complete as far as I envisioned it.

Now maybe I can get back to Flannel...

And I have thought about writing an SCGI version as well... but it's not something I would use. So what's the point? I don't really like supporting something I don't use regularly, though by virtue of being free software, I'm not really obligated to provide any support. (But I still do because I'm a nice guy. ) Maybe if Apache HTTPD eventually adds a mod_proxy_scgi though...

Ah, no Google hits for mod_proxy_scgi. Oh well. At least this entry may eventually show up.

Apparently, I did not read the AJP13 spec nor my original code very closely:

Note: The content-length header is extremely important. If it is present and non-zero, the container assumes that the request has a body (a POST request, for example), and immediately reads a separate packet off the input stream to get that body.

Here I was requesting the first block. Anyway, a quick and easy fix.

ajp-wsgi is moving along... and actually, development has slowed down immensely (a marked sign of stability?). I've converted all Python WSGI applications on my server to use it now. It may just be psychological, but I do notice application responses being a bit snappier. But who knows.

Trac seemed to be the most non-trivial to convert. It doesn't provide a ready-made application factory to create the WSGI app object. I basically had to mimic (using my config options) the operations that its main() method performed. Other applications (my own blog & shorten projects, moinmoin) had readily-available app objects though. And I'm also glad to say that Paste Deploy-based apps are easily deployed with ajp-wsgi as well.

Anyhow, I went ahead and decoupled the C WSGI code from the AJP code today. Now the next time I'm bored, I think I'll write drivers for both ends of an SCGI connection. It would be interesting if I could write that FastCGI->SCGI adapter wholly in C (using the standard C FastCGI dev kit). Actually, I guess I should check if there's already a C SCGI implementation...

I moved all the WSGI stuff out of my AJP C library project into its own project: ajp-wsgi. I polished it up a bit, gave it a command-line interface, wrote a better build system, and even wrote a simple README for it. You can find it here.

Note that this is not a Python extension. Rather, it is a 100% C WSGI implementation... that executes the application in an embedded Python interpreter.

It's moved beyond a proof-of-concept and is quickly becoming more and more practical. (At this moment, my personal wiki is running atop it. Maybe I'll switch my Trac sites and shorten over to it as well.) But make no mistake, it is very much alpha-quality and untested.

Future directions:

• Decouple the WSGI code from AJP, opening it up to an SCGI implementation. I have no interest in making a FastCGI version, so don't ask.
• If not thread pooling, then maybe a thread limit at least?
• Multi-app support. Maybe read configuration from an INI-style file (each section header would define the module:object for the app).
• Figure out/understand the oddity with mod_proxy_ajp. It seems to stream the request body whether solicited or not. It also sends an unsolicited EOF packet. If this is the case, then mod_jk seems like a saner implementation... but maybe this is the future direction of the protocol?

A continuation from the weekend's entry... I actually finished the WSGI implementation in C and glued it to the AJP library I wrote. It was an interesting endeavor... programming for Python in C.

I was going to cop out and just implement wsgi.input in Python, but I went all the way and wrote that in C as well. And I'm glad too, because it's far more efficient. Data copies are greatly minimized. And data is streamed from the server. Assuming the application reads wsgi.input in decent-sized chunks, the memory usage will always remain manageable. (For example, I uploaded a 600+ MB file and hashed it. The server never used more than 2-3MB of memory.)

And from my braindead (i.e. "Hello World!") benchmarks, the server is capable of about 900 requests per second. This is a 10X improvement compared to the pure-Python server serving the same application. Not bad at all.

I'm glad to say that as far as I can tell, the server is pretty close to 100% WSGI compliance. At least, wsgiref.validate doesn't complain. One thing I haven't tested is its ability to handle erroneous WSGI applications. But I'm not sure if I'll want to do much validation (e.g. are the header names/values strings? Is the status code/reason code a string?). My goal is to make this server a production-grade server... so it will assume that your application is WSGI compliant itself.

Anyhow, still a bit of work to do. It would be nice if it was configurable somehow. Also, I should probably investigate if I can just turn it into a simple Python extension module (vs. being a C server that embeds a Python interpreter). I haven't looked how the hybrid FastCGI servers are packaged, but I'm sure it's something much more sane than the route I went.