Allan Saddi's projects blog

I just released a new snapshot, it includes the following changes:

• fix fork bomb and check number of children against minSpare not maxSpare
• Restore check of the absolute number of children against maxSpare and document rationale.
• Add configurable timeout (default: no timeout) to be used when the WSGI application is called. Only applies to forked servers!

Here are links to the Hg repository and the PyPI page.

If there are any issues, please open a ticket at the flup Trac and/or submit a patch.

I just finished beta testing (both closed and open) the upcoming MMO game, Champions Online. Now, I won't talk about the actual game, since this is my software/programming blog, and not my gaming blog. But this game had something that really tickled the code monkey inside of me: the absolutely best in-game bug reporting UI I have ever seen.

The workflow was straightforward:

1. Pick a bug category (e.g. Character, Missions (quests), UI, Graphics, Sound, etc.)


2. Use a context-sensitive drop-down to select the exact aspect or component that was buggy. For example, if you chose the Missions category, the drop-down would contain a list of your active missions and for convenience, it would also include a few of your most recently completed missions.


3. Alternatively, type some text in the text box.


4. Hit "Search." This would actually bring up existing bug reports that matched your criteria. I found this very useful, even if I did not have a bug to report — it let me see if I could expect any problems with what I was currently testing.


5. Clicking on an existing bug report would let you see the original description as well as a log of changes/responses. It was great for finding workarounds (to allow one to continue playing/testing).


6. If there was an existing report that matched your bug, you could add more info or simply "me too" the bug which would increment the number of users who have reported that bug (so developers can judge severity) and update the modification time. Conveniently, the bug would also be tracked as if it were your own, showing up when you viewed your own reports.


7. Lastly, if there are no existing bugs that match yours, you could open a new bug report. And here, I thought, was another neat addition: when you submitted the report, it would automatically gather and send your system info, and in addition to this it would take a screen shot of what you were seeing and attach it to the report. This made graphics and UI bugs quite painless to report.

Anyway, I was averaging a few bug reports a day, so I got a lot of experience with that interface. It was always a good feeling to see bugs that I had personally reported get fixed in subsequent builds. (Though the responses/handling became abysmally slow during open beta... understandable, I guess, since there were far more testers.) I hope future online games adopt something similar, especially the transparency (being able to view others' bug reports) and the convenience (automatic screen shots, etc.)

I didn't realize this, but all my SHA implementations were still living in my Subversion repository. Well, after some meticulous converting from both my archived CVS and SVN repositories, I've managed to finally move them over to Mercurial.

Anyway, I'm not sure what else in Subversion I'd like to move over. Certainly, there's nothing left in there that's all that interesting to the public. So I will take down the public-facing SVN repo... someday.

Searching Google, I see nothing links directly svn.saddi.com. However, there's quite a lot of documentation that references flup and fcgi.py within it. Well, if anyone's been checking out that flup, it's 2 years out of date. And the standalone fcgi.py module... I'd rather just forget about that since it's hopelessly out of sync from the flup version. (It's basically the flup threaded fcgi server without thread pools.)

I've written up brief guidelines on submitting patches for flup. (Though it also pertains to any of my other public projects.)

And I'd like to thank everyone who has recently contributed! Don't worry, there's nothing in the guidelines that you weren't already doing.

See this entry for details.

I guess I've sat on these changes long enough:

• Import Paste factories (and dependencies...) from PasteScript
• Be tolerant of EAGAIN when sending messages to parent process.
• Add forceCGI keyword argument to FastCGI servers to programmatically force CGI behavior.
• Merge Tommi Virtanen's "single server" (sequential server) patch.
• Update ez_setup.py.
• Re-seed random module after each fork.
• Add an indication as to which header fails assertion when passing in non-string header names and/or values.

The only major change are the Paste factories — they should be a lot closer to the (deprecated) factories for flup servers found in PasteScript. In fact, they are the same factories found in PasteScript. Defaults will have changed, with respect to the previous factory implementations.

I'm looking to release in a week or so. Testing/comments are welcome:

http://hg.saddi.com/flup-server/

And, as you probably have noticed, I don't cut releases that often (not a big fan of "release early, release often"...) But my public repository is always up-to-date. So it's almost the same thing... without the overhead.

Well, Python 3.0 was released a few days ago. I admit, I really haven't been keeping up much with Python development. Up until yesterday, Python 2.5 was all that I've used.

Anyway, I was curious. I was curious about all the backwards-incompatible changes to the language/libraries. I was curious about how WSGI dealt with Python 3.0. With the list of proposed WSGI 1.0 amendments for Python 3.0 (and the Web-SIG posts it referenced), I attempted to port both flup and ajp-wsgi.

It was interesting, at least, to see the issues in two different lights. Though flup and ajp-wsgi do the same thing conceptually, flup is pure-Python and ajp-wsgi is pure-C. The most pain for both, unsurprisingly, came from the new str/bytes distinction. While it was an annoying issue to deal with (and I realize as far as the spec is concerned, all details about it still haven't been worked out), I do welcome the change.

I've published both repositories, for the curious. They work for simple "Hello World" type applications. I kind of doubt that any frameworks have been ported to Python 3.0 yet. I probably won't touch them again until the spec has been clarified more. (Encoding of environ key/values? file_wrapper support with e.g. text-mode files?)

Interestingly, I didn't actually test the FastCGI and SCGI adapters with "real" web servers. (I don't have anything set up.) I used the flup client implementations to cross-test clients/servers on 2.x/3.0. So those flup WSGI servers might actually be broken.

The repositories:

• http://hg.saddi.com/ajp-wsgi-py3.0/
• http://hg.saddi.com/flup-py3.0/

A very minor update to flup. If you aren't having any problems now, then there's probably no need to upgrade.

This update concerns certain web servers & configurations that don't correctly supply PATH_INFO and/or QUERY_STRING to flup. When missing these environment entries, the flup servers will now attempt to deduce PATH_INFO and/or QUERY_STRING from REQUEST_URI (if it is present) before defaulting to the empty string (as it did before).

Thanks to Richard Davies for the patch.

Just a few updates on some non-happenings. I recently discovered hg convert and hg transplant. I decided to re-do my Mercurial repositories (most originated in Subversion, while others came from CVS). I was entirely obsessed with converting the entire history to HG. (Projects like yafic and newsfish had the most history, being converted from CVS to SVN, then SVN to HG. Unfortunately, I didn't preserve history when going to SVN.) Anyhow, I'm pretty happy with the results.

On another note, I went ahead and upgraded Trac to 10.4. I also installed the Mercurial plugin and switched to that. Unfortunately, Trac doesn't currently support multiple repositories (see #2086), so my catch-all Trac page, http://trac.saddi.com/projects, only tracks a single project. (And that happens to be yafic since it's had recent activity...) I don't really feel like switching to one Trac environment per project, so I guess I'll wait out that feature request.

With the advent of flup 1.0 and the disappearance of the middleware modules (I don't think anyone really cares about Publisher ), I've been getting a few queries here and there about alternatives.

Rather than continue to answer people individually, here are some alternatives to flup's middleware and publisher modules. However, they represent alternatives that I've personally chosen. (And considering how far removed I am nowadays from the world of Python web development, take my choices with a grain of salt.) Suggestions for other alternatives (alternative alternatives?!?) are welcome.

For users of flup's SessionMiddleware, I recommend Beaker. Switching over is a simple matter of changing the environ key from 'com.saddi.service.session' to 'beaker.session' and ensuring that session.save() is called before the WSGI application returns. (After you've appropriately swapped out flup's SessionMiddleware with Beaker's in the WSGI stack, of course.)

GzipMiddleware and ErrorMiddleware have analogues in Python Paste.

flup's Publisher is probably most like Colubrid, though I haven't personally tried it. (Also remember that flup's Publisher was inspired by mod_python's Publisher, which was apparently inspired by Zope's ZPublisher ).

Anyway, I updated the old flup 0.5 branch and uploaded the source distribution to the Cheese Shop. I really don't want to maintain two branches, so consider this 0.5 release final.

I've been wanting to do this for a while now, especially after hearing that SHA-1 was broken. (Ok, I guess that's rather old news.) Anyhow, I added SHA-256 to yafic. Adding support for future hashes should be fairly trivial now as well.

It can be found at http://www.saddi.com/software/yafic/dist/

I've done very basic testing, so use at your own risk. I plan to start using it regularly to test it a bit more.

I plan to add another feature or two (like XML output) before making an official 1.3 release.

And as usual, it may be found in my Mercurial repository: http://hg.saddi.com/yafic-sha256
(I don't plan to merge it into the main yafic repository until it's undergone more testing.)

I resolved a few more outstanding issues with flup and decided to make a release. I went ahead and called it 1.0. And as I previously mentioned, I went ahead and removed the publisher and middleware modules.

If you've been following the Subversion repository for flup (which has been quiet due to my switching to Mercurial), there have only been 1 or 2 new fixes.

I also uploaded the tarball and 2.5 egg to the Cheese Shop. Further updates will also go to the Cheese Shop.

And on the subject of further updates, consider flup to be in maintenance mode - there won't be any more new development. Bug fixes and bug reports are still welcome, of course. (Though this has been the way I've run flup for quite a while now, so I'm just making a formal announcement. )

And as a bit of editorializing, did I ever mention how much I hate FastCGI? Around the time I first released flup, I had transitioned my apps from FastCGI to AJP, and I never missed FastCGI since. But much to my disappointment, the FastCGI module was the most popular... most likely due to ubiquitous & cheap shared hosting services (e.g. Dreamhost).

Oh well, I don't even use flup myself anymore. I've switched to ajp-wsgi...

I'm currently trying out Mercurial, a distributed SCM that's 99% pure Python. I was able to migrate some of my recent projects (including full history) over to it. My repository is http://hg.saddi.com/.

I haven't fully committed yet, but it's very tempting since it's far easier to install (it's literally a "python setup.py install"). Setting up the web interface was a breeze too (it's built on top of WSGI!)

Oh well, I mention this now because I've finally committed something to my Hg repository. In other words, my SVN and Hg repositories are now out of sync...

Anyway, I know there's a Trac plugin for Hg which I will have to try out as well. Not sure if and when I'll fully commit. Only time will tell and I really haven't been working on any personal projects lately...

As far as Java development goes, for personal projects and for work, I find myself in that odd place between needing only a servlet container (e.g. Tomcat) and needing a full JEE container. I'm sure this situation is quite common. The one thing that I almost always end up needing beyond plain servlets is JTA. As I've mentioned in the past, I've tried various solutions from plugging in JOTM, which I'm not at all too happy with, to using stripped-down versions of full application servers.

Well, the problem for me is still unresolved, but I did try out Jencks over the weekend. It was fairly easy to set up, and it worked as expected. (Just have to remember to set the connection manager's transaction type to XA to actually get XA transactions!) And rather than using DataSourceMCF, one would probably want to use a real MCF for your database for XA functionality.

Which brings me to the point of this post. In the past I've blogged about problems with PostgreSQL/XA/TranQL (here and here). Although the problem was resolved in the driver long ago, I don't believe I ever published the actual PostgreSQL TranQL connector I made. (And believe me, I would rather fold it back into the original project... but I'm having a hard time finding contact information... or a working mailing list... anything.)

Anyway, the connectors: http://maven.saddi.com/org/tranql/

I've tested it with Jencks and Geronimo. (And by "test", I mean I just verified that the PostgreSQL driver is indeed using 2PC.) The source can still be found in the same place: http://www.saddi.com/software/tranql/. You'll probably need to use the latest 8.2 JDBC driver (which includes the above-mentioned fix).

Hopefully anyone searching for TranQL/PostgreSQL will eventually stumble upon it. I mean, I can't be the only person interested in using XA transactions with PostgreSQL in Geronimo, right???

ajp-wsgi 1.0 has been released. It may be downloaded here (ChangeLog). It's been sitting around for a few months now, in heavy use at my server for both "production" and development WSGI apps. I think it's pretty safe to say it's at least on par with flup's ajp WSGIServer now.

The only lingering issue is a core dump with Trac and only Trac, and only when it is killed or SIGHUP'd. Unfortunately, with so many variables (Trac imports quite a few C extension modules), I haven't been able to isolate problem. (And maybe it's just my environment... sadly, I don't have the resources to try elsewhere.)

Anyway, 1.0... so what now?