General

I just finished beta testing (both closed and open) the upcoming MMO game, Champions Online. Now, I won't talk about the actual game, since this is my software/programming blog, and not my gaming blog. But this game had something that really tickled the code monkey inside of me: the absolutely best in-game bug reporting UI I have ever seen.

The workflow was straightforward:

1. Pick a bug category (e.g. Character, Missions (quests), UI, Graphics, Sound, etc.)


2. Use a context-sensitive drop-down to select the exact aspect or component that was buggy. For example, if you chose the Missions category, the drop-down would contain a list of your active missions and for convenience, it would also include a few of your most recently completed missions.


3. Alternatively, type some text in the text box.


4. Hit "Search." This would actually bring up existing bug reports that matched your criteria. I found this very useful, even if I did not have a bug to report — it let me see if I could expect any problems with what I was currently testing.


5. Clicking on an existing bug report would let you see the original description as well as a log of changes/responses. It was great for finding workarounds (to allow one to continue playing/testing).


6. If there was an existing report that matched your bug, you could add more info or simply "me too" the bug which would increment the number of users who have reported that bug (so developers can judge severity) and update the modification time. Conveniently, the bug would also be tracked as if it were your own, showing up when you viewed your own reports.


7. Lastly, if there are no existing bugs that match yours, you could open a new bug report. And here, I thought, was another neat addition: when you submitted the report, it would automatically gather and send your system info, and in addition to this it would take a screen shot of what you were seeing and attach it to the report. This made graphics and UI bugs quite painless to report.

Anyway, I was averaging a few bug reports a day, so I got a lot of experience with that interface. It was always a good feeling to see bugs that I had personally reported get fixed in subsequent builds. (Though the responses/handling became abysmally slow during open beta... understandable, I guess, since there were far more testers.) I hope future online games adopt something similar, especially the transparency (being able to view others' bug reports) and the convenience (automatic screen shots, etc.)

I didn't realize this, but all my SHA implementations were still living in my Subversion repository. Well, after some meticulous converting from both my archived CVS and SVN repositories, I've managed to finally move them over to Mercurial.

Anyway, I'm not sure what else in Subversion I'd like to move over. Certainly, there's nothing left in there that's all that interesting to the public. So I will take down the public-facing SVN repo... someday.

Searching Google, I see nothing links directly svn.saddi.com. However, there's quite a lot of documentation that references flup and fcgi.py within it. Well, if anyone's been checking out that flup, it's 2 years out of date. And the standalone fcgi.py module... I'd rather just forget about that since it's hopelessly out of sync from the flup version. (It's basically the flup threaded fcgi server without thread pools.)

Just a few updates on some non-happenings. I recently discovered hg convert and hg transplant. I decided to re-do my Mercurial repositories (most originated in Subversion, while others came from CVS). I was entirely obsessed with converting the entire history to HG. (Projects like yafic and newsfish had the most history, being converted from CVS to SVN, then SVN to HG. Unfortunately, I didn't preserve history when going to SVN.) Anyhow, I'm pretty happy with the results.

On another note, I went ahead and upgraded Trac to 10.4. I also installed the Mercurial plugin and switched to that. Unfortunately, Trac doesn't currently support multiple repositories (see #2086), so my catch-all Trac page, http://trac.saddi.com/projects, only tracks a single project. (And that happens to be yafic since it's had recent activity...) I don't really feel like switching to one Trac environment per project, so I guess I'll wait out that feature request.

I'm currently trying out Mercurial, a distributed SCM that's 99% pure Python. I was able to migrate some of my recent projects (including full history) over to it. My repository is http://hg.saddi.com/.

I haven't fully committed yet, but it's very tempting since it's far easier to install (it's literally a "python setup.py install"). Setting up the web interface was a breeze too (it's built on top of WSGI!)

Oh well, I mention this now because I've finally committed something to my Hg repository. In other words, my SVN and Hg repositories are now out of sync...

Anyway, I know there's a Trac plugin for Hg which I will have to try out as well. Not sure if and when I'll fully commit. Only time will tell and I really haven't been working on any personal projects lately...

I'd also like to document a news site/comment moderation system that I've been thinking about for a very long time. I'll only paint the broad picture since really, I haven't given it much thought. I credit this discussion from long ago on kuro5hin for inspiration. (Some of "my" ideas may be blatant rips too... so I give credit where credit is due. )

Since moderation systems tend to get gamed... why not just turn it into a game? The site would revolve around fictitious money. I was thinking of having certain actions (like submitting a story or posting a comment) cost money. Not only that, but it would be like an investment. Highly rated comments or active stories would yield profit for the originator. Comments that are universally rated down or stories marked as spam would end up with all or most of the initial investment lost.

It works for people doing moderation/rating as well. For example, it would cost a small amount of money to rate a comment. When you rate a comment, you are basically saying: "I am betting that the average rating of this comment will be X" After some amount of time, an amount of money would be returned to you based on how close you were to the "actual" rating.

It would be a dynamic system. The cost of submitting a story, leaving a comment, rating, etc. would all be variables. As would things such as the return factor for positively-rated stories/comments. It would be adjusted (preferably automatically) to keep the economy adequately sized vs. the number of active users as well as encourage users to say, post more stories under a specific topic.

"New user inflation" (where the economy explodes due to a large influx of new users who get a starting balance) can be combated by transforming the site into something akin to a social network or an invite-only service. Basically, new users start out with a 0 balance and they are given loans, with interest, from existing users. The system enforces the loans by automatically paying it back in installments. (Though what to do with a deadbeat user...?)

Anyhow, the goal of the entire system, ultimately, would be to deter spamming and gaming by the use of multiple accounts. The system can be controlled, albeit indirectly, by setting various prices and rates.

There don't seem to be many AJP C libraries. In fact, there don't seem to be any (according to Google, at least). There's at least one FastCGI C library, which is unsurprising given the ubiquity of FastCGI. So yesterday afternoon, I decided to "read spec, write code" yet again and began a C implementation of the "container" (app server) side of AJP. After not having touched C for over 2-3 years, it was a good feeling to muck around with C and BSD sockets again. (Procedural programming, how I missed you!)

I finished it up in a few hours and it is now fairly complete. It's actually a pretty simple protocol, I've realized. All the complexity comes from the way requests/responses are encoded and decoded. (Otherwise, it's a fairly straightforward 1:1 mapping.)

Of course there were the 3 undocumented spec additions, the first two I had to figure out through experimentation so long ago and the last was conveyed to me by someone who actually looked at the source mod_jk/mod_proxy_ajp source. (As much as I believe in the whole "the best documentation is the source" thing, I don't really like looking at similar/related source code when implementing something.)

Anyhow, I don't think those 3 undocumented additions are documented anywhere (hah!) besides my source (ajp_base.py and ajp.c). So:

• When decoding strings, a string with the length of 0xffff is the same as the empty string. However, its trailing NUL is not in the stream.
• When sending SEND_BODY_CHUNK packets, the packets must be NUL terminated. However, this NUL must not be included in the SEND_BODY_CHUNK's length (but must be included in the packet's length).
• The value following an SSL_KEY_SIZE attribute is not an encoded string, but rather an AJP integer.

Anyway, the ultimate goal of this project is to create a Python WSGI AJP server that is implemented in C as much as possible. At this point, I have a simple proof-of-concept working that makes calls into an embedded Python interpreter. It doesn't implement WSGI at all though.

As far as request/response throughput is concerned, it looks promising. While the threaded pure-Python AJP server could only handle ~86 requests per second (with 100 parallel clients), the threaded C/Python hybrid version was nearly pushing 1000 requests per second. Of course it doesn't include WSGI overhead yet. But we shall see.

All this effort is, of course, inspired by the WSGI servers built upon the FastCGI C library: fcgiapp and python-fastcgi. If I actually used FastCGI, I'd probably be using one of those servers rather than flup's.

I guess.

I've worked on my Python "blog" project for a number of years now. If I remember correctly, under the hood, it started out using flat files for storage and CGI. Eventually I moved it to mod_python+Publisher, then to FastCGI+my own version of Publisher. Sometime before that, I switched to using PostGreSQL... eventually transitioning the site content and then the session data to the database. (The templating engine changed at least once. And of course eventually, came the switch of the whole webserver interface to WSGI.)

Anyhow, what I was aiming for was a user-customizable blog/forum site. Multi-column, a selection of premade components (such as polls and HTML boxes), user profiles, personal bookmarks, etc. It was pretty ambitious and it met most of those goals. Though little did I realize that if I just generalized my goals a bit, I would actually be creating a Content Management System.

I didn't come to that realization until I started playing around with CMS's recently. At first, being in my Java mood, I sought open source Java CMS's. I only stumbled across Lenya. I ran into some difficulty, and promptly tabled it. I figured there would be more Java CMS's out there. But I really didn't have much luck finding anymore... (ok, so I didn't try that hard, either)

I briefly looked at Plone (based on Zope). It seemed a bit overkill for what I wanted... and overtly complex. So I moved on the PHP-based projects. (Note that I eventually did try Plone. But my gut feeling was correct, especially compared to the PHP CMS's.)

So I decided to look at Joomla! and Drupal, based on all the hype around those projects. (Some of it good, some of it bad... i.e. security vulnerabilities that made the front page of Slashdot. )

I tried Drupal first and immediately fell in love. It was utterly simple to use and maintain. It was a breeze to set up multiple instances using the same installation. It had a whole array of 3rd party modules that did what I wanted. (And I found, it could do most of what I wanted out of the box already - I just had to enable the module.) Anyhow, I switched 3 of my (static) sites to Drupal to try it out.

Unfortunately, at this point, I never gave Joomla! a chance. I hear many good things about Joomla! pertaining to the ease of use (and the eye candy). But it had one fatal flaw in my eyes: it only supported MySQL. Hence the reason that I tried Drupal first...

As an aside, it's probably obvious, but I'm not a big fan of the L-M-P in LAMP. 1) I am and have been an avid FreeBSD user for the past decade. 2) I prefer PostGreSQL over MySQL, speed be damned. I like my transactions, thank you. 3) Whether the P stands for Perl or PHP, I'd rather have my apps written in Python and to some extent, Java.

Anyway, the only thing really missing (so far) from Drupal is a good CAPTCHA module. Solving a simple addition problem seems a bit simplistic. I'd be surprised if there weren't bots that could answer those CAPTCHAs already. And the image CAPTCHA support seems a bit lame, especially compared to the last CAPTCHA library I worked with, JCaptcha. Anyhow, I bring this up because even though I normally turn off user registration, I did forget to turn it off for one of my sites. Overnight, some 3-4 "people" (all using rather fake looking yahoo addresses) registered for my site. I'm sure if I had some actual content that you could post comments to, I'd also have a bunch of spam to take care of.

Anyway, one of the sites that was converted happens to be saddi.com. If I get more familiar/comfortable with Drupal, I'll probably be subsuming this blog into its Drupal instance.

I decided to put on my sysadmin hat and do a little work on kalahari (my co-located server). I didn't really have a plan. And I'd been upgrading things here and there over the past few weeks. (Like bringing Subversion up to 1.3.1 from 1.1.x. Thankfully that went painlessly.)

I decided to set up Trac, just to give it a try. Rather than install it through the FreeBSD ports system, I installed most of the dependencies and did a manual install. It was surprisingly painless. The Trac files and environment live in the trac user's home directory... and I run Trac using the WSGI patch + flup's AJP server. Quite simple.

As a digression, sometimes I wonder if people wonder about my affinity for AJP. My webserver has both mod_fastcgi and mod_jk. (Alas, I'm not brave enough to upgrade to httpd 2.2.x yet, for the built-in mod_proxy_xxx modules.) But I prefer to run web applications/services as a different user. CGI/FastCGI would require me to set up the application to run as www:www... and just setting up the permissions alone would be a nightmare. (Not to mention the insecurity of running as the generic www user... not that I don't trust the other users of my server. ) I can't use SCGI because mod_scgi doesn't coexist peacefully with mod_fastcgi under httpd 2.x. And I realize FastCGI apps can be run in an "app-server" mode for lack of a better term (as a manually spawned process), but configuration of AJP seems so much simpler.

Well, that and I run Tomcat as well. I would say the web applications/services on kalahari are now evenly split between Python-based and J2EE-based platforms. (Although one internal app still runs on WebObjects...)

Anyhow, I also decided to revive the old wiki, the main use of which was to serve the help pages for blog.flup.org. I did the same as I did with Trac. Manual installation, WSGI backend + AJP server from flup.

Lastly, I revived flup.org/u (my simple URL shortener). I took it down some time ago because of abuse, but it was always my intent to restore it once I had some basic authentication/authorization built in. Well, with the CAS server now long in place, it was pretty easy. As an aside, I've been itching to write some sort of CAS WSGI middleware, but I see Paste already has something. Still, it would be nice to have some sort of generic filter/interceptor framework. If not for WSGI, then for flup's Publisher.

I have to admit, I've barely done any Python programming since I started my new job. Though, oddly enough, my Java skills and J2EE knowledge have been steadily growing. In fact, lately, I can honestly say I've been enjoying doing Java/J2EE stuff. I can probably thank Eclipse and Spring.

I've had a few personal forays into Java web programming-land in the past (namely Velocity-Struts-Hibernate, and WebObjects). WebObjects I gave up because of the uncertain future and the lagging standards compliance. (I haven't tried WO 5.3 yet though.) Velocity-Struts-Hibernate was just a pain to set up. Plus I didn't like the fact that I had to edit a number of XML files just to add a new action/form...

But apparently, I'm a glutton for punishment because I just recently assembled a skeleton web application consisting of Hibernate (for the persistence layer), Spring (for management of the business layer), Struts (form validator and controller), and Velocity (template engine). Now, either I've become tolerant of editing lots of silly XML files (Spring adds at least one more file to edit), or Spring actually makes it easier to integrate everything.

I'm thinking it's a combination of both... and the fact that I just love Eclipse (which I was introduced to at work). It just makes editing Java so easy. After trying out the refactor feature, I don't think I can go back to using "just" emacs.

Anyhow, I recently deployed CAS as my single sign-on service on kalahari. It currently uses the blog.flup.org user database to authenticate against. (And the development version of blog.flup.org has already been modified to use CAS.) But, as long as I keep the momentum, the first Velocity-Struts-Spring-Hibernate application I deploy will be an account manager. (Providing registration, password changing, & password reset services.)

Hopefully with an SSO service in place, I can deploy more services that I developed but never finished a long time ago. Poor kalahari. It's running 3 different types of application servers, sometimes multiple instances of a particular type...