Allan Saddi's projects blog

I went ahead and installed the original mod_scgi and it opened my eyes to how utterly different (or maybe broken) mod_proxy_scgi's handling of SCRIPT_NAME/PATH_INFO was.

I've been poring over the source for both modules, and though it's been years since I've coded Apache modules, I couldn't find any glaring differences. mod_scgi set the variables itself using some particular logic, while mod_proxy_scgi called ap_add_cgi_vars() to set the variables. That function appears to use the same logic.

So in spite of all that, I just cannot explain why Apache's mod_proxy_scgi is so broken. To give you an idea of how broken, given the request URI of "/foo/bar%20garply", it would set SCRIPT_NAME/PATH_INFO as follows:

SCRIPT_NAME: /foo/bar garply
PATH_INFO: /foo/bar%20garply

The sane thing to do (which mod_scgi does) is:

SCRIPT_NAME: (empty)
PATH_INFO: /foo/bar garply

So not only is SCRIPT_NAME utterly wrong for something mounted at the root, but PATH_INFO remains quoted.

My solution involves re-deriving SCRIPT_NAME/PATH_INFO from REQUEST_URI. Of course, mod_scgi doesn't need this, and in fact, shouldn't use it all since it sets SCRIPT_NAME correctly for non-root-mounted applications (removing the need to manually specify it).

Since I wouldn't trust auto-detection of the SCGI module, I decided to leave it up to the invoker of scgi-wsgi to decide what's right. Which lead to this lovely feature being created (from scgi-wsgi's README):

Due to the inconsistency between the various SCGI connectors, you may need to specify an environment profile using the -E option. The default profile is pass-through. The profiles are described below:

• pass-through passes SCRIPT_NAME and PATH_INFO unchanged from the connector. REQUEST_URI will be unquoted unless the -Q option is used. Any scriptName passed from the command-line (or via WSGI_SCRIPT_NAME) is ignored. This profile is meant for the original mod_scgi available at http://python.ca/scgi/
• request-uri only trusts REQUEST_URI and derives SCRIPT_NAME and PATH_INFO from it. scriptName must be given from the command-line (or via WSGI_SCRIPT_NAME) if the application resides at a non-root location. This profile is meant for Apache HTTPD's mod_proxy_scgi.
• script-name only trusts SCRIPT_NAME and PATH_INFO. It forms REQUEST_URI by their concatenation and then re-derives SCRIPT_NAME and PATH_INFO using scriptName given from the command-line (or WSGI_SCRIPT_NAME). This option is here for completeness.

Other SCGI modules (e.g. for lighttpd and nginx) have not yet been tested, but I imagine their handling of SCRIPT_NAME/PATH_INFO falls under one of the above.

So... yeah. I'm not too happy about it, and it diminishes my motivation to switch over to SCGI. I like the mod_proxy* stuff because it lets me use load balancing and/or set backend connection limits. But between the two, only mod_scgi seems to be the saner implementation. (As mentioned, I've yet to try nginx and lighttpd).

If I can gather code/test cases into a coherent bug report, I'll be posting one with Apache. Because really, the presence of quoted characters in the URL should not throw everything off.

Shortly after my last entry, I switched from a CPU-bound WSGI app to something comparatively more I/O heavy: serving a static 10K page. I then tried it out with the non-threadpool version of scgi-wsgi. The results surprised me. scgi-wsgi was doing well over 700 (nearly 800 at times) requests per second while ajp-wsgi only mustered 300 requests/sec.

Though of course, like before, when I switched to the non-preforking scgi-wsgi, its throughput dropped to a little over 50 requests/sec. ajp-wsgi maintained 300 requests/sec even while forking.

Given the non-threadpool performance of scgi-wsgi, I was spurred to write my own preforking server code (again), this time in C. The result can be found here. Unlike flup's preforking server, this one is based on descriptor passing. (And since I couldn't find my copy of UNIX Network Programming, I have to thank Google for having it browsable online. )

Hooking up the prefork server into scgi-wsgi, I now get similar performance to the threaded version: 700+ requests/sec. With that, scgi-wsgi graduated from 'limbo' to 'alpha.' And since mod_proxy_scgi is now included with vanilla Apache HTTPD, I will probably be transitioning my stuff from ajp-wsgi to scgi-wsgi. (Eat your own dog food and all that.)

As for the future, I would like to merge the two. I'm still entertaining the idea of hooking up my C WSGI code to an embedded HTTP server, similar to what PyCaduceus did (but remaining a top-level program rather than a Python module). I haven't really found an embeddable HTTP server with an interface that I like, though mongoose looks promising.

In the meantime, I'll go ahead and release ajp-wsgi 1.1 and scgi-wsgi 1.1 (eventually).

scgi-wsgi Hg repository (don't forget to clone scgi and preforkserver as well)
scgi-wsgi tarball

I just recently discovered that Apache HTTPD now ships with mod_proxy_scgi. (Probably old news to some, that was 3 patch versions ago.) Couple that with the fact that I stumbled upon PyCaduceus again recently (which proves to me that the C WSGI code written for ajp-wsgi was relatively transport-independent)... So I decided to spend a few hours today to writing an SCGI driver in C.

Well, it was actually pretty easy considering the SCGI spec is only a page long. DJB's netstrings description was just as short.

Replacing AJP in ajp-wsgi was straightforward as well. In fact, building the WSGI environment is a lot simpler with SCGI since the "headers" from the web server don't need to be re-interpreted/converted. (And I've yet to test this, but I don't think specifying the scriptName is necessary anymore.)

Once scgi-wsgi was in a working state, I was curious how it compared to ajp-wsgi. The results were surprising at first, but later made sense once I figured out what was happening. These were on my server using an extremely compute-bound WSGI application:

Driver	Requests/sec
ajp-wsgi threading	~96
scgi-wsgi threading	~92
ajp-wsgi forking	200+
scgi-wsgi forking	~53(!)

So... yeah. scgi-wsgi is slightly slower when threading and significantly slower when forking. Despite SCGI's simplicity, the explanation for this is that AJP uses persistent "backend" connections while SCGI uses one connection per request. In other words, the SCGI version was heavily penalized because there was no thread pooling or process pooling.

So where do I go from here? I'll probably just continue on with ajp-wsgi 1.1 as planned and leave scgi-wsgi in limbo. (I was entertaining the idea of merging the two.) SCGI needs thread/process pools to be competitive. Unless I find C implementations with a friendly-enough license (i.e. not copyleft), it's probably not worth writing my own pool implementations just for scgi-wsgi. The forking reference implementation (now found in Paste, I believe) is still probably the best.

(As an aside, since the SCGI protocol is so simple, moving the implementation to C to avoid the GIL probably doesn't improve things much over a pure-Python implementation.)

Hg Repository

Addendum: Apparently, the C implementations shine over their pure-Python counterparts when uploading files (i.e. large request body). When uploading a ~100MB file, ajp-wsgi and scgi-wsgi took ~6 seconds and ~4 seconds respectively (simplicity wins out!) When using flup ajp/scgi and Paste's scgiserver, I couldn't be bothered to wait for the upload to complete. (It was well over a few minutes for each before I canceled.)

I was curious about what it would take to add forking/multi-process support to ajp-wsgi and apparently the answer is: not much!

Well, it becomes trivial when you don't bother with preforking or process pools. I don't think an AJP backend really needs to worry much about saving fork() time since connections (and therefore processes) are more or less persistant.

I've done some basic stress testing using ApacheBench (ab): 100 concurrent requests, 500 concurrent requests. It's looking promising (no errors!). I haven't done any extensive testing between threaded vs. forking, but the results seem to be the same (using a freshly created Pylons app). Obviously the GIL isn't really being exercised when serving a static ~5K page.

And I did learn one oddity about mod_proxy/mod_proxy_ajp — apparently, when you specify max connections, you're actually specifying max connections per httpd process. On my server, where httpd uses the worker MPM (hybrid thread/process), the hard process limit is apparently 16, so even specifying max=1 means ajp-wsgi can expect up to 16 connections. (Conveniently for me, ajp-wsgi's default process limit is 16.) Something to keep in mind when tweaking ajp-wsgi's maxConnections parameter.

(Also note that if mod_proxy_ajp ever goes over ajp-wsgi's maxConnections limit, users will see HTTP 503 errors.)

Anyway, I think I'll stick to threading for now. But 1.1 alpha is out there for anyone interested...

ChangeLog
Tarball
Hg repository