scgi-wsgi?

I just recently discovered that Apache HTTPD now ships with mod_proxy_scgi. (Probably old news to some, that was 3 patch versions ago.) Couple that with the fact that I stumbled upon PyCaduceus again recently (which proves to me that the C WSGI code written for ajp-wsgi was relatively transport-independent)... So I decided to spend a few hours today to writing an SCGI driver in C.

Well, it was actually pretty easy considering the SCGI spec is only a page long. DJB's netstrings description was just as short.

Replacing AJP in ajp-wsgi was straightforward as well. In fact, building the WSGI environment is a lot simpler with SCGI since the "headers" from the web server don't need to be re-interpreted/converted. (And I've yet to test this, but I don't think specifying the scriptName is necessary anymore.)

Once scgi-wsgi was in a working state, I was curious how it compared to ajp-wsgi. The results were surprising at first, but later made sense once I figured out what was happening. These were on my server using an extremely compute-bound WSGI application:

Driver	Requests/sec
ajp-wsgi threading	~96
scgi-wsgi threading	~92
ajp-wsgi forking	200+
scgi-wsgi forking	~53(!)

So... yeah. scgi-wsgi is slightly slower when threading and significantly slower when forking. Despite SCGI's simplicity, the explanation for this is that AJP uses persistent "backend" connections while SCGI uses one connection per request. In other words, the SCGI version was heavily penalized because there was no thread pooling or process pooling.

So where do I go from here? I'll probably just continue on with ajp-wsgi 1.1 as planned and leave scgi-wsgi in limbo. (I was entertaining the idea of merging the two.) SCGI needs thread/process pools to be competitive. Unless I find C implementations with a friendly-enough license (i.e. not copyleft), it's probably not worth writing my own pool implementations just for scgi-wsgi. The forking reference implementation (now found in Paste, I believe) is still probably the best.

(As an aside, since the SCGI protocol is so simple, moving the implementation to C to avoid the GIL probably doesn't improve things much over a pure-Python implementation.)

Hg Repository

Addendum: Apparently, the C implementations shine over their pure-Python counterparts when uploading files (i.e. large request body). When uploading a ~100MB file, ajp-wsgi and scgi-wsgi took ~6 seconds and ~4 seconds respectively (simplicity wins out!) When using flup ajp/scgi and Paste's scgiserver, I couldn't be bothered to wait for the upload to complete. (It was well over a few minutes for each before I canceled.)