Python

While thinking about how I'm going to implement security in flannel, I thought: it would be nice to be able to reuse all the WSGI auth middleware out there. (Really, coming up with my own auth scheme to fit within flannel's framework did not seem very appealing.)

But it seems, you either protect the entire application, or you don't. Nothing so far that I've seen lets you protect only certain URL patterns. So generalizing upon this, it would be nice to have:

A conditional filter-type middleware that will accept: a bunch of URL patterns and a middleware instance. If the PATH_INFO matches any of the URL patterns, the middleware is invoked... otherwise the request is passed directly to the application. One can build more complex conditional behavior by composing different instances of this filter middleware.

And it would be nice if the pattern language were something simple, like Ant-style path pattern matching. Other nice-to-have features would be pluggable pattern matchers (maybe people would rather use regexes... who knows?), a caching decorator for pattern matchers.

Another wish-list item I had, which is somewhat also security-related:

"Remember Me" middleware: This would be similar to paste.auth.cookie but the cookies would be explicitly persistent. Perhaps signing the username + random junk and stuffing this (along with the signature) into a persistent cookie.

A nice-to-have feature would be including the date the cookie was created into the signed data. This would allow some sort of "expire all persistent login data" feature.

Anyway, stuff to work on if I have time and it interests me enough. I haven't been doing much lately because I haven't been feeling well. A shame though, that a little illness would utterly stop all my Python/flannel momentum.

In an effort to quell a warning from wsgiref's validator, QUERY_STRING will now default to an empty string if it doesn't exist in the environ. Despite not being required to always be present by the WSGI spec, it looks like the cgi module will assume sys.argv[1] is the query string if QUERY_STRING isn't present in the environ.

Also, I changed the keyword parameters in GzipMiddleware a bit:

• mimeTypes -> mime_types
• compresslevel -> compress_level

Lastly, unless I hear from people otherwise, I will be removing flup.publisher and flup.resolver from flup.

And as far as flup.middleware goes, ErrorMiddleware will be disappearing as well. There are better options from Paste, especially paste.evalexception for development. GzipMiddleware and SessionMiddleware are now currently looking for a new home, as I would like to remove them from flup eventually too.

You know a project is getting serious when you create a Trac site and a blog category for it.

So I've been hammering away at this thing for the past 3-4 days. I'm surprised to say that it's actually in a working state now. Maybe functional enough to be classified as a 'toy' web framework? Well, I suppose I need to finish more components first, since having only text fields for form input could get quite frustrating.

But the basics are all there... component rendering, event triggers, parameter binding (both by value and by reference), automatic persistence of session data.

Time to polish it, add more components, refine the interfaces. But I wonder... do I ever really want to publish it? Make it widely available? Document and support it?

Hmmm... I don't know.

But for now, I'd rather write code.

Related to a recent entry I've made, I started thinking about a Python port (at least in concept) of Tapestry. Rather than go for the template/spec/class approach of Tapestry 4, I opted to go for the wholly annotation-based approach of Tapestry 5 (still in development) thus eliminating the need for a spec file.

Annotating methods/functions in Python is pretty simple, just use decorators. Annotating member variables, on the other hand, seemed non-trivial. But then I remembered ORMs like SQLObject and SQLAlchemy and that gave me an idea. However, it would require metaclass magic...

So when I woke up this morning, I grabbed my laptop and started hacking away at a proof-of-concept (while still in bed!) Many hours later (and I did get out of bed, eventually), I had most of the metaclass infrastructure for flannel done. Yes, flannel. (I'm entertaining ideas for a better name though.)

So now I can easily mark variables as a persistent, transient, or parameter type. And the variables will be magically transformed into a Python property with my own setters and getters. Also, I can mark methods with certain "lifecycle decorators" which defines when during the rendering cycle they should be called. Basically something like:

class MyComponent(BaseComponent):
  value1 = Persistent()
  value2 = Transient(42)
  param1 = Parameter('foo')
  param2 = Parameter(required=True)

  @setup_render
  def do_something(self, out):
    pass

Transient variables don't really need to be annotated... but doing so allows them to be reset to an initial value at the start of a processing cycle. Speaking of which, if you aren't familiar with Tapestry, the abbreviated request/response cycle I'm aiming for looks something like:

1. Figure out which page/component the request was targetting via PATH_INFO.
   
2. Get an instance of the page from the pool.
   
3. Restore persistent variables from storage (typically the session), reset transient variables to their initial values, bind parameter variables and ensure all required parameter variables are bound.
   
4. Render the page! (Which recursively renders components within.)
   
5. Save persistent variables to storage.
   
6. Return page to pool.
   

Actions or pages with forms would require slightly different handling, which I haven't thought about yet. Form handling, I think, is the strength of these event-driven frameworks... so I'm aiming to get it right. Basically you "bind" input components (like text fields, checkboxes, etc.) to properties or even property paths and give it an "event handler" to call when the form is submitted. When your event handler is called, you can expect the properties to reflect the user input.

Anyhow, a significant (and unfortunate, I think) part of this project will involve coming up with yet another templating engine. However, it should be a fairly simple tag-based system, since most of the functionality will actually be implemented by components.

It will be interesting. Though I don't expect to take this project very seriously. But maybe if/when I actually get it to where I want it to be, I'll change my mind.

I added FCGIApp and SCGIApp to the (new) flup.client package. I wrote FCGIApp a few months ago for Ian Bicking's WPHP. While attempting to debug a problem with flup's fcgi server on Dreamhost (vs. my old fcgi.py module), I thought of an interesting solution.

Since the problem seemed to stem from the app's long startup time, and since Dreamhost only allows dynamic (i.e. web server-launched) FastCGI apps, why not have the web server-started app be as simple as possible and have it forward requests to a static, manually-launched app server?

In other words:

httpd (mod_fastcgi) -> fcgi.WSGIServer -> FCGIApp -> fcgi.WSGIServer -> app

Yeah, it ends up doing some extra work... which is why I wrote SCGIApp (SCGI is a far simpler protocol):

httpd (mod_fastcgi) -> fcgi.WSGIServer -> SCGIApp -> scgi.WSGIServer -> app

Anyway, it's a trade off and I really don't know how significant it is. But I was never really a fan of web server-launched FastCGI apps. Unless you configure mod_fastcgi explicitly, it has a tendency to launch an application multiple times. And if your application isn't multi-process safe and aware, it can lead to problems.

Consider these two modules experimental for now. I'm not really sure where I want them to live. But they're there for the time being to play around with.

Nothing about Python per-se, but applications/frameworks written in Python that I wish I could find. (And that weren't related to Zope/Plone!)

• A Python portal framework. Perhaps even a standardized Python portlet specification, to facilitate this.


• Similarly, a Python CMS, along the lines of Drupal and Joomla! Is there one?


• A web framework similar to Tapestry (and therefore, similar to WebObjects). But perhaps more Pythonic, whatever that means within that context.

I'm contemplating starting one of those projects myself. But I've been so out of touch with the Python community. Supporting flup has been almost nightmarish lately. (I guess I'm just too nice... it is OSS afterall.)

Poor flup. Reviled and yet seemingly a necessary evil because it was one of the first. (But thankfully, no longer the only choice as more FastCGI servers are released.)

Whatever. Winter break is coming up. I should have time to work on something.

I briefly toyed with Trac some time ago. Though I had to patch it to use WSGI, I found it pretty easy to use and maintain. My efforts eventually stalled, however, probably due to the way my public repository was derived from my real repository (i.e. lots of empty changesets).

Anyway, yesterday I updated to Trac 0.10.2 and was pleasantly surprised to find that it supported WSGI and flup's ajp and scgi servers out-of-the-box. The empty changeset issue is still there (which is my problem to fix). I just might have to split my repository up in the future.

Anyhow, flup has its own Trac site now:

http://trac.saddi.com/flup

The old flup site, http://www.saddi.com/software/flup/ simply redirects there now. (And will probably do so for the foreseeable future.) Please continue to use this URL when referencing flup.

I locked down all the default pages, but all the flup-related pages (save for the front page) are editable by anyone. So feel free to contribute examples, FAQs, etc.

For whatever reason, I had set up the AJP, FastCGI and SCGI servers to ignore EPIPE exceptions. (The exception that's thrown when writing to a disconnected socket, i.e. a web client that cancels a web page load or a download.) I think I had written the original version of fcgi (which I had written a millennia ago, it feels like), to do this... and since it was the basis for all the other servers... I never really questioned why EPIPE was being ignored. (To save the application from having to handle it?) I guess I just never tried downloading large files and then subsequently cancelling.

Anyhow, that behavior's been changed. It's still being ignored, but at a higher level. What this means is that WSGI applications may possibly see EPIPE socket.error exceptions, especially if they use the WSGI write callable. Also, the servers will stop iterating over the application's iterator should an EPIPE exception be thrown. (Which seems to be the correct thing to do anyway.)

Thanks to Ivan Sagalaev for bringing the problem to my attention!

I added a umask keyword parameter to both fcgi.WSGIServer and fcgi_fork.WSGIServer. If given and binding to a UNIX socket, the umask will be changed right before the socket is bound. Afterwards, the umask will be restored.

This should allow one to change the file permissions of the socket without altering the umask of the entire application. (If you need it to be, say, group-writable or world-writable.)

I've added paste.server_factory-compliant factories as well as respective egg entry points for each server within flup. Thanks to Luis Bruno for providing the code.

Additionally, each server now accepts a 'debug' bool keyword, which is True by default. It affects the output of the server-level error handler (which really shouldn't be used, but it's there if you're lazy like me ).

Though I just noticed that PasteScript already includes server runners for the flup servers...

Apparently, the use of signal.SIGHUP was the only thing preventing the use of the (threaded) servers on Windows. I made a minor change to conditionally install the SIGHUP handler. Thanks to Brad Miller for pointing this out.

Now whether the forked servers work on Windows, I really really doubt it.

Though I'm curious, how functional is flup under a CygWin-built Python?

I mentioned the new error404 feature in the last entry - after some review, I decided to change this a bit. There was no reason for it to be at the Resolver level, setting at Publisher made more sense. So I changed things around to reflect this new design. However, if you previously subclassed Publisher to override the _error404 method, that feature is now broken. You'll have to rewrite your custom _error404, which should be trivial.

Resolvers will now take a keyword argument, "error404", which allows you to specify a callable which will be called to render the 404 error page. Previously, you had to subclass Publisher to customize the 404 page (and it was rather low-level - basically a WSGI mini-app). The callable is called using the normal Publisher conventions. Thanks to Scot Doyle for the idea & code!

Also, I decided to look at the Python Egg thing more and converted flup's setup.py to use setuptools. (It does not require setuptools, however. It will fall back to distutils, if needed.)

From now on, the dist directory will contain an egg of the latest version of flup. (I haven't decided on a version numbering scheme yet, so it will always be 0.5. )

I just fixed a rather insiduous 2.3-only problem in fcgi: Apparently, the 2.3 socket module does not define the SHUT_* constants for the shutdown() syscall. This caused an attribute error that was silently being ignored. The end result was that the FastCGI socket connection with the server was not being closed - this caused output to hang in certain situations.

I didn't see the problem because as I said previously, I run 2.4 everywhere - even on my Macs (which normally come with 2.3 out-of-the-box).

Anyhow, big thanks to Eugene Lazutkin for bringing the problem to my attention, helping to isolate the problem (since it was unreproducible on my end, I had a hard time believing it was fcgi's fault ), and finally, providing access to an environment where it was reproducible 100% of the time (which allowed me to figure it out).

Just some superficial rewrites to remove the reliance on decorators (a 2.4 addition). I run 2.4 everywhere I care to run Python... plus the short-hand notation for decorators is so much more convenient. But I guess if such a trivial change would keep compatibility with 2.3, it would benefit more people to make it.

Once Python 2.5 comes around though...