A small HTTP server in Python

The Python http.server module can be run from the commandline with

python3 -m http.server

to provide a simple http server that listens on a configurable port and basically serves any file from the directory it is started from.

That is a good start but I had some additional requirements to serve up files from my raspberry Pi:

  • The server must be able to run as a demonized process, i.e, in the background
  • It must server a limited set of files
  • from a configurable directory and
  • be able to run as a less privileged user
Now of course I could have opted for a light weight http server like nginx or lighttpd, but I decided to write my own so I wouldn't have to install a boatload of additional packages.

restrictedhttpserver.py

The server is based on the daemon module a wrote about in a previous article and can be started from the commandline in much the same way as http.server:


python3 restrictedhttpserver.py


It will run as the user that started it, will listen on port 8000 and start serving any file from the current directory. It will demonize itself as well.

It takes some additional options:

python3 restrictedhttpserver.py -h

usage: restrictedhttpserver.py [-h] [-p PID_FILE] [-l LOG_FILE] [-r ROOT_DIR]
                               [-n NAME] [-u USER] [-f] [-s] [-d]
                               [--bind ADDRESS] [-x] [-e EXT]
                               [port]

Example HTTP Server

positional arguments:
  port                  Specify alternate port [default: 8000]

optional arguments:
  -h, --help            show this help message and exit
  -p PID_FILE, --pid-file PID_FILE
  -l LOG_FILE, --log-file LOG_FILE
  -r ROOT_DIR, --root-dir ROOT_DIR
  -n NAME, --name NAME  name of the server used in log lines
  -u USER, --user USER  drop privileges of running server to those of user
  -f, --force           start a server even if pid file is present already
  -s, --stop            stop a running server
  -d, --debug           Run http server in foreground mode
  --bind ADDRESS, -b ADDRESS
                        Specify alternate bind address [default: all
                        interfaces]
  -x, --nodirlist       never show a directory listing
  -e EXT, --ext EXT     allowed file extensions (without .) May occur more
                        than once
For example if you wanted to run it as user www from the directory /public/www and just server html and css files, you could run it (as root) as follows:

python3 restrictedhttpserver.py -r /public/www -u www -e html -e css

Later you could stop this process by executing
python3 retrictedhttpserver.py -s


For more information have a look at the source code, it is pretty readable I think :-)

Availability

restrictedhttpserver.py and daemon.py are both available from this GitHub repository.

Demonizing a Python process

Properly demonizing a Python process is not easy, even on *nix, and although various sources of good material can be found on the internet (for example here and here) it didn't quite match my requirements and/or I didn't understand it completely.
I therefore decided to create both a daemon module and a http server to test it.

Code availability

Both modules are available on GitHub
The daemon module is called daemon.py and the httpserver is called restrictedhttpserver.py. Both have a fair amount of comments in their source code that are the result of me trying to fully understand what is required to get things working.

A small word of warning here: I only tested it with Python3 on Ubuntu and Raspbian and I do not expect it to work on any other operating system (although most *nix like systems probably will work) and in my world Python2 is no longer a valid option.

Also, the Daemon class makes use of an undocumented attribute in the logging.FileHandler class  (the stream.fileno attribute) which is necessary in order not to close this stream when forking a child process. Theoretically this might break in the future but I did want all logging to happen outside the chroot jail so that the process running as a daemon can log but does not have access to the logging. Whether this is a valid approach remains to be seen as this also prevents proper log rotation.

The Daemon class

The basic requirement are all implemented, an instance of a Daemon class can:
run in a chrooted jail
so it cannot access files outside a given directory
run with lowered privileges
so the risk of accessing files not owned by a specific user is lowered
can have a umask of your choice
which will ensure that newly created files by the daemon are not open to all
log outside its jail
so that we can log events without offering the process managed by daemon any way of altering the logging
and maintains a configurable pid file
so that we can prevent the daemon from starting more than once and providing a way the fond out the process id so that we can terminate the daemon
We also made the Daemon class follow the Singleton pattern, i.e. there can only be one instanced object of the class. This is sensible because a process can only be daemonized once. We also implemented to required methods to facilitate use as a context manager. A typical, minimal code snippet would look something like:
from daemon import Daemon

dm = Daemon()
with dm:
    ... do something forever ...
All configurable options have sensible defaults, but a call could look like this:
from daemon import Daemon

dm = Daemon(user='httpserver', rootdir='/var/www', umask=0o27,
             pidfile='/var/lockhttpserver.pid',
             logfile='/var/log/httpserver.log',
             name="Http Server")
with dm:
    ... do something forever ...
It is important to understand that when the Daemon object is created, the process is not immediately daemonized. This is handled by the context manager (or you could call the daemonize() method directly). There is also a stop() method provided that will check for a running daemon process and terminate it. This could look like this:
from daemon import Daemon
import argparse

parser = argparse.ArgumentParser(description="Example Daemon")
parser.add_argument('-s', '--stop', action='store_true', help='stop a running server')
args = parser.parse_args()

dm = Daemon()
if args.stop:
    dm.stop()
else:
    with dm:
        ... do something forever ...

More code

In a future article I will highlight the small http server that I implemented to test the daemon module.
I have posted an article on my Blender blog that investigates how to speed up the calculating the connectivity of trees with Numpy. Is completely non-Blender specific, so it might be interesting for readers of this blog as well.

Python 3 Web Development Beginner's Guide for free

Guess what, Packt will include my Python 3 Web development book in their Xmas countdown of free gifts on December 17th!

Free e-books at Packt

Packt (the publisher of my books on blender scripting and python web development) is giving away free e-books for Christmas.

I was a bit late in noticing it but so far they have given away some pretty interesting titles so it might be a good idea to check it out for more to come...

A pure Python kd-tree implementation

kd-trees are an efficient way to store data that is associated with a location in any number of dimensions up to twenty or so. Specifically, kd-trees allow for nearest neighbor searches in O(log n) time, something I desperately needed for my Blender tree generation add-on. In this article I highlight some of the design decisions that that shaped my pure Python implementation of a kd-tree module.

Visiting my own post five years later a lot has changed. We are now quite a few versions of Blender ahead of what was available in 2013 and a kd-tree implementation is now part of Blender's Python API.
I highly recommend using that implementation because is written in C and probably quite a bit faster than my pure Python implementation. There is also a bvh-tree implementation that is suited for problems that not such much involve points but geometry that fills space, like triangles.

kd-trees

There exist quite a few implementations in C or C++ that are part of robust and well tested libraries but for my Blender add-on I needed a platform neutral (= pure python) implementation that I could ship with the add-on. There are a few Python implementations but because I could not determine how well tested they were or because they did not quite meet my requirements I decided to develop an implementation myself.

Requirements

My requirements might not be everybody's requirements so besides supporting adding nodes and providing a nearest neighbor search what were the issues that were important to me?

    Comes with test-suite
    It might seem strange to make this the first requirement but because it's a part that is so essential to my add-on and because the code is quite tricky, I opted for test driven development. Adding unit tests to a python module is quite simple and although I can't claim that the tests cover all edge cases, they are nevertheless quite comprehensive. There is also a small performance test included.
    Works with any iterable as position
    A node in a kd-tree consists besides other things primarily of a position attribute. For use in Blender this typically would be a Vector instance but in other situations you might want to use other vector implementations. This kd-tree implementation assumes very little about its position attributes, just that it is subscriptable and that it supports a .dot() member function that returns the sum of the pairwise multiplication of its items (in other words, the dot product which is equal to the distance squared). Individual items should be floats (or anything that supports substraction, addition and comparisons). Probably any vector implementation will meet these requirements. The test-suite implements its own, based on Python's list class.
    Allows easy deletion
    Deleting nodes from a kd-tree is tricky and costly because of the rebalancing that has to be performed. I therefor opted for a much simpler solution: Instead of deleting a node we set its data attribute to None and add an option to the .nearest() member function to ignore nodes without data. Of course this won't make the tree any smaller but even when we delete a third of the nodes in random fashion (a typical use case for my tree add-on) the performance impact on nearest neighbor searches is minimal.

Availability

The module is licensed as GPL and available on Github as part of the spacetree add-on for Blender. It is contained in a single source file that can be downloaded separately (click Raw to download).

Blender 2.6 add-on tutorial

I finished a Blender add-on tutorial the other day that might be an interesting read. And actually I liked the way my cover image turned out, so I reproduce it here :-)



Web2py, a better alternative to CherryPy?

during a short holiday break I did some reading on Web2py and my first impression is that that it looks really good. It seems to be much more complete and data centric than CherryPy. I think I will spend some time experimenting with it the coming months.

Packt Publishing will soon release their 1000th title


My publisher Packt Publishing is about to release their 1000th title, which I think is pretty amazing. Them being altogether very pleasant people to work with I have no reservations pointing you to their site. If you sign up before September 30. there's even some surprise for you in store. You can read all about in their press release.

Being an author myself and having reviewed quite a number of the books published by them, I know it takes some genuine effort to produce a good book. Reaching your 1000th in about 8 years time is therefore a serious achievement. Conratulations!

3D Convex hull in Python, a Blender implementation.

A slight adaption of the code in my previous post to make it directly usable as a add mesh extension in Blender.

A Blender add mesh extension

As of Blender 2.64 there is a native Convex Hull operator availablein Blender.

I won't elaborate too much on my adaption as it has nothing to do with web development. The script can be downloaded from my site and should be installed in Blenders addons directory and enabled in the user preferences. It is tested with Blender 2.61.

I still think it is a nice implementation and although it is pure Python, it performs quite well. Of course there is a native convex hull implementation in Blender game engine but that one isn't accessible from Python. Previous implemenations relied on external programs like qhull, which might be faster for large point sets but having a pure Python implementation that is tightly integrated with Blender feels cleaner. It certainly can handle points sets of up to a couple of thousand points.

I have adapted it to randomize the order of the vertices if the hull algorithm bombs. This will happen if adding a point involves calculating the volume of many degenerate tetrahedrons which is the case when trying to compute the hull of Blenders default uv-sphere. I tried to use Python's decimal module but the results did not improve so now if an error is encountered the vertices are reordered and a second attempt is made. Not clean, but simple.

3D Convex hull in Python

In this article I present a present a reimplementation in pure Python of Joseph O'Rourke's incremental 3D convex hull algorithm from his book Computational Geometry in C.

A convex hull in pure Python

This is the second, rather off topic, article on computational geometry in this blog. The previous article presented an implementation of the marching tetrahedrons algorithm.
The goal of this article is to provide object oriented, pythonic code to compute the convex hull of a collection of 3D points. The code is contained in a single Python module that may be downloaded from GitHub.
A sample of how to use this module is shown below, where we create a a roughly spherical cloud of points, calculate its convex hull and print this hull in STL format to stdout. The resulting object is shown in the image (as seen in Blender).
from random import random
from chull import Vector,Hull

sphere=[]
for i in range(2000):
 x,y,z = 2*random()-1,2*random()-1,2*random()-1
 if x*x+y*y+z*z < 1.0:
  sphere.append(Vector(x,y,z))
h=Hull(sphere)
h.Print()

Difference from the original implementation

The original code restricted the coordinates of points to integers, here there is no such restriction. This might result in errors if the coordinates are large.

Marching Tetrahedrons in Python

In this atricle we show a simple implementation of the Marching Tetrahedrons algorithm in Python.

Marching Tetrahedrons

The listing povided below is a straightforward reimplementation in Python of the ideas and code presented by Paul Bourke. The image

shows the result of sampling a simple lobed function (rendered in Blender). I might implement the code directly in Blender but for now we export to an STL file that can be read by almost any 3D package. Note that the exported triangles do not have normals that point in a uniform direction, in fact we do no export any normals at all. You have to recalculate the normals in your 3D package before you can render the resulte with smooth shading.

class Vector: # struct XYZ
 def __init__(self,x,y,z):
  self.x=x
  self.y=y
  self.z=z
 
 def __str__(self):
  return str(self.x)+" "+str(self.y)+" "+str(self.z)
  
class Gridcell: # struct GRIDCELL
 def __init__(self,p,n,val):
  self.p   = p   # p=[8]
  self.n   = n   # n=[8]
  self.val = val # val=[8]

class Triangle: # struct TRIANGLE
 def __init__(self,p1,p2,p3):
  self.p = [p1, p2, p3] # vertices

# return triangle as an ascii STL facet  
 def __str__(self):
  return """facet normal 0 0 0
outer loop
vertex %s
vertex %s
vertex %s
endloop
endfacet"""%(self.p[0],self.p[1],self.p[2])

# return a 3d list of values
def readdata(f=lambda x,y,z:x*x+y*y+z*z,size=5.0,steps=11):
 m=int(steps/2)
 ki = []
 for i in range(steps):
  kj = []
  for j in range(steps):
   kd=[]
   for k in range(steps):
    kd.append(f(size*(i-m)/m,size*(j-m)/m,size*(k-m)/m))
   kj.append(kd)
  ki.append(kj)
 return ki

from math import cos,exp,atan2

def lobes(x,y,z):
 try:
  theta = atan2(x,y)         # sin t = o 
 except:
  theta = 0
 try:
  phi = atan2(z,y)
 except:
  phi = 0
 r = x*x+y*y+z*z
 ct=cos(theta)
 cp=cos(phi)
 return ct*ct*cp*cp*exp(-r/10)
 
def main():

 data = readdata(lobes,5,41)
 isolevel = 0.1

 #print(data)
 
 triangles=[]
 for i in range(len(data)-1):
  for j in range(len(data[i])-1):
   for k in range(len(data[i][j])-1):
    p=[None]*8
    val=[None]*8
    #print(i,j,k)
    p[0]=Vector(i,j,k)
    val[0] = data[i][j][k]
    p[1]=Vector(i+1,j,k)
    val[1] = data[i+1][j][k]
    p[2]=Vector(i+1,j+1,k)
    val[2] = data[i+1][j+1][k]
    p[3]=Vector(i,j+1,k)
    val[3] = data[i][j+1][k]
    p[4]=Vector(i,j,k+1)
    val[4] = data[i][j][k+1]
    p[5]=Vector(i+1,j,k+1)
    val[5] = data[i+1][j][k+1]
    p[6]=Vector(i+1,j+1,k+1)
    val[6] = data[i+1][j+1][k+1]
    p[7]=Vector(i,j+1,k+1)
    val[7] = data[i][j+1][k+1]
   
    grid=Gridcell(p,[],val)
    triangles.extend(PolygoniseTri(grid,isolevel,0,2,3,7))
    triangles.extend(PolygoniseTri(grid,isolevel,0,2,6,7))
    triangles.extend(PolygoniseTri(grid,isolevel,0,4,6,7))
    triangles.extend(PolygoniseTri(grid,isolevel,0,6,1,2))
    triangles.extend(PolygoniseTri(grid,isolevel,0,6,1,4))
    triangles.extend(PolygoniseTri(grid,isolevel,5,6,1,4))
    
 export_triangles(triangles)

def export_triangles(triangles): # stl format
 print("solid points")
 for tri in triangles:
  print(tri)
 print("endsolid points")
 
def t000F(g, iso, v0, v1, v2, v3):
 return []

def t0E01(g, iso, v0, v1, v2, v3):
 return [Triangle(
 VertexInterp(iso,g.p[v0],g.p[v1],g.val[v0],g.val[v1]),
 VertexInterp(iso,g.p[v0],g.p[v2],g.val[v0],g.val[v2]),
 VertexInterp(iso,g.p[v0],g.p[v3],g.val[v0],g.val[v3]))
 ]

def t0D02(g, iso, v0, v1, v2, v3):
 return [Triangle(
 VertexInterp(iso,g.p[v1],g.p[v0],g.val[v1],g.val[v0]),
 VertexInterp(iso,g.p[v1],g.p[v3],g.val[v1],g.val[v3]),
 VertexInterp(iso,g.p[v1],g.p[v2],g.val[v1],g.val[v2]))
 ]

def t0C03(g, iso, v0, v1, v2, v3):
 tri=Triangle(
 VertexInterp(iso,g.p[v0],g.p[v3],g.val[v0],g.val[v3]),
 VertexInterp(iso,g.p[v0],g.p[v2],g.val[v0],g.val[v2]),
 VertexInterp(iso,g.p[v1],g.p[v3],g.val[v1],g.val[v3]))
 return [tri,Triangle(
 tri.p[2],
 VertexInterp(iso,g.p[v1],g.p[v2],g.val[v1],g.val[v2]),
 tri.p[1])
 ]

def t0B04(g, iso, v0, v1, v2, v3):
 return [Triangle(
 VertexInterp(iso,g.p[v2],g.p[v0],g.val[v2],g.val[v0]),
 VertexInterp(iso,g.p[v2],g.p[v1],g.val[v2],g.val[v1]),
 VertexInterp(iso,g.p[v2],g.p[v3],g.val[v2],g.val[v3]))
 ]

def t0A05(g, iso, v0, v1, v2, v3):
 tri = Triangle(
 VertexInterp(iso,g.p[v0],g.p[v1],g.val[v0],g.val[v1]),
 VertexInterp(iso,g.p[v2],g.p[v3],g.val[v2],g.val[v3]),
 VertexInterp(iso,g.p[v0],g.p[v3],g.val[v0],g.val[v3]))
 return [tri,Triangle(
 tri.p[0],
 VertexInterp(iso,g.p[v1],g.p[v2],g.val[v1],g.val[v2]),
 tri.p[1])
 ]

def t0906(g, iso, v0, v1, v2, v3):
 tri=Triangle(
 VertexInterp(iso,g.p[v0],g.p[v1],g.val[v0],g.val[v1]),
 VertexInterp(iso,g.p[v1],g.p[v3],g.val[v1],g.val[v3]),
 VertexInterp(iso,g.p[v2],g.p[v3],g.val[v2],g.val[v3]))
 return [tri,
 Triangle(
 tri.p[0],
 VertexInterp(iso,g.p[v0],g.p[v2],g.val[v0],g.val[v2]),
 tri.p[2])
 ]

def t0708(g, iso, v0, v1, v2, v3):
 return [Triangle(
 VertexInterp(iso,g.p[v3],g.p[v0],g.val[v3],g.val[v0]),
 VertexInterp(iso,g.p[v3],g.p[v2],g.val[v3],g.val[v2]),
 VertexInterp(iso,g.p[v3],g.p[v1],g.val[v3],g.val[v1]))
 ]

trianglefs = {7:t0708,8:t0708,9:t0906,6:t0906,10:t0A05,5:t0A05,11:t0B04,4:t0B04,12:t0C03,3:t0C03,13:t0D02,2:t0D02,14:t0E01,1:t0E01,0:t000F,15:t000F}

def PolygoniseTri(g, iso, v0, v1, v2, v3):
 triangles = []

 #   Determine which of the 16 cases we have given which vertices
 #   are above or below the isosurface

 triindex = 0;
 if g.val[v0] < iso: triindex |= 1
 if g.val[v1] < iso: triindex |= 2
 if g.val[v2] < iso: triindex |= 4
 if g.val[v3] < iso: triindex |= 8

 return trianglefs[triindex](g, iso, v0, v1, v2, v3)

def VertexInterp(isolevel,p1,p2,valp1,valp2):
 if abs(isolevel-valp1) < 0.00001 :
  return(p1);
 if abs(isolevel-valp2) < 0.00001 :
  return(p2);
 if abs(valp1-valp2) < 0.00001 :
  return(p1);
 mu = (isolevel - valp1) / (valp2 - valp1)
 return Vector(p1.x + mu * (p2.x - p1.x), p1.y + mu * (p2.y - p1.y), p1.z + mu * (p2.z - p1.z))

if __name__ == "__main__":
 main()
 

A jQuery Mobile SelectorImage plugin

I implemented a jQueryMobile plugin to enhance a <select> element with a clickable image map. A full article with examples is available on on my site

A simple, JavaScript only implementation of an clickable image map.

In this age of html5 it is of course not done to use clickable image maps :-) Yet I did want to enhance <select> elements with clickable images to allow for a visual alternative for a drop down list without the hard to maintain imagemaps of old. I therefore implemented a jQueryMobile plugin that takes two maps (these are specified with data- attributes): one to display to the user and one to act as a definition of hotspots. These hotspots are simply colored areas in the hotspot map (instead of polygon definitions in an image map), When the user clicks the image the corresponding location is checked in the hotspot map and the color found is looked up in the list of <option> elements. If there is a match, that option is selected. A working example can be found here. A full explanation, including a small list of known problems can be found here.

Implementing a HTTPS server in Python

Web applications often transfer sensitive data between client and server. Even a session id is not something that should be vulnerable to eavesdropping. It is therefore a very good idea to encrypt all communication and implement a HTTPS server.

Subclassing HTTPServer

Python's ssl module has been cleaned up quite a bit since version 3.x and with a little help from this recipe it was incredibly simple to adapt the HTTPServer class from the http.server module to accept only secure connections:

import ssl
import socket
from socketserver import BaseServer
from http.server import HTTPServer
  
class HTTPSServer(HTTPServer):
 def __init__(self,address,handler):
  BaseServer.__init__(self,address,handler)
  
  self.socket = ssl.SSLSocket(
   sock=socket.socket(self.address_family,self.socket_type),
   ssl_version=ssl.PROTOCOL_TLSv1,
   certfile='test.pem',
   server_side=True)
  self.server_bind()
  self.server_activate()

All we do basically is change the initialization code to create a secure socket instead of a regular one (in line 10). The things to watch out for is the ssl_version: older versions are considered unsafe so we use TLS 1.0 here. Also the certificate file we use here contains both our certificate and our private key. If you want to use a self signed certificate for testing purposes you could generate one with openssl (most UNIX-like operating systems offer binary packages, for a precompiled package for windows check the faq.)

openssl req -new -x509 -keyout test.pem -out test.pem -days 365 -nodes

Note that your browser will still complain about this certificate because it is self signed.

Managing a session id with cookies

When managing sessions in web applications the key to the castle is a sessionid. Most often this sessionid is passed to and from the client by means of a cookie. In this article we explore the tools available in Python to deal with cookies.

Reading and writing cookies

If we are extending the BaseHTTPRequestHandler class from Python's http.server module we have access to the request headers by means of the headers attribute. It provides a convient method get_all() to retrieve headers by name (line 12 in the code sample below):

from http.cookies import SimpleCookie as cookie
...
class ApplicationRequestHandler(BaseHTTPRequestHandler):
 
 sessioncookies = {}

 def __init__(self,*args,**kwargs):
  self.sessionidmorsel = None
  super().__init__(*args,**kwargs)
  
 def _session_cookie(self,forcenew=False):
  cookiestring = "\n".join(self.headers.get_all('Cookie',failobj=[]))
  c = cookie()
  c.load(cookiestring)
  
  try:
   if forcenew or self.sessioncookies[c['session_id'].value]-time() > 3600:
    raise ValueError('new cookie needed')
  except:
   c['session_id']=uuid().hex
  
  for m in c:
   if m=='session_id':
    self.sessioncookies[c[m].value] = time()
    c[m]["httponly"] = True
    c[m]["max-age"] = 3600
    c[m]["expires"] = self.date_time_string(time()+3600)
    self.sessionidmorsel = c[m]
    break

The way we call get_all() provides us with an empty list if there are no cookies in the headers. Either way we end up with a (possibly empty) string that contains the cookies the client sent us. This cookie (or cookies) can be converted to a SimpleCookie object from Python's http.cookie module (line 14).

Cookies are basically key/value pairs with some extra attributes. The whole ensemble is called a morsel. The SimpleCookie object acts as a dictionary that indexes those morsels by key. The whole excersize is aimed at maintaining a session id so we check if our cookie object holds a session_id morsel and use its value (a GUID) as an index into the sessioncookies class variable. This class variabele maintains a dictionary indexed by the GUIDs of the session id cookies we produced. The corresponding values are their timestamps. Line 17 will therefore raise an exception if

  • no session_id cookie was provided by the client,
  • the session_id is unknown to us,
  • the session_id is expired, i.e. older than one hour, or
  • when we explicitely indicated we want a new cookie, no matter what.
In those cases we generate a complete new, random, GUID and store its hexadecimal representation.

At this point we are guaranteed to have a SimpleCookie object that contains a session_id. Our final tasks are to store the timestamp of this cookie and to update or set some additional attributes on this morsel. The client might have sent more than one cookie so we iterate over all morsel and stop at the first session_id morsel we find. We set its httponly attribute to signal to the browset that this cookie should not be manipulated by any client side JavaScript and set both its expires attribute and its max-age before we store this specific morsel in an instance variabele. This way we can add this cookie to the response headers one we have processed the request. An outline is sketched in the snipper below:

...
 def do_GET(self):
  ...
  self._session_cookie()
  ...
  if not (self.sessionidmorsel is None):
    self.send_header('Set-Cookie',self.sessionidmorsel.OutputString())
  ...

Security considerations

At this point we have a tool to manage a session id. Such a session id can be used as a key to access other session information, a topic we cover in a later article. Before we even start thinking of using this we should consider the security issues.

Pythonsecurity.org has a handy checklist that will walk through:

Are session IDs exposed in the URL?
No, we use cookies and those are part of the HTTP headers.
Do session IDs timeout and can users log out?
Our session IDs certainly timeout but providing a log out option should be part of the web application.
When a user logs out or times out, is the session invalidated?
At this point we only dealing with session ids.
Are session IDs rotated after successful login?
That is why we provide the forcenew paramter. After a successful login the web application should call _session_cookie() again with forcenew=True
Are session IDs only sent over TLS/SSL?
That should be implemented in the HTTP server, here we only look at the request handling part.
Are session IDs completely randomly generated?
We use a variant 4 uuid from Python's uuid module which should be completely random. The documentation makes no claims about the quality of the random generator used but a quick look at the code of the uuid module (in Python 3.2) reveals that is uses either system provided functions or the built-in random() function. In other words, we don't know what we get and that is bad! It is probably better use os.urandom() directly which will raise a NotImplentedError if a source of randomness couldn't be found. Generating a a string of 32 hex digits might be done as follows: "%02x"*16%tuple(os.urandom(16))

A jQuery Mobile Address plugin

While playing around with jQuery Mobile and wondering about the support for the new HTML5 input types I realized there wasn't an address extension that was just as easy to use a date picker was for dates. So I wrote an address plug-in.

Adding Google Maps to an input element

The idea is very simple: addresses are text just like an e-mail address or an url. HTML5 makes it possible to identify this a input element types and jQuery Mobile acts on these types as well to display custom widgets for these elements.

I wanted an address type: an input element with that type would allow the user to enter an address or a click on it would open an interactive map. Clicking on a location in that map would fill in the address under the cursor. Writing such a plug-in was a fine exercise in both jQuery Mobile and the Google Maps API and the first results are documented on my website as part of the Erica Web Application framework.

Erica web application framework

I started developing and writing about a new Python web application framework in Python to aid people in understanding concepts in web application development.

Erica web application framework

The framework is named after one of our cutest pygmy goats, Erica, which is of course completely irrelevant :-). More important is that the first article outlining the goals, is already on-line on my website. The articles will build and extend on blog entries made in this blog. The idea is that the website will develop into a small but comprehensive tutorial on implementing a framework while this blog will be the place to watch for updates and place your comments.

Python regular expressions and the functools module

If we use annotations to sanitize incoming data easy access to regular expressions is a must. In this article I'll show how to make the most of Python's bundles functools module to dynamically generate functions that match regular expressions while reducing the overhead of compiling regular expressions as much as possible

Matching against regular expressions

In our framework we annotate function parameters with functions that will pass through or convert incoming arguments if they are ok or raise an exception other wise. For example, if we would want to make sure an argument is an integer, we'd annotate it with Python's built-in int function:

def myfunction(arg:int):
	...

The framework will use this function to check the argument before actually calling the function.

In many situations it would be convenient if we could specify a regular expression that arguments should match. We could extend the framework to check whether the annotation was a string and use that string as a regular expression but it is better to opt for a approach that will allow for the easy definition of categories. Some examples: def myfunc(arg:digits) ... or def myfunc(arg:zipcode) .... In these examples digits and zipcode are functions that take single argument and return that argument if matches a certain pattern or raise an exception if it doesn't.

To construct such functions easily we may utilize Python's re and functoolsmodules:

from re import compile

def regex_compile(pattern):
	return compile("^"+pattern+"$")

def match(pattern,string):
	re = regex_compile(pattern)
	if re.match(string) is None:
		raise ValueError("no pattern match")
	return string

To check whether a string matches a regular expression we define a match function (line 6) that will compile the regular expression and match the string against it. The regex_compile function will make sure the match is anchored at the beginning and the end because we want a complete match, that is string that merely contain the pattern are considered invalid. We don't allow extraneous characters.

Constructing partial functions

So far nothing new under the sun but we still need a simple way to construct functions that match a single argument against a regular expression. Enter the partial function. It will take a function and some arguments and use those to construct a new function that will pass any arguments together with the original arguments to the encapsulated function. This is exactly what we need to construct our categories:

from functools import partial

digits = partial(match,r'\d+')
zipcode = partial(match,r'\d{4}\s*[a-zA-Z]{2}')

print(digits('12345'))
print(zipcode('1234AB'))

Improving efficiency by Memoization

Each time we call digits(a) we actually call match(r'\d+',a) and this will result in compile the regular expression again and again. Compiling regular expressions is a rather expensive operation so we might want to avoid that by reusing compiled expressions. This can simply be accomplished by applying the lru_cache decorator from the functools module to implement a memoize pattern:

from functools import lru_cache
from re import compile

@lru_cache(maxsize=0)
def regex_compile(pattern):
	return compile("^"+pattern+"$")

This simple change will make sure we will reuse compiled regular expressions most of the time. The maxsize parameter should be set as needed: it may be set to zero in which case all regular expression will be cached, possibly a good choice as it is unlikely that your web application sports thousands of regular expressions.

Propagating Python function annotations through decorators

When we decorate a Python function we must take some care to ensure that the decorated function can present itself in the same way as before its decoration. Let's explore what is needed.

Decorating a Python function

Say we have an undecorated function f(). It features a docstring (line 2) and annotated arguments:

def f(a:str):
	"an undecorated function"
	return a

print(f.__name__,f.__annotations__,f.__doc__,f('foo'))

The final line in the previous snippet prints the name of the function, its annotation dictionary and the docstring. The output looks like this:

f {'a': <class 'str'>} an undecorated function foo

Decorating a Python function

Now say we have a function do_something() and we would like to have a decorator that would arrange that a function would call do_something() first. We might set this up like this:

def do_something():
	print('oink')
	
def decorator(f):
	def g(*args,**kwargs):
		do_something()
		return f(*args,**kwargs)
	return g
	
@decorator
def f(a:str):
	"a naively decorated function"
	return a

print(f.__name__,f.__annotations__,f.__doc__,f('foo'))

The final line would produce the following output:

oink
g {} None foo

It is clear that do_something() is executed as it does print oink but although our decorated function can be called as f(), its name, annototations and doctring are different.

Preserving attributes when decorating a Python function

To resolve these matters we can rewrite our decorator as follows:

def decorator(f):
	def g(*args,**kwargs):
		do_something()
		return f(*args,**kwargs)
	g.__name__ = f.__name__
	g.__doc__ = f.__doc__
	g.__annotations__ = f.__annotations__
	return g
	
@decorator
def f(a:str):
	"a less naively decorated function"
	return a

print(f.__name__,f.__annotations__,f.__doc__,f('foo'))

We now simply copy the relevant attributes to the newly created function and if we now examine the output produced by the final line we get what we expect:

oink
f {'a': <class 'str'>} a less naively decorated function foo

Implementing POST method handling in a web application server

In an ongoing project to implement a web application server that is as simple as possible we now implement handling POST requests

Handling POST requests

The HTTP POST method is often the preferred way to transfer information from a form to the server. If we use POST the arguments don't end up in the webserver log for example and it allows us to use a file upload tag.

Depending on the encoding attribute of a form element, POST data might be encode as a number of key/value pairs (one on each line) or a multipart MIME message (useful if you want to upload files). More information can be found here.

Wielding the power of Python's cgi module

Decoding a multipart MIME message isn't trivial but lucky for us we can offload the hard work to an existing module: cgi. It is part of the standard Python distribution and although designed to implemented cgi scripts we can co-opt its functionality for our purpose. All we have to do is add a do_POST() method to our ApplicationRequestHandler class as shown below:

from cgi import FieldStorage
from os import environ

class ApplicationRequestHandler(BaseHTTPRequestHandler):

	def do_POST(self):
		ob=self._find_app()
		
		environ['REQUEST_METHOD'] = 'POST'
		fs=FieldStorage(self.rfile,headers=self.headers)
		kwargs={}
		for name in fs:
			for i in fs.getlist(name):
				if fs[name].file:
					kwargs[name] = fs[name].file
					kwargs[name].srcfilename = fs[name].filename
				else:
					kwargs[name] = fs[name]
		
		return self._execute(ob,kwargs)

We factored out some common code that is also used in the do_GET() method (in the _find_app() and _execute()methods, not shown here), but basically we instantiate a cgi.FieldStorage instance and pass it the input filestream along with a dictionary of the HTTP headers we have enounterd so far. Because the cgi module expects some parameters to be present in the environment we set the REQUEST_METHOD to POST because that information not part of the headers. The FieldStorage instance returned can be used as dictionary with the names of the parameters as keys.

Any file input parameters are a bit special. The contents of the uploaded file are stored in a temporary file and if we would access this parameter the value would be the contents of this file as a byte object. We'd rather pass on the temporary file object to prevent passing around the contents of really big files unnecessarily. We can check if a parameter is an uploaded file by checking its file parameter (line 14). If the argument is a file, we tack on the original file name (that is, the one the user's PC) because we might want to use that later. The final line passes the arguments we have extracted to the _execute() method. This method is factored out from the do_GET() method. It locates the function to execute and checks its arguments against any annotations