A small HTTP server in Python

The Python http.server module can be run from the commandline with

python3 -m http.server

to provide a simple http server that listens on a configurable port and basically serves any file from the directory it is started from.

That is a good start but I had some additional requirements to serve up files from my raspberry Pi:

• The server must be able to run as a demonized process, i.e, in the background
• It must server a limited set of files
• from a configurable directory and
• be able to run as a less privileged user

Now of course I could have opted for a light weight http server like nginx or lighttpd, but I decided to write my own so I wouldn't have to install a boatload of additional packages.

restrictedhttpserver.py

The server is based on the daemon module a wrote about in a previous article and can be started from the commandline in much the same way as http.server:


python3 restrictedhttpserver.py


It will run as the user that started it, will listen on port 8000 and start serving any file from the current directory. It will demonize itself as well.

It takes some additional options:

python3 restrictedhttpserver.py -h

usage: restrictedhttpserver.py [-h] [-p PID_FILE] [-l LOG_FILE] [-r ROOT_DIR]
                               [-n NAME] [-u USER] [-f] [-s] [-d]
                               [--bind ADDRESS] [-x] [-e EXT]
                               [port]

Example HTTP Server

positional arguments:
  port                  Specify alternate port [default: 8000]

optional arguments:
  -h, --help            show this help message and exit
  -p PID_FILE, --pid-file PID_FILE
  -l LOG_FILE, --log-file LOG_FILE
  -r ROOT_DIR, --root-dir ROOT_DIR
  -n NAME, --name NAME  name of the server used in log lines
  -u USER, --user USER  drop privileges of running server to those of user
  -f, --force           start a server even if pid file is present already
  -s, --stop            stop a running server
  -d, --debug           Run http server in foreground mode
  --bind ADDRESS, -b ADDRESS
                        Specify alternate bind address [default: all
                        interfaces]
  -x, --nodirlist       never show a directory listing
  -e EXT, --ext EXT     allowed file extensions (without .) May occur more
                        than once

For example if you wanted to run it as user www from the directory /public/www and just server html and css files, you could run it (as root) as follows:

python3 restrictedhttpserver.py -r /public/www -u www -e html -e css

Later you could stop this process by executing
python3 retrictedhttpserver.py -s


For more information have a look at the source code, it is pretty readable I think :-)

Availability

restrictedhttpserver.py and daemon.py are both available from this GitHub repository.

Demonizing a Python process

Properly demonizing a Python process is not easy, even on *nix, and although various sources of good material can be found on the internet (for example here and here) it didn't quite match my requirements and/or I didn't understand it completely.
I therefore decided to create both a daemon module and a http server to test it.

Code availability

Both modules are available on GitHub
The daemon module is called daemon.py and the httpserver is called restrictedhttpserver.py. Both have a fair amount of comments in their source code that are the result of me trying to fully understand what is required to get things working.

A small word of warning here: I only tested it with Python3 on Ubuntu and Raspbian and I do not expect it to work on any other operating system (although most *nix like systems probably will work) and in my world Python2 is no longer a valid option.

Also, the Daemon class makes use of an undocumented attribute in the logging.FileHandler class  (the stream.fileno attribute) which is necessary in order not to close this stream when forking a child process. Theoretically this might break in the future but I did want all logging to happen outside the chroot jail so that the process running as a daemon can log but does not have access to the logging. Whether this is a valid approach remains to be seen as this also prevents proper log rotation.

The Daemon class

The basic requirement are all implemented, an instance of a Daemon class can:

run in a chrooted jail

so it cannot access files outside a given directory

run with lowered privileges

so the risk of accessing files not owned by a specific user is lowered

can have a umask of your choice

which will ensure that newly created files by the daemon are not open to all

log outside its jail

so that we can log events without offering the process managed by daemon any way of altering the logging

and maintains a configurable pid file

so that we can prevent the daemon from starting more than once and providing a way the fond out the process id so that we can terminate the daemon

We also made the Daemon class follow the Singleton pattern, i.e. there can only be one instanced object of the class. This is sensible because a process can only be daemonized once. We also implemented to required methods to facilitate use as a context manager. A typical, minimal code snippet would look something like:

from daemon import Daemon

dm = Daemon()
with dm:
    ... do something forever ...

All configurable options have sensible defaults, but a call could look like this:

from daemon import Daemon

dm = Daemon(user='httpserver', rootdir='/var/www', umask=0o27,
             pidfile='/var/lockhttpserver.pid',
             logfile='/var/log/httpserver.log',
             name="Http Server")
with dm:
    ... do something forever ...

It is important to understand that when the Daemon object is created, the process is not immediately daemonized. This is handled by the context manager (or you could call the daemonize() method directly). There is also a stop() method provided that will check for a running daemon process and terminate it. This could look like this:

from daemon import Daemon
import argparse

parser = argparse.ArgumentParser(description="Example Daemon")
parser.add_argument('-s', '--stop', action='store_true', help='stop a running server')
args = parser.parse_args()

dm = Daemon()
if args.stop:
    dm.stop()
else:
    with dm:
        ... do something forever ...

More code

In a future article I will highlight the small http server that I implemented to test the daemon module.

I have posted an article on my Blender blog that investigates how to speed up the calculating the connectivity of trees with Numpy. Is completely non-Blender specific, so it might be interesting for readers of this blog as well.

Python 3 Web Development Beginner's Guide for free

Guess what, Packt will include my Python 3 Web development book in their Xmas countdown of free gifts on December 17th!

Free e-books at Packt

Packt (the publisher of my books on blender scripting and python web development) is giving away free e-books for Christmas.

I was a bit late in noticing it but so far they have given away some pretty interesting titles so it might be a good idea to check it out for more to come...

A pure Python kd-tree implementation

kd-trees are an efficient way to store data that is associated with a location in any number of dimensions up to twenty or so. Specifically, kd-trees allow for nearest neighbor searches in O(log n) time, something I desperately needed for my Blender tree generation add-on. In this article I highlight some of the design decisions that that shaped my pure Python implementation of a kd-tree module.

Visiting my own post five years later a lot has changed. We are now quite a few versions of Blender ahead of what was available in 2013 and a kd-tree implementation is now part of Blender's Python API.
I highly recommend using that implementation because is written in C and probably quite a bit faster than my pure Python implementation. There is also a bvh-tree implementation that is suited for problems that not such much involve points but geometry that fills space, like triangles.

kd-trees

There exist quite a few implementations in C or C++ that are part of robust and well tested libraries but for my Blender add-on I needed a platform neutral (= pure python) implementation that I could ship with the add-on. There are a few Python implementations but because I could not determine how well tested they were or because they did not quite meet my requirements I decided to develop an implementation myself.

Requirements

My requirements might not be everybody's requirements so besides supporting adding nodes and providing a nearest neighbor search what were the issues that were important to me?

Comes with test-suite
It might seem strange to make this the first requirement but because it's a part that is so essential to my add-on and because the code is quite tricky, I opted for test driven development. Adding unit tests to a python module is quite simple and although I can't claim that the tests cover all edge cases, they are nevertheless quite comprehensive. There is also a small performance test included.
Works with any iterable as position
A node in a kd-tree consists besides other things primarily of a position attribute. For use in Blender this typically would be a Vector instance but in other situations you might want to use other vector implementations. This kd-tree implementation assumes very little about its position attributes, just that it is subscriptable and that it supports a .dot() member function that returns the sum of the pairwise multiplication of its items (in other words, the dot product which is equal to the distance squared). Individual items should be floats (or anything that supports substraction, addition and comparisons). Probably any vector implementation will meet these requirements. The test-suite implements its own, based on Python's list class.
Allows easy deletion
Deleting nodes from a kd-tree is tricky and costly because of the rebalancing that has to be performed. I therefor opted for a much simpler solution: Instead of deleting a node we set its data attribute to None and add an option to the .nearest() member function to ignore nodes without data. Of course this won't make the tree any smaller but even when we delete a third of the nodes in random fashion (a typical use case for my tree add-on) the performance impact on nearest neighbor searches is minimal.

Availability

The module is licensed as GPL and available on Github as part of the spacetree add-on for Blender. It is contained in a single source file that can be downloaded separately (click Raw to download).

Blender 2.6 add-on tutorial

I finished a Blender add-on tutorial the other day that might be an interesting read. And actually I liked the way my cover image turned out, so I reproduce it here :-)