Demonizing a Python process

Properly demonizing a Python process is not easy, even on *nix, and although various sources of good material can be found on the internet (for example here and here) it didn't quite match my requirements and/or I didn't understand it completely.
I therefore decided to create both a daemon module and a http server to test it.

Code availability

Both modules are available on GitHub
The daemon module is called daemon.py and the httpserver is called restrictedhttpserver.py. Both have a fair amount of comments in their source code that are the result of me trying to fully understand what is required to get things working.

A small word of warning here: I only tested it with Python3 on Ubuntu and Raspbian and I do not expect it to work on any other operating system (although most *nix like systems probably will work) and in my world Python2 is no longer a valid option.

Also, the Daemon class makes use of an undocumented attribute in the logging.FileHandler class  (the stream.fileno attribute) which is necessary in order not to close this stream when forking a child process. Theoretically this might break in the future but I did want all logging to happen outside the chroot jail so that the process running as a daemon can log but does not have access to the logging. Whether this is a valid approach remains to be seen as this also prevents proper log rotation.

The Daemon class

The basic requirement are all implemented, an instance of a Daemon class can:

run in a chrooted jail

so it cannot access files outside a given directory

run with lowered privileges

so the risk of accessing files not owned by a specific user is lowered

can have a umask of your choice

which will ensure that newly created files by the daemon are not open to all

log outside its jail

so that we can log events without offering the process managed by daemon any way of altering the logging

and maintains a configurable pid file

so that we can prevent the daemon from starting more than once and providing a way the fond out the process id so that we can terminate the daemon

We also made the Daemon class follow the Singleton pattern, i.e. there can only be one instanced object of the class. This is sensible because a process can only be daemonized once. We also implemented to required methods to facilitate use as a context manager. A typical, minimal code snippet would look something like:

from daemon import Daemon

dm = Daemon()
with dm:
    ... do something forever ...

All configurable options have sensible defaults, but a call could look like this:

from daemon import Daemon

dm = Daemon(user='httpserver', rootdir='/var/www', umask=0o27,
             pidfile='/var/lockhttpserver.pid',
             logfile='/var/log/httpserver.log',
             name="Http Server")
with dm:
    ... do something forever ...

It is important to understand that when the Daemon object is created, the process is not immediately daemonized. This is handled by the context manager (or you could call the daemonize() method directly). There is also a stop() method provided that will check for a running daemon process and terminate it. This could look like this:

from daemon import Daemon
import argparse

parser = argparse.ArgumentParser(description="Example Daemon")
parser.add_argument('-s', '--stop', action='store_true', help='stop a running server')
args = parser.parse_args()

dm = Daemon()
if args.stop:
    dm.stop()
else:
    with dm:
        ... do something forever ...

More code

In a future article I will highlight the small http server that I implemented to test the daemon module.

Singleton objects

Singleton objects are not some fancy concept but a very practical solution to representing a specific value as an object as Pythons None object shows.

Many programming languages provide syntactical solutions to provide constants. Literals like 123 or "a string" are almost always constants but if you want to give a meaningful name to a constant you need a different approach.

Referring to value by name is simple enough, after all a variable assignment does just that, but you need some way to indicate that that variable shouldn't be altered after the assignment. Python does not provide a way to specify a constant but there are ways around this, see for example (this recipe).

Paradoxically, it is comparatively simple to define a class that allows only a single instantiated object. There are arguments for and against this Singleton Design Pattern and this article is a good starting point if you want to read about it. And whatever the merits of this design pattern, you can't avoid it because many constants in Python are implemented as singleton classes, for example True, False, NotImplemented and None.

And the None implementation illustrates another practical advantage: when comparing a value against a singleton we can check whether they are identical (with the is operator) instead of comparing their values (with the == operator) and although this might not be the primary purpose, it does give us a clear speed advantage as the next snippets shows:

import timeit

s1 = "123 == None"
s2 = "a == None"
s3 = "123 is None"
s4 = "a is None"

count=1000000
t = timeit.Timer(stmt=s1,setup='a=123')
print (s1, "%.2f usec/pass" % (count * t.timeit(number=count)/count))
t = timeit.Timer(stmt=s2,setup='a=123')
print (s2, "%.2f usec/pass" % (count * t.timeit(number=count)/count))
t = timeit.Timer(stmt=s3,setup='a=123')
print (s3, "%.2f usec/pass" % (count * t.timeit(number=count)/count))
t = timeit.Timer(stmt=s4,setup='a=123')
print (s4, "%.2f usec/pass" % (count * t.timeit(number=count)/count))

The results of running this bit of code on my Samsung NC10 netbook give me the following output:

123 == None 0.29 usec/pass
a == None 0.31 usec/pass
123 is None 0.20 usec/pass
a is None 0.21 usec/pass

This is a significant difference and although it doesn't seem to amount to much, this might shave several seconds of your algorithm when you compare for example the results of a large database query, as databases tend to represent NULL values as None.