A small HTTP server in Python

The Python http.server module can be run from the commandline with

python3 -m http.server

to provide a simple http server that listens on a configurable port and basically serves any file from the directory it is started from.

That is a good start but I had some additional requirements to serve up files from my raspberry Pi:

• The server must be able to run as a demonized process, i.e, in the background
• It must server a limited set of files
• from a configurable directory and
• be able to run as a less privileged user

Now of course I could have opted for a light weight http server like nginx or lighttpd, but I decided to write my own so I wouldn't have to install a boatload of additional packages.

restrictedhttpserver.py

The server is based on the daemon module a wrote about in a previous article and can be started from the commandline in much the same way as http.server:


python3 restrictedhttpserver.py


It will run as the user that started it, will listen on port 8000 and start serving any file from the current directory. It will demonize itself as well.

It takes some additional options:

python3 restrictedhttpserver.py -h

usage: restrictedhttpserver.py [-h] [-p PID_FILE] [-l LOG_FILE] [-r ROOT_DIR]
                               [-n NAME] [-u USER] [-f] [-s] [-d]
                               [--bind ADDRESS] [-x] [-e EXT]
                               [port]

Example HTTP Server

positional arguments:
  port                  Specify alternate port [default: 8000]

optional arguments:
  -h, --help            show this help message and exit
  -p PID_FILE, --pid-file PID_FILE
  -l LOG_FILE, --log-file LOG_FILE
  -r ROOT_DIR, --root-dir ROOT_DIR
  -n NAME, --name NAME  name of the server used in log lines
  -u USER, --user USER  drop privileges of running server to those of user
  -f, --force           start a server even if pid file is present already
  -s, --stop            stop a running server
  -d, --debug           Run http server in foreground mode
  --bind ADDRESS, -b ADDRESS
                        Specify alternate bind address [default: all
                        interfaces]
  -x, --nodirlist       never show a directory listing
  -e EXT, --ext EXT     allowed file extensions (without .) May occur more
                        than once

For example if you wanted to run it as user www from the directory /public/www and just server html and css files, you could run it (as root) as follows:

python3 restrictedhttpserver.py -r /public/www -u www -e html -e css

Later you could stop this process by executing
python3 retrictedhttpserver.py -s


For more information have a look at the source code, it is pretty readable I think :-)

Availability

restrictedhttpserver.py and daemon.py are both available from this GitHub repository.

Demonizing a Python process

Properly demonizing a Python process is not easy, even on *nix, and although various sources of good material can be found on the internet (for example here and here) it didn't quite match my requirements and/or I didn't understand it completely.
I therefore decided to create both a daemon module and a http server to test it.

Code availability

Both modules are available on GitHub
The daemon module is called daemon.py and the httpserver is called restrictedhttpserver.py. Both have a fair amount of comments in their source code that are the result of me trying to fully understand what is required to get things working.

A small word of warning here: I only tested it with Python3 on Ubuntu and Raspbian and I do not expect it to work on any other operating system (although most *nix like systems probably will work) and in my world Python2 is no longer a valid option.

Also, the Daemon class makes use of an undocumented attribute in the logging.FileHandler class  (the stream.fileno attribute) which is necessary in order not to close this stream when forking a child process. Theoretically this might break in the future but I did want all logging to happen outside the chroot jail so that the process running as a daemon can log but does not have access to the logging. Whether this is a valid approach remains to be seen as this also prevents proper log rotation.

The Daemon class

The basic requirement are all implemented, an instance of a Daemon class can:

run in a chrooted jail

so it cannot access files outside a given directory

run with lowered privileges

so the risk of accessing files not owned by a specific user is lowered

can have a umask of your choice

which will ensure that newly created files by the daemon are not open to all

log outside its jail

so that we can log events without offering the process managed by daemon any way of altering the logging

and maintains a configurable pid file

so that we can prevent the daemon from starting more than once and providing a way the fond out the process id so that we can terminate the daemon

We also made the Daemon class follow the Singleton pattern, i.e. there can only be one instanced object of the class. This is sensible because a process can only be daemonized once. We also implemented to required methods to facilitate use as a context manager. A typical, minimal code snippet would look something like:

from daemon import Daemon

dm = Daemon()
with dm:
    ... do something forever ...

All configurable options have sensible defaults, but a call could look like this:

from daemon import Daemon

dm = Daemon(user='httpserver', rootdir='/var/www', umask=0o27,
             pidfile='/var/lockhttpserver.pid',
             logfile='/var/log/httpserver.log',
             name="Http Server")
with dm:
    ... do something forever ...

It is important to understand that when the Daemon object is created, the process is not immediately daemonized. This is handled by the context manager (or you could call the daemonize() method directly). There is also a stop() method provided that will check for a running daemon process and terminate it. This could look like this:

from daemon import Daemon
import argparse

parser = argparse.ArgumentParser(description="Example Daemon")
parser.add_argument('-s', '--stop', action='store_true', help='stop a running server')
args = parser.parse_args()

dm = Daemon()
if args.stop:
    dm.stop()
else:
    with dm:
        ... do something forever ...

More code

In a future article I will highlight the small http server that I implemented to test the daemon module.

A pure Python kd-tree implementation

kd-trees are an efficient way to store data that is associated with a location in any number of dimensions up to twenty or so. Specifically, kd-trees allow for nearest neighbor searches in O(log n) time, something I desperately needed for my Blender tree generation add-on. In this article I highlight some of the design decisions that that shaped my pure Python implementation of a kd-tree module.

Visiting my own post five years later a lot has changed. We are now quite a few versions of Blender ahead of what was available in 2013 and a kd-tree implementation is now part of Blender's Python API.
I highly recommend using that implementation because is written in C and probably quite a bit faster than my pure Python implementation. There is also a bvh-tree implementation that is suited for problems that not such much involve points but geometry that fills space, like triangles.

kd-trees

There exist quite a few implementations in C or C++ that are part of robust and well tested libraries but for my Blender add-on I needed a platform neutral (= pure python) implementation that I could ship with the add-on. There are a few Python implementations but because I could not determine how well tested they were or because they did not quite meet my requirements I decided to develop an implementation myself.

Requirements

My requirements might not be everybody's requirements so besides supporting adding nodes and providing a nearest neighbor search what were the issues that were important to me?

Comes with test-suite
It might seem strange to make this the first requirement but because it's a part that is so essential to my add-on and because the code is quite tricky, I opted for test driven development. Adding unit tests to a python module is quite simple and although I can't claim that the tests cover all edge cases, they are nevertheless quite comprehensive. There is also a small performance test included.
Works with any iterable as position
A node in a kd-tree consists besides other things primarily of a position attribute. For use in Blender this typically would be a Vector instance but in other situations you might want to use other vector implementations. This kd-tree implementation assumes very little about its position attributes, just that it is subscriptable and that it supports a .dot() member function that returns the sum of the pairwise multiplication of its items (in other words, the dot product which is equal to the distance squared). Individual items should be floats (or anything that supports substraction, addition and comparisons). Probably any vector implementation will meet these requirements. The test-suite implements its own, based on Python's list class.
Allows easy deletion
Deleting nodes from a kd-tree is tricky and costly because of the rebalancing that has to be performed. I therefor opted for a much simpler solution: Instead of deleting a node we set its data attribute to None and add an option to the .nearest() member function to ignore nodes without data. Of course this won't make the tree any smaller but even when we delete a third of the nodes in random fashion (a typical use case for my tree add-on) the performance impact on nearest neighbor searches is minimal.

Availability

The module is licensed as GPL and available on Github as part of the spacetree add-on for Blender. It is contained in a single source file that can be downloaded separately (click Raw to download).

Blender 2.6 add-on tutorial

I finished a Blender add-on tutorial the other day that might be an interesting read. And actually I liked the way my cover image turned out, so I reproduce it here :-)

Web2py, a better alternative to CherryPy?

during a short holiday break I did some reading on Web2py and my first impression is that that it looks really good. It seems to be much more complete and data centric than CherryPy. I think I will spend some time experimenting with it the coming months.

Marching Tetrahedrons in Python

In this atricle we show a simple implementation of the Marching Tetrahedrons algorithm in Python.

Marching Tetrahedrons

The listing povided below is a straightforward reimplementation in Python of the ideas and code presented by Paul Bourke. The image

shows the result of sampling a simple lobed function (rendered in Blender). I might implement the code directly in Blender but for now we export to an STL file that can be read by almost any 3D package. Note that the exported triangles do not have normals that point in a uniform direction, in fact we do no export any normals at all. You have to recalculate the normals in your 3D package before you can render the resulte with smooth shading.

class Vector: # struct XYZ
 def __init__(self,x,y,z):
  self.x=x
  self.y=y
  self.z=z
 
 def __str__(self):
  return str(self.x)+" "+str(self.y)+" "+str(self.z)
  
class Gridcell: # struct GRIDCELL
 def __init__(self,p,n,val):
  self.p   = p   # p=[8]
  self.n   = n   # n=[8]
  self.val = val # val=[8]

class Triangle: # struct TRIANGLE
 def __init__(self,p1,p2,p3):
  self.p = [p1, p2, p3] # vertices

# return triangle as an ascii STL facet  
 def __str__(self):
  return """facet normal 0 0 0
outer loop
vertex %s
vertex %s
vertex %s
endloop
endfacet"""%(self.p[0],self.p[1],self.p[2])

# return a 3d list of values
def readdata(f=lambda x,y,z:x*x+y*y+z*z,size=5.0,steps=11):
 m=int(steps/2)
 ki = []
 for i in range(steps):
  kj = []
  for j in range(steps):
   kd=[]
   for k in range(steps):
    kd.append(f(size*(i-m)/m,size*(j-m)/m,size*(k-m)/m))
   kj.append(kd)
  ki.append(kj)
 return ki

from math import cos,exp,atan2

def lobes(x,y,z):
 try:
  theta = atan2(x,y)         # sin t = o 
 except:
  theta = 0
 try:
  phi = atan2(z,y)
 except:
  phi = 0
 r = x*x+y*y+z*z
 ct=cos(theta)
 cp=cos(phi)
 return ct*ct*cp*cp*exp(-r/10)
 
def main():

 data = readdata(lobes,5,41)
 isolevel = 0.1

 #print(data)
 
 triangles=[]
 for i in range(len(data)-1):
  for j in range(len(data[i])-1):
   for k in range(len(data[i][j])-1):
    p=[None]*8
    val=[None]*8
    #print(i,j,k)
    p[0]=Vector(i,j,k)
    val[0] = data[i][j][k]
    p[1]=Vector(i+1,j,k)
    val[1] = data[i+1][j][k]
    p[2]=Vector(i+1,j+1,k)
    val[2] = data[i+1][j+1][k]
    p[3]=Vector(i,j+1,k)
    val[3] = data[i][j+1][k]
    p[4]=Vector(i,j,k+1)
    val[4] = data[i][j][k+1]
    p[5]=Vector(i+1,j,k+1)
    val[5] = data[i+1][j][k+1]
    p[6]=Vector(i+1,j+1,k+1)
    val[6] = data[i+1][j+1][k+1]
    p[7]=Vector(i,j+1,k+1)
    val[7] = data[i][j+1][k+1]
   
    grid=Gridcell(p,[],val)
    triangles.extend(PolygoniseTri(grid,isolevel,0,2,3,7))
    triangles.extend(PolygoniseTri(grid,isolevel,0,2,6,7))
    triangles.extend(PolygoniseTri(grid,isolevel,0,4,6,7))
    triangles.extend(PolygoniseTri(grid,isolevel,0,6,1,2))
    triangles.extend(PolygoniseTri(grid,isolevel,0,6,1,4))
    triangles.extend(PolygoniseTri(grid,isolevel,5,6,1,4))
    
 export_triangles(triangles)

def export_triangles(triangles): # stl format
 print("solid points")
 for tri in triangles:
  print(tri)
 print("endsolid points")
 
def t000F(g, iso, v0, v1, v2, v3):
 return []

def t0E01(g, iso, v0, v1, v2, v3):
 return [Triangle(
 VertexInterp(iso,g.p[v0],g.p[v1],g.val[v0],g.val[v1]),
 VertexInterp(iso,g.p[v0],g.p[v2],g.val[v0],g.val[v2]),
 VertexInterp(iso,g.p[v0],g.p[v3],g.val[v0],g.val[v3]))
 ]

def t0D02(g, iso, v0, v1, v2, v3):
 return [Triangle(
 VertexInterp(iso,g.p[v1],g.p[v0],g.val[v1],g.val[v0]),
 VertexInterp(iso,g.p[v1],g.p[v3],g.val[v1],g.val[v3]),
 VertexInterp(iso,g.p[v1],g.p[v2],g.val[v1],g.val[v2]))
 ]

def t0C03(g, iso, v0, v1, v2, v3):
 tri=Triangle(
 VertexInterp(iso,g.p[v0],g.p[v3],g.val[v0],g.val[v3]),
 VertexInterp(iso,g.p[v0],g.p[v2],g.val[v0],g.val[v2]),
 VertexInterp(iso,g.p[v1],g.p[v3],g.val[v1],g.val[v3]))
 return [tri,Triangle(
 tri.p[2],
 VertexInterp(iso,g.p[v1],g.p[v2],g.val[v1],g.val[v2]),
 tri.p[1])
 ]

def t0B04(g, iso, v0, v1, v2, v3):
 return [Triangle(
 VertexInterp(iso,g.p[v2],g.p[v0],g.val[v2],g.val[v0]),
 VertexInterp(iso,g.p[v2],g.p[v1],g.val[v2],g.val[v1]),
 VertexInterp(iso,g.p[v2],g.p[v3],g.val[v2],g.val[v3]))
 ]

def t0A05(g, iso, v0, v1, v2, v3):
 tri = Triangle(
 VertexInterp(iso,g.p[v0],g.p[v1],g.val[v0],g.val[v1]),
 VertexInterp(iso,g.p[v2],g.p[v3],g.val[v2],g.val[v3]),
 VertexInterp(iso,g.p[v0],g.p[v3],g.val[v0],g.val[v3]))
 return [tri,Triangle(
 tri.p[0],
 VertexInterp(iso,g.p[v1],g.p[v2],g.val[v1],g.val[v2]),
 tri.p[1])
 ]

def t0906(g, iso, v0, v1, v2, v3):
 tri=Triangle(
 VertexInterp(iso,g.p[v0],g.p[v1],g.val[v0],g.val[v1]),
 VertexInterp(iso,g.p[v1],g.p[v3],g.val[v1],g.val[v3]),
 VertexInterp(iso,g.p[v2],g.p[v3],g.val[v2],g.val[v3]))
 return [tri,
 Triangle(
 tri.p[0],
 VertexInterp(iso,g.p[v0],g.p[v2],g.val[v0],g.val[v2]),
 tri.p[2])
 ]

def t0708(g, iso, v0, v1, v2, v3):
 return [Triangle(
 VertexInterp(iso,g.p[v3],g.p[v0],g.val[v3],g.val[v0]),
 VertexInterp(iso,g.p[v3],g.p[v2],g.val[v3],g.val[v2]),
 VertexInterp(iso,g.p[v3],g.p[v1],g.val[v3],g.val[v1]))
 ]

trianglefs = {7:t0708,8:t0708,9:t0906,6:t0906,10:t0A05,5:t0A05,11:t0B04,4:t0B04,12:t0C03,3:t0C03,13:t0D02,2:t0D02,14:t0E01,1:t0E01,0:t000F,15:t000F}

def PolygoniseTri(g, iso, v0, v1, v2, v3):
 triangles = []

 #   Determine which of the 16 cases we have given which vertices
 #   are above or below the isosurface

 triindex = 0;
 if g.val[v0] < iso: triindex |= 1
 if g.val[v1] < iso: triindex |= 2
 if g.val[v2] < iso: triindex |= 4
 if g.val[v3] < iso: triindex |= 8

 return trianglefs[triindex](g, iso, v0, v1, v2, v3)

def VertexInterp(isolevel,p1,p2,valp1,valp2):
 if abs(isolevel-valp1) < 0.00001 :
  return(p1);
 if abs(isolevel-valp2) < 0.00001 :
  return(p2);
 if abs(valp1-valp2) < 0.00001 :
  return(p1);
 mu = (isolevel - valp1) / (valp2 - valp1)
 return Vector(p1.x + mu * (p2.x - p1.x), p1.y + mu * (p2.y - p1.y), p1.z + mu * (p2.z - p1.z))

if __name__ == "__main__":
 main()
 

Implementing a HTTPS server in Python

Web applications often transfer sensitive data between client and server. Even a session id is not something that should be vulnerable to eavesdropping. It is therefore a very good idea to encrypt all communication and implement a HTTPS server.

Subclassing HTTPServer

Python's ssl module has been cleaned up quite a bit since version 3.x and with a little help from this recipe it was incredibly simple to adapt the HTTPServer class from the http.server module to accept only secure connections:

import ssl
import socket
from socketserver import BaseServer
from http.server import HTTPServer
  
class HTTPSServer(HTTPServer):
 def __init__(self,address,handler):
  BaseServer.__init__(self,address,handler)
  
  self.socket = ssl.SSLSocket(
   sock=socket.socket(self.address_family,self.socket_type),
   ssl_version=ssl.PROTOCOL_TLSv1,
   certfile='test.pem',
   server_side=True)
  self.server_bind()
  self.server_activate()

All we do basically is change the initialization code to create a secure socket instead of a regular one (in line 10). The things to watch out for is the ssl_version: older versions are considered unsafe so we use TLS 1.0 here. Also the certificate file we use here contains both our certificate and our private key. If you want to use a self signed certificate for testing purposes you could generate one with openssl (most UNIX-like operating systems offer binary packages, for a precompiled package for windows check the faq.)

openssl req -new -x509 -keyout test.pem -out test.pem -days 365 -nodes

Note that your browser will still complain about this certificate because it is self signed.

Erica web application framework

I started developing and writing about a new Python web application framework in Python to aid people in understanding concepts in web application development.

Erica web application framework

The framework is named after one of our cutest pygmy goats, Erica, which is of course completely irrelevant :-). More important is that the first article outlining the goals, is already on-line on my website. The articles will build and extend on blog entries made in this blog. The idea is that the website will develop into a small but comprehensive tutorial on implementing a framework while this blog will be the place to watch for updates and place your comments.

Python regular expressions and the functools module

If we use annotations to sanitize incoming data easy access to regular expressions is a must. In this article I'll show how to make the most of Python's bundles functools module to dynamically generate functions that match regular expressions while reducing the overhead of compiling regular expressions as much as possible

Matching against regular expressions

In our framework we annotate function parameters with functions that will pass through or convert incoming arguments if they are ok or raise an exception other wise. For example, if we would want to make sure an argument is an integer, we'd annotate it with Python's built-in int function:

def myfunction(arg:int):
	...

The framework will use this function to check the argument before actually calling the function.

In many situations it would be convenient if we could specify a regular expression that arguments should match. We could extend the framework to check whether the annotation was a string and use that string as a regular expression but it is better to opt for a approach that will allow for the easy definition of categories. Some examples: def myfunc(arg:digits) ... or def myfunc(arg:zipcode) .... In these examples digits and zipcode are functions that take single argument and return that argument if matches a certain pattern or raise an exception if it doesn't.

To construct such functions easily we may utilize Python's re and functoolsmodules:

from re import compile

def regex_compile(pattern):
	return compile("^"+pattern+"$")

def match(pattern,string):
	re = regex_compile(pattern)
	if re.match(string) is None:
		raise ValueError("no pattern match")
	return string

To check whether a string matches a regular expression we define a match function (line 6) that will compile the regular expression and match the string against it. The regex_compile function will make sure the match is anchored at the beginning and the end because we want a complete match, that is string that merely contain the pattern are considered invalid. We don't allow extraneous characters.

Constructing partial functions

So far nothing new under the sun but we still need a simple way to construct functions that match a single argument against a regular expression. Enter the partial function. It will take a function and some arguments and use those to construct a new function that will pass any arguments together with the original arguments to the encapsulated function. This is exactly what we need to construct our categories:

from functools import partial

digits = partial(match,r'\d+')
zipcode = partial(match,r'\d{4}\s*[a-zA-Z]{2}')

print(digits('12345'))
print(zipcode('1234AB'))

Improving efficiency by Memoization

Each time we call digits(a) we actually call match(r'\d+',a) and this will result in compile the regular expression again and again. Compiling regular expressions is a rather expensive operation so we might want to avoid that by reusing compiled expressions. This can simply be accomplished by applying the lru_cache decorator from the functools module to implement a memoize pattern:

from functools import lru_cache
from re import compile

@lru_cache(maxsize=0)
def regex_compile(pattern):
	return compile("^"+pattern+"$")

This simple change will make sure we will reuse compiled regular expressions most of the time. The maxsize parameter should be set as needed: it may be set to zero in which case all regular expression will be cached, possibly a good choice as it is unlikely that your web application sports thousands of regular expressions.

Propagating Python function annotations through decorators

When we decorate a Python function we must take some care to ensure that the decorated function can present itself in the same way as before its decoration. Let's explore what is needed.

Decorating a Python function

Say we have an undecorated function f(). It features a docstring (line 2) and annotated arguments:

def f(a:str):
	"an undecorated function"
	return a

print(f.__name__,f.__annotations__,f.__doc__,f('foo'))

The final line in the previous snippet prints the name of the function, its annotation dictionary and the docstring. The output looks like this:

f {'a': <class 'str'>} an undecorated function foo

Decorating a Python function

Now say we have a function do_something() and we would like to have a decorator that would arrange that a function would call do_something() first. We might set this up like this:

def do_something():
	print('oink')
	
def decorator(f):
	def g(*args,**kwargs):
		do_something()
		return f(*args,**kwargs)
	return g
	
@decorator
def f(a:str):
	"a naively decorated function"
	return a

print(f.__name__,f.__annotations__,f.__doc__,f('foo'))

The final line would produce the following output:

oink
g {} None foo

It is clear that do_something() is executed as it does print oink but although our decorated function can be called as f(), its name, annototations and doctring are different.

Preserving attributes when decorating a Python function

To resolve these matters we can rewrite our decorator as follows:

def decorator(f):
	def g(*args,**kwargs):
		do_something()
		return f(*args,**kwargs)
	g.__name__ = f.__name__
	g.__doc__ = f.__doc__
	g.__annotations__ = f.__annotations__
	return g
	
@decorator
def f(a:str):
	"a less naively decorated function"
	return a

print(f.__name__,f.__annotations__,f.__doc__,f('foo'))

We now simply copy the relevant attributes to the newly created function and if we now examine the output produced by the final line we get what we expect:

oink
f {'a': <class 'str'>} a less naively decorated function foo

Finding stuck or hot pixels in Nikon D80 images, a multiprocessing approach

Earlier I described a method to find hot or stuck pixels by determining the variance of (sub)pixels over a large set of photos. In this article we take a look at a way to farm out this work to multiple processes.

The parallel algorithm for calculating variance

The parallel algorithm allows us to combine calculated variances from separate sets. With the help of Python's multiprocessing module it is straight forward to implement our hot pixel finding algorithm to one that takes advantage of multiple cores:

import Image
import numpy as np
import sys
from glob import glob
from multiprocessing import Pool,current_process
from time import time

def process_pixels(shape,*args):
	first = True
	for filename in args:
		pic = Image.open(filename)
		if pix.shape != shape:
			print("shapes don't match")
			continue
		if first:
			first = False
			firstpix = pix
			n = np.zeros(firstpix.shape)
			mean = np.zeros(firstpix.shape)
			M2 = np.zeros(firstpix.shape)
			delta = np.zeros(firstpix.shape)
		n += 1
		delta = pix - mean
		mean += delta/n
		M2 += delta*(pix - mean)
	return M2
	
if __name__ == '__main__':
	global pool

	n=int(sys.argv[1])
	pool = Pool(n)
	filenames = []
	for a in sys.argv[2:]:
		filenames.extend(glob(a))
	
	shape = np.array(Image.open(filenames[0])).shape

	s=time()
	results = []
	for i in range(n):
	results.append(pool.apply_async(process_pixels,tuple([shape]+filenames[i::n])))
	for i in range(n):
		results[i]=results[i].get()
	M2 = sum(results)
	print(time()-s)
	
	mini = np.unravel_index(M2.argmin(),M2.shape)
	maxi = np.unravel_index(M2.argmax(),M2.shape)
	print('min',M2[mini],mini)
	print('max',M2[maxi],maxi)
	print('mean',np.mean(M2))

	sorti = M2.argsort(axis=None)
	print(sep="\n",*[(i,M2[i]) for i in [np.unravel_index(i,M2.shape) for i in sorti[:10]]])

	print(time()-s)

Note that because we return complete arrays (line 26) we gain almost nothing for small data sets because of the overhead of shuttling these large arrays (> 30 MByte) between processes. This is illustrated in the following graph with shows the elapsed time as a function of the number of processes, both for a small number of pictures (50) and a large number (480).

Some other notable issues: in the code we do not actually implement the parallel algorithm but we simple add together the variances. Because we're looking for a minimum variance we gain nothing by adding a constant value.

Memory usage is another thing to be aware of (and the reason that there is no entry for six cores in the graph. The algorithm we have implemented uses 5 arrays (the pixel data itself included). That makes for 10 megapixel X 3 colors X 5 arrays X 8 bytes data (because we use 64 bit floats by default) which makes for a whopping 1.2 Gigabyte of data per process or more than 6 Gig with 5 processes. With some other applications open a sixth process wouldn't fit on my test machine. Because we're adding pixel values in the range 0 - 255 we could probably gain a lot by using 32 bit floats or even 16 bit floats here.

Windows 7 Gadgets: mini web applications

Windows 7 gadgets are small applications that are integrated in the Windows desktop. These gadgets are basically webpages displayed in a browser environment that hardly differs from a regular environment. It is therefore entirely possible to turn a gadget into a web application that retrieves data from a remote server with AJAX. In this article we explore some of the possibilities and see how we may use jQuery inside a gadget.

Anatomy of a Windows 7 gadget

There is actually some pretty decent documentation on gadgets available on the web (for example on msdn or here). The trick is to get an example gadget running with as little ballast as possible.

A gadget file is basically a zip file that contains a file gadget.html plus a number of additional files, the most important one, gadget.xml. This zip file does have a .gadget extension instead of a .zip extension. So these simple steps are needed to create a bare bones gadget:

1. Create a directory with a descriptive name, e.g. mygadget
2. In this directory create a file gadget.html
3. a subdirectory named css with a file mygadget.css
4. a file gadget.xml
5. and finally a file icon.png
6. pack the contents of this directory into a file mygadget.zip (i.e. not the toplevel directory itself)
7. rename this file to mygadget.gadget (although 7zip for example can pack to a file with any extension directly)
8. double click this file and follow through the install dialog

With the following gadget.html the result will look like the screenshot

<html xmlns="http://www.w3.org/1999/xhtml">
<head>	
	<title>My Gadget</title>
	<meta http-equiv="Content-Type" content="text/html; charset=unicode" />
	<link href="css/mygadget.css" rel="stylesheet" type="text/css" /> 
</head>
<body> 
<div id="main_image">
<p>42</p>
</div>
</body>
</html>

Not really impressive, I admit, but it shows how simple it is to create a gadget. The necessary gadget.xml mainly describes were to find the actual html code and what image to use in the gadget selector:

<?xml version="1.0" encoding="utf-8" ?>
<gadget>
  <name>Mygadget</name>
  <namespace><!--_locComment_text="{Locked}"-->StartSmall.Gadgets</namespace>
    <version><!--_locComment_text="{Locked}"-->1.0</version>
  <author name="Michel Anders">
    <info url="http://michelanders.blogspot.com" text="Start Small" />
    <logo src="icon.png" />
  </author>
  <copyright><!--_locComment_text="{Locked}"-->&#169; 2011</copyright>
  <description>Basic Gadget</description>
  <icons>
    <icon height="48" width="48" src="icon.png" />
  </icons>
  <hosts>
    <host name="sidebar">
      <base type="HTML" apiVersion="1.0.0" src="gadget.html" />
      <permissions>
        <!--_locComment_text="{Locked}"-->Full
      </permissions>
      <platform minPlatformVersion="1.0" />
    </host>
  </hosts>
</gadget>

Using jQuery in a Windows 7 gadget

Displaying static information is not much fun at all so let's see what options there are to create a more dynamic gadget:

• Refer to external content like images that get refreshed
• Refresh the content ourselves using JavaScript, possibly even interacting with the user

The first option is simple enough and we won't cover that here. The second option is more interesting because it opens up a whole array of possibilities. Let's have a look at how a gadget.html page should look to incorporate jQuery and how we can test if this works:

<html xmlns="http://www.w3.org/1999/xhtml">
<head>	
	<title>My Gadget</title>
	<meta http-equiv="Content-Type" content="text/html; charset=unicode" />
	<link href="css/mygadget.css" rel="stylesheet" type="text/css" />
	<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js" type="text/javascript"></script>
</head>
<body> 
<div id="main_image">
<p>42</p>
</div>
<script type="text/javascript">$("#main_image p").append('<span> 43 44 45 </span>');</script>
</body>
</html>

As you can see this is surprisingly simple. The screenshot proves that the final lines of code are actually executed and change the contents of our basic gadget: Our next step is to check if we can retrieve data from a server.

JSONP in a Windows 7 gadget

Due to the same origin policy it is not entirely straight-forward to retrieve data from a server different from the server we get our webpage from. In the gadget environment every server is considered a different server because the gadget.html file originates from a file system. This means that even if we access a web application server on the same pc, we will be denied access.

Fortunately there is a workaround available in the form of JSONP and jQuery makes it very simple for use to use this. Consider the following gadget.html:

<html xmlns="http://www.w3.org/1999/xhtml">
<head>	
	<title>My Gadget</title>
	<meta http-equiv="Content-Type" content="text/html; charset=unicode" />
	<link href="css/mygadget.css" rel="stylesheet" type="text/css" />
	<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js" type="text/javascript"></script>
</head>
<body> 
<div id="main_image">
<p>42</p>
</div>
<script type="text/javascript">
function Refresh() {
    $.ajax('http://127.0.0.1:8088/value',
		{dataType:'jsonp',scriptCharset: "utf-8",
		success:function(data, textStatus, jqXHR){
			$("#main_image").empty().append('<p>'+String(data)+'</p>');
		}});
}
window.setInterval("Refresh()", 5000);
</script>
</body>
</html>

It will contact an application server every 5 seconds to retrieve a number and will insert this number into the content div. That's all there is to it. The application server that serves these request is equally simple and can be build for example with the applicationserver module from a previous article:

from http.server import HTTPServer, BaseHTTPRequestHandler
from json import dumps
from applicationserver import IsExposed,ApplicationRequestHandler

class Application:

	def value(self,callback:str,_:int=0) -> IsExposed:
		r = "%s(%s);"%(callback,dumps(_ % 17)) 
		return r
		
class MyAppHandler(ApplicationRequestHandler):
	application=Application()
	
appserver = HTTPServer(('',8088),MyAppHandler)
appserver.serve_forever()

The only trick here is that JSONP requests pass an additional paramter (usually called callback that will hold a string (the name of a JavaScript function in the client) and that we have to use this parameter to contruct a return value that looks like a call to this function with the JSON encoded data as its argument (line 8). The _ parameter holds a random number added by JQuery to prevent caching by the browser. So if _ happened to be 123 and callback would be jquery456_123 the value we woudl return would be the string jquery456_123(4);

The data we return here is merely a random number but of course this could be anything and we could style it to be presented in a more readable form.

Correcting stuck or hot pixels in Nikon D80 images

A friend of mine became suddenly aware that her Nikon D80 camera had developed a so called stuck or hot pixel. Repairing a CCD chip in a camera is quite costly and what is more, looking back through the old photos she discovered the defect had been present for over a year. In this article we develop a tiny Python program that masks out a pixel in pretty much the same way that the built-in firmware in de camera does this for pixels that were hot or dead when the camera was constructed.

Correcting stuck or hot pixels in images

Hot or stuck pixels in a camera are annoying and other than replacing the ccd chip, nothing can be done about it with regard to the hardware. More on this in this illuminating article. Sometimes the camera can map out bad pixels but that won't fix old images, so the plan is to create a small program that does this on existing images.

The idea is to replace the offending pixel with a weighted average of the neighboring pixels so it will blend in with the surroundings. Diagonally adjacent pixels have a lower weight then pixels horizontally or vertically adjacent (see picture on the left). Such a program is simple enough to implement using PIL, the Python Imaging Library. There is even a version for Python 3 made available by Christoph Gohlke, although you will have to replace all relative import statements with absolute one if you want that to work. (Simply run the the installer, open all *.py files in site-packages/PIL and replace all occurrences of from . import with a simple import. A decent editor like notepad++ or ultraedit can manage that in one go).

The program will take the name of an image file and the x,y position of the pixel to fix as its arguments, e.g. python correctpixel.py myimage.jpg 1500,1271 and will produce a corrected image with the same name but with a c_ prefix added, in our example c_myimage.jpg. It will use a weighted average of the eight neighboring pixels as depicted in this image (the diagonally adjacent pixels have a weight of 1/sqrt(2).

The simplest way to find the exact location of the pixel to fix is to open the image in Gimp, locate the pixel and hover over it with the mouse: the location is shown in the status bar at the bottom. Depending on your zoom level these may be fractional numbers, but we need just the integral parts.

The program itself is rather straightforward and is shown in its entirety below:

import sys
import Image

def correct(im,xy,matrix):
 pim = im.load()
 x,y = xy
 maxx,maxy = im.size
 n = 0
 sumr,sumg,sumb = 0,0,0
 for dx,dy,w in matrix:
  px,py = x+dx,y+dy
  if px<0 or py<0 or px >= maxx or py >= maxy:
   break
  n += w
  r,g,b = pim[px,py]
  sumr,sumg,sumb = sumr+r*w,sumg+g*w,sumb+b*w
 pim[x,y]=(int(sumr/n),int(sumg/n),int(sumb/n))

w=1/(2**0.5)
matrixd=((0,1,1),(0,-1,1),(1,0,1),(-1,0,1),(-1,-1,w),(-1,1,w),(1,1,w),(1,-1,w))

im = Image.open(sys.argv[1])
xy = tuple(map(int,sys.argv[2].split(',')))
correct(im,xy,matrixd)
im.save('c_'+sys.argv[1],quality=97)

Given an opened image the function correct() will load the data and the loop of the list of pixels offsets and weights given in its matrix argument. It will check whether a neighbor is within the bounds of the image (line 12; if the stuck pixel is on an edge it might not be) and if so, sum its RGB components according to the weight of this pixel. The pixel is finally replaced by summed values divided by the number of summed pixels. We also take care to convert the RGB components to integers again (line 17Y).

The final lines take care of opening the image (line 22), and converting the pixel position argument to a tuple of integers before calling the correct() function. The image is saved again with a name prefixed with c_. The quality parameter is set to a very high value to more or less save the image with the same jpeg quality a the original image.

Note that at the moment the code shown works for jpeg images only (or to be precise, only for images with three bands, R,G,B.) Png images may have and extra transparency band and the code does not deal with that nor does it play nice with indexed formats (like gif).

Function annotations in Python, checking parameters in a web application server , part II

In a previous article I illustrated how we could put Python's function annotations to good use to provide use with a syntactically elegant way to describe which kind of parameter content would be acceptable to our simple web application framework. In this article the actual implementation is presented along with some notes on its usage.

A simple Python application server with full argument checking

The applicationserver module provides a ApplicationRequestHandler class that derives from the Python provided BaseHTTPRequestHandler class. For now all we do is provide a do_GET() method, i.e. we don't bother with HTTP POST methods yet, although it wouldn't take too much effort if we would.

The core of the code is presented below:

class ApplicationRequestHandler(BaseHTTPRequestHandler):
 
 def do_GET(self):
  url=urlsplit(self.path)
  path=url.path.strip('/')
  parts=path.split('/')
  if parts[0]=='':
   parts=['index']
   
  ob=self.application
  try:
   for p in parts:
    if hasattr(ob,p):
     ob=getattr(ob,p)
    else:
     raise AttributeError('unknown path '+ url.path)
  except AttributeError as e:
   self.send_error(404,str(e))
   return
   
  if ('return' in ob.__annotations__ and 
   issubclass(ob.__annotations__['return'],IsExposed)):
   try:
    kwargs=self.parse_args(url.query,ob.__annotations__)
   except Exception as e:
    self.send_error(400,str(e))
    return
    
   try:
    result=ob(**kwargs)
   except Exception as e:
    self.send_error(500,str(e))
    return
  else:
   self.send_error(404,'path not exposed'+ url.path)
   return
   
  self.wfile.write(result.encode())

 @staticmethod
 def parse_args(query,annotations):
  kwargs=defaultdict(list)
  if query != '':
   for p in query.split('&'):
    (name,value)=p.split('=',1)
    if not name in annotations:
     raise KeyError(name+' not annotated')
    else:
     kwargs[name].append(annotations[name](value))
   for k in kwargs:
    if len(kwargs[k])==1:
     kwargs[k]=kwargs[k][0]
  return kwargs
  

The first thing we do in the do_GET() method is splitting the path into separate components. The path is provided in the path member and is already stripped of hostname and query parameters. We strip from it any leading or trailing slashes as well (line 5). If there are no path components we will look for a method called index.

The next step is to check each path component and see if its a member of the application we registered with the handler. If it is, we retrieve it and check whether the next component is a member of this new object. If any of these path components is missing we raise an error (line 16).

If all parts of the path can be resolved as members, we check whether the final path points to an executable with a return allocation. If this return allocation is defined and equal to our IsExposed class (line 22) we are willing to execute it, otherwise we raise an error.

The next step is to check each argument, so we pass the query part of the URL to the static parse_args() method that will return us a dictionary of values if all values checked out ok. If so we call the method that we found earlier with these arguments and if all went well, write its result to the output stream that will deliver the content to the client.

The parse_args() method is not very complicated: It creates a default dictionary whose default will be an empty list. This way we can create a list of values if the query consists of more than one part with the same name. Then we split the query on the & character (line 44) and split each part in a name and a value part (these are separated by a = character). Next we check if the name is present in the annotations dictionary and if not raise a KeyError.

If we did find the name in the annotations dictionary its associated value should be an executable that we pass the value from the query part (line 48). The result of this check (or conversion) is appended to the list in the default dictionary. If the value in the annotation is not an executable an exception will be raised that will not be caught. Likewise will any exception within this callable bubble up to the calling code. The final lines (line 50-53) reduce those entries in the default dictionary that consist of a list with just a single item to just that item before returning.

Conclusion

Python function annotations can be used for many purposes and this example code show a rather elegant (I think) example that let's us specify in clear language what we expect of functions that are part of web applications, which makes it quite easy too catch many input errors in an way that can be adapted to almost any input pattern.

Function annotations in Python, checking parameters in a web application server

Parameter annotations in function definitions are a recent addition to the language. In this article we show how this feature can be put to good use when we build a simple web application server that checks its input parameters rigoreously.

Creating a simple web application server with HTTPServer

It is of course entirely possible to create a web application in a short time when you use an existing Python web application framework. In previous articles and my book on web applications I've used CherryPy extensively and although I recommend it for its flexibility an ease of use, it isn't all that difficult to create a web application framework from scratch.

Python's https.server module provides us with the basic building blocks: the HTTPServer class to handle incoming connections and a BaseHTTPRequestHandler class that processes requests and returns an answer. The main part of developing an application server is therefore sub-classing the BaseHTTPRequestHandler class. The minimum it will have to provide is a do_GET() method that will return results based on any parameters it receives.

Using Python parameter annotations

CherryPy uses classes with methods that serve together as an application: requested URLs are mapped to these methods and any parameters are passed along. CherryPy uses an expose decorator to identify the methods that may be called. Non exposed methods are invisible, i.e. URLs that match those methods do not result in the invocation of that method. This behavior is what we like to mimic in our own web application server.

Another important concept in web applications is the screening of input: we would like to check that incoming data (i.e. the arguments that come along with a query) are within the range of things we deem acceptable. For example, a function that adds its arguments should reject anything that cannot be interpreted as a float. We could write code easily enough that checks any function parameters explicitly but wouldn't it be nice if there was a more syntactically pleasing way of writing this?

Enter Python's function parameter annotations. Python allows us to augment each function parameter with an expression that is evaluated when the function is defined and which is stores in the functions __annotation__ field as a dictionary indexed by parameter name. Such an annotation might be as simple a single string but it can be anything, even a function reference. This function could be called with the value we would like to pass as an parameter to check if this value is ok. This could be done before the function is actually called, for example by the do_GET() method of our request handler.

Assuming we have our applicationserver module available, let's have a look what the definition of a new web application might look like:

from http.server import HTTPServer
from applicationserver import ApplicationRequestHandler,IsExposed

class Application:

 def donothing:
  pass
  
 def index(self) -> IsExposed:
  return 'index oink'
 
 def add(self,a:float,b:float) -> IsExposed:
  return str(a+b)
 
 def cat(self,a:str) -> IsExposed:
  return ' '.join(a)
 
 def opt(self,a:int=42) -> IsExposed:
  return str(a)
  
class MyAppHandler(ApplicationRequestHandler):
 application=Application()
 
appserver = HTTPServer(('',8088),MyAppHandler)
appserver.serve_forever()

The overall idea is to subclass the ApplicationRequestHandler and assign an instance of the Application class to its application field (line 21). This applicationhandler is then passed to a HTTPServer instance that will forward incoming requests to this handler (line 24).

Our ApplicationRequestHandler will try to map URLs of the form http://hostname:8080/foo?a=1&b=2 to member functions of the Application instance. It will only consider member function with a return annotation equal to an IsExposed object. So even though we have defined a donothing() function, it will not be executed when a URL like http://hostname:8080/donothing is received.

We also use annotations to restrict input values for parameters to functions that are exposed. Remember that annotations can be any expression and here we employ that fact to annotate the a and b parameters to the add() method with a reference to the built-in float() function. Our ApplicationRequestHandler will pass an argument to any callable it finds as its corresponding annotation and will only execute the method if this callable returns a value (and not raise an exception). So a URL like http://localhost:8080/add?a=1.23&b=4.56 will return a meaningful result while http://localhost:8080/add?a=1.23&b=spam will fail with an error. Off course we are not restricted to built-in functions here: we can refer to functions that may perform elaborate checking as well, perhaps checking against regular expressions or performing lookups in database tables.

All this shows that Python's function annotations allow for a rather elegant way to describe the expected behavior of methods that perform some sort of action in a web application. In a future article I'll show how to implement the applicationserver module.

More SQLite multithreading woes

In this article we revisit our thread safe persistent dictionary and encounter some irritating performance issues, both SQLite related and caused by Python itself.

SQLite.OperationalError, database is locked

When testing the persistentdict module with many simultaneous threads (more than twenty) I noticed a great number of errors: The many connections open to the same database caused SQLite to raise a lot of SQLite.OperationalError, database is locked exceptions. Getting decent performance with SQLite is by no means easy and because SQLite only locks complete database files and not just tables or rows there is a big chance that threads accessing the same database have to wait to get their turn.

sqlite3.connect, the check_same_thread parameter

Python 3.2 comes with a sqlite3 module that implements (but scarcely documents) a check_same_thread parameter that can be set to false to allow threads to use the same Connection object simultaneously. This is nice since this means we no longer have to implement all sorts of code to provide each thread with its own connection.

But we still have to regulate access to this connection because otherwise a commit in one thread may invalidate a longer running execute in another thread, leaving us with errors like sqlite3.InterfaceError: Cursor needed to be reset because of commit/rollback and can no longer be fetched from.

Python thread switching is really slow

Switching between threads, especially on multi core machines, has never been Python's strongest feature and making our persistent dict thread safe with a lock might hurt performance a lot, depending on what is going on in the threads itself (if there is a lot of I/O going on the impact might not be that big).

A new implementation

The code below shows the new implementation, with a single connection that may be shared by multiple threads. It works but it is really slow: with 40 threads I get just 10 to 20 dictionary assignments (d[1]=2) per second on my dual core Atom. That isn't the fastest machine around but those figures are ridiculously low. We will have to rethink our approach if we want to use SQLite in a multithreaded environment if we need any kind of performance!

"""
 persistentdict module $Revision: 98 $ $Date: 2011-07-23 14:01:04 +0200 (za, 23 jul 2011) $

 (c) 2011 Michel J. Anders

 This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see .

"""

from collections import UserDict
import sqlite3 as sqlite
from pickle import dumps,loads
import threading 

class PersistentDict(UserDict):

 """
 PersistentDict  a MutableMapping that provides a thread safe,
     SQLite backed persistent storage.
 
 db  name of the SQLite database file, e.g.
   '/tmp/persists.db', default to 'persistentdict.db'
 table name of the table that holds the persistent data,
   usefull if more than one persistent dictionary is
   needed. defaults to 'dict'
 
 PersistentDict tries to mimic the behaviour of the built-in
 dict as closely as possible. This means that keys should be hashable.
 
 Usage example:
 
 >>> from persistentdict import PersistentDict
 >>> a=PersistentDict()
 >>> a['number four'] = 4
 
 ... shutdown and then restart applicaion ...
 
 >>> from persistentdict import PersistentDict
 >>> a=PersistentDict()
 >>> print(a['number four'])
 4
 
 Tested with Python 3.2 but should work with 3.x and 2.7.x as well.
 
 run module directly to run test suite:
 
 > python PersistentDict.py
 
 """
 
 def __init__(self, dict=None, **kwargs):
  
  self.db    = kwargs.pop('db','persistentdict.db')
  self.table = kwargs.pop('table','dict')
  #self.local = threading.local()
  self.conn = None
  self.lock = threading.Lock()
  
  with self.lock:
   with self.connect() as conn:
    conn.execute('create table if not exists %s (hash unique not null,key,value);'%self.table)
    
  if dict is not None:
   self.update(dict)
  if len(kwargs):
   self.update(kwargs)
 
 def connect(self):
  if self.conn is None:
   self.conn = sqlite.connect(self.db,check_same_thread=False)
  return self.conn
   
 def __len__(self):
  with self.lock:
   cursor = self.connect().cursor()
   cursor.execute('select count(*) from %s'%self.table)
   return cursor.fetchone()[0]
 
 def __getitem__(self, key):
  with self.lock:
   cursor = self.connect().cursor()
   h=hash(key)
   cursor.execute('select value from %s where hash = ?'%self.table,(h,))
   try:
    return loads(cursor.fetchone()[0])
   except TypeError:
    if hasattr(self.__class__, "__missing__"):
     return self.__class__.__missing__(self, key)
   raise KeyError(key)
   
 def __setitem__(self, key, item):
  h=hash(key)
  with self.lock:
   with self.connect() as conn:
    conn.execute('insert or replace into %s values(?,?,?)'%self.table,(h,dumps(key),dumps(item)))

 def __delitem__(self, key):
  h=hash(key)
  with self.lock:
   with self.connect() as conn:
    conn.execute('delete from %s where hash = ?'%self.table,(h,))

 def __iter__(self):
  with self.lock:
   cursor = self.connect().cursor()
   cursor.execute('select key from %s'%self.table)
   rows = list(cursor.fetchall())
  for row in rows:
   yield loads(row[0])

 def __contains__(self, key):
  h=hash(key)
  with self.lock:
   cursor = self.connect().cursor()
   cursor.execute('select value from %s where hash = ?'%self.table,(h,))
   return not ( None is cursor.fetchone())

 # not implemented def __repr__(self): return repr(self.data)
 
 def copy(self):
  c = self.__class__(db=self.db)
  for key,item in self.items():
   c[key]=item
  return c

Nice discounts on open source titles at Packt

Packt has an offer on open source books, both in print and as e-book, that might interest you. Check out their July offering, there certainly are some interesting titles available, including a few on Python and web development.

linkcheck a Python module to check for broken links

When maintaining a web site or even when hosting a web application it makes sense to check once in a while whether there are any broken links. Here I present a simple module to find broken links in a website that is pure Python and uses no external modules.

Checking for broken links

Checking for broken links can be as simple as shown in the following example:

from linkcheck import LinkChecker

lc = LinkChecker("http://www.example.org")
if lc.check():
    if not lc.follow():
        print("there were problems")
        print("\n".join(lc.failed))
        print("\n".join(lc.other))
    else:
        print("website OK")
else:
    print("cannot open website or homepage is not html")

The check() method tries to open a URL and checks if its content is html. If this went well it returns True. The next step is to use the follow() method the see if we can open any links present in the page and report any failures. If a link points to a page containing HTML this is repeated in a recursive fashion.

The linkcheck module

We highlight some implementation details of the linkcheck module here, but the full code can be found on my website. It is licensed under the GPL and comes with a modest test suite.

The code depends on two crucial elements: LinkParser a html.parse.HTMLParser derived class and LinkChecker. LinkParser

acts on just a few HTML start tags defined in the tagrefs class variable (line 9). Its initializer is passed a baseurl argument that is used to derive relative URLs and a callback parameter that is called for each of the relevant tags that hold a reference to another URL

The LinkChecker class is discussed in detail in an an article on my website. For now I highlight just its main methods:

• __init__(), takes just one mandatory parameter the url we start with (line 28)
• check(), checks if the url passed to __init__() could be opened and points to a HTML file (line 38)
• follow(), finds any links within the HTML a marks any of those links it cannot opened as failed in the failed instance variable (line 59)
• process(), is the call back passed to an instance of the LinkParser class. It will recursively create a new LinkChecker instance to check the link for additional links. (line 67)

from urllib.request import Request,urlopen
from urllib.parse import urlsplit,urljoin,urlunsplit,urldefrag
from urllib.error import HTTPError,URLError
from html.parser import HTMLParser
from re import compile,MULTILINE,IGNORECASE

class LinkParser(HTMLParser):

 tagsrefs = { 'a':'href', 'img':'src', 'script':'src', 'link':'href' }
 
 def __init__(self,baseurl,callback):
  self.callback = callback
  self.baseurl = baseurl
  super().__init__()
  
 def handle_starttag(self, tag, attrs):
  if tag in self.tagsrefs:
   for name,value in attrs:
    if name == self.tagsrefs[tag]:
     newurl=urljoin(self.baseurl,value)
     self.callback(newurl)
     break

class LinkChecker:

 html=compile(r'^Content-Type:\s+text/html$',MULTILINE|IGNORECASE)

 def __init__(self,url,host=None,seen=None,external=True):
  self.url    = url
  self.host   = urlsplit(url).hostname if host is None else host
  self.failed = []
  self.other  = []
  self.notopened = []
  self.duplicates = 0
  self.seen = set() if seen is None else seen
  self.external = external

 def check(self,open=True):
  self.seen.add(self.url)
  if not open :
   self.notopened.append(self.url)
   return False
  try:
   self.req=urlopen(url=self.url,timeout=10)
  except HTTPError as e:
   self.failed.append(self.url)
   return False
  except URLError as e:
   self.other.append(self.url+' ('+str(e)+')')
   return False
  except Exception as e:
   print('Exception',e,type(e))
   self.failed.append(self.url)
   return False
  headers=str(self.req.info())
  m=self.html.search(headers)
  return not(m is None)
  
 def follow(self):
  parser = LinkParser(self.url,self.process)
  try:
   parser.feed(self.req.read().decode())
  except Exception as e:
   self.other.append(self.url+' ('+str(e)+')')
  return len(self.failed)+len(self.other) == 0
  
 def process(self,newurl):
  newurl=urldefrag(newurl)[0]
  if not newurl in self.seen:
   lc = LinkChecker(newurl,self.host,self.seen,self.external)
   samesite = urlsplit(newurl).hostname == self.host
   if lc.check(self.external or samesite) and samesite:
    lc.follow()
   self.failed.extend(lc.failed)
   self.other.extend(lc.other)
   self.notopened.extend(lc.notopened)
   self.seen.update(lc.seen)
   self.duplicates+=lc.duplicates
  else:
   self.duplicates+=1

A Python module providing thread safe SQLite backed persistent storage

In a previous article I wrote about some research I did on Python modules that could provide me with a persistent storage solution for dictionaries. I didn't quite find what I needed especially as none of the modules provided a thread safe solution. In the end I decided to write my own.

Thread safe, SQLite backed persistent storage

The module I wrote, persistentdict, is is freely available on my website along with some notes and examples. It also has a fairly extensive test suite. It provides a single class PersistentDict that behaves almost exactly like a native Python dict but stores its keys and values in a SQLite table instead of keeping it in memory.

Packt Python Book Idea Generator

Packt Publishing invites current and future readers to send in suggestions for subjects to cover in books on Python.

What Python subject do you want authors to write about?

I think this is a pretty neat idea. As a writer myself I choose subjects that interest me most and if a publisher thinks there is a market for it, the game is on. But I have many Python related interests and that probably goes for other authors as well, so if people had a way to show what they would like to see covered in a Python book, the publisher and authors could pick up a subject that would certainly please future readers. That's a classic win-win situation. The people at Packt now offer a such platform so if you have a (Python related) subject you would love to see a book about, check out this poll.