tag:blogger.com,1999:blog-61706320265106598982024-03-18T22:55:23.079+01:00Start SmallArticles, tips & tricks about programming Python applicationsMichelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.comBlogger65125tag:blogger.com,1999:blog-6170632026510659898.post-27207612751890531282018-02-05T13:31:00.000+01:002018-02-05T13:31:03.291+01:00A small HTTP server in PythonThe Python <a href="https://docs.python.org/3.5/library/http.server.html">http.server module</a> can be run from the commandline with<br />
<br /><code>python3 -m http.server</code><br /><br />
to provide a simple http server that listens on a configurable port and basically serves any file from the directory it is started from.<br />
<p>That is a good start but I had some additional requirements to serve up files from my raspberry Pi:<br />
<ul><li>The server must be able to run as a demonized process, i.e, in the background</li>
<li>It must server a limited set of files</li>
<li>from a configurable directory and</li>
<li>be able to run as a less privileged user</li>
</ul>Now of course I could have opted for a light weight http server like <a href="https://www.nginx.com/">nginx</a> or <a href="/www.lighttpd.net">lighttpd</a>, but I decided to write my own so I wouldn't have to install a boatload of additional packages.<br /><br />
<h2>restrictedhttpserver.py</h2>The server is based on the <code>daemon</code> module a wrote about in <a href="http://michelanders.blogspot.nl/2018/01/demonizing-python-process.html">a previous article</a> and can be started from the commandline in much the same way as http.server:<br />
<br /><br />
<code>python3 restrictedhttpserver.py</code><br />
<br /><br />
It will run as the user that started it, will listen on port 8000 and start serving any file from the current directory. It will demonize itself as well.<br />
<p>It takes some additional options:<br />
<pre name="code" style="font-size:8pt">python3 restrictedhttpserver.py -h
usage: restrictedhttpserver.py [-h] [-p PID_FILE] [-l LOG_FILE] [-r ROOT_DIR]
[-n NAME] [-u USER] [-f] [-s] [-d]
[--bind ADDRESS] [-x] [-e EXT]
[port]
Example HTTP Server
positional arguments:
port Specify alternate port [default: 8000]
optional arguments:
-h, --help show this help message and exit
-p PID_FILE, --pid-file PID_FILE
-l LOG_FILE, --log-file LOG_FILE
-r ROOT_DIR, --root-dir ROOT_DIR
-n NAME, --name NAME name of the server used in log lines
-u USER, --user USER drop privileges of running server to those of user
-f, --force start a server even if pid file is present already
-s, --stop stop a running server
-d, --debug Run http server in foreground mode
--bind ADDRESS, -b ADDRESS
Specify alternate bind address [default: all
interfaces]
-x, --nodirlist never show a directory listing
-e EXT, --ext EXT allowed file extensions (without .) May occur more
than once
</pre>For example if you wanted to run it as user <code>www</code> from the directory <code>/public/www</code> and just server html and css files, you could run it (as root) as follows:<br />
<br /><code>python3 restrictedhttpserver.py -r /public/www -u www -e html -e css</code><br />
<br />Later you could stop this process by executing<br /><code>python3 retrictedhttpserver.py -s</code><br />
<br /><br />
For more information have a look at the source code, it is pretty readable I think :-)<br />
<h2>Availability</h2>restrictedhttpserver.py and daemon.py are both available from <a href="https://github.com/varkenvarken/Small-Python-Stuff">this GitHub repository</a>.<br />
Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com41tag:blogger.com,1999:blog-6170632026510659898.post-30457130034428535302018-01-31T16:25:00.000+01:002018-01-31T16:25:41.811+01:00Demonizing a Python processProperly demonizing a Python process is not easy, even on *nix, and although various sources of good material can be found on the internet (for example <a href="https://github.com/ActiveState/code/tree/master/recipes/Python/278731_Creating_a_daemon_the_Python_way">here</a> and <a href="https://github.com/thesharp/daemonize/blob/master/daemonize.py">here</a>) it didn't quite match my requirements and/or I didn't understand it completely.<br />
I therefore decided to create both a daemon module and a http server to test it.<br />
<h3>Code availability</h3>Both modules are available on <a href="https://github.com/varkenvarken/Small-Python-Stuff">GitHub</a><br />
The daemon module is called <a href="https://github.com/varkenvarken/Small-Python-Stuff/blob/master/daemon.py">daemon.py</a> and the httpserver is called <a href="https://github.com/varkenvarken/Small-Python-Stuff/blob/master/restrictedhttpserver.py">restrictedhttpserver.py</a>. Both have a fair amount of comments in their source code that are the result of me trying to fully understand what is required to get things working.<br />
<p>A small word of warning here: I only tested it with Python3 on Ubuntu and Raspbian and I do not expect it to work on any other operating system (although most *nix like systems probably will work) and in my world Python2 is no longer a valid option.<br />
<p>Also, the <code>Daemon</code> class makes use of an undocumented attribute in the <a href="https://docs.python.org/3/library/logging.handlers.html#filehandler">logging.FileHandler</a> class (the stream.fileno attribute) which is necessary in order not to close this stream when forking a child process. Theoretically this might break in the future but I did want all logging to happen outside the chroot jail so that the process running as a daemon can log but does not have access to the logging. Whether this is a valid approach remains to be seen as this also prevents proper log rotation.<br />
<h2>The Daemon class</h2>The basic requirement are all implemented, an instance of a Daemon class can:<br />
<dl><dt>run in a chrooted jail</dt><dd>so it cannot access files outside a given directory</dd>
<dt>run with lowered privileges</dt><dd>so the risk of accessing files not owned by a specific user is lowered</dd>
<dt>can have a umask of your choice</dt><dd>which will ensure that newly created files by the daemon are not open to all</dd>
<dt>log outside its jail</dt><dd>so that we can log events without offering the process managed by daemon any way of altering the logging</dd>
<dt>and maintains a configurable pid file</dt><dd>so that we can prevent the daemon from starting more than once and providing a way the fond out the process id so that we can terminate the daemon</dd>
</dl>We also made the <code>Daemon</code> class follow the Singleton pattern, i.e. there can only be one instanced object of the class. This is sensible because a process can only be daemonized once. We also implemented to required methods to facilitate use as a context manager. A typical, minimal code snippet would look something like:<br />
<pre name="code" language="js">from daemon import Daemon
dm = Daemon()
with dm:
... do something forever ...
</pre>All configurable options have sensible defaults, but a call could look like this:<br />
<pre name="code" language="js">from daemon import Daemon
dm = Daemon(user='httpserver', rootdir='/var/www', umask=0o27,
pidfile='/var/lockhttpserver.pid',
logfile='/var/log/httpserver.log',
name="Http Server")
with dm:
... do something forever ...
</pre>It is important to understand that when the Daemon object is created, the process is not immediately daemonized. This is handled by the context manager (or you could call the <code>daemonize()</code> method directly). There is also a <code>stop()</code> method provided that will check for a running daemon process and terminate it. This could look like this:<br />
<pre name="code" language="js">from daemon import Daemon
import argparse
parser = argparse.ArgumentParser(description="Example Daemon")
parser.add_argument('-s', '--stop', action='store_true', help='stop a running server')
args = parser.parse_args()
dm = Daemon()
if args.stop:
dm.stop()
else:
with dm:
... do something forever ...
</pre><h2>More code</h2>In a future article I will highlight the small http server that I implemented to test the daemon module.<br />
Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com11tag:blogger.com,1999:blog-6170632026510659898.post-5201233692092744042016-03-03T08:57:00.001+01:002016-03-03T08:57:58.203+01:00I have posted an article on <a href="http://www.swineworld.org/2016/03/connections-in-a-tree-numpy-vs-python.html">my Blender blog</a> that investigates how to speed up the calculating the connectivity of trees with Numpy. Is completely non-Blender specific, so it might be interesting for readers of this blog as well.<br />
Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com6tag:blogger.com,1999:blog-6170632026510659898.post-54246588993264301622014-12-16T16:00:00.000+01:002014-12-16T16:00:06.063+01:00Python 3 Web Development Beginner's Guide for freeGuess what, Packt will include my Python 3 Web development book in their <a href="https://www.packtpub.com/packt/offers/christmas-countdown">Xmas countdown of free gifts</a> on December 17th!Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com7tag:blogger.com,1999:blog-6170632026510659898.post-58679527232881066992014-12-12T15:42:00.003+01:002014-12-12T15:42:50.770+01:00Free e-books at PacktPackt (the publisher of my books on <a href="https://www.packtpub.com/hardware-and-creative/blender-249-scripting">blender scripting</a> and <a href="https://www.packtpub.com/web-development/python-3-web-development-beginners-guide">python web development</a>) is giving away <a href="https://www.packtpub.com/packt/offers/christmas-countdown">free e-books</a> for Christmas.<br />
<br />
I was a bit late in noticing it but so far they have given away some pretty interesting titles so it might be a good idea to check it out for more to come...<br />
<div>
<br /></div>
Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com6tag:blogger.com,1999:blog-6170632026510659898.post-25901852893353572242013-04-13T11:02:00.000+02:002018-02-01T07:49:31.893+01:00A pure Python kd-tree implementation<p><a href="http://en.m.wikipedia.org/wiki/Kd-tree">kd-trees</a> are an efficient way to store data that is associated with a location in any number of dimensions up to twenty or so. Specifically, kd-trees allow for nearest neighbor searches in O(log n) time, something I desperately needed for my Blender tree generation add-on. In this article I highlight some of the design decisions that that shaped my pure Python implementation of a kd-tree module.<br />
</p><p style="background:#efe; border:3px solid green; padding:0.2em">Visiting my own post five years later a lot has changed. We are now quite a few versions of Blender ahead of what was available in 2013 and a kd-tree implementation is now <a href="https://docs.blender.org/api/current/mathutils.kdtree.html">part of Blender's Python API</a>.<br />
I highly recommend using that implementation because is written in C and probably quite a bit faster than my pure Python implementation. There is also a <a href="https://docs.blender.org/api/current/mathutils.bvhtree.html">bvh-tree implementation</a> that is suited for problems that not such much involve points but geometry that fills space, like triangles.</p><h2>kd-trees</h2><p>There exist quite a few <a href="http://www.cs.sunysb.edu/~algorith/implement/KDTREE/implement.shtml">implementations in C or C++</a> that are part of robust and well tested libraries but for my Blender add-on I needed a platform neutral (= pure python) implementation that I could ship with the add-on. There are a few <a href="https://github.com/stefankoegl/kdtree">Python implementations</a> but because I could not determine how well tested they were or because they did not quite meet my requirements I decided to develop an implementation myself.<br />
</p><h2>Requirements</h2><p>My requirements might not be everybody's requirements so besides supporting adding nodes and providing a nearest neighbor search what were the issues that were important to me?<br />
<ul><dt>Comes with test-suite</dt>
<dd>It might seem strange to make this the first requirement but because it's a part that is so essential to my add-on and because the code is quite tricky, I opted for test driven development. Adding unit tests to a python module is quite simple and although I can't claim that the tests cover all edge cases, they are nevertheless quite comprehensive. There is also a small performance test included.</dd>
<dt>Works with any iterable as position</dt>
<dd>A node in a kd-tree consists besides other things primarily of a position attribute. For use in Blender this typically would be a Vector instance but in other situations you might want to use other vector implementations. This kd-tree implementation assumes very little about its position attributes, just that it is subscriptable and that it supports a .dot() member function that returns the sum of the pairwise multiplication of its items (in other words, the dot product which is equal to the distance squared). Individual items should be floats (or anything that supports substraction, addition and comparisons). Probably any vector implementation will meet these requirements. The test-suite implements its own, based on Python's <code>list</code> class.</dd>
<dt>Allows easy deletion</dt>
<dd>Deleting nodes from a kd-tree is tricky and costly because of the rebalancing that has to be performed. I therefor opted for a much simpler solution: Instead of deleting a node we set its <code>data</code> attribute to <code>None</code> and add an option to the <code>.nearest()</code> member function to ignore nodes without data. Of course this won't make the tree any smaller but even when we delete a third of the nodes in random fashion (a typical use case for my tree add-on) the performance impact on nearest neighbor searches is minimal.</dd> </ul></p><h2>Availability</h2><p>The module is licensed as GPL and available on Github as part of the <a href="https://github.com/varkenvarken/spacetree">spacetree add-on</a> for Blender. It is contained in a single source file that can be <a href="https://github.com/varkenvarken/spacetree/blob/master/src/add_mesh_space_tree/kdtree.py">downloaded separately</a> (click <code>Raw</code> to download).</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com48tag:blogger.com,1999:blog-6170632026510659898.post-87223268806346681532012-11-08T18:44:00.000+01:002012-11-08T18:44:22.046+01:00Blender 2.6 add-on tutorial<p>I finished a <a href="http://michelanders.blogspot.nl/p/creating-blender-26-python-add-on.html">Blender add-on tutorial</a> the other day that might be an interesting read. And actually I liked the way my cover image turned out, so I reproduce it here :-)<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBLIRJ5bwAWmYr-XbhsuWFM1qvAychfdSW-OpyrIsEDBww7AVG46qGOVCD9XfSMz0JyfzIiFtV6bO2OiKkibtLUjJ8K9lmMC81M88qnG1FU1VWILzXgVHWLtTKDsyiKOGk-qRTK1ZqVOUl/s1600/110712_1520_oink1.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="360" width="540" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBLIRJ5bwAWmYr-XbhsuWFM1qvAychfdSW-OpyrIsEDBww7AVG46qGOVCD9XfSMz0JyfzIiFtV6bO2OiKkibtLUjJ8K9lmMC81M88qnG1FU1VWILzXgVHWLtTKDsyiKOGk-qRTK1ZqVOUl/s1600/110712_1520_oink1.png" /></a></div><br />
<br />
</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com95tag:blogger.com,1999:blog-6170632026510659898.post-12472651969506553772012-10-01T11:24:00.000+02:002012-10-01T11:24:14.425+02:00Web2py, a better alternative to CherryPy?<p>during a short holiday break I did some reading on <a href="http://www.web2py.com">Web2py</a> and my first impression is that that it looks really good. It seems to be much more complete and data centric than CherryPy. I think I will spend some time experimenting with it the coming months.</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com8tag:blogger.com,1999:blog-6170632026510659898.post-56702588366257922542012-09-22T14:51:00.000+02:002012-09-22T14:51:26.783+02:00Packt Publishing will soon release their 1000th title<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc6-e80J3wagc3jrxxA-0cpDOspqkdSMdOevfpcaZCHCU3TXPyFVI3YOrQ50T2tI9BSlbqNWwf4r71KhxWMa0O_FXhkmtq1erNozR8PlQM5GKIg-SSA1nljWaJzAWNOvheHAPC69s6P6Wg/s1600/1000th+Campaign+Banner.jpg" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="192" width="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc6-e80J3wagc3jrxxA-0cpDOspqkdSMdOevfpcaZCHCU3TXPyFVI3YOrQ50T2tI9BSlbqNWwf4r71KhxWMa0O_FXhkmtq1erNozR8PlQM5GKIg-SSA1nljWaJzAWNOvheHAPC69s6P6Wg/s200/1000th+Campaign+Banner.jpg" /></a></div><br />
<p>My publisher <a href="http://www.packtpub.com">Packt Publishing</a> is about to release their 1000th title, which I think is pretty amazing. Them being altogether very pleasant people to work with I have no reservations pointing you to their site. If you sign up before September 30. there's even some surprise for you in store. You can read all about in their <a href="http://www.packtpub.com/news/packt-publishing-reaches-1000-it-titles-and-celebrates-open-invitation">press release</a>.</p><p>Being an author myself and having reviewed quite a number of the books published by them, I know it takes some genuine effort to produce a good book. Reaching your 1000th in about 8 years time is therefore a serious achievement. Conratulations!</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com3tag:blogger.com,1999:blog-6170632026510659898.post-44085372502773535062012-02-26T18:44:00.000+01:002012-08-05T18:16:31.404+02:003D Convex hull in Python, a Blender implementation.<p>A slight adaption of the code in my <a href="http://michelanders.blogspot.com/2012/02/3d-convex-hull-in-python.html">previous post</a> to make it directly usable as a add mesh extension in <a href="http://www.blender.org/">Blender</a>.</p><h2>A Blender add mesh extension</h2><p><b>As of Blender 2.64 there is a native Convex Hull operator <a href="http://wiki.blender.org/index.php/User:Nicholasbishop/Convex_Hull">available</a>in Blender.</b></p><p>I won't elaborate too much on my adaption as it has nothing to do with web development. The script can be <a href="http://www.swineworld.org/blender/chull.py">downloaded from my site</a> and should be installed in Blenders addons directory and enabled in the user preferences. It is tested with Blender 2.61.</p><p>I still think it is a nice implementation and although it is pure Python, it performs quite well. Of course there is a native convex hull implementation in Blender game engine but that one isn't accessible from Python. <a href="http://adrianboeing.blogspot.com/2010/10/convex-hull-generation-in-blender.html">Previous implemenations</a> relied on external programs like <a href="http://www.qhull.org/">qhull</a>, which might be faster for large point sets but having a pure Python implementation that is tightly integrated with Blender feels cleaner. It certainly can handle points sets of up to a couple of thousand points.</p><p>I have adapted it to randomize the order of the vertices if the hull algorithm bombs. This will happen if adding a point involves calculating the volume of many degenerate tetrahedrons which is the case when trying to compute the hull of Blenders default uv-sphere. I tried to use Python's decimal module but the results did not improve so now if an error is encountered the vertices are reordered and a second attempt is made. Not clean, but simple. <br />
</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com4tag:blogger.com,1999:blog-6170632026510659898.post-25374192072577551812012-02-19T15:55:00.000+01:002017-06-09T09:12:33.106+02:003D Convex hull in PythonIn this article I present a present a reimplementation in pure Python of <a href="http://cs.smith.edu/~jorourke/books/compgeom.html">Joseph O'Rourke's</a> incremental 3D convex hull algorithm from his book <a href="http://amzn.com/0521649765">Computational Geometry in C</a>.
<br />
<h2>
A convex hull in pure Python</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIUIwBGrIhDK0glzOzjddZyjYFOSQaRN4fnMGZVMB9dFLM6WKxgmioXlMuoN09-H-y0-hz6QVhJNBzlRX4hT8ZC-0HoGFE2_Hib6K2xXgS0ihlfmQczXMqctX9D3RWAwlKRFCUaff29lBH/s1600/chull.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIUIwBGrIhDK0glzOzjddZyjYFOSQaRN4fnMGZVMB9dFLM6WKxgmioXlMuoN09-H-y0-hz6QVhJNBzlRX4hT8ZC-0HoGFE2_Hib6K2xXgS0ihlfmQczXMqctX9D3RWAwlKRFCUaff29lBH/s320/chull.png" width="320" /></a></div>
This is the second, rather off topic, article on computational geometry in this blog. The <a href="http://michelanders.blogspot.com/2012/02/marching-tetrahedrons-in-python.html">previous article</a> presented an implementation of the marching tetrahedrons algorithm.<br />
The goal of this article is to provide object oriented, pythonic code to compute the convex hull of a collection of 3D points. The code is contained in a single Python module that may be downloaded from <a href="https://github.com/varkenvarken/blenderaddons/blob/master/chull.py">GitHub</a>.<br />
A sample of how to use this module is shown below, where we create a a roughly spherical cloud of points, calculate its convex hull and print this hull in STL format to stdout. The resulting object is shown in the image (as seen in <a href="http://www.blender.org/">Blender</a>).<br />
<pre class="python" name="code">from random import random
from chull import Vector,Hull
sphere=[]
for i in range(2000):
x,y,z = 2*random()-1,2*random()-1,2*random()-1
if x*x+y*y+z*z < 1.0:
sphere.append(Vector(x,y,z))
h=Hull(sphere)
h.Print()
</pre>
<h2>
Difference from the original implementation</h2>
The original code restricted the coordinates of points to integers, here there is no such restriction. This might result in errors if the coordinates are large.
Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com7tag:blogger.com,1999:blog-6170632026510659898.post-46513244596865436002012-02-12T15:50:00.000+01:002012-02-12T15:50:42.276+01:00Marching Tetrahedrons in Python<p>In this atricle we show a simple implementation of the
<a href="http://en.wikipedia.org/wiki/Marching_tetrahedrons">Marching Tetrahedrons</a> algorithm in Python.
</p>
<h2>Marching Tetrahedrons</h2>
<p>The listing povided below is a straightforward reimplementation in Python of the ideas and code presented by <a href="http://paulbourke.net/geometry/polygonise/#tetra">Paul Bourke</a>. The image <div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpsDbfVw7l1eRO9bZy8QeRARwC5lK4Yx-gs1ADEByaWc2wHUg8mWF4VcHGjl0JmvdH6hbNB-amyMc_NUJPnUtpxy2zsU3vgANP3kKu3rHjXY31fJizVfcuOWQl9ENNmI7DYvaRO66UsDyP/s1600/lobes.png" imageanchor="1" style="clear:right; float:right; margin-left:1em; margin-bottom:1em"><img border="0" height="112" width="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpsDbfVw7l1eRO9bZy8QeRARwC5lK4Yx-gs1ADEByaWc2wHUg8mWF4VcHGjl0JmvdH6hbNB-amyMc_NUJPnUtpxy2zsU3vgANP3kKu3rHjXY31fJizVfcuOWQl9ENNmI7DYvaRO66UsDyP/s200/lobes.png" /></a></div>
shows the result of sampling a simple lobed function (rendered in Blender). I might implement the code directly in Blender but for now we export to an STL file that can be read by almost any 3D package. Note that the exported triangles do not have normals that point in a uniform direction, in fact we do no export any normals at all. You have to recalculate the normals in your 3D package before you can render the resulte with smooth shading.
</p>
<pre name="code" class="python">
class Vector: # struct XYZ
def __init__(self,x,y,z):
self.x=x
self.y=y
self.z=z
def __str__(self):
return str(self.x)+" "+str(self.y)+" "+str(self.z)
class Gridcell: # struct GRIDCELL
def __init__(self,p,n,val):
self.p = p # p=[8]
self.n = n # n=[8]
self.val = val # val=[8]
class Triangle: # struct TRIANGLE
def __init__(self,p1,p2,p3):
self.p = [p1, p2, p3] # vertices
# return triangle as an ascii STL facet
def __str__(self):
return """facet normal 0 0 0
outer loop
vertex %s
vertex %s
vertex %s
endloop
endfacet"""%(self.p[0],self.p[1],self.p[2])
# return a 3d list of values
def readdata(f=lambda x,y,z:x*x+y*y+z*z,size=5.0,steps=11):
m=int(steps/2)
ki = []
for i in range(steps):
kj = []
for j in range(steps):
kd=[]
for k in range(steps):
kd.append(f(size*(i-m)/m,size*(j-m)/m,size*(k-m)/m))
kj.append(kd)
ki.append(kj)
return ki
from math import cos,exp,atan2
def lobes(x,y,z):
try:
theta = atan2(x,y) # sin t = o
except:
theta = 0
try:
phi = atan2(z,y)
except:
phi = 0
r = x*x+y*y+z*z
ct=cos(theta)
cp=cos(phi)
return ct*ct*cp*cp*exp(-r/10)
def main():
data = readdata(lobes,5,41)
isolevel = 0.1
#print(data)
triangles=[]
for i in range(len(data)-1):
for j in range(len(data[i])-1):
for k in range(len(data[i][j])-1):
p=[None]*8
val=[None]*8
#print(i,j,k)
p[0]=Vector(i,j,k)
val[0] = data[i][j][k]
p[1]=Vector(i+1,j,k)
val[1] = data[i+1][j][k]
p[2]=Vector(i+1,j+1,k)
val[2] = data[i+1][j+1][k]
p[3]=Vector(i,j+1,k)
val[3] = data[i][j+1][k]
p[4]=Vector(i,j,k+1)
val[4] = data[i][j][k+1]
p[5]=Vector(i+1,j,k+1)
val[5] = data[i+1][j][k+1]
p[6]=Vector(i+1,j+1,k+1)
val[6] = data[i+1][j+1][k+1]
p[7]=Vector(i,j+1,k+1)
val[7] = data[i][j+1][k+1]
grid=Gridcell(p,[],val)
triangles.extend(PolygoniseTri(grid,isolevel,0,2,3,7))
triangles.extend(PolygoniseTri(grid,isolevel,0,2,6,7))
triangles.extend(PolygoniseTri(grid,isolevel,0,4,6,7))
triangles.extend(PolygoniseTri(grid,isolevel,0,6,1,2))
triangles.extend(PolygoniseTri(grid,isolevel,0,6,1,4))
triangles.extend(PolygoniseTri(grid,isolevel,5,6,1,4))
export_triangles(triangles)
def export_triangles(triangles): # stl format
print("solid points")
for tri in triangles:
print(tri)
print("endsolid points")
def t000F(g, iso, v0, v1, v2, v3):
return []
def t0E01(g, iso, v0, v1, v2, v3):
return [Triangle(
VertexInterp(iso,g.p[v0],g.p[v1],g.val[v0],g.val[v1]),
VertexInterp(iso,g.p[v0],g.p[v2],g.val[v0],g.val[v2]),
VertexInterp(iso,g.p[v0],g.p[v3],g.val[v0],g.val[v3]))
]
def t0D02(g, iso, v0, v1, v2, v3):
return [Triangle(
VertexInterp(iso,g.p[v1],g.p[v0],g.val[v1],g.val[v0]),
VertexInterp(iso,g.p[v1],g.p[v3],g.val[v1],g.val[v3]),
VertexInterp(iso,g.p[v1],g.p[v2],g.val[v1],g.val[v2]))
]
def t0C03(g, iso, v0, v1, v2, v3):
tri=Triangle(
VertexInterp(iso,g.p[v0],g.p[v3],g.val[v0],g.val[v3]),
VertexInterp(iso,g.p[v0],g.p[v2],g.val[v0],g.val[v2]),
VertexInterp(iso,g.p[v1],g.p[v3],g.val[v1],g.val[v3]))
return [tri,Triangle(
tri.p[2],
VertexInterp(iso,g.p[v1],g.p[v2],g.val[v1],g.val[v2]),
tri.p[1])
]
def t0B04(g, iso, v0, v1, v2, v3):
return [Triangle(
VertexInterp(iso,g.p[v2],g.p[v0],g.val[v2],g.val[v0]),
VertexInterp(iso,g.p[v2],g.p[v1],g.val[v2],g.val[v1]),
VertexInterp(iso,g.p[v2],g.p[v3],g.val[v2],g.val[v3]))
]
def t0A05(g, iso, v0, v1, v2, v3):
tri = Triangle(
VertexInterp(iso,g.p[v0],g.p[v1],g.val[v0],g.val[v1]),
VertexInterp(iso,g.p[v2],g.p[v3],g.val[v2],g.val[v3]),
VertexInterp(iso,g.p[v0],g.p[v3],g.val[v0],g.val[v3]))
return [tri,Triangle(
tri.p[0],
VertexInterp(iso,g.p[v1],g.p[v2],g.val[v1],g.val[v2]),
tri.p[1])
]
def t0906(g, iso, v0, v1, v2, v3):
tri=Triangle(
VertexInterp(iso,g.p[v0],g.p[v1],g.val[v0],g.val[v1]),
VertexInterp(iso,g.p[v1],g.p[v3],g.val[v1],g.val[v3]),
VertexInterp(iso,g.p[v2],g.p[v3],g.val[v2],g.val[v3]))
return [tri,
Triangle(
tri.p[0],
VertexInterp(iso,g.p[v0],g.p[v2],g.val[v0],g.val[v2]),
tri.p[2])
]
def t0708(g, iso, v0, v1, v2, v3):
return [Triangle(
VertexInterp(iso,g.p[v3],g.p[v0],g.val[v3],g.val[v0]),
VertexInterp(iso,g.p[v3],g.p[v2],g.val[v3],g.val[v2]),
VertexInterp(iso,g.p[v3],g.p[v1],g.val[v3],g.val[v1]))
]
trianglefs = {7:t0708,8:t0708,9:t0906,6:t0906,10:t0A05,5:t0A05,11:t0B04,4:t0B04,12:t0C03,3:t0C03,13:t0D02,2:t0D02,14:t0E01,1:t0E01,0:t000F,15:t000F}
def PolygoniseTri(g, iso, v0, v1, v2, v3):
triangles = []
# Determine which of the 16 cases we have given which vertices
# are above or below the isosurface
triindex = 0;
if g.val[v0] < iso: triindex |= 1
if g.val[v1] < iso: triindex |= 2
if g.val[v2] < iso: triindex |= 4
if g.val[v3] < iso: triindex |= 8
return trianglefs[triindex](g, iso, v0, v1, v2, v3)
def VertexInterp(isolevel,p1,p2,valp1,valp2):
if abs(isolevel-valp1) < 0.00001 :
return(p1);
if abs(isolevel-valp2) < 0.00001 :
return(p2);
if abs(valp1-valp2) < 0.00001 :
return(p1);
mu = (isolevel - valp1) / (valp2 - valp1)
return Vector(p1.x + mu * (p2.x - p1.x), p1.y + mu * (p2.y - p1.y), p1.z + mu * (p2.z - p1.z))
if __name__ == "__main__":
main()
</pre>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com5tag:blogger.com,1999:blog-6170632026510659898.post-82426912211328315602011-10-30T01:00:00.002+02:002011-10-30T01:00:01.478+02:00A jQuery Mobile SelectorImage plugin<p>I implemented a jQueryMobile plugin to enhance a <select> element with a clickable image map. A full article with examples is available on <a href="http://www.swineworld.org/index.html#/articles/erica-web-application-framework-selectorimage-js.html">on my site</a>
</p>
<h2>A simple, JavaScript only implementation of an clickable image map.</h2>
<p>In this age of html5 it is of course not done to use clickable image maps :-) Yet I did want to enhance <select> elements with clickable images to allow for a visual alternative for a drop down list without the hard to maintain imagemaps of old. I therefore implemented a jQueryMobile plugin that takes two maps (these are specified with <code>data-</code> attributes): one to display to the user and one to act as a definition of hotspots. These hotspots are simply colored areas in the hotspot map (instead of polygon definitions in an image map), When the user clicks the image the corresponding location is checked in the hotspot map and the color found is looked up in the list of <option> elements. If there is a match, that option is selected. A working example can be found <a href="http://www.swineworld.org/articles/selectorimage.html">here</a>. A full explanation, including a small list of known problems can be found <a href="http://www.swineworld.org/index.html#/articles/erica-web-application-framework-selectorimage-js.html">here</a>.
</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com4tag:blogger.com,1999:blog-6170632026510659898.post-79192870864892621672011-10-23T01:00:00.000+02:002011-10-23T01:00:02.632+02:00Implementing a HTTPS server in Python<p>Web applications often transfer sensitive data between client and server. Even a session id is not something that should be vulnerable to eavesdropping. It is therefore a very good idea to encrypt all communication and implement a HTTPS server.</p>
<h2>Subclassing HTTPServer</h2>
<p>Python's <code>ssl</code> module has been cleaned up quite a bit since version 3.x and with a little help from <a href="http://code.activestate.com/recipes/442473-simple-http-server-supporting-ssl-secure-communica/">this recipe</a> it was incredibly simple to adapt the <code>HTTPServer</code> class from the <code>http.server</code> module to accept only secure connections:</p>
<pre name="code" class="python">
import ssl
import socket
from socketserver import BaseServer
from http.server import HTTPServer
class HTTPSServer(HTTPServer):
def __init__(self,address,handler):
BaseServer.__init__(self,address,handler)
self.socket = ssl.SSLSocket(
sock=socket.socket(self.address_family,self.socket_type),
ssl_version=ssl.PROTOCOL_TLSv1,
certfile='test.pem',
server_side=True)
self.server_bind()
self.server_activate()
</pre>
<p>All we do basically is change the initialization code to create a secure socket instead of a regular one (in line 10). The things to watch out for is the <code>ssl_version</code>: older versions are considered <a href="https://www.owasp.org/index.php/Transport_Layer_Protection_Cheat_Sheet#Rule_-_Only_Support_Strong_Protocols">unsafe</a> so we use TLS 1.0 here. Also the certificate file we use here contains both our certificate and our private key. If you want to use a self signed certificate for testing purposes you could generate one with <a href="http://www.openssl.org/">openssl</a> (most UNIX-like operating systems offer binary packages, for a precompiled package for windows check the faq.)</p>
<pre name="code" class="python">
openssl req -new -x509 -keyout test.pem -out test.pem -days 365 -nodes
</pre>
<p>Note that your browser will still complain about this certificate because it is self signed.</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com3tag:blogger.com,1999:blog-6170632026510659898.post-21472348570443814882011-10-16T01:00:00.000+02:002011-10-16T01:00:03.180+02:00Managing a session id with cookies<p>When managing sessions in web applications the key to the castle is a <em>sessionid</em>. Most often this sessionid is passed to and from the client by means of a <em>cookie</em>. In this article we explore the tools available in Python to deal with cookies.</p><h2>Reading and writing cookies</h2><p>If we are extending the <code>BaseHTTPRequestHandler</code> class from Python's <code>http.server</code> module we have access to the request headers by means of the <code>headers</code> attribute. It provides a convient method <code>get_all()</code> to retrieve headers by name (line 12 in the code sample below):</p><pre name="code" class="python">from http.cookies import SimpleCookie as cookie
...
class ApplicationRequestHandler(BaseHTTPRequestHandler):
sessioncookies = {}
def __init__(self,*args,**kwargs):
self.sessionidmorsel = None
super().__init__(*args,**kwargs)
def _session_cookie(self,forcenew=False):
cookiestring = "\n".join(self.headers.get_all('Cookie',failobj=[]))
c = cookie()
c.load(cookiestring)
try:
if forcenew or self.sessioncookies[c['session_id'].value]-time() > 3600:
raise ValueError('new cookie needed')
except:
c['session_id']=uuid().hex
for m in c:
if m=='session_id':
self.sessioncookies[c[m].value] = time()
c[m]["httponly"] = True
c[m]["max-age"] = 3600
c[m]["expires"] = self.date_time_string(time()+3600)
self.sessionidmorsel = c[m]
break
</pre><p>The way we call <code>get_all()</code> provides us with an empty list if there are no cookies in the headers. Either way we end up with a (possibly empty) string that contains the cookies the client sent us. This cookie (or cookies) can be converted to a <code>SimpleCookie</code> object from Python's <code>http.cookie</code> module (line 14). </p><p>Cookies are basically key/value pairs with some extra attributes. The whole ensemble is called a <em>morsel</em>. The <code>SimpleCookie</code> object acts as a dictionary that indexes those morsels by key. The whole excersize is aimed at maintaining a session id so we check if our cookie object holds a <code>session_id</code> morsel and use its value (a GUID) as an index into the <code>sessioncookies</code> class variable. This class variabele maintains a dictionary indexed by the GUIDs of the session id cookies we produced. The corresponding values are their timestamps. Line 17 will therefore raise an exception if <ul><li>no session_id cookie was provided by the client,</li>
<li>the session_id is unknown to us,</li>
<li>the session_id is expired, i.e. older than one hour, or</li>
<li>when we explicitely indicated we want a new cookie, no matter what.</li>
</ul>In those cases we generate a complete new, random, GUID and store its hexadecimal representation. </p><p>At this point we are guaranteed to have a <code>SimpleCookie</code> object that contains a session_id. Our final tasks are to store the timestamp of this cookie and to update or set some additional attributes on this morsel. The client might have sent more than one cookie so we iterate over all morsel and stop at the first session_id morsel we find. We set its <code>httponly</code> attribute to signal to the browset that this cookie should not be manipulated by any client side JavaScript and set both its <code>expires</code> attribute and its <code>max-age</code> before we store this specific morsel in an instance variabele. This way we can add this cookie to the response headers one we have processed the request. An outline is sketched in the snipper below:</p><pre name="code" class="python">...
def do_GET(self):
...
self._session_cookie()
...
if not (self.sessionidmorsel is None):
self.send_header('Set-Cookie',self.sessionidmorsel.OutputString())
...
</pre><h2>Security considerations</h2><p>At this point we have a tool to manage a session id. Such a session id can be used as a key to access other session information, a topic we cover in a later article. Before we even start thinking of using this we should consider the security issues.</p><p><a href="http://www.pythonsecurity.org/wiki/sessionmanagement/">Pythonsecurity.org</a> has a handy checklist that will walk through:<br />
<dl><dt>Are session IDs exposed in the URL?</dt>
<dd>No, we use cookies and those are part of the HTTP headers.</dd>
<dt>Do session IDs timeout and can users log out?</dt>
<dd>Our session IDs certainly timeout but providing a log out option should be part of the web application.</dd>
<dt>When a user logs out or times out, is the session invalidated?</dt>
<dd>At this point we only dealing with session ids.</dd>
<dt>Are session IDs rotated after successful login?</dt>
<dd>That is why we provide the <code>forcenew</code> paramter. After a successful login the web application should call <code>_session_cookie()</code> again with <code>forcenew=True</code></dd>
<dt>Are session IDs only sent over TLS/SSL?</dt>
<dd>That should be implemented in the HTTP server, here we only look at the request handling part.</dd>
<dt>Are session IDs completely randomly generated?</dt>
<dd>We use a variant 4 uuid from Python's <code>uuid</code> module which should be completely random. The <a href="http://docs.python.org/py3k/library/uuid.html#module-uuid">documentation</a> makes no claims about the quality of the random generator used but a quick look at the code of the <code>uuid</code> module (in Python 3.2) reveals that is uses either system provided functions or the built-in <code>random()</code> function. In other words, we don't know what we get and that is bad! It is probably better use <code>os.urandom()</code> directly which will raise a NotImplentedError if a source of randomness couldn't be found. Generating a a string of 32 hex digits might be done as follows: <code>"%02x"*16%tuple(os.urandom(16))</code></dd> </dl></p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com3tag:blogger.com,1999:blog-6170632026510659898.post-85288031711223983852011-10-09T01:00:00.005+02:002011-10-09T01:00:03.479+02:00A jQuery Mobile Address plugin<p>While playing around with jQuery Mobile and wondering about the support for the new HTML5 input types I realized there wasn't an address extension that was just as easy to use a date picker was for dates. So I wrote an address plug-in.
</p>
<h2>Adding Google Maps to an input element</h2>
<p>The idea is very simple: addresses are text just like an e-mail address or an url. HTML5 makes it possible to identify this a input element types and jQuery Mobile acts on these types as well to display custom widgets for these elements.
</p>
<p>I wanted an <em>address</em> type: an input element with that type would allow the user to enter an address or a click on it would open an interactive map. Clicking on a location in that map would fill in the address under the cursor. Writing such a plug-in was a fine exercise in both jQuery Mobile and the Google Maps API and the first results are documented
<a href="http://www.swineworld.org/articles/erica-web-application-framework-address-js.html">on my website</a> as part of the Erica Web Application framework.
</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com1tag:blogger.com,1999:blog-6170632026510659898.post-90784506773338320892011-10-02T01:00:00.001+02:002011-10-04T08:11:51.453+02:00Erica web application framework<p>I started developing and writing about a new Python web application framework in Python to aid people in understanding concepts in web application development.<br />
</p><h2>Erica web application framework</h2><p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTHNSQXrdORH5XOtxg2894fqPoOz7jVmVEp5yUMD1mlZs24ZMk1g2WrWhbtnS4P96MIT3y16RPOrHKA3l8bNTR11XlcSijzd8IObq7yhv9uqo8fm2wTIytO42Xiu9-IZ5FXbvsiTvp6Cr-/s1600/erica.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="125" width="125" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTHNSQXrdORH5XOtxg2894fqPoOz7jVmVEp5yUMD1mlZs24ZMk1g2WrWhbtnS4P96MIT3y16RPOrHKA3l8bNTR11XlcSijzd8IObq7yhv9uqo8fm2wTIytO42Xiu9-IZ5FXbvsiTvp6Cr-/s200/erica.png" /></a></div></p><p>The framework is named after one of our cutest pygmy goats, Erica, which is of course completely irrelevant :-). More important is that the <a href="http://www.swineworld.org/index.html#/articles/erica-web-application-framework.html">first article</a> outlining the goals, is already on-line on my website. The articles will build and extend on blog entries made in this blog. The idea is that the website will develop into a small but comprehensive tutorial on implementing a framework while this blog will be the place to watch for updates and place your comments.</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com1tag:blogger.com,1999:blog-6170632026510659898.post-15514124021151557832011-09-25T01:00:00.000+02:002011-09-25T01:00:00.872+02:00Python regular expressions and the functools module<p>If we use <a href="http://michelanders.blogspot.com/2011/08/function-annotations-in-python-checking.html">annotations</a> to sanitize incoming data easy access to regular expressions is a must. In this article I'll show how to make the most of Python's bundles <code>functools</code> module to dynamically generate functions that match regular expressions while reducing the overhead of compiling regular expressions as much as possible</p><h2>Matching against regular expressions</h2><p>In our framework we annotate function parameters with functions that will pass through or convert incoming arguments if they are ok or raise an exception other wise. For example, if we would want to make sure an argument is an integer, we'd annotate it with Python's built-in <code>int</code> function:</p><pre name="code" class="python">def myfunction(arg:int):
...
</pre><p>The framework will use this function to check the argument <em>before</em> actually calling the function.<p/><p>In many situations it would be convenient if we could specify a regular expression that arguments should match. We could extend the framework to check whether the annotation was a string and use that string as a regular expression but it is better to opt for a approach that will allow for the easy definition of categories. Some examples: <code>def myfunc(arg:digits) ...</code> or <code>def myfunc(arg:zipcode) ...</code>. In these examples <code>digits</code> and <code>zipcode</code> are functions that take single argument and return that argument if matches a certain pattern or raise an exception if it doesn't.</p><p>To construct such functions easily we may utilize Python's <code><a href="http://docs.python.org/py3k/library/re.html">re</a></code> and <code><a href="http://docs.python.org/py3k/library/functools.html">functools</a></code>modules:</p><pre name="code" class="python">from re import compile
def regex_compile(pattern):
return compile("^"+pattern+"$")
def match(pattern,string):
re = regex_compile(pattern)
if re.match(string) is None:
raise ValueError("no pattern match")
return string
</pre><p>To check whether a string matches a regular expression we define a <code>match</code> function (line 6) that will compile the regular expression and match the string against it. The <code>regex_compile</code> function will make sure the match is anchored at the beginning and the end because we want a complete match, that is string that merely contain the pattern are considered invalid. We don't allow extraneous characters.</p><h2>Constructing partial functions</h2><p>So far nothing new under the sun but we still need a simple way to construct functions that match a single argument against a regular expression. Enter the <code>partial</code> function. It will take a function and some arguments and use those to construct a new function that will pass any arguments together with the original arguments to the encapsulated function. This is exactly what we need to construct our categories:</p><pre name="code" class="python:firstline[11]">from functools import partial
digits = partial(match,r'\d+')
zipcode = partial(match,r'\d{4}\s*[a-zA-Z]{2}')
print(digits('12345'))
print(zipcode('1234AB'))
</pre><h2>Improving efficiency by Memoization</h2><p>Each time we call <code>digits(a)</code> we actually call <code>match(r'\d+',a)</code> and this will result in compile the regular expression again and again. Compiling regular expressions is a rather expensive operation so we might want to avoid that by reusing compiled expressions. This can simply be accomplished by applying the <code>lru_cache</code> decorator from the <code>functools</code> module to implement a <a href="http://en.wikipedia.org/wiki/Memoization">memoize pattern</a>:</p><pre name="code" class="python">from functools import lru_cache
from re import compile
@lru_cache(maxsize=0)
def regex_compile(pattern):
return compile("^"+pattern+"$")
</pre><p>This simple change will make sure we will reuse compiled regular expressions most of the time. The <code>maxsize</code> parameter should be set as needed: it may be set to zero in which case all regular expression will be cached, possibly a good choice as it is unlikely that your web application sports thousands of regular expressions.</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com1tag:blogger.com,1999:blog-6170632026510659898.post-31509753940356343952011-09-18T01:00:00.000+02:002011-09-18T09:58:47.035+02:00Propagating Python function annotations through decorators<p>When we decorate a Python function we must take some care to ensure that the decorated function can present itself in the same way as before its decoration. Let's explore what is needed.</p><h2>Decorating a Python function</h2><p>Say we have an undecorated function <code>f()</code>. It features a docstring (line 2) and annotated arguments:</p><pre name="code" class="python">def f(a:str):
"an undecorated function"
return a
print(f.__name__,f.__annotations__,f.__doc__,f('foo'))
</pre><p>The final line in the previous snippet prints the name of the function, its annotation dictionary and the docstring. The output looks like this:</p><pre>f {'a': <class 'str'>} an undecorated function foo
</pre><h2>Decorating a Python function</h2><p>Now say we have a function <code>do_something()</code> and we would like to have a decorator that would arrange that a function would call <code>do_something()</code> first. We might set this up like this:</p><pre name="code" class="python">def do_something():
print('oink')
def decorator(f):
def g(*args,**kwargs):
do_something()
return f(*args,**kwargs)
return g
@decorator
def f(a:str):
"a naively decorated function"
return a
print(f.__name__,f.__annotations__,f.__doc__,f('foo'))
</pre><p>The final line would produce the following output:</p><pre>oink
g {} None foo
</pre><p>It is clear that <code>do_something()</code> is executed as it does print <tt>oink</tt> but although our decorated function can be called as <code>f()</code>, its name, annototations and doctring are different.<br />
</p><h2>Preserving attributes when decorating a Python function</h2><p>To resolve these matters we can rewrite our decorator as follows:</p><pre name="code" class="python">def decorator(f):
def g(*args,**kwargs):
do_something()
return f(*args,**kwargs)
g.__name__ = f.__name__
g.__doc__ = f.__doc__
g.__annotations__ = f.__annotations__
return g
@decorator
def f(a:str):
"a less naively decorated function"
return a
print(f.__name__,f.__annotations__,f.__doc__,f('foo'))
</pre><p>We now simply copy the relevant attributes to the newly created function and if we now examine the output produced by the final line we get what we expect:</p><pre>oink
f {'a': <class 'str'>} a less naively decorated function foo
</pre>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com2tag:blogger.com,1999:blog-6170632026510659898.post-64790886748621860962011-09-11T01:00:00.000+02:002011-09-11T01:00:04.009+02:00Implementing POST method handling in a web application server<p>In an <a href="http://michelanders.blogspot.com/2011/08/function-annotations-in-python-checking.html">ongoing project</a> to implement a web application server that is as simple as possible we now implement handling POST requests</p><h2>Handling POST requests</h2><p>The HTTP POST method is often the preferred way to transfer information from a form to the server. If we use POST the arguments don't end up in the webserver log for example and it allows us to use a file upload tag.</p><p>Depending on the <code>encoding</code> attribute of a <code>form</code> element, POST data might be encode as a number of key/value pairs (one on each line) or a multipart MIME message (useful if you want to upload files). More information can be found <a href="http://www.w3.org/TR/html4/interact/forms.html">here.</a></p><h2>Wielding the power of Python's <code>cgi</code> module</h2><p>Decoding a multipart MIME message isn't trivial but lucky for us we can offload the hard work to an existing module: <code>cgi</code>. It is part of the standard Python distribution and although designed to implemented cgi scripts we can co-opt its functionality for our purpose. All we have to do is add a <code>do_POST()</code> method to our <code>ApplicationRequestHandler</code> class as shown below:</p><pre name="code" class="python">from cgi import FieldStorage
from os import environ
class ApplicationRequestHandler(BaseHTTPRequestHandler):
def do_POST(self):
ob=self._find_app()
environ['REQUEST_METHOD'] = 'POST'
fs=FieldStorage(self.rfile,headers=self.headers)
kwargs={}
for name in fs:
for i in fs.getlist(name):
if fs[name].file:
kwargs[name] = fs[name].file
kwargs[name].srcfilename = fs[name].filename
else:
kwargs[name] = fs[name]
return self._execute(ob,kwargs)
</pre><p>We factored out some common code that is also used in the <code>do_GET()</code> method (in the <code>_find_app()</code> and <code>_execute()</code>methods, not shown here), but basically we instantiate a <code>cgi.FieldStorage</code> instance and pass it the input filestream along with a dictionary of the HTTP headers we have enounterd so far. Because the <code>cgi</code> module expects some parameters to be present in the environment we set the <code>REQUEST_METHOD</code> to <code>POST</code> because that information not part of the headers. The <code>FieldStorage</code> instance returned can be used as dictionary with the names of the parameters as keys.</p><p>Any file input parameters are a bit special. The contents of the uploaded file are stored in a temporary file and if we would access this parameter the value would be the contents of this file as a byte object. We'd rather pass on the temporary file object to prevent passing around the contents of really big files unnecessarily. We can check if a parameter is an uploaded file by checking its <code>file</code> parameter (line 14). If the argument is a file, we tack on the original file name (that is, the one the user's PC) because we might want to use that later. The final line passes the arguments we have extracted to the <code>_execute()</code> method. This method is factored out from the <code>do_GET()</code> method. It locates the function to execute and checks its arguments against any annotations</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com3tag:blogger.com,1999:blog-6170632026510659898.post-74796422550403154422011-09-04T00:15:00.012+02:002011-09-04T08:51:33.679+02:00Finding stuck or hot pixels in Nikon D80 images, a multiprocessing approach<p><a href="http://michelanders.blogspot.com/2011/08/finding-stuck-or-hot-pixels-in-nikon.html">Earlier</a> I described a method to find hot or stuck pixels by determining the variance of (sub)pixels over a large set of photos. In this article we take a look at a way to farm out this work to multiple processes.
</p>
<h2>The parallel algorithm for calculating variance</h2>
<p>The
<a href="http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm">parallel algorithm</a>
allows us to combine calculated variances from separate sets. With the help of Python's <code>multiprocessing</code> module it is straight forward to implement our hot pixel finding algorithm to one that takes advantage of multiple cores:
</p>
<pre name="code" class="python">
import Image
import numpy as np
import sys
from glob import glob
from multiprocessing import Pool,current_process
from time import time
def process_pixels(shape,*args):
first = True
for filename in args:
pic = Image.open(filename)
if pix.shape != shape:
print("shapes don't match")
continue
if first:
first = False
firstpix = pix
n = np.zeros(firstpix.shape)
mean = np.zeros(firstpix.shape)
M2 = np.zeros(firstpix.shape)
delta = np.zeros(firstpix.shape)
n += 1
delta = pix - mean
mean += delta/n
M2 += delta*(pix - mean)
return M2
if __name__ == '__main__':
global pool
n=int(sys.argv[1])
pool = Pool(n)
filenames = []
for a in sys.argv[2:]:
filenames.extend(glob(a))
shape = np.array(Image.open(filenames[0])).shape
s=time()
results = []
for i in range(n):
results.append(pool.apply_async(process_pixels,tuple([shape]+filenames[i::n])))
for i in range(n):
results[i]=results[i].get()
M2 = sum(results)
print(time()-s)
mini = np.unravel_index(M2.argmin(),M2.shape)
maxi = np.unravel_index(M2.argmax(),M2.shape)
print('min',M2[mini],mini)
print('max',M2[maxi],maxi)
print('mean',np.mean(M2))
sorti = M2.argsort(axis=None)
print(sep="\n",*[(i,M2[i]) for i in [np.unravel_index(i,M2.shape) for i in sorti[:10]]])
print(time()-s)</pre>
<p>Note that because we return complete arrays (line 26) we gain almost nothing for small data sets because of the overhead of shuttling these large arrays (> 30 MByte) between processes. This is illustrated in the following graph with shows the elapsed time as a function of the number of processes, both for a small number of pictures (50) and a large number (480).</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRCkDiEb6XexrPGKY-p3DTsuGWgdmr1m7pIGffkyBopQ2-2M0bJ8VxtxdEkmfRo-m3ZPdAb5ZxHdMZqZ6je6286Z73-DsOQ02E8dT06z7RM3Uw8J0VmLgtDMXyRO34tGogFiu3xAgI4TVN/s1600/hotpixels-multiprocessing.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="305" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRCkDiEb6XexrPGKY-p3DTsuGWgdmr1m7pIGffkyBopQ2-2M0bJ8VxtxdEkmfRo-m3ZPdAb5ZxHdMZqZ6je6286Z73-DsOQ02E8dT06z7RM3Uw8J0VmLgtDMXyRO34tGogFiu3xAgI4TVN/s400/hotpixels-multiprocessing.png" /></a></div>
<p>Some other notable issues: in the code we do not actually implement the parallel algorithm but we simple add together the variances. Because we're looking for a minimum variance we gain nothing by adding a constant value.
</p><p>Memory usage is another thing to be aware of (and the reason that there is no entry for six cores in the graph. The algorithm we have implemented uses 5 arrays (the pixel data itself included). That makes for 10 megapixel X 3 colors X 5 arrays X 8 bytes data (because we use 64 bit floats by default) which makes for a whopping 1.2 Gigabyte of data per process or more than 6 Gig with 5 processes. With some other applications open a sixth process wouldn't fit on my test machine. Because we're adding pixel values in the range 0 - 255 we could probably gain a lot by using 32 bit floats or even 16 bit floats here.
</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com1tag:blogger.com,1999:blog-6170632026510659898.post-9662694349474775612011-08-28T00:15:00.001+02:002011-08-28T00:15:00.209+02:00Finding stuck or hot pixels in Nikon D80 images<p>In a <a href="http://michelanders.blogspot.com/2011/08/correcting-stuck-or-hot-pixels-in-nikon.html">previous article</a> I presented a tiny Python program that used the Python Image Library (PIL) to correct a so called stuck or hot pixel in an image taken by a digital SLR. In this article we go one step further and see if we can use some simple statistics to identify those pixels automatically.
</p>
<h2>Finding pixels with the least variance</h2>
<p>The idea is to identify those pixels in a bunch of pictures that don't change much from picture to picture, even if the subject and lighting conditions do change. There are of course many ways to define change but the one we explore here is the called the <a href="http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm">variance</a>. Basically we compute for each pixel in a set of pictures its average and then sum (again for each pixel) the differences with its average in each picture. The pixels (or subpixels) that have the smallest sums are likely candidates for being hot or stuck.
</p>
<h2>Using PIL and Numpy for efficient number crunching</h2>
<p>Obviously calculating the variance for each subpixel in a few hundred pictures will entail some serious number crunching if these pictures are from a megapixel camera. We therefore better use a serious number crunching library, like <a href="http://www.scipy.org/">Numpy</a>. Because we use Python 3, I recommend fetching the PIL and Numpy packages from <a href="http://www.lfd.uci.edu/~gohlke/pythonlibs/">Christoph Gohlke's</a> page if you use Windows.
</p>
<p>The code is shown below. The program will open all images given as arguments one by one using PIL's <code>Image.open()</code> function (line 9). PIL images can be directly converted by Numpy by the <code>array()</code> function. Because we might run out of memory we do not keep all images in memory together but process them one by one and calculate the variance using the so called on-line algorithm. The names of the variables used are the same as in <a href="http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#On-line_algorithm">the Wikipedia article</a> but instead of scalars we use arrays. (Note, we do not actually calculate the variance but just the sum of squares of differences from the mean because we interested in the pixel with smallest variance whatever that may be and not in the value of the variance at such.) In the final lines (line 27) we print out the results, retrieving the index of the lowest variance with Numpy's <code>argmin()</code> function.
</p>
<pre name="code" class="python">
import Image
import numpy as np
import sys
from glob import glob
first = True
for arg in sys.argv[1:]:
for filename in glob(arg):
pic = Image.open(filename)
pix = np.array(pic)
if first:
first = False
firstpix = pix
n = np.zeros(firstpix.shape)
mean = np.zeros(firstpix.shape)
M2 = np.zeros(firstpix.shape)
delta = np.zeros(firstpix.shape)
else:
if pix.shape != firstpix.shape:
print("shapes don't match")
continue
n += 1
delta = pix - mean
mean += delta/n
M2 += delta*(pix - mean)
mini = np.unravel_index(M2.argmin(),M2.shape)
print('min',M2[mini],mini)
</pre>
<h2>Results and limitations</h2>
<p>Using a few hundred photo's I could easily identify a previously observed hot pixel. It did take a few minutes though, even on my pc which is a fairly powerful hexacore machine. Besides speed the biggest limitation is that I had to use JPEG images because I did not have RAW images available. JPEG image compression isn't lossless so the hot pixel tends to be smeared out. If you print out not just the pixel with the lowest variance but the ten pixels with the lowest variance, most of them are clustered around the same spot. (example code below).
</p>
<pre name="code" class="python">
sorti = M2.argsort(axis=None)
print(sep="\n",*[(i,M2[i]) for i in [np.unravel_index(i,M2.shape) for i in sorti[:10]]])
</pre>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com2tag:blogger.com,1999:blog-6170632026510659898.post-44327129139954350162011-08-21T00:15:00.016+02:002011-08-24T07:55:38.694+02:00Windows 7 Gadgets: mini web applications<p>Windows 7 gadgets are small applications that are integrated in the Windows desktop. These gadgets are basically webpages displayed in a browser environment that hardly differs from a regular environment. It is therefore entirely possible to turn a gadget into a web application that retrieves data from a remote server with AJAX. In this article we explore some of the possibilities and see how we may use jQuery inside a gadget.</p>
<h2>Anatomy of a Windows 7 gadget</h2>
<p>There is actually some pretty decent documentation on gadgets available on the web (for example
<a href="http://msdn.microsoft.com/en-us/library/ee663278%28v=VS.85%29.aspx">on msdn</a> or
<a href="http://stackoverflow.com/questions/902818/how-to-get-started-with-windows-7-gadgets/1365834#1365834">here</a>). The trick is to get an example gadget running with as little ballast as possible.
</p>
<p>A gadget file is basically a zip file that contains a file <code>gadget.html</code> plus a number of additional files, the most important one, <code>gadget.xml</code>. This zip file does have a <code>.gadget</code> extension instead of a <code>.zip</code> extension. So these simple steps are needed to create a bare bones gadget: </p>
<ol>
<li>Create a directory with a descriptive name, e.g. mygadget</li>
<li>In this directory create a file <code>gadget.html</code></li>
<li> a subdirectory named <code>css</code> with a file <code>mygadget.css</code></li>
<li>a file <code>gadget.xml</code></li>
<li>and finally a file <code>icon.png</code></li>
<li>pack the <em>contents</em> of this directory into a file <code>mygadget.zip</code> (i.e. not the toplevel directory itself)</li>
<li>rename this file to <code>mygadget.gadget</code> (although 7zip for example can pack to a file with any extension directly)</li>
<li>double click this file and follow through the install dialog</a>
</ol>
<p>With the following gadget.html the result will look like the screenshot</p>
<pre name="code" class="html">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>My Gadget</title>
<meta http-equiv="Content-Type" content="text/html; charset=unicode" />
<link href="css/mygadget.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="main_image">
<p>42</p>
</div>
</body>
</html>
</pre>
<img>
<p><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzaYROznfcOyAodcaj2O9pU4ubVi40oQ5SbFpDytMA3CMvGBv2487rUSthh10IrA2-Tx1rcLfeRgmtiN5_JyYKy8ZZA_Mq6icyIsVSQ8Q9BTePi9HJYOXLy2hrzfWp1X11dNQMAO5PGAME/s1600/screenshotgadget.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em"><img border="0" height="200" width="161" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzaYROznfcOyAodcaj2O9pU4ubVi40oQ5SbFpDytMA3CMvGBv2487rUSthh10IrA2-Tx1rcLfeRgmtiN5_JyYKy8ZZA_Mq6icyIsVSQ8Q9BTePi9HJYOXLy2hrzfWp1X11dNQMAO5PGAME/s200/screenshotgadget.png" /></a></div>
Not really impressive, I admit, but it shows how simple it is to create a gadget. The necessary <code>gadget.xml</code> mainly describes were to find the actual html code and what image to use in the gadget selector:
<pre name="code" class="xml">
<?xml version="1.0" encoding="utf-8" ?>
<gadget>
<name>Mygadget</name>
<namespace><!--_locComment_text="{Locked}"-->StartSmall.Gadgets</namespace>
<version><!--_locComment_text="{Locked}"-->1.0</version>
<author name="Michel Anders">
<info url="http://michelanders.blogspot.com" text="Start Small" />
<logo src="icon.png" />
</author>
<copyright><!--_locComment_text="{Locked}"-->&#169; 2011</copyright>
<description>Basic Gadget</description>
<icons>
<icon height="48" width="48" src="icon.png" />
</icons>
<hosts>
<host name="sidebar">
<base type="HTML" apiVersion="1.0.0" src="gadget.html" />
<permissions>
<!--_locComment_text="{Locked}"-->Full
</permissions>
<platform minPlatformVersion="1.0" />
</host>
</hosts>
</gadget>
</pre>
</p>
<h2>Using jQuery in a Windows 7 gadget</h2>
<p>Displaying static information is not much fun at all so let's see what options there are to create a more dynamic gadget:
<ul>
<li>Refer to external content like images that get refreshed</li>
<li>Refresh the content ourselves using JavaScript, possibly even interacting with the user</li>
</ul>
The first option is simple enough and we won't cover that here. The second option is more interesting because it opens up a whole array of possibilities. Let's have a look at how a gadget.html page should look to incorporate jQuery and how we can test if this works:
<pre name="code" class="html">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>My Gadget</title>
<meta http-equiv="Content-Type" content="text/html; charset=unicode" />
<link href="css/mygadget.css" rel="stylesheet" type="text/css" />
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js" type="text/javascript"></script>
</head>
<body>
<div id="main_image">
<p>42</p>
</div>
<script type="text/javascript">$("#main_image p").append('<span> 43 44 45 </span>');</script>
</body>
</html>
</pre>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSXgc05GdEkebrpA_bsdTDS_zPRTbthx6a0dWBt5lRxE2V_T1z0VSeICLpHDxuU35Y5GjuSyZ-QxK7-GVAKca7ggGHCLJ4y0C_KInyDncVl4fqQ6IPuSZcrhentoltIb9Txvu9KwLFDssO/s1600/screenshotgadget2.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em"><img border="0" height="200" width="147" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSXgc05GdEkebrpA_bsdTDS_zPRTbthx6a0dWBt5lRxE2V_T1z0VSeICLpHDxuU35Y5GjuSyZ-QxK7-GVAKca7ggGHCLJ4y0C_KInyDncVl4fqQ6IPuSZcrhentoltIb9Txvu9KwLFDssO/s200/screenshotgadget2.png" /></a></div>
As you can see this is surprisingly simple. The screenshot proves that the final lines of code are actually executed and change the contents of our basic gadget:
<img>
Our next step is to check if we can retrieve data from a server.
</p>
<h2>JSONP in a Windows 7 gadget</h2>
<p>Due to the <a href="http://en.wikipedia.org/wiki/Same_origin_policy">same origin policy</a> it is not entirely straight-forward to retrieve data from a server different from the server we get our webpage from. In the gadget environment <em>every</em> server is considered a different server because the gadget.html file originates from a file system. This means that even if we access a web application server on the same pc, we will be denied access. </p>
<p>Fortunately there is a workaround available in the form of <a href="http://en.wikipedia.org/wiki/JSONP">JSONP</a> and jQuery makes it very simple for use to use this. Consider the following gadget.html:</p>
<pre name="code" class="html">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>My Gadget</title>
<meta http-equiv="Content-Type" content="text/html; charset=unicode" />
<link href="css/mygadget.css" rel="stylesheet" type="text/css" />
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js" type="text/javascript"></script>
</head>
<body>
<div id="main_image">
<p>42</p>
</div>
<script type="text/javascript">
function Refresh() {
$.ajax('http://127.0.0.1:8088/value',
{dataType:'jsonp',scriptCharset: "utf-8",
success:function(data, textStatus, jqXHR){
$("#main_image").empty().append('<p>'+String(data)+'</p>');
}});
}
window.setInterval("Refresh()", 5000);
</script>
</body>
</html>
</pre>
<p>It will contact an application server every 5 seconds to retrieve a number and will insert this number into the content div. That's all there is to it. The application server that serves these request is equally simple and can be build for example with the applicationserver module from <a href="http://michelanders.blogspot.com/2011/08/function-annotations-in-python-checking.html">a previous article</a>:</p>
<pre name="code" class="python">
from http.server import HTTPServer, BaseHTTPRequestHandler
from json import dumps
from applicationserver import IsExposed,ApplicationRequestHandler
class Application:
def value(self,callback:str,_:int=0) -> IsExposed:
r = "%s(%s);"%(callback,dumps(_ % 17))
return r
class MyAppHandler(ApplicationRequestHandler):
application=Application()
appserver = HTTPServer(('',8088),MyAppHandler)
appserver.serve_forever()
</pre>
<p>The only trick here is that JSONP requests pass an additional paramter (usually called <code>callback</code> that will hold a string (the name of a JavaScript function in the client) and that we have to use this parameter to contruct a return value that looks like a call to this function with the JSON encoded data as its argument (line 8). The <code>_</code> parameter holds a random number added by JQuery to prevent caching by the browser. So if <code>_</code> happened to be <code>123</code> and <code>callback</code> would be <code>jquery456_123</code> the value we woudl return would be the string <code>jquery456_123(4);</code></p>
<p>The data we return here is merely a random number but of course this could be anything and we could style it to be presented in a more readable form.</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com4tag:blogger.com,1999:blog-6170632026510659898.post-82714626004713662652011-08-14T00:15:00.016+02:002011-08-14T00:15:01.512+02:00Correcting stuck or hot pixels in Nikon D80 images<p>A friend of mine became suddenly aware that her Nikon D80 camera had developed a so called stuck or hot pixel. Repairing a CCD chip in a camera is quite costly and what is more, looking back through the old photos she discovered the defect had been present for over a year. In this article we develop a tiny Python program that masks out a pixel in pretty much the same way that the built-in firmware in de camera does this for pixels that were hot or dead when the camera was constructed.
</p>
<h2>Correcting stuck or hot pixels in images</h2>
<p>Hot or stuck pixels in a camera are annoying and other than replacing the ccd chip, nothing can be done about it with regard to the hardware. More on this in this
<a href="http://webpages.charter.net/bbiggers/DCExperiments/html/hot_pixels.html">illuminating article</a>. Sometimes the camera can map out bad pixels but that won't fix old images, so the plan is to create a small program that does this on existing images.
</p><p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigtyutDu8nGmp0n46sOIgkzXJSqz-RIfbRd7dh4RrzOtWzNpAxCczhvPK7qOx-ir3c-DD8gqwT51Yz_7QeKoUKo9d84h7DROVK1_wDOi-zC51NgReekeaAW4jhY6zp-o9rgy5twy4FXrv5/s1600/weights.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em"><img border="0" height="200" width="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigtyutDu8nGmp0n46sOIgkzXJSqz-RIfbRd7dh4RrzOtWzNpAxCczhvPK7qOx-ir3c-DD8gqwT51Yz_7QeKoUKo9d84h7DROVK1_wDOi-zC51NgReekeaAW4jhY6zp-o9rgy5twy4FXrv5/s200/weights.png" /></a></div>
The idea is to replace the offending pixel with a weighted average of the neighboring pixels so it will blend in with the surroundings. Diagonally adjacent pixels have a lower weight then pixels horizontally or vertically adjacent (see picture on the left). Such a program is simple enough to implement using
<a href="http://www.pythonware.com/products/pil/index.htm">PIL, the Python Imaging Library</a>. There is even
<a href="http://www.lfd.uci.edu/~gohlke/pythonlibs/">a version for Python 3</a> made available by Christoph Gohlke, although you will have to replace all relative import statements with absolute one if you want that to work. (Simply run the the installer, open all <code>*.py</code> files in site-packages/PIL and replace all occurrences of <code>from . import</code> with a simple <code>import</code>. A decent editor like notepad++ or ultraedit can manage that in one go).
</p>
<p>The program will take the name of an image file and the x,y position of the pixel to fix as its arguments, e.g. <code>python correctpixel.py myimage.jpg 1500,1271</code> and will produce a corrected image with the same name but with a <code>c_</code> prefix added, in our example <code>c_myimage.jpg</code>. It will use a weighted average of the eight neighboring pixels as depicted in this image (the diagonally adjacent pixels have a weight of <code>1/sqrt(2)</code>.
</p><p>The simplest way to find the exact location of the pixel to fix is to open the image in Gimp, locate the pixel and hover over it with the mouse: the location is shown in the status bar at the bottom. Depending on your zoom level these may be fractional numbers, but we need just the integral parts.
</p><p>The program itself is rather straightforward and is shown in its entirety below:
<pre name="code" class="python">
import sys
import Image
def correct(im,xy,matrix):
pim = im.load()
x,y = xy
maxx,maxy = im.size
n = 0
sumr,sumg,sumb = 0,0,0
for dx,dy,w in matrix:
px,py = x+dx,y+dy
if px<0 or py<0 or px >= maxx or py >= maxy:
break
n += w
r,g,b = pim[px,py]
sumr,sumg,sumb = sumr+r*w,sumg+g*w,sumb+b*w
pim[x,y]=(int(sumr/n),int(sumg/n),int(sumb/n))
w=1/(2**0.5)
matrixd=((0,1,1),(0,-1,1),(1,0,1),(-1,0,1),(-1,-1,w),(-1,1,w),(1,1,w),(1,-1,w))
im = Image.open(sys.argv[1])
xy = tuple(map(int,sys.argv[2].split(',')))
correct(im,xy,matrixd)
im.save('c_'+sys.argv[1],quality=97)
</pre>
Given an opened image the function <code>correct()</code> will load the data and the loop of the list of pixels offsets and weights given in its <code>matrix</code> argument. It will check whether a neighbor is within the bounds of the image (line 12; if the stuck pixel is on an edge it might not be) and if so, sum its RGB components according to the weight of this pixel. The pixel is finally replaced by summed values divided by the number of summed pixels. We also take care to convert the RGB components to integers again (line 17Y).
</p><p>The final lines take care of opening the image (line 22), and converting the pixel position argument to a tuple of integers before calling the <code>correct()</code> function. The image is saved again with a name prefixed with <code>c_</code>. The <code>quality</code> parameter is set to a very high value to more or less save the image with the same jpeg quality a the original image.
</p><p>Note that at the moment the code shown works for jpeg images only (or to be precise, only for images with three bands, R,G,B.) Png images may have and extra transparency band and the code does not deal with that nor does it play nice with indexed formats (like gif).
</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com2tag:blogger.com,1999:blog-6170632026510659898.post-11095879331601617972011-08-07T00:15:00.016+02:002011-08-07T00:15:00.661+02:00Function annotations in Python, checking parameters in a web application server , part II<p>In a <a href="http://michelanders.blogspot.com/2011/07/function-annotations-in-python-checking.html"> previous article</a> I illustrated how we could put Python's function annotations to good use to provide use with a syntactically elegant way to describe which kind of parameter content would be acceptable to our simple web application framework. In this article the actual implementation is presented along with some notes on its usage.
</p>
<h2>A simple Python application server with full argument checking</h2>
<p>The <code>applicationserver</code> module provides a <code>ApplicationRequestHandler</code> class that derives from the Python provided <code>BaseHTTPRequestHandler</code> class. For now all we do is provide a <code>do_GET()</code> method, i.e. we don't bother with HTTP POST methods yet, although it wouldn't take too much effort if we would.
</p><p>
The core of the code is presented below:
<pre name="code" class="python">
class ApplicationRequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
url=urlsplit(self.path)
path=url.path.strip('/')
parts=path.split('/')
if parts[0]=='':
parts=['index']
ob=self.application
try:
for p in parts:
if hasattr(ob,p):
ob=getattr(ob,p)
else:
raise AttributeError('unknown path '+ url.path)
except AttributeError as e:
self.send_error(404,str(e))
return
if ('return' in ob.__annotations__ and
issubclass(ob.__annotations__['return'],IsExposed)):
try:
kwargs=self.parse_args(url.query,ob.__annotations__)
except Exception as e:
self.send_error(400,str(e))
return
try:
result=ob(**kwargs)
except Exception as e:
self.send_error(500,str(e))
return
else:
self.send_error(404,'path not exposed'+ url.path)
return
self.wfile.write(result.encode())
@staticmethod
def parse_args(query,annotations):
kwargs=defaultdict(list)
if query != '':
for p in query.split('&'):
(name,value)=p.split('=',1)
if not name in annotations:
raise KeyError(name+' not annotated')
else:
kwargs[name].append(annotations[name](value))
for k in kwargs:
if len(kwargs[k])==1:
kwargs[k]=kwargs[k][0]
return kwargs
</pre>
The first thing we do in the <code>do_GET()</code> method is splitting the path into separate components. The path is provided in the <code>path</code> member and is already stripped of hostname and query parameters. We strip from it any leading or trailing slashes as well (line 5). If there are no path components we will look for a method called <code>index</code>.
</p><p>The next step is to check each path component and see if its a member of the application we registered with the handler. If it is, we retrieve it and check whether the next component is a member of this new object. If any of these path components is missing we raise an error (line 16).
</p><p>If all parts of the path can be resolved as members, we check whether the final path points to an executable with a return allocation. If this return allocation is defined and equal to our <code>IsExposed</code> class (line 22) we are willing to execute it, otherwise we raise an error.
</p><p>
The next step is to check each argument, so we pass the query part of the URL to the static <code>parse_args()</code> method that will return us a dictionary of values if all values checked out ok. If so we call the method that we found earlier with these arguments and if all went well, write its result to the output stream that will deliver the content to the client.
</p><p>The <code>parse_args()</code> method is not very complicated: It creates a default dictionary whose default will be an empty list. This way we can create a list of values if the query consists of more than one part with the same name. Then we split the query on the & character (line 44) and split each part in a name and a value part (these are separated by a = character). Next we check if the name is present in the <code>annotations</code> dictionary and if not raise a <code>KeyError</code>.
</p><p>If we did find the name in the <code>annotations</code> dictionary its associated value should be an executable that we pass the value from the query part (line 48). The result of this check (or conversion) is appended to the list in the default dictionary. If the value in the annotation is not an executable an exception will be raised that will not be caught. Likewise will any exception within this callable bubble up to the calling code. The final lines (line 50-53) reduce those entries in the default dictionary that consist of a list with just a single item to just that item before returning.
</p>
<h2>Conclusion</h2>
<p>Python function annotations can be used for many purposes and this example code show a rather elegant (I think) example that let's us specify in clear language what we expect of functions that are part of web applications, which makes it quite easy too catch many input errors in an way that can be adapted to almost any input pattern.
</p>Michelhttp://www.blogger.com/profile/07415131908018321030noreply@blogger.com0