during a short holiday break I did some reading on Web2py and my first impression is that that it looks really good. It seems to be much more complete and data centric than CherryPy. I think I will spend some time experimenting with it the coming months.
Showing posts with label CherryPy. Show all posts
Showing posts with label CherryPy. Show all posts
Windows 7 Gadgets: mini web applications
Windows 7 gadgets are small applications that are integrated in the Windows desktop. These gadgets are basically webpages displayed in a browser environment that hardly differs from a regular environment. It is therefore entirely possible to turn a gadget into a web application that retrieves data from a remote server with AJAX. In this article we explore some of the possibilities and see how we may use jQuery inside a gadget.
Anatomy of a Windows 7 gadget
There is actually some pretty decent documentation on gadgets available on the web (for example on msdn or here). The trick is to get an example gadget running with as little ballast as possible.
A gadget file is basically a zip file that contains a file gadget.html plus a number of additional files, the most important one, gadget.xml. This zip file does have a .gadget extension instead of a .zip extension. So these simple steps are needed to create a bare bones gadget:
- Create a directory with a descriptive name, e.g. mygadget
- In this directory create a file
gadget.html - a subdirectory named
csswith a filemygadget.css - a file
gadget.xml - and finally a file
icon.png - pack the contents of this directory into a file
mygadget.zip(i.e. not the toplevel directory itself) - rename this file to
mygadget.gadget(although 7zip for example can pack to a file with any extension directly) - double click this file and follow through the install dialog
With the following gadget.html the result will look like the screenshot
<html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>My Gadget</title> <meta http-equiv="Content-Type" content="text/html; charset=unicode" /> <link href="css/mygadget.css" rel="stylesheet" type="text/css" /> </head> <body> <div id="main_image"> <p>42</p> </div> </body> </html>
gadget.xml mainly describes were to find the actual html code and what image to use in the gadget selector:
<?xml version="1.0" encoding="utf-8" ?>
<gadget>
<name>Mygadget</name>
<namespace><!--_locComment_text="{Locked}"-->StartSmall.Gadgets</namespace>
<version><!--_locComment_text="{Locked}"-->1.0</version>
<author name="Michel Anders">
<info url="http://michelanders.blogspot.com" text="Start Small" />
<logo src="icon.png" />
</author>
<copyright><!--_locComment_text="{Locked}"-->© 2011</copyright>
<description>Basic Gadget</description>
<icons>
<icon height="48" width="48" src="icon.png" />
</icons>
<hosts>
<host name="sidebar">
<base type="HTML" apiVersion="1.0.0" src="gadget.html" />
<permissions>
<!--_locComment_text="{Locked}"-->Full
</permissions>
<platform minPlatformVersion="1.0" />
</host>
</hosts>
</gadget>
Using jQuery in a Windows 7 gadget
Displaying static information is not much fun at all so let's see what options there are to create a more dynamic gadget:
- Refer to external content like images that get refreshed
- Refresh the content ourselves using JavaScript, possibly even interacting with the user
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>My Gadget</title>
<meta http-equiv="Content-Type" content="text/html; charset=unicode" />
<link href="css/mygadget.css" rel="stylesheet" type="text/css" />
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js" type="text/javascript"></script>
</head>
<body>
<div id="main_image">
<p>42</p>
</div>
<script type="text/javascript">$("#main_image p").append('<span> 43 44 45 </span>');</script>
</body>
</html>
JSONP in a Windows 7 gadget
Due to the same origin policy it is not entirely straight-forward to retrieve data from a server different from the server we get our webpage from. In the gadget environment every server is considered a different server because the gadget.html file originates from a file system. This means that even if we access a web application server on the same pc, we will be denied access.
Fortunately there is a workaround available in the form of JSONP and jQuery makes it very simple for use to use this. Consider the following gadget.html:
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>My Gadget</title>
<meta http-equiv="Content-Type" content="text/html; charset=unicode" />
<link href="css/mygadget.css" rel="stylesheet" type="text/css" />
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js" type="text/javascript"></script>
</head>
<body>
<div id="main_image">
<p>42</p>
</div>
<script type="text/javascript">
function Refresh() {
$.ajax('http://127.0.0.1:8088/value',
{dataType:'jsonp',scriptCharset: "utf-8",
success:function(data, textStatus, jqXHR){
$("#main_image").empty().append('<p>'+String(data)+'</p>');
}});
}
window.setInterval("Refresh()", 5000);
</script>
</body>
</html>
It will contact an application server every 5 seconds to retrieve a number and will insert this number into the content div. That's all there is to it. The application server that serves these request is equally simple and can be build for example with the applicationserver module from a previous article:
from http.server import HTTPServer, BaseHTTPRequestHandler
from json import dumps
from applicationserver import IsExposed,ApplicationRequestHandler
class Application:
def value(self,callback:str,_:int=0) -> IsExposed:
r = "%s(%s);"%(callback,dumps(_ % 17))
return r
class MyAppHandler(ApplicationRequestHandler):
application=Application()
appserver = HTTPServer(('',8088),MyAppHandler)
appserver.serve_forever()
The only trick here is that JSONP requests pass an additional paramter (usually called callback that will hold a string (the name of a JavaScript function in the client) and that we have to use this parameter to contruct a return value that looks like a call to this function with the JSON encoded data as its argument (line 8). The _ parameter holds a random number added by JQuery to prevent caching by the browser. So if _ happened to be 123 and callback would be jquery456_123 the value we woudl return would be the string jquery456_123(4);
The data we return here is merely a random number but of course this could be anything and we could style it to be presented in a more readable form.
Function annotations in Python, checking parameters in a web application server
Parameter annotations in function definitions are a recent addition to the language. In this article we show how this feature can be put to good use when we build a simple web application server that checks its input parameters rigoreously.
Creating a simple web application server with HTTPServer
It is of course entirely possible to create a web application in a short time when you use an existing Python web application framework. In previous articles and my book on web applications I've used CherryPy extensively and although I recommend it for its flexibility an ease of use, it isn't all that difficult to create a web application framework from scratch.
Python's https.server module provides us with the basic building blocks: the HTTPServer class to handle incoming connections and a BaseHTTPRequestHandler class that processes requests and returns an answer. The main part of developing an application server is therefore sub-classing the BaseHTTPRequestHandler class. The minimum it will have to provide is a do_GET() method that will return results based on any parameters it receives.
Using Python parameter annotations
CherryPy uses classes with methods that serve together as an application: requested URLs are mapped to these methods and any parameters are passed along. CherryPy uses an expose decorator to identify the methods that may be called. Non exposed methods are invisible, i.e. URLs that match those methods do not result in the invocation of that method. This behavior is what we like to mimic in our own web application server.
Another important concept in web applications is the screening of input: we would like to check that incoming data (i.e. the arguments that come along with a query) are within the range of things we deem acceptable. For example, a function that adds its arguments should reject anything that cannot be interpreted as a float. We could write code easily enough that checks any function parameters explicitly but wouldn't it be nice if there was a more syntactically pleasing way of writing this?
Enter Python's function parameter annotations. Python allows us to augment each function parameter with an expression that is evaluated when the function is defined and which is stores in the functions __annotation__ field as a dictionary indexed by parameter name. Such an annotation might be as simple a single string but it can be anything, even a function reference. This function could be called with the value we would like to pass as an parameter to check if this value is ok. This could be done before the function is actually called, for example by the do_GET() method of our request handler.
Assuming we have our applicationserver module available, let's have a look what the definition of a new web application might look like:
from http.server import HTTPServer
from applicationserver import ApplicationRequestHandler,IsExposed
class Application:
def donothing:
pass
def index(self) -> IsExposed:
return 'index oink'
def add(self,a:float,b:float) -> IsExposed:
return str(a+b)
def cat(self,a:str) -> IsExposed:
return ' '.join(a)
def opt(self,a:int=42) -> IsExposed:
return str(a)
class MyAppHandler(ApplicationRequestHandler):
application=Application()
appserver = HTTPServer(('',8088),MyAppHandler)
appserver.serve_forever()
The overall idea is to subclass the ApplicationRequestHandler and assign an instance of the Application class to its application field (line 21). This applicationhandler is then passed to a HTTPServer instance that will forward incoming requests to this handler (line 24).
Our ApplicationRequestHandler will try to map URLs of the form http://hostname:8080/foo?a=1&b=2 to member functions of the Application instance. It will only consider member function with a return annotation equal to an IsExposed object. So even though we have defined a donothing() function, it will not be executed when a URL like http://hostname:8080/donothing is received.
We also use annotations to restrict input values for parameters to functions that are exposed. Remember that annotations can be any expression and here we employ that fact to annotate the a and b parameters to the add() method with a reference to the built-in float() function. Our ApplicationRequestHandler will pass an argument to any callable it finds as its corresponding annotation and will only execute the method if this callable returns a value (and not raise an exception). So a URL like http://localhost:8080/add?a=1.23&b=4.56 will return a meaningful result while http://localhost:8080/add?a=1.23&b=spam will fail with an error. Off course we are not restricted to built-in functions here: we can refer to functions that may perform elaborate checking as well, perhaps checking against regular expressions or performing lookups in database tables.
All this shows that Python's function annotations allow for a rather elegant way to describe the expected behavior of methods that perform some sort of action in a web application. In a future article I'll show how to implement the applicationserver module.
A SQLite multiprocessing proxy, part 2
In a previous article we decided to use Python's multiprocessing module to leverage the power of multi-core machines. Our use case is all about web applications served by CherryPy and so multi-processing isn't the only interesting part: our application will be multi-threaded as well. IN this article we present a first implementation of a multi-threaded application that hands off the heavy lifting to a pool of subprocesses.
The design
The design is centered on the following concepts:
- The main process consists of multiple threads,
- The work is done by a pool of subprocesses,
- Transferring data to and from the subprocesses is left to the pool manager
Sample code
We start of by including the necessary components:
from multiprocessing import Pool,current_process from threading import current_thread,Thread from queue import Queue import sqlite3 as dbapi from time import time,sleep from random import randomThe most important ones we need are the
Pool class from the multiprocessing module and the Thread class from the threading module. We also import queue.Queue to act as a task list for the threads. Note that the multiprocessing module has its own Queue implementation that is not only thread safe but can be used for inter process communication as well but we won't be using that one here but rely on a simpler paradigm as we will see.
The next step is to define a function that may be called by the threads.
def execute(sql,params=tuple()): global pool return pool.apply(task,(sql,params))It takes a string argument with SQL code and an optional tuple of parameters just like the
Cursor.execute() method in the sqlite3 module. It merely passes on these arguments to the apply() method of the multiprocessing.Pool instance that is referred to by the global pool variable. Together with SQL string and parameters a reference to the task() function is passed, which is defined below:
def task(sql,params): global connection c=connection.cursor() c.execute(sql,params) l=c.fetchall() return lThis function just executes the SQL and returns the results. It assumes the global variable
connection contains a valid sqlite3.Connection instance, something that is taken care of by the connect function that will be passed as an initializer to any new subprocess:
def connect(*args): global connection connection = dbapi.connect(*args)
Before we initialize our pool of subprocess let's have a look at the core function of any thread we start in our main process:
def threadwork(initializer=None,kwargs={}):
global tasks
if not ( initializer is None) :
initializer(**kwargs)
while(True):
(sql,params) = tasks.get()
if sql=='quit': break
r=execute(sql,params)
It calls an optional thread initializer first and then enters a semi infinite loop in line 5. This loops starts by fetching an item from the global tasks queue. Each item is a tuple consisting of a string and another tuple with parameters. If the string is equal to quit we do terminate the loop otherwise we simple pass on the SQL statement and any parameters to the execute function we encountered earlier, which will take care of passing it to the pool of subprocesses. We store the result of this query in the r variable even though we do nothing with it in this example.
For this simple example we also need an database that holds a table with some data we can play with. We initialize this table with rows containing random numbers. When we benchmark the code we can make this as large as we wish to get meaningful results; after all, our queries should take some time to complete otherwise there would be no need to use more processes.
def initdb(db,rows=10000):
c=dbapi.connect(db)
cr=c.cursor()
cr.execute('drop table if exists data');
cr.execute('create table data (a,b)')
for i in range(rows):
cr.execute('insert into data values(?,?)',(i,random()))
c.commit()
c.close()
The final pieces of code tie everything together:
if __name__ == '__main__':
global pool
global tasks
tasks=Queue()
db='/tmp/test.db'
initdb(db,100000)
nthreads=10
for i in range(100):
tasks.put(('SELECT count(*) FROM data WHERE b>?',(random(),)))
for i in range(nthreads):
tasks.put(('quit',tuple()))
pool=Pool(2,connect,(db,))
threads=[]
for t in range(nthreads):
th=Thread(target=threadwork,kwargs={'initializer':thread_initializer})
threads.append(th)
th.start()
for th in threads:
th.join()
After creating a queue in line 5 and initializing the database in line 8, the next step is to fill a queue with a fair number of tasks (line 12). The final tasks we add to the queue signal a thread to stop (line 14). We need as many of them as there will be threads.
In line 17 we initialize our pool of processes. Just two in this example, but in general the number should be equal to the number of cpu's in the system. If you omit this argument the number will default to exactly that. Next we create (line 21) and start (line 23) the number of threads we want. The target argument points to the function we defined earlier that does all the work, i.e. pops tasks from the queue and passes these on to the pool of processes. The final lines simply wait till all threads are finished.
What's next?
In a following article we will benchmark and analyze this code and see how we can improve on this design.
A SQLite multiprocessing proxy
This is the first article in a series on improving the performance of Python web applications by leveraging the possibilities of the multiprocessing module. We'll focus on CherryPy and SQLite but the conclusions should be general enough for any Python based platform
Use case
Due to well known restrictions in the most common Python implementation, multithreading solutions will probably not help to solve performance issues (with the possible exception of serving slow network connections). The multiprocessing module offers an API similar to the threading module and might be an alternative when we want to divide the workload on a multicore machine.
The use case we're interested in is a CherryPy server that serves many requests, backed by a SQLite database. CherryPy is multithreaded by design and this approach is sensible as a web server may spend more time waiting for data to be transmitted over relatively slow network connections than actually doing work.
CherryPy however is also an excellent framework to host web applications and many web applications rely on some sort of database back-end. SQLite is a good choice for such a back-end as it comes bundled with Python (reducing the number of external dependencies), is easy to use and performs well enough. With some tricks it will even play nice in a multithreaded environment.
A disadvantage of using SQLite is that we do not have a separate database server: the SQLite engine is part of the same process that runs the Python interpreter. This means that it has the same handicap as any multithreaded application on CPython (the most common implementation of Python) and will not benefit from any extra cores or processors available on the server.
Now we could switch to MySQL or any other stand-alone database back-end but this would add quite an amount to the maintenance burden of our web application. Wouldn't it be nice if we could devise a way to use SQLite together with the multiprocessing module to have the best of both worlds: the ease of use of SQLite and the performance benefits of a stand-alone database server?
In this series of articles I will explore the possibilities and hopefully will come up with a solution that will provide:
- a dbapi proxy (we'll use
sqlite3module but it should be general enough for any dbapi compliant database) - that will use the multiprocessing module to increase performance and
- can be used from a multithreaded environment.
In the next article in this series I will explore the options to make threads and processes play nice, focusing on inter process communication.
Python and Javascript, using the Flot plugin, part 6
In the sixth installment of this series we look at how we can make the graph more interactive by making full use of the extensibility of the Flot plugin.
Creating interactive plots
The Flot plugin mimics most other jQueryUI plugins in that although it works perfectly well without any configuration options it also provides a number of ways to extend its functionality. Here we will see how we may attach a hovering label to any point in the graph, something that may prove useful if the graph consists of many points. An example is shown in this image (the mousepointer itself is not visible in this screenshot):
var p = $.plot($("#tempandwind") ,da ,{
series: { lines: { show: true }, points: { show: true } },
xaxis : { mode:"time",ticks:12,twelveHourClock:false},
y2axis: { position:"right"},
grid : { hoverable: true }
});
$("#tempandwind").bind("plothover",highlight);
The Flot plugin does not generate the custom plothover events by defaults so we need to turn that on as can be seen in line 5. With plothover events enables we can now bind a function to this event as can be seen in the last line.
The Flot website shows a number of interesting examples on how to enhance an extend the Flot plugin. The highlight() we define here is a slightly adapted version of some of the Javascript behind this example. Lets have a look at our implementation:
var previousPoint = null;
function highlight(event, pos, item) {
if (item) {
if (previousPoint != item.dataIndex) {
previousPoint = item.dataIndex;
$("#tooltip").remove();
var y = item.datapoint[1].toFixed(2);
showTooltip(item.pageX, item.pageY,
item.series.label + " = " + y);
}
} else {
$("#tooltip").remove();
previousPoint = null;
}
};
Any function bound to the plothover event is passed three arguments: an event object, the position in the canvas and an item argument that holds graph specific data, i.e. the x and y values of the datapoint that we are hovering above, its coordinates in the canvas and the label of the dataseries that the point belongs to. If item is null or undefined we are not hovering above a datapoint. We check for this in the code and remove the tooltip if so.
Because we do not want to redraw the hovering information if we are still above the same data point we check if the index in the series of datapoints is different from the one we stored in the global variable previousPoint. If this is the case, we remove the tooltip, convert the value of the datapoint to a number with only two decimals to keep things readable and call the showTooltip() function to draw the a label at to correct position.
function showTooltip(x, y, contents) {
$('' + contents + '').css( {
position: 'absolute',
display: 'none',
top: y + 5,
left: x + 5,
border: '1px solid #fdd',
padding: '2px',
'background-color': '#fee',
opacity: 0.80
}).appendTo("body").fadeIn(200);
};
The showTooltip() function is straight from the example on the Flot page but deserves some explanation because it shows off some nifty jQuery features.
It creates a div element ans styles it directly with the css() method. Its display attribute is initially none as the element is added to the body element with the appendTo() method but is made gradually visible with the fadeIn() method. Each of these methods returns the element it is operating on so all methods can be chained.
Python and Javascript: using the Flot plugin, part 5
Using the Flot plugin
In article 5 of this series we look at how we may actually configure the Flot plugin to show the data.
The Javascript code: minimalweather.js
Let's have a look at a rather minimal implementation of the client side of our app. It will only show temperature and wind speed but it is a rather good example of what is possible. The Javascript code is encapsulated in a jQuery $(document).ready() function. This way we ensure that we only convert HTML elements to jQuery widget when we're absolutely sure they are present.
The first step we take is setting some general AJAX parameters. We set cache to false which will instruct jQuery to add a _ (underscore) parameter to every AJAX call. This parameter will have a random value and this will make the URL different each time, thereby preventing the browser to cache result. After all we are not interested in stale weather data. We also set async to false. AJAX calls normally return immediately but signal completion to a function that is given to them as a parameter. However, because our graphs consist of multiple dataseries we want to retrieve the data from more than one datasource. To draw the complete graph we need all data to be available, so by setting async to false we don't start doing anything else unless the last AJAX call is finished. This is a bit against the grain (after all the first A in AJAX stands for asynchronous) but it does make out code simpler.
The second task is to set a default for the reporting period and level of detail in the graphs. The period might be a day, a week or a month and the default level of detail is a average value per hour. PyWWS compatible weather stations are normally capable of logging in a much finer detail and that is what we retrieve if detail is set to raw. (My weather station can log a value every five minutes).
Next we define two functions: hourly(), that will issue two getJSON() calls to retrieve temperature and windspeed averages over the last hour and raw() that will do the same but for 5 minute intervals. Because we designed the server side of the application to produce JSON serialized data all the hard work of converting this data to arrays of timestamp/value pairs in a safe way is done by the getJSON() function which will pass the result as the data argument to the function it calls on completion.
$(document).ready(function(){
$.ajaxSetup({cache:false,async:false});
var temp_out;
var wind_ave;
var period="day";
var detail="hourly";
function hourly(){
$.getJSON('./hourly/temp_out' ,{"period":period},
function(data){ temp_out = data; });
$.getJSON('./hourly/wind_ave' ,{"period":period},
function(data){ wind_ave = data; });
}
function raw(){
$.getJSON('./raw/temp_out' ,{"period":period},
function(data){ temp_out = data; });
$.getJSON('./raw/wind_ave' ,{"period":period},
function(data){ wind_ave = data; });
}
Now that we have functions in place to retrieve data we need something to convert these two data sets (temperature and wind speed) to an object that can be passed as an argument to the Flot plugin. That is what the setdata() function does: it creates an object that defines two dataseries with an appropriate label. It also defines a second y-axis to use by the wind speed dataseries. We initialize the da variable to hourly data.
var da;
function setdata(){
da = [
{data:temp_out,label:"temperature"},
{data:wind_ave,label:"wind",yaxis:2}
];
};
hourly();
setdata();
With the data present and a data argument ready we can finally convert the div with the tempandwind id to a graph. The flot plugin is a little different from most plugins as it does not provide a member function on any jQuery selection (unlike for example the button plugin which can be called as $("#mybutton").button() ). The Flot plugin just provides a plot() function which is passed a jQuery selection as its first argument. The second argument is an object which describes the data series and the final argument is an option object. Here we configure all series to show lines as well as points and configure the x-axis to behave as a time axis with a maximum of twelve tick marks. If it decides to show hours as ticks we inform it to use a 24 hour clock (it might show days as well, it decides that automatically although this may be set explicitly). Note that the width and height of the HTML element that we want to convert to a graph must be set explicitly beforehand. We have take care of that in the HTML contained in the basepage.html file.
var p = $.plot($("#tempandwind") ,da ,{
series: { lines: { show: true }, points: { show: true } },
xaxis : { mode:"time",ticks:12,twelveHourClock:false},
y2axis: { position:"right"}
});
We will also configure some buttons to let the user interact with the graph so we must define some way to reload and redisplay data. We therefore define a replot() function which will retrieve either hourly data or detailed data based on the contents of the detail variable and then create a new data description object. We then use the setData() method of the Flot plugin to indicate we have new data and its setupGrid() method to recalculate things like axes and ticks. The graph is then redrawn by calling the draw() method.
function replot(){
if (detail == "hourly") {
hourly();
}else{
raw();
}
setdata();
p.setData(da);
p.setupGrid();
p.draw();
};
Interaction with the graph is now a simple matter of binding functions to click events. These functions set either the detail or the period variable to a suitable value and then call the replot() function.
$("#d2").click(function(){
detail="raw";
replot();
});
$("#d1").click(function(){
detail="hourly";
replot();
});
$("#p1").click(function(){
period="day";
replot();
});
$("#p2").click(function(){
period="week";
replot();
});
$("#p3").click(function(){
period="month";
replot();
});
What is left (although we could have done it much earlier) is to style the tabs and buttons with regular jQueryUI widgets:
$("#tabs").tabs();
$("#detail").buttonset();
$("#period").buttonset();
});
The result of all this work is a functional web application that shows temperature and wind speed on its first tab:
And it is interactive of course: Clicking the week button for example results in this overview:
Of course we there is a lot more we can do but that is covered in coming articles.
Other parts of this series
Python and Javascript: using the Flot plugin, part 4
First steps in creating a web application
It is nice to have some modules that serve up weather data but it is of course not enough. In this article we show how to use those modules as components in a CherryPy application. We also see what the HTML looks like that we use to structure the information in the web application and load all necessary Jascript files and supporting CSS.
Serving a CherryPy application
The first steps in setting up our web application is importing the cherrypy module and the classes from the weather package that we created earlier:
import os
current_dir = os.path.dirname(os.path.abspath(__file__))
import cherrypy
from weather.services import Hourly,Raw,Monthly
basepage = "".join(open(os.path.join(current_dir,'basepage.html')).readlines())
In the last line we also read in a file called basepage.html that we will look at later as it forms the basis of our application (we use a separate file so that we don't have to mix to much Python and HTML. Syntax highlighters don't like that). The next step is to configure the tree of URLs that serve the different kinds of data:
class Root:
hourly = Hourly('/home/michel/sitescripts/pywws/weatherdata')
raw = Raw('/home/michel/sitescripts/pywws/weatherdata')
monthly= Monthly('/home/michel/sitescripts/pywws/weatherdata')
@cherrypy.expose
def index(self):
return basepage
We configure the server to listen on any address and by default it will listen on port 8080. If you need another port you may configure this with the server.socket_port option.
cherrypy.config.update({'global':{'server.socket_host':'0.0.0.0'}})
Finally we start the CherryPy server by passing an instance of the Root class we defined to serve the application to the quickstart() function. We pass in additional configuration items to make sure we have a log file in a place we can access and that any reference to an URL that starts with /static is mapped to a directory with the same name relative to the location from where started the script.
If we save this script as weatherservice.py we can run it with python weatherservice.py. To test it we may direct our webserver either to the name of the host where we run the script or to localhost, e.g. http://localhost:8080/ or http://www.example.com:8080/.
cherrypy.quickstart(Root(),
config={
'/':{
'log.access_file' : os.path.join(current_dir,"access.log"),
'log.screen' : False
},
'/static':{
'tools.staticdir.on' :True,
'tools.staticdir.dir' :current_dir+"/static"
}
}
)
Structuring data with HTML
We serve a single basic HTML page to structure all the data elements in our web application and use AJAX calls from a small piece of Javascript to fill in the actual data. The base page starts of with ahead section that accomplishes several things: loading the jQuery and jQueryUI libraries, provinding access to a graphical canvas, even in Internet Explorer (that is what the conditional comments do) and load the Flot plugin. The final two lines incorporate our own application specific Javascript (weather.js) and CSS file (weather.css).
<html> <head> <title>Weather Overview</title> <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.min.js" type="text/javascript"></script> <script src="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.11/jquery-ui.min.js" type="text/javascript"></script> <link rel="stylesheet" href="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.11/themes/smoothness/jquery-ui.css" type="text/css" media="all" /> <!--[if IE]><script language="javascript" type="text/javascript" src="/static/js/flot/excanvas.min.js"></script><![endif]--> <script language="javascript" type="text/javascript" src="/static/js/flot/jquery.flot.js"></script> <script language="javascript" type="text/javascript" src="/static/js/weather.js"></script> <link rel="stylesheet" href="/static/css/weather.css" type="text/css" media="all" /> </head>
The body of the HTML is structured with two div elements. The first one is structured with an unordered list in a pattern that is suitable to apply jQueryUI's tab widget. Each of the tabs contains another div, each with its own id and marked as a graph class. These we will convert to graphs with the Flot plugin later.
<body>
<div id="tabs">
<ul>
<li><a href="#tabs-1">Temperature and wind</a></li>
<li><a href="#tabs-2">Humidity and rain</a></li>
<li><a href="#tabs-3">Summary and extremes</a></li>
<li><a href="#tabs-4">Indoor values</a></li>
</ul>
<div id="tabs-1">
<div id="tempandwind" class="graph" style="width:840px;height:300px"></div>
</div>
<div id="tabs-2">
<div id="humidityandrain" class="graph" style="width:840px;height:300px"></div>
</div>
<div id="tabs-3">
<p>here will be a table</p>
</div>
<div id="tabs-4">
<div id="indoorvalues" class="graph" style="width:840px;height:300px"></div>
</div>
</div>
The second div will hold radio buttons to select a reporting period and the level of detail in the graphs. These will be styled as jQueryUI button widgets and fitted with click handlers to make the graphs interactive.
<div id="nav"> <div id="period"> <input type="radio" id="p1" name="period" checked="checked" /> <label for="p1">Day</label> <input type="radio" id="p2" name="period"/> <label for="p2">Week</label> <input type="radio" id="p3" name="period" /> <label for="p3">Month</label> </div> <div id="detail"> <input type="radio" id="d1" name="detail" checked="checked" /> <label for="d1">Low</label> <input type="radio" id="d2" name="detail"/> <label for="d2">High</label> </div> </div> </body> </html>
In the next part we will look at the necessary Javascript and how to use the Flot plugin.
Other parts of this series
Python and Javascript: using the Flot plugin, part 3
In a previous article I sketched the road map for implementing a small web application to present data from a small weather station with the help of PyWWS and the Flot plug-in. In this article we show how to provide data from the pywww/DataStore module as JSON encoded information that we can use in AJAX calls.
Producing JSON encoded weather data with CherryPy
Our web application is a single web page with some added Javascript that relies on a number of
services that provide JSON encoded data to AJAX calls.
These services are implemented as Python classes within a CherryPy application. The first class we define is called Basic and its initializer takes a single tz argument. This will be the timezone used to display data.
class Basic:
def __init(self,tz=pytz.timezone('Europe/Amsterdam')):
self.tz=tz
@cherrypy.expose
def default(self,name,_=None,start=None,end=None):
start,end = self.verifystartend(start,end)
if name in { 'temp_out','hum_out','wind_ave',
'wind_gust','wind_dir','rain'}:
return self.getdata(name,start,end)
The crucial method is default(). It is exposed to the CherryPy engine with the cherrypy.expose decorator. A exposed method with the name default will receive any request that cannot be mapped to a more specific name. The name attribute will hold the final part of the URL. The _ argument will hold a random string that we simply ignore: it is added to any AJAX call by jQuery to prevent the browser from caching results. The start and stop arguments are optional and may contain a date/time argument in YYYYMMDDHH format. If the end argument is absent, it defaults to now, if the start argument is absent it defaults to 24 hours before end. These defaults are calculated by the verifystartend() method (not shown).
The next check is to see whether name is one of the known types of data in the DataStore. If all is well we simply pass the name, start and end arguments to the getdata() method that we encounter in the next section. The getdata() method is not part of the Basic class but will be provided by a mixin class called Service. The idea is that Basic and its subclasses provide web application logic to be embedded in CherryPy, while Service provides an interface to produce JSON encoded data based on the data provided by PyWWS.
We define two subclasses of Basic. The first is called Hourly and should return weather data that are the hourly averages. Its initializer takes a weatherdata argument which should point to the directory where PyWWS stores its data and an optional tz argument to hold the timezone:
class Hourly(Service,Basic):
def __init__(self, weatherdata,
tz=pytz.timezone('Europe/Amsterdam')):
super().__init__(tz)
self.datastore = DataStore.hourly_store
self.weatherdata=weatherdata
The most important bit is that __init__() creates a datastore instance variable and initializes it to the hourly_store class from the DataStore module. The datastore and weatherdata variables will be used by the getdata() method from the Service mixin.
The Raw class is very similar to the Hourly class only its datastore variable is initialized to the data_store class from the DataStore module which produces non-averaged weather data.
class Raw(Service,Basic):
def __init__(self, weatherdata,
tz=pytz.timezone('Europe/Amsterdam')):
super().__init__(tz)
self.datastore = DataStore.data_store
self.weatherdata=weatherdata
The Service mixin class is where all the hard work is happening. The bulk of the work is done by its getdata() method. It first creates a suitable instance of a pywws DataStore and then converts the data it retrieves from this datastore with the dumps function from the json module.
def getdata(self,what,start,end):
ds=self.datastore(self.weatherdata)
return dumps(
[( self.millisecondsfromnaiveutc(
data['idx']), data[what])
for data in ds[
start.astimezone(
Service.utc).replace(
tzinfo=None):
end.astimezone(
Service.utc).replace(
tzinfo=None)]
]
)
The single argument to the dumps() function is a list of tuples, each consisting of a timestamp in milliseconds and a floating point value as this is the format that the Flot library expects. This data is retrieved from the datastore with the slice notation: ds[a:b] will retrieve all the data between a and b (if a and b are datetime instances).
All this work was needed to be able to request URLs like http://localhost:8080/temp_out and receive a response like [[123456,14.1],[123654,14.2],[123789,14.4]] for example.
In the next installment of this series we will look into the HTML and Javascript code needed to make this web application really work.
Other parts of this series
Python and Javascript: using the Flot plugin
Good graphs are not only about data, they should look good as well to attract attention. The Flot plugin for jQuery produces excellent graphs, is simple to use and shows nicely how to marry Python and Javascript.
In my opinion when designing a web application, the client side (the stuff that happens in the browser) is often not receiving the attention it deserves. Sure enough we design for low latency using AJAX and mimic screen interactions of regular applications as close as possible but often is still feels like we're looking a old web page.
One of the areas that can benefit tremendously from a well chosen library is graphs. In the coming months I will try to describe a simple web application that displays data from a weather station. The server side is all python of course and will make use of the PyWWS library to acquire the data. On the client side will employ Flot, a jQuery pluging that can produce attractive graphs in simple manner. A sample of the first draft of the app is show below:
A SQLite thread safe password store
Prompted by one of the reviewers of my upcoming book I decided I needed a simple, thread safe, and reasonably secure password store backed by SQLite.
The design criteria were straight forward and based in part on Storing passwords - done right! and the practical recommendations onPythonSecurity.org:
- based on SQLite
- allow for a reasonable amount of threads (its intended use is within a CherryPy application)
- able to use a salt with a configurable number of random bits
- able to apply key stretching with a configurable number of iterations
- use any secure hash algorithm from Python's hashlib module
Example
from dbpassword import dbpassword
dbpw = dbpassword('/var/password.db')
# later, from any thread
dbpw.update(user,plaintextpassword) # update or set a new password
if dbpw.check(user,plaintextpassword) :
... do stuff ...
else:
... warn off user ...
The dbpassword module
Warning! I am not a cryptographer so I cannot guarantee the following code is safe enough for your needs.
'''
dbpassword.py Copyright 2011, Michel J. Anders
This program is free software: you can redistribute it
and/or modify it under the terms of the GNU General Public
License as published by the Free Software Foundation,
either version 3 of the License, or (at your option) any
later version.
This program is distributed in the hope that it will be
useful, but WITHOUT ANY WARRANTY; without even the implied
warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE. See the GNU General Public License for more
details.
You should have received a copy of the GNU General Public
License along with this program. If not, see
CherryPy Arguments vs. Sqlite's lazy typing
Some bugs prove extremely hard to squash as this little tale shows.
Look a this bit of code:
import sqlite3 as sqlite
with sqlite.connect("/tmp/testdb.db") as conn:
conn.execute("create table if not exists test ( a default 1)")
conn.execute("insert into test default values")
c = conn.cursor()
c.execute("select count(*) from test")
print(c.fetchone()[0])
c.execute("select * from test where a = ?",(1,))
print(c.fetchone()[0])
c.execute("select * from test where a = ?",('1',))
print(c.fetchone()[0])
All it does is creating a table with a single column called a which has a numerical default of 1.
As expected the first print call produces a positive integer (incremented each time you run this snippet unless you delete /tmp/testdb.db in the mean time.) and the second call to print will select all records where a = 1, and printing the first column of the first record therefore will yield the number 1. Nothing special so far.
However, the final print call will raise an exception, stating that NoneType is not subscriptable because the select statement doesn't match any record. This seems logical as the a column contains numbers whereas in the final print call we select records matching the string '1' and a number is never equal to a string.
But what if we would change the table definition to
conn.execute("create table if not exists test ( a integer default 1)")
Now the final print call will succeed without a problem because sqlite will acknowledge the affinity of the a column and convert the string to an integer before comparing them.
This caused me some severe headaches because some of the queries in a CherryPy application I was working on returned different results although the queries where identical except from the location in the code, with identical arguments even.
But what might look identical to the eye proved to be different on close inspection: the column I checked was declared with an integer affinity and in the piece of code that did a successful query, the argument to compare against was an integer. The faulty code however was in a CherryPy method that received its arguments from the outside (a GET request from a browser) and all incoming arguments in CherryPy are strings!
Moral of the story: even though Sqlite allows you to assign any type of value to any column just like Python variables, be careful when comparing values!
Daemonizing CherryPy
It is documented but without a proper example I found it very hard to find out how to daemonize a CherryPy server. After some trial and error it proved to be not so hard at all.
The CherryPy documentation is a little haphazard at times and although the Daemonizer plugin is documented I found it a bit difficult to understand without any examples. There is a bit more available now in the new documentation but that is quite hidden so it never hurts to show an example:
cherrypy.process.plugins.Daemonizer(cherrypy.engine).subscribe()
...
...
cherrypy.quickstart(Root(),config={
'/':
{ 'log.access_file' : os.path.join(current_dir,"access.log"),
'log.screen': False,
'tools.sessions.on': True
}})
Note that this only works on UNIX like systems (it certainly won't work on Windows XP). Also note the configuration parameters. Make sure sure you log your accesses explicitly otherwise you will have a hard time finding where the logging of your daemonized process went (hint: probably nowhere...)Sqlite multithreading woes
Registering a user defined regexp function with Sqlite from CherryPy
Multithreading can be tricky and can actually trip you in unexpected ways. And in some situations multithreading is almost unavoidable, for example when using CherryPy as it usually instantiates a fair number of threads to efficiently serve multiple HTTP requests in parallel. The situation that caught me unawares was the combination with Sqlite.
Sqlite can be used in a multithreaded fashion but you must make sure that each thread has it's own connection object to communicate with the database. This is easily accomplished by registering a function with CherryPy that will be called for each newly started thread. This might look as follows:
import sqlite3
import threading
data=None
db='/tmp/example.db'
def initdb():
global data,db
sql='create table if not exists mytable (col_a, col_b);'
conn=sqlite3.connect(db)
c = conn.cursor()
c.execute(sql)
conn.commit()
conn.close()
data=threading.local()
def connect(thread_index):
global data,db
data.conn = sqlite3.connect(db)
data.conn.row_factory = sqlite3.Row
if __name__ == "__main__":
initdb()
cherrypy.engine.subscribe('start_thread', connect)
<... code to start the cherrypy engine ...>
There are two functions in the example above. The first one, initdb(), is used to initialize the database, that is, to create any tables necessary if they are not defined yet and to prepare some storage that is unique for each thread. Normally, all global data is shared between threads so we have to take special measures to provide each thread with its own private data. This is accomplished by the call to threading.local(). The resulting object can be used to store data as attributes and this data is private to each thread. initdb() needs to be called only once before starting the CherryPy engine.
The second function, connect(), should be called once for every thread. It creates a database connection and stores a reference to this connection in the conn attribute of the global variable data. Because this was setup to be private data for each thread, we can use it to store a separate connection object.
In the main section of the code we simply call initdb() once and use the cherrypy.engine.subscribe() function to register our connect() function to be executed at the start of a new thread. The code to actually start CherryPy is not shown in this example.
User defined functions
Now how can this simple setup cause any troubles? Well, most database configuration actions in Sqlite are performed on connection objects and when we want them to work in a consistent way we should apply them to each and every connection. In other words, those configuration actions should be part of the connect() function. An example of that is shown in the last line of the connect() function where we assign a sqlite3.Row factory to the row_factory attribute of a connection object. Because we do it here we make sure that we may consistently access columns by name in any record returned from a query.
What I failed to do and what prompted this post was register a user defined function for each connection. Somehow it seemed logical to do it only once when initializing the database, but even if that connection wasn't closed it was impossible to use that function in a query. And user defined functions are not a luxury but a bare necessity if you want to use regular expressions in Sqlite!
Sqlite supports the REGEXP operator in queries so you may use a query like:
select * from mytable where a regexp '^a.*b$';
This will select any record that has a value in its a column that starts with an a and ends with a b. However, although the syntax is supported, it still raises a Sqlite3.OperationalError exception because the regexp function that is called by the regexp operator is not defined. If we want to use regular expressions in Sqlite we have to supply an implementation of the regexp function ourselves. Fortunately this is quite simple, a possible implementation is shown below:
import re
def regex(pattern,string):
if string is None : string = ''
return re.search(pattern,str(string))!=None
Note that this isn't a very efficient implementation as we compile a pattern again and again each time the function is called even when it may be called hundreds of times with the same pattern in a single query. It does the job however.
All that is left to do now, is register this function. Not, as I did, as part of the initdb() function, but as part of the connect() function that is called for each thread:
def connect(thread_index):
global data,db
data.conn = sqlite3.connect(db)
data.conn.row_factory = sqlite3.Row
data.conn.create_function('regexp',2,regex)
The create_function() method will make our newly defined function available. It takes a name, the number of arguments and a reference to our new function as arguments. Note that despite what the Sqlite documentation states, our regular expression function should be registered with the name regexp (not regex!).
A side note on multiprocessing
If you have a multiprocessor or multicore machine, multithreading will in general not help you to tap into the full processing power of your server. In this article I explore ways to use Python's multiprocessing module in combination with Sqlite.
Starting a stand alone webapp
The concept of a web application is broader than you might guess: using the webbrowser as a graphical user interface (GUI) makes sense even if you are just running a stand alone application on a local machine. After all, every machine has a browser intalled nowadays, so using this as a GUI might save you from headaches trying to find a cross platform GUI toolkit that looks good and is familiar to the user.
In order to use the browser as a user interface we have to start it up in a platform independent way, start a web application framework serving our application at the same time and make sure that the new web browser window points to the correct location. This sounds like a lot of work but in Python this is actually rather straightforward.
Let's have a look at the following code:
import cherrypy
import webbrowser
import threading
def openbrowser():
webbrowser.open('http://127.0.0.1:8080')
class Root(object):
@cherrypy.expose
def index(self):
return 'Hi There'
if __name__ == "__main__":
threading.Timer(3.0,openbrowser).start()
cherrypy.quickstart(Root(),config={})
If you save the code above as webapp.py and you have CherryPy installed, you can start the program by typing the following in a terminal (or dos-box):python webapp.py
It will start up a webserver running on your local machine that listens on port 8080. It will also start up a new browser window and direct it to http://127.0.0.1:8080. The Python version is not relevant here as we do not use any 3.x specific constructs.
The trick is to utilize Python's bundled module webbrowser to open a browser window in a cross platform compatible way as implemented in the openbrowser() function. We do not call this function right away though, because the browser then might start before there is a webserver running. We cannot start CherryPy first either, because the quickstart() function does not return. Therefor we instantiate a Timer object from the threading module and tell it to call our openbrowser() function after three seconds, which should be plenty of time for the CherryPy server to start.
Subscribe to:
Posts (Atom)