Showing posts with label PyWWS. Show all posts
Showing posts with label PyWWS. Show all posts

Python and Javascript: using the Flot plugin, part 5

Using the Flot plugin

In article 5 of this series we look at how we may actually configure the Flot plugin to show the data.

The Javascript code: minimalweather.js

Let's have a look at a rather minimal implementation of the client side of our app. It will only show temperature and wind speed but it is a rather good example of what is possible. The Javascript code is encapsulated in a jQuery $(document).ready() function. This way we ensure that we only convert HTML elements to jQuery widget when we're absolutely sure they are present.

The first step we take is setting some general AJAX parameters. We set cache to false which will instruct jQuery to add a _ (underscore) parameter to every AJAX call. This parameter will have a random value and this will make the URL different each time, thereby preventing the browser to cache result. After all we are not interested in stale weather data. We also set async to false. AJAX calls normally return immediately but signal completion to a function that is given to them as a parameter. However, because our graphs consist of multiple dataseries we want to retrieve the data from more than one datasource. To draw the complete graph we need all data to be available, so by setting async to false we don't start doing anything else unless the last AJAX call is finished. This is a bit against the grain (after all the first A in AJAX stands for asynchronous) but it does make out code simpler.

The second task is to set a default for the reporting period and level of detail in the graphs. The period might be a day, a week or a month and the default level of detail is a average value per hour. PyWWS compatible weather stations are normally capable of logging in a much finer detail and that is what we retrieve if detail is set to raw. (My weather station can log a value every five minutes).

Next we define two functions: hourly(), that will issue two getJSON() calls to retrieve temperature and windspeed averages over the last hour and raw() that will do the same but for 5 minute intervals. Because we designed the server side of the application to produce JSON serialized data all the hard work of converting this data to arrays of timestamp/value pairs in a safe way is done by the getJSON() function which will pass the result as the data argument to the function it calls on completion.

$(document).ready(function(){
  $.ajaxSetup({cache:false,async:false});

  var temp_out;
  var wind_ave;
  
  var period="day";
  var detail="hourly";
   
  function hourly(){
   $.getJSON('./hourly/temp_out' ,{"period":period},
   function(data){ temp_out = data; });
   $.getJSON('./hourly/wind_ave' ,{"period":period},
   function(data){ wind_ave = data; });
   }
  
  function raw(){
   $.getJSON('./raw/temp_out' ,{"period":period},
   function(data){  temp_out = data; });
   $.getJSON('./raw/wind_ave' ,{"period":period},
   function(data){  wind_ave = data; });
  }

Now that we have functions in place to retrieve data we need something to convert these two data sets (temperature and wind speed) to an object that can be passed as an argument to the Flot plugin. That is what the setdata() function does: it creates an object that defines two dataseries with an appropriate label. It also defines a second y-axis to use by the wind speed dataseries. We initialize the da variable to hourly data.

  var da;
  
  function setdata(){
   da = [
    {data:temp_out,label:"temperature"},
    {data:wind_ave,label:"wind",yaxis:2}
    ];
  };
   
  hourly();
  setdata();

With the data present and a data argument ready we can finally convert the div with the tempandwind id to a graph. The flot plugin is a little different from most plugins as it does not provide a member function on any jQuery selection (unlike for example the button plugin which can be called as $("#mybutton").button() ). The Flot plugin just provides a plot() function which is passed a jQuery selection as its first argument. The second argument is an object which describes the data series and the final argument is an option object. Here we configure all series to show lines as well as points and configure the x-axis to behave as a time axis with a maximum of twelve tick marks. If it decides to show hours as ticks we inform it to use a 24 hour clock (it might show days as well, it decides that automatically although this may be set explicitly). Note that the width and height of the HTML element that we want to convert to a graph must be set explicitly beforehand. We have take care of that in the HTML contained in the basepage.html file.

  
  var p = $.plot($("#tempandwind")  ,da ,{
    series: { lines: { show: true }, points: { show: true } },
    xaxis : { mode:"time",ticks:12,twelveHourClock:false},
    y2axis: { position:"right"}
  });

We will also configure some buttons to let the user interact with the graph so we must define some way to reload and redisplay data. We therefore define a replot() function which will retrieve either hourly data or detailed data based on the contents of the detail variable and then create a new data description object. We then use the setData() method of the Flot plugin to indicate we have new data and its setupGrid() method to recalculate things like axes and ticks. The graph is then redrawn by calling the draw() method.

  function replot(){
 if (detail == "hourly") {
  hourly();
 }else{
  raw();
 }
 setdata(); 
 p.setData(da);
 p.setupGrid();
 p.draw();
  };

Interaction with the graph is now a simple matter of binding functions to click events. These functions set either the detail or the period variable to a suitable value and then call the replot() function.

  
 $("#d2").click(function(){
  detail="raw";
  replot();
 });

 $("#d1").click(function(){
  detail="hourly";
  replot();
 });

 $("#p1").click(function(){
  period="day";
  replot();
 });
 
 $("#p2").click(function(){
  period="week";
  replot();
 });
 
 $("#p3").click(function(){
  period="month";
  replot();
 });

What is left (although we could have done it much earlier) is to style the tabs and buttons with regular jQueryUI widgets:

 
 $("#tabs").tabs();
 $("#detail").buttonset();
 $("#period").buttonset();
});

The result of all this work is a functional web application that shows temperature and wind speed on its first tab:

And it is interactive of course: Clicking the week button for example results in this overview:

Of course we there is a lot more we can do but that is covered in coming articles.

Other parts of this series

Python and Javascript: using the Flot plugin, part 4

First steps in creating a web application

It is nice to have some modules that serve up weather data but it is of course not enough. In this article we show how to use those modules as components in a CherryPy application. We also see what the HTML looks like that we use to structure the information in the web application and load all necessary Jascript files and supporting CSS.

Serving a CherryPy application

The first steps in setting up our web application is importing the cherrypy module and the classes from the weather package that we created earlier:

import os
        
current_dir = os.path.dirname(os.path.abspath(__file__))

import cherrypy

from weather.services import Hourly,Raw,Monthly

basepage = "".join(open(os.path.join(current_dir,'basepage.html')).readlines())

In the last line we also read in a file called basepage.html that we will look at later as it forms the basis of our application (we use a separate file so that we don't have to mix to much Python and HTML. Syntax highlighters don't like that). The next step is to configure the tree of URLs that serve the different kinds of data:

class Root:

    hourly = Hourly('/home/michel/sitescripts/pywws/weatherdata')
    raw    = Raw('/home/michel/sitescripts/pywws/weatherdata')
    monthly= Monthly('/home/michel/sitescripts/pywws/weatherdata')
    
    @cherrypy.expose
    def index(self):
        return basepage

We configure the server to listen on any address and by default it will listen on port 8080. If you need another port you may configure this with the server.socket_port option.

                        
cherrypy.config.update({'global':{'server.socket_host':'0.0.0.0'}})    

Finally we start the CherryPy server by passing an instance of the Root class we defined to serve the application to the quickstart() function. We pass in additional configuration items to make sure we have a log file in a place we can access and that any reference to an URL that starts with /static is mapped to a directory with the same name relative to the location from where started the script.

If we save this script as weatherservice.py we can run it with python weatherservice.py. To test it we may direct our webserver either to the name of the host where we run the script or to localhost, e.g. http://localhost:8080/ or http://www.example.com:8080/.

cherrypy.quickstart(Root(),
        config={
            '/':{
                'log.access_file'  : os.path.join(current_dir,"access.log"),
                'log.screen'       : False
                },
            '/static':{
                'tools.staticdir.on'  :True,
                'tools.staticdir.dir' :current_dir+"/static"
                }
               }
)

Structuring data with HTML

We serve a single basic HTML page to structure all the data elements in our web application and use AJAX calls from a small piece of Javascript to fill in the actual data. The base page starts of with a head section that accomplishes several things: loading the jQuery and jQueryUI libraries, provinding access to a graphical canvas, even in Internet Explorer (that is what the conditional comments do) and load the Flot plugin. The final two lines incorporate our own application specific Javascript (weather.js) and CSS file (weather.css).
<html>
<head>
<title>Weather Overview</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.min.js" type="text/javascript"></script>
<script src="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.11/jquery-ui.min.js" type="text/javascript"></script>
<link rel="stylesheet" href="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.11/themes/smoothness/jquery-ui.css" type="text/css" media="all" />
<!--[if IE]><script language="javascript" type="text/javascript" src="/static/js/flot/excanvas.min.js"></script><![endif]-->
<script language="javascript" type="text/javascript" src="/static/js/flot/jquery.flot.js"></script>
<script language="javascript" type="text/javascript" src="/static/js/weather.js"></script>
<link rel="stylesheet" href="/static/css/weather.css" type="text/css" media="all" />
</head>

The body of the HTML is structured with two div elements. The first one is structured with an unordered list in a pattern that is suitable to apply jQueryUI's tab widget. Each of the tabs contains another div, each with its own id and marked as a graph class. These we will convert to graphs with the Flot plugin later.

<body>
<div id="tabs">
  <ul>
    <li><a href="#tabs-1">Temperature and wind</a></li>
    <li><a href="#tabs-2">Humidity and rain</a></li>
    <li><a href="#tabs-3">Summary and extremes</a></li>
    <li><a href="#tabs-4">Indoor values</a></li>
  </ul>
  <div id="tabs-1">
    <div id="tempandwind" class="graph" style="width:840px;height:300px"></div>
  </div>
  <div id="tabs-2">
    <div id="humidityandrain" class="graph" style="width:840px;height:300px"></div>
  </div>
  <div id="tabs-3">
    <p>here will be a table</p>
  </div>
  <div id="tabs-4">
    <div id="indoorvalues" class="graph" style="width:840px;height:300px"></div>
  </div>
</div>

The second div will hold radio buttons to select a reporting period and the level of detail in the graphs. These will be styled as jQueryUI button widgets and fitted with click handlers to make the graphs interactive.

<div id="nav">
 <div id="period">
 <input type="radio" id="p1" name="period" checked="checked" />
 <label for="p1">Day</label>
 <input type="radio" id="p2" name="period"/>
 <label for="p2">Week</label>
 <input type="radio" id="p3" name="period" />
 <label for="p3">Month</label>
 </div>
 <div id="detail">
 <input type="radio" id="d1" name="detail" checked="checked" />
 <label for="d1">Low</label>
 <input type="radio" id="d2" name="detail"/>
 <label for="d2">High</label>
 </div>
</div>
</body>
</html>

In the next part we will look at the necessary Javascript and how to use the Flot plugin.

Other parts of this series

Python and Javascript: using the Flot plugin, part 3

In a previous article I sketched the road map for implementing a small web application to present data from a small weather station with the help of PyWWS and the Flot plug-in. In this article we show how to provide data from the pywww/DataStore module as JSON encoded information that we can use in AJAX calls.

Producing JSON encoded weather data with CherryPy

Our web application is a single web page with some added Javascript that relies on a number of services that provide JSON encoded data to AJAX calls. These services are implemented as Python classes within a CherryPy application. The first class we define is called Basic and its initializer takes a single tz argument. This will be the timezone used to display data.

class Basic:

    def __init(self,tz=pytz.timezone('Europe/Amsterdam')):
        self.tz=tz
        
    @cherrypy.expose
    def default(self,name,_=None,start=None,end=None):
        start,end = self.verifystartend(start,end)
        if name in { 'temp_out','hum_out','wind_ave',
               'wind_gust','wind_dir','rain'}:
            return self.getdata(name,start,end)

The crucial method is default(). It is exposed to the CherryPy engine with the cherrypy.expose decorator. A exposed method with the name default will receive any request that cannot be mapped to a more specific name. The name attribute will hold the final part of the URL. The _ argument will hold a random string that we simply ignore: it is added to any AJAX call by jQuery to prevent the browser from caching results. The start and stop arguments are optional and may contain a date/time argument in YYYYMMDDHH format. If the end argument is absent, it defaults to now, if the start argument is absent it defaults to 24 hours before end. These defaults are calculated by the verifystartend() method (not shown).

The next check is to see whether name is one of the known types of data in the DataStore. If all is well we simply pass the name, start and end arguments to the getdata() method that we encounter in the next section. The getdata() method is not part of the Basic class but will be provided by a mixin class called Service. The idea is that Basic and its subclasses provide web application logic to be embedded in CherryPy, while Service provides an interface to produce JSON encoded data based on the data provided by PyWWS.

We define two subclasses of Basic. The first is called Hourly and should return weather data that are the hourly averages. Its initializer takes a weatherdata argument which should point to the directory where PyWWS stores its data and an optional tz argument to hold the timezone:

class Hourly(Service,Basic):

    def __init__(self, weatherdata,
           tz=pytz.timezone('Europe/Amsterdam')):
        super().__init__(tz)
        self.datastore = DataStore.hourly_store
        self.weatherdata=weatherdata
The most important bit is that __init__() creates a datastore instance variable and initializes it to the hourly_store class from the DataStore module. The datastore and weatherdata variables will be used by the getdata() method from the Service mixin.

The Raw class is very similar to the Hourly class only its datastore variable is initialized to the data_store class from the DataStore module which produces non-averaged weather data.

class Raw(Service,Basic):

    def __init__(self, weatherdata,
           tz=pytz.timezone('Europe/Amsterdam')):
        super().__init__(tz)
        self.datastore = DataStore.data_store
        self.weatherdata=weatherdata
The Service mixin class is where all the hard work is happening. The bulk of the work is done by its getdata() method. It first creates a suitable instance of a pywws DataStore and then converts the data it retrieves from this datastore with the dumps function from the json module.
            
    def getdata(self,what,start,end):
        ds=self.datastore(self.weatherdata)
        return dumps(            
                [( self.millisecondsfromnaiveutc(
          data['idx']), data[what]) 
                 for data in ds[
                     start.astimezone(
        Service.utc).replace(
                       tzinfo=None):
                     end.astimezone(
        Service.utc).replace(
                      tzinfo=None)]
    ]
               )
The single argument to the dumps() function is a list of tuples, each consisting of a timestamp in milliseconds and a floating point value as this is the format that the Flot library expects. This data is retrieved from the datastore with the slice notation: ds[a:b] will retrieve all the data between a and b (if a and b are datetime instances).

All this work was needed to be able to request URLs like http://localhost:8080/temp_out and receive a response like [[123456,14.1],[123654,14.2],[123789,14.4]] for example.

In the next installment of this series we will look into the HTML and Javascript code needed to make this web application really work.

Other parts of this series

Python and Javascript: using the Flot plugin, part 2

In a previous article I sketched the road map for implementing a small web application to present data from a small weather station with the help of PyWWS and the Flot plugin. In this article we tackle the conversion of the pywww/DataStore module to Python 3.

Converting the pywws/DataStore module to Python 3

Because pywws is written for Python 2.x we need to convert it to Python 3 because I don't want to continue developing for Python 2.x now that many frameworks and libraries are converted to version 3. In principle the 2to3 tool should be able to do the bulk of the work but we cannot be sure before we try and test.

For our web application we will only need to use the pywws/DataStore module to get access to stored weather data and fortunately it has no dependencies on other pywws modules. Converting this module with the 2to3 tool and comparing the differences shows only 3 changes:

  • the ConfigParser module is now called configparser
  • map() returns a map object instead of a list
  • open() takes an 'r' mode argument instead of 'rb' in those cases where the result of the open() function is expected to return unicode strings instead of bytes.
The second change is the most fundamental. A map object is a generator and not a plain list. This means is has to be explicitly converted to a list if an operation expects a plain list. For an expansion from a list to a number of arguments this is correctly done by the 2to3 utility. For example, the following line
  return datetime(*map(int, (date_string[0:4],
                                    date_string[17:19])))
is translated to
  return datetime(*list(map(int, (date_string[0:4],
                                    date_string[17:19])))
All in all this is a fairly painless conversion process (but remember that I converted just the Datastore module, although with 455 lines this is pretty big). The next article will show how we implement server side component of the weather application based on PyWWS and CherryPy.

Python and Javascript: using the Flot plugin

Good graphs are not only about data, they should look good as well to attract attention. The Flot plugin for jQuery produces excellent graphs, is simple to use and shows nicely how to marry Python and Javascript.

In my opinion when designing a web application, the client side (the stuff that happens in the browser) is often not receiving the attention it deserves. Sure enough we design for low latency using AJAX and mimic screen interactions of regular applications as close as possible but often is still feels like we're looking a old web page.

One of the areas that can benefit tremendously from a well chosen library is graphs. In the coming months I will try to describe a simple web application that displays data from a weather station. The server side is all python of course and will make use of the PyWWS library to acquire the data. On the client side will employ Flot, a jQuery pluging that can produce attractive graphs in simple manner. A sample of the first draft of the app is show below:

In later articles I will show how to implement both server and client side.