Creating a Node.js application on the BeagleBone

Well, as usual, it has taken a log time to get to another post! Hopefully it was worth the wait.

WARNING 1 - I am very much of a beginner with Node.js! Please Google before asking me any questions about the programming on this project. The code in this project more or less works and demonstrates basic concepts, but totally lacks the kind of error handling and security that production code should always incorporate.

WARNING 2 - I am also NOT an expert on the BeagleBone and this project really does not use any of the GPIO features of the board (in fact, the Node.js could be run on Windows as easily as the BB). Anyone wanting to understand the board should first look at Derek Molloy's fantastic blog and YouTube videos!

Motivation and background

 

I first pre-ordered my BeagleBone way back in 2011 when the product was first announced. At the time I was interested in the BeagleBoard and the RaspBerry Pi wasn't yet released and it seemed to have good specs for the cost. When I got it, I booted it up and played around with it a bit, but there wasn't a ton of documentation available yet and I was very busy with work. Lately I've been coming back around to my embedded electronics work and when I pulled it out I found that now there was tons of new material around on the device. Just to be clear, mine is the original "white" BeagleBone not the newer "Black" model.

After updating the Angstrom image and booting it up, I saw that the BB provides it's own on-board IDE which is a locally version of normally cloud-based Cloud9 (more on Cloud9) and runs this new and mysterious thing called "Node.js". I had a small amount of JavaScript knowledge, so I bought a text on Node.js (Sams Tech Yourself Node.js) so I could start puzzling it out.

Rather than plunge into the depths of doing GPIO with the BB right away and introducing too many variables, I decided to basically use the BB as a web server and to come up with a simple goal so I could  try to learn some Node.js and prepare for something more substantial later on. I decided something that would be "simple" to do would be figue out how to screen scrape my BlueLine Innovations Wifi PowerCost Monitor (more details here) to provide a visual indication of my power usage and the outside temp. Since I am already using Plot

Gear and setup

 

No elaborate wiring diagrams or anything here. The equipmentused is:

• BeagleBone (White, running Angstrom Linux image)
• BlueLine Innovations PowerCost Monitor™ WiFi Bridge
• BlueLine Innovations Energy Monitor outside meter sensor

Here are the BeagleBone and the PowerCost Monitor:

Here is the outdoor portion strapped onto my utility power meter. Seems a bit funky, but it has been out there for a few years now.

In this project, the Outdoor monitor communicates with the Wifi gateway over a proprietary protocol and then the Gateway serves up a small web page which looks like this:

The Node.js running on the BeagleBone captures the webpage at regular intervals, stores the temperature and usage data in a Tingo database (more on that later) and then serves up a web page that shows a table of recent results with a graph. Here is the layout:

The Web Page 

This is the  resulting power & temp usage monitor web page. It shows:

• a graph at the top of power use and temp over time
• a listing of the data points with their timestamps 
• a settings area where the number of data samples and the interval between samples can be set.

The code

 

This project makes use of many different Node.js and JavaScript library. Specifically:

• Node.js - asynchronous, non-blocking, server-side JavaScript
• Express - web framework for Node.js
• Jade - HTML templating engine for Node.js
• TingoDB - This is a "noSQL" database alternative to MongoDB. (I couldn't get Mongo to run on the BeagleBoard Angstrom release, but TingoDB works fine - probably because it is written entirely in JavaScript.)
• Tungus - This is a driver to allow use of the Mongoose implements mongoose.js driver API.
• Mongoose - object modelling tool for Mongo (in this case being used with TingoDB via Tungus - say that five times fast!)
• Request - A simple HTTP client for Node.js
• Flot - A charting package for jQuery
• jQuery - JavaScript library for HTML document manipulation

Getting all these elements to work together was a  fair challenge given my complete ignorance at the start of this project!

I won't bother with a complete step-by-step build process here. There are many number of great tutorials on the web that walk through creation of Node.js/Express/Jade websites. Christopher Buecheler has a great one over here, for instance. Suffice it to say, first install Node.js, then use npm to install all the dependencies. If you get error messages with the npm installation, Google them and you will pretty much always find an answer. After that, I used Express to create the basic shell of the website.

This is a comparatively simple, single page Node.js project. Besides the libraries above, there are 4 critical files:

• app.js
• layout.jade
• index.jade
• style.css

The complete source code files are at the end of the article and I will highlight critical snippets of code to help understand how it all hangs together. Other code in the files (especially app.js) are set up by Express when you create the website.

Once I had the basic Express/Jade site built and included all the dependencies, I ended up with a directory structure that looked like:

App.js notes:

 

App.js is the server code that runs the whole thing. It also initializes the database, sets environment variables, uses the Request library to extract data from the Wifi PowerCost Monitor website and store that in the TingoDB database. Finally, when the gets a "GET" request from a browser, it renders the data on to the Jade engine.

Here are the sections that manage the database. This code links to the dependencies:

var tungus = require('tungus');
var Engine = require('tingodb');
var mongoose = require('mongoose');

This code initializes Mongoose with Tingo and names the database:

var db = mongoose.connect('tingodb://readingsdb');

Then Mongoose connects and prints to the console:

mongoose.connect('tingodb://readingsdb', function (err){
    if (!err) {
        console.log('connected to databse');
    } else {
        throw err;
    }
});

Then we set up the schema for the database:

var Schema = mongoose.Schema;
var ObjectId = Schema.ObjectId;
var ReadingsSchema = new Schema({
  time: { type : Date, default: Date.now },
    usage2: Number,
    temp2: Number
    });
var Readings = mongoose.model('Readings', ReadingsSchema);

I decided to set the refresh rate (how often the monitor browser reloads) and the interval time (how long between data samples) via environment variables, which I do here:

process.env['SAMPLES'] = '15';

process.env['REFRESHINT'] = '120';

The screen scrape and data extraction is done by the getData function which uses the Request library to load the PowerCost Monitor webpage into a variable, then uses simple JavaScript string commands to isolate the two values we want. It loads the values into the database created above and used a JavaScript setInterval function to repeat the process however often has been set for the refresh rate.

One minor note is I find I have to subract 2 from the temperature I get from the PowerCost Wifi Monitor to get the correct temperature. It just seems to be a slight calibration issue.

function getData (){
    request({uri: "http://192.168.1.33/pcmconfig.htm"}, function (error, response, body){
        if (!error && response.statusCode == 200) {
            var n = body.search("Present Demand");
           usage = body.substr((n+44),5);
           usage = usage.trim();
           console.log(usage);
           var n2 = body.search("Sensor Temp");
           temp = body.substr((n2+40),3);
           temp = temp.trim();
           temp = temp - 2;
           console.log(temp);
           var readingInfo = new Readings({
               temp2: temp,
               usage2: usage
            });
        readingInfo.save(function(err, readingInfo){
              if (err) return console.error(err);
            console.dir(readingInfo);
            });
        setTimeout(getData,(process.env.REFRESHINT * 1000));
    }});
}
//run the above function
getData();

Now we have our data and stored it into a database so we need to render it over to Jade. When a GET request is received from a browser, this function is run:

app.get('/', function(req, res){
  Readings.find({}, {}, { sort: { 'time' : -1}, limit: process.env.SAMPLES }, function(err, readings) {
    if (err) return console.error(err);
      res.render('index', 
        { title: 'Power Usage and Temp from PowerCost Monitor',
          refreshRate: process.env.REFRESHINT,
          sampleNum: process.env.SAMPLES,
          readings: readings
              });
           }
          );
       }
    );

This renders the title, the environment variables and an object with the actual temp & power readings over to Jade via the "res.render" function.


If the user wants to change the sample interval or the refresh rate, they enter them in the the "Settings" fields and when you hit "Submit" it triggers the app.post function, whichwill set the refresh rate and sample number environment variables and then re-render the page:

app.post('/', function(req, res){
    process.env['SAMPLES'] = req.param("samples");
    process.env['REFRESHINT'] = req.param("interval");
    console.log('Samples is set to:  ' + process.env.SAMPLES);
    console.log('Refresh is set to:  ' + process.env.REFRESHINT);    
      Readings.find({}, {}, { sort: { 'time' : -1}, limit: process.env.SAMPLES }, function(err, readings) {
    if (err) return console.error(err);
      res.render('index', 
        { title: 'Power Usage and Temp from PowerCost Monitor',
          refreshRate: process.env.REFRESHINT,
          sampleNum: process.env.SAMPLES,
          readings: readings
              });
           }
          );
       });

Finally, app.js contains the code to actually run the webserver, which is just as Express generated it:

http.createServer(app).listen(app.get('port'), function(){
  console.log('Express server listening on port ' + app.get('port'));
});

Layout.jade

 

Over the course of development, I went back and forth over whether to include the bulk of the client-side JavaScript in the Layout.jade file (essentially the headers of the webpage) or at the end. After reading a few articles (such as this one) , I decided to put the main client-side JS at the end of the page (in this case, the Index.jade file) and it does seem to perform better.

Not really too much to the Layout.jade file, but note this line:

      meta(http-equiv="refresh" content="#{refreshRate}" ) 

It sets the refresh rate based on the environment variable mentioned earlier. I tried to go with having the table and the Flot chart in DIVs and just refreshing the DIVs, but this seemed to not work easily so I will tackle this in the future.

Index.jade

 

This is the other major working part of this project and comes in two parts. First is the Jade portion:

It took me a while to learn how to get the variables to come across properly and to get used to how finicky Jade is about spaces! The coding and hierarchy for the resulting HTML page rely entirely on how you have used spaces to indent, so you have to be very careful! Also note that I have set up the size of the placeholder DIV for the Flot chart. It seems like you need to do this to ensure the chart generates properly.

extends layout

block content    
    h1= title
    
    h2 Graph over time
    br
    #placeholder(style='width:700px;height:300px')
    
    h2 Table of results
    #tablediv
      table
       thead
         tr 
           td 
             b Timestamp
           td
             b Temperature
           td
             b Power usage
         tbody
             each reading in readings
               tr
                 td #{reading.time}
                 td #{reading.temp2}
                 td #{reading.usage2}

The section below is the form for doing the settings. The thing to notice here is that this form also picks up the current values of the environment variables for refresh and interval. The defaults are 15 samples and an interval of 180 seconds.

    h2 Settings
    form#dataSamples(name="dataSamples",method="post",action="/")
        label(for "samples") Data samples to graph:
        input#setSamplesInput(type="text", placeholder="samples", name="samples", value="#{sampleNum}")
        br
        label(for "interval") Time between refreshes (seconds):       
        input#setIntervalInput(type="text",placeholder="interval", name="interval", value="#{refreshRate}")
        br
        button#btnSubmit(type="submit") submit

Then at the bottom of the Index.jade file is the client-side JavaScript which is really what makes everything happen. This points to the jQuery and Flot libraries (which are in the /public folder in the directory tree).

When the page finishes loading, this translates the data object of the database readings from the app.js res.render statement into two separate arrays (one for temp and one for power usage) using the JSON.stringify command so they are ready for plotting. It uses the $.plot function in the Flot library to create the plot. It also sets the labeling and axis formatting as well. (Note that to get the time-based X Axis, you need to include the Flot Time library as well as Flot itself.

    script(src='jquery/dist/cdn/jquery-2.1.0.js')
    script(src='flot/jquery.flot.js')
    script(src='flot/jquery.flot.time.js')
    script.
        window.onload = function(){
           if (window.console)console.log("Executing script");
           var temp_data = [];
           var usage_data = [];
           var reading_data = !{JSON.stringify(readings)};
           for (var i = 0; i < #{sampleNum}; i++)
             temp_data.push([new Date(reading_data[i].time), reading_data[i].temp2]);
           for (var i = 0; i < #{sampleNum}; i++)
             usage_data.push([new Date(reading_data[i].time), reading_data[i].usage2]);
           $.plot($("#placeholder"), [{
               label: "Temperature (C)",
               data: temp_data, 
               },{
               label: "Power Usage (KW)",
               data: usage_data,
               yaxis: 2
              }], {
               xaxes: [{
                 mode: "time",
                 timeformat: "%H:%M",
                 timezone: "browser"
               }],
               yaxes: [{},{position: "right"}]
              });
            };

I suspect there is a simpler way to do this that allows Flot to pick up the data directly from the rendered data object without having to use JSON.stringify array process, but I couldn't find it!

Style.css

Not really much to note here. Just some basic formatting for the few elements I am using. The complete listing is at the end.

Conclusion and next steps

So, that's it for now. As I said at the beginning, this code is certainly not complete! It needs a lot more error checking and testing in failure modes. For instance, what if the network is down or the PowerCost Wifi Monitor is not responding?

Astute observers will also note that the way I have done the "Settings" feature only really works with one browser window at a time. When I have tested changing settings with multiple browsers I have found that it sometimes doesn't pick up the environment variables properly and the refresh rate gets into an "undefined" state and then it just continually loops without any pause - obviously bad!

My next steps will be to learn the Socket.io library in Node.js and then to start connecting the BeagleBone to my DigiX board which I got through Kickstarter a few months ago! Check it out here:

Complete source code

 

app.js

 

This is the complete source code for the app.js file discussed above.

/**
 * Module dependencies.
 */


//Load Dependencies
var express = require('express');
var routes = require('./routes');
var user = require('./routes/user');
var http = require('http');
var path = require('path');
var tungus = require('tungus');
var Engine = require('tingodb');
var mongoose = require('mongoose');
var request = require("request");

//Initialize Mongoose with Tingo
var db = mongoose.connect('tingodb://readingsdb');

//Set up some global variables
var usage;
var temp;
var readings;

//Set environment variables for defaults
process.env['SAMPLES'] = '15';
process.env['REFRESHINT'] = '120';


//Mongoose connects to database
mongoose.connect('tingodb://readingsdb', function (err){
    if (!err) {
        console.log('connected to databse');
    } else {
        throw err;
    }
});

//Set up schema for database
var Schema = mongoose.Schema;
var ObjectId = Schema.ObjectId;

var ReadingsSchema = new Schema({
  time: { type : Date, default: Date.now },
    usage2: Number,
    temp2: Number
    });

var Readings = mongoose.model('Readings', ReadingsSchema);

//Do the simple screen scrape and pop the results into the database.
//Rinse and repeat according to the environment variable for refresh rate
function getData (){
    request({uri: "http://192.168.1.33/pcmconfig.htm"}, function (error, response, body){
        if (!error && response.statusCode == 200) {
            var n = body.search("Present Demand");
           usage = body.substr((n+44),5);
           usage = usage.trim();
           console.log(usage);
           var n2 = body.search("Sensor Temp");
           temp = body.substr((n2+40),3);
           temp = temp.trim();
           temp = temp - 2;
           console.log(temp);
           var readingInfo = new Readings({
               temp2: temp,
               usage2: usage
            });
        readingInfo.save(function(err, readingInfo){
              if (err) return console.error(err);
            console.dir(readingInfo);
            });
        setTimeout(getData,(process.env.REFRESHINT * 1000));
    }});
}

//run the above function
getData();

//Below are all set by default in Express
var app = express();

// all environments
app.set('port', process.env.PORT || 3008);
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'jade');
app.use(express.favicon());
app.use(express.logger('dev'));
app.use(express.json());
app.use(express.urlencoded());
app.use(express.methodOverride());
app.use(app.router);
app.use(express.static(path.join(__dirname, 'public')));

// development only
if ('development' == app.get('env')) {
  app.use(express.errorHandler());
}

//When the Submit button is pressed, it runs this function and re-renders the page
app.post('/', function(req, res){
    process.env['SAMPLES'] = req.param("samples");
    process.env['REFRESHINT'] = req.param("interval");
    console.log('Samples is set to:  ' + process.env.SAMPLES);
    console.log('Refresh is set to:  ' + process.env.REFRESHINT);    
      Readings.find({}, {}, { sort: { 'time' : -1}, limit: process.env.SAMPLES }, function(err, readings) {
    if (err) return console.error(err);
      res.render('index', 
        { title: 'Power Usage and Temp from PowerCost Monitor',
          refreshRate: process.env.REFRESHINT,
          sampleNum: process.env.SAMPLES,
          readings: readings
              });
           }
          );
       });

//This serves up the page and renders the variables into the Jade template engine.
app.get('/', function(req, res){
  Readings.find({}, {}, { sort: { 'time' : -1}, limit: process.env.SAMPLES }, function(err, readings) {
    if (err) return console.error(err);
      res.render('index', 
        { title: 'Power Usage and Temp from PowerCost Monitor',
          refreshRate: process.env.REFRESHINT,
          sampleNum: process.env.SAMPLES,
          readings: readings
              });
           }
          );
       }
    );

//Start up the server!
http.createServer(app).listen(app.get('port'), function(){
  console.log('Express server listening on port ' + app.get('port'));
});

Layout.jade

doctype html
html
  head
      title= title
      link(rel='stylesheet', href='/stylesheets/style.css')
      meta(http-equiv="refresh" content="#{refreshRate}" ) 

  body
  
    block content

Index.jade

extends layout

block content    
    h1= title
    
    h2 Graph over time
    br
    #placeholder(style='width:700px;height:300px')
    
    h2 Table of results
    #tablediv
      table
       thead
         tr 
           td 
             b Timestamp
           td
             b Temperature
           td
             b Power usage
         tbody
             each reading in readings
               tr
                 td #{reading.time}
                 td #{reading.temp2}
                 td #{reading.usage2}

    h2 Settings
    form#dataSamples(name="dataSamples",method="post",action="/")
        label(for "samples") Data samples to graph:
        input#setSamplesInput(type="text", placeholder="samples", name="samples", value="#{sampleNum}")
        br
        label(for "interval") Time between refreshes (seconds):       
        input#setIntervalInput(type="text",placeholder="interval", name="interval", value="#{refreshRate}")
        br
        button#btnSubmit(type="submit") submit
        
    script(src='jquery/dist/cdn/jquery-2.1.0.js')
    script(src='flot/jquery.flot.js')
    script(src='flot/jquery.flot.time.js')
    script.
        window.onload = function(){
           if (window.console)console.log("Executing script");
           var temp_data = [];
           var usage_data = [];
           var reading_data = !{JSON.stringify(readings)};
           for (var i = 0; i < #{sampleNum}; i++)
             temp_data.push([new Date(reading_data[i].time), reading_data[i].temp2]);
           for (var i = 0; i < #{sampleNum}; i++)
             usage_data.push([new Date(reading_data[i].time), reading_data[i].usage2]);
           $.plot($("#placeholder"), [{
               label: "Temperature (C)",
               data: temp_data, 
               },{
               label: "Power Usage (KW)",
               data: usage_data,
               yaxis: 2
              }], {
               xaxes: [{
                 mode: "time",
                 timeformat: "%H:%M",
                 timezone: "browser"
               }],
               yaxes: [{},{position: "right"}]
              });
            };

Style.css

body {
  padding: 50px;
  font: 14px "Lucida Grande", Helvetica, Arial, sans-serif;
}

table
{
border-collapse:collapse;
}
table,th, td
{
border: 1px solid black;
}

td
{
padding:6px;
}