OWL Intuition to EmonCMS



by Jeremy Poulter

Overview

Along with the install of our solar panels came a monitoring system from OWL called OWL Intuition and while maybe only an average product it does have one saving grace, OWL have published the API needed to receive the generation and consumption data from the system. This allows us to easily pipe the data into other ecosystems.  

OWL Intuition

OWL Intuition is a 'smart' energy monitoring and control system from The OWL. It features monitoring of electricity usage and solar power generation as well as heating and hot water control.

The API for interacting with the OWL Intuition is published on the support site. At a high level the data received from the various wireless nodes is received by the Network OWL and translated in to XML fragments that are then multicast on the local IP network (as well as being sent to the OWL servers).

EmonCMS

Part of the OpenEnergyMonitor.org, EmonCMS is a powerful web application for processing, logging and visualising energy and environmental data.

Making the connection

The first step is to read the messages from the OWL Intuition. To do this Node.js is used. The primary reason form this is that there is an existing library written by Alasdair Allan that handles the receiving and parsing of the multicast packets from the Network OWL.

First we need to install the library. This is easily done with the following command;

npm install owlintuition

Now we have the library installed lets give it a quick try and see what we get with the following script;

// -----------------------------
// Load all the required modules

// Module to parse data from the OWL Intuition 
var OWL = require('owlintuition');

// Some helpful utilities for debugging
var util = require('util');

// If debug is enabled then map the log function to console.log else map it to a dummy function
debug = true;
var log = debug ? console.log : function () { };

// Create an instance of the OWL Intuition listener
var owl = new OWL();
log("Starting monitor");
owl.monitor();

// Handle the electricity event. 
owl.on('electricity', function (event) {
    log("electricity = " + util.inspect(JSON.parse(event), { "depth": null }));
});

// Handle the solar event
owl.on('solar', function (event) {
    log("solar = " + util.inspect(JSON.parse(event), { "depth": null }));
});

// Handle the heating event.
owl.on('heating', function (event) {
    log("heating = " + util.inspect(JSON.parse(event), { "depth": null }));

});

// Eandle the weather event.
owl.on('weather', function (event) {
    log("weather = " + util.inspect(JSON.parse(event), { "depth": null }));
});

All looks to be working, we get an electricity, immediately followed by a solar message;

electricity = { id: '4437190052E2',
  signal: { rssi: '-70', lqi: '36' },
  battery: '100%',
  channels: { '2': null } }
solar = { id: '4437190052E2',
  current:
   [ { generating: '0.00', units: 'w' },
     { exporting: '0.00', units: 'w' } ],
  day:
   [ { generated: '21774.56', units: 'wh' },
     { exported: '9168.31', units: 'wh' } ] }

Excellent, getting something that looks usable, but then...

d:\users\jeremy\documents\visual studio 2015\Projects\owltest\owltest\node_modules\owlintuition\owl.js:176
                                                        'battery':buff.heating.battery.level,
                                                                                      ^
TypeError: Cannot read property 'level' of undefined
    at Socket.<anonymous> (d:\users\jeremy\documents\visual studio 2015\Projects\owltest\owltest\node_modules\owlintuition\owl.js:176:38)
    at Socket.emit (events.js:110:17)
    at UDP.onMessage (dgram.js:472:8)

Not so good. The owlintuition module has thrown an exception when receiving a message. Lets have a closer look;

        } else if ( buff.heating ) {
            var heating = { 'id':buff.heating.id,
                            'signal':buff.heating.signal,
                            'battery':buff.heating.battery.level,
                            'temperature':buff.heating.temperature };
        
            self.emit( 'heating', JSON.stringify(heating) );
        
        } else if ( buff.weather ) {

So the owlintuition module does not like the data being sent for the heating message. So we need to look at the content of buff.heating, a simple console.log will suffice;

{ ver: '2',
  id: '4437190052E2',
  timestamp: '1441921515',
  zones:
   { zone:
      { id: '20013B9',
        last: '1',
        signal: { rssi: '-72', lqi: '49' },
        battery: { level: '2670' },
        conf: { flags: '0' },
        temperature:
         { state: '0',
           flags: '144',
           until: '1443916800',
           zone: '0',
           current: '21.75',
           required: '5.00' } } } }

I am getting a different message format compared to what the owlintuition module is expecting. It looks like OWL have added support for zones since the owlintuition module was originally written.

So now I have some choices to make, the API only returns a single zone so I can either filter out theextra zones or return all the zones and break backwards compatibility. Given that I only have one zone and I am not currently planning to add any more zones I chose to maintain the backward compatibility and just return zone 0;

            if(buff.heating.zones && buff.heating.zones.zone)
            {
                var heating = { 'id':buff.heating.id,
                                'signal':buff.heating.zones.zone.signal,
                                'battery':buff.heating.zones.zone.battery.level,
                                'temperature':buff.heating.zones.zone.temperature };

                self.emit( 'heating', JSON.stringify(heating) );
            }
            else
            {
                var heating = { 'id':buff.heating.id,
                                'signal':buff.heating.signal,
                                'battery':buff.heating.battery.level,
                                'temperature':buff.heating.temperature };

                self.emit( 'heating', JSON.stringify(heating) );
             }

This change can be found on my fork of the repository; https://github.com/jeremypoulter/node-owlintuition and have also been pushed upstream.

So we are now successfully receiving the electric and heating data from the OWL Intuition.

The next step is to get the data in to EmonCMS via its API. To send data to EmonCMS we need a few bits of information;

  • The EmonCMS endpoint
  • The write API key, found on he EmonCMS accounts page
  • A node ID, a number between 1 and 30
  • The actual data, formatted using JSON or CSV

We will use JSON to encode the data as it makes it a little easier to handle on the server as we have a human readable name for each value.

To actually send this to EmonCMS it is a simple HTTP GET request to the EmonCMS endpoint with the data passed via the query part of the URL;

// Make a HTTP connection to a particular server and send the data.
function postData(urlString, node, key, packet)
{
  urlParsed = url.parse(urlString);

  // EmonCMS does not look to like properly encoded query strings...
  var post_data = 'json='+JSON.stringify(packet)+
                  '&apikey='+key+
                  '&node='+node;
  
  log( "post_data = '" + post_data + "'");

  // An object of options to indicate where to post to
  var post_options = {
    host: urlParsed.host,
    port: '80',
    path: urlParsed.path+"?"+post_data,
    method: 'GET',
  };

  // Set up the request
  var post_req = http.request(post_options, function(res) {
    res.setEncoding('utf8');
    res.on('data', function (chunk) {
        log('Response: ' + chunk);
    });
  });
  
  post_req.on('error', function(e) {
    log("Got error: " + e.message);
  });
  post_req.end();
}

This function can be then used in our electricity and heating event listeners as follows;

owl.on('electricity', function( event ) {
  data = JSON.parse(event);
  log( "electricity = " + util.inspect(data, {"depth": null}) );
  packet = {
    signalRssi: data.signal.rssi, 
    signalLqi: data.signal.lqi,
    battery: data.battery,
    ch1Current: data.channels['0'][0].current,
    ch2Current: data.channels['1'][0].current,
    ch3Current: data.channels['2'][0].current,
    ch1Day: data.channels['0'][1].day,
    ch2Day: data.channels['1'][1].day,
    ch3Day: data.channels['2'][1].day
  }
  postData("http://emoncms.org/input/post.json", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", 20, packet);
});

owl.on('heating', function( event ) {
  data = JSON.parse(event);
  log( "heating = " + util.inspect(data, {"depth": null}) );
  packet = {
    signalRssi: data.signal.rssi, 
    signalLqi: data.signal.lqi,
    battery: data.battery,
    tempCurrent: data.temperature.current,
    tempRequired: data.temperature.required,
    state: data.temperature.state,
    flags: data.temperature.flags
  }
  postData("http://emoncms.org/input/post.json", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", 20, packet);
});

So we are basically done now. We can get the data from the OWL Intuition and upload it to EmonCMS. So putting it all together we end up with;

// -----------------------------
// Load all the required modules

// Module to parse data from the OWL Intuition 
var OWL = require('owlintuition');

// Some helpful utilities for debugging
var util = require('util');

// Module for making HTTP connections
var http = require('http');

// URL parsing module
var url = require('url');

// Configuration file management
var config = require('config');

// Get the config into some handy vars
var feeds = config.get('feeds');
var nodes = config.get('nodes');
var debug = config.has('debug') && config.get('debug');

// If debug is enabled then map the log function to console.log else map it to a dummy function
var log = debug ? console.log : function () { };

// Create an instance of the OWL Intuition listener
var owl = new OWL();
log("Starting monitor");
owl.monitor();

// Handle the electricity event. 
//
// Sends the raw data from the electricity monitor. This is essentially the same data that an emonTX
// would send. 
// 
// We will pull out the three channels of data and a few other key bits of data (signal strength, 
// battery level) and send them to emonCMS.
owl.on('electricity', function( event ) {
  data = JSON.parse(event);
  log( "electricity = " + util.inspect(data, {"depth": null}) );
  packet = {
    signalRssi: data.signal.rssi, 
    signalLqi: data.signal.lqi,
    battery: data.battery,
    ch1Current: data.channels['0'][0].current,
    ch2Current: data.channels['1'][0].current,
    ch3Current: data.channels['2'][0].current,
    ch1Day: data.channels['0'][1].day,
    ch2Day: data.channels['1'][1].day,
    ch3Day: data.channels['2'][1].day
  }
  reportToEmon(nodes.electricity, packet);
});

// Handle the solar event
//
// A simplified version of the electricity event with just the PV generation and 
// consumption data. Since all this is in the electicity event we do not need to 
// do anything with this event.
owl.on('solar', function (event) {
    log("solar = " + util.inspect(JSON.parse(event), { "depth": null }));
});

// Handle the heating event.
//
// Status data from the heating controller. Again pull out some key data and send
// to emonCMS.
owl.on('heating', function( event ) {
  data = JSON.parse(event);
  log( "heating = " + util.inspect(data, {"depth": null}) );
  packet = {
    signalRssi: data.signal.rssi, 
    signalLqi: data.signal.lqi,
    battery: data.battery,
    tempCurrent: data.temperature.current,
    tempRequired: data.temperature.required,
    state: data.temperature.state,
    flags: data.temperature.flags
  }
  reportToEmon(nodes.heating, packet);
});

// Eandle the weather event.
//
// This is info on the current weather state, just data pulled from the internet.
// We can retrieve in more detail directly from source as needed, so we will ignore 
// this.
owl.on('weather', function( event ) {
  log( "weather = " + util.inspect(JSON.parse(event), {"depth": null}) );
});

// Send data to emonCMS.
//
// Sends the packet with the passed in node ID to all the configured emonCMS servers.
function reportToEmon(node, packet)
{
  log( "node = " + node );
  log( "packet = " + util.inspect(packet, {"packet": null}) );
  for(var i = 0; i < feeds.length; i++) {
    postData(feeds[i].url, node, feeds[i].key, packet);
  }
}

// Make a HTTP connection to a particular server and send the data.
function postData(urlString, node, key, packet)
{
  urlParsed = url.parse(urlString);

  // EmonCMS does not look to like properly decoded query strings...
  var post_data = 'json='+JSON.stringify(packet)+
                  '&apikey='+key+
                  '&node='+node;
  
  log( "post_data = '" + post_data + "'");

  // An object of options to indicate where to post to
  var post_options = {
    host: urlParsed.host,
    port: '80',
    path: urlParsed.path+"?"+post_data,
    method: 'GET',
  };

  // Set up the request
  var post_req = http.request(post_options, function(res) {
    res.setEncoding('utf8');
    res.on('data', function (chunk) {
        log('Response: ' + chunk);
    });
  });
  
  post_req.on('error', function(e) {
    log("Got error: " + e.message);
  });
  post_req.end();
}

In the final script I have also added the ability to configure the endpoint(s) to send the data to as well as the API key. This allows you to send data to a local instillation of EmonCMS as well as cloud based one.

For the latest code see; https://github.com/jeremypoulter/emonOwl.

Running the Script

There is one problem with this so far. The script needs to be running all the time. We can easily do this using another node.js project, PM2.

Installation is nice and simple using NPM, via the following command;

sudo npm install pm2 -g
pm2 startup
sudo su -c "env PATH=$PATH:/usr/local/bin pm2 startup linux -u pi"
sudo chmod +x /etc/init.d/pm2-init.sh
sudo update-rc.d pm2-init.sh defaults

I had to make an additional change to the init script, /etc/init.d/pm2-init.sh. I needed to change the PM2_HOME variable as follows;

export PM2_HOME="/home/pi/.pm2"

Now PM2 is installed we need to start it so we can continue with the setup;

sudo /etc/init.d/pm2-init.sh start

So now we have PM2 starting up on boot, we need to tell it to start our script;

pm2 start emonowl.js
pm2 save

Tidy-up

As a matter of convenience I developed this script on a desktop machine so I now need to get the script on to something where it can run all the time. The script has a number of Node.js modules that it depends on. Node.js has a nice system to handle this, NPM (Node Package Manager).

I was a little put of by my previous experience with package managers on Linux systems which, while incredibly easy to use from an end user point of view, they take a lot of effort to create and test the package. NPM turned out to be nothing like this.  

NPM has some good documentation with details of getting started, https://docs.npmjs.com/, so I will not go in to the details here.

Word of Warning for EmonBase

My goal for this project was to run it on a Raspberry Pi running EmonCMS and already capturing data from other OpenEnergyMonitor nodes. All the instructions so far should work on any Linux device with Node.js installed, but the particular setup of EmonCMS on a Raspberry Pi added a few additional challenges.

Using these instructions to install EmonCMS UFW is installed and that blocked the multicast traffic. As this is a none critical system the simplest solution was to disable UFW;

sudo ufw disable

One problem out of the way. The next problem was the main root file system is mounted as read only, including PM2's default location for storing its config, so this needs changing to a directory. So I don't have to remember the new location all the time I just moved the directoy and created a sym-link to the new location;

rpi-rw
cd /home/pi
mv .pm2 data
ln -s data/.pm2 
rpi-ro

1 Comment

Top