Huninn mesh Part 2 - Architecture, Abstractions and Constraints

Huninn mesh is collection of technologies including a wireless mesh network; a set of physical devices including sensors, relays and gateways that use the implement the wireless network technology; a time series database for storing and manipulating data sampled from sensors; and, a UI for viewing and manipulating sensor data.

In my previous post, I discussed the characteristics of the wireless mesh network devices, describing the system up to and including a gateway. 

In this second post, I examine system architecture from the gateway into the cloud.  I look at the role of a gateway, introduce the device and time series databases, as well as providing a high level view of various abstractions that make it a general purpose system for ingesting and managing time series data. 

This post lays the foundation for a follow on article that will describe the characteristics and structure of the time-series database.

The Gateway

A Huninn mesh Gateway provides the bridge between devices – for example physical Huninn mesh network devices – and a server that resides in the cloud.  The server performs the dual roles of device management and data ingestion from sensors.  (Note that this is a different server from the one discussed in an earlier blog post whose job it is to manage Huninn mesh devices in a wireless network.)

Every device in the Huninn mesh system has a globally unique identity that is allocated and managed in a device database by the server.  This database maintains knowledge of, amongst other things, the gateway to the device, the measurement types supported by the device, the sensor type for each (hardware type and, therefore, the format of messages from/to the device), the sample interval for measurements, etc.

Messages are and moved between the gateway and server using a message queue.  Rabbit MQ was chosen for this role because it is proven, lightweight, scalable and supported on a wide variety of platforms. 

Individual messages are encoded using Protocol Buffers.  For a physical Huninn mesh sensor, the payload might include a timestamp, the temperature, pressure and humidity along with the unique device ID.

A Huninn mesh gateway:

• is specified by an interface;
• knows about all of the devices it manages;
• understands how to ‘talk’ to each device that it manages;
• passes messages from the device to the Huninnmesh cloud via a message queue; and,
• passes messages, received via a message queue from the Huninn mesh cloud, to the device.

Any system that implements the gateway interface can work with the Huninn mesh cloud server.  This is the first high-level abstraction adopted by the system.

A gateway is completely decoupled from a network of physical Huninn mesh devices.  However, a Network of physical Huninn mesh sensors is the most common deployment as shown in the following diagram.

A gateway communicates with the Huninn mesh server using Rabbit MQ.

This figure illustrates that a single gateway g1 manages three sensors, s1… s3 and that two of these s1 & s3 measure temperature, pressure and humidity and the s2 just temperature and pressure.  Measurements are passed to the gateway – in this case a physical Huninn mesh gateway – that publishes them to a Rabbit MQ exchange. 

As noted, this represents a typical Huninn mesh network deployment – where sensors publish measurements that the server collates into a time series. 

The utility of the gateway abstraction becomes apparent when considering how easily other data sources can be incorporated into the system.  Examples of other data sources are external weather measurements (from a web feed) and a Modbus bridge.

Modbus is a serial communication protocol commonly used to interface with HVAC plant equipment in building automation systems.  Read/write access for up to 247 devices can be supported on a single Modbus network

Huninn mesh has built a Modbus Bridge that makes use of the gateway abstraction.  It is built as a standard Linux executable, implements the Modbus protocol and both publishes and subscribes to the Huninn mesh device message queue.  The executable works just like a standard Huninn mesh device: it sleeps, wakes, makes a read Modbus register(s) over the serial interface, packages and publishes the measurement(s) and then sleeps again.

A Modbus gateway might be integrated alongside a physical Huninn mesh network thus:

The gateway abstraction means that it is a simple matter to add other sensors into the system, in this case, Modbus.

In this diagram, the Modbus interface is shown as g2 in red.  This gateway talks to a Modbus device mb1 that reads registers r1…r3. 

This serves to illustrate the point that gateways manage their own devices and know how to communicate with them.  In the diagram two different protocols are employed on the left hand side of the gateway: Huninn mesh and Modbus.   The back end system neither knows nor cares how the gateway delivers messages to the devices it manages, so long as it does deliver them.

The other example cited above is of weather observations, sampled from standard web services.  The weather gateway talks HTTP over the Internet to a RESTful interface providing weather conditions.  As with all Huninn mesh sensors, the feed wakes, reads, publishes and sleeps and, by so doing, fulfils the gateway specification.

In summary, any form of sensor can be integrated into the Huninn mesh system via a software gateway.  Thus, any ‘sensor’, be it hardware or software, can be integrated into the system, publishing data that will be collated into the time-series database.  This is why the Huninn mesh system has wider applicability than just Huninn mesh network devices.

Virtualising a Device

The second significant abstraction is that device addressing.  

Recall that each physical Huninn mesh device has a globally unique identity assigned at the factory.  Devices may have multiple sensors ‘onboard’ (say, temperature, pressure and relative humidity). 

A simple (naive) addressing scheme might be to expose the GUID.Sensor to all users of the system.  This is not what's done, however.  Instead, end users of devices deal with a virtual device ID and this is assigned and managed by the device database.  A virtual device ID is a 64 bit integer.

The reasons why this approach has been adopted will be discussed in a later post when examining the characteristics of the time-series database.   Suffice to say for now that this provides considerable flexibility and significant performance advantages to both internal system components and end users of sensor data. 

The Device Base Period

One important constraint placed upon sensors providing measurements to the Huninn mesh system is that period in which they sample must be a power of two times the base period.  The base period is constant for the entire network.

To illustrate, the base period for the network might be 5ms.  Every sensor in this network is constrained so that it can only return samples at one of: 5ms, 10ms, 20ms, 40ms, 80ms, 160ms, 320ms, 640ms, 1280ms, 2,560ms, 5,120ms and so on. 

The importance of this rule will become apparent when considering how the time-series database, RESTful API and other services are implemented. 

Summary

This post has introduced a series of architectural ideas that underpin the server side of the Huninn mesh network.  A follow on post describing the time series database structure will make clear why each of these abstractions and constraints is useful.