Realtime Temperature Monitoring System using Raspberry Pi

Realtime temperature sensing is one of the common requirement. There are a lot of digital thermometers and temperature monitoring devices available in online shopping sites. But most of them just monitors and displays the realtime values. It does not have any intelligence.  The one we are going to build is a smart temperature monitoring system. This system can be used for monitoring atmospheric temperature as well as liquid temperature.

The following blog post explains the set up of a digital temperature monitoring system.

Digital Temperature Monitoring System

We will enhance the above system by adding analytical capability. So that we can analyse and show the temperature trends. The block diagram below shows the high level architecture of the system.

temperature_monitoring

As shown in the above diagram, the system has three blocks.

  • Edge Device & Sensor (Raspberry Pi & Sensor)
  • Data Storage Server
  • Dashboards for the end user

The following are the software components required for this project

  • MQTT for sending the data from the edge to the server.
  • PostgreSQL for storing the data in the server.
  • Python based backend
  • HTML Web UI

I am not going to explain the working of MQTT in this blog post. This was already explained in one of my earlier posts.

Before we start implementing the solution, lets summarize the story line.

  • The requirement is to perform realtime temperature monitoring and analyse the trends & patterns using the historic data.
  • A temperature sensor is attached to a Raspberry Pi which acts as the edge device.
  • Need provision to support multiple edge devices.
  • Capability to monitor the temperature from anywhere

Bird’s eye view of the system

Here we have considered multiple edge devices and also considered the provision of web and mobile application.

temperature_monitoring_full

Data Model Design

In the PostgreSQL database, we need two base tables for storing the data. We will be able to store data from multiple edge devices located at different locations using this data model. This is a very basic data model. We can enhance this based on our requirement.

  • device_info – This has the metadata of the edge devices. This includes the location details of the device. The column names are given below
    • device_id, device_name, location
  • temperature_data – We store the temperature data from each of the edge devices in this tables. The column names are given below.
    • device_id, timestamp, value

Now let us start developing the application from the edge device. We will modify the program to send the messages to an MQTT topic with the timestamp. The temperature readings will be sent to the server once in every minute. The message format will be as follows. We will be using epoch timestamp in seconds and temperature in Degree Celsius.

{"device_id":"xxx", "timestamp":1584284353, "value": 27.01}

Now lets develop a small python program that send this values to the MQTT topic. For this, we need an MQTT broker to be up and accessible from the Raspberry Pi

Here my central server is a CentOS 7 server and I will be using mosquitto MQTT. The installation steps are explained very detailed in this blog post.

In the central server, these messages will be collected and stored in the database tables.

A sample view of the temperature_data table is shown below.

device_id timestamp value
device_01 1587825234 27
device_02 1587825234 28
device_03 1587825234 23
device_04 1587825234 28
device_05 1587825234 30
device_06 1587825234 26
device_07 1587825234 22
device_08 1587825234 28
device_09 1587825234 32
device_10 1587825234 29
device_11 1587825234 31

Now from this table, we can query and get the required information based on the user requirement. We can either develop custom visualization using javascript or we can query the DB using workbenches or we can even connect & visualize data using visualization tools like Apache Superset, PowerBI etc.

timeseries_chart01

With this I have explained the highlevel architecture and implementation of a sample IoT system. This system can be scaled further by using a proper time series database instead of the Postgres DB.

Sample program to send email using Send Grid

A sample program to send email to multiple users using Send Grid is attached below. The user emails can be provided in the list.

 

The to_email specifies the recipients. The from_email specifies the sender. You can provide the recipient details either as a list of emails addresses or a list of tuples containing email address and the label.

That means

to_emails = ['receiver01@mail.com', 'receiver02@mail.com', 'receiver03@mail.com']

or

to_emails = [('receiver01@mail.com', 'Receiver 02'), ('receiver02@mail.com', 'Receiver 02'), ('receiver03@mail.com', 'Receiver 03')]

 

Also in the from_email if you are simply passing the email address, the recepient will receive an email with the sender name as the name in the email address. If you want proper labels in the email, provide the details in a tuple.

from_email=('amal@gmail.com', 'Amal G Jose')

You have to grab the token from the SendGridto get this email service enabled.

What is MQTT and Where is it used widely ?

MQTT is a very light weight protocol used commonly in sensor communications. MQTT stands for MQ Telemetry Transport. Unlike other messaging protocols, this is very simple, light weight and requires only very low bandwidth. Because of these features, it is widely used in sensor networks. Now with the emerge of Internet of Things, MQTT became very popular. Since it is very light, it consumes very less power and is ideal for low power applications. MQTT messages are delivered asynchronously through a publish-subscribe mechanism. The message packets are specially crafted in such a way to reduce the data size.

The basic communication model in an MQTT system is shown below.

mqtt

Common terminologies in MQTT

Publisher – The one who publishes the messages. Usually the sensors emits the messages. For example in a temparature monitoring scenario, the temparature sensor emits the temparature frequently and the values are displayed in the mobile application. Here the sensor is publisher and the mobile application is the subscriber.

Subscriber – Subscriber is the one who subscribes the messages.

MQTT Client – A client can be a subscriber or publisher. A single client can act as a publisher as well as subscriber. MQTT client libraries are available in almost all programming languages. The complete list is available in MQTT wiki.

MQTT Broker/Server -All the clients are connected to the broker. This is the heart of the MQTT system. Depending upon the implementation, a broker can handle thousands of concurrently connected MQTT clients. The broker needs to be configured in such a way that it is accessible to all the clients. A client can be configured within any NAT network, but the broker needs to directly accessible from all the clients. There are several broker variants available. Some of them are Mosquitto, HiveMQ, Mosca, emqttd etc. Based on your choice and your application needs, you can choose one among these.

Topic – This is a UTF-8 string used by the broker to filter the messages for each connected client. There can be N number of topics. Topic is case sensitive.

MosquittoMQTT is one of the widely used MQTT servers. I have used Mosquitto in many projects. This can be easily installed in Linux and Windows.

By default MQTT uses TCP port 1883 for communication and for SSL enabled communication, it uses port 8883. These are configurable values and it can be changed.

Security

The messages can be secured with username and password. Communication with SSL is also possible. Apart from these, there is no advanced security mechanism present in MQTT. But we can have our own custom message encryption/decryption logic.

There are a lot to explain about MQTT. Instead of writing more, I feel it is better to demonstrate the working of a real implementation. I will be publishing a post on the real implementation of MQTT soon.

HDFS Operations Using Java Program

We are familiar with Hadoop Distributed File System operations such as copyFromLocal, copyToLocal, mv, cp, rmr etc.
Here I am explaining the method to do these operations using Java API. Currently I am explaining the programs to do copyFromLocal and copyToLocal functions only.

Here I used eclipse IDE for programming which is installed in my windows desktop machine.
I have a hadoop cluster. The cluster machines and my destop machine are in the same network.

First create a java project and inside that create a folder named conf. Copy the hadoop configuration files (core-site.xml, mapred-site.xml, hdfs-site.xml) from your hadoop installation to this conf folder.

Create another folder named source which we are using as the input location and put a text file inside that source folder.
One thing you have to remember is that the source and destination locations will be given appropriate permissions. Otherwise read/write will be blocked.

Copying a File from Local to HDFS

The command is
hadoop fs -copyFromLocal

package com.amal.hadoop;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

/**
 * @author amalgjose
 *
 */
public class CopyFromLocal {

	public static void main(String[] args) throws IOException {
		
		Configuration conf =new Configuration();
		conf.addResource(new Path("conf/core-site.xml"));
		conf.addResource(new Path("conf/mapred-site.xml"));
		conf.addResource(new Path("conf/hdfs=site.xml"));
		FileSystem fs = FileSystem.get(conf);
		Path sourcePath = new Path("source");
		Path destPath = new Path("/user/training");
		if(!(fs.exists(destPath)))
		{
			System.out.println("No Such destination exists :"+destPath);
			return;
		}
		
		fs.copyFromLocalFile(sourcePath, destPath);
		
	}
}

Copying a File from HDFS to Local

The command is
hadoop fs -copyToLocal

package com.amal.hadoop;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
/**
 * @author amalgjose
 *
 */
public class CopyToLocal {
public static void main(String[] args) throws IOException {
		
		Configuration conf =new Configuration();
		conf.addResource(new Path("conf/core-site.xml"));
		conf.addResource(new Path("conf/mapred-site.xml"));
		conf.addResource(new Path("conf/hdfs=site.xml"));
		FileSystem fs = FileSystem.get(conf);
		Path sourcePath = new Path("/user/training");
		Path destPath = new Path("destination");
		if(!(fs.exists(sourcePath)))
		{
			System.out.println("No Such Source exists :"+sourcePath);
			return;
		}
		
		fs.copyToLocalFile(sourcePath, destPath);
		
	}
}

Simple Tag Cloud Generation Using Java program

A Tag cloud is a visual representation of text data. In this tags are words, where the importance is highlighted using colour or font size. This is very popular now to analyse contents of websites. This helps in quickly perceiving the most important words. The importance is calculated by counting the number of occurance. Thus based on occurance, weightage is given to each word(tag). After analysing the whole text, it is displayed based on it weightage. Thus tag cloud will be generated. open cloud is a java library for generating tag clouds. Here I used Open cloud library for the generation of Tag cloud. Normally we need a webserver for getting a good UI of the TagCloud, here we are displaying the cloud using Swing. This is a sample program for the generation of a simple tag Cloud. For this download the Open Cloud Library.

package tagcloud;

import java.util.Random;

import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.SwingUtilities;

import org.mcavallo.opencloud.Cloud;
import org.mcavallo.opencloud.Tag;

public class TestOpenCloud {

private static final String[] WORDS = { "amal", "india", "hello", "amal", "birthday", "amal", "hello", "california", "america", "software",
 "cat", "bike", "car", "christmas", "city", "zoo", "amal", "asia", "family", "festival", "flower", "flowers", "food",
 "little", "friends", "fun", "amal", "outing", "india", "weekend", "india", "software", "me", "music", "music", "music",
 "new", "love", "night", "nikon", "morning", "love", "park", "software", "people", "portrait", "flower", "sky", "travelling",
 "spain", "summer", "sunset", "india", "city", "india", "amal", "uk", "usa", "", "water", "wedding","cool","happy","friends","best","trust","good",
 "enjoy","cry","laugh"};

protected void initUI() {
 JFrame frame = new JFrame(TestOpenCloud.class.getSimpleName());
 frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
 JPanel panel = new JPanel();
 Cloud cloud = new Cloud();
 Random random = new Random();
 for (String s : WORDS) {
 for (int i = random.nextInt(50); i > 0; i--) {
 cloud.addTag(s);
 }
 }
 for (Tag tag : cloud.tags()) {
 final JLabel label = new JLabel(tag.getName());
 label.setOpaque(false);
 label.setFont(label.getFont().deriveFont((float) tag.getWeight() * 10));
 panel.add(label);
 }
 frame.add(panel);
 frame.setSize(800, 600);
 frame.setVisible(true);
 }

public static void main(String[] args) {
 SwingUtilities.invokeLater(new Runnable() {
 @Override
 public void run() {
 new TestOpenCloud().initUI();
 }
 });
 }

}