Stream Processing Framework in Python – Faust

March 14, 2020 Leave a comment

I was looking for a highly scalable streaming framework in python. I was using spark streaming till now for reading data from streams with heavy through puts. But somehow I felt spark a little heavy as the minimum system requirement is high.

Last day I was researching on this and found one framework called Faust. I started exploring the framework and my initial impression is very good.

This framework is capable of running in distributed way. So we can run the same program in multiple machines. This will enhance the performance.

I tried executing the sample program present in their website and it worked properly. The same program is pasted below. I have used CDH Kafka 4.1.0. The program worked seamlessly.

To execute the program, I have used the following command.

python sample_faust.py worker -l info

The above program reads the data from Kafka and prints the message. This framework is not just about reading messages in parallel from streaming sources. This has integrations with an embedded key-value data store RockDB. This is opensourced by Facebook and is written in C++.

Filed under python Tagged with , , , , , , , , , , , , , , , , ,

Green House Farming

February 22, 2020 Leave a comment

What is Green House Farming ?

Green House Farming is a technique in which the crops are cultivated in a controlled enclosed environment. This is basically to control the effects due to whether changes.

Advantages of Green House

  • Plants will get an environment with consistent temperature and Humidity
  • Plants will be protected from birds and other organisms
  • The moisture content in the soil will not evaporate easily
  • Easy to control pests
  • Easy to maintain the fields
  • The environment will not get affected because of the external weather

The picture of one of the greenhouses that I visited in the recent past is shared below.

greenhouse

Rose plants are cultivated in this Green House. The plants are planted in well arranged lines and drip irrigation is established across the plantation.

The newly formed rose buds are wrapped with a net protector to maintain proper shapes and protect the buds from other damages. These nets will ensure controlled development during budding. A high quality rosebud will be large in size (long bud with a well formed, heavy base). If you observe closely, you can see these nets in the buds present in the above picture. A sample image of the rose bud net is posted below.

rosebud_net

Filed under Internet of Things Tagged with , , , , , , , , , , , , , , , , ,

How to check whether a Raspberry Pi is 32 bit or 64 bit ?

August 9, 2018 Leave a comment

The latest version of Raspberry Pi comes with 64 bit CPU, but prior to that it was with 32 bit CPU. Some softwares and applications are dependent on CPU and OS architecture.

There are various options to check the architecture.

Method 1:

type the following command and check the response

uname -m

You will get a response something like armv7l or armv8.

ARMv7 and below are 32-bit. AMRv8 introduces the 64-bit instruction set.

Method 2:

Install lshw using the command

apt-get install lshw

Then type the command lshw.  You will be able to find the architecture from the response of the command.

Filed under Linux World, Tips and Tricks Tagged with , , , , , , , , , , , , ,

What is MQTT and Where is it used widely ?

June 29, 2018 Leave a comment

MQTT is a very light weight protocol used commonly in sensor communications. MQTT stands for MQ Telemetry Transport. Unlike other messaging protocols, this is very simple, light weight and requires only very low bandwidth. Because of these features, it is widely used in sensor networks. Now with the emerge of Internet of Things, MQTT became very popular. Since it is very light, it consumes very less power and is ideal for low power applications. MQTT messages are delivered asynchronously through a publish-subscribe mechanism. The message packets are specially crafted in such a way to reduce the data size.

The basic communication model in an MQTT system is shown below.

mqtt

Common terminologies in MQTT

Publisher – The one who publishes the messages. Usually the sensors emits the messages. For example in a temparature monitoring scenario, the temparature sensor emits the temparature frequently and the values are displayed in the mobile application. Here the sensor is publisher and the mobile application is the subscriber.

Subscriber – Subscriber is the one who subscribes the messages.

MQTT Client – A client can be a subscriber or publisher. A single client can act as a publisher as well as subscriber. MQTT client libraries are available in almost all programming languages. The complete list is available in MQTT wiki.

MQTT Broker/Server -All the clients are connected to the broker. This is the heart of the MQTT system. Depending upon the implementation, a broker can handle thousands of concurrently connected MQTT clients. The broker needs to be configured in such a way that it is accessible to all the clients. A client can be configured within any NAT network, but the broker needs to directly accessible from all the clients. There are several broker variants available. Some of them are Mosquitto, HiveMQ, Mosca, emqttd etc. Based on your choice and your application needs, you can choose one among these.

Topic – This is a UTF-8 string used by the broker to filter the messages for each connected client. There can be N number of topics. Topic is case sensitive.

MosquittoMQTT is one of the widely used MQTT servers. I have used Mosquitto in many projects. This can be easily installed in Linux and Windows.

By default MQTT uses TCP port 1883 for communication and for SSL enabled communication, it uses port 8883. These are configurable values and it can be changed.

Security

The messages can be secured with username and password. Communication with SSL is also possible. Apart from these, there is no advanced security mechanism present in MQTT. But we can have our own custom message encryption/decryption logic.

There are a lot to explain about MQTT. Instead of writing more, I feel it is better to demonstrate the working of a real implementation. I will be publishing a post on the real implementation of MQTT soon.

Filed under Internet of Things Tagged with , , , , , , , ,

Booting Raspberry Pi2 for the first time

July 6, 2015 Leave a comment

Today I and my colleagues were trying to install Raspbian OS on Raspberry Pi 2 B version. The steps that we followed are explained below.
Before explaining the steps, I am listing down the hardware and softwares that we used.

  • Raspberry Pi2 B version
  • NOOBS 1.4.1 Offline and Network Installer (Release date 2015-05-11).
  • Transcent 16 GB Class 10 memory card
  • Raspberry Pi charger (We used a 5Volt 2A charger)
  • Memory Card Reader.
  • SD Formatter (https://www.sdcard.org/downloads/formatter_4/)
  • Windows Laptop
  • Monitor
  • HDMI Cable

We used the steps mentioned in the raspberry pi website for reference. There is an excellent video that explains each and every step for the setup.

pi21GB
The steps we followed are listed below.

  • Inserted the memory card into the Card reader and plugged into the windows machine.
  • Formatted the memory card using the SD Formatter with “FORMAT SIZE ADJUSTMENT” set to “ON”.
  • Copied the NOOBS offline installer zip file to the formatted memory card and extracted it.
  • Unplugged the card reader and put the memory card into the Pi2.
  • Supplied power to the Pi2 which is connected to the monitor using HDMI Cable
  • The Pi2 booted up. We have to select the OS to be installed.
  • Select one OS. Now the OS will be installed and your pi will be ready to use.

Note:
We faced an issue while performing the installation. The Pi2 was not booting up. Finally after some struggle, we figured out the issue. We used a phone instead of a memory card reader for connecting the memory card to the computer for formatting. Because of this, the Pi was not booting. We figured out this issue from the status of the ACT LED (Green LED). The green LED should blink in case of normal operation. If the green LED is not blinking, means the Pi is not reading the SD card. In our case the green LED was not blinking. So we started investigating and reviewing the steps that we followed for the SD card set up and finally we found the culprit… Our Mobile Phone . So my suggestion is to use a card reader. Never use any camera or mobile phone instead of card reader. This may unnecessarily waste your time and you will not get any clue of the error also.

Filed under Internet of Things Tagged with , , , , , , , , , , , , ,

Wifi Adapter Set up for Rasperry Pi

April 29, 2015 5 Comments

For connecting Raspberry Pi with internet, we have three options, one is through LAN cable, another is using a supported internet dongle and the other way is to using a wifi adapter. Here explaining about the steps that I did for configuring a Netgear N150 WNA1000MV2 wifi adapter with Raspberry pi. Raspberry pi supports lot of wifi adapters. The list of the hardware supported by raspberry pi is available in this url. I had one Netgear Wifi adapter (N150 WNA1000M) and it was working fine without any config changes. Because of that reason, I bought another wifi adapter with the same model number from an online shopping site. But when I unpacked it, I realized that they upgraded the product and a new version of the same product came with model number WNA1000MV2. When I connected it with the pi, it was not working. My raspberry pi OS was Raspbian.

Raspberry-Pi

Netgear1

A portion of the output of dmesg command that contains the details of the wifi adapter is shown below.

dmesg

[ 3.401856] usb 1-1.2: new high-speed USB device number 4 using dwc_otg
[ 3.523552] usb 1-1.2: New USB device found, idVendor=0846, idProduct=9043
[ 3.532493] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3.541771] usb 1-1.2: Product: WNA1000Mv2
[ 3.547744] usb 1-1.2: Manufacturer: Realtek
[ 3.553858] usb 1-1.2: SerialNumber: 00e04c000001

To make this working, we have to do some modifications.
Add the following contents to the /etc/rc.local file

modprobe 8192cu
echo "0846 9043" > /sys/bus/usb/drivers/rtl8192cu/new_id
ifdown wlan0
ifup wlan0

Then add the following contents to /etc/modprobe.d/netgear_n150.conf file

install 8192cu /sbin/modprobe --ignore-install 8192cu; echo "0846 9043" > /sys/bus/usb/drivers/rtl8192cu/new_id

This worked perfectly for me. Hope this will help..!!!

Filed under Internet of Things Tagged with , , , , , , , , , , , , , , ,

A Small experiment using GPS module

February 9, 2015 3 Comments

Full form of GPS is global positioning system. GPS is one of the very superb inventions that help us very much. GPS finds the position by making satellites as reference. It is accurate upto some meters. Now a days almost everyone in the world are using gps for navigation. Here I am sharing my experience with a gps module. GPS modules are available in electronic shops. The picture of a gps module is shown below.

gps1The gps unit that I bought is having an RS232 as well as a TTL interface. We can connect this device to a computer, raspberry pi or any micro-controller very easily. I connected and did the experiments using this device on raspberry pi, laptop and micro-controller. Here I am explaining a sample python program that will find the coordinates using this gps module. I used a python library called pynmea for reading and parsing the data returned by the gps module.

First power up the gps module. This depends on the type of gps module that you are using. The module I used supports 12V and 5V input voltage. Then connect the gps module to your computer using the RS232. If your laptop or PC is not having the RS232 port, don’t worry, you can use an RS232 to USB converter cable. Check the serial port in which this device is connected (This can be found from the device manager). Before we start writing the program, ensure that the pynmea library is installed in your computer.

First we will check whether the device is working properly. So we will write a small python program to read the data from the serial port.

__author__ = 'Amal G Jose'

import serial
ser = serial.Serial()
ser.baudrate = 9600
ser.port = 'COM5'
ser.open()
while True:
    print ser.readline()

In my case the device was connected to COM5 port. Change the port accordingly and execute this program. If you are getting values in the console, then you are good to go.

Then execute the following program to parse the data and find the coordinates.

Filed under Internet of Things Tagged with , , , , , , , , , , ,

Some keys do not produce the correct characters in Raspberry Pi

February 4, 2015 Leave a comment

I have a raspberry pi B+ with raspbian OS installed. In this set up, most of the keys were not producing the proper characters. Among the characters, ‘|’ was the character that I missed a lot, because I was not able to do any grep in the linux command line. I tried re-configuring the keyboard, but that didn’t helped me. Then I found the solution.

I changed the keyboard layout to US.

The command for doing the same is given below.

setxkbmap -layout us

If the above solution is not working, you can manually edit the following file and change the keyboard layout. Change the layout to US (use small letter).

/etc/default/keyboard

Filed under Internet of Things Tagged with , , , , , , , , , ,

We connected computers in the past, Now computers are connecting us…!!

January 24, 2015 Leave a comment

A new revolution is going to happen in the world with connected devices and human life. We can call it as Internet of Things. Now lot of discussions and experiments are happening around the world to make this concept a reality. This is a very simple concept from the point of view of a person who knows electronics. But the problem is that for making this internet of things a reality and to make it usable by every individuals, it needs much more effort. For developing an application that makes our life easier by connecting things, we need the skillsets of a software engineer and electronics engineer. An electronics engineer can design the circuit, wire up and get the signals in the proper way. But those signals are just some electrical signals which will be useful only to the technical people. These systems were already in the past. Then what is different in IoT..??

IoT includes connecting systems, sensing each and every members in the network and controlling of every single node in the network based on the feedback. The control may not be based on a pre-written script, it can be dynamic. This requires extensive analytic knowledge along with electronics knowledge. If we sense and control every member of a large network, the data size will be high. Here we have big data solutions. So if we combine the electronics and analytic knowledge, we can build superb systems that can revolutionize the world. If we make everything connected and everything under control, there are some pros and cons. This will be just like the atomic energy. Good people used it for energy sustenance by generating energy and bad people used it as atom bomb for destroying people. Hope IoT will be used in the right way..!!!!

Filed under General Tagged with , , , , ,

GLens – An Environment Saviour

December 16, 2014 Leave a comment

GLens enables Real Time Data Acquisition and Monitoring for Industrial Emissions, Effluent and Ambient pollution. Any analyzers make or model with RS232/RS485/RS222/Modbus is seamless integrated in a plug and play model. Provides pre-build reports,alarms and alerts as per the regulatory standards for Pollution monitoring.
The Ad-hoc reporting module provides capability to analyze the data upto 2 second granularity over the collected parameters and provide forecasting using Holt-Winters forecast models. GLens integrates with any analyzer (make or model) in a plug and play mode and enabling real time monitoring of the parameters instantaneously.
With open data exchange format any analyzer, manufacturer or industry could enable connectivity of these monitored parameters on to the platform.
For more details visit the following link GLens

Filed under BigData World Tagged with , , , , , , ,