How to convert a csv file to json file ?

Sometimes we may get dataset in csv format and need to be converted to json format.  We can achieve this conversion by multiple approaches. One of the approaches is detailed below. The following program helps you to convert csv file into multiline json file.  Based on your requirement, you can modify the field names and reuse this program.

The sample input is give below.

1001,Amal,Jose,100000
1002,Edward,Joe,100001
1003,Sabitha,Sunny,210000
1004,John,P,50000
1005,Mohammad,S,75000

 

Output multiline json is given below.

{"EmpID": "1001", "FirstName": "Amal", "LastName": "Jose", "Salary": "100000"}
{"EmpID": "1002", "FirstName": "Edward", "LastName": "Joe", "Salary": "100001"}
{"EmpID": "1003", "FirstName": "Sabitha", "LastName": "Sunny", "Salary": "210000"}
{"EmpID": "1004", "FirstName": "John", "LastName": "P", "Salary": "50000"}
{"EmpID": "1005", "FirstName": "Mohammad", "LastName": "S", "Salary": "75000"}

 

 

Basic statistics using Python

Python comes with a built-in statistics module. This will help us to perform the statistical calculations very easily.

The following are the commonly used statistical functions.

Arithmetic Mean

Arithmetic mean is the average of a group of values. The mathematical equation is

Mean = Sum of group of values / Total number of values in the group

Mean vs Average: What’s the Difference?

Answer: Both are same. No difference

Suppose we have a list of values as shown below.

values = [1,2,3,4,5,6,7,8]

For calculating the mean, without using any built-in function, we have to use the following snippet of the code

values = [1,2,3,4,5,6,7,8]
sum = 0
for value in values:
    sum += value

mean = sum/len(values)
print("Sum -->:", sum)
print("Total Count-->:", len(values))
print("Arithmetic Mean-->:", mean)

The above program involves multiple steps. Instead of writing the entire logic, we can easily calculate the mean using the following code snippet

import statistics
values = [1,2,3,4,5,6,7,8]
print("Arithmetic Mean--> ", statistics.mean(values))

Arithmetic Mode

Arithmetic mode refers to the most frequently occurred value in a data set. Mode can be calculated very easily using the statistics.mode() function

import statistics
values = [1,2,2,2,2,2,2,1,2,3,4,5,2,3,4,5,6,66,6,6,6,6]
print(statistics.mode(values))

Arithmetic Median

Median is basically the mid value in the numerical data set. The median is calculated by ordering the numerical data set from lowest to highest and finding the number in the exact middle. If the count of total numbers in the group is an odd number, the median will be the number which is in the exact middle of the ordered list. If the count of total numbers is an even number, then the median will be the mean of the numbers that reside in the middle of the ordered list.

This can be simply calculated by the statistics.median() function.

import statistics
values = [21,1,2,3,4,5,6,7,8,24,29,50]
print("Arithmetic Median--> ", statistics.median(values))

 

Functions as Objects in Python

Python is very powerful. It is easy to learn. Applications can be developed very quickly using python because of the simplicity.

Everything in python is an object. This includes functions also. Are you aware of the following features of functions in python. I was not aware during my initial few years.

  • Functions can be the elements inside various data structures like lists, dictionaries etc.

Few examples

Function as argument to another function

A Sample program in python to explain the implementation of using function as an argument of another function is given below.

Functions as elements within data structures like list or dict()

A simple implementation of passing list of functions as argument to another function is shared below.

I hope this will help someone. 🙂

 

What is a Stack ?. How to implement Stack in Python ?

What is a Stack ?

Stack is a structure in which items are stored and collected in LIFO order. LIFO means Last In First Out. We can see several stacks in our day to day life. A simple example of stack using paper is shown below. In this arrangement, the paper is stacked from bottom to top order and it will be taken back from top to bottom order.

stack

 

The insert and delete operations are often called push and pop. The schematic diagram of a STACK is given below. Here you can see how the items are pushed and taken out from the STACK.

 

stack01

In Python world, Stack can be implemented in the following methods.

  • list
  • queue.LifoQueue
  • collection.deque

 

Stack Implementation using LIST in Python

The native data structure list can be used as a stack. A simple list is given below.

[1,2,3,4,5,6,7,8]

The push operation can be performed by using the append() function in the list and the pop operation can be performed using pop() function. This usage of append() and pop() function will create a LIFO behavior and this can be used as a simple implementation of stack. The performance of the stack created using list will reduce with larger data. This is ideal for handling small amount of data.

The following program shows a simple implementation of stack using python list

 

Stack Implementation using LifoQueue (Queue) in Python

Stack can be implemented using the LifoQueue function in the Python Queue module. A simple implementation is given below. The program is self explanatory.

Stack Implementation using Deque in Python Collections module.

This approach is similar to that of the implementation using LIST. This will be more efficient than the implementation using the list. The sample program is given below. The program is self explanatory.

How to set Kafka Heap Size?

Setting Kafka Heap size is simple, by default Kafka runs with 512MB as the heap size. For increasing the heap size, set the following environment variable and restart Kafka.

export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"

Kafka will check for KAFKA_HEAP_OPTS before it starts and if there is no value set for this variable, it assigns 512MB as the value, else it will pick up the configured value.

Gunicorn Connection in Use: (‘0.0.0.0’, 8000)

I develop web services using python flask. One of the common error that I see while deploying the application is “Gunicorn Connection in Use: (‘0.0.0.0’, 8000)”.

This means that the port 8000 is busy with some other running process. But when I check the status of the port with the following command, I get empty response. That means there are no active application using the port. Some stale process is making the port busy.

netstat -tulpn | grep 8000

I even tried with the ps command to see any active process, but that also did not help.

ps -aux

If the ps command list the process, we can kill the process directly using the kill command

kill -9 {PID}

In my case I do not have the PID. So the only option to kill these kind of zombie application by using the below command.

sudo fuser -k {PORT}/tcp

In my case, the port number is 8000, so the command will be.

sudo fuser -k 8000/tcp

This trick helped me several times, hope this helps someone else also.

 

SELinux modes – Simple explanation

Everyone who uses linux might be familiar with SELinux. The full form of SELinux is Security-Enhanced Linux. It is a kernel level security module that enhances the access level security policies.

In this post I will be quickly explaining about the various modes in SELinux.

There are three modes in SELinux

  • Enforcing
  • Permissive
  • Disabled

In CentOS and RHEL systems , the SELinux configurations are controlled using the configuration file /etc/sysconfig/selinux.

The changes made to this file needs a system reboot. We can disable the SELinux permanently only with a system reboot. But we can set the SELinux into permissive mode without reboot. This can be easily performed by issuing a setenforce command. The details are explained in my another blog post.

Here we can set SELinux to any of the modes mentioned above.

In the Enforcing mode, SELinux is completely active and it will allow access only using the SELinux policies. User can configure the policies to enable access to their application.

In the Permissive mode, the SELinux will be monitoring and logging all the activities that would have been denied if it is in the enforcing state. The SELinux will not block any activities in this state.

In the Disabled mode, SELinux will be completely disabled.

Sample program to send email using Send Grid

A sample program to send email to multiple users using Send Grid is attached below. The user emails can be provided in the list.

 

The to_email specifies the recipients. The from_email specifies the sender. You can provide the recipient details either as a list of emails addresses or a list of tuples containing email address and the label.

That means

to_emails = ['receiver01@mail.com', 'receiver02@mail.com', 'receiver03@mail.com']

or

to_emails = [('receiver01@mail.com', 'Receiver 02'), ('receiver02@mail.com', 'Receiver 02'), ('receiver03@mail.com', 'Receiver 03')]

 

Also in the from_email if you are simply passing the email address, the recepient will receive an email with the sender name as the name in the email address. If you want proper labels in the email, provide the details in a tuple.

from_email=('amal@gmail.com', 'Amal G Jose')

You have to grab the token from the SendGridto get this email service enabled.

Green House Farming

What is Green House Farming ?

Green House Farming is a technique in which the crops are cultivated in a controlled enclosed environment. This is basically to control the effects due to whether changes.

Advantages of Green House

  • Plants will get an environment with consistent temperature and Humidity
  • Plants will be protected from birds and other organisms
  • The moisture content in the soil will not evaporate easily
  • Easy to control pests
  • Easy to maintain the fields
  • The environment will not get affected because of the external weather

The picture of one of the greenhouses that I visited in the recent past is shared below.

greenhouse

Rose plants are cultivated in this Green House. The plants are planted in well arranged lines and drip irrigation is established across the plantation.

The newly formed rose buds are wrapped with a net protector to maintain proper shapes and protect the buds from other damages. These nets will ensure controlled development during budding. A high quality rosebud will be large in size (long bud with a well formed, heavy base). If you observe closely, you can see these nets in the buds present in the above picture. A sample image of the rose bud net is posted below.

rosebud_net

How to clear/delete the cached Kerberos ticket ?

In Linux

kdestroy

 

In Windows

klist purge