How to set url length in Nginx Request (error code: 414, uri too large)

May 15, 2020 Leave a comment

Today I got one annoying error after deploying the new version of the web application in nginx web server. Initially I thought the web app was buggy, but when I inspected the requests and response, I found the following error.

error code: 414, uri too large

On checking more details around this, I found that this issue can be fixed by adjusting few configurations in nginx. The parameter to modify is large_client_header_buffers.

This parameter sets the maximum number and size of buffers used for reading large client request header. A request line cannot exceed the size of one buffer, or the 414 (Request-URI Too Large) error is returned to the client. A request header field cannot exceed the size of one buffer as well, or the 400 (Bad Request) error is returned to the client. Buffers are allocated only on demand. By default, the buffer size is equal to 8K bytes. If after the end of request processing a connection is transitioned into the keep-alive state, these buffers are released.

Syntax : large_client_header_buffers number size ;

The default value is 4 and the size is 8 KB. You can increase this value to a higher value to fix this issue.

large_client_header_buffers 16 128k;

If you are facing issues even after making these changes, then add the following configuration to the server block in nginx.

fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;

client_max_body_size 24M;
client_body_buffer_size 128k;

client_header_buffer_size 5120k;
large_client_header_buffers 16 5120k;

 

Hope this helps 🙂  Please comment if you are facing any issues.

 

Filed under Linux World, Tips and Tricks Tagged with , , , , , , , , , , , , , , , , , , , , ,

How to enable Compression in Nginx

May 15, 2020 Leave a comment

If you are using Nginx webserver, you have to try this out. I started exploring for this option when I faced slowness in rending the pages in one of the applications. When I checked the backend services, the web service API was given the responses quickly, but the page was taking more time to load. On further inspection, I figured out that the API was sending a a data of size 5 MB. So the browser will first download this data and this speed depends on the bandwidth.

I explored further and figured out the way of compressing API body using the webserver. This will speed up the API calls as the data transferred over the network will be very less. In my case, the 5 MB data became few KBs (size of compressed data). This will get extracted quickly in the browser and it gets rendered in the screen.

The required configurations are pasted below for quick reference.

Make the changes in the config file /etc/nginx/nginx.conf. Add the following content to the nginx.conf file.

gzip on;
gzip_vary on;
gzip_min_length 1024;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
gzip_disable "MSIE [1-6]\.";

 

The explanation of each of the parameters are given below.

  • gzip on; – enables gzip compression in nginx
  • gzip_vary on: – tells proxies to cache both compressed and normal versions of a resource
  • gzip_types – The type of files that can be compressed
  • gzip_disable “MSIE [1-6]\.”;  – Disable compression for Internet Explorer versions 1-6
  • gzip_min_length 1024; – Nginx will compress the files greater than this size and it will not compress anything less than this size.
  • gzip_proxied – compress data even for clients that are connecting via proxies

 

This is one of the way to improve the performance of APIs. The developers can focus on enhancing their logic and the admin or DevOps engineer can focus on enhancing the web server with optimal configuration. There are so many tricks and magics we can do in the web server. I will try to post some of those in my blog.

Hope this helps. 🙂 . Please comment if you face any issues.

Filed under Linux World Tagged with , , , , , , , , , , , , , , , , ,

Production deployment of a Python Web Service (Flask / Tornado Application)

May 3, 2020 Leave a comment

Python Flask and Tornado are two of the most popular frameworks in python for developing RESTful services.

Do you know how to develop and deploy a production grade python application. ?

A sample python flask service is given below. This is a sample flask web service. This has only one endpoint (/requestme) at is a GET method. (sample_flask.py). I am not focusing on the coding standards. My goal is to show you the production implementation of a python application.

We can run this program in the command line by executing the following command.

> python sample_flask.py

The service will be up and running in port 9090. You will be able to make requests to the application by using the URL http://ipaddress:9090/requestme.

How many requests will this python web service can handle ? 

10 or 20 or 100 ?? … Any guess ??

Definitely this is not going to handle too many requests. This is good for development trials and experimental purpose. But we cannot deploy something like this in production environment.

How to scale python applications  ?

Refer to the below diagram. The diagram has multiple instances of flask applications with Gunicorn WSGI proxied and load balanced through Nginx web server.

haproxy_python

Production Deployment of Python Flask Application

Sample Nginx configuration that implements the reverse proxy and load balancing is given below. 

This is a sample configuration and this does not have the advanced parameters.

server {
listen 80;
server_name myserverdomain

location / {
proxy_pass http://upstream_backend/requestme;
  }
}

upstream backend {
server gunicornapplication1:8080;
server gunicornapplication2:8080;

}

 

The upstream section routes the requests to the two gunicorn backends and the requests are routed in round robin manner. We can add as many backend servers as we need based on the load.

How to run the python applications with gunicorn ?

First lets install gunicorn

> pip install gunicorn

Now it is simple, run the following command.

> gunicorn -w 4 app:app

Now the our application will run with 4 workers. Each worker is a separate process and will be able to handle requests. The gunicorn will take care of handling the requests between each of the workers.

We can start multiple gunicorn instances like this and keep it behind the nginx. This is the way to scale our python applications.

Hope this helps 🙂 

Filed under Linux World, python, Tips and Tricks Tagged with , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

“The Zen of Python”, by Tim Peters

May 2, 2020 1 Comment

Every Python Developer should try and read these statements periodically. In the python interpreter, type the following statement.

> import this

You will see the following response. Read it and refresh.

python_zen

Filed under python, Tips and Tricks Tagged with , , , , , , , , , , , , , , , , , ,

Switch Case Statements in Python

May 2, 2020 Leave a comment

Switch case statements are very popular conditional control statements in almost all programming languages. But surprisingly this is not available in python.

Question: Is there any switch case statements in python ?

Answer: The direct answer is NO

Alternative options for switch case statements in python

Option 1: Using If – elif – else statements. An example is given below.

if case == "case1":
    execute_func_case1()
elif case == "case2":
    execute_func_case2()
elif case == "case3":
    execute_func_case3()
else:
    execute_default_func()

Wow. Excellent.  The above code looks good right ?. It works exactly like switch-case statements, then why need switch-case statements in Python ?

Have you noticed a problem ?. The above if-elif-else conditions are fine as long as we have less number of cases. Imagine the situation with 10 or more elif conditions. Now you got the problem right ?.

Lets try the second option

Option 2: Using List in Python as an alternative to switch case statements

An example is given below.

def add(a, b):
    return a + b

def sub(a, b):
    return a-b

case_funcs = [add, sub]

case_funcs[0](1, 2)
case_funcs[1](1, 2)

 

In the above program, we don’t have to use if-elif-else blocks, instead, we can call using the position or index of the list and call the function. This looks better than the previous option right ?. But what about the default case ?. Also what if someone types an option greater than the size of the list ?. It will throw exception and there is no way to handle default case.

Option 3: Using Dictionary as alternative to switch case statements in python

An example is given below.

def add(a, b):
    return a + b

def sub(a, b):
    return a-b

case_funcs = {'sum':add, 'subtract':sub}

case_funcs['sum'](1,2)

 

Here the implementation is much similar to the switch case statement. We use a key to identify or route to the required case or function. The keys can be anything and are not limited by the indices or positions.

Now lets talk about the drawbacks of the above implementation. The above method will throw KeyError if we pass an unknown key. Also there is no default case statement. How will we handle these problems?

Check the below program

def add(a, b):
    return a + b

def sub(a, b):
    return a-b

def default(a, b):
    return "Default Return"

case_funcs = {'sum':add, 'subtract':sub}

# sum is the key for the add(). 
# default is the default function that gets called for non existent keys
# (1, 2) are the arguments for the function
print(case_funcs.get('sum',default)(1,2))

 

Python dictionary has a get() method that returns the value based on the key. This has one more feature. We can configure a default value for non-existent keys. Wow now we got the solution.

So by using this feature, we can implement the switch-case like feature in python.

Filed under python, Tips and Tricks Tagged with , , , , , , , , , , , , , , , , ,

Route internet traffic through a specific interface in Linux Servers – CentOS / RHEL

April 29, 2020 Leave a comment

I have a server with multiple network interfaces. Out of that two of them were connected to two different networks. My requirement is to route the internet traffic through the second interface. The Server is running with CentOS 7 operating system. The second interface is connected to a network with higher internet bandwidth and the first interface is connected to a network with lower bandwidth. So I have to make the second interface as the primary/default interface.

The details of the network interfaces are given below.

Details of first interface:

Interface name -> eno2

IP Address:  192.168.0.208

Gateway 192.168.0.1

Internet Provider : ISP01

Details of second interface:

Interface name -> eno3

IP Address:  172.31.0.208

Gateway 172.31.0.1

Internet Provider : ISP02

The diagrammatic view of the server and network connections are given below.

internet_routing

Problem statement:

The internet traffic is going through the first interface. I want to change it to the second interface.

How to check the route of the internet traffic ?

This can be checked by using the traceroute command. Execute the following commands and check hops. If it is going through the gateway of the first network, means traffic is routed through the first interface. If it is going through the gateway of the second network, means that the traffic is routed through the second interface.

traceroute google.com

When I executed this command, I got the following output that proves that the traffic was going through the first interface. 192.168.0.1 is the gateway of the first interface. You can see the details in the below screenshot.

traceroute_02

How to list the default traffic routes ?

To list the existing routes in the system, type the following command in the terminal.

ip route list

This will list all the routes and that shows the default routes also. The following screenshot shows the details of the routes in my system.

iproute_list01

In the above image, you can see two default routes. The first one has a priority of 102 and second one with priority 103. So based on the priority, the traffic goes through the first interface (gateway 192.168.0.1, ISP-01).

How to change the default / primary route ?

In my case there were two default routes. So making one interface as the default or primary route will solve the problem.

This can be enabled by configuring the DEFROUTE parameter in the network interface settings. The network interface configurations are present in the following directory.

/etc/sysconfig/network-scripts

In this directory, you can see files that starts with ifcfg-. In my case, the two files that I have to deal with are ifcfg-eno2 and ifcfg-eno3.

In this file, there will be a parameter DEFROUTE. If it is configured with value “yes” means that interface is a default route. If the value is “no” means it is not a default interface.

So make DEFROUTE=no in the first interface (ifcfg-eno2) and DEFROUTE=yes in the second interface (ifcfg-eno3).

Restart the network after making these changes. The command is shared below.

service network restart

Now check the route list and see the default route/s. The command is given below.

ip route list

The screenshot from my system is shared below.

iproute_list02

Now you can see that the second interface (gateway 172.31.0.1 & ISP-02) became the default interface and the first interface got removed from the default list. It is present in the available interfaces, but not the default interface.

Now let us test the internet traffic through traceroute command. As per our configuration, the traffic should go through the second interface. The command is given below.

traceroute google.com

The screenshot from my system is given below.

traceroute_01

As per the screenshot, the traffic is going as expected based on our configuration. It is going through the second interface.

The steps for Ubuntu and other operating systems are also similar. Here I have explained based on CentOS & RHEL operating systems.

Hope this article helps someone. 🙂

 

 

 

 

Filed under Linux World Tagged with , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

bash: ifconfig: command not found debian centos ubuntu rhel

April 27, 2020 Leave a comment

This article will be helpful for Ubuntu, CentOS, RHEL, Debian, Raspbian OS users.

ifconfig is a very commonly used command to find the ipaddress of a linux machine. Sometimes we may get an error like

bash: ifconfig: command not found

The reason for this can be because of one of the following reasons.

  • The ifconfig command is not available in the system.
    • In case of this issue, we have to install the net-tools package
  • The ifconfig script is not available in the PATH
    • In this case, we have to try executing /sbin/ifconfig command

In these cases, we can get the ip address details by using another command. We don’t have to install any additional package for issuing the following command

ip addr

If the net-tools package is missing in the system, we can enable ifconfig by installing the package. The command to install the package is given below.

For Ubuntu, Raspbian, Debian OS use the following command

sudo apt-get install net-tools

For RHEL / CentOS users, use the following command

sudo yum install net-tools

Now try ifconfig in the commandline and you will be able to see the ip address details.

 

Filed under Linux World Tagged with , , , , , , , , , , , , , , , ,

How to install Chromium browser in Raspberry Pi debian ?

April 27, 2020 Leave a comment

By default, there will not be any browser included in the debian OS for raspberry Pi. Chromium is one of the best browser for raspberry pi. We can install chromium browser in raspberry pi debian OS using the following simple steps,

sudo apt-get update
sudo apt-get install chromium-browser

Now after doing this, check the home icon in raspberry pi and you will be able to see an additional option “Internet” and on expanding, you can see chromium browser. All set 🙂

Filed under Linux World Tagged with , , , , , , , , , , , , , , , , , ,

Simple Python Program to get the system status of a raspberry Pi

April 27, 2020 Leave a comment

The following program helps you to monitor the status of a raspberry pi. The common parameters that we monitor are CPU, Memory & Disk. This program can be used to get the info of any operating system. This is not limited to raspberry pi.

Here I am providing a simple program that provides the details of all these three parameters. You can enhance this program further and make it part of your application.

Filed under Linux World, python Tagged with , , , , , , , , , , , , , , , , , ,

Covid-19 patient monitoring using Raspberry Pi

April 27, 2020 Leave a comment

Image Source: CBC Canada

A team from the University of Toronto has created a new way for health-care workers to monitor COVID-19 patients — without having to set foot in their hospital rooms.

The university got a call from Mount Sinai hospital with a requirement to figure out a way to monitor vital signs both continuously and remotely.

The students developed a raspberry pi based continuous remote monitoring system for monitoring the status of the patients without the direct intervention of the nurses.

They attached a standard fingertip probe, already in use in hospitals to monitor vital signs, to a “very, very small computer about the size of a credit card,”

That simple computer, called a Raspberry Pi, can then connect to the internet — allowing health-care workers to check on patients from any nursing station computer, or on their smartphone.

The concept is very simple, the finger clip monitors the respiratory status of the COVID-19 patients. The clip detects the light absorbed by the patient’s blood. The colour vary depending upon the oxygen level in the blood and this is the key factor to monitor. This data will be captured from the finger clip by the Raspberry pi and it gets transmitted to the devices in the nursing station.

The prototype was successful based on several trial deployments made in the Mount Sinai hospital.

References:

https://www.cbc.ca/news/canada/toronto/u-of-t-covid-19-monitoring-system-1.5540089

University of Toronto supports COVID-19 patient monitoring with Raspberry Pi

 

 

Filed under General Tagged with , , , , , , , , , , , , , , , ,

← Older posts

Newer posts →