Python code to list all the running EC2 instances across all regions in an AWS account

This code snippet will help you to get the list of all running EC2 instances across all regions in an AWS account. I have used python boto3 package for developing the code. This code will dynamically pick up all the aws ec2 regions. So the code will work perfectly without any modification even if a new region gets added to the AWS.

Note: Only the basic api calls just to list the instance details are mentioned in this program . Proper coding convention is not followed . 🙂

Recovering a corrupted EC2 instance

Amazon Web Services is one of the most popular cloud service providers. I am a customer of Amazon. I like the services provided by Amazon very much. Compared to other cloud service providers AWS is simple, secure and advanced. I use EC2 machines for my project related activities as well as my personal experiments. Since I mostly work on open source software, 99.99 % of my EC2 instances are Linux instances. The only way to access these instances is through ssh. I use putty as the ssh client. If something happens to the ssh server, we will not be able to access the server. Sometimes the ssh server crashes due to overload. This can be resolved by rebooting the instance.

Sometimes because of wrong configs in the sshd config file, the ssh server may stop. The ssh server will not start until we make that file proper and restart the service. But for making these changes we have to access the machine.

By default we don’t have direct root login into the machine. We usually login to one user which is a sudo user and using sudo privileges, we access the root. If something happens to the sudoers file or if some wrong entry made in sudoers file, the root access will be revoked.

These are some of the commonly occurred situations where users loose access or super user privilege in the ec2 machine. Most of the users terminate and leave the instance in this situation.

If the instance is an EBS backed instance, we don’t have to terminate and leave the machines in this kind of situations. We can recover these instances. It is simple and can be done in few steps. If the instance is with ephemeral storage, we cannot do anything, because shutting down will clear all the data in the instance.

  1. Start a new instance in the same availability zone as that of the EBS of the broken machine. Micro or nano instance type is fine. If you already have an instance, no need of this instance.
  2. Stop the broken machine. Note down the mount locations
  3. Detach the EBS from the instance.
  4. Attach the EBS to the second EC2 instance (The newly launched one)
  5. Mount the EBS to some directory in the second EC2 instance.
  6. Navigate through the files and directories and make the required changes.
  7. Unmount the EBS
  8. Detach the EBS from the second instance
  9. Attach the EBS to the first instance
  10. Use the same mount location as that of the orginal
  11. Start the instance.

This should fix the problem.

Increasing the inodes in the disk

I faced an issue while storing large number of small files in the disk. In my linux machine, I was unable to store data because the inodes were getting filled before the storage reaches its maximum limit. This issue was annoying me and wasted a lot of storage. This happened to me several times. Initially I just did a temporary workaround for this issue by clearing old files. But since it became a frequent problem for me, I searched for the solution for this and finally I figured out a work around. The workaround is by choosing a different type of file system while formatting the disk. I found an option to specify the number of inodes while formatting the disk, but I am not sure about the optimal number of inodes that I can specify. I saw some threads in some forums regarding the issues related to improper number of inodes.

/etc/mke2fs.conf file contains various file types with the various inode ratio. The lower the inode ratio, the more you can create files in your file system.

The syntax is given below

mkfs.ext4 -T usage-type /dev/something

The usage type which gives more number of inodes in news. I used news usage-type for my requirement. This gives more inodes as compared to ext4. After doing this, mount the drive and type df -i. This will give the inodes in the new disk.

Hue Error – DatabaseError: database is locked

You may face this error in Hue while using Impala or Hive. This is because of the lock happening in the backend database used in Hue. Hue uses a backend database to store all the metadata and history. By default it uses sqlite, which is not suitable for multiuser environments. The usage of the sqlite causes this issue.

We can resolve this by using mysql, postgresql or oracle database as the metastore for hue.

The detailed steps are explained in this document

Changing the python version in pyspark

pyspark will pick one version of python from the multiple versions of python installed in the machine. In my case, I have python 3, 2.7 and 2.6 installed in my machine and pyspark was picking python 3 by default. If we have to change the python version used by pyspark, set the following environment variable and run pyspark.

export PYSPARK_PYTHON=python2.6

similarly we can configure any version of python with pyspark. Ensure that python2.6 or whatever you are specifying is available

Programmatic way to reboot EC2 instances

Sometimes we might have to reboot EC2 instances. If the requirement is to restart EC2 instances regularly, we can achieve it by writing a small piece of code. I also came across a similar requirement and a portion of the code I used is given below.

 

bashrc file not loading automatically

Recently I faced an issue in my CentOS linux machine. When I login to the machine, the bashrc file was not getting loaded and because of this, the environment variables present in the bashrc file was also not getting loaded.

The solution for this issue is given below.

Create a file with the name .profile in the user’s home directory and add the following content to the file.

if [ -f ~/.bashrc ]; then
    source ~/.bashrc
fi

How to hide or obfuscate python source code ?

Sometimes we may have the requirement to provide applications without source code. In Java it is very easy and people are widely using also. If we want to hide our source code in python what we will do ??

I checked for several solutions for obfuscating the source code . One is using pyminifier. This is  a good tool. This will rename the methods and variables. So that the obfuscated code will look more complicated. But still if you spend some time, we can read it.

Another best way to hide the source code completely is by using the built-in compiler in the python itself. This will generate a byte code and we can use that for execution.

python -OO -m py_compile  <your code.py>

This will generate a .pyo file. Rename the .pyo file to .py extension. You can use this for execution. This will work just like the actual code.

NB : If your program imports modules obfuscated like this, then you have to rename them with a .pyc suffix instead

Enabling LDAP authentication in Apache HTTPD server

Apache httpd server is a widely used open source webserver. By default, the applications deployed in this server will be open to the network and can be accessible without any authentication.

If we want to secure it using some credentials, what we will do .?

Apache httpd server provides several ways to add authentication.

Here I am explaining a basic configuration that enables ldap authentication with apache httpd server. Through this, we can integrate the applications deployed in the apache server with enterprise ldap. We can integrate apache server with LDAP in two steps. These steps are tested with apache httpd version 2.2

Step 1:

Open httpd.conf file and check for the below lines. If it is already present, we are good to go, else add these lines.

LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so

Step 2:

Add the following configuration at the end of the httpd.conf file

<Directory /var/www/html>
AuthType Basic
AuthName "Web Site: Login with user id"
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPURL ldap://ldap.myserver.com:389/ou=people,dc=unix,dc=myserver,dc=com
Require valid-user
</Directory>

 

Now this will allow all the users present in the LDAP directory to access your application.
Now restart the httpd server and try using it. You will be asked for credentials while accessing the webpages 🙂

Utility to get the complete details of a Linux system

This is a small shell script that captures almost all the necessary details of a linux system. I tested this script in CentOS and Redhat operating systems. You can access this script directly from github.