Advertisements

How to extract a tar.gz file quickly in Linux

Recently I got a tar.gz file or around 30 GB and on extraction it will become approximately 4TB. I want to speed up the extraction as the normal extraction was taking approximately a day. I searched a lot and finally figured out a solution.

The solution was pigz. This is an advanced version of gzip. It uses multiple threads for reading, writing and checksum calculations. The extraction happens in a single thread. But overall performance is far better than the normal gzip.

The command to install pigz in CentOS or RHEL is given below. Ensure epel repository is enabled in your system

yum install pigz

The command to extract a tar.gz file using pigz is given below.

pigz -dc compressed.tar.gz | tar xf -

If you want to see the progress of the extraction process, you need to use Pipe Viewer (pv). PV (“Pipe Viewer”) is a tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.

Pipe viewer can be installed in CentOS or RHEL using the following command

yum install pv

Using pv, we can monitor the progress of the decompression process

pigz -dc compressed.tar.gz | pv | tar xf -

 

Advertisements

How to migrate docker images from one server to another without using a docker registry/repository ?

Copying docker image from one server to another server is an easy task. The following steps will explain you about this. Before getting into the actual steps, lets get the understanding of few terminologies.

What is a docker image ?

An image is an immutable master copy. We can correlate docker image with an ISO image of an operating system. Once we run this image, it will create a container. We can run any number of containers from the same image.

What is a docker container ?

Container is basically a running copy of the image with life. Alterations can be made on the container. Basically changes can be applied on top of the base image while running it as a container. A container can be called as a booted image.

Docker save, export and load commands

docker save will save a docker image to the disk. This saved file includes all the layers of images and the metadata required to chain these layers to rebuild the current image. So the docker save command will preserve the history of all the layers present in the current image. We can copy this saved file to another server to load the image and run containers.

The syntax is

docker save -o [filename] [imagename]:[version]

The above command will save the image into the given file name. You can also provide the complete path along with the file name.

The docker load command will load the image back from file into the system. To load this image from the file, use the following command.

docker load -i [saved image file name]

Docker export will create a snapshot of the container. Basically it will save the current state of the container as an image. It will not preserve the details of the layers present in the parent image of the container. This will save the container’s file system as a tar file. This command does not export the contents of volumes associated with the container.

Docker save needs to be performed on a docker image and docker export is performed on a docker container.

To copy a docker image from one host to another host in a single shot, the following command will help. For executing this command, the bzip2 package needs to be installed in your unix operating system

docker save [image]:[version] | bzip2 | ssh username@hostname 'bunzip2 | docker load'

Note: For installing bzip2 in centos/rhel, use the following command

yum install bzip2

For ubuntu

apt-get install bzip2

I hope this article helped you. 🙂

How to find and kill a process locking a particular port in Linux?

Sometimes because of some issue or bug, our application may stop working, but the port will be locked. This kind of issue is very common with MySQL server, Elasticsearch, WebServices, Tomcat etc. In such scenarios, we have to find the zombie process and kill it to free up the locked port.

How to find the process that locks the port?

Use the following command

netstat -tulpn | grep <port>

This output of this command will contain the process id. Now we just need to kill the process.

Verify the process

Before killing the process, figure out what process it is and ensure we are not killing any required processes.

ps -aux | grep <process id>

The output of the above command will give the details of the process.

How to Kill a process ?

After confirming the details, you can kill the process

kill -9 <process id>

Now verify whether the port is still locked or not by executing the netstat command again

What is Swap memory and How to clear Swap usage in Linux ?

What is Swap Space ?

Swap is a space on disk that is used by the system when the available memory in the RAM (physical memory) is completely utilized. This is basically to increase the available virtual memory in the system. The swap memory will be used once the physical memory is full. Since this is residing in the disk, the processing speed using this memory will be slow compared to the processing in physical memory (RAM).

Why we need swap space ?

Suppose we have a system with 4GB RAM. When we start the system the memory usage will be less. But as we open applications or start running processes, the memory utilization will increase. If it reaches the 4GB utilization, we will not be able to use any additional applications and we will have to wait to get some free space in the RAM. With swap memory, the allocated space in the disk will be used in case of any additional requirement and the applications will still run even after crossing the max limit of system RAM. As already explained, the performance of swap will be very slow as compared to RAM.

How the memory management works internally ?

The Linux kernel has a memory management process. This process monitors all the processes and identifies the less frequently used memory pages (or blocks). In case of additional memory requirement exceeding the RAM limit comes, this memory management program will utilize the space in system hard disk allocated for “swapping” or paging these less frequently used memory blocks. In this way the RAM will be freed up and the active memory for running live application will become available in the system.

How to clear the swap memory usage?

If you want to clear the swap memory, you can execute the following command in the terminal as root user.

swapoff -a && swapon -a

WARNING.!!!: Be careful doing this, as this may affect your system’s stability, especially if its already low on RAM. Better not to set these swap clearing scripts as cronjob.

Linux commands to check the disk utilization, size of directory or file

  • Command to check the disk utilization
df -h

The ‘-h’ option will provide the utilization in human readable format.

  • Command to check the size of a directory
du -sh <directory name>
  • Command to check the size of a file
du -sh <file name>
  • Command to check the size of files in a directory

Go inside the directory and execute the following command

du -sh *

 

bashrc file not loading automatically

Recently I faced an issue in my CentOS linux machine. When I login to the machine, the bashrc file was not getting loaded and because of this, the environment variables present in the bashrc file was also not getting loaded.

The solution for this issue is given below.

Create a file with the name .profile in the user’s home directory and add the following content to the file.

if [ -f ~/.bashrc ]; then
    source ~/.bashrc
fi

Linux Filesystem colour codes

When we fire ls –all in linux cli, files may be listed in different colours  

The color code of the files is as follows:

Blue: Directory file

White: Normal file

Green: Executable file

Yellow: Device file

Magenta: Picture file

Cyan: link file

Red: Compressed file

File Symbol

-(Hyphen) = Normal file

d=directory

l=link file

b=Block device file

c=character device file

Changing the Default GUI boot mode of Ubuntu

If you want to change the default GUI mode  boot of ubuntu machine, just do the following changes. Here actually we are not removing the GUI, just changing the default booting as text mode. By doing this you can reduce the memory consumption.

Open the terminal as root user or sudo user.

open /etc/default/grub as and add text to the

sudo nano /etc/default/grub

And in the file, change this line:

GRUB_CMDLINE_LINUX_DEFAULT="text"

Then update your Grub

sudo update-grub

To view the changes, reboot the system. Then the system will always boot in text mode. This is applicable to Ubuntu OS only.

If you want to get the UI without changing the configuration from text mode, execute startx command in the terminal. Then the UI will be loaded.

SSH Key based access to Unix Servers

Access to Linux and Unix systems via Secure Shell (SSH) is standard practice.  It offers encrypted access to enable you to administer your server which is vital over the big bad internet.

There are different ways to access SSH: password, user keys and host-based keys.  Passwords are the most common but are less secure than key-based access.  Passwords are susceptible to keylogger attacks, as well as more likely to fool users into a “man-in-the-middle” attack (one where you think you’re logging onto your server, but you are actually proxying your connection through another server which has been compromised and is recording every keystroke and data transfer.)

Key based access is more secure as it requires two parts of a key to be present before access is granted.  When dealing with cloud based services such as Rackspace and Amazon Web Services, key based access is enabled by default.  Key based access is also known as “passwordless access” as access is granted by your key, not by asking for any passwords.  The exception to this is if you put a password on your key (but you can enable services that ask for this password once and it is cached for the rest of your session).

Setting this up on your Linux server is very simple, and most installations of SSH (OpenSSH) enable both password and key-based access by default.  Let’s assume user@client needs to access user@server

Ensure OpenSSH is installed on your Linux server (server)

Debian/Ubuntu

sudo apt-get install openssh-server

CentOS/Fedora/RedHat/Oracle Enterprise Linux

sudo yum install openssh-server

Ensure the following lines has been uncommented from /etc/ssh/sshd_config

RSAAuthentication yes
PubkeyAuthentication yes

Restart OpenSSH

Debian/Ubuntu

sudo /etc/init.d/ssh restart

CentOS/Fedora/RedHat/Oracle Enterprise Linux

sudo /etc/init.d/sshd restart

On your Linux client (desktop or other server you’ll be using to connect to the server configured in steps 1-3)

Generate your public and private keys

ssh-keygen -t rsa

You will see output like the following:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa):
Created directory ‘/home/user/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
79:e1:08:77:c2:0d:c4:ff:35:22:64:9a:4d:03:b8:67 user@client
The key’s randomart image is:
+–[ RSA 2048]—-+
|       ++.                      |
|      …o=                    |
|      ..+O+.                 |
|      .oE*+.. o             |
|       oS oo o .            |
|         .  .                     |
|                                  |
|                                  |
|                                  |
+—————–+

This produces two important pieces of data.  Your PRIVATE KEY (~/.ssh/id_rsa) and your PUBLIC KEY (~/.ssh/id_rsa.pub).  You must keep your PRIVATE KEY safe.  Your public key can be given to anyone.  Without your private key your public key is just a string of characters and you can’t generate a private key from a public key.  Equally, you can’t generate a public key from a private key.  Together they make your key-pair.
To enable your private key to access the server running SSH configured in steps 1-3 (server) you simply copy the contents of your public key onto the server.
Copy the public key from your client machine to server

scp .ssh/id_rsa.pub user@server:
(enter your password)

Login to server

ssh user@server
(enter your password)

Copy the public key to authorized_keys

cat .ssh/id_rsa.pub >> .ssh/authorized_keys

Change the permission of authorized_keys file to 600 (rw——-)

chmod 0600 .ssh/authorized_keys

This creates the directory .ssh/ and relevant authorized_keys file with the correct permissions (anything less strict will not work).  You can put in a number of public keys in here, line-by-line.  When there are multiple entries it allows multiple people to connect to that account using their keys.  This becomes useful when a team of system administrators require access to systems with minimal accounts installed, but each are accountable for audit purposes as to who logged onto the system.

Log out of that session and log back in again and you shouldn’t be asked for a password.

How to Change the Hostname of Ubuntu Server ?

We can change the hostname of a machine by several ways. I am mentioning one way to change the hostname of Ubuntu server.

  1. Open the terminal or Login with putty.exe as root user (if you are working remotely)
  2. Goto /etc/
  3. Type nano hostname
  4. Change the HOSTNAME to your preferred machine name
  5. Press Cntrl+X
  6. Save the configuration by pressing Y
  7. Log off or reboot

Note: Add the new hostname and ipaddress in the /etc/hosts file also