How to attach a new EBS to an EC2 instance

May 24, 2016 1 Comment

Nowadays majority of us are using some cloud services. Amazon Web Services is one of the popular provider among all the other cloud service providers. Just like we upgrade our harddisk or mounting new drives to physical machines, we can attach new block storages to Amazon EC2 also. Amazon provides a service called EBS (Elastic Block Storage). There are various types of EBS with various speed and cost. Example are magnetic, SSD etc.

Attaching a new EBS to a running EC2 instance is very simple. We can do this programatically as well as using the console. Here I am explaining the basic steps to perform this operation using the console.

  1. Launch an EBS in the same region and same availability zone as that of the EC2 instance
  2. Note down the instance id of the EC2 instance
  3. Attach the EBS to the EC2. This can be done by using the attach option available in the EBS. The EBS will be listed under the Volumes section in EC2 service page of AWS console.
  4. Login to the EC2 instance and switch to the root user
  5. Type lsblk to list all the block devices
  6. Identify the new block device.
  7. Create a new directory to mount the EBS.
  8. Format the newly mounted storage. The command is mkfs -t ext4 /dev/<device-name>
  9. Mount the EBS on the directory. The command is mount /dev/<device-name>  <mount-dir>
  10. Check for the new storage. The command is df -h

 

 

Filed under Linux World Tagged with , , , , , , , ,

Disable SELinux without reboot

May 23, 2016 1 Comment

To disable the SELinux by modifying /etc/sysconfig/selinux file, we have to perform a reboot. In some cases, we may not be able to perform a reboot because this involves a downtime of the system. In this situations we can disable SELinux by using a simple command. This will not disable SELinux permanently. The effect will last until the next reboot, but you have the option to edit the selinux file so that it will be in the disabled state even after  the reboot also. The steps for disabling selinux permanently are explained in my previous post.

The command the check the status of SELinux is given below.

sestatus

This may show enforcing or permissive or disabled. In permissive mode, SELinux will not block anything, but merely warns you. The line will show enforcing when it’s actually blocking.

To disable the SELinux temporarily we can use the following command. This has to be executed as root or using sudo.

setenforce 0

After this command execution we can check the status of selinux using sestatus command. If it is permissive, we are good to go. 🙂

Filed under Linux World Tagged with , , , , , , , , , , , , , , ,

Disable SELinux in CentOS and RHEL

May 23, 2016 Leave a comment

Security-Enhanced Linux (SELinux) is a security architecture integrated into the 2.6.x kernel using the Linux Security Modules. It is a project of the United States National Security Agency (NSA) and the SELinux community. SELinux integration into Red Hat Enterprise Linux was a joint effort between the NSA and Red Hat.

Most of the application needs SELinux to be turned off. Turning off selinux is simple. You can use the following steps to turn off selinux in RHEL or CentOS 6 and 7 operating systems.

Open the file /etc/sysconfig/selinux . The contents will be similar as below.

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted

 

The contents are self explanatory. Change the value of SELINUX as disabled and save the file. Then reboot the system.

Filed under Linux World Tagged with , , , , , , , , ,

Good Quote.!!

May 14, 2016 Leave a comment

“A beginning programmer writes her programs like an ant builds her hill, one piece at a time, without thought for the bigger structure. Her programs will be like loose sand. They may stand for a while, but growing too big they fall apart.

Realizing this problem, the programmer will start to spend a lot of time thinking about structure. Her programs will be rigidly structured, like rock sculptures. They are solid, but when they must change, violence must be done to them.

The master programmer knows when to apply structure and when to leave things in their simple form. Her programs are like clay, solid yet malleable.”

— Master Yuan-Ma, The Book of Programming

Filed under General

Heterogeneous storages in HDFS

May 14, 2016 Leave a comment

From hadoop 2.3.0 onwards, hdfs supports heterogeneous storage. What is this heterogeneous storage? What are the advantages of using this?.

Hadoop came as a processing system for processing and storing huge data, a scalable batch processing system. But now it became the platform for DataLake for Enterprises. In large enterprises, various types of data needs to be stored and processed for advanced analytics. Some of these data are required frequently, some are not required frequently, some are required very rarely. If we store all these in the same platform or hardware, the cost will be more. For example, if we are using a cluster in AWS. We have EC2 nodes for our cluster nodes. EC2 uses EBS and ephemeral storage. Depending upon the type of storage, the cost varies. S3 storage is cheaper than EBS storage, but access speed will be less. Similarly glacier will be cheaper compared to S3, but again the data retrieval will take time. Similarly, if we want to keep data in different storage types depending upon the priority and requirement, we can use this feature in hadoop. This feature was not available in earlier versions of hadoop. This is available in hadoop version 2.3.0 onwards. Now datanode can be defined as a collection of storages. Various storage policies available in hadoop are Hot, Warm, Cold, All_SSD, One_SSD and Lazy_Persist.

  • Hot – for both storage and compute. The data that is popular and still being used for processing will stay in this policy. When a block is hot, all replicas are stored in DISK.
  • Cold – only for storage with limited compute. The data that is no longer being used, or data that needs to be archived is moved from hot storage to cold storage. When a block is cold, all replicas are stored in ARCHIVE.
  • Warm – partially hot and partially cold. When a block is warm, some of its replicas are stored in DISK and the remaining replicas are stored in ARCHIVE.
  • All_SSD – for storing all replicas in SSD.
  • One_SSD – for storing one of the replicas in SSD. The remaining replicas are stored in DISK.
  • Lazy_Persist – for writing blocks with single replica in memory. The replica is first written in RAM_DISK and then it is lazily persisted in DISK.

Filed under BigData World Tagged with , , , , , , , , , , , , , , , , ,