Victory goes to the player who makes the next-to-last mistake
November 20, 2019 Leave a comment
I faced this issue while distributing the downloaded packages in cloudera manager.
The solution that worked for me is to add the IP Address – Hostname mapping in all the /etc/hosts files of all the cloudera manager server and agents
/etc/hosts 192.168.0.101 cdhdatanode1
August 7, 2017 Leave a comment
In an namenode HA enabled hadoop cluster, one of the namenodes will be active and the other will be standby. If you want to perform some operations on HDFS programmatically, some of the the libraries or packages need the details of active namenode (some of the packages in python need the details of active namenode, they will not support the nameservice). In this case, the easiest way to get the status is to issue a GET request similar to the one given below on each of the namenodes. This will help us to identify the status of each namenode.
GET REQUEST
curl 'http://namenode.1.host:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus'
SAMPLE OUTPUT
{
"beans" : [ {
"name" : "Hadoop:service=NameNode,name=NameNodeStatus",
"modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
"State" : "active",
"SecurityEnabled" : false,
"NNRole" : "NameNode",
"HostAndPort" : "namenode.1.host:8020",
"LastHATransitionTime" : 0
} ]
}
May 14, 2016 Leave a comment
From hadoop 2.3.0 onwards, hdfs supports heterogeneous storage. What is this heterogeneous storage? What are the advantages of using this?.
Hadoop came as a processing system for processing and storing huge data, a scalable batch processing system. But now it became the platform for DataLake for Enterprises. In large enterprises, various types of data needs to be stored and processed for advanced analytics. Some of these data are required frequently, some are not required frequently, some are required very rarely. If we store all these in the same platform or hardware, the cost will be more. For example, if we are using a cluster in AWS. We have EC2 nodes for our cluster nodes. EC2 uses EBS and ephemeral storage. Depending upon the type of storage, the cost varies. S3 storage is cheaper than EBS storage, but access speed will be less. Similarly glacier will be cheaper compared to S3, but again the data retrieval will take time. Similarly, if we want to keep data in different storage types depending upon the priority and requirement, we can use this feature in hadoop. This feature was not available in earlier versions of hadoop. This is available in hadoop version 2.3.0 onwards. Now datanode can be defined as a collection of storages. Various storage policies available in hadoop are Hot, Warm, Cold, All_SSD, One_SSD and Lazy_Persist.
August 1, 2015 1 Comment
While checking the details of a YARN applications, if you are getting a message similar to “Log Aggregation not enabled”. You can follow the below steps to enable it. This issue occurs in EMR, because in most of the AMI’s the log aggregation is not enabled by default. It is very simple to enable it. Add the following configuration to the yarn-site.xml of all the yarn hosts and restart the cluster. (full cluster restart is not required. Restarting all the nodemanagers will be fine)
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/tmp/logs</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>259200</value>
</property>
<property>
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>3600</value>
</property>
July 29, 2015 1 Comment
HDFS Data encryption is an excellent feature that came recently. With this we can encrypt the data in hdfs. We can create multiple encryption zones with different encryption keys. In this way, we can secure the data in hdfs properly. For more details, you can visit these websites. Reference1, Reference2
I am using a cluster installed with CDH. I created some encryption keys and zones.
The command I used for creating a key is given below.
# As the normal user, create a new encryption key hadoop key create amalKey # As the super user, create a new empty directory and make it an encryption zone hadoop fs -mkdir /user/amal hdfs crypto -createZone -keyName amalKey -path /user/amal # chown it to the normal user hadoop fs -chown amal:hadoop /user/amal # As the normal user, put a file in, read it out hadoop fs -put test.txt /user/amal/ hadoop fs -cat /user/amal/test.tx
After some days, I deleted the encryption zone and I deleted the encryption key also.
The command I used for deleting the encryption key is given below
hadoop key delete <key-name>
After the deletion, I tried creating the key with the same name. But I got an exception that the key is still present in the disabled state. When I list the keys, I am not able to see the key. The exception that I got was given below.
amalKey has not been created. java.io.IOException: HTTP status [500], exception [com.cloudera.keytrustee.TrusteeKeyProvider$DuplicateKeyException], message [Key with name "amalKey" already exists in "com.cloudera.keytrustee.TrusteeKeyProvider@6d88562. Key exists but has been disabled. Use undelete to enable.] java.io.IOException: HTTP status [500], exception [com.cloudera.keytrustee.TrusteeKeyProvider$DuplicateKeyException], message [Key with name "amalKey" already exists in "com.cloudera.keytrustee.TrusteeKeyProvider@6d88562. Key exists but has been disabled. Use undelete to enable.] at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:159) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:545) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:503) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:676) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:684) at org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483) at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
In the error logs, it says to use purge option to permanently delete the key and undelete to recover the deleted key. But I was not able to find these options with hadoop key command. I googled it and I couldn’t figure out this issue. Finally I got the guidance from one guy from cloudera to execute the purge & undelete commands through rest api of keytrustee and he gave a nice explanation for my issue. I am briefly putting the solution for this exception below.
The delete operation on the Trustee key provider is a “soft delete”, meaning that is possible to “undelete” the key. It is also possible to “purge” the key to delete it permanently. Because these operations are not part of the standard Hadoop key provider API, they are not currently exposed through Hadoop KeyShell (hadoop key). However, you can call these operations directly via the Trustee key provider REST API.
See the examples below.
Use KeyShell to list existing keys
$ ./bin/hadoop key list -provider kms://http@localhost:16000/kms Listing keys for KeyProvider: KMSClientProvider[http://localhost:16000/kms/v1/] amal-testkey-1
Use KeyShell to delete an existing key
$ ./bin/hadoop key delete amal-testkey-1 -provider kms://http@localhost:16000/kms Deleting key: ajy-testkey-1 from KeyProvider: KMSClientProvider[http://localhost:16000/kms/v1/] amal-testkey-1 has been successfully deleted. KMSClientProvider[http://localhost:16000/kms/v1/] has been updated.
Use KeyShell to verify the key was deleted
$ ./bin/hadoop key list -provider kms://http@localhost:16000/kms Listing keys for KeyProvider: KMSClientProvider[http://localhost:16000/kms/v1/]
Use the KeyTrustee key provider REST API to undelete the deleted key
$ curl -L -d "trusteeOp=undelete" "http://localhost:16000/kms/v1/trustee/key/amal-testkey-1?user.name=amal&trusteeOp=undelete"
Use KeyShell to verify the key was restored
$ ./bin/hadoop key list -provider kms://http@localhost:16000/kms Listing keys for KeyProvider: KMSClientProvider[http://localhost:16000/kms/v1/] amal-testkey-1
Use the KeyTrustee key provider REST API to purge the restored key
$ curl L -d "trusteeOp=purge" "http://localhost:16000/kms/v1/trustee/key/amal-testkey-1?user.name=amal&trusteeOp=purge"
Use KeyShell to verify the key was deleted
$ ./bin/hadoop key list -provider kms://http@localhost:16000/kms Listing keys for KeyProvider: KMSClientProvider[http://localhost:16000/kms/v1/]
Use the KeyTrustee key provider REST API to attempt to undelete the purged key
$ curl -L -d "trusteeOp=undelete" "http://localhost:16000/kms/v1/trustee/key/amal-testkey-1?user.name=amal&trusteeOp=undelete"
{
"RemoteException" : {
"message" : "Key with name amal-testkey-1 not found in com.cloudera.keytrustee.TrusteeKeyProvider@6d88562",
"exception" : "IOException",
"javaClassName" : "java.io.IOException"
}
}
Configure ACLs for KeyTrustee undelete, purge and migrate operations
ACLs for the KeyTrustee specific undelete, purge and migrate operations are configured in kts-acls.xml. Place this file in the same location as your kms-acls.xml file. See example below.
<property>
<name>keytrustee.kms.acl.UNDELETE</name>
<value>*</value>
<description>
ACL for undelete-key operations.
</description>
</property>
<property>
<name>keytrustee.kms.acl.PURGE</name>
<value>*</value>
<description>
ACL for purge-key operations.
</description>
</property>
<property>
<name>keytrustee.kms.acl.MIGRATE</name>
<value>*</value>
<description>
ACL for purge-key operations.
</description>
</property>
Note: In kerberized environments, the requests will be a little different. It will be in the following format
Eg :
curl -L --negotiate -u [username] -d "trusteeOp=undelete" "http://localhost:16000/kms/v1/trustee/key/amal-testkey-1?user.name={username}&trusteeOp=undelete"
July 24, 2015 2 Comments
Hadoop comes with various scheduling algorithms such as FIFO, Capacity, Fair, DRF etc. Here I am briefly explaining about setting up fair scheduler in hadoop. This can be performed in any distribution of hadoop. By default hadoop comes with FIFO scheduler, some distribution comes with Capacity Scheduler as the default scheduler. In multiuser environments, a scheduler other than the default FIFO is definitely required. FIFO will not help us in multiuser environments because it makes us to wait in a single queue based on the order of job submission. Creating multiple job queues and assigning a portion of the cluster capacity and adding users to these queues will help us to manage and utilize the cluster resources properly.
For setting up a fair scheduler manually, we have to make some changes in the resource manager node. One is a change in the yarn-site.xml and another is the addition of a new configuration file fair-scheduler.xml
The configurations for a basic set up are given below.
Step 1:
Specify the scheduler class in the yarn-site.xml. If this property exists, replace it with the below value else add this property to the yarn-site.xml
<property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> </property>
Step 2:
Specify the Fair Scheduler allocation file. This property has to be set in yarn-site.xml. The value should be the absolute location of fair-scheduler.xml file. This file should be present locally.
<property> <name>yarn.scheduler.fair.allocation.file</name> <value>/etc/hadoop/conf/fair-scheduler.xml</value> </property>
Step 3:
Create the allocation configuration file
A sample allocation file is given below. We can have advanced configurations in this allocation file. This is an allocation file with a basic set of configurations
There are five types of elements which can be set up in an allocation file
Queue element :– Representing queues. It has the following properties:
User elements :– Representing user behaviors. It can contain a single properties to set maximum number apps for a particular user.
userMaxAppsDefault element :– Setting the default running app limit for users if the limit is not otherwise specified.
fairSharePreemptionTimeout element :– Setting the number of seconds a queue is under its fair share before it tries to preempt containers to take resources from other queues.
defaultQueueSchedulingPolicy element :– Specifying the default scheduling policy for queues; overriden by the schedulingPolicy element in each queue if specified.
<?xml version="1.0"?> <allocations> <queue name="queueA"> <minResources>1000 mb, 1 vcores</minResources> <maxResources>5000 mb, 1 vcores</maxResources> <maxRunningApps>10</maxRunningApps> <aclSubmitApps>hdfs,amal</aclSubmitApps> <weight>2.0</weight> <schedulingPolicy>fair</schedulingPolicy> </queue> <queue name="queueB"> <minResources>1000 mb, 1 vcores</minResources> <maxResources>2500 mb, 1 vcores</maxResources> <maxRunningApps>10</maxRunningApps> <aclSubmitApps>hdfs,sahad,amal</aclSubmitApps> <weight>1.0</weight> <schedulingPolicy>fair</schedulingPolicy> </queue> <queue name="queueC"> <minResources>1000 mb, 1 vcores</minResources> <maxResources>2500 mb, 1 vcores</maxResources> <maxRunningApps>10</maxRunningApps> <aclSubmitApps>hdfs,sree</aclSubmitApps> <weight>1.0</weight> <schedulingPolicy>fair</schedulingPolicy> </queue> <user name="amal"> <maxRunningApps>10</maxRunningApps> </user> <user name="hdfs"> <maxRunningApps>5</maxRunningApps> </user> <user name="sree"> <maxRunningApps>8</maxRunningApps> </user> <user name="sahad"> <maxRunningApps>2</maxRunningApps> </user> <userMaxAppsDefault>5</userMaxAppsDefault> <fairSharePreemptionTimeout>30</fairSharePreemptionTimeout> </allocations>
Here we created three queues queueA, queueB and queueC and mapped users to these queues. While submitting the job, the user should specify the queue name. Only the user who has access to the queue can submit jobs to a particular queue. This is defined in the acls. Another thing is scheduling rules. If we specify scheduling rules, the jobs from a particular user will be directed automatically to a particular queue based on the rule. I am not mentioning the scheduling rule part here.
After making these changes, restart the resource manager.
Now go to the resource manager web ui. In the left side of the UI, you can see a section named Scheduler. Click on that section, you will be able to see the newly created queues.
Now submit a job by specifying a queue name. You can use the option as below. The below option will submit the job to queueA. All the queues that we created are the sub-pools of root queue. Because of that, we have to specify queue name in the fomat parentQueue.subQueue
-Dmapred.job.queue.name=root.queueA
Eg: hadoop jar hadoop-examples.jar wordcount -Dmapred.job.queue.name=root.queueA <input-location> <output-location>
If you are running a hive query, you can set these property in the below format. This property should be set at the top.
set mapred.job.queue.name=root.queueA
July 23, 2015 1 Comment
Cloudera Manager is an Infrastructure management and monitoring tool provided by cloudera. This has now became a very excellent tool to manage bigdata infrastructure. The pain of administrators has been reduced by 80% with this cloudera manager. Almost everything required for an administrator is integrated into this great software and is very user friendly. Cloudera Manager became this muhc powerful recently. So lot of existing clusters are still running without using cloudera manager. If you want to manage an existing cluster using cloudera manager, the following steps may help you. For this you have to completely uninstall the existing hadoop set up. No data loss will happen because we are not touching any data. The configurations also will remain the same. These are just pointers.
July 21, 2015 3 Comments
Heavy mapreduce jobs may run for several hours. There can be several jobs and checking the status of mapreduce jobs manually will be a boring task. I don’t like this J. If we try to manage java programs using a script, it will not be a clean approach. Using scripts for managing java programs is bad approach. I consider these kind of designs as worst designs.
My requirement was to get notification on completion of mapreduce jobs. These are some critical mapreduce jobs and I don’t want to frequently check the status and wait for its completion.
Hadoop is providing a useful configuration to solve my problem. It is very easy to achieve this solution. Just a few lines of code will help us. Add these three lines to the Driver class
conf.set("job.end.notification.url", "http://myserverhost:8888/notification/jobId=$jobId?status=$jobStatus");
conf.setInt("job.end.retry.attempts", 3);
conf.setInt("job.end.retry.interval", 1000);
By setting these properties, hadoop sends an http request on completion of the job. We need a small piece of code for creating a webservice that accepts this http request and send email. For creating the webservice and email utility I used python language because of simplicity. Once the mapreduce job completes, it sends an http request to the URL mentioned by the configuration job.end.notification.url. The variables jobId and jobStatus will be replaced with the actual values. Once a request comes to the webservice, it will parse the arguments and call the email sending module. This is a very simple example. Instead of email, we can make different kind of notifications such as sms, phone call or triggering some other application etc. The property job.end.notification.url is very helpful in tracking asynchronous mapreduce jobs. We can trigger another action also using this trigger. This is a clean approach because we are not running any other script to track the status of the job. The job itself is providing the status. We are using the python program for just collecting the status and making notifications using the status.
The python code for the webservice and email notification are given below.
