Victory goes to the player who makes the next-to-last mistake
December 17, 2019 Leave a comment
The error “dependency xml is not available” can be resolved by installing the following packages.
For CentOS/RHEL
yum install libxml2 libxml2-devel
For Ubuntu
apt-get install libxml2-devDecember 6, 2019 Leave a comment
Banners can be placed in two places, before the login and after successful login. Usually people keep warning messages or banners before login and welcome messages or banners after the login.
Messages before login are configured in the sshd configuration and the welcome messages/banners configured after login are configured in the motd configuration.
These messages are configured using the sshd configuration. Usually system administrators keep these messages in a file called issue.net and configure the file location in the sshd configuration. In this example, I am going to keep an issue.net file in /etc/issue.net location (By default this file exists in some of the operating systems)
nano /etc/issue.net
WELCOME TO AMALGJOSE.COM. ALL THE ACCESS ARE TRACKED AND RECORDED IF YOU ARE NOT THE AUTHORIZED USER, PLEASE DISCONNECT IMMEDIATELY
Now save the issue.net file and open SSHD configuration file.
nano /etc/ssh/sshd_config
Search for the keyword Banner in the sshd_config file. You may see a line something like the one below.
#Banner none
Now uncomment the line and instead of none, specify the banner file name with the path. In our case, it will be like the line below.
Banner /etc/issue.net
Now save and close the sshd_config file and restart the sshd service
service sshd restart
(Note: Instead of restart, you can also do a reload to avoid the disconnection of existing connections)
Now try to login to the linux machine using putty. After entering the username, you will see the configured message on the screen.
You can format this message based on your wish. You can also add ascii-arts to the issue.net file and create fancy messages. The details about generating ascii art messages are explained in my previous post.
Now we will discuss about the configuration of messages that shows up after the login. Usually this will be a welcome message/banner.
The messages that shows up after successful login are usually welcome messages. This is configured in a different file. The file name is motd. This file is located in the location /etc/motd
Edit the file and add the required contents to the file. After that save and close the file. Sample contents are given below.
nano /etc/motd
#####################WELCOME TO AMALGJOSE.COM############################# DO NOT STORE ANY CRITICAL FILES IN THIS SERVER AS THE DISKS WILL BE CLEARED FREQUENTLY
You can also add ascii arts into this file. In this example I have used a simple text content.
Now after saving the file, logout and login again. You will be able to see the message.
All set…!!!. Hope this information helps you.
December 6, 2019 Leave a comment
Do you know what is ascii arts ? Have you ever seen it ?
ASCII arts are the arts that show up on the commandline while we login to linux server using ssh clients such as putty. A Sample ASCII art is given below.
It is very easy to create an art like this. There are a lot of online ASCII art generators. We just need to enter the text and choose the required styling. The tool will automatically create it for us. Some of the links are given below.
http://www.network-science.de/ascii/
http://patorjk.com/software/taag/#p=display&f=Graffiti&t=Type%20Something%20
Copy the ASCII art from this tool and paste it in the following file of your CentOS/RHEL machines.
/etc/motd
Save the file. Logout from the system and login again. You will be able to see the ASCII art on login. You can even use this ascii art in your shell scripts, programs etc.
December 6, 2019 Leave a comment
I was checking for tools to benchmark the performance of DNS servers in my network. The reason behind this performance test was to identify the root cause of the internet slowness within my network. One of the good free tool that I found online is DNS Benchmark Tool.
This is a very light weight and portable tool. This is just 180KB and helps us to perform the DNS speeed test. With this tool I figured out one anonymous DNS server running in an individuals laptop also.
I found this tool as a useful utility.
November 21, 2019 Leave a comment
To check the IP address of a Linux server, type the following command in the terminal/commandline.
ifconfig
The below command also will help in finding the ip address.
ip addr
November 20, 2019 Leave a comment
I faced this issue while distributing the downloaded packages in cloudera manager.
The solution that worked for me is to add the IP Address – Hostname mapping in all the /etc/hosts files of all the cloudera manager server and agents
/etc/hosts 192.168.0.101 cdhdatanode1
November 20, 2019 Leave a comment
If you are facing an error like “Failed to collect NTP metrics”. The following solution might help you. This is because of the lack of ntp server in the server. The below solution will work for CentOS/RHEL systems. NTP will sync the system time with the network time.
yum install ntp systemctl enable ntpd systemctl restart ntpd
October 21, 2019 Leave a comment
When we hear about Delta Lake, the first question that comes to our mind is
“What is Delta Lake and How it works ?”.
“Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark and big data workloads”
But the question is how it is possible to maintain transactions in the Big Data world ?. The answer is very simple. It is using Delta Format.
Delta Lake stores data in Delta Format. Delta format is a versioned parquet format along with a scalable metadata. It stores the data as parquet internally and it tracks the changes happening to the data in the metadata file. So the metadata will also grow along with the data.
Delta format solved several major challenges in the Big Data Lake world. Some of them are listed below
I will elaborate this post by explaining each of the above features and explain more about the internals of Delta Lake.
October 21, 2019 Leave a comment
EMR versions 5.24.x and higher versions has Apache Spark version 2.4.2 and higher. So Delta Lake can be enabled in EMR versions 5.24.x and above. By default Delta Lake is not enabled in EMR. It is easy to enable Delta Lake in EMR.
We just need to add the delta jar to the spark jars. We can either add it manually or can be performed easily by using a custom bootstrap script. A Sample script is given below. Upload the delta-core jar to an S3 bucket and download it to the spark jars folder using the below shell script. The delta core jar can be downloaded from maven repository. You can even build it yourselves also. The source code is available in github.
Adding this as a bootstrap action will automatically perform this activity while provisioning the cluster. Keep the below script in an S3 location and pass it as bootstrap script.
copydeltajar.sh
#!/bin/bash aws s3 cp s3://mybucket/delta/delta-core_2.11.0.4.0.jar /usr/lib/spark/jars/
You can launch the cluster either by using the aws web console or by using the aws cli.
aws emr create-cluster --name "Test cluster" --release-label emr-5.25.0 \ --use-default-roles --ec2-attributes KeyName=myDeltaKey \ --applications Name=Hive Name=Spark \ --instance-count 3 --instance-type m5.xlarge \ --bootstrap-actions Path="s3://mybucket/bootstrap/copydeltajar.sh"
