Enabling Log Aggregation in YARN

While checking the details of a YARN applications, if you are getting a message similar to “Log Aggregation not enabled”. You can follow the below steps to enable it. This issue occurs in EMR, because in most of the AMI’s the log aggregation is not enabled by default. It is very simple to enable it. Add the following configuration to the yarn-site.xml of all the yarn hosts and restart the cluster. (full cluster restart is not required. Restarting all the nodemanagers will be fine)

<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

<property>
    <description>Where to aggregate logs to.</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/tmp/logs</value>
</property>

<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>259200</value>
</property>

<property>
    <name>yarn.log-aggregation.retain-check-interval-seconds</name>
    <value>3600</value>
</property>

Enabling LDAP authentication in Apache HTTPD server

Apache httpd server is a widely used open source webserver. By default, the applications deployed in this server will be open to the network and can be accessible without any authentication.

If we want to secure it using some credentials, what we will do .?

Apache httpd server provides several ways to add authentication.

Here I am explaining a basic configuration that enables ldap authentication with apache httpd server. Through this, we can integrate the applications deployed in the apache server with enterprise ldap. We can integrate apache server with LDAP in two steps. These steps are tested with apache httpd version 2.2

Step 1:

Open httpd.conf file and check for the below lines. If it is already present, we are good to go, else add these lines.

LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so

Step 2:

Add the following configuration at the end of the httpd.conf file

<Directory /var/www/html>
AuthType Basic
AuthName "Web Site: Login with user id"
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPURL ldap://ldap.myserver.com:389/ou=people,dc=unix,dc=myserver,dc=com
Require valid-user
</Directory>

 

Now this will allow all the users present in the LDAP directory to access your application.
Now restart the httpd server and try using it. You will be asked for credentials while accessing the webpages 🙂