Azure Data Lake Storage is a scalable file system from Microsoft for storing large data. This is suitable for Enterprise Data Lakes. This file system is very popular now a days because of the huge Azure adoption happening across enterprises.
The ABFS connector and Hadoop Azure Data Lake connector modules provides support for integration with the Azure Data Lake Storages.
These connectors are already present in the hadoop distribution provided by Azure – HDInsights. So Azure HDInsights users does not have to make any changes in their system to interact with Azure Data Lake Store (ADLS Gen2).
For more details. Refer to the Apache Hadoop Website
A sample pyspark program that interacts with the Azure Data Lake Storage is given below. Here I am demonstrating delete and check operations.
This works very well, thank you for posting.