Programmatic Data Upload to Amazon S3

S3 is a service provided by Amazon for storing data. The full form is Simple Storage Service. S3 is a very useful service for less price. Data can be uploaded to and downloaded from S3 very easily using some tools as well as program. Here I am explaining  a sample program for uploading file to S3 using a python program.

Files can be uploaded to S3 in two approaches. One is the normal upload and another is the multipart upload. Normal upload sends the file serially and is not suitable for large files. It will take more time. For large files, multipart upload is the best option. It will upload the file by dividing it into chunks and sends it in parallel and collects it in S3.

This program is using the normal approach for sending the files to S3. Here I used the boto library for uploading the files.

Advertisements

About amalgjose
I am an Electrical Engineer by qualification, now I am working as a Software Engineer. I am very much interested in Electrical, Electronics, Mechanical and now in Software fields. I like exploring things in these fields. I like travelling, long drives and very much addicted to music.

2 Responses to Programmatic Data Upload to Amazon S3

  1. naveen says:

    hi
    i am looking oozie installation in emr cluster (oozie 4.0.0 or 4.0.1 with apache hadoop 2.4) ,can u pls help me in this(i found two scripts in github but not solving )…

    • amalgjose says:

      Hi,
      oozie installation is pretty simple. You have to install oozie on the master node. For that you can write a custom bootstrap script or you can do a manual installation. A sample custom script is present in this url
      https://github.com/lila/emr-oozie-sample/blob/master/config/config-oozie.sh

      This will not work directly, because all the urls mentioned in this script is not available now. So you can take this and make the changes as per the new oozie present in apache repository. Change the folder names accordingly.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: