How to install s3cmd on Ubuntu 18.04 and CentOS 7.

Introduction

This tutorial is for the AWS S3cmd, How to install S3cmd in Ubuntu 18.04 and CentOS 7. S3cmd is a tool for managing objects in Amazon S3 storage. With the help of the S3cmd command, we can make the buckets, and remove buckets.

Also manages the buckets and the objects inside them(files and folders). We can keep the versions of files and set the retention policy for the files and folders.

To set up s3cmd we need Access Key and Secret key, which can create on AWS IAM roles, you can refer to the following link.

4 topics will be cover in this tutorial for AWS S3cmd.

1. How to install s3cmd on Ubuntu 18.04.
2. How to install s3cmd on CentOS 7.
3. Configure S3cmd with access and secret keys.
4. S3cmd examples via command line.

How to Install S3cmd on Ubuntu 18.04

So to install the S3cmd command line, we have to update the repository by using the apt command.

$ sudo apt update

After the package update, let’s install the s3cmd by using the apt command.

$ sudo apt install s3cmd

If you want to set up the S3cmd you can jump to that section.

How to Install S3cmd on CentOS 7

So to install the S3cmd Command line, we have to update the repository by using the yum command.

# yum update

# yum install epel-release

# yum install s3cmd

How to Setup or Configure S3cmd with access and secret keys.

Then use the command “s3cmd –configure” to access the S3 bucket, let’s configure the s3cmd tool.

$ s3cmd --configure

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.

Access Key: XXXXTastethelinux

Secret Key: XXXXTastethelinux#####$!@@!#

Default Region [US]: region_of_the_Bucket

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. 
S3 Endpoint [s3.amazonaws.com]: Press Enter

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: Press Enter

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]: Press Enter

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: Press Enter

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: Press Enter

New settings:
Access Key: XXXXTastethelinux
Secret Key: XXXXTastethelinux#####$!@@!#
Default Region: region_of_the_Bucket
S3 Endpoint: s3.amazonaws.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n

Save settings? [y/N] y

Configuration saved to '/home/tastethelinux-ashish/.s3cfg'

So we have configured the s3cmd tool with the Access Key and Secret Key.

S3cmd examples via Command line.

1. List all the buckets in AWS s3 using s3cmd.

# s3cmd ls

OUTPUT:
2019-12-30 10:14 s3://tla
2019-11-06 11:26 s3://tla-image
2020-07-01 11:07 s3://tastethelinux-backup
2019-11-07 13:59 s3://tla-logs

So we have 4 buckets(tla, tla-image, tastethelinux-backup, and tla-logs)into the AWS s3 Storage.

2. Create a New bucket into AWS S3.

So let’s create a bucket with the name “test-tla” using the command line.

# s3cmd mb s3://test-tla

OUTPUT:
Bucket 's3://test-tla/' created

3. Upload a file into the S3 bucket.

# s3cmd put tla.txt s3://tla/

OUTPUT:
tla.txt -> s3://tla/tla.txt [1 of 1]
100619 of 100619 100% in 2s 668.35 kB/s done

So, we have uploaded a file tla.txt into a tla bucket.

4. Upload a folder or a directory in the s3 bucket.

# s3cmd put script s3://tla/

OUTPUT:
ERROR: Parameter problem: Use --recursive to upload a directory: Script

So, we are getting the above error while uploading the Directory or folder into the bucket. let’s try a different command then.

# s3cmd put Script s3://tla/ -r

OUTPUT:
upload: 'Script/s3_Script.sh' -> 's3://tla/Script/s3_Script.sh' [1 of 4]
801 of 801 100% in 0s 7.27 kB/s done
upload: 'Script/nodeJs.sh' -> 's3://tla/Script/nodeJs.sh' [2 of 4]
349 of 349 100% in 0s 6.90 kB/s done
upload: 'Script/java' -> 's3://tla/Script/java' [3 of 4]
192 of 192 100% in 0s 6.47 kB/s done
upload: 'Script/tla-rotate.sh' -> 's3://tla/Script/tla-rotate.sh' [4 of 4]
153 of 153 100% in 0s 1703.20 B/s done

So we have uploaded the folder into the S3 bucket. We have 4 files “s3_Script.sh, nodeJs.sh, java, and tla-rotate.sh”.

We have used the -r option to Upload the folder also –recursive will work.

5. Delete the file from the S3 bucket.

# s3cmd del s3://tla/Script/java

OUTPUT:
delete: 's3://tla/Script/java'

6. Delete a folder from the s3 bucket.

# s3cmd del s3://tla/Script

OUTPUT:
delete: 's3://tla/Script'

7. Delete the Bucket from AWS S3.

# s3cmd rb s3://tla

OUTPUT:
Bucket 's3://tla/' removed

If you want to learn more examples for the S3cmd command then follow the link. We will see “How to Download the larger file from AWS S3 bucket.

s3cmd download failed for the larger file

So, While Downloading from s3 Bucket I got the Error Download failed after the timeout error.

In a screenshot, while downloading a file from the S3 bucket it gets failed. With an error, “Download failed for /test/test1 Skipping that file. This is usually a transient error, please try again later“

The total file size is around 80 GB, but after downloading 71 GB it’s got failed. So to resume use the “–continue” option with the s3cmd command.

$ s3cmd --continue get s3://tastethelinux/tastethelinux.tar.gz

After using the –continue option it will start to download the file from where it left. Mean to say after the 71 GB.

Conclusion

In this guide, we have covered an Installation for s3cmd in Ubuntu and CentOS. Then we set up the S3cmd via command line with the Access key and secret keys. After that, we have seen some examples for the AWS S3 bucket. And how to download the larger file from the S3 bucket. If you found any issues or feedback let us know in the comment section.