This tutorial is for the AWS S3cmd, How to install S3cmd in Ubuntu 18.04 and CentOS 7. S3cmd is a tool for managing objects in Amazon S3 storage. With the help of the S3cmd command, we can make the buckets, and remove buckets.
Also manages the buckets and the objects inside them(files and folders). We can keep the versions of files and set the retention policy for the files and folders.
To set up s3cmd we need Access Key and Secret key, which can create on AWS IAM roles, you can refer to the following link.
4 topics will be cover in this tutorial for AWS S3cmd.
- How to install s3cmd on Ubuntu 18.04.
- How to install s3cmd on CentOS 7.
- Configure S3cmd with access and secret keys.
- S3cmd examples via command line.
How to Install S3cmd on Ubuntu 18.04
So to install the S3cmd command line, we have to update the repository by using the apt command.
$ sudo apt update
After the package update, let’s install the s3cmd by using the apt command.
$ sudo apt install s3cmd
If you want to set up the S3cmd you can jump to that section.
How to Install S3cmd on CentOS 7
So to install the S3cmd Command line, we have to update the repository by using the yum command.
# yum update
# yum install epel-release
# yum install s3cmd
How to Setup or Configure S3cmd with access and secret keys.
Then use the command “s3cmd –configure” to access the S3 bucket, let’s configure the s3cmd tool.
$ s3cmd --configure Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: XXXXTastethelinux Secret Key: XXXXTastethelinux#####$!@@!# Default Region [US]: region_of_the_Bucket Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: Press Enter Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: Press Enter Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]: Press Enter When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: Press Enter On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name: Press Enter New settings: Access Key: XXXXTastethelinux Secret Key: XXXXTastethelinux#####$!@@!# Default Region: region_of_the_Bucket S3 Endpoint: s3.amazonaws.com DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] n Save settings? [y/N] y Configuration saved to '/home/tastethelinux-ashish/.s3cfg'
So we have configured the s3cmd tool with the Access Key and Secret Key.
S3cmd examples via Command line.
1. List all the buckets in AWS s3 using s3cmd.
# s3cmd ls OUTPUT: 2019-12-30 10:14 s3://tla 2019-11-06 11:26 s3://tla-image 2020-07-01 11:07 s3://tastethelinux-backup 2019-11-07 13:59 s3://tla-logs
So we have 4 buckets(tla, tla-image, tastethelinux-backup, and tla-logs)into the AWS s3 Storage.
2. Create a New bucket into AWS S3.
So let’s create a bucket with the name “test-tla” using the command line.
# s3cmd mb s3://test-tla OUTPUT: Bucket 's3://test-tla/' created
3. Upload a file into the S3 bucket.
# s3cmd put tla.txt s3://tla/ OUTPUT: tla.txt -> s3://tla/tla.txt [1 of 1] 100619 of 100619 100% in 2s 668.35 kB/s done
So, we have uploaded a file tla.txt into a tla bucket.
4. Upload a folder or a directory in the s3 bucket.
# s3cmd put script s3://tla/ OUTPUT: ERROR: Parameter problem: Use --recursive to upload a directory: Script
So, we are getting the above error while uploading the Directory or folder into the bucket. let’s try a different command then.
# s3cmd put Script s3://tla/ -r OUTPUT: upload: 'Script/s3_Script.sh' -> 's3://tla/Script/s3_Script.sh' [1 of 4] 801 of 801 100% in 0s 7.27 kB/s done upload: 'Script/nodeJs.sh' -> 's3://tla/Script/nodeJs.sh' [2 of 4] 349 of 349 100% in 0s 6.90 kB/s done upload: 'Script/java' -> 's3://tla/Script/java' [3 of 4] 192 of 192 100% in 0s 6.47 kB/s done upload: 'Script/tla-rotate.sh' -> 's3://tla/Script/tla-rotate.sh' [4 of 4] 153 of 153 100% in 0s 1703.20 B/s done
So we have uploaded the folder into the S3 bucket. We have 4 files “s3_Script.sh, nodeJs.sh, java, and tla-rotate.sh”.
We have used the -r option to Upload the folder also –recursive will work.
5. Delete the file from the S3 bucket.
# s3cmd del s3://tla/Script/java OUTPUT: delete: 's3://tla/Script/java'
6. Delete a folder from the s3 bucket.
# s3cmd del s3://tla/Script OUTPUT: delete: 's3://tla/Script'
7. Delete the Bucket from AWS S3.
# s3cmd rb s3://tla OUTPUT: Bucket 's3://tla/' removed
If you want to learn more examples for the S3cmd command then follow the link. We will see “How to Download the larger file from AWS S3 bucket.
s3cmd download failed for the larger file
So, While Downloading from s3 Bucket I got the Error Download failed after the timeout error.
The total file size is around 80 GB, but after downloading 71 GB it’s got failed. So to resume use the “–continue” option with the s3cmd command.
$ s3cmd --continue get s3://tastethelinux/tastethelinux.tar.gz
After using the –continue option it will start to download the file from where it left. Mean to say after the 71 GB.
In this guide, we have covered an Installation for s3cmd in Ubuntu and CentOS. Then we set up the S3cmd via command line with the Access key and secret keys. After that, we have seen some examples for the AWS S3 bucket. And how to download the larger file from the S3 bucket. If you found any issues or feedback let us know in the comment section.