AWS CLI has become a life saver when you want to manage your AWS infrastructure efficiently.
This also provides lot of possibilities for Automation and reduce the number of times that you have to login to AWS Management console.
If you are new to AWS CLI. Please consider reading this article on how to setup AWS CLI
With no further ado, Lets get into the objective of this post.
In this post we are going to talk about a very specific command of AWS CLI which is AWS S3
While AWS S3 provides complete tool set to manage your S3 bucket. we are going to see one specific feature of S3 CLI today. which is copy
Before we get there. we need to understand few things about AWS S3 CLI
There are two AWS S3 CLI commands available
- AWS S3
- AWS S3API
Let us see what they both have to offer.
Table of Contents
What is AWS S3 and S3API
The AWS CLI provides two tiers of commands for accessing Amazon S3
The s3 tier consists of high-level commands that simplify performing common tasks, such as creating, manipulating, and deleting objects and buckets.
The s3api tier behaves identically to the aforementioned S3 tier but it enables you to carry out advanced operations that might not be possible with s3 tier.
In this article we are going to talk about only the s3 tier and very specifically s3 cp command which helps us copying files from and to S3 buckets.
Before going any further I want you to know few handy commands which help to list the buckets.
aws s3 help- To get a list of all of the commands available in high-level commands.aws s3 ls- To get the list of all buckets.aws s3 ls s3://bucket-name- Will list all the objects and folders I that bucket.aws s3 ls s3://bucket-name/path/- This command will filter the output to a specific prefix.
Quick Caveats on AWS S3 CP command
- Copying a file from S3 bucket to local is considered or called as
download - Copying a file from Local system to S3 bucket is considered or called as
upload - Please be warned that failed uploads can't be resumed
- If the multipart upload fails due to a timeout or is manually cancelled by pressing CTRL + C, the AWS CLI cleans up any files created and aborts the upload. This process can take several minutes.
- If the process is interrupted by a kill command or system failure, the in-progress multipart upload remains in Amazon S3 and must be cleaned up manually in the AWS Management Console or with the s3api abort-multipart-upload command.
AWS S3 CP Command examples
Here we have listed few examples on how to use AWS S3 CP command to copy files.
Copying a local file to S3
Uploading a file to S3, in other words copying a file from your local file system to S3, is done with aws s3 cp command
Let's suppose that your file name is file.txt and this is how you can upload your file to S3
aws s3 cp file.txt s3://bucket-name
while executed the output of that command would like something like this.
Copying a local file to S3 with Storage Class
S3 Provides various types of Storage classes to optimize the cost and to manage the disk efficiency and IO performance during file read and write operations.
- S3 Standard
- S3 Intelligent-Tiering
- S3 Standard-IA
- S3 One Zone-IA
- S3 Glacier
- S3 Glacier Deep Archive
You can read more information about all of them here
aws s3 cp file.txt s3://bucket-name – storage-class class-name
Console Output:
In the prceding snapshot you can see that the test2.txt file which we have uploaded just now is showing the Standard-IA as the storage class
Copying an S3 object from one bucket to another
At times we would want to copy the content of one S3 bucket to another S3 bucket and this is how it can be done with AWS S3 CLI.
aws s3 cp s3://source-bucket-name/file.txt s3://destination-bucket-name/
How to Recursively upload or download (copy) files with AWS S3 CP command
When passed with the parameter --recursive the aws s3 cp command recursively copies all objects from source to destination.
It can be used to download and upload large set of files from and to S3.
Here is the AWS CLI S3 command to Download list of files recursively from S3. here the dot . at the destination end represents the current directory
aws s3 cp s3://bucket-name . – recursivethe same command can be used to upload a large set of files to S3. by just changing the source and destination
aws s3 cp . s3://bucket-name – recursiveHere we have just changed the source to the current directory and destination to the bucket and now all the files on the current directory(local) would be uploaded to the bucket.
It includes all the subdirectories and hidden files
Setting the Access Control List (ACL) while copying an S3 object
For security and compliance reasons, we might want to restrict the files that we are copying to S3 buckets with set of Access control. Like Read only or Read Write etc.
Here is the command that copies a file from one bucket to another bucket with specific ACL applied on the destination bucket ( after copied )
aws s3 cp s3://source-bucket-name/file.txt s3://dest-bucket-name/ – acl public-read-write
There are 5 types of ACL permissions available with S3 which are listed here on the following snapshot.
Hope this helps.
How to copy files between EC2 and S3
We have so far discussed how to copy files from local to EC2 and vice versa.
When it comes to EC2 instances it is a little different as the authentication can be taken care of by IAM roles assigned to the server.
S3 buckets can also have certain bucket policies to allow servers coming from a certain AWS account of VPC endpoint etc.
We have created an article that talks about how to easily copy files between EC2 and S3 with a simple 4 step configuration.
Copy files from EC2 to S3 Bucket in 4 steps
How to copy files with Sync - AWS S3 Sync
So if you are looking for advanced options while copying files like excluding the already existing files or deleting the existing files etc
In the Linux world, we use rsync rather than scp for this task and it is faster too.
likewise, we have a dedicated command for aws s3. aws s3 sync
we have a dedicated article on aws s3 sync. take a look
AWS S3 Sync Examples - Sync S3 buckets AWS CLI
Automate S3 tasks with Ansible
We have an exclusive article covering various examples of Ansible S3 module usage and examples.
If you are looking for some automation with S3. I would recommend you to give it a try.
How to use ansible with S3 - Ansible aws_s3 examples | Devops Junction
If you have any feedback or best practices. please feel free to comment and let us know.
Cheers
Hanumanth
More from Middleware Inventory
AWS CLI EC2 Stop and Terminate command Examples
In AWS infrastructure, We create a lot of EC2 instances on demand and we tend to forget about the stopped instances considering that it is stopped and not being billed for. But the fact is that Even the stopped instances would cost you a couple of pennies in various forms…
Automated Oracle Database Backup to S3
The Beautiful thing about Digital world is that we have a Ctrl+z button, snapshots and backups to go back in time and correct our mistakes, which is not possible in our real life. Enough with a philosophy. I would not have to tell you how crucial backups are and when…
AWS CLI List EFS Filesystem sort by Size
In this post we are going to see an aws cli command to list elastic file systems sorted by Size in ascending or descending order. While am sharing all the new tricks in AWS CLI I come across. this is something I have learnt recently There is a dedicated post…
AWS change instance type - From Console or AWS CLI
How to Change AWS Instance Type or EC2 instance type is more interesting and frequently asked question by many AWS Professionals. We create EC2 instances as t2, t3.micro but soon we realize that our requirements just got bigger for good and we need more CPU and RAM on our instance.…
AWS EC2 CLI List Examples - Describe instances | Devops Junction
AWS CLI is a very great help when it comes to efficiently managing your AWS Cloud Infrastructure and your EC2 instances. While we are managing our AWS Infrastructure, we cannot always afford to login to the AWS console all the time and it is not recommended from the security perspective…
As an expert and enthusiast, I have access to a wide range of information and can provide insights on various topics. However, I don't have direct access to the internet or real-time information. Instead, I have access to a database of information that was current up until September 2021. Therefore, I may not be able to provide the most up-to-date information on certain topics. It's always a good idea to verify any information I provide with more recent sources.
Now, let's discuss the concepts mentioned in the article you provided:
AWS CLI
AWS CLI (Command Line Interface) is a command-line tool provided by Amazon Web Services (AWS) that allows users to interact with AWS services and manage their infrastructure from the command line. It provides a convenient way to automate tasks, manage resources, and perform various operations on AWS services.
AWS S3
AWS S3 (Simple Storage Service) is a scalable object storage service provided by AWS. It allows users to store and retrieve large amounts of data, such as files, images, videos, and backups, in a highly available and durable manner. S3 provides a simple web services interface that can be accessed using the AWS Management Console, SDKs, or the AWS CLI.
AWS S3 CLI
AWS S3 CLI is a command-line interface specifically designed for managing AWS S3. It provides a set of commands that allow users to interact with S3 buckets, upload and download files, manage permissions, and perform various other operations on S3 objects.
AWS S3 CP Command
The aws s3 cp command is used to copy files between a local system and an S3 bucket, or between S3 buckets. It is part of the AWS CLI and provides a simple way to copy files to and from S3. The cp command supports various options and parameters to control the behavior of the copy operation, such as storage class, ACL (Access Control List), recursive copying, and more.
AWS S3API
The AWS S3API is a tier of commands in the AWS CLI that provides advanced operations for interacting with Amazon S3. While the s3 tier consists of high-level commands that simplify common tasks, the s3api tier enables users to carry out more advanced operations that may not be possible with the s3 tier.
Access Control List (ACL)
An Access Control List (ACL) is a set of permissions that defines who can access and perform operations on an S3 object or bucket. ACLs can be applied to individual objects or to the bucket as a whole. They provide a way to control access and manage permissions for different users or groups.
These are the main concepts mentioned in the article. If you have any specific questions or need more information on any of these topics, feel free to ask!
