How to Download an Entire S3 Bucket in AWS
Have you ever tried downloading a file from AWS S3 or Simple Storage Service?
Well, you select a file and click on download, very simple – the file gets downloaded. easy peasy. 🙂
Now tell me, have you tried downloading multiple files or the entire bucket?
I have tried and here is my observation-
As soon as you select multiple files, the button to download gets disabled as you can see in the below screenshot.
Console is of no help in this situation. But don’t worry, In this tutorial, I will help you download your entire bucket content using AWS CLI commands.
So, are you ready?
Alright, let’s go.
How to Download an Entire S3 Bucket in AWS: Prerequisite-
Before using AWS CLI to download your entire bucket, you need to install CLI on your machine and configure it using your credentials(access key/secret key).
Here is a step-by-step tutorial on how to do it – How to Install and Configure AWS CLI in your System.
But, in short after downloading the CLI, you can use aws configure command to configure CLI using your credentials.
Enter your access key, secret key , default region and output format one by one and that’s it. You can verify it using a simple command like aws s3 ls to see if it works fine.
Suggested Read: All You need to Know about AWS CloudShell – Your Browser Based CLI
Download Your Entire Bucket using S3 Sync Command
If you have seen my previous post on AWS S3 CLI Commands Cheat Sheet, you might say that , hey I can use the cp command as well as the sync command so why to use sync here?
Well, you are absolutely right. You can use both options. However, cp is convenient when you are only downloading a few files. sync will actually sync your bucket and local directory or vice versa. It can even be used to sync two s3 bucket or folders.
On top of that, it only copies file that are additional. Imagine if your download fails halfway, it will only copy files which are not already present in your folder.
As the name implies, sync really means to sync two file locations.
“sync command recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files.”
Sync Syntax-
aws s3 sync s3://<your_source_s3_bucket> <your_local_path>For example-
If I want to download all the content of my bucket BucketName into my local working directory, this is how the command will look like.
aws s3 sync s3://BucketName .This command will start downloading all the objects in BucketName to the current directory.
Here .(dot) represent your current working directory. And you can change it to something like below to download content into s3content directory.
aws s3 sync s3://BucketName ./s3content/Suggestion: Before running s3 sync, I suggest you to dry run it once to see what is being downloaded.
aws s3 sync s3://demo-talk-with-anu . --dryrun
Now that we have seen, what’s being downloaded, let’s fire the actual sync command.
aws s3 sync s3://demo-talk-with-anu .
You will see the download progress as output as shown above.
If you check your current working directory, you will see the content is downloaded here including folder and it’s content(remember it downloads recursively).
If you try running sync command again- as you can see below – nothing happens.
And, it confirms that sync command only copies new files into the destination.
Conclusion
In this post on how to download an entire S3 bucket in AWS, we have seen that downloading an entire S3 bucket is not possible at the moment using the AWS console. And the download button gets disabled as soon as you select multiple files.
Therefore, we used AWS CLI s3 sync command to download the entire S3 bucket.
Hope the post was useful to you. Feel free to leave a comment or provide feedback.
Enjoyed the content?
Subscribe to our newsletter below to get awesome AWS learning materials delivered straight to your inbox.
Don’t forget to motivate me by-
- Adding a comment below on what you liked and what can be improved.
- Follow us on
- Share this post with your friends