How to Check If a Key Exists in S3 Bucket using Boto3 Python

Dear Reader, I hope you are doing awesome. Here I am doing equally good and back with another post :). In today’s post, you will learn how to check if a key exists in S3 bucket using Boto3 Python.

Amazon S3 is one of the most used AWS services. As it says the Simple Object Storage or S3 service stores everything as an object and you can find your object uniquely using the key. There are many times when we want to check if a file/object is there in the bucket. You can do that by checking if the key exists in the bucket instead of retrieving it. We’ll learn to do it using boto3 today.

Boto3: Boto3 is AWS SDK for Python that makes it easy for you to create a Python application that utilises AWS services like S3 or simple storage service.

In this example post using boto3, you’ll learn to find out if a key exists in s3.

Don’t want to miss any posts from us? join us on our Facebook group, and follow us on Facebook, Twitter, LinkedIn, and Instagram. You can also subscribe to our newsletter below to not miss any updates from us.

Prerequisite

• An AWS Account: Set up AWS Free Tier Account in Right Way
• Python
• AWS Credentials
• Boto3: I have got you covered – How to setup Boto3 in VS Code Easily
• An IDE like VS Code

How to Check If a Key Exists in S3 Bucket using Boto3 Python

You can use the Boto3 library to check if a key (object or file) exists in an S3 bucket. Boto3 as we said is the AWS SDK for Python that makes it easier for you to interact with an AWS service from your application.

To be honest, there are many ways in which you can know if an object or file is present inside an s3 bucket. But not all solutions are efficient or fit for all use cases. So here are a few ways that you can use based on your use case.

Method 1: Using s3.head_object Method

There are mainly two ways in which you interact with AWS using boto3 – Resource and Client. However after the Resource was deprecated recently, it makes sense to use Client only to do what we want to achieve.

In the below section, we are using the client.head_object method to find out if a key exists in a bucket.

import boto3

# Create S3 Client
s3 = boto3.client('s3')

# Bucket and Key that we want to check
demo_bucket_name = 'ck-demo-bucket-18th'
demo_key_name = 'terraform.drawio.png'

# Use head_object to check if the key exists in the bucket
try:
    resp = s3.head_object(Bucket=demo_bucket_name, Key=demo_key_name)
    print('Key exists !!!')
except s3.exceptions.ClientError as e:
    if e.response['Error']['Code'] == '404':
        print('Key does not exist !!!')
    else:
        # Handle Any other type of error
        raise

Explanation:

We imported boto3 and created an s3 client. After that, we call head_object method passing the bucket and key parameter. If the head_object succeeds without any exception, the key is present in the bucket. In this case, you can even print the response of the head_object to see the complete metadata that’s returned. In case the key doesn’t exist in the bucket, a S3.Client.exceptions.NoSuchKey is thrown. You can either directly catch that you or also catch it s3.exceptions.ClientError and benefit here I feel is you can see if there is another error like 403 in case you don’t have permission.

Success Response: Key Exists in Bucket

Error Response:

If you try to print the error response as –

except s3.exceptions.ClientError as e:
    print(e.response)

This is what it looks like-

Using the above error, you can catch a 404 and know if the key doesn’t exist. On the other hand, If you get 403 you know its access denied. and the same applies to any other 4XX error.

Method 2: Using list_objects_v2 Method

Most of the time we are interested in one object and to know if one key exists in a bucket. head_object kind of goes the first choice for that. However, if there is a requirement to see if a folder exists in s3 or not, how would you do that?

Well, you can use list_objects_v2 and specify the folder name as a prefix. For example, I have a logs folder in my s3 bucket, If I want to make sure that logs folder exists in my bucket using boto3 python, I can do it using below code.

import boto3

s3 = boto3.client('s3')

demo_prefix_to_check = 'logs'
demo_bucket_name = 'ck-demo-bucket-18th'

result = s3.list_objects_v2(Bucket=demo_bucket_name, Prefix=demo_prefix_to_check )

if 'Contents' in result:
    print('Exists !!!')
else:
    print('Does not exists !!!')

Output:

Exists !!!

Explanation:

The list_objects_v2 response contains an element called Contents. It contains the details about the prefix you have called for. That means if it’s there, then the folder exists. If the Contents key is not present in the response, it means that the folder doesn’t exist in the bucket.

Success Response – Prefix Exists

As you can see the Contents fiend has the details about the key/prefix.

Error Response – Prefix Doesn’t Exist

If you look closely at the above response, the Contents field is not present at all in the response.

And, this is the exact logic we have used in our code. If for a key/prefix, Contents is returned that means it exists in the s3 bucket otherwise it doesn’t.

Method 3: Using the s3.get_object method:

There are times when we want to know if an object exists in s3 and then we want to get it or retrieve it. Instead of making two subsequent calls why not combine both of them and just call s3.get_object.

If the exists, you will get the content as a response. If it doesn’t an error is thrown and you know it doesn’t exists;

import boto3

# Create S3 Client
s3 = boto3.client('s3')

# Bucket and Key that we want to check
demo_bucket_name = 'ck-demo-bucket-18th'
demo_key_name = 'terraform.drawio.png'

# Do the get operation to know if the key exists
try:
    resp = s3.get_object(Bucket=demo_bucket_name, Key=demo_key_name)
    print('Key exists in the bucket')
except s3.exceptions.NoSuchKey:
    print('Key does not exist in the bucket')

Output:

Key exists in the bucket

Explanation

In this code, we use the s3.get_object method to retrieve the contents of the specified key. If the key exists, the method returns without throwing an exception, and we print a message indicating that the key exists in the bucket. If the key doesn’t exist, the method throws NoSuchKey an exception, and we print a message indicating that the key doesn’t exist in the bucket. Note that this method incurs the additional overhead of retrieving the contents of the key, which may not be desirable if you only need to check for its existence.

Best Way to Check If a Key Exists in S3 Bucket using Boto3 Python?

All three methods (list_objects_v2, head_object, and get_object) have their own advantages and disadvantages, and the best method to use depends on your specific use case.

Here’s a comparison of the three methods:

• list_objects_v2: This method is useful if you want to retrieve a list of all the objects in a bucket with a certain prefix. However, if the bucket has a large number of objects, this method may be slower and consume more resources than the other two methods.
• head_object: This method is useful if you only want to check if a key exists in a bucket and don’t need to retrieve the contents of the key. It’s typically faster and consumes fewer resources than list_objects_v2 because it only retrieves the metadata for the specified key. However, if the key doesn’t exist, this method throws a ClientError exception, which you’ll need to catch and handle appropriately.
• get_object: This method is useful if you need to retrieve the contents of a key as well as check if it exists. However, it’s typically slower and consumes more resources than the other two methods because it retrieves the contents of the key. If the key doesn’t exist, this method throws a NoSuchKey exception, which you’ll need to catch and handle appropriately.

In general, if you only need to check if a key exists in a bucket and don’t need to retrieve its contents, using head_object is often the best choice because it’s typically faster and consumes fewer resources than the other two methods. However, if you need to retrieve the contents of a key or a list of all the objects in a bucket, you should use get_object or list_objects_v2, respectively.

Conclusion:

In this post, you learnt how to check If a key exists in S3 bucket using Boto3 Pythons. We also learnt some important points such as-

• head_object kind of goes as first choice if you just need to check if key exists and is most efficient.
• Using separate resources has advantages when cross-referencing security groups to avoid cyclic dependency.

I hope you found this post helpful. Feel free to drop your questions in the comment section.

Enjoyed the content?

Subscribe to our newsletter below to get awesome AWS learning materials delivered straight to your inbox.

Type your email…

SUBSCRIBE

Don’t forget to motivate me by-

• Adding a comment below on what you liked and what can be improved.
• Follow us on
  • Facebook, Twitter, LinkedIn and Instagram
• Share this post with your friends

Suggested Read:

• Attach an IAM Role to an EC2 Instance using Terraform