How to setup S3 bucket lifecycle configuration using CloudFormation

By PreetiAWS, CloudFormation, S3
How to setup S3 bucket lifecycle configuration using CloudFormation

How to setup S3 bucket lifecycle configuration using CloudFormation

AWS S3 or Simple Storage Service is one of the backbone services of AWS. When you use AWS as your cloud provider, there is almost always a chance that, you use S3 buckets in some way or other.

When you use S3 as your storage option, I am sure you want to know how you can manage your data/object lifecycle while storing your data into it cost efficiently.

S3 provides a feature called lifecycle configuration which automatically handles this. All you need is to define the lifecycle rules for your objects. Like, when you want to move an object from one storage class to other or to expire them altogether and rest is taken care of.

Suggested Read : AWS S3 Storage Classes: All You Need to Know

This post will help you setup S3 bucket lifecycle configuration using CloudFormation. But before doing that, I will give a brief introduction to lifecycle configuration.

What is S3 Lifecycle Configuration?

Lifecycle configuration is set of rules that defines the action S3 takes on your objects.

S3 object lifecycle configuration rules allows two types of action

  • Transition: transitions to a cheaper storage class
  • Expiration: permanently deletes an object

Sample Usecase that we will implement today using CloudFormation

You have stored some log files in S3 in logs folder. your logs are being accessed frequently for first 30 days. For next 90 days they are infrequently accessed , and then required to be stored for an year due to compliance reasons. Finally, after an year logs can be permanently deleted.

How do you deal with this usecase?

Now, I know you can manually go to AWS S3 console and move objects between storage classes and delete them after an year. But that’s crazy !!!

All you need is to lifecycle rule on the bucket specially on prefix /log as below.

  • Transition the objects to Standard IA after 30 days
  • Transition to Glacier after 90 days
  • Delete the object after 1 year/365 days

Simple right !!!

Let’s implement this same lifecycle rule using CloudFormation

Steps to setup S3 bucket lifecycle configuration using CloudFormation

Let’s see the step by step instruction to setup S3 bucket life-cycle configuration using CloudFormation.

Step 1: Provide proper permission

If you are not an admin user, you should explicitly provide s3:PutLifecycleConfiguration apart from create bucket permission for your user/role. Additionally, you will also needs cloudformation:* as well to be able to do CloudFormation stack creation, updation etc.

Step 2: Prepare a template

You can use YAML or JSON for your template. I prefer YAML for writing my templates. But don’t worry, If you want it in JSON, I will provide JSON template as well.

To configure life-cycle rules, you will need LifecycleConfiguration parameter of AWS::S3::Bucket resource.

A sample lifecycle configuration may look like below.

LifecycleConfiguration:
  Rules:
    - Id: Rule for log prefix
      Prefix: logs
      Status: Enabled
      Transitions:
        - TransitionInDays: 30
          StorageClass: STANDARD_IA
      ExpirationInDays: 365

Few things to note there –

  • Transitions are used for moving objects to a cheaper storage class while ExpirationInDays permanently deletes an objects after the no of specified days of creation
  • Use prefix to set different rules for different folders(logically)
  • If you want to apply rule to whole bucket, remove Prefix parameter altogether
  • You can create maximum of 1000 rules per bucket
  • Status can be Enabled which means rule applies currently
  • Status can be Disabled if you want experiment with correct set of rule and don’t want them to apply instantly
  • You can add multiple transitions in a single rule if they all apply to same set of objects
  • if you use expiration and transition both at the same time as we did in above template, time unit for both should be same like both to be in days or to provide exact date. you can’t mix and match here. Check rule documentation from AWS for more

Template to setup S3 bucket lifecycle configuration CloudFormation : YAML

In this template, we are creating an s3 bucket and lifecycle configuration rule as per above discussed usecase.

AWSTemplateFormatVersion: 2010-09-09
Description: Template to setup lifecycle configuration
Parameters:
  BucketName:
    Type: String
    Description: Name of the bucket on which lifecycle configuration will apply
    Default: lifecycle-config-demo-bucket
Resources:
  DemoBucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Ref BucketName
      LifecycleConfiguration:
        Rules:
          - Id: Rule for log prefix
            Prefix: logs
            Status: Enabled
            Transitions:
              - TransitionInDays: 30
                StorageClass: STANDARD_IA
              - TransitionInDays: 90
                StorageClass: GLACIER
            ExpirationInDays: 365
Outputs:
  BucketName:
    Value: !Ref DemoBucket
    Description: Name of the sample Amazon S3 bucket with a lifecycle configuration.

Template to setup S3 bucket lifecycle configuration using CloudFormation: JSON

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Template to setup lifecycle configuration",
    "Parameters": {
        "BucketName": {
            "Type": "String",
            "Description": "Name of the bucket on which lifecycle configuration will apply",
            "Default": "lifecycle-config-demo-bucket"
        }
    },
    "Resources": {
        "DemoBucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                "BucketName": {
                    "Ref": "BucketName"
                },
                "LifecycleConfiguration": {
                    "Rules": [
                        {
                            "Id": "Rule for log prefix",
                            "Prefix": "logs",
                            "Status": "Enabled",
                            "Transitions": [
                                {
                                    "TransitionInDays": 30,
                                    "StorageClass": "STANDARD_IA"
                                },
                                {
                                    "TransitionInDays": 90,
                                    "StorageClass": "GLACIER"
                                }
                            ],
                            "ExpirationInDays": 365
                        }
                    ]
                }
            }
        }
    },
    "Outputs": {
        "BucketName": {
            "Value": {
                "Ref": "DemoBucket"
            },
            "Description": "Name of the sample Amazon S3 bucket with a lifecycle configuration."
        }
    }
}

Step3: Create a Stack using prepared template

Now, we know the basics and we have the template so let’s go and create the stack.

  1. Grab the YAML or JSON template from above as per your convenience.
  2. Put a name of your choice in the template for your bucket name
  3. Save the template with .yml or .json as per the choice of template and follow below steps.
  4. Login to AWS Management Console, navigate to CloudFormation and click on Create stack
  5. Click on “Upload a template file”, upload your saved .yml  or .json file and click Next
  6. Enter the stack name and click on Next. In configuration, keep everything as default and click on Next.
  7. In the events tab of stack, you can view the status.
  8. Once stack is successfully created, you will see success events like below in the events tab.
CloudFormation events tab

Since stack creation is successful, let’s verify the bucket to see if lifecyle rule is created.

Verify created lifecycle rule:

Go to Resources tab and click on the Physical Id and you will be navigated to the s3 bucket we just created.

Physical Id of bucket

Click on the lifecycle rule name and you will be navigated to created bucket.

Click on Management tab. Do you see the created rule like below.

Created rule

Click on the Lifecycle rule name and finally you will be able to see the created lifecycle like below.

AWS S3 Lifecycle configuration using Cloudformation

Congratulations !!!

You have successfully learnt to setup S3 bucket lifecycle configuration using CloudFormation.

Clean Up

If you are creating this lifecycle configuration for learning purpose. Don’t forget to delete your CloudFormation stack so that your queue is deleted and you don’t bear any cost.

Happy Learning !!!

Conclusion:

In this post, we learnt how to setup S3 bucket lifecycle configuration using CloudFormation.

  • We learnt a bit about lifecycle configuration
  • Then we saw the actions that we can perform like transition and expiration
  • We saw how to set them using CloudFormation
  • We created the bucket with lifecycle configuration and verified in the S3 console.

I hope you found this post useful. Do leave a comment to motivate me to write more such content.

Enjoyed the content?

Subscribe to our newsletter below to get awesome AWS learning materials delivered straight to your inbox.

Don’t forget to motivate me by-

Suggested Read:

Leave a Reply

Your email address will not be published. Required fields are marked *