How to Create DynamoDB table with Global Secondary Index using CloudFormation

How to Create DynamoDB table with Global Secondary Index using Cloudformation
Sharing is Caring:

How to Create DynamoDB table with Global Secondary Index using CloudFormation?

Few days ago, I wrote a post on Creating a DynamoDB table using CloudFormation. Back then, I didn’t cover Global Secondary Indexes or GSI.

You might ask, why?

Well, I wanted to give GSI special attention as there are a lot to talk about them.

So, here I am with this post. Technically my aim for the post is to help you create a DynamoDB table with Global Secondary Index using CloudFormation.

Apart from that, I will let you know few issues that you may encounter while adding GSI using CloudFormation and their solutions.

Before we prepare our template, Let’s start with understanding GSI.

What is Global Secondary Index?

In DynamoDB, If you want to perform query on a non key attribute, you need to create secondary index.

DynamoDB supports two types of indexes.

  1. Local Secondary Index
  2. Global Secondary Index

You can read about secondary indexes in more details here.

Let’s cover a bit of Global Secondary Index or GSI here.

Global Secondary Index

GSI is an index with a partition key and sort key(optional) that can be different from the base table. It can help you speed up queries on non key attributes thus providing you better flexibility while querying your data. Ideally your GSI can span across partitions in the base table and thus justifies the name global.

GSI = Partition Key + optional sort key

  • You can create a GSI at the time of table creation or later
  • You can create maximum of 20 global secondary indexes on a table

Things you need to specify while creating DynamoDB table with GSI using CloudFormation?

  1. IndexName: Name of your index, must be unique across indexes on this table
  2. KeySchema: Key schema for your GSI
  3. Projection: The attributes that you would like to project on your GSI. Can have three value – ALL | INCLUDE | KEYS_ONLY.
    • ALL– means, all the attributes from base table will be projected into your GSI and this is most expensive.
    • KEYS_ONLY– Only the index and primary keys are projected into the index.
    • INCLUDE– Keys are always projected but using INCLUDE you can specify specify list of other non key attributes that you would like to have in the projection
  4. ProvisionedThroughput: If you have setup base table with BillingMode: PAY_PER_REQUEST then ProvisionedThroughput can’t be specified otherwise you must specify RCU and WCU using ProvisionedThroughput

Note: When Projection attribute is set to INCLUDE, this is how you need to write it down in the template.

Projection:
  ProjectionType: INCLUDE 
  NonKeyAttributes: 
    - Attribute1
    - Attribute2
    - Attribute3

How does a GSI look in a CloudFormation template?

Resources:
  EmployeeTable:
    Type: AWS::DynamoDB::Table
    Properties:
      # Various parameters 
      # .....
      GlobalSecondaryIndexes:
        -
          IndexName: Location-index
          KeySchema:
            - 
              AttributeName: LocationId
              KeyType: HASH
          Projection:
            ProjectionType: ALL
          ProvisionedThroughput: 
             ReadCapacityUnits: 5
             WriteCapacityUnits: 5

Note: Please note that when BillingMode is set to PAY_PER_REQUEST, you can not specify ProvisionedThroughput. Either you provide ProvisionedThroughput RCU, WCU values on base table and GSI both or make BillingMode on base table to be PAY_PER_REQUEST and don’t specify ProvisionedThroughput anywhere.

Template to Create DynamoDB table with Global Secondary Index using CloudFormation JSON

In this template we are specifying a table Employee with primary key as EmployeeId. We are also adding two Global Secondary Indexes named Location-index and Department-index.

AWSTemplateFormatVersion: 2010-09-09
Description: AWS CloudFormation Template To Create a DynamoDB With GSI

Resources:
  EmployeeTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: Employee
      AttributeDefinitions:
        - 
          AttributeName: EmployeeId
          AttributeType: S
        - 
          AttributeName: LocationId
          AttributeType: S
        - 
          AttributeName: DepartmentId
          AttributeType: S
      KeySchema:
        - 
          AttributeName: EmployeeId
          KeyType: HASH
      GlobalSecondaryIndexes:
        -
          IndexName: Location-index
          KeySchema:
            - 
              AttributeName: LocationId
              KeyType: HASH
          Projection:
            ProjectionType: ALL
        -
          IndexName: Department-index
          KeySchema:
            - 
              AttributeName: DepartmentId
              KeyType: HASH
          Projection:
            ProjectionType: ALL
      BillingMode: PAY_PER_REQUEST
 
Outputs:
  Employee:
    Description: Table Created using this template.
    Value: !Ref EmployeeTable

Template to Create DynamoDB table with Global Secondary Index using CloudFormation JSON

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "AWS CloudFormation Template To Create a DynamoDB With GSI",
    "Resources": {
        "EmployeeTable": {
            "Type": "AWS::DynamoDB::Table",
            "Properties": {
                "TableName": "Employee",
                "AttributeDefinitions": [
                    {
                        "AttributeName": "EmployeeId",
                        "AttributeType": "S"
                    },
                    {
                        "AttributeName": "LocationId",
                        "AttributeType": "S"
                    },
                    {
                        "AttributeName": "DepartmentId",
                        "AttributeType": "S"
                    }
                ],
                "KeySchema": [
                    {
                        "AttributeName": "EmployeeId",
                        "KeyType": "HASH"
                    }
                ],
                "GlobalSecondaryIndexes": [
                    {
                        "IndexName": "Location-index",
                        "KeySchema": [
                            {
                                "AttributeName": "LocationId",
                                "KeyType": "HASH"
                            }
                        ],
                        "Projection": {
                            "ProjectionType": "ALL"
                        }
                    },
                    {
                        "IndexName": "Department-index",
                        "KeySchema": [
                            {
                                "AttributeName": "DepartmentId",
                                "KeyType": "HASH"
                            }
                        ],
                        "Projection": {
                            "ProjectionType": "ALL"
                        }
                    }
                ],
                "BillingMode": "PAY_PER_REQUEST"
            }
        }
    },
    "Outputs": {
        "Employee": {
            "Description": "Table Created using this template.",
            "Value": {
                "Ref": "EmployeeTable"
            }
        }
    }
}

Steps to Create DynamoDB table with Global Secondary Index using CloudFormation

Now, we know the basics and we have the template so let’s go and create the stack.

  1. Grab the YAML or JSON template from above.
  2. Save the template with .yml or .json as per the choice of template and follow below steps.
  3. Login to AWS Management Console, navigate to CloudFormation and click on Create stack
  4. Click on “Upload a template file”, upload your saved .yml  or .json file and click Next
  5. Enter the stack name and click on Next. In configuration, keep everything as default and click on Next.
  6. In the events tab of stack, you can view the status.
  7. Once stack is successfully created, you can go to DynamoDB service and verify your table and all it’s GSI.

Important Notes:

  • You can create more then one GSI at the time of table creation
  • You can not create or delete more then one GSI in a single CloudFormation stack update
  • So ideally once your table is created, you can add indexes one by by. If you try to add more then one GSI you will get error like : cannot perform more than one gsi creation or deletion in a single update
  • All the attributes present in the KeySchema(Primary key or Index) must be present in the AttributeDefinitions
  • Do not add any extra attribute in AttributeDefinitions
  • Do not use BillingMode: PAY_PER_REQUEST and ProvisionedThroughput at a time. Please mind you can only specify one out of two otherwise your stack creation will fail with Property ProvisionedThroughput can’t be used with PAY_PER_REQUEST BillingMode.
  • If you are created indexes on more then one table at a time, add dependson attribute so that they are created sequentially or your stack will fail

I hope you are able to create your table successfully. Even after following all the above, If you face problem, Please feel free to drop in the comment. We will help you at the earliest.

Clean Up

If you are creating this DynamoDB table with GSI for learning purpose. Don’t forget to delete your CloudFormation stack so that your table is deleted and you don’t bear any cost.

Happy Learning !!!

Conclusion

In this post, you learnt to create DynamoDB table with Global Secondary Index using CloudFormation. You also learnt few issues that might occur during the stack creation and also how to solve them.

I hope you found the post useful.

Enjoyed the content?

Subscribe to our newsletter below to get awesome AWS learning materials delivered straight to your inbox.

Don’t forget to motivate me by-

Suggested Read:

Sharing is Caring:

Leave a Reply

Your email address will not be published. Required fields are marked *