How to Create DynamoDB table with Global Secondary Index using CloudFormation

How to Create DynamoDB table with Global Secondary Index using Cloudformation

How to Create DynamoDB table with Global Secondary Index using CloudFormation?

A few days ago, I wrote a post on Creating a DynamoDB table using CloudFormation. Back then, I didn’t cover Global Secondary Indexes or GSI.

You might ask, why?

Well, I wanted to give GSI special attention as there is a lot to talk about them.

So, here I am with this post. Technically my aim for the post is to help you create a DynamoDB table with Global Secondary Index using CloudFormation.

Apart from that, I will let you know a few issues that you may encounter while adding GSI using CloudFormation and their solutions.

Before we prepare our template, Let’s start with understanding GSI.

Related: How to Create a DynamoDB Global Table using CloudFormation

What is Global Secondary Index?

In DynamoDB, If you want to perform a query on a non-key attribute, you need to create a secondary index.

DynamoDB supports two types of indexes.

  1. Local Secondary Index
  2. Global Secondary Index

You can read about secondary indexes in more detail here.

Let’s cover a bit of the Global Secondary Index(GSI) here.

Global Secondary Index

GSI is an index with a partition key and sort key(optional) that can be different from the base table. It can help you speed up queries on non-key attributes thus providing you better flexibility while querying your data. Ideally, your GSI can span across partitions in the base table and thus justifies the name global.

GSI = Partition Key + optional sort key

  • You can create a GSI at the time of table creation or later
  • You can create maximum of 20 global secondary indexes on a table

Things you need to specify while creating DynamoDB table with GSI using CloudFormation?

  1. IndexName: The name of your index, must be unique across indexes on this table
  2. KeySchema: Key schema for your GSI
  3. Projection: The attributes that you would like to project on your GSI. Can have three values – ALL | INCLUDE | KEYS_ONLY.
    • ALL- means, all the attributes from the base table will be projected into your GSI and this is the most expensive.
    • KEYS_ONLY- Only the index and primary keys are projected into the index.
    • INCLUDE- Keys are always projected but using INCLUDE you can specify list of other non-key attributes that you would like to have in the projection
  4. ProvisionedThroughput: If you have set up base table with BillingMode: PAY_PER_REQUEST then ProvisionedThroughput can’t be specified otherwise you must specify RCU and WCU using ProvisionedThroughput

Note: When the Projection attribute is set to INCLUDE, this is how you need to write it down in the template.

Projection:
  ProjectionType: INCLUDE 
  NonKeyAttributes: 
    - Attribute1
    - Attribute2
    - Attribute3

How does a GSI look in a CloudFormation template?

Resources:
  EmployeeTable:
    Type: AWS::DynamoDB::Table
    Properties:
      # Various parameters 
      # .....
      GlobalSecondaryIndexes:
        -
          IndexName: Location-index
          KeySchema:
            - 
              AttributeName: LocationId
              KeyType: HASH
          Projection:
            ProjectionType: ALL
          ProvisionedThroughput: 
             ReadCapacityUnits: 5
             WriteCapacityUnits: 5

Note: Please note that when BillingMode is set to PAY_PER_REQUEST, you can not specify ProvisionedThroughput. Either you provide ProvisionedThroughput RCU, WCU values on the base table and GSI both or make BillingMode on the base table to be PAY_PER_REQUEST and don’t specify ProvisionedThroughput anywhere.

Template to Create DynamoDB table with Global Secondary Index using CloudFormation JSON

In this template, we are specifying a table Employee with the primary key as EmployeeId. We are also adding two Global Secondary Indexes named Location-index and Department-index.

AWSTemplateFormatVersion: 2010-09-09
Description: AWS CloudFormation Template To Create a DynamoDB With GSI

Resources:
  EmployeeTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: Employee
      AttributeDefinitions:
        - 
          AttributeName: EmployeeId
          AttributeType: S
        - 
          AttributeName: LocationId
          AttributeType: S
        - 
          AttributeName: DepartmentId
          AttributeType: S
      KeySchema:
        - 
          AttributeName: EmployeeId
          KeyType: HASH
      GlobalSecondaryIndexes:
        -
          IndexName: Location-index
          KeySchema:
            - 
              AttributeName: LocationId
              KeyType: HASH
          Projection:
            ProjectionType: ALL
        -
          IndexName: Department-index
          KeySchema:
            - 
              AttributeName: DepartmentId
              KeyType: HASH
          Projection:
            ProjectionType: ALL
      BillingMode: PAY_PER_REQUEST
 
Outputs:
  Employee:
    Description: Table Created using this template.
    Value: !Ref EmployeeTable

Template to Create DynamoDB table with Global Secondary Index using CloudFormation JSON

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "AWS CloudFormation Template To Create a DynamoDB With GSI",
    "Resources": {
        "EmployeeTable": {
            "Type": "AWS::DynamoDB::Table",
            "Properties": {
                "TableName": "Employee",
                "AttributeDefinitions": [
                    {
                        "AttributeName": "EmployeeId",
                        "AttributeType": "S"
                    },
                    {
                        "AttributeName": "LocationId",
                        "AttributeType": "S"
                    },
                    {
                        "AttributeName": "DepartmentId",
                        "AttributeType": "S"
                    }
                ],
                "KeySchema": [
                    {
                        "AttributeName": "EmployeeId",
                        "KeyType": "HASH"
                    }
                ],
                "GlobalSecondaryIndexes": [
                    {
                        "IndexName": "Location-index",
                        "KeySchema": [
                            {
                                "AttributeName": "LocationId",
                                "KeyType": "HASH"
                            }
                        ],
                        "Projection": {
                            "ProjectionType": "ALL"
                        }
                    },
                    {
                        "IndexName": "Department-index",
                        "KeySchema": [
                            {
                                "AttributeName": "DepartmentId",
                                "KeyType": "HASH"
                            }
                        ],
                        "Projection": {
                            "ProjectionType": "ALL"
                        }
                    }
                ],
                "BillingMode": "PAY_PER_REQUEST"
            }
        }
    },
    "Outputs": {
        "Employee": {
            "Description": "Table Created using this template.",
            "Value": {
                "Ref": "EmployeeTable"
            }
        }
    }
}

Steps to Create DynamoDB table with Global Secondary Index using CloudFormation

Now, we know the basics and we have the template so let’s go and create the stack.

  1. Grab the YAML or JSON template from above.
  2. Save the template with .yml or .json as per the choice of template and follow the below steps.
  3. Login to AWS Management Console, navigate to CloudFormation and click on Create stack
  4. Click on “Upload a template file”, upload your saved .yml  or .json file and click Next
  5. Enter the stack name and click on Next. In the configuration, keep everything as default and click on Next.
  6. In the events tab of the stack, you can view the status.
  7. Once the stack is successfully created, you can go to the DynamoDB service and verify your table and all its GSI.

Important Notes:

  • You can create more than one GSI at the time of table creation
  • You can not create or delete more than one GSI in a single CloudFormation stack update
  • So ideally once your table is created, you can add indexes one by one. If you try to add more than one GSI you will get an error like: cannot perform more than one gsi creation or deletion in a single update
  • All the attributes present in the KeySchema(Primary key or Index) must be present in the AttributeDefinitions
  • Do not add any extra attribute in AttributeDefinitions
  • Do not use BillingMode: PAY_PER_REQUEST and ProvisionedThroughput at a time. Please mind you can only specify one out of two otherwise your stack creation will fail with Property ProvisionedThroughput can’t be used with PAY_PER_REQUEST BillingMode.
  • If you are creating indexes on more than one table at a time, add depends on attribute so that they are created sequentially or your stack will fail

I hope you are able to create your table successfully. Even after following all the above, If you face problems, Please feel free to drop in a comment. We will help you at the earliest.

Clean Up

If you are creating this DynamoDB table with GSI for learning purposes. Don’t forget to delete your CloudFormation stack so that your table is deleted and you don’t bear any cost.

Happy Learning !!!

Conclusion

In this post, you learnt to create DynamoDB table with Global Secondary Index using CloudFormation. You also learnt a few issues that might occur during the stack creation and also how to solve them.

I hope you found the post useful.

Enjoyed the content?

Subscribe to our newsletter below to get awesome AWS learning materials delivered straight to your inbox.

Don’t forget to motivate me by-

Suggested Read:

Leave a Reply

Your email address will not be published. Required fields are marked *