How to Configure AWS SQS Dead Letter Queue using CloudFormation

How to Configure AWS SQS Dead Letter Queue using CloudFormation
Sharing is Caring:

Dear Reader, Hope you are doing good. In today’s post, I will help you configure AWS SQS dead letter queue using CloudFormation.

But, before we configure an SQS DLQ using CloudFormation, let’s try to understand what exactly is a dead letter queue and how it works. Feel free to skip below section If you already are familiar with SQS DLQ.

What is AWS SQS Dead Letter Queue?

As, you might already know, Amazon SQS or Simple Queue Service is a fully managed message queue Service provided by AWS. In the world of micro-services architecture, Amazon SQS helps you decouple your application components so that they can work and fail independently.

Message producers sends messages to the queue and the consumer processes it. Sounds perfect right?

What happens if a message processing fails?

What if a consumer is unable to process a message even in multiple tries?

You don’t want to keep trying to process a single message infinitely right?

You want to proceed with the next message. But, of course you would like to debug your app later for any such failed messages.

Right?

How do you do that?

Well, to deal with such messages, SQS provides you a feature called dead letter queue. you can store any message that failed to process after few attempts into a dead letter queue.

In other words, SQS DLQ allows us to store message that couldn’t be processed by a consumer due to whatsoever reasons like corrupted message or coding errors etc. So that you can check DLQ later to take action on the messages.

To be more specific, SQS DLQ stores the messages which was received successfully by a consumer but could not be processed even after multiple tries.

To understand SQS DLQ and it’s working in details, please free to check out my previous article on SQS dead letter queue.

By this time, I assume you have got clear idea about what SQS DLQ is and how it works. Let’s setup a dead letter queue for a application queue using CloudFormation.

How to Configure AWS SQS Dead Letter Queue using CloudFormation?

Let’s see the step by step instruction to configure AWS SQS dead letter queue using CloudFormation.

Step 1: Provide proper permission

  • If you are not an admin user, you should explicitly provide sqs:CreateQueue and sqs:SetQueueAttributes permission for your user/role.
  • Additionally, you will also need cloudformation:* to be able to do CloudFormation stack creation, updation etc.

Note: If you are unable to figure out correct permission in the beginning, you can use sqs:* but please it’s not safe to allow all action and you should only use it till you find correct set of permission.

Step 2: Prepare a template

You can use YAML or JSON for your template. I prefer YAML for writing my templates. But don’t worry, If you want it in JSON, I will provide JSON template as well.

How do you configure a dead letter queue?

To configure a dead letter queue for your actual queue, all you need is to create two queues(1 source queue and 1 to be used as dead letter)

Set redrive policy of source queue to send failed message to dead letter queue.

AWS::SQS::Queue resource with RedrivePolicy looks like below.

 MySourceQueue: 
    Type: AWS::SQS::Queue
    Properties: 
      RedrivePolicy: 
        deadLetterTargetArn: !GetAtt MyDeadLetterQueue
        maxReceiveCount: 5

deadLetterTargetArn : Here you provide the actual ARN of queue you would like to use as dead letter queue.

maxReceiveCount: The number of times a message is delivered to the source queue before being moved to the dead-letter queue.

Few things to note there –

  • The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a standard queue.
  • By default CloudFormation creates a standard queue for you. However, If you need a FIFO queue, you can specify a parameter FifoQueue as true.
  • Once you create a queue, you can not change it’s type. For example you can not convert a Standard queue to FIFO and vice versa after the creation.
  • If you are creating a FIFO queue, name must end with .fifo suffix.
  • Your queue name must be unique for your account and region combination.
  • SQS does not create a queue automatically. You must first create a queue and then use it as dead letter queue.
  • Both source queue and dead letter queue must be present in same account and region

Template to Create Amazon SQS Queue using CloudFormation : YAML

In this template, we are declaring two amazon queue , one standard and one FIFO. After the creation of queue we are outputting queue URL and Arn for later use.

AWSTemplateFormatVersion: '2010-09-09'
Description: Template to configure a sample queue

Parameters:

  MySourceQueueName:
    Type: String
    Description: Standard Queue Name
    Default: app-queue
  MyDeadLetterQueueName:
    Type: String
    Description: Fifo Queue Name
    Default: app-dead-letter-queue

Resources:

 MySourceQueue: 
    Type: AWS::SQS::Queue
    Properties: 
      QueueName: !Ref MySourceQueueName
      RedrivePolicy: 
        deadLetterTargetArn: !GetAtt MyDeadLetterQueue.Arn
        maxReceiveCount: 5
      
 MyDeadLetterQueue: 
    Type: AWS::SQS::Queue
    Properties: 
      QueueName: !Ref MyDeadLetterQueueName     
          
Outputs: 
  SourceQueueURL: 
    Description: URL of Source Queue
    Value: !Ref MySourceQueue
  SourceQueueARN: 
    Description: ARN of source queue
    Value: !GetAtt MySourceQueue.Arn
  DeadLetterQueueURL: 
    Description: URL of dead-letter queue
    Value: !Ref MyDeadLetterQueue
  DeadLetterQueueARN: 
    Description: ARN of dead-letter queue
    Value: !GetAtt MyDeadLetterQueue.Arn

Template to Create Amazon SQS Queue using CloudFormation: JSON

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "Template to configure a sample queue",
    "Parameters": {
        "MySourceQueueName": {
            "Type": "String",
            "Description": "Standard Queue Name",
            "Default": "app-queue"
        },
        "MyDeadLetterQueueName": {
            "Type": "String",
            "Description": "Fifo Queue Name",
            "Default": "app-dead-letter-queue"
        }
    },
    "Resources": {
        "MySourceQueue": {
            "Type": "AWS::SQS::Queue",
            "Properties": {
                "QueueName": {
                    "Ref": "MySourceQueueName"
                },
                "RedrivePolicy": {
                    "deadLetterTargetArn": {
                        "Fn::GetAtt": [
                            "MyDeadLetterQueue",
                            "Arn"
                        ]
                    },
                    "maxReceiveCount": 5
                }
            }
        },
        "MyDeadLetterQueue": {
            "Type": "AWS::SQS::Queue",
            "Properties": {
                "QueueName": {
                    "Ref": "MyDeadLetterQueueName"
                }
            }
        }
    },
    "Outputs": {
        "SourceQueueURL": {
            "Description": "URL of Source Queue",
            "Value": {
                "Ref": "MySourceQueue"
            }
        },
        "SourceQueueARN": {
            "Description": "ARN of source queue",
            "Value": {
                "Fn::GetAtt": [
                    "MySourceQueue",
                    "Arn"
                ]
            }
        },
        "DeadLetterQueueURL": {
            "Description": "URL of dead-letter queue",
            "Value": {
                "Ref": "MyDeadLetterQueue"
            }
        },
        "DeadLetterQueueARN": {
            "Description": "ARN of dead-letter queue",
            "Value": {
                "Fn::GetAtt": [
                    "MyDeadLetterQueue",
                    "Arn"
                ]
            }
        }
    }
}

Step3: Create a Stack using prepared template

Now, we know the basics and we have the template so let’s go and create the stack.

  1. Grab the YAML or JSON template from above as per your convinience.
  2. Save the template with .yml or .json as per the choice of template and follow below steps.
  3. Login to AWS Management Console, navigate to CloudFormation and click on Create stack
  4. Click on “Upload a template file”, upload your saved .yml  or .json file and click Next
  5. Enter the stack name and click on Next. In configuration, keep everything as default and click on Next.
  6. In the events tab of stack, you can view the status.
  7. Once stack is successfully created, you can go to SQS service and verify your queue.
  8. Also, you can check output tab of your CloudFormation stack to view queue url and Arn of queue.
configure sqs queue using cloudformation success

Clean Up

If you are configuring this SQS queue with dead letter queue for learning purpose. Don’t forget to delete your CloudFormation stack so that your queue is deleted and you don’t bear any cost.

Happy Learning !!!

Conclusion:

In this post, we configured AWS SQS dead letter queue using CloudFormation. Let’s summarize what we learnt.

We learnt that-

  • SQS dead letter queue works great to store messages that couldn’t be processed successfully even after certain number of attempts named as MaxReceiveCount
  • Messages can be stored in DLQ for latter debugging or other actions
  • AWS doesn’t automatically creates a deal letter queue. You must first create a queue and then use it as dead letter queue
  • So, ideally we created two queue and used one as dead letter queue by specifying it as target in the source queue Redrive policy.
  • Both source queue and dead letter queue should be of same type
  • Also, both the queue should be present in the same account and same region

I hope you found this post helpful.

Enjoyed the content?

Subscribe to our newsletter below to get awesome AWS learning materials delivered straight to your inbox.

Don’t forget to motivate me by-

Suggested Read:

Sharing is Caring:

2 thoughts on “How to Configure AWS SQS Dead Letter Queue using CloudFormation

    1. You mean EC2 keypair .pem file to .ppk? You can use puttygen which comes bundled with putty. Let me know if you need help. I can come up with a tutorial

Leave a Reply

Your email address will not be published. Required fields are marked *