Subscribe an SQS Queue to an SNS Topic using CloudFormation
Dear Reader, I hope you are doing great. Remember a few days ago, I shared a post with you on how to create an SNS topic and subscription using AWS CloudFormation. In this post, I will help you subscribe an SQS queue to an SNS topic using CloudFormation.
So are you ready?
Alright !!!
Don’t want to miss any posts from us? join us on our Facebook group, and follow us on Facebook, Twitter, LinkedIn, and Instagram. You can also subscribe to our newsletter below to not miss any updates from us.
A Bit of Background
Sometimes we need to send the same message to multiple queues. For example, you might send the order details to the shipping, billing, order and marketing queues.
Instead of sending the same message to each queue separately, you can create an SNS topic and publish the message to the topic. All the queues that are interested in your message can subscribe to the topic to get a copy of the message when it is published.
Once an SQS queue is subscribed to an SNS topic, every time a message is published to the topic, SNS sends the message to all the subscribed queues.
Let’s see how can we accomplish all this via CloudFormation in an automated way.
Prerequisite:
- An AWS Account
- Basic Knowledge of CloudFormation
- Basic Knowledge of YAML/JSON
- Necessary Permission(We’ll discuss this later in this post)
Steps to Subscribe an SQS Queue to an SNS Topic using CloudFormation
In this post, we’ll create the SQS queue and SNS topic in the same stack and then we will create a subscription between them. However if you are doing it out of the stack, no worries. All you need to do is pass the topic ARN and Queue Arn.
Let’s see the step-by-step instruction to subscribe an SQS queue to an SNS topic using CloudFormation.
Step 1: Provide proper permission
Since I am creating the SNS topic and queue as part of the stack as well, I will need permission for that as well. Let’s see the permission in detail.
- sns:CreateTopic permission to create an SNS topic.
- sqs:CreateQueue permission to create an SQS queue.
- sns:Subscribe permission to be able to create a subscription.
In addition to these permissions, you should also have permission to create/update/delete a CloudFormation stack. I prefer cloudformation:* for making the work easier.
Note: It’s always recommended to follow the principle of least privilege. For example, If a user is only responsible for creating a stack, no need to give cloudformation:*
Step 2: Prepare a template
You can use YAML or JSON for your template. I prefer YAML for writing my templates. But don’t worry, If you want it in JSON, I’ll provide a JSON template as well. However, you can read this to can convert a JSON template to YAML and Vice Versa.
To create a subscription, all you need is a AWS::SNS::Subscription resource. Or in other words, if I say, AWS::SNS::Subscription resource subscribes an endpoint to an Amazon SNS topic. for example below resource subscribes an SQS queue endpoint to the SNS topic-
DemoSQSSNSSubscription:
Type: AWS::SNS::Subscription
Properties:
Protocol: sqs
TopicArn: !Ref DemoTopic
Endpoint: !GetAtt DemoQueue.ArnExplanation–
- Protocol sqs specifies that the endpoint that subscribes to this topic is an SNS queue.
- TopicArn is the Arn of the topic to which the DemoQueue subscribes.
- Endpoint is the queue subscribing to the topic and specifies queue ARN.
Additional points to remember–
- The endpoint owner must confirm the subscription if the SNS topic and SQS are not in the same AWS account.
- The queue also needs the policy to allow SNS to publish messages into the queue.
- For the subscription to work you must allow SNS to send a message to SQS by creating an SQS queue policy.
Template to Subscribe an SQS Queue to an SNS Topic using CloudFormation: YAML
In this template, we are creating one SQS queue, one SNS topic, a queue policy to allow SNS to send messages to the queue and a subscription resource to establish the subscription between the queue and topic.
You can change the name of or Arn based on your details.
AWSTemplateFormatVersion: 2010-09-09
Description: AWS CloudFormation Template to create SQS-SNS Subscription
Parameters:
QueueName:
Type: String
Description: Name of the Queue
Default: DemoQueue
TopicName:
Type: String
Description: Name of the SNS Topic
Default: DemoTopic
Resources:
#Resource to create an SNS Topic
DemoTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: "Demo Topic for this tutorial"
TopicName: !Ref TopicName
#Resource to create an SQS Queue
DemoQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Ref QueueName
#Resource to create SQS-SNS Subscription
DemoSubscription:
Type: AWS::SNS::Subscription
Properties:
Protocol: sqs
TopicArn: !Ref DemoTopic
Endpoint: !GetAtt DemoQueue.Arn
DemoQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref DemoQueue
PolicyDocument:
Id: AllowSendMessage
Statement:
- Sid: AllowSendReceiveWithinAccount
Effect: Allow
Principal:
AWS:
- !Ref AWS::AccountId
Action:
- sqs:SendMessage
- sqs:ReceiveMessage
Resource:
- !GetAtt DemoQueue.Arn
- Sid: AllowSNSTopicToSendMessage
Effect: Allow
Principal: '*'
Action:
- sqs:SendMessage
Resource:
- !GetAtt DemoQueue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref DemoTopic
Outputs:
QueueArn:
Description: Queue Arn
Value: !GetAtt DemoQueue.Arn
TopicArn:
Description: Topic Arn
Value: !Ref DemoTopic
Template to Subscribe an SQS Queue to an SNS Topic using CloudFormation: JSON
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "AWS CloudFormation Template to create SQS-SNS Subscription",
"Parameters": {
"QueueName": {
"Type": "String",
"Description": "Name of the Queue",
"Default": "DemoQueue"
},
"TopicName": {
"Type": "String",
"Description": "Name of the SNS Topic",
"Default": "DemoTopic"
}
},
"Resources": {
"DemoTopic": {
"Type": "AWS::SNS::Topic",
"Properties": {
"DisplayName": "Demo Topic for this tutorial",
"TopicName": {
"Ref": "TopicName"
}
}
},
"DemoQueue": {
"Type": "AWS::SQS::Queue",
"Properties": {
"QueueName": {
"Ref": "QueueName"
}
}
},
"DemoSubscription": {
"Type": "AWS::SNS::Subscription",
"Properties": {
"Protocol": "sqs",
"TopicArn": {
"Ref": "DemoTopic"
},
"Endpoint": {
"Fn::GetAtt": [
"DemoQueue",
"Arn"
]
}
}
},
"DemoQueuePolicy": {
"Type": "AWS::SQS::QueuePolicy",
"Properties": {
"Queues": [
{
"Ref": "DemoQueue"
}
],
"PolicyDocument": {
"Id": "AllowSendMessage",
"Statement": [
{
"Sid": "AllowSendReceiveWithinAccount",
"Effect": "Allow",
"Principal": {
"AWS": [
{
"Ref": "AWS::AccountId"
}
]
},
"Action": [
"sqs:SendMessage",
"sqs:ReceiveMessage"
],
"Resource": [
{
"Fn::GetAtt": [
"DemoQueue",
"Arn"
]
}
]
},
{
"Sid": "AllowSNSTopicToSendMessage",
"Effect": "Allow",
"Principal": "*",
"Action": [
"sqs:SendMessage"
],
"Resource": [
{
"Fn::GetAtt": [
"DemoQueue",
"Arn"
]
}
],
"Condition": {
"ArnEquals": {
"aws:SourceArn": {
"Ref": "DemoTopic"
}
}
}
}
]
}
}
}
},
"Outputs": {
"QueueArn": {
"Description": "Queue Arn",
"Value": {
"Fn::GetAtt": [
"DemoQueue",
"Arn"
]
}
},
"TopicArn": {
"Description": "Topic Arn",
"Value": {
"Ref": "DemoTopic"
}
}
}
}Step3: Create a Stack using the prepared template
Now, we know the basics and we have the template so let’s go and create the stack.
In this section, we are creating the stack using the console. However please feel free to create the stack using CLI. Here is a guide on how to it- Deploy a CloudFormation Template using AWS CLI.
Meanwhile, let’s go ahead the deploy the template using the AWS console.
- Grab the YAML or JSON template from above at your convenience.
- Save the template with .yml or .json as per the choice of template and follow the below steps.
- Login to AWS Management Console, navigate to CloudFormation and click on Create stack
- Click on “Upload a template file”, upload your saved .yml or .json file and click Next
- Enter the stack name and click on Next. In the configuration, keep everything as default and click on Next.
- In the events tab of stack, you can view the status.
- Once the stack is successfully created, you can go to the resource tab to see what is created and navigate to the resource.
Validate the Subscription
Once the stack is successfully created, subscription between SQS and SNS is established. Time to validate it.
Navigate to the created SNS topic in my case DemoTopic and click Publish message
put a subject and message and click Publish message. Message is successfully published and you get success message like this.
Let’s navigate to our DemoQueue to see if the message is received. As you notice below, available message is 1. That means message is recived by the queue.
Cross-Region SQS/SNS Subscription
If your SNS topic and SQS queue are in different region, then while you are creating subscription, specify the region like below-
DemoSubscription:
Type: AWS::SNS::Subscription
Properties:
Protocol: sqs
Endpoint: !GetAtt DemoQueue.Arn
Region: !Ref TopicRegion
TopicArn: !Ref DemoTopicClean Up
If you are creating this CloudFormation stack just for learning purpose. Don’t forget to delete your it so that all your resource is deleted and you don’t bear any cost accidently.
Happy Learning !!!
I also recommend to setup a cost budget in your account to avoid any such billing shocks.
Conclusion:
In this post, we learnt how to subscribe an SQS queue to an SNS topic using CloudFormation.
- We also saw the required permission to create an SQS queue, SNS topic, subscription and altogether a CloudFormation stack.
- We also saw how to set up cross-region subscription between SNS topic and SQS queue.
- We verified by publishing the message on the topic and receiving it on the SQS queue.
I hope you found this post helpful. Please let me know in the comment section.
Enjoyed the content?
Subscribe to our newsletter below to get awesome AWS learning materials delivered straight to your inbox.
Don’t forget to motivate me by-
- Adding a comment below on what you liked and what can be improved.
- Follow us on
- Subscribe to our newsletter to get notified each time we post new content
- Share this post with your friends