Pushing messages from Amazon SQS to DynamoDB
Amazon SQS is one of the most popular managed message queuing services. It’s used to decouple components and enable asynchronous processes in large applications. Some use cases include high-throughput, system-to-system messaging, buffer and batch jobs, request offloading, collecting application logs and metrics, triggering SMS/Email services, collecting metadata, etc. SQS combined with tools such as Lambda and DynamoDB forms a powerful framework to easily tackle the challenges associated with such operations.
In this article, I will demonstrate the process of asynchronously collecting data from SQS and storing it into DynamoDB using AWS Lambda. Such a setup could be useful in many scenarios — for example, collecting and storing metadata in a wide variety of applications — from e-commerce websites to streaming platforms.
Let us assume an application would like to collect the device and location information of a user as metadata and store it in a dynamoDB table. Let’s have a look at the overall architecture of the system at first :
It’s a fairly simple system to implement. The main application pushes messages into the message queue (SQS). The queue in turn triggers a lambda function and sends the message received, as a payload. The lambda function may contain code to perform a wide variety of operations on the data received. In this case, it saves data to a DynamoDB table and logs some information to Cloudwatch. For the sake of simplicity, we shall use the AWS Console to push messages to SQS, instead of a full-fledged application as it would serve the purpose of demonstrating the idea. So let’s walk through the steps now :
Step I: Create an SQS Queue
- Visit the SQS console and click on ‘Create Queue’ under ‘Get Started’:
2. Choose the queue ‘Type’ as ‘Standard’ and enter a name for the queue. Leave the other options as they are set by default, for the sake of simplicity. Scroll to the bottom of the page and click on ‘Create Queue’ as shown below:
3. Once the queue gets created, you get redirected to the details page of the queue. This console has many options and can be used to manage all the aspects of the queue. However we are only concerned with sending messages. Click on the ‘Send and receive messages’ button (second from right) :
4. In the ‘Send Message’ section, enter a message inside the ‘message body’ text box and click on the ‘Send Message’ button as shown :
5. To view the sent message in the queue, scroll down to the ‘Receive messages’ section and click on ‘Poll for messages’. This starts polling the queue and the messages fetched are displayed in this section:
At this stage, we have a functional queue that we can push messages to and poll messages from. The content of the message is available under the ‘body’ property of the message object. This is where we would find our required payload in a stringified JSON form. The next step would be to create a lambda function, which would be eventually triggered by the above queue.
Step II : Creating the Lambda Function
- Visit the AWS Lambda Console and click on the ‘Create Function’ button :
2. In the create function page, choose ‘Author from scratch’ . Under basic details, enter the function name. Then click on ‘Change default execution role’ under ‘Permissions’ section as shown :
3. Select ‘Create a new role from AWS policy templates’ . Enter the role name and the policy templates as shown :
4. Leave the other options as they are by default and click on ‘Create function’ at the bottom of the page. This would create a barebones Node.js lambda function and show it in an editor :
5. Overwrite the contents of the editor with the following code and click on ‘Deploy’ :
exports.handler = async (event) => {
console.log(event.Records); // array property has the messages from SQS
return;
};
At this stage we have a lambda function ready to receive messages from an SQS queue and log them. The logs can be viewed in Cloudwatch. The next step would be the most important one, where we shall make the SQS queue trigger the above lambda function.
Step III : Adding lambda trigger to SQS queue [IMPORTANT]
- Visit the SQS Console again and click on the queue created eariler:
2. Click on ‘Lambda Triggers’ and click on ‘Configure Lambda function trigger’ button.
3. Search for the lambda function created earlier by name in the drop down and select it. Click on the ‘Save’ button. This would successfully add the lambda trigger to the SQS queue :
4. Now to test our setup, we shall send a message to the queue (as shown earlier in the article). The message must get passed on to the lambda function.
5. Go back to the lambda console and click on the ‘Monitor’ tab. Then click on the ‘View logs in CloudWatch’ button. This will open a new tab and take you to CloudWatch.
6. In the CloudWatch log group page, you would find a log stream pretaining to the latest execution of the lambda function. Click on that to see the logs :
7. The log of the message object is highlighted below. Expand it to view the entire object. The message we sent will appear inside the ‘body’ field of the message object.
So at this stage, we have a functional setup where a message sent into SQS gets pushed to a lambda function which in turn logs it into CloudWatch. Now for the final piece of the puzzle, we have to setup a DynamoDB table and write some code to insert data into it.
Step IV : DynamoDB setup and integration
- Go to the DynamoDB console and click on ‘Create Table’:
2. Provide a table name and a partition key as shown below. Leave the other settings as they are by default. Scroll to the bottom of the page and click on ‘Create Table’ button :
3. The table would take some time to get created. In the meantime, go back to the Lambda console. Paste the following code inside the editor and click on ‘Deploy’ (like we did previously) :
The above code parses the message received from SQS and writes it into the dynamoDB table we created. It also logs some additional information to CloudWatch in case of success as well as error. Comments are provided for better understanding. Note that we do not need to install the aws-sdk dependency in the lambda environment as it’s provided by default.
Final Test
1.Go to the SQS console and send the following message :
{“user_id":"user_1","device":"iPhoneX","location":"Kolkata"}
2.Now visit the dynamo DB console and click on the table you created earlier :
3.Click on ‘Explore table items’ button at the right :
4. …..and voila! You can see an entry in the table with the desired data stored:
This concludes our demo. We could push some more messages into the queue and get the following results:
Also, you could check Cloudwatch logs for the information being logged there:
Thank you for reading through. Hope this was helpful to you 😊
