Triggering AWS Lambda from Amazon SQS to store data into a DynamoDB Table
Project Overview —
To trigger a AWS Lambda function using Amazon SQS. This Lambda function will process messages from the SQS queue and insert the message data as records into a DynamoDB table.
SOLUTIONS ARCHITECTURE OVERVIEW -
First Let's understand the real world use case -
- E-commerce platform: A Lambda function can be triggered by an SQS queue containing incoming orders, and store them in DynamoDB for later processing. This can help ensure that orders are processed in a timely manner and can help track sales data.
- Data pipeline: You might have a data pipeline where you’re ingesting large amounts of data from different sources. Using SQS to queue up data to be processed and a Lambda function to process the data and store it in DynamoDB can help simplify the pipeline and make it more efficient.
- IoT applications: In an IoT application, you may have devices that send data to an SQS queue, such as temperature readings from sensors. A Lambda function can be triggered by the SQS queue to process the data and store it in DynamoDB for analysis.
- Social media platforms: A Lambda function triggered by an SQS queue can help you process incoming posts, comments or messages in real-time, and store them in DynamoDB for later analysis or processing. This can help provide insights into user engagement, sentiment analysis or trending topics.
Overall, the solution of using a Lambda function triggered by an SQS queue to store records in DynamoDB is a common pattern in building scalable and fault-tolerant distributed systems.
STEP BY STEP GUIDE -
we walk through on how to create the Lambda function and configure it to trigger from SQS. We then use a Python script on an EC2 instance to send a high volume of messages to an SQS queue. We also verify the Lambda function is being invoked and the messages are being saved in DynamoDB.
- AWS Account with Admin Access.
- Linux EC2 Instance with Admin IAM role.
AWS Services Usage —
Lambda, SQS, DynamoDB, EC2, IAM, CloudWatch
STEP 1 :
- Navigate to AWS IAM.
- Create a IAM role with name lambda-execution-role
- Attach below mentioned policies to the role
- Trust Relationships is for Lambda
STEP 2 :
- Navigate to AWS SQS.
- Create standard SQS Queue & Name it “Messages”
- Keep other settings as default.
STEP 3 :
- Navigate to AWS DynamoDB.
- Go Tables -> Create Table.
- Table Name = Message
- Partition Key = MessageId (String)
- Table settings = Customize settings
- Table Class = DynamoDB Standard
- Read/Write capacity settings = On-demand
- Encryption at rest = Owned by Amazon DynamoDB
- Click the Create Table button.
STEP 4 :
- Navigate to AWS Lambda.
- Create the Lambda Function.
- On the Create function page, select Author from scratch.
- Under Basic Information, set the following parameters for each field:
- Function name: Enter SQS-DDB.
- Runtime: Select Python 3.9 from the dropdown menu.
- Architecture: Select x86_64.
- Under Permissions, Select Use an existing role.
- Under Existing role, select lambda-execution-role from the dropdown menu.(created in step 1)
- Click the Create function button.
STEP 5 :
- On the Lambda Console our function SQS-DDB is created.
- Click the + Add trigger button to add SQS Trigger.
- Under SQS queue, click the search bar and select Messages.(step 2)
- Ensure that the checkbox next to Activate trigger is checked.
- Click Add button.
STEP 6 :
- Double-click on lambda_function.py from under the Code tab section.
- Copy the Source Code into the Lambda Function from My GitHub Repo.
- Replace the code contents of the function you copied from Github Repo.
- After that click the Deploy button.
STEP 7 :
- Log In to the Linux EC2 Instance which is part of prerequisite.
- Copy the python script to create fake messages from my Github Repo.
- Create a generate_fake_message.py file & paste the code in the file.
- Now run the script as follows to create fake messages which will be pushed to SQS Queue & eventually invoking Lambda Function.
- ./generate_fake_message.py -q Messages -i 0.1
- After a few seconds, hit Control + C to stop the command from continuing to run.
STEP 8 :
- Confirm Messages Were Inserted into the DynamoDB Table.
- Select the Message table.
- Click Explore table items and review the list of items that were inserted from our script.
- We should soon see a spike in the table Number of Messages Received in SQS Queue Monitoring.
- Congratulations on successfully completing this hands-on AWS lab!
- Source code & detailed steps can be found at My Github Repo.
- IMP NOTE — This DEMO/POC might incur some charges if kept active for long time. So please make sure to clean up the environment once done.
I am Kunal Shah, AWS Certified Solutions Architect, helping clients to achieve optimal solutions on the Cloud. Cloud Enabler by choice, DevOps Practitioner having 7+ Years of overall experience in the IT industry.
I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.
#aws #community #builders #devops #sqs #serverless #infrastructure
You can reach out to me @ acloudguy.in