AWS Security Monitoring using CloudWatch Agent

AWS Cloud Hands on Lab Practice Series

Elevating Security with AWS CloudWatch Monitoring and Alerts

Project Overview —

This project revolves around AWS SysOps best practices where we configure an Amazon Linux instances to send log files to Amazon CloudWatch and then create Amazon CloudWatch alarms and notifications to alert for a specified number of login failures on our EC2 instances. Finally, we create a CloudWatch alarm and notification to monitor outgoing traffic through a NAT gateway. By leveraging AWS CloudWatch capabilities we stay proactive & cover observability of our AWS Infrastructure.

SOLUTIONS ARCHITECTURE OVERVIEW -

First Let’s understand the real world use case -

• Web Application Security Monitoring — Real-time alerts will help us identify and respond to unauthorized access attempts. Monitoring failed logins across our instances helps us spot potential brute-force attacks. We can take immediate action, such as blocking IP addresses, when CloudWatch alarms trigger.
• Compliance and Auditing for Critical Services — Detailed logs and alerts provide the necessary evidence for compliance audits. We can set up CloudWatch alarms to detect deviations from security and compliance policies in real-time. Notifications can be used to inform security teams or compliance officers when an alarm is triggered, enabling swift remediation.
• Network Traffic Analysis for Cost Optimization-Real-time alerts will help us identify spikes in outgoing traffic that could lead to unexpected AWS data transfer costs. Monitoring traffic patterns allows us to make informed decisions about scaling resources up or down. We can automate scaling actions or adjust your application’s behavior based on traffic trends.
• Application Performance Monitoring-Set up custom CloudWatch dashboards to visualize critical metrics and detect performance bottlenecks. Create CloudWatch alarms to trigger notifications when metrics exceed predefined thresholds, indicating potential performance issues. We can integrate CloudWatch with AWS X-Ray for end-to-end tracing and monitoring of application requests.
• Serverless Application Monitoring-Set up custom CloudWatch Metrics to track specific business-related events or performance metrics within your serverless application. Use AWS CloudWatch Insights to perform advanced log analytics and gain deeper insights into function behavior. Implement AWS Step Functions to orchestrate serverless workflows and monitor their execution using CloudWatch Metrics and Alarms.

These use cases demonstrate the versatility of Amazon CloudWatch in monitoring and managing various AWS resources and services, from traditional EC2 instances to serverless applications. By leveraging CloudWatch, we can enhance security, optimize costs, and maintain compliance across various scenarios in real-world AWS environments. By effectively utilizing CloudWatch features, we can proactively manage and secure your infrastructure while meeting specific business needs and objectives.

Prerequisite —

• AWS Account with Admin privileges
• Access to CloudFormation, EC2, CloudWatch, IAM

AWS Services Usage —

AWS VPC, IAM, EC2, CloudWatch

STEP BY STEP GUIDE -

STEP 1 : Deploying the proposed architecture Infrastructure.

• Download the CloudFormation template from the Repo.
• Create a new stacks for both the templates.
• This CloudFormation stacks will deploy resources required for this hands on Lab (2 EC2 Instance, NLB, VPC, Nat Gateways, Lambda Functions, IAM roles & policies)
• It will take around 5–10 mins for entire Infrastructure to become active.

STEP 2 : Deploying the CloudWatch Agent on DB Instance

• Go to EC2 dashboard, select Database Server Instance.
• Connect to Instance through AWS session manager.
• Now, Install the CloudWatch agent package using below command
• sudo yum install -y amazon-cloudwatch-agent
• The CloudWatch agent installer creates a new user named cwagent.
• Now provide read and run access to the /var/log/secure file for the cwagent user.
• sudo setfacl -m u:cwagent:rx /var/log/secure
• We are only sending security logs to CloudWatch & hence we gave access to /var/log/secure only.

STEP 3 : Start the CloudWatch Agent on DB Instance

• Run the below command to start the CloudWatch agent configuration wizard
• sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
• We will see the wizard opens with the following options:
• Enter 1 for Linux
• For Are you using EC2 or On-Premises hosts, enter 1 for EC2.
• For Do you want to turn on StatsD daemon, enter 2 for no.
• For Do you want to monitor metrics from CollectD. enter 2 for no.
• For Do you want to monitor any host metrics, enter 2 for no.
• Do you have any existing CloudWatch Log Agent configuration file to import for migration, enter 2 for no.
• For Do you want to monitor any log files, enter 1 for yes.
• For Log file path, enter /var/log/secure
• For Log group name, enter database_server_security_logs
• For Log stream name, press ENTER to keep the default value of Instance ID
• For Log Group Retention in days, enter 9 for 90
• For Do you want to specify any additional log files to monitor, enter 2 for no.
• For Do you want to store the config in SSM parameter store, enter 2 for no.
• Now lets start the CloudWatch agent service:
• sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-ahgent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
• We will see the information about the configuration file that is loaded by the agent.
• Check CloudWatch agent service status- sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-ahgent-ctl -m ec2 -a status

STEP 4 : Generate Auth failures in the security logs

• Run the following command to log in as dbdev :
• su dbdev
• when prompted for a password, try to give wrong creds (to create auth failures)
• Repeat this process for 4–5 times to generate logs.

STEP 5 : Check log files in CloudWatch Logs

• Navigate to CloudWatch -> logs -> log groups
• On the log groups, select the group by the name database_server_security_logs log group.
• On the database_server_security_logs, choose Log streams tab.
• We will see one log stream from DB server Instance.
• This will have security logs from DB server Instance.

STEP 6 : Create a Metric Filter

• On the database_server_security_logs page, choose the Metric filters tab.
• Now create metric filter → Define pattern
• For Filter pattern, enter “authentication failure”
• In the Test pattern, select the EC2 Instance ID of DB server.
• Choose Test pattern -> Next
• For Assign metric, For Filtername give db server auth failures
• For Metric namespace, enter authentication failures & create new is on.
• For Metric name, enter db server auth failures
• Metric value 1 and Default value 0
• Now review & create -> choose Create metric filter.

STEP 7 : Create a CloudWatch alarm & SNS notification.

• On the database_server_security_logs page, choose the Metric filters tab, we will see db server auth failures filter.
• Choose Create alarm -> Specify metrics & conditions.
• For Metric name -> db server auth failures
• For Statistic -> Sum and Period -> 5 mins. (your choice)
• In the Conditions -> Threshold type select Static -> Greater -> 2
• Expand the additional configuration section & for Missing data treatment, select Treat missing data as good (not breaching threshold)
• Select Next -> Configure actions -> Notification
• For Alarm state trigger, select In alarm
• For Select an SNS topic, select Create New Topic.
• For Create a new Topic, enter DB_Cloudwatch_alarm_notifications.
• For Email endpoints -> enter your email address.
• Choose Create Topic -> Next -> Add name & description.
• For Alarm name -> database server authentication failures alarm
• For Alarm description -> Alarms and notifies for more than 2 auth failures over a span of 5 minutes ( you can edit according to your use case)
• Choose Next -> Preview and Create -> Choose Create alarm
• Check email & we must confirm email address & then SNS can send notification messages.

8 : Test the alarm & SNS notification.

• Login to DB server again & hit su dbdev
• when prompted for a password, try to give wrong creds (to create auth failures)
• Repeat this process for 4–5 times to generate logs.
• Now we should get the notification on email as soon as alarm get in In alarm state.

STEP 9 : Lets monitor NAT Gateway

• Create New alarm → select metric -> Metrics -> AWS namespaces Choose NATGateway.
• Choose Nat Gateway Metrics
• Search for BytesOutToDestination in metric search bar.
• Select the NatGatewayId that matches the NatGatewayId value listed.
• Choose Select metric -> Metric Name -> BytesOutToDestination
• For NatGatewayId, verify it matches the NatGatewayId value
• Statistic -> Sum & Period -> 15 mins
• In the conditions -> Threshold type -> Static -> Greater -> 2000000
• Choose Next -> Configure actions -> Notification -> Alarm state trigger -> In alarm
• Select SNS -> Existing topic ->DB_Cloudwatch_alarm_notifications (created earlier) -> Next
• Add Name & description according to your use case.
• Preview and Create, Choose Create alarm.

STEP 10 : Generate traffic & activate the alarm

• Upload any file which is higher than 5 MB to S3 bucket.
• NAT Gateway outgoing traffic alarm will go In alarm state.
• Check your email for notification email.

STEP 11 : Decommission

• Delete all the created alarm & SNS topics
• Delete the CloudFormation stacks from CloudFormation.

Congrats ! We have successfully completed lab for AWS Security Monitoring using CloudWatch Agent

I am Kunal Shah, AWS Certified Solutions Architect, helping clients to achieve optimal solutions on the Cloud. Cloud Enabler by choice, DevOps Practitioner having 7+ Years of overall experience in the IT industry.

I love to talk about Cloud Technology, DevOps, Digital Transformation, Analytics, Infrastructure, Dev Tools, Operational efficiency, Serverless, Cost Optimization, Cloud Networking & Security.

#aws #community #builders #devops #cloudwatch #natgateway #observability #proactive #incident #management #centralize #network #security #logging #monitoring #operations #controlled #infrastructure #authentication #control #acloudguy

You can reach out to me @ acloudguy.in