Send Slack alerts when large AWS EC2 instances are launched

Once you begin collecting AWS data in Splunk using AWS Addon for AWS, there are a lot of detection capabilities that open up. We will cover the use case of detecting when large EC2 instances are launched, such as instance types 2xlarge, 4xlarge or 8xlarge with or without GPU.

Due to their high CPU and memory specs they are prime targets for bitcoin miners and capturing the launch of such instances can indicate a compromised AWS Access Key. 

There are also cases when such instances are launched by mistake in multi-account AWS setup with hundreds of developers working at the same time. Since very large instance types also have a high price tag it's sensible to monitor and alert early on such events.

To create this alert we will use the CloudTrail input provided via AWS Addon for AWS. 

• The SPL search query will look for the instance types most often used in Crypto Currency mining,
• We will detect the user who performed this action, if he is logged in the console, or using AssumeRole
• The RunInstance event also reveals the users's IP address, which we use to detect instance creation actions from suspicious countries
• We will detect the AWS Account ID of the account where the launch action has been performed
• See how many instances have been launched together in the same API call

index=logging-audit sourcetype="aws:cloudtrail" eventSource="ec2.amazonaws.com" eventName="RunInstances"
| spath output=instanceType path=requestParameters.instanceType
| spath output=minCount path=requestParameters.instancesSet{}.items{}.minCount
| search instanceType IN ("p3.2xlarge", "p3.8xlarge", "p3.16xlarge", "p3dn.24xlarge", "p2.xlarge", "p2.8xlarge", "p2.16xlarge", "g3s.xlarge", "g3.4xlarge", "g3.8xlarge", "g3.16xlarge", "t2.nano")
| iplocation sourceIPAddress as Country 
| stats count by eventSource, eventName, awsRegion, userName, userIdentity.accountId, sourceIPAddress, Country, userIdentity.type, requestParameters.instanceType, responseElements.instancesSet.items{}.instanceId, responseElements.instancesSet.items{}.networkInterfaceSet.items{}.privateIpAddress, minCount
| fields - count

Note that we are applying command iplocation on field sourceIPAddress. The resulting country is then printed in the Slack alert.

Next, we configure the an alert to run in realtime from the SPL query above.

Alert Message uses values from the query result to provide context to sysadmins. This is done using the $result.field$ syntax. The field names are highlighted by surrounding text with asterisks.

Alert Message

A large EC2 instance has been launched.

*Instance Type*: $result.requestParameters.instanceType$
*AWS Account*: $result.userIdentity.accountId$
*Country*: $result.Country$
*User*: $result.userName$

For testing purposes we will pretend that also t2.nano is a large instance type and add it to the watch list. Then, using the AWS console we launch a t2.nano instance.

Soon, the instance launch is detected and an alert is pushed to the specified Slack channel

Notices All product names, logos, and brands are property of their respective owners. Splunk is a trademark of Splunk Inc. or its subsidiaries, registered or used in many jurisdictions worldwide.