Generate configuration files for Splunk Add-on for AWS using Ansible

The Splunk Add-on for AWS is an addon supporting data collection from AWS services. At the time of writing, it can seamlessly collect AWS config, AWS config rules, AWS Cloudtrail, CloudWatch, Cloudwatch logs and AWS inspector, Kinesis, S3 via SQS and billing data.

This article touches the method of collecting data using IAM roles. This involves setting up an IAM role for EC2, assigning it to the Splunk instance where the AWS Add-on is installed, then configure that role for collection jobs.

To collect data from multiple accounts, a role is needed in each AWS account with a policy that allows reading from the services that need to be collected, which will be assumed by the role asigned to the EC2 running Splunk Add-on. Within accounts, I called the role Splunk_Role.

The Addon relies on configurations that tell it what inputs to collect and where from. Splunk Add-on for AWS provides two methods of collecting inputs: using the UI and using configuration files.

While the UI may be a practical option when collecting inputs from one AWS account, it can become painful to click through the UI to setup collection of multiple inputs from hundreds of AWS accounts.

Fortunately, inputs can also be configured using configuration files. These files are stored in folder $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/
Some of the inputs are concatenated in inputs.conf, other have individual configuration files.

Ansible playbook

Ansible uses the concept of roles. A role is a mechanism of breaking a playbook into multiple files, simplifying complex playbooks and making them easier to reuse.

For this example, we will create an individual role for each type of Splunk AWS Add-on input and have the execution of the playbook produce resulting configuration files locally in folder output/. All the roles will be then called from the main playbook which will also assemble inputs according to the expected Splunk structure.

Structure of the playbook

├── group_vars
│   └── all
├── output
│   ├── aws_description_tasks.conf
│   ├── inputs.conf
│   ├── splunk_ta_aws_iam_roles.conf
│   └── tmp
│       └── cloudwatch.conf
├── roles
│   ├── aws_description
│   │   ├── files
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   │   └── aws_description_tasks.conf.j2
│   │   └── vars
│   │       └── main.yml
│   ├── cloudwatch
│   │   ├── files
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   │   └── cloudwatch.conf.j2
│   │   └── vars
│   │       └── main.yml
│   └── splunk_ta_aws_iam_roles
│       ├── tasks
│       │   └── main.yml
│       ├── templates
│       │   └── splunk_ta_aws_iam_roles.conf.j2
│       └── vars
│           └── main.yml
└── site.yml

To create .config files, the playbook uses JINJA2 language which is a templating language for python. It allows us to create consistent configuration files, iterating through a set of variables. 

Variables

The playbook uses local and global variables. Global variables are defined in group_vars/all and are used by all roles

---
accounts:
  - accountid: 321321321321
    accountname: zebras-prod
  - accountid: 654654654654
    accountname: zebras-acc
  - accountid: 987987987987
    accountname: zebras-test

Role splunk_ta_aws_iam_roles

An entry is being created for each of the roles to be assumed within AWS Accounts. The role name in accounts is always Splunk_Role. This role must be deployed in the accounts beforehand.
The name of the IAM role to be assumed is for simplicity, the account number

{% for item in accounts %}
[{{ item.accountid }}]
arn = arn:aws:iam::{{item.accountid}}:role/Splunk_Role
{% endfor %}

Role aws_description

This JINJA template uses global variables found in group_vars, specified iterating with_items and variables local to the role, specified in vars/main.yml

Role definition

---
  - name: Generate aws metadata file (aws_description_tasks.yml)
    template:
      src: aws_description_tasks.conf.j2
      dest: output/aws_description_tasks.conf
      remote_src: true
    with_items: "{{ accounts }}"

JINJA2 template

{% for item in accounts %}

[{{ item.accountid }}-{{ item.accountname }}_description]
account = {{ account }}
apis = ec2_volumes/3600, ec2_instances/3600, ec2_reserved_instances/3600, ebs_snapshots/3600, classic_load_balancers/3600, application_load_balancers/3600, vpcs/3600, vpc_n
etwork_acls/3600, cloudfront_distributions/3600, vpc_subnets/3600, rds_instances/3600, ec2_key_pairs/3600, ec2_security_groups/3600, ec2_images/3600, ec2_addresses/3600, la
mbda_functions/3600, s3_buckets/3600, iam_users/3600
index = {{ item.accountid }}-{{ item.accountname }}
regions = {{ regions }}
sourcetype = aws:description
aws_iam_role = {{ item.accountid }}

{% endfor %}

Role cloudwatch

At the end of the definition for CloudWatch inputs splunk appends a UUID
In order to generate a UUID with Ansible, we're harnessing filter to_uuid

[aws_cloudwatch://{{ item.accountid }}-{{ item.accountname }}_cloudwatch_{{ 9999999999999999999999 | random | to_uuid }}_]
aws_account = {{ account }}
aws_iam_role = {{ item.accountid }}
aws_region = {{ regions }}
index = {{ item.accountid }}-{{ item.accountname }}
metric_dimensions = [{"ServiceName":[".*"],"Currency":[".*"]}]
metric_names = ".*"
metric_namespace = AWS/Billing
period = 300
polling_interval = 3600
sourcetype = aws:cloudwatch
statistics = ["Average","Sum","SampleCount","Maximum","Minimum"]
use_metric_format = false

Playbook

Run the playbook and see the results in folder output/

ansible-playbook site.yml

Browse the code of this Ansible playbook on GitHub https://github.com/monids/splunk-aws-addon-ansible

Notices All product names, logos, and brands are property of their respective owners. Splunk is a trademark of Splunk Inc. or its subsidiaries, registered or used in many jurisdictions worldwide.