Automating ELK Stack Installation

Last time we installed an ELK stack on AWS. Today let’s setup an automation script using Python 2.7 to automate the installation of an Elk server.

Let’s make our calls for necessary modules.

import os
import boto3

We set up our access keys using environment variables so we don’t accidentally publish this information to a public repository. Then set the region we want our AWS EC2 instance.

os.environ[“AWS_ACCESS_KEY_ID”]
os.environ[“AWS_SECRET_ACCESS_KEY”]
os.environ[“AWS_DEFAULT_REGION”] = “us-west-2”

Let’s create the bash script that will pass to our instance once it is created. Our bash commands need to contain information for installing Java, creating the repositories for ElasticSearch, Logstash, and Kibana. We also need to include commands for starting our services and configuring the config files.

#Bash commands for installing elk stack
userdata = “””#!/bin/bash
sudo su
cd ~
wget –no-cookies –no-check-certificate –header “Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie” “http://download.oracle.com/otn-pub/java/jdk/8u73-b02/jdk-8u73-linux-x64.rpm”
yum -y localinstall jdk-8u73-linux-x64.rpm
rpm –import http://packages.elastic.co/GPG-KEY-elasticsearch
#create new repo for elasticsearch using rawgit.com to create a downloadable link to get the file needed
wget “https://cdn.rawgit.com/lanerjo/aws_ELK_stack_launcher/master/elasticsearch.repo” -P /etc/yum.repos.d/
#install elastic search
yum -y install elasticsearch
#edit elasticsearch config
sed -i ‘$network.host: localhost’ /etc/elasticsearch/elasticsearch.yml
service elasticsearch start
service enable elasticsearch
#kibana
#add kibana repo used rawgit.com to create a downloadable link to get the file needed
wget ‘https://cdn.rawgit.com/lanerjo/aws_ELK_stack_launcher/master/kibana.repo’ -P /etc/yum.repos.d/
#install kibana
yum -y install kibana
#edit kibana config
sed -i ‘$server.host: “localhost”‘ /opt/kibana/config/
#start kibana
service kibana start
#install logstash
#add logstash repo
wget ‘https://cdn.rawgit.com/lanerjo/aws_ELK_stack_launcher/master/logstash.repo’ -P /etc/yum.repos.d/
#install logstash
yum -y install logstash
service logstash start
“””

Finally we create and start our instance, pass our bash script and return needed information about our server.

#creating the ec2 instance on AWS using a predefined security group, t2 micro size, and amazon linux machine image
ec2 = boto3.resource(‘ec2′)
instances = ec2.create_instances(
ImageId=’ami-7172b611′,
InstanceType=’t2.micro’,
KeyName=’AWS_Testing’,
MinCount=1,
MaxCount=1,
SecurityGroupIds=[‘Jenkins’],
UserData=userdata,
)
#start the instance and print to command instance id, state, public dns, public ip
for instance in instances:
print(“Waiting until running…”)
instance.wait_until_running()
instance.reload()
print((instance.id, instance.state, instance.public_dns_name,
instance.public_ip_address))

Running this script from command line will start our ELK stack automated installation on a new AWS EC2 Instance.

Up next: creating our own private GIT server.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s