My Journey to the cloud…

In pursuit of excellence….

Collect Cloudwatch metrics (including custom one) and upload to S3 bucket

Recently I wrote a script to pull the cloudwatch metrics (including the custom ones – Memory utilization) using CLI. Objective is to have have the data published to S3 and then using Athena/QuickSight, create a dashboard so as to have a consolidated view of all the servers across All the AWS accounts for CPU and Memory utilization.

This dashboard will help to take a right decision on resizing the instances thereby optimizing the overall cost.
Script is scheduled (using crontab) to run every one hour. There are 2 parts of the script
1. – This is the main script
2. – This is a wrapper and internally calls python script.

How the script is called :

/path/ <Destination_AWS_Account ID> <S3_Bucket_AWS_Account_ID> [<AWS_Region>]

Wrapper script –

if [[ $# -lt 2 ]]; then
  echo "Usage: ${0} <AccountID> <S3_Bucket_AccountID>"
  exit 1
NOW=$(date +"%m%d%Y%H%M")
AWS_DEFAULT_REGION=${3} ## 3rd Argument is the Account Default Region is diff than the CLI server
## Reset Env variables
reset_env () {
        unset AWS_SESSION_TOKEN
        unset AWS_DEFAULT_REGION
        unset AWS_ACCESS_KEY_ID
} #end of reset_env
## Set Env function
assume_role () {
source </path_to_source_env_file/filename> ${AccontID}
# Function assume_role ends
assume_role ${AccontID}
if [[ ! -z "$3" ]]; then
## Generate CSV file
python <path_of_the_script>/ ${AccontID} ${csvfile}
## Upload generated CSV file to S3
assume_role ${s3_AccountID}
echo ${csvfile}
echo "Uploading data file  to S3...."
aws s3 cp ${csvfile} <Bucket_Name>

Main python Script –

# To Correct indent in the code - autopep8
import sys
import boto3
import logging
import pandas as pd
import datetime
from datetime import datetime
from datetime import timedelta

AccountID = str(sys.argv[1])
csvfile = str(sys.argv[2])
logger = logging.getLogger()
# define the connection
client = boto3.client('ec2')
ec2 = boto3.resource('ec2')
cw = boto3.client('cloudwatch')

# Function to get instance Name
def get_instance_name(fid):
    ec2instance = ec2.Instance(fid)
    instancename = ''
    for tags in ec2instance.tags:
        if tags["Key"] == 'Name':
            instancename = tags["Value"]
    return instancename

# Function to get instance ID (mandatory for Custom memory Datapoints)
def get_instance_imageID(fid):
    rsp = client.describe_instances(InstanceIds=[fid])
    for resv in rsp['Reservations']:
        v_ImageID = resv['Instances'][0]['ImageId']
    return v_ImageID

# Function to get instance type (mandatory for Custom memory Datapoints)
def get_instance_Instype(fid):
    rsp = client.describe_instances(InstanceIds=[fid])
    for resv in rsp['Reservations']:
        v_InstanceType = resv['Instances'][0]['InstanceType']
    return v_InstanceType

# all running EC2 instances.
filters = [{
    'Name': 'instance-state-name',
    'Values': ['running']

# filter the instances
instances = ec2.instances.filter(Filters=filters)

# locate all running instances
RunningInstances = [ for instance in instances]
# print(RunningInstances)
dnow =
cwdatapointnewlist = []

for instance in instances:
    ec2_name = get_instance_name(
    imageid = get_instance_imageID(
    instancetype = get_instance_Instype(
    cw_response = cw.get_metric_statistics(
                'Name': 'InstanceId',
        Statistics=['Average', 'Minimum', 'Maximum']

    cw_response_mem = cw.get_metric_statistics(
                'Name': 'InstanceId',
                'Name': 'ImageId',
                'Value': imageid
                'Name': 'InstanceType',
                'Value': instancetype
        Statistics=['Average', 'Minimum', 'Maximum']

    cwdatapoints = cw_response['Datapoints']
    label_CPU = cw_response['Label']
    for item in cwdatapoints:
        item.update({"Label": label_CPU})

    cwdatapoints_mem = cw_response_mem['Datapoints']
    label_mem = cw_response_mem['Label']
    for item in cwdatapoints_mem:
        item.update({"Label": label_mem})

# Add memory datapoints to CPUUtilization Datapoints

    for cwdatapoint in cwdatapoints:
         timestampStr = cwdatapoint['Timestamp'].strftime(
             "%d-%b-%Y %H:%M:%S.%f")
         cwdatapoint['Timestamp'] = timestampStr
         cwdatapoint.update({'Instance Name': ec2_name})
         cwdatapoint.update({'Instance ID':})

df = pd.DataFrame(cwdatapointnewlist)
df.to_csv(csvfile, header=False, index=False)

Sample Flat file (CSV format) is as shown below.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

About Me

I’m a Hands-On Technical & Entrprise Solutions Architect based out of Houston, TX. I have been working on Oracle ERP, Oracle Database and Cloud technologies for over 20 years and still going strong for learning new things.

You can connect me on Linkedin and also reach out to me

I am certified for 8x AWS, OCP (Oracle Certified Professionals), PMP, ITTL and 6 Sigma.


This is a personal blog. Any views or opinions represented in this blog are personal and belong solely to the blog owner and do not represent those of people, institutions or organizations that the owner may or may not be associated with in professional or personal capacity, unless explicitly stated.
All content provided on this blog is for informational purposes only. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site.

The owner will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information. Any script available on the blog post MUST be tested before they are run against Production environment.


%d bloggers like this: