build a custom metrics on aws cloud
TRANSCRIPT
Create custom Cloudwatch metrics on AWS cloud & New Relic
Ahmad Karawash
PhD in Technology of Information, Book Editor,
CCA, Latece, ACM & IEEE member
12/23/2015 1
Amazon Metrics• Metrics are used to monitor various resources in Amazon Web
Services like EBS volumes, EC2 instances and RDS instances.
12/23/2015 2
AWS resources covered by CloudWatch
Amazon Ec2 EBS Volumes AutoScaling Groups Elastic load Balancers Amazon Route 53 RDS DB instances DynamoDB tables ElastiCache clusters RedShift clusters SQS queues SNS topics Storage Gateways
12/23/2015 3
Example of available metrics
• Memory Utilization – Memory allocated by applications and the operating system, exclusive of caches and buffers, in percentages.
• Memory Used – Memory allocated by applications and the operating system, exclusive of caches and buffers, in megabytes.
• Memory Available – System memory available for applications and the operating system, in megabytes.
• Disk Space Utilization – Disk space usage as percentages.• Disk Space Used – Disk space usage in gigabytes.• Disk Space Available – Available disk space in gigabytes.• Swap Space Utilization – Swap space usage as a percentage.• Swap Space Used – Swap space usage in megabytes.
12/23/2015 4
Cloudwatch advantage
• AWS provides some additional monitoring scripts for adding custom Cloudwatch metrics.
12/23/2015 5
Use case Example
• A user might have to take automated actions based on a particular parameter. This parameter can be the number of active users in a system which can be stored in an RDS instance. A script can be used to measure this value and if the number of users is zero for a particular time period, an alarm needs to be triggered and the EC2 instance needs to be terminated.
12/23/2015 6
Prerequisites
• Install python• Sudo apt-get install python python dev
• Sudo apt-get install python-pip
• Install boto• Sudo pip install boto
• Create RDS Database or local Database for the new custom metric, for example RoomTemperatureDatabase.
12/23/2015 7
put_metric_data
• put_metric_data ( $namespace, $metric_data, $opt ): Publishes metric data points to Amazon CloudWatch. Amazon Cloudwatchassociates the data points with the specified metric. If the specified metric does not exist, Amazon CloudWatch creates the metric.
• If you create a metric with the PutMetricData action, allow up to fifteen minutes for the metric to appear in calls to the ListMetricsaction.
12/23/2015 8
RoomTemperatureMeasureDatabase
12/23/2015 9
Create a Custom Metric and connect to external databaseRun the python code from Ubuntu console: (how to run check: https://www.youtube.com/watch?v=zKUu4in858A}
1. from boto.ec2.cloudwatch import connect_to_region
2. import MySQLdb
3. import boto.ec2.cloudwatch
4. db=MySQLdb.connect(“Public DNS","username","password","databasename")
5. cursor=db.cursor()
6. n=cursor.execute(" select value from Temperature ORDER BY id DESC LIMIT 1 ")
7. n1=cursor.fetchone()
12/23/2015 10
Create a Custom Metric and connect to external database8. s=str(n1)
9. table=string.maketrans( '', '', )
10. number=s.translate(table,"(){}<>,L")
11. numbers=int(number)
12. reg='ap-west-1'
13. conn_cw=boto.ec2.cloudwatch.connect_to_region(reg, aws_access_key_id=’your_access_key’,aws_secret_access_key=’your_secret_key’)
14. conn_cw.put_metric_data(namespace='my_namespace',name='my_metric',value=numbers,dimensions={'InstanceId':‘instance-val'})
12/23/2015 11
Create a Custom CloudwatchMetric• After you connect successively to the database, “put-metric-data”
method will add instance of a new custom metric into you cloudwatch
12/23/2015 12
How to check whether the custom metric was created
12/23/2015 13
How to check whether the custom metric was created
12/23/2015 14
Creating a CloudWatch alarm for the custom metric• Now you can monitor you own metric using Cloudwatch
• For example, you can add alarm:1. from boto.ec2.cloudwatch import connect_to_region
2. import boto.ec2.cloudwatch
3. metric=cw.list_metrics(dimensions={'InstanceId':’i-a1b2c3d4’},metric_name='my_metric')[0]
4. metric.create_alarm(name='my_alarm', comparison='<=', threshold=0, period=300,evaluation_periods=1, statistic='Average', alarm_actions='arn:aws:automate:ap-northeast-1:ec2:terminate')
12/23/2015 15
Example for other proposed use case• For example:
• if you want to create a new custom metrics that measure the temperature of the room, you can create your own database on the cloud to store the data and update it and you have to create a database to be used by cloud watch.
• For that purpose, you can use the same code I have shown in slides 8-9, but only apply changes to the database query.
12/23/2015 16
Real Implementation
• Using PHP and MySQL, I have created my own database that get temperature value periodically from weather web service (http://weather.yahooapis.com/forecastrss)
• Using Python, I have created a new custom metric that measure the temperature called temperature_metric you can check it online
• I can manage this metric easily by pushing my own data to cloudwatch and benefit from it for monitoring purpose.
• I Can add any notification system compatible with cloudwatch.
• Also, I can do my own alarm system using PHP code to forward emails when required.
12/23/2015 17
New Relic Broadcast alert
12/23/2015 18
In order to Broadcast metric notifications, we can use New Relic, because it is compatible with CloudWatch and use various channels such as:
E-mail: Sends an email message to the email address specified in the configuration.HipChat: Sends a message to the HipChat room specified in the configuration.PagerDuty: Sends a notification through the NetOpsadministration toolPagerDuty. This is an advanced option for the more critical notifications that need to addressed immediately—the alerts can even be configured to call you on the phone!Slack: Sends a notification to the Slack channel specified in the configuration.
New Relic Broadcast alert
12/23/2015 19
VictorOps: Similar to PagerDuty, this option sends a notification through the NetOps tool VictorOps. The functionality in the two tools is quite similar, so the choice depends mostly on what you're already using.Webhook: Sends a notification to a URL you define. Use this option if you want to send notifications to a channel that isn't currently supported directly by New Relic Alerts—or if you want to create your own custom solution...Campfire: Sends a notification to the Campfire chat room specified in the configuration.OpsGenie: Sends a notification using the OpsGenie NetOpsalert system. OpsGenie is another tool similar to PagerDutyand VictorOps that can be used to make sure your team notices the alerts as they arise.
New Relic Broadcast alert
12/23/2015 20
There is also a plan to open more notification channels in the future versions, they may add access to Lambda functions
For full details about how to connect New Relic to CloudWatch and start getting notification please read this web article: http://code.tutsplus.com/tutorials/get-started-with-monitoring-your-web-application-using-new-relic-alerts--cms-24187
