Figure Out which EC2 Instance failed ELB Health Check on AWS

I have been using AWS for years and finally got around to wanting the ability to figure out exactly which EC2 instance failed the health check when the Alarm I had setup on the Load Balancer went off. The interwebs were lacking in a clear and easy to follow example on how to make this happen so I thought I might take a few minutes to explain how to set this up.




My preferred setup is to get emailed with the information about which instance(s) were unhealthy right away so I can troubleshoot the issue as soon as possible.  Kudos to this StackOverflow article for pointing me in the right direction.

Step 1: Create a new Topic to get triggered Alarm on your Load Balancer that triggers off Health Checks

Go to the monitoring tab on the ELB that you want to troubleshoot. There's an option to "Create Alarm" on the right hand side. Select that, create a topic if you don't have one already that gets notified anytime the Unhealth Hosts threshold is greater than zero.

ELB create Alarm and Topic

Step 2: Create a IAM role with the Permissions

The lamda function is going to need permissions to read the state of the ELB instances and send an email using SES. The only part of this which is a little bit tricky is that adding the second type of permissions requires finding the additional link.

Step 3: Create the Lambda

• 
  

  

Step 4: Add the Code & Test

import boto3
 import json
 import datetime
 from time import mktime
 class MyEncoder(json.JSONEncoder):
     def default(self, obj):
         if isinstance(obj, datetime.datetime):
             return int(mktime(obj.timetuple()))
         return json.JSONEncoder.default(self, obj)
 def lambda_handler(event, context):
     elb = boto3.client('elb')
     client = boto3.client('ses') 
     response = elb.describe_instance_health(LoadBalancerName='') 
     client.send_email(     Source='fromemail',     
           Destination={'ToAddresses': [         'samplemeail',      ] }, 
           Message={     'Subject': {         
                            'Data': 'AWS Notify MeEvent'},
                          'Body': { 'Text': { 'Data': json.dumps(response, cls = MyEncoder)}}} ) 
     return json.dumps(response, cls = MyEncoder)