Set up and Configure Ganglia Custom Metrics through Python module

In this tutorial you will learn how to monitor the health of your application through Ganglia using Python script.You can use shell script as well,but i will be using python script in this tutorial assuming ganglia is setup properly on CentOS or RHEL.

You can also refer to this tutorial for pushing custom metric into Ganglia through Gmetric.

Suppose you have an application running,and you want to use ganglia to monitor its health (like memory Used, Memory Free, No of Threads) along with application specific data (Message in Queue, Cache build time,etc). Let us assume your application publish this status through a url:

http://localhost:4080/dummyApp/healthStatus

Output:
jvm-maxm-mb:
jvm-min-mb
jvm-free-mb
no-of-threads:
msg-in-queue:
max-resp-time:
min-resp-time:
avg-resp-time:
cache-build-avg:

1. Now create a Python script to parse the value from above health status output(custom_metrics.py)

# Metric Initialization
def metric_init(params):
global descriptors, SERVER_STATUS_URL
 
if "metric_group" not in params:
params["metric_group"] = "custom_metrics"

if "refresh_rate" not in params:
params["refresh_rate"] = 2

if "url" not in params:
params["url"] = "http://localhost:4080/dummyApp/healthStatus"

SERVER_STATUS_URL = params["url"]

metric_obj_template = {
'name': 'jvm-maxm-mb',
        'call_back': get_value,
        'time_max': 90,
        'value_type': 'uint',
        'units': '',
        'slope': 'both',
        'format': '%u',
        'description': 'Application Server Metrics',
        'groups': "dummy"
}
metrics = get_metrics()[0]['data']
descriptors = list();
for m in metrics:
descriptors.append(create_desc({
"name"       : m,
"value_type" : "float",
"units"      : "",
"call_back"   : get_value,                
"format"     : "%f",
"description": m,
'groups': "dummy"
}))
return descriptors

# Create Metric object from template
def create_desc(prop):
d = metric_obj_template.copy()
for k,v in prop.iteritems():
d[k] = v
return d

# Get value for a metric
def get_value(name):

metrics = get_metrics()[0]

try:
result = metrics['data'][name]
except StandardError:
result = 0

return result

# Read metrics from REST API
def get_metrics():
global METRICS_GLOBAL_STORE_OBJ, SERVER_STATUS_URL, descriptors

try:
req = urllib2.Request(SERVER_STATUS_URL + "")

# Read stats from REST API
res = urllib2.urlopen(req, None, 2)

metrics = {};
for line in res:
split_line = line.rstrip().split(":")
metric_name = split_line[0]
metrics[metric_name] = split_line[1]
except urllib2.URLError:
traceback.print_exc()

METRICS_GLOBAL_STORE_OBJ = {
'time': time.time(),
'data': metrics
    }

return [METRICS_GLOBAL_STORE_OBJ]

# Clean up method
def metric_cleanup():
'''Clean up the metric module.'''
pass

# Main function for debug
if __name__ == '__main__':
try:
params = {
            'url'  : 'http://localhost:4080/dummyApp/healthStatus',
        }
metric_init(params)
get_metrics()
for d in descriptors:
v = d['call_back'](d['name'])
print 'value for %s is %s' % (d['name'],  v)
except KeyboardInterrupt:
os._exit(1)

2. Next create a custom metric configuration file (custom_metrics.conf)

modules {
module{
name = "custom_metrics"
language = "python"
param url {
value = "http://localhost:4080/dummyApp/healthStatus"
}

param metric_group {
value = "dummy"
}
}
}


collection_group {
  collect_every  = 2
  time_threshold = 60

  metric {
    name  = "jvm-maxm-mb"
    title = "JVM Maximum in MB"
    value_threshold = 0
  }

  metric {
    name  = "jvm-totm-mb"
    title = "JVM Total Memory in MB"
    value_threshold = 0

  }
. .  .  .  .  .  . .  .   .<put your other custom metrics with name as coming from the health service url>
}

3. Copy Custom metrics python module "custom_metrics.py" to Ganglia python modules directory1.      (/usr/lib64/ganglia/python_modules)

cp custom_metrics.py /usr/lib64/ganglia/python_modules

Make sure the script is executable before copying to ganglia python module directory
chmod +x custom_metrics.py

1.      4. Copy Custom metrics configuration file to Ganglia nodes configuration directory ( /etc/ganglia//conf.d )

  cp custom_metrics.conf /etc/ganglia/conf.d

5. Add following module and configuration entry into Ganglia nodes gmond.conf 

  vi /etc/ganglia/gmond.conf

 Look for modules section

 Add below module ( Make sure path and params values are exists)

module {

  name = "python_module"

  path = "/usr/lib64/ganglia/modpython.so"

  params = "/usr/lib64/ganglia/python_modules/"

 }

Now restart ganglia:

/etc/init.d/gmond stop

/etc/init.d/gmond start

For Setting up ganglia please refer to the previous tutorials