Fairly regularly I give a training on Weblogic Tuning and Troubleshooting, where I talk about JVMs, Garbage Collections, and some subsystems of Weblogic, JMS and JDBC for instance, and how to tune and troubleshoot them.

One of the larger parts of the training is the Weblogic Diagnostic Framework. I find it quite interesting, but also pretty complex. And maybe therefor hardly used in the every day Weblogic administration. And that might be a pity, because it can be quite powerfull. You can find the use of it in Fusion Middleware, with preconfigured policies and actions. But I guess that other tooling on Weblogic diagnostics and monitoring, like WLSDM also rely on it (although I don't know for sure).

Configuring WLDF might be quite hard, and during the last run of the training, I figured that it might help to turn the solution of the workshop into a script. At least to show what you're doing when executing the labs. But, certainly also to show how you can put your configurations into a script that you can extend and reuse over different environments.

This week I got a question on the Oracle community,To be notify of les warning Logs, that made me remember this script. Maybe it's not exactly the answer, but I think at least it can be a starting point. And I realized that I did not write about it yet.

 11g vs 12c

I stumbeld upon a nice 11g blog about this subject. In 12c Oracle renamed this part of the WLDF: where in 11g it was called "Watches and Notifications" it is now called "Policies and Actions". When working with the console, you'll find that the console follows the new naming. But in WLST the APIs still have the old 11g naming. So keep in mind that Policies are Watches and Actions are Notifications.

Documentation about Configuring Policies and Actions can be found here.

I'm not going to explain all the concepts of subject, but going through my base script step by step, and then conclude with some remarks and ideas.

Diagnostic Module

Just like JMS resources, the Diagnostic Framework combines the resources into WLDFSystemResource Modules. A Diagnostic Module is in essence an administration unit to combine the resources.  A diagnostic module can be created by the following WLST function:

#
def createDiagnosticModule(diagModuleName, targetServerName):
  module=getMBean('/WLDFSystemResources/'+diagModuleName)
  if module==None:
    print 'Create new Diagnostic Module'+diagModuleName
    edit()
    startEdit()
    cd('/')
    module = cmo.createWLDFSystemResource(diagModuleName)
    targetServer=getMServer(targetServerName)
    module.addTarget(targetServer)
    # Activate changes
    save()
    activate(block='true')
    print 'Diagnostic Module created successfully.'
  else:
    print 'Diagnostic Module'+diagModuleName+' already exists!'
  return module

The script is created in a way that it first checks if the Diagnostic Module already exists. You'll see that all the functions in this article work like this. This helps also in the use of it in Weblogic under Kubernetes environments. Diagnostic modules are registered under '/WLDFSystemResources'. And it is created with the createWLDFSystemResource() method. Also like a JMSModules you need to target them. This function does this based on the targetServerName and uses the getMServer() function to get the MBean to target:

#
def getMServer(serverName):
  server=getMBean('/Servers/'+serverName)
  return server

Many Weblogic resources need to be targetted. I notice that I do this in many different ways in different scripts all over this blog and in my work. Maybe I need to write a more generic, smarter way of doing this. In this case I simply just target to a single server, but it could be list of servers and/or clusters.
The function is called from a main function as follows:

import os,sys, traceback
#
adminHost=os.environ["ADM_HOST"]
adminPort=os.environ["ADM_PORT"]
admServerUrl = 't3://'+adminHost+':'+adminPort
#
adminUser='weblogic'
adminPwd='welcome1'
ttServerName=os.environ["TTSVR_NAME"]
diagModuleName='TTDiagnostics'
#
...
def main():
  try:
    print 'Connect to '+admServerUrl
    connect(adminUser,adminPwd,admServerUrl)
    createDiagnosticModule(diagModuleName, ttServerName)
...

Collectors

Collectors are also called Harvesters. They monitor MBeans Attributes and regularly store the data in the Harvested Data Archive of the targetted Managed Server. You can find it under Diagnostics->Log Files in the Weblogic Console. The Weblogic console also includes a Monitoring Dashboard. That can be get to via the console Home page using the 'Monitoring Dashboard' link.

Without collectors, you can only view  MBean Attributes in the Graphs from the moment you start a graph. It will collect the MBeans from that moment onwards, until you pause/stop the collection.
However, using a collector you can view the Attribute values from the history back.

A Collector is created within a diagnostic module. You need to  define a metricType: the MBean Type, for instance 'JDBCDataSourceRuntimeMBean'.  Then a namespace, in this case the ServerRuntime. You can specify a set of instances of the particular MBean Type, or provide None to watch all the instances of the particular type. And from the instances you specify a comma separated list of attributes you want to harvest.

This leads to the following function:

#
def createCollector(diagModuleName, metricType, namespace, harvestedInstances,attributesCsv):
  harvesterName='/WLDFSystemResources/'+diagModuleName+'/WLDFResource/'+diagModuleName+'/Harvester/'+diagModuleName 
  harvestedTypesPath=harvesterName+'/HarvestedTypes/';
  print 'Check Collector '+harvestedTypesPath+metricType
  collector=getMBean(harvestedTypesPath+metricType)
  if collector==None:
    print 'Create new Collector for '+metricType+' in '+diagModuleName
    edit()
    startEdit()
    cd(harvestedTypesPath)
    collector=cmo.createHarvestedType(metricType)
    cd(harvestedTypesPath+metricType)
    attributeArray=jarray.array([String(x.strip()) for x in attributesCsv.split(',')], String)
    collector.setHarvestedAttributes(attributeArray)
    collector.setHarvestedInstances(harvestedInstances)
    collector.setNamespace(namespace)
    # Activate changes
    save()
    activate(block='true')
    print 'Collector created successfully.'
  else:
    print 'Collector '+metricType+' in '+diagModuleName+' already exists!'
  return collector

This creates the Collecter using createHarvestedType() for the MBean Type (metricType). The list of Attributes is provided as comma separated string. But the setter on the collector (setHarvestedAttributes(attributeArray)) expects a jarray, so the csv-list needs to be translated. It is created by a (to me a bit peculiar when I first saw it) python construct:

    attributeArray=jarray.array([String(x.strip()) for x in attributesCsv.split(',')], String)

It splits the csv string with the comma as a separator and then loops over the resulting values. For each value it constructs a String, that is trimmed. The resulting String values are fed to a String based jarray.array factory.

The following line added to the main function will call the function, when you want to watch all inststances:

createCollector(diagModuleName, 'weblogic.management.runtime.JDBCDataSourceRuntimeMBean','ServerRuntime', None, 'ActiveConnectionsCurrentCount,CurrCapacity,LeakedConnectionCount')

In the case you do want to select a specific set of instances, you need to do that as follows:

    harvestedInstancesList=[]
    harvestedInstancesList.append('com.bea:ApplicationRuntime=medrec,Name=TTServer_/medrec,ServerRuntime=TTServer,Type=WebAppComponentRuntime')
    harvestedInstances=jarray.array([String(x.strip()) for x in harvestedInstancesList], String)    
    createCollector(diagModuleName, 'weblogic.management.runtime.WebAppComponentRuntimeMBean','ServerRuntime', harvestedInstances,'OpenSessionsCurrentCount') 

The thing in this case is that the instances self are described in an expression that uses commas. You could construct these expressions using properties of course. And then use the construct above to add those to a jarray.

Actions

When you want to have WLDF to take action upon a certain condition, you need to create an Action for it. A simple one is to create a message on a JMS queue. But according to the documentation you could have the following types:

• Java Management Extensions (JMX)
• Java Message Service (JMS)
• Simple Network Management Protocol (SNMP)
• Simple Mail Transfer Protocol (SMTP)
• Diagnostic image capture
• Elasticity framework (scaling your dynamic cluster)
• REST
• WebLogic logging system
• WebLogic Scripting Tool (WLST)

I created a script for a JMS Action, just by recording the configuration within the console and transformed it into the following script:

#
def createJmsNotificationAction(diagModuleName, actionName, destination, connectionFactory):
  policiesActionsPath='/WLDFSystemResources/'+diagModuleName+'/WLDFResource/'+diagModuleName+'/WatchNotification/'+diagModuleName
  jmsNotificationPath=policiesActionsPath+'/JMSNotifications/'
  print 'Check notification action '+jmsNotificationPath+actionName
  jmsNtfAction=getMBean(jmsNotificationPath+actionName)
  if jmsNtfAction==None:
    print 'Create new JMS NotificationAction '+actionName+' in '+diagModuleName
    edit()
    startEdit()
    cd(policiesActionsPath)
    jmsNtfAction=cmo.createJMSNotification(actionName)
    jmsNtfAction.setEnabled(true)
    jmsNtfAction.setTimeout(0)
    jmsNtfAction.setDestinationJNDIName(destination)
    jmsNtfAction.setConnectionFactoryJNDIName(connectionFactory)
    # Activate changes
    save()
    activate(block='true')
    print 'JMS NotificationAction created successfully.'
  else:
    print 'JMS NotificationAction '+actionName+' in '+diagModuleName+' already exists!'
  return jmsNtfAction
  

For other types, just click on the record link in the console and perform the configuration. Then transform it in a function as above.

I think this function does not need much explanation. It can be called as follows, using the JNDI names of the destination and a connection factory:

createJmsNotificationAction(diagModuleName, 'JMSAction', 'com.tt.jms.WLDFNotificationQueue', 'weblogic.jms.ConnectionFactory')

Policies

A Policy identifies a situation to trap for monitoring or diagnostic purposes. It constitutes of an expression that identifies the situation and one of more actions to follow up on it when the expression evaluates to true. The default language for the expression is the WLDF Query Language, but it is deprecated and superceeded by the Java Expression Language (EL).

Another aspect of the policy is the alarm. When an event in Weblogic is fired, that correlates to the policy, you might not want to have the handlers executed every time it occurs. If, for instance, a JMS queue hits a high count, and you define a policy with an email-action, you might not want a email-message for every new message posted on the queue. Then not only queue is flooded but it will flood your inbox as well. In the next function the alarm is set as 'AutomaticReset', after 300 seconds. When fired, the policy is then disabled for the given amount of time, and then is automatically enabled again.

#
def createPolicy(diagModuleName, policyName, ruleType, ruleExpression, actions):  
  policiesActionsPath='/WLDFSystemResources/'+diagModuleName+'/WLDFResource/'+diagModuleName+'/WatchNotification/'+diagModuleName
  policiesPath=policiesActionsPath+'/Watches/'
  print 'Check Policy '+policiesPath +policyName
  policy=getMBean(policiesPath +policyName)
  if policy==None:
    print 'Create new Policy '+policyName+' in '+diagModuleName
    edit()
    startEdit()
    cd(policiesActionsPath)
    policy=cmo.createWatch(policyName)
    policy.setEnabled(true)
    policy.setExpressionLanguage('EL')
    policy.setRuleType(ruleType)
    policy.setRuleExpression(ruleExpression)
    policy.setAlarmType('AutomaticReset')
    policy.setAlarmResetPeriod(300000)
    cd(policiesPath +policyName)
    set('Notifications', actions)
    schedule=getMBean(policiesPath +policyName+'/Schedule/'+policyName)
    schedule.setMinute('*')
    schedule.setSecond('*')
    schedule.setSecond('*/15')
    # Activate changes
    save()
    activate(block='true')
    print 'Policy created successfully.'
  else:
    print 'Policy '+policyName+' in '+diagModuleName+' already exists!'
  return policy

A policy can drive multiple actions. Therefor they must also be provided as an jarray. For that the following lines are added to the main function:

    actionsList=[]
    actionsList.append('com.bea:Name=JMSAction,Type=weblogic.diagnostics.descriptor.WLDFJMSNotificationBean,Parent=[TTDomain]/WLDFSystemResources[TTDiagnostics],Path=WLDFResource[TTDiagnostics]/WatchNotification[TTDiagnostics]/JMSNotifications[JMSAction]')
    actions=jarray.array([ObjectName(action.strip()) for action in actionsList], ObjectName)    
    createPolicy(diagModuleName,'HiStuckThreads', 'Harvester', 'wls:ServerHighStuckThreads(\"30 seconds\",\"10 minutes\",5)', actions)

As you can see, the JMSAction created earlier is coded as an expression and added to the list. As mentioned earlier with the Harvested Instances, you could wrap this into a separate function to build up the expression based on properties. In the example above, the rule is defined as: 'wls:ServerHighStuckThreads(\"30 seconds\",\"10 minutes\",5)', and added as a hardcoded parameter to the call to the createPolicy() function.

Another example, is:

    ruleExpression='wls:ServerGenericMetricRule(\"com.bea:Name=MedRecGlobalDataSourceXA,ServerRuntime=TTServer,Type=JDBCDataSourceRuntime\",\"WaitingForConnectionHighCount\",\">\",0,\"30 seconds\",\"10 minutes\")'
    createPolicy(diagModuleName,'OverloadedDS', 'Harvester', ruleExpression, actions)

In this example the rule is quite long and would make the line to create the policy quite long. But again, this could be abstracted into a function that builds the expression.

Conclusion

I put the complete script on github. It is a starting point showing how to setup collectors, policies and actions using WLST. It could be extended with functions that create the different expressions based on properties. This would make the scripting more robust, because you would not need to formulate your expressions for every purpose when you want different values.

When I started with this script during the training, I imagined that  you could define a library for several types of collectors, actions and policies. You could drive those with a smart property or xml-configuration file that define all the policies that you want to add to the environment. You could even create different property files for different kinds of environments. You could have different weblogic domains for particular applications, but also for OSB, SOA, BI Publisher, etc. Based on the kind of environment you may want different sets of weblogic resources monitored.

If you make sure that all your functions are re-entrant, you could easily add them to the scripts run from your Docker files, to build up and start your Kubernetes Weblogic Operator domain. See my earlier posts about my cheat sheet, part 1 and part 2.