Sunday, 2 February 2020

Virtualbox 6.1.2 and Vagrant 2.2.7 - the working combination

Today I found out that Vagrant 2.2.7 has been released. A few weeks ago, the Oracle VirtualBox celebrated the release of 6.1.2. The thing with VirtualBox 6.1.2 was that it wasn't compatile with Vagrant 2.2.6, since that version of Vagrant lacked the support of the Virtualbox 6.1 base-release. It was solvable, as described by Tim Hall, with a solution of Simon Coter. Happily, as expected, Vagrant 2.2.7 supports 6.1.x now. So, I was eager to try that out. And it works indeed.

However, the first time I 'upped' a Vagrant project, I hit the error:
VBoxManage.exe: error: Unknown option: --clipboard

Sadly this was due to the following lines in my Vagrantfile:
    # Set clipboard and drag&drop bidirectional
    #vb.customize ["modifyvm", :id, "--clipboard", "bidirectional"]
    #vb.customize ["modifyvm", :id, "--draganddrop", "bidirectional"]
I did not try the --draganddrop option. But assumed that it would fail too. Commenting those out (as in the example) got my Vagrantfile ok again.
I use this to have bi-directional clipboard and draganddrop, which is off by default. So, I have to figure why this is.
After startup of the new VM, I tested the clipboard functionality and although these lines are commented out, it worked as such. Apparently I don't need those lines anymore.

Since it did not let me go, I tried:
C:\Program Files\Oracle\VirtualBox>vboxmanage modifyvm

VBoxManage modifyvm         
                            [--name ]
                            [--groups , ...]
                            [--description ]
                            [--clipboard-mode disabled|hosttoguest|guesttohost|
                            [--draganddrop disabled|hosttoguest|guesttohost|

Apparently the the option changed to --clipboard-mode.

Friday, 24 January 2020

Configure Weblogic Policies and Actions using WLST

Fairly regularly I give a training on Weblogic Tuning and Troubleshooting, where I talk about JVMs, Garbage Collections, and some subsystems of Weblogic, JMS and JDBC for instance, and how to tune and troubleshoot them.

One of the larger parts of the training is the Weblogic Diagnostic Framework. I find it quite interesting, but also pretty complex. And maybe therefor hardly used in the every day Weblogic administration. And that might be a pity, because it can be quite powerfull. You can find the use of it in Fusion Middleware, with preconfigured policies and actions. But I guess that other tooling on Weblogic diagnostics and monitoring, like WLSDM also rely on it (although I don't know for sure).

Configuring WLDF might be quite hard, and during the last run of the training, I figured that it might help to turn the solution of the workshop into a script. At least to show what you're doing when executing the labs. But, certainly also to show how you can put your configurations into a script that you can extend and reuse over different environments.

This week I got a question on the Oracle community,To be notify of les warning Logs, that made me remember this script. Maybe it's not exactly the answer, but I think at least it can be a starting point. And I realized that I did not write about it yet.

 11g vs 12c

I stumbeld upon a nice 11g blog about this subject. In 12c Oracle renamed this part of the WLDF: where in 11g it was called "Watches and Notifications" it is now called "Policies and Actions". When working with the console, you'll find that the console follows the new naming. But in WLST the APIs still have the old 11g naming. So keep in mind that Policies are Watches and Actions are Notifications.

Documentation about Configuring Policies and Actions can be found here.

I'm not going to explain all the concepts of subject, but going through my base script step by step, and then conclude with some remarks and ideas.

Diagnostic Module

Just like JMS resources, the Diagnostic Framework combines the resources into WLDFSystemResource Modules. A Diagnostic Module is in essence an administration unit to combine the resources.  A diagnostic module can be created by the following WLST function:
def createDiagnosticModule(diagModuleName, targetServerName):
  if module==None:
    print 'Create new Diagnostic Module'+diagModuleName
    module = cmo.createWLDFSystemResource(diagModuleName)
    # Activate changes
    print 'Diagnostic Module created successfully.'
    print 'Diagnostic Module'+diagModuleName+' already exists!'
  return module

The script is created in a way that it first checks if the Diagnostic Module already exists. You'll see that all the functions in this article work like this. This helps also in the use of it in Weblogic under Kubernetes environments. Diagnostic modules are registered under '/WLDFSystemResources'. And it is created with the createWLDFSystemResource() method. Also like a JMSModules you need to target them. This function does this based on the targetServerName and uses the getMServer() function to get the MBean to target:
def getMServer(serverName):
  return server

Many Weblogic resources need to be targetted. I notice that I do this in many different ways in different scripts all over this blog and in my work. Maybe I need to write a more generic, smarter way of doing this. In this case I simply just target to a single server, but it could be list of servers and/or clusters.
The function is called from a main function as follows:
import os,sys, traceback
admServerUrl = 't3://'+adminHost+':'+adminPort
def main():
    print 'Connect to '+admServerUrl
    createDiagnosticModule(diagModuleName, ttServerName)


Collectors are also called Harvesters. They monitor MBeans Attributes and regularly store the data in the Harvested Data Archive of the targetted Managed Server. You can find it under Diagnostics->Log Files in the Weblogic Console. The Weblogic console also includes a Monitoring Dashboard. That can be get to via the console Home page using the 'Monitoring Dashboard' link.

Without collectors, you can only view  MBean Attributes in the Graphs from the moment you start a graph. It will collect the MBeans from that moment onwards, until you pause/stop the collection.
However, using a collector you can view the Attribute values from the history back.

A Collector is created within a diagnostic module. You need to  define a metricType: the MBean Type, for instance 'JDBCDataSourceRuntimeMBean'.  Then a namespace, in this case the ServerRuntime. You can specify a set of instances of the particular MBean Type, or provide None to watch all the instances of the particular type. And from the instances you specify a comma separated list of attributes you want to harvest.

This leads to the following function:

def createCollector(diagModuleName, metricType, namespace, harvestedInstances,attributesCsv):
  print 'Check Collector '+harvestedTypesPath+metricType
  if collector==None:
    print 'Create new Collector for '+metricType+' in '+diagModuleName
    attributeArray=jarray.array([String(x.strip()) for x in attributesCsv.split(',')], String)
    # Activate changes
    print 'Collector created successfully.'
    print 'Collector '+metricType+' in '+diagModuleName+' already exists!'
  return collector

This creates the Collecter using createHarvestedType() for the MBean Type (metricType). The list of Attributes is provided as comma separated string. But the setter on the collector (setHarvestedAttributes(attributeArray)) expects a jarray, so the csv-list needs to be translated. It is created by a (to me a bit peculiar when I first saw it) python construct:
    attributeArray=jarray.array([String(x.strip()) for x in attributesCsv.split(',')], String)

It splits the csv string with the comma as a separator and then loops over the resulting values. For each value it constructs a String, that is trimmed. The resulting String values are fed to a String based jarray.array factory.

The following line added to the main function will call the function, when you want to watch all inststances:
createCollector(diagModuleName, '','ServerRuntime', None, 'ActiveConnectionsCurrentCount,CurrCapacity,LeakedConnectionCount')

In the case you do want to select a specific set of instances, you need to do that as follows:
    harvestedInstances=jarray.array([String(x.strip()) for x in harvestedInstancesList], String)    
    createCollector(diagModuleName, '','ServerRuntime', harvestedInstances,'OpenSessionsCurrentCount') 

The thing in this case is that the instances self are described in an expression that uses commas. You could construct these expressions using properties of course. And then use the construct above to add those to a jarray.


When you want to have WLDF to take action upon a certain condition, you need to create an Action for it. A simple one is to create a message on a JMS queue. But according to the documentation you could have the following types:
  • Java Management Extensions (JMX)
  • Java Message Service (JMS)
  • Simple Network Management Protocol (SNMP)
  • Simple Mail Transfer Protocol (SMTP)
  • Diagnostic image capture
  • Elasticity framework (scaling your dynamic cluster)
  • REST
  • WebLogic logging system
  • WebLogic Scripting Tool (WLST)
I created a script for a JMS Action, just by recording the configuration within the console and transformed it into the following script:

def createJmsNotificationAction(diagModuleName, actionName, destination, connectionFactory):
  print 'Check notification action '+jmsNotificationPath+actionName
  if jmsNtfAction==None:
    print 'Create new JMS NotificationAction '+actionName+' in '+diagModuleName
    # Activate changes
    print 'JMS NotificationAction created successfully.'
    print 'JMS NotificationAction '+actionName+' in '+diagModuleName+' already exists!'
  return jmsNtfAction

For other types, just click on the record link in the console and perform the configuration. Then transform it in a function as above.

I think this function does not need much explanation. It can be called as follows, using the JNDI names of the destination and a connection factory:
createJmsNotificationAction(diagModuleName, 'JMSAction', '', 'weblogic.jms.ConnectionFactory')


A Policy identifies a situation to trap for monitoring or diagnostic purposes. It constitutes of an expression that identifies the situation and one of more actions to follow up on it when the expression evaluates to true. The default language for the expression is the WLDF Query Language, but it is deprecated and superceeded by the Java Expression Language (EL).

Another aspect of the policy is the alarm. When an event in Weblogic is fired, that correlates to the policy, you might not want to have the handlers executed every time it occurs. If, for instance, a JMS queue hits a high count, and you define a policy with an email-action, you might not want a email-message for every new message posted on the queue. Then not only queue is flooded but it will flood your inbox as well. In the next function the alarm is set as 'AutomaticReset', after 300 seconds. When fired, the policy is then disabled for the given amount of time, and then is automatically enabled again.
def createPolicy(diagModuleName, policyName, ruleType, ruleExpression, actions):  
  print 'Check Policy '+policiesPath +policyName
  policy=getMBean(policiesPath +policyName)
  if policy==None:
    print 'Create new Policy '+policyName+' in '+diagModuleName
    cd(policiesPath +policyName)
    set('Notifications', actions)
    schedule=getMBean(policiesPath +policyName+'/Schedule/'+policyName)
    # Activate changes
    print 'Policy created successfully.'
    print 'Policy '+policyName+' in '+diagModuleName+' already exists!'
  return policy

A policy can drive multiple actions. Therefor they must also be provided as an jarray. For that the following lines are added to the main function:

    actions=jarray.array([ObjectName(action.strip()) for action in actionsList], ObjectName)    
    createPolicy(diagModuleName,'HiStuckThreads', 'Harvester', 'wls:ServerHighStuckThreads(\"30 seconds\",\"10 minutes\",5)', actions)

As you can see, the JMSAction created earlier is coded as an expression and added to the list. As mentioned earlier with the Harvested Instances, you could wrap this into a separate function to build up the expression based on properties. In the example above, the rule is defined as: 'wls:ServerHighStuckThreads(\"30 seconds\",\"10 minutes\",5)', and added as a hardcoded parameter to the call to the createPolicy() function.

Another example, is:
    ruleExpression='wls:ServerGenericMetricRule(\"com.bea:Name=MedRecGlobalDataSourceXA,ServerRuntime=TTServer,Type=JDBCDataSourceRuntime\",\"WaitingForConnectionHighCount\",\">\",0,\"30 seconds\",\"10 minutes\")'
    createPolicy(diagModuleName,'OverloadedDS', 'Harvester', ruleExpression, actions)
In this example the rule is quite long and would make the line to create the policy quite long. But again, this could be abstracted into a function that builds the expression.


I put the complete script on github. It is a starting point showing how to setup collectors, policies and actions using WLST. It could be extended with functions that create the different expressions based on properties. This would make the scripting more robust, because you would not need to formulate your expressions for every purpose when you want different values.

When I started with this script during the training, I imagined that  you could define a library for several types of collectors, actions and policies. You could drive those with a smart property or xml-configuration file that define all the policies that you want to add to the environment. You could even create different property files for different kinds of environments. You could have different weblogic domains for particular applications, but also for OSB, SOA, BI Publisher, etc. Based on the kind of environment you may want different sets of weblogic resources monitored.

If you make sure that all your functions are re-entrant, you could easily add them to the scripts run from your Docker files, to build up and start your Kubernetes Weblogic Operator domain. See my earlier posts about my cheat sheet, part 1 and part 2.

My Weblogic on Kubernetes Cheatsheet, part 2.

In my previous blog-post I published the first part of my Kubernetes cheatsheet, following the Weblogic Operator tutorial. In this part 2, I'll publish the scripts I created for the next few chapters in the tutorial.

Install Traefik Software Loadbalancer

The fourth part of the tutorial is about installing the Treafic Software Loadbalancer service. It is described in this part: 3. Install Traefik Software Loadbalancer. And also uses Helm to install the service.

Install Treafik is just a quest of (nice Dutch idiom 😉) installing the proper Helm chart.
SCRIPTPATH=$(dirname $0)
echo Install traefik
helm install stable/traefik \
--name traefik-operator \
--namespace traefik \
--values kubernetes/samples/charts/traefik/values.yaml  \
--set "kubernetes.namespaces={traefik}" \
--set "serviceType=LoadBalancer"

A simple script to check the Loadbalancer Service:
SCRIPTPATH=$(dirname $0)
echo Get service traefik
kubectl get service -n traefik

I did not change the name or the namespace of the Treafik loadbalancer. But I could do easily by moving the name and namespace to the But I haven't had the need, but I can imagine that it could raise.

The scriptlet above will show all the treafik information. But to show only the public IP that you need to get to your application, you can use:
SCRIPTPATH=$(dirname $0)
echo Get traefik public IP
kubectl describe svc traefik-operator --namespace traefik | grep Ingress | awk '{print $3}'

Again, I could move the serivce name and namespace to
When the Weblogic domain is created, there will be some additional commands and therefor scriptlets, that update the Treafik configuration.

Deploy WebLogic Domain

Now it gets really interesting: we're getting to create the actual Weblogic pods that will run the Weblogic Domain. This part is described in the fifth part: Deploy Weblogic Domain.

This script will do a sequence of things. You could argue that several of the scripts described earlier could have been combined, like this.
Anyway,  this one does the following:
  • Create a Weblogic Domain Namespace
  • Create and label a Kubernetes secret within that namespace for the Admin boot credentials.
  • Create a secriet for the Docker registry on the Oracle infrastructure (Oracle Container Image Repository).
SCRIPTPATH=$(dirname $0)
function prop {
    grep "${1}" $SCRIPTPATH/|cut -d'=' -f2
echo Create weblogic domain namespace $WLS_DMN_NS
kubectl create namespace $WLS_DMN_NS
WLS_USER=$(prop 'weblogic.user')
WLS_PWD=$(prop 'weblogic.password')
echo Create a Kubernetes secret $WLS_DMN_CRED in namespace $WLS_DMN_NS containing the Administration Server boot credentials for user $WLS_USER
kubectl -n $WLS_DMN_NS create secret generic $WLS_DMN_CRED \
  --from-literal=username=$WLS_USER \
echo Label the $WLS_DMN_CRED in namespace $WLS_DMN_NS secret with domainUID $WLS_DMN_NAME
kubectl label secret $WLS_DMN_CRED \
  -n $WLS_DMN_NS \
  weblogic.domainUID=$WLS_DMN_NAME \
  weblogic.domainName=$WLS_DNM_NAME \
echo Create secret for oci image repository $OCIR_CRED
OCIR_USER=$(prop 'ocir.user')
OCIR_PWD=$(prop 'ocir.password')
OCIR_EMAIL=$(prop '')
OCI_TEN=$(prop 'oci.tenancy')
OCI_REG=$(prop 'oci.region')
kubectl create secret docker-registry $OCIR_CRED \
  -n $K8S_NS \
  --docker-server=${OCI_REG} \
  --docker-username="${OCI_TEN}/${OCIR_USER}" \
  --docker-password="${OCIR_PWD}" \

The scripts starts with a function to read credential properties. It relies on the file:
ocir.password=something difficult

I like the way this works with the prop function, because you can abstract these properties without the need to have them as a environment variable. Especially credentials you would not have in an en environment variable. Actually, passwords should be stored even more secure than in a plain property file, of course. But, I think I would move the OCID's used in seting up the OKE cluster to the in a next iteration.

When the Weblogic domain namespace is created, you can add it to the Treafik configuration:

SCRIPTPATH=$(dirname $0)
echo Upgrade traefik with namespace $WLS_DMN_NS
helm upgrade \
  --reuse-values \
  --set "kubernetes.namespaces={traefik,$WLS_DMN_NS}" \
  --wait \
  traefik-operator \


The next step in the tutorial is to create the Weblogic Domain by starting the pods. To do so, you need to create a domain.yaml, as in this example. You'll need to update the properties according to your environment, specifying things like the project domain container image, names, credential sets, etc.
For this script, one property is of special interest:
  # - "NEVER" will not start any server in the domain
  # - "ADMIN_ONLY" will start up only the administration server (no managed servers will be started)
  # - "IF_NEEDED" will start all non-clustered servers, including the administration server and clustered servers up to the replica count
  serverStartPolicy: "IF_NEEDED"

The property serverStartPolicy defines if the Weblogic pods should start or stop. To have them started, you would set the property to "IF_NEEDED" and to stop to "NEVER". I found that I had to change this property several times and then run kubectl apply on the yaml. Soon I thought that this should be more convenient. So I created a copy of the domain.yaml, called domain.yaml.tpl. In that file I changed the property to:
  # - "NEVER" will not start any server in the domain
  # - "ADMIN_ONLY" will start up only the administration server (no managed servers will be started)
  # - "IF_NEEDED" will start all non-clustered servers, including the administration server and clustered servers up to the replica count
  serverStartPolicy: "$SVR_STRT_POLICY"
  #serverStartPolicy: "NEVER"

Now using a script I can create a copy of the domain.yaml.tpl to domain.yaml while replacing the environment variable to the desired value. A few years ago I found the Linux command envsubst that can read in a file from stndin to replace all the environment variables with there corresponding value and output the result to stdout.

So to start the domain, I would use:
SCRIPTPATH=$(dirname $0)
echo Start Domain $K8S_DMN_NAME
kubectl apply -f $WLS_DMN_YAML

It will set the SVR_STRT_POLICY to "IF_NEEDED", and then streams the domain.yaml.tpl to the domain.yaml through envsubst. Then perform kubectl apply on the resulting yaml.
Obviously, you can use this also to abstract properties as cluster replicas and java options out of the yaml file and have them in a property file or

And of course, the stopping the domain will work accordingly, but then using the value "NEVER".
SCRIPTPATH=$(dirname $0)
echo Stop Domain $K8S_DMN_NAME
kubectl apply -f $WLS_DMN_YAML

When your domain is started, you probably want to be able to reach it from out of the internet. To do so you need to create an Ingress rule. This is done as follows:
SCRIPTPATH=$(dirname $0)
echo Upgrade traefik with namespace $WLS_DMN_NS
cat << EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
  name: traefik-pathrouting-1
  namespace: $WLS_DMN_NS
  annotations: traefik
  - host:
      - path: /
          serviceName: medrec-domain-cluster-medrec-cluster
          servicePort: 8001
      - path: /console
          serviceName: medrec-domain-adminserver
          servicePort: 7001          

The thing with this script is that it does not leverage the script, except for the namespace. There are still a few hardcoded names for the service and the Ingress names. But, just like the namespace, it is easy to move those to the script.

When started your domain, you would check the pods. This can be done with this script:
SCRIPTPATH=$(dirname $0)
echo Get K8s pods for $WLS_DMN_NS
kubectl -n $WLS_DMN_NS get pods -o wide

Initially, you would see 3 pods, one for the Admin server and two Managed servers. But after changing the domain.yaml.tpl following the sixth part: Scaling WebLogic Cluster, you would see more (or less) pods. Mind that if you use my start and stop scripts, you should not change the domain.yaml file itself but the domain.yaml.tpl file.


These scripts helped me a lot in documenting the setup. You can just follow the tutorial, which leaves you with a working environment. But it's just a recipe that you need to follow over and over again. Also, I found at a few points that it is tricky to fill in the correct value.

Lately, I started to look into Terraform for AWS and OCI. I figure that the tutorial that I followed with these scripts can be broken down into a few parts that are infact setting up OCI resources (Instances, Services like Treafik, etc.) and provisioning OKE. So it would be interesting to see where we can use Terraform to setup the OCI resources and services, leaving only the scripts to setup OKE itself.

Wednesday, 15 January 2020

Javascript in ANT

Earlier I wrote about an ANT script to scan JCA adapters files in your projects home, subversion working copy or github local repo.

In my current project we use sensors to kick-of message-archiving processes, without cluttering the BPEL process. I'm not sure if I would do that like that if I would do on a new project, but technically the idea is interesting. Unfortunately, we did not build a registry what BPEL processes make use of it and how. So I tought of how I could easily find out a way to scan that, and found that based on the script to scan JCA files, I could easily scan all the BPEL sensor files. If you have found the project folders, like I did in the JCA scan script, you can search for the *_sensor.xml files.

So in a few hours I had a basic sript. Now, in a second iteration, I would like to know what sensorActions the sensors trigger. For that I need to interpret the accompanying *_sensorAction.xml file. There for, based on the found sensor filename I need to determine the name of the sensor action file.

The first step to that is to figure out how to do a substring in ANT. With a quick google on "ant property substring", I found a nice stackoverflow thread, with a nice example of an ANT script defininition based on Javascript:
  <scriptdef name="substring" language="javascript">
    <attribute name="text"/>
    <attribute name="start"/>
    <attribute name="end"/>
    <attribute name="property"/>
       var text = attributes.get("text");
       var start = attributes.get("start");
       var end = attributes.get("end") || text.length();
       project.setProperty(attributes.get("property"), text.substring(start, end));

And that can be called like:
    <substring text="${}" start="0" end="20"   property=""/>
    <echo message="Sensor Action file: ${sensorAction.file.name1}"></echo>

The javascript substring() function is zero-based, so the first character is indexed by 0.
Not every sensor file name has the same length, the file is called after the BPEL file that it is tight too. And so to get the base name, the part without the "_sensor.xml" postfix, we need to determine the length of the filename. A script that determines that can easily be extracted from the script above:
  <scriptdef name="getlength" language="javascript">
    <attribute name="text"/>
    <attribute name="property"/>
       var text = attributes.get("text");
       var length = text.length();
       project.setProperty(attributes.get("property"), length);

Perfect! Using this I could create the logic in ANT to determine the sensorAction file name. However, I thought that it would be easier to determine the filename in Javascript all the way. Using the strength of the proper language at hand:
  <!-- Script to get the sensorAction filename based on the sensor filename. 
  1. Cut the extension "_sensor.xml" from the filename.
  2. Add "_sensorAction.xml" to the base filename.
  <scriptdef name="getsensoractionfilename" language="javascript">
    <attribute name="sensorfilename"/>
    <attribute name="property"/>
       var sensorFilename = attributes.get("sensorfilename");
       var sensorFilenameLength = sensorFilename.length();
       var postfixLength = "_sensor.xml".length();
       var sensorFilenameBaseLength=sensorFilenameLength-postfixLength;
       var sensorActionFilename=sensorFilename.substring(0, sensorFilenameBaseLength)+"_sensorAction.xml";
       project.setProperty(attributes.get("property"), sensorActionFilename);
And then I can get the sensorAction filename as follows:
    <getsensoractionfilename sensorfilename="${}" property=""/>
    <echo message="Sensor Action file: ${}"></echo>

Superb! I found ANT a powerfull language/tool already. But with a few simple JavaScript snippets you can extend it easily.
Notice by the way also the use of xslt in the Scan JCA adapters files article. You can read xml files as properties, but to do that conveniently you need to transform a file like the sensors.xml in a way that you can easily reference the properties following the element-hierarchy. This is also explained in the Scan JCA adapters files article.
I'll go further with my sensors scan script. Maybe I'll write about it when done.

Friday, 10 January 2020

My Weblogic on Kubernetes Cheatsheet, part 1.

Last week I had the honour to present at the UKOUG TechFest 19, together with my 'partner in crime', I think I can say now: Simon Haslam. We combined our sessions into a part 1 and a part 2.

For me this presentation is the result of having done a workshop at the PaaSForum in Mallorca, and then to work that around into a setup where I was able to run the MedRec Weblogic sample application against a managed Database under Kubernetes.

Kubernetes  Weblogic Operator Tutorial

I already wrote a blog about my workshop at the PaaSForum this year, but Marc Lameriks from Amis, did a walkthrough on the workshop. It basically comes down to this tutorial, which you can do as a self-paced tutorial. Or checkout a Meetup in your neighbourhoud. If you're in the Netherlands, we'll be happy to organized one, or if you like I would come over to your place and we could set something up. See also the links at the end of part 2 of our presentations for more info on the tutorial for instance.

I did the tutorial more or less three times now, once at the PaaSForum, then I re-did it, but deliberately changed namespace-names, domain-name, etc. Just to see where the dependencies are, and actually to see where the pitfalls are. It's based on my method to get to know an unfamiliar city: deliberately get lost in it. Two years ago we moved to another part of Amersfoort. To get to know my new neighbourhood, I often took another way home then I when I left. And this is basically what I did with the tutorial too.

The last time I did it was to try to run a more real-life application with an actual database. And therefor I setup a new OKE cluster, this time in a compartment of our own company cloud subscription. Interesting in that is that you work with a normal customer-alike subscription within a compartment. Another form of a deliberate D-Tour. But also to setup a database and see that configuration overrides to change your runtime datasource-connection pool actually works.


When doing the tutorial, you'll find that besides all the configurations on the Cloud Pages, to setup your OKE Cluster, configure Oracle Pipelines, you'll find that you'll have to enter a lot of commandline-commands. Most of them are kubectl commands, some helm, and a bit of OCI commandline interface. Doing it the first time I soon got lost in the meaning of them and what I was doing with it. Also, most kubectl commands work with namespaces where your Weblogic has another namespace then the Weblogic Operator. And as is my habit nowadays, I soon put the commands in smart but simple scripts. And those I want to share with you. Maybe not all, but at least enough so you'll get the idea.

I also found the official kubectl cheat sheet and this one on github. But those are more explanations of the particular commands.

I found it helpfull to set up this Cheatsheet following the tutorial. I guess this helps in relating the commands in what they're meant for.

Shell vs. Alias

At the UKOUG TechFest, someone pointed that you could use aliases too. Of course. You  could do an alias like
alias k=kubectl

However, you'll still need to extend every command with the proper namespace, pod naming, etc.
Therefor, I used the approach of creating a script that I can include in every script, and a property file to store the credentials to put in secrets. Then call (source) the script in every other script.

Setup Oracle Kubernetes Engine instance on Oracle Cloud Infrastructure

These scripts refer to the first part of the tutorial: 0. Setup Oracle Kubernetes Engine instance on Oracle Cloud Infrastructure.

It all starts with my Here you'll find all the particular necessary variables that are used in most other scripts. I think in a next iteration I would move the OIC_USER, OCID_TENANCY and OCID_CLUSTERID to my credential properties file. But I introduced that later on, during my experiments.

echo Set OKE Environment
export OCID_USER="ocid1.user.oc1..{here goes that long string of characters}" 
export OCID_TENANCY="ocid1.tenancy.oc1..{here goes that other long string of characters}"
export OCID_CLUSTERID="{yet another long string of characters}"
export REGION="eu-frankfurt-1" # or your other region
export CLR_ADM_BND=makker-cluster-admin-binding
export K8S_NS="medrec-weblogic-operator-ns"
export K8S_SA="medrec-weblogic-operator-sa"
export HELM_CHARTS_HOME=/u01/content/weblogic-kubernetes-operator
export WL_OPERATOR_NAME="medrec-weblogic-operator"
export WLS_DMN_NS=medrec-domain-ns
export WLS_USER=weblogic
export WLS_DMN_NAME=medrec-domain
export WLS_DMN_CRED=medrec-domain-weblogic-credentials
export OCIR_CRED=ocirsecret
export WLS_DMN_YAML=/u01/content/github/weblogic-operator-medrec-admin/setup/medrec-domain/domain.yaml
export WLS_DMN_UID=medrec-domain
export MR_DB_CRED=mrdbsecret
export ADM_POD=medrec-domain-adminserver
export MR1_POD=medrec-domain-medrec-server1
export MR2_POD=medrec-domain-medrec-server2
export MR3_POD=medrec-domain-medrec-server3
export DMN_HOME=/u01/oracle/user_projects/domains/medrec-domain
export LCL_LOGS_HOME=/u01/content/logs
export ADM_SVR=AdminServer
export MR_SVR1=medrec-server1
export MR_SVR2=medrec-server2
export MR_SVR3=medrec-server3

This stores the most important credentials. That allows me to abstract those from the scripts. However, as mentioned, I should move the OIC_USER, OCID_TENANCY and OCID_CLUSTERID variables to this file.

After having setup the OKE Cluster in OCI, and configured your OCI CLI, the first actuall command you issue is to create a Kube Config file, using the OCI CLI. This one is executed only once, normally for every setup. So this script is merely to document my commands:

SCRIPTPATH=$(dirname $0)
echo Create Kubeconfig -> Copy command from Access Kube Config from cluster
mkdir -p $HOME/.kube
oci ce cluster create-kubeconfig --cluster-id $OCID_CLUSTERID --file $HOME/.kube/config --region $REGION --token-version 2.0.0 

The SCRIPTPATH variable declaration is a trick to be able to refer to other scripts relatively from that variable. Then as you will see in all my subsequent scripts, I source here the script. Doing so I can refer to the particular variables in the oci command. There for, as described in the tutorial, you should note down your OCID_CLUSTERID and update that into the file, as well as the REGION variable.

Note by the way, that recently Oracle Kubernetes Engine upgraded to only support the Kubeconfig token version 2.0.0. See also this document.

This one is a bit dumb, and could as easily be created by an alias:
SCRIPTPATH=$(dirname $0)
echo Get K8s nodes
kubectl get node

Even the call to the doesn't add anything, really but it is a base for the other scripts and when needing to add namespaces it makes sense.

The last part of setting up the OKE cluster is to create a role binding. This is done with:
SCRIPTPATH=$(dirname $0)
echo Create cluster role binding
echo kubectl create clusterrolebinding $CLR_ADM_BND --clusterrole=cluster-admin --user=$OCID_USER
kubectl create clusterrolebinding $CLR_ADM_BND --clusterrole=cluster-admin --user=$OCID_USER

Install WebLogic Operator

The second part of the tutorial is about seting up your project environment with Github and have Oracle Pipelines build your projects image. This is not particularly related to K8S, so no relevant scripts there. 
The next part of the tutorial is about installing the operator: 2. Install WebLogic Operator.

Installling Weblogic Operator is done using Helm. As far as I have understood is Helm a sort of package manager for Kubernetes. Funny thing in naming is that where Kubernetes is Greek for the steering officer on a ship, helm is the steering device of a ship. It makes use of Tiller, the server side part of Helm. A tiller is the "steering stick" or lever that manages the Helm device. (To be honest, to me it feels a bit the otherway around, I guess I would have named the server side Helm and the client Tiller).

As a first step is to create a Helm Cluster admin role binding, a kubernetes namespace for the Weblogic Operator and a serviceaccount within this namespace. To do so the script does the following:
SCRIPTPATH=$(dirname $0)
echo Create helm-user-cluster-admin-role
cat << EOF | kubectl apply -f -
kind: ClusterRoleBinding
  name: helm-user-cluster-admin-role
  kind: ClusterRole
  name: cluster-admin
- kind: ServiceAccount
  name: default
  namespace: kube-system
echo Create namespace $K8S_NS
kubectl create namespace $K8S_NS
echo kubectl create serviceaccount -n $K8S_NS $K8S_SA
kubectl create serviceaccount -n $K8S_NS $K8S_SA

Installing the Weblogic operator is done with this script. Notice that you need to execute the helm command within the folder in which you checked out the Weblogic Operator github repository.
SCRIPTPATH=$(dirname $0)
echo Install Weblogic Operator $WL_OPERATOR_NAME
helm install kubernetes/charts/weblogic-operator \
  --name $WL_OPERATOR_NAME \
  --namespace $K8S_NS \
  --set image=oracle/weblogic-kubernetes-operator:2.3.0 \
  --set serviceAccount=$K8S_SA \
  --set "domainNamespaces={}"

The script will cd to the Weblogic Operator local repository and executes helm. In the begin of the script the current folder is saved as SCRIPTPATH. After running the helm command, it does a cd back to it.

During my investigations the Weblogic Operator was upraded. If you take a closer look to the command in the tutorial, you'll notice that the image that is used is oracle/weblogic-kubernetes-operator:2.0, but I used oracle/weblogic-kubernetes-operator:2.3.0 in the script above.
I found it usefull to be able to delete the operator to be able to re-install it again. To delete the weblogic operator run the script:

SCRIPTPATH=$(dirname $0)
echo Delete Weblogic Operator $WL_OPERATOR_NAME
helm del --purge $WL_OPERATOR_NAME 

Again in this script the helm command is surrounded by a cd to the helm charts folder of the Weblogic Operator local github repository, and back again to the current folder.

After having installed the Weblogic Operator, you can list the pods of the kubernetes namespace it runs in, using this script:
SCRIPTPATH=$(dirname $0)
echo Get K8s pods for $K8S_NS
kubectl get po -n $K8S_NS

You can check the Weblogic Operator installion by performing a helm list of the Weblogic Operator charts. I wrapped that ino this script:
SCRIPTPATH=$(dirname $0)
echo List Weblogic Operator $WL_OPERATOR_NAME


If you would have followed the workshop, and maybe used my scripts, uptil now you have installed the Weblogic Operator. Let's not make this article too long and call this Part 1. And quickly move on to part 2, to install/configure and monitor the rest of the setup. Maybe at the end I move these contents to an easy to navigate set of articles.

Monday, 2 December 2019

Create a Vagrant box with Oracle Linux 7 Update 7 Server with GUI

Yesterday and today I have been attending the UKOUG TechFest '19 in Brighton. And it got me eager to try things out. For instance with new Oracle DB 19c features. And therefor I should update my vagrant boxes to be able to install one. But I realized my basebox is still on Oracle Linux 7U5, and so I wanted to have a neatly fresh, latest OL 7U7 box.

Use Oracle's base box

Now, last year I wrote about how to create your own Vagrant Base Box: Oracle Linux 7 Update 5 is out: time to create a new Vagrant Base Box. So I could create my own, but already quite some time ago I found out that Oracle supplies those base boxes.

They're made available at, and there are boxes for OL6, OL7 and even OL8. I want to use OL 7U7, and thus I got started with that one. It's neatly described at the mentioned link and it all comes down to:

$ vagrant box add --name <name> <url>
$ vagrant init <name>
$ vagrant up
$ vagrant ssh

And in my case:

$ vagrant box add --name ol77
$ vagrant init ol77
$ vagrant up
$ vagrant ssh

Before you do that vagrant up, you might want to edit your vagrant file, to add a name for your VM:
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at

  # Every Vagrant development environment requires a box. You can search for
  # boxes at = BOX_NAME


  # Provider-specific configuration so you can fine-tune various
  # backing providers for Vagrant. These expose provider-specific options.
  # Example for VirtualBox:
  config.vm.provider "virtualbox" do |vb| = VM_NAME
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #   # Customize the amount of memory on the VM:
  #   vb.memory = "1024"

Otherwise your VM name in Virtual box would be someting like ol7_default_1235897983, something cryptic with a random number.

If you do a vagrant up now it will boot up nicely.

VirtualBox Guest Additions

The VirtualBox GuestAdditions are from version 6.12, while my VirtualBox installation already has 6.14. I found it handy to have a plugin that auto-updates it. My co-Oracle-ACE Maarten Smeets wrote about that earlier. It comes down to executing the following in a command line:
vagrant plugin install vagrant-vbguest

If you do a vagrant up now, it will update the guest additions. However, to be able to do so, it needs to install all kinds of kernel packages to compile the drivers. So, be aware that this might take some time, and you'll need internet connection.

Server with GUI

The downloaded box is a Linux Server install, without a UI. This probably is fine for most of the installations you do. But I like to be able to log on to the desktop from time to time, and I want to be able to connect to that using MobaXterm, and be able to run a UI based installer or application. A bit of X-support is handy. How to do that, I found at this link.

GUI support is one of the group packages that are supported by Oracle Linux 7, and this works exactly the same as RHEL7 (wonder why that is?).

To list the available packages groups are supported, you can do:

[vagrant@localhost ~]$ sudo  yum group list
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
Available Environment Groups:
   Minimal Install
   Infrastructure Server
   File and Print Server
   Cinnamon Desktop
   MATE Desktop
   Basic Web Server
   Virtualization Host
   Server with GUI
Available Groups:
   Backup Client
   Compatibility Libraries
   Console internet tools
   Development tools
   E-mail server
   Educational Software
   Electronic Lab
   Fedora Packager
   General Purpose Desktop
   Graphical Administration Tools
   Graphics Creation Tools
   Hardware monitoring utilities
   Input Methods
   Internet Applications
   KDE Desktop
   Legacy UNIX Compatibility
   Network Infrastructure Server
   Networking Tools
   Office Suite and Productivity
   Performance Tools
   Scientific support
   Security Tools
   Smart card support
   System Management
   System administration tools
   Technical Writing
   TurboGears application framework
   Web Server
   Web Servlet Engine

(After having executed vagrant ssh.)
You'll find 'Server with GUI' as one of the options. This will install all the necessary packages to run Gnome. But, if you want to have KDE there's also package group for that.

To install it you would run:
[vagrant@localhost ~]$ sudo yum groupinstall 'Server with GUI'
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
Resolving Dependencies
--> Running transaction check
---> Package ModemManager.x86_64 0:1.6.10-3.el7_6 will be installed
--> Processing Dependency: ModemManager-glib(x86-64) = 1.6.10-3.el7_6 for package: ModemManager-1.6.10-3.el7_6.x86_64
--> Processing Dependency: libmbim-utils for package: ModemManager-1.6.10-3.el7_6.x86_64
--> Processing Dependency: libqmi-utils for package: ModemManager-1.6.10-3.el7_6.x86_64
--> Processing Dependency: for package: ModemManager-1.6.10-3.el7_6.x86_64
 python-firewall                              noarch    0.6.3-2.0.1.el7_7.2                 ol7_latest            352 k
 systemd                                      x86_64    219-67.0.1.el7_7.2                  ol7_latest            5.1 M
 systemd-libs                                 x86_64    219-67.0.1.el7_7.2                  ol7_latest            411 k
 systemd-sysv                                 x86_64    219-67.0.1.el7_7.2                  ol7_latest             88 k

Transaction Summary
Install  303 Packages (+770 Dependent packages)
Upgrade               (   7 Dependent packages)

Total download size: 821 M
Is this ok [y/d/N]:

It will list a whole bunch of packages with dependencies that it will install. If you're up to it, at this point you would confirm with 'y'. Notice that there will be a bit over a 1000 packages installed, so it will be busy with that for a while.
This is because it will install the complete Gnome Desktop environment.
You could also do:
[vagrant@localhost ~]$ sudo yum groupinstall 'X Window System' 'GNOME'

That will install only the minimum, necessary packages to run Gnome. I did not try that yet.
If it finished installing all the packages, the one thing that is left, is to change the default runlevel, since obviously you want to start in the GUI by default. I think most in the cases, at least.
This is done by:
[vagrant@localhost ~]$ sudo systemctl set-default

I could have put that in a provision script, like I've done before. And maybe I will do that.

Package the box

You will have noticed that it would have stamped quite some time to update the kernel packages for installing the latest Guest Additons and the GUI desktop. To prevent us from doing that over and over again, I thought it was wise to package the box into a ol77SwGUI box (Server with GUI). I described that in my previous article last year:
vagrant package --base ol77_default_1575298630482_71883 --output d:\Projects\vagrant\boxes\

The result

This will deliver you a Vagrant Box/VirtualBox image with:
  • Provider: VirtualBox
  • 64 bit
  • 2 vCPUs
  • 2048 MB RAM
  • Minimal package set installed
  • 32 GiB root volume
  • 4 GiB swap
  • XFS root filesystem
  • Extra 16GiB VirtualBox disk image attached, dynamically allocated
  • Guest additions installed
  • Yum configured for Oracle Linux yum server. _latest and _addons repos enabled as well as _optional_latest, _developer, _developer_EPEL where available.
  • And as an extra addon: Server with GUI installed.
Or basically more or less what I have in may own base box. What I'm less happy with is the 16GiB extra disk image attached. I want a bigger disk for my installations, or at least the data. I'll need to figure out what I want to do with that. Maybe I add an extra disk and reformat the lot with a disk spanning Logical Volume based filesystem.


I found that that the box from Oracle lacks Video Memory to run VM in GUI mode. Because GUI wasn't in the VM, it 8 MB was sufficient.
I added the following to change the video memory:
  config.vm.provider "virtualbox" do |vb| = VM_NAME
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #   # Customize the amount of memory on the VM:
  #   vb.memory = "1024"
    vb.customize ["modifyvm", :id, "--vram", "128"]

Based on the hint found here on StackOverflow.
I also added this on my GitHub vagrant project.

Thursday, 28 November 2019

SOA Suite 12c Stumbling on parsing Ampersands

Yesterday I ran into a problem parsing xml in BPEL. A bit of context: I get messages from a JMS queue, that I read 'Opaque'. Because I want to be able to dispatch the messages to different processes based on a generic WSDL, but with a different payload.

So after the Base64 Decode, for which I have a service, I need to parse the content to XML. Now, I used to use the oraext:parseEscapedXML() function for it. This function is known to have bugs, but I traced that down to BPEL 10g. And I'm on now.

Still I got exceptions as:

<bpelFault><faultType>0</faultType><subLanguageExecutionFault xmlns=""><part name="summary"><summary>An error occurs while processing the XPath expression; the expression is oraext:parseEscapedXML($Invoke_Base64EncodeDecodeService_decode_OutputVariable.part1/ns5:document)</summary></part><part name="code"><code>XPath expression failed to execute</code></part><part name="detail"><detail>XPath expression failed to execute.
An error occurs while processing the XPath expression; the expression is oraext:parseEscapedXML($Invoke_Base64EncodeDecodeService_decode_OutputVariable.part1/ns5:document)
The XPath expression failed to execute; the reason was: oracle.fabric.common.xml.xpath.XPathFunctionException: Expected ';'.
Check the detailed root cause described in the exception message text and verify that the XPath query is correct.


<bpelFault><faultType>0</faultType><subLanguageExecutionFault xmlns=""><part name="summary"><summary>An error occurs while processing the XPath expression; the expression is oraext:parseEscapedXML($Invoke_Base64EncodeDecodeService_decode_OutputVariable.part1/ns5:document)</summary></part><part name="code"><code>XPath expression failed to execute</code></part><part name="detail"><detail>XPath expression failed to execute.
An error occurs while processing the XPath expression; the expression is oraext:parseEscapedXML($Invoke_Base64EncodeDecodeService_decode_OutputVariable.part1/ns5:document)
The XPath expression failed to execute; the reason was: oracle.fabric.common.xml.xpath.XPathFunctionException: Expected name instead of  .
Check the detailed root cause described in the exception message text and verify that the XPath query is correct.

It turns out that it was due to ampersands (&amp;) in the message. The function oraext:parseEscapedXML() is known to stumble on that.

A work around is suggested in a forum on Integration Cloud Service (ICS).  It suggests to use oraext:get-content-as-string() first. And feed the contents to oraext:parseEscapedXML(). It turns out that that helps, although I had to fiddle around with xpath expressions, to get the correct child element, since I also got the parent element surrounding the part I actually wanted to parse.

But then I found this blog, suggesting that it was replaced by oraext:parseXML() in 12c (I found that it is actually introduced in 11g).

Strange that I didn't find this earlier. Digging deeper down memory-lane, I think I must have seen the function before.  However, it shows I'm still learning all the time.