Monday 2 December 2019

Create a Vagrant box with Oracle Linux 7 Update 7 Server with GUI

Yesterday and today I have been attending the UKOUG TechFest '19 in Brighton. And it got me eager to try things out. For instance with new Oracle DB 19c features. And therefor I should update my vagrant boxes to be able to install one. But I realized my basebox is still on Oracle Linux 7U5, and so I wanted to have a neatly fresh, latest OL 7U7 box.

Use Oracle's base box

Now, last year I wrote about how to create your own Vagrant Base Box: Oracle Linux 7 Update 5 is out: time to create a new Vagrant Base Box. So I could create my own, but already quite some time ago I found out that Oracle supplies those base boxes.

They're made available at, and there are boxes for OL6, OL7 and even OL8. I want to use OL 7U7, and thus I got started with that one. It's neatly described at the mentioned link and it all comes down to:

$ vagrant box add --name <name> <url>
$ vagrant init <name>
$ vagrant up
$ vagrant ssh

And in my case:

$ vagrant box add --name ol77
$ vagrant init ol77
$ vagrant up
$ vagrant ssh

Before you do that vagrant up, you might want to edit your vagrant file, to add a name for your VM:
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at

  # Every Vagrant development environment requires a box. You can search for
  # boxes at = BOX_NAME


  # Provider-specific configuration so you can fine-tune various
  # backing providers for Vagrant. These expose provider-specific options.
  # Example for VirtualBox:
  config.vm.provider "virtualbox" do |vb| = VM_NAME
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #   # Customize the amount of memory on the VM:
  #   vb.memory = "1024"

Otherwise your VM name in Virtual box would be someting like ol7_default_1235897983, something cryptic with a random number.

If you do a vagrant up now it will boot up nicely.

VirtualBox Guest Additions

The VirtualBox GuestAdditions are from version 6.12, while my VirtualBox installation already has 6.14. I found it handy to have a plugin that auto-updates it. My co-Oracle-ACE Maarten Smeets wrote about that earlier. It comes down to executing the following in a command line:
vagrant plugin install vagrant-vbguest

If you do a vagrant up now, it will update the guest additions. However, to be able to do so, it needs to install all kinds of kernel packages to compile the drivers. So, be aware that this might take some time, and you'll need internet connection.

Server with GUI

The downloaded box is a Linux Server install, without a UI. This probably is fine for most of the installations you do. But I like to be able to log on to the desktop from time to time, and I want to be able to connect to that using MobaXterm, and be able to run a UI based installer or application. A bit of X-support is handy. How to do that, I found at this link.

GUI support is one of the group packages that are supported by Oracle Linux 7, and this works exactly the same as RHEL7 (wonder why that is?).

To list the available packages groups are supported, you can do:

[vagrant@localhost ~]$ sudo  yum group list
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
Available Environment Groups:
   Minimal Install
   Infrastructure Server
   File and Print Server
   Cinnamon Desktop
   MATE Desktop
   Basic Web Server
   Virtualization Host
   Server with GUI
Available Groups:
   Backup Client
   Compatibility Libraries
   Console internet tools
   Development tools
   E-mail server
   Educational Software
   Electronic Lab
   Fedora Packager
   General Purpose Desktop
   Graphical Administration Tools
   Graphics Creation Tools
   Hardware monitoring utilities
   Input Methods
   Internet Applications
   KDE Desktop
   Legacy UNIX Compatibility
   Network Infrastructure Server
   Networking Tools
   Office Suite and Productivity
   Performance Tools
   Scientific support
   Security Tools
   Smart card support
   System Management
   System administration tools
   Technical Writing
   TurboGears application framework
   Web Server
   Web Servlet Engine

(After having executed vagrant ssh.)
You'll find 'Server with GUI' as one of the options. This will install all the necessary packages to run Gnome. But, if you want to have KDE there's also package group for that.

To install it you would run:
[vagrant@localhost ~]$ sudo yum groupinstall 'Server with GUI'
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
Resolving Dependencies
--> Running transaction check
---> Package ModemManager.x86_64 0:1.6.10-3.el7_6 will be installed
--> Processing Dependency: ModemManager-glib(x86-64) = 1.6.10-3.el7_6 for package: ModemManager-1.6.10-3.el7_6.x86_64
--> Processing Dependency: libmbim-utils for package: ModemManager-1.6.10-3.el7_6.x86_64
--> Processing Dependency: libqmi-utils for package: ModemManager-1.6.10-3.el7_6.x86_64
--> Processing Dependency: for package: ModemManager-1.6.10-3.el7_6.x86_64
 python-firewall                              noarch    0.6.3-2.0.1.el7_7.2                 ol7_latest            352 k
 systemd                                      x86_64    219-67.0.1.el7_7.2                  ol7_latest            5.1 M
 systemd-libs                                 x86_64    219-67.0.1.el7_7.2                  ol7_latest            411 k
 systemd-sysv                                 x86_64    219-67.0.1.el7_7.2                  ol7_latest             88 k

Transaction Summary
Install  303 Packages (+770 Dependent packages)
Upgrade               (   7 Dependent packages)

Total download size: 821 M
Is this ok [y/d/N]:

It will list a whole bunch of packages with dependencies that it will install. If you're up to it, at this point you would confirm with 'y'. Notice that there will be a bit over a 1000 packages installed, so it will be busy with that for a while.
This is because it will install the complete Gnome Desktop environment.
You could also do:
[vagrant@localhost ~]$ sudo yum groupinstall 'X Window System' 'GNOME'

That will install only the minimum, necessary packages to run Gnome. I did not try that yet.
If it finished installing all the packages, the one thing that is left, is to change the default runlevel, since obviously you want to start in the GUI by default. I think most in the cases, at least.
This is done by:
[vagrant@localhost ~]$ sudo systemctl set-default

I could have put that in a provision script, like I've done before. And maybe I will do that.

Package the box

You will have noticed that it would have stamped quite some time to update the kernel packages for installing the latest Guest Additons and the GUI desktop. To prevent us from doing that over and over again, I thought it was wise to package the box into a ol77SwGUI box (Server with GUI). I described that in my previous article last year:
vagrant package --base ol77_default_1575298630482_71883 --output d:\Projects\vagrant\boxes\

The result

This will deliver you a Vagrant Box/VirtualBox image with:
  • Provider: VirtualBox
  • 64 bit
  • 2 vCPUs
  • 2048 MB RAM
  • Minimal package set installed
  • 32 GiB root volume
  • 4 GiB swap
  • XFS root filesystem
  • Extra 16GiB VirtualBox disk image attached, dynamically allocated
  • Guest additions installed
  • Yum configured for Oracle Linux yum server. _latest and _addons repos enabled as well as _optional_latest, _developer, _developer_EPEL where available.
  • And as an extra addon: Server with GUI installed.
Or basically more or less what I have in may own base box. What I'm less happy with is the 16GiB extra disk image attached. I want a bigger disk for my installations, or at least the data. I'll need to figure out what I want to do with that. Maybe I add an extra disk and reformat the lot with a disk spanning Logical Volume based filesystem.


I found that that the box from Oracle lacks Video Memory to run VM in GUI mode. Because GUI wasn't in the VM, it 8 MB was sufficient.
I added the following to change the video memory:
  config.vm.provider "virtualbox" do |vb| = VM_NAME
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #   # Customize the amount of memory on the VM:
  #   vb.memory = "1024"
    vb.customize ["modifyvm", :id, "--vram", "128"]

Based on the hint found here on StackOverflow.
I also added this on my GitHub vagrant project.

Thursday 28 November 2019

SOA Suite 12c Stumbling on parsing Ampersands

Yesterday I ran into a problem parsing xml in BPEL. A bit of context: I get messages from a JMS queue, that I read 'Opaque'. Because I want to be able to dispatch the messages to different processes based on a generic WSDL, but with a different payload.

So after the Base64 Decode, for which I have a service, I need to parse the content to XML. Now, I used to use the oraext:parseEscapedXML() function for it. This function is known to have bugs, but I traced that down to BPEL 10g. And I'm on now.

Still I got exceptions as:

<bpelFault><faultType>0</faultType><subLanguageExecutionFault xmlns=""><part name="summary"><summary>An error occurs while processing the XPath expression; the expression is oraext:parseEscapedXML($Invoke_Base64EncodeDecodeService_decode_OutputVariable.part1/ns5:document)</summary></part><part name="code"><code>XPath expression failed to execute</code></part><part name="detail"><detail>XPath expression failed to execute.
An error occurs while processing the XPath expression; the expression is oraext:parseEscapedXML($Invoke_Base64EncodeDecodeService_decode_OutputVariable.part1/ns5:document)
The XPath expression failed to execute; the reason was: oracle.fabric.common.xml.xpath.XPathFunctionException: Expected ';'.
Check the detailed root cause described in the exception message text and verify that the XPath query is correct.


<bpelFault><faultType>0</faultType><subLanguageExecutionFault xmlns=""><part name="summary"><summary>An error occurs while processing the XPath expression; the expression is oraext:parseEscapedXML($Invoke_Base64EncodeDecodeService_decode_OutputVariable.part1/ns5:document)</summary></part><part name="code"><code>XPath expression failed to execute</code></part><part name="detail"><detail>XPath expression failed to execute.
An error occurs while processing the XPath expression; the expression is oraext:parseEscapedXML($Invoke_Base64EncodeDecodeService_decode_OutputVariable.part1/ns5:document)
The XPath expression failed to execute; the reason was: oracle.fabric.common.xml.xpath.XPathFunctionException: Expected name instead of  .
Check the detailed root cause described in the exception message text and verify that the XPath query is correct.

It turns out that it was due to ampersands (&amp;) in the message. The function oraext:parseEscapedXML() is known to stumble on that.

A work around is suggested in a forum on Integration Cloud Service (ICS).  It suggests to use oraext:get-content-as-string() first. And feed the contents to oraext:parseEscapedXML(). It turns out that that helps, although I had to fiddle around with xpath expressions, to get the correct child element, since I also got the parent element surrounding the part I actually wanted to parse.

But then I found this blog, suggesting that it was replaced by oraext:parseXML() in 12c (I found that it is actually introduced in 11g).

Strange that I didn't find this earlier. Digging deeper down memory-lane, I think I must have seen the function before.  However, it shows I'm still learning all the time.

Thursday 10 October 2019

Oracle Ground Breakers Appreciation Day - Something about Weblogic....

Our most appreciated Oracle ACE Director Tim Hall organizes this yearly initiative, with this years name Oracle Ground Breakers Appreciation Day, and appointed this day to blog about our favorite Oracle Technology, Service or sub-community.

Last week I presented the 'Oracle Kubernetes Managed Weblogic Revival', the introduction of the Weblogic Kubernetes Operator opens the future for Weblogic.

This week I deliver our Weblogic 12c Tuning and Troubleshooting training for ATOS The Netherlands in Groningen. So, hmmm. what to blog, on this years Ground Breakers Appreciation day? There are several other technologies that I use and follow, but mostly around Fusion Middleware: SOA Suite, BPM Suite and Oracle Service Bus. But also Oracle Integration Cloud, that in fact heavily depend on this technologies. And honestly, bottom line here is Oracle Weblogic.

I frequently hear voices that state that Customers should move away from Weblogic. Honestly, I don't relate to that. It has served customers very well over the last decade under the Oracle brand and before. And I still think it was a smart move of Oracle to acquire it and make it a strategic part of the Oracle platform.

Last few years I've been active on the forums, where I've grown to level 13, almost level 14, by answering questions and participating in discussions around Fusion Middleware technologies. My first thank therefor is to this community, for having me participating.

My second thank goes to the whole Weblogic and related Fusion Middleware toolstack. During the Tuning and Troubleshooting training I again realize how smart and rich the Weblogic Suite is. Although, I stated before that Oracle could do something about the footprint. It seems to me that there are quite some duplicate libraries or different versions of the same library. And maybe some old parts could be cut out: maybe only support SAML2.0 and improve that, for instance.

One great, but quite rarely used feature of Weblogic is the Weblogic Diagnostic Framework. And especially the Policies and Actions part. It is quite difficult to configure, the console's UI does not help here and there, and to think of usages of it. However, every time I present it, I find myself thinking: I should use this more often in my daily developments.

So I started to create a wlst script to create a Diagnostic Module, create a few collectors,  a JMS Notification Action and 2 policies on it.  It is actually the solution to Lab 6 of our training. To me it is a start to be able to expand this. You could  create a version per technology: OSB, SOA Suite, or custom application like MedRec. And you can create a more generic version that based on different property files configure different collectors, policies and actions specific for that target environment.

WLDF Diagnostic Module

The script first creates a diagnostic module like this:
def createDiagnosticModule(diagModuleName, targetServerName):
  if module==None:
    print 'Create new Diagnostic Module'+diagModuleName
    module = cmo.createWLDFSystemResource(diagModuleName)
    # Activate changes
    print 'Diagnostic Module created successfully.'
    print 'Diagnostic Module'+diagModuleName+' already exists!'
  return module

It checks if the Diagnostic Module already exists as a WLDFSystemResource. If not, it will create it as module = cmo.createWLDFSystemResource(diagModuleName) and target it to a targetServer.


Then for creating a collector I created the following function:
def createCollector(diagModuleName, metricType, namespace, harvestedInstances,attributesCsv):
  print 'Check Collector '+harvestedTypesPath+metricType
  if collector==None:
    print 'Create new Collector for '+metricType+' in '+diagModuleName
    attributeArray=jarray.array([String(x.strip()) for x in attributesCsv.split(',')], String)
    # Activate changes
    print 'Collector created successfully.'
    print 'Collector '+metricType+' in '+diagModuleName+' already exists!'
  return collector

Again, it first checks for the existing of the collector as a so called HarvestedType, within a WLDFResource in the Diagnostic Module. If not it creates it. Here you need to provide the metricType as a HavervestedType. And then attributes that you want to collect. The function expects it as a comma separated values string, that it converts to an array via a List.
Then you can provide Metric Type Instances or None if you want to collect it over all instances.

You can call this as:
createCollector(diagModuleName, '','ServerRuntime', None, 'ActiveConnectionsCurrentCount,CurrCapacity,LeakedConnectionCount')

or if you want to add instances, it's also done by creating an array:
    harvestedInstances=jarray.array([String(x.strip()) for x in harvestedInstancesList], String)    
    createCollector(diagModuleName, '','ServerRuntime', harvestedInstances,'OpenSessionsCurrentCount')

This is a bit more complicated, since the strings describing the instances that you want to add are comma seperated values them selfs.


Creating an action is again pretty simple, for a JMS Notification that is:
def createJmsNotificationAction(diagModuleName, actionName, destination, connectionFactory):
  print 'Check notification action '+jmsNotificationPath+actionName
  if jmsNtfAction==None:
    print 'Create new JMS NotificationAction '+actionName+' in '+diagModuleName
    # Activate changes
    print 'JMS NotificationAction created successfully.'
    print 'JMS NotificationAction '+actionName+' in '+diagModuleName+' already exists!'
  return jmsNtfAction

There are different types of actions, so they're created differently. You can add one using the console and record that. It's what I did and then transformed the recorded script to functions as shown here.


Policies can be created with the following function. You need to provide a rule type and a rule expression, plus a array of actions you want to add:
def createPolicy(diagModuleName, policyName, ruleType, ruleExpression, actions):  
  print 'Check Policy '+policiesPath +policyName
  policy=getMBean(policiesPath +policyName)
  if policy==None:
    print 'Create new Policy '+policyName+' in '+diagModuleName
    cd(policiesPath +policyName)
    set('Notifications', actions)
    schedule=getMBean(policiesPath +policyName+'/Schedule/'+policyName)
    # Activate changes
    print 'Policy created successfully.'
    print 'Policy '+policyName+' in '+diagModuleName+' already exists!'
  return policy

An example of calling this is:
    actions=jarray.array([ObjectName(action.strip()) for action in actionsList], ObjectName)    
    createPolicy(diagModuleName,'HiStuckThreads', 'Harvester', 'wls:ServerHighStuckThreads(\"30 seconds\",\"10 minutes\",5)', actions)

As you can see the actions to add are actually expressions to the MBeans of the actions configured earlier. It apparently depend on the type and the diagnostic module that contains it. So I could create a function that assembles this expression. If you want a custom rule expression you can create it as follows:
ruleExpression='wls:ServerGenericMetricRule(\"com.bea:Name=MedRecGlobalDataSourceXA,ServerRuntime=TTServer,Type=JDBCDataSourceRuntime\",\"WaitingForConnectionHighCount\",\">\",0,\"30 seconds\",\"10 minutes\")'
    createPolicy(diagModuleName,'OverloadedDS', 'Harvester', ruleExpression, actions)

Again this is an expression that could be assembled using a function.


The complete script can be reviewed and downloaded from my GitHub Repo.

I hit two flies with one beat: Thank you Ground Breakers, fellow ACEs and other Oracle enthousiasts, and I guess my first article about the Weblogic Diagnostic Framework (but not my first one to include WLST scripts...). Happy OGB Appreciation Day y'all!

Tuesday 27 August 2019

SOASuite Composite Sensors: the why and how...


Long time ago BPEL PM was acquired by Oracle, and as part of the first release of SOA Suite (10g), it was a more or less standalone component. For initiated BPEL flow instances in the soa infrastructure database there were 2 tables:
  1. cube_instance: bpel flow instances
  2. ci_indexes: 6 indexes related to the bpel flow that can be set with an embedded java call

These 2 tables store the BPEL instances, along with a set of indexes that you could, and in 11g and 12c still can, set with a value that you determine during the flow. Yes, these tables still exist in the soa infra database. So, let's say in your BPEL you have several string based variables that you fill with a value from the input message using an assing. Then within an Embedded Java activity, you can do something like:

//Get Variables 
String messageType = (String) getVariableData("messageType"); 
String messageId = (String) getVariableData("messageId"); 
String processId = (String) getVariableData("processId"); 
String referenceNr = (String) getVariableData("referenceNr"); 
String branchId = (String) getVariableData("branchId"); 
String cmrNr = (String) getVariableData("cmrNr"); 

//Set Title and indexes 
setFlowInstanceTitle("MyProcessFlow " + messageType + '-" + messageId); 

When you spin of a set of new instances, you can use the following query to find the particular instances:
select ci.flow_id, ci.cmpst_id cube_composite_id, ci.cikey cube_cikey, cix.index_1, cix.index_2, cix.index_3, cix.index_4, cix.index_5, cix.index_6 
from cube_instance ci
join ci_indexes cix on ci.cikey = cix.cikey
where index_1 like '123456789';

With the flow_id you can query the SCA flow instance (in 12c) and/or find the instance in EM.
select * from sca_flow_instance fi where fi.flow_id=100173;

Unfortunately, not even in 10g you can query on the indexes in EM directly. You need to query on them in the database and copy and paste the resulting flow-id in EM - FMW Control.
You might have done this in the past, or still do. You might have created a JSP that helps you with this. We did in 10g at least.

Define Composite Sensors

Since 11g, there is a much more convenient way to do. And it's all declarative and usable from EM. It's called Composite Sensors. You can read more about it in the docs
I haven't blogged about it earlier, because, ...., honestly I haven't used them much until lately.

Composite Sensors can be set in the composite editor:

This will get you to the following dialog:

Select one of the Services or References and click on the blue plus icon:
In this dialog, set a Name, check/validate the Service and Operation, and click on the pencil icon to define an Expression:

  • Variables: clicking this will provide you a navigator that will allow you to drilldown the variable structure of the service operation message type, to select the element to sense the value.
  • Expression: this will show you the expression builder you should be familiar with: it's the same as the one in the assign activity copy rules in bpel. It allows you to create more complex xpath statements like: substring($in.payload/doc:RegisterDocument/doc:Document/doc:BinairyObject/@fileName,0, 100)
  • Properties: allows you to select endpoint properties, for instance JCA properies as JMS Type, JMS CorrelationID. The same as the properties on a BPEL Invoke.
Make sure you have the Enterprise Manager checkbox checked. It should be on by default.

A good source for creating the Composite sensors is the Embedded Java that sets the indexes of the BPEL, as described in the intro of this article. Create a sensor for every index, and base it on the Service Operation on which the variable is based from which the setting of the indexes are based.

I would highly recommend to create an Excel sheet to register which Sensors are defined on which Service/Operation and how they are filled. For instance, you could have several services that work with documents. And on all those composites you might have sensors that fetch the DocumentID. One of your developers would define a sensor called docId, another uses documentId, again another would define docNumber, etc., etc.  An end user or administrator would need and know all those variants. Wouldn't it be much easier that you could just search on documentId  over all those composites? Thus, introduce a method in your team that everyone uses the same sensor name for elements that mean the same.

Search on CompositeSensor values

On the Soa Infra dashboard in Enterprise Manager - Fusion MiddleWare Control (em) you can quickly search on a Sensor:
Fill in a Sensor Name and a search value and click on the Search Instances button.
These are free format fields, so it make sense to have a list of possible sensors that can be distributed along your admins or end users.

In the Search Instance panel of the flow instances tab, you have a more comprehensive search possibility:

This is not available when you click directly on the Flow Instances tab, without performing a search first.
In that case you need to click on Add/Remove Filters on the Flow Instances tab:

In this dialog, check the Flow Instance checkbox:
Having done that, you can add up to 6 sensor search conditions. Click on the magnifier glass to search on a sensor:

Here you can search on a composite on which you know there is sensor. Then you can select a sensor and an operator to search on. Unfortunately this is the only place to choose an operator, which means that you need to search for a sensor through a Composite Revision, before being able to choose an operator. Would be nice being able to just type in a sensor name (or copy and paste it from your excel sheet), select an operator and type a value to search over composites.

What is nice is that if you select a particular flow instance, you can view its composite sensor values:

This is especially handy, when in a busy environment where there are several instances of the same composite within a certain timeframe. Then you can quite easily click through the instances and identify if the particular one is the one you're interesting in. In stead of the need to open the flow trace, click to the bpel flow, select the receive activity and open the XML. In many cases this can be a very tedious job.

Tuesday 16 July 2019

Weblogic under Kubernetes: the weblogic topology of the future

Already 4 months ago I attended the PaaSForum 2019 in Mallorca. As every year it was great to meet members of the big EMEA Oracle Partner family.

And of course a lot of interesting talks and workshops. This year I was especially interested in announcements around SOA Suite and Project Helidon as a Microservice framework. But certainly also Weblogic under Kubernetes.
And actually, to me, the Kubernetes Weblogic Operator that was this years most enthusing subject.

With his WebLogic on Kubernetes talk Maciej Gruszka, Director Product Management, enlightened the future Oracle envisions for WebLogic. He started with stating that 'Weblogic is not dead!'. Well, he got me with that already!

The road ahead is making WebLogic fit to run in Docker and managed by Kubernetes. It might not be exactly what I had in mind, but it is certainly great news to learn that WebLogic will be around and alive for a future ahead. Oracle thrives to make future releases of Weblogic available as Docker images.

Today already, WebLogic is fully supported to run in a Docker container. And according to Marciej, the team is busy with the SOA and OSB teams to get those products fit and available for Docker too. It might even be possible that future releases are going to be delivered as a Docker image.

What is the Weblogic Operator?

To run in a Kubernetes managed cluster, Kubernetes needs to be able to perform lifecycle operation on a Weblogic Managed server. For that  the Weblogic Operator for Kubernetes is created and introduced. A Kubernetes Operator is a sort of Adapter on top of a non-Kubernetes system that translates Kubernetes lifecycle commands to operations within the specific application.

The Weblogic Operator  uses Kubernetes API to implement operations like:
  • Provisioning
  • Life cycle maangement
  • Updates
  • Scaling
  • Security
Besides the Weblogic Operator, Oracle also provides an Exporter for Prometheus and Elastic Stack, for monitoring and logging. Since the managed servers are within a container, you'll need to export events and logfiles to have them accessible and introspectible, even when the container is down or recreated from an updated image.

Some interesting links


There are actually two topologies to choose from:
  • Domain within the Docker Image
  • Domain on a Persistent Volume
With the first one the container is actually stateless. All it needs to know is within the container. The Admin Console can be used for diagnostic and monitoring purposes, but not for updating the domain. Because spinning a new container will have it read the domain from the internal container image.

With the persistent volume topology the domain is stored outside the container. Changes are persisted. This topology is more in line with an On Premises installation of Weblogic. However, High Availability and Disaster Recovery is limited, because Persistent Volume needs to be shared and the domain configuration needs to be synced across datacenters. With 'In Image' Domains, things get simpler, because the domain is transported within the container. Downside is that changes in the domain require creating a new image through the CI/CD pipeline.

Most customers seem to choose for the 'Domain in Image' topology. In practice, domains don't change that much.

You can  adapt specific artifacts like data source connections, urls and username/passwords using Configuration Overrides.


At the PaaSForum we got the chance to play around with Kubernetes and Weblogic. The workshop is described here: You should fork this to a repository with your own Github account, because it contains the files and scripts to create an image, the tutorial works you through configuring Oracle Container Pipelines (Worker) and for that it needs a Github repo.

There is a Domain In Image variant and  a persistent volume variant of the tutorial.

Steps to follow for the Domain In Image variant

  1. Setup Oracle Kubernetes Engine instance on Oracle Cloud Infrastructure. You'll need a trial accound on It will then guide you through the setup of an Kubernetes cluster on OCI.
  2. Build WebLogic container image using Oracle Container Pipelines (Wercker). The second time I did the workshop I decided to change all the labels, namespaces and the domain name. Every where there is a reference to 'sample', I entered 'makker'. In this step the image is created from your fork of the github repo. If you change the name of the domain, there are two files to edit:
    1. The Dockerfile.create is called at the initial creation of the image. If there is a base image, the Dockerfile.update is called, to update the image. The Dockerfile.create creates an image with a complete domain, including the application. But the Dockerfile.update only updates the application. So you need to update the Dockerfile.create to change the domain name in the DOMAIN_NAME environment variable in the top of the file. 
    2. The Dockerfile.create copies the scripts folder into the image. That folder contains a wlst script, called At the top, a variable domain_name is declared with the same domain name assigned to it.
    If you do not change it, and want rename the domain to start it with a different name using Kubernetes later on, then you need to remove the image from the image repository, and then run the Oracle Container Pipelines-pipeline again.
  3.  Install WebLogic Operator: installs the Weblogic Operator.
  4.  Install and configure Traefik: this installs a Traefik loadbalancer on your environment. It will loadbalance over your Weblogic managed servers.
  5. Deploy WebLogic domain: this step lets you prepare your Kubernetes cluster to run the Weblogic domain. Reuse the same domain name as explained in step 2.
  6. Scaling WebLogic cluster: This one I found particularly cool. In this step you update the domain resource yaml file, to update the number of managed servers in the domain. After that, automagically a new Kubernetes pod is spawned that starts a new Managed Server. By the way, the domain will have a dynamic cluster with predefined Managed Servers based on Server Templates.
  7. Override domain configuration:  this will show you how to perform domain configuration overrides to update the datasource.
  8. Update the appliation: The whole point of this exercise is to show you how to setup a CI/CD chain that when you update your application, the image is updated and the domain can be restarted through Kubernetes, with the new image.
  9. Assing the Weblogic Pods to specific nodes or licenced nodes. The latter is important because Weblogic is licensed, so you can't just run it on any number of nodes.
The tutorial is quite elaborate and descriptive. If you stick to the naming, it will guide you through the proces ending up with a running environment. The fun is in being self-wise and choose your own naming. That's how I tripped at step 5  Deploy Weblogic Domain. I could have stuck with the given name. But I didn't feel like it, it was more fun to understand where it was used. Now you can take advantage of it.


I refrained discussing why you would want to run Weblogic under Docker. I have thoughts and had discussions about it. However, it made me enthousiastic that this way Weblogic can be taken with us into the containerized future.

For me the next things to explore are:
  • Create a database on another OCI image, and create a new domain, with a sample application that actually uses that database. It would be fun to create an actuall application on it.
  • Try the same with a persistent volume. A few months ago I was busy with creating java classes to start Kafka. The goal was to create Weblogic Startup classes to have Kafka started at startup of a Weblogic server. Now, it may not seem logical to you, but wouldn't it be great to combine the two and have Kafka embedded in a Weblogic cluster on a Kubernetes Cluster? Well, at least it seems fun to me. Since Kafka needs to log it's messages in a persistent log, we need to do this with a Persistent Volume.
  • Check out other topologies and related technologies. Like accessing the logs. I really would like to be able to  inspect the Weblogic log files within the container.
Have fun with the tutorial.

Monday 24 June 2019

Debugging code - Identifying the bug

Julia Evans is a very smart woman in IT who creates very nice, funny and insight full comics, she calls 'zines', on Linux and Coding topics.

This morning I read she came up with a question that triggered me:

Last week I  realized I'm already 25 years in IT, after that my forced membership of a famous Dutch shooting club ended after 9 months (the kind of shooting club where you got free clothing, survival courses, and in my case also a truck-driving-license. Which other club offers that?). 

Anyway, during the years I discovered that despite of all the smart people I got to know and work with, this part of our work isn't obvious. In very many cases people seem to 'just do something'. No offense, but for developers it's often frustrating and just not fun to work on bugs or problems. And administrators that are confronted with a problem are often 'too busy with other stuff.' So they try something, don't find the thing, and at a later moment try something else. So, when I got involved I ask the obvious questions and in most cases I try out the same thing myself. Even though I do believe them, I want, I need,  to see the behavior with my own eyes.

By the way, I'm always reluctant to call it a bug. A bug is only a bug when you have reproduced it and based on common interpretation, together with the tester (if he found the issue), the functional/solution designer come to a consensus that the code does not do what it is supposed to. The functional specs are interpreted by both the tester and the developer. And in a certain way also the designer. It might be that the tester finds an anomaly, but that it is either a miss-interpretaton from him or a problem with the formulating of the specs. There are cases that the coder is right. But, of course, your program can work with an unexpected logic.

But, back to Julia's tweets:  they triggered me, so I jotted down some thoughts that I  got and are the basis of my search for issues.

To me it starts with identifying a case where it goes wrong, but equally importantly, together with a similar situation where it goes right. And, as far as possible, creating a unit test for both. Since, my work is mostly done on message processing platforms (Oracle SOA Suite, BPM Suite, Service Bus), I love it when a tester can hand me over a triggering message of the case and the involving response messages. I can then add them to my unit-test-set in SoapUI/ReadyAPI.

Then I add instrumentation (log lines, etc.) on key positions, that identify to which points the code is executed and what lines aren't reached. SOA Suite produces a flow trace of the execution. But often expressions are used that are quite complex one liners. I then split those up into several separate assignments to 'in between' variables. In Java, JavaScript, etc., I do not like complex one-liners. I prefer several variables for 'in between' values, and assignments with short expressions. That helps with line-by-line debugging.

Next, I iteratively narrow that gap between the point I can conclude the code reaches and the point I find not reached, until the statement or point of execution that fails can be identified.
In the log lines I add key variable values that are involved.

In very rare, very difficult, cases, I sometimes break down the code, cut away all the code that is not touched, until I get a minimal working Mickey Mouse (or in Dutch: Jip en Janneke) case. From there I build it up, and test iteratively in very small steps, until it breaks.

Also, very important, for difficult problems, I document very meticulously what I have done and concluded. My slogan here is: 'Deduction, my dear Whatson!' When having a problem, one can quickly come up with some potential causes and tests to check those. A unit test for a potential cause can go two ways, it can confirm or disapprove the suspicion. Both outcomes have consequences for the follow-up. Disaproving a potential cause, can strikethrough other potential paths as well. 

But, approving it, need additional steps to narrow down. I see it as a decision-tree to follow.

What I have found through the years, is that structurally document the steps done with the particular conclusions and the follow-ups is not quite obvious. But in many cases I found them important. Especially working in a Taskforce, or when I got hired to get involved in a case. In those cases the customer that hired you has the right to have something in hands that represents what he payed for.
I once was involved in a case that turned out to be a database bug. So I could not help the customer to solve it. But they where very pleased in the structured method I used to check out what could be the problem. And for those administrators and developers that got to do this as  a side job, besides there regular things: please do yourself a favor and document. I found Google Docs very usefull in this.

Oh, and by the way: I work with BPEL, BPMN, Oracle Service Bus, Java, Pl/Sql, XSLT, XQuery, Python/Jython/WLST, sometimes JavaScript, you name it. And actually, my way of structured code or systems analysis comes down to the same procedures. Regardless of technology.