Monday, 24 June 2019

Debugging code - Identifying the bug

Julia Evans is a very smart woman in IT who creates very nice, funny and insight full comics, she calls 'zines', on Linux and Coding topics.

This morning I read she came up with a question that triggered me:

Last week I  realized I'm already 25 years in IT, after that my forced membership of a famous Dutch shooting club ended after 9 months (the kind of shooting club where you got free clothing, survival courses, and in my case also a truck-driving-license. Which other club offers that?). 

Anyway, during the years I discovered that despite of all the smart people I got to know and work with, this part of our work isn't obvious. In very many cases people seem to 'just do something'. No offense, but for developers it's often frustrating and just not fun to work on bugs or problems. And administrators that are confronted with a problem are often 'too busy with other stuff.' So they try something, don't find the thing, and at a later moment try something else. So, when I got involved I ask the obvious questions and in most cases I try out the same thing myself. Even though I do believe them, I want, I need,  to see the behavior with my own eyes.

By the way, I'm always reluctant to call it a bug. A bug is only a bug when you have reproduced it and based on common interpretation, together with the tester (if he found the issue), the functional/solution designer come to a consensus that the code does not do what it is supposed to. The functional specs are interpreted by both the tester and the developer. And in a certain way also the designer. It might be that the tester finds an anomaly, but that it is either a miss-interpretaton from him or a problem with the formulating of the specs. There are cases that the coder is right. But, of course, your program can work with an unexpected logic.

But, back to Julia's tweets:  they triggered me, so I jotted down some thoughts that I  got and are the basis of my search for issues.

To me it starts with identifying a case where it goes wrong, but equally importantly, together with a similar situation where it goes right. And, as far as possible, creating a unit test for both. Since, my work is mostly done on message processing platforms (Oracle SOA Suite, BPM Suite, Service Bus), I love it when a tester can hand me over a triggering message of the case and the involving response messages. I can then add them to my unit-test-set in SoapUI/ReadyAPI.

Then I add instrumentation (log lines, etc.) on key positions, that identify to which points the code is executed and what lines aren't reached. SOA Suite produces a flow trace of the execution. But often expressions are used that are quite complex one liners. I then split those up into several separate assignments to 'in between' variables. In Java, JavaScript, etc., I do not like complex one-liners. I prefer several variables for 'in between' values, and assignments with short expressions. That helps with line-by-line debugging.

Next, I iteratively narrow that gap between the point I can conclude the code reaches and the point I find not reached, until the statement or point of execution that fails can be identified.
In the log lines I add key variable values that are involved.

In very rare, very difficult, cases, I sometimes break down the code, cut away all the code that is not touched, until I get a minimal working Mickey Mouse (or in Dutch: Jip en Janneke) case. From there I build it up, and test iteratively in very small steps, until it breaks.

Also, very important, for difficult problems, I document very meticulously what I have done and concluded. My slogan here is: 'Deduction, my dear Whatson!' When having a problem, one can quickly come up with some potential causes and tests to check those. A unit test for a potential cause can go two ways, it can confirm or disapprove the suspicion. Both outcomes have consequences for the follow-up. Disaproving a potential cause, can strikethrough other potential paths as well. 

But, approving it, need additional steps to narrow down. I see it as a decision-tree to follow.

What I have found through the years, is that structurally document the steps done with the particular conclusions and the follow-ups is not quite obvious. But in many cases I found them important. Especially working in a Taskforce, or when I got hired to get involved in a case. In those cases the customer that hired you has the right to have something in hands that represents what he payed for.
I once was involved in a case that turned out to be a database bug. So I could not help the customer to solve it. But they where very pleased in the structured method I used to check out what could be the problem. And for those administrators and developers that got to do this as  a side job, besides there regular things: please do yourself a favor and document. I found Google Docs very usefull in this.

Oh, and by the way: I work with BPEL, BPMN, Oracle Service Bus, Java, Pl/Sql, XSLT, XQuery, Python/Jython/WLST, sometimes JavaScript, you name it. And actually, my way of structured code or systems analysis comes down to the same procedures. Regardless of technology.

Thursday, 6 June 2019

Weblogic Signs SAML2 requests and responses with SHA-256

Today I reviewed a few responses on a 'What's new in Weblogic' question.
One of the responses mentioned the whats-new document.

Now, I'm not used to study these documents. But today I browsed through them and one thing caught my eye.

I did some implementations of Weblogic as a SAML2 Service Provider against MS ADFS. I'm even invited to do a talk 'SSO with ADFS for Apex Using Weblogic and ORDS: How I did it and Where I Tripped' at the UKOUG Southern Technology Summit 2019, july 2nd.

What's interesting here is that about 2 years ago I already wrote about my earlier experiences, and mentioned that Weblogic 12c did not support SHA-256 for the signing of SAML requests. So you had to configure ADFS to use SHA-1. In my latest implementation it stroke me that I did not have to force my ADFS counterpart to set that, at least I think I didn't. ADFS as you might expect for really some time now, uses SHA2 (SHA-256) as a default. But only today I saw that under Manageability Improvements -> Security is mentioned that Weblogic also has SHA2 as a default now.

Knowing this will improve my talk greatly. I'm glad I saw this. It might seem to be a minor thing, but I think it's quite important.

I use Weblogic mostly as a FMW Infrastructure for SOA Suite, OSB, etc. And occasionally I do assignments with specifics like SAML2. If you're interested in what changed in a specific Weblogic version, I think it's important to know what you're looking for. Know the functionality that you're actively using or interested in.

Wednesday, 24 April 2019

Test Remote Asynchronous Request Response services

A few years ago, I described how you can test Asynchronous Request Response services.

The thing with Asynchronous Request Response services is, as I used to describe it, that they're in essence two complementary Request-Only (Fire and Forget) services. That is, the client submits a request to the Asynchronous Request Response service, and at a certain point waits for the response by listening to an endpoint.

To make this work, the responding Asynchronous Request Response service should be told, which endpoint it should call with the response and which correlation id should be used. The WS-Addressing standard is used for that. All nicely explained in the before mentioned article.

In most customer-cases the problem is that your Client SoapUI or ReadyAPI project should catch the response, but the service is running on a SOA Suite in the datacenter and is not allowed to get to your local machine.

MobaXterm makes it very easy to create a tunnel. You can have a remote tunnel, that enables a local listening endpoint, that forwards every request to a remote service. Very handy if you have a Vagrant project with only a NAT NetworkAdapter, where Vagrant enabled a ssh endpoint on port 2222. You can easily create a Local tunnel on port 7101, for instance, to the remote ssh session on port 2222, that enables you to get to the weblogic console on the remote VM running on http://darlin-vce:7101/console.

To create a tunnel, just open the MobaSSHTunnel - Grahpical port forwarding tool:
This will open:

You can create a new SSH tunnel or edit a current one using the cogs icon under settings. For instance, to be able to do the Local port forwarding to get to your Weblogic console on your Vagrant box, create a tunnel as follows:

On the left you can enter a local port. That is the port you can use on your localhost. On the top right you can enter an host and port for the address to post your request to (does not need to be localhost). Then bottom right you need to provide an ssh session. A bit inconvenient is that you can't select a session from the sessions pane. Provide a host, port and user to connect to your ssh server.

What happens is that MobaXterm creates an SSH session, and a local endpoint. Every thing posted to the local endpoint is posted on the remote server to the give address. In this case I can go on my browser and enter https;//localhost:7101/console and it will bring me to the Weblogic Console on my Vagrant box. Neat, isn't it?

To get the remote Async Service respond to your local machine, you can also create a we need a tunnel that works the otherway around: we need Remote Port Forwarding:

Configuring is similar to Local port forwarding, however, now on the remote server a listen endpoint is created, and everything that is posted to the localhost:7777 adress (in this example) is forwarded to the address entered on the local server. In this case it is forwarded to localhost:7777, but it could be something else.

In our ReadyAPI project I created a Groovy script as follows:
def testCase = testRunner.testCase
def env =
if (env != "o02-12c"  &&  env != "o02" ) { "Environment: "+env+", so set callbackIp to "+InetAddress.localHost.hostAddress
  testRunner.testCase.setPropertyValue( "callbackIp", InetAddress.localHost.hostAddress)
} else { "Environment: "+env+", so set callbackIp to localhost"
  testRunner.testCase.setPropertyValue( "callbackIp", "localhost")

In ReadyAPI you can define environments, with the project property it can be queried.

If the environment points to one of our development environments, I set the callbackIp testcase property to "localhost". But for the default environment, I use InetAddress.localHost.hostAddress to get the local ip address. This will be the ip address of our CD/CI tool, that runs ReadyAPI from a script.

You can set the WS-Addressing ReplyTo address as follows, for instance:
<soapenv:Envelope xmlns:soapenv="" 

Then this address is used to do the callback. Make sure the tunnel is started:

You can also have the tunnel auto started (with the blue man-running-icon) or auto-reconnected (with the purple lightning icon).

This may also be very relevant in testing services on Oracle SOA Cloud Service, or Integration Cloud.

Happy tunneling!

Thursday, 18 April 2019

Split your Vagrant provisioners

For a while now, I'm quite into Vagrant in combination with VirtualBox. A few years ago I started with trying to script FMW environments, and since my discovery, and resulting fancy, of Vagrant, I also created a project for creating and provisioning a SOA Suite box.

Until now, my project all had one shell-type provisioner looking like:
  config.vm.provision "shell", inline: <<-SHELL
    export SCRIPT_HOME=/vagrant/scripts
    echo _______________________________________________________________________________
    echo 0. Prepare Oracle Linux
    echo _______________________________________________________________________________
    echo 1. Create Filesystem
    echo _______________________________________________________________________________
    echo 2. Create Oracle User
    echo _______________________________________________________________________________
    echo 3. Java SDK 8
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/'

This seems quite simple, but for my SOA Suite box, I had quite a lengthy provisioner, that somehow failed at running the RCU and therefor with the creation of the domain.  There is a synchronization thingy with the database. The database is up, but at the time it reaches the RCU creation, it isn't able to connect. When running it seperately it works like a charm.

So, don't know how to solve that, but I want to re-provision only the part of the RCU and domain creation. Last night I fiddled around with it. Following the Vagrant Up basic usage explanation, you can create multiple provisioners with different names and different types. You can then force the provisioning for certain provisioners by type or by name.

I played around with that, because I couldn't get the syntax right. Although the explanation is proper, I wanted to have it slightly different and did not got it at first. Finally, I got it working.

Let's look into it.

First I split up my shell script, and found that I can put those in a variable. I now have a init script, that adapts the Linux OS, creates a new filesystem and creates an oracle user :
  $initScript = <<-SCRIPT
    export SCRIPT_HOME=/vagrant/scripts
    echo _______________________________________________________________________________
    echo 0. Prepare Oracle Linux
    echo _______________________________________________________________________________
    echo 1. Create Filesystem
    echo _______________________________________________________________________________
    echo 2. Create Oracle User

And one for installing the FMW software:
  $installFMWScript = <<-SCRIPT
    echo _______________________________________________________________________________
    echo 3. Java SDK 8
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/'
    echo _______________________________________________________________________________
    echo 4. Database 12c
    sudo runuser -l oracle -c '/vagrant/scripts/database/'
    echo _______________________________________________________________________________
    echo 5.1 SQLCL and SQLDeveloper
    sudo runuser -l oracle -c '/vagrant/scripts/database/'
    echo _______________________________________________________________________________
    echo 5.2 SQLDeveloper
    sudo runuser -l oracle -c '/vagrant/scripts/database/'
    echo _______________________________________________________________________________
    echo 6. Fusion Middleware
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/'
    echo _______________________________________________________________________________
    echo 6.1 Fusion Middleware - SOA
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/'
    echo _______________________________________________________________________________
    echo 6.2 Fusion Middleware - SB    
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/'
    echo _______________________________________________________________________________
    echo 6.3 Fusion Middleware - OHS    
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/'
    echo _______________________________________________________________________________
    echo 7. BPM Quickstart
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/'

And one for configuring FMW, that is running the RCU and creating the domain:
  $configFMWScript = <<-SCRIPT
    echo _______________________________________________________________________________
    echo 8.1 Fusion Middleware - RCU SOA   
    sudo runuser -l oracle -c '/home/oracle/bin/'    
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/'
    echo _______________________________________________________________________________
    echo 8.2 Fusion Middleware - Create Domain    
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/fmw1221_domain/'
    echo !!! TODO: Machine configuration update to use Plain - 5555
    echo !!! TODO: Modify domain creation and property naming to create machine in accordance to nodemanager config.
    echo _______________________________________________________________________________
    echo 8.3 Fusion Middleware - Modify Nodemanager     
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/fmw1221_domain/'
    echo _______________________________________________________________________________
    echo 8.4 Fusion Middleware - Create Nodemanager service
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/fmw1221_domain/'

Cool, so far, right?

Now, after that we need to define the 3 provisioners:

  config.vm.provision "init", type: "shell", inline: $initScript
  config.vm.provision "installFMW", type: "shell", inline: $installFMWScript
  config.vm.provision "configFMW", type: "shell", inline: $configFMWScript

These provisioners

  • init -> provisioning/config of Oracle Linux, creation of oracle user, etc. This will be about equal for every box.
  • installFMW -> installation of all FMW software.
  • configFMW -> run the RCU and create domein.
Having that in place, you can run a specific provisioner. For instance during up:
  • vagrant up --provision-with configureFMW
or when reloading the box:
  • vagrant reload --provision-with configureFMW
But the following also works:
  • vagrant provision --provision-with configureFMW
Another cool thing is that you also could set a run option on the provisioner.
  config.vm.provision "init", run: "once", type: "shell", inline: $initScript
  config.vm.provision "installFMW", type: "shell", run: "once", inline: $installFMWScript
  config.vm.provision "configFMW", type: "shell", run: "once", inline: $configFMWScript

The run option has the following possible values:
  • "once": this is actually the default, the provisioner is only executed at first up. Or if you force it to run as described above.
  • "always": the provisioner is executed at every up. This can be used for something you want to be done every time you do up. A good one would be to start the database.
  • "never": this one is interesting. This makes the provisioner optional. That means it won't be executed, unless you ask for it. A good one would be to drop the RCU and delete the domain. So that you can reprovision the repository and the domain.
Happy Vagrant Upping!

Friday, 29 March 2019

My Seemless Linux Desktop using VirtualBox, Vagrant, MobaXterm and TotalCommander

Years ago I played around with VMWare Unity Mode in VMWare player or VirtualBox's Seemless mode. In those modes you start an application on your virtual machine, but the windows appear as running on your host system. Back in the days I ran OpenSuse on my laptop, and had Windows XP or 7 on a Virtual Machine for those cases I had to run PowerPoint.

I wasn't too enthousiastic about those modes. Of the two I found that VMWare implemented it the most transparent. But it was quite hard to work with multiple screens, and to start the applications using the embedded menu. Now, I didn't use this in a long time, so it might have been improved. But, lately I work with my VM's (mostly Oracle Linux)  using Vagrant most of the time. And I use MobaXterm to connect with them and shortly, I use it to start my Oracle tools from MobaXterm.

With the XWindows Server in MobaXterm, working with either SQLDeveloper or JDeveloper is very convenient. And it will allow me to cleanup even my local installments of SQLDeveloper or JDeveloper.

So, I have been looking for automating some startup and working on the most convenient setup for me. And I found it (for now) in the combination of:
  • VirtualBox
  • Vagrant
  • MobaXTerm 
  • And my All time favorite cockpit: TotalCommander

Let's go through them.


It must have been around 2003 that I got introduced into VMWare Workstation by my former colleague Robert. It was an eyeopener to see that you could run multiple PCs on your laptop, separating different versions of Oracle Products that could even 'talk' to eachother!
Since all is saved in a folder of files, very handy was that I could do one install and share it with colleagues.

Since Oracle acquired VirtualBox, and because of several direction-changes at VMWare (introduction and revocation of VMWare Server, VMWare player that couldn't create VMs and then it could, and then it couldn't anymore,...), after a while it made sense to me to switch to VirtualBox fully. I had a time that I had both on my laptop.

So, it's VirtualBox now and we've come to version 6.0.4 at time of writing. Downloadable from the VirtualBox Download Page. Choose the download for your platform. Installing follows the familiar NNF-Pattern (Next-Next-Finish). And don't forget to download and install the Platform independent Extension Pack. I guess this is piece of cake for you followers.


This is quite new for me, since a bit over a year now. I transferred my main installations in a Vagrant project and it's still work in progress.

I already wrote a bit about my Vagrant solutions. Last few years I worked on scripting my installations of Oracle Products. Vagrant allows me to automate the creation and provisioning of my VM's. Relieving me from the need to keep multiple VMs up to date. Sharing can be both simpler but also harder. Simpler, because I could share my vagrant project and scripts. But also harder, because the provision scripts should be placed in the proper order. I should implement it in a VCS repo. But also the install-binaries should be placed with the proper name in the proper place.

I now have a Software Stage Repository on my second laptop disk, and  separate Vagrant projects. But they all have copies of provision scripts for several products. So, database, java, SOA/BPM QuickStart, Weblogic is duplicated per particular Vagrant Project. I want to split it up in a common provision folder and a Choose&Select approach in my Vagrant Projects in a way that I have a simpel Vagrant provisioning in where I can refer to the provisioning of particular products.

Anyway,  starting up and eventual provisioning of a box is simple: just issue the vagrant up command in the folder with the vagrant file is all you have to do. Suspending  a box is done using vagrant suspend.

My respected con-colleague Maarten Smeets wrote quite a bit about Vagrant and lately about a few good tips.

Vagrant has reached version 2.2.4 recently and can be downloaded here. It follows the famous NNF-pattern. But although it allows you to choose the install directory, it is very devoted to be installed in the c:\HashiCorp\Vagrant\. I quit trying to force it elsewhere.


Many of my even respected coworkers stick with their all-time favorite Putty. Putty stays ubiquitous. It is simple, but I allways have found it a bit archaic. I thought I once noticed that the support was  terminated. However, just now they just released the stunning version number of 0.71. I don't mean to be sarcastic, and Putty has it's own right of existance. But give MobaXterm a try. It's loaded with nice features, including an SCP client that can follow your SSH session. And as said, also a XWindows Server. So, connect with ssh to your Linux server and run jmc, visualVM, Oracle Universal Installer, Weblogic Configurator, JDeveloper, SQLDeveloper, etc. etc. and the UI will pop-up on your desktop. Also MobaXterm includes cygwin so you're able to run a terminal session on Windows. It even allows you to do ps -ef to show your running windows apps!

I also discovered that really easy way to implement a tunnel with MobaXterm!

You can download it here, in a portable and an installer version. I choose the portable. The free edition includes games (why?) and  a limit number of sessions, macros and tunnels. But the professional edition only costs a few bucks/euros.


Ever since the introduction of Windows '95, I disliked the Windows Explorer. Luckily I soon discovered Windows Commander, under pressure of Microsoft, renamed to Total Commander. And it's even  a better name, because it's about the second tool I install on a new Windows Desktop. Just after Firefox, to be able to close IE/Edge...

It's my cockpit, allows me to navigate to hot folders quickly, introspect files, navigate through archives, edit them or unpack them, multi-rename files, compare files, etc., etc. I just don't make coffee with it. One of the nice features is the button bar, where you can launch applications. And this is the thing I use for this blog.

Tie it all together

I realize that I overloaded you with sales talk about my favorite tools.

After installing all the tools and having your Vagrant project in place all can be tied together.

I have a project that provisions a VM with an Oracle Database, SQLDeveloper, SOASuite, and BPM QuickStart.

My TotalCommander Toolbar looks like:

You can right-click anywhere in the toolbar to edit it, create new buttons. A 'button' that is left empty (no command) is presented as a separation bar.

Startup & suspend the VM

Let's take a look at the Vagrant SOA Start button:

It's simple: the command is vagrant up and important here is that it should be executed in the folder where the vagrant file resides. I provide a tool tip, and I created an icon file from the Vagrant Logo using my favorite image app IrfanView.

Clicking the button will fire up the VM and potentially provision it. I copied the button to create a button to suspend the VM. The command there is vagrant suspend. But for the rest it is exactly the same.

MobaXterm local terminal

When you start MobaXterm you'll get to:
When you click on the 'Start local terminal' button, you get a shell window running in the user home folder that is presented by the tool:

You can ssh to a remote server from this terminal. Of course you can create a session to a remote server. Doing so for the first time, logging on to the particular user, allows you to save the password for that user. I've already done that, and then I can do a ssh oracle without the need to provide a password.

Start Database

On my remote server that is started already (using the TotalCommander Button), I have a script that starts the database.

When the VM is started with vagrant, by default it fires up an ssh deamon on port 2222. The command to start my database on the remote server is:
ssh oracle@localhost -p 2222  /home/oracle/bin/

And as you can see in the screendump: I put that in a script in the home folder.

MobaXterm provides several commandline options, that allows you to run  a script command at startup of MobaXterm. That is what I used to create a StartDatabase Button:
The command is just MobaXterm (I should put MobaXterm in a version-less folder name, with a version-less executable, or create a script for that).
As parameters I provided -newtab ./ This is to ensure that the script is started on a new tab in MobaXterm, in a new potential session. Little side affect is that it creates a tab on the MobaXterm every button click. So, I might end up closing a few tabs...

Start SQLDeveloper and JDeveloper (BPM QuickStart)

To start SQLDeveloper I have a script with the following content:
nohup ssh oracle@localhost -p 2222  /home/oracle/bin/  > sqldev.out 2>&1 &

And the button looks like:

Similarly, I have a button to start JDeveloper. And it alls and up in the following desktop:

It might need some tweeking. But for now I love it and it works like a charm.