Wednesday 28 November 2018

Using ANT to investigate JCA adapters

My current customer has a SOA Suite implementation dating from the 10g era. They use many queues (JMS serves by AQ) to decouple services, which is in essence a good idea.

However, there are quite a load of them. Many composites have several adapter specifications that use the same queue, but with different message selectors. But also over composites queues are shared.

There are a few composites with ship loads of .jca files. You would like to replace those with a generic adapter specification, but you might risk eating messages from other composites. This screendump is an anonymised of one of those, that actually still does not show every adapter. They're all jms adapter specs actually.

So, how can we figure out which queues are used by which composites and if they read or write?
I wanted to create a script that reads every .jca file in our repository and write a line to a CSV file for each JCA file, containing:
  • Name of place
  • Name of the jca file
  • Type of the adapter
  • Is it an activation (consume) or interaction (type)
  • What is the location (eis JNDI )
  • Destination
  • Payload
  • Message selector (when consumption)
Amongst some other properties.

Using ANT to scan jca Files

I found that ANT is more than capable for the job. I put my project on GitHub, so you can all the files there.

First let's talk the first parts of scanJCAFiles.xml.

Since I want to know the project that the .jca file belongs to, I first select all the .jpr files in the repository. Because the project folders are spread over the repository, although structured they're not neatly in a linear row of folders, finding the .jpr files gives me a list of all the projects. 
  <!-- Initialisatie -->
  <target name="clean" description="Clean the temp folder">
    <delete dir="${jcaTempDir}"/>
    <mkdir dir="${jcaTempDir}"/>
  <!-- Perform all -->
  <target name="all" description="Scan All SOA applications" depends="clean">
    <echo file="${outputFile}" append="false"
          message="project name,jcaFile,adapter-config-name,adapter-type,connection factory location,endpoint type,class,DestinationName,QueueName,DeliveryMode,TimeToLive,UseMessageListener,MessageSelector,PayloadType,ObjectFieldName,PayloadHeaderRequired,RecipientList,Consumer${line.separator}"></echo>
    <foreach param="project.file" target="handleProject" delimiter=';' inheritall="true">
        <fileset id="dist.contents" dir="${svnRoot}" includes="**/*.jpr"/>
Side note, as can be seen in the snippet, I re-create a folder for transformed jca files (as described later) and I create a new output file, in which I write a header row with all the column names, using echo to a file with the append properties to false.

So, I do a foreach over a fileset, using the svnRoot property in the, that includes ever .jpr file anywhere in the structure. For each file the handleProject target is called with the file in the project.file property. Foreach is an antcontrib addition to ANT. So you need to add that as a task definition (one thing I do as a first thing).

  <taskdef resource="net/sf/antcontrib/antlib.xml">
      <pathelement location="${ant-contrib.jar}"/>

With the name of the .jpr file I have the name of the project and the location:
  <target name="handleProject">
    <echo message="projectFile: ${project.file}"></echo>
    <dirname property="project.dir" file="${project.file}"/>
    <echo message="project dir: ${project.dir}"></echo>
    <basename property="" file="${project.file}" suffix=".jpr"/>
    <foreach param="jca.file" target="handleJca" delimiter=";" inheritall="true">
        <fileset id="dist.contents" dir="${project.dir}" includes="**/*.jca"/>

In this snippet the dirname ANT task trims the filename from the project.file property, to provide me the project folder into the project.dir property. The can be determined from the project.file using the basename task. Nice touch is that it allows you to trim the suffix (.jpr) from it. Within the project location I can find all the .jca file, and in the same way as the .jpr files I can use a foreach on the project.dir and call the handleJca target for each .jca file.

Using XSL to pre-process the jca files

Fortunately, jca files are simply XML files and ANT turns out to be able to read XML as a property file, using the xmlproperty task, which came in handy. Those properties can be appended to an output file using echo, very easily.

However, there are two main problems with the structure of the jca files:
  1. The jca files for the interaction type (the write kind) are different from the activation type (the read kind), So I would need to distinguish those.
  2. The properties like DestinationName, payload and message selector are name value pair properties in the .jca file. The  interprets the names of the properties as separate property values with the name. I can't select specifically the Destination Name for instance.
So I decided to create an xml stylesheet to transform the JCA files to a specific schema, that merges the endpoint interaction and activation elements and has the properties I'm interested in as separate elements. To do so, I created an xsd from both types of jca files. JDeveloper can help me with that:
Just follow the wizard, but emtpy the target namespace. As said I did this for both kinds of jca files (the interaction and activation kinds) and merge them into jcaAdapter.xsd with a xsd:choice:

Out of that I created jcaAdapterProps.xsd where the xsd:choice elements are merged into spec element. I changed the targetnamespace and created specific property elements:
That allowed me to create the XSL Map jcaAdapter.xsl easily:

For the xmlproperty task it is important that the resulting xml is in a default namespace and that the elements depend on the default namespaces, they should not reference a specific namespace (not even the default one).

With that I can finish off with the handleJca target of my script:
  <target name="handleJca">
    <basename property="" file="${jca.file}"/>
    <property name="jca.file.props" value="${jcaTempDir}/${}.props"/>
    <echo message="Jca File: ${}"></echo>
    <xslt style="${jcaPropsXsl}" in="${jca.file}" out="${jca.file.props}"/>
    <xmlproperty file="${jca.file.props}" collapseattributes="true"/>
    <!-- see -->
    <property name="cf.location" value="${adapter-config.connection-factory.location}"/>
    <property name="ep.class" value="${adapter-config.endpoint.spec.className}"/>
    <property name="ep.type" value="${adapter-config.endpoint.spec.type}"/>
    <property name="ep.DestinationName" value="${adapter-config.endpoint.spec.DestinationName}"/>
    <property name="ep.DeliveryMode" value="${adapter-config.endpoint.spec.DeliveryMode}"/>
    <property name="ep.TimeToLive" value="${adapter-config.endpoint.spec.TimeToLive}"/>
    <property name="ep.UseMessageListener" value="${adapter-config.endpoint.spec.UseMessageListener}"/>
    <property name="ep.MessageSelector" value="${adapter-config.endpoint.spec.MessageSelector}"/>
    <property name="ep.PayloadType" value="${adapter-config.endpoint.spec.PayloadType}"/>
    <property name="ep.QueueName" value="${adapter-config.endpoint.spec.QueueName}"/>
    <property name="ep.ObjectFieldName" value="${adapter-config.endpoint.spec.ObjectFieldName}"/>
    <property name="ep.PayloadHeaderRequired" value="${adapter-config.endpoint.spec.PayloadHeaderRequired}"/>
    <property name="ep.RecipientList" value="${adapter-config.endpoint.spec.RecipientList}"/>
    <property name="ep.Consumer" value="${adapter-config.endpoint.spec.Consumer}"/>
    <echo file="${outputFile}" append="true"
With the xslt task the jca file is transformed to the jcaTempDir folder. And using the xmlproperty task the transformed .jca is read as an xml property file. Because the property references are quite long, I copy them in a shorter named property and then echo them as a comma separated line into the outputFile using the append attribute to true.

Note that I used collapseattributes attribute set to true.


And that is actually about it. ANT is very handy to find and process files in a controlled way. Also the combination with XSL makes it powerfull. In this project I concentrated on JMS and AQ adapters, as far as the properties are concerned. But you can extend this for DB Adapters and File Adapters, etc. quite easily. Maybe even create an output file per type.

I can't share the output with you, due to company policy contraints. Just try it out.

Thursday 22 November 2018

How to query your JMS over AQ Queues

At my current customer we use queues a lot. They're JMS queues, but in stead of Weblogic JMS, they're served by the Oracle database.

This is not new, in fact the Oracle database supports this since 8i through Advanced Queueing. Advanced Queueing is Oracle's Queueing implementation based on tables and views. That means you can query the queue table to get to the content of the queue. But you might know this already.

What I find few people know is that you shouldn't query the queue table directly but the accompanying AQ$ view instead. So, if your queue table is called MY_QUEUE_TAB, then you should query AQ$MY_QUEUE_TAB. So simply prefix the table name with  AQ$. Why? The AQ$ view is created automatically for you and joins the queue table with accompanying IOT tables to give you a proper and convenient representation of the state, subscriptions and other info of the messages. It is actually the supported wat of query the queue tables.

A JMS queue in AQ is implemented by creating them in queue tables based on the Oracle type$_jms_text_message type.

That is in fact a quite complex type definition that implements common JMS Text Message based queues. There are a few other types to support other JMS message types. But let's leave that.

Although the payload of the queue table is a complex type, you can get to its attributes in the query using the dot notation. But for that it is mandatory to have a table shortname and prefix the view columns with the table shortname.

The$_jms_text_message has a few main attributes, such as text_lob for the content and header for the JMS header attributes. The header is based on the type$_jms_header. You'll find the JMS type there. But also the properties attribute based on$_jms_userproparray. That in its turn  is a varray based on aq$_jms_userproperty. Now, that makes it a bit complex, because we would like to know the values of the JMS properties, right?

We use those queues using the JMS adapter of SOA Suite and that adds properties containing the composite instance ID, ECID, etcetera. And if I happen to have a message that isn't picked up, it would be nice to know which Composite Instance enqueued this message, wouldn't it?

Luckily, a Varray can be considered as a collection of Oracle types. And do you know you  can query those? Simply provide it to the table() function and Oracle threats it as a table. When you know which properties you may expect, and their types, you can select them in the select clause of your query.  I found the properties that are set by SOA Suite and added them to my query. But you could find others as well.

Putting all this knowledge together, I came up with the following  query:

select qtb.queue
, qtb.msg_id
, qtb.msg_state
,qtb.user_data.header.type type
,qtb.user_data.header.userid userid
,qtb.user_data.header.appid appid
,qtb.user_data.header.groupid groupid
,qtb.user_data.header.groupseq groupseq
--, properties
, (select str_value from table ( prp where = 'tracking_compositeInstanceId') tracking_compositeInstanceId
, (select str_value from table ( prp where = 'JMS_OracleDeliveryMode') JMS_OracleDeliveryMode
, (select str_value from table ( prp where = 'tracking_ecid') tracking_ecid
, (select num_value from table ( prp where = 'JMS_OracleTimestamp') JMS_OracleTimestamp
, (select str_value from table ( prp where = 'tracking_parentComponentInstanceId') tracking_prtCptInstanceId
, (select str_value from table ( prp where = 'tracking_conversationId') tracking_conversationId
,qtb.user_data.text_lob text
from AQ$MY_QUEUE_TAB qtb
where qtb.queue = 'MY_QUEUE'
order by enq_timestamp desc;

This delivered me an actual message that was not picked up by my process. And I could use  the property tracking_compositeInstanceId to find my soa composite instance in EM.

Very helpful if you are able to pause the consumption of your messages.

This also shows you how to query tables with complex nested tables.

Monday 19 November 2018

URL Resolving in an Enterprise Deployment

A few blogs ago I wrote about issues we encountered with persistence of settings in an Enterprise Deployment with seperate Admin and Managed Server domains.

For one of the problems, the  mdm-url-resolver.xml, used to store the Global Tokens, we had a Service Request with support. After over a year, we got an answer from development, that as per design SOA updates will only update the mdm-url-resolver.xml in the soa managed server.

Besides the workaround in my previous article, there is a Java custom system property that refers to the mdm-url-resolver.xml you want to use: 

With this property set, SOA Suite will use this file, and does not have it affacted by the domain config.
I did not try it myself yet, but I think it is advisable to put this file on a shared disk. Otherwise you would need to create a copy of it for each managed server and update every one.
Unfortunately I did not find this Java system property in the documentation. I did find a blog that mentions it, but not where can be found the documentation.

So, for global tokens this seems a workable approach. But the same behavior we saw with the UMS Driver property files. I don't have a property like this for those property files. As soon as I find it, I will update this blog post.

Monday 5 November 2018

List the server group memberships of your domain

Last few years I posted on installation of Fusion Middleware. One of the features of FMW12c is the concept of Server Groups. As can be read here: 'Server groups target Fusion Middleware applications and services to one or more servers by mapping defined application service groups to each defined server group.'

For Fusion Middleware Topology I found this article on SOASuite 12c topology from the A-team very helpful.

Server groups are set at creation of the domain. Relating to SOA and OSB it is important to determin where the WebServices Manager Policy Manager is targetted. By default SOA and OSB servers have the SOA-MGD-SVRS or OSB-MGD-SVRS-COMBINED respectively, which means that those servers/clusters have the Policy Manager targetted automatically. If OSB or SOA is the only component in the domain, then this is sufficient. But if you have a domain that combine those components (and/or BAM or MFT), then there should be a separate WSM cluster that target the PM, since you want it targetted to only one cluster. In that case SOA and OSB should have the server groups SOA-MGD-SVRS-ONLY and  OSB-MGD-SVRS-ONLY.

But how to know if you have targetted the proper server groups to your servers? You can use the getServerGroups([serverName]) wlst command for that.

I created a simple wlst script for it. Save the following script as Change the domainHome variable at the top of the scripts to the location of your domain (I did not bothered to parameterize this...):

# List ServerGroups for a domain
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.0, 2018-11-05
# Usage:
#   wlst
# When       Who                      What
# 20181105   Martien van den Akker    Create
import sys, traceback
scriptName = sys.argv[0]
def main():
  if (len(allServers) > 0):
    for wlserver in allServers:
      wlserverName = wlserver.getName()
      print('Groups of server: '+wlserverName)
      for serverGroup in serverGroups:
# Main

The getServerGroups() command is an wlst offline commmand, so you need to read the domain for it.
You can remove groups and or set new groups using the setServerGroups() command. If you do so, you need to update the domain (updateDomain()) and then close the domain (closeDomain()). And of course you need to restart your domain. This means by the way, also the AdminServer, since it needs to re-read the domain. It is even recommended to stop the domain before updating it in offline mode.

By the way, my scripts to create a FMW Domain also make use of the setServerGroups() command, as can be read in this article, but if you reuse them, make sure you have the correct ServerGroups set (Maybe I should parameterize those too).

Friday 2 November 2018

MobaXterm 11.0

Recently I wrote about MobaXterm as a welcome replacement of Putty. Looking for a 64-bit version of MobaXterm, I found that they released version 11.0 only yesterday.

Nice: one of the improvements says:
  • Improvement: updated PuTTY-based SSH engine to the latest version
Another welcome improvement:
  • Improvement: improved SFTP / FTP / S3 sessions performances, especially when remote folder contains many files/folders
 However, did not found a specific 64 bit version.

Friday 26 October 2018

Recursion in XSLT

Last week I helped someone on the Oracle community forums with transforming a comma separated string to a list of elements. He needed this to process each element in BPM Suite, but it is a use case that can come around in SOA Suite or even in Oracle Integration Cloud.

You would think that you could do something like a for-each and trimming the element from the variable.


One typical thing with XSLT is that variables are immutable. That means that you can declare a variable and assign a value to it, but you cannot change it. So it is not possible to assign a new value to a variable based on a substring of that same variable.

To circumvent this, you should implement a template that conditionally calls itself until an end-condition is met. This is a typical algorithm called recursion. Recursion is a way of implementing a function that calls itself, for example to calculate the faculty of a number. Recursion can help circumventing the immutability of variables, because with every call to the function you can pass (a) calculated and thus different value(s) through the parameter(s).

I wrote about this earlier, but last week a co-worker asked a similar question, but just the other way around: transforming a list into a comma separated string.

So, apparently it's time to write an article about it.

Transforming CSV to a List

I refactored the xsd's from the question as follows. First the source xsd:
<?xml version="1.0" encoding="windows-1252" ?>
<xsd:schema xmlns:xsd="" xmlns:tns=""
            targetNamespace="" elementFormDefault="qualified">
  <xsd:element name="ApprovalRoute" type="tns:approvalRouteByInvoiceNatureResponse"/>
  <xsd:complexType name="approvalRouteByInvoiceNatureResponse">
      <xsd:element type="xsd:string" name="approvalRoute" minOccurs="0"/>
      <xsd:element type="xsd:boolean" name="autoApprove" minOccurs="0"/>

And the target schema is:
<?xml version="1.0" encoding="windows-1252" ?>
<xsd:schema xmlns:xsd="" xmlns:tns=""
            targetNamespace="" elementFormDefault="qualified">
  <xsd:element name="ApprovalRoute" type="tns:ApprovalRouteType"/>
  <xsd:complexType name="ApprovalRouteType">
      <xsd:element name="Approver" type="xsd:string" minOccurs="0" maxOccurs="unbounded"/>

To start with, we have an ApprovalRoute element based on a complex type with the approvalRoute sub-element being the comma-separated list of approvers. Then as a target we have an ApprovalRoute, based on a list of Approver elements.

I generated the following source xml to transform:
<?xml version="1.0" encoding="UTF-8" ?>
<ApprovalRoute xmlns:xsi=""
               xsi:schemaLocation=" SOA/Schemas/Approvals-Source.xsd"

Now, we need to split the approvalRoute value in a part before the first comma, and after the first comma. The value before the first comma can be put in an element. But the remainder has to be fed into the same template again. Then, at the end there is no comma in the remainder, so the part before the comma will be empty. There is no comma anymore, so we should not call the template with the remainder, but simply put the remainder in an element. Therefor, the non-existence of the comma can be the end-condition.

Remember, using recursion, you should always have a finalizing condition. To be honest, in my first piece of code in the answer of the question, I forgot about that. But, to my defence: I just put it together by heart and haven't been able to test.

The explanation above results in the following template:
<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0"
                xmlns:ns0="" xmlns:xsl="
  <!-- -->
  <xsl:template match="/">
      <xsl:call-template name="parseDelimitedString">
        <xsl:with-param name="delimitedStr" select="/ns0:ApprovalRoute/ns0:approvalRoute"/>
  <xsl:template name="parseDelimitedString">
    <xsl:param name="delimitedStr"/>
    <!-- -->
    <xsl:variable name="firstItem" select="substring-before($delimitedStr, ',')"/>
    <xsl:variable name="restDelimitedStr" select="substring-after($delimitedStr, ',')"/>
      <xsl:value-of select="$firstItem"/>
      <xsl:when test="contains($restDelimitedStr, ',')">
        <xsl:call-template name="parseDelimitedString">
          <xsl:with-param name="delimitedStr" select="$restDelimitedStr"/>
          <xsl:value-of select="$restDelimitedStr"/>

(I created this as an XSL Map, but removed the comments that were included by JDeveloper.
I tested this with the following input:
<?xml version="1.0" encoding="UTF-8" ?>
<ApprovalRoute xmlns:xsi=""
               xsi:schemaLocation=" SOA/Schemas/Approvals-Source.xsd"

And this resulted in the following output:
<?xml version = '1.0' encoding = 'UTF-8'?>
<tns:ApprovalRoute xmlns:tns="" xmlns:xsi="" xsi:schemaLocation=" file:/D:/Projects/2018-ODC/XSL-Demo/XSL-Demo/SOA/Schemas/Approvals-Target.xsd">

This I used for input for the following xslt.

The other way around: List to CSV

For didactional reasons I'll show the other way around too. Although, we'll see that this can be done easier.

In this case I mean to loop over a series of elements, starting with an index of 1, and adding the elements to a partial string. That means I have 3 parameters:
  • loopApprovers: the parent element, containing all the elements to loop over
  • index: the loop index, with a default of 1
  • partialApprovalRoute: the partial CSV list, defaulted to an empty string

The template loopApprovers can be called with only the approvalRoute. Then with an index of 1, the template is called recursively the first time, with a partialApprovalRoute assigned with the first Approver occurence and an index increased with 1.
For the other occurences where index > 1 and index <= count of elements, the template is called again recursively, but with an increased index and the indexed element added to the partialApprovalRoute separated with a comma.
Then the end situation is when the template is called where index exceeds the count of elements. Then just the partialApprovalRoute is 'returned'  (by the value-of instruction) where it is substringed to a 20000 characters:

<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0"
  <xsl:template match="/">
        <xsl:call-template name="loopApprovers">
          <xsl:with-param name="approvalRoute" select="/ns0:ApprovalRoute"/>
  <xsl:template name="loopApprovers">
    <xsl:param name="approvalRoute"/>
    <xsl:param name="index" select="1"/>
    <xsl:param name="partialApprovalRoute" select="''"/>
      <xsl:when test="number($index)=1">
        <xsl:call-template name="loopApprovers">
          <xsl:with-param name="approvalRoute" select="$approvalRoute"/>
          <xsl:with-param name="index" select="$index+1"/>
          <xsl:with-param name="partialApprovalRoute" select="$approvalRoute/ns0:Approver[1]"/>
      <xsl:when test="number($index)> 1 and number($index)&lt;=count($approvalRoute/ns0:Approver)">
        <xsl:call-template name="loopApprovers">
          <xsl:with-param name="approvalRoute" select="$approvalRoute"/>
          <xsl:with-param name="index" select="$index+1"/>
          <xsl:with-param name="partialApprovalRoute"
        <xsl:value-of select="substring($partialApprovalRoute,1,20000)"/>

Simpler transformation from list to csv

As can be found here for instance, a for-each does not necessarily need to return an element. It can return just a value. So, it can be a bit simpeler:
<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="1.0"
  <!-- -->
  <xsl:template match="/">
        <xsl:call-template name="loopApprovers">
          <xsl:with-param name="approvalRoute" select="/ns0:ApprovalRoute"/>
  <xsl:template name="loopApprovers">
    <xsl:param name="approvalRoute"/>
    <xsl:variable name="approvalRouteCsv">
      <xsl:for-each select="$approvalRoute/ns0:Approver">
        <xsl:value-of select="concat(substring(.,1,20000),',')"/>
    <xsl:value-of select="substring($approvalRouteCsv,1,string-length($approvalRouteCsv)-1)"/>


Understanding Recursion with XSLT will help you with solving much complexer problems in transformations. The last example of transforming a list to a comma separated list is of course structural easier. But the recursive variant allows for more calculations or conditional processing.

Friday 12 October 2018

Enable X11 on Oracle Cloud Infrastructure

Today my colleague was starting with the installation of Oracle Database on the Oracle Cloud Infrastrcture, for a customer. He phoned me for help on enabling VNC to have a graphical UI to install the database.

Install an ssh client with XServer emulator

Most of my co-workers of around my age, have grown up with Putty. And apparently we as computer consultants are quite hooked to our tools. I know, only yesterday I mentioned it to a co-worker, that everywhere I come the two first tools I install are:
  1. Firefox (unfortunately to be downloaded using IE or Edge, it's from then on immediately the last time I use those browsers, as far as I'm concerned)
  2. Total Commander (of course downloaded with Firefox): I really hate Windows Explorer, ever since it is introduced with Windows 95/NT. Windows 3.1 had FileManager (yes I"m that old...)

    I liked that much better than the successor Windows Explorer. But little filemanager beat the revival of Norton Commander: Total Commander. I use it almost all of my carreer, and so much that quite early on I bought a key for what we would call 'an apple and an egg'.

    But about the the third tool I install is:
  3. MobaXterm:this tool is so much richer than Putty. It includes a SCP/SFTP client that can follow your terminal. Which means, every time you cd to a certain folder, your SFTP window will cd to it as well. I like the fonts, the looks more, it's more comfortable. But it also includes a XServer, with no additional install! It's free, with a limited number of sessions though. However, the cost for a lifetime license is really low.
So, as soon as I see a co-worker work with Putty, I recommend MobaXterm (no, I don't have shares).
Quite because of the same reasons I would recommend Total Commander over Win Explorer.

When connecting to a server, MobaXterm, by default (although you can uncheck it) will do X11Forwarding.

However, in the case of my colleague, unfortunately X11-forwarding was disabled:

We installed xclock which additionally installed several X-libraries. We checked XForwarding in /etc/ssh/sshd_config. All with no luck. But, we were so close. In the end, the answer (thanks Radu) was in this whitepaper. We needed to set the X11UseLocalhost property to no in /etc/ssh/sshd_config.

The complete setup, following the whitepaper:
  1. Log into the instance (obviously)
  2. Configure SSHD to not use localhost for X11:
    1. Open /etc/ssh/sshd_config in your favorite editor.
    2. Search for the line that has X11UseLocalhost (it’scommented out).
    3. Uncomment  the line by removing the # from the beginning.
    4. Set the property to no.
    5. While you're at it, check if the property X11Forwarding is set to yes.
    6. Save the file.
    7. Restart the ssh daemon by executing: sudo systemctl restart sshd
  3. Install xauth by executing: sudo yum-y install xauth
  4. Install xterm (used to verify X configuration) by executing: sudo yum -y install xterm
  5. Also isntall xclock for simple testing of the X Forwarding: sudo yum -y install xclock
Now, reconnect using MobaXterm, and you should see that X11-forwarding is enabled:

When running xclock on the remote terminal will show a clock on your local desktop.
As can be seen in the screendump, you might run into the message 'Missing charsets in String to FontSet conversion'. This can be solved following this hint by RedHat. It is caused by improper locale environment variable. Run the following:
export LC_ALL=C

You might want to add this to your ~/.bash_profile. Then run xclock again.

But, but, but... I can't log on to oracle...

To make things slightly more complicated, in most situations, you can't logon as the oracle user. You get a user to connect, and then have to sudo to oracle. In those cases you ned to redirect your xauth.

So, perform the following:
  1. Connect as the user provided 
  2. List your xauth by executing: xauth list $DISPLAY (you might need to check the DISPLAY variable)
    This would provide a line like:
    darlin123/unix:11  MIT-MAGIC-COOKIE-1  1231a6f34cca12394d3233456230df26
  3. Sudo to oracle: sudo su - oracle
  4.  Then set the DISPLAY using the port from the xauth list above:
    export DISPLAY=darlin123:11
    (In some examples explaining this X forwarding across users, you might see export DISPLAY=localhost:11. But, remember: we disabled the use of localhost above).
  5. Then add the autorisation with:
    xauth add darlin123/unix:11  MIT-MAGIC-COOKIE-1  1231a6f34cca12394d3233456230df26

    Also take over the port in the display, as well as the GUID (in green) from the xauth list.

That should work!

ODC Appreciation Day : SOA Suite 12c and the Community

Yesterday Tim - - Hall had the ODC Appreciation Day initiative.

I gladly join in, however yesterday I hadn't had the change to write something. I did a tweet to remember it:
But today, on the day-after I do like to write something.

SOA Suite 11g/12c

Just yesterday I mentioned a coworker that I indeed still like SOA Suite.
I worked at Oracle Consulting when Oracle acquired Collaxa in 2004, and some other companies whose technologies were at the base of Oracle SOASuite. Since then I work with SOASuite and BPEL, and I still like it.

When in 2008 BEA was acquired, SOASuite 11g was released in 2009 based on Weblogic. It was actually delayed because of making Weblogic as a strategic platform.

The thing with SOA Suite11g and 12c, based on Weblogic, is that it has become a quite draconic system, as I used to call it. The footprint is quite large, both on disk as well as in memory.

I understand the MicroServices up coming for two reasons:
  • Especially when it comes together with containerization, MicroService applications startup quite fast. SOA Suite startups can last 10 minutes to about a quarter of an hour.
  • We must admit that we messed up SOA projects. Yes, we made them quite complex and we did not manage to do proper Service Governance, not even Service Registration, Service Reuse, etc. Driven by project-funding, time-pressures, lack of enforcing proper scrum project procedures, etc., we deliver mostly one-purpose services, not following the proper SOA principles. Also yesterday, at my current customer, my co-worker and I concluded that there are mostly point-to-point integrations. Back in 2008 I invented the term POPOTOPI for it.
Are MicroServices the answer then? I'm not really sure, I must admit:
  • It seems to me that with MicroServices initiatives we're going to build services in Java again. Indeed with frameworks like SpringBoot and JPA. But still we seem to program the logic in Java again. And when we're able to mess up BPEL with lacking standards (no use of Scopes and local variables for instance), Java allows us for even bigger messes. Are we really going to program services in Java frameworks with 50+ possible elements?
    Where end user Application development earlier transformed from for instance Oracle Forms to Java/ADF, now transforms to Low Code (VBCS, OutSystems, Oracle Forms 😉?, etc.) I expect a same transformation in MicroServices.
  • With containerization we apparently create CI/CD initiatives delivering the complete Service Application with an app server (often Tomcat) complete in a container. So a new bug-release deliveres a container with a complete installation. Which add to a large library of the different versions of the containers.
  • MicroServices have good thoughts underneath it, as said it is an answer to problems encountered in SOA projects. But why not build MicroServices in SOA Suite or OSB? SOA Suite and OSB are the current LowCode environments in Services. 
I don't see that the technology is the problem here. I've seen several times with failing projects that the problem is not the SOA Technologies, but more the project politics and governance.

So, I still like working with SOA Suite. However, I would like some improvements:
  • JDeveloper could be made more stable. Although on the same IDE platform, SQLDeveloper is as stable as a castle. JDeveloper, however, is stuffed with loads of designers, addons, build by numerous teams. But, please, prevent us from all those Null-pointer exceptions, and exception dialogs asking me to save my work and quite or continue.
  • In a SOA Suite installation I encounter loads of jars of different version for the same library at different location. It seems to me that we could get rid of a significant amount of those.
    The main differentiator between Oracle 9i database and Oracle 10g was the foot print. I think with FMW (SOA Suite, but also OSB and other products), could take advantage of a great "spring cleaning". 
  • One of the new features of Java 9, is the modularisation. In SOA Suite we have loads of adapters installed (some of them not activated), and other functionality that aren't used. So a modularisation of Weblogic and FMW products on top of it would be a great idea. Only install what I need and provide a package manager that allows me to install let's say an SAP Adapter or SAML (either 1 or 2.0) support when I need it later on.

 How about Cloud?

Yes, cloud. Of course cloud is important, and I completely understand why Oracle has a near 100% focus on Cloud. But, I seem to be one of the few people that has difficulty with believing that in let's say 5 years 100% of all our customers are for 100% of their business 'in the cloud'. Let alone, the Oracle Cloud. And to straighten my statements: I don't see IaaS as actual Cloud. To me, IaaS is the same as 'On Premise', but in another Data Center. Of course Oracle might have different License policies if you want to run SOA Suite on Amazon or Azure or a local IaaS provider, with regards to  running on Oracle Cloud. But I don't encounter any customers that won't run their software on Virtual Machines (VMWare ESX or alike). So even in their own DataCenters, software is run virtualized. When not on an external IaaS provider, they are essentially their own IaaS provider.

And although Oracle Integration Cloud is promising, there is much I'd rather do on SOA Suite. And I expect that there are several good reasons for current customers to stay on a On Premise or local IaaS serviced SOA Suite installation.

Also, Oracle Integration Cloud, and other Oracle PaaS-es, are based on the exact same platforms and engines (BPEL/BPM Process Engine, Business Rules, WSM, Service Bus, etc., etc.). So, my above suggested improvements would improve the PaaS-es as well.

So, Oracle Please add SOA Suite and OSB to the Fusion Middleware 19c category in the Supported Systems and Configurations:
And as has been asked on the forums this week a few times: please add support for Java 9/10/11 on the upcoming Weblogic and FMW releases.

Since a few years I try to keep up and be active on the forums for SOA, Weblogic, etc. It's great to be of any help, answering questions and collect points (to me it's a bit of playing something like Forge of Empires). I sometimes get questions via email. But it's better to go over there. You could go to Stack Overflow, but why not form a community at Oracle's own community page? Come over and join us. Meet me there. But when you do, please:
  • Edit your profile and enter a proper name: I do like to know how you want to becalled, instead of user 12345656 of alike,
  • Ask a question together with your error-message, in stead of just posting your error message. It happens that some one just posts the error message and I often must refrain my self from answering "Congratulations!".
  • I'm putting in my time answering questions. The only rewards I get for it are points. Please, give me my points by marking my answers as help-full (50 points) or 'Answered' (even 100 points and I can register them for my ACE review).

I'm looking forward for another year of Oracle Fusion Middleware and other infrastructure technologies in the Oracle Developer Community.

Thursday 4 October 2018

Persisting of settings in a SOA Suite Enterprise Deployment

About a year ago, at my previous customer, a co-worker and I encountered and described a persistence problem with setting the Global Tokens in SOA Suite.

What are Global Tokens again?

The problem with a middleware product as Oracle Service Bus, SOA Suite (and the same probably counts for MuleSoft, or any other integration tool) is that when you move an integration through the development lifecycle from development, to test, preproduction and production, you need to update the endpoints. When I have an integration with a (BPEL) Process that does a check-in of a document in WebCenter Content, for instance, then on the test environment it should do the check-in to another WCC server than on pre-production or production. We don't want to have our test documents in production, do we?

To solve that, in OSB we have customization files, and in SOA Suite 11g and onwards, we use config plans. But, in 11g PatchSet 6 (, SOA Suite introduced Global Tokens. That way you can create a token that refers to the WCC host, eg. ${wcc_url}, and use that as a reference in your binding properties.

These properties can be set using Enterprise Manager FMW Control 12c:
which lead to:
Where you can add tokens or import a file with the tokens.

These settings are stored in the mdm-url-resolver.xml file, in the $DOMAIN_HOME/config/fmwconfig folder.

What about Enterprise Deployment?

The Enterprise Deployment Guide of SOA Suite 12c is quite complex. But in short, as we implemented it, we installed Fusion Middleware Infrastructure and SOA Suite on one node/host. Then, of course, ran the Repository Creation Utility, and configured the domain for the AdminServer. That domain was configured on a shared disk, let's say, /u01/data/oracle/domains/soa1_domain. Then it is cloned to cater for the managed servers, using pack/unpack, to local storage, for instance, /u02/data/oracle/domains/soa1_domain. In short we have 2 domain homes:
  • ASERVER_DOMAIN_HOME=/u01/data/oracle/domains/soa1_domain
  • MSERVER_DOMAIN_HOME=/u02/data/oracle/domains/soa1_domain
Where  /u01 is mounted on a shared disk and /u02 on local storage.

So, the AdminServer runs from the ASERVER_DOMAIN_HOME domain, that is on shared storage. This way, when the host running the AdminServer goes down, the AdminServer can be brought up on the second host. The Managed Servers run on a clone of the domain on local storage.

Side note: in 12c we have a per-domain NodeManager by default. So cloning the domain, implicitly clones the nodemanager config. And running that against another adapter, allows for a nodemanager for the Admin server and one for the ManagedServers.

Why is this important? Well, this allows for a High Available setup, including functionality as Zero Downtime Patching.

What is the problem then?

Updating the Global Tokens is done in FMW Control, that runs on the AdminServer. It stores the properties in the mdm-url-resolver.xml. But, which particular mdm-url-resolver.xml file? Well, the changes are stored in the one in $MSERVER_DOMAIN_HOME/config/fmwconfig!
After that you need to restart the SOA Server, to get the properties loaded. And then something very smart happens. When starting the SOA Server, the AdminServer sends it's copy from the $ASERVER_DOMAIN_HOME/config/fmwconfig to the SOA Server. And so the changes are cleared by the version from the AdminServers domain!

So, in an Enterprise Deployment configuration of the SOA Suite a restart of the SOA Server, will clear the changes of the Global Tokens.

But there is more!

As I wrote above, we found this a year ago. And we created a Prio 1 Service Request. The issue is very straight forward, reproducable, and in the status Development Working for about a year now:
(I'm not writing this to bash Support by the way. No offense intended, altough I would really like a patch by now...)

But, today another co-worker and I encountered a very similar problem with configuring the email driver of the User Messaging Services. A description on how to configure that can be found here. The email driver settings are stored in driverconfig.xml in the $MSERVER_DOMAIN_HOME/config/fmwconfig/servers/soa_server1/applications/usermessagingdriver-email/configuration.

And again, restarting the domain, or soa_server1, these are overwritten by the driverconfig.xml at the same subfolder location in the $ASERVER_DOMAIN_HOME! And since this works like this for the Global Tokens and the email-driver it problably works like this for other UMS drivers, or even other functionality.

The workaround?

Is quite simple: copy the updated mdm-url-resolver.xml or driverconfig.xml in the
$MSERVER_DOMAIN_HOME to the counterparts in $ASERVER_DOMAIN_HOME. Then start the servers again. On startup the AdminServer will copy it's variant (that is a copy of the correct, updated one) to the SOA Server again.


I still do like SOA Suite. It's an impressive Middleware suite. But I really hope Oracle does invest in making it more stable, decreasing the footprint and adapt the functionality to the Enterprise Deployment Guide. Since, the behavior above does not match the recommendations as described in the EDG. And I think SOA Suite, OSB and even Weblogic could be a lot smaller and faster with the same functionality. I encounter a lot of duplicated libraries throughout the ORACLE_HOME. Or several different versions of the same library. I assume those can be reduced quite a bit. And that will benefit both the Cloud variants of the software as the On Premise variant.

Tuesday 25 September 2018

Split your flow trace in BPEL

A quick one today. In the past we suffered from very large flowtraces in SOA Suite 11g, due to which it could happen that a flow trace wasn't parsable in EM. And therefor could not be shown.

Also, you might have other reasons to split up your flow trace. Maybe because you want to have a callee BPEL process that may run for a longer time run on, while the calling BPEL project is redeployed (although I haven't tested that yet, so I'm not sure if that would work).

I did know it should be possible to split up the flow traces by changing the ECID (Execution Context ID). But, haven't seen it and wasn't able to find it. But, today I found the how-to in Oracle Support Note 2278472.1. So, as  a note-to-myself, here it is.

In the invoke activity to a child process you should add the following property:
<bpelx:toProperty name="tracking.ecid" variable="ora:generateGUID()"/>

This will update the tracking.ecid to a GUID.You should/need to do this on the invoke only (not on the receive). It should not cause any collission or conflict, since it generates a Global Unique Identifier.

Tuesday 18 September 2018

SOA 12c MDS configuration

In a previous post I described how to have the integrated Weblogic of your SOA/BPM QuickStart refer to the same filebased MDS that you might refer to in the SOADesignTimeRepository in JDeveloper.

These days for my current customer I'm looking into upgrading 11g, as can be read from my previous posts. This customer also has a legacy with projects migrated from 10g.

In the 11g workspace there was a reference to the database MDS in the Development database. In 12c we have a designtime mds reference. I would recommend to refer that to the mds artefacts in your VCS (Version Control System: svn or git) working copy. To do so, call-up the Resources pane in JDeveloper and right click on the SOADesignTimeRepository:
Then navigate to the location in your working copy:
Mind that SOASuite expects an apps folder within this folder, so resulting references in the composite.xml, etc. are expected to start with oramds:/apps/....

Now, I migrated a few projects including a adf-config.xml. In stead of the DB MDS repo, I replaced it with a file-based reference in the adf-config.xml, refering to the SOADesignTimeRepository. If you create a new 12c SOA Application, the adf-config.xml will look like:
<?xml version="1.0" encoding="windows-1252" ?>
<adf-config xmlns="" xmlns:adf=""
  <adf:adf-properties-child xmlns="">
    <adf-property name="adfAppUID" value="MyApplication-1234"/>
  <sec:adf-security-child xmlns="">
    <CredentialStoreContext credentialStoreClass=""
  <adf-mds-config xmlns="">
    <mds-config xmlns="">
          <namespace path="/soa/shared" metadata-store-usage="mstore-usage_1"/>
          <namespace path="/apps" metadata-store-usage="mstore-usage_2"/>
          <metadata-store-usage id="mstore-usage_1">
            <metadata-store class-name="oracle.mds.persistence.stores.file.FileMetadataStore">
              <property name="partition-name" value="seed"/>
              <property name="metadata-path" value="${}/integration"/>
          <metadata-store-usage id="mstore-usage_2">
            <metadata-store class-name="oracle.mds.persistence.stores.file.FileMetadataStore">
              <property name="metadata-path" value="${soamds.apps.home}"/>

In the metadata-store-usage with id mstore-usage_2 you'll find the reference ${soamds.apps.home} in the metadata-path property. This refers to the folder as choosen in your SOADesignTimeRepository.

Now, I found in the past several times that although the adf-config.xml was similar to the above, that the MDS references did not work. In those cases, as a workaround, I put the absolute path reference in the metadata-path property.

Today I found something similar because of the upgrade, and searched on MDS-01333: missing element "mds-config" This resulted in this article, that gave me the hint.

It turns out that the snippet:
  <adf:adf-properties-child xmlns="">
    <adf-property name="adfAppUID" value="MyApplication-1234"/>

get's in the way. the UID refers to the application name and some generated number. It turns out not enough to change it the name of the application with a generated number. I haven't found what the proper number should be. So I just removed that snippet and then it worked.

Monday 17 September 2018

SOA Bundelpatch for available (since juli 2018)

Because of the version level my previous customer was on, I mostly worked with the version of the BPM QuickStart. Recently I started at an other customer that is still on SOA Suite 11g. Since I'm looking into upgrading those the latest 12c version, I installed BPM QuickStart again.

Doing a patch search on, I found out that juli 17th, 2018, a SOA BundlePatch on was released. It's patch 28300397.

The readme shows quite a list of bugs solved. The version of JDeveloper and the main components stay unaffected. The version changes are shown in the extensions. The vanilla, unpatched, JDeveloper shows:

And the patched JDeveloper shows:

Since it's been a year already since was released (august 2017, if I recollect correctly), this bundle patch is welcome.

By the way, the reason that I was looking into the patches, was that I created a few .xsl files to pre-upgrade our 11g projects. And the didn't reformat properly. JDeveloper behaves strangely, apparenlty it does not recognize an .xsl file as xml. When you copy and paste it into an .xml file it does format properly. I think I have to dig into the preferences to see if this can be tweaked.

To install it, unzip the patch. I'm used to create a patches folder within the OPatch folder in the Oracle Home:
And unzip the patch in to that folder. Because the unzip functionality in Windows is limited to 256 characters in the resulting path names, it is advised to use a tool like 7Zip. Since I use TotalCommander for about everything, (file related that is), I get a neat dialog mentioning this and allowing me to keep the names.

Make sure you have closed JDeveloper.

Then open a command window and navigate to the patches folder:
Microsoft Windows [Version 10.0.17134.285]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Windows\system32>cd \oracle\JDeveloper\12213_BPMQS\OPatch\patches\28300397

C:\oracle\JDeveloper\12213_BPMQS\OPatch\patches\28300397>set ORACLE_HOME=C:\oracle\JDeveloper\12213_BPMQS

First set the the ORACLE_HOME variable to the location where you installed JDeveloper, C:\oracle\JDeveloper\12213_BPMQS in my case.

Using opatch apply you can apply the patch:
c:\Oracle\JDeveloper\12213_SOAQS\OPatch\patches\28300397>..\..\opatch apply
Oracle Interim Patch Installer version
Copyright (c) 2018, Oracle Corporation.  All rights reserved.

Oracle Home       : c:\Oracle\JDeveloper\12213_SOAQS
Central Inventory : C:\Program Files\Oracle\Inventory
   from           :
OPatch version    :
OUI version       :
Log file location : c:\Oracle\JDeveloper\12213_SOAQS\cfgtoollogs\opatch\opatch2018-09-17_12-05-54PM_1.log

OPatch detects the Middleware Home as "C:\Oracle\JDeveloper\12213_SOAQS"

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   28300397

Do you want to proceed? [y|n]
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = 'c:\Oracle\JDeveloper\12213_SOAQS')

Is the local system ready for patching? [y|n]
User Responded with: Y
Backing up files...
Applying interim patch '28300397' to OH 'c:\Oracle\JDeveloper\12213_SOAQS'
ApplySession: Optional component(s) [ oracle.mft, ] , [ oracle.soa.workflow.wc, ] , [ oracle.integ
emina, ] , [ oracle.mft.apachemina, ] , [ oracle.bpm.plugins, ] , [ oracle.oep.examples, ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.soa.all.client,

Patching component oracle.integration.bam,

Patching component oracle.rcu.soainfra,

Patching component oracle.rcu.soainfra,

Patching component oracle.soacommon.plugins,

Patching component oracle.oep,

Patching component oracle.integration.soainfra,

Patching component oracle.integration.soainfra,

Patching component oracle.soa.common.adapters,

Patching component oracle.soa.procmon,
Patch 28300397 successfully applied.
Log file location: c:\Oracle\JDeveloper\12213_SOAQS\cfgtoollogs\opatch\opatch2018-09-17_12-05-54PM_1.log

OPatch succeeded.


Answer 'y' on the questions to proceed and if the oracle home is ready to be patched. And with  opatch lsinventory you can check if the patch (and possibly others) is applied:

c:\Oracle\JDeveloper\12213_SOAQS\OPatch\patches\28300397>..\..\opatch lsinventory
Oracle Interim Patch Installer version
Copyright (c) 2018, Oracle Corporation.  All rights reserved.

Oracle Home       : c:\Oracle\JDeveloper\12213_SOAQS
Central Inventory : C:\Program Files\Oracle\Inventory
   from           :
OPatch version    :
OUI version       :
Log file location : c:\Oracle\JDeveloper\12213_SOAQS\cfgtoollogs\opatch\opatch2018-09-17_12-11-19PM_1.log

OPatch detects the Middleware Home as "C:\Oracle\JDeveloper\12213_SOAQS"

Lsinventory Output file location : c:\Oracle\JDeveloper\12213_SOAQS\cfgtoollogs\opatch\lsinv\lsinventory2018-09-17_12-11-19PM.txt

Local Machine Information::
ARU platform id: 233
ARU platform description:: Microsoft Windows Server 2003 (64-bit AMD)

Interim patches (5) :

Patch  28300397     : applied on Mon Sep 17 12:08:10 CEST 2018
Unique Patch ID:  22311639
Patch description:  "SOA Bundle Patch"
   Created on 6 Jul 2018, 01:58:16 hrs PST8PDT
   Bugs fixed:
     26868517, 27030883, 25980718, 26498324, 26720287, 27656577, 27639691
     27119541, 25941324, 26739808, 27561639, 26372043, 27078536, 27024693
     27633270, 27073918, 27210380, 27260565, 27247726, 27880006, 27171517
     26573292, 26997999, 26484903, 27957338, 27832726, 27141953, 26851150
     26696469, 27494478, 27150210, 27940458, 26982712, 27708925, 26645118
     27876754, 24922173, 27486624, 26571201, 26935112, 26953820, 27767587
     26536677, 27311023, 26385451, 26796979, 27715066, 27241933, 24971871
     26472963, 27411143, 27230444, 27379937, 27640635, 26957183, 26031784
     26408150, 27449047, 27019442, 26947728, 27368311, 26895927, 27268787
     26416702, 27018879, 27879887, 27929443

Patch  26355633     : applied on Wed Sep 12 12:00:33 CEST 2018
Unique Patch ID:  21447583
Patch description:  "One-off"
   Created on 1 Aug 2017, 21:40:20 hrs UTC
   Bugs fixed:

Patch  26287183     : applied on Wed Sep 12 11:59:56 CEST 2018
Unique Patch ID:  21447582
Patch description:  "One-off"
   Created on 1 Aug 2017, 21:41:27 hrs UTC
   Bugs fixed:

Patch  26261906     : applied on Wed Sep 12 11:59:11 CEST 2018
Unique Patch ID:  21344506
Patch description:  "One-off"
   Created on 12 Jun 2017, 23:36:08 hrs UTC
   Bugs fixed:
     25559137, 25232931, 24811916

Patch  26051289     : applied on Wed Sep 12 11:58:45 CEST 2018
Unique Patch ID:  21455037
Patch description:  "One-off"
   Created on 31 Jul 2017, 22:11:57 hrs UTC
   Bugs fixed:


OPatch succeeded.

Friday 14 September 2018

Advanced SoapUI: Mocking a Async Request Response Service supporting WS-Addressing

Lately, I sat down with my appreciated colleague Frank, prepping for a SoapUI/ReadyAPI training due next week. After having solved some issues he had, we agreed upon designing an advanced lab.

In the past I wrote an article about how to test een Asynchronous Request Response BPEL process, with WS-Addressing. This uses SoapUI as a test client, to test an Async BPEL Process and catching the response. Frank suggested to create a SoapUI project that mocks the BPEL Process. And that's a whole other ball-game!

SoapUI does support Mock Services. But those are in fact Synchronous services: upon request they send back a response. They're very flexible in that you can determine and select the responses that are send back in several ways. You can even script the lot using an OnRequest Groovy script.

But in this case we do not want to send back a response. The thing with an Asynchronous Request Response service is that they're actually two complementary Fire & Forget services.
  1. The actual request service is a fire and forget service implemented by the service provider. It does not respond with a message, but it just starts processing the request.
  2. Then the service client implements a CallBack fire and forget service. Upon processing the request to the point that a response is build, the Service Provider will call this service with the actual response as a request.
How would you implement this with SoapUI? First, you create a Project with a TestCase as described in my referenced article. It will invoke the SOAP Service with a request and then bring up a SOAP Mock Response to catch the response.

For the Async MockService we create a MockService that catches the request. But we leave the response empty: we do not want to reply with a response immediately. In stead, we use the On Request script to call a Test Case that simulates the proces. The interesting part is to pass the info from the request: the WS-Addressing elements (ReplyTo-Address and MessageId) and message Content. But let's sort that out step-by-step.

By the way I worked this out as a Lab together with my colleague Frank, in both SoapUI and ReadyAPI simultaneously. So it works in both products. In stead of ReadyAPI's 'Virts', I stick with the SoapUI term MockServices. But the principles and code snippets work one-on-one.

Create a SoapUI project with a MockService

First create a SoapUI project. I used the wsdl and xsd that I published here on github.
Then create a MockService on the BPELProcessAsyncBinding Request binding:

  • Name it for example: BPELProcessAsyncBinding MockService.
  • Service Path: /mockBPELProcessAsyncBinding
  • Port: 8088
We don’t provide a response on purpose: it will be an async service, that will respond by doing an invoke later on.

Add the mockservice’s endpoint to the interface binding:
Remove the original endpoint from the interface, since it is a dummy endpoint ('http://localhost:7101/soa-infra/services/default/helloWorldBPELAsync/bpelprocessasync_client_ep').

Now you can test the MockService with an adhoc request.

Create a 'Client' Test case

In the SoapUI Project, create a TestSuite called TestSuite AsyncBPEL and add a TestCase, called AsyncSvcClient:
Then clone the Adhoc Test Request to the testcase and call it InvokeAsyncService:

To pick up the response we need to add a MockResponse based on the CallBack binding of the wsdl:
Base it on the CallBack Binding of the wsdl:
Take note of the Port and the Path, if you choose to use something else as 8090 and /HelloWorldCallback that I used for this article.

It is important that this step is started as soon as the request is sent. It takes time to startup the MockResponse listener. So, you need to couple it to the corresponding Soap Request step. To do so, you need to get to the properties of the AsyncReceive MockResponse step and set the start step of the MockResponse step to InvokeAsyncService:

This will ensure that when the InvokeAsyncService step is executed the AsyncReceive mock response is started, so that it can be called as soon as the ServiceProvider wants to send back its response.

Note that the xml request of the AsyncReceive step is empty, as well as the response. The response will stay unused, but the request is to capture the callback message from the service provider, as we will see later on.

Setup the Async Service Provider

The MockService inherently is a synchronous mechanism, so normally used to respond with a response message on request. Since we want to implement an asynchronous request-reply mock service, we won’t respond with a message. So the response message stays empty. How are we going to respond then? We will build a second test case, that will be executed on request from a Groovy Script on the MockService. It will build up a context from the request message and providing that to the running testcase we will provide the test case with the information to invoke the AsyncReceive step of the client test case.

Thus we create a new test case, and it will do two things:
  1. Extract the request properties from the context, they will consist of the following properties:
    1. WS Addressing ReplyTo Address
    2. WS Addressing MessageId
    3. HelloWorld Input message (payload elements)
  2. Do the Callback based on the provided information.
To implement this perform the following:
  1. Create a new TestSuite, called AsyncBPELSvcProvider and add a TestCase, called AsyncSvcProvider.
  2. Add a SOAP Request step, named CallBackAsyncSvcClient and based that on the BPELProcessAsyncCallbackBinding:
  3. As a result value provide ‘Hello’ for now.
  4. As an endpoint set http://localhost:8090/HelloWorldCallback. We will change that to a property, later , fetched from the context.
  5. Remove a possible assertion to check on the Soap Response Message (since we won’t get one).
  6. If you want to test now, you can run the AsyncSvcClient but it will wait on the AsyncReceive step. To have that execute, you should manually run the AsyncSvcProvider test case.

Now we need to have the new TestCase called from the OnRequest script of the MockService.
For that we add a few properties to the MockService, to denote the TestSuite and the containing TestCase that implements our ServiceProvider process.
Then using a basic Groovy script that we will extend later on, we make sure that that test case is ran.

  1. Add two Custom Properties:
    1. AsyncTestSuite, with value: AsyncBPELSvcProvider
    2. AsyncSvcProvTestCase, with value: AsyncSvcProvider
  2. On the OnRequest script of the Mock Service:

    Add the following script:
    def mockService = context.mockService
    def method =".Response 1.OnRequest Script""Start "+method)
    def project = mockService.project"Project "
    def asyncTestSuiteName = mockService.getPropertyValue( "AsyncTestSuite")
    def asyncTestSuite = project.getTestSuiteByName(asyncTestSuiteName)"TestSuite: "
    def asyncSvcProvTestCaseName = mockService.getPropertyValue( "AsyncSvcProvTestCase")
    def asyncSvcProvTestCase = asyncTestSuite.getTestCaseByName(asyncSvcProvTestCaseName)"TestCase: "
    //Log Request
    // Set Service Context
    def svcContext = (
    //Invoke Async Service Provider TestCase, false)
    // End Method"End "+method)

    What this does is the following:
    1. Define the mockService and the project objects from the context variable.
    2. Get the TestSuite and TestCase objects based on the MockService property values of the TestCase to be called.
    3. Create a serviceContext, to be used to do property transfer later on.
    4. Run the testCase using the created serviceContext.
  3. Now you can test this by invoking the AsyncSvcClient test case. You might want to remove the current content of the request of the AsyncReceive .

Transfer Request Context properties to ServiceProvider TestCase

Now we want to at least transfer the helloworld input in the request from the MockService to the service provider testcase, so that it can add it to the response message.

In the OnRequest Groovy Script we already created a context. We can simply set additional properties to that context. The values we can extract from the request, by xpath.

  1. Go to the OnRequest groovy script and extend your existing script to reflect the following:
    def mockService = context.mockService
    def method =".Response 1.OnRequest Script""Start "+method)
    def project = mockService.project"Project "
    def asyncTestSuiteName = mockService.getPropertyValue( "AsyncTestSuite")
    def asyncTestSuite = project.getTestSuiteByName(asyncTestSuiteName)"TestSuite: "
    def asyncSvcProvTestCaseName = mockService.getPropertyValue( "AsyncSvcProvTestCase")
    def asyncSvcProvTestCase = asyncTestSuite.getTestCaseByName(asyncSvcProvTestCaseName)"TestCase: "
    //Log Request
    // Added lines ==>
    def groovyUtils = new
    // Set Namespaces and query request
    def holder = groovyUtils.getXmlHolder(mockRequest.getRequestContent())
    holder.namespaces["soapenv"] = ""
    holder.namespaces["bpel"] = ""
    holder.namespaces["wsa"] = ""
    def helloInput =  holder.getNodeValue("/soapenv:Envelope/soapenv:Body/bpel:process/bpel:input")
    // Set Service Context
    def svcContext = (
    // <==Added lines
    //"helloInput: "+svcContext.helloInput)
    //Invoke Async Service Provider TestCase, false)
    // End Method"End "+method)
    This adds the following:
    1. A declaration of the groovyUtils, that is used to get an so called XmlHolder that contains the content of the Request in parsed XML Format.
    2. Declare namespace references in the holder.
    3. Query the helloInput using the xpath expression: "/soapenv:Envelope/soapenv:Body/bpel:process/bpel:input” from the request.
    4. Set this as a helloInput property on the service context.

  • Now we need to extract these properties in the AsyncSvcProvider TestCase, so that we can use it in the request of the callback. To do so add a Groovy Test Step to the AsyncSvcProvider TestCase, as a first step:

    Call it GetContextProperties, and move it as the first step in the TestCase:
  • Add the following to the script:
    def testCase=testRunner.testCase
    def testSuite=testCase.testSuite
    def"."".getContextProperties""Start MethodName: "+methodName)
    def helloInput=context.helloInput" Received HelloInput: "+helloInput)
    testCase.setPropertyValue("helloInput",helloInput)"End MethodName: "+methodName)

    As you can see in the top right corner of the editor, you can see that besides a log variable also a context variable is provided:

    This variable will contain the properties we set in the call to the testcase from the MockService.
    As you can see we get the property from the context, and set it as a TestCase property.
  • Add the helloInput property to the AsyncSvcProvider TestCase. You don’t need to provide a value, it just needs to exist. 
  • Lastly, in the request of the CallBackAsyncSvcClient step, add ${#TestCase#helloInput} to the result:

  • Configure WS-Addressing

    In the previously mentioned blog article you can read how to create a test case that supports WS Addressing to call and test an asynchronous (BPEL) request response service. Now with the above, we have the plumbing in place to add the WS Addressing specifics to simulate and test the Asynchronous RequestResponse Service Provider.

    We need then to provide and process the following:
    • A WS Addressing Reply To Address, based on property values that matches the port and path of the AsyncReceive step.
    • A message id that is used to validate if the response back is using the correct provided messageId header value. In a real life case this message Id is used by the SOA Suite infrastructure to correlate the response to the correct process instance that requested it. This is not supported/implemented in SoapUI, since that tool is not meant for that. But we can add an assertion to check the correct responding of this property.
    To implement this, perform the following:
    1. On the AsyncSvcClient test case add the following properties:
      • callbackURI, with value: HelloWorldCallback
      • callbackPort, with value: 8090
      • callbackHost, with value: localhost
      • wsAddressingReplyToEndpoint, with value: http://${#TestCase#callbackHost}:${#TestCase#callbackPort}/${#TestCase#callbackURI}
      • wsAddressingMessageId, with no value

      You see that the wsAddressingReplyToEndpoint is dynamically build up from the previous properties. The callbackURIand the callbackPort should exactly match the values of the path and the port of the AsyncReceive step (without the initial slash):

      The property wsAddressingMessageId does not need a value: we will generate a value in another Groovy TestStep.
    2.  Add a Groovy TestStep to AsyncSvcClient test case, call it GenerateWSAMessageId,  and move it to the top, and add the following code:
      def testCase=testRunner.testCase
      def testSuite=testCase.testSuite
      def"."".GenerateWSAMessageId""Start "+methodName)
      def wsAddressingMessageId=Math.round((Math.random()*10000000000))
      testCase.setPropertyValue("wsAddressingMessageId", wsAddressingMessageId.toString())"End "+methodName)

      This will do a randomize and multiply it with a big number to create an integer value.
    3. Now we will add the WS Addressing properties to the request. Open the InvokeAsyncService test step and click on the WS-A tab at the bottom:

      Set the following properties:
      • Check Enable WS-A Addressing
      • Set Must understand to TRUE
      • Leave WS-A Version to 200508
      • Check Add default wsa:Action
      • Set Reply to to: ${#TestCase#wsAddressingReplyToEndpoint}
      • Uncheck Generate MessageID
      • Set MessageID to: ${#TestCase#wsAddressingMessageId}
      The Reply To address and the MessageID now are based on the earlier determined properties.
    4. If you would test this, then the request that will be send will look like:
      <soapenv:Envelope xmlns:soapenv="" xmlns:bpel="">
        <soapenv:Header xmlns:wsa="">
          <wsa:Action soapenv:mustUnderstand="1">process</wsa:Action>
          <wsa:ReplyTo soapenv:mustUnderstand="1">
          <wsa:MessageID soapenv:mustUnderstand="1">9094853750</wsa:MessageID>

      You see that the ReplyTo Address is set (as a nested element) and the MessageId. You won’t see this in the Request XML panel, but in the http log or in the script-log since we log the request in the OnRequest script of the MockService. The WS-Addressing properties are added to the soap:header on invoke.
    5. Since we have these elements in the request, we can extract those the same way as we did with the helloInput in the OnRequest script of the MockService. Add the lines denoted with // Added lines ==> and // <==Added lines: from the following script in your script (or copy&paste complete script):
      def mockService = context.mockService
      def method =".Response 1.OnRequest Script""Start "+method)
      def project = mockService.project"Project "
      def asyncTestSuiteName = mockService.getPropertyValue( "AsyncTestSuite")
      def asyncTestSuite = project.getTestSuiteByName(asyncTestSuiteName)"TestSuite: "
      def asyncSvcProvTestCaseName = mockService.getPropertyValue( "AsyncSvcProvTestCase")
      def asyncSvcProvTestCase = asyncTestSuite.getTestCaseByName(asyncSvcProvTestCaseName)"TestCase: "
      //Log Request
      def groovyUtils = new
      // Set Namespaces and query request
      def holder = groovyUtils.getXmlHolder(mockRequest.getRequestContent())
      holder.namespaces["soapenv"] = ""
      holder.namespaces["bpel"] = ""
      holder.namespaces["wsa"] = ""
      def helloInput =  holder.getNodeValue("/soapenv:Envelope/soapenv:Body/bpel:process/bpel:input")
      // Added lines ==>
      def wsaReplyToAddress =  holder.getNodeValue("/soapenv:Envelope/soapenv:Header/wsa:ReplyTo/wsa:Address")
      def wsaInReplyToMsgId =  holder.getNodeValue("/soapenv:Envelope/soapenv:Header/wsa:MessageID")
      // <Added lines
      // Set Service Context
      def svcContext = (
      // Added lines ==>
      // <Added lines
      //"helloInput: "+svcContext.helloInput)
      // Added lines ==>"wsaReplyToAddress: "+svcContext.wsaReplyToAddress)"wsaInReplyToMsgId: "+svcContext.wsaInReplyToMsgId)
      // <Added lines
      //Invoke Async Service Provider TestCase, false)
      // End Method"End "+method)
    6. These context properties need to be extracted in the GetContextProperties of the AsyncSvcProvider test case, to set those as TestCase Properties. So, add the following properties (with no values) to the AsyncSvcProvider test case:
      • wsaReplyToAddress
      • wsaInReplyToMsgId
    7. In the GetContextProperties test step, add the lines with the added properties (or copy and paste the complete script):
      def testCase=testRunner.testCase
      def testSuite=testCase.testSuite
      def"."".getContextProperties""Start MethodName: "+methodName)
      def wsaReplyToAddress=context.wsaReplyToAddress
      def wsaInReplyToMsgId=context.wsaInReplyToMsgId
      def helloInput=context.helloInput" Received wsaReplyToAddress: "+wsaReplyToAddress)" Received wsaInReplyToMsgId: "+wsaInReplyToMsgId)" Received HelloInput: "+helloInput)
      // End"End MethodName: "+methodName)

      (Since the wsaInReplyToMsgId is an integer, it should be "toStringed"...)
    8. As a pre-final step is to adapt the CallBackAsyncSvcClient step to use the wsaReplyToAddress as an endpoint and the wsaInReplyToMsgId as a header property. Edit the endpoint in the step to ${#TestCase#wsaReplyToAddress}:

      Edit the soap header to:
         <soapenv:Header xmlns:wsa="">

    9. The final step is to add an XPath Match assertion on the AsyncReceive to validate the response of the wsaInReplyToMsgId. Call it WSAInReplyToMessageId and provide the following xpath:
      declare namespace wsa='';
      declare namespace bpel='';
      declare namespace soapenv='';

      As an Expected Result value provide: ${#TestCase#wsAddressingMessageId}.
    10. Test the completed AsyncSvcClient.


    This will conclude this setup. And shows how to create a WS-Addressing supporting Asynchronous Request Response Service. I hope you got this far. In that case: I'm impressed. This is quite advanced SoapUI/ReadyAPI! stuff. But it shows the power of the tools. And if you wouldn't use this as it is, you might get some nice tips from it.