Persisting of settings in a SOA Suite Enterprise Deployment

About a year ago, at my previous customer, a co-worker and I encountered and described a persistence problem with setting the Global Tokens in SOA Suite.

What are Global Tokens again?

The problem with a middleware product as Oracle Service Bus, SOA Suite (and the same probably counts for MuleSoft, or any other integration tool) is that when you move an integration through the development lifecycle from development, to test, preproduction and production, you need to update the endpoints. When I have an integration with a (BPEL) Process that does a check-in of a document in WebCenter Content, for instance, then on the test environment it should do the check-in to another WCC server than on pre-production or production. We don't want to have our test documents in production, do we?

To solve that, in OSB we have customization files, and in SOA Suite 11g and onwards, we use config plans. But, in 11g PatchSet 6 (11.1.1.7), SOA Suite introduced Global Tokens. That way you can create a token that refers to the WCC host, eg. ${wcc_url}, and use that as a reference in your binding properties.

These properties can be set using Enterprise Manager FMW Control 12c:

which lead to:

Where you can add tokens or import a file with the tokens.

These settings are stored in the mdm-url-resolver.xml file, in the $DOMAIN_HOME/config/fmwconfig folder.

What about Enterprise Deployment?

The Enterprise Deployment Guide of SOA Suite 12c is quite complex. But in short, as we implemented it, we installed Fusion Middleware Infrastructure and SOA Suite on one node/host. Then, of course, ran the Repository Creation Utility, and configured the domain for the AdminServer. That domain was configured on a shared disk, let's say, /u01/data/oracle/domains/soa1_domain. Then it is cloned to cater for the managed servers, using pack/unpack, to local storage, for instance, /u02/data/oracle/domains/soa1_domain. In short we have 2 domain homes:

• ASERVER_DOMAIN_HOME=/u01/data/oracle/domains/soa1_domain
• MSERVER_DOMAIN_HOME=/u02/data/oracle/domains/soa1_domain

Where  /u01 is mounted on a shared disk and /u02 on local storage.

So, the AdminServer runs from the ASERVER_DOMAIN_HOME domain, that is on shared storage. This way, when the host running the AdminServer goes down, the AdminServer can be brought up on the second host. The Managed Servers run on a clone of the domain on local storage.

Side note: in 12c we have a per-domain NodeManager by default. So cloning the domain, implicitly clones the nodemanager config. And running that against another adapter, allows for a nodemanager for the Admin server and one for the ManagedServers.

Why is this important? Well, this allows for a High Available setup, including functionality as Zero Downtime Patching.

What is the problem then?

Updating the Global Tokens is done in FMW Control, that runs on the AdminServer. It stores the properties in the mdm-url-resolver.xml. But, which particular mdm-url-resolver.xml file? Well, the changes are stored in the one in $MSERVER_DOMAIN_HOME/config/fmwconfig!
After that you need to restart the SOA Server, to get the properties loaded. And then something very smart happens. When starting the SOA Server, the AdminServer sends it's copy from the $ASERVER_DOMAIN_HOME/config/fmwconfig to the SOA Server. And so the changes are cleared by the version from the AdminServers domain!

So, in an Enterprise Deployment configuration of the SOA Suite a restart of the SOA Server, will clear the changes of the Global Tokens.

But there is more!

As I wrote above, we found this a year ago. And we created a Prio 1 Service Request. The issue is very straight forward, reproducable, and in the status Development Working for about a year now:

(I'm not writing this to bash Support by the way. No offense intended, altough I would really like a patch by now...)

But, today another co-worker and I encountered a very similar problem with configuring the email driver of the User Messaging Services. A description on how to configure that can be found here. The email driver settings are stored in driverconfig.xml in the $MSERVER_DOMAIN_HOME/config/fmwconfig/servers/soa_server1/applications/usermessagingdriver-email/configuration.

And again, restarting the domain, or soa_server1, these are overwritten by the driverconfig.xml at the same subfolder location in the $ASERVER_DOMAIN_HOME! And since this works like this for the Global Tokens and the email-driver it problably works like this for other UMS drivers, or even other functionality.

The workaround?

Is quite simple: copy the updated mdm-url-resolver.xml or driverconfig.xml in the
$MSERVER_DOMAIN_HOME to the counterparts in $ASERVER_DOMAIN_HOME. Then start the servers again. On startup the AdminServer will copy it's variant (that is a copy of the correct, updated one) to the SOA Server again.

Conclusion

I still do like SOA Suite. It's an impressive Middleware suite. But I really hope Oracle does invest in making it more stable, decreasing the footprint and adapt the functionality to the Enterprise Deployment Guide. Since, the behavior above does not match the recommendations as described in the EDG. And I think SOA Suite, OSB and even Weblogic could be a lot smaller and faster with the same functionality. I encounter a lot of duplicated libraries throughout the ORACLE_HOME. Or several different versions of the same library. I assume those can be reduced quite a bit. And that will benefit both the Cloud variants of the software as the On Premise variant.