A little background

In April I wrote about our BPM Server installation, that is actually a single host cluster on dev and test. But installed like it was a dual-node  server, so we had a cloned domain.

A BPM installation has a component that is called the Process Asset Manager. Under the hood it uses a replicated SubVersion server. Each node has one, so they had to synchronize. But since we're on the same host, we needed to differentiate in port numbers. Although we used virtual host names for each server node. In an article in april 'BPM BAC Subversion Server refusing connections' I wrote about that.

It turns out: it did not work appropriately. After some investigation, apparently the bac_node1/subversion server calls the subversion server on bac_node2, with it's own address, but with the port of the other:

...
<Jul 24, 2017, 2:37:29,543 PM CEST> <Debug> <oracle.bpm.bac.svnserver.replication> <darlin01> <bpm_server1> <Active Sync Thread [/54680efc-9328-478e-953c-834bbb250725/]> <<anonymous>> <> <20a54a15-d8d8-4e58-a4f6-11a90e992967-00000008> <1500899849543> <[severity-value: 128] [rid: 0:490:13:19] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <[oracle.bpm.log.Logger:debug] About to synchronize against node Member(Id=2, Timestamp=2017-07-12 08:01:13.309, Address=10.0.0.38:37055, MachineId=9002, Location=site:tst.darwin-it.local,machine:bpm_machine1,process:24566,member:bpm_server2, Role=bpm_cluster), url svn://syncuser@t-bpm-1-bpm-1-vhn.tst.darwin-it.local:8424/54680efc-9328-478e-953c-834bbb250725>
...

Of course there's no listen address so unsurprisingly we get errors like:

...
[2017-05-30T12:55:51.704+02:00] [bpm_server1] [ERROR] [] [oracle.bpm.bac.svnserver.replication] [tid: Active Sync Thread [/b91abb78-6b3d-4448-af6a-e82125f261f0/]] [userId: ] [ecid: ed0bc6ce-fefb-4608-8dbe-46f7206a1573-0000000a,0:527] [APP: OracleBPMBACServerApp] [partition-name: DOMAIN] [tenant-name: GLOBAL] org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server[[
oracle.bpm.bac.subversion.server.repository.exceptions.RepositoryException: org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server
at oracle.bpm.bac.subversion.server.repository.exceptions.RepositoryException.wrap(RepositoryException.java:56)
at oracle.bpm.bac.subversion.server.repository.SVNKitRepositorySession.getRepositoryUUID(SVNKitRepositorySession.java:98)
at oracle.bpm.bac.subversion.server.repository.RepositorySVNSync.sync(RepositorySVNSync.java:74)
at oracle.bpm.bac.subversion.server.repository.RepositorySVNSync.sync(RepositorySVNSync.java:59)
at oracle.bpm.bac.subversion.server.repository.ha.aa.ActiveAARepository$Synchronizer.runImpl(ActiveAARepository.java:341)
at oracle.bpm.bac.subversion.server.repository.ha.aa.ActiveAARepository$Synchronizer.run(ActiveAARepository.java:304)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:62)
at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:77)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1252)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:95)
at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:280)
at oracle.bpm.bac.subversion.server.repository.SVNKitRepositorySession.getRepositoryUUID(SVNKitRepositorySession.java:95)
... 5 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:112)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:68)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:53)
... 10 more

]] 

(to get this page searchable).

Solution

But now, since this week, Oracle Support presents us.... (drum rolls)  Patch 26775572 for version 12.2.1.2.0, the official fix for this issue, is finally ready and available for download.
I hope to be able to install it next monday. But feel free to try with me: 'Patch 26775572: BAC node using the wrong port when attempts synchronization with Virtual IP '

By the way, for about 2 months we have diagnostic patch running that solves this. But named patch is the official delivery of our diagnostic variant.

By the way, since 12.2.1.3.0 is out in the field for quite some time already, I expect that this did not landed in that version yet. If you're running in the same issue for 12.2.1.3.0 then create a SR to request for a port of this patch.

At my current customer I implemented a poor man's devops solution for Release and Deploy. It was based on a framework created as bunch of Ant projects, that I created years ago. It was based on scripts from Edwin Biemond. See for instance here, here and here. I never wrote about my solution, because although I refactored them quite intensively, the basics were already described thoroughly by him.

What I did was that I modularized the lot, split the environment property files, added logging, added OSB 12c  support, based on the config jar tool, etc.

One thing I ran into this week was that at first deployment from our team to the test environment using my framework, the running instances for the BPM projects were aborted.

Now, if you take a look at the deploy.sarLocation target in Edwin's article about deploying soa suite composites  you'll find that he also supported the overwrite and forceDefault properties.

When re-deploying a composite from JDeveloper you're probably familiar with the 'keep running instances' checkbox. I was looking for the ANT alternative in the ${oracle.home}/bin/ant-sca-deploy.xml scripting. First I looked in the 12c docs (see 47.9.4 How to Use ant to Deploy a SOA Composite Application), but it is not documented there.

But when I opened the particular ant-sca-deploy.xml script I read:

 <condition property="overwrite" value="false">
    <not>
      <isset property="overwrite"/>
    </not>
  </condition>
  <condition property="forceDefault" value="true">
    <not>
      <isset property="forceDefault"/>
    </not>
  </condition>
  <condition property="keepInstancesOnRedeploy" value="false">
    <not>
      <isset property="keepInstancesOnRedeploy"/>
    </not>
  </condition>
 ...
  <target name="deploy">
    <input message="Please enter server URL:" addproperty="serverURL"/>
    <input message="Please enter sar location:" addproperty="sarLocation"/>
    <input message="Please enter username:" addproperty="user"/>
    <input message="Please enter password:" addproperty="password">
      <handler classname="org.apache.tools.ant.input.SecureInputHandler" />
    </input>
    <deployComposite serverUrl="${serverURL}" sarLocation="${sarLocation}" realm="${realm}" user="${user}" password="${password}"
      overwrite="${overwrite}" forceDefault="${forceDefault}" keepInstancesOnRedeploy="${keepInstancesOnRedeploy}"
      regenerateRuleBase="${regenerateRuleBase}" configPlan="${configplan}" scope="${scope}"
      sysPropFile="${sysPropFile}" failOnError="${failOnError}" timeout="${timeout}" folder="${folder}"/>
  </target>

So, the script kindly supports the keepInstancesOnRedeploy property. And thus I implemented a deploy.keepInstancesOnRedeploy property the same way as the deploy.forceDefault/deploy.overwrite properties.

This probably is usefull for Maven based (re-)deployments.

At my current assignment I need to trouble shoot the identity service because of a BPM->OID coupling. I use the support document 1327140.1 for it, that suggest to test http://<soa-server>:<port>/integration/services/IdentityService/identity

Doing so in a production mode soa or bpm environment, you'll soon find out that it uses the WebService test client via uri /ws_utc, and that this does not work. Resulting in http-404 Not found errors.

First I found a blog of Maarten of Amis mentioning this as well. But unfortunately, he could not get around it either. But luckily I found note 1915317.1, that tells me that the WebServices test Client is not enabled by default.

You can enable it on your domain via the EM:

And then expand the Advanced node:


Check the 'Enable Web Service Test Page' check box.

Since it is about a production mode environment, you need to 'lock & edit'. The note suggests to do that in the /console and then do the change in /em. And back in /console do the activate. I found that peculiar, since you can do it in the change center in /em as well.

You need to restart the servers (apparently including the AdminServer) to get this in effect.

So now lunch and check if my restart worked.