Monday, 18 December 2017

Create the SOA/BPM Demo User Community, with just WLST.

As said in my previous post (I've learned somewhere you should not post twice on the same day, but spread it out over time), I'm delivering a BPM 12c training. And based it on the BPM Quickstart. Although nice for UnitTests and development, the integrated weblogic lacks a  proficient set of users to test your task definitions.

Oracle has a demo community and a set of ANT and Servlet based scripts to povision your SOA or BPMSuite environment with a set of American literature writers, to be used in demo's and trainings. I some how found this years ago and had it debugged to be used in 12.1.3 a few years ago. However, I did not know where I got it and if it was free to be delivered.

Apparently it is, and you can find it at my oracle support. Our friends with Avio Consulting also  did a good job in making it available and working with 12c. However, I could not make it work smoothly end 2 end. I got it seeded, but figured that I would not need ANT and a Servlet.

Last year, in 2016, I created a bit of WLST scripting to create users for OSB and have them assigned to OSB Application roles. You can read about that here for the user creation, and here for the app-role assignment.

One thing that's missing in those scripts is the setting of the user attributes. So I googled around and found a means to add those too.

First, I had to transform the demo community seeding xml file to a property file. Like this:

#
cdickens.password=welcome1
cdickens.description=Demo User
cdickens.email=cdickens@emailExample.com
cdickens.title=CEO
cdickens.firstName=Charles
cdickens.lastName=Dickens
cdickens.timeZone=America/Los_Angeles
cdickens.languagePreference=en-US
cdickens.workPhone=100000001
cdickens.homePhone=200000001
cdickens.mobile=300000001
cdickens.im=jabber|cdickens@exampleIM.com

The complete, usersAndGroups.properties file is available here.
In an earlier blog, I wrote about how to read a property file. But my prefered method, does not allow me to determine the property to be fetched dynamically. That's why I split my basic createDemoUsers.properties file, that refers to the usersAndGroups.properties file, and contains the properties refering the Oracle/Jdeveloper home and the connection details for the AdminServer. This property file also contains comma separated lists of users, groups and AppRoles to be created or  granted.

The actual createDemoUsers.py file loops over the three lists and creates the particular users and groups, and grants the AppRoles.

To set the attributes, the setUserAttributeValue of the authenticatorMBean can be used as follows:
    #Set Properties
    firstName=userProps.getProperty(userName+".firstName")
    lastName=userProps.getProperty(userName+".lastName")
    displayName=nvl(firstName, " ")+" "+nvl(lastName, " ")
    authenticator.setUserAttributeValue(userName,"displayName",displayName.strip())

I published the complete set of scripts on the GitHub repo I shared with my colleague.
You can download them all, adapt the createDemoUsers.sh, to refer the correct MW_HOME to your JDeveloper environment. For Windows you might translate it to a .bat/.cmd file.

And of course you can use it for your own set of users.

I think I covered near to all of the Demo User community. Except for management-chains: I could not find how to register a manager for a user in Weblogic. Neither in the console, nor inWLST. So, I currently I conclude it cannot be done. But, if you have a tip, please be so good to leave a comment. I would highly appreciate it.

2018-08-22, Update: found this article referencing the BPM roles, from my appreciated former Whitehorses co-worker. Should be able to integrate this in my scripts.




BPM 12.2.1.3: Exception when deploying BPM project with Human tasks

This week I deliver a BPM 12c Workshop, that I based on the 12.2.1.3 BPM QuickStart. When the students worked on the lab on Human Workflow, they hit an error deploying the Composite, where in the log you can find something like:

Caused By: oracle.fabric.common.FabricException: Error occurred during deployment of component: RequestHolidayTask to service engine: implementation.workflow, for composite: HolidayRequestProcess: ORABPEL-30257
 
exception.code:30257
exception.type: ERROR
exception.severity: 2
exception.name: Error while Querying workflow task metadata.
exception.description: Error while Querying workflow task metadata.
exception.fix: Check the underlying exception and the database connection information.  If the error persists, contact Oracle Support Services.
: exception.code:30257
exception.type: ERROR
exception.severity: 2
exception.name: Error while Querying workflow task metadata.
exception.description: Error while Querying workflow task metadata.
exception.fix: Check the underlying exception and the database connection information.  If the error persists, contact Oracle Support Services.
 
Caused By: oracle.fabric.common.FabricDeploymentException: ORABPEL-30257
 
 
Caused By: java.sql.SQLSyntaxErrorException: Column 'WFTM.PACKAGENAME' is either not in any table in the FROM list or appears within a join specification and is outside the scope of the join specification or appears in a HAVING clause and is not in the GROUP BY list. If this is a CREATE or ALTER TABLE  statement then 'WFTM.PACKAGENAME' is not a column in the target table.

Apparently in the repository of 12.2.1.3 a column is missing in the Workflow Metadata table.

Luckily, I stumbled upon a question in the community.oracle.com forum that hit this 'bug' as well; and provided a solution. You need to do an alter table to resolve this:
ALTER TABLE SOAINFRA.WFTASKMETADATA ADD PACKAGENAME varchar (200);

The smart guy that provided the answer, used a separate Database UI tool. But fortunately, JDeveloper is perfectly capable to provide you de means as well.

First open the Resource Pallette in JDeveloper. Make sure that you have started your Integrated WebLogic already (since that will run the DerbyDB.

Then in the Resource Pallette, create a new Database Connection:


Provide the following details:
Give it a name, like soainfraDB, as a Connection Type select 'Java DB / Apache Derby'. You can leave Username and Password empty. Then as a Driver Class, choose the 'org.apache.derby.jdbc.ClientDriver' (not the default). Then as a Host Name provide localhost, as a JDBC Port enter 1527 en as  a Database Name enter soainfra.

You can  Test Connection  and then, if Successfull, hit OK.

Then from the IDE Connections pallette  right click your newly created database connection, and choose 'Open in Databases Window':



And from that right click on the database connection and choose 'Open SQL Worksheet':

There you can enter and execute the alter statement:
After this, deployment should succeed. Since it is persisted in the DerbyDB it will survive restarts.

This might apply to the SOA QuickStart as well (did not try).

Friday, 1 December 2017

OSB: Disable Chunked Streaming Mode recommendation

Intro

These weeks I got involved in a document generation performance issue. This ran for several months, maybe years even. But it stayed quite unclear what the actual issue was.

Often we got complaints that document generation from the front-end application (based on Siebel) was taking very long. End users often hit the button several times, but with no luck. Asking further, it did not mean that there appeared a document in the content management system (Oracle UCM/WCC). So, we concluded that it wasn't so much a performance issue, but an exception along the process of document generation. Since we upgraded BI Publisher to 12c, it was figured that it might got something to do with that. But we did not find any problems with BI Publisher, itself. Also, there was an issue with Siebel it's self, but that's also out of the scope of this article.

The investigation

First, on OSB the retry interval of the particular Business Service was decreased from 60 seconds to 10. And the performance increased. Since the retry interval was shorter, OSB does a retry on shorter notice. But of course this did not solve the problem.

As Service developers we often are quite laconical about retries. We make up some settings. Quite default is an interval of 30 seconds and a retry count of 3. But, we should actually think about this and figure out what the possible failures could be and what a sensible retry setting would be. For instance: is it likely that the remote system is out of order? What are the SLA's for hoisting it back up again? If the system startup is 10 minutes, then a retry count of 3 and interval of 30 seconds is not making sense. The retries are done long before the system's up again. But of course, in our case sensible settings for system outage would cause delays being too long. We apparently needed to cater for network issues.

Last week our sysadmins encountered network failures, so they changed the LoadBalancer of BIP Publisher, to get chunks/packets of one requests routed to the same BI Publisher node. I found SocketReadTimeOuts in the logfiles. And from the Siebel database a query was done and plotted out in Excel showing lots of request in the  1-15 seconds range, but also some plots in ranges around 40 seconds and 80 seconds. We wondered why these were.

The Connection and Read TimeOut settings on the Business Service were set to 30s. So I figured the 40 and 80 seconds range could have something to do with a retry interval of 10s added to a time out of 30 seconds.

I soon found out that in OSB on the Business Service, the Chunked Streaming Mode  was enabled. This is a setting we struggled with a lot. Several issues we encountered were blamed on this one. As a Helpdesk employee would ask you if you have restarted your system, on OSB questions I would ask you about this setting first... Actually, I did for this case, long before I got actively involved.

Chunked Streaming Mode explained

Let's start with a diagram:

In this diagram you'll see that the OSB is fronted by a Load Balancer. But since 12c the Oracle HTTP Server is added to the Weblogic Infrastructure. And following the Enterprise Deployment Guide we added an OHS to the Weblogic Infrastructure Domain, as a co-located OHS Instance. And since the OSB as well as the Service Provider (in our case BI Publisher) are clustered, the OHS will load balance the requests.

Now, the Chunked transfer encoding is an HTTP 1.1 specification. It is an improvement that allows clients to process the data in chunks right after the chunk is read. But in most (of our) cases a chunk on itself is meaning-less, since a SOAP Request/XML Document need to be parsed as a whole.
The Load Balancer also process the chunks as separate entities. So,by default, it will route the first one to the first endpoint, and the other one to the next. And thus each SP Managed Server gets an incomplete message and there for a so-called Bad Request. This happens with big requests, where for instance a report is requested together with the complete content. Then chances are that the request is split up in chunks.

But although the SysAdmins adapted the SP Load Balancer, and although I was involved in the BIPublisher 12c setup, even I forgot about the BIP12c OHS! And even when the LoadBalancer tries to keep the chunks together, then again the OHS will mess with them. Actually, if the LoadBalancer did not keep them together, the OHS instances could reroute them again to the correct end-node.

The Solution

So for all those Service Bus developers amongst you, I'd like you to memorize two concepts: "Chunked Streaming Mode" and "disable", and the latter in combination with the first, of course.
In short: remember to set Chunked Streaming Mode to disable in every SOAP/http based Business Service. Especially with services that send potentially large requests, for instance document check-in services on Content/Document Management Systems.

The proof of the pudding

After some discussion and not being able to test it on the Acceptance Test environment, due to rebuilds, we decided to change this in production (I would/should not recommend that, at least not right away).

And this was the result:


This picture shows that the first half of the day, plenty requests were retried at least once, and several even twice. Notice the request durations around the 40 seconds (30 seconds read timeout + 10 seconds retry interval) and 80 seconds. But since 12:45, when we disabled the Chunked Streaming Mode we don't see any timeout exceptions any more. I hope the end users are happy now.

Or how a simple setting can throw a spanner in the works. And how difficult it is to get such a simple change into production. Personally I think it's a pity that the Chunked Streaming Mode is enabled by default, since in most cases it causes problems, while in rare cases it might provide some performance improvements. I think you should rationalize the enablement of it, in stead of actively needing to disable it.

Friday, 24 November 2017

BPM BAC Subversion Server refusing connections Revised

A little background

In April I wrote about our BPM Server installation, that is actually a single host cluster on dev and test. But installed like it was a dual-node  server, so we had a cloned domain.

A BPM installation has a component that is called the Process Asset Manager. Under the hood it uses a replicated SubVersion server. Each node has one, so they had to synchronize. But since we're on the same host, we needed to differentiate in port numbers. Although we used virtual host names for each server node. In an article in april 'BPM BAC Subversion Server refusing connections' I wrote about that.

It turns out: it did not work appropriately. After some investigation, apparently the bac_node1/subversion server calls the subversion server on bac_node2, with it's own address, but with the port of the other:
...
<Jul 24, 2017, 2:37:29,543 PM CEST> <Debug> <oracle.bpm.bac.svnserver.replication> <darlin01> <bpm_server1> <Active Sync Thread [/54680efc-9328-478e-953c-834bbb250725/]> <<anonymous>> <> <20a54a15-d8d8-4e58-a4f6-11a90e992967-00000008> <1500899849543> <[severity-value: 128] [rid: 0:490:13:19] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000000> <[oracle.bpm.log.Logger:debug] About to synchronize against node Member(Id=2, Timestamp=2017-07-12 08:01:13.309, Address=10.0.0.38:37055, MachineId=9002, Location=site:tst.darwin-it.local,machine:bpm_machine1,process:24566,member:bpm_server2, Role=bpm_cluster), url svn://syncuser@t-bpm-1-bpm-1-vhn.tst.darwin-it.local:8424/54680efc-9328-478e-953c-834bbb250725>
...

Of course there's no listen address so unsurprisingly we get errors like:
...
[2017-05-30T12:55:51.704+02:00] [bpm_server1] [ERROR] [] [oracle.bpm.bac.svnserver.replication] [tid: Active Sync Thread [/b91abb78-6b3d-4448-af6a-e82125f261f0/]] [userId: ] [ecid: ed0bc6ce-fefb-4608-8dbe-46f7206a1573-0000000a,0:527] [APP: OracleBPMBACServerApp] [partition-name: DOMAIN] [tenant-name: GLOBAL] org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server[[
oracle.bpm.bac.subversion.server.repository.exceptions.RepositoryException: org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server
at oracle.bpm.bac.subversion.server.repository.exceptions.RepositoryException.wrap(RepositoryException.java:56)
at oracle.bpm.bac.subversion.server.repository.SVNKitRepositorySession.getRepositoryUUID(SVNKitRepositorySession.java:98)
at oracle.bpm.bac.subversion.server.repository.RepositorySVNSync.sync(RepositorySVNSync.java:74)
at oracle.bpm.bac.subversion.server.repository.RepositorySVNSync.sync(RepositorySVNSync.java:59)
at oracle.bpm.bac.subversion.server.repository.ha.aa.ActiveAARepository$Synchronizer.runImpl(ActiveAARepository.java:341)
at oracle.bpm.bac.subversion.server.repository.ha.aa.ActiveAARepository$Synchronizer.run(ActiveAARepository.java:304)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.tmatesoft.svn.core.SVNException: svn: E210003: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:85)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:62)
at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:77)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1252)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:95)
at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:280)
at oracle.bpm.bac.subversion.server.repository.SVNKitRepositorySession.getRepositoryUUID(SVNKitRepositorySession.java:95)
... 5 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:112)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:68)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:53)
... 10 more

]] 

(to get this page searchable).

Solution

But now, since this week, Oracle Support presents us.... (drum rolls)  Patch 26775572 for version 12.2.1.2.0, the official fix for this issue, is finally ready and available for download.
I hope to be able to install it next monday. But feel free to try with me: 'Patch 26775572: BAC node using the wrong port when attempts synchronization with Virtual IP '

By the way, for about 2 months we have diagnostic patch running that solves this. But named patch is the official delivery of our diagnostic variant.

By the way, since 12.2.1.3.0 is out in the field for quite some time already, I expect that this did not landed in that version yet. If you're running in the same issue for 12.2.1.3.0 then create a SR to request for a port of this patch.


Wednesday, 15 November 2017

SOASuite 12c: keep running instances using ANT

At my current customer I implemented a poor man's devops solution for Release and Deploy. It was based on a framework created as bunch of Ant projects, that I created years ago. It was based on scripts from Edwin Biemond. See for instance here, here and here. I never wrote about my solution, because although I refactored them quite intensively, the basics were already described thoroughly by him.

What I did was that I modularized the lot, split the environment property files, added logging, added OSB 12c  support, based on the config jar tool, etc.

One thing I ran into this week was that at first deployment from our team to the test environment using my framework, the running instances for the BPM projects were aborted.

Now, if you take a look at the deploy.sarLocation target in Edwin's article about deploying soa suite composites  you'll find that he also supported the overwrite and forceDefault properties.

When re-deploying a composite from JDeveloper you're probably familiar with the 'keep running instances' checkbox. I was looking for the ANT alternative in the ${oracle.home}/bin/ant-sca-deploy.xml scripting. First I looked in the 12c docs (see 47.9.4 How to Use ant to Deploy a SOA Composite Application), but it is not documented there.

But when I opened the particular ant-sca-deploy.xml script I read:
 <condition property="overwrite" value="false">
    <not>
      <isset property="overwrite"/>
    </not>
  </condition>
  <condition property="forceDefault" value="true">
    <not>
      <isset property="forceDefault"/>
    </not>
  </condition>
  <condition property="keepInstancesOnRedeploy" value="false">
    <not>
      <isset property="keepInstancesOnRedeploy"/>
    </not>
  </condition>
 ...
  <target name="deploy">
    <input message="Please enter server URL:" addproperty="serverURL"/>
    <input message="Please enter sar location:" addproperty="sarLocation"/>
    <input message="Please enter username:" addproperty="user"/>
    <input message="Please enter password:" addproperty="password">
      <handler classname="org.apache.tools.ant.input.SecureInputHandler" />
    </input>
    <deployComposite serverUrl="${serverURL}" sarLocation="${sarLocation}" realm="${realm}" user="${user}" password="${password}"
      overwrite="${overwrite}" forceDefault="${forceDefault}" keepInstancesOnRedeploy="${keepInstancesOnRedeploy}"
      regenerateRuleBase="${regenerateRuleBase}" configPlan="${configplan}" scope="${scope}"
      sysPropFile="${sysPropFile}" failOnError="${failOnError}" timeout="${timeout}" folder="${folder}"/>
  </target>

So, the script kindly supports the keepInstancesOnRedeploy property. And thus I implemented a deploy.keepInstancesOnRedeploy property the same way as the deploy.forceDefault/deploy.overwrite properties.

This probably is usefull for Maven based (re-)deployments.

Thursday, 2 November 2017

Enable WebService test client on SOA/BPM production mode environments

At my current assignment I need to trouble shoot the identity service because of a BPM->OID coupling. I use the support document 1327140.1 for it, that suggest to test http://<soa-server>:<port>/integration/services/IdentityService/identity

Doing so in a production mode soa or bpm environment, you'll soon find out that it uses the WebService test client via uri /ws_utc, and that this does not work. Resulting in http-404 Not found errors.

First I found a blog of Maarten of Amis mentioning this as well. But unfortunately, he could not get around it either. But luckily I found note 1915317.1, that tells me that the WebServices test Client is not enabled by default.

You can enable it on your domain via the EM:

And then expand the Advanced node:


Check the 'Enable Web Service Test Page' check box.

Since it is about a production mode environment, you need to 'lock & edit'. The note suggests to do that in the /console and then do the change in /em. And back in /console do the activate. I found that peculiar, since you can do it in the change center in /em as well.

You need to restart the servers (apparently including the AdminServer) to get this in effect.

So now lunch and check if my restart worked.

Friday, 8 September 2017

Implementing the KeyStore Service with Fusion MiddleWare 12c

For the passphrases, use the passphrases used earlier.Thinking about TLS (Transport Layer Security, the succesor of Secure Socket Layer, SSL) and WebLogic and Oracle HTTP Server, allways gave me Cold Water Fear. You have to create keystores with keys, wallets, certificate signing requests, import signed and trusted certificate chains. Not to mention the configuration of WebLogic and OHS.

Now, creating keystores with the Java Keytool turns out not that hard. And generating the Certificate signing requests and importing the certificates are also a walk in the park, nowadays. The internet world is full of example so I'm not going to do that here.

But lately, our Service Bus developers found that they needed to replace the configured demo identity and trust key stores with Custom Stores. But this broke the connection between the AdminServer and the Nodemanagers, resulting in TLS/SSL Handshake errors. By default, the nodemanagers work with the demo-identities when running in TLS.

This drove us to work out an infrastructural configuration of TLSin our FMW environments, in a  way that the SB developers can extend that with their certificates.

In this article I want to describe how to configure TLSin Weblogic using the KeyStore Service, and also how to reconfigure the nodemanagers to have them running TLS using the custom stores.

Keystores and the KeyStore Service (KSS)

When implementing in TLS in Fusion MiddleWare 12c you have the choice to use the new KeyStore Service for creating keys and certificates directly in the KSS, or the Java Key Tool.

Until now WebLogic preferred a Java Key Store (JKS) for storing certificates. This is a file that functions as a vault to store your keys and certificates savely. You can use the commandline tool Keytool that is delivered with the JDK. But there are several graphical tools, like Portecle, that can make your encrypting life even more simple.

Since a keystore is a atomic file, you need to copy it or put it on a share to have a complete multi-node clustered domain use the same store.

Per Fusion MiddleWare 12c Oracle introduced the KeyStoreService. The KSS is part of the Oracle Platform Security Services, and stores the keys in so-called Stripes in the MDS. For some components it is necessary to sync them to a keystores.xml file in the domain. But we'll get to that.
The KSS enables you to create, delete and manage keystores from the Enterprise Manager/FustionMiddleWare Control.

As said, you can start with creating your keys using the KSS. We however did choose not to. We started with creating a JKS using the KeyTool. The reaons for that were:
  • Using the keytool, you can already create a store and a Certificate Signing Request (CSR) before configuring your domain.
  • We wanted to make sure we had the configuration of Weblogic and Nodemanager correct, before transition to KSS. 
  • With a JKS you have a backup of your identity in case you mess up your configuration. Keep in mind that you need to import the CSR into the keystore from where you generated the key. Even a new JKS created the exact sameway won't work.
  • Using OraPKI, you can migrate your certificates from a JKS to a wallet. We are still in the 'figure-out-phase' of implementing OHS with KSS.
  • Oh, and we wanted to have scripts and easy to document commands for creating and importing the stores.

Identity and Trust Store

So we started with creating  an Identity Keystore with the Java Keytool, created a CSR and had it signed.
We had several virtual hostnames refering to the hostname (darlin001.org.darwin-it.local) loadbalancer (o-dwn-1.ont.org.darwin-it.local), managed servers (o-dwn-1-dwn-1-vhn.ont.org.darwin-it.local), Admin Virtual host (o-dwn-1-admin-vhn.ont.org.darwin-it.loca), etc. To have them accepted with the same certificate, you need to create the Certificate Signing Request with an extention called 'Subject Alternative Names'. This can be done with the -ext parameter including the SAN keyword as follows:
... -ext SAN=dns:darlin001.org.darwin-it.local,dns:o-dwn-1-admin-vhn.ont.org.darwin-it.local,dns:o-dwn-1-dwn-1-vhn.ont.org.darwin-it.local,dns:o-dwn-1-dwn-2-vhn.ont.org.darwin-it.local,dns:o-dwn-1-ohs-1-vhn.ont.org.darwin-it.local,dns:o-dwn-1-internal.ont.org.darwin-it.local,dns:o-dwn-1-admin.ont.org.darwin-it.local,dns:o-dwn-1.ont.org.darwin-it.local
This is a comma seperated list, with each addres need to be prefixed with dns:. See also the keytool documentation.

Don't forget to add the Common Name (CN) to the SAN, since clients are supposed to ignore the CN, when using a SAN. Also you need to add the -ext SAN parameter to the CSR. You can use it when creating the Key, but that's not enough for the CSR.

For the Trust Store, I copied the cacerts from the JDK. I changed the password and imported the CA and root certificates. But you need to consider if you want to accept all the JDK's trusted certificates. Or only those you really need to trust.

 Scope of the Keystore

We choose to have the domain the scope of the keystore, not the host. So as a CommonName I choose the loadbalancer address. The thing is that we want to have the managed servers use the same keystore. And with a whole server migration, a managed server could migrate to a new host, that is not yet in the certificate. Although we add the known hosts, we could be forced to use a new host. And with whole server migrations, you can't way for your CSR to be signed.
As name of the keystores and the identity alias I choose a reference to our domain name. As an environment shortage I choose 'dwn', that could also refer to the type of environment. Like BPM, OSB, SOA, etc. The first letter ('o') denotes the development environment (Dutch: 'ontwikkeling'). And the digit denotes the environment number, where you could have multiple development or test environment.

Import KeyStores into KSS

Having created your KeyStores and made sure Weblogic works with them, you can import them into the KSS. The UI (EM) does allow you to create a keystore, and import certificates. It does not allow you to import a complete JKS. But you can do so using WLST. This caters also for my quest to script as much as possible.

The commands are executed in WLST Online mode, connected to the AdminServer. So start wlst and connect to the AdminServer.

In the following command commands, I use the following environment variables:
  • $JAVA_HOME: refering to, yes, you're right....
  • $JKS_LOC=/u01/oracle/config/keystore
  • $JKS_NAME=o-dwn-1.jks
  • $JKS_TRUST_NAME=o-dwn-1-trust.jks
Connected to your AdminServer you need to get a handle on the 'KeyStoreService object', that has methods to do the imports,  etc.:
wls:/o-dwn-1_domain/serverConfig/> svc = getOpssService(name='KeyStoreService')

With the handle you can import the Identity keystore:
wls:/o-dwn-1_domain/domainRuntime/> svc.importKeyStore(appStripe='system', name='o-dwn-1-id', password='<password>', aliases='o-dwn-1', keypasswords='<password>', type='JKS', permission=false, filepath='/u01/oracle/config/keystore/o-dwn-1.jks')
Already in Domain Runtime Tree 

Keystore imported. Check the logs if any entry was skipped.

The parameters aliases and keypasswords contain a comma-separated list of key-aliases and corresponding key-passphrases.
My advise would be to use the same password for your key as for your keystore. Many products (like Oracle B2B) allow you to provide only one password, that is used for both.

Importing the Trust Store, works similar:
svc.importKeyStore(appStripe='system', name='o-dwn-1-trust', password='<password>', aliases='ca_dwn_org,our-root-ca', keypasswords='none,none', type='JKS', permission=false, filepath='/u01/oracle/config/keystore/o-dwn-1-trust.jks')
Already in Domain Runtime Tree

Keystore imported. Check the logs if any entry was skipped.

One thing with Trusts: they don't contain Keys. So no keyprases are applicable. But the keypasswords property is mandatory and also need to contain the same number of passphrases as there are aliases named to be imported in the KSS. Apparently it is sufficient to use 'none' as a passphrase.

A wallet can also be imported the same way. For a type then choose 'OracleWallet', and in filepath a reference to the wallet folder.

To check the content of the keystore, we can list it:
wls:/o-dwn-1_domain/domainRuntime/> svc.listKeyStoreAliases(appStripe="system",name="o-dwn-1-id",password='<password>',type="*")
Already in Domain Runtime Tree

o-dwn-1

And for the trust:
wls:/o-dwn-1_domain/domainRuntime/> svc.listKeyStoreAliases(appStripe="system",name="o-dwn-1-trust",password='<password>',type="*")
Already in Domain Runtime Tree

ca_dwn_org
our-root-ca

Now the keystores are in the KSS. You can see them in EM. But some FMW components like nodemanagers, (but it appeared to me even Weblogic Servers), don't work right away. They can't look into the KSS, but use a keystores.xml file in the ${DOMAIN_HOME}/config/fmwconfig. This turns out to be an export of the keystores. To synchronize the KSS with that file you need to do:
wls:/o-dwn-1_domain/domainRuntime/> syncKeyStores(appStripe='system',keystoreFormat='KSS')
Already in Domain Runtime Tree

Keystore sync successful.

List the content of KSS with FMW Control

To list the KSS within FMW Control, logon and navigate to Weblogic Domain -> Security -> Keystore:
This brings you to the content of the KSS with the particular Stripes. For Weblogic we used the system stripe to import our custom keystores:

Select a store, and click the manage button in the bar. It will ask for the keystore password (the one you used creating the JKS with the keytool). Then it shows:


Here you can import, export certificates, create keypairs and generate a CSR for it.

Reconfigure Weblogic

Now we can proceed in configuring Weblogic and the nodemanagers.

For the weblogic servers, logon to the Admin console, navigate to the particular Server, tab Keystores and select 'Custom Identity Keystore and Custom Trust keystore' in the pop-list.

Then enter the following values:
Attribute
Value
Custom Identity Keystore kss://system/o-dwn-1-id
Custom Identity Keystore Type KSS
Custom Trust Keystore kss://system/o-dwn-1-trust
Custom Trust Keystore Type KSS

For the passphrases, use the passphrases used earlier.
Then on the SSL tab you can provide the Identity alias and the key-passphrase.

Restart the Server (or at least Restart the SSL, under the control tab).

Reconfigure Nodemanagers for KSS

For the nodemanager, navigate to the nodemanager.properties file in ${DOMAIN_HOME}/nodemanager.
At the top of the file add the following properties:
KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreFileName=kss://system/o-dwn-1-id
CustomIdentityKeyStoreType=KSS
CustomIdentityKeyStorePassPhrase=
CustomIdentityAlias=o-dwn-1
CustomIdentityPrivateKeyPassPhrase=

Restart the nodemanager and check if it succesfully loads the kss://system/o-dwn-1-id as a keystore and starts the listener in SSL mode.

Then in the AdminServer console, under Domain Structure -> Machines -> <Machine_Name> check if the NodeManager is 'Reachable'.

Conclusion

That's about it. Don't forget to Sync the KSS in wlst, when the nodemanager does not restart properly and the managed/admin server could not start SSL properly.

 I hope I'll soon be able to write down the directions to reconfigure SSL in OHS to use the KSS. But as said we're still in the 'figure-out-phase'.