At one of my customers we have quite an impressive domain configuration. It's a FMW domain with SOA, OSB, BAM, WSM, MFT in clusters of 4 nodes. The thing is that when having started all the servers, the console becomes slooooooowwwww. Not to speak of FMW Control (em).

One suggestion is to set the 'Invocation Timeout Seconds' under MyDomain->Configuration->General->Advanced to a value like 2. And 'Management Operation Timeout' under Preferences->Shared Preferences to a value like 5:

This surely makes the console responsive again. But it actually means that the console gives up right away when querying for the (health) state of the servers. So in stead of a health of 'OK', you get a 'server unreachable' message.

When having a lot of servers in the domain, they all share the same Admin Channel, and this seams to get over flooded. AdminServer does not get the responses in time. Sometimes a new request leads to a proper response, but in fact it takes a lot of time.

To reduce the load on the default channel, we created an admin channel per managed server. Since it's a lot of servers, and we need to do it on several environments, so I created a wlst-script for it.
The script createAdminChannels.py:

#############################################################################
# Create AdminChannels for WebLogic Servers
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.1, 2016-08-24
#
#############################################################################
# Modify these values as necessary
import sys, traceback
scriptName = sys.argv[0]
#
#
lineSeperator='__________________________________________________________________________________'
#
#
def usage():
  print 'Call script as: '
  print 'Windows: wlst.cmd '+scriptName+' -loadProperties localhost.properties'
  print 'Linux: wlst.sh '+scriptName+' -loadProperties environment.properties'
  print 'Property file should contain the following properties: '
  print "adminUrl=localhost:7001"
  print "adminUser=weblogic"
  print "adminPwd=welcome1"
#
#
def connectToadminServer(adminUrl, adminServerName):
  try:
    print(lineSeperator)
    print('Try to connect to the AdminServer')
    try:
      connect(userConfigFile=usrCfgFile, userKeyFile=usrKeyFile, url=adminUrl)
    except NameError, e:
      print('Apparently user config properties usrCfgFile and usrKeyFile not set.')
      print('Try to connect to the AdminServer adminUser and adminPwd properties')
      connect(adminUser, adminPwd, adminUrl)
  except WLSTException:
    message='Apparently AdminServer not Started!'
    print (message)
    raise Exception(message)
#
#
def createAdminChannel(serverName, adminListenPort):
  print(lineSeperator)
  channelName = serverName+'-AdminChannel'
  try:
    cd('/Servers/'+serverName+'/NetworkAccessPoints/'+channelName)
    print('Channel '+channelName +' for '+serverName+' already exists.')
  except WLSTException: 
    try:
      print('Create Admin Channel for server: '+serverName+', with port: '+adminListenPort)
      cd('/Servers/'+serverName)
      cmo.createNetworkAccessPoint(channelName)
      cd('/Servers/'+serverName+'/NetworkAccessPoints/'+channelName)
      cmo.setProtocol('admin')
      cmo.setListenPort(int(adminListenPort))
      cmo.setEnabled(true)
      cmo.setHttpEnabledForThisProtocol(true)
      cmo.setTunnelingEnabled(false)
      cmo.setOutboundEnabled(false)
      cmo.setTwoWaySSLEnabled(false)
      cmo.setClientCertificateEnforced(false)
      print ('Succesfully created channel: '+channelName)
    except WLSTException:
      apply(traceback.print_exception, sys.exc_info())
      message='Failed to create channel '+channelName+'!'
      print (message)
      raise Exception(message)
#
#
def main():
  try:
    print (lineSeperator)
    print ('Start Osb Cluster')
    print (lineSeperator)
    print('\nConnect to AdminServer ')
    connectToadminServer(adminUrl, adminServerName)
    print(lineSeperator)
    print('Start Edit Session')
    edit()
    startEdit()
    #
    #Create Admin Channels
    # Administrators
    print('\nCreate Admin Channels')
    serverNameList=serverNames.split(',')
    serverAdminPortList=serverAdminPorts.split(',')
    #
    idx=0
    for serverName in serverNameList:
      adminPort=serverAdminPortList[idx]
      createAdminChannel(serverName, adminPort)
      idx=idx+1
    # Activate changes
    print(lineSeperator)
    print('Activate Changes')
    save()
    activate(block='true')
    #
    print('\nExiting...')
    exit()
  except NameError, e:
    print('Apparently properties not set.')
    print "Please check the property: ", sys.exc_info()[0], sys.exc_info()[1]
    usage()
  except:
    apply(traceback.print_exception, sys.exc_info())
    stopEdit('y')
    exit(exitcode=1)
#call main()
main()
exit()

The shell script to call it, createAdminChannels.sh:

#!/bin/bash
#############################################################################
# Create AdminChannels
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 2.1, 2016-08-24
#
#############################################################################
#  
. fmw12c_env.sh
export PROPERTY_FILE=$1
echo
echo Create Admin Channels
wlst.sh ./createAdminChannels.py -loadProperties $PROPERTY_FILE

And the example property file, darlin-vce-db.properties:

#############################################################################
# Properties voor Creeëren SOADomain
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.0, 2016-04-15
#
#############################################################################
#
# Properties for AdminServer
adminServerName=Adminserver
adminUrl=darlin-vce-db:7001
# AdminUser
adminUser=weblogic
adminPwd=welcome1
#
serverNames=AdminServer,OsbServer1,SoaServer1
serverAdminPorts=7101,8111,8101
#

Call the script as $> createAdminChannels.sh darlin-vce-db.properties

In the property file you'll need to name every server in the property serverNames. And for each server the particular Admin Listen Port in serverAdminPorts, in the exact same order. Start with the AdminServer.


At the end of the script the changes are activated and then the AdminServer listens over https on the changed port.

Important: the servers need to be down, except for the the AdminServer.

Unfortunately the infrastructure database was apparently down. So I haven't been able to start SOA, BAM, etc. to see if it is performant now. But I have high hopes...

Update wlst.sh
When you update the AdminServer as above, it will by default use the DemoTrust keystore. And probably the listenaddress might not necessarily match the url that is used in the connect-URL.

So by default you might run into errors when trying to connect to the AdminServer using wlst.
First you need to adapt the Admin URL to something like 'tls://darlin-vce-db:7101'. Explicitly prefix with 'tls://' and adapt the port to the new admin-listen-port.

Then the wlst script need to be adapted. The following parameters need to be added to the wlst command:

• -Dweblogic.security.SSL.ignoreHostnameVerification=true
• -Dweblogic.security.TrustKeyStore=DemoTrust

To do so, find out where the wlst.sh/cmd script is located. Under linux you can perform:

[oracle@darlin-vce-db bin]$ which wlst.sh
/u01/app/oracle/FMW12210/oracle_common/common/bin/wlst.sh

Of course after setting the weblogic environment (see one of my earlier blogs describing the fmw12c_env.sh script).

Edit the wlst.sh file and go to the bottom of the file, and add the properties to the JVM_ARGS variable:

...
JVM_ARGS="${WLST_PROPERTIES} ${JVM_D64} ${UTILS_MEM_ARGS} ${CONFIG_JVM_ARGS} -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.TrustKeyStore=DemoTrust"
if [ -d "${JAVA_HOME}" ]; then
 eval '"${JAVA_HOME}/bin/java"' ${JVM_ARGS} weblogic.WLST '"$@"'
else
 exit 1
fi

But setting the CONFIG_JVM_ARGS in a script like fmw12c_env.sh might be a better idea:

...
export CONFIG_JVM_ARGS='-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.TrustKeyStore=DemoTrust'
...