Earlier you could have enjoyed my article on the automatic generation of a SOA/OSB domain. Earlier this week I encountered some issues with a domain created at a customer this way.

I got the change to dive into that this week and luckily not only I learned a lot again, but I found the problem as well. I adapted my scripts. I won't repost them completely, I've created a github account, and try to place them there in the near future.

But I'll cover the changes, especially those that caused the problems.

Enrollment

When you create a domain using the config.sh/.cmd wizard, and choose to configure the nodemanager, you'll be asked for the nodemanager user and password. You'll get a per domain nodemanager for free and the domain is enrolled for the nodemanager for you. You might need to adapt the nodemanager.properties in <domain_home>/nodemanager and set the property SecureListener to false. The default is true, while in the Machine definition in the domain the nodemanager/machine is configured to SSL. If you disable the SecureListener, you need to set the property in the wls-console under Machines to Plain instead of SSL. And adapt the listener-port.

The docs on configuring the nodemanager for 12.2.1 can be found here. One of the biggest changes between 11g and 12cR1 is that in 12cR2 you get a per domain nodemanager, with a nodemanager home in the domain home, instead of a nodemanager home in the FMW_HOME, the home of the binaries. Also in the bin folder of the domain you'll find scripts to start and stop the nodemanager.

So for a domain configured in the config-wizard, you need little to do to get the nodemanager working.

But using the scripts in my previous article, you'll find that although a nodemanager password file is created, something is missed for a proper startup of your servers using the nodemanager. You'll find that although the nodemanager starts, connecting to it using nmConnect(), fails with: a message like:

WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Access to domain 'osb_domain' for user 'weblogic' denied.

You can go to the Domain->security settings in the weblogic console, and change the nodemanager password, but apparently this is not enough. You'll need to explicitly enroll the domain against the nodemanager. To do so:

• Start the AdminServer, using the startWebLogic.sh/.cmd script in the domain home.
• Start wlst.sh/.cmd
• Connect to the AdminServer using:  connect(adminUser, adminPwd, adminURL)
• Perform an NodeManager Enroll using: nmEnroll(soaDomainHome, nodeManagerHome)
•  Stop the AdminServer
• (Re)Start the nodemanager
• In wlst: Connect to the nodemanager using nmConnect(....)
• Perform nmStart('AdminServer')

The following enrollDomain.py script might help:

#############################################################################
# Create a SOA/BPM/OSB domain
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.1, 2016-06-23
#
#############################################################################
# Modify these values as necessary
import sys, traceback
scriptName = 'enrollDomain.py'
#
#Home Folders
soaDomainHome   = domainsHome+'/'+soaDomainName
nodeManagerHome = soaDomainHome+'/'+'nodemanager'
#
#
lineSeperator='__________________________________________________________________________________'
#
#
def usage():
  print 'Call script as: '
  print 'Windows: wlst.cmd '+scriptName+' -loadProperties localhost.properties'
  print 'Linux: wlst.sh '+scriptName+' -loadProperties environment.properties'
  print 'Property file should contain the following properties: '
  print "adminUrl=localhost:7101"
  print "adminUser=weblogic"
  print "adminPwd=welcome1"
#
#
def main():
  try:
    #
    # Section 1: Base Domain + Admin Server
    print (lineSeperator)
    print ('Enroll '+soaDomainName+' for NodeManager')
    print('\nConnect to AdminServer ')
    print (lineSeperator)
    adminURL=adminListenAddress+':'+adminListenPort
    connect(adminUser, adminPwd, adminURL)
    #
    print('\nPerform nmEnroll')
    print (lineSeperator)
    #
    nmEnroll(soaDomainHome, nodeManagerHome)
    #
    print ('\nFinished')
    #
    print('\nExiting...')
    exit()
  except NameError, e:
    print 'Apparently properties not set.'
    print "Please check the property: ", sys.exc_info()[0], sys.exc_info()[1]
    usage()
  except:
    apply(traceback.print_exception, sys.exc_info())
    stopEdit('y')
    exit(exitcode=1)
#call main()
main()
exit()

It can be called with a shell script (enrollDomain.sh) like:

#!/bin/bash
. fmw12c_env.sh
echo
echo Enroll domain
wlst.sh enrollDomain.py -loadProperties darlin-vce-db-osb.properties

I also adapted the function 'createUnixMachine' in the createSoaBpmDomain.py script:

#
# Create a Unix Machine
def createUnixMachine(serverMachine,serverAddress, serverPort, nmType):
  print('\nCreate machine '+serverMachine+' with type UnixMachine')
  print (lineSeperator)
  cd('/')
  create(serverMachine,'UnixMachine')
  cd('UnixMachine/'+serverMachine)
  create(serverMachine,'NodeManager')
  cd('NodeManager/'+serverMachine)
  set('ListenAddress',serverAddress)
  set('ListenPort',int(serverPort))
  set('NMType',nmType)

This allows for the non-default setting of the nodemanager to another listen port and nodemanager type, based on the following properties in the property file:

#
# Server Settings
nmType=Plain
server1Machine=darlin-vce-db
server1Address=darlin-vce-db
server1Port=5555
server2Enabled=false
server2Machine=darlin-vce-db
server2Address=darlin-vce-db2
server2Port=5555

An example of the property file can be found in the earlier article: automatic generation of a SOA/OSB domain.

Logging

In the example property file in my scripts you can set a location for the logging:

# Logs
logsHome=/u01/app/work/logs

If this is a absolute path, 'nothing is on the hand'. Your domain should start correclty. But if you choose to use a relative path, like 'logs', then starting the AdminServer might succeed (in my case) but starting the managed servers fail with a 'No such file or directory' message:

wls:/nm/osb_domain> nmStart('Adminserver')
 
Starting server Adminserver ...
 
Traceback (innermost last):
 
  File "<console", line 1, in ?
 
  File "<iostream", line 188, in nmStart
 
  File "<iostream>", line 553, in raiseWLSTException
 
WLSTException: Error occurred while performing nmStart : Error Starting server Adminserver : Received error message from Node Manager Server: [Server start command for WebLogic server 'Adminserver' failed due to: [No such file or directory]. Please check Node Manager log and/or server 'Adminserver' log for detailed information.]. Please check Node Manager log for details.
 
Use dumpStack() to view the full stacktrace :
 
wls:/nm/osb_domain

Now it turns out that this is caused by the JavaArgs that are generated and set in the script. I found that if you set JavaArgs you need to set redirects for weblogic.Stdout and weblogic.Stderr, like:

'-XX:PermSize=256m -XX:MaxPermSize=512m -Xms1024m -Xmx1532m -Dweblogic.Stdout='+logsHome+'AdminServer.out -Dweblogic.Stderr='+logsHome+'AdminServer_err.out'

Where logFolder should be an absolute path. This has to do with the context in which the nodemanager is starting the server. From that context the relative reference apparently does not evaluate to the proper location.
You can however, leave the Java args empty. So I changed my scripts to not use the getServerJavaArgs function, anymore, but get them from the property file. I replaced the xxxJavaArgsBase  with xxxJavaArgs variables. And left them empty.

The configurator doesn't set the JavaArgs, it leaves them over to the setDomain.sh/.cmd and setStartupEnv.sh/.cmd. If you do so, you can use a relative path, and the servers will start properly.