Tuesday, 12 June 2018

The Medrec 12.2.1.3 Datamodel DDL

Next week I deliver the training 'Weblogic 12c Tuning and Troubleshooting' . One of the labs is to have the sample application MedRec generate Stuck Threads, so that the students can investigate and try to solve that. Or actually configure the server so that it will cause a automatic restart.

To do so I have to deliberately break the application and so I need the source. I have an earlier version of the application, but not the sources. So I have to go to the latest MedRec. I actually like that, because it looks more modern.

The MedRec application is available if you install WebLogic with samples.

You can run the script demo_oracle.ddl  from against the database:
$WL_HOME/samples/server/examples/src/examples/common/ddl

The medrec.ear can be found at:
$WL_HOME/samples/server/medrec/dist/standalone

I ran in quite some confusion and frustration, but I found that this combination, although from the same samples folder, does not work. Not only this medrec.ear expects the tables in plural (PRESCRIPTIONS) where the script creates them in singular (PRESCRIPTION), it expects a separate DRUGS table with a foreign key column DRUG_ID in PRESCRIPTIONS.  And a few other changes.

I had a version of the scripts from earlier versions of WebLogic's MedRec. Based the exceptions in the server log, I refactored/reverse engineered the scripts.

Using those I could succesfully login and view the Patient records of fred@golf.com:


First we need to create a a schema using createDBUserMedrec.sql:
prompt Create user medrec with connect, resource roles;
grant connect, resource to medrec identified by  welcome1;
alter user medrec 
default tablespace users
temporary tablespace temp; 
alter user medrec quota unlimited on users;

Drop tables (if needed) using medrec_dropall.sql:
DROP TABLE "MEDREC"."ADMINISTRATORS";

DROP TABLE "MEDREC"."OPENJPA_SEQUENCE_TABLE"; 

DROP TABLE "MEDREC"."PATIENTS";

DROP TABLE "MEDREC"."PATIENTS_RECORDS";

DROP TABLE "MEDREC"."PHYSICIANS";

DROP TABLE "MEDREC"."PRESCRIPTIONS";

DROP TABLE "MEDREC"."DRUGS";

DROP TABLE "MEDREC"."RECORDS";

DROP TABLE "MEDREC"."RECORDS_PRESCRIPTIONS";

Create the tables using medrec_tables.sql:
CREATE TABLE "MEDREC"."ADMINISTRATORS" (
   "ID" INTEGER NOT NULL,
   "EMAIL" VARCHAR(255),
   "PASSWORD" VARCHAR(255),
   "USERNAME" VARCHAR(255),
   "VERSION" INTEGER,
   PRIMARY KEY ( "ID" ) );


CREATE TABLE "MEDREC"."OPENJPA_SEQUENCE_TABLE" (
   "ID" SMALLINT NOT NULL,
   "SEQUENCE_VALUE" INTEGER,
   PRIMARY KEY ( "ID" ) );


CREATE TABLE "MEDREC"."PATIENTS" (
   "ID" INTEGER NOT NULL,
   "EMAIL" VARCHAR(255),
   "PASSWORD" VARCHAR(255),
   "USERNAME" VARCHAR(255),
   "PHONE" VARCHAR(255),
   "DOB" TIMESTAMP,
   "GENDER" VARCHAR(20),
   "SSN" VARCHAR(255),
   "STATUS" VARCHAR(20),
   "VERSION" INTEGER,
   "FIRSTNAME" VARCHAR(255),
   "LASTNAME" VARCHAR(255),
   "MIDDLENAME" VARCHAR(255),
   "CITY" VARCHAR(255),
   "COUNTRY" VARCHAR(255),
   "STATE" VARCHAR(255),
   "STREET1" VARCHAR(255),
   "STREET2" VARCHAR(255),
   "ZIP" VARCHAR(255),
   PRIMARY KEY ( "ID" ) );


CREATE TABLE "MEDREC"."PATIENTS_RECORDS" (
   "PATIENT_ID" INTEGER,
   "RECORDS_ID" INTEGER );


CREATE TABLE "MEDREC"."PHYSICIANS" (
   "ID" INTEGER NOT NULL,
   "EMAIL" VARCHAR(255),
   "PASSWORD" VARCHAR(255),
   "USERNAME" VARCHAR(255),
   "PHONE" VARCHAR(255),
   "VERSION" INTEGER,
   "FIRSTNAME" VARCHAR(255),
   "LASTNAME" VARCHAR(255),
   "MIDDLENAME" VARCHAR(255),
   PRIMARY KEY ( "ID" ) );


CREATE TABLE "MEDREC"."DRUGS" 
   ( "ID" NUMBER(*,0) NOT NULL ENABLE, 
 "NAME" VARCHAR2(255 BYTE), 
 "FREQUENCY" VARCHAR2(255 BYTE), 
    "PRICE" NUMBER(10,2),
 "VERSION" NUMBER(*,0),
   PRIMARY KEY ( "ID" ) );


CREATE TABLE "MEDREC"."PRESCRIPTIONS" (
   "ID" INTEGER NOT NULL,
   "DATE_PRESCRIBED" TIMESTAMP,
   "FREQUENCY" VARCHAR(255),
   "INSTRUCTIONS" VARCHAR(255),
   "REFILLS_REMAINING" INTEGER,
   "VERSION" INTEGER,
   "DOSAGE" NUMBER, 
   "DRUG_ID" NUMBER, 
   PRIMARY KEY ( "ID" ) );

CREATE TABLE "MEDREC"."RECORDS" (
   "ID" INTEGER NOT NULL,
   "CREATE_DATE" TIMESTAMP,
   "RECORDDATE" TIMESTAMP,
   "DIAGNOSIS" VARCHAR(255),
   "NOTES" VARCHAR(255),
   "SYMPTOMS" VARCHAR(255),
   "VERSION" INTEGER,
   "PATIENT_ID" INTEGER NOT NULL,
   "PHYSICIAN_ID" INTEGER NOT NULL,
   "DIASTOLIC_BLOOD_PRESSURE" INTEGER,
   "HEIGHT" INTEGER,
   "PULSE" INTEGER,
   "SYSTOLIC_BLOOD_PRESSURE" INTEGER,
   "TEMPERATURE" INTEGER,
   "WEIGHT" INTEGER,
   PRIMARY KEY ( "ID" ) );

CREATE TABLE "MEDREC"."RECORDS_PRESCRIPTIONS" (
   "RECORD_ID" INTEGER,
   "PRESCRIPTIONS_ID" INTEGER );


Insert data using medrec_data.sql:
INSERT INTO "MEDREC"."ADMINISTRATORS" (
"ID", "EMAIL", "PASSWORD", "USERNAME", "VERSION" )
VALUES (
201,'admin@avitek.com','weblogic','admin@avitek.com',1
);

COMMIT;

INSERT INTO "MEDREC"."OPENJPA_SEQUENCE_TABLE" (
"ID", "SEQUENCE_VALUE" )
VALUES (
0,251
);

COMMIT;

INSERT INTO "MEDREC"."PATIENTS"
VALUES (
51,'page@fish.com','weblogic','page@fish.com','4151234564',
TIMESTAMP '1972-03-18 00:00:00','MALE','888888888','APPROVED',3,
'Page','Trout','A','Ponte Verde','United States','FL',
'235 Montgomery St','Suite 15','32301'
);
INSERT INTO "MEDREC"."PATIENTS"
VALUES (
52,'fred@golf.com','weblogic','fred@golf.com','4151234564',
TIMESTAMP '1965-04-26 00:00:00','MALE','123456789','APPROVED',3,
'Fred','Winner','I','San Francisco','United States','CA',
'1224 Post St','Suite 100','94115'
);
INSERT INTO "MEDREC"."PATIENTS"
VALUES (
53,'volley@ball.com','weblogic','volley@ball.com','4151234564',
TIMESTAMP '1971-09-17 00:00:00','MALE','333333333','APPROVED',3,
'Gabrielle','Spiker','H','San Francisco','United States','CA',
'1224 Post St','Suite 100','94115'
);
INSERT INTO "MEDREC"."PATIENTS"
VALUES (
54,'charlie@star.com','weblogic','charlie@star.com','4151234564',
TIMESTAMP '1973-11-29 00:00:00','MALE','444444444','REGISTERED',3,
'Charlie','Florida','E','Ponte Verde','United States','FL',
'235 Montgomery St','Suite 15','32301'
);
INSERT INTO "MEDREC"."PATIENTS"
VALUES (
55,'larry@bball.com','weblogic','larry@bball.com','4151234564',
TIMESTAMP '1959-03-13 00:00:00','MALE','777777777','APPROVED',3,
'Larry','Parrot','J','San Francisco','United States','CA',
'1224 Post St','Suite 100','94115'
);

COMMIT;

INSERT INTO "MEDREC"."PHYSICIANS" (
"ID", "EMAIL", "PASSWORD", "USERNAME", "PHONE", "VERSION", "FIRSTNAME", "LASTNAME", "MIDDLENAME" )
VALUES (
1,'mary@md.com','weblogic','mary@md.com','1234567812',4,'Mary','Oblige','J'
);

COMMIT;

Insert into "MEDREC"."DRUGS"  (ID,NAME,FREQUENCY,PRICE,VERSION) values (101,'Advil','1/4hrs',1.0, 2);
Insert into "MEDREC"."DRUGS"  (ID,NAME,FREQUENCY,PRICE,VERSION) values (102,'Codeine','1/6hrs',2.5,2);
Insert into "MEDREC"."DRUGS"  (ID,NAME,FREQUENCY,PRICE,VERSION) values (103,'Drixoral','1tspn/4hrs',3.75,2);

COMMIT;

Insert into "MEDREC"."PRESCRIPTIONS"  (ID,DATE_PRESCRIBED,FREQUENCY,INSTRUCTIONS,REFILLS_REMAINING,VERSION,DOSAGE,DRUG_ID) values (101,to_timestamp('18-JUL-99 12.00.00.000000000 AM','DD-MON-RR HH.MI.SSXFF AM'),'1/4hrs',null,0,2,1,101);
Insert into "MEDREC"."PRESCRIPTIONS"  (ID,DATE_PRESCRIBED,FREQUENCY,INSTRUCTIONS,REFILLS_REMAINING,VERSION,DOSAGE,DRUG_ID) values (102,to_timestamp('30-JUN-93 12.00.00.000000000 AM','DD-MON-RR HH.MI.SSXFF AM'),'1/6hrs',null,1,2,1,102);
Insert into "MEDREC"."PRESCRIPTIONS"  (ID,DATE_PRESCRIBED,FREQUENCY,INSTRUCTIONS,REFILLS_REMAINING,VERSION,DOSAGE,DRUG_ID) values (103,to_timestamp('18-JUL-99 12.00.00.000000000 AM','DD-MON-RR HH.MI.SSXFF AM'),'1tspn/4hrs',null,0,2,1,103);


COMMIT;

INSERT INTO "MEDREC"."RECORDS"
VALUES (
151,TIMESTAMP '1991-05-01 00:00:00',TIMESTAMP '1991-05-01 00:00:00','Allergic to coffee. Drink tea.',
'','Drowsy all day.',2,51,1,85,70,75,125,98,180
);
INSERT INTO "MEDREC"."RECORDS"
VALUES (
152,TIMESTAMP '1991-05-01 00:00:00',TIMESTAMP '1991-05-01 00:00:00','Light cast needed.',
'At least 20 sprained ankles since 15.','Sprained ankle.',
2,53,1,85,70,75,125,98,180
);
INSERT INTO "MEDREC"."RECORDS"
VALUES (
153,TIMESTAMP '1989-08-05 00:00:00',TIMESTAMP '1989-08-05 00:00:00','Severely sprained interior ligament. Surgery required.','Cast will be necessary before and after.','Twisted knee while playing soccer.',2,52,1,85,70,75,125,98,180
);
INSERT INTO "MEDREC"."RECORDS"
VALUES (
154,TIMESTAMP '1993-06-30 00:00:00',TIMESTAMP '1993-06-30 00:00:00','Common cold. Prescribed codiene cough syrup.','Call back if not better in 10 days.','Sneezing, coughing, stuffy head.',2,52,1,85,70,75,125,98,180
);
INSERT INTO "MEDREC"."RECORDS"
VALUES (
155,TIMESTAMP '1999-07-18 00:00:00',TIMESTAMP '1999-07-18 00:00:00','Mild stroke.  Aspirin advised.','Patient needs to stop smoking.','Complains about chest pain.',2,52,1,85,70,75,125,98,180
);
INSERT INTO "MEDREC"."RECORDS"
VALUES (
156,TIMESTAMP '1991-05-01 00:00:00',TIMESTAMP '1991-05-01 00:00:00','Patient is crazy.  Recommend politics.','','Overjoyed with everything.',2,55,1,85,70,75,125,98,180
);

COMMIT;

INSERT INTO "MEDREC"."RECORDS_PRESCRIPTIONS"
VALUES (
154,102
);
INSERT INTO "MEDREC"."RECORDS_PRESCRIPTIONS"
VALUES (
155,101
);
INSERT INTO "MEDREC"."RECORDS_PRESCRIPTIONS"
VALUES (
155,103
);

COMMIT;
To install the datasource you can use this wlst script, createDataSource.py:
#############################################################################
# Create DataSource for WLS 12c Tuning & Troubleshooting workshop
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.1, 2018-01-22
#
#############################################################################
# Modify these values as necessary
import os,sys, traceback
scriptName = sys.argv[0]
adminHost=os.environ["ADM_HOST"]
adminPort=os.environ["ADM_PORT"]
admServerUrl = 't3://'+adminHost+':'+adminPort
ttServerName=os.environ["TTSVR_NAME"]
adminUser='weblogic'
adminPwd='welcome1'
#
dsName = 'MedRecGlobalDataSourceXA'
dsJNDIName = 'jdbc/MedRecGlobalDataSourceXA'
initialCapacity = 5
maxCapacity = 10
capacityIncrement = 1
driverName = 'oracle.jdbc.xa.client.OracleXADataSource'
dbUrl = 'jdbc:oracle:thin:@darlin-vce.darwin-it.local:1521:orcl'
dbUser = 'medrec'
dbPassword = 'welcome1'
#
def createDataSource(dsName, dsJNDIName, initialCapacity, maxCapacity, capacityIncrement, dbUser, dbPassword, dbUrl, targetSvrName):
  # Check if data source already exists
  try:
    cd('/JDBCSystemResources/' + dsName)
    print 'The JDBC Data Source ' + dsName + ' already exists.'
    jdbcSystemResource=cmo
  except WLSTException:
    print 'Creating new JDBC Data Source named ' + dsName + '.'
    edit()
    startEdit()
    cd('/')
    # Create data source
    jdbcSystemResource = create(dsName, 'JDBCSystemResource')
    jdbcResource = jdbcSystemResource.getJDBCResource()
    jdbcResource.setName(dsName)
    # Set JNDI name
    jdbcResourceParameters = jdbcResource.getJDBCDataSourceParams()
    jdbcResourceParameters.setJNDINames([dsJNDIName])
    jdbcResourceParameters.setGlobalTransactionsProtocol('TwoPhaseCommit')
    # Create connection pool
    connectionPool = jdbcResource.getJDBCConnectionPoolParams()
    connectionPool.setInitialCapacity(initialCapacity)
    connectionPool.setMaxCapacity(maxCapacity)
    connectionPool.setCapacityIncrement(capacityIncrement)
    # Create driver settings
    driver = jdbcResource.getJDBCDriverParams()
    driver.setDriverName(driverName)
    driver.setUrl(dbUrl)
    driver.setPassword(dbPassword)
    driverProperties = driver.getProperties()
    userProperty = driverProperties.createProperty('user')
    userProperty.setValue(dbUser)
    # Set data source target
    targetServer = getMBean('/Servers/' + targetSvrName)
    jdbcSystemResource.addTarget(targetServer)
    # Activate changes
    save()
    activate(block='true')
    print 'Data Source created successfully.'
  return jdbcSystemResource

def main():
  # Connect to administration server
  try:
    connect(adminUser, adminPwd, admServerUrl)
    #
    createDataSource(dsName, dsJNDIName, initialCapacity, maxCapacity, capacityIncrement, dbUser, dbPassword, dbUrl,ttServerName)
    #    
    print("\nExiting...")
    exit()
  except:
    apply(traceback.print_exception, sys.exc_info())
    exit(exitcode=1)
#call main()
main()

Also Medrec needs an administrative user, createUser.py:
print 'starting the script ....'
#
adminHost=os.environ["ADM_HOST"]
adminPort=os.environ["ADM_PORT"]
admServerUrl = 't3://'+adminHost+':'+adminPort
#
adminUser='weblogic'
adminPwd='welcome1'
#
realmName = 'myrealm' 
#
def addUser(realm,username,password,description):
  print 'Prepare User',username,'...'
  if realm is not None:
    authenticator = realm.lookupAuthenticationProvider("DefaultAuthenticator")
    if authenticator.userExists(username)==1:
      print '[Warning]User',username,'has been existed.'
    else:
      authenticator.createUser(username,password,description)
      print '[INFO]User',username,'has been created successfully'


connect(adminUser,adminPwd,admServerUrl)

security=getMBean('/').getSecurityConfiguration()
realm=security.lookupRealm(realmName)
addUser(realm,'administrator','administrator123','MedRec Administrator')

disconnect()


Deploy the medrec application, deployMedRec.py:
#############################################################################
# Deploy MedRec for WLS 12c Tuning & Troubleshooting workshop
#
# @author Martien van den Akker, Darwin-IT Professionals
# @version 1.0, 2018-01-22
#
#############################################################################
# Modify these values as necessary
import os,sys, traceback
scriptName = sys.argv[0]
adminHost=os.environ["ADM_HOST"]
adminPort=os.environ["ADM_PORT"]
admServerUrl = 't3://'+adminHost+':'+adminPort
ttServerName=os.environ["TTSVR_NAME"]
adminUser='weblogic'
adminPwd='welcome1'
#
appName = 'medrec'
appSource = '../ear/medrec.ear'
#
# Deploy the application
def deployApplication(appName, appSource, targetServerName):
  print 'Deploying application ' + appName + '.'
  progress = deploy(appName=appName,path=appSource,targets=targetServerName)
  # Wait for deploy to complete
  while progress.isRunning():
   pass
  print 'Application ' + appName + ' deployed.'
#
#
def main():
  # Connect to administration server
  try:
    connect(adminUser, adminPwd, admServerUrl)
    #
    deployApplication(appName, appSource, ttServerName)
    #    
    print("\nExiting...")
    exit()
  except:
    apply(traceback.print_exception, sys.exc_info())
    exit(exitcode=1)
#call main()
main()


Monday, 11 June 2018

Add Weblogic12c Vagrant project

Last week I proudly presented a talk on  how to create, provision and maintain VMs with Vagrant, including installing Oracle software (Database, FusionMiddlware,etc.). It was during the nlOUG Tech Experience 18.  I've written about it in the last few posts.

Today I added my WLS12c Vagrant project to Github. Based on further insights I refactored my database12c installation a bit and pushed an updated Database12c Vagrant project to Github as well.

Main updates are that I disliked the 'Zipped' subfolder in the stage folder paths for the installations. And I wanted all the installations extracted into the same main folder. Then I can clean that up more easily.

You should check the references to the stage folder. Download the appropriate Oracle installer zip files and place them in the appropriate folder:



Coming up: a SOA 12c (including SOA, BPM and OSB) complete with creation of the domain (based on my scripts from 2016 and my TechExperience talk of last year), and configuration of nodemanager. Ready to start up after 'up'.

I also uploaded my slides of my talk to slideshare:
 

Friday, 4 May 2018

Enhance Vagrant provisioning: install java and database

In my previous blog posts (here and here), I wrote about how to create a base box and a create and start a virtual machine out of it. I started with provisioning, to have the vagrant user adapt the kernel settings, add a install user/owner and create a filesystem on an added disk.

Now let's make the provisioning a bit more interesting and install actual software in it.

Prepare new Vagrant project 

For this article I copied the project created from the previous blog. I called it ol75_db12c, since the goal is database 12c. But we'll also add java.
Now edit the Vagrantfile, since we want a new VM with another name:
So adapt the VM_NAME variable to something like "OL75U5_DB12c". You see how convenient it is to have those properties set as a global variable?

You could already try to do vagrant up to try it out. Remember, you can just do vagrant destroy to recreate it.

Also remove (or don't copy) the .vagrant subfolder, otherwise Vagrant would probably assume the box is already provisioned.  If that is the case, then either do a vagrant destroy to destroy the VM altogether, or vagrant provision to just re-provision the box.

Java

In the copied project, lets start with Java. There are several possibilities to install java, you could download the RPM from OTN. But one of the recommended practice I found in installing Java on a server is to put it in a path that hasn't got the java version (especially  the update) in it. When using it to install Weblogic, for instance, this path ends up in several places in scripts. Although it's a handfull, it's more than once.
Upgrading java is then just bringing down the servers/services using it, backup the version and unzip/untar the new version in the same folder. And there you have it: I like a java distribution that comes in an archive.

In Oracle Support you can find it by searching for the document All Java SE Downloads on MOS (Doc ID 1439822.1). There you find the current versions of the Java SE pacakges. For this article I used the public version JDK 8 Update 172, that you can download as patch 27412872:
 This contains the rpm as well as a .tar.gz package, that we'll use. Make sure that you download the x86_64 version:
And copy the download in the Stages folder in your vagrant main project folder (see previous blog).

To install it, I have the following script installJava.sh:
#!/bin/bash
#
#Download a zip with tar.gz containing complete JDK
#On MOS: Search for Doc ID 1439822.1
#Download latest 1.8 (public) patch, eg.:
#27412872  Oracle JDK 8 Update 172 (complete JDK, incl. jmc, jvisualvm)
#
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/fmw12c_env.sh
#
TMP_DIR=/tmp
STAGE_HOME=/media/sf_Stage
EXTRACT_HOME=$STAGE_HOME/Extracted
JAVA_ZIP_HOME=$STAGE_HOME/Java
JAVA_INSTALL_HOME=$EXTRACT_HOME/Java
JAVA_INSTALL_TMP=$EXTRACT_HOME/jdk
JAVA_INSTALL_ZIP=p27412872_180172_Linux-x86-64.zip
JAVA_INSTALL_TAR=jdk-8u172-linux-x64.tar.gz
JAVA_INSTALL_NAME=jdk1.8.0_172

#
echo "Checking Java Home: "$JAVA_HOME
if [ ! -f "$JAVA_HOME/bin/java" ]; then
 #
  #Unzip Java
if [ ! -f "$JAVA_INSTALL_HOME/$JAVA_INSTALL_TAR" ]; then
    if [ -f "$JAVA_ZIP_HOME/$JAVA_INSTALL_ZIP" ]; then
      echo Unzip $JAVA_ZIP_HOME/$JAVA_INSTALL_ZIP  to $JAVA_INSTALL_HOME/$JAVA_INSTALL_TAR
      mkdir -p $JAVA_INSTALL_HOME
      unzip -o $JAVA_ZIP_HOME/$JAVA_INSTALL_ZIP -d $JAVA_INSTALL_HOME
    else
      echo JAVA Zip File $JAVA_ZIP_HOME/$JAVA_INSTALL_ZIP does not exist!
    fi  
  else
    echo $JAVA_INSTALL_TAR already unzipped
  fi
  # Install jdk
  echo Install jdk 
  echo create folder $JAVA_INSTALL_TMP
  mkdir -p $JAVA_INSTALL_TMP
  echo create JAVA_HOME $JAVA_HOME
  mkdir -p $JAVA_HOME
  echo Untar $JAVA_ZIP_HOME/$JAVA_INSTALL_TAR to $JAVA_INSTALL_TMP
  tar -xf $JAVA_INSTALL_HOME/$JAVA_INSTALL_TAR -C $JAVA_INSTALL_TMP
  echo Move $JAVA_INSTALL_TMP/$JAVA_INSTALL_NAME/* to $JAVA_HOME
  mv  $JAVA_INSTALL_TMP/$JAVA_INSTALL_NAME/* $JAVA_HOME
  #cp -R $JAVA_INSTALL_RPM/* $JAVA_HOME
  #sudo rpm -ihv $JAVA_INSTALL_HOME/$JAVA_INSTALL_RPM  
else
  echo jdk 1.8 already installed
fi

That uses the fmw12c_env.sh script:
#!/bin/bash
echo set Fusion MiddleWare 12cR2 environment
export ORACLE_BASE=/app/oracle
export INVENTORY_DIRECTORY=/app/oraInventory
export JAVA_HOME=$ORACLE_BASE/product/jdk

The script checks java already exists. If not then it checks if the tar or zip file exist in /media/sf_Stage/Java. If the tar file does not exist it will unzip the zip file. If the tar file does exist, then it will create a temp folder to extract the tar file into. Then it will create the java-home folder and move the extracted jdk folder to it.

I put these files in the scripts/fmw folder of my project:

Call script from provisioning

Now we're at the point that took me a lot of time to figure out last winter. How do I call this scripts form the provisioning. Simple question, but let me try to explain the difficulty.
The provisioning is done using the vagrant user. The vagrant user is in the sudoers list, so it's able to run a script using the permissions of another user, usually the super user (root). But it is still the running vagrant user who owns the resulting files and folders. I could do something like sudo su - oracle -c "script"... However, it still results in all the files owned by vagrant. So, if I run the Java install script, the complete java tree is owned by vagrant. But I want oracle to own it. Now,  I could create another base box and replace vagrant by oracle as the install user. But that is not the idea.

It took me some time to finally find the great utility runuser. This allows me to run the script as another substitute user. The runuser utility must be run as root, but that's no problem since vagrant is in the sudoers list.

So add the following lines to your provisioning part of the Vagrantfile:
    echo _______________________________________________________________________________
    echo 3. Java SDK 8
    sudo runuser -l oracle -c '/vagrant/scripts/fmw/installJava.sh'

With the -l argument I denote the user and with -c the command to run.

First try

Having done that, you could do a first try of the provisioning by upping the box.
Open a command window and start it the first time with vagrant up. Then if all goes well the provisioning ends with:
    darwin: _______________________________________________________________________________
    darwin: 3. Java SDK 8
    darwin: set Fusion MiddleWare 12cR2 environment
    darwin: Checking Java Home: /app/oracle/product/jdk
    darwin: Unzip /media/sf_Stage/Java/p27412872_180172_Linux-x86-64.zip to /media/sf_Stage/Extracted/Java/jdk-8u172-linux-x64.tar.gz
    darwin: Archive:  /media/sf_Stage/Java/p27412872_180172_Linux-x86-64.zip
    darwin:   inflating: /media/sf_Stage/Extracted/Java/jdk-8u172-linux-x64.rpm
    darwin:   inflating: /media/sf_Stage/Extracted/Java/jdk-8u172-linux-x64.tar.gz
    darwin:   inflating: /media/sf_Stage/Extracted/Java/readme.txt
    darwin: Install jdk
    darwin: create folder /media/sf_Stage/Extracted/jdk
    darwin: create JAVA_HOME /app/oracle/product/jdk
    darwin: Untar /media/sf_Stage/Java/jdk-8u172-linux-x64.tar.gz to /media/sf_Stage/Extracted/jdk
    darwin: tar: jdk1.8.0_172/bin/ControlPanel: Cannot create symlink to `jcontrol': Protocol error
    darwin: tar: jdk1.8.0_172/man/ja: Cannot create symlink to `ja_JP.UTF-8': Protocol error
    darwin: tar: jdk1.8.0_172/jre/bin/ControlPanel: Cannot create symlink to `jcontrol': Protocol error
    darwin: tar: jdk1.8.0_172/jre/lib/amd64/server/libjsig.so: Cannot create symlink to `../libjsig.so': Protocol error
    darwin: tar: Exiting with failure status due to previous errors
    darwin: Move /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/bin /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/COPYRIGHT /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/db /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/include /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/javafx-src.zip /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/jre /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/lib /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/LICENSE /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/man /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/README.html /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/release /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/src.zip /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/THIRDPARTYLICENSEREADME-JAVAFX.txt /media/sf_Stage/Extracted/jdk/jdk1.8.0_172/THIRDPARTYLICENSEREADME.txt to /app/oracle/product/jdk

It it all went well you can just destroy it:
d:\Projects\vagrant\ol75_db12c>vagrant destroy
    darwin: Are you sure you want to destroy the 'darwin' VM? [y/N] y
==> darwin: Forcing shutdown of VM...
==> darwin: Destroying VM and associated drives...

d:\Projects\vagrant\ol75_db12c>

Install database

Installing the database is a bit more complex. I'll not post every file, the code can be found here in github, as in fact the rest of my project files.
It expects the database installation as two zip files, V46095-01_2of2.zip and V46095-01_1of2.zip in the folder /media/sf_Stage/DBInstallation/12.1.0.2/x86_64/Zipped. Like for the rest it works much like my earlier install Fusion Middleware scripts. It unzips the files, installs the database based on the template response files. And it runs the database creation utility.

I also added scripts to install/unzip sqldeveloper and sql commandline.

To install the database and possibly SQLDeveloper and/or SQLcl, just add the following lines to your vagrant file:
    echo _______________________________________________________________________________
    echo 4. Database 12c
    sudo runuser -l oracle -c '/vagrant/scripts/database/installDB.sh'
    echo _______________________________________________________________________________
    echo 5.1 SQLCL and SQLDeveloper
    sudo runuser -l oracle -c '/vagrant/scripts/database/installSqlcl.sh'
    echo _______________________________________________________________________________
    echo 5.2 SQLDeveloper
    sudo runuser -l oracle -c '/vagrant/scripts/database/installSqlDeveloper.sh'

And run vagrant up again.

Conclusion

I didn't get much into the install of the database. I feel that I wrote quite a lot in the past on installing Oracle software. In the upcoming period I'll wrap these lasts blogs into a presentation on creating boxes with Vagrant. This an input to my talk on this subject at the NLOUG Tech Experience 2018.

By now you should be able to replicate my findings and create other boxes as well. In the next blogs I might write about transforming my other FMW install scripts into vagrant project. We should now be able to create Vagrant projects to install OSB, SOA/BPM Suite, SOA/BPM Quickstart, etc.

Lately I did a Docker install in an Ubuntu 16 base box. Ubuntu has some peculiarities in creating users and logical volume management as opposed to RedHat/Oracle Linux. I might also write about that. But feel free to leave a comment if you have particular wishes. However, please have a bit of patience with me, since I first need to get my talk on the Tech Experience ready.


Tuesday, 1 May 2018

Installing haveged from Oracle repositories

In my previous blog, I mentioned the installation of the utility Haveged that is used to increase your entropy on non-gui systems. Since you can download it from the Oracle yum repository, I figured that I could install it from yum too, instead of using rpm. This prevents me from having to download it myself, and ensures I have the one that is applicable to my version of Linux.

Too bad it isn't in the default ol7_latest repo. So you have to either add the yum repo above or enable it.

To add it, you can do:
[vagrant@darlin-vce ~]$ sudo yum-config-manager --add-repo https://yum.oracle.com/repo/OracleLinux/OL7/developer_EPEL/x86_64

To remove it, you can remove the corresponding file from /etc/yum.repos.d:
[vagrant@darlin-vce ~]$ cd /etc/yum.repos.d/
[vagrant@darlin-vce yum.repos.d]$ ls
public-yum-ol7.repo
yum.oracle.com_repo_OracleLinux_OL7_developer_EPEL_x86_64.repo
[vagrant@darlin-vce yum.repos.d]$ sudo rm yum.oracle.com_repo_OracleLinux_OL7_developer_EPEL_x86_64.repo
[vagrant@darlin-vce yum.repos.d]$ ls
public-yum-ol7.repo

But it appears already registered in the repository list, but disabled:

[vagrant@darlin-vce yum.repos.d]$ sudo yum repolist all
Loaded plugins: langpacks, ulninfo
repo id                                     repo name                                                                                                        status
ol7_MODRHCK/x86_64                          Latest RHCK with fixes from Oracle for Oracle Linux 7Server (x86_64)                                             disabled
ol7_MySQL55/x86_64                          MySQL 5.5 for Oracle Linux 7 (x86_64)                                                                            disabled
ol7_MySQL56/x86_64                          MySQL 5.6 for Oracle Linux 7 (x86_64)                                                                            disabled
ol7_MySQL57/x86_64                          MySQL 5.7 for Oracle Linux 7 (x86_64)                                                                            disabled
ol7_UEKR3/x86_64                            Latest Unbreakable Enterprise Kernel Release 3 for Oracle Linux 7Server (x86_64)                                 disabled
ol7_UEKR3_OFED20/x86_64                     OFED supporting tool packages for Unbreakable Enterprise Kernel on Oracle Linux 7 (x86_64)                       disabled
ol7_UEKR4/x86_64                            Latest Unbreakable Enterprise Kernel Release 4 for Oracle Linux 7Server (x86_64)                                 enabled:    641
ol7_UEKR4_OFED/x86_64                       OFED supporting tool packages for Unbreakable Enterprise Kernel Release 4 on Oracle Linux 7 (x86_64)             disabled
ol7_addons/x86_64                           Oracle Linux 7Server Add ons (x86_64)                                                                            disabled
ol7_ceph/x86_64                             Ceph Storage for Oracle Linux Release 2.0 - Oracle Linux 7.2 or later (x86_64)                                   disabled
ol7_ceph10/x86_64                           Ceph Storage for Oracle Linux Release 1.0 - Oracle Linux 7.1 or later (x86_64)                                   disabled
ol7_developer/x86_64                        Oracle Linux 7Server Development Packages (x86_64)                                                               disabled
ol7_developer_EPEL/x86_64                   Oracle Linux 7Server Development Packages (x86_64)                                                               disabled
ol7_developer_gluster310/x86_64             Oracle Linux 7Server Gluster 3.10 Packages for Development and test (x86_64)                                     disabled
ol7_developer_gluster312/x86_64             Oracle Linux 7Server Gluster 3.12 Packages for Development and test (x86_64)                                     disabled
ol7_developer_nodejs4/x86_64                Oracle Linux 7Server Node.js 4 Packages for Development and test (x86_64)                                        disabled
ol7_developer_nodejs6/x86_64                Oracle Linux 7Server Node.js 6 Packages for Development and test (x86_64)                                        disabled
ol7_developer_nodejs8/x86_64                Oracle Linux 7Server Node.js 8 Packages for Development and test (x86_64)                                        disabled
ol7_developer_php70/x86_64                  Oracle Linux 7Server PHP 7.0 Packages for Development and test (x86_64)                                          disabled
ol7_developer_php71/x86_64                  Oracle Linux 7Server PHP 7.1 Packages for Development and test (x86_64)                                          disabled
ol7_developer_php72/x86_64                  Oracle Linux 7Server PHP 7.2 Packages for Development and test (x86_64)                                          disabled
ol7_latest/x86_64                           Oracle Linux 7Server Latest (x86_64)                                                                             enabled: 26,602
....

To enable it just:
sudo yum-config-manager --enable ol7_developer_EPEL

As a response it will display the current configuration.
Then install haveged by:
sudo yum -q -y install haveged

To check if haveged is installed, do a yum -list:
[vagrant@darlin-vce ~]$ sudo yum list haveged
Loaded plugins: langpacks, ulninfo
Installed Packages
haveged.x86_64

This is drawn by combining this RHEL Tech-Doc with the repo content of my freshly installed OL7U5 box.

And that solves the note I had left in my previous blog post.

Base box ready? Let's create a box!

Last week I wrote about creating our own Vagrant base box, based on the new and fresh Oracle Linux 7 Update 5. As a reaction on my article I got noted that Oracle also keeps base boxes for their latest linuxes at http://yum.oracle.com/boxes. They're also a great start.

To begin, I created a project structure for all my vagrant projects:
As you can see, I have a vagrant main projects folder on my D: drive, D:\Projects\vagrant. Besides my actual vagrant projects, like oel74, oel74_wls12c etc., I have a boxes folder and a Stage folder. The boxes folder, as you can guess, contains my base boxes:
The Stage folder contains all my installation-binaries, for database, Weblogic, Java, FusionMiddleware and so on.

Today I want to create a basic VM that will be a base for further VM's, like database, weblogic, etc.
I want the VM to have:
  • Linux prepared with correct kernel settings for database, FusionMiddleware, etc.
  • Filesystem created on a second disk. I did not add a second disk to the base box, only a root disk. Thus we need to extend the VM with one.
  • Create an oracle user that can sudo.

Initialize a vagrant project

 Begin with creating a folder, like ol75 in the structure, for this project. Open a command window and navigate to it:
Microsoft Windows [Version 10.0.16299.371]
(c) 2017 Microsoft Corporation. All rights reserved.

C:\Windows\system32>d:

D:\>cd d:\Projects\vagrant\ol75\

d:\Projects\vagrant\ol75>vagrant help init
==> vagrant: A new version of Vagrant is available: 2.0.4!
==> vagrant: To upgrade visit: https://www.vagrantup.com/downloads.html

Usage: vagrant init [options] [name [url]]

Options:

        --box-version VERSION        Version of the box to add
    -f, --force                      Overwrite existing Vagrantfile
    -m, --minimal                    Use minimal Vagrantfile template (no help comments). Ignored with --template
        --output FILE                Output path for the box. '-' for stdout
        --template FILE              Path to custom Vagrantfile template
    -h, --help                       Print this help

d:\Projects\vagrant\ol75>

The vagrant command init creates a new vagrant project, with a so called Vagrantfile in it. By default, you'll get a Vagrantfile with the most common settings and comments explaining  the most common additional settings. But using the -m or --minimal setting a just enough vagrant file is created, without comments. I do like a file with the most common settings in the comments, as it allows me to quickly extend it without having to lookup every thing. So I create a basic file:
d:\Projects\vagrant\ol75>vagrant init
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

d:\Projects\vagrant\ol75>dir /w
 Volume in drive D is DATA
 Volume Serial Number is 62D7-9456

 Directory of d:\Projects\vagrant\ol75

[.]           [..]          Vagrantfile
               1 File(s)          3,081 bytes
               2 Dir(s)  651,847,397,376 bytes free

Now, let's expand the file bit by bit. So, open it in your favorite ASCII editor, like Notepad++, for instance.
# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "base"

A vagrant file has about the following format:
Vagrant.configure("2") do |config|
  # ...
end

Apparently you can have multiple occurences with multiple configuration versions ("1" refers to Vagrant 1.0 configuration, "2" to 1.1 and onwards).

As with many other scripted languages, I find it convenient to have all the configurable values declared as global variables at the top of the file. So let's declare some:
# -*- mode: ruby -*-
# vi: set ft=ruby :
#
BOX_NAME="ol75"
BOX_URL="file://../boxes/OL75v1.0.box"
VM_MEMORY = 12288 # 12*1024 MB
VM_CPUS=4
VMS_HOME="d:\\VirtualMachines\\VirtualBox"
VM_NAME="OL7U5"    
VM_DISK2=VMS_HOME+"\\"+VM_NAME+"\\"+VM_NAME+".disk2.vdi"
VM_DISK2_SIZE=1024 * 512
# Stage folders
STAGE_HOST_FOLDER="d:/Projects/vagrant/Stage"
STAGE_GUEST_FOLDER="/media/sf_Stage"
Then find the line ‘Vagrant.configure("2") do |config|’. This starts the block that configures the box. All the following configuration is done between that line and the corresponnding closing end line.
 

Start with renaming the box based on the global variable:
  config.vm.box = BOX_NAME

And add the following lines:
  config.vm.box_url=BOX_URL
  config.vm.define "darwin"

The config.vm.box_url directive refers to the base box used. Often this is just the name of the box that is automatically downloaded from the Hashicorp/Vagrant repository. You can also use a URL to a box on internet, it will be downloaded automatically. But we created our own box, so that we exactly know and control what's in it. It does not need to be downloaded, it's available right away.

Side note: manage your boxes

At start up this box is added to your local Vagrant box repository. You can see the boxes in your repository with the command box list:
d:\Projects\vagrant\ol75>vagrant box list
OL7U4           (virtualbox, 0)
OL7U4-1.1b      (virtualbox, 0)
Ubuntu16.0.4LTS (virtualbox, 0)

d:\Projects\vagrant\ol75>

These boxes are found in the .vagrant.d/boxes folder your user profile:
My new ol75 box is not added yet, as you can see. It will be done at first up. Since they're eating up costly space on my SSD drive, it's sensible to remove unused boxes. For instance, the  OL7U4 one is superseded by the OL7U4-1.1b. The Ubuntu one I still use, although the Ubuntu version is a bit old. So, I should at least remove the OL7U4 one. It can be done with the command box remove ${name}:

d:\Projects\vagrant\ol75>vagrant box remove OL7U4
Removing box 'OL7U4' (v0) with provider 'virtualbox'...

d:\Projects\vagrant\ol75>vagrant box list
OL7U4-1.1b      (virtualbox, 0)
Ubuntu16.0.4LTS (virtualbox, 0)

d:\Projects\vagrant\ol75>

Support for multi-machine boxes

The other line I added was the config.vm.define directive. This one allows to define multi-machine definitions in one Vagrant project. It can come in handy when you have a project where one VM is servicing your database, while another is doing your front-end application. You can have them started and provisioned automatically, where the provisioning can be done in stages. Or you can switch off auto-start for certain machines and start those explicitly. It's a nice-to-know, but I'll leave it for now. I use it for noting the machine in the provisioning logs.
Read more about multi-machine configs here.

SSH Vagrant User

Below the config.vm.* lines, within the config block, you can add some config lines for the vagrant username/password:
  config.ssh.username="vagrant"
  config.ssh.password="vagrant"
  config.ssh.port=2222

The password line should be optional, since we injected a key for the vagrant user. You'll see that Vagrant will replace that key. The ssh.port will direct Vagrant to create a port-forwarding for the local port 2222 to the ssh port on the vm.

Provider config


Within the config block, below the line config.fm.define add the following block:
  config.vm.provider :virtualbox do |vb|
    vb.name = VM_NAME
    vb.gui = true
    vb.memory = VM_MEMORY
    vb.cpus = VM_CPUS
    # Set clipboard and drag&drop bidirectional
    vb.customize ["modifyvm", :id, "--clipboard", "bidirectional"]
    vb.customize ["modifyvm", :id, "--draganddrop", "bidirectional"]
    # Create a disk  
    unless File.exist?(VM_DISK2)
      vb.customize [ "createmedium", "disk", "--filename", VM_DISK2, "--format", "vdi", "--size", VM_DISK2_SIZE , "--variant", "Standard" ]
    end
    # Add it to the VM.
    vb.customize [ "storageattach", :id , "--storagectl", "SATA", "--port", "2", "--device", "0", "--type", "hdd", "--medium", VM_DISK2]
  end

This block begins setting the following properties:
Property
Meaning
vb.name Name of the VM to create. This is how the VM will appear in VirtualBox.
vb.gui This toggles the appearance of the UI of the VM. If false, it is started in the background. It is then only reachable through the network settings. Using the VirtualBox manager the GUI can then be brought to appear using the show button.
vb.memory This sets the memory available to the VM. In the global variables I have set it to 12GB (12*1024 MB)
vb.cpus Number of CPU cores availabe to the VM. In the globals I have set it to 4.

Using the customize provider command you can change the VM's configuration. It is in fact an API to the VboxManage utility of VirtualBox.

With modifyvm we set both the properties --clipboard and --draganddrop to bidirectional. When the GUI is shown (vb.gui = true) then it allows us to copy and paste into the VM for instance.

Then using  the createmedium command a standard vdi disk is created. The name VM_DISK2 is based on the variables VMS_HOME and VM_NAME. See the top of the file. It's convenient to have the file created in the same folder as the VM is created in. So check the VirtualBox preferences:

The Default Machine Folder preference is used to create the VM in, in a subfolder indicated by the name of the VM. So make sure that the VMS_HOME variable matches the value in that preference.

The value for --format the createmedium command I used is vdi. This is the default Virtual Disk Image format of VirtualBox. When exporting a VM into an OVA ( Open Virtual Appliance package, which is a tar archive file) the VDI disks are converted to the VMDK  (Virtual Machine Disk).
The size is set with the --size parameter, in my example set to the VM_DISK2_SIZE that I created in the top of the file as 512GB (1024 * 512).
The --variant Standard indicates a dynamically allocated file, that grows with the filling of it. So, you won't loose the complete filesize on diskspace. The 512GB limits the growth of the disk.

I want the disk to persist and not recreated at every startup, so I surrounded that command with the
unless File.exist?(VM_DISK2) block.

Using the storageattach I add it to my SATA controller using the --storagectl SATA parameters. It's added to --port 2 and --device 0, since it is my second drive. It needs to appear secondly in Linux. Then ofcourse the --type is hdd and the --medium is VM_DISK2.

You see a special variable :id. This refers to the VM that is created in VirtualBox. Of course I want the disk attached to the proper VM.

Shared/Synced folders

Vagrant by default creates a folder-link, a so-called Synced Folder, the wrapper around VirtualBox’s Shared Folder functionality. The default refers to the Vagrant project folder from which the VM is created and provisioned. Thus, in fact the folder where the Vagrantfile resides. That folder is mounted in the VM as /vagrant. So, navigating to the /vagrant folder in the VM will show you the files from the vagrant-project folder and child folders. This is convenient, because subfolders in that folder, for instance a scripts folder with provisioning scripts, are immediately available at startup.

Since we want to install software from the Stage folder on our host, we need a mapping to that.
So find the following line:
 # config.vm.synced_folder "../data", "/vagrant_data"


This is an example line for configuring additional synced folders. Add a line below it as follows:
  config.vm.synced_folder STAGE_HOST_FOLDER, STAGE_GUEST_FOLDER

This maps the folder on the host as denoted in the global variable STAGE_HOST_FOLDER, and then mount that as the value from the global STAGE_GUEST_FOLDER. Notice that I doing so I map the folder /media/sf_Stage in the VM to the folder d:/Projects/vagrant/Stage on the host. Wich is a sub-folder in my main vagrant project folder.

Provisioning

Having the VM configured, the provisioning part is to be configured. Vagrant allows for several provisioners like Puppet, Chef, Ansible, Salt, and Docker. But a Shell snippet is provided in our Vagrantfile. I'll expand that one. At the bottom of the file, right above the closing end of our configure block, we'll find the snippet:
# config.vm.provision "shell", inline: <<-SHELL
  #   apt-get update
  #   apt-get install -y apache2
  # SHELL
Replace it with the following block:
  config.vm.provision "shell", inline: <<-SHELL
    export SCRIPT_HOME=/vagrant/scripts
    echo _______________________________________________________________________________
    echo 0. Prepare Oracle Linux
    $SCRIPT_HOME/0.PrepOEL.sh
    echo _______________________________________________________________________________
    echo 1. Create Filesystem
    $SCRIPT_HOME/1.FileSystem.sh
    echo _______________________________________________________________________________
    echo 2. Create Oracle User
    $SCRIPT_HOME/2.MakeOracleUser.sh
    #
   SHELL
This provides an inline script. I like that because it allows me to see directly what happens in helicopter view. But, I do not like to have all the detailed steps in here, so I only call sub-scripts within this block. You can also uses external scripts, even remote ones, as described in the doc.

As can be seen the scripts I'll describe below have to be placed in the scripts folder as part of the vagrant project folder. Remember that this folder is mapped as a synched folder to /vagrant in the VM.

I  have the following script, called 0.PrepOEL.sh, to  update the Oracle Linux installation in the VM:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/install_env.sh
echo Installing packages required by the software
sudo yum -q -y install compat-libcap1* compat-libstdc* libstdc* gcc-c++* ksh libaio-devel* dos2unix system-storage-manager
echo install Haveged
sudo rpm -ihv $STAGE_HOME/Linux/haveged-1.9.1-1.el7.x86_64.rpm
echo 'Adding entries into /etc/security/limits.conf for oracle user'
if grep -Fq oracle /etc/security/limits.conf
then
    echo 'WARNING: Skipping, please verify!'
else
    echo 'Adding'
    sudo sh -c "sed -i '/End of file/i # Oracle account settings\noracle soft core unlimited\noracle hard core unlimited\noracle soft data unlimited\noracle hard data unlimited\noracle soft memlock 3500000\noracle hard memlock 3500000\noracle soft nofile 1048576\noracle hard nofile 1048576\noracle soft rss unlimited\noracle hard rss unlimited\noracle soft stack unlimited\noracle hard stack unlimited\noracle soft cpu unlimited\noracle hard cpu unlimited\noracle soft nproc unlimited\noracle hard nproc unlimited\n' /etc/security/limits.conf"
fi

echo 'Changing /etc/sysctl.conf'
if grep -Fq net.core.rmem_max /etc/sysctl.conf
then
    echo 'WARNING: Skipping, please verify!'
else
    echo 'Adding'
    sudo sh -c "echo '
#ORACLE
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 4294967295
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 4194304
'>>/etc/sysctl.conf"
/sbin/sysctl -p
fi
It uses the following script, install_env.sh:
#!/bin/bash
echo set Install environment
export STAGE_HOME=/media/sf_Stage
export SCRIPT_HOME=/vagrant/scripts
to set a few HOME variables.
It first install required packages using sudo yum . These packages are required for most of the Oracle software setup, like database and/or FusionMiddleware.
Then from the Linux folder in the STAGE_HOME folder, the tool Haveged is installed. You can download it from the Oracle yum repository. So, I should be able to have it installed with yum as well. One improvement point noted.

Why Haveged you ask? On non-gui, terminal only virtualized systems, the entropy maybe low, which causes slow encryption/decryptions. Installing and configuring FusionMiddleware, for instance, or starting WebLogic maybe very slow.

Then it set some limits for the oracle  user and kernel configs. These can be found in the install guides of the Oracle software.

To create the file system I use the script 1.FileSystem.sh:
#!/bin/bash
echo Create folder for mountpoint /app
sudo mkdir /app
echo Create a Logical Volume group and Volume on sdb
sudo ssm create -s 511GB -n disk01 --fstype xfs -p pool01 /dev/sdb /app
sudo ssm list
sudo sh -c "echo \"/dev/mapper/pool01-disk01       /app                    xfs     defaults        0 0\" >> /etc/fstab"


This one creates a folder  /appfor the file mount. Then it uses ssm (System Storage Manager) to create a 511GB (because of overhead I can't create a filesystem of 512GB) filesystem using a Logical Volume in a Logical Volume Group on the sdb device in Linux. For more info on how this works, read my blog article on it.

That leaves us to create an oracle user. For that I use the script 2.MakeOracleUser.sh:
#!/bin/bash
#
# Script to create a OS group and user
#
SCRIPTPATH=$(dirname $0)

ENV=${1:-dev}

function prop {
    grep "${1}" $SCRIPTPATH/makeOracleUser.properties|cut -d'=' -f2
}

# As we are using the database as well, we need a group named dba
echo Creating group dba
sudo /usr/sbin/groupadd -g 2001 dba

# We also need a group named oinstall as Oracle Inventory group
echo create group oinstall
sudo /usr/sbin/groupadd -g 2000 oinstall

#
# Create the Oracle user
echo Create the oracle user
sudo /usr/sbin/useradd -u 2000 -g oinstall -G dba oracle
echo Setting the oracle password to...
sudo sh -c "echo $(prop 'oracle.password') |passwd oracle --stdin"
sudo chown oracle:oinstall /app
#
# Add Oracle to sudoers so he can perform admin tasks
echo Adding oracle user to sudo-ers.
sudo sh -c "echo 'oracle           ALL=NOPASSWD:        ALL' >> /etc/sudoers"
#
# Create oraInst.loc and grant to Oracle
echo Create oraInventory folder
sudo chown -R oracle:oinstall /app
sudo mkdir -p /app/oracle/oraInventory
sudo chown -R oracle:oinstall /app/oracle
echo Create oraInst.loc and grant to Oracle
sudo sh -c "echo \"inventory_loc=/app/oracle/oraInventory\" > /etc/oraInst.loc"
#sudo sh -c "echo \"\" > /etc/oraInst.loc"
sudo sh -c "echo \"inst_group=oinstall\" >> /etc/oraInst.loc"
sudo chown oracle:oinstall /etc/oraInst.loc

It uses a property file makeOracleUser.properties for the oracle password:
oracle.password=welcome1

Using the prop function this property is read.

The groups dba and oinstall are created. Then the oracle user is created and it's password set. The filesystem mounted on /app is assigned to the oracle user to own. And then the user is added to tue /etc/sudoers file.

Lastly the oraInventory and the oraInst.loc file are created.

Up, up and up it goes…!

If everything went alright, you’re now ready to fire up your VM.
So open a command window and navigate to your vagrant project folder, if not done already.
Then simply issue the following command:
d:\Projects\vagrant\ol75>vagrant up

And then you wait…. And watch carefully to see that Vagrant imports the box, creates the VM, and provisions it.

Some other helpfull commands

I'll finish with some other helpfull commands:
Command
Meaning
vagrant up Start a VM, and provision it the first time.
vagrant halt Stop the VM.
vagrant suspend Remove the VM.
vagrant destroy Number of CPU cores availabe to the VM. In the globals I have set it to 4.
vagrant box list Lists the base boxes in your repository.
vagrant box remove Remove a listed base box from your repository.
vagrant package --base <VM Name> --output <box filename> Package the VM <VM Name> from the provider (VirtualBox) into a base box with given <box filename>.

Next stop: installing software as an oracle (or othernon-vagrant) user  user.

Tuesday, 24 April 2018

Oracle Linux 7 Update 5 is out: time to create a new Vagrant Base Box

It's been busy, so unfortunately it's already been almost two weeks I wrote my introductory story on Vagrant. Today I happen to have an afternoon off, and I noticed that Oracle Linux 7 Update 5 is out. I based my first boxes on 7.4, so nice moment to start with creating a new Base Box.

De essentials on creating a Vagrant base box can be read here. But I'm going to guide you trough the process step by step, so I hope you will be able to repeat this yourself, using this guide-through.

First of, Vagrant recommends Packer to automate the creation of base boxes. But I'm a bit confused, because in this guide it is apparently stated that this is deprecated by march 2018. I haven't tried Packer yet, but I feel that over the years I created a base VM only a few times. I used to create a base VM that I import/clone to create new VMs over and over again. And often, I start of with a VM that already contains a pre-installed database for instance.

Vagrant has a built in command to create a base box out of an existing VM. That is what I use.

Base box requirements

What is a Base Box actually? Well, it's in fact sort of a template that is used by Vagrant to create and configure a new VM and provision that. It should contain the following
  • An OS: I use Oracle Linux 7 Update 5 for this story. I also have a base box with Ubuntu. Ubuntu has some peculiarities I want to discuss later on in this series. For this base box I'll install a server-with-gui. But further as basic as possible.
  • A vagrant user. The vagrant user is used for provisioning the box. We'll place a public insecure key in it, that will be replaced by Vagrant at first startup. We'll add vagrant to the sudoers list, so the user can sudo without passwords.
  • A started ssh daemon:  Vagrant connects via ssh using the vagrant-user to do the provisioning.
  • A NAT (Network Address Translation) Adapter as the first one: needed to do kernel/package updates without further network configuration.
  • VirtualBox GuestAdditions installed: Vagrant makes use of shared folders to map the project folder to get to the scripts. Also it's convenient to add an extra stage folder mapping. 
  • Password of root: not a requirement, but apparently it's a bit of a standard to set the root password to vagrant as ease of sharing. But at least note down the passwords.
That's about it. Maybe I forget something, but since it's digital, I can edit it later... So let's get started.

Download  Oracle Linux

All the serious enterprise stuff of Oracle can be downloaded at edelivery. Search for Oracle Linux:
Then add the 7.5 version to the Cart by clicking it:

Follow the wizard instructions and you'll get to:
I downloaded V975367-01.iso        Oracle Linux Release 7 Update 5 for x86 (64 bit), 4.1 GB.

Create the VM

The ISO is downloading, so let's create a VM in VirtualBox. I assume VirtualBox with VirtualBox Extension Pack is installed. And for later on Vagrant of course.

From the Oracle VM VirtualBox Manager, create a new VM, I called it OL75, for Oracle Linux 64 bit:
I followed the wizard and gave it 10240 MB memory and a 128GB dynamically allocated virtual disk:

In the VM Settings, I set the number of processors to 4 and for now I kept everything to the default.

In the meantime my download is ready, so in the VM Settings, under Storage I added the disk by clicking the disk icon next to the IDE controller:

Then navigate to your downloaded iso:
and select it. Now the VM is ready to kick-off:


It will startup automatically after a minute, but let's not wait that long.

I don't need much, but in the Sofware Selection I do want Server with GUI:
But with out selecting other packages. What I might need later on, I'll install at provisioning.

I do not like default local domain networknames. So I changed the network hostname to darlin-vce.darwin-it.local:
Hostname darlin stands for Darwin Linux and vce for Virtual Course Environment.

Then hit Begin Installation:


Soon in the installation the installer asks for the Root password:
And the password is as said: vagrant.
Then I add also a vagrant with the same password:
Having done that, we need to wait for the installer to finish. At the end of the Install, do a reboot:

This leads to 2 questions to be answered. One is about accepting the licensing. I assume that can be answered without guidance. The other is about connecting the network.

You need to switch on the network adapter, but to have it done automatically you need to configure it and check the box Automatically connect to this network when it is available on the General tab. You'll need to have this done, otherwise Vagrant will have difficulties in connecting to the box.
Then finish the configuration:

Install guest additions

To be able to install the guest additions, we need to add some kernel packages. We could have done that by installing additional kernel packages. But I wanted to have a as basic as possible installation. And the following is more fun...

So open a terminal and switch to the super user:

[vagrant@darlin-vce ~]$ su -
Password: 
Last login: Tue Apr 24 09:41:21 EDT 2018 on pts/0
...


Then stop package kit, because it will probably hold a lock pausing yum:
[root@darlin-vce ~]# systemctl stop packagekit

And then install the packages kernel-uek-devel kernel-uek-devel-4.1.12-112.16.4.el7uek.x86_64, that are suggested by the GuestAdditions installer, by the way:
[root@darlin-vce ~]# yum -q -y install kernel-uek-devel kernel-uek-devel-4.1.12-112.16.4.el7uek.x86_64
No Presto metadata available for ol7_UEKR4
warning: /var/cache/yum/x86_64/7Server/ol7_latest/packages/cpp-4.8.5-28.0.1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Public key for cpp-4.8.5-28.0.1.el7.x86_64.rpm is not installed
Public key for kernel-uek-devel-4.1.12-124.14.1.el7uek.x86_64.rpm is not installed
Importing GPG key 0xEC551F03:
 Userid     : "Oracle OSS group (Open Source Software group) "
 Fingerprint: 4214 4123 fecf c55b 9086 313d 72f9 7b74 ec55 1f03
 Package    : 7:oraclelinux-release-7.5-1.0.3.el7.x86_64 (@anaconda/7.5)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle

Having done that, insert the GuestAdditions CD:
It brings the following pop-up, click Run:

And provide the Administrator password:

In my case the script ran and during that the display got messed up. But after a reset of the VM (I waited until I got the impression it was done), the VM got up with a Hi-res display, indicating that the install went ok. Also the bi-directional clipboard worked.

Configure vagrant user

Again in a terminal switch to super user and add the following line to the /etc/sudoers file:
vagrant ALL=(ALL) NOPASSWD: ALL

Exit and as vagrant user create a .ssh folder in the vagrant home folder, cd to it and create the file authorized_keys:
[vagrant@darlin-vce ~]$ mkdir .ssh
[vagrant@darlin-vce ~]$ cd .ssh
[vagrant@darlin-vce .ssh]$ vi authorized_keys

Insert the following content:
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key

This is the insecure key of vagrant that can be downloaded here.
It will be replaced by Vagrant at first startup.

Package the box

So, now we have a base install that can function as a base box for Vagrant. Thus we can now shut it down to export it to an OVA (just as a backup for VirtualBox) and then create are base box out of it.

After creating your export of the OVA, that I skip describing here, you just open a command window. I assume you have Vagrant installed.

To package the box, you use the package subcommand of vagrant:

Microsoft Windows [Version 10.0.16299.371]
(c) 2017 Microsoft Corporation. All rights reserved.

d:\Projects\vagrant>vagrant package --base OL75 --output d:\Projects\vagrant\boxes\OL75v1.0.box
==> OL75: Exporting VM...
==> OL75: Compressing package to: d:/Projects/vagrant/boxes/OL75v1.0.box

d:\Projects\vagrant>

Conclusion

Well, that concludes this part of the series. We have our own base box and it's barely 3GB. Next: create a VM with it. Stay tuned.




Thursday, 19 April 2018

Garbage First in JDeveloper

At my current customer we work with VDI's: Virtual Desktop Images, that at several times a day very, very slow. Even so slow that it more or less stalls for a minute or two.

JDeveloper is not known as a Ferrari under the IDE's. One of the causes is that by default heap settings is very poor: 128M-800M. Especially when you use it in  SOA or BPM Quickstart then at startup it will need to grow several times. But very soon working in it you'll get out of memory errors.

Because of the VDI's I did several changes to try to improve performance.
Main thing is set Xms and Xmx both at 2048M. I haven't found needing more up to this day.

But I found using the Garbage First collector gives me a slightly better performance.

To set it, together with the heap, add/change the following options in the ide.conf in ${JDEV_HOME}\jdeveloper\ide\bin\:
# Set the default memory options for the Java VM which apply to both 32 and 64-bit VM's.
# These values can be overridden in the user .conf file, see the comment at the top of this file.
#AddVMOption  -Xms128M
#AddVMOption  -Xmx800M
AddVMOption  -Xms2048M
AddVMOption  -Xmx2048M
AddVMOption  -XX:+UseG1GC 
AddVMOption  -XX:MaxGCPauseMillis=200

Find more on the command line options in this G1GC tutorial.

You can also use the ParNew incombination with the ParOld or ConcMarkSeep collector, as suggested in this blog. But from Java9 onwards G1GC is the default, and I expect that it better fits the behavior of JDeveloper, as in SOASuite and OSB installations.