Friday 21 February 2020

My Weblogic on Kubernetes Cheatsheet, part 3.

Oracle Kubernetes

In two previous parts I already wrote about my Kubernetes experiences and the important commands I learned:
My way of learning and working is to put those commands in little scriptlets, one more usefull then the other. But all with the goal to keep those together.

It its time to write part 3, in which I will present some maintenance functions, mainly to connect with your pods.

Get node and pod info

getdmnpod-status.sh

In part 2 I ended with the script getdmnpods.sh. You can parse the output using awk to get just the status of the pods:

#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Get K8s pod statuses for $WLS_DMN_NS
kubectl -n $WLS_DMN_NS get pods -o wide| awk '{print $1 " - "  $3}'

getpods.sh

With getdmnpods.sh you can get the status of the pods running your domain. There's also a weblogic operator pod. To show this, use:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Get K8s pods for $K8S_NS
kubectl get po -n $K8S_NS

getstmpods.sh

Then also the kubernetes cluster infrastructure consist of a set of pods. Show these using:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Get K8s pods for kube-system
kubectl -n kube-system get pods


getnodes.sh


On OCI your cluster is running on a set of nodes. These OCI Instances are actually running your system. You can show those, with their IP's and Kubernetes versions using:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Get K8s nodes
kubectl get node

getdmnsitlogs.sh


Of course you want to see some logs, especially when something went wrong. Perhaps you want to see some specific loggings. For instance, this script show the logs of the admin pod, grepping the logs situational related to the situational config:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Get situational config logs for $WLS_DMN_NS server $ADM_POD
kubectl -n $WLS_DMN_NS logs $ADM_POD | grep -i situational

Weblogic Operator

When I was busy with getting the MedRec Sample application deployed to Kubernetes, at one point I got stuck because, as I later learned, my Kubernetes Operator's version was behind.

list_wlop.sh 

I learned I could get Weblogic Operator information as follows:

#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo List Weblogic Operator $WL_OPERATOR_NAME
cd $HELM_CHARTS_HOME
helm list $WL_OPERATOR_NAME
cd $SCRIPTPATH

delete_weblogic_operator.sh 

When you find that the operator needs an update, you can remove it with this script:

#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Delete Weblogic Operator $WL_OPERATOR_NAME
cd $HELM_CHARTS_HOME
helm del --purge $WL_OPERATOR_NAME 
cd $SCRIPTPATH

install_weblogic_operator.sh


Then of course, you want to install it with the proper function. This can be done using:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Install Weblogic Operator $WL_OPERATOR_NAME
cd $HELM_CHARTS_HOME
helm install kubernetes/charts/weblogic-operator \
  --name $WL_OPERATOR_NAME \
  --namespace $K8S_NS \
  --set image=oracle/weblogic-kubernetes-operator:2.3.0 \
  --set serviceAccount=$K8S_SA \
  --set "domainNamespaces={}"
cd $SCRIPTPATH

Take note of the image named in this script. Make sure that it matches the image with the latest-greatest operator version. In this script I apparently still use 2.3.0, but as of November 15th, 2019 2.4.0 is released.

upgrade_weblogic_operator.sh

Besides an install and delete chart, there is also an operator upgrade Helm chart:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Upgrade Weblogic Operator $WL_OPERATOR_NAME with domainNamespace $WLS_DMN_NS
cd $HELM_CHARTS_HOME
helm upgrade \
  --reuse-values \
  --set "domainNamespaces={$WLS_DMN_NS}" \
  --wait \
  $WL_OPERATOR_NAME \
  kubernetes/charts/weblogic-operator
cd $SCRIPTPATH

Connect to the pods

The containers in the pods are running Linux (I know this is a quite blunt statement). So you might want to be able to connect to them. In case of Weblogic, you might want to be able to run wlst.sh to navigate to the MBean tree to investigate certain settings and find out why certain settings won't work in runtime.

admbash.sh and mr1bash.sh

To get to the console of the container you can run for the AdminServer the script admbash.sh:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh

echo Start bash in $WLS_DMN_NS - $ADM_POD
kubectl exec -n $WLS_DMN_NS -it $ADM_POD /bin/bash

And for one of the managed servers a variant of mr1bash.sh:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Get K8s pods for $WLS_DMN_NS
kubectl -n $WLS_DMN_NS get pods -o wide
kubectl exec -n medrec-domain-ns -it medrec-domain-medrec-server1 /bin/bash

On the commandline you can then run wlst.sh and connect to your AdminServer.

dwnldAdmLogs.sh and dwnldMr1Logs.sh


The previous scripts can help to navigate through your container and find the contents. However, you'll find that the containers lack certain basic bash commands like vi. The cat command does exist, but not very convenient investigating large log files. So, very soon I found the desire to download the log files to investigate them with a proper editor. You can do it for the admin server using dwnldAdmLogs.sh:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
#
LOG_FILE=$ADM_SVR.log
OUT_FILE=$ADM_SVR.out
#
echo From $WLS_DMN_NS/$ADM_POD download $DMN_HOME/servers/$ADM_SVR/logs/$LOG_FILE to $LCL_LOGS_HOME/$LOG_FILE
kubectl cp $WLS_DMN_NS/$ADM_POD:$DMN_HOME/servers/$ADM_SVR/logs/$LOG_FILE $LCL_LOGS_HOME/$LOG_FILE
echo From $WLS_DMN_NS/$ADM_POD download $DMN_HOME/servers/$ADM_SVR/logs/$OUT_FILE to $LCL_LOGS_HOME/$OUT_FILE
kubectl cp $WLS_DMN_NS/$ADM_POD:$DMN_HOME/servers/$ADM_SVR/logs/$OUT_FILE $LCL_LOGS_HOME/$OUT_FILE

And for one of the managed servers a variant of dwnldMr1Logs.sh:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
#
LOG_FILE=$MR_SVR1.log
OUT_FILE=$MR_SVR1.out
#
echo From $WLS_DMN_NS/$MR1_POD download $DMN_HOME/servers/$MR_SVR1/logs/$LOG_FILE to $LCL_LOGS_HOME/$LOG_FILE
kubectl cp $WLS_DMN_NS/$MR1_POD:$DMN_HOME/servers/$MR_SVR1/logs/$LOG_FILE $LCL_LOGS_HOME/$LOG_FILE
echo From $WLS_DMN_NS/$MR1_POD download $DMN_HOME/servers/$MR_SVR1/logs/$OUT_FILE to $LCL_LOGS_HOME/$OUT_FILE
kubectl cp $WLS_DMN_NS/$MR1_POD:$DMN_HOME/servers/$MR_SVR1/logs/$OUT_FILE $LCL_LOGS_HOME/$OUT_FILE

I found these scripts very handy, because I can quickly repeatedly download the particular log files.

Describe kube resources


Many resources in Kubernetes can be described. In my case I found it very usefull when debugging the configuration overrides.

descjdbccm.sh


One subject in the Weblogic Operator tutorial workshop is to do configuration overrides, and one of the steps is to create a configuration map. This is one example of a resource that can be desribed:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Describe jdbc configuration map of $WLS_DMN_NS
kubectl describe cm jdbccm -n $WLS_DMN_NS

Usefull to see what the latest overrides values are.

override_weblogic_domain.sh

To perform the weblogic override I use the following script:

#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Delete configuration map jdbccm for Domain $WLS_DMN_UID 
kubectl -n $WLS_DMN_NS delete cm jdbccm
#echo Override Weblogic Domain $WLS_DMN_UID using $SCRIPTPATH/medrec-domain/override
kubectl -n $WLS_DMN_NS create cm jdbccm --from-file $SCRIPTPATH/medrec-domain/override
kubectl -n $WLS_DMN_NS label cm jdbccm weblogic.domainUID=$WLS_DMN_UID

Obviously the descjdbccm.sh is very usefull in combination with this script.

descmrsecr.sh


Another part in the configuration overrides is the storage of the database credentials and connection URL. We store those in a secret that is referenced in the overide files. This is smart, because you now only need to create or update the secret and then run the configuration override script. To describe the secret you can use:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Describe secret $MR_DB_CRED of namespace $WLS_DMN_NS
kubectl describe secret $MR_DB_CRED -n $WLS_DMN_NS

Since it is a secret, you can show the names of the attributes in the secret, but not their values.

create_mrdbsecret.sh


You need to create or update secrets. Apparently you need to delete it first to be able to (re)create it. This script does it for two secrets, for two datasources:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
#
function prop {
    grep "${1}" $SCRIPTPATH/credentials.properties|cut -d'=' -f2
}
#
MR_DB_USER=$(prop 'db.medrec.username')
MR_DB_PWD=$(prop 'db.medrec.password')
MR_DB_URL=$(prop 'db.medrec.url')
#
echo Delete Medrec DB Secret $MR_DB_CRED for $WLS_DMN_NS
kubectl -n $WLS_DMN_NS delete secret $MR_DB_CRED
echo Create Medrec DB Secret $MR_DB_CRED for $MR_DB_USER and URL $MR_DB_URL
kubectl -n $WLS_DMN_NS create secret generic $MR_DB_CRED --from-literal=username=$MR_DB_USER --from-literal=password=$MR_DB_PWD --from-literal=url=$MR_DB_URL
kubectl -n $WLS_DMN_NS label secret $MR_DB_CRED weblogic.domainUID=$WLS_DMN_UID
#
SMPL_DB_CRED=dbsecret
echo Delete Medrec DB Secret $SMPL_DB_CRED for $WLS_DMN_NS
kubectl -n $WLS_DMN_NS delete secret $SMPL_DB_CRED
echo Create DB Secret dbsecret $SMPL_DB_CRED for  $WLS_DMN_NS
kubectl -n $WLS_DMN_NS create secret generic $SMPL_DB_CRED --from-literal=username=scott2 --from-literal=url=jdbc:oracle:thin:@test.db.example.com:1521/ORCLCDB
kubectl -n $WLS_DMN_NS label secret $SMPL_DB_CRED weblogic.domainUID=$WLS_DMN_UID

This script gets the MedRec database credentials from a property file. Obviously you need to store those values in a save place. So you might figure that having them in a property file might not be a very safe way. You could change the script of course to ask for the particular password. And you might want to adapt it to be able to load different property files per target environment.

Can I?

The Kubernetes API has of course an authorization schema. One of the first things in the Weblogic Operator tutorial is that when you create your OKE Cluster you should make sure that you have the authorization to access your Kubernetes cluster using a system admin account.

To check if you're able to call the proper API's for your setup you can use the following scripts:

canideploy.sh

#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo K8s Can I deploy?
kubectl auth can-i create deploy

canideployassystem.sh


#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo K8s Can I deploy as system?
kubectl auth can-i create deploy --as system:serviceaccount:kube-system:default

Conclusion

At this point I showed you my scriptlets up to now. There is a lot to investigate still. For instance, there are Terraform examples to create your OKE cluster from scratch with Terraform. This is very promising as an alternative to the on-line wizards. Also I would like to create some (micro-)services to get data from the MedRec database and run them in pods side by side with the MedRec application. Maybe even with a OJet front end.

No comments :