In two previous parts I already wrote about my Kubernetes experiences and the important commands I learned:
My way of learning and working is to put those commands in little scriptlets, one more usefull then the other. But all with the goal to keep those together.
It its time to write part 3, in which I will present some maintenance functions, mainly to connect with your pods.
Get node and pod info
getdmnpod-status.sh
In part 2 I ended with the script getdmnpods.sh. You can parse the output using awk to get just the status of the pods:#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Get K8s pod statuses for $WLS_DMN_NS
kubectl -n $WLS_DMN_NS get pods -o wide| awk '{print $1 " - " $3}'
getpods.sh
With getdmnpods.sh you can get the status of the pods running your domain. There's also a weblogic operator pod. To show this, use:#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Get K8s pods for $K8S_NS kubectl get po -n $K8S_NS
getstmpods.sh
Then also the kubernetes cluster infrastructure consist of a set of pods. Show these using:#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Get K8s pods for kube-system kubectl -n kube-system get pods
getnodes.sh
On OCI your cluster is running on a set of nodes. These OCI Instances are actually running your system. You can show those, with their IP's and Kubernetes versions using:
#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Get K8s nodes kubectl get node
getdmnsitlogs.sh
Of course you want to see some logs, especially when something went wrong. Perhaps you want to see some specific loggings. For instance, this script show the logs of the admin pod, grepping the logs situational related to the situational config:
#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Get situational config logs for $WLS_DMN_NS server $ADM_POD kubectl -n $WLS_DMN_NS logs $ADM_POD | grep -i situational
Weblogic Operator
When I was busy with getting the MedRec Sample application deployed to Kubernetes, at one point I got stuck because, as I later learned, my Kubernetes Operator's version was behind.list_wlop.sh
I learned I could get Weblogic Operator information as follows:#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo List Weblogic Operator $WL_OPERATOR_NAME cd $HELM_CHARTS_HOME helm list $WL_OPERATOR_NAME cd $SCRIPTPATH
delete_weblogic_operator.sh
When you find that the operator needs an update, you can remove it with this script:#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Delete Weblogic Operator $WL_OPERATOR_NAME cd $HELM_CHARTS_HOME helm del --purge $WL_OPERATOR_NAME cd $SCRIPTPATH
install_weblogic_operator.sh
Then of course, you want to install it with the proper function. This can be done using:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Install Weblogic Operator $WL_OPERATOR_NAME
cd $HELM_CHARTS_HOME
helm install kubernetes/charts/weblogic-operator \
--name $WL_OPERATOR_NAME \
--namespace $K8S_NS \
--set image=oracle/weblogic-kubernetes-operator:2.3.0 \
--set serviceAccount=$K8S_SA \
--set "domainNamespaces={}"
cd $SCRIPTPATH
Take note of the image named in this script. Make sure that it matches the image with the latest-greatest operator version. In this script I apparently still use 2.3.0, but as of November 15th, 2019 2.4.0 is released.
upgrade_weblogic_operator.sh
Besides an install and delete chart, there is also an operator upgrade Helm chart:#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
echo Upgrade Weblogic Operator $WL_OPERATOR_NAME with domainNamespace $WLS_DMN_NS
cd $HELM_CHARTS_HOME
helm upgrade \
--reuse-values \
--set "domainNamespaces={$WLS_DMN_NS}" \
--wait \
$WL_OPERATOR_NAME \
kubernetes/charts/weblogic-operator
cd $SCRIPTPATH
Connect to the pods
The containers in the pods are running Linux (I know this is a quite blunt statement). So you might want to be able to connect to them. In case of Weblogic, you might want to be able to run wlst.sh to navigate to the MBean tree to investigate certain settings and find out why certain settings won't work in runtime.admbash.sh and mr1bash.sh
To get to the console of the container you can run for the AdminServer the script admbash.sh:#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Start bash in $WLS_DMN_NS - $ADM_POD kubectl exec -n $WLS_DMN_NS -it $ADM_POD /bin/bash
And for one of the managed servers a variant of mr1bash.sh:
#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Get K8s pods for $WLS_DMN_NS kubectl -n $WLS_DMN_NS get pods -o wide kubectl exec -n medrec-domain-ns -it medrec-domain-medrec-server1 /bin/bash
On the commandline you can then run wlst.sh and connect to your AdminServer.
dwnldAdmLogs.sh and dwnldMr1Logs.sh
The previous scripts can help to navigate through your container and find the contents. However, you'll find that the containers lack certain basic bash commands like vi. The cat command does exist, but not very convenient investigating large log files. So, very soon I found the desire to download the log files to investigate them with a proper editor. You can do it for the admin server using dwnldAdmLogs.sh:
#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh # LOG_FILE=$ADM_SVR.log OUT_FILE=$ADM_SVR.out # echo From $WLS_DMN_NS/$ADM_POD download $DMN_HOME/servers/$ADM_SVR/logs/$LOG_FILE to $LCL_LOGS_HOME/$LOG_FILE kubectl cp $WLS_DMN_NS/$ADM_POD:$DMN_HOME/servers/$ADM_SVR/logs/$LOG_FILE $LCL_LOGS_HOME/$LOG_FILE echo From $WLS_DMN_NS/$ADM_POD download $DMN_HOME/servers/$ADM_SVR/logs/$OUT_FILE to $LCL_LOGS_HOME/$OUT_FILE kubectl cp $WLS_DMN_NS/$ADM_POD:$DMN_HOME/servers/$ADM_SVR/logs/$OUT_FILE $LCL_LOGS_HOME/$OUT_FILE
And for one of the managed servers a variant of dwnldMr1Logs.sh:
#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh # LOG_FILE=$MR_SVR1.log OUT_FILE=$MR_SVR1.out # echo From $WLS_DMN_NS/$MR1_POD download $DMN_HOME/servers/$MR_SVR1/logs/$LOG_FILE to $LCL_LOGS_HOME/$LOG_FILE kubectl cp $WLS_DMN_NS/$MR1_POD:$DMN_HOME/servers/$MR_SVR1/logs/$LOG_FILE $LCL_LOGS_HOME/$LOG_FILE echo From $WLS_DMN_NS/$MR1_POD download $DMN_HOME/servers/$MR_SVR1/logs/$OUT_FILE to $LCL_LOGS_HOME/$OUT_FILE kubectl cp $WLS_DMN_NS/$MR1_POD:$DMN_HOME/servers/$MR_SVR1/logs/$OUT_FILE $LCL_LOGS_HOME/$OUT_FILE
I found these scripts very handy, because I can quickly repeatedly download the particular log files.
Describe kube resources
Many resources in Kubernetes can be described. In my case I found it very usefull when debugging the configuration overrides.
descjdbccm.sh
One subject in the Weblogic Operator tutorial workshop is to do configuration overrides, and one of the steps is to create a configuration map. This is one example of a resource that can be desribed:
#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Describe jdbc configuration map of $WLS_DMN_NS kubectl describe cm jdbccm -n $WLS_DMN_NS
Usefull to see what the latest overrides values are.
override_weblogic_domain.sh
To perform the weblogic override I use the following script:#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Delete configuration map jdbccm for Domain $WLS_DMN_UID kubectl -n $WLS_DMN_NS delete cm jdbccm #echo Override Weblogic Domain $WLS_DMN_UID using $SCRIPTPATH/medrec-domain/override kubectl -n $WLS_DMN_NS create cm jdbccm --from-file $SCRIPTPATH/medrec-domain/override kubectl -n $WLS_DMN_NS label cm jdbccm weblogic.domainUID=$WLS_DMN_UID
Obviously the descjdbccm.sh is very usefull in combination with this script.
descmrsecr.sh
Another part in the configuration overrides is the storage of the database credentials and connection URL. We store those in a secret that is referenced in the overide files. This is smart, because you now only need to create or update the secret and then run the configuration override script. To describe the secret you can use:
#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo Describe secret $MR_DB_CRED of namespace $WLS_DMN_NS kubectl describe secret $MR_DB_CRED -n $WLS_DMN_NS
Since it is a secret, you can show the names of the attributes in the secret, but not their values.
create_mrdbsecret.sh
You need to create or update secrets. Apparently you need to delete it first to be able to (re)create it. This script does it for two secrets, for two datasources:
#!/bin/bash
SCRIPTPATH=$(dirname $0)
#
. $SCRIPTPATH/oke_env.sh
#
function prop {
grep "${1}" $SCRIPTPATH/credentials.properties|cut -d'=' -f2
}
#
MR_DB_USER=$(prop 'db.medrec.username')
MR_DB_PWD=$(prop 'db.medrec.password')
MR_DB_URL=$(prop 'db.medrec.url')
#
echo Delete Medrec DB Secret $MR_DB_CRED for $WLS_DMN_NS
kubectl -n $WLS_DMN_NS delete secret $MR_DB_CRED
echo Create Medrec DB Secret $MR_DB_CRED for $MR_DB_USER and URL $MR_DB_URL
kubectl -n $WLS_DMN_NS create secret generic $MR_DB_CRED --from-literal=username=$MR_DB_USER --from-literal=password=$MR_DB_PWD --from-literal=url=$MR_DB_URL
kubectl -n $WLS_DMN_NS label secret $MR_DB_CRED weblogic.domainUID=$WLS_DMN_UID
#
SMPL_DB_CRED=dbsecret
echo Delete Medrec DB Secret $SMPL_DB_CRED for $WLS_DMN_NS
kubectl -n $WLS_DMN_NS delete secret $SMPL_DB_CRED
echo Create DB Secret dbsecret $SMPL_DB_CRED for $WLS_DMN_NS
kubectl -n $WLS_DMN_NS create secret generic $SMPL_DB_CRED --from-literal=username=scott2 --from-literal=url=jdbc:oracle:thin:@test.db.example.com:1521/ORCLCDB
kubectl -n $WLS_DMN_NS label secret $SMPL_DB_CRED weblogic.domainUID=$WLS_DMN_UID
This script gets the MedRec database credentials from a property file. Obviously you need to store those values in a save place. So you might figure that having them in a property file might not be a very safe way. You could change the script of course to ask for the particular password. And you might want to adapt it to be able to load different property files per target environment.
Can I?
The Kubernetes API has of course an authorization schema. One of the first things in the Weblogic Operator tutorial is that when you create your OKE Cluster you should make sure that you have the authorization to access your Kubernetes cluster using a system admin account.To check if you're able to call the proper API's for your setup you can use the following scripts:
canideploy.sh
#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo K8s Can I deploy? kubectl auth can-i create deploy
canideployassystem.sh
#!/bin/bash SCRIPTPATH=$(dirname $0) # . $SCRIPTPATH/oke_env.sh echo K8s Can I deploy as system? kubectl auth can-i create deploy --as system:serviceaccount:kube-system:default
