shell script to collect todays cache details of weblogic jvm and send email

 

The following shell script collects today's cache details of weblogic jvm and sends out an email.
The script can ssh to many linux servers and only collect cache details were cache has been created on the present day.

The servers should already have keyless SSH enabled between them

Below is the script

 #!/bin/sh
echo -e "\e[0;36m++++++ \e[0m"
echo -e "\e[0;36m Author : robin.richard@****.com \e[0m"
sleep 5
echo -e "\e[0;36m++++++ \e[0m"
printf "\n ======================= \n Begining cache collection in  PROD  \n ======================= \n"
sleep 4
rm -r report.txt
printf "\n ======================= \n purging old data \n ======================= \n"
echo -e "\e[0;36m++++++ \e[0m"
sleep 5
#rm -r report.txt
for i in oracle@server oracle@server oracle@server oracle@server oracle@server; do ssh $i 'echo  "Server: $(hostname) - Date: $(date)" && find /<path>/servers/*/tmp/_WL_user/ -daystart -ctime 0 -print' >> report.txt; done
sleep 4
printf "\n ======================= \n Completed Cache collection \n ======================= \n"
echo “ prod  cache details” | mail -s “Prod_cache_details” robin.richard@****.com,rramaneedi@****.com,hsinghchouhan@****.com < report.txt
printf "\n ======================= \n Sending Email \n ======================= \n"
echo -e "\e[36;41m++++++ \e[0m"
printf "\n ======================= \n Exiting the system \n ======================= \n"
echo -e "\e[36;41m++++++ \e[0m"

You can also put the script in bash profile as follows
vi ~oracle/.bashrc
alias cachecollector='/home/oracle/cache_collector/robin.sh &'

 

also you can try

for i in oracle@server oracle@server oracle@server oracle@server oracle@server; do ssh $i 'echo  "Server: $(hostname) - Date: $(date)" && echo -e "\e[0;36m++++++ \e[0m" && find /<path>/servers/*/tmp/_WL_user/ -daystart -ctime 0 -print | xargs ls -ltr && echo -e "\e[0;36m++++++ \e[0m"' >> report.txt; done 

or 

for i in oracle@server oracle@server oracle@server oracle@server oracle@server ; do ssh $i 'echo  "Server: $(hostname) - Date: $(date)" && find /<path>/servers/*/tmp/_WL_user/ -daystart -ctime 0 -print' >> report.txt; done

or 

for i in oracle@server oracle@server oracle@server oracle@server oracle@server ; do ssh $i 'echo "********" && echo  "Server: $(hostname) - Date: $(date)" && find /<path>servers/*/tmp/_WL_user/ -daystart -ctime 0 -print && echo "********"' >> report.txt; done

Print index.html from todays date

find /<path>/*/index.html -daystart -ctime 0 -print

Print folders only of todays creation date

find /<path>/*/<path>s/*/tmp/_WL_user/ -mindepth 1 -maxdepth 1 -ctime 0 -print

java.lang.ClassNotFoundException: javafx.application.Application

 PROBLEM : While installing vertex receiving javafx issues

ERROR: 

Exception in thread "main" java.lang.NoClassDefFoundError: javafx/application/Application
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at com.vertexinc.tps.install.app.InstallApp.main(InstallApp.java:22)
Caused by: java.lang.ClassNotFoundException: javafx.application.Application
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)

REASON: javafx has been removed from java11, however it is still available in java8

so use JDK 8 version

solr failed to start

 

Error : OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory. (error = 1)

Context: this is a lucene solr 9.0.0 + zk 3.8.0. on a linux server

Solution:  

increment the heap memory at SOLR_HEAP="24g" to accommodate increased collection load.

The file /*/solr/solr-9.0.0/bin/solr.in.sh

 

 

Extract Private Key from oracle wallet

 To Extract Private Key from a wallet we would need orapki keytool and openssl.

<<<<<<<<<< Display the contents of the wallet using orapki >>>>>>>>>

/<fmw_home>/oracle_common/bin/orapki wallet display -wallet /<domain_home>config/fmwconfig/components/OHS/instances/ohs1/keystores/wallet/cwallet.sso

<<<<<<<<<<<< copy the wallet to new location say /tmp/robin/wallet >>>>>>>>>>>

convert from pkcs12 to jks and provide password

/<fmw_home>/oracle_common/bin/orapki wallet pkcs12_to_jks -wallet /tmp/robin/wallet -jksKeyStoreLoc /tmp/robin/ewallet.jks 

welcome1

 

<<<<<<<<<<<< <<<<< import keystore >>>>>>>>>>>>>>>>>>>>>>>>

keytool -importkeystore -srckeystore /tmp/robin/ewallet.jks -srcstoretype JKS -deststoretype PKCS12 -destkeystore /tmp/robin/result/ewallet.p12

<<<<<<<<<<<<<<<<<< private key will be in r.txt >>>>>>>>>>>>>>>>>>>>>>>>>>>>

openssl pkcs12 -in /tmp/robin/result/ewallet.p12 -passin pass:welcome1 -out /tmp/robin/result/r.txt -nodes

OHS not starting due to cert issue

 ERROR MSG:

<date> <Info> <Security> <BEA-090909> <Using the configured custom SSL Hostname Verifier implementation: weblogic.security.utils.SSLWLSHostnameVerifier$NullHostnameVerifier.>
This Exception occurred at <date>.
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
Error: Error occurred while performing nmConnect : Cannot connect to Node Manager. : sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
Use dumpStack() to view the full stacktrace :

FACTS:

1. Nodemanager is running in SSL mode

2.OHS is configured to run on SSL mode

3. OHS uses wallet with default self signed cert

Solution:

<<<<<<<<TO DISPLAY the contents of the WALLET>

oracle_home/oracle_common/bin/orapki wallet display -wallet domain_home/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1/keystores/default/cwallet.sso

<<<<<<<<< export cert to a file >>>>>>>>>>
oracle_home/oracle_common/bin/orapki wallet export -wallet . -dn "CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY" -cert domain_home/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1/keystores/cert.txt

<<<<< read the cert in a file to check expiry of cert >>>>>>>>>

cd domain_home/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1/keystore
/oracle_home/oracle_common/bin/orapki cert display -cert cert.txt -complete

<<<<<<<< take a backup of .sso and create a new wallet >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
oracle_home/oracle_common/bin/orapki wallet create -wallet domain_home/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1/keystores/default/wallet -auto_login_only

<<<<<<<<<<<<<<<<<<<<< add new cert to the wallet (self signed cert) >>>>>>>>>>>>>>
oracle_home/oracle_common/bin/orapki wallet add -wallet domain_home/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1/keystores/default -dn "CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY" -keysize 2048 -self_signed -validity 3650 -auto_login_only

start the ohs

bash script to check URL status from a linux box

 

Wrote a script to check valid urls from linux box, To be executed as below from a linux box

We can keep adding url’s to url.txt. Result will be stored in validURLlist.txt

./url_check.sh < url.txt

 

 #

rm -r validURLlist.txt
while read i
do
  wget --spider $i > /dev/null 2>1
  if [ $? == 0 ]
  then
    echo "em_result:success: "$i  >> validURLlist.txt

  else
 echo "em_result:failue: "$i  >> validURLlist.txt
  fi
done 

Servlet failed with an IOException. java.io.IOException: HTTPS hostname wrong: should be

Error msg : 

Servlet failed with an IOException.java.io.IOException: HTTPS hostname wrong:  should be XXXXX

Context:

app is exclusively talking  on TLS1.2 and higher 

Solution:

Enable following params in weblogic startup:
-Dweblogic.ssl.SSLv2HelloEnabled=false -DUseSunHttpHandler=true -Dhttps.protocols=TLSv1.2

upgrade jdk to jdk 1.8_311 or higher.

restart domain.

access denied on webmethods IS page console

 problem: 

Access Denied

Services Necessary to show the Integration Server Administrator are currently unavailable on this port.

This is most likely due to port security restrictions.

If this is the only port available to access the Integration Server, contact webMethods Support.

 

Solution:

cd /***/IntegrationServer/instances/default/packages/WmRoot/config/ 

check listeners.cnf for exclude name="default"

make it include 

example : <value name="default">include</value>

also check file /******/IntegrationServer/instances/default/config/is_jaas.cnf

for corruption

restart IS.

 

READ_TIMEOUT in webserver

 Setup

Browser - webserver (ohs) - appserver (weblogic)

Situation

Getting below exception in ohs logs:

Exception type [READ_TIMEOUT] (no read after 120 seconds)

Solution

weblogic console - domain - config - JTA

changed Timeout Seconds from 30 to 3600 and restarted.

weblogic managed server not coming up after weblogic password change

 Situation:

1. Console password was changed on my realms and domain- security - general

2.boot .properties was updated.

The jvm's were stopped and admin started

admin came up but managed server did not come up. (no nodemanager involved)

 

Error: logs indicate boot.properties related issue

Solution : 

Copy security folder from admin (boot.properties are already encrypted)

to managed server 

remove the ldap folder of managed server / data.

start...

wlst script to stop and start and check status of managed server weblogic jvm

cd /opt/robin/wls/oracle/middleware/user_projects/domains/robin/bin

$ more stop_m1.sh

/opt/robin/wls/oracle/middleware/wlserver/common/bin/wlst.sh stop_m1.py > /opt/robin/wls/oracle/middleware/user_projects/domains/robin/servers/m1/logs/robin.log

$ more stop_m1.py

# wlst script to stop managed servers`
import os
import sys

connect('weblogic','Welcome1','t3://server:7010')
print'connecting'


exitonerror=false

# Stop all managed Servers
shutdown('m1', 'Server', force="true")
print'm1 shutdown'

exit()

----------------------------------------------------------

 $ more start_m1.py

# wlst script to start managed servers`
import os
import sys

connect('weblogic','Welcome1','t3://server:7010')
print'connecting'


exitonerror=false

# Start all managed Servers
start(name="m1", block="true")
print'm1 start'

exit()

$ more start_m1.sh

/opt/robin/wls/oracle/middleware/wlserver/common/bin/wlst.sh start_m1.py > /opt/robin/wls/oracle/middleware/user_projects/domains/robin/servers/m1/logs/robin.log

$ more checkstatus.py

# wlst script to check status of jvm

import os
import sys


connect('weblogic','Welcome1','t3://server:7010')
print'connecting'


exitonerror=false

servers=cmo.getServers()
print "-------------------------------------------------------"
print "\t"+cmo.getName()+" domain status"
print "-------------------------------------------------------"
for server in servers:
        state(server.getName(),server.getType())
print "-------------------------------------------------------"

exit()

$ . ./setDomainEnv.sh
$ java weblogic.WLST checkstatus.py

Symantec NetBackup PureDisk change URL / port / ip address

How to get the puredisk url working on another address.

Login to the server using a putty session and enter the below command

ps -ef | grep java

The result shows us that, it is a tomcat server hosting the webpage.

HOSTNAME:~ # ps -ef | grep java
root     25012 24974  0 10:56 pts/0    00:00:00 grep java
root     29395     1  0 May29 ?        00:00:19 /opt/VRTSjre/jre1.5/bin/java -Dvrtsat.libpath=/opt/VRTSat/lib/x64 -Duuid.libpath=/etc/vx/uuid/bin -Dvrtsweb.home=/opt/VRTSweb -Dvrtsweb.runtime.dir=/var/VRTSweb -Dvrtsweb.log.dir=/var/VRTSweb/log -Dvcs.home=/opt/VRTSvcs -Dvcs.conf=/etc/VRTSvcs -Xmx1024m -Dvrtsweb.max.heap.size=1024 -server -Xrs -Djava.awt.headless=true -Dsun.java2d.noddraw=true -Xms64m -Xss512k vrts.tomcat.bootstrap.Main start

The configuration must have been under /opt/VRTSvcs.
Go to the config folder /opt/VRTSweb/conf

open the file vrtsweb.xml.

# more vrtsweb.xml

<Ports securePortRedirection="false">
                <Port port="8080" protocol="http" address="127.0.0.1" />
                <Port port="8009" protocol="ajp"  address="127.0.0.1" />
        </Ports>

We see that the page can be accessed on ip 127.0.0.1 & ports 8080 & 8009
However 127.0.0.1 is internal to the server.

So edit the file and change it like below:

# vi vrtsweb.xml

<Ports securePortRedirection="false">
                <Port port="8080" protocol="http"                                  />
                <Port port="8009" protocol="ajp"  address="127.0.0.1" />
        </Ports>

Change the listen address from 127.0.0.1 to all ( earlier the port 8080 was listening to only 127.0.0.1, now it listens on all ip’s)
You can also change the port here if you want.

save and exit the file and restart the tomcat. (service puredisk restart)

check the ip's the host is configured to using command <ip addr>

The result will show you all the ip's the host is listening on under <inet>

simply access the url on any of the <inet> ip address and port 8080,
for example ip:8080/PureDisk/index.do

If there is an address translation in the server, simply configure a tunnel at your putty, to access it as localhost.

High CPU utilization due to java process on windows

Issue : java is using up most of the CPU in a windows server.

Artifacts needed : powershell

Solution : Note down the java pid causing high cpu using task manager
check high CPU using taskbar using process tab and details tab.
check pid causing high CPU

If jvm is started using service, please stop the service.

Open the cmd in admin mode and right click ta the top bar of cmd and go to properties
Under "layout" tab, of "screen buffer size", increase the Height to 5000 and click OK.
exit the cmd and open again in admin mode.

Start the jvm from cmd line in admin mode and do not close the cmd window.

wait for the CPU to peak again and take down new pid from task manager.

Now on the cmd where the jvm is running, press ctrl break atleast 3 times within an interval of 5secs each, which will generate 3 threaddump

right click on cmd top bar and edit - select all, then press ENter,

Now copy all the contents to a notepad and save, this is your threaddump.

Open powershell in admin mode and enter the below command:

(get-process -id XXXX).threads | sort-object {$_.TotalProcessorTime} > c:\1.txt

XXXX represents the high cpu java pid.

in the output 1.txt, the (bottom) last stack, the Id is the thread id and TotalProcessorTime is the most used cpu time of the pid in decimal format taking maximum processor time

Below is an example:

BasePriority            : 8
CurrentPriority         : 10
Id                      : 7860
IdealProcessor          :
PriorityBoostEnabled    : True
PriorityLevel           : Normal
PrivilegedProcessorTime : 00:20:56.5781250
StartAddress            : 140726260225200
StartTime               : 5/19/2017 9:14:18 AM
ThreadState             : Running
TotalProcessorTime      : 00:49:12.1718750
UserProcessorTime       : 00:28:15.5937500
WaitReason              :
ProcessorAffinity       :
Site                    :
Container               :

TotalProcesorTime is the sum of UserProcessorTime and PrivilagedProcessorTime.
Usertime is related to application requests and Privileged is related to kernel ops.

So 3 iterations over an period time showing the same Thread ID with highest TotalProcessorTime is the real culprit.


convert the last Id from decimal to hexadecimal add 0X at the front (if needed)
search this hex value in threaddump generated earlier.
The found stack trace is the culprit.

Testing Infinispan application on Windows with Jboss EAP 7

The following lab is setup using jboss-eap-7.0.0 running on oracle hotspot JDK 1.8 which runs on windows 7 64bit. The lab showcases storing and retrieving values in a distributed cache environment.
The infinispan application is a basic example that shows how to store and retrieve data to/from the cache. Users can access the cache either from a servlet or from a JSF page through request scoped beans.
Infinispan is configured in clustered distributed mode with synchronous replication. Entries have their lifespan (expiration) and are removed from the cache after 60 seconds since last update.

Artifacts needed

OS: windows 7 enterprise version 64 bit
JAVA: any JDK above 1.7, I have used oracle hotspot 1.8 64 bit. (jdk-8u121-windows-x64)
Development tool: apache-maven-3.3.9-bin
Application server: jboss-eap-7.0.0

Install JAVA

JAVA is the first requirement of any application server installation. Use following command to check if you have java installed already on your system.
# java -version
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
Else go ahead and install java jdk-8u121-windows-x64.exe
Then set the classpath and java_home as shown below:
From the desktop, right click the Computer icon.
Choose Properties from the context menu.
Click the Advanced system settings link.
Click Environment Variables. In the section System Variables, find the CLASSPATH environment variable and select it. Click Edit. If the CLASSPATH environment variable does not exist, click New.
Add the path after   ;
Variable name CLASSPATH
Variable value   ;C:\Program Files\Java\jdk1.8.0_101\bin
In the Edit System Variable (or New System Variable) window, specify the value of the JAVA_HOME environment variable. Click OK. Close all remaining windows by clicking OK
Variable name JAVA_HOME
Variable Value C:\Program Files\Java\jdk1.8.0_101\jre

Quickstarts

Download the quickstart from the jboss website check reference for link.
Unzip the archive

Maven

Download maven from the maven homepage and unzip the archive
apache-maven-3.3.9-bin.zip
Unzip the package and run the .bat file
C:\infinispan\apache-maven-3.3.9-bin\apache-maven-3.3.9\bin\mvn.cmd
Check it by running the command mvn -v

Take the settings.xml from the quickstart and place it under the maven installation directory.
In my example settings.xml is under C:\infinispan\jboss-jdg-quickstarts-jdg-7.0.x\jboss-jdg-quickstarts-jdg-7.0.x
Which I will copy it to
C:\Users\richarob\.m2

Add maven path under system variable, as done in 2.2
Variable path PATH
Variable value C:\infinispan\apache-maven-3.3.9-bin\apache-maven-3.3.9\bin
If you have any proxy settings on the browser, you need to implement it on the maven as well.
Go to tools menu of browser and copy the proxy settings if any.
Now paste this address in a new browser and open the result in a notepad, which will give you the host and port through which you gain access to internet from within your company.
Edit the settings.xml under C:\Users\richarob\.m2\settings.xml to add the host and port under http and https protocol of proxies tag.

<proxies>
    <proxy>
      <id>optional</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>web-proxy.in.corp.net</host>
      <port>8080</port>
    </proxy>
  <proxy>
      <id>optional_https</id>
      <active>true</active>
      <protocol>https</protocol>
      <host>web-proxy.in.hpecorp.net</host>
      <port>8080</port>
    </proxy>
   </proxies>

Install JBOSS EAP

Download JBOSS EAP archive file by visiting the official site provided in the references section.
Unzip jboss-eap-7.0.0.zip in the desired directory.


Jboss EAP setup

Start the first instance
C:\infinispan\jboss-eap-7.0.0\jboss-eap-7.0\bin>standalone.bat
Access the console http://localhost:8080/
You need to add user to the console
C:\infinispan\jboss-eap-7.0.0\jboss-eap-7.0\bin>add-user.bat

Complete creation of management user and refresh console and login, which redirects you to http://localhost:9990/console/App.html#home

Start the second instance
Make a second copy of EAP folder as 2jboss-eap-7.0.0 and start
C:\infinispan\2jboss-eap-7.0.0\jboss-eap-7.0\bin>standalone.bat -Djboss.socket.binding.port-offset=100
Access the console at http://localhost:10090/console/App.html#home 
The port offset works like (100+9990=10090)

Now Go ahead and deploy the first package as below:
C:\infinispan\jboss-jdg-quickstarts-jdg-7.0.x\jboss-jdg-quickstarts-jdg-7.0.x\helloworld-jdg>mvn clean package jboss:deploy
This will create a war file under
C:\infinispan\jboss-jdg-quickstarts-jdg-7.0.x\jboss-jdg-quickstarts-jdg-7.0.x\helloworld-jdg\target\ jboss-helloworld-jdg.war

Test infinispan application

Login to both of the consoles and deploy the war file

After deployment you will be able to access the application under below URL's

http://localhost:8080/jboss-helloworld-jdg/put.jsf  

http://localhost:8180/jboss-helloworld-jdg/put.jsf 

Access first server at http://localhost:8080/jboss-helloworld-jdg and insert key "Robin" with value "Richard"
Access second server at http://localhost:8180/jboss-helloworld-jdg and do the following
Click on "Get Some"
Get the value for key "robin"
Click "Put Some More"
Insert key "super" with value "cool"
Access the first server at http://localhost:8080/jboss-helloworld-jdg and do the following
Click on "Get Some"
Get all mappings by clicking on "Get All"
All data entered on each server was replicated to the other server
Entries expire and simply disappear after 60 seconds from last update.
 

References 

http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html
https://github.com/jboss-developer/jboss-jdg-quickstarts
http://maven.apache.org/download.cgi
http://maven.apache.org/install.html
https://github.com/jboss-developer/jboss-developer-shared- resources/blob/master/guides/CONFIGURE_MAVEN.md#configure-maven---for-use-with-command-line
https://github.com/jboss-developer/jboss-jdg-quickstarts
https://developers.redhat.com/products/eap/download/
https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.1/html/Installation_Guide/Install_JBoss_Enterprise_Application_Platform_6_Zip_Installation.html
https://access.redhat.com/documentation/en-US/JBoss_Operations_Network/3.1/html/How_to_Manage_JBoss_Servers/eap6-management-users.html
http://blog.infinispan.org/
http://infinispan.org/docs/stable/user_guide/user_guide.html