Apply weblogic Patch to unix

January 2011 weblogic patches are available at
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=REFERENCE&id=1263333.1

( Note: If you're wondering about  what "No Error Correction (NEC)" is: it means if you find an issue on WLS-10.3.1,there will be no fix provided, the fix will be provided only for WLS-10.3.2 onwards)

Copy all patches along with the patch-catalog.xml file to the Unix box at
*/bea1033/utils/bsu/cache_dir

now if the patch comes with an xml which is like  patch-catalog_xxxx.xml (ex : patch-catalog_4444.xml) then you would have to rename this  patch-catalog_xxxx.xml to  patch-catalog.xml in cache_dir
make sure that if a file patch-catalog.xml is already present in cache_dir before you dropping the patch there
you would have to back it up first and then rename patch-catalog-xxxx.xml to patch-catalog.xml

On the Unix machine go to the bsu folder.
( Ex : */bea1033/utils/bsu/cache_dir ) and run the following command :

Syntax :
$ ./bsu.sh -prod_dir= -patchlist= -verbose -install -log=/tmp/error.log

ex:
$ ./bsu.sh -prod_dir=*/bea1033/wlserver_10.3 -patchlist=B1WK -verbose -install -log=/tmp/error.log

If you want to install multiple patches at the time use the below command :

$ ./bsu.sh -prod_dir=*/bea1033/wlserver_10.3 -patchlist=B1WK,P2MT -verbose -install -log=/tmp/error.log


You can later use the below command to check the output

*\bea1033\utils\bsu>bsu -report

restart domain

To remove a patch

On the Solaris machine go to the bsu folder. ( Ex : */bea1033/utils/bsu ) and run the following command :

bsu -remove -patchlist=4MHU -prod_dir=D:\bea\weblogic92 –verbose

If you get the error:
"Patch id **** is not associated with the given product dir."
Then the patch does not belong to the WLS version you are in, most likely bug for the patch has already been fixed in the current version.

If you get the below error:
com.bea.plateng.common.util.logging.LogWriter -  java.io.IOException: Unable to backup file D:\work\wls\patch_wls1033\patch_jars\BUG*.jar to D:\work\wls\patch_wls1033\patch_jars\BUG*.jar53888.bak
com.bea.plateng.patch.CommandLinePatchInstaller - An unexpected error was encountered removing patch DFYK.  That resource may be busy.
The patch was not removed, and has been restored to its original state.  For further information, enable logging and run patch removal again.
com.bea.plateng.patch.PatchRemovalException: java.io.IOException: Unable to backup file D:\work\wls\patch_wls1033\patch_jars\BUG*.jar to D:\work\wls\patch_wls1033\patch_jars\BUG*.jar53888.bak


The OS has the JAR locked,  So please kill all OS related weblogic processes like nodemanager and beasvc service.

 If you get the below error:

    2014-06-25 02:48:59,328 DEBUG [Main Thread] com.bea.plateng.common.util.logging.LogWriter - Restoring /data/XXX//wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar from /XXXX/patch_wls1036/backup/backup.jar
    2014-06-25 02:48:59,332 ERROR [Main Thread] com.bea.plateng.patch.FilePatchRemover - Error restoring files from backup archive.
    2014-06-25 02:48:59,333 ERROR [Main Thread] com.bea.plateng.patch.PatchRemover - Error encountered removing patch.
    2014-06-25 02:48:59,335 DEBUG [Main Thread] com.bea.plateng.patch.CommandLineProgress - Command line progress stopped
    2014-06-25 02:48:59,335 DEBUG [Main Thread] com.bea.plateng.common.util.logging.LogWriter - Result: Failure
    2014-06-25 02:48:59,335 DEBUG [Main Thread] com.bea.plateng.common.util.logging.LogWriter - Failure condition follows:

The patch was not removed, and has been restored to its original state.  For further information, enable logging and run patch removal again.
com.bea.plateng.patch.PatchRemovalException: java.io.FileNotFoundException: /XXX/wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar (No such file or directory)

 while patching being rolled back, its looking for jar file in wrong location.

bash-3.2$ grep -i /XXXX/wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar patch-backup.xml
   <target>/XXXX/wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar</target>
   <target>/XXXX/wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar</target>

 

 

 

If you get the below error:

 

Exception in thread "Thread-0" java.lang.NullPointerException
        at com.bea.plateng.patch.PatchTargetHelper.loadPatchProfiles(PatchTargetHelper.java:447)
        at com.bea.plateng.patch.PatchTarget.<init>(PatchTarget.java:268)
        at com.bea.plateng.patch.PatchTargetFactory.create(PatchTargetFactory.java:30)
        at com.bea.plateng.patch.ProductAliasTarget.constructPatchTargetList(ProductAliasTarget.java:88)
        at com.bea.plateng.patch.ProductAliasTarget.<init>(ProductAliasTarget.java:46)
        at com.bea.plateng.patch.ProductAliasTargetHelper.getProdAliasTargetList(ProductAliasTargetHelper.java:55)
        at com.bea.plateng.patch.ProductAliasTargetHelper.getAllHomeToProdAliasesTargetMap(ProductAliasTargetHelper.java:32)
        at com.bea.plateng.patch.ProductAliasTargetHelper.checkProfilesInProductAliases(ProductAliasTargetHelper.java:133)
        at com.bea.plateng.patch.Patch$1.run(Patch.java:376)
        at java.lang.Thread.run(Thread.java:748)
java.lang.NullPointerException

 

 Solution:

 

Ensure  all the files and directories under weblogic home are under same owner and group.
 

Creating custom jvm scripts for starting through node manager

The primary aim of nodemanager is to restart managed servers when the go down and also you can start or stop managed servers from the console.
Now StartScriptEnabled=true will use the managed server script. If your start script is named startWebLogic.sh or startWebLogic.cmd, Node Manager uses one of those scripts as the default. So now when you start a managed server from console, it will use its own startWebLogic.sh or startWebLogic.cmd to come up via nodemanager since StartScriptEnabled=true.
If you would like to Set different environment properties to each jvm then doing it in the console is one of the best methods, however you can also modify the startweblogic.sh script to the example as below

if ["${SERVER_NAME}"==ms1]
then
JAVA_OPTIONS="-Xverboselog:/osb/oracle/bea/user_projects/domains/osb_domain/servers/OSBJMS2/logs/ms1gc1.log -Xverbose:memory ${JAVA_OPTIONS}"
echo %JAVA_OPTIONS%
exit
fi

if ["${SERVER_NAME}"==ms2]
then
JAVA_OPTIONS="-Xverboselog:/osb/oracle/bea/user_projects/domains/osb_domain/servers/OSBJMS2/logs/ms2gc2.log -Xverbose:memory ${JAVA_OPTIONS}"
echo %JAVA_OPTIONS%
exit
fi

high availability of admin server

<Warning> <Management> <BEA-141191> <The prepare phase of the configuration update failed with an exception:
weblogic.management.DeploymentException: Exception occured while downloading files
       at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.doDownload(ConfigDataUpdate.java:83)
       at weblogic.deploy.internal.targetserver.datamanagement.DataUpdate.download(DataUpdate.java:56)
       at weblogic.deploy.internal.targetserver.datamanagement.Data.prepareDataUpdate(Data.java:98)
       at weblogic.management.provider.internal.RuntimeAccessDeploymentReceiverService.downloadFiles(RuntimeAccessDeploymentReceiverService.java:898)
       at weblogic.management.provider.internal.RuntimeAccessDeploymentReceiverService.updateDeploymentContext(RuntimeAccessDeploymentReceiverService.java:564)
       Truncated. see log file for complete stacktrace
java.io.IOException: Posted content doesn't set it's Content-Length
       at weblogic.deploy.service.datatransferhandlers.MultipartParser.parseResponse(MultipartParser.java:97)
       at weblogic.deploy.service.datatransferhandlers.MultipartParser.<init>(MultipartParser.java:82)
       at weblogic.deploy.service.datatransferhandlers.MultipartParser.<init>(MultipartParser.java:45)
       at weblogic.deploy.service.datatransferhandlers.HttpDataTransferHandler.getDataAsStream(HttpDataTransferHandler.java:88)
       at weblogic.deploy.service.datatransferhandlers.DataHandlerManager$RemoteDataTransferHandler.getDataAsStream(DataHandlerManager.java:153)
       Truncated. see log file for complete stacktrace

Managed server going into Admin mode after Adminserver fails over.

As per KM 1294379.1, Released that there were 0sized files under the config directory of domain which were not being synchronized, removing those files fixed the problem.

Take threaddumps, jra recording and others with a single command.

Create a file ctrlhandler.act in the weblogic domain
contents of ctrlhandler.act (For R27 Jrockit version) are
 ............................................
set_filename filename=/tmp/output.txt
print_class_summary
heap_diagnostics
print_threads
timestamp
startjrarecording recordingtime=60s filename="/tmp/robin"
stop
..................................................
  For R28 version use
start_flightrecording duration=60s filename="/tmp/robin.jfr"

For windows the command is ctrl + break on the cmd screen, where weblogic is running.
For unix, start the weblogic as . ./startweblogic.sh & and kill -3 <pid> to activate.

To find the current working directory use   ls -l /proc/<PID>/cwd 
i recommend to keep set_filename as that of current working directory.
 
ref:
http://download.oracle.com/docs/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/ctrlbreakhndlr.html

Thread dump in windows

on the command prompt (cmd)
go to the weblogic path ex: D:\work\wls\user_projects\domains\base_domain>
set environment variables ex: D:\work\wls\user_projects\domains\base_domain>setEnv.cmd
enter below command
ex: D:\work\wls\user_projects\domains\base_domain> java weblogic.Admin -url t3://localhost:7001 -username weblogic -password welcome1 THREAD_DUMP
replace t3://localhost:7001 with your console address
replace weblogicwith your console username
replace welcome1with your console password
enter the last command 5 times in 5 seconds.

You can also use WLST
bin>setDomainEnv.cmd
connect('username','password','t3://ip:port') threadDump('true', 'td1.txt') 

         if you are using SSL
 bin>setDomainEnv.cmd
>java -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.TrustKeyStore=DemoTrust weblogic.WLST 
>connect('username','password','t3s://ip:port') threadDump('true', 'td1.txt') 

Platform MBean Server Used

For you to find "Platform MBean Server Used" and enable it. The weblogic version has to be 10.3.3 or above.
For lower versions: us the below method.

1. start your weblogic admin and open the console.
2, invoke *\wlserver_10.3\common\bin wlst.cmd or wlst.sh
3.
If you have configured SSL run below, else move to step 4
java -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.TrustKeyStore=DemoTrust weblogic.WLST
4.
wls:/offline> connect()
Please enter your username [weblogic] :weblogic
Please enter your password [welcome1] :
Please enter your server URL [t3://localhost:7001] :
wls:/base_domain/edit> startEdit()
wls:/base_domain/edit !> cd ('JMX')
wls:/base_domain/edit/JMX !> ls()
drw-   base_domain
wls:/base_domain/edit/JMX !> cd ('base_domain')
wls:/base_domain/edit/JMX/base_domain !> ls()

-rw-   CompatibilityMBeanServerEnabled              true
-rw-   DomainMBeanServerEnabled                     true
-rw-   EditMBeanServerEnabled                       true
-rw-   InvocationTimeoutSeconds                     0
-rw-   ManagementEJBEnabled                         true
-rw-   Name                                         base_domain
-rw-   Notes                                        null
-rw-   PlatformMBeanServerEnabled                   false
-rw-   PlatformMBeanServerUsed                      false
-rw-   RuntimeMBeanServerEnabled                    true
-r--   Type                                         JMX

-r-x   freezeCurrentValue                           Void : String(attributeName

-r-x   isSet                                        Boolean : String(propertyName)
-r-x   restoreDefaultValue                          Void : String(attributeName

-r-x   unSet                                        Void : String(propertyName)

wls:/base_domain/edit/JMX/base_domain !> cmo.setPlatformMBeanServerUsed(true)
wls:/base_domain/edit/JMX/base_domain !> save()
Saving all your changes ...
Saved all your changes successfully.
wls:/base_domain/edit/JMX/base_domain !> exit()

For 10.3.5 and above you can simply do the below

1.) Set the Platform MBean Server Enabled from Admin Console Domain >Configuration > General >Advanced >Platform MBean Server Enabled

2.) Enable this system property in server startup scripts

-Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder

 ref: http://download.oracle.com/docs/cd/E14571_01/web.1111/e13728/understandwls.htm#JMXCU239

Script to check the status of application in weblogic domain & send email

Place the script under *\wlserver_10*\common\bin
Trigger it like wlst.sh robin.py (for windows: wlst.cmd robin.py)
To redirect the output to a file  ./wlst.sh robin.py > robin.log
To send by email  mailx -s "status" "robin4444@blogspot.com" < robin.log

below is robin.py code

#wlst script to check status of deployed applications

import os
import sys

username='weblogic'
password='welcome1'
url='t3://localhost:7001'
connect(username, password, url)
print ' Application Deployed '

cd("AppDeployments")
app = ls(returnMap='true')

domainRuntime()

cd("AppRuntimeStateRuntime/AppRuntimeStateRuntime")

print ' Targetted on Admin '
i=1

for appName in app:
        appState = cmo.getIntendedState(appName, "AdminServer")
        print appName, '','==>',     appState

i=i+1
print ' '
print ' Targetted on Cluster '
i=1

for appName in app:
        appState = cmo.getIntendedState(appName, "new_Cluster_1")
        print appName, '','==>',     appState

i=i+1
print ' '
print ' visit robin4444.blogspot.com today '


change the below values to suite your respective domain's values
weblogic welcome1 t3://localhost:7001 AdminServer new_Cluster_1

so, Usage will be like   
./wlst.sh robin.py > robin.log; mailx -s "status" "robin4444@blogspot.com" < robin.log

check the JDBC statistics

wls/common/bin
./wlst.sh

Initializing WebLogic Scripting Tool (WLST) …

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Please enter your username [weblogic] :
Please enter your password [weblogic] :
Please enter your server URL [t3://localhost:7001] :t3://localhost:7003
Connecting to t3://localhost:7003 with userid weblogic …
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

wls:/base_domain/serverConfig> serverRuntime()
Location changed to serverRuntime tree. This is a read-only tree with ServerRunt
imeMBean as the root.
For more help, use help(serverRuntime)
wls:/base_domain/serverRuntime> cd(‘JDBCServiceRuntime’)
wls:/base_domain/serverRuntime/JDBCServiceRuntime> cd(‘AdminServer′)
wls:/base_domain/serverRuntime/JDBCServiceRuntime/AdminServer> cd(‘JDBCDataSourceRuntimeMBeans’)
wls:/base_domain/serverRuntime/JDBCServiceRuntime/AdminServer/JDBCDataSourceRuntimeMBeans> cd(‘MyDs’)
wls:/base_domain/serverRuntime/JDBCServiceRuntime/AdminServer/JDBCDataSourceRuntimeMBeans/MyDs>ls()

Start weblogic without DB

To force WLS to not access the back end database during start up.
Datasource -> config -> connection pool -> advanced -> Connection Creation Retry Frequency (seconds)
Changing it to a non-zero value will allow WLS to boot without accessing database.

Warning: JMS queue 'weblogic.wsee.DefaultQueue' is not found

Error message:
<Server started in RUNNING mode>
 <WSEE:18>Warning: JMS queue 'weblogic.wsee.DefaultQueue' is not found, as a result, Web Service async responses via jms transport is not supported. If the target service uses JMS transport, the responses will not be able to come back.<JmsQu eueListener.connect:287>

Solution:

add this in startup command line:

-Dweblogic.wsee.skip.async.response=true

cant save changes in console when using BIG IP

Have configured BIG IP for https traffic and not able to save the changes when accessing through https://<BIG_IP>:<PORT>/console

however able to make changes by http://<DNS_of_machine>:<PORT>/console  

Issue was resolved after enabling “Weblogic Plugin Enabled” parameter at the Domain -> Configuration ->Web Applications Tab .

change java version / JDK of a weblogic domain

1> Take a backup and edit <DOMAIN_HOME>\bin\setDomainEnv.sh /.cmd

after the lines:
set WL_HOME=D:\work\wls\wlserver_10.3
for %%i in ("%WL_HOME%") do set WL_HOME=%%~fsi

set BEA_JAVA_HOME=D:\work\wls\jrockit_160_17_R28.0.0-679

set SUN_JAVA_HOME=D:\work\wls\jdk160_18

add the line
set JAVA_VENDOR=Oracle     for jrockit
set JAVA_VENDOR=Sun        for sun jdk

so the setDomainEnv.sh or setDomainEnv.cmd will look like below for jrockit

set WL_HOME=D:\work\wls\wlserver_10.3
for %%i in ("%WL_HOME%") do set WL_HOME=%%~fsi

set BEA_JAVA_HOME=D:\work\wls\jrockit_160_17_R28.0.0-679

set SUN_JAVA_HOME=D:\work\wls\jdk160_18

set JAVA_VENDOR=Oracle

if "%JAVA_VENDOR%"=="Oracle" (


However if you have JVM tuning parameters set in your startup scripts, you would have to set the corresponding jvm parameters of the new java.
so in that case its best create a new domain with above changes, else you can do the changes in current domain.

2> else look for the below lines in setdomainenv.sh

# We need to reset the value of JAVA_HOME to get it shortened AND
# we can not shorten it above because immediate variable expansion will blank it

under the above lines add the new jdk path and vendor so the result would be like

# We need to reset the value of JAVA_HOME to get it shortened AND
# we can not shorten it above because immediate variable expansion will blank it
JAVA_HOME="/u01/CR-root/Oracle/Middleware/combinedjdk/jdk1.6.0_37"
JAVA_VENDOR="Sun"
export JAVA_HOME
export JAVA_VENDOR

save it in vi mode as esc:wq

ref : Doc ID 1058804.1

Weblogic server takes long to start

Only on server present in the domain and it took more then 45mins to come up.

Have deleted the cache present in tmp, cache & stage folders within user_projects\domains\<DOMAIN>\servers\<myserver> after stopping the server.
Also 2GB diagnostics file under /user_projects/domains/DOMAIN_NAME/servers/SERVER_NAME/data/store/diagnostics.
The server is now fine.

weblogic installation fail

Error message: A fatal error has occurred. This application will terminate.


User had downloaded a generic versoin of weblogic (64bit) which came as *.zip and had unzipped the file in linux to find a installer.jar file which she ran and received the error.

The generic file must be run the way it is ignoring the *.zip extension.
for 64 bit jvm ex: java -d64 -jar *generic.zip
for 32 bit jvm ex: java -jar *generic.zip
Provide full permissions to the file (chmod -r 777 *.zip)

Linux flavour template
$ java -d64 -Djava.io.tmpdir=<your temp dir> -jar wls1033_generic.jar -log=/tmp/wls.log

Windows template
d:\java -d64 -jar wls1034_generic.jar -Djava.io.tmpdir=<your temp dir>


If you are unable to set the java try below:
/home/misc/jdk1.6.0_25/bin/java -jar wls1034_generic.jar -mode=console