Few managed servers automatically disconnect from datasource

Of the many managed servers from a single box targeted onto a datasource, a few would automatically
disconnect suddenly.


<Error> <Deployer> <BEA-149205> <Failed to initialize the application '*' due to error weblogic.application.ModuleException: .
weblogic.application.ModuleException:
      at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:302)
      at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)
      at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)
      at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
      at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:159)
      Truncated. see log file for complete stacktrace
Caused By: weblogic.common.ResourceException: weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: IO Error: Connection reset
      at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:276)
      at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1249)
      at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1166)
      at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:249)
      at weblogic.jdbc.common.internal.ConnectionPool.doStart(ConnectionPool.java:1154)
      Truncated. see log file for complete stacktrace

Enabled the parameter "Connection Creation Retry Frequency" to 60 in all the servers. which solved the issue, it will keep trying to reestablish connection every 60 secs, after there has been a disconnection of DB with weblogic. So every time DB gets disconnected, the weblogic reconnects to it within 60secs.

PosixSocketMuxer

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '71' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "633" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@158051e[null null null]",
which is more than the configured time (StuckThreadMaxTime) of "600" seconds
. Stack trace:
weblogic.socket.DevPollSocketMuxer.read(DevPollSocketMuxer.java:77)
weblogic.servlet.internal.MuxableSocketHTTP.requeue(MuxableSocketHTTP.java:236)
weblogic.servlet.internal.VirtualConnection.requeue(VirtualConnection.java:333)
weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1544)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1462)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

we can see in the thread dump, all the muxer threads are stuck waiting for the lock so none of those thread can poll.

Switching to weblogic.socket.PosixSocketMuxer from the weblogic.socket.DevPollSocketMuxer solved this issue.

To enable PosixSocketMuxer via WLST:
edit()
cd ('Servers/'MyServerName')
startEdit()
set('MuxerClass','weblogic.socket.PosixSocketMuxer')
activate()

Failed to create App/Comp mbeans for AppDeploymentMBean odi.em

Error msg
<Error> <Deployer> <BEA-149605> <Failed to create App/Comp mbeans for AppDeploymentMBean odi.em. Error - weblogic.management.DeploymentException: .
weblogic.management.DeploymentException:
      at weblogic.servlet.internal.WarDeploymentFactory.findOrCreateComponentMBeans(WarDeploymentFactory.java:69)
      at weblogic.application.internal.MBeanFactoryImpl.findOrCreateComponentMBeans(MBeanFactoryImpl.java:48)
      at weblogic.application.internal.MBeanFactoryImpl.createComponentMBeans(MBeanFactoryImpl.java:110)
      at weblogic.application.internal.MBeanFactoryImpl.initializeMBeans(MBeanFactoryImpl.java:76)
      at weblogic.management.deploy.internal.MBeanConverter.createApplicationMBean(MBeanConverter.java:91)
      Truncated. see log file for complete stacktrace
Caused By: java.io.FileNotFoundException: File not found
      at java.util.zip.ZipFile.open(Native Method)
      at java.util.zip.ZipFile.<init>(ZipFile.java:117)
      at java.util.zip.ZipFile.<init>(ZipFile.java:133)
      at weblogic.servlet.utils.WarUtils.existsInWar(WarUtils.java:84)
      at weblogic.servlet.utils.WarUtils.isWebServices(WarUtils.java:76)
      Truncated. see log file for complete stacktrace

EM as part of SOA deployed on weblogic, after starting weblogic, the EM application is looking for ODi references. The resource it is looking for is odi.em while starting.However weblogic still starts up after EM is uninstalled. The resource odi.em is being referenced in weblogic.xml , unjar and remove that reference and recompile the EM application.

Rotate .out log files

1) If you are using nodemanager you can redirect .out to .log and control them via console (server -> logging tab under console)
traverse to */user_projects/domains/base_domain/bin/nodemanager
take a backup of wlscontrol.sh and edit it
change OutFile=$ServerDir/logs/$ServerName.out to OutFile=$ServerDir/logs/$ServerName.log

ELSE

2) You can make changes to the OS parameters under /etc/logrotate.conf
add the below lines at the end of the file

<Location of logs directory>/*.out
{
copytruncate
rotate 4
size=20Mb
}

ELSE

3)Add the flag  -Dweblogic.log.RedirectStdoutToServerLogEnabled=true to redirect .out to .log and control them via console (server -> logging tab under console)
and if you are starting via nodemanager add below additional parameters
nodemanager.properties under */wlserver_10.3/common/nodemanager
StartScriptName=startWebLogic.cmd
StartScriptEnabled=true

ELSE

4)Use OS command "logrotate" (via cron) to achieve your target. You can combine it with copytruncate to rotate logs.
http://linuxcommand.org/man_pages/logrotate8.html

certicom is unable to read cacerts

Error msg:
java.io.IOException: PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11
       at weblogic.server.channels.DynamicSSLListenThread.<init>(DynamicSSLListenThread.java:64)
       at weblogic.server.channels.DynamicListenThreadManager.createListener(DynamicListenThreadManager.java:289)
       at weblogic.server.channels.AdminPortService.bindListeners(AdminPortService.java:76)
       at weblogic.server.channels.EnableAdminListenersService.start(EnableAdminListenersService.java:39)
       at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
       at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
       at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.security.cert.CertificateParsingException: PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11
       at com.certicom.security.cert.internal.x509.X509V3CertImpl.<init>(Unknown Source)
       at com.certicom.tls.interfaceimpl.CertificateSupport.addTrustedCertificate(Unknown Source)
       at com.certicom.net.ssl.SSLContext.addTrustedCertificate(Unknown Source)
       at com.bea.sslplus.CerticomSSLContext.addTrustedCA(Unknown Source)
       at weblogic.security.utils.SSLContextWrapper.addTrustedCA(SSLContextWrapper.java:62)
       at weblogic.security.utils.SSLContextManager.createServerSSLContext(SSLContextManager.java:424)
       at weblogic.security.utils.SSLContextManager.getChannelSSLContext(SSLContextManager.java:336)
       at weblogic.security.utils.SSLContextManager.getSSLServerSocketFactory(SSLContextManager.java:91)
       at weblogic.server.channels.DynamicSSLListenThread.<init>(DynamicSSLListenThread.java:59)
       ... 6 more


After looking at the execption it seems that WLS 10.3 does not support cipher suite with Object ID 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
Please take a backup and replace the cacerts file located in JDK_HOME/jre/lib/security with the cacerts located file in WL_HOME/server/lib.

Looks like the 10.3.0.0 version of certicom is unable to read the cacerts of JDK 1.6 u29.

A version attribute is required

WLS 10.3.4 
error message:
<Warning> <Munger> <BEA-2156203> <A version attribute was not found in element webservices in the deployment descriptor in weblogic.utils.classloaders.ChangeAwareClassLoader@49e98c6d finder: weblogic.utils.classloaders.CodeGenClassFinder@65254b11 annotation: bea_wls9_async_response@bea_wls9_async_response.war/WEB-INF/webservices.xml. A version attribute is required, but this version of the Weblogic Server will assume that the JEE5 is used. Future versions of the Weblogic Server will reject descriptors that do not specify the JEE version.>

I applied the parameter -Dweblogic.wsee.skip.async.response=true in setdomainenv.sh
and error message disappeared. 
else you could always apply patch 11720907 or cumulative patch 12770299.

Timeout read connections from JDBC side on weblogic.

When a client requests data from the DB through a thread and there is no reply from the server the client indefinitely waits for the reply.
You can avoid this situation by timing out the thread from JDBC side. It will end up with a "Socket READ TIMEOUT" error so you don't indefinitely wait for a reply from server.Make sure the time period you set is strictly not shorter than any of the normal db queries of your application, else it will timeout normal connections.

oracle.net.READ_TIMEOUT
works for jdbc driver versions <=10.2
does NOT work for jdbc driver versions >= 11.1

oracle.jdbc.ReadTimeout
works for jdbc driver versions >= 10.1.0.5
Does not work for jdbc driver versions <10.1.0.5

You can set it in the JDBC_Data_*-jdbc.xml in the domain/config/JDBC folder. Timeout in miliseconds.

<jdbc-driver-params>
     <url>jdbc:oracle:thin:@localhost:1521:XE</url>
    <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name>
     <properties>
      <property>
        <name>user</name>
        <value>SYSTEM</value>
      </property>
      <property>
        <name>oracle.jdbc.ReadTimeout</name>
         <value>18000</value>
      </property>
     </properties>
    <password-encrypted>{3DES}s447QVesIco=</password-encrypted>
   </jdbc-driver-params>

also you can set it via console in

Admin Console -> JDBC Data Source: Configuration: Connection Pool, and set "Properties" as follows. Note to list each property=value pair on a separate line.

oracle.net.READ_TIMEOUT=30000 

oracle.jdbc.ReadTimeout=30000

To check the JDBC driver version:
 D:\work\wls\wlserver_10.3\server\lib>java -jar ojdbc6.jar
Oracle 11.1.0.7.0-Production JDBC 4.0 compiled with JDK6

Activate changes taking long time

Activate changes was taking forever....
I went ahead and deleted crc.ser and ver.ser under
user_projects\domains\base_domain\config\configCache.
Then i found out that the config.xml copy under the different jvm's present in other physical machines are not similar in size.
restart those jvm's whose config.xml size is not equal to that of the admins.
Activate changes is working fine now.

Apply weblogic Patch to unix

January 2011 weblogic patches are available at
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=REFERENCE&id=1263333.1

( Note: If you're wondering about  what "No Error Correction (NEC)" is: it means if you find an issue on WLS-10.3.1,there will be no fix provided, the fix will be provided only for WLS-10.3.2 onwards)

Copy all patches along with the patch-catalog.xml file to the Unix box at
*/bea1033/utils/bsu/cache_dir

now if the patch comes with an xml which is like  patch-catalog_xxxx.xml (ex : patch-catalog_4444.xml) then you would have to rename this  patch-catalog_xxxx.xml to  patch-catalog.xml in cache_dir
make sure that if a file patch-catalog.xml is already present in cache_dir before you dropping the patch there
you would have to back it up first and then rename patch-catalog-xxxx.xml to patch-catalog.xml

On the Unix machine go to the bsu folder.
( Ex : */bea1033/utils/bsu/cache_dir ) and run the following command :

Syntax :
$ ./bsu.sh -prod_dir= -patchlist= -verbose -install -log=/tmp/error.log

ex:
$ ./bsu.sh -prod_dir=*/bea1033/wlserver_10.3 -patchlist=B1WK -verbose -install -log=/tmp/error.log

If you want to install multiple patches at the time use the below command :

$ ./bsu.sh -prod_dir=*/bea1033/wlserver_10.3 -patchlist=B1WK,P2MT -verbose -install -log=/tmp/error.log


You can later use the below command to check the output

*\bea1033\utils\bsu>bsu -report

restart domain

To remove a patch

On the Solaris machine go to the bsu folder. ( Ex : */bea1033/utils/bsu ) and run the following command :

bsu -remove -patchlist=4MHU -prod_dir=D:\bea\weblogic92 –verbose

If you get the error:
"Patch id **** is not associated with the given product dir."
Then the patch does not belong to the WLS version you are in, most likely bug for the patch has already been fixed in the current version.

If you get the below error:
com.bea.plateng.common.util.logging.LogWriter -  java.io.IOException: Unable to backup file D:\work\wls\patch_wls1033\patch_jars\BUG*.jar to D:\work\wls\patch_wls1033\patch_jars\BUG*.jar53888.bak
com.bea.plateng.patch.CommandLinePatchInstaller - An unexpected error was encountered removing patch DFYK.  That resource may be busy.
The patch was not removed, and has been restored to its original state.  For further information, enable logging and run patch removal again.
com.bea.plateng.patch.PatchRemovalException: java.io.IOException: Unable to backup file D:\work\wls\patch_wls1033\patch_jars\BUG*.jar to D:\work\wls\patch_wls1033\patch_jars\BUG*.jar53888.bak


The OS has the JAR locked,  So please kill all OS related weblogic processes like nodemanager and beasvc service.

 If you get the below error:

    2014-06-25 02:48:59,328 DEBUG [Main Thread] com.bea.plateng.common.util.logging.LogWriter - Restoring /data/XXX//wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar from /XXXX/patch_wls1036/backup/backup.jar
    2014-06-25 02:48:59,332 ERROR [Main Thread] com.bea.plateng.patch.FilePatchRemover - Error restoring files from backup archive.
    2014-06-25 02:48:59,333 ERROR [Main Thread] com.bea.plateng.patch.PatchRemover - Error encountered removing patch.
    2014-06-25 02:48:59,335 DEBUG [Main Thread] com.bea.plateng.patch.CommandLineProgress - Command line progress stopped
    2014-06-25 02:48:59,335 DEBUG [Main Thread] com.bea.plateng.common.util.logging.LogWriter - Result: Failure
    2014-06-25 02:48:59,335 DEBUG [Main Thread] com.bea.plateng.common.util.logging.LogWriter - Failure condition follows:

The patch was not removed, and has been restored to its original state.  For further information, enable logging and run patch removal again.
com.bea.plateng.patch.PatchRemovalException: java.io.FileNotFoundException: /XXX/wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar (No such file or directory)

 while patching being rolled back, its looking for jar file in wrong location.

bash-3.2$ grep -i /XXXX/wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar patch-backup.xml
   <target>/XXXX/wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar</target>
   <target>/XXXX/wlserver_10.3/server/lib/schema/weblogic-domain-binding.jar</target>

 

 

 

If you get the below error:

 

Exception in thread "Thread-0" java.lang.NullPointerException
        at com.bea.plateng.patch.PatchTargetHelper.loadPatchProfiles(PatchTargetHelper.java:447)
        at com.bea.plateng.patch.PatchTarget.<init>(PatchTarget.java:268)
        at com.bea.plateng.patch.PatchTargetFactory.create(PatchTargetFactory.java:30)
        at com.bea.plateng.patch.ProductAliasTarget.constructPatchTargetList(ProductAliasTarget.java:88)
        at com.bea.plateng.patch.ProductAliasTarget.<init>(ProductAliasTarget.java:46)
        at com.bea.plateng.patch.ProductAliasTargetHelper.getProdAliasTargetList(ProductAliasTargetHelper.java:55)
        at com.bea.plateng.patch.ProductAliasTargetHelper.getAllHomeToProdAliasesTargetMap(ProductAliasTargetHelper.java:32)
        at com.bea.plateng.patch.ProductAliasTargetHelper.checkProfilesInProductAliases(ProductAliasTargetHelper.java:133)
        at com.bea.plateng.patch.Patch$1.run(Patch.java:376)
        at java.lang.Thread.run(Thread.java:748)
java.lang.NullPointerException

 

 Solution:

 

Ensure  all the files and directories under weblogic home are under same owner and group.
 

Creating custom jvm scripts for starting through node manager

The primary aim of nodemanager is to restart managed servers when the go down and also you can start or stop managed servers from the console.
Now StartScriptEnabled=true will use the managed server script. If your start script is named startWebLogic.sh or startWebLogic.cmd, Node Manager uses one of those scripts as the default. So now when you start a managed server from console, it will use its own startWebLogic.sh or startWebLogic.cmd to come up via nodemanager since StartScriptEnabled=true.
If you would like to Set different environment properties to each jvm then doing it in the console is one of the best methods, however you can also modify the startweblogic.sh script to the example as below

if ["${SERVER_NAME}"==ms1]
then
JAVA_OPTIONS="-Xverboselog:/osb/oracle/bea/user_projects/domains/osb_domain/servers/OSBJMS2/logs/ms1gc1.log -Xverbose:memory ${JAVA_OPTIONS}"
echo %JAVA_OPTIONS%
exit
fi

if ["${SERVER_NAME}"==ms2]
then
JAVA_OPTIONS="-Xverboselog:/osb/oracle/bea/user_projects/domains/osb_domain/servers/OSBJMS2/logs/ms2gc2.log -Xverbose:memory ${JAVA_OPTIONS}"
echo %JAVA_OPTIONS%
exit
fi

high availability of admin server

<Warning> <Management> <BEA-141191> <The prepare phase of the configuration update failed with an exception:
weblogic.management.DeploymentException: Exception occured while downloading files
       at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.doDownload(ConfigDataUpdate.java:83)
       at weblogic.deploy.internal.targetserver.datamanagement.DataUpdate.download(DataUpdate.java:56)
       at weblogic.deploy.internal.targetserver.datamanagement.Data.prepareDataUpdate(Data.java:98)
       at weblogic.management.provider.internal.RuntimeAccessDeploymentReceiverService.downloadFiles(RuntimeAccessDeploymentReceiverService.java:898)
       at weblogic.management.provider.internal.RuntimeAccessDeploymentReceiverService.updateDeploymentContext(RuntimeAccessDeploymentReceiverService.java:564)
       Truncated. see log file for complete stacktrace
java.io.IOException: Posted content doesn't set it's Content-Length
       at weblogic.deploy.service.datatransferhandlers.MultipartParser.parseResponse(MultipartParser.java:97)
       at weblogic.deploy.service.datatransferhandlers.MultipartParser.<init>(MultipartParser.java:82)
       at weblogic.deploy.service.datatransferhandlers.MultipartParser.<init>(MultipartParser.java:45)
       at weblogic.deploy.service.datatransferhandlers.HttpDataTransferHandler.getDataAsStream(HttpDataTransferHandler.java:88)
       at weblogic.deploy.service.datatransferhandlers.DataHandlerManager$RemoteDataTransferHandler.getDataAsStream(DataHandlerManager.java:153)
       Truncated. see log file for complete stacktrace

Managed server going into Admin mode after Adminserver fails over.

As per KM 1294379.1, Released that there were 0sized files under the config directory of domain which were not being synchronized, removing those files fixed the problem.

Take threaddumps, jra recording and others with a single command.

Create a file ctrlhandler.act in the weblogic domain
contents of ctrlhandler.act (For R27 Jrockit version) are
 ............................................
set_filename filename=/tmp/output.txt
print_class_summary
heap_diagnostics
print_threads
timestamp
startjrarecording recordingtime=60s filename="/tmp/robin"
stop
..................................................
  For R28 version use
start_flightrecording duration=60s filename="/tmp/robin.jfr"

For windows the command is ctrl + break on the cmd screen, where weblogic is running.
For unix, start the weblogic as . ./startweblogic.sh & and kill -3 <pid> to activate.

To find the current working directory use   ls -l /proc/<PID>/cwd 
i recommend to keep set_filename as that of current working directory.
 
ref:
http://download.oracle.com/docs/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/ctrlbreakhndlr.html

Thread dump in windows

on the command prompt (cmd)
go to the weblogic path ex: D:\work\wls\user_projects\domains\base_domain>
set environment variables ex: D:\work\wls\user_projects\domains\base_domain>setEnv.cmd
enter below command
ex: D:\work\wls\user_projects\domains\base_domain> java weblogic.Admin -url t3://localhost:7001 -username weblogic -password welcome1 THREAD_DUMP
replace t3://localhost:7001 with your console address
replace weblogicwith your console username
replace welcome1with your console password
enter the last command 5 times in 5 seconds.

You can also use WLST
bin>setDomainEnv.cmd
connect('username','password','t3://ip:port') threadDump('true', 'td1.txt') 

         if you are using SSL
 bin>setDomainEnv.cmd
>java -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.TrustKeyStore=DemoTrust weblogic.WLST 
>connect('username','password','t3s://ip:port') threadDump('true', 'td1.txt') 

Platform MBean Server Used

For you to find "Platform MBean Server Used" and enable it. The weblogic version has to be 10.3.3 or above.
For lower versions: us the below method.

1. start your weblogic admin and open the console.
2, invoke *\wlserver_10.3\common\bin wlst.cmd or wlst.sh
3.
If you have configured SSL run below, else move to step 4
java -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.TrustKeyStore=DemoTrust weblogic.WLST
4.
wls:/offline> connect()
Please enter your username [weblogic] :weblogic
Please enter your password [welcome1] :
Please enter your server URL [t3://localhost:7001] :
wls:/base_domain/edit> startEdit()
wls:/base_domain/edit !> cd ('JMX')
wls:/base_domain/edit/JMX !> ls()
drw-   base_domain
wls:/base_domain/edit/JMX !> cd ('base_domain')
wls:/base_domain/edit/JMX/base_domain !> ls()

-rw-   CompatibilityMBeanServerEnabled              true
-rw-   DomainMBeanServerEnabled                     true
-rw-   EditMBeanServerEnabled                       true
-rw-   InvocationTimeoutSeconds                     0
-rw-   ManagementEJBEnabled                         true
-rw-   Name                                         base_domain
-rw-   Notes                                        null
-rw-   PlatformMBeanServerEnabled                   false
-rw-   PlatformMBeanServerUsed                      false
-rw-   RuntimeMBeanServerEnabled                    true
-r--   Type                                         JMX

-r-x   freezeCurrentValue                           Void : String(attributeName

-r-x   isSet                                        Boolean : String(propertyName)
-r-x   restoreDefaultValue                          Void : String(attributeName

-r-x   unSet                                        Void : String(propertyName)

wls:/base_domain/edit/JMX/base_domain !> cmo.setPlatformMBeanServerUsed(true)
wls:/base_domain/edit/JMX/base_domain !> save()
Saving all your changes ...
Saved all your changes successfully.
wls:/base_domain/edit/JMX/base_domain !> exit()

For 10.3.5 and above you can simply do the below

1.) Set the Platform MBean Server Enabled from Admin Console Domain >Configuration > General >Advanced >Platform MBean Server Enabled

2.) Enable this system property in server startup scripts

-Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder

 ref: http://download.oracle.com/docs/cd/E14571_01/web.1111/e13728/understandwls.htm#JMXCU239

Script to check the status of application in weblogic domain & send email

Place the script under *\wlserver_10*\common\bin
Trigger it like wlst.sh robin.py (for windows: wlst.cmd robin.py)
To redirect the output to a file  ./wlst.sh robin.py > robin.log
To send by email  mailx -s "status" "robin4444@blogspot.com" < robin.log

below is robin.py code

#wlst script to check status of deployed applications

import os
import sys

username='weblogic'
password='welcome1'
url='t3://localhost:7001'
connect(username, password, url)
print ' Application Deployed '

cd("AppDeployments")
app = ls(returnMap='true')

domainRuntime()

cd("AppRuntimeStateRuntime/AppRuntimeStateRuntime")

print ' Targetted on Admin '
i=1

for appName in app:
        appState = cmo.getIntendedState(appName, "AdminServer")
        print appName, '','==>',     appState

i=i+1
print ' '
print ' Targetted on Cluster '
i=1

for appName in app:
        appState = cmo.getIntendedState(appName, "new_Cluster_1")
        print appName, '','==>',     appState

i=i+1
print ' '
print ' visit robin4444.blogspot.com today '


change the below values to suite your respective domain's values
weblogic welcome1 t3://localhost:7001 AdminServer new_Cluster_1

so, Usage will be like   
./wlst.sh robin.py > robin.log; mailx -s "status" "robin4444@blogspot.com" < robin.log

check the JDBC statistics

wls/common/bin
./wlst.sh

Initializing WebLogic Scripting Tool (WLST) …

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Please enter your username [weblogic] :
Please enter your password [weblogic] :
Please enter your server URL [t3://localhost:7001] :t3://localhost:7003
Connecting to t3://localhost:7003 with userid weblogic …
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

wls:/base_domain/serverConfig> serverRuntime()
Location changed to serverRuntime tree. This is a read-only tree with ServerRunt
imeMBean as the root.
For more help, use help(serverRuntime)
wls:/base_domain/serverRuntime> cd(‘JDBCServiceRuntime’)
wls:/base_domain/serverRuntime/JDBCServiceRuntime> cd(‘AdminServer′)
wls:/base_domain/serverRuntime/JDBCServiceRuntime/AdminServer> cd(‘JDBCDataSourceRuntimeMBeans’)
wls:/base_domain/serverRuntime/JDBCServiceRuntime/AdminServer/JDBCDataSourceRuntimeMBeans> cd(‘MyDs’)
wls:/base_domain/serverRuntime/JDBCServiceRuntime/AdminServer/JDBCDataSourceRuntimeMBeans/MyDs>ls()

Start weblogic without DB

To force WLS to not access the back end database during start up.
Datasource -> config -> connection pool -> advanced -> Connection Creation Retry Frequency (seconds)
Changing it to a non-zero value will allow WLS to boot without accessing database.

Warning: JMS queue 'weblogic.wsee.DefaultQueue' is not found

Error message:
<Server started in RUNNING mode>
 <WSEE:18>Warning: JMS queue 'weblogic.wsee.DefaultQueue' is not found, as a result, Web Service async responses via jms transport is not supported. If the target service uses JMS transport, the responses will not be able to come back.<JmsQu eueListener.connect:287>

Solution:

add this in startup command line:

-Dweblogic.wsee.skip.async.response=true

cant save changes in console when using BIG IP

Have configured BIG IP for https traffic and not able to save the changes when accessing through https://<BIG_IP>:<PORT>/console

however able to make changes by http://<DNS_of_machine>:<PORT>/console  

Issue was resolved after enabling “Weblogic Plugin Enabled” parameter at the Domain -> Configuration ->Web Applications Tab .