Few managed servers automatically disconnect from datasource

Of the many managed servers from a single box targeted onto a datasource, a few would automatically
disconnect suddenly.


<Error> <Deployer> <BEA-149205> <Failed to initialize the application '*' due to error weblogic.application.ModuleException: .
weblogic.application.ModuleException:
      at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:302)
      at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)
      at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)
      at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
      at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:159)
      Truncated. see log file for complete stacktrace
Caused By: weblogic.common.ResourceException: weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: IO Error: Connection reset
      at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:276)
      at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1249)
      at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1166)
      at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:249)
      at weblogic.jdbc.common.internal.ConnectionPool.doStart(ConnectionPool.java:1154)
      Truncated. see log file for complete stacktrace

Enabled the parameter "Connection Creation Retry Frequency" to 60 in all the servers. which solved the issue, it will keep trying to reestablish connection every 60 secs, after there has been a disconnection of DB with weblogic. So every time DB gets disconnected, the weblogic reconnects to it within 60secs.

PosixSocketMuxer

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '71' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "633" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@158051e[null null null]",
which is more than the configured time (StuckThreadMaxTime) of "600" seconds
. Stack trace:
weblogic.socket.DevPollSocketMuxer.read(DevPollSocketMuxer.java:77)
weblogic.servlet.internal.MuxableSocketHTTP.requeue(MuxableSocketHTTP.java:236)
weblogic.servlet.internal.VirtualConnection.requeue(VirtualConnection.java:333)
weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1544)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1462)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

we can see in the thread dump, all the muxer threads are stuck waiting for the lock so none of those thread can poll.

Switching to weblogic.socket.PosixSocketMuxer from the weblogic.socket.DevPollSocketMuxer solved this issue.

To enable PosixSocketMuxer via WLST:
edit()
cd ('Servers/'MyServerName')
startEdit()
set('MuxerClass','weblogic.socket.PosixSocketMuxer')
activate()

Failed to create App/Comp mbeans for AppDeploymentMBean odi.em

Error msg
<Error> <Deployer> <BEA-149605> <Failed to create App/Comp mbeans for AppDeploymentMBean odi.em. Error - weblogic.management.DeploymentException: .
weblogic.management.DeploymentException:
      at weblogic.servlet.internal.WarDeploymentFactory.findOrCreateComponentMBeans(WarDeploymentFactory.java:69)
      at weblogic.application.internal.MBeanFactoryImpl.findOrCreateComponentMBeans(MBeanFactoryImpl.java:48)
      at weblogic.application.internal.MBeanFactoryImpl.createComponentMBeans(MBeanFactoryImpl.java:110)
      at weblogic.application.internal.MBeanFactoryImpl.initializeMBeans(MBeanFactoryImpl.java:76)
      at weblogic.management.deploy.internal.MBeanConverter.createApplicationMBean(MBeanConverter.java:91)
      Truncated. see log file for complete stacktrace
Caused By: java.io.FileNotFoundException: File not found
      at java.util.zip.ZipFile.open(Native Method)
      at java.util.zip.ZipFile.<init>(ZipFile.java:117)
      at java.util.zip.ZipFile.<init>(ZipFile.java:133)
      at weblogic.servlet.utils.WarUtils.existsInWar(WarUtils.java:84)
      at weblogic.servlet.utils.WarUtils.isWebServices(WarUtils.java:76)
      Truncated. see log file for complete stacktrace

EM as part of SOA deployed on weblogic, after starting weblogic, the EM application is looking for ODi references. The resource it is looking for is odi.em while starting.However weblogic still starts up after EM is uninstalled. The resource odi.em is being referenced in weblogic.xml , unjar and remove that reference and recompile the EM application.