Thursday, February 6, 2014

deadlock in file rotation

configuration : wls 10.3.6 + coherence + OSB

Situation : all logs are being written to one file.

Error:

Below is the chain of events

ExecuteThread: '9'
Blocked trying to get lock: weblogic/work/CalendarQueue@0x2913e168[thin lock]   <--1
     ^-- Holding lock: weblogic/logging/LoggingPrintStream@
0x2a2a1f68[thin lock]      at sun/nio/cs/StreamEncoder.implFlush(StreamEncoder.java:278)[inlined]    <--4

"weblogic.cluster.MessageReceiver"   id=94 idx=0x158 tid=14612 prio=5 alive, blocked, native_blocked, daemon   
Blocked trying to get lock: weblogic/logging/FileStreamHandler@0x29bddfd8[thin lock]                                           <--2
Holding lock: weblogic/work/C****Queue@0x2913e168[recursive]     <--1
Holding lock: weblogic/work/C****Queue@0x2913e168[thin lock]

"Timer-2"   id=25 idx=0x58 tid=12819 prio=5 alive, blocked, native_blocked, daemon   
Blocked trying to get lock: com/bea/logging/StdoutHandler@0x23dc37f0[thin lock]  <--3
Holding lock: weblogic/logging/FileStreamHandler@0x29bddfd8[recursive]     <--2
at com/bea/logging/RotatingFileOutputStream$1.run(RotatingFileOutputStream.java:166)   
Holding lock: weblogic/logging/FileStreamHandler@0x29bddfd8[thin lock]

"Timer-5"   id=40 idx=0x88 tid=13405 prio=5 alive, blocked, native_blocked, daemon   
Blocked trying to get lock: weblogic/logging/LoggingPrintStream@0x2a2a1f68[thin lock] <--4
Holding lock: com/bea/logging/StdoutHandler@0x23dc37f0[recursive]     <--3
at com/bea/logging/StdoutHandler.publish(StdoutHandler.java:39)   
Holding lock: com/bea/logging/StdoutHandler@0x23dc37f0[thin lock]

Circular (deadlocked) lock chains
Chain 2:
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x5c tid=3592 waiting for weblogic/logging/FileStreamHandler@0xaa22cbb8 held by:
"Timer-2" id=24 idx=0x54 tid=3589 waiting for weblogic/work/C****Queue@0xa88491c0 held by:
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x5c tid=3592 


Fix : disable the below in the console - server
Redirect stdout logging enabled
Redirect stderr logging enabled
& redirect coherence logs to separate file 

In a nutshell write individual logs seperately.

or apply the patch17070169 for 10.3.6

1 comment:

Blogger said...

Did you know that that you can make money by locking selected pages of your blog / website?
Simply join AdscendMedia and implement their Content Locking tool.