Provide user with monitor role access to view JMS messages (via console & WLST)

1. enable JMX policy editor
Login to console - security realms - myrealm -configuration - general - enable :Use Authorization Providers to Protect JMX Access - save - activate changes - restart

2. create user with monitor role
Login to console - security realm - myrealm - users&groups - users-new - create new user -save - click on that user again - groups - select monitor on the left table and move it to right - save -

3. create policy
Login to console - security realm - myrealm - roles & policies - Realm Policies - JMX Policy Editor - global scope - next - weblogic.management.runtime - JMSDestinationRuntimeMBean - next - Operations: Permission to Invoke - create policy -add conditions - Predicate List: user - next -
type your user and add - finish - save

4. now login to console as user (with monitor role) and try reading a message


WLST:

 For granular approach of specific permission of get messages only:

cmo.createPolicy('type=<jmx>, operation=invoke, application=, mbeanType=weblogic.management.runtime.JMSDestinationRuntimeMBean, target=getMessages','{Rol(Monitor)}')

For a broader permissions remove the targets:

cmo.createPolicy('type=<jmx>, operation=invoke, application=, mbeanType=weblogic.management.runtime.JMSDestinationRuntimeMBean','{Rol(Monitor)}')


I was able to get the resource details by enabling audit logging
http://docs.oracle.com/cd/E12840_01/wls/docs103/secwlre/xacmlusing.html#wp1268689

java.lang.OutOfMemoryError: GC overhead limit exceeded

Issue : java.lang.OutOfMemoryError: GC overhead limit exceeded in jvm logs
details : Sun JDK 1.6, parallel collector GC,
solution : apply the param in java_options -XX:+UseGCOverheadLimit     
It uses a policy that limits the proportion of the machines time that is spent in GC before an OutOfMemory error is thrown.
however this parameter will not avoid outofmemory at a later stage.

Create User and set Role in WLST

connect('weblogic','welcome1','t3://localhost:8001')

edit()

wls:/base_domain/serverConfig>startEdit(-1,-1,'false')

serverConfig()

easeSyntax()

wls:/base_domain/serverConfig/cd SecurityConfiguration/base_domain/Realms/myrealm/AuthenticationProviders/DefaultAuthenticator

wls:/base_domain/serverConfig/SecurityConfiguration/base_domain/Realms/myrealm/AuthenticationProviders/DefaultAuthenticator> cmo.createUser ('robin','welcome1','test')

wls:/base_domain/serverConfig/SecurityConfiguration/base_domain/Realms/myrealm/RoleMappers/XACMLRoleMapper> cmo.setRoleExpression ('','Anonymous','Usr(robin)')

wls:/base_domain/serverConfig/SecurityConfiguration/base_domain/Realms/myrealm/RoleMappers/XACMLRoleMapper> save
<function save 1>

wls:/base_domain/serverConfig/SecurityConfiguration/base_domain/Realms/myrealm/RoleMappers/XACMLRoleMapper> activate
<function activate 2>

Set the MaxRequestParamterCount Value

Error:
weblogic.utils.http.MaxRequestParameterExceedException
at weblogic.utils.http.QueryParams.checkLimit(QueryParams.java:52)
at weblogic.utils.http.QueryParams.getCurrentAndCheck(QueryParams.java:46)
at weblogic.utils.http.QueryParams.put(QueryParams.java:79)

Description:
The default value of MaxRequestParamterCount is 10,000, crossing which would cause a MaxRequestParameterExceedException

Solution:
1.We can apply patch 13656558 for version 10.3.6, but since it is a security patch it is not freely available
Instead we can upgrade our version to 10.3.6.0.4 by applying patch 16083651.
(patch 16083651already in itself includes the patch 13656558).

2. Follow Doc ID 1505598.1

The MaxRequestParamterCount can be set in 3 places:

    On the WebAppContainerMBean, which is under the DomainMBean and has domain wide effect.
    On the WebServerMBean, which is under ServerMBean and has effect on only JVM. 
    This overrides settings on the WebAppContainerMBean.
    On the VirtualHostMBean. Virtual hosts, like servers, are under DomainMBean. 
    This overrides settings on the WebAppContainerMBean.

Use WLST to set it
 
WebAppContainerMBean:

$ connect('<user>','<pwd>','<admin_url>')
$ edit()
$ startEdit()
$ cmo.getWebAppContainer().setMaxRequestParamterCount(1000)
$ save()
$ activate()
$ exit()

WebServerMBean:

$ connect('<user>','<pwd>','<admin_url>')
$ edit()
$ startEdit()
$ cd('Servers/<server-name>')
$ cmo.getWebServer().setMaxRequestParamterCount(1000)
$ save()
$ activate()
$ exit()

For VirtualHostMBean:
$ connect('<user>','<pwd>','<admin_url>')
$ edit()
$ startEdit()
$ cd('VirtualHosts/<virtualhost>')
$ cmo.setMaxRequestParamterCount(1000)
$ save()
$ activate()
$ exit()



I tried making the change in my 12.1.1.0.3 and it works fine.

Adminserver.log
####<Aug 21, 2013 10:45:03 AM IST> <Info> <Management> <****> <> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1377062103640> <BEA-000000> <Version: WebLogic Server 12.1.1.0.3 PSU Patch for BUG14736141 Mon Dec 17 02:10:51 MST 2012


cmd

D:\work\wls\wls12.1.1\user_projects\domains\base_domain\bin>setDomainEnv.cmd
D:\work\wls\wls12.1.1\user_projects\domains\base_domain>cd D:\work\wls\wls12.1.1
\wlserver_12.1\common\bin
D:\work\wls\wls12.1.1\wlserver_12.1\common\bin>
D:\work\wls\wls12.1.1\wlserver_12.1\common\bin>wlst.cmd
wls:/base_domain/config> connect('weblogic','welcome1','t3://localhost:8001')
wls:/base_domain/config> easeSyntax()
wls:/base_domain/serverConfig/Servers/m1/WebServer/m1>
wls:/base_domain/serverConfig/Servers/m1/WebServer/m1> edit
<function edit 3>
wls:/base_domain/edit> cd Servers
wls:/base_domain/edit/Servers> cd m1
wls:/base_domain/edit/Servers/m1> cd WebServer
wls:/base_domain/edit/Servers/m1/WebServer> cd m1
wls:/base_domain/edit/Servers/m1/WebServer/m1> startEdit ()  
Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
wls:/base_domain/edit/Servers/m1/WebServer/m1 !> cmo.getMaxRequestParamterCount()
10000
wls:/base_domain/edit/Servers/m1/WebServer/m1 !> cmo.setMaxRequestParamterCount(-1)
wls:/base_domain/edit/Servers/m1/WebServer/m1 !>ls
wls:/base_domain/edit/Servers/m1/WebServer/m1 !> ls
dr--   Targets
dr--   WebServerLog
-rw-   AcceptContextPathInGetRealPath               false
-rw-   AuthCookieEnabled                            true
-rw-   Charsets                                     null
-rw-   ChunkedTransferDisabled                      false
-rw-   ClientIpHeader                               null
-rw-   DefaultWebAppContextRoot                     null
-rw-   DeploymentOrder                              1000
-rw-   FrontendHTTPPort                             0
-rw-   FrontendHTTPSPort                            0
-rw-   FrontendHost                                 null
-rw-   HttpsKeepAliveSecs                           60
-rw-   KeepAliveEnabled                             true
-rw-   KeepAliveSecs                                30
-rw-   MaxPostSize                                  -1
-rw-   MaxPostTimeSecs                              -1
-rw-   MaxRequestParamterCount                      -1

wls:/base_domain/edit/Servers/m1/WebServer/m1 !> save
<function save 4>
wls:/base_domain/edit/Servers/m1/WebServer/m1 !> save ()
Saving all your changes ...
Saved all your changes successfully.  

Reference:

Doc ID 1505598.1

popup appears multilpe times

Problem : 
ADF application times out with message

"Page Expired The page has expired. Click OK to continue."

if this message is left for a while, sometimes the following message is displayed multiple times:

"Because of inactivity, your session has timed out and is no longer active.
Click OK to reload the page."

solution : patch 13366844 for WLS 10.3.6

JCA binding

error message : 
 <Error> <oracle.soa.adapter> <BEA-000000> <JCABinding=>  SCA_Oracle*_Aceva:Oracle*_FileWrite [ Write_ptt::Write(body) ]  Could not invoke operation 'Write' against the 'File Adapter' due to:
BINDING.JCA-11063
Unable to acquire mutex for interaction.
Unable to acquire mutex for interaction.
Unable to acquire lock on resource "/*/*/soa_nfs/data/*/file/*PAYMENT/output*.tmp" for "DatabaseMutex::acquireNoSave"

This is a SOA bug 13344070

Native Library(terminalio) not found

Error message :
<Mar 4, 2013 9:31:55 AM PST> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:<Mar 4, 2013 9:31:55 AM PST> <Error> <Security> <BEA-090782> <Server is Running in Production Mode and Native Library(terminalio) to read the password securely from commandline is not found.>
<Mar 4, 2013 9:31:55 AM PST> <Notice> <WebLogicServer> <BEA-000388> <JVM called WLS shutdown hook. The server will force shutdown now

Soln:
Looks like  library "terminalio" is missing on your system, this is a OS related issue, however you can workaround this in weblogic by the following method.

Go to the domain / server / security folder for example :
user_projects\domains\base_domain\servers\AdminServer\security
if security folder is not found under adminserver, go ahead an create it. and provide it 777 permissions (chmod -r -R 777 security)
for unix system ->

cd user_projects\domains\base_domain\servers\AdminServer\
mkdir security
chmod 777 security/

under security folder create a new file called boot.properties and enter the values of weblogic username and password and save + exit.

for unix system ->

cd security/
vi boot.properties
esc + i
username=XXXX
password=XXXX

esc + :wq

for windows system ->

under security
right click -> New Text Document.txt
open it
username=XXXX
password=XXXX

file - save as -
file name : boot.properties
save as type : All files

save

repeat the same procedure for your managed server as well.

start your admin server

jdk1.6.0_37/bin/java: not found

Description : after installing wls 10.3.6 64 bit and jdk 1.6u37 64 bit sparc we are not able to start weblogic
Error message : jdk1.6.0_37/bin/java: not found

Solution : download the 64 bit jdk as well as 32 bit jdk from
http://www.oracle.com/technetwork/java/archive-139210.html

Place them on winSCP and upload using binary mode.

Provide chmod 777 *.sh to both the files.

-rwxrwxrwx   1 slcruser other    12721088 Jan 31 00:16 jdk-6u37-solaris-sparcv9.sh
-rwxrwxrwx   1 slcruser other    76926483 Jan 31 00:17 jdk-6u37-solaris-sparc.sh

install the 32 bit version first ( jdk-6u37-solaris-sparc.sh) and later the 64 bit version (jdk-6u37-solaris-sparcv9.sh) on the same path as 32 bit version and provide this new path of jdk in setdomainenv.sh

refer my earlier blog
"http://robin4444.blogspot.in/2011/05/change-java-version-jdk-of-weblogic.html"
on how to update setdomainenv.sh with the new jdk path.

GSSException No valid credentials provided Mechanism level Failed to find any Kerberos Ticket Key

Error message :
Caused By: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Key)

    Cause:
This may occur if no valid Kerberos credentials are obtained. In particular, this occurs if you want the underlying mechanism to obtain credentials but you forgot to indicate this by setting the javax.security.auth.useSubjectCredsOnly system property value to false (for example via -Djavax.security.auth.useSubjectCredsOnly=false in your execution command).

    Solution:
Be sure to set the javax.security.auth.useSubjectCredsOnly system property value to false if you want the underlying mechanism to obtain credentials, rather than your application or a wrapper program (such as the Login utility used by some of the tutorials) performing authentication using JAAS.

Stop the admin server
Modify the startWebLogic.sh file located in DOMAIN_HOME/bin
add the flag -Djavax.security.auth.useSubjectCredsOnly=false in java_options
save and start your admin server

below is a typical example assuming your weblogic is on linux box

Modify the startWebLogic.sh file located in DOMAIN_HOME/bin

Locate the following line:
# Start WebLogic

Below this line, add the following, replacing the correct path to krb5Login.conf:
JAVA_OPTIONS=”${JAVA_OPTIONS} -Djava.security.auth.login.config=/opt/Oracle/Middleware/user_projects/domains/base_domain/krb5Login.conf –Djavax.security.auth.useSubjectCredsOnly=false –Dweblogic.security.enableNegotiate=true”
export JAVA_OPTIONS

Save the file. Start the Administration Server


For Windows
Modify the startWebLogic.cmd file located in DOMAIN_HOME\bin (default is c:\Oracle\Middleware\user_projects\domains\base_domain\bin\startWebLogic.cmd)

Locate the following line:
@REM START WEBLOGIC

Below this line, add the following:

set JAVA_OPTIONS=%JAVA_OPTIONS% -Djava.security.auth.login.config=%DOMAIN_HOME%\krb5Login.conf –Djavax.security.auth.useSubjectCredsOnly=false –Dweblogic.security.enableNegotiate=true

Ref : http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/Troubleshooting.html

webserver ip address is being recorded instead of client ip address after enabling WebLogic Plug-In Enabled option in console

webserver ip address is being recorded instead of client ip address after enabling WebLogic Plug-In Enabled option in console (<weblogic-plugin-enabled>true</weblogic-plugin-enabled>)


1. Checked the plugin version by hitting the url
http://<host>:<port>/?__WebLogicBridgeConfig  (console ip and port)
found it to be 1.1

WebLogic Server Plugin version 1.1,
<WLSPLUGINS_11.1.1.6.0_WINDOWS.X64_RELEASE>

Ref : Doc id 1285304.1

2. Made sure that the customer has enabled WebLogic Plug-In Enabled option at the below places

Environment >>> clusters >>  CLUSTER NAME > configuration -> general tab -> WebLogic Plug-In  Enabled (checked)
Environment >>> SERVER >> SERVER NAME > General -> Advanced -> WebLogic Plug-In Enabled (checked)
Domain_Name >>> Configuration Tab > Web Applications Tab  -> WebLogic Plug-In Enabled (checked)

3. Enabled debugs in webserver config file
Debug ALL
DebugConfigInfo ON

Ref : Doc ID 780007.1

Later realized that customer has installed  Microsoft TMG server in the same machine as that of webserver which is masking the client ip.

Memory utilization / usage of java / weblogic processes on RAM for unix / linux

Get the output of below commands

1. cat /proc/meminfo
2. ps -A --sort -rss -o comm,pmem | head -n 11
3. ps -A --sort -rss -o pid,comm,pmem,rss
4. free -m
5. while read command percent rss; do if [[ "${command}" != "COMMAND" ]]; then rss="$(bc <<< "scale=2;${rss}/1024")"; fi; printf "%-26s%-8s%s\n" "${command}" "${percent}" "${rss}"; done < <(ps -A --sort -rss -o comm,pmem,rss | head -n 11)
6. ps -aux

2. COMMAND
ps -A --sort -rss -o comm,pmem | head -n 11

RESULT
COMMAND         %MEM
java            13.4
java            11.8
java            10.7
java             5.8
java             5.1
java             5.0
emagent          0.5
java             0.3
perl             0.0
ntpd             0.0

The java / weblogic processes are using only 52.1% of the total memory

4. COMMAND
free -m

RESULT
             total       used       free     shared    buffers     cached
Mem:         56378      55716        662          0        175       3245
-/+ buffers/cache:      52295       4083
Swap:         4094          0       4094


When you run free command, look for the line which says “-/+ buffers/cache”. This line which shows numbers under used and free, is the RAM you literally have used and literally

have free because of ram allocated to cache and buffers,
this is the line you should refer to when you want to see how much free RAM you have because RAM allocated to buffers and cache will be instantly released and given to a

process whenever it needs it.
So you have 4GB RAM free now.

5. COMMAND
while read command percent rss; do if [[ "${command}" != "COMMAND" ]]; then rss="$(bc <<< "scale=2;${rss}/1024")"; fi; printf "%-26s%-8s%s\n" "${command}" "${percent}" "${rss}"; done < <(ps -A --sort -rss -o comm,pmem,rss | head -n 11)

RESULT
COMMAND                   %MEM    RSS
java                      13.4    7610.14
java                      11.8    6686.60
java                      10.7    6088.11
java                      5.8     3274.82
java                      5.1     2893.45
java                      5.0     2821.26
emagent                   0.5     312.08
java                      0.3     175.11
perl                      0.0     9.97
ntpd                      0.0     4.78

RSS is the portion of memory that a process is using that is held in pysical RAM and is calculated in kb's.

6. COMMAND
ps -aux

RESULT
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2064   620 ?        Ss   Oct20   0:03 init [5]
root         2  0.0  0.0      0     0 ?        S<   Oct20   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   Oct20   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S<   Oct20   0:00 [watchdog/0]

shows you the rss memory as well as the virtual memory of all the pid's, you may then look for your respective pid in the list.
Please note that here the VSZ - virtual size is in kilobytes, RSS - real memory size is in byte units.



Some useful AIX commands:
svmon -Pg -t 1 |grep Pid ; svmon -Pg -t 10 |grep "N"          top 10 processes using the most paging space

svmon -P -O sortseg=pgsp                                               shows paging space usage of processes

ps gv | head -n 1; ps gv | egrep -v "RSS" | sort +6b -7 -n -r 

Cruise to lakshadweep

Organized by SPORTS
Booking agent: mintvalley kochi
Islands : Kalpeni & minicoy
Ship : MV lakshadweep sea
package : coral reef

Lakshadweep is one of the best places to travel if you would like to get away from the city for a couple of days. Nice warm shallow beaches, warm hearted people, world class facilities, amazing sea creatures and super duper food.
Mintvalley tours kochi (pretty - 0484 6062011) are one of the best agencies to reach out to for booking a slot on SPORTS. Make sure you plan your vacation a month in advance and always choose coral reef package over samudram package. Samudram is a 3 island tour with 158 co-passengers which creates unnecessary competition for fewer resources, while coral reef takes you to 2 islands with 8 passengers.
Make sure you travel by MV Lakshadweep which is one of the newest ships with world class facilities.
It is better to travel by 1st class which includes individual cabin with attached rest room comfortable bed, TV, radio, study table and window to the sea and centralized AC.  You also have an agent of SPORTS accompanying you on board the ship.
Food + tea + snacks on board the ship and islands are part of the package, which was a pleasant surprise.
The ship usually leaves by around 6pm even though the agency asks you to report by 1:30pm to sports office. Make sure you take a taxi or cab to sports office on indira Gandhi road, willington island, kochi as auto’s usually demand more than double the fare once they reach the destination and it might get ugly.
Once you reach the sports office on the second floor they offer you a complimentary t-shirt and cap and provide you with boarding pass and original bill of the trip. They then take you via auto (paid by them)to the boarding point where you and your luggage are checked thoroughly many times by the CISF, who themselves have no clue, as to what are they looking for. You are then loaded onto a bus and your luggage on a truck to the harbor.  The journey from here on is nothing but pure heaven.  Porters take your luggage to your cabin and the ship takes more than an hour to start its journey. If they leave by 6pm the first island kalpeni can be reached by 6:30 in the morning. A mechanized boat takes you from ship to the island where you are greeted by a sports agent who takes you to the report in a jeep.
Kalpeni is a peaceful island with predominately sunni Muslim population, preachers from the main land have slowly turned them into an orthodox yet peaceful population. Only men are seen everywhere while women generally stay home. Everyone speaks hindi and Malayalam. Snorkeling is the major highlight of this island as you can see corals and ornamental fishes in plenty, you also have canoeing and local sightseeing via jeep. You are also taken to local cloth factory where you get quality t-shirts at throw away price.  Make sure you buy tuna pickle from kalpeni. By 6 in the evening you are dropped back to the shore where a boat takes you back to the ship.
Minicoy is the most awesome island of Lakshadweep, I was truly taken aback by its natural beauty and the warmth of the people. Minicoy is ruled by women and they still retain their cultural identity and their way of life unlike kalpeni. Apart from snorkeling canoeing and visit to the lighthouse you also get to visit a local village. These people are a stark contrast to the city dwellers. Unity, honesty, community workmanship and selflessness are every evident in all the islanders. They speak very good hindi and malyalam and are in tune with the latest bollywood news. Their waters are loaded with tuna and islands with coconut trees. No dogs or crows on the island yet.  The boat drops you back to the ship and you reach kochi by 9am. No security check on arrival to the mainland.
Make sure you take a bottle of amla drink (concentrated vitamin C) to keep you from feeling sea sick.
Also carry an extra pair of T-shirt and tracks on the island and mineral water as well. Pics available on my flickr link.

Error in storeUserConfig windows environment

Below is the error:

wls:/nm/formsdomain> storeUserConfig('D:\nm\nmuserconfigfile.secure','D:\nm\nmuserkeyfile.secure','true')
Currently connected to Node Manager to monitor the domain formsdomain.
Creating the key file can reduce the security of your system if it is not kept in a secured location after it is created. Do you want to create the key file? y or n  -> y
Error: Not able to write secret key file.  Check the location and priviledges of the specified file location.

Solution:
1. Set the environment by using setdomainenv before executing storeUserConfig.
2. Ensure the folders have enough permissions where the .secure files are kept. 
3. Use \\ instead of \ in storeUserconfig command for example : D:\\nm\\nmuserconfigfile.secure\\
since windows has the habit of assuming \ as a char or special symbol, like \n is referred as new line it is best to give \\ to indicate a path.

FMW admin server unable to start with other admin users account

FMW-weblogic admin server is working fine when started by user who created it. However it is not start able by other admin account users of the box.

error message:
Caused By: oracle.security.jps.JpsRuntimeException: oracle.security.jps.JpsException: [PolicyUtil] Exception while getting default policy Provider
    at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:291)
    at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:282)
    at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:261)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

MW_HOME\user_projects\domains\bifoundation_domain\config\fmwconfig\bootstrap\
file cwallet.sso, right click on it and properties - security add users to admin group (give permissions)

if its still not working, restart the machine.

This is because the installation was done by one user and run by another user who does not have the same set of permissions.

custom banner on weblogic console

If you would like to have your own customized banner on weblogic console.
Go to wlserver_10.3\server\lib\consoleapp\webapp\framework\skins\wlsconsole\images
and edit the gif file in paint to create your own banner and clean restart weblogic server.
However doing this in production systems is illegal.

cannot open file _WLS_ADMINSERVER000000.DAT

Error message:1

<BEA-280061> <The persistent store "XXXX" could not be deployed: weblogic.store.PersistentStoreException: [Store:280020]There was an error while reading from the log file
weblogic.store.PersistentStoreException: [Store:280020]There was an error while reading from the log file
    at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:796)
    at weblogic.store.io.file.Heap.open(Heap.java:182)
    at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:88)
    at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:332)
    at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:323)

java.io.IOException: Error reading from file, No such file or directory, errno=2
    at weblogic.store.io.file.direct.DirectIONative.read(Native Method)
    at weblogic.store.io.file.direct.DirectFileChannel.read(DirectFileChannel.java:134)
    at weblogic.store.io.file.StoreFile.read(StoreFile.java:282)
    at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:794)
    at weblogic.store.io.file.Heap.open(Heap.java:182)
    at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:88)
    at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:332)
    at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:323)

Error message :2 

weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.
    at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
    at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
    at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
    at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)
    at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191)
    at weblogic.store.admin.DefaultStoreService.start(DefaultStoreService.java:60)
    at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.io.IOException: Error from open(), Permission denied, errno=13
    at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
    at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54)
    at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
    at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
    at weblogic.store.io.file.StoreFile.openDirect(StoreFile.java:168)
    at weblogic.store.io.file.Heap.openStoreFile(Heap.java:398)
    at weblogic.store.io.file.Heap.open(Heap.java:325)
    at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
    at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
    at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
    at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)
    at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191)
    at weblogic.store.admin.DefaultStoreService.start(DefaultStoreService.java:60)
    at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Solution 1: Stop weblogic and delete the .Dat file from *\user_projects\domains\<base_domain>\servers\<AdminServer>\data\store\default and diagnostics
start the JVM

Solution 2:  Follow solution1 and then login to console - server - config - services - change :Synchronous Write Policy: to cache-flush. select Advanced - uncheck :Enable File Locking - save -activate changes and restart.

Error : Caused By: java.io.IOException: Error from fcntl() for file locking

Soln:
1.) Uncheck "Enable File locking " for persistent stores

==> This option is not supported while using NFS filesystem

2.) Maintain different directories for filestores of different JMS Servers and maintain a different file store for each JMS server.

ref :
http://docs.oracle.com/cd/E14571_01/web.1111/e13814/storetune.htm#CACFDGCA
http://docs.oracle.com/cd/E24329_01/web.1211/e24390/storetune.htm#CAEGGIEG
http://docs.oracle.com/cd/E17904_01/web.1111/e13701/store.htm#i1143516
Doc ID 957377.1
Doc ID 1278828.1 

Two weblogic consoles in the same set of browser

Two weblogic consoles when opened in the same set of browser, logs out.
For example, if you open both consoles in firefox or IE, then both logs out, however if you open one console in firefox and another in IE, it does not logout.

That is because the default Console Cookie Name is ADMINCONSOLESESSION, and when two cookies are created in browser with same cookie name, there is a conflict.

To avoid this:
Go to console - domain - config - gen - adv -
edit the Console Cookie Name of ADMINCONSOLESESSION to something else - save - restart

Else use ip:port/console for one and hostname:port/console for other.

ANT script to deploy application

Create a file called Build.xml and below are its contents:
...............................................................................................................................................
<project name="Test Deployment" default="deploy">
<taskdef name="wldeploy" classname="weblogic.ant.taskdefs.management.WLDeploy">
<classpath>
<pathelement location="D:/work/wls/wls9.2.4/weblogic92/server/lib/weblogic.jar"/>
</classpath>
</taskdef>
<target name="deploy">
<wldeploy
        action="deploy" verbose="true" debug="true" upload="true" remote="true" stage="true" library="true"
        name="conference.shared.lib" source="C:\test\test.war"
        user="weblogic" password="weblogic"
        adminurl="t3://localhost:7001" targets="examplesServer" />
</target>
</project>
...............................................................................................................................................
# change the appropriate values to reflect your environment.

Start the weblogic server.
Run the setdomainenv.sh from the domain folder and then traverse to the file containing build.xml
and type $ ant
This will deploy the file.

The input line is too long - during installing windows service

Error: The input line is too long - during installing windows service.

Desc: MS Windows 32bit has 2K limitation on the length of the command line. If the classpath setting in the " installSvc.cmd " is very long, the 2K limitation could be exceeded. Consequently the set up of the MS Windows Service may fail with the error.

Soln: Start the jvm in the command prompt and copy the classpath from startup log.

Place the classpath in a seperate .txt file and provide complete access to it.

now call the file in the installsvc file.

by replacing -classpath \"%CLASSPATH%\" option with the following option:

-classpath @pathname\filename

 Ex:JAVA_OPTIONS% -classpath @C:\classpath.txt -Dweblogic.Name 

ref: http://docs.oracle.com/cd/E17904_01/web.1111/e13708/winservice.htm#i1188175

weblogic as windows service

To CREATE SERVICE
Go to${bea_dir}\wlserver_10.3\server\bin where bea_dir is home directory of weblogic installation.
Create a text file with following values:

echo off
SETLOCAL
set DOMAIN_NAME=base_domain
set USERDOMAIN_HOME=D:\work\wls\user_projects\domains\base_domain
set SERVER_NAME=AdminServer
set WLS_USER=weblogic
set WLS_PW=weblogic1
set PRODUCTION_MODE=false
set ADMIN_URL=http://localhost:7001
set JAVA_VENDOR=Sun (or set JAVA_VM=-hotspot)
set JAVA_HOME=D:\work\wls\jdk160_18
set MEM_ARGS=-Xms256m -Xmx512m
call "D:\work\wls\wlserver_10.3\server\bin\installSvc.cmd"
ENDLOCAL

save and rename to createSvc.cmd

Setting admin url and username password is optional if your domain picks up username/password from boot.propeties (domain_home\servers\AdminServer\security), no need to mention them in the script.

For Jrockit:
set JAVA_VENDOR=BEA
set JAVA_HOME=C:\bea\jrockit_160_05


execute the createSvc.cmd from command prompt so you can check if there are any errors.

This should create service with name of "beasvc sampleDomain_Adminserver_name" in the registry under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services

check services under (start - run - services.msc) to find your service with beasvc prefix. change it to automatic and start the service and check logs.

INCLUDING CLASSPATH
If you have variables set in classpath which need to be included, edit the installSvc.cmd and look for the line where it calls commenv and replace it with setdomainEnv path.

call "%WL_HOME%\common\bin\commEnv.cmd"

TO REMOVE SERVICE
Go to${bea_dir}\wlserver_10.3\server\bin where bea_dir is home directory of weblogic installation.
Create a text file with following values:

echo off
SETLOCAL
set DOMAIN_NAME=base_domain
set SERVER_NAME=AdminServer
call "D:\work\wls\wlserver_10.3\server\bin\uninstallSvc.cmd"
ENDLOCAL

save and rename to removeSvc.cmd
execute the createSvc.cmd from command prompt so you can check if there are any errors.
(details should be the same as that of createSvc.cmd)


TO CHANGE SERVICE NAME
take a backup of the installSvc.cmd
edit installSvc.cmd and look for beasvc and change the name you wanted to.
"%WL_HOME%\server\bin\beasvc" -install -svcname:"beasvc %DOMAIN_NAME%_%SERVER_NAME%"

TO ENABLE MANAGED SERVICE TO START AFTER ADMIN SERVICE
 make a copy of installSvc.cmd say installSvc2.cmd
edit installSvc2.cmd and make it dependent on admin server script

for example

rem *** Install the service
"%WL_HOME%\server\bin\beasvc" -install -svcname:"ROBIN_%DOMAIN_NAME%_%SERVER_NAME%" -depend:"beasvcAdminService"

so now managed service will depend on admin service before starting.
----------------------------------------------------
1> If you get the below error in the server log

javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found. Either it was has not been exported or it has been collected by the distributed garbage collector.]

remove the service by removeSvc.cmd
edit createSvc.cmd
remove line          set ADMIN_URL=http://localhost:7001
save and run.
----------------------------------------------------
2>  If you get the below error in the server log

<BEA-150000> <An error occurred while establishing a connection

javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]

Caused By: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector  

ensure the user has admin rights to the box
remove the service by removeSvc.cmd
edit createSvc.cmd
remove line          set ADMIN_URL=http://localhost:7001
save and run. 
----------------------------------------------------
3>  If you get the below error
'oracle.fabric.common.classloaderurl.handler' is not recognized as an internal or external command

Double quotation marks break the command syntax.

In the installSvc.cmd,
Change the %JAVA_OPTIONS% to \"%JAVA_OPTIONS%\"
(have to add \"  \")

For example
set CMDLINE="%JAVA_VM% %MEM_ARGS% %JAVA_OPTIONS% -classpath
will be come
set CMDLINE="%JAVA_VM% %MEM_ARGS% \"%JAVA_OPTIONS%\" -classpath

Doc ID 1276229.
----------------------------------------------------
To debug:
Go to the installSvc.cmd file and right at the location where you call the beasvc.exe file, insert the following options.

-debug –log:”path to log.txt”

So the entire string would look something like this.

"%WL_HOME%\server\bin\beasvc" –install  -debug  -log:c:\path.txt -svcname:"beasvc %DOMAIN_NAME%_%SERVER_NAME%" -javahome:"%JAVA_HOME%" -execdir:"%USERDOMAIN_HOME%" -maxconnectretries:"%MAX_CONNECT_RETRIES%" -host:"%HOST%" -port:"%PORT%" -extrapath:"%EXTRAPATH%" -password:"%WLS_PW%" -cmdline:%CMDLINE%


Also you can debug by traversing to the folder containing beasvc.exe
*\wlserver_10.3\server\bin> beasvc -debug "affected windows service name" 

Windows service log rotation
http://docs.oracle.com/cd/E14571_01/web.1111/e13708/winservice.htm#i1193313

reset / change weblogic username password

To change the admin password in a weblogic environment, please follow the below steps:

STEP 1 :  Firstly, Stop the server if running and make sure that weblogic.jar is in the classpath. (this could be checked by running setDomainEnv and env in linux or set in windows)

STEP 2 : Open a command prompt and navigate to your Domain directory’s bin folder and run “setDomainEnv.sh “
After you run setdomain the prompt should go back to domain folder from bin folder.
if it doesn't happen run the command as . ./setdomainEnv.sh (in words- dot space dot slash).
Run echo $CLASSPATH to check if weblogic.jar is set.
Run Echo %CLASSPATH% for windows.

STEP 3 : Now run the following command :

“Java weblogic.security.utils.AdminAccount NewAdminUser NewAdminPassword . “

where ” NewAdminUser ” and ” NewAdminPassword ” are the new user and password you would like to create.

Note : The “ . “ (period) at the end of the command is very important..!! as it tells the command that this should be run in the domain directory.

STEP 4 : The above command creates a file “ DefaultAuthenticatorInit.ldift “ in the domain directory.

STEP 5 :  Now copy the newly created “DefaultAuthenticatorInit.ldift “ to “*\bea10.3\user_projects\domains\change_password\security “ (where change_password is your affected domain)

Note : You can take a backup or delete  “DefaultAuthenticatorInit.ldift “ which is already present in the security folder

STEP 6 :  Now edit the boot.properties file ( located in “ *\bea10.3\user_projects\domains\change_password\servers\AdminServer\security “ & “ *\bea10.3\user_projects\domains\change_password\servers\managedserver\security “ ) with the new user name and password that we created ( i.e NewAdminUser  and NewAdminPassword )

Delete the file “ DefaultAuthenticatormyrealmInit.initialized “ located in “C:\bea10.3\user_projects\domains\change_password\servers\AdminServer\data\ldap “

Now start the server and login with the NewAdminUser  and NewAdminPassword.

To reset the admin password in a weblogic environment, please follow the below steps:

STEP 7 : After logging into the console with the NewAdminUser  and NewAdminPassword. go to security realms - myrealm (or your respective realm) - users and groups - click on your original user - passwords - change password - save - activate changes.

STEP 8 :  Edit the boot.properties file ( located in “ *\bea10.3\user_projects\domains\change_password\servers\AdminServer\security “ & “ *\bea10.3\user_projects\domains\change_password\servers\managedserver\security “ ) with the original user and password that we just reset.

jvm freeze

The stack trace :
"[ACTIVE] ExecuteThread: for queue: 'weblogic.kernel.Default (self-tuning)'" id=16 idx=0x58 tid=* prio=5 alive, waiting, native_blocked, daemon
-- Waiting for notification on: netscape/ldap/LDAPSearchListener@0x0000000184932A78[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:1973)[inlined]
at java/lang/Object.wait(Object.java:474)[optimized]
at netscape/ldap/LDAPMessageQueue.waitForMessage(LDAPMessageQueue.java:200)
^-- Lock released while waiting: netscape/ldap/LDAPSearchListener@0x0000000184932A78[recursive]
at netscape/ldap/LDAPMessageQueue.waitFirstMessage(LDAPMessageQueue.java:101)
^-- Lock released while waiting: netscape/ldap/LDAPSearchListener@0x0000000184932A78[fat lock]
at netscape/ldap/LDAPConnection.sendRequest(LDAPConnection.java:1796)
^-- Holding lock: netscape/ldap/LDAPConnection@0x0000000184924B50[biased lock]
at netscape/ldap/LDAPConnection.search(LDAPConnection.java:2566) 


 The StackoverFlow issue observed is caused by a misconfiguration of LDAP server.

Soln:
1. Open the console
2. through the menu Security realms->myRealm->Providers->Authenticaton->[provider]->Configuration->Provider Specific
3. Change the Group Membership Searching to limited
4. Change the Max Group Membership Search Level to 16
5. Save and restart WebLogic Server

BEA-000402 BEA-000438

Error message:
Warning> <Socket> <BEA-000402> <There are: 5 active sockets, but the maximum number of socket reader threads allowed by the configuration is: 4. You may want to alter your configuration.>
are because of the original messages of not picking up libmuxer.so

or

<Error> <Socket> <BEA-000438> <Unable to load performance pack. Using Java I/O instead. Please ensure that a native performance library is in:

Soln:
1. login to console - servers - configuration - tunning - make sure 'Enable Native IO' is clicked.
2. run the setdomainenv and then run env command.this will tell us the current libmuxer.so being used.
3. You need to make sure the corresponding libmuxer.so is used which could be done by

take a backup and at the end of your commEnv.sh file please add the lines

LD_LIBRARY_PATH=<path of the libmuxer.so>${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH

path of libmuxer.so
wlserver_10.3\server\native\<server type>\<bit version>\libmuxer.so

Ref : Doc ID 965299.1

SSL trouble on Weblogic 12c (12.1) nodemanger

On 12.1 version of weblogic:
Trying to configure SSL on nodemanger.
After setting the CustomIdentityKeyStorePassPhase  and CustomIdentityPrivateKeyPassPhase attributes in the nodemanager.properties file in clear text and restarting the nodemanager,
the values are not being replaced with the encrypted value.

On 10.3.X series it is working fine and values are being encrypted.

Workaround:
If you take the nm_data.properties file from a 10.3.x version (wlserver_10.3\common\nodemanager) and place it in (wlserver_12.1\common\nodemanager) path and then run a fresh startnodemanager.sh
The  CustomIdentityKeyStorePassPhase  and CustomIdentityPrivateKeyPassPhase values in nodemanager.properties are getting encrypted and also it deletes the nm_data.properties file.

To encrypt clear text passwords

To encrypt a clear text value which could be used in weblogic file
run the setdomainenv from the domain folder.
java weblogic.security.Encrypt
It will ask you for password and you may enter the value to be encrypted, The resultant encrypted value could be used in any weblogic file.

Check JDBC driver version

To check the JDBC driver version

run . ./setdomainenv.sh (dot space dot slash)
traverse to

cd *\wlserver_10.3\server\lib
java -jar ojdbc6.jar (where ojdbc6.jar is the driver's jar file)

you would get the result as
Oracle 11.1.0.7.0-Production JDBC 4.0 compiled with JDK6

so 11.1.0.7.0 is the version.

for db2 driver
wlserver_12.1\server\lib\wldb2.jar

D:\work\wls\wls12.1.1\wlserver_12.1\server\lib>java -cp wldb2.jar weblogic.jdbc.db2.DB2Driver
[FMWGEN][DB2 JDBC Driver]Driver Version: 4.2.1.035064 (F044246.U015812) 

WSDL excpetion in weblogic

Error message: WARNING: Input Action on WSDL operation PreparedOperation and @Action on its associated Web Method preparedOperation did not match and will cause problems in dispatching the request

or 

Error message : WARNING: Input Action on WSDL operation RegisterOperation and @Action on its associated Web Method registerOperation did not match and will cause problems in dispatching the requests
 

Weblogic version: 12c (12.1.1)

Solution : 1. change the domain from development mode to production mode.
                This could be done by opening the console and domain name --> configuration --> general
                click production mode enabled - save -activate changes - restart.
                2. Also a patch 13606167 is available.

Weblogic console showing old version after upgrade

After upgrading from 10.3.3 to 10.3.5, the console page is still showing old version.

config.xml & startup logs are showing the correct version.

under the middleware directory directory where the patch for upgrade installer is kept, i found two patch directories patch_wls1033 & patch_wls1035.
I deleted patch_wls1033 as my weblogic has now been upgraded to 10.3.5
The console is showing the correct version now.

Connection has been administratively disabled.

Seeing error message :
java.sql.SQLException: Connection has been administratively disabled.

intermittent issueand connection hangs for requests

Added the following parameter and problem solved
-Dweblogic.resourcepool.max_test_wait_secs=

domain creation stuck

Weblogic Domain creation stuck/froze at creating domain security information.......
Have taken a backup of config.sh (wlserver_10.3\common\bin)
added the variable -Djava.security.egd=file:/dev/./urandom in JVM_ARGS

it now looks like 
JVM_ARGS="-Djava.security.egd=file:/dev/./urandom -Dprod.props.file='${WL_HOME}/.product.properties' -Dpython.cachedir=/tmp/cachedir ${JVM_D64} ${MEM_ARGS} ${CONFIG_JVM_ARGS}"

rerunning config.sh solved the issue. 

Reset password in weblogic portal

Error message

Caused By: javax.security.auth.login.FailedLoginException: [Security:090302]
Authentication Failed: User canopiadmin denied
       at weblogic.security.providers.authentication.shared.DBMSAtnLoginModuleImpl.login(DBMSAtnLoginModuleImpl.java:285)
       at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
       at java.security.AccessController.doPrivileged(Native Method)
       at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       Truncated. see log file for complete stacktrace
 
In weblogic portal the authentication happens via DB, since it uses SQL based authentication.
1. We need to enter the original username password in database
2. enable plain text password in config.xml if its not already so, since the above entered credentials are in plaintext and table will have encrypted values.

to enable plaintext in config.xml as

<wls:plaintext-passwords-enabled>true</wls:plaintext-passwords-enabled>

so it would look like below in config.xml

<realm>
     <sec:authentication-provider xsi:type="wls:sql-authenticatorType">
       <sec:control-flag>SUFFICIENT</sec:control-flag>
       <wls:data-source-name>p**nDataSource</wls:data-source-name>
       <wls:plaintext-passwords-enabled>true</wls:plaintext-passwords-enabled> 

3. restart the servers.

plaintext password is reachable via console as
console - security realm - myrealm - providers - authentication - select the respective sql authenticator - config - provider specific - Plaintext Passwords Enabled

You may disable plain text after successful login,  

To disable WLDF diagnostics

Access the Administration Console for you domain.
Click Lock & Edit to start a new configuration session.
Select the Diagnostics->Diagnostic Modules node in the left pane of the console.
disable or stop or delete the respective module and carry out a restart.
Also please check if any of the options are enabled under console - data source - configuration - diagnostics  

Else

1. Shut down WebLogic Server.
2. Clear the <SERVER>/data/store/diagnostics directories
3. Apply the below 3 parameters at JAVA_OPTIONS of your setdomainenv  file

-D_Offline_FileDataArchive=true
To control diagnostics data collected from the server instance and the applications running on them. If true, WLDF archives the data collected (default: false).

-Dcom.bea.wlw.netui.disableInstrumentation=true
To control Workshop page flow event reporting. Set to false by default in Workshop 10.1 and earlier). Set to true by default (in Workshop 10.2 and later).

-Dweblogic.connector.ConnectionPoolProfilingEnabled=false
JDBC Connection Pool profiling data (default: false).

4. Restart WebLogic Server.
 

 Else

Take a backup of config.xml and delete the lines from and to <server-diagnostic-config>
restart

Else
 
You can adopt a retirement policy by rotating the .dat file
console -  diagnostics - archives - select the respective server - create NEW 'Data Retirement Policies' followed by a restart.

Few managed servers automatically disconnect from datasource

Of the many managed servers from a single box targeted onto a datasource, a few would automatically
disconnect suddenly.


<Error> <Deployer> <BEA-149205> <Failed to initialize the application '*' due to error weblogic.application.ModuleException: .
weblogic.application.ModuleException:
      at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:302)
      at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:199)
      at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:517)
      at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
      at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:159)
      Truncated. see log file for complete stacktrace
Caused By: weblogic.common.ResourceException: weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: IO Error: Connection reset
      at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:276)
      at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1249)
      at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1166)
      at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:249)
      at weblogic.jdbc.common.internal.ConnectionPool.doStart(ConnectionPool.java:1154)
      Truncated. see log file for complete stacktrace

Enabled the parameter "Connection Creation Retry Frequency" to 60 in all the servers. which solved the issue, it will keep trying to reestablish connection every 60 secs, after there has been a disconnection of DB with weblogic. So every time DB gets disconnected, the weblogic reconnects to it within 60secs.

PosixSocketMuxer

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '71' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "633" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@158051e[null null null]",
which is more than the configured time (StuckThreadMaxTime) of "600" seconds
. Stack trace:
weblogic.socket.DevPollSocketMuxer.read(DevPollSocketMuxer.java:77)
weblogic.servlet.internal.MuxableSocketHTTP.requeue(MuxableSocketHTTP.java:236)
weblogic.servlet.internal.VirtualConnection.requeue(VirtualConnection.java:333)
weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1544)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1462)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

we can see in the thread dump, all the muxer threads are stuck waiting for the lock so none of those thread can poll.

Switching to weblogic.socket.PosixSocketMuxer from the weblogic.socket.DevPollSocketMuxer solved this issue.

To enable PosixSocketMuxer via WLST:
edit()
cd ('Servers/'MyServerName')
startEdit()
set('MuxerClass','weblogic.socket.PosixSocketMuxer')
activate()

Failed to create App/Comp mbeans for AppDeploymentMBean odi.em

Error msg
<Error> <Deployer> <BEA-149605> <Failed to create App/Comp mbeans for AppDeploymentMBean odi.em. Error - weblogic.management.DeploymentException: .
weblogic.management.DeploymentException:
      at weblogic.servlet.internal.WarDeploymentFactory.findOrCreateComponentMBeans(WarDeploymentFactory.java:69)
      at weblogic.application.internal.MBeanFactoryImpl.findOrCreateComponentMBeans(MBeanFactoryImpl.java:48)
      at weblogic.application.internal.MBeanFactoryImpl.createComponentMBeans(MBeanFactoryImpl.java:110)
      at weblogic.application.internal.MBeanFactoryImpl.initializeMBeans(MBeanFactoryImpl.java:76)
      at weblogic.management.deploy.internal.MBeanConverter.createApplicationMBean(MBeanConverter.java:91)
      Truncated. see log file for complete stacktrace
Caused By: java.io.FileNotFoundException: File not found
      at java.util.zip.ZipFile.open(Native Method)
      at java.util.zip.ZipFile.<init>(ZipFile.java:117)
      at java.util.zip.ZipFile.<init>(ZipFile.java:133)
      at weblogic.servlet.utils.WarUtils.existsInWar(WarUtils.java:84)
      at weblogic.servlet.utils.WarUtils.isWebServices(WarUtils.java:76)
      Truncated. see log file for complete stacktrace

EM as part of SOA deployed on weblogic, after starting weblogic, the EM application is looking for ODi references. The resource it is looking for is odi.em while starting.However weblogic still starts up after EM is uninstalled. The resource odi.em is being referenced in weblogic.xml , unjar and remove that reference and recompile the EM application.

Rotate .out log files

1) If you are using nodemanager you can redirect .out to .log and control them via console (server -> logging tab under console)
traverse to */user_projects/domains/base_domain/bin/nodemanager
take a backup of wlscontrol.sh and edit it
change OutFile=$ServerDir/logs/$ServerName.out to OutFile=$ServerDir/logs/$ServerName.log

ELSE

2) You can make changes to the OS parameters under /etc/logrotate.conf
add the below lines at the end of the file

<Location of logs directory>/*.out
{
copytruncate
rotate 4
size=20Mb
}

ELSE

3)Add the flag  -Dweblogic.log.RedirectStdoutToServerLogEnabled=true to redirect .out to .log and control them via console (server -> logging tab under console)
and if you are starting via nodemanager add below additional parameters
nodemanager.properties under */wlserver_10.3/common/nodemanager
StartScriptName=startWebLogic.cmd
StartScriptEnabled=true

ELSE

4)Use OS command "logrotate" (via cron) to achieve your target. You can combine it with copytruncate to rotate logs.
http://linuxcommand.org/man_pages/logrotate8.html

certicom is unable to read cacerts

Error msg:
java.io.IOException: PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11
       at weblogic.server.channels.DynamicSSLListenThread.<init>(DynamicSSLListenThread.java:64)
       at weblogic.server.channels.DynamicListenThreadManager.createListener(DynamicListenThreadManager.java:289)
       at weblogic.server.channels.AdminPortService.bindListeners(AdminPortService.java:76)
       at weblogic.server.channels.EnableAdminListenersService.start(EnableAdminListenersService.java:39)
       at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
       at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
       at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.security.cert.CertificateParsingException: PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11
       at com.certicom.security.cert.internal.x509.X509V3CertImpl.<init>(Unknown Source)
       at com.certicom.tls.interfaceimpl.CertificateSupport.addTrustedCertificate(Unknown Source)
       at com.certicom.net.ssl.SSLContext.addTrustedCertificate(Unknown Source)
       at com.bea.sslplus.CerticomSSLContext.addTrustedCA(Unknown Source)
       at weblogic.security.utils.SSLContextWrapper.addTrustedCA(SSLContextWrapper.java:62)
       at weblogic.security.utils.SSLContextManager.createServerSSLContext(SSLContextManager.java:424)
       at weblogic.security.utils.SSLContextManager.getChannelSSLContext(SSLContextManager.java:336)
       at weblogic.security.utils.SSLContextManager.getSSLServerSocketFactory(SSLContextManager.java:91)
       at weblogic.server.channels.DynamicSSLListenThread.<init>(DynamicSSLListenThread.java:59)
       ... 6 more


After looking at the execption it seems that WLS 10.3 does not support cipher suite with Object ID 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
Please take a backup and replace the cacerts file located in JDK_HOME/jre/lib/security with the cacerts located file in WL_HOME/server/lib.

Looks like the 10.3.0.0 version of certicom is unable to read the cacerts of JDK 1.6 u29.

A version attribute is required

WLS 10.3.4 
error message:
<Warning> <Munger> <BEA-2156203> <A version attribute was not found in element webservices in the deployment descriptor in weblogic.utils.classloaders.ChangeAwareClassLoader@49e98c6d finder: weblogic.utils.classloaders.CodeGenClassFinder@65254b11 annotation: bea_wls9_async_response@bea_wls9_async_response.war/WEB-INF/webservices.xml. A version attribute is required, but this version of the Weblogic Server will assume that the JEE5 is used. Future versions of the Weblogic Server will reject descriptors that do not specify the JEE version.>

I applied the parameter -Dweblogic.wsee.skip.async.response=true in setdomainenv.sh
and error message disappeared. 
else you could always apply patch 11720907 or cumulative patch 12770299.

Timeout read connections from JDBC side on weblogic.

When a client requests data from the DB through a thread and there is no reply from the server the client indefinitely waits for the reply.
You can avoid this situation by timing out the thread from JDBC side. It will end up with a "Socket READ TIMEOUT" error so you don't indefinitely wait for a reply from server.Make sure the time period you set is strictly not shorter than any of the normal db queries of your application, else it will timeout normal connections.

oracle.net.READ_TIMEOUT
works for jdbc driver versions <=10.2
does NOT work for jdbc driver versions >= 11.1

oracle.jdbc.ReadTimeout
works for jdbc driver versions >= 10.1.0.5
Does not work for jdbc driver versions <10.1.0.5

You can set it in the JDBC_Data_*-jdbc.xml in the domain/config/JDBC folder. Timeout in miliseconds.

<jdbc-driver-params>
     <url>jdbc:oracle:thin:@localhost:1521:XE</url>
    <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name>
     <properties>
      <property>
        <name>user</name>
        <value>SYSTEM</value>
      </property>
      <property>
        <name>oracle.jdbc.ReadTimeout</name>
         <value>18000</value>
      </property>
     </properties>
    <password-encrypted>{3DES}s447QVesIco=</password-encrypted>
   </jdbc-driver-params>

also you can set it via console in

Admin Console -> JDBC Data Source: Configuration: Connection Pool, and set "Properties" as follows. Note to list each property=value pair on a separate line.

oracle.net.READ_TIMEOUT=30000 

oracle.jdbc.ReadTimeout=30000

To check the JDBC driver version:
 D:\work\wls\wlserver_10.3\server\lib>java -jar ojdbc6.jar
Oracle 11.1.0.7.0-Production JDBC 4.0 compiled with JDK6