EM12c – Webtier is unable to start 

I have been asked to review why the webtier is down and also unable to start as well and every time failed to start the webtier. If we can see the status of OMS as below the webtier is down and the OMS is up.

[oracle@bimpoda1 bin]$ emctl status oms
 Oracle Enterprise Manager Cloud Control 12c Release 5
 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
 WebTier is Down
 Oracle Management Server is Up
[oracle@bimpoda1 opmn]$ emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Node Manager Not Running
Oracle Management Server is Up
Admin Server Port is occupied by another process. Please stop process to free port.

Next reviewed various oms, opmn, ohs and etc found few error messages such as “Terminating Process: ohs1~OHS~OHS~1 (24:7307)” & as “apachectl hardstop: httpd (no pid file) not running”.

> opmn.log
[2017-02-06T11:00:29-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:01:30-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:01:30-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:02:31-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:02:31-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:03:32-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:03:32-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:04:33-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:04:33-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:05:34-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:05:34-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:06:35-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:06:35-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:07:36-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:07:36-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:08:37-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:08:37-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:09:38-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:09:38-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:10:39-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:10:39-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:11:40-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:11:40-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:12:41-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:12:41-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:13:42-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:13:42-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:14:43-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:14:43-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:15:44-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:15:44-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:16:45-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:16:45-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:17:46-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:17:46-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:18:47-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:18:47-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:19:48-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:19:48-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[oracle@bimpoda1 ohs1]$ vi console~OHS~1.log
[oracle@bimpoda1 ohs1]$ pwd
/u01/app/oracle/product/12.1.0.2/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1
[oracle@bimpoda1 ohs1]$
--------
17/02/06 11:15:44 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:16:45 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:17:46 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:18:47 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:19:48 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:20:49 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:21:50 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:22:51 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
[oracle@bimpoda1 diagnostics]$ cd ..
[oracle@bimpoda1 WebTierIH1]$ cd bin/
[oracle@bimpoda1 bin]$ ./opmnctl status
Processes in Instance: instance1
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
ohs1 | OHS | 7307 | Stop
[oracle@bimpoda1 bin]$ cd ../config/OPMN/opmn/states/
[oracle@bimpoda1 states]$ ls -ltr
total 4
-rw-r----- 1 oracle oinstall 558 Feb 6 10:59 p24
[oracle@bimpoda1 states]$
opmn.log
[2017-02-06T11:00:29-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:01:30-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:01:30-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:02:31-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:02:31-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:03:32-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:03:32-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:04:33-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:04:33-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:05:34-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:05:34-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:06:35-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:06:35-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:07:36-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:07:36-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:08:37-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:08:37-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:09:38-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:09:38-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:10:39-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:10:39-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:11:40-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:11:40-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:12:41-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:12:41-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:13:42-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:13:42-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:14:43-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:14:43-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:15:44-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:15:44-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:16:45-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:16:45-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:17:46-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:17:46-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:18:47-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:18:47-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:19:48-05:00] [opmn] [NOTIFICATION:1] [663] [pm-process] Stopping Process: ohs1~OHS~OHS~1 (24:7307)
[2017-02-06T11:19:48-05:00] [opmn] [ERROR:1] [] [libopmnohs] Forcefully Terminating Process: ohs1~OHS~OHS~1 (24:7307)
[oracle@bimpoda1 ohs1]$ vi console~OHS~1.log
[oracle@bimpoda1 ohs1]$ pwd
/u01/app/oracle/product/12.1.0.2/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1
[oracle@bimpoda1 ohs1]$
--------
17/02/06 11:15:44 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:16:45 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:17:46 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:18:47 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:19:48 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:20:49 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:21:50 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
17/02/06 11:22:51 Stop process
--------
/u01/app/oracle/product/12.1.0.2/oms/Oracle_WT/ohs/bin/apachectl hardstop: httpd (no pid file) not running
[oracle@bimpoda1 diagnostics]$ cd ..
[oracle@bimpoda1 WebTierIH1]$ cd bin/
[oracle@bimpoda1 bin]$ ./opmnctl status
Processes in Instance: instance1
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
ohs1 | OHS | 7307 | Stop
[oracle@bimpoda1 bin]$ cd ../config/OPMN/opmn/states/
[oracle@bimpoda1 states]$ ls -ltr
total 4
-rw-r----- 1 oracle oinstall 558 Feb 6 10:59 p24
[oracle@bimpoda1 states]$

 

Finally, i’ve seen few processes are running even though all the services stopped. Below are the sessions which i used grep to check for the weblogic processes and i can see that the sessions are running from few days.

oracle 44848 44778 12 10:57 ? 00:03:35 /u01/app/oracle/product/12.1.0.2/oms/jdk16/jdk/bin/java -server -Xms256M -Xmx1740M -XX:PermSize=128M -XX:MaxPermSize=768M -XX:CompileThreshold=8000 -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -Dweblogic.Name=EMGC_OMS1 -Djava.security.policy=/u01/app/oracle/product/12.1.0.2/oms/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.system.BootIdentityFile=/u01/app/oracle/product/12.1.0.2/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.ReverseDNSAllowed=false -DINSTANCE_HOME=/u01/app/oracle/product/12.1.0.2/gc_inst/em/EMGC_OMS1 -DORACLE_HOME=/u01/app/oracle/product/12.1.0.2/oms/oms -Ddomain.home=/u01/app/oracle/product/12.1.0.2/gc_inst/user_projects/domains/GCDomain -Djava.awt.headless=true -Ddomain.name=GCDomain -Doracle.sysman.util.logging.mode=dual_mode -Djbo.doconnectionpooling=true -Djbo.txn.disconnect_level=1 -Docm.repeater.home=/u01/app/oracle/product/12.1.0.2/oms/oms -Djbo.ampool.minavailablesize=1 -Djbo.ampool.timetolive=-1 -Djbo.load.components.lazily=true -Djbo.max.cursors=5 -Djbo.recyclethreshold=50 -Djbo.ampool.maxavailablesize=50 -Djava.security.egd=file:///dev/./urandom -Dweblogic.debug.DebugWebAppSecurity=true -Dweblogic.SSL.LoginTimeoutMillis=300000 -Djps.auth.debug=true -Djps.authz=ACC -Djps.combiner.optimize.lazyeval=true -Djps.combiner.optimize=true -Djps.policystore.hybrid.mode=false -Djps.subject.cache.key=5 -Djps.subject.cache.ttl=600000 -Doracle.apm.home=/u01/app/oracle/product/12.1.0.2/oms/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -da -Dplatform.home=/u01/app/oracle/product/12.1.0.2/oms/wlserver_10.3 -Dwls.home=/u01/app/oracle/product/12.1.0.2/oms/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/product/12.1.0.2/oms/wlserver_10.3/server -Dcommon.components.home=/u01/app/oracle/product/12.1.0.2/oms/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/app/oracle/product/12.1.0.2/gc_inst/user_projects/domains/GCDomain -Djrockit.optfile=/u01/app/oracle/product/12.1.0.2/oms/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/app/oracle/product/12.1.0.2/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/servers/EMGC_OMS1 -Doracle.domain.config.dir=/u01/app/oracle/product/12.1.0.2/gc_inst/user_projects/domains/GCDomain/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/app/oracle/product/12.1.0.2/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/app/oracle/product/12.1.0.2/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/app/oracle/product/12.1.0.2/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/app/oracle/product/12.1.0.2/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/app/oracle/product/12.1.0.2/oms/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/product/12.1.0.2/oms/oracle_common/modules/oracle.oamprovider_11.1.1,/u01/app/oracle/product/12.1.0.2/oms/oracle_common/modules/oracle.jps_11.1.1,/u01/app/oracle/product/12.1.0.2/oms/oms/sysman/jlib -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.jdbc.remoteEnabled=false -Doracle.apm.home=/u01/app/oracle/product/12.1.0.2/oms/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -Dweblogic.management.discover=false -Dweblogic.management.server=https://bimpoda1.oracle-ckpt.com:7101 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/product/12.1.0.2/oms/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server
oracle 79286 78703 5 11:14 pts/5 00:00:39 /u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/jdk/bin/java -Xmx166M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -Dwatchdog.pid=78703 -cp /u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/jdbc/lib/ojdbc5.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/ucp/lib/ucp.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/modules/oracle.http_client_11.1.1.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/lib/xmlparserv2.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/lib/jsch.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/lib/optic.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/modules/oracle.dms_11.1.1/dms.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/modules/oracle.odl_11.1.1/ojdl.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/modules/oracle.odl_11.1.1/ojdl2.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/sysman/jlib/log4j-core.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/jlib/gcagent_core.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/sysman/jlib/emagentSDK-intg.jar:/u01/app/oracle/product/12.1.0.2/agent/core/12.1.0.5.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain
[oracle@bimpoda1 states]$ kill -9 44848
[oracle@bimpoda1 states]$ kill -9 43267

Then i’ve considered to kill all the sessions related to weblogic, then started the OMS and i can able to startup even the webtier.

[oracle@bimpoda1 opmn]$ emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
WebTier is Up
Oracle Management Server is Up
[oracle@bimpoda1 opmn]$

Conclusion: The old processes which are already running with weblogic sessions and those created lock and hence other processes unable to startup, after killing them then the webtier is able to started.