Apache Related Common Issues in R12 and 11i.

Apache Related Common Issues in R12 and 11i.


Apache Common Issues
General Information to Troubleshoot Apache Issues.
1)If you mistype any directive in Apache (httpd.conf or any conf file), Apache will not start & it will not write even in error_log. In these case it will report any error message to file
$COMMON_TOP/admin/log/$CONTEXT_NAME/adapcctl.txt

So first check in the above file

2)Found *.pid file in Apache_Top/Apache/logs/ even after bounce of an Apache web server (When Apache starts it creates .pid file in logs directory and when you shutdown apache it cleans this .pid file). So if this file is already there then Apache will not start.
Fix: Simply move this file





3)Could not bind port (Port already in Use) or Sometimes apache and concurrent Manager will interchange their port no.’s while starting results error and both of them will not up
Fix: To overcome this up the services in this manner
 

1)Listener
2)Apache web server
3)Concurrent manager
4)adstrtal.sh

4)Error 500 – Internal server error
Fix: 1) Clear Cache (Application Server Cache)and start the services
5)If there are no error messages in error_log or error_log_pls
then there might be issues in starting Jserv component of web server (which caters Java requests by mod_jserv)

6)If you don’t know port number of your web server you can check it in file $IAS_ORACLE_HOME/Apache/Apache/conf/httpd.conf with directive Listen & Port

7)Once you are able to access http://hostname.domainname:port and if you are getting any error message that means your web server (Core Apache) is working but there are various components called as modules and there may be issues with those modules which we will check these modules . few modules of our interest are
-mod_jserv
-mod_pls
-mod_oprocmgr
-mod_ossl (If you have ssl enabled on web server)
-mod_osso (If single sign on server is enabled on web server)

8)Once in a while adapcctl.sh start will not work then ultimate you have to go and invoke apachectl script file in Apache top/apache/bin to start Apache webserver.
09)Error : Your session is no longer valid while trying to loging to applications.
The issue is resolved by changing the profile option session_cookie_domain to null from domain.com

10)HTTP-403 forbidden message after entering the username and password in the “Apps Logon Links” page. [Not authorised to view this page]
Check for Apps password in $APACHE_TOP/modplsql/cfg/wdbsvr.app ,Comment out custom_auth entry in $APACHE_TOP/modplsql/cfg/wdbsvr.app and bounce apache.

11)Error message: You have insufficient privileges for the current operation.
Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.
Example :
update icx_parameters set SESSION_COOKIE_DOMAIN=’NEW_VALUE’ where SESSION_COOKIE_DOMAIN=’OLD_VALUE’
12)AppsLocalLogin?.jsp page error outs….

• Edit the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties
• Set wrapper.bin.parameters=-DLONG_RUNNING_JVM=false
• add the line wrapper.bin.parameters=-DCACHEMODE=LOCAL
• Restarted Apache for these changes to take effect and retested the issue

13) Login to the applications URL fails with
500 Internal server Error
java.lang.NoClassDefFoundError at
oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:126) at
oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:170) at com.evermind[Oracle
Containers for J2EE 10g (10.1.3.0.0)
].server.http.HttpApplication.loadServlet(HttpApplication.java:2231) at com.evermind[Oracle
Containers for J2EE 10g (10.1.3.0.0)
].server.http.HttpApplication.findServlet(HttpApplication.java:4617) at com.evermind[Oracle
Containers for J2EE 10g (10.1.3.0.0)
].server.http.HttpApplication.findServlet(HttpApplication.java:4541) at com.evermind[Oracle
Containers for J2EE 10g (10.1.3.0.0)
Performed the below steps, but it did not resolved the issue
1) Bounced Apache server, cleared cache and validated ports
2) Bounced all Middle tier services and killed defunct processes
3) Bounced Unix Server

Solution
=======
Java object Caching Configuration parameter was Enabled in oc4j.properties
$INST_TOP/ora/10.1.3/j2ee/oacore/config
Disabled LONG_RUNNING_JVM=false from True
Bounced Apache server and could login to the applications URL without issues
                           ISSUES AND SOLUTIONS
ISSUE 1:
Description
When Apache script adapcctl.sh is issued.It shows that Apache is started.On checking the status Apache is stopped
Cause
The error log file in the folder
cd $APACHE_TOP/Apache/logs
Showed
fopen: No such file or directory httpd: could not log pid to file /var/opt/_/Apache/logs/httpd_pls.pid
SOLUTION 1
Created the missing directories /_/Apache/logs under /var/opt folder
Bounced Apache
**********************************************************************
ISSUE 2:
Description
oacore was not comming up in the configurator node.
opmnctl status
ias-component process-type pid status
OC4J oacore N/A Down
OC4J oacore N/A Down
OC4J oacore N/A Down
OC4J oacore N/A Down
Cause
Check the oacore error log.
Example:
Below Log file gives root cause. //mtlog/_/logs/ora/10.1.3/opmn/oacore_default_group_2/oacorestd.err
14:11:56 Error initializing server: //inst/apps/_/ora/10.1.3/j2ee/oacore/config/server.xml,
Fatal error at line 70 offset 1 in file://inst/apps/_/ora/10.1.3/j2ee/oacore/config/server.xml: .:
XML-20100: (Fatal Error) Expected ‘EOF’.
SOLUTION 2
Edit the /sid/inst/apps/SID_hostname/ora/10.1.3/j2ee/oacore/configserver.xml file and remove the extra line in the file. Restart and check oacore will be starting. This sudden change in server.xml seems to be because of a bug.
Bug – 6702510
************************************************************************
ISSUE 3
AppsLocalLogin?.jsp page was displaying errors
Cause
Extract from the Jserv log file =========================== [27/02/2008 16:57:14:447 CST] weboamLocal/oracle.apps.fnd.oam. servlet.ui.OAMServlet:
Sucessfully initialize oaosu.OAMServlet [27/02/2008 16:57:14:447 CST] Servlet Zone rootauohstrzt11 initia lization complete [27/02/2008 16:57:15:101 CST]
Creating noun for: oracle.jsp.JspServlet [27/02/2008 16:57:15:102 CST] Creating noun for: AppsLocalLogin?.jsp
[27/02/2008 16:57:15:120 CST] oracle.jsp.JspServlet: init [27/02/2008 16:57:28:066 CST] Creating noun for: oracle.jsp.JspServlet
[27/02/2008 16:57:28:067 CST] Creating noun for: AppsLocalLogin?.jsp [27/02/2008 16:57:28:078 CST] oracle.jsp.JspServlet: init
[27/02/2008 16:57:29:753 CST] JspServlet?: unable to dispatch to requested page: Exception:java.lang.NoClassDefFoundError
SOLUTION 3:
Edited the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties
a) Set wrapper.bin.parameters=-DLONG_RUNNING_JVM=false
b) Add the line wrapper.bin.parameters=-DCACHEMODE=LOCAL
c) Restarted Apache for these changes to take effect and retested the issue
********************************************************************
ISSUE 4
New Message when starting Apache using adapcctl.sh start
Cause
It takes 7/10 minutes to clear this cache directory if present
apache Web Server Listener is not running.
Starting Apache Web Server Listener (dedicated HTTP) …
Removing gantt cache directory
/ppomwi/applmgr/r12/apps/apps_st/comn/webapps/oacore/html/cabo/images/cache
Successfully removed the gantt cache directory:/tcge1i/applmgr/common/html/cabo/images/cache/gantt
Apache Web Server Listener (PLSQL) is not running.
Starting Apache Web Server Listener (dedicated PLSQL) …
SOLUTION 4
$COMMON_TOP/webapps/oacore/html/cabo/images/cache
Wait for atleast 7 to 10 minutes and this cache directory will be cleared and Apache will be started successfully
***********************************************************************
ISSUE 5
Apache server is up and running.
But adapcctl.sh status shows
adapcctl.sh version 115.54
Apache Web Server Listener is not running. Apache Web Server Listener (PLSQL) is not running.
adapcctl.sh: exiting with status 1

Cause
Incorrect Lock file path given in httpd.conf
LockFile? $APACHE_TOP/Apache/log/PPOMWI_httpd.lock
and Incorrect path given in adapcctl.sh script.
SOLUTION 5:
Modified the path of lock file in $APACHE_TOP/Apache/conf/httpd.conf file.
LockFile? /var/opt/SID_/httpd.lock
and Modified the path of adapcctl.sh file by commenting the wrong path
************************************************************************
ISSUE 6

After selecting the E-Business home page got internal server error Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.

Cause
Found that all the properties files as below
jserv.properties jserv_restrict.properties zone.properties zone_restrict.properties
were pointing to a dbc file name _.dbc
which was not physically present at the path
/sid/applmgr/11510/fnd/11.5.0/secure/_
$ cd //applmgr/11510/fnd/11.5.0/secure/_
$ ls .dbc
SOLUTION 6:
copied the file PROD.dbc
which was physically present at path
//applmgr/11510/fnd/11.5.0/secure/_
to the dbc file name which was being pointed by all the properties files
This resolved the issue.
**********************************************************************
ISSUE 7
Instance is very slow / inaccessible with a warning on login page:
Warning
Low-level logging is currently enabled. Your application will not perform as well while Low-level logging is on.
Cause
Logging was enabled through profile values:
FND: Debug Log Enabled — Yes
FND: Debug Log Level — Statement
FND: Debug Log Module — %
SLA: Enable Diagnostics — Yes
FND Validation Level — Error
SOLUTION 7
1. Change these values to:
FND: Debug Log Enabled — No
FND: Debug Log Level — ( Blank )
FND: Debug Log Module — ( Blank )
SLA: Enable Diagnostics — No
FND Validation Level — None
2. Bounce Apache on all the MTs and clear Apache cache.
*********************************************************************
ISSUE 8
Login to the applications URL fails with
500 Internal Server Error java.lang.NoClassDefFoundError at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect?.java:126) at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect?.java:170) at com.evermind[Oracle Containers for J2EE? 10g (10.1.3.0.0) ].server.http.HttpApplication.loadServlet(HttpApplication?.java:2231) at com.evermind
Performed the below steps, but it did not resolved the issue
1) Bounced Apache server, cleared cache and validated ports
2) Bounced all Middle tier services and killed defunct processes
3) Bounced Unix Server

Cause
No Cause Entered
SOLUTION 8
Java object Caching Configuration parameter was Enabled in oc4j.properties $INST_TOP/ora/10.1.3/j2ee/oacore/config
Disabled LONG_RUNNING_JVM=false from True
Bounced Apache server and could login to the applications URL without issues
Java Caching is a mechanism used to improve performance by storing data in the Middle Tier JVM memory, instead of retrieving the data from the database every time it is requested.
***********************************************************************
ISSUE 9
When trying to access the URL (Either Big IP or the node level), it hangs for a long time and getting redirected to a different URL and gives a blank page. For example, When try to access the ipayments URL which is hosted on node2, it hangs for a long time and its getting redirected towards the Sales URL which is hosted on node3, and displays a blank page.

Cause
All the host reference and the URL directives are pointing towards node3 and sales URL respectively, in the below files. Hence every attempt tries to launch the sales URL and get fails as the apache configuration files and parameters are pointing towards the correct one.
jserv.conf forms.properties zone.properties jserv.properties
SOLUTION 9
Stop Apache. Edit the files jserv.conf, forms.properties, zone.properties, jserv.properties, jserv.conf to replace the host reference and the URL directives to the correct one (In our case, its node3 . Clear the cache and restart the Apache.
ISSUE 10
Apache login issues occurs in multi MT environment. Unable to login to oracle applications.

Cause
Issue due to distributed cache.
SOLUTION 10
Set is_distributed to false in javacache.xml Bounce Apache.
ISSUE 11
Apache server is down. Apache logfile did not provide useful information.
Started the Apache server, but the Apache Web listener did not started.
1) Verified no log files (Apache,Jserv) crossed 2GB limit. error_log, access.log, mod_jserv
2) Veified that the port is listening (netstat -an |grep ) Apache port(httpd.conf), pls listener(httpd_pls.conf)
3) Verified that the Apache and Jserv configuration files are not changed recently httpd.conf & Jserv.conf
4) Shutdown all the Middle tier services and check for ipcs (semaphores) did not resolve the outage.
$ ipcs on application middle tier as Application user ap

Cause
No Cause Entered
SOLUTION 11
Rebooted the server and started the services. All services came back normal.
************************************************************************
ISSUE 12
Unable to login to the home page URL. Login fails with the error:
Error Page You have encountered an unexpected error. Please contact the system administrator for assistance.
Click here for exception details
Checks:
Upon clicking the link, can observe the error:
oracle.apps.fnd.framework.OAException: Application: FND, Message Name: SQL_PL SQL_ERROR. Tokens: ROUTINE = createSession(int)(userId=6,sessionMode=’null’,’2D8 FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412?’); REASON = java.s ql.SQLException: ORA-01422: exact fetch returns more than requested number of ro ws ORA-06512: at “APPS.FND_SESSION_MANAGEMENT”, line 295 ORA-06512: at “APPS.FND_SESSION_MANAGEMENT”, line 455 ORA-06512: at line 1 ; ERRNO = 1422;
Cause
FND_NODES table was updated with the same server_id for all Nodes.
SOLUTION 12
Update the table FND_NODES with correct server_ids for all nodes as in $FND_TOP/secure/sid_host/sid.dbf
Bounced apache on all apache nodes.
and Run select node_name, server_id from fnd_nodes;
to confirm the values.
Below is the example for the solution:
Found three hosts using same server_id:
SQL> select node_name, server_id from fnd_nodes; NODE_NAME SERVER_ID
AUOHSACPS06 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412
AUOHSACPS03 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412
AUOHSACPS02 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412
reset server_id with original server_id from each hosts .dbc file @ $FND_TOP/secure:
SQL> select node_name, server_id from fnd_nodes; NODE_NAME SERVER_ID
AUOHSACPS06 2D8FF21843830F82E040558CDBC6195C27690802237516122831445560236127
AUOHSACPS03 2D90005D5A13C774E040558CDBC61D9436546148971863330582284123371818
AUOHSACPS02 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412
************************************************************************
ISSUE 13
Users getting ‘404 page not found errors’ when tyring to access the URL
(OR)
Getting Exception in thread “main” java.lang.ClassFormatError: Truncated class file
(OR)
adstrtal.sh or adstpall.sh is throwing below error message
You are running adstrtal.sh version 115.16
Exception in thread “main” java.lang.ClassFormatError: oracle/apps/ad/context/AppsContext (Truncated class file) at java.lang.ClassLoader.defineClass0(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader?.java:539) at java.security.SecureClassLoader.defineClass(SecureClassLoader?.java:123) at java.net.URLClassLoader.defineClass(URLClassLoader?.java:251) at java.net.URLClassLoader.access$100(URLClassLoader?.java:55) at java.net.URLClassLoader$1.run(URLClassLoader?.java:194) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader?.java:187) at java.lang.ClassLoader.loadClass(ClassLoader?.java:289) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:274) at java.lang.ClassLoader.loadClass(ClassLoader?.java:235) at java.lang.ClassLoader.loadClassInternal(ClassLoader?.java:302) at oracle.apps.ad.autoconfig.ServiceControl.(ServiceControl?.java:319) at oracle.apps.ad.autoconfig.ServiceControl.main(ServiceControl?.java:662) Check logfile //applmgr/common/admin/log/_/08061104.log for details
Exiting with status 1
(OR)
[06/08/2007 11:03:12:181] (ERROR) an error returned handling request via protocol “ajpv12” [06/08/2007 11:03:12:181] (EMERGENCY) ajp12: can not connect to host 140.85.188.36:20021 [06/08/2007 11:03:13:184] (EMERGENCY) ajp12: can not connect to host 140.85.188.36:20021 [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth did not receive challenge size [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth fail
Cause
Extract from the JVM log file: //product/iAS/Apache/Jserv/logs/jvm/OACoreGroup.0.stderr
Exception in thread “main” java.lang.ClassFormatError: Truncated class file at org.apache.jserv.JServServletManager.load_init(JServServletManager?.java:765) at org.apache.jserv.JServServletManager.loadServlet(JServServletManager?.java:659) at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager?.java:789) at org.apache.jserv.JServServletManager.init(JServServletManager?.java:447) at org.apache.jserv.JServ.start(JServ.java:625)
Verified that there are no log files that exceed 2GB limit (OS limit).
Clearing the Apache and Jserv cache did not resolve the issue. Unmount / mount product specific file system did not resolve
SOLUTION 13:
Try each of these solutions until the problem is resolved, in order to have the least impact on the system.
1. If you can determine the specific class file from the log files, use the ‘touch’ command.
e.g. touch classfile
If necessary, restart Apache and clear server cache.
2. Shut down all MT services, exit the Apps owner and ask the System Admin to unmount and remount the /sid/applmgr file system (mount point above $JAVA_TOP). Ask the System Admin to kill any processes that may be holding the file system open. Then log in to the Apps owner and restart MT services.
3. Shutdown all the MT services and ask System Admin to reboot the server if the actions above don’t resolve the outage.
************************************************************************
ISSUE 14:
Got login error – Internal Server Error after bounced all services on . find following message from Jserv log file:
$ tail -8 mod_jserv.log
[07/05/2007 18:41:06:915] (ERROR) ajp12: Servlet Error: java.lang.NoClassDefFoundError: null
[07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol “ajpv12”
[07/05/2007 18:41:06:916] (ERROR) balance: 11906 internal servlet error in server auohsnops03.oracleoutsourcing.com:20720
[07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol “balance”
[07/05/2007 20:07:52:732] (ERROR) ajp12: Servlet Error: java.lang.NoClassDefFoundError: null
No runaway, no lock on port 20720.

SOLUTION 14:
Shutdown all services on MT. Removed all log file (standard error…) from Jserv/jvm. Started All services with successful login.
************************************************************************
ISSUE 15
Customer reported Not able to login with “Internal Server Error” on .
We could not start some of services during bounced All Services with following error message: adapcctl.sh version 115.47
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Apache Web Server Listener is not running (dedicated HTTP) …
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Apache Web Server Listener is already stopped.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
adapcctl.sh: exiting with status 2
Unable to login the application URL, facing the error while login:
Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, ap@domain.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
Cause:
SQL> select END_DATE from fnd_user where user_name=’GUEST’;
END_DATE
———
05-OCT-10
SOLUTION 15
We found out the apps and applsys were not in sync.
Just manual change the apps password from sqlplus.
Restarted all services and login successfully.
or
Update the FND_USER table with END_DATE=null and bounced apache
SQL> update fnd_user set END_DATE=null where user_name=’GUEST’;
1 row updated.
SQL> commit;
Commit complete.
SQL> select END_DATE from fnd_user where user_name=’GUEST’;
END_DATE
———–
************************************************************************



ISSUE 16
Unable to login to oracle applications:
error message: You have insufficient privileges for the current operation.
Checks
Unable to login to oracle applications:
error message: You have insufficient privileges for the current operation.
SOLUTION 16
Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.
Example :
update icx_parameters set SESSION_COOKIE_DOMAIN=’NEW_VALUE’ where SESSION_COOKIE_DOMAIN=’OLD_VALUE’;
************************************************************************
ISSUE 17
Apache home page is not coming up even after bounced
Found in jvm log asking for java version
SOLUTION 17
//product/iAS/Apache/Apache/bin/java.sh: /usr/java/jdk1.3.1_15/bin/java: not found
Check the version in /usr/java and update that version in java.sh file and Bounced Apache.
************************************************************************
ISSUE 18
Unable to access applications
cause
All applications are up and running.
“Solaris Error: 12: Not enough space” errors are encounterd in database listener logfile.
SOLUTION 18:
Check if the OS is SUN Solaris.
Check if /tmp filesystem is 100% full.
Check Swap space utilization.
Check the application process that is hagging resources. (Check if any sql sessions are hagging resources)
Clear the process in issue after getting required approval.
NOTE:
The above is applicable only for environments on Solaris OS.
************************************************************************
ISSUE 19
Not able to login to APPS. JSP Error:
Unable to create an user session or ORA-01403 : no data found has been detected in FND.SIGNON.AUDIT_USER
Checks
Check number of rows in dual table.
Select * from dual;
DUAL table should always contain only one row.
The table named DUAL is a small table in the data dictionary that Oracle and user-written programs can reference to guarantee a known result.
This table has one column called DUMMY and one row containing the value X.
SOLUTION 19
DUAL table should always contain only one row and one column.
Insert a row with value X into dual table if the dual table contains 0 rows. If the dual table contains more than one row then delete the rows by keeping one row.
The rows in dual table not equal to zero.
************************************************************************
ISSUE 20
Signature for “https URL redirecting to http URL”
Issue:
https URL redirecting to http URL and the page cannot be displayed error occurs.
SOLUTION 20
Check the httpd.conf and url_fw.conf, in url_fw.conf the Rewrite rule was
RewriteRule ^/$/OA_HTML/AppsLocalLogin.jsp.
Changed into
RewriteRule https://: /OA_HTML/AppsLocalLogin.jsp.