Oracle Application Technology

Today support team sought a help for the issue they were encountering in the TEST instance.

Error as given by the support team

ErrBufORA-20100: File o7725967.tmp creation for FND_FILE failed.
You will find more information on the cause of the error in request log.

I tried a quick test to see if there is any issue with utl_file_dir

SQL>exec FND_FILE.PUT_LINE(FND_FILE.LOG, 'THIS IS A TEST');
BEGIN FND_FILE.PUT_LINE(FND_FILE.LOG, 'THIS IS A TEST'); END;

*
ERROR at line 1:
ORA-20100: File o7725970.tmp creation for FND_FILE failed.
You will find more information on the cause of the error in request log.
ORA-06512: at "APPS.FND_FILE", line 410
ORA-06512: at "APPS.FND_FILE", line 517
ORA-06512: at line 1

Above test threw an error. Then I verified 2 more things

1) APPLPTMP directory should be the first one appearing when UTL_FILE_DIR parameter is queried – this looked Ok

SQL>show parameter utl_file

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
utl_file_dir string /d05/apptest/common/temp, /usr
/tmp, /d12/oratest/product/9.2.
0/appsutil/outbound/TEST, /d05/
apptest/11.5/xxbb/4.0.0/edi/ou
t, /d05/appltest/11.5/xxdd/4.0.
0/US/utl_file_dir, /d05/appltest
t/common/admin/log/TEST_host_name
, /d05/appltest/common/admi
n/out/TEST_host_name, /d05/ap
pltest/common/admin/log/TEST_host_name,
/d12/oratest/product/
9.2.0/ccr/hosts/host_name/st
ate, /d05/appltest/11.5/xxcal/s
mart, /d05/appltest/11.5/xxcal/
bartender

2)Missing permissions on APPLPTMP directory – I found that the permission was not correct. I gave the permission (777) to the directory and this resolved the issue.

To confirm, I ran the same procedure again and it returned the expected result.

SQL>exec FND_FILE.PUT_LINE(FND_FILE.LOG, 'THIS IS A TEST');

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.15

Other probable causes of error

3)APPLPTMP directory is missing &

4)APPLPTMP directory pointing to wrong location

Hope this helps.

Recently I got a request from a bank where the customer transmits some of the files. Bank was upgrading and changing their target server and hence we need to accommodate certain changes on our side to do the testing before the same is done in production environment.

Command to list the existing Key IDs on the keyring

appluser.DEV.pdc-xyz-ibm> /usr/local/bin/pgp -kv
Pretty Good Privacy(tm) Version 6.5.8
(c) 1999 Network Associates Inc.
Uses the RSAREF(tm) Toolkit, which is copyright RSA Data Security, Inc.
Export of this software may be restricted by the U.S. government.
Type bits      keyID      Date       User ID
RSA  1024      ********** 2009/07/02 *** DEFAULT SIGNING KEY ***
                                     XY_ABCBANK_TEST
RSA  1024      ********** 2009/07/06 abc pqr <a href="mailto:abc.pqr@domain.com">abc.pqr@domain.com</a>>
RSA  2048/2048 ********** 2012/02/21 expires 2014/05/01
                                      File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
RSA  2048/2048 ********** 2009/03/04 *** KEY EXPIRED ***
                                      ABCBANK_ECS_QA_2011 <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
4 matching keys found.

Bank sends the new public key from their TEST/QA server. This is done so as to do point to point testing from our DEV/QA to Bank DEV/QA server.

Once the new public key is recived,this needs to be added into the keyring using below command

/usr/local/bin/pgp -ka <BANK_PUBLIC_KEY.ASC>
Pretty Good Privacy(tm) Version 6.5.8
(c) 1999 Network Associates Inc.
Uses the RSAREF(tm) Toolkit, which is copyright RSA Data Security, Inc.
Export of this software may be restricted by the U.S. government.
Looking for new keys...
RSA  2048/2048 ************ 2012/02/21 File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
sig?           ************             (Unknown signator, can't be checked)

keyfile contains 1 new keys. Add these keys to keyring ? (Y/n) Y
New userid: "File_Transfer_Services <a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>".
New signature from keyID ********* on userid File_Transfer_Services <a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>

Keyfile contains:
   1 new key(s)
   1 new signatures(s)
   1 new user ID(s)
Summary of changes :

New userid: "File_Transfer_Services <a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>".
New signature from keyID ********** on userid File_Transfer_Services <a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>

Added :
   1 new key(s)
   1 new signatures(s)
   1 new user ID(s)

Comand to extract the key for the user XY_ABCBANK_TEST (copies ID’s key to keyfile from keyring )

XY_ABCBANK_TEST is the user ID signing the encrypted file

/usr/local/bin/pgp -kx XY_ABCBANK_TEST /tmp/pgp_key_XY_Nov2013.asc

Getting a warning message while encrypting and signing the file

Key for user ID: File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
2048-bit RSA key, Key ID ******, created 2012/02/21, expires 2014/05/01
WARNING:  Because this public key is not certified with a trusted
signature, it is not known with high confidence that this public key
actually belongs to: "File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>".

Now to overcome the error, we need to sign the public key received from 3rd party using our secret key on key ring

pgp -ks <HER_USERID>-u <YOUR_USERID>keyring

appluser.DEV.pdc-xyz-ibm> pgp -ks File_Transfer_Services -u XY_ABCBANK_TEST_2013
Pretty Good Privacy(tm) Version 6.5.8
(c) 1999 Network Associates Inc.
Uses the RSAREF(tm) Toolkit, which is copyright RSA Data Security, Inc.
Export of this software may be restricted by the U.S. government.
Key for user ID: File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
2048-bit RSA key, Key ID ***********, created 2012/02/21, expires 2014/05/01
          Key fingerprint =  XY PQ 86 14 PO E9 9F 11  9Q E7 2G 80 69 75 U8 37  F6 8S 7C C3
READ CAREFULLY:  Based on your own direct first-hand knowledge, are
you absolutely certain that you are prepared to solemnly certify that
the above public key actually belongs to the user specified by the
above user ID (y/N)? y

You need a pass phrase to unlock your secret key.
Key for user ID "XY_ABCBANK_TEST_2013"
Key does not have a passphrase.

Attach a regular expression to this signature, or
press enter for none:
appluser.DEV.pdc-xyz-ibm>

Reference URL for other various PGP command summary

http://www.dsj.net/pgp/pgphelp.html

http://www.cs.wm.edu/pgp.html

-Anand

Today our development team complained of ‘Development’ instance of Oracle 11 Ebiz is not accessible. Even the login page is not loading.

I tried to login and got the login page but later on after keying the login credential, it was taking extremely long time to open and later on threw ‘page not found’ error. I thought of bouncing apache.

When I tried to bounce apache, I got below error.

ORACLE error 604 in afpodbinit

Cause:  afpodbinit failed due to ORA-00604: error occurred at recursive SQL level 1
ORA-01013: user requested cancel of current operation
ORA-06508: PL/SQL: could not find program unit being called
ORA-06512: at "APPS.FND_GLOBAL", line 101
ORA-06512: at "APPS.FND_GLOBAL",.

The SQL statement being executed at the time of the error was: begin fnd_global.bless_next_init('FND_PERMIT_0000'); fnd_global.initialize(:session_id, :user_id, :resp_id, :resp_appl_id, :security_group_id, :site_id, :login_id, :conc_login_id, :prog_appl_id, :conc_program_id, :conc_request_id, :conc_priority_request, :form_id, :form_application_id, :conc_process_id, :conc_queue_id, :queue_appl_id, :server_id); fnd_profile.put('ORG_ID', :org_id); fnd_profile.put('MFG_ORGANIZATION_ID', :mfg_org_id); fnd_profile.put('MFG_CHART_OF_ACCOUNTS_ID', :coa); fnd_profile.put('APPS_MAINTENANCE_MODE', :amm); end; and was executed from the file &ERRFILE.

Apache Web Server Listener :httpd ( pid 17197 ) is running.
Stopping Apache Web Server Listener (dedicated HTTP) ...

Oracle error -6508: ORA-01013: user requested cancel of current operation
ORA-06508: PL/SQL: could not find program unit being called
has been detected in FND_GLOBAL.INITIALIZE.

Oracle error -6508: ORA-01013: user requested cancel of current operation
ORA-06508: PL/SQL: could not find program unit being called
has been detected in FND_GLOBAL.INITIALIZE.
Apache Web Server Listener (PLSQL) :httpd ( pid 17521 ) is running.
Stopping Apache Web Server Listener (dedicated PLSQL) ...
Oracle error -6508: ORA-01013: user requested cancel of current operation
ORA-06508: PL/SQL: could not find program unit being called
has been detected in FND_GLOBAL.INITIALIZE.

adapcctl.sh: exiting with status 0

Later on when I tried to run adstpall.sh on Web tier, I got following in the error.

Setting service Oracle Apache Server DEV_pdc-al-lnx12 to mode 3
Executing service control script:
/d01/wfmdev/common/admin/scripts/DEV_pdc-al-lnx12/adapcctl.sh stop
Timeout specified in context file: 100 second(s)

script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception

adapcctl.sh version 115.50

Later on when I looked into the database, I got following latches/library pin waits and hence decided to go for database bounce.

    1    71 library cache lock            JDBC Thin Client     APPS        ############ 5.0440E+17        302   1117
    1    87 library cache lock               ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        302   1084
                                          (TNS V1-V3)

    1    88 library cache lock                                 APPS        ############ 5.0440E+17        302    817
    1   104 library cache lock            JDBC Thin Client     APPS        ############ 5.0440E+17        302    520
    1   132 library cache lock                                 APPS        ############ 5.0440E+17        302   1169
    1   150 library cache lock               ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        302    817
                                          (TNS V1-V3)

    1   164 library cache lock            JDBC Thin Client     APPS        ############ 5.0440E+17        302   1117
    1   233 library cache lock               ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        302    189
                                          (TNS V1-V3)

    1   142 library cache pin                ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        302    219
                                          (TNS V1-V3)

    1   171 library cache pin                ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        202     11

DB bounce helped to resolve the issue. ‘adstpall.sh’ and ‘adstrtal.sh’ executed successfully. Application was very much accessible.

-Anand

Over the weekend, while running adconfig on one of our cloned instance, I got an error

applint..pdc-al-ibm3> adconfig.sh
Enter the full path to the Context file: /d05/applint/11.5/admin/INT_pdc-al-ibm3.xml
Invalid range "a-Z" in transliteration operator at /d05/applint/product/iAS/Apache/perl/lib/5.00503/vars.pm line 17.
Compilation failed in require at /d05/applint/product/iAS/Apache/perl/lib/5.00503/AutoLoader.pm line 3.
BEGIN failed--compilation aborted at /d05/applint/product/iAS/Apache/perl/lib/5.00503/AutoLoader.pm line 3.
Compilation failed in require at /usr/opt/perl5/lib/5.8.2/aix-thread-multi/POSIX.pm line 7.
BEGIN failed--compilation aborted at /usr/opt/perl5/lib/5.8.2/aix-thread-multi/POSIX.pm line 7.
Compilation failed in require at /d05/applint/11.5/ad/11.5.0/bin/adconfig.pl line 88.
BEGIN failed--compilation aborted at /d05/applint/11.5/ad/11.5.0/bin/adconfig.pl line 88

I searched over Metalink and found a hit –

Invalid Range “A-Z” In Transliteration Operator [ID 279306.1]

Followed the approach and that resolved the issue. Post implementing the solution, I was able to run adconfig successfully.

-Anand

Y’day one of our PROD support team called up and asked that their ftp program is failing with the below error

/d05/applprod/11.5/xxzz/4.0.0/bin/ZZAAR1240F: No such file or directory usdsop: exec failed during spawn/d05/applprod/11.5/xxzz/4.0.0/bin/ZZAAR1240F Program exited with status 1

I first checked if the given shell script has the required permission to be executed by the application owner. and it was Ok – 755.

Then I opened the shell script on the server and tried to see if there is anything within the code that is causing the error. And there I saw the junk character (ctrl V + ctrl M). Looks like the file was edited in Windows and transferred to UNIX.

I removed all the junk character using the below command

:%s/(ctrl-v)(ctrl-m)//g
and press Enter key.

Then I asked the support team to run the program. program completed successfully and file got transferred.

-Anand

Couple of days back, I had the strange issue in our PROD environment.

End user called up and said that certain print job (custom job) has been running long and they were not able to see the output coming from the printer

We are on 11.5.9 with database version 9.2.0.6

Issue analysis/findings and resolution

• What has been done to the PROD system
  • There were some code migrations. Although these code migrations did touch some of the objects related to the offending program but it was very unlikely to cause this strange behaviour.
  • Application tier/concurrent tier server was bounced due to some h/w change. Although this supposedly h/w change was later on reverted due to some other reason
  • application patch was applied. This patch was exclusively related to Oracle workflow.
• Initial analysis showed that there were many duplicate requests submitted by the user. Hence those long running duplicate requests were cancelled.
• Despite cancelling some of the requests, many more requests kept coming up and thereby pegging the concurrent queue
• When tried to find what these long running requests are doing at the backend, I couldn’t find the Oracle process ID. ORACLE_PROCESS_ID column in FND_CONCURRENT_REQUESTS was NULL for all those concurrent requests. This was strange behaviour since this particular request has been completing successfully till sometime (may be around 10-12 hrs before) ago. And this is the reason, possibility of ‘SRW Exit and Init’ trigger code not being there was ruled out.
• Many other concurrent jobs (Oracle standard as well custom) in this particular manager were completing successfully.
• No errors were seen in the concurrent manager log – specific to the custom job which was giving issue and in question.
• No errors were seen in the startup script. All the components had come gracefully when the application was bounced.
• Later on I decided to bounce the concurrent manager. Even after bouncing the concurrent manager, behaviour remained the same i.e. when the concurrent managers came back, huge number of requests started showing running. Strange thing was none of “those” requests were having Oracle process ID associated.
• Ran cmclean.sql script – since the behaviour was related to concurrent processing. Logged a Sev1 SR with oracle and Oracle also confirmed to run cmclean.sql and then start the concurrent manager
• Despite running cmclean.sql, problem is not resolved.
• Then it was decided to bounce the whole application along with the database. But even database bounce didn’t help and still all “those” requests kept coming up – but without any associated ‘Oracle process ID’. This really made the thing worst – as I was not able to identify if there is anything happening on the database
• Database was perfectly fine –
  • No unusual database waits
  • No other long running jobs
  • No database locks
  • CPU utilization was normal
  • No persistent latches
• Later on I tried submitting a request for the same concurrent job to see if there is some issue with the schedule. But even my job remained in the queue – due to many other requests already in queue.
• Oracle was also not much forthcoming – despite we being on Sev1 SR.
•  I tried one last chance – before jumping into some unknown troubleshooting steps.
  • shut down all the services on app and web tier using adstpall
  •  Tried checking if there is any old defunct process on app and web tier. I saw many (around 300-400) dead/defunct processes owned by applmgr. These sessions were print queue processes and many of them were ora_rw20_run. Ensured that no ‘applmgr’ process is left
  • Killed all the defunct processes owned by applmgr
  • bounced the database and listener
  • Restarted the app and web services
  • ‘Concurrent job’ in question started showing as completing successfully. Number of pending jobs also started receding.
• This resolved the issue and PROD started working as expected but the exact root cause was not known

What could have caused it – Dead/defunct processes on the application tier as a result of unclean bounce of server.

• We had server bounce due to some h/w maintenance and I remember AIX guy had some issue while unmounting the application filesystem and it took significantly more time to bring down the AIX box as compared to other instances.
• Many print queue processes were showing of the same timestamp – as of server bounce.
• Even relinking FND as suggested by Oracle through SR didn’t help to address the issue.

This issue haunted for several days before I happened to find the root cause and solution. Since it was never clear what has been causing this issue, so only evident solution that had worked everytime is bounce applicaiton services along with database.

Last time, when the issue came, I just happned to see the TOP command output and found VNC process taking around 6-7% CPU – which didnot look normal. I killed the VNC process and as soon as VNC process was killed, all the requests showing as running (phase code and status code = R) went away. Concurrent Manager queue which was pegged started coming to normal.

All these concurrent jobs which was running long without any databas handle process were POSTSCRIPT report and hence need some sort of DISPLAY (VNC in our case).

Currently working to figure out if upgrading VNC would resolve the frequent problem. I will update the post once I find the permamnent solution (not bouncing VNC process).

-Anand M

Today I completed the vision instance installation of Oracle R12.1.1 on Linux machine. I had some hiccups but finally were able to resolve all the error and got the login page successfully. I will be covering all the installation steps in this blog – all the issues faced, their resolutions etc.

Oracle R12 Version – 12.1.1

OS – RHEL5 update 3

Single Node and Multi user Installation.

Metalink Note ID followed – 761566.1

Before starting installation, pls make sure all the below packages/utility is installed

• ar
• gcc
• g++
• ld
• ksh
• make
• X Display server

Installation size of Oracle E-business Suite 12.1.1

• Fresh install with Vision demo database – 243 GB (Application Tier FS – 35 GB + DB Tier FS – 208 GB)
• For my envionment
  • /d02 –> Application and Database FS  – 350 GB
• Staging Area
  • /d03 –> Oracle provided media will be copied into the respective directory structure
  • My staging area consists of
    • /d03/R12.1.1_X86
      • startCD
      • oraAS
      • oraDB
      • oraAppsDB
      • oraApps
• OS User and Group
  • created a group called ‘dbaerp’ and created 2 users – ‘applerp’ (application user) and ‘oraerp’ (Oracle filesystem user)
  • created a base directory – /d02/ebs/R12VIS
  • Change ownership and grant write permission to dbaerp group
  • cd /d02
  • mkdir -m 775 -p /ebs/R12VIS
  • chown -R oraerp:dbaerp ebs
• Kernel parameters (Related to Oracle E-Business Suite – R12) – After making any changes inthe system file, use “sysctl -p” command tor restart the system to invoke the new settings. vi /etc/sysctl.conf
  • kernel.shmmax = 68719476736
  • kernel.shmall = 4294967296
  • kernel.sem =  250 32000 100 128
  • fs.file-max = 6815744
  • net.ipv4.ip_local_port_range = 9000 65500
  • net.core.rmem_default = 262144
  • net.core.wmem_default = 262144
  • net.core.rmem_max = 4194304
  • net.core.wmem_max = 1048576
• $ vi /etc/security/limits.conf
  • Added below entries in the file
    • hard         nofile      65536
    • soft           nofile      4096
    • hard         noproc   16384
    • soft           noproc    2047
• vi /etc/hosts – make sure hosts file is formatted as below
  • [ip_address] [node_name].[domain_name] [node_name]
• vi /etc/sysconfig/network – Pls make sure it is formatted as below
  • HOSTNAME=[node_name].[domain_name]
• vi  /etc/sysconfig/networking/profiles/default/network – make sure ‘network’ file doesnot exist. In my case, it was not there.
• vi /etc/resolv.conf – Add or update the following entries to these minimum settings on each node
  • options attempts:5
  • options timeout:15
• OS Library patch for Oracle HTTP server – Downlaod and apply the patch 6078836 from MOS to fix an issue with the Oracle HTTP server (missing libdb.so)
• 

RPM required for Oracle E-Business Suite R12.1.1 – As per Metalink Note ID – 761566.1

• Check Linux Kernel – uname -r
• 

• OS Kernel
• 
• Hence for my RHEL5 update 3, following RPMs are needed – In my case, many of them were already installed and some of these RPMS need to be installed.

• Command used to query the RPM –
  • rpm -qa –queryformat “%{NAME}-%{VERSION}.%{RELEASE} (%{ARCH})\n” |grep <RPM_NAME>
  • openmotif21-2.1.30-11.EL5.i3861
  • 
  • xorg-x11-libs-compat-6.8.2-1.EL.33.0.1.i386
  • binutils-2.17.50.0.6-9.0.1.x86_642 – For update 3
  • gcc-4.1.2-14.el5.x86_64
  • gcc-c++-4.1.2-14.el5.x86_64
  • glibc-2.5-18.i686 (32-bit)
  • glibc-2.5-18.x86_64
  • glibc-common-2.5-18.x86_64
  • glibc-devel-2.5-18.i386 (32-bit)
  • glibc-devel-2.5-18.x86_64
  • libgcc-4.1.2-14.el5.i386
  • libgcc-4.1.2-14.el5.x86_64
  • libstdc++-devel-4.1.2-14.el5.i386
  • libstdc++-devel-4.1.2-14.el5.x86_64
  • libstdc++-4.1.2-14.el5.i386
  • libstdc++-4.1.2-14.el5.x86_64
  • libXi-1.0.1-3.1.i386
  • libXp-1.0.0-8.1.el5.i386
  • libXp-1.0.0-8.1.el5.x86_64
  • libaio-0.3.106-3.2.i386
  • libaio-0.3.106-3.2.x86_64
  • libgomp-4.1.2-14.el5.x86_64
  • make-3.81-1.1.x86_64
  • gdbm-1.8.0-26.2.1.i386
    • 
• • gdbm-1.8.0-26.2.1.x86_64
  • sysstat-7.0.0-3.el5.x86_64
  • util-linux-2.13-0.45.el5.x86_64
  • compat-libstdc++-296-2.96-138.i386
  • compat-libstdc++-33-3.2.3-61.i386
• Additionally few of the RPMs are required for 11gR1 database (which is bundles with 12.1.1 release) on database tier.
  • compat-libstdc++-33-3.2.3-61.x86_64
  • elfutils-libelf-devel-0.125-3.el5.x86_64
  • elfutils-libelf-devel-static-0.125-3.el5.x86_64
  • libaio-devel-0.3.106-3.2.x86_64
  • unixODBC-2.2.11-7.1.i386
  • unixODBC-devel-2.2.11-7.1.i386
  • unixODBC-2.2.11-7.1.x86_64
  • unixODBC-devel-2.2.11-7.1.x86_64
  • kernel-headers-2.6.18-8.el5.x86_64
• I had some error installing some of these but those were dependecies error viz. error when  installed openmotif21-2.1.30-11.EL5.i3861
  • 

Installation Process

• Media Check – MD5 Checksums for R12.1.1 Rapid Install Media [ID 802195.1]
  • cd /d03/R12.1.1_X86 –> Staging area
  • Download md5sum_xxxxx.txt –> Depending on OS
  • 

• Standard Installation – Login as root
• Export Display – Start the VNC process (if not done already) and export the display
• Cd to staging area – $ cd /d03/R12.1.1_X86/startCD/Disk1/rapidwiz
• /rapidwiz
• 

Issues and resolution

• Post Installation setp threw fllowing errors
  • HTTP Error
  • Login Pager Error
  • JSP Error
  • Virtual Page Error
  • Help page error
• Troubleshooting step
  • I stopped all the services and tried starting it again all using adstrtall command but adopmnctl command errored out.
    • 

• Upon investigation, I saw some error related to APPS listener in the Autoconfig log file
• 

• Researching error over Metalink, I found a Metalink Note – 1324667.1 and performed the actions as advised
  • Clean the Data Topology Model
    • sqlplus apps/apps_password
    • Exec FND_CONC_CLONE.SETUP_CLEAN;
    • COMMIT;
    • EXIT
  • Run autoconfig on all tier – first on DB tier and then app tier.
• Autoconfi completed on both the nodes successfully.  I started the services using adstpall. All the services came up successfully but Apache.
  • 

• Apache still not coming up and is giving error. Error log shows
  • 

• Error log was pointing to libclntsh.so.10.1. This file was very much there in 10.1.2_HOME/lib and 10.1.3_HOME/lib as well. I googled a lot and every forum/thread was mentioning that this file should be there in lib as well as lib32. I didnot find lib32 folder either in 10.1.2_HOME or in 10.1.3_HOME
  • Pls remember I had already applied the patch 6078836 – which is a OS library patch and required for a known HTTP issue.
  • Hence I thought to create a lib32 folder under both the homes and create a soft link to the actual physical location
    • created lib32 folder in $10.1.2_ORACLE_HOME & $10.1.3_ORACLE_HOME
    • created soft link for libclntsh.so.10.1
      • ln -s /d02/ebs/R12VIS/apps/tech_st/10.1.2/lib/libclntsh.so.10.1 libclntsh.so
      • ln -s /d02/ebs/R12VIS/apps/tech_st/10.1.3/lib/libclntsh.so.10.1 libclntsh.so
  • Restarted apache and got another error this time. Error related to libclntsh.so was no more there.
  • 

• This error was pointing to some SSL configuration in httpd.conf file. Hence commented the said configuration.
  • # Include the SSL definitions and Virtual host container
  • #include “/d02/ebs/R12VIS/inst/apps/R12VIS_adc-al-lnx45/ora/10.1.3/Apache/Apache/conf/ssl.conf”
• Restarted Apache. This resovled the apache issue and all the services came up without any issue.
• 

• Launched the login page and I was able  to see the login page
  • 

-Anand

Today after applying a patch in Production environment, one of the queues “FND_CP_GSM_OPP_AQ” got invalid
I tried to recompile this using below command but got ‘Invalid Alter command’ error.

SQL> alter queue FND_CP_GSM_OPP_AQ compile;
alter queue FND_CP_GSM_OPP_AQ compile
*
ERROR at line 1:
ORA-00940: invalid ALTER command

I searched over the Metalink and found one note (311851.1) related to compiling INVALID queue object. As per Metalink Note
“If the OPP service is running and there are messages in FND_CP_GSM_OPP_AQ queue, applying patches will invalidate this queue. The patch/patches internally validate and compile the schema making the data in the queue to be inconsistent”

Solution is to recreate the queue using $FND_TOP/patch/115/sql/afopp002.sql file as ‘APPLSYS’ user

I ran the said script and it resolved my INVALID queue issue.

-Anand