Oracle Application Technology

Development team informed me of an issue where the concurrent job (that needs post processing) errored. On reveiwing the request log file, i notice an isue with ‘Output post Processor’. I checked the OPP in the ‘Administer Concurrent Manager’ screen and found

Actual=4 and Target=0 processes

I tried to restart but still getting the same status. Later on I tried to query the “FNDOPP” process on the application tier
$ ps -ef|grep -i FNDOPP|grep -v grep

and this resulted in 0 process.

I then looked into the manager log file and found below error

Jan 19, 2016 8:02:44 AM oracle.ias.cache.CacheInternal logLifecycleEvent
INFO: JOC is initialized from oracle.apps.jtf.cache.IASCacheProvider.init, ver=11.1.1.2.0, distribute=true, vid=996, coordinator=0, discover list=[[129.80.88.133:12345] segID=1]
Unable to initialize state monitor.
oracle.apps.fnd.cp.gsm.GenCartCommException: ORA-01403: no data found
ORA-06512: at "APPS.FND_CP_GSM_IPC", line 539
ORA-06512: at line 1

	at oracle.apps.fnd.cp.gsm.GenCartComm.initService(Unknown Source)
	at oracle.apps.fnd.cp.gsm.GenCartComm.<init>(Unknown Source)
	at oracle.apps.fnd.cp.gsf.GSMStateMonitor.init(Unknown Source)
	at oracle.apps.fnd.cp.gsf.GSMStateMonitor.<init>(Unknown Source)
	at oracle.apps.fnd.cp.gsf.GSMServiceController.init(GSMServiceController.java:117)
	at oracle.apps.fnd.cp.gsf.GSMServiceController.<init>(GSMServiceController.java:72)
	at oracle.apps.fnd.cp.gsf.GSMServiceController.main(GSMServiceController.java:448)

Solution that resolved the issue

I found “Service Manager” was down. So I restarted “Service Manager” and then restarted “Output Post Processor”.
and Actual=4 and Target=4 processes

I asked the development team to submit the job again. This time job completed successfully.

Today support team sought a help for the issue they were encountering in the TEST instance.

Error as given by the support team

ErrBufORA-20100: File o7725967.tmp creation for FND_FILE failed.
You will find more information on the cause of the error in request log.

I tried a quick test to see if there is any issue with utl_file_dir

SQL>exec FND_FILE.PUT_LINE(FND_FILE.LOG, 'THIS IS A TEST');
BEGIN FND_FILE.PUT_LINE(FND_FILE.LOG, 'THIS IS A TEST'); END;

*
ERROR at line 1:
ORA-20100: File o7725970.tmp creation for FND_FILE failed.
You will find more information on the cause of the error in request log.
ORA-06512: at "APPS.FND_FILE", line 410
ORA-06512: at "APPS.FND_FILE", line 517
ORA-06512: at line 1

Above test threw an error. Then I verified 2 more things

1) APPLPTMP directory should be the first one appearing when UTL_FILE_DIR parameter is queried – this looked Ok

SQL>show parameter utl_file

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
utl_file_dir string /d05/apptest/common/temp, /usr
/tmp, /d12/oratest/product/9.2.
0/appsutil/outbound/TEST, /d05/
apptest/11.5/xxbb/4.0.0/edi/ou
t, /d05/appltest/11.5/xxdd/4.0.
0/US/utl_file_dir, /d05/appltest
t/common/admin/log/TEST_host_name
, /d05/appltest/common/admi
n/out/TEST_host_name, /d05/ap
pltest/common/admin/log/TEST_host_name,
/d12/oratest/product/
9.2.0/ccr/hosts/host_name/st
ate, /d05/appltest/11.5/xxcal/s
mart, /d05/appltest/11.5/xxcal/
bartender

2)Missing permissions on APPLPTMP directory – I found that the permission was not correct. I gave the permission (777) to the directory and this resolved the issue.

To confirm, I ran the same procedure again and it returned the expected result.

SQL>exec FND_FILE.PUT_LINE(FND_FILE.LOG, 'THIS IS A TEST');

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.15

Other probable causes of error

3)APPLPTMP directory is missing &

4)APPLPTMP directory pointing to wrong location

Hope this helps.

Recently I got a request from a bank where the customer transmits some of the files. Bank was upgrading and changing their target server and hence we need to accommodate certain changes on our side to do the testing before the same is done in production environment.

Command to list the existing Key IDs on the keyring

appluser.DEV.pdc-xyz-ibm> /usr/local/bin/pgp -kv
Pretty Good Privacy(tm) Version 6.5.8
(c) 1999 Network Associates Inc.
Uses the RSAREF(tm) Toolkit, which is copyright RSA Data Security, Inc.
Export of this software may be restricted by the U.S. government.
Type bits      keyID      Date       User ID
RSA  1024      ********** 2009/07/02 *** DEFAULT SIGNING KEY ***
                                     XY_ABCBANK_TEST
RSA  1024      ********** 2009/07/06 abc pqr <a href="mailto:abc.pqr@domain.com">abc.pqr@domain.com</a>>
RSA  2048/2048 ********** 2012/02/21 expires 2014/05/01
                                      File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
RSA  2048/2048 ********** 2009/03/04 *** KEY EXPIRED ***
                                      ABCBANK_ECS_QA_2011 <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
4 matching keys found.

Bank sends the new public key from their TEST/QA server. This is done so as to do point to point testing from our DEV/QA to Bank DEV/QA server.

Once the new public key is recived,this needs to be added into the keyring using below command

/usr/local/bin/pgp -ka <BANK_PUBLIC_KEY.ASC>
Pretty Good Privacy(tm) Version 6.5.8
(c) 1999 Network Associates Inc.
Uses the RSAREF(tm) Toolkit, which is copyright RSA Data Security, Inc.
Export of this software may be restricted by the U.S. government.
Looking for new keys...
RSA  2048/2048 ************ 2012/02/21 File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
sig?           ************             (Unknown signator, can't be checked)

keyfile contains 1 new keys. Add these keys to keyring ? (Y/n) Y
New userid: "File_Transfer_Services <a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>".
New signature from keyID ********* on userid File_Transfer_Services <a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>

Keyfile contains:
   1 new key(s)
   1 new signatures(s)
   1 new user ID(s)
Summary of changes :

New userid: "File_Transfer_Services <a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>".
New signature from keyID ********** on userid File_Transfer_Services <a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>

Added :
   1 new key(s)
   1 new signatures(s)
   1 new user ID(s)

Comand to extract the key for the user XY_ABCBANK_TEST (copies ID’s key to keyfile from keyring )

XY_ABCBANK_TEST is the user ID signing the encrypted file

/usr/local/bin/pgp -kx XY_ABCBANK_TEST /tmp/pgp_key_XY_Nov2013.asc

Getting a warning message while encrypting and signing the file

Key for user ID: File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
2048-bit RSA key, Key ID ******, created 2012/02/21, expires 2014/05/01
WARNING:  Because this public key is not certified with a trusted
signature, it is not known with high confidence that this public key
actually belongs to: "File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>".

Now to overcome the error, we need to sign the public key received from 3rd party using our secret key on key ring

pgp -ks <HER_USERID>-u <YOUR_USERID>keyring

appluser.DEV.pdc-xyz-ibm> pgp -ks File_Transfer_Services -u XY_ABCBANK_TEST_2013
Pretty Good Privacy(tm) Version 6.5.8
(c) 1999 Network Associates Inc.
Uses the RSAREF(tm) Toolkit, which is copyright RSA Data Security, Inc.
Export of this software may be restricted by the U.S. government.
Key for user ID: File_Transfer_Services <<a href="mailto:connectivity.services.engineering@abc.com">connectivity.services.engineering@abc.com</a>>
2048-bit RSA key, Key ID ***********, created 2012/02/21, expires 2014/05/01
          Key fingerprint =  XY PQ 86 14 PO E9 9F 11  9Q E7 2G 80 69 75 U8 37  F6 8S 7C C3
READ CAREFULLY:  Based on your own direct first-hand knowledge, are
you absolutely certain that you are prepared to solemnly certify that
the above public key actually belongs to the user specified by the
above user ID (y/N)? y

You need a pass phrase to unlock your secret key.
Key for user ID "XY_ABCBANK_TEST_2013"
Key does not have a passphrase.

Attach a regular expression to this signature, or
press enter for none:
appluser.DEV.pdc-xyz-ibm>

Reference URL for other various PGP command summary

http://www.dsj.net/pgp/pgphelp.html

http://www.cs.wm.edu/pgp.html

-Anand

Today our development team complained of ‘Development’ instance of Oracle 11 Ebiz is not accessible. Even the login page is not loading.

I tried to login and got the login page but later on after keying the login credential, it was taking extremely long time to open and later on threw ‘page not found’ error. I thought of bouncing apache.

When I tried to bounce apache, I got below error.

ORACLE error 604 in afpodbinit

Cause:  afpodbinit failed due to ORA-00604: error occurred at recursive SQL level 1
ORA-01013: user requested cancel of current operation
ORA-06508: PL/SQL: could not find program unit being called
ORA-06512: at "APPS.FND_GLOBAL", line 101
ORA-06512: at "APPS.FND_GLOBAL",.

The SQL statement being executed at the time of the error was: begin fnd_global.bless_next_init('FND_PERMIT_0000'); fnd_global.initialize(:session_id, :user_id, :resp_id, :resp_appl_id, :security_group_id, :site_id, :login_id, :conc_login_id, :prog_appl_id, :conc_program_id, :conc_request_id, :conc_priority_request, :form_id, :form_application_id, :conc_process_id, :conc_queue_id, :queue_appl_id, :server_id); fnd_profile.put('ORG_ID', :org_id); fnd_profile.put('MFG_ORGANIZATION_ID', :mfg_org_id); fnd_profile.put('MFG_CHART_OF_ACCOUNTS_ID', :coa); fnd_profile.put('APPS_MAINTENANCE_MODE', :amm); end; and was executed from the file &ERRFILE.

Apache Web Server Listener :httpd ( pid 17197 ) is running.
Stopping Apache Web Server Listener (dedicated HTTP) ...

Oracle error -6508: ORA-01013: user requested cancel of current operation
ORA-06508: PL/SQL: could not find program unit being called
has been detected in FND_GLOBAL.INITIALIZE.

Oracle error -6508: ORA-01013: user requested cancel of current operation
ORA-06508: PL/SQL: could not find program unit being called
has been detected in FND_GLOBAL.INITIALIZE.
Apache Web Server Listener (PLSQL) :httpd ( pid 17521 ) is running.
Stopping Apache Web Server Listener (dedicated PLSQL) ...
Oracle error -6508: ORA-01013: user requested cancel of current operation
ORA-06508: PL/SQL: could not find program unit being called
has been detected in FND_GLOBAL.INITIALIZE.

adapcctl.sh: exiting with status 0

Later on when I tried to run adstpall.sh on Web tier, I got following in the error.

Setting service Oracle Apache Server DEV_pdc-al-lnx12 to mode 3
Executing service control script:
/d01/wfmdev/common/admin/scripts/DEV_pdc-al-lnx12/adapcctl.sh stop
Timeout specified in context file: 100 second(s)

script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception

adapcctl.sh version 115.50

Later on when I looked into the database, I got following latches/library pin waits and hence decided to go for database bounce.

    1    71 library cache lock            JDBC Thin Client     APPS        ############ 5.0440E+17        302   1117
    1    87 library cache lock               ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        302   1084
                                          (TNS V1-V3)

    1    88 library cache lock                                 APPS        ############ 5.0440E+17        302    817
    1   104 library cache lock            JDBC Thin Client     APPS        ############ 5.0440E+17        302    520
    1   132 library cache lock                                 APPS        ############ 5.0440E+17        302   1169
    1   150 library cache lock               ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        302    817
                                          (TNS V1-V3)

    1   164 library cache lock            JDBC Thin Client     APPS        ############ 5.0440E+17        302   1117
    1   233 library cache lock               ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        302    189
                                          (TNS V1-V3)

    1   142 library cache pin                ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        302    219
                                          (TNS V1-V3)

    1   171 library cache pin                ?  @pdc-al-lnx12  APPS        ############ 5.0440E+17        202     11

DB bounce helped to resolve the issue. ‘adstpall.sh’ and ‘adstrtal.sh’ executed successfully. Application was very much accessible.

-Anand