Oracle Application Technology

Today I got a call from Development team that one the concurrent jobs submitted (in Development instance) by them has not been processed since long. I looked into the job and it showed it is in ‘Pending Standby’ status even though the manager has enough worker to process

Somehow it looked to me concurrent manager issue. Others jobs were also not getting processed. I thought of bouncing the CCM (concurrent manager). I stopped the CCM using adcmctl.sh but the job which is submitted to bring down the concurrent manager also went into ‘pending standby’ mode.

Finally I had to manually kill the FNDLIBR process at OS level and started the CCM again. But none of the FNDLIBR process came up. I looked into the ICM log and saw below error

Routine &ROUTINE has attempted to start the internal concurrent manager.  The ICM is already running.  Contact you system administrator for further assistance.afpdlrq received an unsuccessful result from PL/SQL procedure or function FND_DCP.Request_Session_Lock.
Routine FND_DCP.REQUEST_SESSION_LOCK received a result code of 1 from the call to DBMS_LOCK.Request.
Possible DBMS_LOCK.Request resultCall to establish_icm failed
The Internal Concurrent Manager has encountered an error.

Review concurrent manager log file for more detailed information

I tried to correlate this with the Database crash which happened in the same instance in the morning. and looked like database crash could have caused this. I did ‘tnsping to FNDSM’ and it returned the result properly.

Hence followed below approach to resolve the issue

• Stopped all the services on application and Web Tier
• since none of the FNDLIBR process had come up earlier, I was good not to kill any FNDLIBR processes.
• Bouned the database (Database was taking time while going down – due to crash)
• started the services on application and web tier

All the services came up without any issue. All the concurrent managers got started. I checked the concurrent jobs and they also completed normal.

Later on I researched the issue over Google and found a similar thread.

https://forums.oracle.com/forums/thread.jspa?messageID=9245583

-Anand