MISSING TASK TIMER EXECUTIONS from Event Manager - Due to max retry reached.

Problem statement :-

There is a scheduled task which meant to start a BPD and create an instance in EM queue which should run at particular time but BPM instance was not created even after timer task went out of EM queue.

Explaination :-

Whenever BPM executes  EM (= Event Manager) task, it very early in that thread tries to lock the instance with a SELECT for UPDATE WITH RS on the according row in the BPD instance table.

If another thread already locked that row, the EM task may fail after a while (lock wait timeout).

You can see something like below log in logs

wle_ucaexcept E   CWLLG0181E: An exception occurred during execution of task 16,513,005.  Error: PreparedStatementCallback; SQL [SELECT BPD_INSTANCE_ID FROM LSW_BPD_INSTANCE WHERE BPD_INSTANCE_ID = ? FOR UPDATE WITH RS]; DB2 SQL Error: SQLCODE=-911, SQLSTATE=40001, SQLERRMC=68, DRIVER=3.64.133; nested exception is com.ibm.db2.jcc.am.SqlTransactionRollbackException: DB2 SQL Error: SQLCODE=-911, SQLSTATE=40001, SQLERRMC=68, DRIVER=3.64.133

The Event Manager is going to retry such a task, by default for 5 times!

If if fails 5 times it would have been just discarded in earlier BPM releases, but with APAR JR47860 installed, it would be rescheduled for 2099 and you could manually resubmit it!

Let my show you how the message looks like, after which the 5 retries have been reached (important is the message CWLLG0197W):

wle_ucaexcept E   CWLLG0197W: Task Notify BPD 741581 of notification failed 5  times.  The task will not be re-executed.

If you face similar situation, try to find CWLLG0197W in logs.

Problem Conclusion :-

The problem is solved by enhancing the Event Manager such that Event Manager tasks that were retried until reaching the re-execute-limit can be resumed by administrative means once the exceptional situation is resolved.

With this code fix in place when a task reaches the maximum limit When a work item has reached the maximum retry execution or the queue is full, the items will be listed in the Event Manager console page and marked with a scheduled execution date of "2099-02-01." Due to localization it may also appear as "1/2/99".

• With this APAR upon reaching the re-execute-limit, the respective Event Manager task is put on hold.
• In addition, a task is created and assigned to the EM administrator (as specified via notify-error in the 80EventManager.xml).  
• Note: The system could be configured such that the EM administrator is notified about such tasks via e-mail. Thus the EM administrator could be notified about the exceptional situation.
• To resume Event Manager tasks that are on hold, an administrative command is provided that allows replaying such Event Manager tasks so that they can be scheduled by the Event Manager again: BPMReplayOnHoldEMTasks

Parameters :-

• getNumberOfTasks - retrieves the number of Event Manager tasks that are on hold
• Set this parameter to true if the BPMReplayOnHoldEMTasks command should retrieve the number of Event Manager tasks available for replay.
• maxNumberOfTasksToReplay - replays on-hold Event Manager tasks up to a maximum number specified Use this parameter to set an upper limit for the number of on-hold Event Manager tasks to be replayed.
• bpdInstanceId - replays on-hold Event Manager tasks for the BPD instance specified Specifies for which BPD instance on-hold Event Manager tasks should be replayed.

Note that the parameters are mutually exclusive.

To invoke BPMReplayOnHoldEMTasks you must start wsadmin and connect it to the process server or process center.

                 E.g., wsadmin -conntype SOAP -port 4080 -host PC1.mycompany.com -user admin -password admin -lang jython  

Examples

Query the number of available on-hold Event Manager tasks in the system:

wsadmin>AdminTask.BPMReplayOnHoldEMTasks ('[-getNumberOfTasks true]')

 'The BPMReplayOnHoldEMTasks command found 20 on hold Event Manager Task(s) ready for replay.'

    

Replay 13 on-hold Event Manager tasks:

wsadmin>AdminTask.BPMReplayOnHoldEMTasks ('[-maxNumberOfTasksToReplay 13]')

'The BPMReplayOnHoldEMTasks command replayed 13 on hold Event Manager Task(s).'

    

Replay on-hold Event Manager tasks for BPD instance 49:

wsadmin>AdminTask.BPMReplayOnHoldEMTasks ('[-bpdInstanceId 49]')

'The BPMReplayOnHoldEMTasks command replayed 1 on hold Event Manager Task(s).'

Replay all on-hold Event Manager tasks:

wsadmin>AdminTask.BPMReplayOnHoldEMTasks();

'The BPMReplayOnHoldEMTasks command replayed 20 on hold Event Manager Task(s).'

More details can be found at below link regarding the IFIX

http://www-01.ibm.com/support/docview.wss?uid=swg1JR47860