Wednesday, August 10, 2016

MISSING TASK TIMER EXECUTIONS from Event Manager - Due to max retry reached.

Problem statement :-


There is a scheduled task which meant to start a BPD and create an instance in EM queue which should run at particular time but BPM instance was not created even after timer task went out of EM queue.


Explaination :-


Whenever BPM executes  EM (= Event Manager) task, it very early in that thread tries to lock the instance with a SELECT for UPDATE WITH RS on the according row in the BPD instance table.

If another thread already locked that row, the EM task may fail after a while (lock wait timeout).


You can see something like below log in logs


wle_ucaexcept E   CWLLG0181E: An exception occurred during execution of task 16,513,005.  Error: PreparedStatementCallback; SQL [SELECT BPD_INSTANCE_ID FROM LSW_BPD_INSTANCE WHERE BPD_INSTANCE_ID = ? FOR UPDATE WITH RS]; DB2 SQL Error: SQLCODE=-911, SQLSTATE=40001, SQLERRMC=68, DRIVER=3.64.133; nested exception is com.ibm.db2.jcc.am.SqlTransactionRollbackException: DB2 SQL Error: SQLCODE=-911, SQLSTATE=40001, SQLERRMC=68, DRIVER=3.64.133



The Event Manager is going to retry such a task, by default for 5 times!
If if fails 5 times it would have been just discarded in earlier BPM releases, but with APAR JR47860 installed, it would be rescheduled for 2099 and you could manually resubmit it!
Let my show you how the message looks like, after which the 5 retries have been reached (important is the message CWLLG0197W):



wle_ucaexcept E   CWLLG0197W: Task Notify BPD 741581 of notification failed 5  times.  The task will not be re-executed.



If you face similar situation, try to find CWLLG0197W in logs.



Problem Conclusion :-



The problem is solved by enhancing the Event Manager such that Event Manager tasks that were retried until reaching the re-execute-limit can be resumed by administrative means once the exceptional situation is resolved.
With this code fix in place when a task reaches the maximum limit When a work item has reached the maximum retry execution or the queue is full, the items will be listed in the Event Manager console page and marked with a scheduled execution date of "2099-02-01." Due to localization it may also appear as "1/2/99".

  • With this APAR upon reaching the re-execute-limit, the respective Event Manager task is put on hold.
  • In addition, a task is created and assigned to the EM administrator (as specified via notify-error in the 80EventManager.xml).  
    • Note: The system could be configured such that the EM administrator is notified about such tasks via e-mail. Thus the EM administrator could be notified about the exceptional situation.
  • To resume Event Manager tasks that are on hold, an administrative command is provided that allows replaying such Event Manager tasks so that they can be scheduled by the Event Manager again: BPMReplayOnHoldEMTasks

Parameters :-

  • getNumberOfTasks - retrieves the number of Event Manager tasks that are on hold
    • Set this parameter to true if the BPMReplayOnHoldEMTasks command should retrieve the number of Event Manager tasks available for replay.
  • maxNumberOfTasksToReplay - replays on-hold Event Manager tasks up to a maximum number specified Use this parameter to set an upper limit for the number of on-hold Event Manager tasks to be replayed.
  • bpdInstanceId - replays on-hold Event Manager tasks for the BPD instance specified Specifies for which BPD instance on-hold Event Manager tasks should be replayed.
Note that the parameters are mutually exclusive.
To invoke BPMReplayOnHoldEMTasks you must start wsadmin and connect it to the process server or process center.
                 E.g., wsadmin -conntype SOAP -port 4080 -host PC1.mycompany.com -user admin -password admin -lang jython  


Examples

Query the number of available on-hold Event Manager tasks in the system:
wsadmin>AdminTask.BPMReplayOnHoldEMTasks ('[-getNumberOfTasks true]')
 'The BPMReplayOnHoldEMTasks command found 20 on hold Event Manager Task(s) ready for replay.'
    
Replay 13 on-hold Event Manager tasks:
wsadmin>AdminTask.BPMReplayOnHoldEMTasks ('[-maxNumberOfTasksToReplay 13]')
'The BPMReplayOnHoldEMTasks command replayed 13 on hold Event Manager Task(s).'
    
Replay on-hold Event Manager tasks for BPD instance 49:
wsadmin>AdminTask.BPMReplayOnHoldEMTasks ('[-bpdInstanceId 49]')
'The BPMReplayOnHoldEMTasks command replayed 1 on hold Event Manager Task(s).'


Replay all on-hold Event Manager tasks:

wsadmin>AdminTask.BPMReplayOnHoldEMTasks();
'The BPMReplayOnHoldEMTasks command replayed 20 on hold Event Manager Task(s).'



More details can be found at below link regarding the IFIX





Tuesday, August 2, 2016

BPMDeleteSnapshot and BPMSnapshotCleanup commands - Issues

Abstract

Because issues have been discovered with the BPMDeleteSnapshot and BPMSnapshotCleanup commands, these commands require the interim fixes that are listed in this document. 

Follow 
http://www-01.ibm.com/support/docview.wss?uid=swg21669992#7512

check if you have all below IFIx installed on top of 7512 before going ahead.

APAR7.5.1.2
JR48877 
Symptom
A participant group (called “team” in V8.5) is not visible in the Process Admin Console of Process Server under Role Bindings when a snapshot is installed after you use theBPMDeleteSnapshot command.
Prevention 
Interim fix JR48877
Repairs
Interim fix JR49635
Express: 7.5.1.2-WS-BPM-IFJR48877
Standard: 7.5.1.2-WS-BPM-IFJR48877
Advanced: 7.5.1.2-WS-BPM-IFJR48877
JR49374
Symptom
You receive the following error after you run theBPMSnapshotCleanup or BPMDeleteSnapshot command:
CWLLG1414E - An exception occurred while getting user attributes
Prevention 
Interim fix JR49374
Repairs
Interim fix JR49374 or modify each user attribute definition in your process application and save it
Express: 7.5.1.2-WS-BPM-IFJR49374
Standard: 7.5.1.2-WS-BPM-IFJR49374
Advanced: 7.5.1.2-WS-BPM-IFJR49374
JR49606
Symptom
A participant group (called “team” in V8.5) is not visible in the Process Admin Console of IBM Process Center under Role Bindings when a snapshot is installed after you use theBPMSnapshotCleanup command.
Prevention 
Interim fix JR49606
Repairs
Interim fix JR49928
Express: 7.5.1.2-WS-BPM-IFJR49606
Standard: 7.5.1.2-WS-BPM-IFJR49606
Advanced: 7.5.1.2-WS-BPM-IFJR49606
JR49635 
This repair-only interim fix applies to the issue that is addressed in JR48877. JR49635 looks for and repairs issues when importing a snapshot. Although JR48877 repairs most of the issues, you still need interim fix JR49635 if there are more than 1000 participant groups (called “teams” in V8.5). JR49635 repairs the damage done by the issue that is fixed by JR48877 in the event that you encounter the issue before you apply JR48877.
Express: 7.5.1.2-WS-BPM-IFJR49635
Standard: 7.5.1.2-WS-BPM-IFJR49635
Advanced: 7.5.1.2-WS-BPM-IFJR49635
JR49848
Symptom
After you run the BPMSnapshotCleanup command, you can no longer save artifacts that have advanced content in Process Designer in a process application or toolkit from IBM Integration Designer
Prevention 
Interim fix JR49848
Repairs
Interim fix JR49848 or edit the LSW_SCA_CONNECTOR table of the BPMDB database and put the outOfSync value into the LAST_SYNCHRONIZED column for the Advanced Integration Service that is affected
Express: Not applicable
Standard: Not applicable
Advanced: 7.5.1.2-WS-BPM-IFJR49848
JR49928
This repair-only interim fix applies to the issue that is addressed in JR49606. JR49928 looks for and repairs issues when the server starts. You do not require interim fix JR49928 if the interim fixes JR48877 and JR49606 are applied before snapshots are deleted.
Express: 7.5.1.2-WS-BPM-IFJR49928
Standard: 7.5.1.2-WS-BPM-IFJR49928
Advanced: 7.5.1.2-WS-BPM-IFJR49928
JR50444
When the server starts or when you use theBPMInstallOfflinePackage administrative command, you might see the DATAINTEGRITYVIOLATIONEXCEPTION. This fix checks whether there are exposed items before deleting the groups, preventing you from deleting those groups.
Express: 7.5.1.2-WS-BPM-IFJR50444
Standard: 7.5.1.2-WS-BPM-IFJR50444
Advanced: 7.5.1.2-WS-BPM-IFJR50444
JR50460
An exception occurs while history information for a process app or toolkit is being loaded when that history information contains references to deleted snapshots.
Express: 7.5.1.2-WS-BPM-IFJR50460
Standard: 7.5.1.2-WS-BPM-IFJR50460
Advanced: 7.5.1.2-WS-BPM-IFJR50460
JR50667
Deleting a snapshot destroys most smart folders. Smart folders are used for the entire project's track or branch. It is not expected that the smart folders will be deleted when snapshots are deleted.
Express: 7.5.1.2-WS-BPM-IFJR50667
Standard: 7.5.1.2-WS-BPM-IFJR50667
Advanced: 7.5.1.2-WS-BPM-IFJR50667
JR50671
When snapshots are deleted, the corresponding LSW installation table entries are not deleted.    
Express: 7.5.1.2-WS-BPM-IFJR50671
Standard: 7.5.1.2-WS-BPM-IFJR50671
Advanced: 7.5.1.2-WS-BPM-IFJR50671
JR50718
The BPMDeleteSnapshot command proceeds even though there are running instances of stand-alone human services.  
Express: 7.5.1.2-WS-BPM-IFJR50718
Standard: 7.5.1.2-WS-BPM-IFJR50718
Advanced: 7.5.1.2-WS-BPM-IFJR50718
JR50720
The BPMDeleteSnapshot command proceeds even though there business process definition (BPD) instances in another snapshot that point to tasks from the snapshot that is being deleted.
Express: 7.5.1.2-WS-BPM-IFJR50720
Standard: 7.5.1.2-WS-BPM-IFJR50720
Advanced: 7.5.1.2-WS-BPM-IFJR50720
JR50721
The BPMSnapshotCleanup command proceeds even though there are running      
instances of stand-alone human services.          
Express: 7.5.1.2-WS-BPM-IFJR50721
Standard: 7.5.1.2-WS-BPM-IFJR50721
Advanced: 7.5.1.2-WS-BPM-IFJR50721
JR50742
When the server starts or when you use theBPMInstallOfflinePackage administrative command, you might see a DATAINTEGRITYVIOLATION exception.   
Express: 7.5.1.2-WS-BPM-IFJR50742
Standard: 7.5.1.2-WS-BPM-IFJR50742
Advanced: 7.5.1.2-WS-BPM-IFJR50742
JR52327
When you run the BPMSnapshotCleanup command with -ignoreDependency = true, toolkits that have other toolkits and process apps depending on them can be deleted without warning, leading to other problems.
Express: 7.5.1.2-WS-BPM-IFJR52327
Standard: 7.5.1.2-WS-BPM-IFJR52327
Advanced: 7.5.1.2-WS-BPM-IFJR52327