Wednesday, August 10, 2016

MISSING TASK TIMER EXECUTIONS from Event Manager - Due to max retry reached.

Problem statement :-


There is a scheduled task which meant to start a BPD and create an instance in EM queue which should run at particular time but BPM instance was not created even after timer task went out of EM queue.


Explaination :-


Whenever BPM executes  EM (= Event Manager) task, it very early in that thread tries to lock the instance with a SELECT for UPDATE WITH RS on the according row in the BPD instance table.

If another thread already locked that row, the EM task may fail after a while (lock wait timeout).


You can see something like below log in logs


wle_ucaexcept E   CWLLG0181E: An exception occurred during execution of task 16,513,005.  Error: PreparedStatementCallback; SQL [SELECT BPD_INSTANCE_ID FROM LSW_BPD_INSTANCE WHERE BPD_INSTANCE_ID = ? FOR UPDATE WITH RS]; DB2 SQL Error: SQLCODE=-911, SQLSTATE=40001, SQLERRMC=68, DRIVER=3.64.133; nested exception is com.ibm.db2.jcc.am.SqlTransactionRollbackException: DB2 SQL Error: SQLCODE=-911, SQLSTATE=40001, SQLERRMC=68, DRIVER=3.64.133



The Event Manager is going to retry such a task, by default for 5 times!
If if fails 5 times it would have been just discarded in earlier BPM releases, but with APAR JR47860 installed, it would be rescheduled for 2099 and you could manually resubmit it!
Let my show you how the message looks like, after which the 5 retries have been reached (important is the message CWLLG0197W):



wle_ucaexcept E   CWLLG0197W: Task Notify BPD 741581 of notification failed 5  times.  The task will not be re-executed.



If you face similar situation, try to find CWLLG0197W in logs.



Problem Conclusion :-



The problem is solved by enhancing the Event Manager such that Event Manager tasks that were retried until reaching the re-execute-limit can be resumed by administrative means once the exceptional situation is resolved.
With this code fix in place when a task reaches the maximum limit When a work item has reached the maximum retry execution or the queue is full, the items will be listed in the Event Manager console page and marked with a scheduled execution date of "2099-02-01." Due to localization it may also appear as "1/2/99".

  • With this APAR upon reaching the re-execute-limit, the respective Event Manager task is put on hold.
  • In addition, a task is created and assigned to the EM administrator (as specified via notify-error in the 80EventManager.xml).  
    • Note: The system could be configured such that the EM administrator is notified about such tasks via e-mail. Thus the EM administrator could be notified about the exceptional situation.
  • To resume Event Manager tasks that are on hold, an administrative command is provided that allows replaying such Event Manager tasks so that they can be scheduled by the Event Manager again: BPMReplayOnHoldEMTasks

Parameters :-

  • getNumberOfTasks - retrieves the number of Event Manager tasks that are on hold
    • Set this parameter to true if the BPMReplayOnHoldEMTasks command should retrieve the number of Event Manager tasks available for replay.
  • maxNumberOfTasksToReplay - replays on-hold Event Manager tasks up to a maximum number specified Use this parameter to set an upper limit for the number of on-hold Event Manager tasks to be replayed.
  • bpdInstanceId - replays on-hold Event Manager tasks for the BPD instance specified Specifies for which BPD instance on-hold Event Manager tasks should be replayed.
Note that the parameters are mutually exclusive.
To invoke BPMReplayOnHoldEMTasks you must start wsadmin and connect it to the process server or process center.
                 E.g., wsadmin -conntype SOAP -port 4080 -host PC1.mycompany.com -user admin -password admin -lang jython  


Examples

Query the number of available on-hold Event Manager tasks in the system:
wsadmin>AdminTask.BPMReplayOnHoldEMTasks ('[-getNumberOfTasks true]')
 'The BPMReplayOnHoldEMTasks command found 20 on hold Event Manager Task(s) ready for replay.'
    
Replay 13 on-hold Event Manager tasks:
wsadmin>AdminTask.BPMReplayOnHoldEMTasks ('[-maxNumberOfTasksToReplay 13]')
'The BPMReplayOnHoldEMTasks command replayed 13 on hold Event Manager Task(s).'
    
Replay on-hold Event Manager tasks for BPD instance 49:
wsadmin>AdminTask.BPMReplayOnHoldEMTasks ('[-bpdInstanceId 49]')
'The BPMReplayOnHoldEMTasks command replayed 1 on hold Event Manager Task(s).'


Replay all on-hold Event Manager tasks:

wsadmin>AdminTask.BPMReplayOnHoldEMTasks();
'The BPMReplayOnHoldEMTasks command replayed 20 on hold Event Manager Task(s).'



More details can be found at below link regarding the IFIX





Tuesday, August 2, 2016

BPMDeleteSnapshot and BPMSnapshotCleanup commands - Issues

Abstract

Because issues have been discovered with the BPMDeleteSnapshot and BPMSnapshotCleanup commands, these commands require the interim fixes that are listed in this document. 

Follow 
http://www-01.ibm.com/support/docview.wss?uid=swg21669992#7512

check if you have all below IFIx installed on top of 7512 before going ahead.

APAR7.5.1.2
JR48877 
Symptom
A participant group (called “team” in V8.5) is not visible in the Process Admin Console of Process Server under Role Bindings when a snapshot is installed after you use theBPMDeleteSnapshot command.
Prevention 
Interim fix JR48877
Repairs
Interim fix JR49635
Express: 7.5.1.2-WS-BPM-IFJR48877
Standard: 7.5.1.2-WS-BPM-IFJR48877
Advanced: 7.5.1.2-WS-BPM-IFJR48877
JR49374
Symptom
You receive the following error after you run theBPMSnapshotCleanup or BPMDeleteSnapshot command:
CWLLG1414E - An exception occurred while getting user attributes
Prevention 
Interim fix JR49374
Repairs
Interim fix JR49374 or modify each user attribute definition in your process application and save it
Express: 7.5.1.2-WS-BPM-IFJR49374
Standard: 7.5.1.2-WS-BPM-IFJR49374
Advanced: 7.5.1.2-WS-BPM-IFJR49374
JR49606
Symptom
A participant group (called “team” in V8.5) is not visible in the Process Admin Console of IBM Process Center under Role Bindings when a snapshot is installed after you use theBPMSnapshotCleanup command.
Prevention 
Interim fix JR49606
Repairs
Interim fix JR49928
Express: 7.5.1.2-WS-BPM-IFJR49606
Standard: 7.5.1.2-WS-BPM-IFJR49606
Advanced: 7.5.1.2-WS-BPM-IFJR49606
JR49635 
This repair-only interim fix applies to the issue that is addressed in JR48877. JR49635 looks for and repairs issues when importing a snapshot. Although JR48877 repairs most of the issues, you still need interim fix JR49635 if there are more than 1000 participant groups (called “teams” in V8.5). JR49635 repairs the damage done by the issue that is fixed by JR48877 in the event that you encounter the issue before you apply JR48877.
Express: 7.5.1.2-WS-BPM-IFJR49635
Standard: 7.5.1.2-WS-BPM-IFJR49635
Advanced: 7.5.1.2-WS-BPM-IFJR49635
JR49848
Symptom
After you run the BPMSnapshotCleanup command, you can no longer save artifacts that have advanced content in Process Designer in a process application or toolkit from IBM Integration Designer
Prevention 
Interim fix JR49848
Repairs
Interim fix JR49848 or edit the LSW_SCA_CONNECTOR table of the BPMDB database and put the outOfSync value into the LAST_SYNCHRONIZED column for the Advanced Integration Service that is affected
Express: Not applicable
Standard: Not applicable
Advanced: 7.5.1.2-WS-BPM-IFJR49848
JR49928
This repair-only interim fix applies to the issue that is addressed in JR49606. JR49928 looks for and repairs issues when the server starts. You do not require interim fix JR49928 if the interim fixes JR48877 and JR49606 are applied before snapshots are deleted.
Express: 7.5.1.2-WS-BPM-IFJR49928
Standard: 7.5.1.2-WS-BPM-IFJR49928
Advanced: 7.5.1.2-WS-BPM-IFJR49928
JR50444
When the server starts or when you use theBPMInstallOfflinePackage administrative command, you might see the DATAINTEGRITYVIOLATIONEXCEPTION. This fix checks whether there are exposed items before deleting the groups, preventing you from deleting those groups.
Express: 7.5.1.2-WS-BPM-IFJR50444
Standard: 7.5.1.2-WS-BPM-IFJR50444
Advanced: 7.5.1.2-WS-BPM-IFJR50444
JR50460
An exception occurs while history information for a process app or toolkit is being loaded when that history information contains references to deleted snapshots.
Express: 7.5.1.2-WS-BPM-IFJR50460
Standard: 7.5.1.2-WS-BPM-IFJR50460
Advanced: 7.5.1.2-WS-BPM-IFJR50460
JR50667
Deleting a snapshot destroys most smart folders. Smart folders are used for the entire project's track or branch. It is not expected that the smart folders will be deleted when snapshots are deleted.
Express: 7.5.1.2-WS-BPM-IFJR50667
Standard: 7.5.1.2-WS-BPM-IFJR50667
Advanced: 7.5.1.2-WS-BPM-IFJR50667
JR50671
When snapshots are deleted, the corresponding LSW installation table entries are not deleted.    
Express: 7.5.1.2-WS-BPM-IFJR50671
Standard: 7.5.1.2-WS-BPM-IFJR50671
Advanced: 7.5.1.2-WS-BPM-IFJR50671
JR50718
The BPMDeleteSnapshot command proceeds even though there are running instances of stand-alone human services.  
Express: 7.5.1.2-WS-BPM-IFJR50718
Standard: 7.5.1.2-WS-BPM-IFJR50718
Advanced: 7.5.1.2-WS-BPM-IFJR50718
JR50720
The BPMDeleteSnapshot command proceeds even though there business process definition (BPD) instances in another snapshot that point to tasks from the snapshot that is being deleted.
Express: 7.5.1.2-WS-BPM-IFJR50720
Standard: 7.5.1.2-WS-BPM-IFJR50720
Advanced: 7.5.1.2-WS-BPM-IFJR50720
JR50721
The BPMSnapshotCleanup command proceeds even though there are running      
instances of stand-alone human services.          
Express: 7.5.1.2-WS-BPM-IFJR50721
Standard: 7.5.1.2-WS-BPM-IFJR50721
Advanced: 7.5.1.2-WS-BPM-IFJR50721
JR50742
When the server starts or when you use theBPMInstallOfflinePackage administrative command, you might see a DATAINTEGRITYVIOLATION exception.   
Express: 7.5.1.2-WS-BPM-IFJR50742
Standard: 7.5.1.2-WS-BPM-IFJR50742
Advanced: 7.5.1.2-WS-BPM-IFJR50742
JR52327
When you run the BPMSnapshotCleanup command with -ignoreDependency = true, toolkits that have other toolkits and process apps depending on them can be deleted without warning, leading to other problems.
Express: 7.5.1.2-WS-BPM-IFJR52327
Standard: 7.5.1.2-WS-BPM-IFJR52327
Advanced: 7.5.1.2-WS-BPM-IFJR52327

Monday, July 18, 2016

Getting Incorrect SOAP response post BPM deployment - getting Snapshot name for default snapshot

Problem Statement -
There is one anomaly with the BPM deployment, post our deployment and Active Default conversion on environment, Snapshot Namespace is getting appended in the BPM response. This generally doesnt happen for an Active Default snapshot.

Snapshot name should not be appended in the BPM WS response if Snapshot is Default. Problem is sometimes snpashot name also gets appended with response and sometimes it does not.         


Reason for Problem -
For BPM 7.5.1.x release on Process Server. The inbound web service will remember the first call style. And keep in use that style in follow up processing. So in customer case. Before switch the default snapshot customer called this web service with snapshot ID. So that after switch although they can invoke that with default URL. But the namespace with snapshot still in used until next restart and first invoke issued with default URL. The same to no switch but first invoke issued with snapshot URL. This is a limitation for Process Server web service on BPM 7.5.1.x release. It haven't been resolved until the re implemented of the inbound web service with new engine on BPM V8.5. 

Solution -
Testfix for JR56318 is published along with the read me on our ftp server for download.                                                      
                                                                        
Prerequisite APAR JR50211 can be downloaded on fixcentral 

https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm~We
bSphere&product=ibm/WebSphere/IBM+Business+Process+Manager+Advanced&release=All&platform=All&function=aparId&apars=JR50211                      
<https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm~W
ebSphere&product=ibm/WebSphere/IBM+Business+Process+Manager+Advanced&release=All&platform=All&function=aparId&apars=JR50211>

Thursday, June 23, 2016

Reviewing IBM BPM caches

Below is the details of IBM BPM Caching details in Process Admin console.
When you go to IBM BPM Process Admin -> Manage Caches you get a table. Below is the description for that columns.

AbbreviationMeaningExplanation
CACache accessShows the number of times the cache was refreshed and accessed.
UCAUnrefreshed cache accessShows the number of times the cache was accessed but not refreshed.
UCPUnrefreshed cache percentageShows the percentage of uncached access versus cached access for this cache.
Last A.Last accessedShows the most recent access.
StatusStatusShows whether the cache is on or off.
ActionsActionsAllows you to show or reset the cache.


More details about Cache - Must Read

https://www.ibm.com/support/knowledgecenter/en/SS964W/com.ibm.wbpm.admin.doc/topics/perf_tuning_instrument_data.html

https://www.ibm.com/support/knowledgecenter/en/SS964W/com.ibm.wbpm.admin.doc/topics/perf_tuning_other.html

Wednesday, May 18, 2016

Deleting a toolkit snapshot in IBM BPM Environment - 7.5.1.2+


We can delete a toolkit snapshot using BPMDeleteSnapshot command in IBM BPM 7.5.1.2+ environments. First we need to check if all the required IFixes are installed for these commands. that can be checked by following below post.
http://randombpmsolutions.blogspot.co.uk/2016/08/bpmdeletesnapshot-and.html

Toolkits can have dependencies on process app or another toolkit.The toolkits cannot be deleted while those dependencies exist. To delete toolkits, complete the following steps:

  • Deactivate the toolkit snapshot of the dependency with the BPMDeactivate command.
  • Stop the toolkit snapshot with the BPMStop command.
  • Get a list of toolkits and process applications that have dependencies on the toolkit snapshot with the following command: BPMShowSnapshot -showDependents snapshot_name.
  • Starting with the root of the list of reported dependencies, remove each dependency. For each dependency that you want to remove, delete the snapshot of the toolkit or process application.
  • After you resolve all of the dependencies, delete the snapshot with the BPMDeleteSnapshot command.
Below DB query gives the details about all toolkits which does not have nay dependency on and processApp or another Toolkit which makes it a candidate to be deleted.


select  p.SHORT_NAME, s.NAME, s.CREATED_ON from LSW_SNAPSHOT s, LSW_PROJECT p where s.PROJECT_ID = p.PROJECT_ID and

s.SNAPSHOT_ID not in (select d.TARGET_SNAPSHOT_ID from LSW_SNAPSHOT s, LSW_PROJECT_DEPENDENCY d, LSW_PROJECT p  where s.SNAPSHOT_ID = d.TARGET_SNAPSHOT_ID and p.PROJECT_ID = s.PROJECT_ID group by d.TARGET_SNAPSHOT_ID)
and p.IS_TOOLKIT = 'T'


Below DB query gives the list of toolkits having dependency with number of dependent ProcessAll/Toolkit snapshot. this query can be used while selecting the toolkits which can be deleted easily.

select p.SHORT_NAME, s.NAME,d.TARGET_SNAPSHOT_ID, count(d.PROJECT_DEPENDENCY_ID) as dependancy from LSW_SNAPSHOT s, LSW_PROJECT_DEPENDENCY d, LSW_PROJECT p  where s.SNAPSHOT_ID = d.TARGET_SNAPSHOT_ID and p.PROJECT_ID = s.PROJECT_ID group by d.TARGET_SNAPSHOT_ID, s.NAME, p.SHORT_NAME


Wednesday, April 20, 2016

Ad-hoc groups in IBM BPM - when it will be deleted

Ad hoc group - If a team calculates its members using a service, the service returns a set of users and groups. This list of users and groups is then persisted as a reusable entry in the database. Ad hoc groups are immutable. Ad hoc groups can also be created by using a list of users or a list of groups

in IBM BPM Ad-hoc groups are getting created when we use routing option 'List of Users'

this ad hoc groups are temporary in nature and continue to exist until they have at least one available instance.

Ad hoc group will be deleted when all the task related to ad-hoc groups are deleted or reassigned.

Wednesday, March 23, 2016

IBM BPM Task status codes

Below are the Task Status Codes for IBM BPM 

It is mentioned in PS database lsw_task_status_codes table. This is just for ref.



STATUS_ID NAME STATUS_VALUE
1 New 11
2 Received 12
3 Replied 13
4 Forwarded 14
5 Sent 21
6 Actioned 31
7 Closed 32
8 Special 41
9 Deleted 91
10 Alert 61
11 Help_Request 62
12 Comment 63
13 Answered_Help_Request 65
14 Ignored_Help_Request 66
15 Followed 70
16 Tagged 71
17 Collaboration 72