After IBM BPM 7511 upgrade old task started from deactivated snapshot failing

Recently we have upgraded to IBM BPM 7511 versino from 7510 and after upgrade we monitored that instances are failing for all the sanpshots which are deactivated. That means if you have started an instance from snpashot 1.0 and then deactivated that snpashot and instance migration was also not performed. All the running instances having referance to snpashot 1.0 will fail.

After investigation we found that it is due to new properties added in 7511.
Business Process Manager Version 7.5.1 Fix Pack 1 added the following new properties:

1. block-deactivated-snapshot-favorite-progression block-deactivated-snapshot-task-progression
2. 
   

These two properties exists in the "99Local.xml" configuration file, are defined, by default, with value "True". This default value for these properties will stop in-flight tasks on the deactivated snapshot.

If you want to process or excute in-flight tasks or instances on deactivated snapshots, you have to change this property to "False"

Note: You do not want to modify the 99Local.xml configuration file directly. Instead, use the 100Custom.xml file to override the values using following steps

To execute these in-flight tasks, complete the following steps: 

1) Locate the 100Custom.xml file.

2) Add the following XML code to the file:

 <properties>
   <server merge="mergeChildren">
     <block-deactivated-snapshot-favorite-progression merge="replace">false
     </block-deactivated-snapshot-favorite-progression>
     <block-deactivated-snapshot-task-progression merge="replace">false
     </block-deactivated-snapshot-task-progression>
   </server>
 </properties>

3) Restart the server.

UCA or Scheduled taks from older version are not beig deactivated - IBM BPM 7.5.1

We have faced this problem earlier and it was very BIG problem for us as the UCA was involved in some very sensitive tasks.

Problem Statement ::
While you install new snapshot for any project and deactivate older snapshot, all older UCAs(scheduled) should be deactivated and only UCAs from new version should be there in Event Manager Queue. but in IBM BPM 7.5.1 this is not the case, All UCAs for older snpashot are still there in Event Manager queue as well as All UCAs from newer version. so basically we have multiple Schduled UCAs for doing same work and executing multiple times.
Earlier we thought like UCAs are executing multiple time but while investigating we realized that its older UCAs which are still there in EM queue.

Solution::
This problem is fixed by IBM and they provided an Ifix for solving this issue.
I-Fix is also included in IBM BPM  7.5.1.1 Fix pack

For more information, 
http://www-01.ibm.com/support/docview.wss?uid=swg1JR41966

How to stop a service which is in loop or taking long time to execute in IBM BPM

Sometimes on Development or test a service might get stuck in loop or take too much time to execute, If it takes unusual time to execute you need to halt it by force. Follow below steps to halt or force stop a service.

• log on to ProcessAdmin console as admin/tw_admin.
• Go to Monitoring -> Process Monitor
• You'll see a page displaying summary of all the services. On the top of the page, you'll see "Process", "Services".
• Click on Services
• If you service is still running, you'll find it under "Active Services Currently Executing"
• Click on the service and you will be routed to a web page with the serive detials
• You'll also notice a button "halt service". This should stop your service.
• If this does not work, the only option I can think of is to restart Lombardi(IBM BPM app node) unless there is webapi call to teminate tasks (which I am not very sure of).

Disabling the tracking at cluster level when not required

Few days before we were dealing with an interesting problem in which we were getting db lock wait issues on LSW_PERF_DATA_TRANSFER in ProcessDB. While investigating we found that table(in process server DB) was used for tracking purpose and when we asked all development teams they said they are not using tracking at all.

When we check the applications we found out even though project team were saying that they are not using tracking at all, "Enable Tracking" and "Auto Tracking" option were checked in almost all BPDs. There were lots of project executing that it was not possible to make everyone uncheck that options immediately and get new version deployed. As no project wants to use tracking we decided to disable tracking at farm level.

We have disabled it by changing configuration in dmgr and syncing it with all nodes.

Details are there in below link,

http://www-01.ibm.com/support/docview.wss?uid=swg21611013

NOTE :: Do not change "00Static.xml" file. Always change/add configuration details in "100Custom.xml" file

Text format problem in text input box IBM BPM 7.5.1

One of our testing team came back with an issue few weeks before. They were entering some text in first screen and bind it to a variable and use the same variable to populate the next screen. The problem they were facing was all the formatting was removed in next screen. i.e. if they input text with new line(multiple line in text area) in next screen they were seeing everything in one line.

Apparently this is the known issue with 7.5.1 and one fix is available for this.

http://www-01.ibm.com/support/docview.wss?uid=swg1JR42355 

This is also included in 7.5.1.1 upgrade.

OutOfMemory due to high number of snapshots installed in Process Server

How it started.

one of our Process Server farm was crashing after generating lots of heap dumps. CPU usage was almost 100% continuously and AppTarget Node ran out of heap. This is for 7.5.1.0 version.

Investigation
Using the IBM Heap Analyzer tool, we observed that we had a single instance of  com.lombardisoftware.server.ejb.persistence.versioning.BranchManager worth of 4GB+

After some digging we found below iFix.

http://www-01.ibm.com/support/docview.wss?uid=swg1JR42522

JR42522: BusinessDataAliasCache performance and memory leak fix for IBM Business Process Manager

Solution::

Upgrade to 7.5.1.1(we took this road) or install iFix provided in above link

Get list of toolkit which is not being used in any Process App

We are using IBM BPM 7.5.1.1 and having lost of Process Apps(100+) and Toolkits(50+)

We need to do housekeeping in order to make our Development environment fast. We need to figure out which all Process Apps or Toolkits are not used and we can safely Archive.

I have created one query which get list of all toolkits which are not used by any process app as dependency.

Below is the query.

select NAME, SHORT_NAME, LSW_PROJECT.LAST_MODIFIED from LSW_PROJECT where IS_TOOLKIT='T' and SHORT_NAME not in (select NAME from LSW_PROJECT_DEPENDENCY group by NAME) order by LSW_PROJECT.LAST_MODIFIED

We distributed the output of above query to all development teams and after their approval(basically if they do not raise any concern in 7 days) we Archive toolkits. 

Multiple System toolkit in IBM BPM 7.5

We recently experienced a problem regarding dependency of  multiple system toolkit in same process app.

we were getting null pointer exception whenever we were opening that Process App in Process Designer. The solution is to apply APAR JR41389 or updating the DB and deleting toolkit ref.

We have solved this by updating the DB tables and deleting it from process app.

Open the below link for details.

http://www-01.ibm.com/support/docview.wss?uid=swg21576147

Custom Saved Search in IBM BPM 7.5.1 and 8

Follow below link to create a custom Saved seach in IBM BPM 8.0 and 7.5.1

Instruction for 7.5.1 are after 8.0 part. 

http://www.ibm.com/developerworks/bpm/bpmjournal/1209_cline/1209_cline.html

To migrate or not to migrate – that is the question

Few days ago we have been asked about the risk factor in migrating in-flight instances in IBM BPM 7.5.1

After few discussion, calls and google search we came to conclusion that in-flight data migration is a good but risky approach as instances might fail during migration. To lower risk we have recommended below steps.

- Regressive testing of instance migration from snapshot A to snapshot B in lower environment.
- If possible make prod like environment by restoring prod DB in lower environment and do instance migration to new snapshot before going live.
- Take backup of LIVE DB before doing data migration.

And most important thing
- Take advice from actual development team if they have removed any variable, task or component where token can be while migration
Also, are they comfortable about their Process Model and Process data structure as described below
- Process Modeling should be “Encapsulate what varies”
- Process Data Structure should be “Open for extension, closed for modification”
find more information on Process Model and Process data Structure @
http://bpmstech.blogspot.co.uk/2011/05/lombardi-best-practices.html
Other helpful links,
http://www-01.ibm.com/support/docview.wss?uid=swg21615128
http://www-01.ibm.com/support/docview.wss?uid=swg27038196
http://www-01.ibm.com/support/docview.wss?uid=swg21439774

CPU Starvation and OOM Error occured while access Role Binding Tab

Problem::

Problem occurs while accessing Role Binding tab on the Process Server Environment where there are lots of projects and snapshots deployed.

We have faced CPU Starvation and OutOfMemory Error and sometime server failure while we access Role Binding tab in one of our very heavily used server.

Explaination ::

In IBM BPM 7.5 version, whenever you open Role Binding or Envrionment Variable Tab, Every 15 second product send a fresh request to get latest deatails regarding Role Binding and Environment Variables.

If system is having lots of data and request does not completes in 15 second, then another request will also be there in system. Each request needs Memory to hold data and CPU to process data. After some time there will be lots of request pending and processing in system that it will force JVM to throw OOM Exception and CPU to starve.

Solution ::

JR48822  is available for IBM BPM 7.5.1, IBM BPM 8.5.5 and IBM BPM 8.0.1.2

this is already included in next fixpack for all releases which area as given below,

Version 7.5.1 Fix Pack 2 for the IBM Business Process Manager products
Version 8.5 Refresh Pack 5 for the IBM Business Process Manager products

Download Version 8.0.1 Fix Pack 3 for the IBM Business Process Manager products