Showing posts with label outofmemory. Show all posts
Showing posts with label outofmemory. Show all posts

Monday, June 1, 2015

IBM BPM PC servre throwing OOM error and High CPU usage after 7.5.1.2 upgrade from 7.5.1.1

Problem::
Getting OutofMemory error and High CPU usage of applicaiton server just after upgrading to IBM BPM 7.5.1.2
We have raised PMR to IBM for same and solution we got was as below.

Solution ::
After upgrading BPM 7.5.1.1 to 7.5.1.2 all snapshots will be some kind 
of 'fixed'. Hence, those are loaded in the heap and could allocate a   
lot of memory, depending on the amount. This operation is called       
UserAttributeDefinitionRepair and is an one time operation. So when it 
is completed once, it is not executed again. If the system crashes     
during this operation, BPM tries to re-run it again and will probably  
fail again.                                                            
                                                                        
One can verify, that this process is running, doing the following: To  
check if the 'UserAttributeDefinitionRepair' is still running, please  
check table LSW_SYSTEM and look for the PROPKEY                        
'UserAttributeDefinitionRepair'. If its PROPVALUE is not COMPLETED,    
than the process is not finished yet.                                  
                                                                        
In order to regulate the number of loaded snapshots one can adjust the 
Branch Context Cache Size. We suggested to use the following action    
plan:                                                                  
                                                                        
1) Change the branch-context-max-cache-size to 64 and monitor the      
system, if OOM occurs (testing now)                                    
2) Change the branch-context-max-cache-size to 32 and monitor the      
system, if OOM occurs,                                                 
3) Install the recommended fixes for BPMSnapshotCleanup and dependency 
ifixes on top of 7.5.1.2.                                              
 - 7512 tab                                                             
4) Run BPMSnapshotCleanup command as instructed to clean up unneeded   
 snapshots.                                                            
main.doc/managinglib/topic/managing_snapshots_i.html?cp=SSFPJS_7.5.1   
5) After clean up database, start server and monitor the system.       
                                                                        
Steps 1 and 2 might be the most crucial. Reducing the cache size causes
to be not that much Snapshots be loaded in the heap at once. As a      
result, the whole process will take longer, but it will also not fail. 
This value can be adjusted after the process is once finished (see     
above: PROPVALUE shows COMPLETED then).                                
                                                                        
Steps 3 to 5 are recommended in order to remove old, unused snapshots, 
that influence the whole process. Keeping the system clean is a good   
approach in general. Those cleaning operations are just available on   
BPM 7.5.1.2 and not 7.5.1.1. See:                                      
ofmemory-regarding-poversions/                                         
                                                                        
In order to prevent the whole problem, you can use the following       
process for further migrations from BPM 7.5.1.1 to BPM 7.5.1.2:        
                                                                        
1) Identify the process apps and toolkits which have the largest numbers
of snapshots by running DB queries. Take note.                         
2) Repeat the following steps to each process app or toolkit will be   
 cleaned.                                                              
    a) Create a new snapshot;                                          
    b) Export snapshot created in a);                                   
    c) Archive and delete process app or toolkit;                      
    d) Re-import snapshot exported in b).                              
                                                                        
    Note: If multiple versions of toolkit snapshots are referenced,    
please do the following:                                               
    a1) Export all toolkit snapshots to keep one by one;               
    b1) Archive and delete the toolkit;                                 
    c1) Import toolkit snapshot in the desired sequence, import the    
oldest snapshot first. Do NOT import them with dependent process apps  
since it may potentially messed up the toolkit snapshot levels.         

Monday, March 11, 2013

CPU Starvation and OOM Error occured while access Role Binding Tab

Problem::
Problem occurs while accessing Role Binding tab on the Process Server Environment where there are lots of projects and snapshots deployed.
We have faced CPU Starvation and OutOfMemory Error and sometime server failure while we access Role Binding tab in one of our very heavily used server.

Explaination ::
In IBM BPM 7.5 version, whenever you open Role Binding or Envrionment Variable Tab, Every 15 second product send a fresh request to get latest deatails regarding Role Binding and Environment Variables.
If system is having lots of data and request does not completes in 15 second, then another request will also be there in system. Each request needs Memory to hold data and CPU to process data. After some time there will be lots of request pending and processing in system that it will force JVM to throw OOM Exception and CPU to starve.

Solution ::
JR48822  is available for IBM BPM 7.5.1, IBM BPM 8.5.5 and IBM BPM 8.0.1.2

this is already included in next fixpack for all releases which area as given below,
Version 7.5.1 Fix Pack 2 for the IBM Business Process Manager products
Version 8.5 Refresh Pack 5 for the IBM Business Process Manager products