Problem::
Getting OutofMemory error and High CPU usage of applicaiton server just after upgrading to IBM BPM 7.5.1.2
We have raised PMR to IBM for same and solution we got was as below.
Solution ::
After
upgrading BPM 7.5.1.1 to 7.5.1.2 all snapshots will be some kind
of 'fixed'.
Hence, those are loaded in the heap and could allocate a
lot of
memory, depending on the amount. This operation is called
UserAttributeDefinitionRepair
and is an one time operation. So when it
is
completed once, it is not executed again. If the system
crashes
during this
operation, BPM tries to re-run it again and will probably
fail
again.
One can
verify, that this process is running, doing the following: To
check if
the 'UserAttributeDefinitionRepair' is still running, please
check table
LSW_SYSTEM and look for the
PROPKEY
'UserAttributeDefinitionRepair'.
If its PROPVALUE is not COMPLETED,
than the
process is not finished
yet.
In order to
regulate the number of loaded snapshots one can adjust the
Branch
Context Cache Size. We suggested to use the following
action
plan:
1) Change
the branch-context-max-cache-size to 64 and monitor
the
system, if
OOM occurs (testing
now)
2) Change
the branch-context-max-cache-size to 32 and monitor
the
system, if
OOM
occurs,
3) Install
the recommended fixes for BPMSnapshotCleanup and dependency
ifixes on
top of
7.5.1.2.
-
7512
tab
4) Run
BPMSnapshotCleanup command as instructed to clean up unneeded
snapshots.
main.doc/managinglib/topic/managing_snapshots_i.html?cp=SSFPJS_7.5.1
5) After
clean up database, start server and monitor the
system.
Steps 1 and
2 might be the most crucial. Reducing the cache size causes
to be not
that much Snapshots be loaded in the heap at once. As
a
result, the
whole process will take longer, but it will also not fail.
This value
can be adjusted after the process is once finished
(see
above:
PROPVALUE shows COMPLETED
then).
Steps 3 to
5 are recommended in order to remove old, unused snapshots,
that
influence the whole process. Keeping the system clean is a
good
approach in
general. Those cleaning operations are just available on
BPM 7.5.1.2
and not 7.5.1.1. See:
ofmemory-regarding-poversions/
In order to
prevent the whole problem, you can use the
following
process for
further migrations from BPM 7.5.1.1 to BPM
7.5.1.2:
1) Identify
the process apps and toolkits which have the largest numbers
of
snapshots by running DB queries. Take
note.
2) Repeat
the following steps to each process app or toolkit will be
cleaned.
a)
Create a new
snapshot;
b)
Export snapshot created in a);
c)
Archive and delete process app or
toolkit;
d)
Re-import snapshot exported in
b).
Note:
If multiple versions of toolkit snapshots are
referenced,
please do
the
following:
a1)
Export all toolkit snapshots to keep one by
one;
b1)
Archive and delete the toolkit;
c1)
Import toolkit snapshot in the desired sequence, import
the
oldest
snapshot first. Do NOT import them with dependent process apps
since it
may potentially messed up the toolkit snapshot
levels.
