Generating heap dump on live WAS JVM script

Hi All!

There has been quite a while since my last post. This will be short one.
Recently we had to diagnose some memory leak on WebSphere Portal. We allready had post-mortem heap dumps. To better understand what is going on we needed ones from live server to.
There is an easy way to generate heap dump on live WebSphere JVM using scripting. John Pape blogged on this subject recently. In nutshell one needs to connect to server using wsadmin and execute MBean generateHeapDump method.

This task is very easy using following script: Save script above as generateHeapDump.py in your was_profile_root/bin directory and execute following command:

wsadmin.bat -f generateHeapDump.py -lang jython

This should result with new heap dump in was_profile_root directory.
To analyze it download IBM Support Assistant and use IBM Monitoring and Diagnostic Tools for Java™ - Memory Analyzer tool.
Alternative is to directly download MAT from eclipse. Hope this helps to stop those leaks!

WebSphere Application Server SIBus as MQ queue manager

You may find yourself developing (legacy) message driven beans (MDBs) using listener port mechanism to connect to WMQ. If you want to test the MDBs (point-2-point) you need connection to WebShere MQ (at least if you are on WAS v6.x). Typically developers don't have dedicated WMQ so that could be a problem. I have used two workarounds. If you only have WAS on your disposal you could do:

1. (Scenario one) Test MDB using activation specification (JCA 1.5 AS) instead of LP
2. (Scenario two) Use SIBus to mimic WMQ and test as you would if you had WMQ

Scenario one

Both solution requires use of service integration bus (SIBus). If you are okey to use AS resource instead of LP then go ahead and create needed queues on SIBus, JNDI queues, queue conn. factories and AS. You can replace LP with AS in development tool (open ejb-jar.xml in RAD) or when application is deployed in WAS admin console. Because you created default messaging provider resources if you like to test MQ scenario you should have MQ resource set in JNDI also. That is inconvenient so you could use built in SIBus feature called MQ Client Link (not MQ Link, that's something different).

Scenario two

You define new MQ Client Link in your SIBus messaging engine (details here). You need give link name (not so important), define MQ channel name and Queue manager name. Those fields are important and you can enter something like WAS.JMS.SVRCONN and DEVQM respectively. So far so good. BTW this feature is added to SIBus so WAS v5 apps could be migrated to WAS v6 and use default messaging provide instead v5's embedded messaging (details here).
To keep things simple if you already have resources defined to connect to WMQ (queue, queue conn. factory, LP) all you need to do is change queue connection factory. Accordingly to given queue manager name and MQ channel name you should change it so it has properties like one below. 

Picture of queue manager connection factory (important fields shown)

You can find correct port on servers page in admin console. Look for SIB_MQ_ENDPOINT_ADDRESS port.
By default, this is 5558. Everything defined, after server restart, you can use JNDI resources and test MDB. Also note that JNDI mq queues have the same base queue name as the queues defined on SIBus (same as on WMQ).  One can also use JNDI resources from client applications (let say to put message).

WebSphere Process Server JNDI client - JUnit template

Sooner or later you would like to test your code out of container. Of course you would like to use BusinessFlowManager and HumanTaskManager. To make use of them, they should be fetched from JNDI. Problem you may face is that WPS is running with security turned on. Writing JNDI client could be challenging because of JAAS. Official documentation is always missing some vital information. You end up surfing the net looking to some guidance. The same thing I did. The best one I found was blog from Nerdy Eddy. In first part he describes writing client with both SUN JRE and IBM one. To get around security check, EVERYONE group was given permission to use naming service. That will not help using BPC and HTM API. Better solution is one from second part where JAAS login is performed. You can see all the nasty exceptions you can get trying to connect. I got a many of them also. The end result for me was JUnit 4 template.
Here is the code:

import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;

import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.rmi.PortableRemoteObject;
import javax.security.auth.Subject;
import javax.security.auth.login.LoginContext;
import javax.security.auth.login.LoginException;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import com.ibm.bpe.api.ActivityServiceTemplateData;
import com.ibm.bpe.api.BusinessFlowManager;
import com.ibm.bpe.api.BusinessFlowManagerHome;
import com.ibm.task.api.HumanTaskManager;
import com.ibm.task.api.HumanTaskManagerHome;
import com.ibm.websphere.security.auth.WSSubject;
import com.ibm.websphere.security.auth.callback.WSCallbackHandlerImpl;

/**
 * @author mresetar
 *
 */
public class SecureBfmAndHtmIntegrationTestAbstract {
    private static final Logger LOG = Logger.getLogger(SecureBfmAndHtmIntegrationTestAbstract.class.getCanonicalName());
    private static final String IIOP_URL = "iiop://localhost:2810";
    protected InitialContext ic;
    protected BusinessFlowManager bfm;
    protected HumanTaskManager htm;
    protected Subject subject;

    protected String getUsername() {
        return "admin";
    }

    protected String getPassword() {
        return "admin";
    }

    protected <T> T fetchRemoteHomeObject(Class<T> homeClass, String jndiName) throws NamingException {
        LOG.finest(String.format(
                "Trying to fetch remote home object of class %s with JNDI name %s", homeClass.getCanonicalName(), jndiName));
        final Object obj = ic.lookup(jndiName);

        return (T) PortableRemoteObject.narrow(obj, homeClass);
    }

    @Before
    public void init() throws Exception {
        Properties props = new Properties();
        props.put(InitialContext.INITIAL_CONTEXT_FACTORY, "com.ibm.websphere.naming.WsnInitialContextFactory");
        props.put(InitialContext.PROVIDER_URL, IIOP_URL);
        props.put(InitialContext.SECURITY_PRINCIPAL, getUsername());
        props.put(InitialContext.SECURITY_CREDENTIALS, getPassword());
        ic = new InitialContext(props);

        BusinessFlowManagerHome processHome = fetchRemoteHomeObject(BusinessFlowManagerHome.class,
                "com/ibm/bpe/api/BusinessFlowManagerHome");

        HumanTaskManagerHome htmHome = fetchRemoteHomeObject(HumanTaskManagerHome.class, "com/ibm/task/api/HumanTaskManagerHome");

        LoginContext lc = null;
        try {
            final WSCallbackHandlerImpl callbackHandlerImpl = new WSCallbackHandlerImpl(getUsername(), getPassword());

            lc = new LoginContext("WSLogin", callbackHandlerImpl);

        } catch (LoginException le) {
            LOG.log(Level.SEVERE, "Cannot create LoginContext.", le);
            throw le;
        } catch (SecurityException se) {
            LOG.log(Level.SEVERE, "Cannot create LoginContext.", se);
            throw se;
        }

        try {
            lc.login();
            subject = lc.getSubject();
        } catch (LoginException le) {
            LOG.log(Level.SEVERE, "Error while logging in!", le);
            throw le;
        }

        bfm = processHome.create();
        htm = htmHome.create();
    }

    @After
    public void afer() throws NamingException {
        if (ic != null) {
            ic.close();
        }
    }

    @Test
    public void testFetchWaitingActivites() throws Exception {
        WSSubject.doAs(subject, new java.security.PrivilegedAction() {
            public Object run() {
                ActivityServiceTemplateData[] waitingActivities = null;
                try {
                    waitingActivities = bfm.getWaitingActivities("_PI:90030123.e6d8e040.b8c09cf5.5dbc0000");
                    System.out.println(waitingActivities);
                } catch (Exception e) {
                    LOG.log(Level.SEVERE, "", e);
                }
                return waitingActivities;
            }
        });
    }
}

I did changes to sas.client.props and ssl.client.props. To run JUnit I had add VM arguments in Elipse's Run Configurations window to following:

-Djava.security.auth.login.config=file:///D:/IBM/WID62/pf/wps/properties/wsjaas.conf
-Dcom.ibm.CORBA.ConfigURL=file:///D:/IBM/WID62/pf/wps/properties/sas.client.props
-Djava.security.policy=file:///D:/IBM/WID62/pf/wps/properties/security.policy
-Dcom.ibm.SSL.ConfigURL=file:///D:/IBM/WID62/pf/wps/properties/ssl.client.props
-DtraceSettingsFile=TraceSettings.properties -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager  -Djava.util.logging.configureByServer=true

Working directory I changed to bin (build directory) because I wanted to use TraceSettings.properties file for logging. I also changed JRE to WPS JRE. To be able to use BPC API I also added org.eclipse.equinox.registry_3.2.1.R32x_v20060814.jar from WPS plugins directory to class path.

Running the JUnit code above you should get similar to:

Processing Trace Settings File: TraceSettings.properties
[2009.11.08 22:38:00:734 CET] 0000000a WsLoggerConfi W com.ibm.ws.logging.WsLoggerConfigurator getExtensionPointLoggingConfiguration Unable to get extension point - com.ibm.wsspi.extension.logger-properties
[2009.11.08 22:38:00:906 CET] 0000000a SecureBfmAndH 3   Trying to fetch remote home object of class com.ibm.bpe.api.BusinessFlowManagerHome with JNDI name com/ibm/bpe/api/BusinessFlowManagerHome
[2009.11.08 22:38:01:484 CET] 0000000a SASRas        A   JSAS0006I: Security connection interceptor initialized.
[2009.11.08 22:38:01:828 CET] 0000000a SASRas        A   JSAS0001I: Security configuration initialized.
[2009.11.08 22:38:01:875 CET] 0000000a SSLConfig     W   CWPKI0041W: One or more key stores are using the default password.
[2009.11.08 22:38:01:875 CET] 0000000a SSLConfigMana I   CWPKI0027I: Disabling default hostname verification for HTTPS URL connections.
[2009.11.08 22:38:01:906 CET] 0000000a SASRas        A   JSAS0003I: Authentication mechanism: BASICAUTH
[2009.11.08 22:38:02:000 CET] 0000000a SASRas        A   JSAS0007I: Client request interceptor registered.
[2009.11.08 22:38:03:343 CET] 0000000a SecureBfmAndH 3   Trying to fetch remote home object of class com.ibm.task.api.HumanTaskManagerHome with JNDI name com/ibm/task/api/HumanTaskManagerHome
[2009.11.08 22:38:03:796 CET] 0000000a SecureBfmAndH E   
                                 com.ibm.bpe.api.ObjectDoesNotExistException: CWWBA0015E: The object '_PI:90030123.e6d8e040.b8c09cf5.5dbc0000' does not exist; it might have been deleted in the meantime.
 at com.ibm.bpe.framework.BusinessProcessServiceImpl.getWaitingActivities(BusinessProcessServiceImpl.java:29151)
 at com.ibm.bpe.framework.BusinessProcessServiceImpl.getWaitingActivities(BusinessProcessServiceImpl.java:29105)
 at com.ibm.bpe.framework.BusinessFlowManagerBean.getWaitingActivities(BusinessFlowManagerBean.java:16905)
 at com.ibm.bpe.api.EJSRemoteStatelessGenericBusinessFlowManagerEJB_a412961d.getWaitingActivities(Unknown Source)
 at com.ibm.bpe.api._EJSRemoteStatelessGenericBusinessFlowManagerEJB_a412961d_Tie.getWaitingActivities__CORBA_WStringValue(_EJSRemoteStatelessGenericBusinessFlowManagerEJB_a412961d_Tie.java:16226)
 at com.ibm.bpe.api._EJSRemoteStatelessGenericBusinessFlowManagerEJB_a412961d_Tie._invoke(_EJSRemoteStatelessGenericBusinessFlowManagerEJB_a412961d_Tie.java:1131)
 at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:627)
 at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:480)
 at com.ibm.rmi.iiop.ORB.process(ORB.java:512)
 at com.ibm.CORBA.iiop.ORB.process(ORB.java:1571)
 at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2717)
 at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2582)
 at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:62)
 at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
 at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)

Setup java.util.logging in WAS client (unit test)

If you run WebSphere client it is very useful to have client logging set up. Recently I had the same need when I run some JUnit integration test. In that test I instantiated SCA POJO component that had a lot of log statements logged in. Naturally I wanted to have those log message printed out on console when test is run. After short googling I found out that there are two (easiest) options available:

1. Using java.util.logging.LogManager built in functionality reading initial configuration from logging.properties file
2. Enabling trace on client using IBM trace support using trace settings file

I created junit run configuration using both options. There is one issue that I experienced though, I was unable to use eclipse variables like "workspace_loc". I believe that this is because this variable has windows file syntax that is not best understood when reading property file. So when defining run configuration you should use bin (or other output directory) as working directory under Arguments tab. Property file is placed in java source folder.

When using logging.properties I had

-Djava.util.logging.config.file=logging.properties

under VM arguments. My logging.properties has content:

# Add handlers to the root logger.
# These are inherited by all other loggers.
handlers=java.util.logging.ConsoleHandler, java.util.logging.FileHandler

# Set the logging level of the root logger.
# Levels from lowest to highest are
# FINEST, FINER, FINE, CONFIG, INFO, WARNING and SEVERE.
# The default level for all loggers and handlers is INFO.
.level=INFO

# Specify logging levels for specific namespaces.
org.example.level=FINEST

# Configure the ConsoleHandler.
# ConsoleHandler uses java.util.logging.SimpleFormatter by default. 
# Even though the root logger has the same level as this,
# the next line is still needed because we're configuring a handler,
# not a logger, and handlers don't inherit properties from the root logger.
java.util.logging.ConsoleHandler.level=FINEST

# Configure the FileHandler.
# FileHandler uses java.util.logging.XMLFormatter by default. 
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.FileHandler.level=FINEST

# The following special tokens can be used in the pattern property
# which specifies the location and name of the log file.
#   / - standard path separator
#   %t - system temporary directory
#   %h - value of the user.home system property
#   %g - generation number for rotating logs
#   %u - unique number to avoid conflicts
# FileHandler writes to %h/demo0.log by default.
java.util.logging.FileHandler.pattern=E:/temp/org_example_test.log

For other run configuration, using IBM trace settings file under VM settings I have

-DtraceSettingsFile=TraceSettings.properties -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager  -Djava.util.logging.configureByServer=true

  Where TraceSetting.properties has content:

# 5724-I63, 5724-H88 (C) COPYRIGHT International Business Machines Corp. 2000, 2004
# All Rights Reserved * Licensed Materials - Property of IBM


########################################################################
# A property to specify where the output messages and trace (if enabled)
# should be routed. Valid values include stdout or a fully qualified
# file name. If a file is specified but cannot be opened, stdout is used.
# 
# Sample invocations
# To specify stdout - traceFileName=stdout
# To specify a file on NT - traceFileName=c:\\MyTraceFile.log or
#                           traceFileName=c:/MyTraceFile.log
# To specify a file on Unix - traceFileName=/usr/MyTraceFile.log
########################################################################
#traceFileName=e:\\test\\MyTraceFile.log
traceFileName=stdout

########################################################################
# Specify trace strings here. Trace strings take the form of:
# logger={level}={type} where:
#     level = entryexit || debug || event || all
#     type =  enabled || disabled
# examples:
# com.ibm.ejs.ras.SharedLogBase=all=enabled enables all tracing for the
#         single logger created in class com.ibm.ejs.ras.SharedLogBase.
# com.ibm.ejs.*=debug=enabled enables debug tracing for all loggers with
#         names starting with com.ibm.ejs.
#
# Multiple trace strings can be specified, one per line.
########################################################################
org.example.*=all=enabled

Tools for connecting to IBM WAS Service Integration Bus

For the past year me and my colleagues at CROZ were doing a lot of work that included using service integration bus (SIB) and WAS 6.1 as JMS provider. Very often we needed to look are there any messages in queue or send test message. Two basic tools that does the job comes with WAS distribution:

• WAS admin console


• Universal test client (UTC)
  

Admin console can be used for looking queue (and topic) status at runtime. You can use UTC for sending and posting message using JMS API. However admin console is very cumbersome to use because you need to make 10 clicks to see message content and UTC is okay only for basic scenario.

So what other tools you can use to connect to SIB? Preferably stand alone one.

There are actually 3 nice tools from IBM and few from open source community that are very handy. From IBM you have:

1. Service Integration Bus Explorer
2. Service Integration Bus Performance
3. IBM Client Application Tool for JMS

There are also very nice open source tools that can be connected to WAS SIB. I will mention :

1. Hermes JMS
2. Apache JMeter

Service Integration Bus Explorer

Tool that we used the most was SIB Explorer. It is stand alone tool from IBM alphaWorks. Installation is simple and described here. Trickier part is downloading SWT libraries. I have in my swt dir (version number may vary)

• org.eclipse.swt.win32.win32.x86_3.3.2.v3347a.jar
• org.eclipse.swt_3.3.2.v3347.jar
• swt-gdip-win32-3347.dll
• swt-win32-3347.dll

The reason this tool is very handy is that everything is there and you don't need to click too much. If you like to see what messages are in what queue you just open queue points and there you have all queues with their current depth. However for putting new messages in queue it is not so great because you can't add any message property (for instance targetService for SOAP/JMS).

Service Integration Bus Performance

If you need to get some SIB performance related data this tool can be very useful. You can use it to monitor thread pools, queues, topic spaces, communication and data store in near real time (with refresh every 2s). We didn't use it much as we have ITCAM but data it provides can be used for performance tuning. Also you can detect some anomalies like messages with inadequate reliability level.

IBM Client Application Tool for JMS

If you need to experiment a lot with putting messages in queue than you should probably use this tool. It also come from IBM alphaWorks but it doesn't require SWT libraries. However this tool connects to WAS JNDI so you need to have JMS JNDI resources defined. Dan Zrobok has wrote how to connect to the WAS from this tool. Once you connect you can send and receive message (tabs Message Producer and PtoP Message Consumer). Very nice feature is registering new Message Listener. Window pops up for every new message that is deliverd to destination that listener listens to.

Hermes JMS

One more tool you can use for exploring SIB destinations and posting is Hermes JMS. It can be quite handy if you have (or wanna have) messages stored in files and use them for testing. Also you can save messages to file from destinations for later use. Unfortunately Hermes JMS doesn't come with SIB adapter (only with WebSphere MQ one) so you must set it up on your own. It isn't trivial but you can start with downloading "IBM Client for JMS on J2SE with IBM WebSphere Application Server" and installing it. There are also some resources on the web available describing the whole procedure.

Apache JMeter

Another tool that can be very useful is Apache JMeter. With JMeter you can send and receive P2P messages using JMS Point-to-Point sampler. Publish-Subscribe functionality is also available. I was using JMeter for performance testing and also for functional testing. It is really great because you can set up JMS test case and test your messaging infrastructure with it. To set it up connect to the WAS SIB, you will also need IBM Client for JMS on J2SE with IBM WebSphere Application Server. If you need instructions on any of these tools please reply and maybe I'll blog about it.

Hello World! REST service using Jersey and WebSphere Application Server v6.1 (WAS)

I was wondering how easy it is to create Hello World REST service using Jersey and deploying it to WAS 6.1. Long story short: It is really easy. So here are the steps:

1. Download jersey-archive-1.0.2.zip from http://jersey.dev.java.net/.
2. Extract jersey-archive-1.0.2.zip.
3. Go to your favorite tool for WAS development. Mine was Rational Software Architect 7.5 (RSA) but the same steps works for 7.0.
4. Create new Dynamic web project (HelloWorldRest). In RSA 7.5 you should choose WAS 6.1 as your target platform. Also choose Servlet spec 2.4.
5. Copy jersey jars from lib folder. That’s the folder where you extracted jersey archive in step 2. Put them in the WEB-INF/lib folder.
6. Now you must add Servlet configuration and mapping. You can do this through wizard or you can copy the following in web.xml just after display-name element:

   
    Jersey Servlet
    ServletContainer
    ServletContainer
    com.sun.jersey.spi.container.servlet.ServletContainer
   
   
    ServletContainer
    /resources/*
   
   

   With this in place we have configured Jersey servlet. It will serve REST services with pattern resources in URI.
7. Now we can create service implementation class. We are creating simple POJO class (in JAX-RS term Root Resource Class) with one method like one below:

   /**
    * 
    */
   package org.example;
   
   import javax.ws.rs.GET;
   import javax.ws.rs.Path;
   import javax.ws.rs.Produces;
   import javax.ws.rs.QueryParam;
   
   @Path("/helloworld")
   public class HelloWorldResource {
   
    @GET
    @Produces("text/plain")
    public String sayHello(@QueryParam("world") String world) {
     return "Hello " + world;
    }
   
   }
   

   The interesting bits are four annotations:
   1. @Path("/helloworld") – this annotation exposes methods in this class through helloword path. Annotation value is the thing that we will add after resources in URI to call this service
   2. @GET – annotated method (sayHello) is available through HTTP GET method
   3. @Produces("text/plain") – method will return plain text
   4. @QueryParam("world") – with this annotation we are mapping URI parameter world to method parameter world
8. Deploy project on server!

And that’s it. Now we can test service using browser. If you enter following URI: http://localhost:9080/HelloWorldRest/resources/helloworld?world=World! you should get response in clear text, as expected: Hello World!. Next we can create a simple stand alone client for this service. I have:

1. Created a new Java project called HelloWorldRestJava.
2. Created new folder called lib.
3. Copied jars from WEB-INF/lib to lib folder. Added them to project classpath.
4. Created new class with main method:

   package org.example;
   
   import com.sun.jersey.api.client.Client;
   import com.sun.jersey.api.client.WebResource;
   
   public class HelloWorldRestClient {
   
    public static void main(String[] args) {
     Client client = new Client();
     WebResource webResource = client.resource("http://localhost:9080/HelloWorldRest");
     String response = webResource.path("resources").path("helloworld")
       .queryParam("world", "World!").get(String.class);
     System.out.println("Response: " + response);
    }
   }
   

5. Run the client as Java application.

If you do the same you should see Response: Hello World! in the console. Now when you saw how easy it is, you can continue to play with Jersey.