Goodbye 2007

December 31, 2007

Well, today is the final day of 2007. As a famous blogger I know it is important for me to say a few words. For posterity and all.

There, I feel better now.

Happy New Year.


Schedule your own Java thread dumps

December 29, 2007

So, if you ever tried to figure out what a Java program that appears to be hung is doing you are probably very familiar with the Java thread dump feature. Basically you send a signal to the JVM which responds by, essentially, writing a stack trace of each thread in the JVM to the standard output device. In fact, a thread dump contains more useful information that just a stack trace, it also show the state of each thread (i.e. runnable, waiting, etc) and which Java monitors (synchronized locks) are owned and/or being waited on.

Here is a sample snippet of a thread dump:

Full thread dump Java HotSpot(TM) Client VM (1.5.0_07-b03 mixed mode):

"Timer-5" daemon prio=10 tid=0x092d5720 nid=0x73 in Object.wait() [0x9b52f000..0x9b52fd38] 
at java.lang.Object.wait(Native Method) - waiting on <0xa2a4b978> (a java.util.TaskQueue) 
at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0xa2a4b978> (a java.util.TaskQueue) 
at java.util.TimerThread.run(Timer.java:462)

"Timer-4" prio=10 tid=0x0925d418 nid=0x72 in Object.wait() [0x9b4ed000..0x9b4edab8] 
at java.lang.Object.wait(Native Method) - waiting on <0xa2a49570> (a java.util.TaskQueue) 
at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0xa2a49570> (a java.util.TaskQueue) 
at java.util.TimerThread.run(Timer.java:462) 

As you can see a thread dump contains a lot of very useful information.

The method used to “request” a thread dump is to send a signal to the running JVM. In Unix this is the SIGQUIT signal which may be generated via either:

kill -3 <pid>

or

kill -QUIT <pid>

where <pid> is the process id the the JVM. You can also enter Ctrl-\ in the window where the JVM is running.

On Windows a thread dump is requested by sending a Ctrl-Break to the JVM process. This is pretty simple for foreground windows but requires a program (akin to Unix kill) to be used for JVMs running in the background (i.e. services).

The problem with requesting a thread dump is that it requires manual intervention, i.e. someone has to enter the kill command or press the Ctrl-Break keys to generate a thread dump. If you are having problems with your production site in the wee hours of the morning your support staff probably won’t appreciate getting out of bed to capture a few dumps for you. In addition, a single thread dump is not as useful as a series of dumps taken over a period of time. With a single dump you only get a snapshot of what is happening. You might see a thread holding a monitor that is causing others thread to block but you have no idea how long that condition has existed. The lock might have been released a millisecond after the dump was taken. If you have, say, 5 dumps taken over 20 minutes and the same thread is holding the monitor in all of them then you know you’ve got a problem to investigate.

The solution I’m going to propose makes use of JNI to request a thread dump of the current JVM and capture that output to a file which may be time stamped. This allows dump output to be segregated from everything else the JVM is sending to STDOUT.

Before you invest any more time in this article let me state that the solution I’m going to present here only partially works for windows. It is possible to programmatically request a thread dump under Windows but due to a limitation in Win32, the Microsoft C runtime, or both the capture to a separate file does not work. Even though Win32 provides APIs for changing the file handles used for STDOUT/STDERR, changing them after a process has started executing does not seem to make any difference. If you do all your Java work on Windows, you’ve been warned – don’t read to the end and then send me a nasty email saying I wasted your time!

Ok, the first thing we need to do is create a Java class that will serve as an interface to our native routine that captures thread dumps:

package com.utils.threaddump;

public class ThreadDumpUtil
{
public static int performThreadDump(final String fileName)
{
return(threadDumpJvm(fileName));
}

private static native int threadDumpJvm(final String fileName);

static
{
System.loadLibrary("libthreaddump");
}
}

This class loads a native library called libthreaddump when it is loaded and then exposes a static method to request a thread dump from Java code specifying the name of the file that should contain the captured dump.

Running this file through the javah tool generates a C header named com_utils_threaddump_ThreadDumpUtil.h which is used to help build our native routine.

The C code for the Unix variant follows:

#include <signal.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <string.h>
#include <errno.h>
#include "com_utils_threaddump_ThreadDumpUtil.h"

#define FILE_STDOUT             1
#define FILE_STDERR             2

JNIEXPORT jint JNICALL 
Java_com_nm_utils_threaddump_ThreadDumpUtil_threadDumpJvm(JNIEnv *env, jclass clazz, jstring fileName)
{
    /* get my process id */
    pid_t pid = getpid();
    
    /* open the file where we want the thread dump written */
    char* fName = (char*) (*env)->GetStringUTFChars(env, fileName, NULL);
    if (NULL == fName)
    {
        printf("threadDumpJvm: Out of memory converting filename");
        return((jint) -1L);
    }
    
    int fd = open(fName, O_WRONLY | O_CREAT, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
    if (-1 == fd)
    {
        printf("threadDumpJvm: Open of file %s failed: %d[%s]\n", fName, errno, strerror(errno));
	    (*env)->ReleaseStringUTFChars(env, fileName, fName);
        return((jint) -2L);
    }
    
    /* redirect stdout and stderr to our thread dump file */
    int fdOut = dup(FILE_STDOUT);
    int fdErr = dup(FILE_STDERR);
    dup2(fd, FILE_STDOUT);
    dup2(fd, FILE_STDERR);
    close(fd);
    (*env)->ReleaseStringUTFChars(env, fileName, fName);
    
    /* send signal requesting JVM to perform a thread dump */
    kill(pid, SIGQUIT);
    
    /* this is kind of hokey but we have to wait for the dump to complete - 10 secs should be ok */
    sleep(10);
    
    /* replace the original stdout and stderr file handles */
    dup2(fdOut, FILE_STDOUT);
    dup2(fdErr, FILE_STDERR);
    return((jint) 0L);
}

Following are the compile command lines I’ve used on a couple of Unix systems to build this dynamic library:

Mac OSX:
gcc -o liblibthreaddump.dylib -dynamiclib -I. -I$JAVA_HOME/include -L/usr/lib -lc libthreaddump_unix.c

Solaris:
gcc -o liblibthreaddump.so -G -I/$JAVA_HOME/include -I/$JAVA_HOME/include/solaris libthreaddump_unix.c -lc

Here is the C code for the Windows version of the native library:

#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include "com_nm_utils_threaddump_ThreadDumpUtil.h"

#define FILE_STDOUT             1
#define FILE_STDERR             2

JNIEXPORT jint JNICALL 
Java_com_nm_utils_threaddump_ThreadDumpUtil_threadDumpJvm(JNIEnv *env, jclass clazz, jstring fileName)
{
	auto	HANDLE		fd;
	auto	HANDLE 		fdOut;
	auto	HANDLE		fdErr;
	auto	long		retValue = 0L;
	auto	char* 		errText = "";
	auto	DWORD 		pid = GetCurrentProcessId();
	    
    /* open the file where we want the thread dump written */
    char* fName = (char*) (*env)->GetStringUTFChars(env, fileName, NULL);
    if (NULL == fName)
    {
        printf("threadDumpJvm: Out of memory converting filename");
        return((jint) -1L);
    }
	
	fd = CreateFile((LPCTSTR) fName, GENERIC_WRITE, FILE_SHARE_WRITE, 
					NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
    if (INVALID_HANDLE_VALUE == fd)
    {
        printf("threadDumpJvm: Open of file %s failed: %ld\n", fName, (long) GetLastError());
	    (*env)->ReleaseStringUTFChars(env, fileName, fName);
        return((jint) -2L);
    }

    /* redirect stdout and stderr to our thread dump file */
    fdOut = GetStdHandle(STD_OUTPUT_HANDLE);
    fdErr = GetStdHandle(STD_ERROR_HANDLE);
    printf("fdOut=%ld fdErr=%ld\n", (long) GetStdHandle(STD_OUTPUT_HANDLE), (long) GetStdHandle(STD_ERROR_HANDLE));
    if (!SetStdHandle(STD_OUTPUT_HANDLE, fd))
    	printf("SetStdHandle failed: %ld\n", (long) GetLastError());
    	
    SetStdHandle(STD_ERROR_HANDLE, fd);
    
    printf("fdOut=%ld fdErr=%ld\n", (long) GetStdHandle(STD_OUTPUT_HANDLE), (long) GetStdHandle(STD_ERROR_HANDLE));
    if (0 == GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, 0))  // pid fails here????
    {
    	retValue = (long) GetLastError();
    	errText = "Generate CTRL-BREAK event failed";
   	}
	else
	{
	    /* this is kind of hokey but we have to wait for the dump to complete - 10 secs should be ok */
    	Sleep(10000L);
    }

    printf("This is a test message\n");
    
    /* replace the original stdout and stderr file handles */
    SetStdHandle(STD_OUTPUT_HANDLE, fdOut);
    SetStdHandle(STD_ERROR_HANDLE, fdErr);
	CloseHandle(fd);
    (*env)->ReleaseStringUTFChars(env, fileName, fName);
    
    if (0L != retValue)
    {
    	printf("threadDumpJvm: Error generating thread dump: %s\n", errText);
    }
        
	return((jint) retValue);
}

Remember – the file capture will not work here, it simply creates an empty file and the thread dump goes to the original STDOUT device.

Here is the command I used to create a Windows DLL using Microsoft Visual C++ 6.0:

cl -I. -I%JAVA_HOME%\include -I%JAVA_HOME%/include/win32 -LD libthreaddump_win32.c -Felibthreaddump.dll

That’s it. All the tools needed to request a thread dump any time you like. I used these tools to diagnose problems with an ATG application cluster to research problems being reported by the ATG ServerMonitor component. The ATG ServerMonitor issues warning and error log message for various reasons like the JVM being low on memory or a application request thread executing for an extended period of time. In a future post I’ll discuss how I extended the ATG ServerMonitor to capture thread dumps under these conditions.


Amazon Kindle Computer Books

December 29, 2007

Dummies

I just checked out the Amazon site for the best selling Kindle edition computer books. Eight of the top ten are “for Dummies” books. WTF?


Living Will

December 29, 2007

Livingwill

Last night my sister and I were sitting in the den and I said to her, ‘I never want to live in a vegetative state, dependent on some machine and fluids from a bottle to keep me alive.
That would be no quality of life at all, If that ever happens, just pull the plug.’
So she got up, unplugged the computer, and threw out my wine.
She’s such a bitch.


Just Wondering

December 20, 2007

Image001-1


W Can’t Be Shocked!

December 20, 2007

Ok, as Rod Serling used to say, “Submitted for your consideration”. So, consider the shocker site (visual instructions below):
 Blog Wp-Content Images Shocker-Hand
And then check out our fearless leader Gettin’ Down!
  Pnjfztinpky R2Px62--Cci Aaaaaaaacjk Krmwu-Vjrde S1600 Shocker-35838
Nuff said.


Rock on Colbie

December 16, 2007

200712160843Of late I’ve been listening to some Colbie Caillat and I have to tell you, this girl has talent. Oh sure, she’s cute and all but close your eyes and listen to her sing. Not bad, eh? Colbie just released her first studio album, Coco, which in some parts of Texas means horse. That’s probably not why she picked that title but, it might be.

Give a listen and see what you think.