1 Prepare Java for Troubleshooting

Set up the Java environment and command-line options to enable gathering relevant data for troubleshooting.

1. Update the Java version: Use the latest Java version to avoid spending time on troubleshooting issues in Java that were fixed. Often, a problem caused by a bug in the Java runtime is fixed in the latest update release. Working with the latest Java version helps avoid some known and common issues.
2. Set up the Java environment to debug: Consider the following scenarios while setting up a bigger Java application, starting an application with a launcher script, or running distributed Java on several machines.
   1. Make it easy to change the Java version: Using the latest Java version helps avoid many runtime issues. If your application starts by running a script, ensure that you have to update the Java path in only one place. If you run in a distributed system, then think about easy ways to change the Java versions across all of the machines.
   2. Make it easy to change the Java command-line options: Sometimes, while troubleshooting, you may want to change Java options; for example, to add a verbose output, to turn off a feature, or to tune Java for better performance. Prepare the systems for these changes.
      In a Java application that is running remotely, for example in a testing framework or a cloud solution, ensure that you can still change the Java flags easily. Sometimes, the application takes command-line parameters, or you may want to try a flag quickly to reproduce a problem. Prepare the systems to make these changes easy.

Set up JVM options and flags to enable gathering relevant data for troubleshooting.

1. Enable core files: If Java crashes, for example due to a segmentation fault, the OS saves to disk a core file (complete dump of the memory). On Linux and Solaris, core files are sometimes disabled by default. To enable core files on Linux/Solaris, it is usually enough to run the ulimit -c unlimited before starting the application command. Some systems may have different ways to handle these limits.

   Note:

   The core files take up a lot of disk space, especially when run with a large Java heap.

   To decide whether to enable core files, consider what you would do if you had a crash in your system. Would you want to see a core file? Many Java users won't have much use for a core file. However, if you would want to debug a possible crash either in a native debugger such as gdb or by using the Serviceability Agent, then ensure that you enable core files before the starting the application.

   Many times, crashes are hard to reproduce; therefore, enable core files before the starting the application.

2. Add -XX:+HeapDumpOnOutOfMemoryError to the JVM flags: The -XX:+HeapDumpOnOutOfMemoryError flag saves a Java Heap dump to disk if the applications runs into an OutOfMemoryError.

   Like core files, heap dumps can be very large, especially when run with a big Java heap.

   Again, think about what you would do if the application runs into an OutOfMemoryError. Would you want to inspect the heap at the time of the error? In that case, turn flag by default so that you get this data if the application runs into an unexpected OutOfMemoryError.

3. Run a continuous Java flight recording: The Java Flight Recorder (JFR) is a commercial feature. You can use it for free on developer desktops and laptops, and for evaluation purposes in test, development, and production environments. However, to enable JFR on a production server, you must have a commercial license.

   Set up Java to run with a continuous flight recording. Continuous flight recordings are a circular buffer of JFR events. If the application runs into an issue, you can dump the data from the last hour of the run. The JFR events can be helpful to debug a wide range of issues from memory leaks to network errors, high CPU usage, thread blocks, and so on.

   The overhead of running with a continuous flight recording is very low. See How to Produce a Flight Recording for producing a continuous Java Flight Recording.

4. Add -verbosegc to the JVM command-line: The flag -verbosegc logs basic information about Java Garbage Collector. This log helps you find the following:

   • Does garbage collection run for a long time?

   • Does the free memory decrease over time?

   The garbage collector log helps diagnose issues when the application throws an OutOFMemoryError or the application runs into performance issues; therefore, turning on the -verbosegc flag by default helps troubleshoot issues.

   Note:

   Use log rotation so that an application restart doesn't delete the previous logs. Since JDK7, the flags UseGClogFileRotation and NumberOfGCLogFiles can be used to set up for log rotation. For a description of these flags, see Debugging Options for Java HotSpot VM.
5. Print Java version and JVM flags: Before filing a bug on Java or seeking help from a forum, have the basic information handy in the log files. For example, it's helpful to print the Java version and the JVM flags used.

   If your application starts with a script, run java -version to print the Java version and print the command line before executing it. Another alternative is to add -XX+PrintCommandLineFlags and -showversion to the JVM arguments.

6. Set up JMC JMX for remote monitoring: JMX can be used to connect to a Java application remotely using tools such as Mission Control or Visual VM. Unless you can run these tools on the same machine that is running your application, setting this up can be helpful later on to monitor the application, send diagnostic commands, manage flight recordings, and so on. There is no performance overhead if you enable JMX.

   Another alternative, is to enable JMX after a Java application has started is to use the diagnostic command ManagementAgent.start. Run jcmd <pid> help ManagementAgent.start for a list of flags that can be sent with the command.

   See The jcmd Utility.

If your application runs into a problem and you want to debug the problem further, ensure that you collect any relevant data before restarting the system, especially if restarting will remove previous files.

• It is important to gather the following files:

  • Core files for crash issues.

  • hs_err printed text file for Java crashes.

  • Log files: Java and application logs.

  • Java heap dumps for -XX:+HeapDumpOnOutOfMemoryError.

  • Java flight recordings (if enabled). If the problem didn't terminate the application, dump the continuous recordings.

• If the application stopped responding, then gather the following files:

  • Stack traces: Take several stack traces using jcmd <pid> Thread.print before restarting the system.

  • Dump flight recordings (if enabled).

  • Force a core file: If the application can't be closed properly, then stop the application, and force a core file using kill -6 <pid> on Linux or Solaris systems.