Saturday, December 18, 2010

Automatically Retry Failed Jobs in Quartz

How do you handle job failures in Quartz? There are a few things you can do:
  • Do nothing. Let the job fail and log the error.
  • Retry continuously until the job succeeds.
  • Retry n times and then disable the job.
In this post, I will describe how you can configure your jobs to be retried on failure.

Retrying continuously until success:
If you want to keep trying over and over again until the job succeeds, all you have to do is throw a JobExecutionException with a flag to tell the scheduler to fire it again when it fails. The following code shows how:

class MyJob implements Job {

  public MyJob() {
  }

  public void execute(JobExecutionContext context)
                  throws JobExecutionException {
    try{
        //do something
    }
    catch(Exception e){

        Thread.sleep(10000); //sleep for 10 secs

        JobExecutionException e2 = new JobExecutionException(e);
        //fire it again
        e2.refireImmediately();
        throw e2;
    }
  }
}
Retrying n times:
It gets a bit more complicated if you want to retry a certain number of times only. You have to use a StatefulJob and hold a retryCounter in its JobDataMap, which you increment if the job fails. If the counter exceeds the maximum number of retries, then you can disable the job if you wish.
class MyJob implements StatefulJob {

  public MyJob() {
  }

  public void execute(JobExecutionContext context)
                                 throws JobExecutionException {
    JobDataMap dataMap = context.getJobDetail().getJobDataMap();
    int count = dataMap.getIntValue("count");

    // allow 5 retries
    if(count >= 5){
        JobExecutionException e = new JobExecutionException("Retries exceeded");
        //unschedule it so that it doesn't run again
        e.setUnscheduleAllTriggers(true);
        throw e;
    }


    try{
        //do something

        //reset counter back to 0
        dataMap.putAsString("count", 0);
    }
    catch(Exception e){
        count++;
        dataMap.putAsString("count", count);
        JobExecutionException e2 = new JobExecutionException(e);

        Thread.sleep(10000); //sleep for 10 secs

        //fire it again
        e2.refireImmediately();
        throw e2;
    }
  }
}

Saturday, December 11, 2010

Throw a checked exception from a method without declaring it

When you try to throw a checked exeption from a method in Java, the compiler will force you to either catch it or declare it to be thrown in the method declaration. For example, the following method fails to compile:
public void go(){
    throw new IOException();
}
The compiler says:
Dummy.java:15: unreported exception java.io.IOException;
must be caught or declared to be thrown
    throw new IOException();
    ^
1 error
However, if you declare it to be thrown in the method declaration, like this:
public void go() throws IOException{
    throw new IOException();
}
then it compiles successfully.

But is it possible to throw a checked exception WITHOUT declaring it to be thrown? You could wrap it in a RuntimeException (for example, new RuntimeException(new IOException()), but then you lose the ability to catch the checked exception in the caller method, because you have changed the type of the exception.

There is another trick you can use. I first noticed this being done in the JDK's java.lang.Class#newInstance0 method:

private T newInstance0()
    throws InstantiationException, IllegalAccessException
{
    ...
    // Run constructor
    try {
        return tmpConstructor.newInstance((Object[])null);
    } catch (InvocationTargetException e) {
        Unsafe.getUnsafe().throwException(e.getTargetException());
        // Not reached
        return null;
    }
    ...
}
As shown above, the method throws an InvocationTargetException but only declares the exceptions InstantiationException and IllegalAccessException to be thrown! The Sun utility class sun.misc.Unsafe is being used to throw a checked exception without declaring it. We could use the same approach, but unfortunately our code throws a security exception when we try to use it: java.lang.SecurityException: Unsafe. However, we can bypass this by using reflection to get hold of the unsafe variable directly, as shown below:
public void go() {
  getUnsafe().throwException(new IOException());
}

private Unsafe getUnsafe() {
  try {
    Field field = Unsafe.class.getDeclaredField("theUnsafe");
    field.setAccessible(true);
    return (Unsafe) field.get(null);
  } catch (Exception e) {
    throw new RuntimeException(e);
  }
}
So, why did the JDK developers do it this way? I don't know, but I'm sure they had a good reason. Although this is a neat trick to impress your friends, I wouldn't recommend using it. This is non-standard, undocumented, Sun-specific code which could change in future releases and may not be compatible with all JVMs. It also makes code harder to understand for others and violates the POLA.

Further Reading:
Avoiding Checked Exceptions [O'Reilly]

Saturday, October 30, 2010

Formatting XML quickly

There are many times when I have come across badly formatted XML and need to prettify it instantly in order to aid readability or simply to paste into another document, like a blog post. There are plugins available (for example, Textpad's XMLTidy) which tidy up xml, but they involve pasting XML into a file and running a macro to clean it up, which can be slow, especially if you don't have an editor open.

So, I decided to write my own Java utility to format XML instantly. All you have to do is select some XML text, hit CTRL+C to save it to your clipboard, hit CTRL+ALT+F to invoke my formatting utility and finally hit CTRL+V to paste the nicely formatted XML somewhere else. This has made working with XML so much easier!

This is how you can set it up too:

The Java Source Code
Save the following source code to a file called XMLTidy.java and compile it using javac.

import java.awt.Toolkit;
import java.awt.datatransfer.DataFlavor;
import java.awt.datatransfer.StringSelection;
import java.awt.datatransfer.Transferable;
import java.awt.datatransfer.UnsupportedFlavorException;
import java.io.IOException;
import java.io.StringReader;
import java.io.StringWriter;
import java.io.Writer;

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;

import org.w3c.dom.Document;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;

import com.sun.org.apache.xml.internal.serialize.OutputFormat;
import com.sun.org.apache.xml.internal.serialize.XMLSerializer;

/**
 * A useful utility for formatting xml.
 * Retrieves xml text from the system clipboard, formats it
 * and resaves it to the clipboard.
 */
public class XMLTidy {


  /**
   * Formats the specified xml string
   *
   * @param src the xml text to format
   * @return formatted xml
   * @throws ParserConfigurationException
   * @throws SAXException
   * @throws IOException
   */
  private static String tidyXml(String src)
      throws ParserConfigurationException, SAXException, IOException {
    DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
    DocumentBuilder db = dbf.newDocumentBuilder();
    InputSource is = new InputSource(new StringReader(src));
    Document document = db.parse(is);
    OutputFormat format = new OutputFormat(document);
    format.setLineWidth(65);
    format.setIndenting(true);
    format.setIndent(2);
    Writer out = new StringWriter();
    XMLSerializer serializer = new XMLSerializer(out, format);
    serializer.serialize(document);
    return out.toString();
  }

  /**
   * @return the text in the clipboard
   */
  private static String getClipboard() {
    Transferable t = Toolkit.getDefaultToolkit().getSystemClipboard()
        .getContents(null);
    try {
      if (t != null &&
          t.isDataFlavorSupported(DataFlavor.stringFlavor)) {
        String text = (String) t.getTransferData(DataFlavor.stringFlavor);
        return text;
      }
    } catch (UnsupportedFlavorException e) {
      e.printStackTrace();
    } catch (IOException e) {
      e.printStackTrace();
    }
    return "";
  }

  /**
   * @param str the text to set in the clipboard
   */
  private static void setClipboard(String str) {
    StringSelection ss = new StringSelection(str);
    Toolkit.getDefaultToolkit().getSystemClipboard().setContents(ss, null);
  }


  /**
   * Formats the xml supplied as an argument.
   * If no arguments are specified, formats the xml
   * in the clipboard.
   * @param args
   * @throws Exception
   */
  public static void main(String[] args) throws Exception {
    String in = args.length > 0 ? args[0] : getClipboard();
    if (in != null) {
      in = in.trim();
      if (in.charAt(0) == '<') {
        setClipboard(tidyXml(in));
      }
    }
  }
}
The Launcher Script
Create a bat file to launch the java program:
@echo off
%JAVA_HOME%\bin\java -cp \path\to\XMLTidy\classes XMLTidy %1
The Keyboard Shortcut
Finally create a keyboard shortcut to the launcher script as follows:
  • First, create a shortcut to the launcher script, by right-clicking the bat file and selecting "Create a shortcut".
  • Right-click the shortcut file and select "Properties".
  • Enter a "Shortcut key" on the Shortcut tab. For example, the shortcut key I use is CTRL+ALT+F
Try it out!
  • Select some badly formatted XML and copy it (using CTRL+C, for example).
  • Invoke XMLTidy by using the keyboard shortcut, CTRL+ALT+F.
  • Paste the XML (using CTRL+V, for example). The XML will be nicely formatted!

Sunday, October 17, 2010

Redirect stdout to logger

Recently, whilst using an external jar in my project, I found that it was doing a lot of logging to stdout using System.out.print statements. Even exceptions were being printed using e.printStackTrace(). We all know that this is bad practice and we should always consider using a logger instead. It was really annoying, seeing these messages come up on my console. Since my application already uses SLF4J for logging, I decided to redirect stdout and stderr to my logger. The following code snippet shows how:
private static final Logger LOGGER = Logger.getLogger(MyClass.class);

/**
 * Redirects stdout and stderr to logger
 */
public static void redirect(){
 System.setOut(new PrintStream(System.out){
  public void print(String s){
   LOGGER.info(s);
  }
 });
 System.setErr(new PrintStream(System.err){
  public void print(String s){
   LOGGER.error(s);
  }
 });
}

Saturday, October 16, 2010

Logback: Change root logger level programmatically

A couple of years ago, I wrote about how it is possible to change log4j logging levels using JMX. I'm now using logback, which is intended to be the successor of log4j and provides several advantages over it.

The following code snippet can be used to change the root logger's logging level in logback:

import org.slf4j.LoggerFactory;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.Logger;

Logger root = (Logger)LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME);
root.setLevel(Level.DEBUG); //change to debug

Sunday, October 10, 2010

Creating tables with PDFBox

Apache PDFBox is a useful Java library for working with PDF documents. It allows you to create new PDF documents and extract data from existing documents.

However, the library doesn't provide an API for creating tables within PDF documents. So I wrote my own method which uses basic operations like drawLine to draw the table cells and drawString to fill in the content. The code is shown below. Note, that this code could be improved to handle longer strings of text properly by wrapping text within cells, which it doesn't do at present.

/**
 * @param page
 * @param contentStream
 * @param y the y-coordinate of the first row
 * @param margin the padding on left and right of table
 * @param content a 2d array containing the table data
 * @throws IOException
 */
public static void drawTable(PDPage page, PDPageContentStream contentStream,
                            float y, float margin,
                            String[][] content) throws IOException {
    final int rows = content.length;
    final int cols = content[0].length;
    final float rowHeight = 20f;
    final float tableWidth = page.findMediaBox().getWidth()-(2*margin);
    final float tableHeight = rowHeight * rows;
    final float colWidth = tableWidth/(float)cols;
    final float cellMargin=5f;

    //draw the rows
    float nexty = y ;
    for (int i = 0; i <= rows; i++) {
        contentStream.drawLine(margin,nexty,margin+tableWidth,nexty);
        nexty-= rowHeight;
    }

    //draw the columns
    float nextx = margin;
    for (int i = 0; i <= cols; i++) {
        contentStream.drawLine(nextx,y,nextx,y-tableHeight);
        nextx += colWidth;
    }

    //now add the text
    contentStream.setFont(PDType1Font.HELVETICA_BOLD,12);

    float textx = margin+cellMargin;
    float texty = y-15;
    for(int i = 0; i < content.length; i++){
        for(int j = 0 ; j < content[i].length; j++){
            String text = content[i][j];
            contentStream.beginText();
            contentStream.moveTextPositionByAmount(textx,texty);
            contentStream.drawString(text);
            contentStream.endText();
            textx += colWidth;
        }
        texty-=rowHeight;
        textx = margin+cellMargin;
    }
}

public static void main(String[] args){
    PDDocument doc = new PDDocument();
    PDPage page = new PDPage();
    doc.addPage( page );

    PDPageContentStream contentStream =
                    new PDPageContentStream(doc, page);

    String[][] content = {{"a","b", "1"},
                          {"c","d", "2"},
                          {"e","f", "3"},
                          {"g","h", "4"},
                          {"i","j", "5"}} ;

    drawTable(page, contentStream, 700, 100, content);
    contentStream.close();
    doc.save("test.pdf" );
    }

Tuesday, September 07, 2010

Lock-Ordering Deadlock

The code snippet below shows an example of a lock-ordering deadlock.
/**
 * Swaps the balances of two accounts.
 * Deadlock-prone!
 *
 * @param a
 * @param b
 */
public void swap(Account a, Account b) {
 synchronized (a) {
  synchronized (b) {
   double aBal = a.getBalance();
   double bBal = b.getBalance();
   a.setBalance(bBal);
   b.setBalance(aBal);
  }
 }
}
It acquires locks on both Account objects and then swaps their balances. It may look harmless at first, but if you look closely you will see that a deadlock can occur if two threads call swap at the same time, for the same accounts, but in the opposite order:

Thread_1: swap(a,b)
Thread_2: swap(b,a)

It is possible that Thread_1 will acquire the lock on account a and wait for the lock on account b, while Thread_2 is holding the lock on b and waiting for the lock on a.

In order to fix this, we need to make sure that the locks are always acquired in the same order. Here are a few different approaches that can be taken to resolve this:

1. Synchronizing the method
Remove the nested lock acquisitions and synchronize the method instead.


/**
 * Swaps the balances of two accounts.
 *
 * @param a
 * @param b
 */
public synchronized void swap(Account a, Account b) {
 double aBal = a.getBalance();
 double bBal = b.getBalance();
 a.setBalance(bBal);
 b.setBalance(aBal);
}
2. Inducing a lock ordering
To induce a lock ordering, you can compare the two accounts based on a unique, immutable key such as an account number. If your accounts are not Comparable, you can use System.identityHashCode instead. In case, for some reason, the two accounts being passed in are the same, you need to aquire a tie-breaking lock before aquiring the account locks.
private static final Object tieBreaker = new Object();

public void swap(Account a, Account b) {
 final int c = a.compareTo(b);
 if (c > 0) {
  synchronized (a) {
   synchronized (b) {
    double aBal = a.getBalance();
    double bBal = b.getBalance();
    a.setBalance(bBal);
    b.setBalance(aBal);
   }
  }
 } else if (c < 0) {
  synchronized (b) {
   synchronized (a) {
    double aBal = a.getBalance();
    double bBal = b.getBalance();
    a.setBalance(bBal);
    b.setBalance(aBal);
   }
  }
 } else {
  synchronized (tieBreaker) {
   synchronized (a) {
    synchronized (b) {
     double aBal = a.getBalance();
     double bBal = b.getBalance();
     a.setBalance(bBal);
     b.setBalance(aBal);
    }
   }
  }
 }
}
3. Using tryLock
Use tryLock to acquire both locks, but backoff and retry if they cannot both be acquired.
public void swap(Account a, Account b)
                        throws InterruptedException {
 Random random = new Random();
 while(true){
  if(a.getLock().tryLock()){
   try{
    if(b.getLock().tryLock()){
     try{
      double aBal = a.getBalance();
      double bBal = b.getBalance();
      a.setBalance(bBal);
      b.setBalance(aBal);
      return;
     }
     finally{
      b.getLock().unlock();
     }
    }
   }
   finally{
    a.getLock().unlock();
   }
  }
  Thread.sleep(random.nextInt(1000));
 }
}

Dell Studio - Restoring the Function Keys

I really like my new Dell Studio 17 laptop. But it does come with one really annoying feature. The function keys F1, F2 etc don't work as they should. They have been overlayed with "multimedia" keys which are enabled by default, so pressing F3, for example, brings up the battery dialog, F5, pops up the brightness control and F9 increases volume. In order to use the original F5 behaviour (e.g. to reload a webpage) you would have to combine it with the Function key (Fn) i.e. press Fn+F5. This is a big design defect, in my opinion. I would rather have my original F1, F2 keys back and combine them with the Fn key for multimedia, which I do not use as often.

There is a way to reconfigure the function keys. It involves changing a setting in the BIOS. You need to perform the following steps:

  1. When the laptop starts up (and you see the Dell Studio logo), quickly press F2 to enter the Setup Utility.
  2. Go to the "Advanced" tab.
  3. Move down to "Function Key Behaviour" and use the + key to change it from "Multimedia" to "Function".
  4. Press F10 to Save and Exit.

Monday, September 06, 2010

Java Bit Twiddling

This post explains some of the less commonly used operators in Java, namely the bitwise and bit shift operators and how they can be cleverly used to multiply or divide integers, test for evenness etc. The operators are shown in the table below:
Name Description
~a (NOT) Negates each bit. 0s becomes 1s and vice versa. ~a=(-a)-1
a & b (AND) In each pair of corresponding bits, the result is 1 if the first bit is 1 AND the second bit is 1, 0 otherwise.
a | b (OR) In each pair of corresponding bits, the result is 1 if the first bit is 1 OR the second bit is 1 OR both bits are 1, and otherwise the result is 0
a ^ b (XOR) In each pair of corresponding bits, the result in each position is 1 if the two bits are different, and 0 if they are the same.
a << n (LEFT-SHIFT) Shifts bit pattern to the left by n positions and places 0s on the right.
a >> n (RIGHT-SHIFT) Shifts bit pattern to the right by n positions and places the sign-bit on the left, thus preserving the sign of the operand.
a >>> n (UNSIGNED-RIGHT-SHIFT) Shifts bit pattern to the right by n positions and places 0s on the left.
Examples of bitwise and bit shift operators in action:
a = 3 (11)
b = 6 (110)

~a = -4 (11111111111111111111111111111100)
a & b = 2 (10)
a | b = 7 (111)
a ^ b = 5 (101)
a << 2 = 12 (1100)
b >> 1 = 3 (11)
b >>> 1 = 3 (11)
The code used to generate this is shown below:
final int a = 3; // 011
final int b = 6; // 110

System.out.printf("a = %s\t(%s)\n", a,
                 Integer.toBinaryString(a));
System.out.printf("b = %s\t(%s)\n", b,
                 Integer.toBinaryString(b));

/**
 * NOT
 */
int not = ~a;
System.out.printf("~a = %s\t(%s)\n", not,
                 Integer.toBinaryString(not));

/**
 * AND
 */
int and = a & b;
// 011
// 110
//----&
// 010
System.out.printf("a & b = %s\t(%s)\n", and,
                 Integer.toBinaryString(and));

/**
 * OR
 */
int or = a | b ;
// 011
// 110
//----|
// 111
System.out.printf("a | b = %s\t(%s)\n", or,
                 Integer.toBinaryString(or));

/**
 * XOR
 */
int xor = a ^ b ;
// 011
// 110
//----^
// 101
System.out.printf("a ^ b = %s\t(%s)\n", xor,
                 Integer.toBinaryString(xor));

/**
 * LEFT-SHIFT
 */
int lshift = a << 2;
// 00011
// ----- <<2
// 01100
System.out.printf("a << 2 = %s\t(%s)\n", lshift,
                 Integer.toBinaryString(lshift));

/**
 * RIGHT-SHIFT
 */
int rshift = b >> 1;
// 110
// ----- >>1
// 011
System.out.printf("b >> 1 = %s\t(%s)\n", rshift,
                 Integer.toBinaryString(rshift));

/**
 * UNSIGNED-RIGHT-SHIFT
 */
int urshift = b >>> 1;
// 110
// ----- >>1
// 011
System.out.printf("b >>> 1 = %s\t(%s)\n", urshift,
                 Integer.toBinaryString(urshift));

Usage of bit manipulation:
So, where do you use this? In the olden days, they were used for efficiency but most modern compilers today perform optimisations for you, so you don't have to worry about it. The bitwise and bit shift operators are not used very frequently in day-to-day programming, but here are a few scenarios where they can be used - mainly to impress you colleagues :)

1. Checking if an integer is even or odd
If an integer is odd, its rightmost bit will always be 1 and if it is even, it will always be 0. For example, 4 is 100 and 5 is 101.

public boolean isEven(int a){
    return a & 1 == 0;
}
2. Multiplying/Dividing by powers of two
Shifting to the left is just like multiplying by a power of two. For example 3 << 2 is 12. In general, a << n is a*(2^n). Keep in mind that a left shift can result in "overflow" which occurs when you exceed the limits of the range of the data type you are using and can cause unexpected behaviour.

A right shift is an integer division by a power of 2. Shifting right 1 position is the same as dividing by two. This approach is used in the Collections Framework, such as the Arrays class to find the middle element of an array. In general, a >> n is a/(2^n). Be careful, when right shifting negative integers. For example, 5 >> 1 gives 2, but -5 >> 1, gives -3.

3. Is an integer positive or negative?
Shift the integer right 31 places. The value of the far left bit is copied to the other bits. The far left bit is 1 when the value is negative and 0 otherwise.

public boolean isNegative(int a){
    return (a >> 31) != 0;
}
4. Negating an integer
You can use the NOT operator to determine the negative of any integer as shown in the snippet below. The basic idea is you invert all the bits and then add 1.
int num = 4;
int neg = ~num + 1; //-4
Note that you can also do the reverse and determine the positive of a negative integer by subtracting 1 and then inverting the bits.

5. Check if the nth bit is set
This is a more generalised version of the even/odd check. We move the 1 bit n places to the left and then apply the AND operation. If the nth bit was not set, this would result in 0.

public boolean isBitSet(int a, int n){
    return a & (1 << n) != 0;
}
6. Setting the nth bit
a | (1<<n)
7. Unsetting the nth bit
a & ~(1<<n)
8. Toggling the nth bit
a ^ (1<<n)
9. Unsetting the rightmost 1-bit
a & (a-1)
10. Checking if number is a power of 2
If the number is a power of 2, there will be only one 1-bit. If we unset this bit, we will get 0.
return (a & (a-1)) == 0;
11. Swapping values without a temporary variable
Swap two variables without a temporary variable using XOR:
x = x ^ y;
y = x ^ y;
x = x ^ y;
References:
Bit Twiddling Hacks
Java Ranch - Bit Shifting
Low Level Bit Hacks You Absolutely Must Know

Thursday, September 02, 2010

Python Cheat Sheet

I've been playing around with python over the last few days. It's really easy to use and I've written the following cheat sheet so that I don't forget how to use it:

Collections:

###################### TUPLE - immutable list
rgb = ("red", "green", "blue")

assert len(rgb) == 3
assert min(rgb) == "blue"
assert max(rgb) == "red"

#find the last element
assert rgb[len(rgb) - 1] == "blue"

assert "red" in rgb
assert "yellow" not in rgb

#slice the list [1,3)
rgb[1:3] #('green', 'blue')

#concatenate twice
rgb * 2

#iterate over list
for e in rgb:
 print e

###################### LIST - mutable list
colors = ["black", "white"]
assert len(colors) == 2

#add an element
colors.append("green")
assert len(colors) == 3
assert colors[2] == "green"

#remove an element
colors.remove("white")
assert "white" not in colors

colors.append("green")

#count occurrences of an element
assert colors.count("green") == 2

#return first index of value
assert colors.index("black") == 0

#insert element
colors.insert(1, "orange")
assert colors.index("orange") == 1

#remove the last element
e = colors.pop()
assert e == "green"

#replace a slice
colors[1:3] = ["red"];

colors.reverse()
colors.sort()

###################### SET
l = ["a", "b", "a", "c"]
assert len(set(l)) == 3

###################### DICTIONARY- mappings

dict = {"alice":22, "bob":20}
assert dict.get("alice") == 22

#add an element
dict["charlie"] = 25
assert dict.get("charlie") == 25

#get list of keys
assert len(dict.keys()) == 3

#get list of values
assert len(dict.values()) == 3

#check for key
assert dict.has_key("alice")
Loops:
for i in range(1, 10, 2):
    print i

while True:
    print 1
    break
Methods:
Printing out the factorial of a number.
def factorial(n):
    """prints factorial of n using recursion."""
    if n == 0:
        return 1
    else:
        return n * factorial(n - 1)

print factorial(4)
Classes:
class Person:

    def __init__(self, name, age):
        self.name = name
        self.age = age

    def equals(self, p):
        return isinstance(p, Person) and \
               self.name == p.name and \
               self.age == p.age


    def toString(self):
        return "name: %s, age: %s" % (self.name, self.age)


p = Person("Bob", 20)
q = Person("Charlie", 20)
assert p.equals(p)
assert p.equals(q)==False
print p.toString()
Read user input:
string = raw_input("Enter your name: ")
number = input("Enter a number:")
File I/O:
#write to file
filename = "C:/temp/test.txt"
f = file(filename, "w")
f.write("HELLO WORLD\nHELLO WORLD")
f.close()

#open a file in read-mode
f = file(filename, "r")

#read all lines into a list in one go
assert len(f.readlines()) == 2

#print out current position
print f.tell()

#go to beginning of file
f.seek(0)

#read file line by line with auto-cleanup
with open("C:\\temp\hello.wsdl") as f:
    for line in f:
        print line
Pickling (Serialisation):
#pickling
f = file("C:\\temp\dictionary.txt", 'w')
dict = {"alice":22, "bob":20}
import pickle
pickle.dump(dict, f)
f.close()

#unpickling
f = file("C:\\temp\dictionary.txt", 'r')
obj = pickle.load(f)
f.close()
assert obj["alice"] == 22
assert obj["bob"] == 20
Exception handling:
try:
    i = int("Hello")
except ValueError:
    i = -1
finally:
    print "finally"


#creating a custom exception class
from repr import repr

class MyException(Exception):
    def __init__(self, value):
        self.value = value
    def __str__(self):
        return repr(self.value)

try:
    raise MyException(2*2)
except MyException as e:
    print 'My exception occurred, value:', e.value
Lambdas:
def make_incrementor (n):
    return lambda x: x + n

f = make_incrementor(2)
print f(42)

print map(lambda x: x % 2, range(10))

Sunday, August 29, 2010

Facebook Chat with Pidgin

In a previous post, I wrote about how you can use Pidgin to replace Office Communicator (OCS). Now, I will show you how you can use Pidgin for Facebook chat, thereby making Pidgin a truly universal chat client!
  • Start the Pidgin client.
  • Add a new Account with the following details:
    • Protocol: XMPP
    • Username: yourfacebookusername
    • Domain: chat.facebook.com
    • Password: facebookpassword
    • On the Advanced tab, untick "Require SSL/TLS".

Friday, August 27, 2010

Leaving DB

Today is my last day at Deutsche Bank.

I've been at DB for six years. I started as a graduate and, over the years, have learnt so much, not just about technology, business and processes, but also about myself. I've had the opportunity to work with some of the best minds in the bank and have made some great friends. I would like to thank all my colleagues for their guidance and support and for making my time at DB so enjoyable. I will miss you all!

Leaving DB is by far the hardest decision I've ever had to make, but I feel that the time has come for me to move on to new challenges.

I'm certainly planning to keep blogging and tweeting, so stay tuned to find out what I am getting up to.

Journey onward!

Wednesday, August 25, 2010

Software Inventory of my Windows Machine

As I will be leaving my computer soon, I have decided to make a quick list of useful applications currently installed:
CategoryNameDescription
DatabaseAqua Data StudioDatabase Query Tool
DatabaseJaySQLJDBC Database Tool
DatabaseToadOracle Database Tool
DevAntBuild Tool
DevAxis 1.4Web Services Engine
DevAxis 2Web Services Engine
DevChainsawLog Viewer
DevCoberturaCode Coverage Tool
DevEclipseIDE
DevGroovyProgramming Language
DevGWTGoogle Web Toolkit
DevHermesJMSJMS Browser
DevHSQLDBJava Database Engine
DevJADJava Decompiler
DevJavaProgramming Language
DevMavenProject Build Tool
DevPerlProgramming Language
DevPuTTySSH Client
DevPython 2.6Programming Language
DevSciTESource Code Editor
DevsoapUIWeb Service Caller
DevSQLiteSQL database engine
DevTomcatServlet Engine
DevTortoiseCVSCVS Client
DevWinMergeDifferencing and Merging Tool
Devxmlbeans-2.4.0XML to Java type binding
EditorAltova XML SpyXML Editor
EditorEmacs 22.3Text Editor
EditorTextPad 4Text Editor
PerformanceCachemanXPWindows Tuneup Utility
PerformanceCCleanerPC Cleaner
PerformanceDefragglerDisk Defragmenter
PerformanceHijackThisSystem Scanner
ProductivityAlt-Tab Task SwitcherMS Powertoys
ProductivityCalculator PowertoyMS Powertoys
ProductivityClipXClipboard History Manager
ProductivityCmdHere PowertoyMS Powertoys
ProductivityLClockClock
ProductivityPassword SafePassword Manager
ProductivityProcessExplorerTask Manager
ProductivityPsToolsWindows Tools
ProductivitySlickRunQuick Application Launcher
ProductivityTaskixReorder taskbar items
ProductivityTweak UIMS Powertoys
ProductivityUnxUtilsGNU Utilities for Windows
Productivityxplorer² liteWindows File Manager
UtilitiesConvert Image To PDFPDF Converter
UtilitiesdoPDF 7PDF Converter
UtilitiesFoxitPDF Reader
UtilitiesWinRARArchive Manager
UtilitiesxmltidyTextPad addon
WebMozilla FirefoxWeb Browser
WebPidginInstant Messenger
And, ofcourse, browsing wouldn't be the same without my Firefox addons:
Firefox Addons
Adblock Plus
All-in-One Sidebar
British English Dictionary
Cache Status
Colorful Tabs
Delicious Bookmarks
Download Statusbar
DownThemAll!
dragdropupload
FaviconizeTab
Firebug
Fission
Flashblock
Forecastfox
FoxClocks
Greasemonkey
Mouse Gestures Redox
Tab Mix Plus
Tab Preview
Ubiquity
URL Fixer
If you think you have better alternatives for any of the applications above, please write a comment!

Saturday, August 21, 2010

Faster XPaths with VTD-XML

I've recently started using VTD-XML for applying XPaths on large XML documents. DOM is a memory hog and is too slow. However, VTD-XML allows you to run XPaths and provides random access to nodes, similar to DOM, but much more efficiently. You can't apply XPaths with a SAX parser nor can you access nodes randomly or traverse the document easily.

VTD-XML was 60 times faster compared to DOM when processing my XML document (20MB).

This post shows you how to use VTD-XML for fast XPath evaluation.

Sample XML:
I will use the following XML document in the examples below.

<?xml version="1.0"?>
<catalog>
 <book id="bk101">
  <author>Gambardella, Matthew</author>
  <author>Doe, John</author>
  <title>XML Developer's Guide</title>
  <genre>Computer</genre>
  <price>44.95</price>
  <publish_date>2000-10-01</publish_date>
 </book>
 <book id="bk102">
  <author>Ralls, Kim</author>
  <title>Midnight Rain</title>
  <genre>Fantasy</genre>
  <price>5.95</price>
  <publish_date>2000-12-16</publish_date>
 </book>
 <book id="bk103">
  <author>Corets, Eva</author>
  <title>Maeve Ascendant</title>
  <genre>Fantasy</genre>
  <price>5.95</price>
  <publish_date>2000-11-17</publish_date>
 </book>
</catalog>
Loading the XML document:
The following code parses the XML file and creates the navigator and autopilot objects.
final VTDGen vg = new VTDGen();
vg.parseFile("books.xml", false);
final VTDNav vn = vg.getNav();
final AutoPilot ap = new AutoPilot(vn);
Selecting all titles:
Print out all the title nodes using an XPath expression of /catalog/book/title. First call selectXPath to compile the expression and then use evalXPath to move the cursor to the selected nodes in the result.
ap.selectXPath("/catalog/book/title");
while (ap.evalXPath() != -1) {
  int val = vn.getText();
  if (val != -1) {
    String title = vn.toNormalizedString(val);
    System.out.println(title);
  }
}
Selecting all book ids and authors:
This one is a bit more involved as a book can have many authors. In the code below, I first run an XPath to select the books and then iterate over the children, selecting the author nodes.
ap.selectXPath("/catalog/book");
while (ap.evalXPath() != -1) {
  int val = vn.getAttrVal("id");
  if(val != -1){
    String id = vn.toNormalizedString(val);
    System.out.println("Book id: " + id);
  }

  if(vn.toElement(VTDNav.FIRST_CHILD,"author")){
    do{
      val = vn.getText();
      if(val != -1){
        String author = vn.toNormalizedString(val);
        System.out.println("\tAuthor:" + author);
      }
    }while(vn.toElement(VTDNav.NEXT_SIBLING,"author"));
  }
  vn.toElement(VTDNav.PARENT);
}
The output is:
Book id: bk101
 Author:Gambardella, Matthew
 Author:Doe, John
Book id: bk102
 Author:Ralls, Kim
Book id: bk103
 Author:Corets, Eva
Related Posts:
Using XPath with DOM

Friday, August 20, 2010

Fixing a ConcurrentModificationException

Question:
The following code throws a ConcurrentModificationException. What additional code can you add between the <FIXME>...</FIXME> tags in order to prevent this exception from being thrown?
final List<String> list = new ArrayList<String>();
list.add("HELLO");
final Iterator<String> iter = list.iterator();
System.out.println(iter.next());
list.add("WORLD");
//<FIXME>

//</FIXME>
System.out.println(iter.next());
Solution:
In this example, a ConcurrentModificationException is thrown because the Iterator detects that the list over which it is iterating has been changed. If you look into the source code for these classes you will find that when an Iterator is created, it contains an int variable called expectedModCount which is initialised to the modCount of the backing list. Whenever the backing list is structurally modified (with an add or remove operation, for example) then the modCount is incremented. As a result, the iterator's expectedModCount no longer matches the list's modCount and the iterator throws a ConcurrentModificationException.

In order to prevent this exception from being thrown, we need to bring the expectedModCount of the iterator and the modCount of the list back in line with each other. Here are a couple of ways this can be done:

1. Reflection:
Reflection is the easiest way to change the internal counters of the iterator and list. In the fix below, I have set the expectedModCount of the iterator to the same value as the modCount of the list. The code no longer throws the ConcurrentModificationException.

final List<String> list = new ArrayList<String>();
list.add("HELLO");
final Iterator<String> iter = list.iterator();
System.out.println(iter.next());
list.add("WORLD");
//<FIXME>
/* Using Reflection */
try{
  //get the modCount of the List
  Class cls = Class.forName("java.util.AbstractList");
  Field f = cls.getDeclaredField("modCount");
  f.setAccessible(true);
  int modCount = f.getInt(list);

  //change the expectedModCount of the iterator
  //to match the modCount of the list
  cls = iter.getClass();
  f = cls.getDeclaredField("expectedModCount");
  f.setAccessible(true);
  f.setInt(iter, modCount);
}
catch(ClassNotFoundException e){
  e.printStackTrace();
}
catch(NoSuchFieldException e){
  e.printStackTrace();
}
catch(IllegalAccessException e){
  e.printStackTrace();
}
//</FIXME>
System.out.println(iter.next());
2. Integer Overflow:
Another approach is to keep modifying the list until the integer modCount overflows and reaches the same value as expectedModCount. At the moment, modCount=2 and expectedModCount=1. In the fix below, I repeatedly change the list (by calling trimToSize), forcing modCount to overflow and reach expectedModCount. This code took 38s to run on my machine.
final List<String> list = new ArrayList<String>();
list.add("HELLO");
final Iterator<String> iter = list.iterator();
System.out.println(iter.next());
list.add("WORLD");
//<FIXME>
for(int i = Integer.MIN_VALUE ; i < Integer.MAX_VALUE ; i++){
  ((ArrayList)list).trimToSize();
}
//</FIXME>
System.out.println(iter.next());

Sunday, August 15, 2010

DateFormat with Multiple Threads

The DateFormat class is not thread-safe. The javadocs state that "Date formats are not synchronized. It is recommended to create separate format instances for each thread. If multiple threads access a format concurrently, it must be synchronized externally."

The following code shows how you would typically use DateFormat to convert a String to a Date in a single-threaded environment. It is more efficient to get the format as an instance variable and use it multiple times so that the system doesn't have to fetch the information about the local language and country conventions multiple times.

public class DateFormatTest {

  private final DateFormat format =
            new SimpleDateFormat("yyyyMMdd");

  public Date convert(String source)
                      throws ParseException{
    Date d = format.parse(source);
    return d;
  }
}
This code is not thread-safe. We can test it out by invoking the method using multiple threads. In the calling code below, I create a thread pool with 2 threads and submit 5 date conversion tasks to it. I then examine the results.
final DateFormatTest t = new DateFormatTest();
Callable<Date> task = new Callable<Date>(){
    public Date call() throws Exception {
        return t.convert("20100811");
    }
};

//lets try 2 threads only
ExecutorService exec = Executors.newFixedThreadPool(2);
List<Future<Date>> results =
             new ArrayList<Future<Date>>();

//perform 5 date conversions
for(int i = 0 ; i < 5 ; i++){
    results.add(exec.submit(task));
}
exec.shutdown();

//look at the results
for(Future<Date> result : results){
    System.out.println(result.get());
}
When the code is run the output is unpredictable - sometimes it prints out the correct dates, sometimes the WRONG ones (e.g. Sat Jul 31 00:00:00 BST 2012!) and at other times it throws a NumberFormatException!

How can you use DateFormat concurrently?
There are different approaches you can take to use DateFormat in a thread-safe manner:

1. Synchronization
The easiest way of making this code thread-safe is to obtain a lock on the DateFormat object before parsing the date string. This way only one thread can access the object at a time, and the other threads must wait.

public Date convert(String source)
                    throws ParseException{
  synchronized (format) {
    Date d = format.parse(source);
    return d;
  }
}

2. ThreadLocals
Another approach is to use a ThreadLocal variable to hold the DateFormat object, which means that each thread will have its own copy and doesn't need to wait for other threads to release it. This is generally more efficient than synchronising sa in the previous approach.

public class DateFormatTest {

  private static final ThreadLocal<DateFormat> df
                 = new ThreadLocal<DateFormat>(){
    @Override
    protected DateFormat initialValue() {
        return new SimpleDateFormat("yyyyMMdd");
    }
  };

  public Date convert(String source)
                     throws ParseException{
    Date d = df.get().parse(source);
    return d;
  }
}

3. Joda-Time
Joda-Time is a great, open-source alternative to the JDK's Date and Calendar API. It's DateTimeFormat is "thread-safe and immutable".

import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
import java.util.Date;

public class DateFormatTest {

  private final DateTimeFormatter fmt =
       DateTimeFormat.forPattern("yyyyMMdd");

  public Date convert(String source){
    DateTime d = fmt.parseDateTime(source);
    return d.toDate();
  }
}

Saturday, August 14, 2010

Using Compressed JMS Messages

If you are publishing large XML messages onto a JMS topic or queue, compression will give you much better performance because less data is sent over the network. Also, your JMS server can hold more messages and there is less risk of running out of memory. XML messages are great candidates for compression, due to the repetitive nature of XML.

Compressing JMS messages
The following code shows how you can create a compressed BytesMessage and publish it onto a topic:

InputStream in = null;
GZIPOutputStream out = null;
try {
  ByteArrayOutputStream bos = new
                            ByteArrayOutputStream(1024 * 64);
  out = new GZIPOutputStream(bos);

  String filename = "input.xml";
  in = new BufferedInputStream(new FileInputStream(filename));

  byte[] buf = new byte[1024 * 4];
  int len;
  while ((len = in.read(buf)) > 0) {
      out.write(buf, 0, len);
  }
  out.finish();

  //publish it
  BytesMessage msg = session.createBytesMessage();
  msg.writeBytes(bos.toByteArray());
  publisher.publish(msg);
}
catch (IOException e) {
  e.printStackTrace();
}
finally {
  if (in != null) {
    try {
        in.close();
    }
    catch (IOException ignore) {
    }
  }
  if (out != null) {
    try {
        out.close();
    }
    catch (IOException ignore) {
    }
  }
}
Decompressing JMS messages
The following code shows how you can decompress a JMS BytesMessage when your subscriber receives it and write it to file:
if (mesg instanceof BytesMessage) {
 final BytesMessage bMesg = (BytesMessage) mesg;

 byte[] sourceBytes;
 try {
    sourceBytes = new byte[(int) bMesg.getBodyLength()];
    bMesg.readBytes(sourceBytes);
    System.out.println("Read " + sourceBytes.length + " bytes");
 }
 catch (JMSException e1) {
    throw new RuntimeException(e1);
 }
 GZIPInputStream in = null;
 OutputStream out = null;
 try {
    in = new GZIPInputStream(
         new ByteArrayInputStream(sourceBytes));
    String filename = "message.xml";
    out = new FileOutputStream(filename);
    byte[] buf = new byte[1024 * 4];
    int len;
    while ((len = in.read(buf)) > 0) {
        out.write(buf, 0, len);
    }
    System.out.println("Wrote to " + filename);
 }
 catch (IOException e) {
    e.printStackTrace();
 }
 finally {
    if (in != null)
        try {
            in.close();
        }
        catch (IOException ignore) {
        }
    if (out != null)
        try {
            out.close();
        }
        catch (IOException ignore) {
        }
 }
}

Thursday, August 12, 2010

Java Monitoring Tools

JDK 6.0 comes bundled with a number of handy, but often overlooked, utilities to monitor, manage and troubleshoot Java applications. They can be found in the bin directory of your installation. Here are a few of them explained:

1) jps - report java process status
This command prints information about active java processes running on a given machine. The output contains the JVMID, the name of the main class and any arguments passed to it or the JVM.

sharfah@starship:~> jps -lmv
3936 sun.tools.jps.Jps -lmv -Xms8m
5184 test.TestClient -Xmx1024m

2) jstat - statistics monitoring tool
This command allows you to monitor memory spaces of a JVM. For example, using the "-gcutil" option will show you the utilisation of the eden (E), survivor (S0, S1), old (O) and permanent (P) generations and how long the minor and major GCs are taking. You can gather these statistics continuously by specifying a sampling interval and how many samples you wish to take.

sharfah@starship:~> jstat -gcutil 5184 1s 5
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
  0.00   0.00   3.04  85.47  11.68      5    0.009     1    0.023    0.032
  0.00   0.00   3.04  85.47  11.68      5    0.009     1    0.023    0.032
  0.00   0.00   3.04  85.47  11.68      5    0.009     1    0.023    0.032
  0.00   0.00   3.04  85.47  11.68      5    0.009     1    0.023    0.032
  0.00   0.00   3.04  85.47  11.68      5    0.009     1    0.023    0.032

3) jstack - stack trace
Prints out a complete thread dump of your application (just like "kill -3"). Useful for investigating what your application is doing and identifying deadlocks.

4) jmap - memory map
Use this command to print a histogram of the heap to show you the number of instances of each java class and how much memory they are occupying. You can also dump the heap to a file in binary format and then load it into Eclipse Memory Analyser as I described here.

sharfah@starship:~> jmap -histo 5184
sharfah@starship:~> jmap -dump:format=b,file=heap.bin 5184

5) jhat - heap analysis tool
This command reads a binary heap file (produced by the jmap command, for example). It launches a local webserver so that you can browse the heap using a web browser. The cool thing is being able to execute your own queries using Object Query Language (OQL) on the heap dump.

sharfah@starship:~> jhat heap.bin

6) jinfo - configuration info
Prints out java system properties (like the classpath and library path) and JVM command line flags. Doesn't work on Windows though! Also allows you to enable/disable/change VM flags.

sharfah@starship:~> jinfo -flag PrintGCDetails  4648
-XX:-PrintGCDetails
sharfah@starship:~> jinfo -flag +PrintGCDetails 4648

Friday, July 02, 2010

Brain Teaser: Find the ages

Question:
I was visiting a friend one evening and remembered that he had three daughters. I asked him how old they were. "The product of their ages is 72," he answered. Quizzically, I asked, "Is there anything else you can tell me?" "Yes," he replied, "the sum of their ages is equal to the number of my house." I stepped outside to see what the house number was. Upon returning inside, I said to my host, "I'm sorry, but I still can't figure out their ages." He responded apologetically, "I'm sorry, I forgot to mention that my oldest daughter likes strawberry shortcake." With this information, I was able to determine all of their ages. How old is each daughter?

Answer:
The first clue is that the product of their ages is 72. Therefore, the list of all possible sets of numbers whose product is 72 is:

72 1 1
36 2 1
24 3 1
18 4 1
18 2 2
12 3 2
12 6 1
9 8 1
9 4 2
8 3 3
6 6 2
6 4 3
The second clue involves the sums of their ages which are:
72 + 1 + 1 = 74
36 + 2 + 1 = 39
24 + 3 + 1 = 28
18 + 4 + 1 = 23
18 + 2 + 2 = 22
12 + 3 + 2 = 17
12 + 6 + 1 = 19
9 + 8 + 1 = 18
9 + 4 + 2 = 15
8 + 3 + 3 = 14
6 + 6 + 2 = 14
6 + 4 + 3 = 13
Out of all these sums, two of them are equal:

8 + 3 + 3 = 14
6 + 6 + 2 = 14
The final clue is that his "oldest" daughter likes cake, which means that we can eliminate the second one of the above. Therefore, his daughter's ages are:

8 + 3 + 3 = 14
More Interview Posts

Wednesday, June 23, 2010

Power Set Algorithm

The power set of a set S is the set of all subsets of S including the empty set and S itself. For example, the power set of {1, 2, 3} is {{}, {1}, {2}, {3}, {1, 2}, {1, 3}, {2, 3}, {1, 2, 3}}. In an interview, I was asked to write an algorithm to find the power set of a given set.

The problem can be solved using recursion. We know that if the initial set has only one element {a} then its power set is {{}, {a}}.

Here is the algorithm:

public <E extends Comparable<E>> List<List<E>> powSet(List<E> list){

  List<List<E>> result = new ArrayList<List<E>>();
  if(list.isEmpty()){
    result.add(new ArrayList<E>());
    return result;
  }
  else if(list.size() == 1){
    List<E> empty = new ArrayList<E>() ;
    List<E> oneE = new ArrayList<E>(list) ;
    result.add(empty);
    result.add(oneE);
    return result;
  }
  else{
    E first = list.remove(0);
    List<List<E>> subsets = powSet(list);

    for(List<E> subset : subsets){
      result.add(subset);

      List<E> tmp = new ArrayList<E>(subset) ;
      tmp.add(first);
      Collections.sort(tmp);
      result.add(tmp);
    }
    return result;
  }
}

More Interview Posts

Friday, June 18, 2010

Design Pattern Interview Questions

  1. Explain the Flyweight, Builder, Mediator and Memento patterns.
  2. Describe the decorator pattern and how it is used within the java.io package.

    Answer: Decorators are used to provide additional functionality to an object of some kind. The key to a decorator is that a decorator "wraps" the object decorated and looks to a client exactly the same as the object wrapped. This means that the decorator implements the same interface as the object it decorates. For example, a BufferedReader is just a decorator for a Reader.

  3. Observer Pattern:
    • Draw the class diagram for the Observer Pattern.
    • What instance variables would you have in the Observable class?
    • What would happen if an Observer removes himself from the observers list when he is notified?
    • What is thread-safety and how can you make the Observer Pattern thread-safe?
  4. Name your favourite design pattern.
  5. Explain the Model-View-Controller (MVC) compound design pattern. What patterns is it composed of?
More Interview Posts

Thursday, June 17, 2010

Java Interview Questions II

Here are a few more java questions I have been asked in interviews recently:
  1. Garbage Collection:
    • How does Garbage Collection work?
    • Where does garbage collection start from?
    • When do static fields get garbage collected?
    • When does a thread get garbage collected?
  2. What is a potential problem with the following code? How can it be resolved?
    public void swap(Account a, Account b){
     synchronized (a) {
      synchronized (b) {
       double d = a.getBalance();
       double e = b.getBalance();
       a.setBalance(e);
       b.setBalance(d);
      }
     }
    }
    
    Answer: A deadlock will occur if one thread calls swap(a,b) and another calls swap(b,a) at the same time. The easiest way to resolve this issue is to remove the two synchronized blocks and make the method synchronized instead.
  3. How many occurences of "NO" does the following program print?
    
    public class Main {
    
      public static void test(String s, String t) {
        if (s == t) {
          System.out.println("YES");
        }
        else {
          System.out.println("NO");
        }
        if (s.equals(t)) {
          System.out.println("YES");
        }
        else {
          System.out.println("NO");
        }
      }
    
      public static void main(String[] args) {
        String s = new String("HELLO");
        String t = new String("HELLO");
        test(s, t); // no, yes
    
        s = "HELLO";
        t = "HELLO";
        test(s, t); // yes, yes
    
        s = s + "!";
        t = t + "!";
        test(s, t); // no, yes
    
        s = "HELLO" + "!";
        t = "HELLO" + "!";
        test(s, t); // yes, yes
      }
    }
    
  4. What does the following code print?
    String s = "Welcome!";
    s.substring(3, 7);
    System.out.println(s);
    
    Answer: Welcome!
  5. What are the possible outputs of the following program? What changes can you make to print out "HelloI HelloII HelloIII HelloIV" in order?
    public static void main(String[] args) {
     Thread t1 = new Thread(new Runnable(){
      public void run() {
       System.out.println("HelloI");
       System.out.println("HelloII");
      }
     });
     Thread t2 = new Thread(new Runnable(){
      public void run() {
       System.out.println("HelloIII");
       System.out.println("HelloIV");
      }
     });
     t1.start();
     t2.start();
    }
    
    Answer: Possible outputs are:
    HelloI HelloII HelloIII HelloIV
    HelloI HelloIII HelloIV HelloII
    HelloI HelloIII HelloII HelloIV
    HelloIII HelloIV HelloI HelloII
    HelloIII HelloI HelloII HelloIV
    HelloIII HelloI HelloIV HelloII
    
    To print them in order add t1.join() after t1.start().
  6. You have a list of integers. Write an algorithm to find the maximum of the differences of each element with those ahead of it in the list.
  7. What does the following code print?
    public class A {
    
      private String s = "A1";
    
      public A(){
        dump();
        s = "A2";
        dump();
      }
    
      public void dump(){
        System.out.println(s);
      }
    
      public static void main(String[] args) {
        A a = new B();
      }
    }
    
    class B extends A {
    
      private String s = "B1";
    
      public B(){
        dump();
        s = "B2";
        dump();
      }
    
      public void dump(){
        System.out.println(s);
      }
    }
    
    Answer:
    null
    null
    B1
    B2
  8. How would you increment a counter in a threadsafe way?
  9. How does AtomicInteger work? Does it use any internal synchronisation?
  10. What does the following code print?
    int i = -100;
    int a = ~i + 1;
    a >>= 2;
    System.out.println(a);
    
    Answer: 25
  11. Write a unix command to find all files containing the string "ErrorMessage".

    Answer: find . -type f -exec grep -l "ErrorMessage" {} \;

More Interview Posts

Java Interview Questions I

Here are some java questions I have been asked in interviews recently:
  1. Explain what the following JVM args are used for:
    • -Xss
    • -Xmx
    • -Xms
    • -XX:PermSize
    • -XX:NewSize
    • -server vs -client
  2. What is the difference between the heap and the stack? Explain what is present on the heap and stack when the following method is called:
    public void f(int i){
      Integer a = i ;
      double d = i ;
    }
    
    Answer: The primitive int 'i' is boxed into an Integer object and placed on the heap. So you have a "new Integer(i)" object present on the heap, and variables a, d and i on the stack (as well as the method f). [Further Reading]
  3. What are the main collections found in the Collections API?
  4. What is the difference between a List and a Set?
  5. How does a HashMap work?
  6. Given the following classes: Collection, Set, Map, List, ArrayList, Vector, LinkedList, Stack, HashMap, TreeMap, HashSet, TreeSet state which ones are interfaces, concrete classes and those that can be cast to Collection.
  7. What happens when the following code is executed?
    List<String> list = new ArrayList<String>() ;
    list.add("foo");
    list.add("bar");
    list.add("baz");
    
    for(String s : list){
        list.remove(s);
    }
    
    Answer: A ConcurrentModificationException is thrown.
  8. Given the following class:
    public class Person {
    
      private String name;
    
      public Person(String name){
        this.name = name;
      }
    
      public boolean equals(Object o){
        return name.equalsIgnoreCase(((Person)o).name);
      }
    
      public int hashCode(){
        return 0;
      }
    
      public static void main(String[] args) {
        Set<Person> set = new HashSet<Person>();
        set.add(new Person("David"));
        set.add(new Person("John"));
        set.add(new Person("DAVID"));
        set.add(new Person("Fred"));
    
        Map<Person, String> map = new HashMap<Person, String>();
        map.put(new Person("David"),"David");
        map.put(new Person("John"),"John");
        map.put(new Person("DAVID"),"DAVID");
        map.put(new Person("Fred"),"Fred");
      }
    }
    
    i) What is the size of the set and map? What is the result of map.get(new Person("David"))?
    ii) Delete the equals method and repeat i)
    iii) Delete the hashCode method and repeat i)
    iv) Delete both the equals and hashCode methods and repeat i)

    Answers:
    i) 3, 3, DAVID
    ii) 4, 4, null
    iii) 4, 4, null
    iv) 4, 4, null

  9. Describe the different kinds of exceptions?
  10. What happens when a RuntimeException is thrown but not caught by anything?
  11. How are JNI Exceptions handled?
  12. What does the following code print?
    public static void main(String[] args) {
     try {
      try {
       throw new Exception();
      }
      catch (Exception e) {
       System.out.println("FooI");
       throw e;
      }
      finally {
       System.out.println("FinallyI");
      }
     }
     catch (Exception e) {
      System.out.println("FooIII");
     }
     finally {
      System.out.println("FinallyII");
     }
    }
    
    Answer: FooI FinallyI FooIII FinallyII
More Interview Posts

Wednesday, June 16, 2010

Designing a Lottery System

I was asked this question in an interview: You need to design a database to hold customers and their lottery tickets. A lottery ticket has a sequence of 6 numbers e.g. 04 12 18 28 35 41. Once you have designed your tables, write a query which will print out a report of all customers whose tickets match three or more numbers of the winning number. Assume every customer has only one lottery ticket.

The simplest way is to have two tables:

  • Customer: with an id, name etc
  • Ticket: with an id, customer id and number (which will hold one of the numbers only, so you will get six records per ticket)
Given a winning number, the query to find all customers and how many numbers they matched would be:
SELECT c.name, COUNT(t.num) AS matches
FROM customer c, ticket t
WHERE c.id = t.customer_id
AND t.num IN
(
'05', /*winning number*/
'12',
'19',
'28',
'35',
'42'
)
GROUP BY customer
HAVING COUNT(t.num) >= 3
ORDER BY matches DESC

Tree Traversal

Given the following tree, how would you print out the nodes in the following order: 0, 1, 2, 3, 4, 5? (interview question)
     0
   /   \
  1     2
 / \   /
3   4 5
Breadth-first Traversal:
The problem can be solved with breadth-first (level-order) traversal, using an intermediate queue and iteration:
public void breadthFirstIterative(Node root){
  Queue<Node> q = new LinkedList<Node>();
  q.add(root);
  while(!q.isEmpty()){
    Node node = q.poll();
    System.out.println(node.data);
    if(node.left != null){
      q.add(node.left);
    }
    if(node.right != null){
      q.add(node.right);
    }
  }
}
You can also solve it using recursion, but this is not efficient and not recommended.
public void breadthFirstRecursive(Node root) {
  Map<Integer, List<Node>> map =
               new TreeMap<Integer, List<Node>>();
  breadthFirst2(root, map, 0);
  for (int i : map.keySet()) {
      List<Node> nodes = map.get(i);
      for(Node node : nodes){
          System.out.println(node);
      }
  }
}

private void breadthFirst2(Node node,
        Map<Integer, List<Node>> map,
                        int level) {
  List<Node> nodes = map.get(level);
  if (nodes == null) {
      nodes = new ArrayList<Node>();
      map.put(level, nodes);
  }
  nodes.add(node);
  if (node.left != null) {
      breadthFirst2(node.left, map, level + 1);
  }
  if (node.right != null) {
      breadthFirst2(node.right, map, level + 1);
  }
}
Depth-first Traversal:
Just for completeness, I will also show you the algorithm for depth-first traversal, which will not print out 0, 1, 2, 3, 4, 5 but will print out 0, 1, 3, 4, 2, 5. Here is the algorithm using a stack and iteration:
public void depthFirstIterative(Node root) {
    Stack<Node> s = new Stack<Node>();
    s.push(root);
    while (!s.isEmpty()) {
        Node node = s.pop();
        System.out.println(node);
        if (node.right != null) {
            s.push(node.right);
        }
        if (node.left != null) {
            s.push(node.left);
        }
    }
}
You can also solve it using recursion:

public void depthFirstRecursive(Node root) {
    System.out.println(root);
    if (root.left != null) {
        depthFirstRecursive(root.left);
    }
    if (root.right != null) {
        depthFirstRecursive(root.right);
    }
}

Monday, June 14, 2010

Detecting a Cycle in Linked List

How do you detect a cycle in a linked list? A cycle occurs if one node points back to a previous node in the list. You could solve this by using an intermediary data structure, but a more efficient algorithm is Floyd's "hare and tortoise" algorithm.

The idea is to have two iterators; the hare and the tortoise. The hare runs along the list faster than the tortoise and if the list has a cycle, the hare will eventually overtake the tortoise. The following code illustrates the algorithm:

public boolean hasCycle(Node root){
  Node tortoise = root;
  Node hare = tortoise.next;

  while (hare != tortoise) {
    // tortoise moves one place
    tortoise = tortoise.next;

    // hare moves two places
    hare = hare.next;
    if (hare != null) {
      hare = hare.next;
    }

    if (tortoise == null || hare == null) {
      // end of list, no cycles
      return false;
    }
  }
  return true;
}
(This is an interview question.)

Saturday, June 12, 2010

Mounting a Windows Hard Drive on Linux

Today, I had problems with my Windows hard drive. The operating system (XP) refused to start up and was complaining about corrupt system files. I wanted to back up my files in case things got worse. Luckily, I have a USB external hard drive enclosure and an Ubuntu Linux netbook so this is what I did:
  • I took out the hard drive from my Windows laptop.
  • I inserted it into my Akasa Integral External Enclosure and plugged it via USB into my Ubuntu netbook.
  • Ubuntu wasn't about to mount it automatically and gave me an error saying that the hard drive had problems and that I should "force" it to mount using a command.
  • I opened a terminal and typed the following commands to mount the disk manually:
  • $ sudo mkdir /media/disk
    $ sudo mount -t ntfs /dev/sdb1 /media/disk -o force
    $ cd /media/disk
    
  • I was then able to browse the contents of the Windows drive and back up important files.
  • Finally, I unmounted the drive as follows:
  • $ sudo umount /dev/sdb1
    

Saturday, May 08, 2010

Fun with Pre/Post Operators

Think you know how pre and post operators work? Test yourself by working out what the following programs print. Answers at the end of the post:

1.

void function(int val){
  if(val > 0){
    function(val-1);
  }
  print(val);
}

function(5);
2.
void function(int val){
  if(val > 0){
    function(--val);
  }
  print(val);
}

function(5);
3.
void function(int val){
  if(val > 0){
    function(val--);
  }
  print(val);
}

function(5);
4.
int a = 1;
b = a++ + ++a;
print(b);
Answers:
The first program is simple enough and prints out the numbers 5, 4, 3, 2, 1, 0.

The second program prints out the numbers 4, 3, 2, 1, 0, 0. --val decrements the value of the variable before assigning it. For example, if a=5, and b=--a, then b=4 and a=4.

The third program throws a StackOverflowError. val-- assigns the value of the variable first, before decrementing it. For example, if a=5, and b=a--, then b=5 and a=4.

The fourth program prints 4, because b = 1 + 3.

Tuesday, April 20, 2010

The Last Digit of a^b

Given two integer numbers: the base a (0 < a < 20) and the index b (0 <= b <= 2,147,483,000). You have to find the last digit of a^b.

This is quite a simple problem to solve, provided you know a few rules. Here is the code which is pretty self explanatory:

import java.util.*;

public class Main {
  public static int f(int a, int b) {

    if(b==0)return 1;

    //find last digit of a
    int da = a % 10;

    if(da==0)return 0;
    if(da==5)return 5;
    switch(b%4){
    case 0:
      return da%2!=0?1:6;
    case 1:
      return da;
    case 2:
      return da*da%10;
    default:
      return da*da*da%10;
    }
  }
  public static void main(String[] a) {
    Scanner c = new Scanner(System.in);
    c.nextLine();
    while(c.hasNext())
      System.out.println(f(c.nextInt(),c.nextInt()));
  }
}
Links:
SPOJ LASTDIG: The last digit

Monday, April 19, 2010

The Next Palindrome

I recently came across a challenging problem on SPOJ called The Next Palindrome. It involves finding the next palindrome given a very large integer (with more than 1000000 digits). For example, the next palindrome after 2133 is 2222.

A palindrome is any string which reads the same when read from left-to-right and right-to-left, for example, 818. Here is a method which tests if a string is a palindrome:

private static boolean isPalindrome(String s) {
  for (int i = 0, j = s.length() - 1; i < j; i++, j--) {
    if (s.charAt(i) != s.charAt(j)) {
      return false;
    }
  }
  return true;
}

The obvious way to solve the next palindrome problem is by brute-force i.e. keep incrementing the integer until you hit a palindrome.

public static String bruteForce(String s){
  BigInteger i = new BigInteger(s) ;
  while(true){
    i = i.add(BigInteger.ONE);
    final String result = i.toString();
    if(isPalindrome(result)){
      return result;
    }
  }
}
This approach is VERY slow for large numbers! A faster way is "mirroring" the number about its centre. For example, the mirror of 65432 is 65456, which is the next palindrome. The following method mirrors a string:
public static String mirror(String s) {
  final char[] arr = s.toCharArray();
  int midpoint = arr.length / 2;
  int i=arr.length%2==0?midpoint:midpoint+1;
  while (i < arr.length) {
    arr[i] = arr[midpoint - 1];
    i++;
    midpoint--;
  }
  return new String(arr);
}
BUT sometimes a mirrored number might be smaller than the original. For example, the mirror of 12345 is 12321, which is smaller. In this case, we need to increment the original number from the middle and then mirror it. So when 12345 is incremented from the middle, you get 12445 which mirrors to 12421, the next palindrome. Incrementing from the middle can get tricky if there is a 9 in the middle. For example, 12945 increments to 13045 which then mirrors to 13031. The following method increments a number from the middle:
public static String incrementFromMiddle(String s) {
  final char[] arr = s.toCharArray();
  final int midpoint = arr.length / 2;
  int currPoint=arr.length%2==0?midpoint-1:midpoint;
  boolean found = false;

  while (currPoint >= 0 && !found) {
    char c = arr[currPoint];
    char inc;
    if (c == '9') {
      inc = '0';
    }
    else {
      inc = (char) (c + 1);
      found = true;
    }
    arr[currPoint] = inc;
    if (!found) {
      currPoint--;
    }
  }

  if (!found) {
    // we have fallen off the start of the string
    // example 999 has become 009. Add a one on to give: 1009
    final char[] newarr = new char[arr.length + 1];
    newarr[0] = '1';
    System.arraycopy(arr, 0, newarr, 1, arr.length);
    return new String(newarr);
  }
  else {
    return new String(arr);
  }
}
Complete Code Listing:
That's it! Here is the complete java program to find the next palindrome: (Also available here)
import java.io.BufferedOutputStream;
import java.io.FileDescriptor;
import java.io.FileOutputStream;
import java.io.PrintStream;
import java.math.BigInteger;
import java.util.Scanner;

/**
 * Finds the Next Palindrome.
 */
public class Main {

  /**
   * Brute Force method.
   * Keeps incrementing by one until a palindrome is found.
   * @param s
   * @return the next palindrome
   */
  public static String bruteForce(String s){
    BigInteger i = new BigInteger(s) ;
    while(true){
      i = i.add(BigInteger.ONE);
      final String result = i.toString();
      if(isPalindrome(result)){
        return result;
      }
    }
  }

  /**
   * Mirrors a string around the centre.
   * Example: abc -> aba and abcd -> abba
   * @param s
   * @return mirrored string
   */
  public static String mirror(String s) {
    final char[] arr = s.toCharArray();
    int midpoint = arr.length / 2;
    int i = arr.length % 2 == 0 ? midpoint : midpoint + 1;
    while (i < arr.length) {
      arr[i] = arr[midpoint - 1];
      i++;
      midpoint--;
    }
    return new String(arr);
  }

  /**
   * Increments the middle digit.
   * Example: 12345 -> 12445 and 1234 -> 1334
   * 9s get incremented to 0 and carried over.
   * Example: 12945 -> 13045
   * @param s
   * @return incremented string
   */
  public static String incrementFromMiddle(String s) {

    final char[] arr = s.toCharArray();
    final int midpoint = arr.length / 2;
    int currPoint = arr.length % 2 == 0 ? midpoint - 1 : midpoint;
    boolean found = false;

    while (currPoint >= 0 && !found) {
      char c = arr[currPoint];
      char inc;
      if (c == '9') {
        inc = '0';
      }
      else {
        inc = (char) (c + 1);
        found = true;
      }
      arr[currPoint] = inc;
      if (!found) {
        currPoint--;
      }
    }

    if (!found) {
      // we have fallen off the start of the string
      // example 999 has become 009. Add a one on to give: 1009
      final char[] newarr = new char[arr.length + 1];
      newarr[0] = '1';
      System.arraycopy(arr, 0, newarr, 1, arr.length);
      return new String(newarr);
    }
    else {
      return new String(arr);
    }
  }



  /**
   * Next Palindrome using the mirroring approach.
   * @param s
   * @return
   */
  public static String getNextPalindrome(String s) {
    String mirrored = mirror(s);

    //the mirror might be smaller than original, so increment it and try again.
    if (compare(mirrored, s) <= 0) {
      mirrored = mirror(incrementFromMiddle(s));
    }
    return mirrored;
  }

  /**
   * @param s
   * @return true if the specified string is a palindrome.
   */
  private static boolean isPalindrome(String s) {
    //compare first and last chars, and so on...
    for (int i = 0, j = s.length() - 1; i < j; i++, j--) {
      if (s.charAt(i) != s.charAt(j)) {
        return false;
      }
    }
    return true;
  }

  /**
   * Compares two numerical mirrored strings.
   * Assumes they have the same length.
   * Only compares the second half of the strings.
   * @param s
   * @param t
   * @return -1, 0 or 1 if s<t, s==t or s>t
   */
  private static int compare(String s, String t){
    //only check second half
    final int midpoint = s.length() / 2;
    int i = s.length() % 2 == 0 ? midpoint : midpoint + 1;
    for (; i < s.length(); i++) {
      if(s.charAt(i) < t.charAt(i)){
        return -1 ;
      }
      else if (s.charAt(i) > t.charAt(i)){
        return 1;
      }
    }
    return 0;
  }

  /**
   * The main method.
   *
   * @param args
   */
  public static void main(String[] args) {
    // fast output as there is no flush on \n
    final FileOutputStream fdout =
         new FileOutputStream(FileDescriptor.out);
    final BufferedOutputStream bos =
         new BufferedOutputStream(fdout, 1024);
    final PrintStream ps = new PrintStream(bos, false);
    System.setOut(ps);

    try {
      final Scanner scanner = new Scanner(System.in);
      scanner.next();
      while (scanner.hasNext()) {
        final String s = scanner.next();
        System.out.println(getNextPalindrome(s));
      }
    }
    catch (Exception e) {
      e.printStackTrace();
    }
    finally {
      ps.close();
    }
  }
}
Links:
SPOJ PALIN: Next Palindrome

Monday, March 08, 2010

Null Characters

On a few occasions (normally when I am running out of disk space), I have seen ^@ symbols appear in my log files. These are "file holes" and contain null characters. The null character (or NUL char) has an ASCII code of 0 and appears as ^@ when viewed in 'vi' or 'less'.

Create a dummy file containing null characters:
In order to create a file with null characters, simply print \000. For example:

sharfah@starship:~> perl -e \
'print "hello \000world\000\nfoo bar\n";' > file-with-nulls
sharfah@starship:~> less file-with-nulls
hello ^@world^@
foo bar
Find lines containing null characters:
Use the following command in order print out lines containing null characters:
sharfah@starship:~> perl -ne '/\000/ and print;' file-with-nulls \
| less
hello ^@world^@
You can also perform an octal dump of the file to check if it has null characters:
sharfah@starship:~> od -b file-with-nulls | grep ' 000'
0000000 150 145 154 154 157 040 000 167 157 162 154 144 000 012 146 157
Delete null characters:
There are various ways in which this can be done. In the following examples, I have used tr and sed to remove the unwanted characters.
sharfah@starship:~> tr -d '\000' < file-with-nulls | less
hello world
foo bar
sharfah@starship:~> sed 's/\x0//g' < file-with-nulls | less
hello world
foo bar
Do not use the strings command because it will create a newline when it encounters a null characeter. "The strings utility looks for ASCII strings in a binary file. A string is any sequence of 4 or more printing characters ending with a newline or a null character."
sharfah@starship:~> strings file-with-nulls | less
hello
world
foo bar

Further reading:
ASCII Character Set
Identifying and removing null characters in UNIX [stackoverflow]
File holes

Friday, February 26, 2010

HSQLDB - Java In-Memory Database

I've started using HSQLDB because of its in-memory database capabilities. My app reads a file, builds a database and populates it with data. It then allows me to run SQL queries against the data very easily. HSQLDB has many other useful features (like persistence) which I have not tried (nor have the need for) yet.

It is really easy to get up and running with HSQLDB.

  • Download HSQLDB from here.
  • Add hsqldb.jar to your classpath.
  • Create a connection like this:
    Class.forName("org.hsqldb.jdbc.JDBCDriver");
    Connection conn = DriverManager.getConnection(
                         "jdbc:hsqldb:mem:mydb", "SA", "");
    
  • Create a table like this:
    String bookTableSQL = "create memory table MY_TABLE ("+
    " TITLE varchar(256) not null primary key,"+
    " AUTHOR varchar(256) not null"+
    ");"
    Statement st = conn.createStatement();
    st.execute(bookTableSQL);
    
  • You can then insert records into the table and query it just as you would with an ordinary database table.
HSQLDB also has Text Tables which allows you to treat CSV (or other delimited) files as SQL tables.

Wednesday, February 24, 2010

Freezing columns in a JTable

If you have a really large JTable, you might sometimes want to "freeze" the first column, or the first X columns, so that they do not scroll with the rest of the table. It is quite easy to do so. All you have to do is create a second JTable with the data you want to freeze and use it as the row header view on your scroll pane. I have wrapped this functionality up, in a class called FrozenTablePane to which you provide a JTable and the number of columns to freeze.
/**
 * A ScrollPane which "freezes" the specified number of
 * columns of a JTable.
 *
 * @author fahdshariff
 *
 */
import javax.swing.JScrollPane;
import javax.swing.JTable;
import javax.swing.JViewport;
import javax.swing.table.DefaultTableModel;
import javax.swing.table.JTableHeader;
import javax.swing.table.TableModel;

public class FrozenTablePane extends JScrollPane{

  public FrozenTablePane(JTable table, int colsToFreeze){
    super(table);

    TableModel model = table.getModel();

    //create a frozen model
    TableModel frozenModel = new DefaultTableModel(
                                model.getRowCount(),
                                colsToFreeze);

    //populate the frozen model
    for (int i = 0; i < model.getRowCount(); i++) {
      for (int j = 0; j < colsToFreeze; j++) {
        String value = (String) model.getValueAt(i, j);
        frozenModel.setValueAt(value, i, j);
      }
    }

    //create frozen table
    JTable frozenTable = new JTable(frozenModel);

    //remove the frozen columns from the original table
    for (int j = 0; j < colsToFreeze; j++) {
      table.removeColumn(table.getColumnModel().getColumn(0));
    }
    table.setAutoResizeMode(JTable.AUTO_RESIZE_OFF);

    //format the frozen table
    JTableHeader header = table.getTableHeader();
    frozenTable.setBackground(header.getBackground());
    frozenTable.setAutoResizeMode(JTable.AUTO_RESIZE_OFF);
    frozenTable.setEnabled(false);

    //set frozen table as row header view
    JViewport viewport = new JViewport();
    viewport.setView(frozenTable);
    viewport.setPreferredSize(frozenTable.getPreferredSize());
    setRowHeaderView(viewport);
    setCorner(JScrollPane.UPPER_LEFT_CORNER, frozenTable.getTableHeader());
  }
}

Tuesday, February 23, 2010

Display any ResultSet in a JTable

I have created an extension of JTable, called ResultSetTable, to display any SQL ResultSet. It first gets the column names by looking at the meta data of the result set and then builds the table.
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;

import javax.swing.JTable;
import javax.swing.table.DefaultTableModel;

/**
 * A JTable used to display a SQL ResultSet.
 * @author fahdshariff
 *
 */
public class ResultSetTable extends JTable{

  private final DefaultTableModel dataModel;

  public ResultSetTable(ResultSet rs)
                       throws SQLException{

    super();
    dataModel = new DefaultTableModel();
    setModel(dataModel);

    try {
      //create an array of column names
      ResultSetMetaData mdata = rs.getMetaData();
      int colCount = mdata.getColumnCount();
      String[] colNames = new String[colCount];
      for (int i = 1; i <= colCount; i++) {
        colNames[i - 1] = mdata.getColumnName(i);
      }
      dataModel.setColumnIdentifiers(colNames);

      //now populate the data
      while (rs.next()) {
        String[] rowData = new String[colCount];
        for (int i = 1; i <= colCount; i++) {
          rowData[i - 1] = rs.getString(i);
        }
        dataModel.addRow(rowData);
      }
    }
    finally{
      try {
        rs.close();
      }
      catch (SQLException ignore) {
      }
    }
  }
}

Tuesday, February 16, 2010

Eclipse Memory Utilisation

On my PC, Eclipse is currently hogging 518 MB of RAM! Want to find out why Eclipse is using up so much memory? This is how:
  • Find the process id of Eclipse. You can use the jps command to help.
  • Now dump out the java object heap using the jmap command. For example: jmap -dump:format=b,file=C:\temp\eclipse_heap.bin 3080
  • Open the heap file (C:\temp\eclipse_heap.bin) in Eclipse.
Eclipse will take a couple of minutes to load the object heap into its Memory Analyser. It will then provide lots of useful information such as:
  • A Leak Suspects Report: My main suspect was org.maven.ide.components.nexus_indexer occupying 24,201,456 (24.29%) bytes.
  • A Histogram which lists the number of instances per class. There were 583,760 char[] objects and 439,572 String objects on my heap.
  • Dominator Tree which lists the biggest objects and what they keep alive. Mine were:
    • org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader @ 0x3f46368 org.maven.ide.components.nexus_indexer (23.1 MB)
    • org.eclipse.jdt.internal.core.JavaModelManager @ 0x4710728 (15.2 MB)
    • org.eclipse.jdt.internal.core.search.indexing.IndexManager @ 0x475c3e8 (9.7 MB)
    • org.eclipse.jdt.internal.ui.text.spelling.SpellCheckEngine @ 0x3d617a8 (6.5 MB)
  • Top Consumers which lists the most expensive objects grouped by class and package.
As a result of my findings, I have now removed the Maven Plugin and have also disabled the Spell Checker. Memory is down to 356 MB and the biggest object is org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager (26.1 MB).

Further Reading:
Eclipse Memory Analyzer, 10 useful tips/articles