Sometimes a casual pick can lead to big problems. Early on in developing our new social media product we decided to use JSON for storing our data in HDFS. This idea has worked out really well for us. The performance is good, it's easy to read if you want to take a peak at a file by hand, and there are libraries for every language under the sun, making it easy to write streaming jobs in your language of choice.

What didn't work out so well for us was the library we chose. The net.sf.json library is a big improvement over the code it's based on from www.json.org, but it has minor memory leak, apparently it is using a threadlocal hashmap of objects seen to make cycle detection faster. This fails miserably when we try to send gigabytes of data through the same Mapper object -- it holds a reference to every object deserialized.

Since we were unfamiliar with Hadoop when we started, we didn't identify the library as the cause of the problems. We expected some performance hiccups, and when we ran out of memory, we just gave it more. But still the problems didn't go away. Finally a heap dump pointed us in the right direction, and once we knew what to look for, it was easy to spot the suspect line of code.

At this point we did what we should have done at the outset: ask the mailing list. A quick note to the core-users Hadoop mailing list gave us one person who had previously had trouble with json-lib (and got it fixed quickly after reporting the bug), and two who recommended Jaskson JSON Processor. Jackson is even easier to use than json-lib.

Here's some sample code that shows how to read and write objects.

public class JsonUtils {

    private static final Log log = LogFactory.getLog(JsonUtils.class);

    private static final ObjectMapper mapper = new ObjectMapper();

    public static String jsonFromObject(Object object) {

        StringWriter writer = new StringWriter();

        try {

            mapper.writeValue(writer, object);

        } catch (RuntimeException e) {

            throw e;

        } catch (Exception e) {

            log.error("Unable to serialize to json: " + object, e);

            return null;

        }

        return writer.toString();

    }



    public static User userFromJson(String json) {

        return (User) objectFromJson(json, User.class);

    }



    static  T objectFromJson(String json, Class klass) {

        T object;

        try {

            object = mapper.readValue(json, klass);

        } catch (RuntimeException e) {

            log.error("Runtime exception during deserializing "

                    + klass.getSimpleName() + " from "

                    + StringUtils.abbreviate(json, 80));

            throw e;

        } catch (Exception e) {

            log.error("Exception during deserializing " + klass.getSimpleName()

                    + " from " + StringUtils.abbreviate(json, 80));

            return null;

        }

        return object;

    }

}

Feel free to use the above if you find it useful. It was written at home and I think is actually an improvement over what I wrote at work (the first time seeing the library).