Introduction

Today if you ever need to consume a web API, or produce one, chances are you’ll need to use JSON.
And there is good reasons why JSON has become so prevalent in the web world:

• first JSON is really well integrated with Javascript, the Esperanto of the web (and that’s not an understatement with the rise of server-side Javascript platforms like Node.js): the stringification of a Javascript objects tree gives a JSON document (but not all the JSON document are valid Javascript objects),
• secondly JSON is less verbose than XML (see my other article on the subject) and can be used for most scenarios where XML was historically used

So whatever the language and platform you’ll use you’ll need a strong JSON parsing component.

In the Java world there is at least two good candidates: Gson and Jackson.
In this article I’ll illustrate how to use Gson: I’ll start with a (not so) simple use case that just works, then show how you can handle less standard situations, like naming discrepancies, data represented in a format not handled natively by Gson or types preservation.

The source code, along with a set of JUnit tests, are available in this archive:

A simple Java-JSON mapping

The model

Here is a simplified representation of a file system:

    +--------------------+
    |   FileSystemItem   |
    +--------------------+
    | name: string       |
    | creationDate: date |
    | hidden: boolean    |
    +--------------------+
       ^               ^
       |               |
+------------+      +--------+
|    File    |<--*--| Folder |-----+
+------------+      +--------+     |*
| size: long |      |        |<----+
+------------+      +--------+

So File and Folder inherit from FileSystemItem, a File has a size and a Folder can contains any number of files and folders.
This model is interesting for at least four reasons:

• it uses all the primitives types of JSON: string, boolean and numbers, plus date which is not a native JSON type but is very common
• it uses complex objects and arrays
• it uses inheritance
• it uses a recursive data structure: the Folder

It’s a good illustration of all the interesting points of bidirectional Java-Json mapping.

The Java implementation

Here is an implementation of this model in Java:

FileSystemItem:

import java.util.Date;

public class FileSystemItem
{
	private String name;
	private Date creationDate;
	private Boolean hidden;
	
	public String getName() {
		return name;
	}
	
	public void setName(String name) {
		this.name = name;
	}
	
	public Date getCreationDate() {
		return creationDate;
	}
	
	public void setCreationDate(Date creationDate) {
		this.creationDate = creationDate;
	}
	
	public Boolean isHidden() {
		return hidden;
	}
	
	public void setHidden(Boolean hidden) {
		this.hidden = hidden;
	}
	
	public FileSystemItem(String name)
	{
		this(name, null);
	}
	
	public FileSystemItem(String name, Date creationDate)
	{
		this.name = name;
		this.creationDate = creationDate;
	}
}

File:

import java.util.*;

public class File extends FileSystemItem
{
	private long size;

	public long getSize() {
		return size;
	}

	public void setSize(long size) {
		this.size = size;
	}
	
	public File(String name)
	{
		super(name);
	}
	
	public File(String name, Date creationDate)
	{
		super(name, creationDate);
	}
	
	public File(String name, long size)
	{
		super(name);
		this.size = size;
	}
	
	public File(String name, Date creationDate, long size)
	{
		super(name, creationDate);
		this.size = size;
	}
}

And Folder:

import java.util.*;

public class Folder extends FileSystemItem
{
	private File[] files;

	public File[] getFiles() {
		return files;
	}
	
	private Folder[] folders;

	public Folder[] getFolders() {
		return folders;
	}
	
	public Folder(String name, FileSystemItem...items)
	{
		super(name);
		
		List<File> files = new ArrayList<File>();
		List<Folder> folders = new ArrayList<Folder>();
		
		for(FileSystemItem item : items)
		{
			if (item instanceof File)
			{
				files.add((File)item);
			}
			else
			{
				folders.add((Folder)item);
			}
		}
		
		this.files = files.toArray(new File[files.size()]);
		this.folders = folders.toArray(new Folder[folders.size()]);
	}
}

OOP design pedantic guy, before you ask: yes FileSystemItem could/should have been abstract but this is intended to demonstrate the type lost issue.

Mapping with Gson

With Gson the class responsible for serializing from Java to JSON and deserializing from JSON to Java is named, logically enough, Gson.
More specifically we respectively use the “toJson” and “fromJson” methods.

Let’s try to JSON serialize this file system tree:

+ /
|
+----+ /tmp
|    |
|    +--+ cache.tmp
|
+----+ /opt
|    |
|    +--+ /opt/Chrome
|       |
|       +--+ /opt/Chrome/chrome
|
+----+ /home
     |
     +--+ /home/john
     |
     +--+ /home/kate

Here is a Java code that creates the tree, serializes it to JSON and deserializes it back to Java:

Date time = Calendar.getInstance().getTime();
	
Folder root = new Folder
("/",
		new Folder("tmp",
				new File("cache.tmp", time)),
		new Folder("opt",
				new Folder("Chrome",
						new File("chrome", 123456))),
		new Folder("home",
				new Folder("john"),
				new Folder("kate"))				
);
	
Gson gson = new Gson();
String json = gson.toJson(root);
	
Folder outputRoot = gson.fromJson(json, Folder.class);

The JSON representation of the tree, stored into the “json” variable, is:

{
  "files" : [  ],
  "folders" : [ { "files" : [ { "creationDate" : "Jun 8, 2013 4:41:29 PM",
              "name" : "cache.tmp",
              "size" : 0
            } ],
        "folders" : [  ],
        "name" : "tmp"
      },
      { "files" : [  ],
        "folders" : [ { "files" : [ { "name" : "chrome",
                    "size" : 123456
                  } ],
              "folders" : [  ],
              "name" : "Chrome"
            } ],
        "name" : "opt"
      },
      { "files" : [  ],
        "folders" : [ { "files" : [  ],
              "folders" : [  ],
              "name" : "john"
            },
            { "files" : [  ],
              "folders" : [  ],
              "name" : "kate"
            }
          ],
        "name" : "home"
      }
    ],
  "name" : "/"
}

You can run the test named “canSerializeATree” to be convinced it works as expected.

Yes it’s as simple as that: 3 lines of code to serialize and deserialize … at least when you don’t have too specific requirements.

Helping Gson by customizing the JSON mapping

As you’ve seen in the first part mapping a relatively complex data structure was quite straightforward because Gson handles a great part of the hard-work.
But it can’t do everything and sometimes it needs some help to correctly serialize and deserialize.

In this second part I will address some common use-cases that require a customization of the mapping.

Naming discrepancies

The first use-case is maybe the more common, and if you’ve ever done some object XML mapping it’s likely you’ve encountered it: the properties names are not the same in the JSON document and in the classes.

This is so common that solving it is as simple as adding an annotation on the field: @SerializedName.

Here is a basic Mail class that uses this annotation:

import com.google.gson.annotations.SerializedName;

public class Mail
{
	@SerializedName("time")
	private int timestamp;
	
	public int getTimestamp()
	{
		return timestamp;
	}
	
	public Mail(int timestamp)
	{
		this.timestamp = timestamp;
	}
}

When we serialize it as JSON:

int timestamp = (int)new Date().getTime() / 1000;
Mail mail = new Mail(timestamp);
		
Gson gson = new Gson();
String json = gson.toJson(mail);

we obtain:

{"time":629992}

Instead of:

{"timestamp":629992}

You can test it yourself running the “canCustomizeTheNames” unit-test.

Mapping boolean integers

Recently I’ve used a third-party API that uses JSON but represents boolean values with integers, 1 for true and 0 for false; this is (or at least was) a common pattern for languages that does not benefit from a true boolean type.

Mapping from the third-party API

If you intend to map these values to your Java object representation that uses a “true” boolean value you’ll have to help Gson a little.
As an example here is a naive attempt:

Gson gson = new Gson();
File file = gson.fromJson("{hidden: 1}", File.class); 

When you run it Gson yells at you with this exception:

com.google.gson.JsonSyntaxException:
java.lang.IllegalStateException:
Expected a boolean but was NUMBER at line 1 column 11

Because indeed 1 is not a valid boolean value.

So we will provide Gson with a little hook, a custom deserializer for booleans, i.e. a class that implements the JsonDeserializer<Boolean> interface:

import java.lang.reflect.Type;
import com.google.gson.*;

class BooleanTypeAdapter implements JsonDeserializer<Boolean>
{
	  public Boolean deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException
	  {
		  int code = json.getAsInt();
		  return code == 0 ? false :
			  	 code == 1 ? true :
			     null;
	  }
}

To use it we’ll need to change a little the way we get the Gson mapper instance, using a factory object, the GsonBuilder, a common pattern in Java:

GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(Boolean.class, new BooleanTypeAdapter());
Gson gson = builder.create();
File file = gson.fromJson("{hidden: 1}", File.class); 

And this time no more exceptions and if you debug the “BooleanTypeAdapter” class you’ll see that as expected Gson calls its “deserialize” method; so we are now able to read the values sent by the third-party.

Mapping to the third-party API

But what if we use a two directional mapping: we not only need to deserialize but also to serialize back some data?

To illustrate the point say the third party API sends us this kind of file representation:

public class ThirdPartyFile
{
	private String name;
	private int hidden;
	
	public String getName() {
		return name;
	}
	
	public void setName(String name) {
		this.name = name;
	}
	
	public int isHidden() {
		return hidden;
	}
	
	public void setHidden(int hidden) {
		this.hidden = hidden;
	}	
}

If the client sends a file using its local File representation we obtain this JSON:

{name: "test.txt", hidden: true}

But if the third-party tries to parse this JSON, what we simulate with this code:

Gson gson = new Gson();
ThirdPartyFile file = gson.fromJson("{name: \"test.txt\", hidden: true}", ThirdPartyFile.class);

it will be surprised:

com.google.gson.JsonSyntaxException:
java.lang.IllegalStateException:
Expected an int but was BOOLEAN at line 1 column 14

This is the exact opposite of the error we got when trying to deserialize the third-party API JSON.

To ensure we produce JSON that is valid for the API we need to customize the serialization too, and this is as simple as implementing the JsonSerializer<Boolean> interface.

We don’t need to create a dedicated class, but can simply enhance our previous implementation with this new capability:

import java.lang.reflect.Type;
import com.google.gson.*;

class BooleanTypeAdapter implements JsonSerializer<Boolean>, JsonDeserializer<Boolean>
{
	  public JsonElement serialize(Boolean value, Type typeOfT, JsonSerializationContext context)
	  {
		  return new JsonPrimitive(value ? 1 : 0);
	  }
	
	  public Boolean deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException
	  {
		  int code = json.getAsInt();
		  return code == 0 ? false :
			  	 code == 1 ? true :
			     null;
	  }
}

Here is a program that simulates the interactions between the client and the third-party:

GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(Boolean.class, new BooleanTypeAdapter());
Gson gson = builder.create();

// We generate some JSON for the third-party API...
String json = gson.toJson(file);

// we send the JSON to the API...

// it parses the file in its own representation...
ThirdPartyFile otherFile = gson.fromJson(json, ThirdPartyFile.class);

// then it returns us the same file as its JSON representation...
String otherJson = gson.toJson(otherFile);

// we finally try to read it and ensure all is fine.
File outFile = gson.fromJson(otherJson, File.class);

This scenario has been implemented in the “canMapToAndFromThirdParty” JUnit test.

Mapping status codes

A similar use-case I’ve experienced with an API is the mapping of numerical error codes, like 0 and 1, to more meaningful enums values, like “OK” and “KO“.

Here is a simplification of the situation:

enum Status
{
	OK,
	KO
}

public class Response
{
	private Status status;
	
	public Status getStatus()
	{
		return status;
	}
	
	public Response(Status status)
	{
		this.status = status;
	}
}

If you naively try to parse some JSON like “{status:1}” the target status property will desperately remain null.

Once again, custom adapter to the rescue:

import java.lang.reflect.Type;
import com.google.gson.*;

class StatusTypeAdapter implements JsonSerializer<Status>, JsonDeserializer<Status>
{
	  public JsonElement serialize(Status value, Type typeOfT, JsonSerializationContext context)
	  {
		  return new JsonPrimitive(value.ordinal());
	  }
	
	  Status[] statuses = Status.values();
	  
	  public Status deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException
	  {
		  int code = json.getAsInt();
		  
		  return code < statuses.length ? statuses : null;
	  }
}

And here is a sample:

GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(Status.class, new StatusTypeAdapter());
Gson gson = builder.create();

Response responseOK = gson.fromJson("{status:0}", Response.class);
Response responseKO = gson.fromJson("{status:1}", Response.class);

As expected this time "responseOK" status is "Status.OK" and "responseKO" status is "Status.KO".
If you want to check it live run the "canMapStatusCodes" JUnit test.

Fixing the date format

Another common requirement when dealing with JSON is fixing the date representation to an agreed format.
E.g. if you're conversing with an API that uses an ISO 8601 format like "yyyy-MM-ddTHH:mm:ss", e.g. 2013-06-08T18:05:23, then you need to inform Gson it should use another date format to interpret the dates it reads and to serialize the dates it writes.
This is done using the "setDateFormat" method of the GsonBuilder class, here is an example:

Calendar calendar = Calendar.getInstance();
calendar.set(2013, 05, 8, 18, 05, 23);
Date time = calendar.getTime();

GsonBuilder builder = new GsonBuilder();
builder.setDateFormat("yyyy-MM-dd'T'HH:mm:ss");
Gson gson = builder.create();

String json = gson.toJson(time); 

json will be :

"2013-06-08T18:05:23"

instead of the default:

"Jun 8, 2013 6:05:23 PM"

The corresponding unit-test is named "canCustomizeTheDateFormatting".

Mapping a composite string

Sometimes the JSON representation of a data is compacted, as an example I've recently used an API that represents a conversion from a file format A to another format B as "A2B", e.g. "PDF2XLS", instead of using a more verbose representation like "{ from: "PDF", to: "XLS" }".

To show how you can map such representations with Gson I'll use a similar use case, currency pairs, a perfect candidate because the compacted version, e.g. EUR/USD, is almost always used.

So here is our CurrencyPair class:

public class CurrencyPair
{
	private String baseCurrency;
	private String counterCurrency;
	
	public String getBaseCurrency()
	{
		return baseCurrency;
	}
	
	public String getCounterCurrency()
	{
		return counterCurrency;
	}
	
	public CurrencyPair(String baseCurrency, String counterCurrency)
	{
		this.baseCurrency = baseCurrency;
		this.counterCurrency = counterCurrency;
	}
}

But we don't want its JSON representation to be:

{
    baseCurrency: "EUR",
    counterCurrency: "USD",
}

but simply:

"EUR/USD"

So once again we write a dedicated type adapter that will take care of serializing the compacted representation and deserializing the expanded one:

import java.lang.reflect.Type;
import com.google.gson.*;

class CurrencyPairTypeAdapter implements JsonSerializer<CurrencyPair>, JsonDeserializer<CurrencyPair>
{
	  public JsonElement serialize(CurrencyPair value, Type typeOfT, JsonSerializationContext context)
	  {
		  return new JsonPrimitive(value.getBaseCurrency() + "/" + value.getCounterCurrency());
	  }
	
	  public CurrencyPair deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException
	  {
		  String currencyPairStr = json.getAsString();
		  
		  String[] tokens = currencyPairStr.split("/");
		  
		  return new CurrencyPair(tokens[0], tokens[1]);
	  }
}

Here is a sample of round-trip:

CurrencyPair EURUSD = new CurrencyPair("EUR", "USD");

GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(CurrencyPair.class, new CurrencyPairTypeAdapter());
Gson gson = builder.create();

String json = gson.toJson(EURUSD);

CurrencyPair outEURUSD = gson.fromJson(json, CurrencyPair.class);

The unit-test for it is named "canWriteAndReadCurrencyPairs".

Preserving type information

One of the "shortcomings" of the reduced verbosity of JSON is that you lose the type of any serialized object whereas with XML, whether you want it or not, you keep it, which is one of the reason for the verbosity of XML.

A non-issue?

Let's see the difference with a concrete example, the representation of a File named "test":
In XML you get:

<file name="test" />

In JSON you get:

{ name: "test" }

So you lose the type information, is it a file, a folder, a user...? You can't say by just looking at the JSON document.
You have to rely on the type of the property corresponding to the object, e.g. if the file instance is a part of a Database class:

class Database
{
    File file;
}

it would be serialized as:

{ file:{
       name: "test"
   }
}

So the parser looks at the target property in the Database class, "file", and sees it's of type File so it knows that "{ name: "test" }" should represent a File instance and will correctly deserialize it; so seems like a non-issue...

Well, not always

But what if the target property can have more than one possible type, i.e. when it's using some form of polymorphism?

As an example we will create another folder class, say Folder2 (original isn't it?), but this time instead of storing the files and the sub-folders in two distinct collections we'd like to use only one collection, an array of their common super-class FileSystemItem.

public class Folder2 extends FileSystemItem
{
	private FileSystemItem[] items;

	public FileSystemItem[] getItems() {
		return items;
	}
	
	public Folder2(String name, FileSystemItem...items)
	{
		super(name, null, null);
		
		this.items = items;
	}
}

If you remember the previous Folder class you'll agree that this is a cleaner implementation with far less code and no more plumbing.

Using it we will reproduce this simple file system:

+ /
|
+----+ /tmp
|    
+----+ test.txt

So we start by naively using directly a Gson parser:

Folder2 folder = new Folder2("/", new Folder2("tmp"), new File("test.txt"));

Gson gson = new Gson();

String json = gson.toJson(folder);

And here is the resulting JSON:

{ "items" : [ { "items" : [  ],
        "name" : "tmp"
      },
      { "name" : "test.txt",
        "size" : 0
      }
    ],
  "name" : "/"
}

As you see we've lost all the typing information, and if we deserialize it we will obtain two FileSystemItem instances instead of a Folder2 and a File.

Indeed all the information that the parser has, when it looks at the items property, is it's an array of FileSystemItem.
You may reply that this is not completely true: the first object has an "items" property which means it can't be a File but only a Folder, and the second one has a "size" property so it can only be a File, and you would be right; but, first in the general case this is not true and you can have true ambiguity, and secondly imagine the guess of the JSON parser is wrong (because it can't know all the existing types which may exist in other jars for example) and you deserialize the wrong type, it could have really annoying consequences.

Note that the issue would be more noticeable if FileSystemItem was an abstract class because in this case even instantiating a FileSystemItem would be impossible.

The solution

So we need to preserve the type information by serializing it along with the data.
As always this is done by intercepting the normal serialization process but this time this is not trivial.
Hopefully a clever guy has already implemented all the necessary plumbing for us in the RuntimeTypeAdapterFactory class

When you add an instance of this class to the mapping process, each object in the generated JSON will be enriched with a new property that will holds the original type name; I've choosen to name it "$type" because it is obvious, with virtually no possibility of collision with a "normal" property, and this convention is used by another great JSON library, Json.NET.

Here is how to use it:

Folder2 folder = new Folder2("/", new Folder2("tmp"), new File("test.txt"));

RuntimeTypeAdapterFactory<FileSystemItem> factory = RuntimeTypeAdapterFactory.of(FileSystemItem.class, "$type")
		 .registerSubtype(File.class)
		 .registerSubtype(Folder2.class);

GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapterFactory(factory);
Gson gson = builder.create();

String json = gson.toJson(folder);

This time here is the resulting JSON:

{ "items" : [ { "$type" : "Folder2",
        "items" : [  ],
        "name" : "tmp"
      },
      { "$type" : "File",
        "name" : "test.txt",
        "size" : 0
      }
    ],
  "name" : "/"
}

And if we deserialize it we retrieve our original types: a Folder and a File.

The unit test used to check this is named "canPreserveTypeInformationForASimpleTree".

Conclusion

From now on you know how to use Gson to map an objects tree and to customize the mapping process when you have specific needs.

If you've used another library, like Jackson, to do a similar job I'd like to hear from you, so please let a comment with your feedback.

If you have any issue with the code, you've spotted a typo, or some questions don't hesitate to ask in the comments.

---

To follow the blog please subscribe to the RSS feed