Android and Protocol Buffers

For quite a while now, I’ve been trying to figure out what one would use on a greenfield project for passing stuff back and forth to the server. There are many options. So far, in the projects we’ve worked on…

– Custom XML. That worked, but was rough.
– XML-RPC calls. Ehh.
– Json. Now I feel like we’re getting somewhere. The class definitions are pretty loose, but at least its structured data. Using it with Java can be kind of rough, as you wind up with really basic classes, but its still pretty good.
– Seam remoting. An Ajax api for Jboss Seam where you can pass and receive Java transfer object, and Javascript can read/write them. I wrote a client for Android way before the G1 came out. (By “way” I mean a few months)

What I really want is a way to deal with real objects and classes. Structured data. You can get a custom XML message and manually inflate that, but its extra work, and when things change, you don’t know about it automatically. Worse, on some projects, there’s no real documentation on the data you’ll get back, so we’ve been surprised on many occasions.

XML-RPC and Json does a pretty good job, but you still wind up with loose data structure. OK, but whatever.

The Seam Remoting project worked fairly well for me. Its VERY specific, in that you have to be using Seam on the server to use it, and its proprietary, which is generally ALWAYS bad, but it allowed me to write some simplified TO classes, and deploy them both on the server and phone, and use that to talk back and forth with the server.

Seam remoting falls apart in a few places. first of all, it uses XML to encode the data, and its really verbose. We used remoting at a place I used to work. They had a trading system and we used remoting to communicate with it. Just by implementing a few simple tweaks, I cut the full XML size down to about 1/3 of the original, and it was still beefy (and my cut down was not resilient to object structure changes, so if you deployed you needed to make sure the client refreshed Javascript).

With my new project I started using my seam remoting stuff. Now, it used to work fine, but I was using jdom instead of the regular XML classes. I converted it to the standard XML stuff, and promptly ran into a number of bugs. I started fixing them. Then realized, wow, what a waste of time.

Enter protocol buffers. In a nutshell, its a google product. You write message definitions in a text file, and compile them to the language of your choice. For Java (which is all I care about today), you get structured class files, which is exactly what I want, and you can stream/inflate them right into/out of streams and byte arrays. Perfect for the interweb. I’ll let you do the research on protocol buffers if you want, or click here: http://tinyurl.com/2v9wbj5.

Implementing them on Android, and in my case, a Servlet back end, is what I’m going to talk about here. Any way you can shove a byte array up to the server and back would allow you to use protocol buffers, but some ways are better than others. In the past, we’ve wound up passing files in a multipart message, but in this case, I made the body of the message the raw byte data. I would have to check with the actual protocol. I assume its still encoded in some way, like base 64, but possibly not, which would be sweet. In any case, having a singular object made the whole thing simpler.

On the phone, here’s the code I use to call a remote servlet with PB:

public static Object callServerWithPayload(String methodName, byte[] data, String... params)
{
DefaultHttpClient client = new DefaultHttpClient();

StringBuilder sb = new StringBuilder();
sb.append(SERVER_ADDRESS + SERVLET_PATH);
sb.append(“/”).append(methodName);

for (String param : params)
{
sb.append(“/”).append(URLEncoder.encode(param));
}

HttpPost method = new HttpPost(sb.toString());

method.setEntity(new ByteArrayEntity(data));

try
{
HttpResponse response = client.execute(method);
return response.getEntity().getContent();
}
catch (Exception e)
{
App.logException(e);
}

The way I call the servlet is as follows:

http://server.domain/rpcproto/methodName/param1/param2/param3

“rpcproto” is just the name of the servlet. Everything after that is part of the request.

“methodName” lets the server know what you’re trying to do.

The params are optional. Some calls actually don’t need to push an object to the server, so not needing a custom message object for those makes life a lot easier.

The ‘data’ byte array was passed into this method, but would have come from the ‘writeTo’ method on the actual PB data object. Here’s an example…

BaseResult.UserTO.Builder userTO = BaseResult.UserTO.newBuilder();

userTO.setName(“Mike Faker”);
userTO.setNotifyEmail(“fakeguy@rrespapps.com”);
userTO.setUsername(“fakeguy@rrespapps.com”);

BaseResult.UserTO userTOData = userTO.build();

ByteArrayOutputStream bout = new ByteArrayOutputStream(10000);
try
{
userTO.build().writeTo(bout);
}
catch (IOException e)
{
reportError(e);
}

So, build a PB object, stuff it into a byte array, send it to server. On the way back, just read from the input stream and inflate the data object:

HttpResponse response = client.execute(method);
InputStream inputStream = response.getEntity().getContent();
BaseResult.CreateUserResult createUserResult = BaseResult.CreateUserResult.parseFrom(inputStream);
output.setText("User id: "+ createUserResult.getUser().getUserId());

By the way, the “BaseResult” object is just what I called my .proto file, “base_result.proto”. All the data objects were included inside that.

On the server. You can actually just read from the input stream and write to the output stream directly. Its all pretty simple stuff.

String[] split = StringUtils.split(request.getPathInfo(), "/");

String methodName = split[0];

response.setContentType(“application/octet-stream”);

MobileClient mobileClient = (MobileClient) Component.getInstance(“mobileClient”);
ServletOutputStream outputStream = response.getOutputStream();

if (methodName.equals(“createUser”))
{
long userId = Long.parseLong(split[1]);

ServletInputStream inputStream = request.getInputStream();

BaseResult.UserTO userTO = BaseResult.UserTO.parseFrom(inputStream);

mobileClient.createUser(userTO, userId).writeTo(outputStream);
}
else
{
throw new RuntimeException(“Unknown method: ” + methodName);
}

outputStream.flush();

The “MobileClient” thing is actually an EJB with methods for our mobile support. Don’t worry about that.

Call this code from a ‘doPost’ implementation.

Simple. Fast. Use it.

Negatives of protocol buffers are:

– Data model is simplistic. No “Map” type, missing other primitive types (Date, etc).
– Language support is fairly small. C++, Java, and Python. There are several 3rd party implementations, but that’s always a little scary.

Negatives weren’t enough to dissuade me, and so far, its working out great.

-Kevin