Data Synchronization

There are topics for which I claim to be an expert.  This is not one of them.  I say this because its one of those things that doesn’t seem to have a fully graspable scope.  I’d say I’m somewhat of an expert on SQLite and Android.  Its easy to say that because there are a limited number of topics about which one might care about, and if there’s something specific that needs research, you can kind of “dig it out” and test it.

One can grasp the scope.  Every project is different, but what’s core to SQLite doesn’t change all that much.

How do you become an expert in data synchronization?  It seems like every app is completely different.  Also, worse, its one of those problems that seems simple on a cursory look, so I find “experts” looking for generic or even automatic solutions.

Also also, when I’m talking about “data synchronization”, I’m mostly talking about data on a mobile client syncing with the server.  Not so much multiple devices with each other, or server push-based.  Just FYI.  I’m definitely no expert on those.  Anyway…

I think looking for something automatic is a waste of time, in most cases.  I’ll admit some level of insecurity here.  I say automatic, or at least heavily generic, isn’t realistic, but in the back of my mind I keep expecting some genius to prove me wrong.  Over time, though, I expect that less and less.

The problem isn’t so much that you can’t point a smart process at a database and sync rows.  You can do that.  You could probably add an “updated” flag, set up some rules, and even detect rudimentary dependencies based on joins.  You could also explicitly describe dependencies, and even specify what happens when syncs collide.  Maybe a callback!  Then you could even provide for some type of fallback scenario where native code gets called for weird data entities, etc.

What you’re really doing there is pulling the sock inside out.  You wind up with app-specific syncing logic.  Its just put in a different place.  Data syncing seems to be one of those “One Ring” developer problems.  The domain is deceptively simple when looked at in the abstract, and a “smart” developer could code a framework to make it all happen.  A more obvious example would be using a CMS as a base for your web application.  At some point, the majority of your screens are custom coded, and you’re probably spending more time coding around the generic aspects of the CMS (ie. don’t base a custom web app on WordPress, drupal, etc, unless you’re serving mostly content).

BTW, if you didn’t get the “One Ring” reference, you’re probably not nerdy enough for this post.  Its from LOTR.

That was a long disclaimer.  Summary, I think data synchronization is mostly domain specific, so you shouldn’t spend too much time trying to do it “generically”.  However, I love being proved wrong.

A Tale of two Apps

I’m going to make up some terms and concepts as I go along here, and I am drinking a little bit of Kentucky’s finest as I do it (Bulleit), so bear with me.  I can’t blame spelling errors on autocorrect this time.

For my money, there are basically two types of synchronization mechanisms: state-based and command-based.  

State-based looks at the data to see if anything has changed since the last sync.  Dirty flag, or its more informative cousin, the “updated at” column.  This is simple.  If its updated, sync it.

Command-based is a little more complex.  As the app updates data locally, it records commands locally to update the server.  For example, if you added an email to a contact, you might have an “add email” command.  You might make it simpler with an “update contact” command, but there are good reasons not to do that too much.

State-based isn’t bad.  It is certainly the simpler of the two, and if you’re coding from scratch, is probably the safer.  The logic is (relatively) simple.  When you sync, rip through the db table, look for the dirty flag, send data.  Rinse and repeat.  You need to make sure you get some form of success code from the server before clearing the flag, but if your local state is stable, there’s not too much that can go wrong.

State-based falls apart when your logic gets beyond simple.  It cannot easily handle dependencies beyond direct parent-child.  Say you had an app for managing little league teams.  Each player would belong to a team, and each team would have a captain (who is also a player).  Can’t “sync” the team without the captain, can’t add a player without the team.  You CAN sort out that logic, but it with multiple dependencies.  Welcome to bug-town.  It would also be a nightmare if you had data that isn’t in your database (for example, uploading media).

Command-based is not a walk in the park.  First of all, its is not comforting.  There’s that nagging fear of somehow dropping a command.  A server update may fail, which may leave something locally different than whats on the server, and your app won’t ever try again.  What happens to everything else in the chain?  What if everything else depends on it?

That is the nightmare scenario of command-based.  If you’re not careful, you can seriously mess things up.  I do not have a magic solution.  You first need to accept that remote syncing is not easy.  Its so not-easy, I tend to classify app development difficulty into the following buckets:

1. All local – easy

2. All download (reader app) – easy

3. All upload (picture app) – easy

4. Up/down, no syncing (requires data connection, no local storage) – medium/easy

5. Up/down, with syncing (can be offline) – difficult to very difficult

A full syncing app, with offline capability, can be MUCH more difficult than the other types.  This extends beyond just the syncing mechanism.  You have the whole “server” thing to deal with, and very often, that is coded by somebody else.  Touch Track, our testing framework, was built specifically for this problem.  That’s a blog post for another day, though.

Rambling.  Kentucky and time are reducing my efficiency.

Lets assume I’ve convinced you that offline syncing is hard, that command-based can be scary, but complex apps would be difficult without it.  As a former boss would often say, careful with that ax Eugene.  The implication being, its sharp.  Sometimes you need sharp tools, but, you know, be careful.

The Superbus!

First of all, the name will probably change.  Superbus is our open source command-based data syncing framework.

https://github.com/touchlab/Superbus

As you can probably guess, it does not try to automate syncing your db to your server process.  What it does is put some sanity around command-based syncing.  At a very high level, its an Android Service that accepts commands, implemented by you, and processes them in order.  That’s basically it.  The details, of course, are somewhat more involved.

The first decision with the bus is how you’re going to persist your commands.  This is critical.  A sync service wouldn’t be much use if you could lose your commands on app shutdown.  you can implement your own storage, but generally you’d want to use SQLite, or our Json file provider.

SQLite seems a little more secure.  SQLite is very good at not corrupting itself, and if something goes wrong, it goes really wrong, so while you might get an error, you’ll know about it.  The down side, you’ll either need many tables to store your data, or you may need to keep custom data in a text field (or whatever).  Also, unless you have some exotic logic, you’ll need to have a single “Command” class, and have either a branch or some kind of dispatch to your custom classes.  Blah.

Json is more flexible.  Assuming the file system is sufficiently stable, and I’m going to assume linux is, you shouldn’t have an consistency issues, but its always a nagging concern in the back of my mind because this provider is new.  However, you can put custom data structures into the json, and the provider knows how to instantiate different command classes by name.  This means you don’t need some central “Command” instance like the SQLite provider.

A third implementation, which I’m just now thinking of, would blend the two.  Keep a SQLite database with a classname field, and a json field.  Best of both worlds.  Coming soon.

If persistence isn’t critical, you can skip that part.  We’ve done this, but the value is limited.  You’re probably better off with an intent service, or something similar.

Command instances have some specific callbacks, and methods can throw specific exceptions.  If you need to do something when errors happen, you can implement the error method.  From here you can roll back changes, show a notification, etc.  Sync code is inherently specific to the data being synced.  Sometimes you can respond automatically.  Sometimes the user needs to do something.

For exceptions, you can throw either TransientException or PermanentException.  So far, the only thing I’ve ever thrown a TransientException for is IOException, and only when its due to network issues.  The sync bus will make a few attempts, with delay, then go to sleep.  Your commands stay persisted, and will be attempted later under better conditions.  PermanentExceptions are thrown when you can’t recover.  Database issues, server errors, etc.  There’s a problem that won’t resolve itself.  Your command will be removed, the error method will be called, and you need to deal with it.

That’s basically it.  The logic is pretty simple.  Define your commands, submit them to the service, life is good.  Or not, if your back end is a disaster.

Some little tweaks.  You can define different priorities, if some operations should jump to the front/back.  For example, if you need to upload an image, and other operations don’t depend on it, I’d highly suggest giving that operation a low priority.  Commands process serially, so if something is taking a long time, the rest need to wait.

Speaking of serial processing, the next thing to be added is a way to process some commands in parallel.  The logic is still being worked out.  In its simple form, commands that don’t have dependencies could be marked “parallel”.  Until the next “serial” command is encountered, the service would process commands in parallel (with some sort of reasonable pool cap).

This is a little too simplistic, though.  Another option might be swim lanes.  If multiple families of commands are independent, it would make sense to process them independently.  You could do this now, though, by declaring multiple bus services, though, so the necessity of coding this specifically into the bus is dubious.

Seriously rambling now.  Must finish later.