Synchronization

I have spent the past two weeks procrastinating (to some extent) on working on my top secret project. The problem I've been struggling with is just a really hard one to solve and I've gotten quite used to instant gratification with my code. Much of what I've been working on has ended up with results by the end of a hacking session. Not so much with my current task -- synchronization.

One of the key features of the top secret project is that it is always available. Whether by a browser or a native app like an android client, the user is expected to be able to interact with the app no matter if they have internet connectivity or not. This means I have to spend a lot of time working on offline access as a requirement for letting anyone use the app. I thought I could get away with dogfooding my app while I was in Toronto, but I quickly realized that without 3G data on the phone the top secret project would just not function correctly.

One of the challenges I've faced so far is a temporal one. My first thought when deciding to do offline access was that the client would do the synchronization and call back to the server to push a canonical dataset into the datastore. After several nights of hacking I was fed up. I couldn't get the synchronzation to work at all. I found other things (like Statusinator) to work on instead.

On the flight to Chicago I had a "breakthrough" that really should have been my first thought. Do synchronization on the server side! My idea is as such:

1. Client creates local data, assigns hypothetically-unique UUID to record, tags it as existing locally only.



1. Stores:

   {"key": "possibly-unique-key", "value": "some-value", updated: "2008-10-27 04:41:01", "created": "2008-10-27 04:41:01", "is_pending": true}





2. Client requests sync session with server, gets data for min/max records to sync. **All further RPCs have a sync_uuid.



1. Client Sends:
   
   

   {"device_uuid": "some-possibly-id"}

2. Client Recieves:
   
   

   {"sync_uuid":
   "some-unique-session-based-on-device-uuid-and-user", "last_sync": null,
   "max_checkin": "2008-10-27 04:41:01"}



3. Client pushes record to server, is_pending to denote that the server is receiving a note with an unrecognizable key.



1. Client sends:
   
   

   [{"key": "possibly-unique-key", "value": "some-value", updated: "2008-10-27 04:41:01", 
   "created": "2008-10-27 04:41:01", "is_pending": true}]





4. Server processes record, changes the key to a valid server key, stores the original as an attribute on the record for future book keeping: local_id.



1. Server Stores:
   
   
   

   {"key": "some-real-key", "local_id": "possibly-unique-key", "value": "some-value", 
   updated: "2008-10-27 04:41:01", "created": "2008-10-27 04:41:01", "is_pending": false}





5. Client requests updates from the server, Server responds with all new or updated records modified serverside, along with the newly added records from step 3.



1. Client requests, asking for all records modified after last_sync (or all records, if None) but before max_checkin.
2. Server responds:
   
   
   

   [{"key": "some-real-key", "local_id":
   "possibly-unique-key", "value": "some-value", updated: "2008-10-27
   04:41:01", "created": "2008-10-27 04:41:01"}]





6. Client parses record for "local_id" attribute, and replaces the record in the local datastore with the copy from the server, stripping the local_id attribute and removing the pending bit.



1. Client stores:
   
   
   

   {"key": "some-real-key", "value": "some-value", updated: "2008-10-27 04:41:01", 
   "created": "2008-10-27 04:41:01", "is_pending": false}





7. Client tells server sync sesison is complete, using the newest record received from the "push/pull" to specifiy the end date of the session. (just as last_sync is the start). To save a write during the server-to-client record update, the client is the one noting the end date for the session here, instead of the server.



1. Client sends: last_update.
2. Server stores: stores last_update as last_sync.

I haven't thought this through all the way yet, but I think this will work just fine for a google-gears based browser client just as it will work for my android client. There is something that still bothers me and I'm having a hard time scoping it out in my head: What happens when clock scews occur?

Remaining questions:

• Do I leave around "local_id?" When is an appropriate time to strip those records? I don't want the server modifying the records withouth confirmation from the client that the info is no longer needed.
• What are the error condtions when the sync fails at each of the above steps, how does this pattern resolve conflicts that occur when records are modified after a failed sync?

Labels: code, project, Python

11:02 PM 0 comments links to this post

Catchup with two hands

its been six days and its time again to leave SF for a new place. I'm going to be in Toronto until the 15th. I'm in for work but ahould have plenty of time for fun. Heck, as I'm good at doing, I am in Toronto over a 3 day weekend too.

I have taken this whole week off of work, including Friday but excluding part of monday. In that time I have  been able to catch up on several months of life. I got a haircut, went clothes shopping and had several Hendricks (great gin, thanks for the recommendation josebiro) Martinis. I have also added a few more needed features to the top-secret prokect and am still looking for someone to do some UI design work, I could also use someone who is interested in mobile applications, python, java and/or javascript because I think I need more people to work on this if I'm going to get this completed in any reasonable amount of time.

On one hand, I see that I am making steady progress. Every day that I sit down and focus on it, I walk away at the end with something newer and better than before. On the otherhand, some of the ideas that propelled me to work on this project seem so far away in the future that I'm worried ill never get to the cool problems that will make this project stand out. Heck, I've no doubt pumped some people (including myself) up about this and I'm sure it won't meet anyone's expectations.

On the other hand, I'm still happy about the work that I have completed, I just wish it hadn't taken four months to get here. My thought is that in the next few days ill start using the app on my own and see if in its current state it is usable. If not, I'm going to have to take a hard look at what it will take to make my project get there. if not I will have several months of rewrite ahead of me and I will surely have lost the first to market edge.

Labels: Android, Java, project, Toronto, travel

7:04 PM 0 comments links to this post

Top Secret Project Update

I've spent another weekend hacking away at my top secret project and I feel that every day I work on it that I am getting closer to being able to use it. In fact, I'm just a few major-usability bugs away from using it day to day! After I spend a few weeks working on it, then a few more weeks fixing what I see as show stoppers I hope to be able to show it off to a couple of people.

Looking at what I've done so far this summer leaves me both depressed and excited. I'm a bit sad because I am so far away from what I hoped to have accomplished by this point. At the same time, I'm excited about what I have been able to do. I've learned a ton about Java and realized that it is a pretty awesome language. I feel it is a good way for me to stretch my brain. With python you can throw everything against a wall, 'import antigravity' and have the crap float away, leaving you with something usable. With Java, I feel I have to plan things out a bit better. I have to live with the consequences of my decisions and deal with every shortcut I inevitable take. As a result I'm much more careful about what I do. Even if I do take a shortcut, these days they don't tend to last very long as I get irritated with the way my code looks or interacts and I refactor until I'm happy.

One thing I haven't gotten around to learning is unit testing. I thought I would have more time to work on the project this summer and fully expected to be twice as far along as I am today, with solid unit testing coverage. Instead I'm not even at a usable point with my project and I've not written one Java unit test. The python side of my app has a few tests but still nothing worthy of being called "coverage."

Labels: code, Java, project, Python

12:12 PM 0 comments links to this post

Traveling with a Project

This summer I spent most of my time on the road for work. Somehow, I still managed to find some time to work on my project. While I didn't work on it as much as I did while I was home (probably 2-3 hours a week instead of 8) I was nevertheless able to make valuable progress on it. One of the things that helped was an obligation to my teammate to finish and another was that I kep in contact with my teammate. One of the best things you can do to keep interested in your project is to discuss it with someone who really cares. In my case I called my teammate at midnight (with Skype) a few times to discuss recent changes and code checkins. This allowed me to keep the project at the top of my mind and when an opportunity came around where I wasn't running around a new city drunk off my ass I knew that I should pull out my laptop and get working. It was successful. I was also lucky enough to have great flights both to and from Europe in which to get work done. I've always said that an airplane is one of the best places to concentrate. Even a crying child can be ignored with a decent pair of headphones. There is no internet, no where to walk, nobody to talk to, no toys/tech/games to distract and nothing but your laptop and the carefully prepared contents of its hard drive.

Labels: project, travel

11:27 AM 0 comments links to this post

How to work on a Project

This comes from a conversation I was having with my friend Chris the other day
about how you can keep yourself and friends working on a project.

So this is what worked for me... Something that has allowed my project to not die after a day of working on it -- what usually happens to projects that i start.

First and foremost: Get everyone to set aside a specific time each week to work on the project. Ideally it would be everyone at the same time at the same place, but thats not likely to be possible. At the very least you should have pairs of people.  Most of my friends are already highly motivated but having someone to bounce ideas off of is invaluable. If you have to wait until the next time someone is online that idea will probably bounce away instead of back. There are other reasons for this that I will come to shortly.

Because you're working on a personal project, the only initial rewards are going to be the feeling of satisfaction of having produced a good idea or from helping another person on your team.

Make sure everyone shares their ideas, and document those ideas. You'll need some reference material later when you thing "Well, why didn't we do this in the first place?" This is the second reason for having pairs or groups of people working at the same time. At the end of each session you can suggest people show their work with an example or through some documentation. This not only keeps people concentrating on goals but it establishes that each person on the team is working for someone else -- the most important part of this plan is to make sure people are vested in the work everyone is doing. It is vital that each member of the team takes as much or more pride in the work each of their teammates than their own.

Next, spend some time planning what the team is going to focus on. Even if you don't have a clear idea of what the project will be, give people focus area to lead. Have someone investigate javascript frameworks while you have someone else analyze some patterns for the client-server interaction. Or have someone start writing up use cases for how they expect the project to be used. If you immediately jump into writing code you might leave your teammates who have less a clear idea of waht to do in the lurch. Know that each person has some series of talents that nobody else does so be sure to spend the time upfront to realize what those are.

Do not expect the same quantity of work from each person, but do expect each person to make progress, even if that progress is a discussion about how they had to re-write some system for a third time. If someone is not completing any work you can expect that attitude to be more influential than that of the other people completing some tasks. Even if someone's time is spent researching, a short summary to another teammate or in a document will help spread some knowledge and capture some progress.

As the project moves along, the tasks people will be doing will tend to be more tightly coupled with the work from different people. By this point your team is probably working well together and because you've already encouraged people to help each other being a dependency or or dependent on someone else is not a new feeling.

Finally and most importantly, make sure everyone is having fun. The best way to kill a personal project it to treat your teammates poorly. The second best way is to make the work people are doing boring so be sure that people find something they enjoy to work on.

Labels: code, project

9:32 AM 0 comments links to this post

Android/App Engine prototyping problems

Almost a year ago, Google unleashed its Android SDK in a preview capacity. One of the pain points I experienced with it was the tedium in creating ContentProviders. Today, I'm feeling the same tedium. A ContentProvider is the sole mechanism for an android application to share information with external packages. ContentProviders can be backed by any kind of data, file, database or live server with custom backend. The popular method is creating a ContentProvider that is backed by a database. Creating such a beast is a time consuming operation and making it with a SQL database is quite contrary to the BigTable based datastore API in Google App Engine (GAE).
In my top secret project, I'm using the HttpClientService api that I've defined in my Missing SVN repository (the top-secret project is indeed different than Missing) to interface with my server. Currently, the client does no caching of data -- all information it needs is pulled at user request, from the interwebs. When testing on my local machine, this is not a problem. I have a fast workstation and there isn't much packet loss or latency on a loopback interface.

Unfortunately, this won't work in the "real world." I'm now at the point where I have to implement a sql-lite backed ContentProvider that tries to model a table-based GAE datastore api. Not only does this mean I have to keep track of two different schema, one for GAE and one for Android, but it means I have to implement the same "get my data" api twice, with a layer of abstraction between the android UI and the ContentProvider I'm resenting having to create.

What I'm considering doing is using something like Google's Protocol Buffers or Facebook's Thrift to define my data model and create stub interfaces for both GAE and Android. This seems like a bit of overkill for the current state of my project but even in the not-yet-ready-to-show-anyone stage of this project I'm having to consider these very-high-time-cost coding excersizes. This is going to consume time when I should be iterating on features and trying to get the app in a state where I can finally start to use it. I think I've proto-typed the android app as far as I can go but I don't want to start spending hordes of time on this when I have several core concepts not-yet-implemented.

I'm also feeling the pain of knowing that the stuff I prototyped at the beginning of the project are going to have to be fully re-written and if I don't provide a more featureful content provider.

One of the ways you can provide an abstracted interface to the ContentProvider is by wrapping it in a Service, which of course requires yet another interface declaration, this one with java primatives and simple classes using the AIDL interface spec provided by Android. I figure I'll have to cross that bridge some day, I'm just glad I don't have to do it right now.

Labels: Android, appengine, facebook, Google, Java, missing, project

5:21 PM 0 comments links to this post

Don't get distracted!

One of the things I've learned from starting my last few projects is that if you get distracted by project infrastructure you get distracted from what made you excited about the project in the first place.

An example would be a project Kevin and I tried starting a few months back -- an everyblock-like appliance that could sit on your mantle and passively provide you intersting information about your neighborhood. After several hours spent setting up an SVN repository, the correct mailing lists, researching frameworks and languages we were tired; not because of the amount of work involved, rather it was because we had grown weary of the boringness of the project.

Even though we hadn't written a single line of code for the project the energy we expended on the infrastructure was the exact energy we should have used to sketch out a more serious design, start prototyping and collaborating. Instead of getting more interested in our project we instead grew tired of it and like so many other projects, it died.

There are a few things you can do to mitigate this problem.

1. Spend some time *now*, before you have a project, researching project resources. Things like SVN repositories, mailing lists and web sites are easy to get going but suck up the most valuable moments of your project's birth.
2. Your initial decisions are both temporary and perminent. Think of it this way: You may decide to change from django, to ruby-on-rails, to C# but the design vs feature tradeoffs you think of with any of them will probably have lasting impact on your future design decisions.
3. Make sure you work on the project you were excited about, not some deriviative of it that you think you need to get working on the real project.

Labels: code, project

2:52 PM 0 comments links to this post

Design Patterns... BAH!

I started reading "Head First Design Patterns" last week, and taking advantage of my new-found knowledge I tried to refactor Missing's http client stack. Its taken me about eight hours and it works again. Its as ugly as it was before, maybe even more so.

The kookiness of the design stems from the fact that I want the ability to respond to an HTTP response, possibly from a different thread. Android has a cool thread/message-queuing class called a Handler, that allows you to post a Runnable to the handler and it will get executed on the thread in which the handler was instantiated.

Because I wanted to keep my http stack android-agnostic (not that I had a good reason for it, in fact I didn't even think of doing it at first.) I removed the knowledge of the Handler from the .http package.

Instead I created a decorator for the HttpResponseRunnable that does know about Android's handlers. For the sake of re-use on another project, I put it in the .http package... DOH!

On the positive side, I did manage to implement a design that allows me to dynamically create http requests and responses without having to subclass all the time. I don't know if in the long run that will be better.

Things I need to think about:

1. What is the best way to chain http operations where each request depends on data from the previous request?
2. How would I design the HttpService from the ground up? Where can I look at code that has similar features to the ones I'm coding up to see what design they used.
3. Why am I constantly re-writing this stack?
4. If I am going to have a large number of http callbacks, if there are many types of end points, do I want keep the ability to define them quickly, or do I go with something more verbose and more abstract?
   

Do you know Java? Want to discuss some of this with me? I'm a newb and could use insight from someone with more experience than I.

Labels: Android, code, Java, missing, project

4:42 PM 0 comments links to this post