REST MESS

04 December 2013

While working on a hobby project called freezr I came across a few assumptions I had made which turned out to be wrong. I’m going to write a bit about these assumptions, since I found solving the resulting problems very frustrating.

I had decided to write freezr API-first instead of UI-first. The reason for this decision was based on that

I had a very good understanding of the problem and what kind of actions it offered to the user, so there was no need to research the problem through UI prototypes etc. (If you do not have a good understanding of the problem, you should always start with UI mockups and prototypes.)
I am very much in favor of delegating the web server to provide interfaces to core logic and let the browser be the UI (e.g. HTML5 web application). This means that “web server’s” role is really to provide an HTTP API to the core services, and the only bit of user-visible “web serving” happens when it bootstraps the browser-based application.
I am not a UI designer nor UI developer. I’m much more a service architect and developer. I can do UIs that are best described as “engineering UIs”, functional, but not pretty and definitely not having thought too much about usability. In particular, I was hoping to get someone else to actually do the UI bit for me — another reason for postponing UI development.

So I worked on freezr for a few weeks, on and off, and got it to a situation where the service itself was functional (albeit lacking a lot of production-level stuff like authentication, access control etc.) and passed quite a lot of unit and integration tests. The integration tests drove the service via its defined REST API alone.

I wrote the server using Django. I had a few reasons to pick up Django, one of them being familiarity with it. That’s familiarity, not liking. I don’t like Django that much, I’ve struggled with it in the past, but I still find it a quick way to get an web app from zero to development demo. Albeit, it is always a bit of a frustrating experience. I could have used Flask, but I’m not as familiar with it, and the times I’ve used it I’ve found writing quite a bit of boilerplate code for things that come as default in Django.

(As a side note: I don’t like node.js, so I’m not going to use Meteor or its ilk. I find it frustrating to write in a language that has practically zero thought given towards developer friendliness, orthogonality or understandable error and exception handling. If I could decide, I’d replace JavaScript with a standardised and well designed bytecode interpreter where browsers would provide a JavaScript-to-bytecode compiler shim for backwards compability. It could even use LVVM representation directly. This would give it much better re-targetability from other languages.)

Anyway, I ended up using Django with the Django REST framework. I have worked with TastyPie and found it superlatively frustrating experience when trying to do anything “out of the tastypie box” so I was absolutely sure that I would not touch it even with a long stick (unless it had a very, very sharp end with a nuclear option installed). (TastyPie might have gotten better since, so you shouldn’t take my opinion as anything else than an opinion.)

So I wrote a REST interface using REST framework. I think it ended up nice and orthogonal. I especially liked the way how the framework made it easy to provide URIs for resource references. Like this (edited for brevity):


GET /api/account/1/
HTTP 200 OK

{
    "id": 1,
    "domain": "http://localhost:8000/api/domain/1/",
    "name": "AWS account",
    "access_key": "AKIAJH3LIPN74P3XO3UQ",
    "active": true,
    "projects": [
        "http://localhost:8000/api/project/1/",
        "http://localhost:8000/api/project/2/"
    ],
    "regions": [
        "us-east-1"
    ],
    "instances": [
        "http://localhost:8000/api/instance/13/",
        "http://localhost:8000/api/instance/14/",
        "http://localhost:8000/api/instance/15/",
        "http://localhost:8000/api/instance/16/",
        "http://localhost:8000/api/instance/17/",
        "http://localhost:8000/api/instance/18/"
    ],
    "updated": "2013-11-29T13:19:17.963Z",
    "log_entries": [
        {
            "type": "info",
            "time": "2013-11-28T20:58:19.900Z",
            "message": "Regions changed",
            "details": "Added: us-east-1\nRemoved: none"
        },
        {
            "type": "info",
            "time": "2013-11-28T20:58:38.929Z",
            "message": "Refreshed 1 regions in 1.15 seconds, total 6 / added 6 / deleted 0 instances",
            "details": null
        },
	...
    ],
    "url": "http://localhost:8000/api/account/1/"
}

Using URIs for resource references makes the whole API theoretically to have a very nice property: as long as the “root” point is known, it is possible to find all resources in the system without any need of the resource URL syntax. The interface itself will tell you that instance 15 is located at http://localhost:8000/api/instance/15/ without you having to know anything about the URL structure. For all you care, you could have instance 15 in a completely different URL from other instances like http://fnord:6643/ISTORE.JCL/?iid=i-6a56cd3. You, as a web browser application programmer would not have to do anything to support distributed resources!

I just love the idea. I though the REST API was just what REST is really meant to be — simple, using HTTP primitives, clean URLs, with the whole data model traversable without having to know about the particular service’s URL structure (the web server would tell the root URL during bootstrapping).

Hooray. Time to go do some UI development.

For the UI side I decided to try out Ember.js. I knew its data layer wasn’t yet final, but I thought, what the heck, I’m doing pretty simple REST API here, that shouldn’t be a problem.

It was.

This is not Ember’s fault in itself. It is just that Ember’s REST interface is designed to work with a particular flavor of REST interfaces. The REST API that I had defined did not conform to this model. I searched the net for a solution, and found ember-data-django-rest-adapter which … didn’t work out too well either. It is not final either so I should not expect too much, but it had the same problem as with Ember’s default REST adapter: it was making a lot of assumptions about the REST protocol. In particular, it didn’t work with resource URIs. Well, no problem, I can just HyperlinkedModelSerializer to get IDs instead. And it wanted to pluralize resources in URIs, e.g. a project was fetched from /api/project/ID/ but list of projects from /api/projects. Oh god. Then I found it actually was expecting hasMany relations as [{"id":1},{"id":2}] and not just [1,2].

No, I’m not going down that rabbit hole.

If there is competition for most stupid convention ever, I’d nominate the idea that computers are required to pluralize human words when using a computer-oriented API to distinguish between fetching a resource versus many resources. Quick, what’s the plural of “locus”? What if your API describes shoe pairs (e.g. shoes), is the resource point for fetching records of many shoes then “shoess”?

Frustration and amazement.

I came to realize that:

Most of the backend to browser development is done in a tightly linked manner. They are collectively developed and either both of those work well together (you picked rails-friendly framework for rails backend), or either the browser side gives in (custom resource adapter) or the server side caves in (doing whatever is required for the responses to conform to client expectations).
There is no universal “way of doing REST”. I though I understood this, but I had just thought the disperancies were in resource access and action definitions, not so much in how the resources are serialized and deserialized to/from JSON format.

(Example of different action definitions: In freezr, a project is frozen with a POST to /api/project/1/freeze/. Another and entirely valid choice would have been to apply PATCH to /api/project/1/ with content of { 'state': 'freezing' }, where instead of defining an action, the request would declare the desired state.)

In reality there are many ways to do these, and most frameworks are designed to work only with one particular REST protocol without thought given to reconfigurability for different use cases. (The configuration of REST adapters mostly concerns with endpoint URL and what combinations of operations is used for different idioms like is partial change PUT or PATCH, can you POST over an existing record?)
I don’t know jack shit.

To fix the last problem I’m going to do some research on different REST interface patterns and which server- and client-side frameworks use then, and write a follow-up blog post on what I find out.

REST MESS What's in an API?

REST MESS
What's in an API?