Some random thoughts on webapp performance

A recent discussion got me thinking back in terms of website development, so I thought I'd jot down a few ideas that I'd like to try to implement when I get some free tuits. Maybe they'll spur some creativity in others, or at least you can tell me why I'm insane and this couldn't possibly work.

Server-side vs client-side rendering

Back when node.js was springing up in popularity, a lot of the focus around it was to do with how it uses an asynchronous event-loop to manage a lot of connections per process (since Javascript is single-threaded, parallelism is only managed by pre-emptive multitasking similar to how things worked back in the Windows 3.1 days).  I was confused by this for a few reasons.  Why not use process pooling to run multiple copies of the node server then?  You'd be able to peg each core with a separate process at least, and at least improve how many tasks you could do in parallel.  This would at least scale the number of requests you could serve simultaneously and maximize your hardware use.  Have each listen on a different port, throw a loadbalancer in front, voila!  Maybe they're doing that now, but there was no talk of it at the time.

Anyway, I've gone on a tangent.  For me, the killer use-case for having Javascript on the server is to share libraries on both server and client.  On the server side, your data source would be a database or an API or whatever you're using, on the client side, the server's REST interface would be the data source (or if you set up CORS, the original REST API could be the source on the client as well).  This would buy you a few things.

Pre-render the whole page server-side.  

One of the biggest problems with SPAs (Single Page Apps) is it completely offloads the rendering to the client.  You send down a minimal set of javascript code and templates, then the javascript code fires off a bunch of AJAX requests to pull down the data and renders the templates.  This works well on modern laptops and desktops, but for phones, resources are much more constrained.  Not only are you doing all of the rendering on their CPU, you're hammering the network with all of the AJAX requests to pull in data.  Rather than do that, you could, using the same exact rendering path as the client-side, pre-render the full page's initial state and send only that over the network.  Initial page load times will be much faster, and no more "the page is here, but wait while we load the data" frustration.

Render updates client-side.  

Now, here's the rub.  Once you have the page downloaded, you don't want to have to hit the server to re-render the whole page every time something changes.  That would just be silly.  Since you've got the same rendering paths available client-side, everything being Javascript and all, you could simply re-render the section of the page that was updated by the new data received by whatever AJAX request you sent.  Smaller, incremental updates done with a minimum of overhead.

It seems that my ideas were a bit ahead of their time.  I had a conversation with a coworker about this back in 2012 or so, and a recent poll suggests that this sort of dual-rendering path is finally becoming popular.

https://medium.com/javascript-scene/javascript-scene-tech-survey-d2449a529ed

The only advantage left for SPAs is that you can push out all of your content to the edges by using a CDN, but you can't push your REST interface to the edge, so all of those AJAX requests to grab data are still hitting the same limitation.  You could still put all of your Javascript and templates in a CDN and get some of the benefit there, I suppose.

Queued AJAX requests/multipart REST

Maybe something like this exists, but a quick googling hasn't found any results based on keywords I thought to look for.  I was thinking more about how to reduce the number of requests required by a website.  For context, at my previous job, we had a UI that was broken down into a bunch of components, and each component was responsible for updating itself via AJAX.  Some did this based on a timer, others did this based on a publish/subscribe framework that would alert them when their data source had potentially changed.  I wanted to expand this to use websockets to listen for updates coming in from the server-side as well, in case you had multiple users accessing the same account at the same time, but that use-case wasn't deemed common enough to justify the development effort, and the idea sat.  Anyway, in this case, we were very often firing off a bunch of AJAX GET requests to pull the latest data and re-render sections of the page.  So, given a framework like that, which I imagine isn't all that uncommon, I was thinking, "what if REST had a multipart GET request to grab a bunch of resources in a single request".  I haven't fully fleshed the idea out, but similar to how you upload files using multipart/form-data, you could do a multipart GET request.  The request would look something like:

GET /
Content-Type: multipart/rest; boundary=----------------------------12345
------------------------------12345
GET /resource/A
------------------------------12345
GET /resource/B

And the response would be something like:

Content-Type: multipart/rest; boundary=----------------------------12345
------------------------------12345
Location: /resource/A
Content-Type: application/json
Content-Length: 10
Status: 200 OK
{"a": "b"}
------------------------------12345
Location: /resource/B
Status: 404 Not Found

This would avoid the overhead of the network and HTTP protocol for all the additional requests and let you get a bunch of resources in a single request.  Since this would require modifying the HTTP spec, something a little easier would be to just have an API endpoint that does this for you in a single body:

GET /multiple?resource=/resource/A&resource=/resource/B

And the response would just encapsulate all of the resources in a single body:

Content-type: application/json
Status: 200 OK
{ "/resource/A": {"a": "b"}, "/resource/B": null }

To take advantage of something like this and still let your development be sane, you'd need a construct to queue up your AJAX requests rather than firing them immediately, then you could fire off a single "multiple" request on a predefined interval, say 100ms.  Something like this (this code would not work as-is):

var queue = [];
function get(url) {
    queue.push(url);
}
function parse_multiple(response) {
    for url in response {
        event.fire('data-received', url, response[url]);
    }
}
function multiget() {
    var query_params = ""; // build the query params from queue
    queue = [];
    $.ajax('GET', '/multiple?' + query_params, parse_multiple);
}
setTimeout(100, multiget);

Now, every 100ms you fire off a single AJAX request to get everything that needs to be updated rather than firing off a ton of ad-hoc AJAX requests for every individual item on the page.  Seems like that could work, but I haven't yet put it to the test.

On a similar note, to reduce the amount of redrawing you do in the browser, you could queue up DOM changes and do all of them at once on a timer as well.  That's a little more difficult to pull off, but if you built the new dom nodes in memory and just pointed them at the node that they were replacing, you could cache that in an object and since redraws only happen in between function calls (due to the single-threaded nature of Javascript, it has to wait until it exits user code to execute the browser code, unless something has changed recently).  So you could replace all the pending nodes in a single function and then let the browser redraw the whole thing after your function exited.  One redraw every 100ms is better than 100 redraws at random intervals, and fast enough that the user wouldn't notice the lag.  I dunno, maybe I'm taking crazy pills, since I haven't seen anyone attempt to do something like this in the major frameworks (or maybe they did while I wasn't looking).

Ok, good, I wrote those down so I can stop thinking about them and get back to what I'm supposed to be working on.  Hopefully some day I'll find out if they're feasible ideas or not.

Introducing python-ambariclient

Apache Ambari is an open-source project that configures and manages Hadoop clusters.  The product I work on at work configures and manages Hadoop clusters... in the cloud (ooooh).  Now that Ambari has matured enough to be stable and have a fairly usable API, we've decided to discontinue the part of our product that overlaps with Ambari and use and contribute to that project instead.  Our product isn't going anywhere, just the ~50% of our codebase that does pretty much exactly what Ambari does will be replaced with Ambari and we'll put those resources we would spend building and maintaining our own code into improving Ambari.  Given that Ambari's primary audience is Java developers, their main efforts at a client library are written in Groovy, another JVM-backed language that works with Java seamlessly.  They also have a Python client, but it's fairly immature, incomplete, and buggy.  Efforts to contribute to it proved onerous, mostly due to concerns about breaking backwards-compatibility, so we decided that I would create a new client that we'd release to the public.  And so I'm here to announce our Python Ambari client libraries, aptly named python-ambariclient.

There were a few things I wanted out of the client that I felt we weren't able to accomplish easily with the existing one:

1. An easy-to-intuit, consistent interface that mimicked the API structure.
2. Native support for polling the long-running background operations that are common to working with Ambari.
3. Easy to add new types of objects to as the Ambari API added new features.
4. Minimize the number of actual HTTP requests executed.
5. An ORM-style interface that felt natural to use coming from projects like SQLAlchemy and libcloud.

To accomplish all those goals, I felt like a vaguely promises-style API would suit it best.  This would allow us to delay firing off HTTP requests until you actually needed the response data to proceed, and I wanted the method-chaining style reminiscent of Javascript projects like jquery.  I was able to accomplish both, and I think it turned out pretty well.  It's a good example of what I've always wanted in an API client.  So, let's dive in to some of the design decisions.

Delegation and Collections and Models, oh my

The main API client is just an entry point that delegates all of the actual logic to a set of collection objects, each of which represents a collection of resources on the Ambari server.  For those who are used to REST APIs, this might make sense, but here's some examples to show what I mean:

# get all of the users in the system
users = ambari.users
for user in users:
    print user.user_name

# get all of the clusters in the system
clusters = ambari.clusters
for cluster in clusters:
    print cluster.identifier

The collections are iterable objects that contain a list of model objects, each representing a resource on the server.  There are some helper methods on the collections to do bulk operations, such as:

# delete all users (this will likely fail or break everything if it doesn't)
ambari.users.delete()

# update all users with a new password (bad idea, but hey)
ambari.users.update(password='new-password')

If you want to get a specific model out of a collection, that's easily accomplished by passing a single parameter into the accessor for the collection.

# get the admin user
admin_user = ambari.users('admin')

# get a specific cluster
cluster = ambari.clusters(cluster_name)

# get a specific host
host = ambari.hosts(host_name)

Additionally, you can get a subset of a collection by passing in multiple arguments.

# get a subset of all hosts
hosts = ambari.hosts([hostname1, hostname2, hostname3])

So, this is just the basic entry point model collections.  In Ambari, there's a large hierarchy of related resource and sub-resources.  Users have privileges, clusters have hosts, services have components, etc.  To handle that, each model object can have a set of related collections for the objects that are contained by it.  So, for example:

# get all hosts on a specific cluster
ambari.cluster(cluster_name).hosts

# get a specific host on that cluster
host = ambari.cluster(cluster_name).hosts(host_name)

Some of the hierarchies are very deep.  These are the deepest examples I can find so far:

# get a repo for a specific OS for a specific version of a specific stack
ambari.stacks(stack_name).versions(stack_version).operating_systems(os_type).repositories(repo_id)

# get a component for a specific service for a specific version of a specific stack
ambari.stacks(stack_name).versions(stack_version).services(service_name).components(component_name)

Obviously those are outliers, in general use you only need to go one or two levels deep for most things, but it's good to know the pattern holds even for deep hierarchies.

When you get to the individual model objects, they behave much like a normal ORM.  They have CRUD methods like create, update, delete, and they use attribute-based accessors for the fields returned by the API for that resource.  For example:

cluster = ambari.clusters(cluster_name)
print cluster.cluster_id
print cluster.health_report

There's no fancy data validation or type coercion like in SQLAlchemy, just a list of field names that define which attributes are available, but really that's all that I think is necessary in an API client.  The server will do more robust validation, and I didn't see any places where automatic coercion made sense.  What I mean by automatic coercion is automatically converting datetime fields into datetime objects, or things of that nature.  I'm not doing that, and it's possible that that decision turns out to be shortsighted, but I'm guessing the simplicity of the current design will win out.

Wait for it...

Because the client is a promises style API, it doesn't necessarily populate the objects when you expect.  For the most part, if it can't accomplish what you're requesting without populating the object with data from the server, it will do it automatically for you.  Many operations also are fairly asynchronous, and what you as a user really care about is that you are safe to operate on a resource.  To accomplish that, there is a method called wait() on each object.  Calling wait() will basically do whatever is required for that model or collection to be in a "ready" state for you to act on it.  Whether that's simply just requesting data from the server or waiting for a background operation to complete or waiting for a host to finish registering itself with the Ambari server, the method is the same.  .wait():

# wait for a recently-added host to be available in Ambari
ambari.hosts(host_name).wait()

# wait for a bootstrap call to finish and all hosts to be available
ambari.bootstrap.create(hosts=[hostname1, hostname2], **other_params).wait()

I have a request

In the Ambari API, if your POST or PUT command triggers a background operation, a 'request' object is returned in the response body.  It will look something like this:

{
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/testcluster/requests/1",
  "Requests" : {
    "id" : 1,
    "status" : "InProgress"
  }
}

If any API call returns this information, the Ambari client will automatically recognize that and store that information away.  Then, if you call .wait() on the object, it will poll the Ambari API until that request has completed.  At some point, it will start throwing exceptions if the request doesn't complete successfully, but that logic hasn't been built in yet.

# install all registered components on a host and wait until that's done
ambari.clusters(cluster_name).hosts(host_name).components.install().wait()

And to be consistent and obey the principle of least surprise, you can chain off wait() calls to do further actions, so this also works:

# install and start all registered components on a host and wait until it's done
ambari.clusters(cluster_name).hosts(host_name).components.install().wait().start().wait()

It's not generally a great idea to just have a huge long method chain like that, but it's possible.  It would be written better like:

components = ambari.clusters(cluster_name).hosts(host_name).components
components.install().wait()
components.start().wait()

Wait, that's it?

I wanted it to be extremely easy to add new model classes to the client, because that was one of my biggest complaints with the existing client.  So most of the common logic is built into two base classes, called QueryableModel and DependentModel.  Now defining a model class is as simple as defining a few pieces of metadata, for example:

class Cluster(base.QueryableModel):
    path = 'clusters'
    data_key = 'Clusters'
    primary_key = 'cluster_name'
    fields = ('cluster_id', 'cluster_name', 'health_report', 'provisioning_state',
              'total_hosts', 'version', 'desired_configs',
              'desired_service_config_versions')
    relationships = {
        'hosts': ClusterHost,
        'requests': Request,
        'services': Service,
        'configurations': Configuration,
        'workflows': Workflow,
    }

1. 'path' is the piece of the URL that should be appended to access this model.  i.e. /api/v1/clusters
2. 'data_key' defines which part of the returned data structure contains the data for this particular model.  The Ambari API returns the main model's data in a subordinate structure because it also returns a lot of related objects.
3. 'primary_key' is the field that is used to generate the URLs to a specific resource.  i.e. /api/v1/clusters/cluster_name
4. 'fields' is a list of field names that should be returned in the model's data.
5. 'relationships' is a list of accessors that should build related collection objects. i.e. ambari.clusters(cluster_name).hosts == collection of ClusterHost models

Some objects are not represented by actual URLs on the server and are only returned as related objects to other models.  These are called DependentModels in my client.  Here's a pretty simple one:

class BlueprintHostGroup(base.DependentModel):
    fields = ('name', 'configurations', 'components')
    primary_key = 'name'

class Blueprint(base.QueryableModel):
    path = 'blueprints'
    data_key = 'Blueprints'
    primary_key = 'blueprint_name'
    fields = ('blueprint_name', 'stack_name', 'stack_version')
    relationships = {
        'host_groups': BlueprintHostGroup,
    }

When you get a specific blueprint, it returns something like this:

{
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/blueprints/blueprint-multinode-default",
  "configurations" : [
    {
      "nagios-env" : {
        "properties" : {
          "nagios_contact" : "greg.hill@rackspace.com"
        }
      }
    }
  ],
  "host_groups" : [
    {
      "name" : "namenode",
      "configurations" : [ ],
      "components" : [
        {
          "name" : "NAMENODE"
        }
      ],
      "cardinality" : "1"
    }
  ],
  "Blueprints" : {
    "blueprint_name" : "blueprint-multinode-default",
    "stack_name" : "HDP",
    "stack_version" : "2.1"
  }
}

As you can see, the 'Blueprints' key is the 'data_key', so that structure has the data related to the blueprint itself.  The 'host_groups' and 'configurations' structures are related objects that don't have URLs associated with them.  For those, we can define DependentModel classes to automatically expand them into usable objects.  So, now this works:

for host_group in ambari.blueprints(blueprint_name).host_groups:
    print host_group.name
    for component in host_group.components:
        print component['name']

I tried to make things act consistently even where they weren't consistent in the API.  It should be noted that objects that are backed by URLs are also returned in related collections like this, and the client will automatically use that data to prepopulate the related collections to avoid more HTTP requests.  For example, here is a very trimmed down cluster response:

{
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/testcluster",
  "Clusters" : {
  },
  "requests" : [
    {
      "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/testcluster/requests/1",
      "Requests" : {
        "cluster_name" : "testcluster",
        "id" : 1
      }
    }
  ],
  "services" : [
    {
      "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/testcluster/services/GANGLIA",
      "ServiceInfo" : {
        "cluster_name" : "testcluster",
        "service_name" : "GANGLIA"
      }
    }
  ]
} 

As you can see, both the 'requests' and 'services' related collections were returned here.  So, if you were to then, do:

for service in ambari.clusters(cluster_name).services:
    print service.service_name

It would only have to do the single GET request to populate the cluster object, then use the data returned there to populate the service objects.  There is a caveat here.  When getting collections in the Ambari API, it generally only returns a minimal subset of information, usually just the primary_key and possibly the primary_key of its parent (in this case, service_name and cluster_name).  If you want to access any other fields on that object, it will have to do another GET call to populate the remaining fields.  It does this for you automatically:

for service in ambari.clusters(cluster_name).services:
    print service.maintenance_state

'maintenance_state' was not among the fields returned by the original call, so it will do a separate GET request for  http://c6401.ambari.apache.org:8080/api/v1/clusters/testcluster/services/GANGLIA to populate that information and then return it.

Smoothing out the rough edges

The Ambari API is mostly consistent, but there are some warts from old designs or one-off pieces.  The bootstrap API and the configurations are the worst offenders in this regard.  All efforts were made to make those areas behave like the other areas as much as possible.  I didn't want the user to have to know that, for example, bootstrap requests aren't the same as every other asynchronous task, or that even when a bootstrap finishes the hosts are not visible to Ambari until their agents have booted up and registered themselves.  So, I overloaded the wait() method on those objects so that it just does the needful.

# wait until these hosts are in a ready state
ambari.hosts([hostname1, hostname2]).wait()

Similarly, adding a host to a cluster normally involves manually assigning all of the components, but an upcoming Ambari feature will make it so you simply have to pass in a blueprint and host_group and it will automatically do it for you.  I pre-emptively smoothed this out in the client so you can do this now, it just involves a few more API requests to be made automatically on your behalf.  Wherever things are inconsistent on the API server, my client makes them consistent to the user.

# add a new host to an existing host_group definition
ambari.clusters(cluster_name).hosts.create(host_name, blueprint=blueprint_name, host_group=host_group_name)

When the server-side is updated to include support for this, I can simply pass the information along and let it sort it out.  There are a few other cases where warts in the API were smoothed over, but for the most part the idioms in the client matched up with the API server pretty well.

Where do we go from here?

There was one feature that I really wanted to have that I wasn't able to wrap my head around sufficiently to implement in a clean, intuitive way.  That is the ability to act on collections of collections.  Wouldn't it be awesome if this worked?

# restart all components on all hosts on all clusters
ambari.clusters.hosts.components.restart().wait()

The .wait() would get a list of clusters, then get a list of hosts per cluster in parallel, then get a list of components for each host in parallel, then call the restart API method for each of those, gobble up all the request objects, and wait until all of them completed before returning.  This should be possible, but it will require a bit more thought into how to implement it sanely, and there wasn't enough bang for the buck for our use-cases to justify spending the time right now.  But maybe I'll get back to it later.

What's it to me?

I realize Ambari is a niche product, and that most of this post will be gobbledy-guck for most of you, but I think the general principles behind the client's design apply well to any REST-based API client.  I hope that people find them useful and maybe lift a few of them for their own projects.  Most of all, I think this is probably the best client library I've ever written, and it embodies pretty much everything I've wanted in a client library in the past.  We plan on rewriting the client library for our own API in a similar fashion and releasing that to the public in the near future*.

* Usual disclaimer about forward-looking statements and all that.  I make no guarantee that this will actually happen.

Promise-based REST API clients

In software development, the concept of promises (sometimes also called futures) is deceivingly simple.  There's a pretty good wikipedia article that explains it better than I do, and the opening line is about a good a summary as I can think of:

In computer science, future, promise, and delay refer to constructs used for synchronizing in some concurrent programming languages. They describe an object that acts as a proxy for a result that is initially unknown, usually because the computation of its value is yet incomplete.

What that means in normal people English is that you call a method to do something (i.e. compute a value, gather some data, etc) and instead of doing what you requested immediately and making you wait, it returns a "promise" that it will eventually do what you requested.  While this is generally done to make concurrency easier, I think the concept works well for hierarchical REST API clients as well.

Let me explain.  No, no, there is too much.  Let me sum up.

REST APIs are generally hierarchical.  You often see structures like:

GET /artists - get a list of artists
GET /artists/metallica - get the artist Metallica
GET /artists/metallica/albums - get a list of albums by Metallica

And so on.  What I wanted from an API client was to do something along the lines of this Python snippet:

for album in client.artists('metallica').albums:
    album.delete() # take that Metallica

What I didn't want to happen in the above snippets was to do all of these HTTP requests

GET /artists
GET /artists/metallica
GET /artists/metallica/albums
DELETE /artists/metallica/albums/killemall
DELETE /artists/metallica/albums/ridethelightning
...

But instead just do:

DELETE /artists/metallica/albums

(or if the API didn't support bulk-delete in that way, just do a single GET followed by a series of DELETE calls)

So, enter promises.  If each method in that chain simply returned the promise of fetching the underlying data, I could chain off of it and only actually fire off HTTP requests for the things I actually needed to get.

I should've taken the blue pill

So, that makes pretty good sense, but that's not all.  For your 3 easy payments of $29.95 you get not only this nice, easy-to-use method chaining and minimal HTTP request overhead, but also implicit parallelized requests as well.  What if, given the above example, you could also do something like:

for album in client.artists.albums:
     print album.tracklisting

And that just automatically did basically this:

1. GET /artists
2. for every $artist, GET /artists/$artist/albums (in parallel)
3. for every album from every $artist, print the tracklisting

Now, let's see how deep this rabbit hole can go:

for track in client.artists.albums.tracks:
    print "%d - %s" % (track.number, track.name)

1. GET /artists
2. for every $artist, GET /artists/$artist/albums (in parallel)
3. for every $album from every $artist, GET /artists/$artist/albums/$album/tracks (in parallel)
4. for every track for every $album from every $artist, print the number and name

I am serious, and stop calling me Shirley

I think this is a really powerful idea that could make an API client very easy to use and actually, you know, fun.  But maybe I'm alone here, since every API client ever is just the same old boring bunch of methods, sometimes in namespaces, sometimes with actual model-style classes for the resources, if you're lucky.  Granted, this idiom could be abused to fire off hundreds of HTTP requests in parallel and potentially overwhelm the server, either by accident or malice.  But, in my opinion the ease-of-use outweighs the potential for mass destruction.  I'd like to see a client that works this way, even if I have to build it myself (especially if I get to build it).  I've managed to build a client that does the first half of this, and does it pretty well IMO (blog post coming soon).  I have a good idea of how to accomplish the second part, but I haven't yet overcome the effort-to-payoff ratio on that one.  It would be cool, and potentially useful, but it's a lot of work for such a small benefit.  Still, that itch needs scratching, and one day it shall be scratched.  Even though I realize that maybe it means I'm insane.