
API, HTTP, REST, Standard, Universal, Web
Devi's producing a public website explaining pharmaceuticals to the public. The browser applications will pull down drug data from the server, and authorised users - such as drug manufacturers - will be allowed to upload new information. It's envisioned that third-party applications will also connect to the web server and provide their own user-interfaces. For example, pharmacies will be able to use the information via their point-of-sale systems, and doctors will have access from their specialised consulting software. Devi opts to follow a RESTful architecture for all of the services exposed on the server. This makes it easy for the browser side of her own web app to access the data and exposes services in a standardised manner.
The browser-side of an Ajax App needs to call server-side services.
There might be more than one browser-side application sharing the same server-side service.
As well, it's often desirable for third-party applications to invoke the server-side service too.
With numerous different applications - and a variety of developers - accessing the service, it ought to be easy to use.
Taken as a whole, the server-based services in a single "application" form an Application Programming Interface ("API"). It's important that services work consistently within the API.
The basic architecture of the web does not force any type of service architecture - any given functionality could be exposed in a wide variety of styles.
Expose web services according to RESTful principles. Representational State Transfer (REST) is an architectural style - or "pattern" - guiding on the architecture of web services. Like "Ajax" itself, "REST" is a broad term and can be broken down into many smaller patterns and idioms (enough to fill up an entire book!). The explanation here can't possibly do justice to the entire concept, but it's worthwhile being familiar with the general principles of REST. Because REST is such a large topic, be prepared for the reality that most real-world systems you'll encounter are only somewhat RESTful; they'll follow some principles and break others.
To motivate REST, let's consider a football service. Sufficiently authorised clients can read game records, upload new games, correct existing games, or delete games. We'll assume your backend's already written, but how would you offer it as a web service? How will client developers call it, and how will they learn about its interface?
To wit, following are a few random ways you might expose a game service - note that these aren't necessarily RESTful!
GET call to a special newGame service: http://example.com/newGame?id=995&bluebaggers=150&redlegs=60.
POST or PUT call to a special newGame service, http://example.com/newGame, with an XML body like: <game id="995"><score team="bluebaggers">150</score><score team="redlegs">60</score></game>.
POST or PUT call to a special newGame service, http://example.com/newGame, with a CGI-style body: id=995&bluebaggers=150&redlegs=60.
GET call to a multi-purpose gameMaintenance service, http://example.com/gameMaintenance/command=newGame&id=995&bluebaggers=150&redlegs=60.
POST call to a multi-purpose gameMaintenance service with a command-qualified URL, http://example.com/gameMaintenance/command=newGame with an XML body like: <game id="995"><score team="bluebaggers">150</score><score team="redlegs">60</score></game> .
And we could go on. And then we could look at the other functions here. The point is this: As a service designer, you have many options for representing a single function. Is that, in itself, a problem? You might argue it's not a problem at all - just choose any old option, using some darts and a blindfold if need be. Then just create the service and document its usage - any competent developer will then be able to deal with it, right? That argument, however, is flawed on several counts:
You might assume that the browser script for your web app, and perhaps some known external clients, are the only users of your service, making it relatively easy to create and document any service. In fact, your application resides within a larger internet ecosystem. Among the entities involved: the web browser (e.g. Firefox) which impacts on issues like back-button handling and bookmarking; the web server (e.g. Apache) which impacts on issues like caching and compilation; network routers and caches all along the way; robot applications such as those crawling the web on behalf of search engines; personal web agents which trawl through the web on behalf of individuals. Your application will perform better and more reliably if you follow certain conventions that are common among these entities.
Ideally, a service should be intuitive and self-documenting: if a service is based on familiar conventions, you only have to specify the bare minimum for a developer to be able to use it. That's simply not possible if everyone chooses alternatives according to their own personal taste. The standards-based interface usually trumps the optimised interface that's technically superior but based on unfamiliar conventions. There are thousands of web services available on the internet; developers should be able to assess them and put them into action with as little effort as possible.
In the absence of any guiding principles, there's a disturbingly high likelihood you'll end up in tedious debates with colleagues over the most trivial issues, trying to arrive at an optimal API. (The blindfold and darts - or an impatient manager - would alleviate this kind of paralysis, but the other issues would still remain.)
So we've seen the argument for a convention-based approach to web services, but what's the convention? REST is one such convention consisting of many different guidelines, and has broad industry support. Just about any functionality can be presented as a RESTful API, and by doing so, it gains the advantage of a familiar, intuitive, style. Many services on the net already follow RESTful principles, to differing degrees. Also, many real-world entities such as browsers and caches tend to assume REST. These entities work better if REST is in place; a browser, for instance, might give better feedback; a cache will retain all the data that the programmer expected it to retain. And what will happen if REST is not in place? It's probably not the end of the world, but you'll find little things will go wrong and work in unexpected ways. For instance, hitting the browser's Reload button might cause a purchase order to be resubmitted. Or a cache might end up caching data it shouldn't, or not caching any data at all. Why these unfortunate events might occur will become clear as we learn about the RESTful principles.
So much for abstract motivation: what exactly are the RESTful principles all about? REST ultimately presents the server as a big blob of "resources" such as people, cars, or football games. Clients interact by inspecting and changing that state. Putting a purchase order on the server, for example, will trigger the server to effect the purchase described in the order. This contrasts with the Remote Procedural Call (RPC) approach, where the client would call a remote purchase() procedure. REST is based on data manipulation, RPC is is based on procedural calls.
REST sees the universe as consisting of resources and operations performed on, and with, those resources. The resources and the operations can be represented as the HTTP concepts of URLs and call methods.
A resource is something like a "business entity" in modelling lingo. It's an entity you wish to expose as part of an API. Almost always, it's a noun, e.g. a person, a car, or a football game. Each resource is represented with a unique URL. So when you say a RESTful URL, it will be a thing rather than an action.
REST leverages the existing HTTP methods, particularly GET, POST, PUT, and DELETE. Note that the XMLHttpRequest object, and some wrapper libraries such as AjaxCaller, support all these methods. If they're unfamiliar to you, consult the W3C HTTP 1.1 Specification.
Another way to say it: In HTTP, any request a client can make involves a URL and an HTTP method. With REST, the URL is designed to represent a noun and the HTTP method always maps to one of several standard verbs, which will be performed against that noun.
How is the URL combined with the HTTP methods in practice? Returning to the example, a game is clearly an entity in this API. So REST says we create a scheme to expose each particular game as a unique URL: http://example.com/games/995.
So far, all pretty unequivocal. Anyone familiar with REST will always be driven to create a unique URL for each game. There might be differences of opinion about the precise naming, but the basic concept will always be the same if you follow REST. Now that we have this unique URL, we can implement many functions around it, by leveraging the standard HTTP methods.
To get game results, the client simply issues a GET call on the URL. GET is the standard way to perform queries on the resource indicated by the URL. The response format is not dictated by REST, but XML is common due to it's self-describing and self-validating nature.
To delete the game, the client issues a DELETE call on the URL. DELETE is the standard way to delete a resource.
To update a game, the client builds a fresh game message and uploads it in as the body of a PUT command. PUT is the standard way to update a resource's value.
To create a new game, the correct action will depend on whether the client can guess what the game's URL will be. If the service just allocates the URL arbitrarily, perhaps based on an auto-incrementing counter, then the client won't be able to guess it. When that happens, the new game is POSTed to a URL like http://example.com/games, and the server responds with the new game's URL (http://example.com/games/995). On the other hand, the client might be able to guess the URL, based on a naming convention. If the client guesses the new game would have URL http://example.com/games/2000/9/25, then it can simply PUT the new game to that URL.
To perform an arbitrary transaction, you'll also use one or more of these standard methods. So maybe the client wants a way to double the winner's score. In this case, there's probably no special service - the client just reads the old match, updates it, and PUTs it back to the server. For more complicated transactions, you might design a new resource specifically to encapsulate the request (e.g. a "purchase order" resource), and the client will then be able to POST resources like this to the server. Many businesses work the same way - contracts/orders are passed around in order to request things are done and verify they've been done. See the section on "Arbitrary Transactions" below for more details.
In summary, each HTTP method will cause a well-defined action on the resource represented by the URL it operates on. The methods can be compared to SQL commands: GET is like "SELECT", DELETE is like "DELETE", POST is like "INSERT" with an auto-generated ID, PUT is like "INSERT OR UPDATE IF EXISTS" with an ID specified.
Again, notice how most of this is unequivocal. True, you still have to decide on a precise URL convention and message formats, but that's fairly minimal. Just by declaring "this is a REST interface", you're conveying many implicit rules to a REST-savvy developer. Inform that developer of the URL and message conventions, and they'll be able to intuit most of the API. Compare that to an ad hoc, roll-your-own, API, where each detail must be explained piecemeal. Considering that there are thousands of web services out there, it soon becomes clear why REST is an attractive option.
As explained in the previous section, each resource receives a unique URL. URLs generally look like nouns.
REST doesn't tell you what names to call your resources - there are infinite possibilities and the choice is up to you. They'll often reflect your conceptual model of the system and look similar to your database table names.
With REST, each HTTP method indicates the kind of action to be performed. Moreover, the REST API designer doesn't have to decide what each method means, because there are only a handful of methods, and REST standardises the meaning of each. GETs for queries, POSTs for inserts, and so on, as explained above. Notice how resource names are not standardised in the same way. Businesses have different models, so you can't standardise on those things. But REST is saying that you can indeed model the resources so that the set of required actions is standard, thus ensuring APIs are consistent with one another.
Fortunately, the standard HTTP methods, when combined with a well-design URL and resource scheme - are enough to let a client express just about anything it needs to. It's true that in an ideal world, there would be a few extra actions to make life a bit easier. For example, it would be convenient if there was a standard LOCK action to let a client deal exclusively with a particular resource[8]. However, you can still work around this by introducing a lock resource and letting clients lock things by manipulating those resources with the standard HTTP methods.
Refining the previous point, GET calls must be read-only. When you perform a read-only query of any kind, always use GET. Conversely, when your call will affect any data on the server, use one of the other methods: PUT, POST, and DELETE all change state. There are some caveats here - for example, if issuing a query ends up leaving a log message somewhere, does that constitute a change? Technically, yes; but commonsense says no, it's not a significant change to the underlying resource, so it's acceptable to log the message in response to a GET request. Just be aware that the log might not read as intended; for instance, some clients will get the content from a cache, which means no log message will occur.
The reverse problem can happen too. GET requests are often cached, but not the other types of request. When you perform a query with POST, you deny the possibility of caching in the browser, within the client's network, or on your own server.
If you learn nothing else about REST, be sure to learn this guideline; of all the REST conventions, this is the best known and widely applied across the net (see sidebar.)
In stateful interaction, the server remembers stuff from the past. For example, you could do this:
Greg's Browser: Give me the next game to work on.
Server: Work on game 995.
...
Greg's Browser: Here's the result: bluebaggers 150, redlegs 60.
Server: Thanks.
(Server stores "bluebaggers 150, redlegs 60" for game 995.)
The server here has an implicit conversational state; it remembers the game Greg's working on. That makes testing harder, because you have to set the whole history up to test any particular interaction. Another problem here is the fragile nature of the transaction; what if Greg's browser failed, and ended up requesting two different games. When Greg uploads the score, the server could easily misinterpret which game he's talking about. Also, you can't cache effectively because a query result might depend on something that happened earlier.
Furthermore, an objective of REST is to be able to switch clients at any time and receive the same result. If you switch "Greg's Browser" to "Marcia's Browser", the server will respond differently to the game result above, because it will assume Marcia's Browser is working on a different game number. While users are unlikely to switch browsers mid-conversation, other clients like caches and robots may well switch. Substitutability makes networks more scaleable.
The upshot is that everything should be passed in at once. It's okay for the client to remember details about the conversation - REST says nothing about the client's activity - but the server must not. The server responses should be based on the server's global state and not on the state of the conversation, i.e.:
Greg's Browser: Give me the next game to work on.
Server: Work on game 995.
...
Greg's Browser: Here's the result
for Game 995: bluebaggers 150, redlegs 60.
Server: Thanks, I'll store "bluebaggers 150, redlegs 60" for Game 995.
So are cookies used at all? Yes, cookies can be used, but mainly for authentication. They should only affect if the server will accept a call, but they should in no way influence how the server will respond. If authentication fails, an appropriate response, such as a 401 Forbidden header, will result. If authentication succeeds, the same call will always have the same effect, independent of the client who made the call. So if Greg's Browser says "Get All Games", Greg will see exactly the same thing as if the all-powerful administrator asked for the same thing, as long as Greg's allowed to see them. If the security policy forbids Greg from seeing all games, Greg's query will be denied. What won't happen is for Greg to receive just a minimal list of his own games - if that happened, the server would be using the cookie to determine the browser's response. To get a minimal list, the query must be explicit - "Get All of Greg's Games" - and of course, it must be Greg or the administrator that issues this request. If Marcia made that request, she'd get an authentication error.
An alternative to cookies is to use HTTP Basic Authentication, but this doesn't work too well in an Ajax context, because you don't have any control over the UI. The browser will typically pop up a dialog box, whereas you probably want to integrate any authentication directly into the interface, especially if using a pattern like “Lazy Registration”. HTTP Digest Authentication does allow more flexibility, but browser and server support is limited. Thus, while HTTP authentication is seen as "more pure", cookies are a reasonable workaround, provided you use them only for authentication.
"Idempotent" means that once you pass a message to service, there's no additional effect of passing the same message again. Consider the deletion message, "Delete Game 995". You can send it once, or you can send it ten times in succession, and the world will be the same either way. Likewise, a RESTful GET is always idempotent, because it never affects state. The basic conventions on GET/DELETE/PUT/POST, described above, are intended to support idempotency.
In resource representations returned by GET queries, use hyperlinks liberally to refer to related resources. With judicious use of hyperlinks, you can and should break information down. Instead of providing one massive response, include a modicum of information and point to further information using resource identifiers.
RESTful services can and should document themselves. The exact mechanism is not well-defined. But a couple of examples include:
Base URLs can explain how the service works. For example, in the example above, http://example.com/game/995 represents a particular game, but http://example.com/game represents no real entity, so its result could be an instructive document, in human-friendly language, with appropriate hyperlinks.
Error responses should also be in human-friendly language and with examples and hyperlinks.
As an extension to the previous point, RESTful services rely on standards such as Document Type Definitions (DTDs) and XML Schema to verify data formats as well as document what's acceptable.
The REST definition is especially geared for creation, reading, updating, and deleting (CRUD) operations. How about transactions and arbitrary actions, such as "Pause the printer" or "Email the catalogue to this user"? By definition, application-specific actions don't fit neatly with the standard REST actions. There will always be some pragmatic judgment required as to a suitably RESTful interface. A few possibilities have been mooted, with viewpoints varying:
If it makes sense to do so, reformulate the action as a standard persistence operation. For updates that are complex or entail business logic, you can provide a read-only service - accessed with GET queries - to help the client compute the new state.
You can POST a message to the resource in question, for example, POST a "pause" message to a URL representing the printer.
The server can can expose a stateless "processor" resource to perform the action, e.g. a printer controller, and have clients POST command-like messages there, with URLs referencing the resources to be acted upon.
Being a broad architectural style, REST will always have different interpretations. The ambiguity is exacerbated by the fact that there aren't nearly enough standard HTTP methods to support common operations. The most common example is the lack of a search method, meaning that different people will design search in different ways. Given that REST aims to unify service architecture, any ambiguity must be seen as weakening the argument for REST.
Another issue is portability - while GET and POST are standard, you may encounter browsers and servers which don't deal consistently with DELETE, PUT, and other methods.
The main alternative to REST is RPC (see “RPC Service”). It's equally broad in definition, but the essential idea is that services are exposed at procedures. You end up POSTing into verb-like URLs such as /game/createGame?gameId=995 instead of RESTful, noun-like URLs such as /game/995. In fact, the distinction is significant enough that some service providers, such as Amazon, actually provide separate APIs for each. As a general rule, any set of services could be exposed as either REST or RPC; it's just a question of API clarity and ease-of-implementation. Note also there is some overlap; as discussed in the “RPC Service” solution, RPC can still follow certain RESTful principles.
From an implementation perspective, REST and RPC differ in that REST requires some explicit design, whereas RPC tends to follow directly from the back-end software model. In the example, it's likely there will be a Game class with a createGame() method - that's just how most server-side software gets done. So it's a no-brainer to tack on a /game/createGame web service that mediates between the client and the back-end method. In fact, there are many frameworks that will completely automate the process for you.
With REST, there's no direct mapping between web service and backend implementation - an impedence mismatch. You need to take a step back and explicitly design your API to be RESTful. If you're following feature-driven design, the API is the first thing you'll produce anyway, since the design will be "pulled" by the needs of clients, rather than "pushed" from the available technology. Once you've designed the API, the web service implementation will effectively be a kind of middleware Adaptor (see Gamma et. al, 1995) to the back-end services.
To summarise crudely:
Any web service functionality can be exposed as either REST or RPC. There's a good argument that REST APIs are somewhat clearer, though that view is far from universal.
REST APIs may be clearer, but they do require more design and maintenance because they tend to diverge from the back-end technology. However, adaptor-style implementations of this nature are quite easy, so the overhead is not substantial and often justified in the effort to provide a clean interface, unaffected by the incidentals of the back-end implementation.
I'm not aware of any Ajax Apps which access a truly RESTful interface on the server-side. Consequently, the example here is a public API which conforms closely to REST. Note that several prominent industry interfaces are not covered here, because though they promote themselves as RESTful, they tend to break quite a few basic principles. For example, Amazon's REST API. In industry parlance, REST is sometimes synonymous with "not SOAP" and it's sometimes assumed that an interface is RESTful as long as it uses GET for reads and POST for writes.
Atom is a feed protocol built around RESTful principles. Blogger offers a good description on its use of Atom in its public API. The API lets third-party applications read and change Blogger blogs. Blogger themselves could theoretically build an Ajax App that directly calls on the API.
A blog entry is one important resource in blogger's API. For example, entry ID 1000 for user 555 "lives at" http://blogger.com/atom/555/1000. So to read that entry, you just issue a GET on that URL. To change the entry there, you just PUT an XML document to the same URL. To add a new entry, you don't yet know its ID, so you POST to a URL containing only the user ID, viz. http://blogger.com/atom/555. Note that each of these operations uses HTTP Basic Authentication and runs over HTTPS.
The Basic Shop Demo employs an ad hoc web service to expose and manipulate store items and user shopping carts. In this demo, the cart service is refactored to become more RESTful. You can contrast this to the Refactoring Illustration in “RPC Service”. Note that some constraints make the RESTful service less than ideal here:
URLs would ideally be prefixed with http://ajaxshop.com, but because it's running within the Ajax Demo framework, the prefix is longer: http://ajaxify.com/run/shop/rest
URLs would ideally avoid CGI-style variables and look like "/content/Movies" instead of /content?category=music. That's not possible here because there's no use URL-rewriting (to make the demos easy to install).
There are three resources: categories, items, and carts. GET is used to read each of these. Only carts can be modified, either by adding an item or clearing the cart. Both of these are handled by POST, rather than PUT, since they are changes rather than replacements. Let's look at each service in more detail.
An XML list of categories is exposed by GETting
http://ajaxify.com/run/shop/categories.phtml.
Here, categories is the resource and the HTTP method is
GET since we're reading the resource. To avoid listing all category
information here, there's a link to each specific category resource:
<categories>
<category xlink="http://ajaxify.com/run/shop/rest/category.phtml?name=Books">Books</category>
<category xlink="http://ajaxify.com/run/shop/rest/category.phtml?name=Songs">Songs</category>
<category xlink="http://ajaxify.com/run/shop/rest/category.phtml?name=Movies">Movies</category>
</categories>
To drill down to an individual category, say "Movies", you GET http://ajaxify.com/run/shop/category.phtml?name=Movies, which provides the name and items. Since an item is defined solely by its name, and we can't perform operations on items themselves, there's no need to give it a URL.
<category>
<name>Movies</name>
<item>Contact</item>
<item>Gatica</item>
<item>Solaris</item>
</category>
As the system scales up, the list of all items gets excessive for someone who just wants to know the category name. So the next service provides just the items. If we wanted, we could then remove the list of items in the category name.
Being RESTful, we can't just access the shopping cart out using the current session - each operation must specify the cart's owner. The session is used here, but only for authentication.
To read the shopping cart of user 5000, we GET http://ajaxify.com/run/shop/rest/cart.phtml?userId=5000:
<cart userId="5000">
<item>
<name>Hackers and Painters</name>
<amount>2</amount>
</item>
<item>
<name>Accidental Empires</name>
<amount>4</amount>
</item>
</cart>
To test the authentication, visit the web app http://ajaxify.com/run/shop/rest and add a few items. Cut-and-paste your assigned user ID over the "5000" in the above URL. You'll be able to see your cart. Then try a different user ID, say 6000, and you'll be greeted with the following message along with a 401 (Forbidden) header:
You don't have access to Cart for user 6000.
Being RESTful, the cart URL remains the same whether we're manipulating or reading it. So when we make changes to user 5000's cart, we use the same URL as for reading it, http://ajaxify.com/run/shop/rest/cart.phtml?userId=5000 (which would instead end with the cleaner /cart/5000 if we could use URL-rewriting).
To update the cart, we simply PUT a new cart specification to the cart's URL. If the user's just cleared the cart, the following will be uploaded:
<cart>
</cart>
If the user's added an item, the browser still does the same thing: just upload the whole cart, e.g.:
<cart userId="65590">
<item>
<name>Hackers and Painters</name>
<amount>2</amount>
</item>
<item>
<name>Accidental Empires</name>
<amount>5</amount>
</item>
</cart>
Note that the working with PUT is quite similar to working with POST. As explained in “XMLHttpRequest Call”, XMLHttpRequest has a requestType parameter. In the example here, the details are in any event abstracted by the AjaxCaller wrapper library:
ajaxCaller.putBody("cart.phtml?userId="+userId, null, onCartResponse, true,
null, "text/xml", cartXML);
How to mail the cart contents is more subjective, as discussed in the "Arbitrary Actions" section in the Solution. The approach here is to post an email address to the cart URL, e.g.:
<email-address>
ajaxmail@example.com
</email-address>
“RPC Service” is an alternative to RESTful Service, as highlighted in the Solution above.
“XML Message”s are often used as the format of a REST service's response, and sometimes its input as well. XML fits well here because it's standards-based, broadly supported and understood, self-documenting, and capable of self-verifying when used with DTDs and XML Schemas. XML Messages tend to appear as the body of PUT and POST calls as well as in responses.
A RESTful transaction often involves passing a resource back and forth, with each side augmenting its state. For instance, a server can deliver an initially empty shopping cart, the client can add an item to it, the server can set the total price, and so on. Instead of transforming to and from a custom data structure, it sometimes makes sense for the browser to retain the incoming “XML Message”s in an “XML Data Island” and transform it according to the interaction.
“Unique URLs” also relates to URL design, though they differ in scope. “Unique URLs” relates to the URLs of the Ajax App itself, while RESTful Service involves the URLs of web services it uses. It's sometimes said that "Ajax breaks REST" because many Ajax Apps have a single URL regardless of state. Ironically, though, Ajax actually facilitates REST by encouraging a clean API to service the browser application. As for the browser application, “Unique URLs” do help make an Ajax App a bit more RESTful, though that's somewhat beside the point since it's the web services that a client would want to use.
REST is often likened to Unix - with its consistent usage of pipes and filters - and SQL - with its standard persistence and querying commands - but can also be compared to windows-based GUI platforms. There are standard actions like "Cut", "Paste", "Popup Context Menu". Each object (e.g. documents, icons, folders) accepts such actions and interprets them in a manner consistent with the action's meaning.
Roy Fielding's Dissertation which introduced the REST Concept.
Thanks to Tony Hill from the Thomson Corporation for reviewing this pattern and providing some great insights into REST.
[8] The WebDav protocol does just that. It adds several methods to HTTP, to support locking, property inspection, and moving resources around. These make REST easier to implement, but unfortunately most internet entities won't understand them, so it defeats one of the main purposes of REST on the open internet.