AjaxPatterns
This is the original book draft version for the Cross-Domain Proxy Ajax pattern, offered for historical purposes and as an anti-spam measure.
You're probably better off visiting the Live Wiki Version for Cross-Domain Proxy instead, where you can make changes and see what others have written.

Cross-Domain Proxy

Aggregator, Fusion, Mash-Up, Mashup, Mediator, Mix, Proxy, Tunnel

Developer Story

Dave's working on a job-hunting website and wants to "mash up" content from various websites. Each job ad is accompanied by a review from a corporate forum website, recent company news, and a stock ticker. To get these details into the browser, the server queries several sites and exposes their content as a web service which the browser can query.

Problem

How can you augment your application with services available elsewhere on the web?

Forces

  • There's a lot of data and useful services on the web, much of it freely available. It's usually more effective to leverage that content than to replicate it yourself.

  • The "same-origin policy" constrains most ??? techniques, meaning that the browser script can only talk to the server from whence it came, and no-one else.

  • External servers often use protocols and message formats that would be difficult for browser-side scripts to process. They may also require authentication, and such details should not be exposed in the browser.

  • To comply with service agreements, website owners often have to control and monitor all traffic to and from their web service, which is impossible if their server is being bypassed.

Solution

Create proxying and mediating “Web Service”s to facilitate communication between the browser and external domains. As explained in “XMLHttpRequest Call”, the same-origin policy means the browser script can only talk to the server from whence it came, the "base server". Hence, any communication with external domains must go via the base server.

The simplest form of proxy is a dumb Web Service that simply routes traffic between browser and external server. The service can accept the remote URL as a parameter. That URL will then be accessed synchronously, and the service will then output its response. All this follows the Proxy pattern (Gamma et al., 1995).

A cleaner type of Cross-Domain Proxy is more closely based on the Facade or Adaptor pattern (Gamma et al.). In this case, the base server presents the interface that's most suitable to the browser script, and performs whatever manipulation is necessary to interact with the external script. For example, the external server might present a complex, over-engineered, SOAP-based “RPC Service”, but you want the browser to deal in simple “Plain-Text Message”s. The likely solution is to present the plain-text Web Service, dress up requests into SOAP messages, and undress responses into “Plain-Text Message”s.

The Facade/Adaptor approach is generally better for two reasons. First, it keeps the client simple, protecting it from dealing with the details of the protocol used to communicate with the external server. Second, it's more secure; the first approach will allow any client to call out to any server they feel like, whereas the second approach will allow you to exert much more control over what kind of communication takes place.

A Cross-Domain Proxy is implemented using some form of HTTP client library to access the remote server. Generally, the connection should be quite straightforward - specify the URL and grab the response as a string. However, it might get more complex if the remote web service relies on cookies and authentication, parameters which the proxy might have to pass on from the Ajax request that initiated the call.

Decisions

What external content will be accessed?

In theory, anything you can do with a browser can theoretically be accomplished by an automated HTTP client. However, there are likely to be legal constraints, and technical challenges too if there is serious usage of Javascript or browser plugins. Furthermore, your script will be vulnerable to changes in the site layout, and many content providers have deliberately performed subtle changes to prevent automated access. Thus, relying on the ever-increasing collection of public APIs and structured data would make a better choice.

You can find a collection of public APIs at http://wsfinder.com. Some popular APIs include: Amazon, Delicious, EvDB, Flickr, Google Maps, Google Search, Technorati, Yahoo Maps.

How will you connect to the external server?

If you're scraping content directly from a public website, you'll need to use the underlying web protocol, HTTP. And even if you're talking to an API, you'll probably be communicating with HTTP anyway. In some cases, API publishers will provide code to access the content, rather than just publishing the specification. Also, with more complex web services protocols like SOAP, you don't have to write a lot of low-level code yourself. In many cases, though, the easiest thing to do is talk HTTP, and manually format and parse messages yourself.

Many scripting languages feature built-in libraries to communicate in HTTP and a language are often well-equipped for Cross-Domain Proxy implementations due to strong support for regular expression manipulation. A PHP example is featured in the WPLicense code example below.

In the Java world, for example, the standard API provides some relatively low-level support for HTTP clients, but you can use some of the website testing libraries to quickly extract content from external sites. HttpUnit is a good example, and it also has some support for Javascript manipulation. To grab some content, do this:

    import com.meterware.httpunit.WebConversation;
    import com.meterware.httpunit.GetMethodWebRequest;
    import com.meterware.httpunit.WebRequest;
   
    public class PatternSorter {
   
      public String getContent(String url) {
        try {
          WebConversation wc = new WebConversation();
          WebRequest request = new GetMethodWebRequest(url);
          String response = wc.getResponse(request).getText();
        } catch(Exception ex) {
          throw new RuntimeException("Could not get content from " + url, ex);
        }
      }
   
      public static void main(String[] args) {
          System.out.println("http://mahemoff.com");
      }
   
    }

How will you deal with errors and delays in accessing the service?

At times, there will be errors accessing an external web service. Think twice before attributing blame: the problem might be at your end, or somewhere along the network. This has implications for any error messages you might show users. Ideally, you should be able to detect an error has occurred and log the details for immediate attention. In responding to the user, you have several options:

  • Ensure you have a good server-side framework in place to detect errors, so they're not directly - ignorantly - passed on to users. In addition, be sure to detect timeouts and handle them proactively, e.g. respond with an appropriate error message.

  • In some cases, you will simply have to admit failure and show an error message to users. If possible, suggest an alternative for them or tell them when they should try again.

  • Provide reduced functionality.

  • Switch to a pre-arranged substitute web service.

  • Rely on cached results, if they're still relevant.

Under what licensing terms will you access the remote service?

The legals of invoking external services are tricky, vaguely-defined, and always evolving. Some “Web Service”s are open for public use, others require you hold an API key, and others are critical enough to warrant authentication via digital signatures. In each case, there are usage terms involved, even if they're not explicit, and you will need to investigate issues such the authentication mechanism, number of queries per day, how the data will be used, server uptime, support level.

Real-World Examples

WPLicense

The WPLicense Wordpress plugin presents the blogger with some forms to specify their preferred license statement, based on the Creative Commons model. The server is acting as a middleman between the browser and the Creative Commons API.

Housing Maps (Craigslist and Google Maps)

Housing Maps is a mashup between Google Maps and Craigslist, a community-oriented classifieds advertising website. What does that mean? It means you get to see a list of advertised homes on one side and view a map of those homes one the other side, using the familiar Google Maps thumbtacks to pinpoint the locations of the advertised homes.

Figure 1.30. Housing Maps

Housing Maps

A few features:

  • Click on a thumbtack and you'll see a “Popup” balloon appear, showing a summary of the classified ad: price, terms, address, photos. The balloon contains a hyperlink to the ad at Craigslist

  • Change a category (e.g. price) and you'll see both the thumbtacks and home lists update. Change the city and you'll see the map change as well.

  • You can pan and zoom as on Google Maps.

Cross-Domain Proxy is used to grab housing data from CraigsList, while the map images are fetched directly from Google's server into the browser.

Bill Gates Wealth Clock

This was not Ajaxian, but noteworthy as the first prominent mashup application and a precursor to the Ajaxian Cross-Domain Proxy pattern. Web guru Philip Greenspun mixed data from several sources to produce a "wealth clock" showing Bill Gates' worth at any time (No longer working, unfortunately, but the code is still available.

Figure 1.31. Bill Gates Wealth Clock

Bill Gates Wealth Clock

The price per stock was extracted from Yahoo Finance. The number of stocks in Gates' portfolio came from Microsoft reports (it was not automatically extracted). And the US population came from the US Census Bureau.

Here's what Greenspun said back in 2001:

(M)any of the most interesting and challenging problems of the coming decades will center around the Internet and Web as a distributed computing environment. An "island" site such as amazon.com or photo.net is probably less representative of the future than http://www.webho.com/WealthClock (Bill Gates Personal Wealth Clock).

Another prominent usage of cross-domain proxying in "Web 1.0" was meta-searching, as performed by crawlers such as MetaCrawler.com. Again, all the work was done in the server, and the browser response was usually opened while the results came back to the server, so at least some information could be shown while requests were still out.

CPaint Library

CPaint is a ??? library. A special "proxy" URL can be established, pointing to a proxy on the originating server, so that remoting to external domains has virtually the same API as to the originating server.

Code Example [WPLicense]

See the “Live Form” Code Example for a full background on the WPLicense's license selection process, but here's a quick summary:

  • A field lets the user select a license type (e.g. Creative Commons versus Public Domain versus Sampling).

  • Once the license type is chosen, it's immediately sent to the server, and the server responds with a form consisting of questions relevant to this particular type. If the license is "Creative Commons", for example, one of the questions is "Allow commercial uses of this work?"

  • Each time the user changes one of the license options, the server updates some hidden fields. These fields will be uploaded to the server, to be persisted, when the user eventually submits the form.

What all this glosses over is the Cross-Domain Proxy pattern that goes on behind the scenes. At each of these three stages, the server is actually interacting with CreativeCommons.org via its public API, which you can read about at http://api.creativecommons.org. The plugin provides a clean separation between web server logic and cross-domain mediation: a separate php file (ccwsclient.php) hosts several API facade operations, accepting and returning data structures in standard PHP style. This client in turn delegates all the XML infrastructure stuff to a 3rd party library (MiniXML).

Let's now zoom in on the three steps.

1. Retrieving License Types

The server doesn't have the license types hard-coded. Instead, it retrieves them via a web-service. This is the core of the Cross-Domain Proxy functionality - the PHP function file_get_contents() will retrieve content from the specified URL.

    $WS_ROOT = "http://api.creativecommons.org/rest/1.0/";
    ...
    $xml = file_get_contents($WS_ROOT);

In fact, you can see exactly what's pumped into $xml by visiting http://api.creativecommons.org/rest/1.0/, then View Page Source. Alternatively, use a command-like application like curl ("curl http://api.creativecommons.org/rest/1.0/"), which yields the following XML (reformatted):

    <licenses>
      <license id="standard">Creative Commons</license>
      <license id="publicdomain">Public Domain</license>
      <license id="recombo">Sampling</license>
    </licenses>

Once we know what license types are available, it's a matter of transforming the XML into an HTML selector. XSLT could be used here, but it's just as easy to do it manually:

    foreach($license_classes as $key => $l_id) {
      echo '<option value="' . $key . '" >' . $l_id . '</option>';
    };

2. Retrieving License Questions

Presenting the questions gets interesting, because the server must now present questions, and accept answers, without the programmer knowing in advance what those questions will be.

Browser requests for license questions invoke a URL which depends on the user's chosen license type. If the license type is "Creative Commons", which has the id "standard", the URL is http://api.creativecommons.org/rest/1.0/license/standard/. Visit that URL, and you'll see the questions and possible answers. For example:

    <field id="derivatives">
     <label xml:lang="en">Allows modifications of your work?</label>
     <description xml:lang="en">The licensor permits others to copy, distribute and perform only
unaltered copies of the work, not derivative works based on it.</description>
     <type>enum</type>
     <enum id="y">
       <label xml:lang="en">Yes</label>
     </enum>
     <enum id="sa">
       <label xml:lang="en">ShareAlike</label>
     </enum>
     <enum id="n">
       <label xml:lang="en">No</label>
     </enum>
    </field>

Equipped with all that information about each question, the server must now transform it into an HTML selector. Ultimately, a loop is used to traverse the entire data structure and generate a selector for each field:

    foreach ($fields as $f_id=>$f_data) {
      $result .= '<tr><th><nobr>' . $f_data['label'] . '</nobr></th><td>';

 
     // generate the appropriate widget
     if ($f_data['type'] == 'enum') {
        $result .= '<select class="lic_q" id="'.$f_id.'" lic_q="true" size=
"1">';

        foreach ($f_data['options'] as $enum_id=>$enum_val) {
          $result .= '<option value="'. $enum_id . '">' . $enum_val . '</option>';
        } // for each option
        ...

As explained in the “Live Form” discussion, this new form HTML is directly placed onto a div.

3. Handling User Answers

The Creative Commons' issueLicense web service accepts XML input of all the license criteria, and outputs a document containing a URL for the license criteria, along with some other, related, information. In a similar manner to the previous code, all of this information is transformed into HTML. This time, it's used to populate several fields, rather than generate any new input fields.

Alternatives

A fairly old cross-domain technique is to use “On-Demand Javascript”; see the discussion of Cross-Domain Loading in that pattern. The main benefit over Cross-Domain Proxy is reduced resources - the base server is bypassed, so there's no bandwidth or processing costs involved. However, there's a major constraint: the server must be exposing a suitable script to fetch, because can't just extract arbitrary information from a server. Additional problems include lack of server-side logging, inability to reach services that require authentication, and the security concerns described in “On-Demand Javascript”.

Shared document.domain

When we speak of the "same-origin" policy, we're not necessarily referring to the true domain a document is served from; each document has a mutable domain property (viz., document.domain) which turns out to be the critical factor in cross-domain calls. If two documents declare the same domain property, regardless of their true origin, they should be able to communicate with each other using XMLHttpRequest. Jotspot developer Abe Fettig has explained how to exploit this knowledge for making cross-domain communication practical with “XMLHttpRequest Call”. The trick relies on embedding the external document, and having that document - as well as your own document - define the same document.domain property. Thus, as with the “On-Demand Javascript” alternative, it does have one key constraint: the external server must explicitly co-operate. In addition, you can't just declare an arbitrary domain; it has to be a "parent" of the true domain, i.e. "shop.example.com" can declare "example.com" but not "anyoldmegacorp.com". Due to these constraints, it's best suited to a situation where there's some direct relationship between the respective site owners.

Images

Images have always worked across domains, with various benefits (syndication) and more than a few problems ("bandwidth theft") along the way. As noted in the Alternatives section of “XMLHttpRequest Call”, images can be used for general-purpose remoting, usually with 1-pixel images that will never be rendered. While that's a hack you'll unlikely need these days, transfer of legitimate images remains a useful capability. When you run the Google Maps API in your web page, for example, it pulls down map images directly from Google's own servers.

Related Patterns

Performance Optimisation Patterns

Since external calls can be expensive, the Performance Optimisation Patterns apply. They may be applied at the level of the browser or the server or both. For example, caching can take place in the browser, the server, or both.

Want to Know More?

WSFinder: A Wiki of public web services.

Live Wiki Version for Cross-Domain Proxy