Cross-Domain Proxy
Time your website with
WebWait - from the creator of AjaxPatterns.orgFrom Ajax Patterns
There's an archived version of this pattern available, taken from the Ajax Pattern book draft, showing roughly how it appeared before the page became publicly editable.
Evidence: 3/3
Tags: Aggregator, Fusion, Mash-Up, Mashup, Mediator, Mix, Proxy, Tunnel
Contents |
|
In a Blink
Note: Throughout this pattern, "web service developer" refers to the owner of a web service being accessed, which may be as simple as a standard website being scraped with an HTTP client script.
Ajax mashup eg map pulled in for a user registration page
Goal Story
TODO Change to technical story
Bill is bored with his accounting job and decides to start a new career as a lion tamer. He visits his favourite recruitment meta-search site and performs a search. A results area immediately appears below the search form, and begins being populated with postings taken from various corporate websites. Since the results are in different formats, the application has taken the liberty of parsing them to extract company names, and augmented each listing with a phone number extracted from an external server. Each listing has a couple of links too. Clicking on the "finance" link for one position, the area below expands, and a moment later, shows an annual stock chart which the server has just pulled in from a finance web service. Clicking on "Location", the stock chart fades to make way for a map locating the company's headquarters, with a photo beside it.
Problem
How can you augment your application with services available elsewhere on the web?
Forces
- There's a lot of data and useful services on the web, much of it freely available. It's more effectively to leverage that content than go to the effort of duplicating it yourself.
- The "same-domain policy" constrains most Web Remoting techniques, meaning that the browser script can only talk to the server from whence it came, and no-one else.
- External servers often use protocols and message formats that would be difficult for browser-side scripts to process. They may also require authentication, and such details should not be exposed in the browser.
- To comply with service agreements, website owners often have to control and monitor all traffic to and from their web service, which is impossible if their server is being bypassed.
Solution
TODO
Include http://www.softwareas.com/cross-domain-portlets-with-startcom-gadgetswidgetsstartlets
Decisions
Is it legal to access the web service?
First up: The legals of Cross-Domain Proxys are tricky, vaguely-defined, and always evolving. I'm no lawyer, so take everything here with a grain of salt and seek legal advice if in doubt.
You know that little "legal" link you've been ignoring for the past decade? Now would be a good time to open it up and have a read. Many websites include a license agreement. By accessing a server, in a browser or via a script, you're implicitly agreeing to comply with that agreement. And in many cases, the agreement will explicitly prohibit usage by a script altogether. Here's what Google has to say about its main content:
The Google Services are made available for your personal, non-commercial use only. You may not use the Google Services to sell a product or service, or to increase traffic to your Web site for commercial reasons, such as advertising sales. You may not take the results from a Google search and reformat and display them, or mirror the Google home page or results pages on your Web site. You may not "meta-search" Google. If you want to make commercial use of the Google Services, you must enter into an agreement with Google to do so in advance. Please contact us for more information.
So that pretty much rules out a script, huh? Then what's with all the mashup projects? Well, a few things are happening there:
- Firstly, people are simply taking their chances. While companies like Google are keen to protect their data from competitors, they have often closed a blind eye to hackers playing around with the services on offer. When people start charging for those services, that's when they can expect to hear from company lawyers in the near future.
- Public APIs are a big trend in Web 2.0. Here, the web service owner explicitly defines how the interaction takes place, and often provides suporting documentation and code. In many cases, you will need set up an account to gain access, and will be given a limited number of queries per day. If you wish to make more queries, or access the server in ways other than provided by the API, you'll need to negotiate directly with the company.
- Just because the standard terms prohibit certain usage, that doesn't mean it can't happen. It simply means contacting the company and discussing a possible arrangement. Given that the service already exists, the incremental cost of further queries are usually trivial compared to the value they can provide. Thus, it is in many web service owners' interests to license their services.
How can information be accessed from an external server?
If you're scraping content directly from a public website, you'll need to use the underlying web protocol, HTTP. And even if you're talking to an API, you'll probably be communicating with HTTP anyway. In some cases, API publishers will provide code to access the content, rather than just publishing the specification. Also, with more sophisiticated web services protocols like SOAP, you don't have to write a lot of low-level code yourself. In many cases, though, the easiest thing to do is talk HTTP, and manually format and parse messages yourself.
Many scripting languages feature built-in libraries to communicate in HTTP and a language are often well-equipped for Cross-Domain Proxy implementations due to strong support for regular expression manipulation. A PHP example is featured in the code examples below.
In the Java world, the standard API provides some relatively low-level support for HTTP clients, but you can use some of the website testing libraries to quickly extract content from external sites. HttpUnit is a good example, and it also has some support for Javascript manipulation. To grab some content, do this:
import com.meterware.httpunit.WebConversation;
import com.meterware.httpunit.GetMethodWebRequest;
import com.meterware.httpunit.WebRequest;
public class PatternSorter {
public String getContent(String url) {
try {
WebConversation wc = new WebConversation();
WebRequest request = new GetMethodWebRequest(url);
String response = wc.getResponse(request).getText();
} catch(Exception ex) {
throw new RuntimeException("Could not get content from " + url, ex);
}
}
public static void main(String[] args) {
System.out.println("http://mahemoff.com");
}
}
What web services can be accessed?
In theory, anything you can do with a browser can theoretically be accomplished by an automated HTTP client. However, there are likely to be legal constraints, and technical challenges too if there is serious usage of Javascript or browser plugins. Furthermore, your script will be vulnerable to changes in the site layout, and many content providers have deliberately performed subtle changes to prevent automated access. Thus, relying on the ever-increasing collection of public APIs and structured data would make a better choice.
You can find a collection of public APIs at [1]. Some popular APIs include: Amazon, Delicious, EvDB, Flickr, Google Map, Google Search, Technorati, Yahoo Maps.
What if the service cannot be accessed?
At times, there will be errors accessing an external web service. Think twice before attributing blame: the problem might be at your end, or somewhere along the network. This has implications for any error messages you might show users.
Ideally, you should be able to detect an error has occurred and log the details for immediate attention. In responding to the user, you have several options:
- In some cases, you will simply have to admit failure and show an error message to users. If possible, suggest an alternative for them or tell them when they should try again.
- Provide reduced functionality.
- Switch to a pre-arranged substitute web service.
- Rely on cached results, if they're still relevant.
Real-World Examples
Mixendo Cross-domain XHR (MiXHR)
There is a nice implementation of this Cross-domain Proxy pattern from the guys at mixendo.com. It's the first brick of their platform to facilitate building mash-up applications, and it comes with some nice additions:
- a light-weight JS library that, once included, let you use your XMLHttpRequest object as always - but cross-domain enabled!
- supports all length of URLs, and HTTP modes, so that you can do POST and GET without worry
- supports cookies
- includes a Security enhancement: an enforced Access Control over a list of servers that the web application wants to connect to, so that users are aware (and can control) where data is going
- cross-browser, of course
It is the first step towards a platform aimed at changing the way you browse the web, making mash-up easier: for application developers, and for users. You can of course find more information on their Cross-Domain XMLHttpRequest service on the Mixendo Developers Documentation website. They talk about cross-domain Security, alternative methods, and more.
Housing Maps (craigslist and Google Maps)
Housing Maps is a mashup between Google Maps and craigslist, a community-oriented classifieds advertising website. What does that mean? It means you get to see a list of advertised homes on one side and view a map of those homes one the other side, using the familiar Google Map thumbtacks to pinpoint the locations of the advertised homes.
A few features:
- Click on a thumbtack and you'll see a Popup balloon appear, showing a summary of the classified ad: price, terms, address, photos. The balloon contains a hyperlink to the ad at cragislist.com.
- Change a category (e.g. price) and you'll see both the thumbtacks and home lists update. Change the city and you'll see the map change as well.
- You can pan and zoom as on Google Maps.
An interesting historical note: this website appeared soon after Google Maps itself came out, and rapidly gained prominence as a powerful demonstration of combining services from different websites. For this reason, Tim O'Reilly labelled it none other than "the first Web 2.0 application". In fact, a slew of Ajaxian Google Maps mashups continue to appear, all of them good examples of the Cross-Domain Proxy pattern.
ByTheOwner began using google maps as a search function on our site in late 2005. Remax.ca soon followed, but they have yet to master the tool. Notice how small their font is.
Bill Gates Wealth Clock
This was not Ajaxian, but noteworthy as the first well-known mashup application and a precursor to the Ajaxian Cross-Domain Proxy pattern. All-round web guru Philip Greenspun mixed data from several sources to produce a "wealth clock" showing Bill Gates' worth at any time (No longer working, unfortunately, but the code is still available. Here's the sort of thing you'd see in the brower:
Microsoft Stock Price: $26.89 Bill Gates's Wealth: $60.732674 billion U.S. Population: 291,604,291 Your Personal Contribution: $208.27
The price per stock was extracted from Yahoo Finance. The number of stocks in Gates' portfolio came from Microsoft reports (it was not automatically extracted). And the US population came from the US Census Bureau.
Here's what Greenspun said in 2001:
(M)any of the most interesting and challenging problems of the coming decades will center around the Internet and Web as a distributed computing environment. An "island" site such as amazon.com or photo.net is probably less representative of the future than http://www.webho.com/WealthClock (Bill Gates Personal Wealth Clock).
Another well-known type of cross-domain mediation in Web 1.0 was meta-searching, as performed by crawlers such as MetaCrawler.com. Again, all the work was done in the server, and the browser response was usually opened while the results came back to the server, so at least some information could be shown while requests were still out.
CPaint Framework
CPaint is a Web Remoting framework. A special "proxy" URL can be established, pointing to a proxy on the originating server, so that remoting to external domains has virtually the same API as to the originating server.
HTTP proxy
HTTP proxy lets you transparently pass all the incoming requests to the some predefined host. Servlet simply performs a role of HTTP proxy for your requests.
Code Example
WPLicense
The WPLicense plugin is a good example of a server application talking to an API in order to service Web Remoting calls. In this case, the API belongs to Creative Commons, which is nice because it's completely public - no API key required. As we'll see below, you can simulate the server's request just by typing URLs into your browser.
See the Live Form#WPLicense discussion for a full background on the license selection process, but here's a quick summary:
- A field lets the user select a license type (e.g. Creative Commons versus Public Domain versus Sampling).
- Once the license type is chosen, its immediately sent to the server, and the server responds with a form consisting of questions relevant to this particular type. If the license is "Creative Commons", for example, one of the questions is "Allow commercial uses of this work?"
- Each time the user changes one of the license options, the server is called some hidden fields. These fields will be uploaded to the server, to be persisted, when the user eventually submits the form.
What all this glosses over is the Cross-Domain Proxy pattern that goes on behind the scenes. At each of these three stages, the server is actually interacting with CreativeCommons.org via its public API, which you can read about at [2]. The plugin provides a clean separation between web server logic and cross-domain mediation: a separate php file (ccwsclient.php) hosts several API facade operations, accepting and returning data structures in standard PHP style. This client in turn delegates all the XML infrastructure stuff to a 3rd party library (MiniXML).
Let's now zoom in on the three steps.
1. Retrieving License Types
The server does not have the license types hard-coded. Instead, it retrieves them via a web-service:
$WS_ROOT = "http://api.creativecommons.org/rest/1.0/"; ... $xml = file_get_contents($WS_ROOT); [ccwsclient.php]
In fact, you can see exactly what's pumped into $xml by visiting [3], then View Page Source. Alternatively, use a command-like application like curl ('curl <url>' will do the trick).
<licenses> <license id="standard">Creative Commons</license> <license id="publicdomain">Public Domain</license> <license id="recombo">Sampling</license> </licenses> [Result from http://api.creativecommons.org/rest/1.0/. (Reformatted)]
Once we know what license types are available, it's a simple matter of transforming the XML into an HTML selector. XSLT could be used here, but it's just as easy to do a little hacking:
foreach ($licenses as $l) {
$l_classes[strval($l->attribute('id'))] = strval($l->getValue());
}
return $l_classes;
[ccwsclient.php]
foreach($license_classes as $key => $l_id) {
echo '<option value="' . $key . '" >' . $l_id . '</option>';
};
[wpLicense.php]
2. Retrieving License Questions
Presenting the questions gets interesting, because the server must now present questions, and accept answers, without the programmer knowing in advance what those questions will be.
Browser requests for license questions invoke a URL which depends on the user's chosen license type. If the license type is "Creative Commons", which has the id "standard", the URL is http://api.creativecommons.org/rest/1.0/license/standard/. Visit that URL, and you'll see the questions and possible answers. For example:
<field id="derivatives">
<label xml:lang="en">Allows modifications of your work?</label>
<description xml:lang="en">The licensor permits others to copy, distribute and perform only
unaltered copies of the work, not derivative works based on it.</description>
<type>enum</type>
<enum id="y">
<label xml:lang="en">Yes</label>
</enum>
<enum id="sa">
<label xml:lang="en">ShareAlike</label>
</enum>
<enum id="n">
<label xml:lang="en">No</label>
</enum>
</field>
[4]
Equipped with all that information about each question, the server must now transform it into an HTML selector. Ultimately, a loop is used to traverse the entire data structure and generate a selector for each field:
foreach ($fields as $f_id=>$f_data) {
$result .= '<tr><th><nobr>' . $f_data['label'] . '</nobr></th><td>';
// generate the appropriate widget
if ($f_data['type'] == 'enum') {
$result .= '<select class="lic_q" id="'.$f_id.'" lic_q="true" size="1">';
foreach ($f_data['options'] as $enum_id=>$enum_val) {
$result .= '<option value="'. $enum_id . '">' . $enum_val . '</option>';
} // for each option
...
As explained in the Live Form discussion, this new form HTML is directly placed onto a DIV.
3. Handling User Answers
The Creative Commons' issueLicense web service accepts XML input of all the license critieria, and outputs a document containing a URL for the license criteria, along with some other, related, information. In a similar manner to the previous code, all of this information is transformed into HTML. This time, it's used to populate several fields, rather than generate any new input fields.
Alternatives
On-Demand Javascript
A weak alternative is On-Demand Javascript. Unlike with XMLHttpRequest and Frames/IFrames, it is actually possible to fetch some content from a remote website as part of a script element, and you can create that script element dynamically. The content must be valid Javascript, but if you have any control over the other server, you can export arbitrary data as a JSON Message (a JSON Message is a Javascript expression, which also means it's also a valid script).
Note: To actually access the data from a remote JSON source via a dynamic script tag, you need to actually use a variation called JSON-P (JSON with Padding), which involves having the JSON returned with a function call wrapped around it. Typically, in the URL of the script tag request, you pass a parameter like "?...callback=foo&..." and the service will then return something like "foo({a:1,b:2...});". The requesting page then has a function defined called "foo", which will be called with the response JSON as expected.
JSON-P XHR
A variation of the JSON-P On-Demand Javascript approach is called jXHR. jXHR uses JSON-P for the cross-domain calls, but it exposes an API on the jXHR object instance that is compatible with the native XHR API, which makes cross-domain calls (using JSON-P) look exactly like normal same-domain Ajax calls. This makes using jXHR in an existing site/application easier than just doing a custom JSON-P solution, since the code shouldn't need to change much if at all. Also, jXHR provides basic error handling, which is something that most custom JSON-P solutions do not provide.
Flash-enabled XHR
The Flash-enabled XHR pattern is another solution for cross-domain scripting. It takes advantage of Flash's ability to make cross-domain Ajax calls by checking the target server for an opt-in policy file to authorize the connection.
flXHR is an implementation of this pattern, taking the additional step of implementing the identical API to the normal native XHR Ajax object. This means flXHR can be dropped into any existing web code and enable cross-domain Ajax calls in the same way as normal Ajax calls. In particular, this makes it very easy to use flXHR even with Javascript frameworks like jQuery, Dojo, etc. There are even plugins which do the work of adapting the framework to use flXHR, so you don't even have to think about how to do it.
Another benefit of the Flash-enabled XHR solutions is that content-type for the request and for the response are not restricted as they are with On-Demand Javascript for instance. You can send and receive text, html, xml, json, even binary data.
There are other similar solutions to flXHR, such as CrossXHR, F4A, FlashXMLHttpRequest, etc. However, flXHR is the only currently available solution which implements the exact API, making flXHR the only one which is a truly drop-in replacement for native XHR Ajax calls.
Related Patterns
Performance Improvement Patterns
Since external calls can be expensive, the performance improvement patterns apply. They may be applied at the level of the browser or the server or both. For example, caching external responses can be achieved in the server, which would naturally be effective if there were
Visual Metaphor
A language translator in a United Nations meeting mediates the conversation between two diplomats.
