AjaxPatterns
This is the original book draft version for the On-Demand Javascript Ajax pattern, offered for historical purposes and as an anti-spam measure.
You're probably better off visiting the Live Wiki Version for On-Demand Javascript instead, where you can make changes and see what others have written.

On-Demand Javascript

Behaviour, Bootstrap, CrossDomain, Dynamic, Javascript, LazyLoading, OnDemand, ScriptTag

Goal Story

Bill's logging into his bank website. The login form came up straight away, and as he types in his username and password, the browser is quietly downloading the Javascript necessary for the rest of the application.

Problem

How can you deploy lots of Javascript code?

Forces

  • Ajax Apps make heavy use of Javascript. Richer browser behaviour means bulkier Javascript to download.

  • Downloading Javascript has a performance impact. Interaction cannot fully begin until all initial Javascript has been loaded.

  • Bandwidth is also a concern with heavy Javascript content. Often, not all Javascript that's loaded is actually used, leading to a waste of bandwidth.

Solution

Download and run Javascript snippets. The initial page load includes some Javascript code, which - among other things - contains the bootstrapping code necessary to pull down further Javascript. There are two techniques available: Script Tag Creation and Service Eval. Each will now be described, following by three main applications of the pattern.

.

With Script Tag Creation, you use DOM manipulation to inject a new script element into the page. Perhaps surpisingly, the effect will be exactly the same as if the <script> tag had been encountered on startup: the referenced Javascript will be downloaded and executed automatically. The tag can be attached to either the head or the body, though the former is more common as that's where you'd usually find script tags:

    var head = document.getElementsByTagName("head")[0];
    script = document.createElement('script');
    script.id = 'importedScriptId';
    script.type = 'text/javascript';
    script.src = "http://path.to.javascript/file.js";
    head.appendChild(script);

How will you know when the script has loaded? It seems browsers vary in their behaviour here, with some loading the script synchronously and some not. In IE, you can be notified a mechanism similar to XMLHttprequest's onreadystatechange property. With other browsers, you might need to keep polling for some indication it's loaded (where the indication depends on what the script is meant to do). If you happen to control the server script, what you could do is implement a notification mechanism, i.e. complete the script by calling back to a listener, if it exists. There's actually a promising proposal, JSONP, which aims to have a simple, flexible, mechanism like this, standard across the entire industry.

Script Tag Creation doesn't let you retrieve any old text; it has to be valid Javascript. And your script won't be able to read it directly because the browser will use it only for evaluation. So how can you get hold of some data from an external server? The most common way is for the remote Javascript to assign a variable to the required data:

    var characters = new Array("Mario", "Sonic", "Lara");

The content is usually a data structure, just like a standard “JSON Message”, but requires an assignment as well, in order to be referenced in the code. It could also be a function that returns the desired value. Either way, the browser code has to use whatever name is mentioned in the script, which isn't ideal. Again, the idea of a flexible script mechanism is worth considering, as it would let the caller decide on the variable name.

Service Eval is the other technique for On-Demand Javascript, though not as prominent as Script Tag Creation for reasons we'll discuss later. Under Service Eval, a “Web Service” is called with a standard “XMLHttpRequest Call”, it outputs some Javascript as response content, and the Javascript is then executed with an eval() call. We're just inspecting the XMLHttpRequestCall's responseText property, which we could manipulate before evaluating, so the body doesn't have to be a complete, valid, piece of Javascript (unlike with Script Tag Creation).

Any code not inside a function will be executed immediately. To add new functionality for later on, the Javascript can add directly to the current window or to an object that's already known to exist. For example, the following can be sent:

    self.changePassword = function(oldPassword, newpassword) {
      ...
    }

The XMLHttpRequest callback function just needs to treat the response as plain-text and pass it to eval(). A warning: here again, asynchronicity rears its ugly head. You can't assume the new code will be available immediately after requesting it, so don't do this:

    if (!self.changePassword) {
      requestPasswordModuleFromServer();
    }
    changePassword(old, new) // Won't work the first time because
                             // changePassword's not loaded yet.

Instead, you either need to make a synchronous call to the server, add a loop to keep checking for the new function, or explicitly make the call in the response handler.

On-Demand Javascript has three distinct applications:

Lazy Loading

Defer loading of bulky Javascript code until later on. Works with either On-Demand Javascript technique (Service Eval or Script Tag Creation).

Behaviour Message

Have the server respond with a kind of "Behaviour Message" which dictates the browser's next action. Works with either On-Demand Javascript technique (Service Eval or Script Tag Creation).

Cross-Domain Scripting

Using Script Tag Creation, bypass the standard "same-origin" policy that normally necessitates a “Cross-Domain Proxy”. Only works with Script Tag Creation.

Let's look at Lazy Loading first. Conventionally, best practice has been to avoid include Javascript unobtrusively - by including it in one or more script tags:

    <html>
      <head>
        <script type="text/javascript" src="search.js"></script>
        <script type="text/javascript" src="validation.js"></script>
        <script type="text/javascript" src="visuals"></script>
      </head>
      ...
    </html>

Lazy Loading builds on this approach to suggest just a minimal initialisation module in the initial HTML:

    <html>
      <head>
        <script type="text/javascript" src="init.js"></script>
      </head>
      ...
    </html>

The initialisation module declares whatever actions are required to start up the page and perhaps enough to cover typical usage. In addition, it must perform a bootstrapping function, pulling down new Javascript on demand.

The second application, Behaviour Message is a variant of the “HTML Message” pattern. Whereas Lazy Loading sets up library code for ongoing use, Behaviour Messages take some transient code and runs eval() on it immediately - the script is not wrapped inside a function (although it may define some functions that it calls). Effectively, the browser is asking the server what to do next.

Cross-Domain Scripting is the third application of this pattern. script tags have always been able to include source Javascript from external domains. The rule not only applies to static <script> tags, but also dynamically created script tags as in the Script Tag Creation technique. Thus, unlike with XMLHttpRequest and IFrame, your script can directly access external content this way. And because the src property can be any URL, you can pass in arguments as CGI variables. The idea is becoming rather popular, with companies such as Yahoo! offering Javascript APIs specifically for this approach (see Real-World Examples below).

Running a script from an external domain can be useful when you trust it, and ideally control it. Other times, it's a definite security risk. Douglas Crockford, creator of JSON, warns of the havoc an external script can wreak (emphasis mine):

That script can deliver the data, but it runs with the same authority as scripts on the base page, so it is able steal cookies or misuse the authorization of the user with the server. A rogue script can do destructive things to the relationship between the user and the base server. ... The unrestricted script tag hack is the last big security hole in browsers. It cannot be easily fixed because the whole advertising infrastructure depends on the hole. Be very cautious.

Decisions

Will you use Service Eval or Script Tag Creation?

The choice is between Service Eval and Script Tag Creation depends on several factors. Service Eval has the following benefits over Script Tag Creation:

  • Being based on XMLHttpRequest, there's a standard mechanism for being notified when the script is ready, so there's no risk of calling functions that don't yet exist.

  • You get access to the raw script code.

  • There's more flexibility on the message format: you can, for example, send several Javascript snippets inside different XML nodes, and have the browser script extract them out.

And Script Tag Creation has a couple of benefits over Service Eval:

  • You can load Javascript from external domains. This is the only significant functional difference between the two styles.

  • The Javascript will automatically be evaluated in much the same way as Javascript linked in the static HTML is evaluated when the <script> tag is first encountered. Thus, you don't have to explicitly add variables and functions to the document in order to use them later; you just declare them normally.

With Lazy Loading, How will you break modules down?

You'll need to decide how to carve up your Javascript. Standard principles of software development apply: a module should be well-focused, and inter-module dependencies avoided where possible. In addition, there are web-specific concerns:

  • Ideally, any given module is either not used at all, or used in entirety. What you don't want is a 5000-line module that's downloaded to retrieve a 3-line function. It's a waste of bandwidth that defeats the main purpose of On-Demand Javascript.

  • You need one or more modules present on startup. At least one is required to kick off further downloads as required.

  • Keep in mind that code will probably be cached locally. Alexander Kirk has done some experimentation with caching of On-Demand Javascript, and it turns out that all major browsers will cache code created with Script Tag Generation, and, as you can usually ensure responses from XMLHttpRequest are also cached.

With Lazy Loading, at what stage will the script download the Javascript?

It's easiest to download the Javascript just before it's required, but that's not always the best approach. There will be some delay in downloading the Javascript, which means the user will be waiting around if you grab it at the last possible moment. When the user's actions or the system state suggest some Javascript will soon be needed, consider downloading it immediately.

Predicting if Javascript is needed is an example of “Predictive Fetch” - grabbing something on the hunch that it might be needed. You need to make a trade-off about the likelihood it's required against the hassle caused if you hold off until later. For example, imagine you have some Javascript to validate a completed form. If you wait until the end, you'll definitely not be wasting bandwidth, but at the expense of the user's satisfaction. You could download it when there's one field to go, or two fields, or when the form's first loaded. Each option increases the chance of a wasted download, but increases the chance of a smoother validation procedure.

Real-World Examples

MapBuilder

MapBuilder is a framework for mapping websites (Figure 1.15, “Mapbuilder”). It uses On-Demand Javascript to reduce the amount of code being downloaded. The application is based on a Model-View-Controller paradigm. Model information is declared in an XML file, along with Javascript required to load corresponding widgets. When the page starts up, only the required Javascript is downloaded, instead of the entire code base. This is therefore a partial implementation of this pattern; it does ensure that only the minimal subset of Javascript is ever used, but it doesn't load pieces of the code in a lazy, on-demand, style.

Figure 1.15. Mapbuilder

Mapbuilder

Delicious/Yahoo! APIs

Social bookmarking website Delicious, and its parent company Yahoo!, both offer JSON-based APIs, with appropriate hooks to allow direct access from the browser via Script Tag Creation. The Delicious API will create a new object with the result (or populate it if it's already present). The Yahoo! API allows you to specify a callback function in your script which the JSON will be passed to.

Dojo Packaging Framework

Dojo is a comprehensive framework aiming to simplify Javascript development. As such, it provides a number of scripts, of which an individual project might only use a subset. To manage the scripts, there's a Java-like package system, which lets you pull in new Javascript as required. You need only include a single Javascript file directly:

    <script type="text/javascript" src="/dojo/dojo.js"></script>

You then pull in packages on demand with the dojo API:

    dojo.hostenv.moduleLoaded("dojo.aDojoPackage.*");

Running the above command will cause Dojo to automatically download the modules under dojo.aDojoPackage [3].

JSAN Import System

The JavaScript Archive Network (JSAN) is an online repository of scripts. As well, it includes a library for importing Javascript modules, also a convention similar to Java. The following call:

    JSAN.use('Module.To.Include');

is mapped to Module/To/Include.js. JSAN.includePath defines all the possible top-level directories where this path can reside. If the includePath is <a href="/",">/js</a>, JSAN will look for /Module/To/Include and /js/Module/To/Include.

Code Example [AjaxPatterns On-Demand Javascript Wiki]

Introducing On-Demand Javascript to the Wiki Demo

In the Basic Wiki Demo, all Javascript is downloaded at once. But many times, users will only read from the wiki - why download the code to write to it? So this demo refactors to On-Demand Javascript in three stages:

  1. The uploadMessage function is extracted to a second Javascript file, upload.js. There's no On-Demand Javascript yet because both files are included.

  2. A further refactoring introduces On-Demand Javascript by ensuring that upload.js file is downloaded only if and when an upload occurs. This version uses Script Tag Creation.

  3. In yet another refactoring, the Script Tag Creation technique is replaced with Service Eval.

Separate Javascript: Extracting upload.js

In the first refactoring, the upload function is simply moved to a separate Javascript file. The initial HTML includes the new file:

    <script type="text/javascript" src="wiki.js"></script>
    <script type="text/javascript" src="upload.js">

The new upload.js now contains the uploadMessage function. Reflecting the separation, a couple of parameters are introduced to help decouple the function from the main wiki script:

    function uploadMessages(pendingMessages, messageResponseHandler) {
      ...
    }

The calling code is almost the same as before:

    uploadMessages(pendingMessages, onMessagesLoaded);

We've thus far gained a bit of modularity, but no boost to performance yet, since both files must be downloaded on startup.

Script Tag Creation On-Demand Javascript

With On-Demand Javascript, the upload.js is no longer required on startup, so its reference no longer appears in the initial HTML, leaving just the main module, wiki.js:

    <script type="text/javascript" src="wiki.js"></script>

A new function has been added to download the script. To avoid downloading it multiple times, a guard clause checks if uploadMessages already exists, and if so, immediately returns. Following the Script Tag Creation technique, it adds a script element to the document's head, initialised with the upload.js URL and a standard Javascript type attribute:

    function ensureUploadScriptIsLoaded() {
      if (self.uploadMessages) { // Already exists
        return;
      }
      var head = document.getElementsByTagName("head")[0];
      script = document.createElement('script');
      script.id = 'uploadScript';
      script.type = 'text/javascript';
      script.src = "upload.js";
      head.appendChild(script);
    }

The calling script just has to invoke this function. However, as mentioned in the Solution, it's possibly downloaded asynchronously, so a check must be made here too:

    ensureUploadScriptIsLoaded();
    if (self.uploadMessages) { // If not loaded yet, wait for next sync
      uploadMessages(pendingMessages, onMessagesLoaded);
      ....

If scripts are loaded asynchronously by the browser, the test will actually fail the first time, because the download function will return before the script's been downloaded. But in this particular application, that doesn't actually matter much - the whole synchronisation sequence is run every five seconds anyway. If the upload function isn't there yet, it should be there in five seconds, and that's fine for our purposes.

Service Eval On-Demand Javascript

The Script Tag Creation code is refactored here to use an Service Eval instead, where an “XMLHttpRequest Call” retrieves upload.js and evals it. The initial script - wiki.js differs only in its implementation of the Javascript retrieval. The text response is simply passed to eval:

    function ensureUploadScriptIsLoaded() {
      if (self.uploadMessages) { // Already exists
        return;
      }
      ajaxCaller.getPlainText("upload.js", function(jsText) { eval(jsText); });

    }

The Javascript response must also change. If it simply defined a global function like before, i.e.:

    function uploadMessages(pendingMessages, messageResponseHandler) {
      ...
    }

then the function would die when the evaluation completes. Instead, we can achieve the same effect by declaring the function as follows (this will attach the function to the current window, so it will live on after the script is evaluated).

    uploadMessages = function(pendingMessages, messageResponseHandler) {
      ...
    }

Related Patterns

The Behaviour Message usage of this pattern is a companion to “HTML Message” and follows a similar server-centric philosophy in which the server-side dynamically controls browser activity.

Apply “Predictive Fetch” to On-Demand Javascript by downloading Javascript when you anticipate it will soon be required.

The Lazy Loading application of this pattern is a like “Multi-Stage Download”, which also defers downloading. The emphasis in “Multi-Stage Download” is downloading semantic and display content rather than downloading Javascript. In addition, that pattern is more about downloading according to a pre-scheduled sequence rather than downloading on demand.

Want To Know More?

  • Dynamic data using the DOM and Remote Scripting A tutorial by Thomas Brattli.



[3] It's possible not all modules will be downloaded, because the precise set can be defined by the package author, and can be made dependent on the calling environment (whether browser or command-line).

Live Wiki Version for On-Demand Javascript