On-Demand Javascript - Ajax Patterns

On-Demand Javascript

From Ajax Patterns

Evidence: 2/3

Tags: Behaviour Bootstrap Dynamic Javascript LazyLoading OnDemand ScriptTag


Contents

In A Blink

JavaScript being pulled from the server after the page has loaded


Technical Story

Devi's concerned about initial user experience, because her new website requires lots of Javascript. She refactors the application, extracting out a few Javascript modules. The initial page load is now much faster, because only a bare-bones initialisation module is pulled down, with the other modules loaded only when they're needed.


Problem

How can you deploy lots of Javascript code?


Forces

  • Ajax applications make heavy use of Javascript. Richer browser behaviour means more script code must be downloaded.
  • Downloading Javascript has a performance impact. Interaction cannot fully begin until all initial Javascript has been loaded.
  • Bandwidth is also a concern with heavy Javascript content. Often, not all Javascript that's loaded is actually used, leading to a waste of bandwidth.


Solution

Download Javascript as and when required, instead of downloading it all on page load. This is a "lazy loading" approach applied to Javascript, and has several benefits:

  • Initial page load is faster.
  • Overall bandwidth usage is less, since only Javascript that's required is used.
  • Deployment is easier as the code takes care of pulling in Javascript, with the coder only ensuring enough is there to kick the process off.
  • You can produce snippets of Javascript on the fly - effectively sending back a kind of "behaviour message" advising the browser on its next action. This is a variant of the core pattern described at the end of this section.
  • You can bypass the standard "same-domain" policy that normally necessitates a Cross-Domain Proxy. Provided a server exposes a segment of valid Javascript, it can be extracted by the browser.

Conventionally, best practice has been to include Javascript unobtrusively - by including it in one or more script tags.

  <html>
    <head>
      <script type="text/javascript" src="search.js"></script>
      <script type="text/javascript" src="validation.js"></script>
      <script type="text/javascript" src="visuals.js"></script>
    </head>
    ...
  </html>

On-Demand Javascript builds on this approach to suggest just a minimal initialisation module in the initial HTML:

  <html>
    <head>
      <script type="text/javascript" src="init.js"></script>
    </head>
    ...
  </html>

The initialisation module declares whatever behaviour is required to start up the page and perhaps enough to cover typical usage. In addition, it must perform a bootstrapping function, pulling down new Javascript on demand.

So how do you load Javascript after the page has loaded? One way is with an XMLHttpRequest Call. The browser can easily accept a Javascript response and evaluate it. Any code not inside a function will be executed immediately. To add new functionality for later on, the Javascript can add directly to the current window or to an object that's already known to exist. For example, the following can be sent:

 self.changePassword = function(oldPassword, newpassword) {
   ...
 }

The callback function just needs to treat the response as plain-text and pass it to eval(). A warning: here again, asynchronicity rears its ugly head. You can't assume the new function will be available immediately. So don't do this:

 if (!self.changePassword) {
   requestPasswordModuleFromServer();
 }
 changePassword(old, new) // Won't work the first time because
                          // changePassword's not loaded yet.

You either need to make a synchronous call to the server, explicitly make the call in the response handler, or use a closure callback - either manually or using a continuation transformer such as Narrative JavaScript or djax. Note that you can't just add a loop to keep checking for the new function, due to the single-threaded nature of JavaScript.

A less obtrusive way to do the same thing involves DOM Manipulation. Simply insert a suitable script element into the DOM and the JavaScript will be loaded. It can be a child of either the head or body and its src property must be pointing to a JavaScript URL. There are several differences with XMLHttpRequest:

  • The JavaScript will automatically be evaluated in much the same way as JavaScript linked in the static HTML is evaluated when the <script> tag is first encountered. You can declare a bare function or variable without adding it to the window.
  • You can load JavaScript from external domains.
  • The URL points to a specific Javascript resource. With XMLHttpRequest, there's more flexibility: you can, for example, send several JavaScript snippets inside different XML nodes.
  • The DOM is affected in a different way. Even if the behaviour is transient, the script will be added to the DOM, whereas it would disappear after an XMLHttpRequest callback eval'd it.

(Watch for compatibility. See http://www.howtocreate.co.uk/loadingExternalData.html)

  • Inserting script element into the DOM means that if the file is cached in the browser then the cache is used. This is not possible with XMLHttpRequest.

There's a variant of the pattern I'll call JavaScript Response in reference to the HTML Response pattern. The standard pattern refers to the situation where you're downloading a new set of functions, classes, and variables. In reality, you're just adding new members to the DOM - some other code will then invoke it. In JavaScript Response, you grab a snippet of Javascript and eval it - the script is not wrapped inside a function (although it may define some functions that it calls). Under this variant, then, the browser is asking the server what to do next. Whereas in the standard pattern, the browser knows what to do, but not how to do it.

One important application is with JSON Messages. Being standard JavaScript, they can be downloaded using On-Demand JavaScript, which is a convenient way to transfer semantic content, i.e. a self-contained object. Because Javascript can be pulled from third-party sources, this provides a mechanism for transferring semantic content from an external server into the browser, and without a Cross-Domain Proxy. Some Web Services, designed for public browser-based usage, therefore output JSON Messages.

If you are exposing a JSON API from your server for external browsers to use, you should be aware of some important security concerns. Any JSON-based services that rely on cookie-based authentication are vulnerable. Otherwise, you allow a malicious website to make cross-domain calls on behalf of the user, to a JSON service. This is unlike XMLHttpRequest Calls, which can't cross domains and IFrame Calls, where external domain details can't be accessed by the parent application. With JSON, there is no such protection...by redefining the ways of JavaScript, such as rewriting the Array constructor, hackers from evil.com can submit an authenticated JSON call to bank.com, and then do what they like with the response, e.g. upload it back to their servers. To prevent this: (a) ensure JSON is used to serve only public data, i.e. that which requires no cookie-based authentication; OR (b) if your web app relies on calling cookie-based JSON services from your own server, make the services POST-based, so that they can't be invoked from script tag inclusion (which always uses GET). This means your browser script will have to issue a POST-based XMLHttpRequest Call and eval() the response. Fortunately, a browser script from an external server won't be allowed to issue the same request.

Changing the src property

<script id="external_script" type="text/JavaScript"></script>
<script>
    document.getElementById('external_script').src = 'http://cross.domain.com/other.js';
</script>

document.write-ing a script element

For more details, especially on why the backslash is necessary before the closing script tag, see Dynamically loading external JavaScript files.

<script>
    document.write('<script src="', 'http://cross.domain.com/other.js', '" type="text/JavaScript"><\/script>');
</script>

Real-World Examples

Ensure

Ensure can load not just javascript, but html snippet and css on the fly. Best you can wait for them to load and then invoke functions that are loaded dynamically. For example, the following loads javascript, some html snippet, related css and then executes a javascript that is only available when the script is dynamically loaded.

   ensure( { js: "popup.js", html: "popup.html", css: "popup.css" }, function()
   {
       PopupManager.show();
   });

UFrame

UFrame is like an ajax IFrame. It makes a DIV behave like an iframe. You can load any page from the same domain, along with its javascripts, css, images etc on the fly. It also supports ASP.NET postback on the dynamically loaded part. Using it is as simple as using an iframe. For example,

This should get replaced with content from Somepage.aspx


Ajile

Ajile is a browser-independent JavaScript extension that provides on-demand script loading, namespace, import and dependency-management support.

Ajile provides cross-domain script loading via dynamic script insertion circumventing the Same Origin Policy problem that limits XMLHttpRequest-based solutions.

  Load    ("http://cross.domain/script.js");  // Load an arbitrary script from an external domain.
  Import  ("my.namespace.*");                 // Load & provide short-name access to all my.namespace modules.
  Include ("my.namespace.Module");            // Load my.namespace.Module
  ImportAs("Mine", "my.namespace.Module");    // Load my.namespace.Module as Mine (importing with aliases)

Ajile further enhances the loading process by providing configurable automatic script loading. This feature allows web pages to be associated with unique and/or shared scripts that are automatically loaded when Ajile is initialized. This automatic loading reduces the number scripts explicitly listed within web pages to just one, Ajile's:

  <script type="text/javascript" src="path/to/com.iskitz.ajile.js"></script>

By acting as a page's controller, Ajile allows script changes to be made independently of the host page. This simplifies page maintenance and improves team efficiency by allowing designers to work in script-free HTML while developers work in a pure scripting environment.

Persevere

Persevere Client is a persistent object mapping JavaScript framework. Persevere uses a persistent object model where the logic for applications can be defined within the persisted objects. The persisted objects as well as the JavaScript functions within are loaded on demand using the JSPON data format (an extension of JSON). Persevere uses JavaScript Strands for continuations, and by utilizing continuations, Persevere can do true inline on-demand loading of JavaScript and data without making synchronous (locking) remote calls. On-demand loading is done transparently as data is needed. Persevere supports lazy loading with standard JavaScript syntax, so the following is an example of referencing a persisted object and function with Persevere:

  myobj.anotherObject.render();

Persevere will transparently retrieve the object referred to by "anotherObject" and the render function if they have not been loaded then execution will continue inline. prediction for object transfers.

MapBuilder

MapBuilder is a framework for mapping websites. It uses On-Demand Javascript to reduce the amount of code being downloaded. The application is based on a Model-View-Controller paradigm. Model information is declared in an XML file, along with Javascript required to load corresponding widgets. When the page starts up, only the required Javascript is downloaded, instead of the entire code base. This is therefore a partial implementation of this pattern; it does ensure that only the minimal subset of Javascript is ever used, but it doesn't load pieces of the code in a lazy, on-demand, style.

Dojo Package System

Dojo is a comprehensive framework aiming to simplify Javascript development. As such, it provides a number of scripts, of which an individual project might only use a subset. To manage the scripts, there's a Java-like package system, which lets you pull in new Javascript as required. You only need include a single Javascript file:

  <script src="dojo.js" type="text/javascript">

You then pull in further packages on demand like this:

  dojo.require("dojo.aDojoPackage");

will download all relevant Javascript modules in dojo.aDojoPackage. The precise set of modules is defined by the dojo.aDojoPackage author, and can be made dependent on the calling environment (whether browser or command-line).

JSAN Import System

The JavaScript Archive Network (JSAN) is an online repository of scripts. As well, it includes a library for importing Javascript modules, also a convention similar to Java. The following call:

 JSAN.use('Module.To.Include');

is mapped to Module/To/Include.js. JSAN.includePath defines all the possible top-level directories where this path can reside. If the includePath is ["/", "/js"], JSAN will look for /Module/To/Include and /js/Module/To/Include.

twoBirds

This AJAX system is providing on-demand web-object functionality. In addition to load the .js file on demand, it also loads .html.tpl and .css files dynamically into the browser cache structure. A given twoBirds web-object has an init() function, that checks whether all files needed are present in the cache, if not waits for them to be loaded, and then calculates the html code of the element to be included in the DOM model. The following call:

tb.element.show( '<parameter object, e.g. target div id>', '<module>', '<element>' );

will load and display an element on the webpage in the given DIV element. twoBirds includes a modular site structure based on module directories.

Since a twoBirds web-object may include other twoBirds web objects, the site overall complexity level is unlimited, still allowing quality management of the code.

As of Mar 31st 2007 twoBirds V2.0.0 has been declared stable, see [1] for the prototype. Contact the programmer frank dot thuerigen -at- phpbuero justAnotherDot de if you want to get a copy of the lib, since system documentation and his ever-forgotten HP are "work in progress" right now.

Refactoring Illustration

Introducing On-Demand Javascript to the Wiki Demo

In the Basic wiki Demo, all Javascript is downloaded at once. But many times, users will only read from the wiki - why download the code to write to it? So this demo refactors to [On-Demand Javascript], in three stages:

1. The <td>uploadMessage function is extracted to a second Javascript file, upload.js. 2. A further refactoring pulls in the upload.js file only if and when an upload occurs. This version uses the DOM-morphing approach. 3. In yet another refactoring, the DOM-morphing approach is substituted with the XMLHttpRequest approach.

Separate Javascript: Extracting upload.js

In the first refactoring, the upload function is simply moved to a separate Javascript file. The initial HTML includes the new file.

  <script type="text/javascript" src="wiki.js"></script>
  *** <script type="text/javascript" src="upload.js"></script> ***

The new upload.js now contains the uploadMessage function. Reflecting the separation, a couple of parameters were introduced to decouple the function from the main wiki script.

 function uploadMessages(pendingMessages, messageResponseHandler) {
   ...
 }

The calling code is almost the same as before:

 uploadMessages(pendingMessages, onMessagesLoaded);

We've thus far gained a bit of modularity, but no boost to performance, since both files must be downloaded on startup.

DOM-Based On-Demand Javascript

This refactoring and the next both achieve On-Demand Javascript, in different ways. Neither approach is intrinsically superior - each approach has its own characteristics, as discussed in the Solution above.

With On-Demand Javascript, the upload.js is no longer required on startup, so its reference no longer appears in the initial HTML, leaving just the main module, wiki.js.

 <script type="text/javascript" src="wiki.js"></script>

To this file, a new function has been added to download the script. To avoid downloading the script multiple times, a check is made against uploadMessages, the function that's downloaded on demand. It adds a script element to the document's head, initialised with the upload.js URL and a standard Javascript type attribute. This new DOM element is at the heart of DOM-based On-Demand Javascript.

 function ensureUploadScriptIsLoaded() {
   if (self.uploadScript) { // Already exists
     return;
   }
   var head = document.getElementsByTagName("head")[0];
   script = document.createElement('script');
   script.id = 'uploadScript';
   script.type = 'text/javascript';
   script.src = "upload.js";
   head.appendChild(script);
 }

If you are going to fetch data from some script every interval, it is necessary to clean up after yourself before creating new DOM node with new script. In other case your DOM will grow lineary with time and consume more and more RAM. Modified previous example, which cleans up previous script before creating new call:

 function periodicChecker() {
   eval("var old = document.getElementById('uploadScript')");
   if (old != null) {
     old.parentNode.removeChild(old);
     delete old;
   }
   var head = document.getElementsByTagName("head")[0];
   var script = document.createElement('script');
   script.id = 'uploadScript';
   script.type = 'text/javascript';
   script.src = "upload.js";
   head.appendChild(script);
 }

The calling script just has to call this function. However, as mentioned in the Solution, it's downloaded asynchronously, so a check must be made here too.

 ensureUploadScriptIsLoaded();
 if (self.uploadMessages) { // If not loaded yet, wait for next sync
   uploadMessages(pendingMessages, onMessagesLoaded);
   ....

In most cases, the test will actually fail the first time, because the download function will return before the script's been downloaded. But in this particular application, that doesn't actually matter much - the whole synchronisation sequence is run every five seconds anyway. If the upload function isn't there yet, it should be there in five seconds, and that's fine for our purposes. If desired, we could be more aggressive - for instance, the script could be downloaded as soon as an edit begins.

XMLHttpRequest-Based On-Demand Javascript

The DOM manipulation technique is refactored here to use an XMLHttpRequest Call instead, where a server call retrieves upload.js and eval's it. As with the previous version, upload.js no longer appears in the initial HTML. Also, the invocation of ensureUploadScriptIsLoaded remains as before. The initial script - wiki.js differs only in the implementation of the Javascript retrieval. The text response is simply passed to eval.

 function ensureUploadScriptIsLoaded() {
   if (self.uploadMessages) { // Already exists
     return;
   }
   ** ajaxCaller.getPlainText("upload.js", function(jsText) { eval(jsText); }); **
 }

The actual Javascript must also change. If it simply defined a global function like before, i.e.:

 function(pendingMessages, messageResponseHandler) {
   ...
 }

the function would die when the evaluation completes. Instead, we can achieve the same effect by altering the browser window object (self).

 self.uploadMessages = function(pendingMessages, messageResponseHandler) {
   ...
 }


Implementation Examples

LABjs (Loading And Blocking Javascript)

LABjs A simple API for replacing blocking script tags in your page with calls to download the files in parallel, asynchronously, allowing the rest of the page to continue without waiting on the scripts. But, since you often have script dependencies, you can also "block" and wait for a set of scripts to load before continuing with other scripts or code execution.

jXHR (JSON-P XHR)

jXHR A simple API wrapper for JSON-P based XHR, but using the API of the native XHR object. This allows you to make cross-domain Ajax calls (JSON-P styled) in the same way as normal same-domain Ajax calls. This allows you to drop jXHR into existing Ajax code with little to no changes. jXHR also provides basic error handling on the JSON-P calls.

Related Patterns

HTML Response

The Javascript Response variant of this pattern is a companion to HTML Response and follows a similar server-centric philosophy in which it is the server-side that pushes instructions to the browser.

Predictive Fetch

Apply Predictive Fetch to On-Demand Javascript by downloading Javascript when you anticipate it will soon be required.

Browser-Side Cache

The nature of Javascript deployment means that a form of browser-side caching occurs naturally. That is, the results of calls are usually retained by virtue of the fact they automatically add new functions to the DOM.

Multi-Stage Download

On-Demand Javascript is a variant of Multi-Stage Download. The emphasis in Multi-Stage Download is on downloading semantic and display content rather than Javascripts. In addition, that pattern is more about downloading according to a pre-scheduled sequence rather than downloading on demand.


Visual Metaphor

If you watch someone learning to use a new product, their experience will usually mimic On-Demand Javascript. They'll read the bare minimum, if anything, and get started playing with it. When they get stuck, they'll consult the instructions, but only enough to get over the next hurdle.


Want to Know More?

Dynamic data using the DOM and Remote Scripting A tutorial by Thomas Brattli.