Background

wshlst uses a lot of Javascript. When developing, I want to have all of my JS files listed in my index.html page. But in production mode, I want a single JS file. So I wrote a script to read in my index.html file, find all the references to my Javascript files, and write out a new HTML file that contains a reference to a single JS file that was created by concatenating and compressing all the JS files.

As an example, have a look at what my index.html page looks like in development mode and in production mode.

Ruby Script → Rake

When a server move prompted me to upgrade to Subversion and Capistrano, I decided to turn my Ruby script into a Rake task that I could trigger from my Capistrano deploy: minify.rake

Capistrano

I added a line to my Capistrano deploy.rb file to run rake minify on the server after updating from Subversion:

run "cd #{release_path} && rake minify RAILS_ENV=#{rails_env}"

Apache

I also added a few lines to my httpd.conf file to support this. First, since the Rake task combines all the JS files into a single JS file but Capistrano deploys the entire app, I wanted to restrict access to everything in the javascripts directory:

# Don't allow access to the javascripts directory (all JS should be in minified js file)
RewriteRule ^/javascripts/ - [F]

Then, because the Rake task reads index.html and outputs a file called index-mini.html, I wanted to send all requests for / and /index.html to /index-mini.html:

# Rewrite index.html to index-mini.html
RewriteRule ^/index.html$ /index-mini.html [QSA]

# Rewrite index to check for static
RewriteRule ^/$ /index-mini.html [QSA]

Why Not Use AssetPackager?

For a normal Rails application, AssetPackager is probably the right solution to this problem. However, wshlst doesn’t use Rails to generate the UI (it’s all done in Javascript), so AssetPackager wouldn’t work.

Sending mouse events such as click and mouseover in JsUnit tests can be really hard.

More Bad News

Prototype doesn’t make it any easier. Sam Stephenson says:

We would very much like to support it in the future. It’s fairly complicated to implement native event firing across all supported browsers, so in 1.6.0, fire works with custom events only.

YUI To The Rescue

YAHOO.util.UserActions can simulate some user actions. Unfortunately, calls to YUI can look a bit clunky in a Prototype-heavy codebase:

var element = new Element("div").insert("Hi");
var offset = element.cumulativeOffset();
YAHOO.util.UserAction.click(element, { shiftKey: true });

YUI + Prototype FTW

A little mixin magic:

Element.addMethods({
  simulateClick: YAHOO.util.UserAction.click.bind(YAHOO.util.UserAction),
  simulateDblClick: YAHOO.util.UserAction.dblclick.bind(YAHOO.util.UserAction),
  simulateMousedown: YAHOO.util.UserAction.mousedown.bind(YAHOO.util.UserAction),
  simulateMouseup: YAHOO.util.UserAction.mouseup.bind(YAHOO.util.UserAction),
  simulateMouseover: YAHOO.util.UserAction.mouseover.bind(YAHOO.util.UserAction),
  simulateMouseout: YAHOO.util.UserAction.mouseout.bind(YAHOO.util.UserAction),
  simulateMousemove: YAHOO.util.UserAction.mousemove.bind(YAHOO.util.UserAction)
});

and now our test code looks nicer:

var element = new Element("div").insert("Hi");
var offset = element.cumulativeOffset();
myElement.simulateClick({ shiftKey: true });

I contemplated having a per-user time zone setting, but that seemed complicated. So I decided to just show how old the item is. For that, I needed to write two things. I’m posting them here in case anyone else finds them useful.

RelativeDate.js

First, I needed a simple relative date formatter, so I wrote a simple one. Given two dates, it returns a string like

4 days ago

or

26 hours ago

Here’s the code and the JsUnit test:

RelativeDate.js
RelativeDateTest.html

ServerTime.js

The timestamps of the items I’m displaying are stored in the database, so they are based on the clock of my database server. The relative dates are calculated in Javascript, so they are based on the clock of the user, which might be hours or just minutes off from the database server time.

The first thing my app does when it loads is checks to see if the user is logged in, so I decided to piggyback the current server time along with the response. Since the client and server talk using JSON, it’s simple to add another property to the response:

def current
  ...
  render_json({
    :success => true,
    :user => user,
    :servertime => dbtime()
  }.to_json)
end

...

def dbtime
  ActiveRecord::Base
    .connection
    .select_all("select unix_timestamp(now()) as now")[0]["now"]
    .to_i
end

I wrote a ServerTime class on the client side and I initialize it when the app receives the current user response from the server:

this.serverTime = new ServerTime(json.servertime);

Here’s the code (it’s very simple):

ServerTime.js

Putting It Together…

And now to display the relative time, my app only has to do this:

var updated = new Date(item.updated_at * 1000);
element.insert(new RelativeDate(updated, app.serverTime.get()).toString());

It’s been a lot of fun to write, and one reason is that the entire UI is written in Javascript. No HTML, no CSS. There’s no HTML generation on the server-side either; it’s all client-side Javascript. I’d really like to write more apps like this.

The Server

There is of course a server-side component. I decided to write the server in Rails because I had just finished working on a Rails project at Pivotal Labs and had Rails on the brain. Plus, my web host, DreamHost, supports Rails and doesn’t support Java, which is the language I’ve written all my other web apps in.

DreamHost’s support of Rails is a bit limited though: they kill any long-running processes, which means that a response from your Rails app could take 10 seconds if no other requests have been made lately. It’s good enough for a little project like Wshlst with only a few users and for initial development, but not much beyond that. (If I were to do it over again, I’d probably just use Ruby CGI, or maybe even PHP. Or use a VPS from Rimu, but that costs money.)

Because all of the UI is in Javascript, the Rails side of things doesn’t have to generate any HTML. Instead, it takes standard Rails requests and returns JSON using JSON for Ruby. For example, the client sends the following AJAX request:

GET http://wshlst.com/user/list

and the server uses ActiveRecord magic to find all the users that are “friends” with the current user and uses JSON for Ruby to convert that list to JSON, which it returns back to the client like this:

[{name: "Andy Acorn", id: 12, email: "andy@example.com"},
{name: "Betty Beets", id: 13, email: "betty@example.com"},
{name: "Chad Carrot", id: 14, email: "chad@example.com"},
{name: "Donna Donut", id: 15, email: "donna@example.com"}]

which Prototype automatically converts to Javascript objects.

The server is of course also responsible for sending static files, but those are all handled by DreamHost’s Apache servers.

The Client

Here’s where it gets interesting. The app consists of an HTML file that does little more than include the Javascript files. (In development mode, there are multiple Javascript files, but when I deploy to production, there’s just one file. Read this blog post on concatenating and compressing Javascript files with Ruby.)

The app relies heavily on Prototype, and also uses Animator.js for a few visual effects. Everything else I wrote from scratch. I like to write a lot of code from scratch when I’m working on a personal project so I can get a good understanding of what’s really going on and be able to better evaluate pre-written libraries when I’m programming for money.

The app’s main class is called Application and acts as a controller. There are many view classes which are combined to form the app’s UI. For example, the “Change Password” box is implemented by a ChangePasswordForm class and consists of a ModalBox with a FormPanel inside. The FormPanel contains instances of Buttons and Fields. Writing a Field class of course takes more time than putting <input type=”text”> in some template file somewhere, but it allows for easy reuse and the ability to add a feature to all fields in just one place. (I hope to post some of the code from Wshlst on this blog in the future.)

Responding to user actions is just a matter of one Javascript method directly calling another method. No parsing of parameters, no layers of actions and filters, no guessing client state on the server side. It’s very liberating.

Lessons Learned

Test early and often in lots of browsers. This is obviously important when doing any web development, but when it’s 100% Javascript, there’s even more chance of incompatibilities. My JsUnit tests caught some issues (Date.now turns out to be Firefox-only), but most issues I had were visual.

MVC is good. Again, I already knew this, but I didn’t do nearly a good enough job with it in this app, probably because it was the first time I wrote an entire UI in Javascript. Next time, I’ll have multiple controllers and keep app logic out of the views as much as possible.

Events are probably the way to go. Wshlst suffers from some tight coupling of objects because they directly notify each other of actions. Next time, I’ll try having all my objects send custom events to and receive custom events from some global event source (perhaps document.body), like I did back in the days when I wrote Mac software.

UI is hard. I’m glad I did all the UI myself, but next time I’d like to build on the work of others. I’m still looking for a good Javascript widget library. Some of the widget libraries I’ve seen are trying to make web apps look application-y, and I’d prefer my web app to look web-y. Also, some libraries don’t have great support for calling everything from Javascript. I think I’ll look into Dojo again now that version 1.0 has been released, and more importantly, now that there’s more documentation.

Try It Out

There’s no way yet to sign up for Wshlst (if there’s enough interest I’ll do it), but the front page of the site lists some demo users you can use to play around: www.wshlst.com.

Read more articles about wshlst.com

It differs from other wish list sites in a few ways: it’s group-oriented, it allows people to add items to other peoples’ wish lists (secretly, if desired), and it allows people to discuss wish list items (again, secretly, if desired). The result is a site that lets a group of people plot and scheme behind each others’ backs to make gift buying easier.

Site Tour

The site’s main page is divided into three panels. The first is called “People” and lists all the people you share wish lists with. For me, it lists everyone in my family and everyone in my wife’s family.

Clicking on a person loads their wish list in the next panel, the “Wish List” panel. As you’d expect, you can add and remove items from your wish list. But you can also add and remove items from other peoples’ wish lists. By default, when you add an item to someone else’s list, it’s not visible to that person. Items can also be marked as “Claimed” so two people don’t end up buying the same gift.

Clicking on a wish list item loads comments about the item in the third panel, called “Discussion”. Comments in the discussion panel are also hidden from the item’s intended recipient by default.

Try It Out

There’s no way to sign up to use the site (yet—if there’s interest from anyone outside my family, I might take the time to add the ability), but there are demo users you can use to try the site out. Details are on the front page of the site: www.wshlst.com.

Read more posts about wshlst.com

The standard solution is to concatenate all the Javascript files into one big file and then compress it. I wanted it to be automated of course, and I wanted to do it at deploy-time, not at run-time.

So I wrote a little script that figures out what Javascript files are being included in the HTML page, compresses them with the YUI Compressor, concatenates them into a file named after the newly-compressed and -concatenated Javascript’s MD5 hash, and creates a copy of the original HTML page that references the new Javascript file instead of the old ones.

It’s pretty specific to my app’s needs, but with some minor tweaking, it should be useable on other projects. Check it out: minify.rb

So in IE, JsUnit has to resort to searching the text of all scripts for strings that look like test function names. Unfortunately, it doesn’t understand comments so commented-out test functions will end up running anyway.

Recently, Dean McNamee and Dan Pupius alerted Edward Hieatt and me to an undocumented Internet Explorer Javascript function called RuntimeObject and suggested JsUnit might be able to use it to discover test functions.

I replaced the text searching with RuntimeObject and tagged it as version 2.2 alpha 25. (I also added an assertEqualsIgnoringOrder function while I was making changes.)

Browse the new version at SourceForge or check it out like this:

Try it out and let us know if there are any problems.

So I wrote a simple AppleScript to move old files to an “Old” subdirectory. I thought I’d post it here in case anyone else needed something similar. It’s a simple script, but AppleScript is hard for me because I write scripts so rarely that I forget everything about it before I need to write another script.

(Many years ago, I actually got paid to write a few AppleScripts.)

set folderName to "Macintosh HD:Users:your-user-name:Temp:”

repeat with fileName in list folder folderName
  set tempFile to (folderName as string) & contents of fileName
  set modificationDate to modification date of (info for alias tempFile)
  set age to ((current date) - modificationDate) / days

  if age > 7 and fileName is not ” Clean Up Temp Dir” then
    tell application “Finder”
      move alias tempFile to alias (folderName & ” Old”)
    end tell
  end if
end repeat

Show Log

There’s now a “Show Log” button that opens a window with a log of what JsUnit has been up to.

New log button (click image for larger view)

New log window (click image for larger view)

New URL Parameters

There are three new URL parameters:

• suppressDialogs=true will suppress JsUnit’s dialogs in interactive mode
• setupPageTimeout=n will change the timeout for the setupPage call
• pageLoadTimeout=n will change the page load timeout

Easier Suite Creation

The jsUnitTestSuite constructor now accepts test pages and subsuites as parameters. So instead of this:

function suite() {
  var suite = new jsUnitTestSuite();
  suite.addTestPage("/foo/barTest.html");
  suite.addTestPage("/foo/bazTest.html");
  return suite;
}

you can do this:

function suite() {
  return new jsUnitTestSuite("/foo/barTest.html", "/foo/bazTest.html");
}

Also, I wanted to be able to view any test page so I could debug the tests with Firebug (as explained here).

I also didn’t want to run a web server all the time, so I wanted a script that would create a static HTML file.

Here’s what I came up with. It’s not configurable, so you’ll have to modify it for your own needs.

JsUnitSuiteBuilder.rb

$("foo").stylize({
  color: red,
  background-color: green,
  text-decoration: underline
});

Implementing stylize is easy with Prototype’s Element.addMethods() method:

var ElementExtensions = {
  stylize: function(element, styles) {
    for (var s in styles) {
      element.style[s] = styles[s];
    }
    return element;
  }
}
Element.addMethods(ElementExtensions);

Update: Prototype 1.6 added a similar setStyle method which works similarly but also allows CSS string-type style declaration (e.g., myElement.setStyle(”color: blue; border: 3px”)). Still, the above is a nice example of Element.addMethods().

Easier said than done

Here’s the error I got:

It was looking for header files in /usr/lib/ruby/1.8/powerpc-darwin8.0, but they weren’t there; they were in /usr/lib/ruby/1.8/universal-darwin8.0.

The Solution

The solution was simple enough: create symlinks to the header files:

$sudo ln -s ../universal-darwin8.0/*.h ./
$sudo gem install termios

There’s even a Monstatic Widget for Yahoo Widgets.

I only wish it had Jabber notification in addition to email notification.

I also read about dwatch but I figured a service would be easier.

Update: I got an email from Arun who works for Site24×7.com. It seems to have more features than Montastic, such as being able to look for a particular keyword in the response (or not in the response), and nice looking reports. I’ll be trying them out alongside Montastic. (Unrelatedly, the makers of Site24×7 also make Zoho, a suite of productivty apps like a word processor, spreadsheet, etc. Somehow I’d never heard of them.)

Update 2: My pals at Pivotal Labs have released Monitwitter, which uses Twitter to show you the status of your servers. Brilliant!

FireBug has a nice debugger, but I couldn’t get it to work with the JsUnit test runner. I could set breakpoints in JsUnit itself, but any breakpoints in my test would be ignored.

And then it hit me: ditch the test runner. I opened my test page directly in Firefox, opened FireBug, clicked the Debugger tab, and chose my test from the “Scripts” menu. Here’s a simplified version:

function testUpdate() {
  var listPanel = new ListPanel();
  listPanel.update(["one", "two", "three"]);

  var children = $(”content”).childNodes;
  assertEquals(”one”, children[0]);
  assertEquals(”two”, children[1]);
  assertEquals(”three”, children[2]);
}

I put a breakpoint at the first assertEquals and switched to the Console tab and typed testUpdate(). I saw the ListPanel getting drawn in the browser (something you don’t see in the JsUnit test runner) and in the Debugger tab I could inspect all my variables.

But even better, I could go back to the Console tab and start evaluating Javascript.

>>> children.length
3

Good.

>>> children[0]
div

Bad.

>>> children[0].firstChild
[Text] “one”

Better.

>>> children[0].firstChild.nodeValue
“one”

Perfect.

PS1='\\h:\\w \\u\\$ '

The code for starting a font and color change is \\e[. The code for bold red is 1;31. The code for closing the starting code for a font and color tag is m. The closing code for a font and color change is \\e[m. Bash thinks that these color commands will increase the length of the prompt, which will cause odd line wrapping behavior. The solution is to wrap all the color codes in brackets (\\[ and \\]).

So, the entire prompt looks like this:

PS1='\\[\\e[1;31m\\]\\h:\\w \\u\\$\\[\\e[m\\] ‘

Eeech. Add that to your .bashrc file and you’re all set. If you don’t want the red, the prompt would be:

PS1='\\[\\e[1m\\]\\h:\\w \\u\\$\\[\\e[m\\] ‘

More colors are listed at the bottom of this page.

Creating the join table

Starting with tables foos and bars, I created a join table bars_foos. (Rails expects the join table’s name to be a concatenation of the two table names in alphabetical order).

Reading attributes

I added an attribute baz to bars_foos. I had originally tried to access it with:

myFoo.baz

but that didn’t work. It turns out that I needed to force Rails to do the join before accessing the attribute, which makes sense when you think about it (but it took me a while before my little brain thought about it correctly):

myFoo.bars.baz

Adding attributes

Use the well-documented push_with_attributes method:

myFoo.bars.push_with_attributes(bar, :baz => "fez")

Updating attributes

Apparently you can’t. Luckily, in my app the attribute is immutable. Other people have suggested deleting and re-inserting the row which I imagine shouldn’t be a problem for most applications.

Recently, I was trying to pass some info from my Javascript into my XSL stylesheet. Here’s how I finally did it:

(Quick note: in real life, I don’t name variables "myThis" and "myThat"; I’m just doing it here to make it clear what is my code and what is part of XSL, AJAXSLT, Javascript, etc.)

The XSL

<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:param name="myParam" select="'myDefault'"/>
  <xsl:template match="/">
    myParam: <xsl:value-of select="$myParam"/>
  </xsl:template>
</xsl:stylesheet>

The Javascript

var myContext = new ExprContext(myXml);
myContext.setVariable("myParam",
    new StringValue("the value for myParam"));
xsltProcessContext(myContext, myXsl, myRootElement);

The Trick

So the big trick, which I figured out after reading lots of the AJAXSLT code, was to set the parameter to a StringValue, rather than a regular Javascript String. StringValue is part of AJAXSLT. Update: Will, in the comments below, mentions that there are four total types: StringValue, NumberValue, BooleanValue and NodeSetValue.

Troubleshooting

Don’t forget to declare your parameter with <xsl:param.../> before using it with <xsl:value-of select="$..."/>, otherwise Ajaxslt will blow up in the middle of rendering.

• It uses WebKit so it works just like Safari
• It doesn’t do any caching
• It takes a URL command-line parameter and loads it
• It quits immediately [update: I added a noquit paramter in case you don't want it to quit immediately]

It currently has the following problems:

• It’s uses AppKit, so it has a UI. I really wanted it to be a command-line-only app, but I haven’t yet figured out how to do that
• Invoking it is painful: /Applications/wkget.app/Contents/MacOS/wkget -url http://www.apple.com
• It only tests WebKit; one day I might make a version that uses Firefox’s JavaScript engine

I called it wkget and you can download it here. [Update: I had an incorrectly-built version up here for a while. It's fixed now.]

Here’s the code (sorry for the formatting; I had to wrap everything for this very narrow column):

@implementation WkgetController
- (void)awakeFromNib {
  [self gotoUrl:self];
  [webView setFrameLoadDelegate:self];
}

- (IBAction)gotoUrl:(id)sender {
  NSString *url = [[NSUserDefaults standardUserDefaults]
      stringForKey:@”url”];

  if (url == nil) {
    url = @”http://www.apple.com/”;
  }

  NSURLRequest *request = [NSURLRequest
      requestWithURL:[NSURL URLWithString:url]
      cachePolicy:NSURLRequestReloadIgnoringCacheData
      timeoutInterval:1.0];

  [[webView mainFrame] loadRequest:request];
}

- (void)webView:(WebView *)sender
  didFinishLoadForFrame:(WebFrame *)frame {
  if (![[NSUserDefaults standardUserDefaults]
      boolForKey:@”noquit”]) {
    [NSApp terminate:self];
  }
}
@end

It took a bit of playing to get it to work. Here’s some sample code. (I forgot to write it up when I did it, so I might have left something out.)

Setup

First, include the scripts. Note: make sure to have a closing </script> tag rather than doing <script … /> because that silently fails on some browsers (yay!).

<!-- AJAXSLT -->
<script src="/ajaxslt/misc.js"
    type="text/javascript"></script>

<script src="/ajaxslt/dom.js"
    type="text/javascript"></script>

<script src="/ajaxslt/xpath.js"
    type="text/javascript"></script>

<script src="/ajaxslt/xslt.js"
    type="text/javascript"></script>

<script src="/ajaxslt/xpathdebug.js"
    type="text/javascript"></script>

Using Ajaxslt for XSL Transformation

For XSL transformation, you’ll need three things: some XML, some XSL, and a node (or DocumentFragment presumably) to place the result in. One way to get the XML and XSL is to load them through an XmlHttpRequest. Ajaxslt also provides a couple of handy functions to convert between text and XML:

var myText = "...";
var myXml = xmlParse(myText);
var myTextAgain = xmlText(myXml);

Once you’ve got the XML and XSL, you’re ready to do the transformation:

var myNode = document.getElementById("myId");
xsltProcessContext(new ExprContext(myXml), myXsl, myNode);

Your node should now contain the result of the transformation.

Using Ajaxslt for XPath

You can also use Ajaxslt for XPath evaluation, which is really handy when writing tests on your rendered document:

var expr = xpathParse(myXpathString);
var result = expr.evaluate(new ExprContext(myXmlDocument));
var resultAsString = result.stringValue();

Troublshooting

The order of the JavaScript include statements matters: if Ajaxslt is giving strange errors, make sure the scripts are included in the exact same order as in the Setup section of this page.

Make sure you know what you’re passing to Ajaxslt: if you’re sending request.responseXML to a method, don’t look at request.responseText to see what you’re passing; use xmlText(request.responseXML).

The root element isn’t selectable: if your XML is <foo id="myId">…</foo>, then you don’t select foo’s id with /foo/@id but rather with just @id.

Case matters in xpath: If you’re evaluating an xpath expression on an HTML DOM, note that the node names are most likely uppercase. So do /HTML/BODY instead of /html/body

One big problem is that it can’t run all your tests. So we wrote a test suite class that runs all tests and can be run from Test::Unit:

require 'test/unit'

Dir.chdir("test")
@@test_files = Dir.glob("**/*_test.rb")
@@test_files.each {|x| require x} 

class TestSuite
  def self.suite
    suite = Test::Unit::TestSuite.new

    @@test_files.each do |filename|
      File.open(filename) do |file|
        file.each_line do |line|
          if match = (/class\s+(\w+Test)/).match(line)
           suite << eval(match[1] + ".suite")
          end
        end
      end
    end

    return suite

  end
end

Here’s what ended up working for me. It’s based on the instructions on Jamie’s Blogomatic.

1. Install the XCode tools from the Tiger DVD.

2. Download and run the Ruby on Rails installer from: http://users.tpg.com.au/aarnold/Ruby%20on%20Rails.zip.

3. Download and run the MySQL installer (I used version 4.1.11): http://dev.mysql.com/downloads/index.html.

4. Start mysql: bash$ sudo /usr/local/mysql/bin/mysqld_safe

5. Put mysql into the background: ctrl-z and then bash$ bg

6. Update gems: bash$ sudo gem update

7. Install the mysql gem from the remote source: bash$ sudo gem install -r mysql — –with-mysql-dir=/usr/local/mysql

8. Compile and install the MySQL API module from here: http://www.tmtm.org/en/mysql/ruby/.

9. All done!