Templates and code generation

Earlier this month Google released Closure Templates, a new templating library intended for generating HTML. Who cares? I personally dislike HTML templating, and avoid it whenever possible in lieu of s-expression based generation tools like CL-WHO. At first look, Closure Templates didn't seem to be anything new or useful.

The one thing that makes Closure Templates somewhat interesting is that the library works in both Java and JavaScript, so you get client and server-side templates in one place. I did a similar thing with uri-template by having the URI template expand into code that executed in both Common Lisp and JavaScript via Parenscript (here I have to state that despite being an author of a templating library, I still dislike templating libraries and even find my own creation annoying at times).

About a week after Closure Templates was released, Andrey Moskvitin (archimag) wrote an implementation in Common Lisp (interesting note: it took him only five days and 1/15 the number of lines of code as the original). The resulting system, cl-closure-template, similarly generates JavaScript via Parenscript.

I still didn't understand the motivation. Yesterday, Andrey was kind enough to explain it. Those pampered by web development in Common Lisp using s-expression HTML generation tools simply don't encounter the problem of trying to fit HTML templating onto the paradigm of incremental page updating via AJAX. So if you want to build an AJAX web application and use HTML templating, give cl-closure-template a try.

[Blog metanote: if you're reading this blog through its feed, you will shortly see this post show up in Russian. a CONS is an object which cares is now syndicated on Russian Lisp Planet, and I'm going to be writing Lisp-related posts in Russian as well as English. All such posts will be tagged 'lisp-ru'. Filter out that tag if you don't want the Russian version of the posts to show up in your feed reader.]

New Parenscript release.

A new version of Parenscript has just been released (see the official release announcement for some more details). This release fixes a number of issues (among them poor compilation speed due to terrible choice of implementation on the part of yours truly, for which I have recently been called a lot of bad names in the Russian Lisp blog world).

Parenscript birthday present

A while ago I mentioned I was going to announce some exciting news about web development tools. Shortly thereafter my priorities changed, and I shelved the project with the intention of picking it up again in the future. Then I remembered about LispNYC's participation in the Google Summer of Code program - if I can't do the project now, why not try to get funding from Google to have a student work on it?

Here is what all this is about:

The PSDE project aims to develop an Emacs-based JavaScript programming and debugging tool for AJAX-based RIAs. The implementation will provide an unobtrusive client-side script that can be loaded alongside any JavaScript web application, and an Emacs interface (REPL) that will enable debugging of remote and mobile clients, including the possibility of no-reload drop-in for live user sessions.

Along with an interactive prompt/REPL, PSDE will provide a profiler, tracing and logging facilities, a DOM inspector, a completion and documentation system for DOM symbols, and other useful web programming tools, both by using the reflective capabilities of JavaScript, and by providing hooks (a la Open Implementation) into the Parenscript compiler to provide metaprogram information that cannot be gleamed using runtime techniques or analysis of JavaScript code.

In addition to providing a valuable tool to the web development community, the OI extensions to the Parenscript compiler included as part of the PSDE project will be useful in their own right to those wishing to create other web development tools based on Parenscript, or web developers using Parenscript and needing advanced JavaScript generation capabilities.

The project has its roots in the "browser Comet REPL" demo I gave during my talk about Parenscript at LispNYC in September 2007, but the idea of making a "browser SLIME" didn't occur to me until this past November, when Daniel Gackle told me: "Why don't you make a browser SLIME?"

If you are a student that might be interested in this project and qualify to participate in Google SoC, or know of such an individual, get in touch: vsedach@gmail.com. I'll be happy to answer any questions and help you with preparing an application. Google will start accepting student applications March 23rd, with the cut-off date being April 3rd.

UPDATE: LispNYC has not been selected as a mentoring organization for SOC 2009. I would like to fund this project myself, but currently cannot afford it.

Parenscript's 4th birthday

Today is a day of many celebrations. As most of you are aware March 14 is π Day. It also happens to be Einstein's birthday.

Four years ago, Manuel Odendahl announced the release of Parenscript to the public. Reflecting on the original post in the context of what Parenscript is today, the power and appropriateness of Manuel's design have more than proven themselves, but also I see the amount of exciting work that is still ahead.

GWT and deferred binding

I was going through last year's Google I/O conference videos and noticed a few GWT-related ones. One in particular seemed interesting: Faster-than-Possible Code: Deferred Binding with GWT by Bruce Johnson.

The technique that GWT dubs "deferred binding" consists of generating feature-specific output code (JavaScript) from a common codebase (Java for GWT, Lisp for Parenscript) and serving it up under different URLs so the resource are cached properly.

I first encountered the idea in the spring of 2007 when Daniel Gackle proposed it as a way of efficiently handling the generation of feature-specific JavaScript code from Parenscript, and we implemented it in a few days.

The system would generate several pages from a common definition written in CL-WHO/Parenscript (we used it to generate browser-specific code and profiler-instrumented code, so in all we'd get a cartesian product of (ie, ff) x (regular, profiled) as output), and serve them up under different URLs that also incorporated a version number (via a mechanism that linked source files to generated resources and tracked the file modification date). This approach generated perfectly cacheable HTTP resources and ensured trouble-free application upgrades.

Something interesting that I noticed when developing the feature-specific code generation mechanism was its resemblance to context-oriented programming: each feature acts as a layer which affects the way that the compiler produces code.

Besides browser-specific code generation, in GWT the technique is used for generating locale-specific resources, and by virtue of its implementation, to provide an extremely obtuse way to emulate Java's broken metaprogramming facilities (see http://www.zenika.com/blog/wp-content/uploads/2007/08/tutorial-binding-en.pdf, for example).

In the intervening time I have thankfully learned a little more about programming for the web, and have come to the inevitable conclusion (if I had started reading comp.lang.javascript sooner, I could have avoided making this mistake - if you're not reading that group yet, start now) that using the approach to generate browser-specific code is fundamentally flawed and will almost certainly ensure that your code will break on browsers that you have not developed for. Working to add support for new browsers to this scheme will not only waste an incredible amount of time that you would not have had to spend at all otherwise, but will actually make your code more brittle and harder to maintain.

The only viable approach to dealing with differences in browser capabilities is feature detection. I can only say that I am glad that Parenscript is not pushing a "framework" on anybody, so my ignorance only impacted one project. Generating feature-specific code is not in itself a bad idea - GWT uses it to also generate locale-specific resources, with the corresponding bandwidth savings and cacheability advantages. I can only hope for the sake of their users that the GWT developers repent and change their stance on generating browser-specific code.

President's day Parenscript release, and a little on how I deploy web apps

It is President's day in the US of A, and that means a new release of Parenscript. Be aware that this one may break your code.

In other news, a couple of weeks ago I decided to sign up at stackoverflow, a community Q&A site for programmers. It has member-driven moderation, good search and tagging facilities, but a much wider scope and lack of (for lack of a better term) "narrative" than Usenet or mailing lists or message boards. The first means that spam, trolling and off-topic messages are kept under control, but the third means you can't participate in the site like you do Usenet, which will hopefully be offset by the second, which means that you can use the site to find answers more effectively than searching Usenet archives and leave your contribution to answering questions that you have some knowledge of.

So far I've answered one question about deploying Lisp web apps. I'm thinking of expanding it in more depth and adding examples and turning it into a blog post ("article" in the old media parlance). Which naturally leads me to remark on my own online community participation: I've stopped reading Usenet and participating in community sites a few years ago, and now follow blogs exclusively. Narrative, personalization, and Internet etiquette.

Web browser fun

The diagram below summarizes my recent cursory survey of web browsers. I think the one important conclusion that can be drawn is this: Webkit will be critical in the future. I know at least one person who doesn't have a computer, but uses Facebook from her mobile phone. There will be more and more people like that. Another important factoid: IE6 isn't dead. IE Mobile 6 will use the JavaScript implementation of IE8, but the layout engine is based off of IE6. That will probably induce nausea in most web developers, but it makes me glad I still take care to develop my web apps to run in IE6.

Other fun things:

The repository version of Parenscript will probably break your code, because your code probably deserves to get broken.

I've released a new version of uri-template, which fixes a bug in how URI-encoding was being performed.

Parenscript tricks and parallelism

The new version of Parenscript features a user-definable obfuscation facility. Among other amusing things, it can be used (due to JavaScript's under-utilized support for Unicode identifiers) to make your code Asian:

(ps:obfuscate-package "LAMBDACHART"
  (let ((code-pt-counter #x8CF0)
        (symbol-map (make-hash-table)))
    (lambda (symbol)
      (or (gethash symbol symbol-map)
          (setf (gethash symbol symbol-map) (make-symbol (string (code-char (incf code-pt-counter)))))))))

LAMBDACHART> (ps (defun foo (bar baz) (+ bar baz)))
"function 賱(賲, 賳) {
    賲 + 賳;
};"

Unrelated, I recently found Marijn Haverbeke's PCall library for parallelism in Common Lisp. The library provides futures (called 'tasks') as parallelizing mechanism, and thread pool (the library is based on bordeaux-threads) management facilities to tweak how the futures are actually executed.

Unlike MultiLisp, which implemented the same futures-based parallel model, there is no macro provided to evaluate a function's arguments in parallel before applying the function to them. That seemed to be a popular facility in the parallel research Lisp systems of the 80s, probably because it is a no-brainer once you consider the Church-Rosser theorem, however upon some reflection and a little coding that construct proves to be not very convenient.

I think the futures approach to parallelism is the most widely useful model available today. It shares all of the conceptual benefits of its cousin delayed/lazy evaluation: futures are declared and used explicitly in the code, without forcing (pun fully intended) any contortions in the control flow of the code using those futures. If you can write a function, then you can define a task that can be executed in parallel.

The model doesn't handle concurrency control beyond the synchronization provided by joining/forcing the future, so if your tasks share state (although you should be writing your code to do the synchronization in the code making and consuming the tasks, so that they don't share state), you'll need to do the synchronization yourself (this is where you take advantage of locks provided in bordeaux-threads).

One interesting thing about the library is Haverbeke's extreme pessimism about native thread overhead (the default thread pool size is 3). On many systems that is certainly justified, but apparently some half-decent OS implementations exist. I'm interested in doing some benchmarks with SBCL using NPTL threads on an AMD64 box to see what kinds of numbers are reasonable.

New Parenscript release!

Details on the project page: http://common-lisp.net/project/parenscript/

This one has been a while in the making, but in the meantime many subtle but important issues have been thought out and addressed. I feel that the project now has a much clearer direction for future development, particularly as a basis for advanced web development tools that are going to be a step beyond anything else out there (announcement coming soon).

Lisping in the NYC

I gave a presentation about Parenscript to CLUDG, and then two weeks later I also gave it to LispNYC. An audio recording from the NY talk and a tarball of the presentation code can now be found here.

New ParenScript release.

I just finished putting up the tarball of the new ParenScript release, which can be download here. Red Daly has been working on a new package/namespace mechanism for ParenScript, which will entail some major changes to the codebase, and this will be the last release before his work is integrated into the main ParenScript source tree.