Symbolics MacIvory auctions

For anyone interested in acquiring a Symbolics system, here is an email I just received from David K. Schmidt, director of sales at Symbolics:

We are in the process of consolidating two storage facilities into one new one. To reduce the amount that needs to be moved we are liquidating some of the inventory that is not needed for future operations.

Along with auctions for 36xx boards and other spare parts we are going to sell the following 4 MacIvory systems in serial auctions one after the other:

1. MacIvory model 3 with 8 MWords in a Quadra 700 with 20 MBs and a 9GB 10000 rpm disk

2. MacIvory model 3 with 8 MWords in a Mac IIci with 20 MBs, a 4GB 7200 rpm disk and a Radius Accelerated 24-bit graphics card

3. MacIvory model 2 with 2.6 MWords in a Mac IIfx with 20 MBs, a 4GB 7200 rpm disk, a Radius Accelerated 24-bit graphics card and an ethernet card

4. MacIvory model 2 with 1.3 MWords in a Quadra 650 with 64 MBs and a 9GB 10000 disk

I will send you a link to each auction as they are put on line.

If you are interested in any 36xx part that you don't see an auction for, email me for a quote.

You can contact David at sales@symbolics-dks.com

Enforcing required slot values

I was reading a blog post by Robert Smith on ideas for future directions for Lisp type systems, where Robert mentioned the following trick to enforce provision of required slot values for structures and classes:

(defstruct foo
  (slot1 (error "Must provide value for slot1")))

That's something I haven't thought of doing before, I hope you find it useful as well. The rest of Robert's article is well worth a read.

CLiki news

cliki.net is finally up on the cliki2 software (the ALU wiki has been running it for a while).

CLiki2 features real user accounts, effective anti-spam protection, and much improved article versioning (you now get graphical diffs like Wikipedia, among other things). Change notification for the whole wiki and for each individual article is provided by ATOM feeds. There is now code coloring support. Pages can be deleted for real. The layout and ATOM feeds have been designed to be accessible, and work great in console browsers.

The biggest casualties in the move were latin1 characters (some pages were a mix of utf-8 and latin1, and latin1 lost). In particular, if your name contains a lot of accents, I aplogize.

So grab yourself an account, read up on tips for writing CLiki pages, and contribute some information about Common Lisp libraries or other topics.

If you want to write about Lisps other than Common Lisp, the ALU wiki is the place. Right now it is a little sparse and could use more contributions.

Bonus link: Check out Takeru Ohta's github account for some neat Common Lisp software: https://github.com/sile

Finding bugs in Quicklisp libraries

One of the great things about Common Lisp is the variety of implementations and the scope of deployment platforms and performance characteristic trade-offs (run-time vs compile-time vs memory size) they encompass. This is also one of the things that complicates library development. Testing on any significant combination of implementations and platforms is not practical for most Free Software library authors.

cl-test-grid is a project that provides automated, distributed testing for libraries available via Quicklisp. You download cl-test-grid on your machine, tell it about which CL implementations you have available, it runs the test suites from many libraries in the current Quicklisp release, and sends the test results to a publicly available report as well as a public bug tracker. The report details test failures by Common Lisp implementations, platform/OS, and library version.

The report is a great resource for library authors and implementation maintainers. If your library is distributed via Quicklisp and isn't tested by cl-test-grid yet, Anton Vodonosov provides some tips for making your test suite cl-test-grid friendly. If you're looking to contribute to Free Lisp software, one of the best ways is to get cl-test-grid and run the tests. See if there are any failures in the test suites, and report the bugs to the library maintainers and help find a fix.

Upcoming presentation about Parenscript

I'm going to be giving a talk about Parenscript to the Montreal Scheme/Lisp Users Group on Thursday, January 19 (meeting details here).

The slides I'm going to be using are here, and a list of links referenced in the talk is below. The last time I gave a presentation on Parenscript was to LispNYC in 2007. Parenscript has received a huge number of changes and improvements since then, and continues to be the best language/compiler to JavaScript and one of the best tools available for web application development. What's also new since 2007 are libraries and tools that extend Parenscript: Red Daly has added CLOS and the Common Lisp condition system to JavaScript as a Parenscript library, and there are now several options for interactive development with SLIME in your browser.

Links:

• Parenscript
  
• Moritz Heidkamp's survey of Lisp-in-JavaScript implementations
  
• Jason Kantz's JSGEN
  
• Peter Seibel's Lispscript
  
• Red Daly's PSOS
  
• Catherine Gaudron and Marc Feeley's JavaScriptScheme Scheme to JavaScript compiler
  
• Web frameworks: BKNR, UCW, Weblocks, teepeedee2
  
• Libraries: Suave, css-lite, clouchdb, uri-template, cl-closure-template
  
• Scheme2JS
  
• Parenscript mailing list thread on multiple value return
  
• Manuel Odendahl's original Parenscript announcement
  
• slime-proxy
  

Don't steal my REPL, or Lisp lessons from ledger

John Wiegley's ledger is a popular double-entry accounting system with a Unix command line interface.

What many people don't know is that version 3 of ledger was written in Common Lisp. This version was never made into an official release. In a FLOSS weekly podcast, Wiegley explains (31:00) that Common Lisp wasn't the best choice for ledger's users.

I emailed John to learn more about this. He replied that there were only two major problems: building cl-ledger was more difficult than building ledger, and that cl-ledger could no longer be scripted from the command line. In effect, the Common Lisp REPL had stolen the place of the Unix command line as ledger's interface.

cl-ledger was written in 2007, and there are now good solutions to these problems. ASDF works well as a build system, but before Quicklisp, dependency management for Common Lisp applications was difficult. Quicklisp solved the biggest outstanding problem in building Common Lisp applications. (PS - you can give Zach a Christmas gift for his work on Quicklisp)

Didier Verna's Command-Line Options Nuker is a widely portable Unix CLI library with many features that you can use to build command-line driven Common Lisp applications.

Announcing: CLiki2 public beta 2

CLiki, the Common Lisp wiki, is a good resource for finding out about Common Lisp libraries and other information. However, the code behind CLiki itself is hard to maintain and add features to.

Andrey Moskvitin and I have been working on a replacement wiki web application on and off for the past eight months. The first public beta came out in the summer. Since then, I've worked the software to the point where I think it's ready as a replacement to power CLiki.

The second beta of CLiki2 is now up at http://74.207.228.11/. Please try it out and let me know what you think. Bugs can be reported at https://github.com/archimag/cliki2/issues or by sending me email: vsedach@gmail.com

Most of the new features center around spam prevention:

• Wikipedia-style history lists and diffs for all pages
  
• Lists of edits by account/IP address
  
• Blacklist of accounts and IP addresses
  
• Atom change feeds for individual pages (as well as all of CLiki)
  
• HTML tag filtering

Other new features include:

• Real article deletion and undeletion
  
• Code coloring using cl-colorize
  
• Working list of uncategorized/orphan articles
  
• Pages that work well in text browsers (and hopefully screen readers)
  
• No JavaScript

Pages are marked up using a mixture of HTML (with some tags and attributes filtered out) as well as a slightly simplified form of CLiki syntax.

Behind the scenes, CLiki2 is powered by Hunchentoot, BKNR-datastore, and Nathan Froyd's diff library.

Source code is at https://github.com/vsedach/cliki2, and is licensed under the Affero GPL.

Montreal Scheme/Lisp Users Group back in action

After a long hiatus, the Montreal Scheme/Lisp Users Group (MSLUG) is back to regular meetings, with the latest one taking place November 24.

Marc Feeley gave two talks, the first presenting his experiences adapting the Gambit Scheme implementation into the Gambit REPL Scheme interpreter app for iOS (most of the difficulties seemed to revolve around the Apple app store screening process). (slides)

Then Marc showed and discussed a demo doing distributed computing using mobile, serialized continuations bouncing around Gambit instances on x86 Macs and ARM-based iPhones/iPods. The slides are well worth checking out, particularly for insight into how serialization of problematic objects like I/O streams/ports is done.

There's at least one Common Lisp project (cl-walker) that claims to be able to serialize continuations, and another library (Storable Functions) which claims to be able to serialize closures (from which you can get continuations with a CPS transformer); I haven't tried either and to my knowledge there hasn't been any works done on mobile continuations for Common Lisp, or that much work for mobile code in CL in general.

The next MSLUG meeting is tentatively January 19th; I'm scheduled to present a talk on Parenscript.

Optional dependencies

Both uri-template and css-lite provide optional support for generating JavaScript templates via Parenscript. However, I did not want to make Parenscript a required dependency for either library.

Previously, this was implemented using the #+parenscript read-time conditional in the source files.

That worked ok if you loaded Parenscript before loading css-lite, but there were two problems:

1. If you initially compiled css-lite without loading Parenscript first, you'd need to go back and re-compile css-lite by hand after loading Parenscript if you wanted the JavaScript output.
   
2. If you loaded the css-lite fasls compiled with Parenscript into a fresh Lisp image without loading Parenscript first, you'd get an error.

Both of these error stem from the fact that ASDF didn't know anything about the optional Parenscript dependency.

Didier Verna has written about optional ASDF dependencies previously (make sure to read the asdf-devel thread on optional dependencies Didier links to if you're interested in this). In short, relying on ASDF's :weakly-depends-on seems quite hairy.

I think I found a simple alternate solution for uri-template that seems to work: put all the Parenscript-dependent code into one file, and then use read-time conditionals in the uri-template.asd list of files like so:

:components ((:file "package")
             ...
             (:file "destructure-uri")
             #+parenscript (:file "parenscript-implementation")
             )

You can see the full implementation in the latest patch to uri-template.

Let me know if you have any ideas about this technique, or optional dependencies in general.

Common Lisp is the best language to learn programming

Now that Conrad Barski's Land of Lisp (see my review on Slashdot) has come out, I definitely think Common Lisp is the best language for kids (or anyone else) to start learning computer programming.

Between Land of Lisp, David Touretzky's Common Lisp: A Gentle Introduction to Symbolic Computation (really great book for people new to programming, available for free) and The Little LISPer (3rd edition, editions four and up use Scheme) you have three really great resources to get started.

Lisp's syntax is a great advantage because it is so simple to learn and has so few special cases. The interactive, iterative development style and real late-binding means you can build programs in parts and add to them as you go. The presence of real metaprogramming means you always have the ability to look at any part of your program and its state to find out what it's doing/what's wrong. The HyperSpec and Common Lisp The Language are two of the best programming language reference manuals ever written.

The best parts about Common Lisp are that it's a language that's hard to outgrow and that it makes difficult things easy. One of the chapters in Land of Lisp explains HTTP and HTML and has you build a basic web server. That chapter is only 15 pages! There's tons of ideas in the language, and because you're not restricted to a particular programming paradigm, you're always discovering better ways of doing things and developing a personal style.

Common Lisp is case sensitive

One Common Lisp feature that needs more publicity is case sensitivity. A common misconception is that Common Lisp is case insensitive, when in fact symbols in Common Lisp are case sensitive.

By default, the Common Lisp reader is case-converting: all unescaped characters in a symbol name get upper-cased. This gives the practical effect of making it seem as though symbol case doesn't matter. This is desirable behavior for interfacing with other case-insensitive languages (such as Fortran; from what I understand the main motivation for the default Common Lisp behavior), but a pain to interface with case-sensitive ones (such as C).

The behavior of the reader can be customized via readtable-case.

The one that might seem to be most useful for having case-sensitive symbols at first glance is :preserve, however remember that all code read in with the default setting (:upcase) is in upper-case, as are all the standard Common Lisp symbols (this is defined by the standard), so this means you will need to spell out all CL and external symbols IN ALL UPPERCASE. To make this less annoying, the :invert readtable-case is the most practical - all-lowercase symbol names become uppercase, all-uppercase become lowercase, and mixed-case stays mixed-case (the important part for case sensitivity). The Lisp printer outputs symbol names correctly this way by default. The only problem now becomes inconsistent spelling of a symbol in all lowercase or all uppercase in old code that expects case conversion. But otherwise you can get case sensitivity for your software by setting readtable-case to :invert today.

An easy way to manage the readtable-case is by using the named-readtables library. I've recommended named-readtables before; besides case sensitivity, it helps manage reader macros.

[This blog post is adapted from the case sensitivity CLiki FAQ entry I wrote. Feel free to make corrections and other suggestions on the CLiki page.]

Programmer myopia

We like to assume that people are basically competent and rational. Computer programmers enjoy pretending they are more competent and rational than people in other professions.

In many cases neither of those assumptions is true.

Two popular memes surrounding programming languages in the 80s and 90s were the assertions that "garbage collection is too slow" and that "dynamic typing doesn't work for large programs."

Many programmers were convinced both those things were true. In hindsight they were completely wrong, as languages such as Tcl (more on the importance of Tcl in the history of programming languages in an upcoming post), Perl, Java and Python "dragged people halfway to Lisp" and changed public perception.

How could so many people who consider themselves above-average in competence and rationality be so wrong? (Assume that every programmer is indeed a special snowflake and the Dunning-Kruger effect doesn't apply).

A hermit spent 10 years writing a program. 'My program can compute the motion of the stars on a 286-computer running MS DOS,' he proudly announced. 'Nobody owns a 286-computer or uses MS DOS anymore,' Fu-Tzu responded.

Eloquent JavaScript, Marijn Haverbeke

The problem is that programmers seem unable to think even a couple of years into the future. People complaining about garbage collection in the 80s were looking back at their existing 8-bit Trash-80s instead of at contemporary computers being produced and the future computers being planned. The idea that computers can be good at automating rote tasks like managing memory and checking and inferring types never occured to them.

People have trouble imagining the future even if the trends, such as Moore's law, are in front of them. It takes a very long time for people to understand the right ideas. Just ask Alan Kay. Being able to find the appropriate point of view really is better than a high IQ.

Here are some other Lisp concepts that programmers believe out of ignorance that will take a long time to dispel:

• tail recursion is unnecessary and makes debugging difficult
  
• macro-based metaprogramming results in unmaintainable programs
  

Lisp linkdump, March 2011 edition.

Here's a bunch of links to interesting Common Lisp stuff I've come across lately:

EOS is a drop-in replacement for the FiveAM test framework which has no dependencies on external libraries (one of the dependencies of FiveAM is heavily implementation-specific and doesn't work on LispWorks 6, among other platforms). I've converted all my FiveAM projects to EOS, and recommend you do the same.

I've ported the Multilisp benchmarks from Marc Feeley's PhD dissertation to Eager Future2. Some of them tax your Lisp implementation a lot, and might reveal bugs. On my 32-bit x86 system, the only implementation to run all benchmarks to completion without segfaulting was SBCL. Try them on yours: load :test.eager-future2 and call benchmark.eager-future2:run-benchmarks.

Speaking of Marc Feeley, here is some other cool work he has been involved in:

Vincent St-Amour and Feeley came up with the PICOBIT R4RS Scheme, which can run Scheme programs in 256 bytes RAM and 8KiB of ROM total (including VM footprint) on PIC microcontrollers.

Feeley gave a talk about Gambit Scheme at ILC2010, and had a really great set of slides which you can get at http://www.iro.umontreal.ca/~gambit/Gambit-inside-out.pdf (warning: 24MB PDF file!)

Common Lisp FORMAT continues to amaze. Peter Seibel pointed out Erik Naggum's cool hack for formatting dates on the Lisp-pro mailing list. I recently learned that instead of passing nil or t or a stream to FORMAT, you can pass a string with a fill-pointer, and "the output characters are added to the end of the string (as if by use of vector-push-extend)."

Jonathan Fischer wrote a good article about feeding binary data from Common Lisp to C.

Other things you can do with C is add s-expressions and macros - c-amplify does just that. c-amplify is motivated by the needs of game development, as was GOAL, Andy Gavin's Lisp system which powered the Jak & Daxter videogame franchise. Now there's a GPL-licensed attempt to create a GOAL-like environment called Score.

Other places besides C you can now run Lisp:

cl-gpu translates a subset of CL to CUDA GPU kernels. Arduino Lisp translates a subset of CL to a subset of C++ that can be compiled to run on the Arduino microcontrollers.

I keep promoting named-readtables (seriously, try it!), but Tobias C. Rittweiler has more cool libraries you should check out:

Hyperdoc provides a way for your Lisp library documentation have fancy Hyperspec-like lookup in SLIME.

Parse-Declarations is useful if you're building Lisp translators or code-walking tools.

There was a very interesting discussion of syntax and s-expressions on Hacker News; one of the things I learned is that according to X-bar linguistic theory, all natural languages basically consist of s-expressions. Of course the great thing about Common Lisp is, like natural languages, it is both homoiconic and its words have different meanings in different contexts. Alan Bawden came up with the "Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo" let quine of the Lisp world (another one of the cool things I learned from Doug Hoyte's Let Over Lambda).

If you've been looking for example code of large CL web applications, you should check out cl-eshop. cl-eshop currently runs the «ЦиFры» St. Petersburg store chain website and is licensed under the AGPL (Affero GPL).

SMP system wanted

Does anyone have an 8- or 16-way machine gathering dust? I need a many-processor box to do tests and tuning for Common Lisp future library implementations (I recently ported the Multilisp benchmarks from Marc Feeley's PhD dissertation to Common Lisp). Age or architecture (as long as it runs SBCL or ECL) not a concern. Will pay shipping. The box will live at Foulab, where it will be given a good home for software projects.

If you have such a system, or know someone who does, get in touch at vsedach at gmail.com

Mouse-copy for Emacs

Zmacs, MCL's Fred, and the Lispworks editor all contain a very nice time-saving feature absent from Emacs called mouse-copy. Rainer Joswig wrote a good description of mouse-copy; in brief it can be summarized as "hold down Ctrl and click on an s-exp to copy it to the current point."

I first found out about mouse copy from working with Jedi, JazzScheme's IDE, and ever since I've wanted it for my Emacs setup.

Michael Weber's redshank extensions to Paredit/SLIME include mouse-copy, but it depends on buffers being in Paredit mode.

Fortunately there's a simple way to get generic mouse-copy in Emacs. Unfortunately it doesn't do the right thing when it comes to spaces. This is easy to fix by borrowing a couple of lines from redshank. This is the mouse-copy I use right now, and it seems to work pretty well.

As you might suspect there's more time-saving things you can do with the mouse. For example, in redshank M-S-mouse-1 generates a skeleton for a make-instance when you click on a defclass definition. Looking at TI Explorer ZWEI sources, Lisp Machines had an entire modifier-key-and-mouse-button-chord convention that really made s-exp manipulation easy and direct. It would be nice to have something like that for Emacs.

If you have tips on better ways of using Emacs for editing Lisp, consider sharing them on CLiki's Emacs tips page.

January Lisp happenings

The ALU wiki is running again. Despite requiring registration and having a captcha on every edit, it seems to have a worse spam problem than CLiki. Both seem to be being spammed by hand instead of by bots. Makes me wish the current SEO/content mill bubble would burst sooner.

One thing the ALU wiki does need is more up-to-date content. In particular, if you're a Lisp consultant or freelancer, please add yourself to the ALU wiki Lisp consultants directory.

LinkedIn gave me some ad credit to try out their ad platform, and I'm planning to run ads targeted at technologists and product managers to this landing page that strongly encourages them to try Common Lisp. Any ideas for how I can make it better?

One thing you might notice when perusing Ediware (what Luís Oliveira branded Edi Weitz's excellent Free Lisp Software) is the uniformly useful documentation right on the project webpage. What you may not realize is that Edi has written some software to help you write documentation like he does. DOCUMENTATION-TEMPLATE takes a package and generates HTML to describe the package's exported symbols (you are writing docstrings, right?).

Speaking of Ediware, one of the least appreciated of Edi's libraries is CL-INTERPOL. Besides regular expressions, it's also handy for things like HTML templating.

One of the highest-impact papers I've managed to overlook has come to my attention recently: Henry Baker's Metacircular Semantics for Common Lisp Special Forms. If you're working on Common Lisp compilers/translators/simulators it's a must-read (I wish I had noticed it sooner because a lot of the techniques are applicable to Parenscript). Techniques like that are also useful if you want to fake CL-style control flow mechanisms in other languages.

Perhaps the coolest such hack I've seen is Red Daly's implementation of the Common Lisp condition system for Parenscript. It's worth reading just for the fact that the implementation code explains how the condition system works better than the Hyperspec manages to.

Bonjure, the Montréal Clojure user's group, is having its next meeting January, Friday 21 at 17:30 at CRIM.

FunctionalJobs.com recently launched. They seem to have bootstrapped their first listings from Lispjobs; hopefully they'll have more Lisp-related jobs in the future.

Free Software license onion

There are tons of Free Software licenses out there, and it can be confusing choosing one for your Lisp project. I'm writing this guide to clear up some confusion, but be aware that I'm not a lawyer and you probably should consult one if you are going to take advantage of this advice for something that you hope to commercialize.

A good strategy for choosing a license is to consider how the project would fit in a software stack - is it a general-purpose utility library? A library to interface to specific services or hardware? A web server that can be used to build applications? A content management system that you one day hope to commercialize? These projects all have different potential uses, but it would be nice if their licensing terms reflected those uses and allowed the content management system to use the web server, allowed the web server to use the service interface library, and allowed the service interface library to use utilities from the general-purpose library.

I think general-purpose utility libraries should be public domain (consider using the unlicense, which is a copyright waiver with a warranty discalimer). One thing people often want to do is copy-paste one or two functions out of them into random projects. Other good things to put into the public domain include benchmark code and test suites (you can release just those parts of your project as public domain).

Next up on the hierarchy are permissive licenses. There are dozens of these around, however I think only those that are LLGPLv2.1 compatible should be used for Free Software written in Lisp (Wikipedia has a chart summarizing Free Software license compatibilities).

There are a few of these (see the GNU license list whether a particular license is GPL-compatible or not), but generally most of them are like the BSD license. I recommend using the ISC license, which is really concise and is used by OpenBSD. Good examples of projects that can be licensed under a permissive license are parsers for data formats and interfaces to C libraries or web services. Permissive-licensed software can include public domain code and code from most other permissive-licensed software.

Projects you do not want to see forked into a closed-source product without giving back to community should be licensed under the LLGPLv2.1 (version 2.1 specifically, as LGPLv3 code cannot be used in a GPLv2 project). Why the LLGPL and not the LGPL or the GPL? The GPL makes it impossible to use a Lisp library as part of a closed-source product (even if you only use it as a library and make no modifications), and the wording of the LGPL does likewise because the "linking clause" basically presumes you're using C.

LLGPL software can incorporate other LLGPL code, public domain code, and LLGPL-compatible permissive license code, but for example LLGPL code can't be put into an ISC-licensed project (the whole project would have to be re-licensed under the LLGPL).

I think it's pretty obvious that you shouldn't license your Lisp library under the GPL if you want other people to actually use it, but how to decide between a permissive license and the LLGPL? I think that aside from practical considerations, many times it comes down to a moral choice. LLGPL forces sharing reciprocity on others. I believe copyright should be abolished or severely reformed, and until those changes start taking place, the LLGPL can be used to similar effects through the impact of works on which you own the copyright (the real pirates aren't the people downloading mp3s, they're the people violating the GPL).

I think the most interesting licensing situation is when you want to develop commercial software, but allow proprietary use and extension only by certain parties (hopefully ones which will pay you lots of money). The general strategy for this is to dual-license your code under the GPL and a commercial license. The most prominent example of a project using this strategy is MySQL.

One implication of this is that in order to make it into your official repository to be dual-licensed, any patches must have their copyright assigned to you/your company by their contributors.

Which version of the GPL to choose for dual-licensing? I think either v2 or v3 is fine (one important thing v3 includes is the anti-tivoization clause, which prevents closed hardware platforms).

One thing the GPL doesn't cover is running proprietary modifications of GPLed software as a service without any distribution going on (for example, Google is rumored to have Linux kernel patches running that haven't been released). The AGPL addresses this issue. I don't know of any software dual-licensed under the AGPL, but I think it can be a promising strategy for a variety of projects.

Character encoding is about algorithms, not datastructures

One thing you might be aware of is that both SBCL and Clozure represent characters using 4 bytes. There's been significant discussion about this already, but I hope I can offer more insight into how you can apply this to your Lisp applications.

First, the one thing most people seem to agree on is that UTF-16 is evil (it should be noted that both CLISP and CMUCL use UTF-16 as their internal character representation).

The important thing about UTF-32 vs. UTF-8 and UTF-16 is that it is not primarily a question of string size, but of algorithms.

Variable-length encodings work perfectly fine for stream algorithms. But string algorithms are written on the assumption of constant-time random access and being able to set parts of a string to certain values without consing. These assumptions can't be satisfied when variable-length encodings are used, and most string algorithms would not run with any level of acceptable performance.

What about immutable strings? Random access is still not constant-time for variable-length encodings, and all the mutator operations are gone. In effect most immutable string algorithms actually end up being stream algorithms that cons a lot.

Chances are, you're already using stream algorithms on your strings even if you're not aware of it. Any kind of search over a string really treats that string as a stream. If you're doing string concatenation, you're really treating your strings as though they were immutable - consider using with-output-to-string to cut down on consing and to simplify your code.

One thing that is special about UTF-8 is that it is the de-facto character encoding standard of the web. When you're reading UTF-8 into a Lisp string, what you're really doing is decoding and copying each character.

Most web application patterns are based around searching for and extracting some string from the HTTP request, and either using that string as a retrieval key in a database or splicing it together with some other strings to form the reply. All these actions can be modeled in terms of streams.

One of the things that makes John Fremlin's tpd2 fast is that it dispenses with the step of decoding and copying the incoming UTF-8 data into a Lisp string (this is also what antiweb does). Using some compiler macros and the cl-irregsexp library, all the searching and templating is done on byte arrays that hold the UTF-8 encoded data (Lisp string literals are converted to UTF-8 byte arrays by compiler macros). The result is a UTF-8 byte array that can be sent directly back to the web browser, bypassing another character encoding and copying step.

I read somewhere that the Exokernel operating system permitted extending this concept further down the software stack by allowing applications to send back pre-formatted TCP/IP packets, although I don't know if that's actually possible or how much of a speedup it would give.

In addition to skipping the overhead of copying and re-encoding characters, working on UTF-8 byte sequences directly means you can use algorithms that depend on working with a small alphabet set to achieve greater performance (an example of this is the Boyer–Moore–Horspool string searching algorithm).

Various Lisp news

I've put uri-template and Eager Future on github and did some work on both.

uri-template got a new release, and now uses named-readtables to provide a modular way to use reader macros. I think named-readtables is a really big deal; if your library defines reader macros, start using named-readtables today. Once it is widely adopted, named-readtables can be used to facilitate things like a global move to readtable-case :invert.

I'm working on some really interesting features for Eager Future, but I'm getting stumped by how finalizers and thread-interrupt interact in SBCL. Any help appreciated (I've posted the problem description to sbcl-help).

I've mentioned CL-JavaScript before, and it's cool to see similar projects. Justin Grant did a toy Ruby to Common Lisp compiler that (no surprise) is a lot faster than Ruby at calculating factorials. The source code is a good illustration of why Common Lisp is the ultimate language implementation language.

In local news, Montreal Clojure user's group will be hosting their first meeting October 26 (details).

In less Lisp-related news, Foulab is hosting a demo party November 27.

Input needed

Recently I've been doing a major cleanup/arson of CLiki. The cleanup effort was inspired by recurring comments on Hacker News of the form: "I didn't find Lisp libraries" or "I couldn't decide which libraries to use."

I would love for everyone to add a description of their Free Software Lisp libraries to CLiki with ASDF-installable tarballs and appropriate topic markers so libraries are easy to find and compare (maybe do it for cl-user.net first; I'm entertaining the idea of writing a scraper that would auto-generate new CLiki pages from cl-user.net entries). Currently this does not seem to be very realistic.

Instead I'm going to ask people who read Planet Lisp (where this blog is syndicated) to contribute to two specific tasks:

• Go through the list of utilities packages on CLiki (add a CLiki entry with the *(utilities) tag for any utilities packages you know that aren't in the list) and add a description of what they contain to the package CLiki page.
  
• Contribute to the Great Macro Debate CLiki page.

Right now there are almost two dozen "utility" CL packages offering everything from my-defun*$%# to map-nthcadadar (that's a joke, but only a slight exaggeration). It's quite hard to decide what to choose why. To me cl-utilities seems to be the most sane package, but it hasn't had development since 2006 (maybe it doesn't need it?). Disclaimer: I use kmrcl and Anaphora in my software.

The Great Macro Debate was a round-table at ILC 2009 that asked the unaskable: are macros evil?

To most post-Y2K Lisp programmers (like me) this seemed ridiculous. We grew up on Paul Graham's kool-aid. Macros are powerful, macros are awesome, they are special, don't use them when you can use a function, everything will be great.

But then all these experienced Lisp programmers came out at ILC and said that macros are bad for software maintenance. How can this be? In the absence of concrete examples, the cognitive dissonance was too great. The only defense mechanism was to tell yourself "bad programmers don't understand macros" and move on.

Two events changed my point of view. The first was encountering defclass* in a commercial project. The second was working with the TPD2 networking code and encountering my-defun (and you thought I was joking?). I came face-to-face with macros that made software maintenance hard.

The point of the Great Macro Debate CLiki page is to collect all relevant information as to why to write macros. That information will then get distilled into a sort of macro style writing document, which will contain examples and recommendations of how to write relevant macros, and what kinds of macros not to write, to make software maintenance easier.