Input needed
Recently I've been doing a major cleanup/arson of CLiki. The cleanup effort was inspired by recurring comments on Hacker News of the form: "I didn't find Lisp libraries" or "I couldn't decide which libraries to use."
I would love for everyone to add a description of their Free Software Lisp libraries to CLiki with ASDF-installable tarballs and appropriate topic markers so libraries are easy to find and compare (maybe do it for cl-user.net first; I'm entertaining the idea of writing a scraper that would auto-generate new CLiki pages from cl-user.net entries). Currently this does not seem to be very realistic.
Instead I'm going to ask people who read Planet Lisp (where this blog is syndicated) to contribute to two specific tasks:
- Go through the list of utilities packages on CLiki (add a CLiki entry with the *(utilities) tag for any utilities packages you know that aren't in the list) and add a description of what they contain to the package CLiki page.
- Contribute to the Great Macro Debate CLiki page.
Right now there are almost two dozen "utility" CL packages offering everything from
my-defun*$%# to map-nthcadadar (that's a joke, but only a slight exaggeration). It's quite hard to decide what to choose why. To me cl-utilities seems to be the most sane package, but it hasn't had development since 2006 (maybe it doesn't need it?). Disclaimer: I use kmrcl and Anaphora in my software.The Great Macro Debate was a round-table at ILC 2009 that asked the unaskable: are macros evil?
To most post-Y2K Lisp programmers (like me) this seemed ridiculous. We grew up on Paul Graham's kool-aid. Macros are powerful, macros are awesome, they are special, don't use them when you can use a function, everything will be great.
But then all these experienced Lisp programmers came out at ILC and said that macros are bad for software maintenance. How can this be? In the absence of concrete examples, the cognitive dissonance was too great. The only defense mechanism was to tell yourself "bad programmers don't understand macros" and move on.
Two events changed my point of view. The first was encountering defclass* in a commercial project. The second was working with the TPD2 networking code and encountering
my-defun (and you thought I was joking?). I came face-to-face with macros that made software maintenance hard.The point of the Great Macro Debate CLiki page is to collect all relevant information as to why to write macros. That information will then get distilled into a sort of macro style writing document, which will contain examples and recommendations of how to write relevant macros, and what kinds of macros not to write, to make software maintenance easier.
12 comments:
To me, it doesn't seem unrealistic to generate wiki pages from the sources.
I've got something like that in my compile script, in which I extract my packages documentation strings to generate "summary.html" files that are presented in the iframes on:
http://www.informatimago.com/develop/lisp/index.html
See http://tinyurl.com/38f97d3
and http://tinyurl.com/3589g6b
The only thing is that you need some discipline in where you put the information, how you format it, and how systematically you write it, so that the automatic publishing process may work smoothly.
The cleanup/arson has indeed been extensive but not all for the better. This is an interesting initiative but a bit too drastic. I am mainly concerned by the pages like the Java page that are now significantly broken and cannot be edited anymore.
Two events changed my point of view. The first was encountering defclass* in a commercial project. The second was working with the TPD2 networking code and encountering my-defun (and you thought I was joking?). I came face-to-face with macros that made software maintenance hard.
if i were you, i'd draw your conclusion more like:
I came face-to-face with macros that made software maintenance for the later joining programmers in the initial phase harder.
i don't see why a defun* is any worse than cl:defun, besides the initial learning obstacle for the newcomer. pushing your argument will end us up writing (setf (fdefinition ...) ...) instead of cl:defun.
or from another perspective: are abstractions introduced by the people writing CLHS are somehow absolute? as if they were some form of higher beings defining our world? or if not, then should we ban cl:defun also? where do you draw the line?
software development is mostly about reifying abstractions. of course abstractions come with a price, but in my experience there was much more to be paid due to unidentified and therefore unformalized implicit abstractions, than the required few C-c RET or M-. needed due to reified abstractions.
so IMO, there's no definitive conclusion to TGMD other than that it's a fuzzy scale on which it's subjective where people/projects end up.
There are bad macro definitions. There are bad function definitions. There are bad variable definitions. There are bad class definitions.
Whether a piece of code is bad or not is independent of whether it defines a macro or not. If you conclude from reading a bad macro definition and its uses that the concept of macros is a bad idea, you're just making a category error.
@pjb - the unrealistic part was referring to my wish to have people contribute well-categorized pages.
@JCB - that was one of the first things I did. It's a CLiki bug (http://www.cliki.net/CLiki%20Bugs). I've been in contact with Drew Crampsie and he's promised to send me a CLiki tarball so I can write a fix.
@p-cos - that's what you said at ILC. I am not saying that macros are bad because there are "bad" macros out there (here it is also important to define good and bad - bad macros are those that make maintenance harder; good macros are those that either make maintenance easier or have no effect on it). There are a lot of guides and advice to writing "good" functions. There are very few guides for writing "good" macros, and I think that is causing problems in the free software Lisp world.
@attila - the fundamental ontological flaw with the SICP assertion that "software is about reifying/layering abstractions" is that it does not differentiate motivations/goals from process. The process of software engineering may involve layering abstractions, but just because you are layering abstractions does not mean you are building useful software ("cargo cult engineering"). Abstractions in and of themselves are completely useless. Abstractions that do not serve to advance the state of your software to some domain-specific goal are worse than useless, because they make maintenance harder for no reason. ANSI CL is canon for Common Lisp programmers. The half-dozen "definer" my-first-lisp-macro projects that shift arguments around are just that.
@vladimir: so, you say that focusing solely on building layers of abstraction on each other doesn't say anything about a whole lot of other important aspects of the question of macros.
and then you seem to focus solely on maintainability.
the ultimately easy to maintain software is an empty file.
Abstractions that do not serve to advance the state of your software to some domain-specific goal are worse than useless, because they make maintenance harder for no reason.
i guess this is the point when we should stop trying to convince each other, because we see this so different that there's not much point in struggling, it'll be just a waste of our time.
e.g. i sometimes even introduce such low level "abstractions" as mere aliases. the purpose of them is communicating my intentions for the reader (including myself days/weeks later), and to provide a possible extension point later, not to mention the added searchability. my colleagues like it and they do the same.
@vladimir
Macros are powerful tools. And with great power comes great responsibility. Use them wisely they're great help, misuse them and they're burden. Due to cl being able to support many different paradigms you will encounter programmers with different styles which you will often disagree with. For myself Hoyte LOL comes to mind, I respect his opinion but I don't want to code like that. Being against macros you're against a degree of freedom trying to shape cl into Python like language. And that new subset might not be the subset you prefer.
@attila
"the ultimately easy to maintain software is an empty file."
Thank you for putting that so succinctly! That sums up my philosophy on writing software.
"i guess this is the point when we should stop trying to convince each other, because we see this so different that there's not much point in struggling, it'll be just a waste of our time."
Agreed. Like I said, we have a fundamentally different ontological view of software development, and I suspect a conflicting moral outlook as well (you might find it amusing, but I think software is evil).
"e.g. i sometimes even introduce such low level "abstractions" as mere aliases. the purpose of them is communicating my intentions for the reader (including myself days/weeks later), and to provide a possible extension point later"
Now you're getting at some valid motivations for writing macros. This is getting close to the kind of stuff I want to be able to list and explain.
Re utility packages, have a look at Alexandria.
"Abstractions that do not serve to advance the state of your software to some domain-specific goal are worse than useless, because they make maintenance harder for no reason."
I write tiny abstractions to make my code a little shorter all the time. I rarely have "some domain-specific goal" -- if I knew where I was going, I wouldn't need such a flexible language as Lisp! Often a bunch of these little unrelated abstractions yield some further insight or larger abstraction, which I never would have otherwise found. Maintenance difficulty I find to often be a function of verbosity more than whether I had a goal when I wrote it.
"ANSI CL is canon for Common Lisp programmers."
Are you one of those people who defines "Common Lisp programmers" as something more specific than "programmers who use a Common Lisp compiler"? I don't consider ANSI CL "canon". (Or at least, less so than AMOP or SICP or PAIP.) It's what my compiler happens to speak, mostly, so all my ideas have to get to that point if they want to be run as a program, but otherwise I see nothing special about the language. Shakespeare invented more new English words than anybody, and today his work is canon!
"The half-dozen "definer" my-first-lisp-macro projects that shift arguments around are just that."
What about LOOP and ITERATE, then? LOOP is probably one of the worst looping constructs around for Lisp, but it's part of CL. If it didn't have the stamp of approval from an early-90's political process, would you have a different opinion of it?
"Maintenance difficulty I find to often be a function of verbosity more than whether I had a goal when I wrote it."
I agree with this assertion 100%, just see my previous comment. But verbosity isn't defined by token length. If all some macro is doing is shortening the name of a Common Lisp function and shuffling the arguments around, that does nothing to reduce the code size, but ensures you must understand what the macro does in order to understand the code. So it's not like there's a trade-off between how much more concise the macro makes your code vs. having to understand the macro - the result is an unmitigated negative effect on maintainability.
"Shakespeare invented more new English words than anybody, and today his work is canon!"
Not all of the words Shakespeare invented are useful or used. The analogy to good macros vs bad macros would be like inventing new words for a technical jargon vs coming up with unrelated synonyms for the 100 most commonly used English words. In the former case your words get adopted right away, in the latter people think you are speaking a foreign language.
"What about LOOP and ITERATE, then? LOOP is probably one of the worst looping constructs around for Lisp, but it's part of CL. If it didn't have the stamp of approval from an early-90's political process, would you have a different opinion of it?"
Yes. That's the reason why I use loop and not iterate (or series).
Pascal Costanza wrote:
"There are bad macro definitions. There are bad function definitions. There are bad variable definitions. There are bad class definitions.
Whether a piece of code is bad or not is independent of whether it defines a macro or not. If you conclude from reading a bad macro definition and its uses that the concept of macros is a bad idea, you're just making a category error."
This kind of argument really irritates me. Here's why: if it were a valid argument, you could substitute any language feature for "macros" (since it says nothing concrete about macros in particular), and therefore justify the addition of absolutely any feature to a language, no matter how badly designed or badly integrated with the rest of the language.
For an argument justifying a language feature to be valid, it must actually say something about the technical or social aspects of using that feature, or the effect that it has on language properties like ease of reasoning, expressiveness, efficiency, difficulty of implementation, and so on.
To include an example of that in this comment, I will mention that I much prefer macro systems for typed languages such as MacroML, that guarantee that the output will be well-typed given that the input is, for all possible applications of the macro. (Such systems also automatically avoid unintended variable capture and other pitfalls of Common Lisp's macro system.)
-- David-Sarah Hopwood
Post a Comment