Lisp Design Patterns

Scaly lizard tail wrapped into a spiral. — Fractal lizard showing us a proper way to design layered software.

There was this question on how to design Lisp software on r/lisp. The answers mostly falling into one of

"Try and see."
"Use a general approach applicable to any language."
"Do a REPL-driven development."
"Build a functional style layered architecture."

The last response is quite close to acceptable architecture advice. But, I believe, neither (except for maybe the REPL answer) does justice to Lisp. The Lisp family of languages have many features and architectural approaches. Even though these patterns differ from the normie ones, they still are this: patterns. In this post, I'm listing some of the general patterns that I've seen in mine and others' code. From more generic (pun intended) to the local ones.

I'm deliberately ignoring OOP patterns. Lispy Object-Oriented systems (CLOS, EIEIO, GOOPS) might be strange and powerful. But most of the techniques from other OOP languages apply to them nonetheless. So let's leave the space for more Lisp-specific patterns!

But to tip my hat to the OOP/Gang of Four people, I'll start from one of their ideas:

Extension Points

There's an odd pattern among the ones Java programmers preach: Extension Point. Most likely no one understands what exactly that pattern means. It's just "a part of your design open to later extension." But if anything suits the "Extension Point" name, it's Lisp code.

All the patterns below are a certain kind of extension point. Varying levels of complexity, assorted dialects, diverse features. But still, these are some of the things that make Lisp software flexible and loveable.

By Contract: Protocols and Back-ends (CL, Clojure)

This top-level pattern is quite universal. My friend recently raged about the overabundance of meaningless interfaces in Java and C #. Golang is built around the idea of interfaces. C++ boasts virtual classes. Clojure allows to define defprotocol-s or multiple versions of and ensure their plasticity.

Common Lisp protocols usually come in the form of defgeneric declarations.

One defines a generic function.
But they don't define any implementation.
Down the line, they write code conforming to the contract this generic function established.
Further down the line, they can write another implementation.
And either of implementations might be drop-in alternatives for the downstream user.

A perfect example of this approach is Shinmera's work like cl-mixed with its heaps of back-ends. Another example is JSON-parser-agnostic NJSON (shameless plug: I designed it.) Here's how a protocol definition looks there:

(defgeneric decode-from-stream (stream)
  (:method (stream)
    (signal 'decode-from-stream-not-implemented))
  (:documentation "Decode JSON from STREAM.
Specialize on `stream' to make NJSON decode JSON."))

Protocol generic requiring at least one implementation

This type of architecture enforces a structure on the back-ends, while being adaptable to back-end swapping. And this type of generalization also allows for...

Monkey Patching (Emacs Lisp, CL)

If one can write a back-end and plug it into the software... Then, well, anyone can do that. Even the user. Even if the software is quite intricate. If it's a mature enough Lisp, it likely has

Function overrides (any Lisp),
Method overrides (any OOP Lisp),
Advices (Emacs, some CL implementations, Clojure),
:around methods (CL),
Slot listeners (CL and some other OOP Lisps, likely),
et cetera, et cetera

This downstream-override approach is colloquially known as "monkey patching." It's a basis for Aspect-Oriented Design, suggested by Gregor Kiczales. The idiomatic example of that design is exactly what the OP of the Reddit post was asking about: logging. In the the Aspect-Oriented-friendly application, logging can be added at any moment, just as an advice/override/:around function. No need to put console.log all over the place, just make a special method and wrap your code into it!

Another example of this approach might be a home-brewed event system in Nyxt (even though I left the team, I still love the code we've done together), based on (setf slot) :around methods in Common Lisp. The logic is: once the slot is setf-ed, invoke a function updating the UI:

(defmethod (setf url) :around (value (buffer document-buffer))
  (call-next-method)
  (set-window-title))

Example of event-invoking setter method

Hooks (Emacs, CL, and others)

Thanks to u/arthurno1 for mentioning this!

Hooks are variables/slots/objects effectively storing a list of functions (usually called "handlers"). Once something happens, the functions stored inside the hook are called. Emacs utilizes hooks as one of the main extension points, having hooks for:

Emacs initialization, closing, saving and other editor-specific actions.
Events.
Mode toggling.
And likely some other purposes I can't recall.

Hooks may be said to be a subset/superset of the monkey-patched listeners. But they are too conventional and useful to ignore.

Fail Fast, Fail Often: Conditions (CL)

I always panic! when I see an error in a non-Lisp language. Because this error usually means stack unwinding at best or program abort at worst. So there's a reason to fear errors and try to prevent them in any way possible.

Not in Lisp. Not in Common Lisp, in particular. CL has this Condition System thingie that errors are part of. Restarts are a notable feature of CL conditions that most other exception/error/condition systems lack. Restarts are basically pieces of code you can invoke from the debugger to fix the error.

Server returned an error code? Just restart to try again!
Something broke the code, but you don't want to deal with it now? Just continue to ignore it.
A typo made a wrong value to be passed to a function? Just use-value and replace the faulty argument.

All the computation is effectively paused until the user chooses a restart to resume it. Which can happen at an arbitrary moment or not even happen at all.

The point partially applies to other Lisps, because of REPL-driven development. Even if something goes wrong, one's just dropped into another level of debug loop. While in there, one can redefine the broken functions and restart the computation.

Or ignore the error and abort out into toplevel REPL.

Shared State: Dynamic Variables (Emacs Lisp, CL)

Ah, the dreaded global thread-unsafe state! Even though most Lisps left the idea of dynamic scope behind, some still have it. Common Lisp and Emacs Lisp both utilize dynamic variables... productively. One particular example might be CL's printer system. There are more than 15 globals which influence printing.

Which sounds like a disaster, right? Modifying global state just to change local logic sounds painful.

Luckily, the main use for dynamic variables is not setting them directly. Binding them lexically with let is a much more widespread pattern. It allows one to override the behavior these variables define, without making risky modifications of the shared state! P() Shameless plug again: my Graven Image library. There's no custom printing in it, and all the APIs rely on the built-in functions like print for it. At times, this makes the output look extremely inconsistent or outright scary. As an intended use-case, the output of Graven Image functions is intended to be altered via printer variables:

(defmethod gimage:apropos* :around (string &optional package external-only docs-too)
  (let ((*print-case* :downcase)
        (*print-level* 2)
        (*print-lines* 2)
        (*print-length* 10))
    (call-next-method)))

An example method overriding (see previous sections) printer variables

Pythonique: Keywords and Destructuring (CL, Guile, Clojure)

I'm not even sure whether I should include that: it's too low-level. But dynamic variables are quite atomic (pun not intended) too, so here goes nothing.

Programming in Guile, I often miss CL's function keyword. There's lambda*/define* for that, but it sucks that keywords are not enabled by default.

(define* (jsc-class-make-constructor
          (number-of-args (if callback (procedure-maximum-arity callback) 0)))
  "Create a constructor for CLASS with CALLBACK called on object initialization..."
  ...)

Use of define* to make functions with keywords in Guile

There are several places keywords are vital:

Non-trivial function behavior: dex:get has 21 keyword arguments because HTTP is hard and &optional arguments just don't cut it.
Heavily Object-oriented code: apply over make-instance is one's best friend.
Structure constructors are using keywords for arguments.
Other side of the OOP-FP spectrum: destructuring of arguments/EDN/data in Clojure. There's even a dedicated destructuring syntax in let!

Keywords might be passed real deep in the code paths. That's why CL's declare (ignore ...) and &allow-other-keys pattern is important. If one can ignore some keys instead of mandating them, their code can scale gracefully. And that's the whole point of design patterns, right?

Update Apr 2024: I now understand and love Clojure map/sequence destructuring. And Scheme match macro. So this point really goes beyond keywords: it's about getting the data out of inputs. Flexibly and within a hand's reach at all times.

Concluding

This is not the final list! You can probably recall some other patterns better than my memory allows me to. So feel free to reach out via Reddit, Hacker News comments, or email.

To use the air time productively: I'm searching for new projects/positions! If your team looks for a CL/Clojure dev with a good knowledge of design patterns (duh), C, and JavaScript, then we might be a good match! Shoot me an email sometime 😉