Fight the System

While there are quite a few libraries for Common Lisp downloadable on the Web, Newbies often have a hard time figuring out how to use them. Neither clicking on setup.exe nor configure; make; make install usually help. Instead, Lispers use system definitions that are kind of like a mixture of Makefiles and pkg-config tools (only different), together with a library called mk:defsystem to handle them.

In Lispspeak, a system is a collection of files that together make up a library, an application or something similar. Usually, most if not all files are Lisp source files, but sometimes other file types are included as well, for example data and configuration files, C sources or documentation.

Some of the files may depend on others – for example because they use macros defined in some other file – so they have to be loaded/compiled in the right order to work properly. Doing so by hand, using LOAD and COMPILE-FILE, may by possible, but is surely not fun – especially when you have a system containing dozens of files, or want to use several systems at once.

In addition to the problem of compiling systems properly, there also has to be a convenient way to find them and load them into a Lisp image. The ANSI Common Lisp standard includes rudimentary support for that, by means of the the functions PROVIDE and REQUIRE, that manage a collection of so-called "modules". Unfortunately, it doesn't really say much about how a Common Lisp implementation should actually handle modules (for example where it should look for installed modules), which makes it awkward to use them in practice.

While there are alternatives^[1] , mk:defsystem is the most common way to manage systems today: Most libraries come with a mk:defsystem-compatible system definition, and mk:defsystem itself works with most popular (and even some not-so-popular) Lisp implementations.

mk:defsystem is maintained as a part of CLOCC, the Common Lisp Open Code Collection, which happens to include lots of other useful CL code, too. You can find CLOCC at http://clocc.sourceforge.net, or download mk:defsystem only from its Sourceforge project site.

For obvious reasons, you cannot use the convenience mk:defsystem provides to load itself, you have to make your Lisp system know about it the hard way. Fortunately (and for exactly that reason), it is only a single source file (defsystem.lisp), so that you can easily LOAD it, like this:

(load "/path/to/your/defsystem.lisp")
    

Since you are likely to use it in almost every Lisp session, it is best to do so in your Lisp startup file. The actual file name of your init file depends on the Lisp implementation and operating system you use. For example, for CLISP on Unix, it would be ~/.clisprc.lisp, CMUCL uses ~/.cmucl-init.lisp. See the documentation of your Lisp of choice when you don't know where to put startup code.

You can also build a new Lisp image that contains mk:defsystem, so you don't have to load it at all. Again, your Lisps documentation will tell you how to do so.

mk:defsystem has the nice feature that it can find system definitions on its own; you only have to give it the name of a system to load or compile, not the physical location where its definition or source files live. This depends on two things: First, the file containing the system definition should be called foo.system (where "foo" is the name of the system). That's not usually something you have to care about, since library authors tend to know and respect this convention. Second, mk:defsystem has to know where to look for these files. It maintains a list of directories called the central registry (in a variable called mk:*central-registry*) where it expects system definitions to be.

Generally, it's a good idea to keep all your Lisp libs in a central directory, for example ~/cl-library/. Additionally, quite a few libraries by default expect their sources to live on a logical host^[2] called cl-library:, so we set this one up in our lisp init file, too.

(load "/path/to/your/defsystem.lisp")
(setf (logical-pathname-translations "cl-library")
      '(("**;*.*.*" "/home/you/cl-library/**/")))
(mk:add-registry-location #p"cl-library:")
      

As an example, we'll install the CL-PPCRE regular expression library using defsystem.

The first step is obviously to download the sources from the CL-PPCRE homepage. Unpack them in your ~/cl-library directory. This should create a subdirectory ~/cl-library/cl-ppcre-${VERSION} containing both Lisp source files and system definitions.

CL-PPCREs system definition is written in a way that it will find the source files based on its truename, that is, the "real" name of the system definition file, even when loaded via a symbolic link. So the easiest way to make everything work is creating a symbolic link to cl-ppcre.system in a directory in your mk:*central-registry*, for example like this:

# ln -s ~/cl-library/cl-ppcre-0.4.0/cl-ppcre.system ~/cl-library/cl-ppcre.system
    

That should be enough to be able to handle CL-PPCRE using mk:defsystem. To verify that everything works, start up your Lisp system and ask mk:defsystem if it finds the system:

* (mk:find-system :cl-ppcre)
System CL-PPCRE not loaded. Shall I try loading it? y

; Loading #p"/home/you/cl-library/cl-ppcre-0.4.0/cl-ppcre.system".
#<DEFSYSTEM: cl-ppcre>
    

Unless you get an error, you can now compile CL-PPCRE by evaluating

(mk:oos :cl-ppcre :compile)
    

(In case you wonder, mk:oos is just a short nickname of the function mk:operate-on-system.)

Once you have compiled CL-PPCRE, you can load it in subsequent Lisp sessions by running

(mk:oos :cl-ppcre :load)
    

Not all libraries are as nice as CL-PPCRE, using the truename of the system definition file to find the directory containing the sources. As stated above, others expect a logical host to set up properly, still others simply hard-code the paths that happen to work on the developers machine.

(mk:defsystem :frobber
              :source-pathname "frobber:"
              :source-extension "lisp"
              :components ((:file "package")
                           (:file "frobnicate")))
      

(mk:defsystem :frobber
              :source-pathname "/home/stronzo/proj/frobber/src/"
              :source-extension "lisp"
              :components ((:file "package")
                           (:file "frobnicate")))
      

The important part of these system definitions is the :source-pathname keyword option, which has to point to the source directory on your machine somehow.

In the first case, the simple solution is to set up a matching logical pathname translation. Again, this is not the place to explain logical pathnames in depth, but something like

(setf (logical-pathname-translations "frobber")
      '(("**;*.*.*" "cl-library:frobber;**;")))
    

should work, given that the sources are in a subdirectory called frobber in your cl-library.

The second case is more ugly. The quick and dirty solution is to simply modify the system definition file so that it points to the right source-pathname.

FIXME: Is there a non-q+d solution, without modifying the system definition? setf'ing mk::component-source-root-directory etc has no effect, apparently systems are always re-initialized by mk:oos.