Common Lisp Tips

Formatting integers in different radixes

Sun, 03 Mar 2013 21:26:00 -0500

format has four directives for formatting integers with different radixes:

~B formats as binary: (format nil "~B" 42) => "101010"
~O formats as octal: (format nil "~O" 42) => "52"
~X formats as hexadecimal: (format nil "~X" 666) => "29A"
~R formats with an arbitrary radix between 2 and 36: (format nil "~36R" 18321) => "E4X"

Each directive takes padding and pad-character options (as well as other options). I define these functions in my init files to quickly dump out hex and bit views of integers:

(defun :hex (value &optional (size 4))
  (format t "~v,'0X" size value))

(defun :bits (value &optional (size 8))
  (format t "~v,'0B" size value))
  
* (:bits 42)
00101010

* (:hex 666)
029A

The lower-level write function accepts a :base argument that specifies a radix to use when printing integers. The default value is taken from *print-base*. For example, (write-to-string 65535 :base 16) => "FFFF". Since write underlies how other standard functions produce output, binding *print-base* will affect pretty much everything.

Literal syntax for integers

Sat, 02 Mar 2013 09:30:30 -0500

There are several ways to write literal integers with different radixes in Common Lisp. #b… is for binary, #o… is for octal, #x… is for hexadecimal, and #r is for an arbitrary radix from 2 to 36. Section 2.4.8.10 has this example:

#2r11010101  ;Another way of writing 213 decimal  
#b11010101   ;Ditto                               
#b+11010101  ;Ditto                               
#o325        ;Ditto, in octal radix               
#xD5         ;Ditto, in hexadecimal radix         
#16r+D5      ;Ditto                               
#o-300       ;Decimal -192, written in base 8     
#3r-21010    ;Same thing in base 3                
#25R-7H      ;Same thing in base 25               
#xACCEDED    ;181202413, in hexadecimal radix

How do I convert an integer to a list of bits?

Thu, 28 Feb 2013 20:54:00 -0500

Novices sometimes ask how to get a convert an integer to a list of bits, often for the purpose of iterating over the bits somehow. Common Lisp has a rich set of functions for directly accessing the bits of an integer, so constructing an intermediate list of bits is rarely necessary.

Here are a few examples of accessing the bits of an integer.

With integer-length and logbitp to test the bit at a particular index:

(defun list-of-bits (integer)
  (loop for index below (integer-length integer)
        collect (if (logbitp index integer) 1 0)))

With ash to shift the integer to the right and logand to test the least-significant bit:

(defun list-of-bits (integer)
  (loop repeat (integer-length integer)
        for i = integer then (ash i -1)
        collect (logand i 1)))

With ash to shift a one-bit mask to the left and logtest to test the mask bit against the integer:

(defun list-of-bits (integer)
  (loop repeat (integer-length integer)
        for mask = 1 then (ash mask 1)
        collect (if (logtest mask integer) 1 0)))

With byte to construct a byte specifier and ldb to extract a field from the integer:

(defun list-of-bits (integer)
  (loop for position below (integer-length integer)
        collect (ldb (byte 1 position) integer)))

update Unfortunately, all these examples are buggy - they collect bits in the wrong order. I used loop/collect to avoid having to reverse the results, but didn’t think it through properly. Here are some updated definitions with dotimes that give the results in the intended order:

(defun list-of-bits (integer)
  (let ((bits '()))
    (dotimes (index (integer-length integer) bits)
      (push (if (logbitp index integer) 1 0) bits))))

(defun list-of-bits (integer)
  (let ((i integer)
        (bits '()))
    (dotimes (j (integer-length integer) bits)
      (push (logand i 1) bits)
      (setf i (ash i -1)))))

(defun list-of-bits (integer)
  (let ((mask 1)
        (bits '()))
    (dotimes (i (integer-length integer) bits)
      (push (if (logtest mask integer) 1 0) bits)
      (setf mask (ash mask 1)))))

(defun list-of-bits (integer)
  (let ((bits '()))
    (dotimes (position (integer-length integer) bits)
      (push (ldb (byte 1 position) integer) bits))))

The optional arguments of deftype

Wed, 27 Feb 2013 21:20:57 -0500

deftype can be used to create user-defined types that expand into built-in types. For example:

(deftype octet-vector (length)
  `(simple-array (unsigned-byte 8) (,length)))

Given this deftype, in a type context, (octet-vector 32) expands into (simple-array (unsigned-byte 8) (32)), or a one-dimensional octet array of length 32.

Lambda lists for deftype are very similar to lambda lists for defmacro, with an important difference: where optional arguments to defmacro with no initialization form default to nil, the default value in deftype is the symbol *. The deftype above could be rewritten to take advantage of this:

(deftype octet-vector (&optional length)
  `(simple-array (unsigned-byte 8) (,length)))

Now, in a type context, a plain octet-vector expands into (simple-array (unsigned-byte 8) (*)), or a one-dimensional octet array of indeterminate length, and (octet-vector 32) expands into (simple-array (unsigned-byte 8) (32)).

For more information, see section 3.4.8, Deftype Lambda Lists.

Trying again with with-simple-restart

Tue, 26 Feb 2013 21:13:12 -0500

I have some code that reads data from a config file. If there’s a problem loading, I’d like the opportunity to fix it, outside of Lisp, and retry loading. It’s easy to do that with with-simple-restart and loop:

(loop 
  (with-simple-restart (try-again "Try again")
    (return 
     (progn 
       (setf *config* (load-config-file))))

If any error occurs during load-config-file, I can either fix up the file and choose the try-again restart, or give up and use an abort restart.

(Thanks to Frode Vatvedt Fjeld for the concise idiom.)

Semicolon style

Sun, 24 Feb 2013 20:32:14 -0500

The standard explains common semicolon comment style in section 2.4.4.2. Section 2.4.4.2.5 includes this short example showing typical use of one through four semicolons:

;;;; Math Utilities

;;; FIB computes the the Fibonacci function in the traditional
;;; recursive way.

(defun fib (n)
  (check-type n integer)
  ;; At this point we're sure we have an integer argument.
  ;; Now we can get down to some serious computation.
  (cond ((< n 0)
         ;; Hey, this is just supposed to be a simple example.
         ;; Did you really expect me to handle the general case?
         (error "FIB got ~D as an argument." n))
        ((< n 2) n)             ;fib[0]=0 and fib[1]=1
        ;; The cheap cases didn't work.
        ;; Nothing more to do but recurse.
        (t (+ (fib (- n 1))     ;The traditional formula
              (fib (- n 2)))))) ; is fib[n-1]+fib[n-2].

The tree-walkers of CL

Mon, 18 Feb 2013 11:11:00 -0500

If you need to visit a tree in some way, there are two built-in functions that, while not explicitly designed for the purpose, can walk a tree effectively.

First, subst-if takes a predicate function that is called for each cons and atom in a tree. As long as the predicate returns nil, no substitution will actually take place.

Here’s a simple way to turn it into a function:

(defun walk-tree (fun tree)
  (subst-if t
            (constantly nil)
            tree
            :key fun))

* (walk-tree 'print '(a b (1 2 . 10) c))

(A B (1 2 . 10) C) 
A 
(B (1 2 . 10) C) 
B 
((1 2 . 10) C) 
(1 2 . 10) 
1 
(2 . 10) 
2 
10 
(C) 
C 
NIL 
(A B (1 2 . 10) C)

Second, tree-equal is normally considered for comparing two trees, but it can also be used to call a function for each atom in a single tree; just pass the same tree to tree-equal, and use the test function to do something with each atom. As long as the test function returns true, the walking continues.

Here’s a function that wraps it up:

(defun walk-tree-atoms (fun tree)
  (tree-equal tree tree
              :test (lambda (element-1 element-2)
                      (declare (ignore element-2))
                      (funcall fun element-1)
                      t)))

* (walk-tree-atoms 'print '(a b (1 2 . 10) c))

A 
B 
1 
2 
10 
C 
NIL 
T

(This tip inspired by a few articles from Rob Warnock.)

Duplicated keyword arguments

Thu, 06 Dec 2012 15:39:00 -0500

If a function call has two pairs of keyword arguments with the same keyword, the leftmost argument pair is used. This can be helpful if a list of keyword arguments is received and you want to selectively override some value in that argument list in application. There’s no need to use mutation or produce a fresh variation of the list.

For example:

(defun write-escaped (object &rest args &key &allow-other-keys)
  (apply 'write object :escape t args))

* (write "hello" :escape nil)
hello

* (write-escaped "hello" :escape nil)
"hello"

See 3.4.1.4 for details.

The four causes of symbol conflicts

Sat, 27 Oct 2012 16:22:00 -0400

A symbol conflict arises when a package operation would result in two distinct symbols with the same name being accessible in a single package. There are exactly four ways this can happen.

Inheriting: When you use (via use-package or the defpackage :use clause) a package that exports a distinct symbol sharing the same name with a symbol already accessible in the using package. The conflict can arise between two inherited symbols, or between an inherited symbol and a present symbol.

Importing: When you import a symbol and a distinct symbol with the same name is already accessible in the target package.

Exporting: When you export a symbol from a package and a distinct symbol with the same name is already accessible in any package that uses that package.

Uninterning: When you unintern a shadowing symbol and the absence of its shadowing results in two distinct symbols with the same name being accessible via inheritance.

Here are short examples of each case. They all assume an environment in which this package definition is in effect:

(defpackage #:auto
  (:documentation "An automotive package.")
  (:use)
  (:intern #:tan)
  (:export #:car))

Inheritance conflict:

(defpackage #:inheritance-conflict
  (:use #:cl #:auto))

=> USE-PACKAGE #[PACKAGE "COMMON-LISP"] causes name-conflicts in
#[PACKAGE "INHERITANCE-CONFLICT"] between the following symbols:
  AUTO:CAR, COMMON-LISP:CAR
   [Condition of type NAME-CONFLICT]
See also:
  Common Lisp Hyperspec, 11.1.1.2.5 [:section]

Import conflict:

(defpackage #:import-conflict
  (:use #:cl))

(import '(auto:car) '#:import-conflict)

=> IMPORT AUTO:CAR causes name-conflicts in
#[PACKAGE "IMPORT-CONFLICT"] between the following symbols:
  AUTO:CAR, COMMON-LISP:CAR
   [Condition of type NAME-CONFLICT]
See also:
  Common Lisp Hyperspec, 11.1.1.2.5 [:section]

Export conflict:

(defpackage #:export-conflict
  (:use #:cl #:auto)
  ;; Avoid inheritance conflict with shadowing
  (:shadow #:car))

(export 'auto::tan '#:auto)

=> EXPORT AUTO::TAN causes name-conflicts in
#[PACKAGE "EXPORT-CONFLICT"] between the following symbols:
  AUTO::TAN, COMMON-LISP:TAN
   [Condition of type NAME-CONFLICT]
See also:
  Common Lisp Hyperspec, 11.1.1.2.5 [:section]

Unintern conflict:

(defpackage #:unintern-conflict
  (:use #:cl #:auto)
  ;; Avoid inheritance conflict with shadowing
  (:shadow #:car))

(unintern 'unintern-conflict::car '#:unintern-conflict)

=> UNINTERN UNINTERN-CONFLICT::CAR causes name-conflicts in
#<PACKAGE "UNINTERN-CONFLICT"> between the following symbols:
  AUTO:CAR, COMMON-LISP:CAR
   [Condition of type NAME-CONFLICT]
See also:
  Common Lisp Hyperspec, 11.1.1.2.5 [:section]

If you use Common Lisp, you have to interact with the package system. I think the best way to get along with the package system is not to keep it at arm’s length while holding your nose, but to learn how it works, inside and out.

A little bit of file-position

Thu, 27 Sep 2012 06:28:00 -0400

When used with one argument, file-position returns an integer representing the stream’s file position:

* (defvar *s* (open "data.bin" :element-type '(unsigned-byte 8))
=> *S*

* (read-byte *s*)
=> 42

* (read-byte *s*)
=> 107

* (file-position *s*)
=> 2

When used with two arguments, file-position sets the stream position:

* (file-position *s* 1)
=> T

* (read-byte *s*)
=> 107

When used this way, the position is often given as an integer, but the function actually accepts a designator. An integer position value represents itself, the keyword :start represents 0 (the first position in the stream), and the keyword :end represents the last position in the stream.

What would life be like without :start and :end? For :start, not too difficult; you could just use 0. But without :end you would need to use some other function, such as file-length, to correctly position the stream at the end.

A brief history of Lisp

Sat, 15 Sep 2012 23:58:04 -0400

For a 1500-word history of Lisp up to the point of Common Lisp standardization, see section 1.1.2 of the spec. It covers the institutions, projects, people, and influential ideas involved in the creation and evolution of Lisp over the course of several decades, from McCarthy at Dartmouth in the 1950s to the many active branches of Lisp in the 1980s.

Putting the R in REPL

Fri, 14 Sep 2012 23:58:10 -0400

Try this in your REPL:

* (let (#'42) (+ . #'5))
=> 47

If you find it perplexing, and you use SBCL, evaluating + next in the REPL might help clear things up.

Using an adjustable displaced array as a cursor on another array.

Fri, 14 Sep 2012 06:29:42 -0400

Scanning a string can be done using input functions withwith-input-from-string:

(with-input-from-string (stream "Hello World!  How do you do? ")
  (let ((c (make-string 4)))
    (loop
      :for pos = (read-sequence c stream)
      :while (= pos (length c))
      :do (print c)
      :finally (terpri))))

"Hell" 
"o Wo" 
"rld!" 
"  Ho" 
"w do" 
" you" 
" do?" 
nil

This can also be done using a displaced adjustable array, without copying the data that is read:

(defun make-vector-cursor (vector &key (size 0) (offset 0))
  (make-array size
              :element-type (array-element-type vector)
              :adjustable t
              :displaced-to vector
              :displaced-index-offset offset))

(defun advance-cursor (cursor &key (size (length cursor)))
  (multiple-value-bind (vector offset) (array-displacement cursor)
    (adjust-array cursor size
                  :element-type (array-element-type vector)
                  :displaced-to vector
                  :displaced-index-offset (+ offset (length cursor)))))

(let ((c (make-vector-cursor "Hello World!  How do you do? " :size 4)))
  (loop
    :do (print c)
    :while (ignore-errors (advance-cursor c))
    :finally (terpri)))

"Hell" 
"o Wo" 
"rld!" 
"  Ho" 
"w do" 
" you" 
" do?" 
NIL

(let ((c (make-vector-cursor "Hello World!  How do you do? " :size 1)))
  (loop
    :for size :from 1
    :do (print c)
    :while (ignore-errors (advance-cursor c :size size))
    :finally (terpri)))

"H" 
"e" 
"ll" 
"o W" 
"orld" 
"!  Ho" 
"w do y" 
"ou do? " 
nil

This post courtesy of Pascal J. Bourguignon

Binding keyword arguments

Sun, 09 Sep 2012 23:56:44 -0400

By default, keyword argument variable bindings match the name of the keyword used to pass the value. For example:

(defun keytest (&key foo)
  (list foo))

* (keytest :foo 42)
=> (42)

However, the variable doesn’t have to match the keyword provided. The following syntax will accept a keyword of :foo in the function call but bind a variable named bar:

(defun keytest (&key ((:foo bar)))
  (list bar))

* (keytest :foo 42)
=> (42)

Binding keywords this way can be helpful when binding a plist via destructuring-bind when you don’t want the plist keys to name the variables you actually want to use in the scope of the destructuring bind. For example, the plist may contain (:customer-id 42 ...), but in your context it makes more sense to bind a variable named old-customer-id.

The “keyword” also need not be a keyword:

(defun keytest (&key ((foo bar)))
  (list bar))

* (keytest 'foo 42)
=> (42)

Using internal symbols as function keyword arguments is one way to indicate that they are not part of a function’s public API.

Multiple export clauses in defpackage

Thu, 06 Sep 2012 23:56:08 -0400

The syntax for defpackage allows multiple export clauses. I like to use this feature to visually group related symbols.

(defpackage #:myproject
  (:use #:cl)
  ;; Web stuff
  (:export #:fetch
           #:parse-url
           #:status)
  ;; File utilities
  (:export #:lines
           #:first-line
           #:touch)
  ...)

Although it has no effect on the semantics, I find it helpful for reading.

A simple REPL

Wed, 05 Sep 2012 23:54:11 -0400

Here’s a very simple REPL that includes the *, **, and *** variables:

(defun repl ()
  (princ "> ")
  (loop 
    (shiftf *** ** * (eval (read))) 
    (format t "~a~&> " *)))

Provided by Stas Boukarev.

Printing package-qualified symbols

Tue, 10 Apr 2012 20:02:36 -0400

When *print-escape* is true, symbols are normally printed with package prefixes only if the current package *package* is not the symbol’s home package. It’s easy to make sure that a symbol is always printed with a package prefix, e.g. for debugging.

When the current package is the keyword package, non-keyword symbols are printed with package prefixes, and keywords are printed with their normal colon prefix. For example, in SBCL:

(in-package #:cl-user)
(let ((*package* (find-package "KEYWORD")))
  (prin1-to-string '(car stream :car quit)))

=> "(COMMON-LISP:CAR COMMON-LISP:STREAM :CAR SB-EXT:QUIT)"

For full details, see 22.1.3.3.1, Package Prefixes for Symbols.

Evaluating the last expression

Tue, 03 Jan 2012 08:50:06 -0500

In the REPL, +, ++, and +++ have as values the three most recently evaluated expressions. A quick way to evaluate the previous expression, especially handy in a REPL without input history, is

#.+

It’s equivalent to (eval +).

(Thanks to Anton Kovalenko.)

Un-displacing an array

Sun, 01 Jan 2012 10:00:06 -0500

Here’s a function to get the underlying array on which a displaced array is based:

(defun undisplace-array (array)
  "Return the fundamental array and the start and end positions into
it of a displaced array."
  (let ((length (length array))
        (start 0))
    (loop
      (multiple-value-bind (to offset) (array-displacement array)
        (if to
            (setq array to
                  start (+ start offset))
          (return (values array start (+ start length))))))))

From an article by Erik Naggum.

Referring to method parameters

Fri, 30 Dec 2011 08:47:00 -0500

In defmethod lambda lists, required parameters that aren’t explicitly specialized default to specializing on the system class t. But there’s a difference between an implicit and explicit specialization on t. The hyperspec explains:

The expansion of the defmethod macro “refers to” each specialized parameter (see the description of ignore within the description of declare). This includes parameters that have an explicit parameter specializer name of t. This means that a compiler warning does not occur if the body of the method does not refer to a specialized parameter, while a warning might occur if the body of the method does not refer to an unspecialized parameter. For this reason, a parameter that specializes on t is not quite synonymous with an unspecialized parameter in this context.

This makes a difference in Clozure CL; here’s what you get if you don’t use an implicitly specialized required parameter:

? (defmethod foo (bar) 42)
;Compiler warnings :
;   In (FOO (T)) inside an anonymous lambda form: Unused lexical variable BAR
#<STANDARD-METHOD FOO (T)>

There is no warning with an explicit specialization:

? (defmethod bar ((baz t)) 42)
#<STANDARD-METHOD BAR (T)>

(Thanks to Hans Hübner.)