Practical macros in Racket and how to work with them

| Macros, Racket | By: Kevin R. Stravers

A macro is central in any Lisp and must be mastered in order to master the language. However, Racket is in a state of macro chaos - at least in the official documentation. There’s define-syntax-rule, syntax-parse, syntax-case, and so many more. How do we make sense of it? In this tutorial we’ll investigate syntax transformers in Racket so we can get a complete picture.

1  What’s all this racket?

I’ve looked through the racket documentation and connected most of the syntax transformers there. Here are the biggest ones and some utility functions.

Arrows denote that one can be defined in terms of the other, e.g.: define-simple-macro can be defined in terms of define-syntax-parser.
Dashed boxes are unimportant tools that ought to be deprecated.
Red boxes indicate what I believe to be important tools, and the circles denote important syntax primitives.
Bold boxes are important but not essential.
Normal boxes are not that important, but good to know.

For the new macro programmer this is quite a lot to take in, and there’s a myriad of other tools that supplement these, but I can safely say that you can ignore everything except define-syntax-parser if you’re new. Why? Because it’s a typechecking syntax transformer and allows you to manipulate syntax in a complex manner. It also prevents you from having to use define-syntax together with syntax-parse. In essence it’s a way to define macros in a clean manner.

2  syntax-case, don’t use it!

Why not use syntax-case? It appears to be an important primitive. Well to put it clearly: syntax-case is the typeless version of syntax-parse.There are some other differences in how arguments are applied and errors are reported, but what’s in bold is the most important aspect. With syntax-parse we can assert that a term is an identifier or a keyword in a declarative manner whereas with syntax-case we’d have to write code that checks types during macro-time. syntax-parse is widely superior to syntax-case.
We should ideally deprecate syntax-case for end-user usage.

3  with-syntax

with-syntax is incredibly useful in syntax-case but inside syntax-parse we should use #:with instead:

Examples:
> (require syntax/parse/define)
> (define-syntax-parser with-example
    [(_ a)
     #:with (b:id ...) #'(one two three)
     #'(list a 'b ...)])
> (with-example 'zero)

'(zero one two three)

The advantage of #:with over with-syntax is the use of types and better error reporting for syntax-parse.

For reference, here’s with-syntax:

> (require syntax/parse/define)
> (define-syntax-parser with-example
    [(_ a)
     (with-syntax ([(b ...) #'(one two three)])
       #'(list a 'b ...))])
> (with-example 'zero)

'(zero one two three)

4  The syntax-parse family

syntax-parse is the primitive of the most advanced syntax transformer in racket (as far as I know). Here are some examples.

> (require syntax/parse syntax/parse/define)
> (define-syntax-parser name
    [(_ a b ...+)
     #'(+ a (- b ...))])
> (name 1 2 3 4)

-4

> (define-syntax name
    (syntax-parser
      [(_ a b ...+)
       #'(+ a (- b ...))]))
> (name 1 2 3 4)

-4

> (define-syntax (name stx)
    (syntax-parse stx
      [(_ a b ...+)
       #'(+ a (- b ...))]))
> (name 1 2 3 4)

-4

These forms are all acceptable depending on your situation. Most of the time we want to use define-syntax-parser because it saves us the effort of typing so much.

5  make-rename-transformer

This special transformer is basically an alias that preserves identifier equality.

> (define-syntax l (make-rename-transformer #'let))
> (let ([a 1] [b 2]) (+ a b))

3

> (l ([a 1] [b 2]) (+ a b))

3

> (free-identifier=? #'let #'l)

#t

6  make-set!-transformer

Another special transformer is the set!-transformer, it allows you to transform a mutation of an identifier.

> (define a 0)
> (define b 1)
> (let-syntax ([a (make-set!-transformer
                      (syntax-parser #:literals (set!)
                        [(set! _ v) #'(set! b v)]
                        [i:id #'a]))])
    (set! a 2)
    (list a b))

'(0 2)

I haven’t had much use for this in my code so far, but I guess it’s fine to keep in mind in case you need it.

7  Syntax taints, what are they?

The documentation on syntax taints is confusing to me. Here’s my synopsis: It prevents the arbitrary use of identifiers: if you extract any part of another macro’s armed result, then that extracted part is tainted and can’t be used further. Allow me to exemplify:

Examples:
> (require syntax/parse/define)
> (define-syntax-parser a
    [(_) (syntax-protect #'(c))])
; (c) is armed here
> (define-syntax-parser b
    [(_)
    ; c is extracted from (c), which taints the result c
     #:with d (car (syntax-e (local-expand #'(a) 'expression #f)))
    ; the macro expander inserts d which results in #'(+ TAINTED:c), so the expander rejects this
     #'(+ d)])
> (b)

eval:22:0: #%top: cannot use identifier tainted by macro

transformation

  in: #%top

This rejects the expression (+ c) because the identifier c is tainted. Why is it tainted? Because syntax-e tainted it. Why did it taint it? Because the syntax-object was armed.

> (require syntax/parse/define)
> (define c 10)
> (define-syntax-parser a
    [(_) (syntax-protect #'c)])
> (define-syntax-parser b
    [(_)
     #:with d #'(a)
     #'(displayln d)])
> (b)

10

This shows that the expander accepts: armed and clean syntax objects, but rejects tainted syntax objects.

8  Literals

syntax-parse allows the use of literals:

> (require syntax/parse/define)
> (define-syntax-parser my-parser
    #:datum-literals (a-word)
    [(_ a-word b-word)
     #'(begin
       (displayln 'a-word)
       (displayln 'b-word))])
> (my-parser a-word 10)

a-word

10

#:literals is also possible. Then there’s a need for an identifier to exist in the enclosing phase:

> (define-syntax-parser my-parser
    #:literals (is-this-bound?)
    [(_ is-this-bound? b-word)
     #'(begin
       (displayln 'a-word)
       (displayln 'b-word))])
> (my-parser is-this-bound? 10)

eval:33:0: syntax-parser: literal is unbound in phase 0

(phase 0 relative to the enclosing module)

  at: is-this-bound?

  in: (syntax-parser #:literals (is-this-bound?) ((_

is-this-bound? b-word) (syntax (begin (displayln (quote

a-word)) (displayln (quote b-word))))))

We can use literals to discriminate between real and fake identifiers:

> (define-syntax-parser is-it-let?
    [(_ (~literal let)) #'#t]
    [(_ (~datum let)) #'#f]
    [_ #'#f])
> (is-it-let? let)

#t

> (let ([let 0])
    (is-it-let? let))

#f

Note that (~literal x) as a pattern is the same as specifying #:literals (x) as keyword argument and using x as a pattern. Similarly for #:datum-literals (x).

9  Experimenting with the lowest level

Using define-syntax we can define simple functions that are essentially macros that don’t pattern match. This style allows you to get to know the low-level API, and I believe it to be very important to experiment with to understand what syntax-parse is actually doing.

Vision is the most important thing, let’s look at what’s going on!
; Note: a macro only takes on argument, which contains the entire syntax object
> (define-syntax (name stx)
    (displayln stx))
> (name hello world)

#<syntax:39:0 (name hello world)>

name: received value from syntax expander was not syntax

  received: #<void>

We need to add a result that is a syntax object:
> (define-syntax (name stx)
    (displayln stx)
    #'(void))
> (name hello world)

#<syntax:41:0 (name hello world)>

Now to extract some values. There are primitives used to extract information from syntax objects.

> (define-syntax (name stx)
    (displayln `("stx" ,stx))
    (displayln `("syntax-e" ,(syntax-e stx)))
    (displayln `("syntax->list" ,(syntax->list stx)))
    (displayln `("syntax-source" ,(syntax-source stx)))
    (displayln `("syntax-line" ,(syntax-line stx)))
    (displayln `("syntax-column" ,(syntax-column stx)))
    (displayln `("syntax-position" ,(syntax-position stx)))
    (displayln `("syntax?" ,(syntax? stx)))
    (displayln `("syntax-span" ,(syntax-span stx)))
    (displayln `("syntax-original?" ,(syntax-original? stx)))
    (displayln `("syntax-source-module" ,(syntax-source-module stx)))
    (displayln `("syntax->datum" ,(syntax->datum stx)))
    #'(void))
> (name hello world)

(stx #<syntax:43:0 (name hello world)>)

(syntax-e (#<syntax:43:0 name> #<syntax:43:0 hello> #<syntax:43:0 world>))

(syntax->list (#<syntax:43:0 name> #<syntax:43:0 hello> #<syntax:43:0 world>))

(syntax-source eval)

(syntax-line 43)

(syntax-column 0)

(syntax-position 43)

(syntax? #t)

(syntax-span 1)

(syntax-original? #f)

(syntax-source-module #f)

(syntax->datum (name hello world))

These are some of the functions that we can use on syntax objects. There’s another one that allows us to turn datums into syntax called datum->syntax. Let’s see if we can construct a simple macro based on this and syntax-e:

We’re gonna make (infix 1 + 2) return (+ 1 2).
> (define-syntax (infix stx)
    (let ([elems (syntax-e stx)])
      (when (not (= (length elems) 4))
        (raise-syntax-error "there should be 3 elements"))
      (datum->syntax stx `(,(caddr elems) ,(cadr elems) ,(cadddr elems)))))
> (infix 1 + 2)

3

Notice how there are 4 elements in the list, because infix is inside it too. We also need to provide a context for datum->syntax. The identifiers used in the result will be referenced from that context. In this case we used stx as the context. If you use #f, then + won’t be found and we have an error. The macro is essentially equivalent to:

> (define-syntax-parser infix
    [(_ a op b)
     #'(op a b)])
> (infix 1 + 2)

3

With syntax-parse the context is dependent on the input. This way we can safely refer to variables from the caller’s scope. This safety is what we call "macro hygiene", and allows us to compose macros without breaking them.

10  Syntax parameters, what are they for?

An anaphoric macro is a macro that can define macro-local variables. This isn’t composable because replacing code with anaphoric macros may break it, I present you exhibit A, the unhygienic macro:

> (define-syntax (aif stx)
    (let ([elems (syntax-e stx)])
      (datum->syntax stx
        `(let ([it ,(cadr elems)])
          (if it ,(caddr elems) ,(cadddr elems))))))
> (define it 10)
> (aif (member 2 '(1 2 3))
    (displayln it)
    (void))

(2 3)

The programmer wanted to print 10 but instead something else got printed. This is a trivial example but quickly balloons with bigger programs and bigger macros.

Let’s instead use syntax-parameters. These can be used hygienically:

> (require racket/stxparam)
> (define-syntax-parameter it (syntax-parser))
> (define-syntax-parser aif
    [(_ condition then otherwise)
     #'(let ([t condition])
       (syntax-parameterize ([it (syntax-parser [_ #'t])])
         (if t then otherwise)))])
> (aif (member 2 '(1 2 3))
    (displayln it)
    (void))

(2 3)

If we now have a declaration of it, that will override the syntax parameter.
> (let ([it 10])
    (aif (member 2 '(1 2 3))
      (displayln it)
      (void)))

10

During normal racket evaluation (from a file) you’ll get a duplicate-identifier error, in this context there’s another error, but the point is that there is an error instead of letting the programmer scratch his head.
> (define it 10)
> (aif (member 2 '(1 2 3))
    (displayln it)
    (void))

eval:53:0: syntax-parameterize: not bound as a syntax

parameter

  at: it

  in: (syntax-parameterize ((it (syntax-parser (_ (syntax

t))))) (if t (displayln it) (void)))

11  I don’t get it, how does syntax-parse work?

syntax-parse works by replacing all syntax objects after the pattern match with the results from the pattern match:

> (syntax-parse #'(this is some syntax)
    [(here is the pattern) #'(pattern is put here)])

#<syntax:58:0 (syntax is put this)>

So stuff like put is not in the pattern, so it’s just pasted as-is. Another cool thing is that you can run arbitrary code in the body:

> (syntax-parse #'(this is some syntax)
    [(here is the pattern)
     (displayln "This is arbitrary code, we could download webpages for use in this macro, whatever you wish")
     #'(pattern is put here)])

This is arbitrary code, we could download webpages for use in this macro, whatever you wish

#<syntax:59:0 (syntax is put this)>

There are also some special pattern forms:

> (syntax-parse #'(this is some syntax)
    [(here ...) #'(here ... put stuff)])

#<syntax:60:0 (this is some syntax put stuff)>

... is a postfix operator that makes syntax-parse consider whatever is before as a list of that pattern. It will expand this list in the expander when it is encountered. This allows us to create pretty complex macros:

...+ means one or more.

> (syntax-parse #'((this is) (some syntax))
    [((here there) ...+) #'(here ... there ... put stuff)])

#<syntax:61:0 (this some is syntax put stuff)>

They can even be nested

> (syntax-parse #'((this is) (some more stuff syntax))
    [((here ... there) ...+) #'(here ... ... there ... put stuff)])

#<syntax:62:0 (this some more stuff is synt...>

Note that the ... operator in the syntax has left-associativity, so here ... ... reduces to (in this case) ((this) (some more stuff)) ... ..., which reduces to (this) ... (some more stuff) ... which reduces to this some more stuff.