Decoding The Racket Implementation

| Racket | By: Kevin R. Stravers

I’ve always wanted to know how big programs work, and to me the Racket implementation appears big. It’s pretty old and feels clunky. Nevertheless, let’s dive in to see if we can make sense of the code.

1  Setup

I start by pulling the code from the git repository hosted at https://github.com/racket/racket, I cd to my projects directory and run the following:

git clone https://github.com/racket/racket racket

Great! Now I enter the directory and see three directories:

 build

 pkgs

 racket

There’s a README.md and INSTALL.txt as well, but a quick search does not reveal where the entry point is nor what language the project is in.

After looking around for a bit I think racket contains the code. So I enter it.

 bin  collects  doc  etc  include  lib  man  share  src

Oh! That looks familiar, almost like the file system on modern linux systems. The obvious place for source is src/:

 build/ cify/ common/ configure COPYING_LESSER.txt COPYING-libscheme.txt COPYING.txt cs/ expander/ foreign/ gracket/ io/ lt/ mac/ Makefile.in myssink/ mzcom/ native-libs/ pkgs-check.rkt pkgs-config.rkt racket/

 README

 regexp/ rktio/ schemify/ setup-go.rkt start/ thread/ utils/ worksp/

There is a README file, let’s take a look!

 "... results of C compilation ..."

Great! Now we know that there is C being compiled. Let’s use git grep to find where the entry point is

git grep -Pi "int main"

 configure:   int main() {

 configure:   int main() {

 configure:   int main () {

 configure:    int main() {

 configure:      int main() {

 configure:     int main() {

 configure:       int main() {

 configure:     int main() {

 configure:    int main(int argc, char **argv) {

 configure:    int main(int argc, char **argv) {

 configure:int main() { return 0; }

 configure:   int main() {

 configure:   int main() {

 configure:   int main() {

 configure:     int main(int argc, char **argv) {

 configure:   int main() {

 cs/c/configure:    int main() {

 cs/c/configure:      int main() {

 cs/c/configure.ac:    int main() {

 cs/c/configure.ac:      int main() {

 cs/c/main.c:int main(int argc, char **argv)

 foreign/libffi/aclocal.m4:int main(){nm_test_func;return 0;}

 foreign/libffi/config.guess: int main (int argc, char *argv[]) {

 foreign/libffi/config.guess: int main (argc, argv) int argc; char *argv[]; {

 foreign/libffi/config.guess: int main ()

 ...

Oh no... seems like there are many entry points, but I want to find the interpreter entry point... Eventually we find something that looks just too nice to not be the interpreter entry point

 ...

 racket/main.c:int MAIN(int argc, MAIN_char **MAIN_argv)

 ...

Alright, let’s see what’s up.

vim racket/main.c

The header explains the following

 This file defines Racket's main(), which is a jumble of

 platform-specific initialization. The included file "cmdline.inc"

 implements command-line parsing. (GRacket also uses "cmdline.inc".

This is actually very useful to know so we know where command line parsing happens.

 The rest of the source code resides in the `src' subdirectory$

 (except for the garbage collector, which is in `gc', `sgc', or$

 `gc2', depending on which one you're using).

Great!

The relevant code comes down to:

 int MAIN(int argc, MAIN_char **MAIN_argv)

 {

 #if defined(DOS_FILE_SYSTEM) && !defined(__MINGW32__)

   load_delayed();

 #endif

   return main_after_dlls(argc, MAIN_argv);

 }

Now when compiling this on my machine (Fedora 26), the ifdef won’t trigger, so I’m interested if we can get some result by adding a print right inside this main function.

 int MAIN(int argc, MAIN_char **MAIN_argv)

 {

 #if defined(DOS_FILE_SYSTEM) && !defined(__MINGW32__)

   load_delayed();

 #endif

   printf("Hello from main :]");

   return main_after_dlls(argc, MAIN_argv);

 }

To compile, I go back to the README file I saw earlier and note that I can configure and make:

 For example, if you want to install into "/usr/local/racket" using

 dynamic libraries, then run:

 [here]configure --prefix=/usr/local/racket --enable-shared

Excellent, I’d like to put the results into a build directory, the README informs me that I can use a build directory and call configure as ../configure. Note that –enable-shared uses shared libraries.

 mkdir build

 cd build

 ../configure --enable-shared

Ok, no problems.

 Configure with --prefix if you wanted to install somewhere else.

 The --prefix option also makes the installed files better conform

 to Unix installation conventions. (The configure script will show

 you specific installation paths when --prefix is used.)

Thank you, having this information available at many stages is a good thing (tho, not too much of it).

Let’s see what build/ contains:

 config.log  config.status  foreign/  gracket/  lt/  Makefile  racket/  rktio/

Okay, there is the familiar makefile, now we can run make and see what happens!

 make

It sure spits out a lot of stuff, oooh a warning!

 ../../../racket/src/error.c: In function 'sch_vsprintf':

 ../../../racket/src/error.c:634:8: warning: 'errkind_str' may be used uninitialized in this function [-Wmaybe-uninitialized]                                                                                               sprintf((char *)t, "%s; %s=%d", es, errkind_str, en);

More warnings:

 In file included from ../../../racket/src/lightning/lightning.h:58:0,

                  from ../../../racket/src/jit.h:87,                                                                                                                                                                                 from ../../../racket/src/jitcommon.c:28:

 ../../../racket/src/jitcommon.c: In function 'gen_struct_slow':                                                                                                                                                    ../../../racket/src/jitcommon.c:1544:39: warning: ?: using integer constants in boolean context, the expression will always evaluate to 'true' [-Wint-in-bool-context]

    jit_movi_i(JIT_V1, ((kind == 3) ? 2 : 1));                                                                                                                                                                      ../../../racket/src/lightning/i386/core.h:507:29: note: in definition of macro 'jit_movi_i'

  #define jit_movi_i(d, is) ((is) ? MOVLir((is), (d)) : XORLrr ((d), (d)) )                                                                                                                                                                      ^~

 ../../../racket/src/jitcommon.c: In function 'common4b':                                                                                                                                                           ../../../racket/src/jitcommon.c:2100:40: warning: ?: using integer constants in boolean context, the expression will always evaluate to 'true' [-Wint-in-bool-context]

        jit_movi_i(JIT_V1, ((i == 1) ? 2 : 1));                                                                                                                                                                     ../../../racket/src/lightning/i386/core.h:507:29: note: in definition of macro 'jit_movi_i'

  #define jit_movi_i(d, is) ((is) ? MOVLir((is), (d)) : XORLrr ((d), (d)) )

 

Compilation fails with a make error:

 Hello from main :]make[6]: *** No rule to make target 'Hello'.  Stop.

1.1  Making the print work

Odd,... maybe I need to include <stdio.h>... no that doesn’t work either. In fact, there are other prints in the same file so this should work.

I remove the print and see what happens. Using grep to find the string "Hello from main" only shows up in binary files, so I’m not sure why make reports an error here. We get some more warnings.

 ../../../racket/gc2/../src/bignum.c: In function 'make_single_bigdig_result':

 ../../../racket/gc2/../src/bignum.c:488:110: warning: 'quick.iso.so.keyex' is used uninitialized in this function [-Wuninitialized]

    SCHEME_SET_BIGPOS(sm, pos);

And some more warnings, but I won’t paste those here. Now compilation actually seems to continue further than before. Maybe configure uses main to generate make rules (which is quite odd).

So instead of putting my print in main I move it to ‘main_after_dlls‘. Well, that doesn’t work either, again, the same makefile error.

Running ‘make -n‘ shows me the commands and we see something that might be the culprit:

 /home/kefin/stuf/docs/racket/racket/src/build/lt/libtool --mode=compile --tag=CC gcc -I. -I../../racket/include -Wall   -g -O2  -pthread  -DUSE_SENORA_GC  -DMZ_USES_SHARED_LIB -DINITIAL_COLLECTS_DIRECTORY='"'"`cd ../../racket/../../collects; pwd`"'"' -DINITIAL_CONFIG_DIRECTORY='"'"`cd ../../racket/../..; pwd`/etc"'"' -c ../../racket/main.c -o main.lo

 /home/kefin/stuf/docs/racket/racket/src/build/lt/libtool --mode=link --tag=CC gcc -rpath /home/kefin/stuf/docs/racket/racket/lib -o racketcgc main.lo  libracket.la libmzgc.la   -pthread -ldl -lm  -ldl -lm -rdynamic

 make[5]: Leaving directory '/home/kefin/stuf/docs/racket/racket/src/build/racket'

 make cstartup

 make[5]: Entering directory '/home/kefin/stuf/docs/racket/racket/src/build/racket'

 make cstartup_`./racketcgc -cu ../../racket/src/startup-select.rkt`

 make[6]: Entering directory '/home/kefin/stuf/docs/racket/racket/src/build/racket'

 ./racketcgc -cu ../../racket/src/compile-startup.rkt cstartup.inc cstartup.zo ../../racket/src/startup.inc ../../racket/src/schvers.h

 make[6]: *** No rule to make target 'Hello'.  Stop.

 make[6]: Leaving directory '/home/kefin/stuf/docs/racket/racket/src/build/racket'

So maybe it’s libtool’s doing? I’m not sure. Let’s try using PRINTF instead of printf, nope doesn’t work. Let’s just put in the expression ‘"Hello";‘, does that work? YES! IT DOES. With a warning that the statement has no effect.

So my hypothesis is that libtool is actually scanning for printf and doing something strange with it.

I create a simple function to see if this at all works:

 void x(const char *p) { }

 ...

 x("Hello");

Okay, this actually compiles, so it just strengthens my idea about printf. Could it be... that the print is taken as a make command? That libtool runs the code and uses it to compile further parts of racket? That’s absurd. By removing the call to ‘x‘ but leaving ‘void x...‘ in, we can check if it’s run-time or compile-time dependent, seems run-time, as this works just fine.

To add another test, let’s create an infinite loop inside main ‘for (;;);‘

Unsurprisingly, it doesn’t stop. So we now know what happens. The compilation process uses the racket interpreter to compile stuff. That surely makes debugging harder for us. What can we do?

Let’s try printing to standard-error instead: ‘fprintf(stderr, "Hello");‘

HURRAH! It works!

Now, trying to run the binaries, stuff is stored in build, and using find: ‘find . -type f -executable‘

 ./lt/libtool

 ./gracket/.libs/gracket3m

 ./gracket/gracket3m

 ./config.status

 ./racket/racket3m

 ./racket/racketcgc

 ./racket/starter

 ./racket/.libs/racket3m

 ./racket/.libs/racketcgc

 ./racket/.libs/libracket-6.90.0.19.so

 ./racket/.libs/libracket3m-6.90.0.19.so

 ./racket/.libs/lt-racketcgc

 ./racket/.libs/lt-racket3m

 ./racket/.libs/libmzgc-6.90.0.19.so

 ./foreign/libffi/config.status

 ./foreign/libffi/a.out

 ./foreign/libffi/libtool

 ./rktio/config.status

Ok cool, so it seems that we may want racket/racket3m or racketcgc, running either results in.

 HelloWelcome to Racket v6.90.0.19 [cgc].

 read (compiled): wrong version for compiled code

   compiled version: 6.9.0.4

   expected version: 6.90.0.19

   in: /home/kefin/stuf/docs/racket/racket/collects/racket/compiled/init_rkt.zo

   context...:

    read-dispatch

    read-syntax

    default-load-handler

    standard-module-name-resolver

    module-path-index-resolve

    module-declared?

 read (compiled): wrong version for compiled code

   compiled version: 6.9.0.4

   expected version: 6.90.0.19

   in: /home/kefin/stuf/docs/racket/racket/collects/racket/compiled/interactive_rkt.zo

   context...:

    read-dispatch

    read-syntax

    default-load-handler

    standard-module-name-resolver

    do-dynamic-require6

 read (compiled): wrong version for compiled code

   compiled version: 6.9.0.4

   expected version: 6.90.0.19

   in: /home/kefin/stuf/docs/racket/racket/collects/racket/compiled/base_rkt.zo

   context...:

    read-dispatch

    read-syntax

    default-load-handler

    standard-module-name-resolver

    do-dynamic-require6

The same error occurs without the Hello printed to stderr.

I try ‘git reset –hard‘ to undo every change I’ve made to check if that’s the cause. Nope, same error.

Asking the racket IRC channel on freenode.net (#racket) to see what’s up.

1.2  Finding the real entry point

Meanwhile, let’s see if the print to standard error is closest to the real entry point. We can do this by running ‘objdump -d racket/racketcgc | less‘, unfortunately: file format not recognized. ‘file racket/racketcgc‘ gives

 racket/racketcgc: POSIX shell script, ASCII text executable, with very long lines

Oh, well that makes sense, let’s look at this file.

 # racketcgc - temporary wrapper script for .libs/racketcgc

 # Generated by ltmain.sh - GNU libtool 1.5.24 (1.1220.2.455 2007/06/24 02:13:29)

Okay, so let’s then ‘objdump -d racket/.libs/racketcgc | less‘

So normally, programs start with the ‘_start‘ symbol, which calls main. Let’s see if - from there - main calls fprintf with stderr:

 0000000000402810 <main>:

   402810:       55                      push   %rbp

   402811:       53                      push   %rbx

   402812:       89 fd                   mov    %edi,%ebp

   402814:       48 8d 3d cc 32 00 00    lea    0x32cc(%rip),%rdi        # 405ae7 <__dso_handle+0x40f>

   40281b:       48 89 f3                mov    %rsi,%rbx

   40281e:       ba 05 00 00 00          mov    $0x5,%edx

   402823:       48 83 ec 18             sub    $0x18,%rsp

   402827:       48 8b 05 ca 57 20 00    mov    0x2057ca(%rip),%rax        # 607ff8 <stderr@GLIBC_2.2.5>

   40282e:       be 01 00 00 00          mov    $0x1,%esi

   402833:       48 8b 08                mov    (%rax),%rcx

   402836:       e8 e5 fe ff ff          callq  402720 <fwrite@plt>

   40283b:       48 8d 35 0e 17 00 00    lea    0x170e(%rip),%rsi        # 403f50 <main_after_stack>

   402842:       48 89 e2                mov    %rsp,%rdx

   402845:       bf 01 00 00 00          mov    $0x1,%edi

   40284a:       89 2c 24                mov    %ebp,(%rsp)

   40284d:       48 89 5c 24 08          mov    %rbx,0x8(%rsp)

   402852:       e8 d9 f9 ff ff          callq  402230 <scheme_main_stack_setup@plt>

   402857:       48 83 c4 18             add    $0x18,%rsp

   40285b:       5b                      pop    %rbx

   40285c:       5d                      pop    %rbp

   40285d:       c3                      retq

   40285e:       66 90                   xchg   %ax,%ax

 

 0000000000402860 <_start>:

   402860:       31 ed                   xor    %ebp,%ebp

   402862:       49 89 d1                mov    %rdx,%r9

   402865:       5e                      pop    %rsi

   402866:       48 89 e2                mov    %rsp,%rdx

   402869:       48 83 e4 f0             and    $0xfffffffffffffff0,%rsp

   40286d:       50                      push   %rax

   40286e:       54                      push   %rsp

   40286f:       49 c7 c0 c0 56 40 00    mov    $0x4056c0,%r8

   402876:       48 c7 c1 50 56 40 00    mov    $0x405650,%rcx

   40287d:       48 c7 c7 10 28 40 00    mov    $0x402810,%rdi

   402884:       ff 15 5e 57 20 00       callq  *0x20575e(%rip)        # 607fe8 <__libc_start_main@GLIBC_2.2.5>

   40288a:       f4                      hlt

   40288b:       0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)

Note how 40287d:, 0x402810 is loaded into ‘rdi‘, which is the address of ‘main‘, and in main we can see the fprintf on 402836 (fwrite in this case, but it is the same).

To make sure I’m not led astray I recompile without the fprintf and lo and behold,

 0000000000402780 <main>:

   402780:       48 83 ec 18             sub    $0x18,%rsp

   402784:       48 89 74 24 08          mov    %rsi,0x8(%rsp)

   402789:       48 8d 35 10 17 00 00    lea    0x1710(%rip),%rsi        # 403ea0 <main_after_stack>

   402790:       89 3c 24                mov    %edi,(%rsp)

   402793:       48 89 e2                mov    %rsp,%rdx

   402796:       bf 01 00 00 00          mov    $0x1,%edi

   40279b:       e8 10 fa ff ff          callq  4021b0 <scheme_main_stack_setup@plt>

   4027a0:       48 83 c4 18             add    $0x18,%rsp

   4027a4:       c3                      retq

   4027a5:       66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)

   4027ac:       00 00 00

   4027af:       90                      nop

 

 00000000004027b0 <_start>:

   4027b0:       31 ed                   xor    %ebp,%ebp

   4027b2:       49 89 d1                mov    %rdx,%r9

   4027b5:       5e                      pop    %rsi

   4027b6:       48 89 e2                mov    %rsp,%rdx

   4027b9:       48 83 e4 f0             and    $0xfffffffffffffff0,%rsp

   4027bd:       50                      push   %rax

   4027be:       54                      push   %rsp

   4027bf:       49 c7 c0 10 56 40 00    mov    $0x405610,%r8

   4027c6:       48 c7 c1 a0 55 40 00    mov    $0x4055a0,%rcx

   4027cd:       48 c7 c7 80 27 40 00    mov    $0x402780,%rdi

   4027d4:       ff 15 16 58 20 00       callq  *0x205816(%rip)        # 607ff0 <__libc_start_main@GLIBC_2.2.5>

   4027da:       f4                      hlt

   4027db:       0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)

No fwrite!

This confirms that this is the absolute entry point of the program. The biggest problem now is getting the correctly compiled libraries to work.

1.3  Response from IRC

They tell me to go up into the root dir and instead run make. After reading the README.md some more, I find ‘make in-place‘, this will put the resulting binaries inside the racket/ directory.

 <kefin_> read (compiled): wrong version for compiled code

 <kefin_> Anyone know what I can do? Using ../configure --enable-shared && make

 <kefin_> compiled version: 6.9.0.4, expected version: 6.90.0.19

 <kefin_> Here's full paste: https://pastebin.com/t987HYcM

 <stamourv> kefin_: try running raco setup

 <kefin_> stamourv: note, I'm compiling to a local dir, my native racket is 6.11, I'll try it anyway

 <tilpner> "v6.90.0.19."

 <tilpner> That sounds wrong

 <kefin_> tilpner: It's commit 234e47a58f

 <stamourv> tilpner: That's a version number for a version that's leading up to 7.0.

 <tilpner> Oh, okay. Is this a clean build?

 <stamourv> That is, 6.12 will be the last release in the 6 series, so everything at this point is going towards 7.

 <kefin_> tilpner: yes, I am building from repo-root/racket/src/build, do i need to build from the makefile in the root directory?

 <stamourv> kefin_: Yes, typing make in the root directory will take care of everything.

 <stamourv> The directory you mention will only build the C parts of the system, which is a very small portion.

 <kefin_> I see. I was under the impression that one could build that and ignore the rest.

 <stamourv> I mean, building that will get you *something*. But that somethins is not full Racket (or even Racket minimal),

Well... this takes a bit longer to build...

1.4  Not quite working

I caught another error

 raco setup: --- summary of errors ---

 raco setup: error: during making for <pkgs>/typed-racket-more/typed/rackunit

 raco setup:   racket/share/pkgs/typed-racket-more/typed/rackunit/type-env-ext.rkt:9:21: except-in: identifier `make-arr' not included in nested require spec

 raco setup:     at: (rep prop-rep object-rep type-rep)

 raco setup:     in: (except-in (rep prop-rep object-rep type-rep) make-arr)

 raco setup:     compiling: <pkgs>/typed-racket-more/typed/rackunit/type-env-ext.rkt

 raco setup: error: during making for <pkgs>/typed-racket-more/typed/rackunit

 raco setup:   racket/share/pkgs/typed-racket-more/typed/rackunit/type-env-ext.rkt:9:21: except-in: identifier `make-arr' not included in nested require spec

 raco setup:     at: (rep prop-rep object-rep type-rep)

 raco setup:     in: (except-in (rep prop-rep object-rep type-rep) make-arr)

 raco setup:     compiling: <pkgs>/typed-racket-more/typed/rackunit/type-env-ext.rkt

 Makefile:85: recipe for target 'in-place-setup' failed

 make[2]: *** [in-place-setup] Error 1

 make[2]: Leaving directory '/home/kefin/stuf/docs/racket'

 Makefile:67: recipe for target 'plain-in-place' failed

 make[1]: *** [plain-in-place] Error 2

 make[1]: Leaving directory '/home/kefin/stuf/docs/racket'

 Makefile:48: recipe for target 'in-place' failed

 make: *** [in-place] Error 2

I’m not sure what this means, the identifier make-arr doesn’t exist but it’s inside an except-in clause.

I re-clone the repository and try again

 make >& /dev/null

 echo $?

 0

Nice! It works I guess.

So now I add back the ‘fprintf(stderr, "Hello");‘

The executable is located in the racket/bin directory as described in the readme file.

1.5  We have liftoff

Finally, it works. When starting racket we now have "Hello" printed right at the start.

 ./racket/bin/racket

 HelloWelcome to Racket v6.90.0.19.

 >

Great! But let’s find out what happens further. By tracing through the code we can get an idea where the red thread of the program is.

 racket/main.c: int MAIN(argc, argv) // Does nothing interesting

 racket/main.c: main_after_dlls(argc, argv) // Puts argc, argv into Main_Args (ma) and calls

 racket/src/salloc.c: scheme_main_stack_setup(1, main_after_stack, &ma) //

         scheme_setup_thread_local_key_if_needed();

         scheme_init_os_thread();

         scheme_rktio = rktio_init();

 racket/src/salloc.c: do_main_stack_setup(1, main_after_stack, &ma);

         void *stack_start;

         scheme_set_stack_base(PROMPT_STACK(stack_start), 1);

 racket/main.c: main_after_stack(&ma)

         Extracts argc/argv from ma

         Processes windows unicode argv to utf-8

         scheme_set_signal_handler(SIGINT, user_break_hit);

         scheme_set_signal_handler(SIGTERM, term_hit);

         scheme_set_signal_handler(SIGHUP, hup_hit);

         pre_filter_cmdline_arguments(&argc, &argv);

 racket/cmdline.inc: run_from_cmd_line(argc, argv, scheme_basic_env, cont_run);

         Parses command line arguments in a primitive manner

         Converts long arguments to short

         Parses each short argument

         If no arguments -> run REPL [besides the -i interactive argument]

         Acquire ENVVARS (PLTSYSLOG, PLTSTDERR, PLTDISABLEGC,..)

         Set collection paths

One thing that surprised me is the amount of unexplained code in ‘cmdline.inc‘, for instance on line 1267 (repository 14448eb1bd1ec0d12e2ce5ffd5155466ac58151b).

 {

   int len, offset;

   GC_CAN_IGNORE char *coldir;

   collects_paths_l = scheme_make_null();

   coldir = extract_coldir();

   offset = 0;

   while (1) {

     len = strlen(coldir XFORM_OK_PLUS offset);

     offset += len + 1;

     if (!coldir[offset])

       break;

     collects_paths_l = scheme_make_pair(scheme_make_path(coldir XFORM_OK_PLUS offset),

                                         collects_paths_l);

 }

This really should be in its own function. My overall sense of ‘cmdline.inc‘ is that it’s a mess that is poorly abstracted into smaller functions (something that can be done).

At the end of the function, the results are moved from local variables into a structure called FinishedArgs (fa).

 fa->a->use_repl = use_repl;

This seems like a code flaw to me. Since this is located in the same function in which the local variable is stored, why not either separate the function or just immediately assign to fa?

Anyway, let’s continue.

At the end of the function we see ‘return cont_run(fa)‘. Alright, so where is ‘cont_run‘? Well that comes as an argument into ‘run_from_cmd_line‘ from ‘main_after_stack‘. Looking at main.c again, we see ‘cont_run‘ declared and it calls

 racket/main.c: cont_run(fa)

 racket/cmdline.inc: finish_cmd_line_run(fa, do_scheme_rep)

Again, ‘finish_cmd_line_run‘ is located in cmdline.inc, what does it do? For starters it’s about 250 lines long. Okay so it’s managable...

So by reading, it simply issues scheme commands to a fresh environment, basically it prepares the environment for the program, setting a base namespace and so on, then calls ‘do_scheme_rep‘

Inside ‘do_scheme_rep‘ we require ‘racket/base‘ and the ‘read-eval-print-loop‘ symbol, then we do a ‘scheme_apply‘ on the resulting object. The code path looks like this:

 racket/cmdline.inc: run_from_cmd_line(argc, argv, scheme_basic_env, cont_run);

 racket/main.c: cont_run(fa);

 racket/cmdline.inc: finish_cmd_line_run(fa, do_scheme_rep);

 racket/main.c: do_scheme_rep(fa->global_env, fa);

 ?: scheme_apply(rep, 0, NULL);

Now using ‘git grep‘ doesn’t yield much information, so I’m going to run gdb on racket to and break on ‘scheme_apply‘:

Well shit, we get a SIGSEGV in ‘scheme_gmp_tls_unload‘ for some reason. I bet it has to do with threads.

Anyway, I asked around on IRC and just needed ‘handle SIGSEGV nostop noprint‘ in GDB, the reason is that the GC causes SIGSEGVs during normal operation and handles them. In our case gdb was interfering with handling.

 <kefin> Anyone able to run gdb on racket? Getting a SIGSEGV in scheme_gmp_tls_unload

 <samth> kefin: you want to do `handle SIGSEGV nostop noprint`

 <samth> racket uses page protection to implement write barriers for the precise GC

 <kefin> samth: thanks that works. Does this mean it SIGSEGVs outside of gdb too but it's just hidden?

 <kefin> Or does gdb interfere too much?

 <samth> kefin: it triggers SEGV in the course of normal operation

 <samth> there's a signal handler that does the marking

 <kefin> Ok. How does GC_CAN_IGNORE work? It's an empty #define afaik

 <samth> kefin: that's a hint to the xform program

 <samth> which transforms the C code to cooperate with the precise collector

 <samth> it basically says that xform can ignore that

 <kefin> Ok. Haven't looked much into what xform is and how the GC works yet

Now it runs fine and breaks on ‘scheme_apply‘, but gdb has no idea where that function is located. Ugh.

Well, we can always fall back on ‘git grep "scheme_apply.*[^;]$"‘, and finally find that this function is located in racket/src/racket/src/fun.c

Well, from here we seem to enter redirection hell. Let’s go boys...

 racket/main.c: do_scheme_rep(fa->global_env, fa);

 racket/src/fun.c: scheme_apply(rep, 0, NULL)

Hmm, it seems that rep doesn’t really depend on anything in fa much.

 racket/src/fun.c: _apply(rep, 0, NULL, 0, 1)

         Sets current thread object, rator, etc

         scheme_current_thread::

                 ku.k.p1 = rep;

                 ku.k.p2 = NULL;

                 ku.k.i1 = 0;

                 ku.k.i2 = 0;

 racket/src/fun.c: scheme_top_level_do(apply_k, eb=1);

 racket/src/fun.c: scheme_top_level_do_worker(apply_k, eb=1, new_thread=0);

This last procedure is a bit hard to understand. It sets up continuation frames/barriers and then calls ‘apply_k‘, however, it may also call ‘apply_again_k‘. This function appears to check the arity and/or return values.

I put fprintf inside ‘apply_again_k‘ and recompile. Let’s see what happens... Nothing. It’s never printed. The comment on the code states

 an abort to the thread start; act like the default prompt handler but remember to jump again

Ok...? What does the thread start mean?

We don’t know (yet) so let’s look further, finally ‘apply_k‘ is called

 racket/src/fun.c: apply_k();

         Depending on the current thread ku.k.p2, calls either

         _scheme_apply_multi_wp or _scheme_apply_wp

To find out what the number means, I put a print, recompiled, and ran some statements. I’m not quite sure, when typing 1 in the REPL and running, we get a sequence of 1000101110001. Not sure how to decode this.

 _scheme_apply_wp(r,n,rs,p) = scheme_do_eval_w_thread(r,n,rs,1,p)

 _scheme_apply_multi_wp(r,n,rs,p) = scheme_do_eval_w_thread(r,n,rs,-1,p)

 scheme_do_eval_w_thread(r,n,e,f,p) = scheme_do_eval(r,n,e,f)

‘scheme_do_eval‘ calls the following

 racket/src/eval.c: scheme_do_eval(obj, num_rands, rands, get_value)

 racket/src/eval.c: do_eval_stack_overflow(obj, num_rands, rands, get_value)

 racket/src/eval.c: scheme_handle_stack_overflow

         Does a longjmp and almost never returns

Oh this is quite nasty., the include in ‘scheme_do_eval‘ ends in a condition for the return.

Alright, let’s re-evaluate then.

 racket/src/eval.c: scheme_do_eval(obj, num_rands, rands, get_value)

         handles each type of obj (apply top)

The first type check is to see if a type is ‘scheme_primitive‘. Alright, I add fprintf to the if to see what happens. When writing ‘1‘ in the REPL, primitive type shows up twice. Inside the if, a cast is made

 prim = (Scheme_Primitive_Proc *)obj;

The type ‘Scheme_Primitive_Proc‘ looks like this (found using git grep)

 typedef struct {

   Scheme_Prim_Proc_Header pp;

   Scheme_Primitive_Closure_Proc *prim_val;

   const char *name;

   mzshort mina;

   /* If mina < 0; mina is negated case count minus one for a case-lambda

      generated by mzc, where the primitive checks argument arity

      itself, and mu.cases is available instead of mu.maxa. */

   union {

     mzshort *cases;

     mzshort maxa;   /* > SCHEME_MAX_ARGS => any number of arguments */

   } mu;

 } Scheme_Primitive_Proc;

Okay great! What do we do here? Oh look, it has ‘name‘, let’s print it!

 check_location_fields

 void

Okay interesting,... what is the location field then?

Instead of looking for that, I looked for ‘scheme_print‘, and found ‘scheme_print_to_string‘, maybe this can enlighten us! There is also one where you can print to a port but that is a ‘Scheme_Object‘, and I don’t know how to specify stderr as a scheme object.

This resulted in a compilation error for some reason, with the reason being bad startup script, errorring with a SIGSEGV. Hmmm. This indiactes perhaps that the string is null and that fprintf should not print it. Let’s try it.

Aaaah, I see. The print function eventually calls ‘scheme_top_level_do‘, thus causing an infinite recursion and eventual stack overflow.

 scheme_print_to_string(obj, len)

 scheme_print_to_string_w_max(obj, len, maxlen=-1)

 scheme_top_level_do(print_to_string_k, 0)

Now it does not appear as if it gets back to the main ‘scheme_do_eval‘ function, so I’m not entirely sure what goes wrong here. Unfortunately, gdb can’t be used either since the program doesn’t compile fully. This is in my opinion a problem for newbies like myself. We can’t fully mess with the code because messing with it makes it uncompilable.

I remove static from the ‘print_to_string‘ function that ‘print_to_string_k‘ eventually calls and forward-declare it in ‘eval.c‘. Let’s see what happens next... but I can’t. There’s a variable coming from the environment called ‘qq_depth‘ and I have no idea what this means. Instead, I’ll write to stderr BEFORE calling the string conversion to see if anything indicates an infinite loop.

Interestingly, it’s only printed once. Is it because we can’t just reinterpret from within the main interpreter? That would certainly make sense, as it would mess up the state of the machine.

I’ll have to get back to the function ‘print_to_string‘ instead... wait. I might’ve given the wrong input. I see it takes a ‘intptr_t *len‘, and I gave it 100. Obviously that’s going to be a problem, so I declare a local variable and grant the address instead.

 prim = (Scheme_Primitive_Proc *)obj;

 intptr_t len;

 GC_CAN_IGNORE char *x = scheme_print_to_string(obj, &len);

 if (x) {

    x[99] = '\0';

    fprintf(stderr, "primitive type %s\n", x);

 } else {

    fprintf(stderr, "primitive unknown\n");

 }

 free(x);

The first line is not my code, but is there to show where the rest of the code is located. This results in a double-free, so I remove my free and ‘GC_CAN_IGNORE‘.

This time I get an xform error. xform is the program that checks variables in the C code. I’ll need to move len and x to the beginning of the block.

This also fails, let’s see what’s going on:

 SIGSEGV MAPERR si_code 1 fault on addr 0x7f04c6363ff8

Okay, interesting. Why is that? I add in a free and get a ‘munmap_chunk‘ error, so it’s definitely not free that’s needed. here is the revised code

 #include <assert.h>

 ...

 intptr_t len;

 char *x;

 ...

 prim = (Scheme_Primitive_Proc *)obj;

 x = scheme_print_to_string(obj, &len);

 if (x) {

    assert(x[len-1] == '\0');

    fprintf(stderr, "primitive type %s\n", x);

 } else {

    fprintf(stderr, "primitive unknown\n");

 }

It might have been the x[99] that caused the sudden SIGSEGV. Let’s try and run this.

 Assertion `x[len-1] == '\0'' failed

What about len instead of len-1? Maybe that’s the way things are here. The build just fails spontaneously. Perhaps we’re writing to parts of memory that aren’t supposed to be zerod. Hmm... the prints work just fine without the null. Perhaps it’s something else. I remove the assert.h include because it’s causing some funky business with other asserts.

Now we get SIGSEGV MAPERR again. I’m setting len-1 to be ’\0’ JUST to be safe here, maybe that cuts off a letter... and it does :O.

How very interesting. But alas, again the MAPERR. Again the idea of a stack overflow crosses my mind and I add a print right before ‘scheme_print_to_string‘ to see if that’s true.

There doesn’t seem to be any indication of an infinite loop here at all. There are all sorts of primitives coming through. If it were a loop we would see a pattern of the same primitives, besides adding a print after the statement confirms that we go in-and-out instead of looping.

 Seg fault (internal error during gc) at 0x7fec405b09b8

 SIGSEGV SEGV_ACCERR SI_CODE 2 fault on 0x7fec405b09b8

I’ll put a ‘GC_CAN_IGNORE‘ on len to see if that helps. ... Same problem. Let’s add one on the ‘char* x‘ whilst having top open to make sure memory isn’t exhausted.

The same thing happened. I’m not sure where to progress to now. I know! Instead of checking for x = nullptr, let’s instead check for len != 0.

Still the same error. It’s so weird, it errs at about the same place every time. Let’s try and not print the specific object on which it fails. Let’s put prints around the ‘scheme_print_to_string‘ procedure, and, interestingly enough, the mapper error ONLY shows up AFTER this function has exited. What?! Let’s instead assign x = ""; to see if there is any difference at all. This in fact compiles just fine...

Using a static int to only run the print ONCE doesn’t make anything crash.

BUT it still compiles sufficiently to run the binary. Hurra! This unfortunately doesn’t shed much light to the subject; we just get a bunch of ‘#<procedure>‘ being printed.

1.6  What does the data look like?

While we can’t advance much on that problem, let’s look at what the data structure ‘Scheme_Object‘ looks like. To understand any code, one must understand the most important data structures.

 # define MZ_HASH_KEY_EX  short keyex;

 ...keyex

 typedef struct Scheme_Object

 {

   Scheme_Type type; /* Anything that starts with a type field

                        can be a Scheme_Object */

 

   /* For precise GC, the keyex field is used for all object types to

      store a hash key extension. The low bit is not used for this

      purpose, though. For string, pair, vector, and box values in all

      variants of Racket, the low bit is set to 1 to indicate that

      the object is immutable. Thus, the keyex field is needed even in

      non-precise GC mode, so such structures embed

      Scheme_Inclhash_Object */

 

   MZ_HASH_KEY_EX

 } Scheme_Object;

Alright, so it’s basically just a placeholder pointer for other types. Great. So we have no type safety here at all. All that’s the same for every object is that the first few bytes are allocated to its type.

 typedef short Scheme_Type;

Okay, so the first 2 bytes (on the machine I’m using) define the type of the ‘Scheme_Object‘.

 typedef struct Scheme_Inclhash_Object

 {

   Scheme_Object so;

   MZ_OPT_HASH_KEY_EX

 } Scheme_Inclhash_Object;

 

 typedef struct Scheme_Simple_Object

 {

   Scheme_Inclhash_Object iso;

 

   union

     {

       struct { mzchar *string_val; intptr_t tag_val; } char_str_val;

       struct { char *string_val; intptr_t tag_val; } byte_str_val;

       struct { void *ptr1, *ptr2; } two_ptr_val;

       struct { int int1; int int2; } two_int_val;

       struct { void *ptr; int pint; } ptr_int_val;

       struct { void *ptr; intptr_t pint; } ptr_long_val;

       struct { struct Scheme_Object *car, *cdr; } pair_val;

       struct { mzshort len; mzshort *vec; } svector_val;

       struct { void *val; Scheme_Object *type; } cptr_val;

     } u;

 } Scheme_Simple_Object;

Okay, so this is pretty standard stuff, we have a string type, a byte-string type, two pointers, two integers (probably rationals), and so on.

 /* A floating-point number: */

 typedef struct {

   Scheme_Object so;

   double double_val;

 } Scheme_Double;

Ok. A double floating-point number, to be specific.

 typedef struct Scheme_Symbol {

   Scheme_Inclhash_Object iso; /* 1 in low bit of keyex indicates uninterned */

   intptr_t len;

   char s[mzFLEX_ARRAY4_DECL];

 } Scheme_Symbol;

Wow! There’s our symbol. How does ‘mzFLEX_ARRAY4_DECL‘ work? Well it can be either 4 or nothing, and if it’s nothing, then it’s a flexible array. I suppose the string is simply zero terminated and extends out of the struct itself.

Great!

Now what’s next?

 typedef struct Scheme_Vector {

   Scheme_Inclhash_Object iso; /* 1 in low bit of keyex indicates immutable */

   intptr_t size;

   Scheme_Object *els[mzFLEX_ARRAY_DECL];

 } Scheme_Vector;

Let’s try putting a print with ‘scheme_make_integer‘ in the main loop... and it works! Great.

 typedef struct {

   Scheme_Object so;

   unsigned short flags;

 } Scheme_Prim_Proc_Header;

 

 typedef struct {

   Scheme_Prim_Proc_Header pp;

   Scheme_Primitive_Closure_Proc *prim_val;

   const char *name;

   mzshort mina;

   /* If mina < 0; mina is negated case count minus one for a case-lambda

      generated by mzc, where the primitive checks argument arity

      itself, and mu.cases is available instead of mu.maxa. */

   union {

     mzshort *cases;

     mzshort maxa;   /* > SCHEME_MAX_ARGS => any number of arguments */

   } mu;

 } Scheme_Primitive_Proc;

So this is the procedure header and procedure itself.

 typedef struct Scheme_Hash_Table

 {

   Scheme_Inclhash_Object iso; /* 0x1 flag => print as opaque (e.g., exports table); 0x2 => misc (e.g., top-level multi_scopes) */

   intptr_t size; /* power of 2 */

   intptr_t count;

   Scheme_Object **keys;

   Scheme_Object **vals;

   void (*make_hash_indices)(void *v, intptr_t *h1, intptr_t *h2);

   int (*compare)(void *v1, void *v2);

   Scheme_Object *mutex;

   intptr_t mcount; /* number of non-NULL keys, >= count (which is non-NULL vals) */

 } Scheme_Hash_Table;

This is quite descriptive.

So now we have some idea of about what objects look like.

1.7  apply top

By adding an fprintf under the ‘apply_top‘ label we see that every character that gets inserted results in an object of ‘#<procedure:readline/rktrl.rkt:188:14>‘ to be issued. This happens to be the following lambda:

 (lambda (_)

   (define next-byte (read-byte real-input-port))

   (if (eof-object? next-byte) -1 next-byte)))

This is interesting, how does a keypress issue this command?

We see inside of ‘pkgs/readline-lib/readline/pread.rkt‘ the history, prompt, and so on

 (define current-prompt   (make-parameter #"> "))

 (define max-history      (make-parameter 100))

 (define keep-duplicates  (make-parameter #f))

I change the prompt to see what’s up,... and it works! So this package is used to read input, however, how is it initialized? Remember that the read-eval-print-loop is used, so I suspect this to be the culprit.

Inside main.c:

 /*************************   do_scheme_rep   *****************************/

 /*                  Finally, do a read-eval-print-loop                   */

 

 static void do_scheme_rep(Scheme_Env *env, FinishArgs *fa)

 {

   /* enter read-eval-print loop */

   Scheme_Object *rep, *a[2];

   int ending_newline = 1;

 

 #ifdef GRAPHICAL_REPL

   if (!fa->a->alternate_rep) {

     a[0] = scheme_intern_symbol("racket/gui/init");

     a[1] = scheme_intern_symbol("graphical-read-eval-print-loop");

     ending_newline = 0;

   } else

 #endif

     {

       a[0] = scheme_intern_symbol("racket/base");

       a[1] = scheme_intern_symbol("read-eval-print-loop");

     }

 

   rep = scheme_dynamic_require(2, a);

 

   if (rep) {

     scheme_apply(rep, 0, NULL);

     if (ending_newline)

       printf("\n");

   }

 }

Right right... so we intern two symbols, call dynamic require, and then apply it. What does apply mean and how does ‘scheme_dynamic_require‘ work? Let’s find out.

In eval.c:

 Scheme_Object *scheme_dynamic_require(int argc, Scheme_Object *argv[])

 {

   Scheme_Object *proc;

   proc = scheme_get_startup_export("dynamic-require");

   return scheme_apply(proc, argc, argv);

 }

Okay, what does ‘get_startup_export‘ do?

 Scheme_Object *scheme_get_startup_export(const char *s)

 {

   Scheme_Object *sym;

   Scheme_Bucket *b;

   sym = scheme_intern_symbol(s);

   b = scheme_instance_variable_bucket_or_null(sym, scheme_startup_instance);

   if (b)

     return (Scheme_Object *)b->val;

   return NULL;

 }

Okay, we’ll need to look at what a ‘Scheme_Bucket‘ is, how ‘scheme_intern_symbol‘ works, and how ‘scheme_instance_variable_bucket_or_null‘ works.

As defined in scheme.h:

 typedef struct Scheme_Bucket

 {

   Scheme_Object so;

   void *val;

   char *key;

 } Scheme_Bucket;

Okay, so a key-value pair. Cool.

 #define scheme_intern_symbol (scheme_extension_table->scheme_intern_symbol)

Oh this is interesting. What is the ‘scheme_extension_table‘? There is another one in symbol.c:

 Scheme_Object *

 scheme_intern_symbol(const char *name)

   /* `name' must be ASCII; this function is not suitable for non-ASCII

      conversion, necause it assumes that downcasing each C char

      is good enough to normalize the case. */

 {

   if (!scheme_case_sensitive) {

     uintptr_t i, len;

     char *naya;

     char on_stack[MAX_SYMBOL_SIZE];

 

     len = strlen(name);

     if (len >= MAX_SYMBOL_SIZE)

       naya = (char *)scheme_malloc_atomic(len + 1);

     else

       naya = on_stack;

 

     for (i = 0; i < len; i++) {

       int c = ((unsigned char *)name)[i];

 

       c = scheme_tolower(c);

 

       naya[i] = c;

     }

 

     naya[len] = 0;

 

     return scheme_intern_exact_symbol(naya, len);

   }

 

   return scheme_intern_exact_symbol(name, strlen(name));

 }

 

 Scheme_Object *

 scheme_intern_exact_symbol(const char *name, uintptr_t len)

 {

   return intern_exact_symbol_in_table(enum_symbol, 0, name, len);

 }

 

 static Scheme_Object *

 intern_exact_symbol_in_table(enum_symbol_table_type type, int kind, const char *name, uintptr_t len)

 {

   return intern_exact_symbol_in_table_worker(type, kind, name, len);

 }

 

 static Scheme_Object *

 intern_exact_symbol_in_table_worker(enum_symbol_table_type type, int kind, const char *name, uintptr_t len)

 {

   Scheme_Object *sym;

   Scheme_Hash_Table *table;

 #if defined(MZ_USE_PLACES) && defined(MZ_PRECISE_GC)

   Scheme_Hash_Table *place_local_table;

 #endif

 

   sym = NULL;

 

   switch(type) {

     case enum_symbol:

       table = symbol_table;

 #if defined(MZ_USE_PLACES) && defined(MZ_PRECISE_GC)

       place_local_table = place_local_symbol_table;

 #endif

       break;

     case enum_keyword:

       table = keyword_table;

 #if defined(MZ_USE_PLACES) && defined(MZ_PRECISE_GC)

       place_local_table = place_local_keyword_table;

 #endif

       break;

     case enum_parallel_symbol:

       table = parallel_symbol_table;

 #if defined(MZ_USE_PLACES) && defined(MZ_PRECISE_GC)

       place_local_table = place_local_parallel_symbol_table;

 #endif

       break;

     default:

       printf("Invalid enum_symbol_table_type %i\n", type);

       abort();

   }

 

 #if defined(MZ_USE_PLACES) && defined(MZ_PRECISE_GC)

   if (place_local_table) {

     sym = symbol_bucket(place_local_table, name, len, NULL, type);

   }

 #endif

   if (!sym && table) {

     sym = symbol_bucket(table, name, len, NULL, type);

   }

   if (!sym) {

     /* create symbol in symbol table unless a place local symbol table has been created */

     /* once the first place has been create the symbol_table becomes read-only and

        shouldn't be modified */

 

     Scheme_Object *newsymbol;

     Scheme_Hash_Table *create_table;

 #if defined(MZ_USE_PLACES) && defined(MZ_PRECISE_GC)

     create_table = place_local_table ? place_local_table : table;

 #else

     create_table = table;

 #endif

     newsymbol = make_a_symbol(name, len, kind);

 

     /* we must return the result of this symbol bucket call because another

      * thread could have inserted the same symbol between the first

      * symbol_bucket call above and this one */

     sym = symbol_bucket(create_table, name, len, newsymbol, type);

   }

 

   return sym;

 }

Well this causes more confusion than I wanted. Apparently this gets a symbol table, creates a symbol bucket.

So back to ‘scheme_dynamic_require‘, the equivalent command being run is

 (dynamic-require 'racket/base 'read-eval-print-loop)

This means that ‘read-eval-print-loop‘ is inside the ‘racket/base‘ namespace, and dynamic-require puts it into the base namespace. The returned scheme object ‘rep‘ is the repl function, which we apply. This is when the main REPL starts.

What I’d like to know is how the function is stored inside racket/base. Let’s go back to the initial list of functions.

 racket/main.c: int MAIN(argc, argv) // Does nothing interesting

 racket/main.c: main_after_dlls(argc, argv) // Puts argc, argv into Main_Args (ma) and calls

 racket/src/salloc.c: scheme_main_stack_setup(1, main_after_stack, &ma) //

         scheme_setup_thread_local_key_if_needed();

         scheme_init_os_thread();

         scheme_rktio = rktio_init();

 racket/src/salloc.c: do_main_stack_setup(1, main_after_stack, &ma);

         void *stack_start;

         scheme_set_stack_base(PROMPT_STACK(stack_start), 1);

 racket/main.c: main_after_stack(&ma)

         Extracts argc/argv from ma

         Processes windows unicode argv to utf-8

         scheme_set_signal_handler(SIGINT, user_break_hit);

         scheme_set_signal_handler(SIGTERM, term_hit);

         scheme_set_signal_handler(SIGHUP, hup_hit);

         pre_filter_cmdline_arguments(&argc, &argv);

 racket/cmdline.inc: run_from_cmd_line(argc, argv, scheme_basic_env, cont_run);

         Parses command line arguments in a primitive manner

         Converts long arguments to short

         Parses each short argument

         If no arguments -> run REPL [besides the -i interactive argument]

         Acquire ENVVARS (PLTSYSLOG, PLTSTDERR, PLTDISABLEGC,..)

         Set collection paths

 racket/main.c: cont_run(fa)

 racket/cmdline.inc: finish_cmd_line_run(fa, do_scheme_rep)

 racket/main.c: do_scheme_rep(fa->global_env, fa);

 racket/fun.c: scheme_apply(rep, 0, NULL);

Okay, so somewhere in here the racket/base namespace has to be loaded into somewhere. My guess is ‘finish_cmd_line_run‘.

In there it calls ‘scheme_builtin_value‘ on "dynamic-require", so let’s see what this does...

 Scheme_Object *scheme_builtin_value(const char *name)

 {

   Scheme_Object *sym, *v;

   Scheme_Bucket *b;

   sym = scheme_intern_symbol(name);

   v = scheme_hash_get(scheme_startup_env->all_primitives_table, sym);

   if (!v) {

     b = scheme_instance_variable_bucket_or_null(sym, scheme_startup_instance);

     if (b)

       return b->val;

   }

   return v;

 }

It just gets the symbol associated with the object from ‘scheme_startup_env->all_primities_table‘. Here is the startup env in schpriv.h

 /* A Scheme_Startup_Env holds tables of primitives */

 struct Scheme_Startup_Env {

   Scheme_Object so; /* scheme_startup_env_type */

   Scheme_Hash_Table *current_table; /* used during startup */

   Scheme_Hash_Table *primitive_tables; /* symbol -> hash table */

   Scheme_Hash_Table *all_primitives_table;

   Scheme_Hash_Table *primitive_ids_table; /* value -> integer */

 };

We also find ‘make_startup_env‘ in env.c, which is called from ‘init_startup_env‘. Adding an fprintf does show that this is run, but from where is it called? from ‘scheme_basic_env‘, given to ‘run_from_cmd_line‘ from ‘main_after_stack‘ main.c

Then inside ‘run_from_cmd_line‘ we do:

 /* Creates the main kernel environment */

 global_env = mk_basic_env();

Great! and ‘global_env‘ is inside FinishedArgs, and indeed, ‘global_env‘ is used inside ‘finish_cmd_line_run‘.

 scheme_init_process_globals();

 scheme_init_true_false();

Not much interesting happens, a lock is made and some objects assigned.

 scheme_init_symbol_table();

Let’s look at this one

 symbol_table = init_one_symbol_table();

Not very interesting

 init_startup_env();

OOOOH that looks more like it! Let’s go!

 static void init_startup_env(void)

 {

   Scheme_Startup_Env *env;

 #ifdef TIME_STARTUP_PROCESS

   intptr_t startt;

 #endif

   REGISTER_SO(kernel_symbol);

   kernel_symbol = scheme_intern_symbol("#%kernel");

I haven’t even looked further but I already know it’s going to be good.

 MZTIMEIT(symbol-type, scheme_init_symbol_type(env));

 ...

 void

 scheme_init_symbol_type (Scheme_Startup_Env *env)

 {

 }

Oh... nothing? Okay next.

         MZTIMEIT(fun, scheme_init_fun(env));

 

         racket/fun.c:

           o = scheme_make_folding_prim(procedure_p, "procedure?", 1, 1, 1);

           SCHEME_PRIM_PROC_FLAGS(o) |= scheme_intern_prim_opt_flags(SCHEME_PRIM_IS_UNARY_INLINED

                                     | SCHEME_PRIM_IS_OMITABLE

                                     | SCHEME_PRIM_PRODUCES_BOOL);

           scheme_addto_prim_instance("procedure?", o, env);

 

 Alright, interesting, so it binds `procedure_p` to the symbol "procedure?", I assume. Let's look at `procedure_p`.

 

         static Scheme_Object *

         procedure_p (int argc, Scheme_Object *argv[])

         {

           return (SCHEME_PROCP(argv[0]) ? scheme_true : scheme_false);

         }

Oh, quite interesting, note how it takes in an array of pointers to ‘Scheme_Object‘. Perhaps this is the actual stack (instead of the heap). I wonder if simply making a single list of ‘Scheme_Object‘ would be faster. What’s also interesting is that the function itself doesn’t appear to check for argc length == 1. What does this? ‘SCHEME_PRIM_IS_UNARY_INLINED‘? Let’s try to remove the flag to see what happens. Well it doesn’t work. It still expects a single input.

And according to scheme.h:

 #define SCHEME_PROCP(obj)  (!SCHEME_INTP(obj) && ((_SCHEME_TYPE(obj) >= scheme_prim_type) && (_SCHEME_TYPE(obj) <= scheme_proc_chaperone_type)))

So that’s that. Now I’d like to know how ‘scheme_make_folding_prim‘ works.

 Scheme_Object *

 scheme_make_folding_prim(Scheme_Prim *fun, const char *name,

                          mzshort mina, mzshort maxa,

                          short folding)

 {

   /* A folding primitive is an immediate primitive, and for constant

      arguments the result must be the same on all runs and platforms. */

   return make_prim_closure(fun, 1, name, mina, maxa,

                            (folding

                              ? SCHEME_PRIM_OPT_FOLDING

                              : 0),

                            1, 1,

                            0, 0, NULL);

 }

mina and maxa represent the max and minimal number of arguments.

 o = scheme_make_folding_prim(procedure_p, "procedure?", 1, 1, 1);

There it is, the 1, 1 are mina and maxa respectively. The last value is "folding". What does this do? It sends a flag to ‘make_prim_closure‘. Let’s change the second number to 2 and see if we can give ‘procedure?‘ two arguments. And it works! Great.

So we now have an idea of how primitives - at least primitive functions - are implemented.

The other initialization function ‘scheme_init_list‘ does similar things but for ‘null‘, ‘pair?‘, and so on. ‘cons‘ for instance is very simple, a call to ‘scheme_make_pair‘ which simply creates a ‘Scheme_Object‘:

 # define cons(car, cdr) scheme_make_pair(car, cdr)

 static Scheme_Object *

 cons_prim (int argc, Scheme_Object *argv[])

 {

   return cons(argv[0], argv[1]);

 }

 Scheme_Object *scheme_make_pair(Scheme_Object *car, Scheme_Object *cdr)

 {

 #ifdef MZ_PRECISE_GC

   return GC_malloc_pair(car, cdr);

 #else

   Scheme_Object *cons;

   cons = scheme_alloc_object();

   cons->type = scheme_pair_type;

   SCHEME_CAR(cons) = car;

   SCHEME_CDR(cons) = cdr;

   return cons;

 #endif

 }

The deeper I go the simpler things start to become.

Only remaining is dynamic-require. The only place where dynamic-require is mentioned in C code is in cmdline.inc and in eval.c, both of which use ‘scheme_builting_value‘ or ‘scheme_get_startup_export‘. ‘scheme_startup_instance‘ is the environment that is used for this, let’s see how it is manipulated. From env.c:

 scheme_startup_instance = scheme_make_instance(scheme_intern_symbol("startup"), scheme_false);

 scheme_init_startup_instance(scheme_startup_instance);

Ah, that’s interesting. A git-grep later and we find ‘startup.c‘:

 void scheme_init_startup_instance(Scheme_Instance *inst)

 {

   /* called per-places */

   scheme_instantiate_linklet_multi(startup_linklet(), inst, 0, NULL, 0);

 }

 

 Scheme_Object *scheme_instantiate_linklet_multi(Scheme_Linklet *linklet, Scheme_Instance *instance,

                                                 int num_instances, Scheme_Instance **instances,

                                                 int use_prompt)

 {

   return do_instantiate_linklet(linklet, instance, num_instances, instances, use_prompt, 1, 1);

 }

 

 static Scheme_Object *do_instantiate_linklet(Scheme_Linklet *linklet, Scheme_Instance *instance,

                                              int num_instances, Scheme_Instance **instances,

                                              int use_prompt, int multi, int top)

 {

   Scheme_Thread *p = scheme_current_thread;

 

   p->ku.k.p1 = linklet;

   p->ku.k.p2 = instance;

   p->ku.k.p3 = instances;

 

   p->ku.k.i1 = multi;

   p->ku.k.i2 = num_instances;

   p->ku.k.i3 = use_prompt;

 

   if (top)

     return (Scheme_Object *)scheme_top_level_do(instantiate_linklet_k, 1);

   else

     return (Scheme_Object *)instantiate_linklet_k();

 }

Ok, so this issues something. But what is the linklet from ‘startup_linklet‘?

 startup.inc: #define EVAL_STARTUP EVAL_ONE_STR(startup_source)

 startup.c:

 static Scheme_Linklet *startup_linklet()

 {

 #define EVAL_ONE_STR(str) return eval_linklet_string(str, -1, 0)

 #define EVAL_ONE_SIZED_STR(str, len) return eval_linklet_string(str, len, 1)

   EVAL_STARTUP;

 }

the startup source is a long string that looks like this

 "(linklet"

 "()"

 "((boot boot)"

 "(1/bound-identifier=? bound-identifier=?)"

 "(1/compile compile)"

 "(compile-to-linklets compile-to-linklets)"

 ...

Not sure how this works.

From here we need to look at eval_linklet_string, which calls scheme_compile_and_optimize_linket

 static Scheme_Linklet *eval_linklet_string(const char *str, intptr_t len, int extract)

 {

   Scheme_Object *port, *expr;

 

   if (len < 0)

     len = strlen(str);

   port = scheme_make_sized_byte_string_input_port(str, -len); /* negative means it's constant */

 

   expr = scheme_internal_read(port, 1, 1, -1, scheme_init_load_on_demand ? scheme_true : scheme_false);

 

   if (extract) {

     /* expr is a linklet bundle; 'startup is mapped to the linklet */

     return (Scheme_Linklet *)scheme_hash_tree_get((Scheme_Hash_Tree *)SCHEME_PTR_VAL(expr),

                                                   scheme_intern_symbol("startup"));

   } else {

     return scheme_compile_and_optimize_linklet(scheme_datum_to_syntax(expr, scheme_false, 0),

                                                scheme_intern_symbol("startup"));

   }

 }

 

 Scheme_Linklet *scheme_compile_and_optimize_linklet(Scheme_Object *form, Scheme_Object *name)

 {

   return compile_and_or_optimize_linklet(form, NULL, name, NULL, NULL, 0);

 }

Linklets are explained here http://blog.racket-lang.org/2018/01/racket-on-chez-status.html

From all this we now have a rough idea of how the interpreter works.

This leads us to ‘scheme_compile_linklet‘.

My major annoyance at both SBCL and Racket implementations is that they use some form of bytecode compiled from earlier versions making it neigh impossible to decode what’s going on