Elia Scotto

Shai-Hulud and the risks of external dependencies

Wed, 29 Jul 2026 00:00:00 GMT

Shai-Hulud is a malware created by TeamPCP that targeted npm to spread across the Node supply chain. Its name refers the giant sandworms that populate the desert planet Arrakis in the sci-fi series Dune by Frank Herbert. If you've seen the movie, they are the creatures that get attracted by vibrations in the sand. The reference is explained clearly in the README:

A sandworm surfaces in the desert. It takes everything.

An estimate of the damage caused by Shai-Hulud through multiple campaigns includes 500 compromised software packages, hundreds of organisations breached, 300 GB of data stolen, and roughly 500,000 credentials exfiltrated to the attackers.

TeamPCP decided to release its source code on 11 May. I’m not a security researcher, but I’ve always been interested in how criminals exploit complex systems and bypass their security. So I forked the repo to see what intrusion techniques or exploits were used. I also took some notes to turn into a code review article later. But the more I read through the code, the more I felt somehow disappointed.

I didn't find any advanced exploit system, but rather a collection of simple tricks to steal credentials, obfuscate strings, and inject malicious code into npm packages. At times, the code was inconsistent, contained a few bugs, and appeared incomplete.

So how did a codebase like this become so harmful? Because it exploited the Achilles' heel in modern development. The trust we place in other people's code and how much we love dependencies.

How it works

The codebase I’ve analysed is Mini Shai-Hulud. There have been three versions of the worm, with what appears to be an increasing number of features from one version to another.

The project uses Bun for its build process, and the README includes it's usage instructions.

Drop this into a CI pipeline. The sandworm crawls through your infrastructure, feeding on credentials from every provider it can reach, then exfiltrates through encrypted channels. If it finds npm tokens or OIDC access, it backdoors packages and publishes them. Downstream consumers install the infection. The worm grows.

Preflight and infection

Once the worm starts running, it first performs some checks, becomes a daemon if it is not running inside a CI environment, and compromises the OpenSearch repository if the infected system has access to it, using an OIDC attack.

The worm uses two separate methods to poison a package. The OIDC attack involves injecting a new optional dependency, @opensearch/setup, pinned to a specific commit in another package, opensearch-project/opensearch-js, which has presumably already been compromised. The other method adds a preinstall script containing a malicious setup file that downloads Bun and then runs the worm.

Credentials harvesting

Because the code has access to the host system, it simply reads configuration files and grabs tokens from processes and apps.

In short, it steals almost everything a developer can access: AWS credentials and secrets, SSH keys, cloud configuration files, cryptocurrency wallets, web chat data, VPN configurations, Kubernetes credentials, HashiCorp Vault secrets, GitHub tokens, Actions secrets, Claude settings, .npmrc, .netrc, Bash, Zsh, and Python shell histories, and MySQL, PostgreSQL, and Redis credentials.

Data exfiltration

Once the malware has collected the credentials, it needs to send them to the attackers. The data is first compressed and encrypted using a public key embedded in the payload.

The malware then tries to contact the typosquatted domain git-tanstack.com. If that fails, it searches GitHub for other compromised repositories through signed commit messages and retrieves a fallback domain from one of them. If even that fails, it creates multiple GitHub repositories with names made from random adjectives and nouns drawn from Dune. The stolen data is later uploaded to these repositories through a GitHub Actions workflow.

A quick GitHub search showed thousands of repositories containing encrypted JSON data. A significant number of these repositories were created only a few days after the release of Mini Shai-Hulud’s source code, making me wonder whether someone had used it in a new campaign.

Embedded payload

The payload contains a few scripts that download and install Bun on the system before executing the malware. Two of the files are configuration files for VS Code and Claude, designed to be injected into repositories so that whenever a user opens the project in the editor or with a coding agent, the worm compromises the machine and replicates itself. There is also a GitHub Actions workflow designed to collect repository secrets and upload them as artifacts. Then there are two public keys: one used to encrypt data and another to verify commit signatures. Finally, there is a Python script that dumps the memory of a GitHub Actions runner.

The payload contains also the DEADMAN_SWITCH, a Bash script that installs a daemon that once minute for the next 24h checks whether the GitHub token has been revoked, and if it has, wipes the home directory.

Obfuscation

The code is obfuscated at build time in two layers. The first makes important strings unreadable in the built code and decodes them only at runtime. This prevents a signature-based malware detector from being triggered by a string such as rm -rf ~, because it is transformed into something like LG+0RPNiAq6wTcBsei6MwLKLjyZ79e67. The second layer obfuscates the entire codebase with javascript-obfuscator library.

Imperfect code

I found a few issues in the codebase. Some expected files are missing, which could potentially prevent execution of a part of the malware. The bootstrapper exits if Bun is already installed, without executing the worm. BASH_LOADER and PYTHON_LOADER appear unused. The fallback-domain verification mechanism uses mismatched markers: it searches for thebeautifulmarchoftime but parses thebeautifulsnadsoftime, so verification with verify_key.pub likely never succeeds. I also found unused TypeScript interfaces and some catch and finally blocks left empty.

The credentials stealing and the DEADMAN_SWITCH script targets macOS and Linux systems. Code targeting Windows appears in the codebase, but it's unused. A good amount of Node developers use macOS, while CI systems usually run Linux, so if those were the main targets, maybe there was little reason to support Windows.

The README claims that the malware was vibe-coded, but it made me wonder whether an agent could have quickly fixed a few of these issues. The the source contains signs of work produced across multiple prompts: inconsistent abstractions, unfinished branches, unused interfaces, and code that appears to have been patched together from few sources, or precedent versions. The author seems fairly knowledgeable about security and build pipelines, but the project has been left in a prototype state.

The important lesson from this codebase it that it doesn’t have to be particularly sophisticated or complex to exploit a supply chain.

How this happened?

What we have ignored for years is that every time we run npm install, we give code written by others access to our machine and the network. It sounds nightmarish, but that’s what happens.

Every time you run a linter, whether directly or through your editor, compromised code can steal your credentials, access your servers and cryptocurrency wallets, send your data to a remote server, and push new versions of your repositories to GitHub with poisoned dependencies, spreading itself like a worm. Dependencies are insecure, and editor extensions are no exception[^1].

Node, the runtime behind most JavaScript projects, has security flaws built into its design, as its creator, Ryan Dahl, has himself acknowledged. We got used to writing web applications faster by relying on a large number of dependencies and on a network of trust, expecting maintainers and the open-source community to do a good job. Very large companies were built using these technologies.

While forward-thinking companies developed the habit of giving back to the community by open-sourcing their code and financially supporting maintainers, we also saw signs that this equilibrium was not always stable.

TeamPCP decided to attack developers because, in some respects, we became an easy target. Also the credentials on our machines are more valuable than ordinary users’ data. Now that writing imperfect but runnable code has become extremely cheap thanks coding agents, we should stop blindly trusting third-party code without security measures in place.

What can we do?

The creator of Odin, Ginger Bill, argued last year that package managers are evil. I recommend reading the article because it’s very interesting. The key point for this discussion is that we tend to trust others and therefore “assume random code off the internet works”.

We have become used to believing that code published online is reliable, and that relying on third-party packages whenever possible is efficient[^2]. We hid the dependency hell behind automated systems, making it nice and easy to use, but almost impossible to verify what code we are actually importing. If a dependency nested four layers down gets compromised, developers will install and run it, without even realising what happened.

Token protection, credential rotation, CI monitoring, delayed dependency updates, agentic auditing and supervision can reduce the risk, but they do not remove the design problem. Node, Python, Rust, Ruby and Go all rely on third-party dependencies whose code execute with the same privileges as the application, without being isolated from the rest of the system.

Going forward, we should seriously consider adopting a zero-trust mindset when developing software.

npm has recently removed the ability for packages to run their own scripts during installation. As a result, it will no longer execute preinstall, install, or postinstall scripts from dependencies, which helped previous versions of Shai-Hulud spread.

One effective approach often proposed online is to reduce the number of dependencies to a minimum. Prefer mature, actively maintained packages with a limited dependency tree and a long, stable release history. This would reduce the probability of being attacked through small, obscure dependencies.

Also we can consider choosing programming languages that provide larger standard libraries and runtimes, so we can write code faster without the need to import many external packages.

When dependencies are necessary, they can be distributed with a reliable release model closer to those used by linux distributions like Debian, with different channels such as stable, testing, and nightly. This would allow code to be audited by a larger community of maintainers, preventing frequent release.

A stronger protection comes from limiting access to third-party code. This has been successfully applied by Apple in macOS, by requiring permission before they can access sensitive folders or other apps’ data.

A more effective approach is to sandbox third-party dependencies. The code runs inside an isolated and restricted environment, preventing it from compromising the system.

These techniques have been partially addressed by the new JavaScript runtime Deno, another project created by Node creator Ryan Dahl. Deno implements runtime permissions and package sandboxing. It provides dependency auditing, minimum-age controls, a larger built-in toolset, and it's almost entirely compatible with npm packages. It does not eliminate the problem of trusting external dependencies, but it reduces the amount of permissions granted to them by default.

Whatever we choose, we should start considering every single dependency as a new tradeoff. We need to become responsible for it, rather than expecting someone else to take that responsibility.

[^1]: One of the latest attack from TeamPCP in May, involved a GitHub developer installing a malicious software extension for VSCode, which ended up in almost 4000 private repositories violated. [^2]: is-string has 83K weekly download.

A Road to Lisp: Which Lisp

Fri, 17 Jul 2026 00:00:00 GMT

Most programming languages evolve as a single language. Python, Java, Javascript, C++, have new versions and standards, multiple implementations, but they still remain the same language. C and C++ can be compiled with GCC or Clang, Python can be compiled with CPython or PyPy, the same JavaScript runs in both Firefox and Chrome, and Java programs can run on JVM or GraalVM.

Lisp is not like that.

The Wikipedia page lists more than 20 different dialects, which means there are many variations to choose from. That’s because Lisp is a family of programming languages. They share the same fundamental syntax but differ in their operators, semantics, standard libraries, and language capabilities.

One of the main concerns Lisp beginners have is which dialect to learn first. I see this question frequently asked in online forums. The answer is that the dialect matters, but not as much as a beginner might think. Learning Lisp is about learning a new type of programming. A new way of thinking about problems using code. You will learn the fundamental concepts with any dialect. Then, once you have learned one, it will be relatively easy to switch to another.

I will briefly present the most relevant dialects that are actively used and maintained. I’ll try to highlight their strengths and weaknesses to help you overcome your indecision and pick one to start your Lisp journey.

If you are a beginner, many of the concepts in this article might be completely new to you. Don’t worry too much about them yet. When I was a beginner, I found many of these concepts fascinating and they pushed me to learn more about the Lisp world.

Common Lisp

Abbreviated as CL, it is the most mature and comprehensive of all Lisp dialects. It’s considered the old-school Lisp, since the language — I will call it a language from now on — was standardised in 1994 with a formal ANSI specification. Thanks to this standardisation, there are multiple implementations of Common Lisp targeting different platforms and use cases.

The most famous implementation is SBCL, which compiles directly to native code. It is fast, open-source, and compatible with modern hardware. With it, well-written Common Lisp code can achieve performance comparable to C and Rust. Because SBCL optimises heavily, it compiles a bit more slowly than other implementations but generates some of the fastest code in the Lisp family.

When I said above that Common Lisp is the most comprehensive dialect, I meant that it offers the broadest set of features among all Lisps. It provides a large amount of functionality out of the box, much of it defined directly in the standard. For example, CL has functions for controlling compilation and evaluation from within the language itself (COMPILE, LOAD, EVAL, COMPILE-FILE, and others). This means that I can use these functions in my code or at the REPL to tell the Lisp process to compile a function, load a file, or evaluate some code. CL also provides DISASSEMBLE, which lets you inspect the machine code generated for a compiled function.

Common Lisp has a condition and restart system, which is one of the most powerful way I ever saw to debug and inspect programs. If a condition occurs during execution, the Lisp process might stop and let you inspect the state of the program and its variables at that point in time. You can then choose to restart the program and maybe retry the operation, or ignore the condition and continue execution. This is also possible because the Common Lisp REPL is deeply integrated with the running system. If your program fails on a remote server, you can connect to its REPL and inspect the live process to understand what went wrong.

Common Lisp supports all major programming paradigms, like functional, imperative, metaprogramming, and object-oriented programming. It offers one of the most advanced object systems, called CLOS, which supports features such as multiple dispatch and generic functions. This makes OOP more flexible than in common object-oriented languages such as Java or C++. It is also worth noting that Common Lisp is dynamically typed but has a rich and expressive type system with optional type declarations.

Standardisation means that the language is stable. The ANSI Common Lisp from 1994 is still the one in use today. Because of it, lispers rarely fall victim to backward incompatibility, and old Common Lisp code often still runs perfectly today. For example, if you go through old Lisp books such as PAIP by Peter Norvig, published in 1991, you’ll find that much of the code still runs on modern implementations. With Common Lisp, you encounter far fewer of the incompatibilities typical of languages like Ruby and Python. Even Common Lisp libraries that haven’t been updated recently will often still run fine on your system because they don’t need to be continually adapted to new versions of the language.

Common Lisp lacks some conveniences that have become common in newer languages, such as concise literals for a wider variety of data structures, persistent immutable collections, lazy sequences and built-in general-purpose pattern matching. Common Lisp was designed in the 1980s by consolidating several existing Lisp dialects into a single language. Its designers chose to retain much of the syntax and patterns of those older dialects, making sure that existing Lisp programmers could adopt the language without having to learn a completely new syntax.

Common Lisp doesn’t have a central direction or corporate sponsor. The community is relatively small and spread across several implementations, projects and communication channels. It’s mostly made up of volunteers who help keep the language alive. It can be a bit difficult to find guides and help: tutorials are often either too simple or too advanced, and documentation is not always easy to find.

It’s currently used in quantum computing at Rigetti Computing, and has been used at Grammarly for its core grammar service and also powers Google Flight Search. Worth mentioning is Kandria, an open-source video game released on Steam and written entirely in Common Lisp.

An interesting fact: Paul Graham originally wrote HackerNews in a custom Lisp dialect called Arc, based on Racket. Today, Daniel Gackle (@dang) reimplemented it in Clarc, a Common Lisp implementation of Arc that runs on SBCL^1, serving around 10 million pages per day.

Where Common Lisp shines

CL compiles to native code on all major operating systems and can achieve fast execution, at the cost of slightly slower startup times, so it’s great for long-running processes. It offers one of the most powerful REPLs among Lisp dialects and can be considered, despite its steep learning curve, one of the fastest languages for writing solid software in a short time.

CL shines when you need to do research and prototyping with quick iterations, as in quantum computing, or when the specifications aren’t fully defined yet and change frequently, requiring the software to adapt quickly, as in startups. Paul Graham wrote a famous and inspiring article about how Common Lisp helped his internet startup beat the competition.

Learning resources

A good starting point is A Road to Common Lisp by Steve Losh, which inspired me to write this series. One of the most accessible books for beginners is Practical Common Lisp by Peter Seibel, which is available entirely online. It’s the best introduction to the language I’ve read, as the most important concepts are neatly separated into chapters. A more playful introduction is Land of Lisp, which uses funny characters and comics to teach programming through games. A more advanced book that I enjoyed is On Lisp by Paul Graham, which is entirely dedicated to macros.

The best IDE support for CL is Emacs + Sly/Slime (Doom already has support for it) or Vim + VLime. A weaker alternative is VSCode + Alive.

One of the best resources for beginners is The Common Lisp Cookbook, which has many up-to-date tutorials. For references and libraries, the Awesome-CL list is worth a look. I also printed my own copy of the Common Lisp Quick Reference booklet, which can be faster than finding information in the CL HyperSpec.

Clojure

Rich Hickey created it during a sabbatical year, frustrated by clients refusing to let him use Common Lisp instead of Java and C# because they were concerned about compatibility and maintenance issues. So he decided to write a new Lisp language targeting the JVM, allowing it to access everything the Java ecosystem had built over decades and making it a practical language from the start. The deal is that Clojure provides the language, while the host platform — the JVM — provides the runtime.

Clojure compiles directly to JVM bytecode, which means it’s compatible with any codebase that runs on the JVM (Java, Kotlin, Scala). This compatibility means that Clojure can coexist with other JVM languages in the same codebase, use Java libraries directly, and access all the features the JVM offers, such as OS portability, garbage collection, a fast and optimised runtime, and so on.

Rich Hickey chose to target the JVM so that he didn't have to reinvent the wheel and build a new compiler for Clojure from scratch. Instead, he focused his efforts on reading research papers on language design and data structures, bringing some of the newest and most effective ideas into the language.

The language at its core is composed of pure functions operating on immutable data structures. Instead of making mutable objects the primary way to represent changing state, it provides explicit mechanisms such as atoms, refs and agents for managing change. This design choice makes Clojure a functional programming language. It supports object-oriented programming, but replaces class-based inheritance with abstractions such as protocols and multimethods. Also the syntax is more modern than Common Lisp’s, with richer literals, destructuring, and overall more compact code.

Clojure also comes with several new features. Its persistent data structures use structural sharing, making immutable updates efficient. It also uses lazy sequences, which allow computations to be performed only when their results are needed. It has strong support for concurrency through constructs such as atoms, refs, agents and futures, and provides records and custom data types when more structured data is needed. EDN (Extensible Data Notation) offers a simple extensible data format based on Clojure’s literals, a better lispy alternative to JSON, YAML and similar. Clojure doesn't have types, but clojure.spec can be used to describe, validate and generate data without introducing a static type system, useful in large applications.

Rich Hickey designed Clojure with the intention of prioritising stability, similar to what standardisation did for Common Lisp. Even though the language is actively developed, you don’t need to expect major deprecations when updating to a newer version, since the maintainers preserve backwards compatibility and mostly add new features through core libraries.

Since Clojure is designed to be hosted on other platforms, it can be ported to other runtimes, like the JavaScript engine that runs in your browser. ClojureScript is a Clojure compiler that targets JavaScript rather than the JVM, so the same language can run on both the server and the browser. This means that Clojure can be used as a full-stack language.

The JVM integration is great, but it comes with some trade-offs. Error messages are contaminated with a lot of Java implementation details and aren’t always self-explanatory, so stack traces are not necessarily helpful when debugging issues. Java profilers like YourKit are compatible, but even there you will see a lot of Java classes, and it can be hard to map them back to the code you wrote.

In general you don’t need to know Java to learn Clojure. I didn’t know Java when I started and never had substantial problems because of that. However, knowing how to tweak the JVM can be helpful when doing performance tuning on the server.

The community around Clojure is small but active and happy to help newcomers pick up the language quickly. It’s known as Clojurians and mostly hangs out on Slack, where you can also collaborate on open-source projects and find job opportunities.

Clojure is actively used by companies and startups around the world, and it’s probably the most widely used Lisp in production. Some big names include Nubank (one of the largest fintech companies in Latin America), which acquired Cognitect (the main sponsor of Clojure’s development), as well as Walmart, Netflix, Apple, Salesforce, Amazon, Cisco, and Grammarly.

Where Clojure shines

Clojure shines in large software systems where the workload consists of a lot of data processing, or more broadly, anywhere you might previously have picked Java. Code can be written and tested quickly through continuous iteration thanks to the REPL, making the language suitable for delivering reliable solutions in less time. Its immutable design helps eliminate some of the hardest concurrency bugs, while its functional orientation guides programmers towards writing programs as a series of data transformations.

Clojure is great in fields where operations need to happen quickly and often involve processing large sets of data, such as finance and trading. It’s also great for startups solving domain-specific problems with complex logic, since the language’s extensibility allows it to be modelled around the problem.

Learning resources

To give Clojure a try, you might want to check out Try Clojure, then solve some exercises like the Clojure Koans, followed by 4Clojure for more advanced stuff.

The official website offers some of the most curated material you can find, with great explanations of all the core features and the motivations behind the design choices that make the language special. Clojure also has one of the best documentation websites I’ve ever found for a language. All the core library functions come with docstrings, and the community has added usage examples, so you can quickly grasp what each function does.

The best pitch for the language is Clojure Distilled. It’s probably one of the shortest explanations of what Clojure has to offer.

After the release of Clojure, Rich Hickey became famous for a series of talks about software design, in which he shared his deep expertise in fundamental topics like simplicity in software design, state and time, databases, concurrency, and language design. Some of his best talks — Simple Made Easy, The Value of Values, Hammock-Driven Development, Are We There Yet?, and Design, Composition, and Performance. I highly recommend spending some time watching them, whether you choose Clojure or not.

Some good books are Clojure for the Brave and True by Daniel Higginbotham, which is free to read online and good for beginners. The one I bought is The Joy of Clojure, which is recommended for programmers who come from OOP languages like Java.

For editors, VSCode has Calva, a complete and rich extension for Clojure. If you use Emacs you can use Cider, plus there are extensions available for most editors/IDEs.

Racket

Racket is a modern dialect descended of Scheme. While other Scheme dialects are generally small languages, Racket has evolved into a fully featured language with its own unique features.

The code is compiled on every major platform and the language has been used in a lot of fields to create web applications, graphical interfaces, database integrations and mostly to quickly develop tools with graphics.

Racket is language-oriented, which means it’s designed for creating new programming languages. Every source file begins with #lang, which defines the language used by that module. You can create entirely new languages with their own syntax and semantics, while still using the Racket ecosystem underneath. Other Lisp dialects can easily build DSLs, but they are tied to Lisp. Racket advances the concept by making it easy to define completely new languages, without requiring them to use Lisp syntax.

Racket also comes with a wide range of features. It includes libraries for cross-platform GUIs, web servers, concurrency and parallelism, regular expressions, pattern matching, classes and objects, and an FFI for calling C code. Its package manager is integrated into the ecosystem, and a large collection of libraries is immediately available. Compared with other Lisp dialects, there is generally less setup required before you can start building something substantial.

Racket offers an advanced macro system, providing the same kind of macro support as other dialects, plus hygienic macros. Hygienic means that identifiers introduced by a macro don’t accidentally capture those in the surrounding code. This makes macros safer to write, especially for beginners, that don’t have to rely on gensym like in the other dialects.

Static typing is supported with Typed Racket, a typed variant of the language that lets programmers add type annotations and have their code checked before execution. Typed and untyped Racket modules can also work together, so typing can be introduced where it is useful without requiring an entire program to be rewritten.

Even though Racket comes from the Scheme tradition and has strong functional programming roots, it supports multiple paradigms. Programs can be written using imperative programming, objects and classes, functional techniques, and metaprogramming.

The language installation comes with DrRacket, a great IDE with everything ready to use the language, available on all major platforms. It combines an editor, REPL, and debugger, in a single programming environment for writing and testing Racket code.

The main downsides are its small ecosystem and limited use in the industry. There are fewer libraries, projects and jobs compared to more popular languages. Performance and deployment can also be weaker points, especially compared with other compiled dialects.

Where Racket shines

Racket shines when designing new programming languages, building compilers and interpreters, or experimenting with new language features. This makes it particularly popular in computer science education and university research. Also, since it’s language-oriented, it’s a great choice when a program needs to provide a simple DSL, for example for customisation, without forcing the user to use a Lisp dialect.

Racket is ideal for prototyping, scripting, and projects where having a comprehensive set of libraries and development tools immediately available is important. It’s worth noting that other dialects offer GUI support through external libraries and wrappers, while in Racket it is available out of the box.

Learning Resources

Racket has a rich website with a lot of helpful information. Important resources for beginners include Quick: An Introduction to Racket with Pictures and The Racket Guide, which is the complete guide to the language. Also worth mentioning is Beautiful Racket by Matthew Butterick, a visually curated guide to the language written by a passionate contributor. If you’re looking to write a compiler, check out this course using Racket.

The best resource available for Scheme is SICP, also known as the Wizard Book. It’s not only about Lisp, but it’s a great book for learning the fundamentals of programming and programming languages. It’s also worth mentioning a rather unusual book, The Little Schemer, which consists of a long series of questions that forces the reader to think before looking at the answer on the page.

Special Mention

Elisp

Elisp is a specialised dialect that’s part of Emacs and is used to customise the editor. It’s an old Lisp dialect that comes with quite a few limitations, but it’s probably one of the best practical uses I’ve ever found for Lisp, beyond using it as a general-purpose programming language. It’s used to customise and extend the functionality of Emacs, since much of the editor itself is written in Elisp. By evaluating Elisp code inside it, you can change how Emacs looks and behaves in real time, without requiring a reload.

Learning resources

An Introduction to Programming in Emacs Lisp is freely available online. Another useful resource is Emacs Lisp Elements by Protesilaos Stavrou, an active Emacs contributor and advocate.

Syntax comparison

We can use a simple instruction interpreter to show some of the syntactic differences between the dialects.

(defun calculate (instructions)
  (loop with result = 0
        for (operation value) in instructions
        do (setf result
                 (case operation
                   (add      (+ result value))
                   (subtract (- result value))
                   (multiply (* result value))))
        finally (return result)))

(calculate '((add 5) (multiply 3) (subtract 4))) ;; => 11

In Common Lisp, I used the LOOP macro to iterate over and destructure each instruction, then perform the operation by updating local state. Notice that it carries an accumulator (result) through the sequence, essentially doing the same job as a reduce.

(defn calculate [instructions]
  (reduce
    (fn [result [operation value]]
      (case operation
        :add      (+ result value)
        :subtract (- result value)
        :multiply (* result value)))
    0
    instructions))

(calculate [[:add 5] [:multiply 3] [:subtract 4]]) ;; => 11

In Clojure, I used a reduction over the sequence of instructions. Notice that it doesn’t use or update any local state, since the reduction uses immutable accumulation. I also used some syntax that is specific to Clojure, such as vectors [] and keywords like :add, and destructured the contents of each vector with [operation value].

(define (calculate instructions)
  (for/fold ([result 0])
            ([instruction (in-list instructions)])
    (match instruction
      [(list 'add value)
       (+ result value)]
      [(list 'subtract value)
       (- result value)]
      [(list 'multiply value)
       (* result value)])))

(calculate '((add 5) (multiply 3) (subtract 4))) ;; => 11

In Racket, we use for/fold to iterate over the instructions while carrying the result from one iteration to the next. We also use pattern matching with match to destructure each instruction and identify the operation at the same time. For example, (list 'add value) matches a two-element list whose first element is the symbol add and binds the second element to value.

Which one to pick?

To summarise the above:

Clojure gives you modern syntax, functional programming and immutable data structures, great tooling and libraries, an almost drop-in replacement for Java, a full-stack language, a strong community, and maybe even a job.
Common Lisp gives you native compilation, high performance, the most powerful REPLs, multi-paradigm programming, and a stable, battle-tested language.
Racket gives you a feature rich language, a great IDE for beginners, a cross-platform GUI, a powerful platform for building new languages and DSLs, easily approachable for students.
Elisp gives you the ability to extend and customise Emacs.

For most programmers looking for a practical and elegant Lisp they could use professionally, Clojure is probably the safest first choice.

If you want to experience the traditional Lisp development model at its fullest, you want native compilation without rely on JVM, an exceptionally interactive environment and a huge language, choose Common Lisp.

If you are a computer science student, interested in compilers and programming-language design, or if you need a quick way to write tools or need an easy way to use a cross-platform GUI, choose Racket. This is probably the easiest to start with between the dialects.

If you have any questions of feedback, please send me an email.

A Road to Lisp: Why Lisp

Thu, 09 Jul 2026 00:00:00 GMT

The question that most programmers face when seeing some Lisp code for the first time is, without doubt, “what the hell is this?”. I asked myself the same thing when I first read its unconventional syntax: all those parentheses, the weird indentation, and who thought to use the first argument of format to print to stdout?

(defun flip-coin-for-real ()
  (<= (random 100) 80))

(defun hello-lisp ()
  (write-line "What is your name?")
  (let ((name (read-line))
	    (learn-it (if (flip-coin-for-real)
		              "should"
		              "should not")))
    (format t "Hello, ~A.~%" name)
    (format t "The Oracle said... you ~A learn Lisp!~%" learn-it)))

After getting comfortable reading code with so many parentheses, I had to learn how to use packages and symbols, how to create new projects and import libraries, how to use the REPL, and how to use conditions and restarts. Most importantly, I had to switch to a new way of thinking when constructing algorithms.

The Lisp journey has a steep learning curve compared to most common languages. But it can also unlock skills that those languages never will. Why? Because in Lisp you can do things that are not possible in other languages. Lisp enables new possibilities and allows algorithms to take different shapes, giving the programmer more power and flexibility.

Paul Graham coined the term Blub paradox to explain why it is difficult for programmers who have only used less powerful languages to understand the power of Lisp. In short, it’s because they are missing the concepts needed to perceive what is lacking in those languages and the advantage Lisp has over them. Louis Armstrong said something similar about jazz:

If you have to ask what jazz is, you'll never know. — Louis Armstrong.

In this article, I'll explain a few features that show why this special language is worth learning. It's not an easy path, and to truly grasp its power you'll need to use Lisp yourself. And even if you don’t end up using Lisp, you will gain a different perspective on what programming languages can do.

Lisp is going to make you a better programmer because it changes the way you think about problems using code. To be more specific, it will teach you an approach that is not possible with other programming languages. You will be able to program Lisp itself and thus adapt the language you use to the problems you are solving. Using Lisp, you will learn to grow the language toward your problem, then write the program in that language.

Extensibility

Lisp is extensible within itself. Experts — called lispers — commonly refer to it as the programmable programming language[^1]. Not only will you write code for your programs, but you can write code that extends Lisp itself.

This is possible thanks to the macro operator. If you've used macros before in languages like C, Rust, or Swift, don’t expect the same thing. Those macros are primarily a way to eliminate boilerplate or generate repetitive code. Beyond that, macros in Lisp allow you to create new constructs that become part of the language itself.

They are one of the hardest features to master for Lisp programmers, so I won't try to teach you how to use them or how they work here. I just want to give you a taste of what they do. For example, C programmers might want to use the while operator to define a simple loop. Common Lisp does not provide it, so the programmer can write a macro and add it to the language.

(defmacro while (condition &body body)
  `(loop while ,condition do
     (progn ,@body)))

The new while macro executes a series of instructions (the body) for as long as the condition is true. It uses loop, which is itself a macro used to write complex iterations (notice that it uses a while symbol as well). progn is a special form that takes multiple expressions, runs them in order, and returns the result of the last one.

Instead of writing (loop while ... do (progn ...)), we can use the new while operator, shortening the code and making it similar to C. We just extended the language available to us.

(let ((counter 3))
  (while (> counter 0)
    (print counter)
    (decf counter)))

;; 3
;; 2
;; 1

The beginner might not grasp what is really magical here, since, other than the unique syntax, defmacro looks pretty similar to defun, which was used in the first code example to define new functions.

To better understand the difference between the two, let’s define a new while as a function, then call it.

(defun fake-while (condition body)
  (loop while condition do
    (funcall body)))
    
(let ((counter 3))
  (fake-while (> counter 0) 
	(progn
	  (print counter)
	  (decf counter))))

loop is the same macro we used before; funcall calls the function received as the first parameter, in this case body. Try to execute the code above in a Common Lisp REPL. You will receive a condition of type error, similar to this: The value 2 is not of type FUNCTION.

This happens because the arguments to fake-while are evaluated immediately. (< x 3) evaluates to t -- true. Because progn is a function, its arguments are evaluated. (print counter) prints 3, and (decf counter) returns 2. Since progn returns the value of its last expression, body becomes 2. So fake-while receives t as its first argument and 2 as its second, not a block of code. It will enter the while since the condition is true, and then funcall expects body to be the function to be called, but body is the value 2. This raises the condition above. The code executed by the compiler is the one below. (funcall 2) raises the condition.

(loop while t
	  do (funcall 2))

The macro while defined earlier works because it keeps the arguments — condition and body — intact, without evaluating them until needed. We can inspect the transformation that a macro operates in the REPL. This transformation is called expansion. macroexpand is a special Lisp command we can use to expand code.

CL-USER> (macroexpand-1 '(while (> counter 0) (print counter) (decf counter)))
(LOOP WHILE (> COUNTER 0)
      DO (PROGN (PRINT COUNTER) (DECF COUNTER)))

The code returned from the macro expansion is the one received and executed by the compiler. We can see that this time the while macro preserved the arguments so they will be executed together with the rest of the code.

A macro, unlike a function, does not evaluate the arguments in advance. Instead, it treats them as pure data. This is possible because (almost) everything you see in Lisp is made of lists. They are the main data structure, and they are used to write code.

It’s lists all the way down

Programs in Lisp are composed of a series of symbolic expressions (abbreviated s-expr). Expression is a mathematical term meaning anything that evaluates to a value; symbolic means that expressions are created using values and symbols.

An s-expression is one of two things:

an atom, which is a unit of data (a number, a string, a symbol, etc)
a list, which is a collection of elements that can be atoms or other lists

1           ;; An atom number
"hello"     ;; An atom string
'(1 "y" :c) ;; A list of atoms (a number, a string, a keyword)
(+ 1 2)     ;; A list composed of a symbol (plus) and two numbers

Lisp means LISt Processing, so the purpose of the language is to process the series of instructions expressed as lists that compose the program.

Since lists are used both as the main data structure and to write the code, we can extract an interesting property of the language. In Lisp, code-as-data means that both code and data are expressed using lists. This property is called homoiconicity.

CL-USER> (+ 1 2)
3
CL-USER> '(+ 1 2)
(+ 1 2)

I’ve used two lists with the same content; the only difference is that I put ' in front of the second one, but the output is different. The first list is treated as code. It gets evaluated, and specifically the symbol + maps to a function that performs the sum of the arguments. Lisp uses what’s called Polish notation, where the function name is the first element of the list and what follows are the arguments. So instead of writing print("hello"), we write (print "hello"), moving the function name inside the parentheses. The output is 3, which is the result of the code execution.

The second list is treated as data. ' is an operator that tells the interpreter not to evaluate the following list and keep it as manipulable data. The output is the same list, not evaluated.

The subtle distinction between code and data is what makes macros possible. We can manipulate code as we manipulate data. By transforming the source code, we are able to write programs that write programs, a capability other languages have spent decades trying to imitate.

Extending the language means that by creating new constructs that are more expressive for your program's specific scope, you can make your code more concise. Repetitive boilerplate code can be replaced with custom macros to shorten the code and save time. New control structures can be created to control resource access and the evaluation of forms. Macros can generate code, improve performance, and, most importantly, create new syntactic abstractions that are easier to use to hide complex and error-prone code.

A live system

Lisp is not only a programming language, it's a live system. To clarify this point, let’s start with how different the workflow is between Lisp and other languages.

With other languages, you open the project in the editor and start writing code. The first thing a lisper does is start a Lisp process, attach it to the REPL, and then load the project into that process.

The Read-Eval-Print Loop is an interactive environment used to evaluate code and immediately see the result. It's a window into the Lisp living process that is currently running your program.

In other languages, once you change the code, you have to stop and compile it, then run the project to observe, test, and debug the changes. If something's wrong, you go back to the write-compile-run-debug cycle.

Instead, a lisper continuously evaluates code in the running process and observes the output directly in the REPL, since the Lisp process is the running program. Every single function can be immediately tested in the REPL as a single unit of code, as well as any other action useful for development, such as querying a database, inspecting variables, or debugging.

A Lisp process is usually kept alive for weeks on end since there is no need to stop it. It can stay attached to the REPL in the editor for as long as needed. New functions, macros, and variables are continuously defined and redefined, a process called binding within an internal memory called the environment. This particular workflow even has a name: REPL-driven development. This is a real killer feature. It has been available in Lisp since its invention in the 60s, and it has been imitated by other languages (where possible), like many other Lisp original features.

Does the word hot-reloading tell you anything? I’ve been lucky since I started my career working with web frontends, where the JavaScript code of the app is hot-reloaded with every hard refresh of the page. And thanks to modern tools, I now only need to save a file and the webpage will automatically refresh itself with the new changes. Under the hood, hot-reloading means there is a process watching the files, recompiling the modules that changed, and a websocket connection is used to inject the changes silently into the page. But not all software stacks are that lucky. Some backend frameworks still require recompilation, the situation is mixed for desktop and mobile development, and low-level languages require full compilation every time.

In Lisp, hot-reloading is not a separate tool or technique. It comes for free when evaluating code in the live environment. Every time I redefine a symbol, it picks up the new definition, whether it’s a function, data, or a macro. I don’t need to stop and compile, run tests separately, or use some special debugger. Everything comes automatically by evaluating in the REPL. A good analogy is that programming in Lisp is like evolving a program rather than constructing it. Once a new Lisp process has started and code gets evaluated into it, that system will become the program itself.

Lisp allows you to approach programming in a new way. You can evolve the programs while writing them, and at the same time, as discussed before, evolve the language to your needs. A Lisp CAD program has evolved by turning Lisp into a language that writes CAD programs. A high-frequency trading system has evolved with a language adapted to write similar trading systems. This is the nature of Lisp, and it offers many advantages.

Extensible software

This is something probably even less known, but the two features explained above (language extensibility and the Lisp live system) allow you to easily create extensible software.

We're used to desktop programs being extensible mainly through plugins, such as in text editors, or sometimes through scripting languages for customization, as in game modding. These solutions usually work well, but they require programmers to design and maintain an entire system for extensibility, and they require users to learn a specific set of APIs to write plugins or extensions. In Lisp, this comes almost for free.

While extending the language, lispers write macros to encapsulate actions and make code easy to reuse. In doing so, beyond just creating new symbols, they are effectively writing a DSL, a domain-specific language for the program. When lispers need to make their software extensible, all they need to do is let users use the DSL they've created.

To understand this concept, let’s look at some use case scenarios. I wrote a custom CMS server for clients’ websites. I wrote some macros for generating webpages server-side.

(html
  (:h1 "Lorem Ipsum")
  (:p "Dolor sit " (:b "amet") "."))

I can expose the same DSL I’ve written internally to the users so they can write their own views and customise their CMS instance.

;; A user can write dynamic pages using the full power of Lisp
(let ((user-name "Alice")
      (items '("Apples" "Bananas" "Oranges")))
  (html
    (:h1 (format nil "Welcome back, ~A!" user-name))
    (:ul (dolist (item items)
           (html (:li item))))))

Notice how the html macro seamlessly integrates with Lisp's variables (user-name), loops (dolist), and string formatting (format). In a traditional templating language, you'd need to learn a separate syntax with special brackets like {{ user_name }} or {% for %}. In Lisp, you don't need a new language since you just use Lisp itself to build your templates.

This integration means your DSL inherits all the power of Lisp: conditionals, recursion, higher-order functions, and even the ability to debug your templates step-by-step in the REPL.

Another scenario: I wrote a math software where users can write formulas and the program renders them on a Cartesian graph. So I wrote a DSL to write math and draw. I can expose a small subset of the language to users so they can draw directly without having to use math formulas.

(graph
  (:curve (lambda (x) (* x x)) 0 0)     ; plot y = x²
  (:point 0 0))                         ; mark the origin

With this DSL, users can draw complex mathematical functions with minimal code. Since it's just Lisp, they can use variables, define helper functions, or even generate multiple curves in a loop. This gives the users a clean interface while leaving the complex rendering code safely hidden away.

This technique for extending programs has been used successfully. AutoCAD uses AutoLISP to automate repeating tasks and create complex geometries. Emacs is the most extensible text editor that has ever existed on this planet, and most of it is implemented in its own Lisp dialect in order to make it infinitely expandable. There's a running joke that hardcore Emacs users never have to leave the editor because it already contains everything they would ever need. In fact, Emacs has been extended to be a PDF reader, a web browser, an email client, an RSS reader, a terminal multiplexer, a Git client, a music player, a chat client, a spreadsheet, an IRC client, and even a window manager for your entire desktop, all thanks to using Lisp under the hood[^2].

The most powerful programming language is Lisp. If you don't know Lisp, you don't know what it means for a programming language to be powerful and elegant. Once you learn Lisp, you will see what is lacking in most other languages. — R. M. Stallman creator of GCC, Emacs, the GNU project.

So why Lisp (or when)

A lot of people have argued that one day in the future there would be a time for Lisp. A time where it becomes popular, almost the default language, when the majority of programmers will come to appreciate what a fantastic experience it is to work with it. A time when Lisp would be taught at universities all over the world so companies would have no issues finding lispers, and engineers would become unafraid of incompatibility or performance issues. Programmers would joyfully spend their days using REPL-driven development, and most of the software we use would become extensible using variations of a single language.

This time never happened, and probably never will.

Still, Lisp has survived from the 60s, becoming the second oldest programming language still in use after Fortran. So many languages were inspired by it, and so many languages are still being created, missing some of the features that helped Lisp survive. So even if there isn't a time for Lisp, that doesn't mean it's not a powerful language worth knowing.

Better programmers than me have said that it's hard to justify Lisp's use by picking a single feature among its extensibility, its interactive environment, the REPL, and a lot of other features we haven't touched yet. It is the combination of all of them that makes Lisp programming what it is.

So get ready to learn what the most powerful programming language truly is.

[^1]: This quote belongs to John Foderaro, co-founder of Franz Lisp. [^2]: To be fair, some people argue that Emacs is "a great operating system, lacking only a decent editor".

A simple Common Lisp web app

Wed, 30 Apr 2025 00:00:00 GMT

One of the drawbacks I’ve found when dealing with Common Lisp is the lack of documentation available. Too often, I find published libraries without an explanation of how they are meant to be used or only partially documented, and I need to dig into the source code to understand what they do and to see all the functions available. Even though reading source code is a proven technique to improve one's grasp of a programming language, most other systems come with extensively documented libraries, something appreciated by beginners and a factor that often contributes to a language's popularity.

In my opinion, this lack of good documentation is one of the reasons Common Lisp is often seen as challenging for beginners, which makes it harder for the language to become popular.

Some time ago, when I looked for guidance on writing a generic web app, I was surprised by the absence of a quickstart page to help me set up a simple server—something the Python community has provided for Flask for many years.

“The most underrated skill to learn as an engineer is how to document. Fuck, someone please teach me how to write good documentation. Seriously, if there's any recommendations, I'd seriously pay for a course (like probably a lot of money, maybe 1k for a course if it guaranteed that I could write good docs.)” A drunk dev on reddit

So I put together a short tutorial on how to build a simple web app in Common Lisp, inspired by the Clojure tutorial written for Luminus. The goal is to write a guestbook demo, involving template rendering, connecting to a database to run queries, and exposing routes to the webpage.

To follow the tutorial, you need a Common Lisp implementation, like SBCL, with Quicklisp, the dependency manager. I also recommend having a REPL integrated with your IDE, like SLY for Emacs or Alive for VSCode.

I’ll use more modern CL libraries that have an interface similar to other languages, so it might be a bit easier to follow along.

The server

First things first, I created a new Common Lisp project. To do that with a simple boilerplate, I loaded cl-project in the environment and called the function make-project with a path and a name for the project.

> (ql:quickload :cl-project)
To load "cl-project":
  Load 1 ASDF system:
    cl-project
; Loading "cl-project"
..
(:CL-PROJECT)

> (cl-project:make-project #P"~/guestbook/" :name "guestbook")
writing ~/guestbook/guestbook.asd
writing ~/guestbook/README.org
writing ~/guestbook/README.markdown
writing ~/guestbook/.gitignore
writing ~/guestbook/src/main.lisp
writing ~/guestbook/tests/main.lisp
T

Then I ran this command to let Quicklisp know where my new project is located.

> (pushnew #P"~/guestbook/" asdf:*central-registry* :test #'equal)

I declared the required libraries in the :depends-on property so that Quicklisp would download them from the repo and load them into the environment.

(defsystem "guestbook"
  :version "0.0.1"
  :license "MIT"
  :depends-on (:alexandria        ;; utils
               :uiop
               :cl-ppcre          ;; regex library
               :cl-syntax-annot   ;; for @export annotation
               :clack             ;; Web libraries
               :lack
               :caveman2          ;; Web framework
               :djula             ;; Template engine
               :cl-dbi)           ;; Database
  :components ((:module "src"
                :components
                ((:file "config") ;; files into src/
                 (:file "db")
                 (:file "web")
                 (:file "core"))))

Now, inside src/core.lisp (I renamed the file from main to core) I added two new functions, to start and stop the server, which will be helpful to use from the REPL.

(defvar *server* nil)

(defparameter *app*
  (lack:builder
	  (:static
      :path (lambda (path)
              (if (ppcre:scan 
                    "^(?:/images/|/css/|/js/|/robot\\.txt$|/favicon\\.ico$)" 
                    path)
                  path
                  nil))
      :root *static-directory*)
  ;; Additional middlewares
    guestbook.web:*web*))

@export
(defun start (&rest args
              &key
                (server :hunchentoot)
                (port 3210)
                (debug nil)
              &allow-other-keys)
  "Starts the server."
  (when *server*
    (restart-case (error "Server is already running.")
      (restart-server ()
        :report "Restart the server."
        (stop))))

  (setf *server* (apply #'clack:clackup *app*
                   :server server
                   :port port
                   :debug debug
                   args))
  (format t "Server started"))

@export
(defun stop ()
  "Stops the server."
  (when *server*
    (clack:stop *server*)
    (format t "Server stopped")
    (setf *server* nil)))

*server* contains the server instance and is defined as a variable since we will need to redefine it. *app* is the web application wrapped with a layer by Lack. start and stop instantiate the server with some logging, or raise errors if the action is not successful.

Lack and Clack are the two libraries I used to wrap the web application. The first one allows to define a series of middlewares in the server; for example, I used :static to tell the server where to find all the static assets in the project, inside the directory pointed to by *static-directory*. Other middlewares are available, like logging, managing sessions or providing authentication features. Clack instead is an abstraction layer for the server that provides some parameters to customise it, for example, to quickly swap which server to use between development mode (hunchentoot) and production (woo).

(unless (null *server*)
    (restart-case (error "Server is already running.")
      (restart-server ()
        :report "Restart the server."
        (stop))))

In this portion of the code, I defined a restart action for the debugger. The Common Lisp debugger always has RETRY and ABORT actions for every exception raised. By declaring a restart-case, we are signalling an error and adding custom choices to the ones offered by default by the debugger. The new option I added is called restart-server and, if selected, it first (stop)s the server and then restarts the operation, so the function runs again without raising an error. It’s a smart way to interact with the REPL and improve the developer experience using the language directly.

More on conditions and restart here and here.

At the top of src/core.lisp I set the package definition and some initialisation function calls.

(in-package :cl-user)
(defpackage guestbook.core
  (:use :cl)
  (:import-from :guestbook.config
                :*static-directory*))
(in-package :guestbook.core)

(syntax:use-syntax :annot)

(syntax:use-syntax :annot), from the package cl-syntax-annot, allows us to use some special decoration notation at the top of the function. At the top of start and stop I added an @export tag, which tells the compiler that a function is exported by the package.

Configuration

I put the configuration parameters for the app into a config file; there are better ways, but they’re not necessary for a project this simple.

@export
(defparameter *application-root* 
  (asdf:system-source-directory :guestbook))
@export
(defparameter *static-directory* 
  (merge-pathnames #P"static/" *application-root*))
@export
(defparameter *template-directory* 
  (merge-pathnames #P"templates/" *application-root*))

@export
(defvar *config*
  `(:databases 
      ((:maindb :sqlite3 
        :database-name ,(namestring (merge-pathnames "guestbook.sqlite"
                                                     *application-root*))))
    :schema-file "db/schema.sql"))

The variable *config* needs a specific format to be used with cl-dbi, which is the library that I used to interface with the database. In a real application, it would be good to differentiate between dev and prod mode with different configurations used, for example to point to different databases or to use a different server with Clack.

Database

I created a new file in db/schema.sql and put the SQL code to create a message table.

CREATE TABLE message (
    id       INTEGER PRIMARY KEY AUTOINCREMENT,
    username VARCHAR(50) NOT NULL,
    ts       DATETIME NOT NULL,
    content  TEXT NOT NULL
);

Then I ran sqlite3 guestbook.sqlite --init db/schema.sql from the terminal to create and initialise a new database in the project root.

(in-package :cl-user)
(defpackage guestbook.db
  (:use :cl)
  (:import-from :guestbook.config
                :*config*
                :*application-root*))
(in-package :guestbook.db)

(syntax:use-syntax :annot)

(defun connection-settings (&optional (db :maindb))
  (cdr (assoc db (getf *config* :databases))))

@export
(defun db (&optional (db :maindb))
	"Returns a cached database connection for DB (defaults to :maindb). 
  Uses `connection-settings` and `dbi:connect-cached`. 
	
	Usage: (db) or (db :testdb)"
  (apply #'dbi:connect-cached (connection-settings db)))

@export
(defvar *connection* nil)

@export
(defmacro with-connection (conn &body body)
  "Executes BODY with *CONNECTION* dynamically bound to CONN.
CONN should be a database connection object, typically from `db`.

Usage: (with-connection (db :maindb)
         (dbi:do-sql *connection* \"SELECT * FROM users\"))"
  `(let ((*connection* ,conn))
     (unless *connection*
       (error "Database connection cannot be NIL"))
     ,@body))

This is the first macro of the project. It’s a simple wrapper that provides a new variable *connection* to use inside the body. It raises an error if the conn value that we passed is not initialised. The function db instead returns a valid connection, cached automatically by the library.

Then I defined some functions to perform CRUD operations on the database. format-timestamp converts a universal timestamp value into a readable date-time string.

(defun format-timestamp (universal-time)
  "Converts a universal time value into a human-readable timestamp string,
formatted as 'YYYY-MM-DD HH:MM:SS'."
  (multiple-value-bind (sec min hour day month year)
      (decode-universal-time universal-time)
    (format nil "~4,'0D-~2,'0D-~2,'0D ~2,'0D:~2,'0D:~2,'0D" year month day hour min sec)))

(defun add-message (name message)
  "Inserts a new message into the database with the given NAME and MESSAGE content."
  (with-connection (db)
    (let ((sql "INSERT INTO message (username, ts, content) VALUES (?, ?, ?)")
          (ts (get-universal-time)))
      (dbi:do-sql *connection* sql (list name ts message)))))


(defun delete-message (id)
  "Deletes the message with the given ID from the database."
  (with-connection (db)
    (let ((sql "DELETE FROM message WHERE id = ?"))
      (dbi:do-sql *connection* sql (list id)))))


(defun get-all-messages ()
  "Retrieves all messages from the database, ordered by timestamp descending.
Timestamps are formatted as human-readable strings."
  (with-connection (db)
    (let* ((sql "SELECT * FROM message ORDER BY ts DESC")
           (messages (dbi:fetch-all
                      (dbi:execute
                       (dbi:prepare *connection* sql)))))
      (mapcar (lambda (row)
                (setf (getf row :|ts|) (format-timestamp (getf row :|ts|)))
                row)
              messages))))

The Web App

Finally we can create the web application which I put inside src/web.lisp. It’s a bit longer then the other files so I will break it into chunks.

(in-package :cl-user)
(defpackage guestbook.web
  (:use :cl
        :caveman2)
  (:import-from :guestbook.config
                :*template-directory*)
  (:import-from :guestbook.db
                :add-message
                :delete-message
                :get-all-messages)
  (:export :*web*))
(in-package :guestbook.web)

(defclass <web> (<app>) ())
(defvar *web* (make-instance '<web>))
(clear-routing-rules *web*)

The web framework I am using is called Caveman, developed by Eitaro Fukamachi, who is a very prolific lisper and the author of multiple libraries that I am using, including Lack, Clack, and cl-dbi, all having great integration with each other. <app> is a class defined by the web framework, and I am extending it and instantiating it in *web*. clear-routing-rules is a function inherited from Ningle, another web framework, which clears the route associations inside the web app instance.

Templates

Our web app needs to render some static templates with data, and to do that there’s a great library called djula, which allows us to use most of the same tags and filters that Django exposes in its template engine.

I’m not going to include the templates source here for space reasons, in the repository they're in the templates/ folder.

(djula:add-template-directory *template-directory*)

(defun render (template-path &optional env)
  "Renders a Djula template from TEMPLATE-PATH.
ENV is an optional plist of variables passed to the template."
  (apply #'djula:render-template*
         (djula:compile-template* (princ-to-string template-path))
         nil
         env))

The first function call tells djula where to find the templates in the project. render is a function that receives a template path as a parameter and compiles the template for better performance.

Routes

To allow users to perform CRUD operations, we need to expose a few routes to be called by the web frontend. There are two ways to define routes in Caveman, but this one seems a bit clearer to me than using annotations. defroute is a macro that receives the route path, some parameters like :method and eventually some arguments, and then defines the body to handle the request. Inside the body, we can access the request object via *request* and extract data from it using, for example, the function request-body-parameters.

The routes are self-descriptive. / renders index.html, passing all the messages from the db; /message handles POST requests and inserts the message in the database if the parameters conform; and /message/delete deletes a message.

(defroute "/" ()
  (render #P"index.html"
          (list :messages (get-all-messages))))


(defroute ("/message" :method :POST) ()
  (let* ((body-params (request-body-parameters *request*))
         (name-param (assoc "name" body-params :test #'string=))
         (message-param (assoc "message" body-params :test #'string=)))
    (if (and (consp name-param) (consp message-param))
      (add-message (cdr name-param)
                   (cdr message-param))
      (format t "Missing body parameters: received ~A~%" body-params)))
  (redirect "/"))


(defroute ("/message/delete" :method :POST) ()
  (let* ((body-params (request-body-parameters *request*))
         (id-param (assoc "id" body-params :test #'string=)))
    (if (consp id-param)
        (let ((id (ignore-errors (parse-integer (cdr id-param)))))
          (when id
            (delete-message id)))
        (format t "Missing id parameter.")))
  (redirect "/"))

I also added another function, which defines a method on the web app class. on-exception is a generic function called when an exception occurs in the web application. This method is specific on the parameter type, since it will run only if the exception has code 404 — not found — and in that case I return a specific custom template.

(defmethod on-exception ((app <web>) (code (eql 404)))
  (declare (ignore app code))
  (render #P"404.html"))

Run the demo

Now I can load the project into the environment with Quicklisp.

> (ql:quickload :guestbook)
To load "guestbook":
  Load 1 ASDF system:
    guestbook
; Loading "guestbook"
......................
(:GUESTBOOK)
CL-USER>

Then I can start the server and point my browser to 127.0.0.1:3210.

> (guestbook.core:start)
Hunchentoot server is started.
Listening on 127.0.0.1:3210.
Server started

I can send messages that get saved with a timestamp and then delete them. Pretty simple.

Reducing boilerplate

Lisp became popular for a series of reasons, and one of the most cited is its ability to reduce boilerplate thanks to the metaprogramming capabilities of the language. Lisp hackers are proud of being able to code solutions faster and do exploratory programming, reducing the size of the code they need to write to reach a solution.

When comparing the lines-of-code count of my Common Lisp guestbook against the Python-Flask version, the latter seems quicker and simpler to write — 36 lines of Python vs. 229 of Lisp. Lisp’s strength is its ability to model itself according to the problem the developer is solving. In this case, I am dealing with a simple guestbook demo, so it may not be strictly necessary, but to present the language better I will try to reduce the size of the program by hiding some code and configuration.

First, let’s have a look at the new guestbook app, written in my new custom web framework called flashcl: 29 LOC, properly formatted and stripped of all comments, and not too hard to read.

(in-package :cl-user)
(defpackage guestbook.core
  (:use :cl :flashcl))
(in-package :guestbook.core)

(init-flashcl :sqlite3 "guestbook.sqlite")

(defmodel message
  ((username :col-type (:varchar 50) :accessor message-username)
   (content  :col-type  :text        :accessor message-content)))

(defroute "/" ()
  (render #P"index.html" (list :messages (db-all 'message))))

(defroute ("/message" :method :POST) ()
  (let ((name (form-param "name"))
        (message (form-param "message")))
    (if (and name message (> (length name) 0) (> (length message) 0))
      (db-add (make-instance 'message :username name :content message))
      (format t "Missing body parameters: received ~A~%" (body-params))))
  (redirect "/"))

(defroute ("/message/delete/:id" :method :POST) (&key id)
  (if id
    (let ((id (ignore-errors (parse-integer id))))
      (when id
        (db-delete (db-find 'message id)))
      (format t "Missing id parameter.")))
  (redirect "/"))

Where did start go? It is now part of the flashcl framework, so it’s imported in the package. Here’s the usage reference.

;; --- Run the Application ---
;; Call run-app function from your REPL or add it here to run on load.
;; Call stop-app to stop the server.
;;
;; Example: (guestook::run-app) or just (run-app) inside the package.
;; (run-app :port 5000 :server :hunchentoot)

Then I slightly modified the template delete button to point to the correct route with the id parameter.

I'm going to show just some the new code that I added to write flashcl. I mostly merged the old code in a new file trying to make it as reusable as I could for other web projects, since it's possible to define new database models and routes quickly, with support for static files and templates. I used mito, another library from Eitaro Fukamachi, which is an ORM that works well with SQLite.

First I imported the library and exported only what’s needed by the user.

(defpackage :flashcl
  (:use #:cl)
  (:import-from #:caveman2
                #:defroute
                #:redirect
                #:*request*
                #:*response*)
  (:import-from #:lack.request
                #:request-parameters)
  (:import-from #:mito
                #:dao-table-class ; Re-export metaclass for use in defmodel
                #:connect-toplevel
                #:ensure-table-exists
                #:select-dao
                #:find-dao
                #:delete-dao
                #:insert-dao)
  (:export ;; Setup
          #:init-flashcl
          ;; App Definition
          #:flashcl-app ;; Variable holding the app instance
          #:defmodel
          ;; Routing & Request/Response
          #:defroute
          #:form-param
          #:render ;; Re-export caveman's render (or wrap it)
          #:redirect ;; Re-export caveman's redirect
          ;; Database
          #:db-all
          #:db-add
          #:db-find
          #:db-delete
          #:dao-table-class ;; Re-export metaclass
          ;; Running
          #:run-app
          #:stop-app))

Then I wrote an initialisation function similar to Flask’s, which sets various variables and creates a new instance of the Caveman webapp.

(defun init-flashcl (db-type db-path &optional (template-dir "templates")
                                               (static-dir "static"))
  "Initializes Flashcl environment. Sets DB/Template paths, connects DB."
  (setf *database-path* (get-absolute-path db-path))
  (setf *static-directory* (get-absolute-path static-dir))

  ;; Set Djula's template directory
  (djula:add-template-directory (get-absolute-path template-dir))
  (format t "Set template directory as ~A~%" (get-absolute-path template-dir))

  ;; Connect to SQLite database
  (handler-case (connect-toplevel db-type :database-name *database-path*)
    (error (c)
      (format *error-output* "~&Error connecting to database ~A: ~A~%" *database-path* c)))

  ;; Define the Caveman2 app instance here
  (setf flashcl-app (make-instance 'caveman2:<app>)))

To make it easy to define a new database model for the application, hiding the mito library, I created a macro that ensures the table gets created correctly. mito automatically adds a few fields to the table definition, like id and created_at or updated_at, so we don't have to worry about them.

(defmacro defmodel (name slots &rest options)
  "Defines a Mito DAO class and ensures its table exists.
   Example: (flashcl:defmodel comment
              ((name :col-type (:varchar 20))
               (comment :col-type :text)))"
  (let ((class-options options)
        (table-name (string-downcase name))) ; Use singular name by default
    
    ;; Add table name inference if not present
    (unless (find :table-name options :key #'car)
      (push `(:table-name ,table-name) class-options))

    `(progn
      (defclass ,name ()
        ,slots
        (:metaclass mito:dao-table-class)
        ,@class-options)
      ;; Ensure table exists after class definition
      ;; Note: This runs at compile/load time when defmodel is processed.
      ;; Ensure DB is connected before loading code using defmodel.
      (handler-case (ensure-table-exists ',name)
        (error (c)
          (format *error-output* "~&Warning: Could not ensure table for ~A (DB might not be connected yet?): ~A~%" ',name c)))
      ;; Return the class name
      ',name)))

Then I added some database helpers to perform a few CRUD actions on the db.

(defun db-all (class-name)
  "Selects all records for the given model class."
  (select-dao class-name))

(defun db-find (class-name id)
  "Finds a model instance by ID."
  (find-dao class-name :id id))

(defun db-add (instance)
  "Inserts a model instance into the database."
  (insert-dao instance))

(defun db-delete (instance)
  "Deletes a model instance from the database."
  (delete-dao instance))

Conclusions

To recap, in this tutorial I used some of the latest libraries in Common Lisp to create a simple guestbook webapp. Then I wrote a simple reusable wrapper to make our code more concise, aiming to challenge the Flask framework in Python. The full source code of the two versions it's here and here.

Ultimately I would like to give my opinion on using Common Lisp to write a web server. Common Lisp shines when dealing with low-level tasks like small systems programming or performing intensive computation at scale. When these complex systems need to communicate with the outside world, perhaps via an API, writing a server is the correct way and Common Lisp is capable of delivering that.

But despite the fact that the language is easily adaptable and comes with performant server libraries (Hunchentoot and Woo), I would say that there are better alternatives for developing generic modern web apps. Hot reloading is now a feature present in many web frameworks, and Common Lisp is not really shining for being ergonomic, nor does it come with many built-ins. Clojure, by contrast, is a modern Lisp dialect with great web frameworks, nice documentation and tons of stable libraries. I would definitely choose the latter for a fresh web project, since with CL I get the feeling that I would end up having to write more code than I should to add custom features and middlewares.

I would like to give a shout-out to Alive, which is the only Common Lisp extension for VSCode that implements the REPL with features similar to what SLIME and SLY bring to Emacs. The extension is still under development and not yet a full replacement for Emacs, especially during debugging, but it has great features and a lot of potential to introduce Common Lisp to newcomers, thanks to the VSCode web interface.

Feel free to email me if you have any questions, recommendations, or even complaints.

Rust 101

Wed, 10 Apr 2024 00:00:00 GMT

I’ve recently had the chance to try Rust for a project and it has been a surprising pleasant experience. Rust is a modern language and I felt quire familiar with it in a few days. It has a simple and effective design and thanks to that, it gets frequently mentioned as a successor to old low level languages C like. Rust is fast and, in a lot of benchmarks, the compiled code performance is comparable to C and C++. I don’t have experience yet with other similar low-level languages as Nim or Zig but I believe that there’s a reason why Rust got extremely popular in the last few years.

In this article I’ll write down a quick overview of the most important concept that I found make Rust a different and more evolved language compared to it’s predecessor. The Rust documentation is really well written with a lot of example and it will allow you to use the language in a very short time.

Memory Safety

Rust is a memory safe language which means that it’s design prevent the programmer to do operation that could provoke memory faults and crashes of the program. If you ever wrote C or C++ you know what kind of bugs I’m talking about. This kind of problems are hard to spot and debug and in complex programs, it gets easy to use an already free memory or to forget to free one.

There have been two main ways to manage program memory. One is letting the memory management in the hands of the programmer (as C, C++), and the other is using a Garbage Collector that will clean the unused allocated memory portions for you (as Lisp, Java, Python, JS). Manually freeing the memory can get complicated with large and complex programs, and automatic garbage collection can slow down your execution at random, and it’s not that easy to control.

Rust bring a different approach; writing programs that are actually memory safe by design. It does that by not exposing the memory management functions, without using an external tool to do the job. It introduces the concept of ownership, a set of rules that check your code at compile time to prevent you to do operations that break the ownership of a variable.

Ownership

This are the ownership rules:

Each value in Rust has an owner.
There can only be one owner at a time.
When the owner goes out of scope, the value will be dropped.

let s1 = String::from("hello"); // s1 is the owner of the String
let s2 = s1; // s2 becomes the new owner, and s1 is invalidated
println!("{}, world!", s1); // This will cause a compile-time error because s1 is no longer valid

The previous code will not compile. The compiler would even give you suggestion to what caused the error: error[E0382]: borrow of moved value: s1. One may think that the string s1 should still be there, but in reality Rust already called drop on the string structure, deallocating the memory segment and making it no longer valid.

Rust is getting in the way of the programmer for a good reasons; following this rules will prevent problems with memory deallocation, like forgetting to deallocate something, accessing a resource already deallocated, or double freeing a variable. In exchange, Rust is enforcing some patterns and providing some data structure to solve all ownership problems. The following is a useful schema I found on reddit, that quickly points you to which data structure is better to use for each scenario (T is the generic data type).

Borrowing

Borrowing is a mechanism that allows you to create references to data owned by another variable, enabling multiple parts of your code to access and potentially modify the same data without transferring ownership.

fn main() {
    let mut s = String::from("hello");

    change(&mut s);
}

fn change(some_string: &mut String) {
    some_string.push_str(", world");
}

In the code above, I've just created an mutable reference to a String object, and passed it to a function that modify the string. This behaviour it's called borrowing, since we borrow the value, and then give it back to the caller when finished using it.

In Rust, we cannot borrow as mutable variable more than once. This restriction of preventing multiple mutable references to the same data, helps preventing data races at compile time.

Structs and objects

Rust allows the creation of custom data types using struct and enum. A struct is between a C struct and a C++ class. It can be used as object and have access to self, but it’s missing a full support for inheritance.

struct Rectangle {
    width: u32,
    height: u32,
}

impl Rectangle {
    fn new(w: u32, h: u32) -> Rectangle {
        Rectangle {
            width: w,
            height: h,
        }
    }

    fn area(&self) -> f64 {
        (self.width * self.height) as f64
    }
}

fn main() {
    let mut rect = Rectangle::new(3, 4);
    println!("Area: {}", rect.area());
}

A new struct Rectangle has been defined, with a simple area method which access the property using the reference to itself.

When a new struct is defined inside a file, it can be encapsulated simply by avoiding the use of pub on the methods. This way, they will not be visible from outside but still accessible from inside.

Traits

Polymorphism can be pefrormed by defining a Trait which adds the possibility to have shared behaviour between structures.

trait Area {
    fn area(&self) -> f64;
}

impl Area for Circle {
    fn area(&self) -> f64 {
        std::f64::consts::PI * self.radius * self.radius
    }
}

Here I've defined a trait Area which every shape struct can implement in it’s own way. The trait it’s only the function signature, with parameters and expected return type.

Enum

Enum are just a way of define a set of values that a variable of that type can take. It’s similar to struct in the sense that accept the use of impl with custom methods and traits.

enum Animal {
    Bird,
    Insect,
    Fish,
    Mammal,
}

Pattern matching

Pattern matching is a construct that is essential in the Rust design. It allows to automatically match the value of a variable using it’s possible states, similar to the statement case in other languages. I use the operator match that receives the variable and give the possibility to execute code when a value is matching.

impl Animal {
    fn can_fly(&self) -> bool {
        match self {
            Animal::Bird => true,
            Animal::Insect => true,
            _ => false, // Covers Fish and Mammal
        }
    }
}

For example, here it's matching all the possible states that a variable of type Animal can have. The option _ is a catch-all pattern, representing all the other possible states that the variable can take.

Result and Option

Two enumerated types that are part of the Rust standard library are Result and Option. They’re values used to deal with errors that need to be managed and potentially absent values. To access the value, the pattern matching method shown above is required.

Result is used in case a function may fail and there's a need to grab the exception and handle it. Rust allows functions to throw errors and then catch them from the caller by performing a match on the function's returned value.

use std::fs::File;
use std::io;

fn read_file(path: &str) -> Result<String, io::Error> {
    let file = File::open(path)?;
    let mut contents = String::new();
    file.read_to_string(&mut contents)?;
    Ok(contents)
}

fn main() {
    match read_file("example.txt") {
        Ok(contents) => println!("File contents: {}", contents),
        Err(error) => println!("Error reading file: {}", error),
    }
}

? is a special operator in Rust that allows to return the error directly, saving you the boilerplate of writing another match.

Option instead is used to return an optional value, since Rust doesn’t have the concept of null (and no Null Pointer Exception).

fn find_first_even(numbers: &[i32]) -> Option<i32> {
    for num in numbers {
        if num % 2 == 0 {
            return Some(*num);
        }
    }
    None
}

fn main() {
    let numbers = vec![1, 3, 5, 7, 8];
    match find_first_even(&numbers) {
        Some(even) => println!("First even number: {}", even),
        None => println!("No even numbers found"),
    }
}

Functional

Rust took closures and iterators from functional programming. It's easy to write inline functions and iterate over array with the set of functions available for functional programming.

let numbers = vec![1, 2, 3, 4, 5];
let doubled: Vec<_> = numbers.iter().map(|x| { x * 2 }).collect();

In the code above I'm using map over a list of numbers, and running the lambda function |x| x * 2 on it, in fact creating a closure. If you need to add the possibility to add iteraction support to a struct, you would need to implement the Iterator trait, as shown in the docs

Analyzer

rust-analyzer is what is going to help you understanding Rust and the patterns that the language is guiding you to use. When integrated inside your favourite editor (I’m using Zed right now), you will see warning and errors generated directly when saving a file.

Let’s try to analyse some broken code

fn main() {
    let s1 = String::from("hello");
    let s2 = s1;
    println!("{}", s1);
}

Why is this code broken?

error[E0382]: borrow of moved value: `s1`
  --> src/main.rs:4:20
   |
65 |     let s1 = String::from("hello");
   |         -- move occurs because `s1` has type `String`, which does not implement the `Copy` trait
66 |     let s2 = s1;
   |              -- value moved here
67 |     println!("{}", s1);
   |                    ^^ value borrowed here after move
   |
   = note: this error originates in the macro `$crate::format_args_nl` which comes from the expansion of the macro `println` (in Nightly builds, run with -Z macro-backtrace for more info)
help: consider cloning the value if the performance cost is acceptable
   |
66 |     let s2 = s1.clone();
   |                ++++++++

Rust is telling that when the code does s2 = s1, it moves the value s1 so it cannot be used anymore in the println call. A solution is to clone s1 into s2 so it will then have two independent instances of the string. Pretty handy to find a quick solution, especially for beginners. The Rust team did a great job for creating high quality error messages to guide the user to find a solution.

Macros

Rust supports different types of macros: declarative and procedural. Macro helps extending the language by removing boilerplate code and even extending programs with external code (e.g. plugins).

Declarative macros are called like functions, they just append ! to their name.

#[macro_export]
macro_rules! vec {
    ( $( $x:expr ),* ) => {
        {
            let mut temp_vec = Vec::new();
            $(
                temp_vec.push($x);
            )*
            temp_vec
        }
    };
}

let v = vec![1, 2, 3, 4]; // Creates a new vec inline

My suggestion is to avoid using macros and always prefer functions where possible. Macros are hard to understand, maintain and document when they get complex.

Cargo and packages

cargo is the official package manager for Rust. It downloads the dependencies and keep track of the Rust version the project is supporting. It can be used for all the task related to the project, like running, building it, generating the documentations, etc. The package registry is crates.io which counts more than 140K packages as today.

Tests

Rust provides a test framework directly in the language which supports both unit and integration tests. cargo test is how to run all the tests for a project.

This is a unit test example, included in the same file of the function implementation.

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        let result = 2 + 2;
        assert_eq!(result, 4);
    }
}

When to use Rust

Rust got really popular in the last few years, but it's not a solution for all coding problems, nor a suitable alternative for all cases. When performance is a requirement, instead of using C or C++, Rust can provide much nicer usability and ergonomics, since it's nearly as fast as C but definitely safer, and the choice will pay off in the long run. Big companies have started adopting Rust effectively with great results and enough performance gains in critical parts of their applications to justify the choice.

Common Lisp data structures

Tue, 12 Dec 2023 00:00:00 GMT

Quick reference for the main data structures in Common Lisp. The code has been tested using SBCL.

Some good rules for Common Lisp that I found say:

Use the appropriate data structure for the situation.
Favor iteration over recursion. Recursion is good for recursive data structures.

List

Lists are used when you need to store a sequence of elements. It supports different data types. They are particularly useful when you need to perform operations that involve recursion or iteration.

Key points: ordered, flexible size, efficient for traversal, slow for random access

;; Declaration
cl-user> (defparameter *list* '(1 2 3 4))
*list*
cl-user> (list 1 2 3 4) ; alternative
(1 2 3 4)

;; Access
cl-user> (nth 3 *list*) ; not performant for long lists
4 (3 bits, #x4, #o4, #b100)
cl-user> (first *list*)
1 (1 bit, #x1, #o1, #b1)

;; Adding
cl-user> (push '0 *list*)
(0 1 2 3 4)

;; Deleting
cl-user> (setf *list* (remove 1 *list*))
(0 2 3 4)

Association list - alist

Association lists are lists, implemented using cons cells, which are used to store key-value pairs where the key is a symbol. They are slower than hash-tables for large amount of data.

Key points: key-value pairs, simple structure, inefficient for large datasets

;; Declaration
cl-user> (defparameter *alist* '((:a . 1) (:b . 2)))
*alist*

;; Access
cl-user> (assoc :a *alist*)
(A . 1)
cl-user> (cdr (assoc :a *alist*))
1

;; Adding
cl-user> (push '(:c . 3) *alist*)
((C . 3) (A . 1) (B . 2))

;; Deleting
cl-user> (setf *alist* (remove :c *alist* :key #'car))
((A . 1) (B . 2))

Property list - plist

Property are used when you need to store key-value pairs and the keys are symbols. They are simpler than hash tables and association lists but can be slower for large amounts of data.

Key points: key-value pairs, lightweight, inefficient for large datasets

;; Declaration
cl-user> (defparameter *plist* '(:key1 value1 :key2 value2))
*plist*

;; Access
cl-user> (getf *plist* :key1)
value1

;; Adding
cl-user> (setf (getf *plist* :new-key) 'new-value)
new-value

;; Deleting
cl-user> (remf *plist* :key1)
t
cl-user> *plist*
(:new-key new-value :key2 value2)

Hash table

Hash tables store key-value pairs and provide efficient lookup and storage. They are highly efficient for large datasets.

Key points: key-value pairs, fast lookups, mutable

;; Declaration
cl-user> (defparameter *hash-table* (make-hash-table))
*hash-table*

;; Adding
cl-user> (setf (gethash 'key1 *hash-table*) 'value1)
value1

;; Access
cl-user> (gethash 'key1 *hash-table*)
value1

;; Deleting
cl-user> (remhash 'key1 *hash-table*)
t

Array

Arrays are multidimensional data structures that hold elements accessible by indices. Arrays are faster than lists for random access, but slower for operations that involve adding or removing elements. Arrays have a fixed size upon creation, but they can be resized using adjust-array.

Key points: multidimensional, fixed size, efficient access by index

;; Declaration
cl-user> (defparameter *array* (make-array 4 :initial-contents '(1 2 3 4)))
*array*

;; Access
cl-user> (aref *array* 2)
3

;; Modifying
cl-user> (setf (aref *array* 1) 5)
5

;; Extending/Shrinking
cl-user> (setf *array* (adjust-array *array* 6 :initial-element 0))
#(1 5 3 4 0 0)

;; Deleting
;; To delete elements, consider creating a new array without the unwanted elements.

Vector

A vector is a one-dimensional array. Vectors are similar to arrays but can dynamically grow and shrink using functions like vector-push and vector-pop. Vectors are particularly useful when you need a sequence that can change size dynamically.

Key points: one-dimensional, dynamic size, efficient access by index

;; Declaration
cl-user> (defparameter *vector* #(1 2 3 4))
*vector*

;; Access
cl-user> (elt *vector* 0)
1

;; Modifying
cl-user> (setf (elt *vector* 3) 5)
5

;; Adding elements
cl-user> (vector-push-extend 6 *vector*)
4
cl-user> *vector*
#(1 2 3 5 6)

;; Removing elements
cl-user> (vector-pop *vector*)
6
cl-user> *vector*
#(1 2 3 5)

;; Converting to a list
cl-user> (coerce *vector* 'list)
(1 2 3 5)

;; Shrinking/Extending
cl-user> (setf (fill-pointer *vector*) 2)
2
cl-user> *vector*
#(1 2)

;; Push extending
cl-user> (vector-push-extend 7 *vector*)
2
cl-user> *vector*
#(1 2 7)

Structure

Structures are user-defined data structures created using defstruct, allowing the grouping of different data types together.

Key points: user-defined, structured, efficient access

;; Declaration
cl-user> (defstruct person name age)
person

;; Creating
cl-user> (make-person :name 'Alice :age 30)
#S(PERSON :NAME ALICE :AGE 30)

cl-user> (defvar *person* (make-person :name 'Bob :age 25))
*PERSON*

;; Access
cl-user> (person-name *person*)
BOB

;; Modifying
cl-user> (setf (person-age *person*) 40)
40

;; Deleting
cl-user> (makunbound '*person*)
*PERSON*

References

How to add a Clojure REPL to a web page

Fri, 24 Mar 2023 00:00:00 GMT

One of the advantages of working with Clojure is that it can be used on top of the JVM as well as in the browser. This is possible thanks to the Clojurescript compiler, which takes your .cljs files and turns them into optimized Javascript with the help of the Google Closure compiler, ready to be embedded into a webpage. Since Clojurescript translate Clojure code to Javascript, in theory, it should be possible to run Clojurescript directly in the browser, since Javascript interpreter is exposed to the user.

I got inspired after experimenting with a cool project to present the Haskell language called Try Haskell, which allows to run Haskell in the browser. It made me thinking about how I could replicate it using Clojure, so that users can type in a REPL on a web page. I started searching for a way to embed a working Clojure REPL in a small web app and this is what I found. There are various way to achieve this. The old method was using the self-hosted cljs compiler, but there's a simpler approach. Thanks to SCI, the Small Clojure Interpreter is now possible to write and execute Clojurescript code on top of Javascript; this means it can be executed in the browser.

The simplest way to try it is to add the library via CDN. Just include Scittle library, same as SCI but loaded with a script tag into a webpage. It doesn't need any additional library or Clojurescript.

<script
  type="text/javascript"
  src="https://cdn.jsdelivr.net/npm/scittle@0.5.14/dist/scittle.js"
></script>

Then you can write your Clojure code into a script tag, or load a clojurescript file.

<script type="application/x-scittle">
  (println "Hello Wolrd")
</script>
<!-- or -->
<script src="app.cljs" type="application/x-scittle"></script>

For the rest of the tutorial, I'll be using SCI so the library get's loaded via Clojurescript. Refer to SCI docs to learn how to do it.

You can try the Clojure REPL with a tutorial at TryClojure.org.

SCI

The core of the app is the integration of SCI. It essentially provide a way to safely evaluate the strings that the user input in the REPL. I said "safe" since it creates a sandboxed environment where we can run our code. The Clojure interpreter runs directly in the browser's JavaScript engine, offering the same functionalities of the developer console.

We can create a SCI context by using the function sci/init with some initialization options.

(ns app.sci
  (:require [sci.core :as sci]))
   
(defonce context (atom (sci/init init-opts)))

I'm, using an atom since it lets me add options during execution by updating the context, as shown in the following function.

(defn extend-ctx
  "Extend default sci context merging `opts`."
  [opts]
  (reset! context (sci/merge-opts @context opts)))

I set a custom print function, by changing the default binding of print-fn for sci.

(defn set-print-fn
  "Setup a custom `print-fn` for sci."
  [f]
  (sci/alter-var-root sci/print-fn (constantly f)))

I created a wrapper around eval-string* for better error management. The error/error-handler function prints errors to console along with the stacktrace. This is the core function of the app: SCI evaluates a string and returns the output of the execution.

(defn eval-string
  "Evaluate `source` using the sci interpreter and return its output.
   If an error occurs during evaluation, raise an exception."
  [source]
  (try (sci/eval-string* @context source)
       (catch :default e
         (error/error-handler e (:src context))
         (let [sci-error? (isa? (:type (ex-data e)) :sci/error)]
           (throw (if sci-error?
                    (or (ex-cause e) e)
                    e))))))

REPL

I created a custom function write-repl!, to print the evaluation output. It appends the value to a vector that stores the output history.

(ns app.repl.core
  (:require [reagent.core :as r]
            [app.sci :as sci]))
  
(defonce repl-history (r/atom []))
  
(defn- write-repl!
  "Append `s` to the REPL history.
  Optional keyword `k` to use as a type."
  ([s]
   (write-repl! s :output))
  ([s k]
   (swap! repl-history conj {:type k :value s})))

(sci/set-print-fn (fn [s] (write-repl! s)))

I extended the context by adding a few functions to the user default namespace. This functions can be called in the REPL, and they're able to interact with the webpage. For example changing some environment variables like prompt symbol color, the current user name, or interacting with the tutorial steps.

(sci/extend-ctx
 {:namespaces {'user {'start start-tutorial
                      'clear clear-repl
                      'restart restart-tutorial
                      'my-name set-name
                      'next-step inc-step!
                      'prev-step dec-step!
                      'set-step (when DEBUG set-step)
                      'set-prompt set-prompt
                      'more (fn [] true)
                      'help print-help}}})

For example the clear-repl function resets the history to an empty vector, returning nil.

(defn clear-repl []
  (reset! repl-history [])
  nil)

All the Clojurescript code is evaluated using eval-string. The typical way to do this is when the user press "Enter" in the REPL HTML input element, but we can also use it to import an entire namespace.

;; Import `repl` namespace helper functions 
(sci/eval-string "(require '[clojure.repl :refer :all])")

Colophon

I released TryClojure more than a year ago but I didn't write any content on how I made it. It's a frontend-only app served by Netlify, with no backend since all the code runs in the browser. I've used a few libraries to build the app. The build process is managed with shadow-cljs, I've used Tailwind for styles, and Reagent for creating UI components.

The missing ClojureScript re-frame quick start tutorial

Sat, 18 Mar 2023 00:00:00 GMT

Re-frame lacks a quick start guide, so this is my attempt to create one, including a TodoMVC tutorial.

When I discovered Clojurescript and started to play around with it, the first library that I used was Reagent. Essentially, it's a wrapper around React, allowing us to write components as Clojurescript functions and instead of JSX, using vectors and keywords[^helix] for the syntax. It has been pretty easy to start with, since I already known how React works and I was comfortable with the Clojure syntax. Like its parent library, it's basically offers a way to define reusable component, with internal state and props received from the parent components.

Since React and Reagent are UI libraries, the application state management is left to the programmer, and that's where something like re-frame comes handy. It implements a model, view, update paradigm using Reagent under the hood, and help separating the state management (model, update) from the UI components (view). The library is similar to Redux having a single central store for state and unidirectional data flow.

The problem I found with re-frame, although it's actually a simple library, is that the documentation doesn't provide a quick start page like Reagent. Instead, it only offers a long, detailed explanation (which is still a recommended reading for fully comprehend how to architecture large apps). If you're already familiar with similar paradigms, a quick guide would be enough to get started with it.

;; Importing the library
(:require [re-frame.core :as rf])

App-db

The entire application state is saved inside a Reagent version of a single Clojure atom called app-db. It's just a map, initially empty, similar to the following one.

(def app-db ; will be called db from now on 
  {})

Re-frame stores the app state in a single source of truth, which, like all Clojure data structures, is immutable and updated using pure functions. This pattern makes state management flow simple to understand, ensures changes in state are predictable, and makes debugging easier.

This is a great explanation on how app-db works, directly from the re-frame README:

at any one time, the value in app-db is the result of performing a reduce over the entire collection of events dispatched in the app up until that time. The combining function for this reduce is the set of registered event handlers.

Re-frame progress in time applying the event handler functions to the previous state of the app. This means that the collection of events is replay-able from the beginning of time, step by step; a dream for debugging an app. To achieve this, it's important that all the event handlers are pure functions (without side-effects).

Dispatch

To modify the user name value inside the app state, one needs to dispatch a new event. So, simply call the dispatch function with a vector containing the event name :user/update-name and the new value for the user name.

(rf/dispatch [:user/update-name new-user-name])

The dispatch function sends the event in the re-frame event-loop for async processing. It can be used from anywhere in the application, so in this case it would most likely happen in a reaction to a click event on a "Save" button.

Events

Now, it's necessary to define a new re-frame event handler using reg-event-db which updates the user name inside the app db. The event handler function receives the current state (db) and the triggering event as arguments. The event is a vector containing the event name (as a keyword) and the parameters.

(rf/reg-event-db
  :user/update-name
  (fn [db [event-kw new-user-name]]
    (assoc-in db [:user :name] new-user-name)))

Now our app-db will look like the following:

{:user {:name "admin"}}

An event handler should be a pure function which returns a new version of the db, only based on its input parameters and the previous state of the app. Again, it should be free from side-effects. A function is considered pure if a given input maps to the same output all the time.

Subscriptions

To show the user name inside a component, create a re-frame subscription called :user/name that extract the value of :name relative to the current :user in the state.

(rf/reg-sub
  :user/name
  (fn [db query-v]
    (get-in db [:user :name]])))

The handler function receives the app-db and query-v as parameters. query-v is the vector supplied to the subscription. In this case above, it's quite simple, and it's not using it. It's just doing an extraction of a value from the state.

The new subscription can be connected to a Reagent component using subscribe, which accepts a vector where the first element is the name of the subscription, and the rest other possible parameters.

(defn user-name []
  ;; Returns an atom that needs to be dereferenced
  (let [username @(rf/subscribe [:user/name])]
    [:h1 username]))

Now every time the value of [:user :name] changes in the db, the related subscription function will re-run, and the new value will be propagated to the views depending on them, updating the Reagent component. We don't have to care about state management if the component mount/refresh because it's all delegated to re-frame. No more large props cascade, no context/provider pattern. A single map contains the entire app state that refreshes the view when gets updated.

Re-frame is implementing a pattern called unidirectional data flow, where the application state can only be modified with dispatched actions. These actions trigger event handler functions that update the state. Than, the changes propagate consistently throughout the application. There is only one way to update the view. As a result, our view is a deterministic function of the data, making the data flow more predictable.

Subscriptions are not only used as extractors but they can also compute derived data from other subscriptions. For example, the user name can be returned in uppercase or lowercase.

(rf/reg-sub
  :user/name-uppercase
  ;; signals function
  (fn [query-v]
    (rf/subscribe [:user/name]))
  ;; computation function
  (fn [username query-v]
    (string/upper-case username)))
    
(rf/reg-sub
  :user/name-lowercase
  (fn [query-v]
    (rf/subscribe [:user/name]))
  (fn [username query-v]
    (string/lower-case username)))

In this case our subscription recevies two functions:

A signals function that returns one or more subscriptions. It can return either a single signal, a vector of signals, or a map where the values are the signals.
A computation function that takes the input values provided by the signals function, supplied as the first argument, and produces a new derived value, which will be the output of the node.

Another subscription that receives a parameter can be created easily.

(rf/reg-sub
  :user/name-prefix-case
  (fn [[_ prefix str-case]]
    (case str-case     ; don't get confused
      :upper (rf/subscribe [:user/name-uppercase])
      :lower (rf/subscribe [:user/name-lowercase])
      (rf/subscribe [:user/name])))
  (fn [username [_ prefix str-case]]
    (str prefix ": " username)))

And it would be invoked as:

(rf/subscribe [:user/name-prefix-case "Username" :lower])

Co-Effects

In case something else is required, other than update the state, such as storing data asynchronously, the event handler need to cause side effects. To do this the previous handler defined with reg-event-db must be replaced with reg-event-fx. When to use one or the other? reg-event-db when only updating the state, reg-event-fx when side effects are needed (mostly defined with :fx); The latter can be seen as a wrapper of the former. Why using a separate function for side-effects? Because our event handler functions have to be pure, to keep the ability to have the state update history as a sequence of changes over time.

In the following code, we replaced the event handler to return a map with :db as the new app state and :fx a vector telling re-frame to dispatch asynchronously another event :user/save-asynch that would save the new user name somewhere like in the database with a server call.

(rf/reg-event-fx
  :user/update-name
  (fn [{db :db} [_ new-user-name_]] ; 1st arg is coeffects, which contains the db
    {:db (assoc-in db [:user :name] new-user-name)
     :fx [[:dispatch [:user/save-asynch]]]}))

:dispatch is a built-in effect used to dispatch a single event. One can even create new custom effects, that are performing side-effects, after the app state ad been updated. This one will show an annoying alert to the user.

(rf/reg-fx
  :alert
  (fn [s] (js/alert s))

To use it, modify the map returned before:

{:db (assoc-in db [:user :name] new-user-name)
 :fx [[:dispatch [:user/save-asynch]]
      (when (> (count new-user-name) 10)
        [:alert "What a long name!"])]

TodoMVC tutorial

For the one of you, like me, who prefer learning with a practical examples, here's a working example of TodoMVC created with Reagent and re-frame.

First I initialized the app-db state with empty values using a new event handler called :initialize-db, which returns a new :db set the initial state of the app.

(rf/reg-event-fx
 :initialize-db
 (fn [_ _]
   {:db {:todo {:items {} :filter :all}}}))

In the app initialization, I dispatched the event synchronously to set the initial state, blocking the event loop. This call must run only once during app loading.

(rf/dispatch-sync [:initialize-db])

I access both the items and the filter with two subscriptions that simply extract the values.

(rf/reg-sub
 :todo/items
 (fn [db _]
   (get-in db [:todo :items])))

(rf/reg-sub
 :todo/filter
 (fn [db _]
   (get-in db [:todo :filter])))

An item is a map composed as {:id uuid :content text :active true}, while the filter is a keyword that accepts [:active :completed :all] as value.

When the user presses Enter on the todo input, the Reagent component adds a item to the collection using the following event handler. It accepts a non-empty string as a parameter.

(rf/reg-event-db
 :todo/add-item
 (fn [db [_ text]]
   (let [id (js/crypto.randomUUID)]
     (if (not (string/blank? text))
       (assoc-in db [:todo :items id] 
                 {:id id :content text :active true})
       db))))
       
;; Usage 
;; (rf/dispatch [:todo/add-item "write a re-frame tutorial"])

I defined other simple actions for todo items.

(rf/reg-event-db
 :todo/remove-item
 (fn [db [_ id]]
   (update-in db [:todo :items] dissoc id)))

(rf/reg-event-db
 :todo/toggle-item
 (fn [db [_ id]]
   (update-in db [:todo :items id :active] not)))

(rf/reg-event-db
  :todo/set-item-content
  (fn [db [_ id text]]
    (assoc-in db [:todo :items id :content] text)))

I defined a similar action for setting the filter content (without performing a value check).

(rf/reg-event-db
 :todo/set-filter
 (fn [db [_ filter-value]]
   (assoc-in db [:todo :filter] filter-value)))
   
;; Usage 
;; (rf/dispatch [:todo/set-filter :completed])

At this point, I'm able to add/remove/toggle items, update the content, and set a new filter. The next step is defining a new subscription that returns a filtered list of items. When the user clicks one of the filters in the UI, the list of items will be automatically filtered.

(rf/reg-sub
 :todo/visible-items
 :<- [:todo/items] ; syntactic sugar for the signal function
 :<- [:todo/filter] 
 (fn [[items todo-filter] _]
   (when (and (map? items) (keyword? todo-filter))
    (let [filter-fn (case todo-filter
                      :active    :active
                      :completed (complement :active)
                      :all       identity)]
      (filter filter-fn (vals items))))))

Another computed subscription is the count of active items left.

(rf/reg-sub
 :todo/active-items-count
 :<- [:todo/items]
 (fn [items _]
   (-> (filter :active (vals items))
       (count))))

What's left to complete the TodoMVC features are two actions: clearing the completed items and toggling all of them. We need two new event handlers to work on the list of items, either removing the ones that are completed or changing the state of all of them.

(rf/reg-event-db
 :todo/clear-completed
 (fn [db _]
   (let [items         (get-in db [:todo :items])
         ;; Extract active items ids and remove them from the items map
         cleaned-items (->> (vals items)
                            (filter (complement :active))
                            (map :id)
                            (reduce dissoc items))]
     (assoc-in db [:todo :items] cleaned-items))))

(rf/reg-event-db
  :todo/toggle-all
  (fn [db _]
    (let [items         (get-in db [:todo :items])
          ;; Is there any active item?
          active        (not-every? (complement :active) (vals items))
          ;; Set toggles as completed or not, updating the items object
          updated-items (reduce (fn [acc id]
                                  (assoc-in acc [id :active] (not active)))
                                items
                                (keys items))]
      (assoc-in db [:todo :items] updated-items))))

You can explore the entire source code in app.cljs after opening the sandbox.

Extra - Debugging re-frame

As React provide it's own Developer Tools with a Chrome extension, re-frame offers some integrated dashboards as libraries like re-frame-10x or re-frisk that are capable to provide a UI to explore the application state, show the events history and even navigate back in time reversing the app state to previous versions. Not bad!

[^helix]: An alternative library which supports the modern way to write React functional components is called Helix.