Project Isidore Design Notebook

Lisp was the language for research into artificial intelligence before the AI winter. Now the field calls itself machine intelligence. The aim remains much like the story of Icarus: to ape the most noble aspect of man, his rational nature. Until that is realized, humans will be first and foremost the most important audience for any computer language or software. Here is how I setup literate programming with lisp.

You likely heard of org-babel, an extension of org-mode that allows one to interleave text and code. It is comparable to a more powerful jupyter notebook. Get lisp in org-babel blocks by, adding to your init.el/user-config.el

(org-babel-do-load-languages
 'org-babel-load-languages
 '((lisp . t)))

When evaluating a code block with C-c C-c (, , in spacemacs), make sure to start SLIME first (M-x slime RET).

(princ "Hello World!")

Some may be familiar with poly-org, a MELPA package which allow multiple major modes. Naturally this comes in handy when using literate programming. It uses font-lock-mode to turn on the relevant major mode when your cursor is inside said code block. This saves you from having to call org-edit-special repeatedly.

Furthermore, for most languages you can only evaluate the entire code block. Not so for lisp. M-x slime-compile-defun and M-x slime-compile-region do as they say on the tin: compile the specific function or highlighted region at cursor. Poly-org breaks these functions slightly as they do not treat #+begin_src and #+end_src as the start and end-of-file respectively. The following emacs lisp snippet fixes that.

(with-eval-after-load "poly-org"
  ;; sly-compile-file sends entire .org file. Narrow to span as done in poly-R
  ;; https://github.com/polymode/poly-org/issues/25
  (when (fboundp 'advice-add)
    (advice-add 'slime-compile-file :around 'pm-execute-narrowed-to-span)
    (advice-add 'slime-compile-defun :around 'pm-execute-narrowed-to-span)
    (advice-add 'slime-load-file :around 'pm-execute-narrowed-to-span)
    (advice-add 'slime-eval-defun :around 'pm-execute-narrowed-to-span)
    (advice-add 'slime-eval-last-expression :around 'pm-execute-narrowed-to-span)
    (advice-add 'slime-eval-buffer :around 'pm-execute-narrowed-to-span)))

Since we've already shaved the editor yak up to this point, the last package I'd like to mention that makes literate programming possible is org-tanglesync]]. To "tangle" a file is, in literate programming parlance, to extract just the source code from a document. Then to de-tangle (two-way sync) has always been a problem, and traditionally org-babel-detangle has relied upon unsightly link comments to do so.

Org-tanglesync keeps your .git controlled source code and your .org mode file in sync. And it does that without any markings or artifacts on the tangled code, making collaboration easier. If a more elegant solution to de-tangle a file exists out in the wild, please do let me know.

1. Finding new Libraries & Surveying the Ecosystem

   Looking for a library? In addition to online search queries, use command(s)

   (ql:system-apropos :library) ; search for term in quicklisp dist
   (ql:who-depends-on :library) ; usage in lisp ecosystem
   (ql-dist:dependency-tree :library) ; number of dependencies upstream
   ;; See quicklisp-stats README for example usage.
   (ql:quickload "quicklisp-stats") ; look at quicklisp download stats
   (ql:quickload "quicksearch")
   (qs:? "bluetooth" :du 100)
   

   In addition, look to Sabracrolleton's detailed reviews of Common Lisp libraries. Another great hint to library quality is the Github page for Zach Beane, who does the thankless job of maintaining Quicklisp.

   Quicklisp in one respect is less like Javascript's Node Package Manager and more like Debian's apt. Zach makes sure that ALL libraries on Quicklisp build together. He takes this burden upon himself so the end users might avoid dependency hell.

   A helper to gather all lisp system's dependencies · GitHub

   • Quicklisp beta releases
   • Quickref
   • GitHub - CodyReichert/awesome-cl: A curated list of awesome Common Lisp frame…
   • CLiki: the common lisp wiki
   • GitHub - 40ants/lisp-project-of-the-day: Here I'll post notes about Quicklisp…
   • A Survey of Simple Vector Addition Performance of the Numerical Computing Lib…

2. System Dependency graph

   Call sly-eval-buffer on the following code block to update the graph.

   ;; https://40ants.com/lisp-project-of-the-day/2020/05/0063-asdf-viz.html
   ;; Inside shell "sudo apt install graphviz".
   (ql:quickload :cl-dot)
   ;; Not present in quicklisp, retrieve from https://github.com/guicho271828/asdf-viz
   (ql:quickload :asdf-viz)
   (ql:quickload :project-isidore) ; also loads cl-ppcre.
   (setf cl-dot:*dot-path* (string-trim '(#\space #\newline) (second (ppcre:split " "(nth-value 0 (uiop:run-program "whereis -b dot" :output :string))))))
   ;; Tilde char "~" in destination pathname throws an error.
   (asdf-viz:visualize-asdf-hierarchy (asdf:system-relative-pathname "project-isidore" "assets/project-isidore-dependency-graph.png") (list (asdf:find-system :project-isidore)))
   ;; "asdf-viz" also can draw class hierarchies and call graphs.
   

   

   Transitive dependencies & Lines of Code from running,

   cd ~/quicklisp/dists/quicklisp/software/
   find . -name '*.lisp' | xargs wc -l
   

   Date	Version (Commit)	# of Libraries	LOC
   2021-11-05	v1.1.0 (7cc0598)	35	251532
   2021-12-23	v1.2.0 (fd7c9f3)	42	264852

3. Javascript Dependencies
   • Highlight.js 11.2.0 Upgrade via editing :head under org-publish-project-alist.
   • Mathjax 2.7.0 See C-h v org-html-mathjax-options and org-html-mathjax-template.

4. Lisp Web Example Projects & Misc. Resources

   Cool libraries to check out:

   Library	Author	Function
   CL-CSV	Edward Marco Baringer	Spreadsheet
   CLOG	David Botton	Websockets
   CL-Unification	Marco Antonetti	Generalized pattern matching
   Cells	Kenny Tilton	Dataflow (GUI)

   • L-99: Ninety-Nine Lisp Problems
   • https://cells-gtk.common-lisp.dev/cgtk-primer.html
   • https://nullprogram.com/blog/2013/06/10/ (WebGPU is the successor to WebGL).
   • GitHub - byulparan/websocket-demo: websocket/webgl demo in CommonLisp
   • https://www.cs.cmu.edu/Groups/AI/html/faqs/lang/lisp/
   • Duncan Bayne / heroku-app-clozure-common-lisp · GitLab. Used as a base template. Hence the AGPL License on my own derived work.
   • GitHub - gongzhitaao/orgcss: Simple and clean CSS for Org-exported HTML Used as a base template.
   • bendersteed / bread and roses · GitLab
   • GitHub - vindarel/lisp-web-template-productlist: A web template with Hunchent…
   • cl-bootstrap/demo at master · rajasegar/cl-bootstrap · GitHub
   • GitHub - mtravers/wuwei: WuWei – effortless Ajax web UIs from Common Lisp
   • https://common-lisp.net/
   • https://google.github.io/styleguide/lispguide.xml
   • Tools of the Trade, from Hacker News. List of SaaS services
   • GitHub - 255kb/stack-on-a-budget: A collection of services with great free ti…
   • GitHub - ripienaar/free-for-dev: A list of SaaS, PaaS and IaaS offerings that…
   • Lisp Web Server From Scratch using Hunchentoot and Nginx | Zaries's Blog
   • ergolib/web at master · rongarret/ergolib · GitHub
   • GitHub - eigenhombre/weeds: Work in progress porting my blog software to Comm…
   • GitHub - mmontone/cl-forms: Web forms handling library for Common lisp
   • lispm comments on Steel Bank Common Lisp (SBCL) 1.4.7 released
   • READ EVAL PRINT — It's alive! The path from library to web-app.
   • GitHub - no-defun-allowed/nqthm: nqthm - the original(ish) Boyer-Moore theore…
   • GitHub - urweb/urweb: The Ur/Web programming language
   • LISA - a production-rule system for the development of Lisp-based Intelligent…
   • EPTCS 359: ACL2 Theorem Prover and its Applications
   • GitHub - hemml/OMGlib: A Common Lisp library to build fully dynamic web inter…
   • https://imps.mcmaster.ca/imps-system/README
   • GitHub - personal-mirrors/maxima: 🔰 Personal hopefully daily updated Maxima m…
   • GitHub - svspire/YACC-is-dead: YACC is dead for Common Lisp (based on http://…

Some measure of quality assurance through unit, integration and regression testing upon commit. This is done in MAKE.LISP, see also TESTS.LISP.

Building an executable binary for all 3 major operating system platforms on the x86-64 architecture. Binaries built this way may be downloaded through the Github release interface. This is done as the final step of MAKE.LISP by calling (asdf:make "system-name"). More esoteric architecture/OS pairs will have to compile from source, refer to the SBCL compiler supported platform table.

Lisp being amenable to image based development means an executable binary in the Lisp world saves the entire global state (the stack is unwound), libraries and SBCL runtime included.

Having to go through initial setup of a new computer dampens the otherwise joyous occasion of unwrapping new hardware. The pain is multiplied if it happens to be one's job to manage many computers. In its most general form, nerds around the world are busy tackling the problem of software reproducibility. NixOS and Guix System are the ones currently well-known. I would prefer Guix over NixOS myself on account of preferring Guile Scheme over the Nix Expression Language (self-admittedly designed with the goal of not being a full-featured, general purpose language). I was not able to get Guix System (version 1.3.0) working on Oracle's aarch64 A1-Flex VMs. Please email me if you have. In the meantime I am using and enjoying consfigurator for declarative server configuration, at least avoiding another Tower of Babel situation.

Current infrastructure diagram:

Note to self, before scaling this solution horizontally, consider the COST of doing so. https://groups.csail.mit.edu/tds/papers/Lynch/MIT-LCS-TM-394.pdf

Prerequisites:

• Project Isidore source code.
• Common Lisp development environment set up.
• Basic familiarity with asymmetric encryption.

• Deploy Project Isidore

  Shell access with Root SSH credentials to a Debian machine is needed. Whether this is a physical server or a virtual private server is up to you. I have a guided setup on Oracle Cloud Infrastructure.

  Read PRODUCTION.LISP to properly supply SSH credentials and SSL keys.

• DNS Resolution - associate domain name with public IP addresses

  Optional: Enable IPv6 on Oracle Cloud Infrastructure.

  Visitors to our website won't want to key in http://140.291.294.154:8080. Our Domain Registrar will point http://my-domain-name.com to http://140.291.294.154. Purchase a domain name if needed.

  I am not getting paid to say this, but if you are in the market for a Domain Name Registrar, I am a happy customer of Porkbun. Unlike Google, you can reach a human being. Three cheers for Porkbun, and may they grow in success without losing their human touch.

  First time customer, brand new account. Foolishly tried to register a domain on my overcharged credit card on the first try. Account was flagged and locked (understandably). Paid off credit card, contacted chat support, account was unlocked and order completed in less than 5 minutes. Lovely experience. Appreciate the cute graphics as well. – review by some stranger named Ben

  Cloudflare seem like an ethical company from what I can gather from their blogs and I'll be making use of their generous free tier in the next steps. Oracle has my gratitude, but my recommendation of Porkbun and Cloudflare comes without reservations.

  Search online for current methods to create a DNS A records (and/or AAAA records for IPV6) specific to your Domain Registrar. Port 80 is the default port for HTTP traffic, as entering http://my-domain-name.com is equivalent to entering http://my-domain-name.com. If you don't want visitors to have to key in http://my-domain-name.com:8080/ then either read on, use OCI's docs to redirect a subdomain to a URL with a port number, or configure your web server to listen on – what is on UNIX a privileged port – 80. Cloudflare can also redirect all HTTP requests to HTTPS instead of doing this on the server NGINX configuration. This allows me to close port 80 and only have port 443 exposed to the public.

• Denial-of-Service protection and Content Delivery Network with Cloudflare

  Cloudflare here acts as a reverse proxy (man-in-the-middle) between the website visitor (client) and the server; it provides protection as well as caching services. Now by registering your domain with Cloudflare your domain will be served as HTTPS to the client. But communications from Cloudflare to our server is still via un-encrypted HTTP. We remedy this by using Origin Certificates.

  After setting up Authenticated Origin Pulls the origin server now only accepts HTTP requests that use Cloudflare's valid client certificate. There will be no more directly connecting to the web server via IP address.

  This helps Cloudflare's Web Application Firewall and its blocking of automated WordPress vulnerability scanning. WordPress powers 43% of all websites. This means any public website will face a barrage of probing attacks targeting WordPress vulnerabilities. As an example, use Web Application Firewall to block all requests with URI paths containing "php" or "wp-includes" from reaching Nginx.

• Developing a remote LISP image

  Of course with the incremental re-defintion in the LISP REPL and our object database, Rucksack, having update-instance-for-redefined-class it would be fun to imagine the flexibility if we could connect a local sly to our production LISP image. Doing so gives us great observation capabilities and is an cool technique to add to our toolbox. What has by now become the stuff of internet legend, Ron Garret's story of Debugging code from 60 million miles away, continues to serve as an inspiration.

  SLY User Manual, version 1.0.42

  For an example of how this application starts a slynk server see da1f7fa. There is no need to allow ingress on OCI's virtual cloud network or in firewalld for port 4005. The 4005 ports on both the local and remote machines are communicate via SSH port 22. Create the SSH tunnel on the local machine,

  ssh -L4005:localhost:4005 pi-server
  

  M-x sly-connect RET "localhost" RET 4005 RET.

  I was playing around with connecting sly to a remote LISP image. I wondered if slynk:create-server would still work after sbcl:save-lisp-and-die. I mean there's no reason to think it wouldn't, but I had to move the slynk:create-server form to the toplevel function used by sbcl:save-lisp-and-die. Otherwise "sudo ss -tulpn | grep LISTEN" didn't show the open port. Mentioning this if it saves somebody else some time.

• Crack open a cold one.

  Optional: with the boys.

The most likely worse case scenario is a frontpage post to some link aggregate website such as Reddit. The number of requests made to the server per client obviously varies depending on architecture and workload.

The plural of anecdote is data I hope, so if I'm allowed to draw a similarity between the application in the above anecdote and my own, I should ballpark for around 300 requests per second. Cloudflare is the star of the story here, and I will continue to sing their praises: their CDN allows the caching of my static assets.

I originally spent some time seeing if there was a convenient way to hide .html from the URL when serving blog entries. Org-publish generates my static blog, and what seemed like a minor annoyance at first proved useful when setting up Cloudflare page rules; it took one rule to tell Cloudflare to cache all URL's ending in .html. They have a very reasonable usage policy for their CDN (see section 2.8 of their EULA). OCI deserves applause in tandem, for their 10TB free data egress. Thank you to abuse-prevention teams in both these companies! It makes the homegrown part of the internet possible.

We use loader.io to benchmark our application.

15 clients per second for a duration of 1 minute. HTTP Resource: https://www.hanshenwang.com/bible/1-1-1/1-1-31

300 clients per second for a duration of 1 minute. HTTP Resource: https://www.hanshenwang.com/about

300 clients per second for a duration of 1 minute. HTTP Resource: https://www.hanshenwang.com/assets/blog/installation-of-spacemacs.html

For a primer on high performance LISP web servers see Woo: a high-performance Common Lisp web server. It should be pointed out that the hunchentoot listed on Woo's benchmark graph is the single threaded version. The multi-threaded version benchmarks are more impressive. The article about Woo also fails to mention quux-hunchentoot which employs a thread-pooling taskmaster as an extension to Hunchentoot.

These stress tests are run with sudo apt install wrk on OCI's A1 Flex VM's with 4 cores and 24GB at commit d0f10c5.

1. Cl-tbnl-gserver-tmgr

   [email protected]:~$ wrk -t4 -c100 -d10 "http://localhost:8081/about"
   Running 10s test @ http://localhost:8081/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency     2.94ms    2.84ms  43.51ms   90.52%
       Req/Sec     0.91k   509.03     1.73k    46.00%
     27146 requests in 10.02s, 50.51MB read
   Requests/sec:   2709.57
   Transfer/sec:      5.04MB
   
   [email protected]:~$ wrk -t4 -c100 -d10 "http://localhost:8081/about"
   Running 10s test @ http://localhost:8081/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency     2.93ms    2.82ms  51.14ms   90.58%
       Req/Sec     0.91k   660.73     2.08k    64.67%
     27306 requests in 10.02s, 50.81MB read
   Requests/sec:   2726.07
   Transfer/sec:      5.07MB
   
   [email protected]:~$ wrk -t4 -c100 -d10 "http://localhost:8081/about"
   Running 10s test @ http://localhost:8081/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency     2.92ms    2.82ms  49.55ms   90.39%
       Req/Sec     1.37k   808.96     2.52k    52.00%
     27355 requests in 10.02s, 50.90MB read
   Requests/sec:   2731.18
   Transfer/sec:      5.08MB
   

2. Default multi-threaded Hunchentoot

   [email protected]:~$ wrk -t4 -c100 -d10 "http://localhost:8082/about"
   Running 10s test @ http://localhost:8082/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency    64.00ms  131.31ms   1.95s    92.41%
       Req/Sec   504.07    391.26     1.67k    71.28%
     14308 requests in 10.05s, 26.62MB read
     Socket errors: connect 0, read 0, write 0, timeout 36
   Requests/sec:   1423.18
   Transfer/sec:      2.65MB
   
   [email protected]:~$ wrk -t4 -c100 -d10 "http://localhost:8082/about"
   Running 10s test @ http://localhost:8082/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency    65.20ms  142.16ms   1.96s    93.24%
       Req/Sec   416.33    321.55     1.55k    69.21%
     14800 requests in 10.08s, 27.54MB read
     Socket errors: connect 0, read 0, write 0, timeout 40
   Requests/sec:   1468.68
   Transfer/sec:      2.73MB
   
   [email protected]:~$ wrk -t4 -c100 -d10 "http://localhost:8082/about"
   Running 10s test @ http://localhost:8082/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency    65.67ms  145.15ms   2.00s    93.67%
       Req/Sec   452.20    421.83     2.00k    74.68%
     14152 requests in 10.06s, 26.33MB read
     Socket errors: connect 0, read 0, write 0, timeout 35
   Requests/sec:   1406.08
   Transfer/sec:      2.62MB
   

3. Clack with Woo and libev-dev

   [email protected]:~$ wrk -t4 -c100 -d10 "http://localhost:8083/about"
   Running 10s test @ http://localhost:8083/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency    43.78ms   29.71ms 221.07ms   88.44%
       Req/Sec   634.03    234.05     0.91k    77.25%
     25272 requests in 10.02s, 47.02MB read
   Requests/sec:   2522.89
   Transfer/sec:      4.69MB
   
   [email protected]:~$ wrk -t4 -c100 -d10 "http://localhost:8083/about"
   Running 10s test @ http://localhost:8083/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency    55.64ms   72.08ms 597.50ms   92.30%
       Req/Sec   630.25    245.64     0.95k    76.80%
     24364 requests in 10.02s, 45.33MB read
   Requests/sec:   2432.66
   Transfer/sec:      4.53MB
   
   [email protected]:~$ wrk -t4 -c100 -d10 "http://localhost:8083/about"
   Running 10s test @ http://localhost:8083/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency    44.21ms   29.66ms 204.73ms   88.33%
       Req/Sec   629.64    227.31     0.89k    78.00%
     25085 requests in 10.01s, 46.67MB read
   Requests/sec:   2505.62
   Transfer/sec:      4.66MB
   

4. Quux-Hunchentoot Thread Pool

   [email protected]:~$ wrk -t4 -c100 -d10 http://localhost:8080/
   Running 10s test @ http://localhost:8080/
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency     3.34ms    4.82ms 166.69ms   94.11%
       Req/Sec   820.70    791.93     2.29k    72.15%
     24371 requests in 10.04s, 160.35MB read
   Requests/sec:   2426.30
   Transfer/sec:     15.96MB
   [email protected]:~$ wrk -t4 -c100 -d10 http://localhost:8080/
   Running 10s test @ http://localhost:8080/
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency     3.13ms    2.97ms  43.67ms   93.27%
       Req/Sec     1.28k   452.76     1.89k    72.50%
     25593 requests in 10.05s, 168.39MB read
   Requests/sec:   2546.40
   Transfer/sec:     16.75MB
   [email protected]nstance:~$ wrk -t4 -c100 -d10 http://localhost:8080/about
   Running 10s test @ http://localhost:8080/about
     4 threads and 100 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency     2.86ms    2.70ms  52.46ms   94.02%
       Req/Sec     0.94k   653.16     2.06k    64.67%
     27966 requests in 10.04s, 52.09MB read
   Requests/sec:   2784.27
   Transfer/sec:      5.19MB
   

Because MAKE.LISP fiddles with compiler knobs in search of performance, a single [email protected] in production can reach upwards of 3300 requests per second. To improve application resiliency Nginx is used to load balance between 12 project-isidore processes each at 2GB of RAM. Lastly, of note is the 4GBps network bandwidth offered by Oracle Cloud. My bottleneck should be at this point the rucksack database, despite offering concurrent transactions.

So with the bare minimum amount of testing done I can say with confidence that my website is well prepared given the restraints on cost, time, and money.

This website is run as a non-profit hobby, for no advertising is or will be displayed. Dollar amounts are in CAD. Thank you to all patrons.

This section is now outdated. I implemented a trivial PostgreSQL mailing list but running what amounts to your own mail server that does not have its messages marked by the big email providers as spam is most definitely non-trivial. As of commit 7a4fc5d, I have switched to Mailchimp. On production instances, there's a savings of 138MB-92.8MB=45.2MB RAM.

PostgreSQL has very clear and structured documentation. Refer to the documentation to install PostgreSQL locally on your computer. Afterwards a good introduction to basic Create, Read, Update, Delete (CRUD) operations is here: 12 Steps to Build and Deploy Common Lisp in the Cloud (and Comparing Rails) |…

Documentation on Postmodern is better than your average Common Lisp library. Still to supplement the official docs are examples and specifically examples using the Data Access Objects.

1. To start PostgreSQL server process

# Install PostgreSQL.
sudo apt install postgresql
# Start server process. PostgreSQL defaults to PORT 5432
sudo service postgresql start
# Login as PostgreSQL Superuser "postgres" via "psql" Client
sudo -u postgres psql
# Create database "Test"
createdb test
# Delete database "Test"
dropdb test
# Login as superuser to create user for database "Test"
sudo -u postgres psql
# See defparameter `*local-db-params*' in MODEL.LISP.
CREATE USER user1 WITH PASSWORD 'user1';
# Use Shell to login as host: localhost with database: test and user:user1
psql -h localhost -d test -U user1
# Once logged in,
test=# select * from tablename;

Design constraints imposed by the current deployment platform, Heroku & Github.

• Heroku managed PostgreSQL free tier limitations = 10000 rows, 1GB disk capacity.
• Heroku free tier dyno memory (automatic dyno shutdown at 1GB) = 512 MB.
• Heroku free tier slug size = 500 MB.
• Github large file limit = 100MB-2GB.

Characteristics of Bible dataset:

• Read-only data.
• Dataset should be available offline.
• Non-expanding dataset.
• Fits within Heroku free tier dyno memory (18-20MB). Online reports of 80-140mb RAM usage by hunchentoot + ironclad.
• Limited developer resources mean instead of programming/debugging in LISP, I would need to master a second domain specific programming language: SQL.
• Very cost sensitive (cut me some slack, I'm a college student).

Object Relational Mappers (ORM) are notoriously hard to get right. It is too bad the pure LISP persistence solutions (Allegrocache) remain proprietary. For open source solutions, I still think bknr.datastore is among the best for now. Rucksack by Arthur Lemmens is also worth playing with, but due to the restrictions of the Heroku ephemeral filesystem, the library with the best fit for my application would be bknr.datastore.

Memory-Centric Data Management A Monash Information Services White Paper by Curt

1. Monash, Ph.D. May, 2006, accessible at http://www.monash.com/whitepapers.html

See pg 668 of weitzCommonLispRecipes2016 for cookbook recipes on BKNR.DATASTORE.

Object Prevalence : An In-Memory, No-Database Solution to Persistence | by Pa…

https://nicklevine.org/lisp-book/contents/chac.pdf

https://cl-pdx.com/static/persistence-lemmens.txt

Rucksack measurements are listed first, then BKNR.Datastore.

PROJECT-ISIDORE/VIEWS> (time (get-bible-text 23))
Evaluation took:
  0.000 seconds of real time
  0.000098 seconds of total run time (0.000095 user, 0.000003 system)
  100.00% CPU
  228,122 processor cycles
  0 bytes consed

PROJECT-ISIDORE/VIEWS> (time (get-bible-text 23))
Evaluation took:
  0.000 seconds of real time
  0.000020 seconds of total run time (0.000020 user, 0.000000 system)
  100.00% CPU
  36,872 processor cycles
  0 bytes consed

PROJECT-ISIDORE/VIEWS> (time (bible-page "1-1-1-2-2-2"))
Evaluation took:
  0.330 seconds of real time
  0.328973 seconds of total run time (0.328973 user, 0.000000 system)
  99.70% CPU
  821,152,853 processor cycles
  30,101,264 bytes consed

PROJECT-ISIDORE/VIEWS> (time (bible-page "1-1-1-2-2-2"))
Evaluation took:
  0.440 seconds of real time
  0.441031 seconds of total run time (0.431896 user, 0.009135 system)
  [ Run times consist of 0.017 seconds GC time, and 0.425 seconds non-GC time. ]
  100.23% CPU
  1,098,042,384 processor cycles
  91,860,080 bytes consed

PROJECT-ISIDORE/VIEWS> (time (bible-page "1-1-1-73-22-21"))
Evaluation took:
  8.990 seconds of real time
  8.998338 seconds of total run time (8.102798 user, 0.895540 system)
  [ Run times consist of 0.077 seconds GC time, and 8.922 seconds non-GC time. ]
  100.09% CPU
  22,446,254,265 processor cycles
  796,903,104 bytes consed

PROJECT-ISIDORE/VIEWS> (time (bible-page "1-1-1-73-22-21"))
Evaluation took:
  0.660 seconds of real time
  0.659741 seconds of total run time (0.641298 user, 0.018443 system)
  [ Run times consist of 0.025 seconds GC time, and 0.635 seconds non-GC time. ]
  100.00% CPU
  1,641,964,703 processor cycles
  327,536,880 bytes consed

A sampling of postmodern DAO speeds,

PROJECT-ISIDORE/MODEL> (time (friend-email (mailinglist-get 2)))
Evaluation took:
  0.010 seconds of real time
  0.002794 seconds of total run time (0.002794 user, 0.000000 system)
  30.00% CPU
  25,589,575 processor cycles
  32,432 bytes consed

"[email protected]"

And a regular SQL query done in postmodern,

PROJECT-ISIDORE/MODEL>  (time (pomo:with-connection (db-params) (pomo:query (:select 'email :from 'mailinglist :where (:= 'id 2)) :single)))
Evaluation took:
  0.010 seconds of real time
  0.003089 seconds of total run time (0.003089 user, 0.000000 system)
  30.00% CPU
  24,084,690 processor cycles
  16,368 bytes consed

"[email protected]"
1 (1 bit, #x1, #o1, #b1)

A sampling of cl-sqlite with the in-memory database,

SQLITE>
(time (execute-single *db* "select id from users where user_name = ?" "dvk"))
Evaluation took:
  0.000 seconds of real time
  0.000047 seconds of total run time (0.000045 user, 0.000002 system)
  100.00% CPU
  108,448 processor cycles
  0 bytes consed

2 (2 bits, #x2, #o2, #b10)

and with a regular disk based database,

SQLITE> (time (execute-single *dba* "select id from users where user_name = ?" "dvk"))
Evaluation took:
  0.000 seconds of real time
  0.000085 seconds of total run time (0.000081 user, 0.000004 system)
  100.00% CPU
  196,908 processor cycles
  0 bytes consed

2 (2 bits, #x2, #o2, #b10)

For this admittedly shallow testing, BKNR.Datastore and Rucksack perform admirably! Quicklisp-stats shows BKNR.Datastore is still in use, with the most recent example being a startup by an ex-Facebook engineer. The most recent mention I can find of Rucksack is by Ravenpack, at ELS2020.

The quicklisp download stats for Rucksack show an 200% increase (from NIL > 223/239) around the months of January and February. I find it plausible that Ravenpack, who does data analysis for the financial sector, have their engineers repull all libraries from quicklisp once per year; I have noticed same patterns for some other libraries.

BKNR.Datastore is the reason I am able to structure the Tabular Douay Rheims in the way that I have. Any disk based solution would have been too slow, forcing me to cache the pages or store them as static files. Many thanks to Hans Hubner, the author of said library. And yes, cl-store or simple serialization would have sufficed for my use case but I'll take Edmund Weitz's word when he recommends BKNR.Datastore over cl-store in his ebib:weitzCommonLispRecipes2016.

The question of ORM's comes up again. As of the current writing, I am aware of a handful of options.

Elephant - interfaces to the C databases (Berkeley DB, PostgreSQL) most likely bitrotted, perhaps the author got hired to work on allegrocache? Quicklisp-stats show 35 downloads total for the past 2 years.

CLSQL - by far the greatest number of backends supported, with the necessary compromises that suggests. Tradition extends back many years, shares ideas with CommonSQL and has a SOLID track record. Packaged for Debian. Great documentation.

Hu.dwim.perec - Originally started as a fork of CLSQL. Greatly extends the ORM capabilities and is kept up to date: I think mostly by one person, Levente Mészáros. This is one of the libraries that show almost exact same download patterns as Rucksack. So what's the catch? Very little documentation. You will have to dive deep into a bunch of hu.dwim.* packages and look at tests. Pretty much pulls in the entire hu.dwim.* ecosystem with it when downloaded from quicklisp. With great power comes…

Mito - Fukamachi-ware. Similar download stats to CLSQL. Very young project, started in 2016. Also see Crane, the stats show close to zero usage though.

If one is willing to make the trade-off of SQL for object persistence, then as far as I know there are really only two pure lisp options. I say "only" but I'm not aware of any other language with libraries comparable to the ones below.

Rucksack (open source) - as mentioned earlier, authored by hacker Arthur Lemmens (worked at Ravenbrook). Small, written in portable lisp, possessing performance that isn't bad at all, a real gem of a project. The mailing lists of Elephant and Rucksack show some attempt made to combine the ideas of Rucksack into a pure lisp backend for Elephant. Rucksack also shows a lack of recent updates, but unlike Elephant, has users to this day. Ain't it beautiful how the stability of the language shines through the library? Don't let the date of the last commit turn you away. Give it a shot, look at the talk-eclm2006.txt file under the /docs folder.

Allegrocache (proprietary, Franz Inc.) - what Rucksack could have been if you threw a bunch of money at the problem. Tightly integrated with the rest of the Allegro CL ecosystem. You do get what you pay for in this case. I have heard they have great support too. Allegrocache was originally based on ObjectStore (Dan Weinreb of Symbolics fame + others). Dan does a fair job at defending object oriented database management systems here. I would like to point out that Glenn D. House of 2Is (DoD contractor) testifies (21:30) to the conclusions found in Prechault and Garret when comparing Lisp v. Java v. C/C++. Grammatech is also a DARPA funded shop that uses Common Lisp. I would also be remiss if I did not mention the recent milestone of CLASP (Common Lisp implementation with C++ interop) version 1.0. The geriatric IP of Symbolics is still closed source; rumor is there are still legacy DoD contracts and that American Express's fraud detection used to (up until the mid 2000's?) use Open Genera.

Jacobs, J. H., M. R. Swanson, and R. R. Kessler. Persistence is Hard, Then You Die! or Compiler and Runtime Support for a Persistent Common Lisp. Technical report, Center for Software Science, University of Utah, 1994. UUCS-94* 004, 1994.

To view the entire revision history of an article, find and click the article title on the repository's blog subfolder. Then click on the History button to view specific, atomic changes. Project release notes are available for a general overview and inputting the article URL into the Internet Wayback Machine is also an option.

To remove an email address from the mailing list, fill out and submit the form on the unsubscribe page. To resubscribe, visit https://www.hanshenwang.com/subscribe.

To access the website offline, download the appropriate executable from the project release page. Releases · HanshenWang/project-isidore · GitHub.

Find the appropriate exectuable matching both your computer's processor architecture and the operating system. Currently only the x86-64 architecture is supported on Ubuntu, MacOS and Windows operating systems.

Project Isidore does not offer signed binaries for MacOS, therefore you will have to manually execute the unsigned binaries. Please see https://lapcatsoftware.com/articles/unsigned.html for more details.

An audit of the source code can be done at any time. Please see the source repository as well as the third party dependencies.

Be advised that the program consumes around 50MB of RAM when used by a single user locally. Please understand the executable is provided AS IS, WITHOUT WARRANTY. See the provided COPYING.txt included in the download.

The documentation is organized according to the best practices outlined here: The documentation system — divio.

The closest thing to a tutorial as understood by the divio documentation system ought to be the development quickstart (present in the README.org) or embedded as close as possible to the end-user interface. How-to guides are meant to be placed here in the user manual. The Reference is auto-generated with the help of the Declt system. The Explanation is this Design Notebook blog article and git commit messages.

The Declt Common Lisp library is used generate the reference manual in .texi texinfo format. GNU Texinfo is able to convert a single .texi source file into online HTML format, PDF documentation as well as others. On a high level, Declt uses ASDF and SBCL contrib sb-introspect to query and extract documentation strings, lambda lists, slot type, allocation and initialization arguments, and definition source files. It requires no special architecture choices in a system, other than to conform to ASDF conventions. Beyond that, writing clear docstrings where the opportunity arises will yield great results.

generate-doc.lisp script is run by a git pre-commit hook. Upon every commit, it will regenerate the reference manual and include changes into the commit. pre-commit is originally located at /project-isidore/.git/hooks/pre-commit.sample.

#!/bin/sh

# Generate updated manual. SBCL must be installed. See documentation for
# environment setup
sbcl --load /home/ben/quicklisp/local-projects/project-isidore/src/generate-doc.lisp
# Add updated manual.html to commit
git add /home/ben/quicklisp/local-projects/project-isidore/assets/reference-manual.html