• Paper 0: inherit a gigantic .bib from advisor and add new references as needed
• Paper n: pick the .bib from some paper k (with 0 ≤ k < n) and add new references as needed

So here I was, sitting with ~50kLOC worth of .bib files I needed to organize. On the other hand, I wanted to collect all the references in a single section of the thesis to avoid repetetition and inconsistencies. So, in principle I could have simply used the good ’ol bibtool utility to merge all my .bib files together:

$ bibtool -s foo.bib bar.bib baz.bib

The problem with this approach is that I would have ended up again with a gigantic .bib file when, in reality, I only needed a handful of entries from it. To solve this, I put together a small bash script that pulls references from .bib files on demand. The trick is to ask for forgiveness rather than for permission:

1. Compile the project with an empty .bib file
2. Parse the log file to find which citations are missing references
3. Extract each of those references from some existing .bib file and append them to the project’s .bib file
4. Compile the project again, now without missing references :D

Finding missing citations

If we inspect the log files created by pdflatex/bibtex, missing citations are reported as:

LaTeX Warning: Citation 'afl' on page 21 undefined on input line 16664.

So we need to find these warnings, extract the citation keys (i.e., afl in the line above) from them and remove any duplicates:

LOGFILE=thesis.log

MISSING=$( \
  grep "LaTeX Warning: Citation" $LOGFILE |\
  awk -F' ' '{print $4}' | tr -d \' |\
  sort | uniq \
)

With the culprits at hand, we can move onto the next step.

Retrieving citations on demand

To find if a concrete reference exists in a given citation file, we can use bibtool’s select resource:

find_ref () {
  bibtool -q -r biblatex '--expand.macros=ON' '--print.all.strings=OFF' '--select{$key "'$1'"}' $2
}

This way, find_ref $key $bibfile will try to find the reference $key in the file $bibfile, returning either its corresponding entry if it can find it, or an empty string otherwise. The -r biblatex option is needed to allow indexing @online entries, whereas the expand.macros and print.all.strings options are needed to ensure that string macros are expanded, so we don’t need to print them everytime we look for a citation key.

Then, we can iterate over the existing .bib files to see if we find each missing citation:

BIBFILES=$(ls bib/*.bib) # Existing .bib files
OUTPUT=references.bib # The project's .bib file

for missing in $MISSING; do
  echo -n "Looking for $missing: "

  # Reference is already there
  FOUND=$(find_ref $missing $OUTPUT)
  if [ ! -z "$FOUND" ]; then
    echo "Already in in $OUTPUT!"
    continue
  fi

  # Reference is in some .bib file
  for bibfile in $BIBFILES; do
    FOUND=$(find_ref $missing $bibfile)
    if [ ! -z "$FOUND" ]; then
      echo "Found in $bibfile!"
      echo "$FOUND" >> $OUTPUT
      break
    fi
  done

  # Reference is M.I.A.
  if [ -z "$FOUND" ]; then
    echo "Could not find any entry!"
  fi

done

The reason we first look in the project’s .bib file is to avoid producing duplicate entries when:

• The user already added them manually, or
• We have an overlapping entry key that matches a previously searched missing reference. ¹

Trying it out

We can now run this little snippet to gather only the references we need:

Looking for afl: Found in bib/foo.bib!
Looking for asan: Found in bib/bar.bib!
Looking for peach: Found in bib/baz.bib!
...

And my curated .bib file now has “only” a little over 2000 lines of code ¯\_(ツ)_/¯

You can find the bash script here in case you want to give it a try.

For the most part, this development container works flawelessly. Except for this: the user’s cabal folder (located at ~/.cabal by default) is not persisted across container runs. This means I need to rebuild from scratch most of the dependencies that cabal saves in its user-wide package store (~/.cabal/store) every time I reopen the project (notably annoying with Hakyll’s dependency of big-chungus pandoc).

There are a couple of ways to solve this:

• By configuring cabal to use a package store located inside the repository’s folder (which is bind-mounted to the host’s file system).

  Problem: this would duplicate the work for any dependency shared across Haskell projects.

• Creating a ~/.cabal folder in my host computer and share it among devcontainers.

  Problem: this folder would be continuously modified by root, a permission hell waiting to happen.

• Mounting the package store to a Docker named volume.

  This is what I ended up doing, as it takes the least work and doesn’t clutter my host’s $HOME.

  The only changes needed are adding the following properties to .devcontainer/devcontainer.json:

  "mounts": [
      "source=cabal_store,target=/home/vscode/.cabal/store,type=volume"
    ],
  "postCreateCommand": "sudo chown vscode /home/vscode/.cabal/store"

  Where cabal_store is the name of the Docker volume to be created and vscode is the Linux user of this devcontainer. The postCreateCommand is there to make sure our container can access and modify the package store (only needed in case your devcontainer logs in to a user other than root). Repeat this for every Haskell devcontainer you want to share the cabal store with.

That’s it! Signing off.

Makefile
keyboards
  |-- preonic
  |     |-- keymap.c
  |     |-- config.h
  |     |-- rules.mk
  |     \-- env
  \-- thekey_v2
        |-- keymap.c
        |-- config.h
        |-- rules.mk
        \-- env

The content of these folders is not important today (but maybe soon). What’s relevant here is that I have a Makefile that builds a given firmware by passing the KBD variable with the folder where it’s defined:

$ make KBD=preonic
$ make KBD=thekey_v2

Now, if I want to build and publish these firmares in CI, I could simply do one after another, something like:

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Build preonic firmware
        run: make KBD=preonic

      - name: Build thekey_v2 firmware
        run: make KBD=thekey_v2

      - name: Create artifact
        uses: actions/upload-artifact@v2
        with:
          name: firmwares
          path: |
            build/*.bin
            build/*.hex

The problem with this approach is that I need to remember to go and change the CI workflow everytime keyboard N+1 suddenly appears. What we want instead is to run something like:

for kbd in $(ls keyboards); make KBD=$kbd; done

One interesting way to do this is to split the workflow into:

1. A find-targets job that “discovers” which jobs to run and saves them in an output named targets.
2. A build job that reads these targets and uses the matrix strategy to run them in parallel.

This looks like:

on: [push]

jobs:

  find-targets:
    runs-on: ubuntu-latest
    outputs:
      targets: ${{ steps.set-targets.outputs.targets }}
    steps:
      - uses: actions/checkout@v2

      - id: set-targets
        run: echo "targets=$(ls keyboards | jq -R '[.]' | jq -s -c 'add')" >> $GITHUB_OUTPUT

  build:
    needs: find-targets
    runs-on: ubuntu-latest
    strategy:
      matrix:
        KBD: ${{ fromJson(needs.find-targets.outputs.targets) }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Build firmware
        run: make KBD=${{ matrix.KBD }}

      - name: Create artifact
        uses: actions/upload-artifact@v2
        with:
          name: ${{ matrix.KBD }}
          path: |
            build/*.bin
            build/*.hex

There are a couple of things worth mentioning here:

1. The actual build step is more complicated as it uses the official qmkfm/qmk_cli Docker image to build the firmwares. The code I show here is deliberately simpler to show the idea.

2. We need the build job to depend on find-targets so they run in the correct order. This is easy to enforce by adding needs: find-targets as shown above.

3. In find-targets, we need to create a JSON array of targets, e.g. ["preonic","thekey_v2"]. For this, the set-targets step first lists the files under keyboards and progressively add them to an empty array using good ’ol jq. This array is then saved to the GITHUB_OUTPUT environment variable associated with this step. Finally, this job retrieves the targets from the output of the set-targets and assigns it to the job’s output targets. For reference, this is the new and cool way to do this now that save-output is getting deprecated.

4. In build, we define the build matrix by retrieving the targets variable from output of find-targets. GitHub will then run the build action once per target, instantiating the matrix.KBD variable with the current target name, which we use later to call make accordingly.

With this in place, GitHub will create build jobs dinamycally on push, and there’s no need to hardcode build targets anywhere :)

It’s been a very long time since I originally built this website with the sole purpose of having my publications in a single place. While doing so, I also found it was quite easy to add the necessary code to generate blog posts (like the one you’re looking at now). So, after all, why not, why shouldn’t I? To the surprise of anybody, I haven’t written anything in the 4 years this site has been up — even though I really wanted to!

Now, the reasons for not having written anything in 4 years despite really wanting to are probably unsurprising: sometimes I feel I have nothing interesting to say. Other times when I find something interesting that I think I could write about, my immediate reaction is to think “why anybody would want to read about this”. Sometimes I feel the effort needed to write something meaningful isn’t really worth it. Sometimes I’m just tired and I rather zoom out and binge-watch a TV show or play a videogame than write a long existential blob. You know, all the bits in the procrastinator’s army knife.

But one day (yesterday actually), I stumbled upon TinyProjects, a small website full of fun week-long projects. Along with that, Ben (the author) also writes daily posts as a sort of journal. After seeing some of his posts, I fell in love with the idea of writing shorter, more frequent, and more open-ended posts. So that’s what I will try, too.

But how often should I write before this becomes a chore? Daily is perhaps too much of a commitment, but monthly is definitely too little. Something between these two sounds reasonable, but I’d like to have a more concrete (however arbitrary) timescale to start feeling I’m getting behind. So here’s one: this website’s source code is built and deployed by a GitHub Action. Its code is generated by Hakyll. Hakyll depends on Pandoc to translate Markdown into HTML. Pandoc takes some 40 minutes to build inside my GitHub Action, and it would be shameful to waste 40+ minutes of CI time building Hakyll+Pandoc for every new silly post I create. Of course, my CI action uses a cache to keep everything hot’n’ready. But there’s a limit to that: GitHub caches action’s dependencies for only 7 days.

So that’s my target, to write stuff often enough to keep alive my website’s CI cache :)

My days at Chalmers are numbered. I will make them count!

Plain Text

Aliquam erat volutpat. Nunc eleifend leo vitae magna. In id erat non orci commodo lobortis. Proin neque massa, cursus ut, gravida ut, lobortis eget, lacus. Sed diam. Praesent fermentum tempor tellus. Nullam tempus. Mauris ac felis vel velit tristique imperdiet. Donec at pede. Etiam vel neque nec dui dignissim bibendum. Vivamus id enim. Phasellus neque orci, porta a, aliquet quis, semper a, massa. Phasellus purus. Pellentesque tristique imperdiet tortor. Nam euismod tellus id erat.

Links

• Here is a link.
• Here is the same link with alt text.

Images

Images are inserted using the following syntax. Captions are optional.

Math

N_f, the flux associated with the ith filter is F_i, and the weight, w_i, is defined as

w_i ≡ σ_F, i⁻²,

where σ_F, i is the uncertainty in the flux associated with the ith filter. Assume that all sums range from i = 1 to i = N_f and use the summation convention on repeated indices. Note that we strike things out like this, italicize things like this or like this, and we can make things bold like this or like this.

Unfortunately, –> and => do not turn into arrows automatically, but you can make arrows like this → and ⇒. You can also implement a Pandoc filter to add features that are not built-in. So, it would be possible to make –> turn into → and introduce new features, such as Graphviz graph rendering. For examples of such filters, check out this list (or keep reading this document).

Here’s a horizontal line / separator:

Table Creation

There are many ways of creating tables. This is an example of the pipe table creation syntax…

Here’s one more example, with a table caption:

Refer to the documentation to learn more about the other types of tables.

Line Blocks

Normally, adjacent lines are combined together, so a set of short formatted lines, such as an address or verse, becomes jumbled. To overcome this, place | at the beginning of each line, followed by a space. This preserves the formatting:

Note the indentation on the final two lines above.

Block Quotes

The syntax for block quotes is extremely simple:

You can write a block quote by putting a single ‘>’ at the beginning of a block of text or by placing the ‘>’ at the beginning of each line of the quoted block, similar to the way e-mail readers handle quoted messages.

Enumerated lists

1. You can use integers or the # symbol in enumerated lists.
2. This is quite convenient. For example, i. You don’t have to count
   2. If you want to change the order, no numbering needs to be changed
3. You can also use roman numerals, obviously.

Example Lists

1. This is Example (1).

2. This is Example (2).

Now we discuss something for a while and introduce the third example…

3. This is Example (3).

You can refer to an example by its label. For instance Example (2).

Syntax Highlighting

A Python syntax highlighting example:

You can include blocks of pre-formatted text, like this. If the text is source code, you can tell Pandoc to perform syntax highlighting. This is graphviz.py:

{-# LANGUAGE OverloadedStrings #-}

import Data.Monoid ((<>))

import System.FilePath
import Hakyll


main :: IO ()
main = hakyll $ do

  match "images/*" $ do
    route   idRoute
    compile copyFileCompiler

  match "css/*" $ do
    route   idRoute
    compile compressCssCompiler

  match "templates/*" $ compile templateBodyCompiler

  match "index.md" $ do
    route   $ setExtension "html"
    compile $ pandocCompiler
      >>= loadAndApplyTemplate "templates/default.html" defaultContext
      >>= relativizeUrls

A C++ syntax highlighting example:

Here’s another example using a different programming language (and a different method of specifying the pre-formatted text block).

#include <iostream>

int main()
{
    // comment
    std::cout << "Hello, World!\n";
    return 0;
}

Compiling this with Pandoc

To convert¹ this Markdown document to HTML, I used the following command:

# BTW, this is a Bash syntax highlighting example.

$ pandoc example.md -s --smart --mathjax \
         --css nrstyle.css \
         --highlight-style pygments \
         --columns=200 \
         --filter graphviz.py \
         -o example.html

An explanation of each parameter (as a bulleted list)

• example.md: the input file
• -s: create a stand-alone document (rather than a document fragment that lacks a header).
• --smart: automatically replace --, ---, and ... with –, —, and … and handle quotation marks properly.
• --mathjax: use the MathJax library for typesetting math in HTML documents.
• --css nrstyle.css: Use the nrstyle.css stylesheet in the HTML output document.
• --highlight-style pygments: Turn on syntax highlighting and use the pygments color scheme.
• --columns=200: set the line length to 200. This makes the tables display properly.
• --filter graphviz.py: Pass Pandoc’s abstract syntax tree through the filter program graphviz.py before rendering the output document.
• -o example.html: Specify the name and type of the output file. The format is inferred from the suffix (file extension).

The same explanation, formatted as a definition list

example.md

The input file

-s

Create a stand-alone document (rather than a document fragment that lacks a header).

--smart

Automatically replace --, ---, and ... with –, —, and … and handle quotation marks properly.

--mathjax

Use the MathJax library for typesetting math in HTML documents.

--css nrstyle.css

Use the nrstyle.css stylesheet in the HTML output document.

--highlight-style pygments

Turn on syntax highlighting and use the pygments color scheme.

--columns=200

set the line length to 200. This makes the tables display properly.

--filter graphviz.py

Pass Pandoc’s abstract syntax tree through the filter program graphviz.py before rendering the output document.

-o example.html

Specify the name and type of the output file. The format is inferred from the suffix (file extension).

The setup consists of three parts:

1. A Docker image with texlive-full and lhs2tex preinstalled. It gets built and deployed automatically to DockerHub using a GitHub action.
2. A GitHub action inside my project that pulls the Docker image (1) and uses it to build the LaTeX documents (via make), exporting all the generated PDFs as an artifact.
3. A couple of config files to instruct VSCode to open my project inside the same Docker image (1) and rebuild the documents after saving a modified source file.

Let’s take a deeper look at each part now.

Build environment

The LaTeX pipeline I use is relatively simple. First, we preprocesses the root .lhs.tex files of my project using lhs2tex. This produces some gigantic .tex files that we then compile with the usual pdflatex+bibtex+pdflatex+pdflatex nonsense. These two steps are defined using a simple Makefile. I wrote this Dockerfile to automate the process of installing the dependencies needed to run the pipeline.

For simplicity, I use a GitHub action to build and deploy this image to DockerHub remotely on every new push. This is quite straightforward to set up using some existing actions:

.github/workflows/docker.yml:

name: Docker Image CI

env:
  IMAGE_NAME: latex-lhs2tex

on:
  push:
    branches: [ main ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Login to Docker Hub
      uses: docker/login-action@v1
      with:
        username: ${{ secrets.DOCKERHUB_USER }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1
      with:
        driver: docker-container
        driver-opts: |
          image=moby/buildkit:master
          network=host

    - name: Build and push
      uses: docker/build-push-action@v2
      with:
        context: ./
        file: ./Dockerfile
        builder: ${{ steps.buildx.outputs.name }}
        push: true
        tags: ${{ secrets.DOCKERHUB_USER }}/${{ env.IMAGE_NAME }}:latest
        cache-from: type=registry,ref=${{ secrets.DOCKERHUB_USER }}/${{ env.IMAGE_NAME }}:buildcache
        cache-to: type=registry,ref=${{ secrets.DOCKERHUB_USER }}/${{ env.IMAGE_NAME }}:buildcache,mode=max

These definitions live in a separate repository so I can reuse it for different LaTeX projects. The secrets.DOCKERHUB_USER and secrets.DOCKERHUB_TOKEN are the credentials used to login into DockerHub and need to be set using the web UI. Finally, in the build step it’s important to setup the cache-from and cache-to so we can additionally cache the intermediate image layers in DockerHub, and our future commits don’t take an eternity to rebuild.

GitHub CI action

With the build environment already deployed to DockerHub, the next step is to create a CI action in the project’s repo that pulls the image and runs make inside the build environment:

.github/workflows/latex.yml:

name: Build LaTeX documents

env:
  BUILD_ENV_IMAGE: agustinmista/latex-lhs2tex

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Set up Git repository
        uses: actions/checkout@v3
        with:
          path: repo

      - name: Fix file permissions in repo
        run: |
          chmod -R 777 repo
          sudo chown -R 1000:1000 repo

      - name: Pull custom Docker image
        run: docker pull ${{ env.BUILD_ENV_IMAGE }}

      - name: Compile LaTeX documents using custom Docker image
        run: |
          docker run \
            -v ${{ github.workspace }}/repo:/home/docker/workdir \
            ${{ env.BUILD_ENV_IMAGE }} \
            -c "make all"

      - name: Upload PDF file
        uses: actions/upload-artifact@v3
        with:
          name: artifact
          path: repo/*.pdf

A little nuance here is that I’m building the build enviroment image using the usual uid:gid=1000:1000 so build containers can modify and create files in the host files without screwing up their permissions. To my surprise, GitHub does not follow this, and the action runner user has instead uid:gid=1001:121. I’ve seen that some projects solve this by building Docker images to be used by GitHub runners using those specific uid:gid values, but this approach would probably not work if we want to also use the image later to build the project locally. The easiest way to solve it is to chown+chmod the cloned project folder repo so they belong to the build container user, but everyone can access them (this is needed by the last step to find and export the generated PDFs as an artifact).

VSCode setup

Now it’s time to bring the same build environment we used in CI to my local machine, so I don’t have to worry anymore about reproductibility. The easiest way to do it in VSCode is by using a development container. We do this by creating the following file in our project’s repository:

.devcontainer/devcontainer.json:

{
  "image": "agustinmista/latex-lhs2tex",
  "customizations": {
    "vscode": {
      "extensions": [
        "mathematic.vscode-pdf",
        "Gruntfuggly.triggertaskonsave"
      ]
    }
  }
}

The important bit here is that we’re telling VSCode that this project is meant to be developed inside the agustinmista/latex-lhs2tex Docker image.

Additionally, we can customize VSCode with specific extensions to be enabled for this particular project. Here I’m including the “Trigger Task on Save” extension that let us run VSCode tasks when we save specific files. These tasks are specified under .vscode/tasks.json. In our case, we can define a make task that simply runs make all:

.vscode/tasks.json:

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "make",
      "type": "shell",
      "command": "make all",
      "problemMatcher": [],
      "group": {
        "kind": "build",
        "isDefault": true
      }
    }
  ]
}

To make this work, we now need to configure the extension to run the make task when we save some modified source file. This can be done by adding the following to the repo’s local .vscode/settings.json:

.vscode/settings.json:

{
  "triggerTaskOnSave.tasks": {
    "make": [
      "*.lhs.tex",
      "notation.fmt",
      "references.bib",
      "path/to/other/source/files/*.tex"
    ],
  },
  "triggerTaskOnSave.showNotifications": true,
  "triggerTaskOnSave.on": true,
  "triggerTaskOnSave.restart": true
}

And that’s it. Next time we open the project, VSCode should ask us if we want to reopen it inside our build environment. Click “Yes” and carry on looking for those missing $ inserted.