Generating auxiliary files with Literate

See also how to interact with literate scripts.

Literate.jl allows to pre and post-process a script in order, for instance, to generate a notebook out of a script. This can be convenient if you want to have a tutorial downloadable as a standalone notebook or as a scrubbed script.

This page presents one way of doing this which is used in DataScienceTutorials and which might inspire your own approach.

The key ingredients are:

In what follows, it is assumed you have your Literate scripts in a folder /scripts/ and that you're using GitHub. It shouldn't be hard to modify that to suit your own case.

Using Literate to generate auxiliary files

Literate can manipulate scripts fairly easily, for instance, to generate notebooks:

scripts = joinpath.("scripts", readdir("scripts"))
nbpath = joinpath("generated", "notebooks")
isdir(nbpath) || mkpath(nbpath)

for script in scripts
   # Generate annotated notebooks
   Literate.notebook(script, nbpath,
                     execute=false, documenter=false)
end

This will go over all scripts in /scripts/ and call Literate.notebook to generate a derived notebook in a /generated/ folder (which you will want to add to your .gitignore).

Push the generated files to a page branch

Start by installing gh-pages with npm:

$> npm install gh-pages

Using the package NodeJS.jl it is then easy to use gh-pages to push the generated notebooks to a folder on the gh-pages branch:

using NodeJS

JS_GHP = """
    var ghpages = require('gh-pages');
    ghpages.publish('generated/', function(err) {});
    """

run(`$(nodejs_cmd()) -e $JS_GHP`)

Now these generated files are available on that branch without polluting your master branch. You can see this live on the DataScienceTutorials repo.

You can link to these generated notebooks with links adapted from:

https://raw.githubusercontent.com/username/project/gh-pages/notebooks/theNotebook.ipynb

See this page for example.

Add the whole process to publish

You may want to re-generate all notebooks prior to pushing updates to GitHub. For this, use the final keyword of publish to which you can pass a function to use before publishing updates. For instance:

function gen_literate()
    scripts = joinpath.("scripts", readdir("scripts"))
    nbpath  = joinpath("generated", "notebooks")
    isdir(nbpath) || mkpath(nbpath)

    for script in scripts
       # Generate annotated notebooks
       Literate.notebook(script, nbpath,
                         execute=false, documenter=false)
    end
    JS_GHP = """
        var ghpages = require('gh-pages');
        ghpages.publish('generated/', function(err) {});
        """
    run(`$(nodejs_cmd()) -e $JS_GHP`)
end
# ... serve etc ...
# ... then finally ...
publish(final=gen_literate)