Evaluation and Caching¤
Note
Understanding and controlling when blocks are evaluated is important for the author of docsets, (not the reader.
Best Practice¤
These are the recommended base settings to control evaluation:
- Before mkdocs build:
export lp_eval=always - Before mkdocs serve:
export lp_eval=on_page_change
They are also set by the make:on_page_change file.
Within the page header you might want to set eval page wide to on_page_change, in order to
prevent CI/CD to re-evaluate the page and commit the .lp.py file for those pages.
Here the Details:
Hashing¤
When the plugin identifies and parses lp blocks within your docset, it builds a hash to identify them.
The hash is built over the complete lp body plus a specific set of header parameters, which might, when changed, result in a different outcome of the evaluation:
# those header params will prevent to use caching when changed, they go into the hash of
# a block which is the cache key:
hashed_headers = [
'asserts',
'body',
'chmod',
'cwd',
'delim',
'dir',
'expect',
'mode',
'new_session',
'pdb',
'post',
'pre',
'session',
'timeout',
]
Warning
It should be clear that the evaluation result might change, even without any change in those headers, due to side-effects outside of our control. It remains upone the author of LP stanzas to decide upon re-evaluation.
Cache Files¤
After a page was evaluated, a file is written, next to the .md source file, ending with .lp.py.
The file contains a hash map, with keys the hashes of each block and value the raw (unformatted) result of the evaluation.
The eval parameter determines now, if, at page build time, a new evaluation is performed or the
result from the cache is taken, if available.
Patching the mkdocs file watcher¤
At first start of mkdocs .lp.py files. That prevents putting them into the site directory but also helps avoid
evaluation loops.
implementation
def patch_mkdocs_to_ignore_res_file_changes():
"""sad. we must prevent mkdocs serve to rebuild each time we write a result file
And we want those result files close to the docs, they should be in that tree.
Also we save tons of rebuilds when preventing to monitor imgs - since often
autocreated, e.g. from kroki or drawio.
"""
import mkdocs
fn = mkdocs.__file__.rsplit('/', 1)[0]
fn += '/livereload/__init__.py'
if not exists(fn):
return app.warning('Cannot patch mkdocs - version mismatch', missing=fn)
s = read_file(fn)
S = 'event.is_directory'
if not S in s:
return app.warning('Cannot patch mkdocs - version mismatch', missing=fn)
if lp_res_ext in s:
return app.info('mkdocs is already patched to ignore %s' % lp_res_ext, fn=fn)
os.system('cp "%s" "%s.orig"' % (fn, fn))
new = S
for ext in [lp_res_ext, '.svg', '.png']:
new += ' or event.src_path.endswith("%s") ' % ext
new += ' or "/autodocs" in event.src_path '
# cannot write a comment due to the ':'
s = s.replace(S, new)
write_file(fn, s)
diff = os.popen('diff "%s.orig" "%s"' % (fn, fn)).read().splitlines()
app.info('Diff', json=diff)
msg = (
'Have patched mkdocs to not watch %s and .svg files. Please restart.' % lp_res_ext
)
app.die(msg, fn=fn)
CI/CD: Comitting Cache Files?¤
Sometimes lp blocks can or should only be evaluated on your local machine - but not on CI/CD elsewhere.
Example: You document how to set up a Kubernetes cluster in the cloud, using your cloud provider credentials, plus it takes 30 minutes until completed.
You want to be able to prevent CI/CD to evaluate the specific blocks - or whole pages -, when running, i.e. when CI/CD is building the docs.
Solution: Set eval to "on_change" or "on_page_change"
- You commit the
.lp.pycache files of these pages and - set
evalto "on_change" or "on_page_change". - The result (cache) files, for pages which should be evaluated on CI/CD (e.g. for additional testing purposes) you do NOT commit.
- Then the final docs will display the result from your local run, for the pages you committed the cache files for.
The eval Parameter¤
Adjusting eval is key for having fast edit-render cycles, while authoring pages.
Predefined Values¤
class Eval:
never = 'never' # not even when not cached
always = 'always' # even when cached. except skipped
on_change = 'on_change' # only when block changed
on_page_change = 'on_page_change' # whan any block (md irrelevant) on page changed
# Default: anything else would confuse user. e.g. cat <filename> would show old still when it changed but no lp change involved:
default = 'always'
- never: No evaluation. Typically given on page level, while writing a bunch of LP statements. Will never eval, even with a cache miss.
- on_change: Re-evaluation when the hash changes
- on_page_change: Re-evaluation of all blocks, when any hash changes on the page
- always: Always evaluate (the default)
Arbitrary Matching¤
When working on a block or a page, you can also restrict evaluation to the current page or even
block only, by specifying the eval parameter like so: <page match>[:<block match>].
Typically you do this via an environ parameter at start up of mkdocs build|serve:
Warning
Such a page match is checked against the full source path to markdown pages, from / (root) folder!
So lp_eval=docs/index would match exactly on your main index.md.
$ lp_eval="mypage" mkdocs build
$ lp_eval="mypage:myblockmatch" mkdocs build
When a block match was given, you can just add the match string within the header, e.g. as a boolean value.
Example:
```bash lp myblockmatch
< statements to eval>
```
When the block is correctly functioning, you can delete the match string and have the result in the cache file, moving on to the next block.
Any non matching block is set to eval="never", i.e. results will be from cache - or when not
present rendered as skipped.
Hint
To cause more than one block evaluated simply add the matching keyword argument in the header for all blocks you want.
Skips¤
The header exclusive argument <lang> lp skip_<this|other|below|above> will skip block execution
accordingly, i.e. you can work your way towards a completely evaluatable page, from block to block.
Note
Behind the scenes, the eval parameter does nothing else than adding skip_this parameters to
non evaluated blocks.