Running Mark IV

From ConTeXt wiki
Revision as of 08:58, 10 August 2008 by Vga (talk | contribs) (File not found: Windows instructions for avoiding spaces)

Introduction

Context Mark IV does not use the kpathsea library to find files and configuration settings. It follows that running mktexlsr or fmtutil has no effect. This page explains the details of setting up and maintaining Mark IV.

The most important motivation for this page is that the TeX Live 2008 edition contains Mark IV, but it is only copied to the installed system, no post-install setup takes place at all. This year, the post-install setup has to be done manually. If you have trouble following these instructions, please direct you questions or complaints on the [|context mailing list], not the tex-live list.

Quickstart

Just for the extremely impatient unix user that knows how to deal with a web2c-based TeX distribution and has a simple, private (single-user) installation, this is the only absolutely required extra knowledge:

 luatools --generate

is the Mark IV replacement for mktexlsr.

 context --make

is the Mark IV replacement for fmtutil.

 context myfile.tex

typesets a file.

Everybody else should read on.

A note for windows users: the luatex executable doest not accept filenames with spaces in their name. This means that if you are on windows you will at least have to set up a texmfcnf.lua as explained below, and you likely will even have to reinstall texlive, because the default texlive installation directory itself contains spaces.

Filename and Configuration databases

Introduction

When you run

 context myfile.tex

both the script 'context' and the Mark IV code inside the actual format file have to find various files. To make this happen as fast as possible, a file name database is used. There is also some configuration information needed and some generated font data to be cached.

All of this data stored inside the 'texmf cache'. To keep this cache up-to-date, you have to (re)run

 luatools --generate

whenever you install or remove files from your tex installation that have to be found by Mark IV, or when you make changes to a the configuration values. Examples of moments when you need to rerun luatools are

  • after updating the context package
  • after adjusting settings in any of the texmf.cnf files
  • after installing font packages from TeX Live or CTAN
  • after adding or removing third-party context modules.

Mark IV can also use the operating system's installed fonts when it is set up properly, but for fonts installed or removed using the operating system's installation method you do need to rerun luatools, such fonts are handled by their own subsystem (which will be explained below).

Cache location

When

 luatools --generate

is run for the first time, it creates the file name database, later runs do updates.

To prevent unnecessary duplication of settings, luatools parses texmf.cnf as the source for discovering configuration values and which directory trees need to be indexed. This also ensures that even though Mark IV does not use kpathsea, it is still runtime-compatible with how kpathsea handles things.

But there is a catch: luatools has to decide where on the harddisk the file name database should be stored (or updated). This needs extra care, because a runtime command like 'context myfile.tex' does not look at texmf.cnf at all. Therefore, the cache location is handled separately from all other configuration settings.

The internal logic used for finding the cache location is as follows:

  • If there is an environment variable named TEXMFCACHE, this will point to the desired harddisk location
  • Otherwise, if reading texmfcnf.lua results in the variable TEXMFCACHE being set, this is the desired location. this is the recommended approach, see below
  • If that doesn't work either, a list of likely environment variables is tested, and the first environment variable that is set is assumed to point to the correct location.
  • As a last resort, the local directory is used.

For completeness: the list of tested environment variables is currently: TMPDIR, TEMPDIR, TMP, TEMP, HOME, HOMEPATH, but there have been quite some variations on this in the past.

texmfcnf.lua

Files with the name texmfcnf.lua can appear anywhere on the search path where there can also be a texmf.cnf file, and multiple texmfcnf.lua files can be present.

All possible locations are tried in order, and any found texmfcnf.lua files are read in order of discovery. All texmfcnf.lua files are read before all texmf.cnf files, and none of the values defined in texmfcnf.lua file(s) cannot be overruled by texmf.cnf files.

Note: context on texlive can not find texmfcnf.lua unless it is placed in <installationdir>texmf/web2c/.

The format of texmfcnf.lua is pretty simple: it is a Lua script that has to return a single table with keys for the variables it wants to define. If there are multiple files with duplicate keys, the first appearance of a particular key 'wins' (this is the same behaviour as for texmf.cnf variables).

texmfcnf.lua could be used to set up the 'normal' texmf.cnf variables, but by far the most important key is TEXMFCACHE. Here is a simple one-line example texmfcnf.lua to set up the texmf cache to be inside your private texlive 2008 directory:

 return { TEXMFCACHE =  '$HOME/.texlive2008/texmf-var' }

Note: Windows users should replace $HOME with $USERPROFILE. TeX Live 2008 users and users of Mark IV versions from after Jul 10, 2008 can use a tilde (~) instead of an explicit environment variable.


Generating and updating formats

Before first use and after each update to either the context distribution or the luatex executable you will have to regenerate the formats. The command for that is:

 context --make cont-en

This will generate a new Mark IV format with the English interface. Without an explicit format argument, it will generate four formats: Mark IV cont-en, Matk IV cont-nl, luatex+mptopdf, and luatex+plain.

The generated formats are stored inside the texmf cache.

Using fonts

Context Mark IV can use non-tex fonts (in OpenType, TrueType or Type1 format) directly. The first time that you run

 context myfile.tex

on a file that uses such non-traditional fonts (which is probably always), the actual fonts themselves have to be loaded in luatex's embedded section of fontforge to extract their metric information. This removes the need for tfm and vf files on your disk, but it is also a somewhat lengthy and often memory-hungry process.

That is why the results of that internal preprocessing are also saved inside the texmf cache. Future runs using the same font (not just this file, but all files using that particular font) can then use the cached version of the metrics instead.

Font name database

Because Luatex does not know about the names of fonts all by itself, Mark IV uses a database of 'known font names' to allow you to specify fonts by their internal name instead of the name of the disk file. The mtxrun script named 'fonts' maintains this database. At any time, you can run

 mtxrun --script fonts --list

to get the list of currently known fonts.

 mtxrun --script fonts --help

will show some other options.

Previously,

 mtxrun --script fonts --reload

was always needed if you wanted to use new fonts by their font name (as opposed to their file names).

In the latest beta (July 2008) versions of Mark IV this is no longer the case. In these new versions, the reload command is run automatically whenever a named font cannot be found.

Nowadays, you only have to do the reload manually when you have installed updates to already installed fonts that cause the font and/or file names to change. This normally only happens for fonts that are still in development like TeX Gyre and Latin Modern.

Downloaded files

If you have the program 'curl' installed, then your input file can contain e.g.

\externalfigure
  [http://www.pragma-ade.com/show-gra.pdf]

and the referenced file will be downloaded automatically and placed in the 'texmf cache' if is not found in the cache already.

In case of trouble

File not found

  • You may get errors like
 I can't find file `C:/Program.tex'.

This is likely caused by the luatex executable not handling filename quoting properly yet. If filenames with spaces work at all, you are just lucky.

This problem is most likely to happen on Windows. There two steps to deal with it:

  • You must put the main texmf tree in path without spaces. If you're installing Mark IV via TeXLive 2008, then simply change the top installation path at install time.
  • If you're running XP or Server 2003 (and not Vista or Server 2008), you must also set the TeX user home and the Lua cache to paths without spaces.
    • In context.cnf set HOMETEXMF to C:/tlhome/$USERNAME or to something similar of your liking. Using $USERNAME in the path will still give a per-user directory, just not under "Documents and Settings". This assumes that the user name doesn't have spaces of course...
    • As mentioned above, create/edit texmfcnf.lua and put a line like return { TEXMFCACHE = 'C:/tlhome/$USERNAME/.texlive2008/texmf-var' }, which sets each user's cache inside his space-free TeX home path.

Cache internals

(this paragraph is for developers and power-users only)

If you look at the folder that TEXMFCACHE points to, you will see a folder named luatex-cache. Inside that, there is the folder context. Inside the context folder, there usually is a single subfolder with a hexadecimal name. This is the md5sum of the name of the 'tex root' folder (aka SELFAUTOPARENT). Below this, there are currently four folders:

  • curl contains downloaded files. The url is converted to something that can be represented on disk by replacing problematic characters by dashes
  • fonts contains the file data/names.tma, this is the 'font name database' (and the derived names.tmc, tmc files are byte-compiled lua code for faster loading). There are also subdirectories containing the preprocessed metrics for non-tex fonts, one folder for each font type.
  • formats contains the Mark IV format file
  • trees has a small set of files with hexadecimal names, one for each of the top-level constituents of TEXMF (containing the filename database for that tree), and one for each of the found texmf.cnf files (containing in preprocessed form all the variables defined by that file). The filenames are md5sums of the contained directory or filename.