Difference between revisions of "Running Mark IV"

From ConTeXt wiki
(more text (still not done))
 
(29 intermediate revisions by 11 users not shown)
Line 1: Line 1:
 
== Introduction ==
 
== Introduction ==
  
Context [[Mark IV]] does not use the kpathsea library to find files and configuration settings. It follows that running [[mktexlsr]] or [[fmtutil]] has no effect. This page explains the details of  
+
ConTeXt [[Mark IV]] does not use the kpathsea library to find files and configuration settings. It follows that running mktexlsr or fmtutil has no effect. This page explains the details of  
installing and maintaining [[Mark IV]].
+
setting up and maintaining [[Mark IV]].
  
 +
== History ==
  
== Quickstart ==
+
The most important original motivation for this page was that the [[TeX Live]] 2008 edition contained Mark IV, but it was only copied to the installed system, no post-install setup took place at all. For the Tex Live 2008 release, the post-install setup had to be done manually. TeX Live 2009 did not come with a usable Mark IV at all because the luatex on the 2009 editions was too old compared to the ConTeXt version. Instructions on how to deal with TeX Live 2009 were needed then.  TeX Live 2010 has a working Context Mark IV out of the box: you can just run 'context' directly.  At most, you could run `luatools --generate; context --make` to deal with any tlmgr updates you may have performed since installing TeX Live.
  
  luatools --generate
+
In TeX Live 2011, not even that is needed and you only have to run `mtxrun --generate` when files are installed manually ''outside'' of tlmgr. If you are running tl 2011, this page is just background information, there should be no need to take actual actions.
  
is the replacement for [[mktexlsr]]
+
=== Special notes (TL 2008/2009): ===
  
  context --make
+
* '''Windows''': Running Mark IV under Windows is the same as on Unix, but see [[Running_Mark_IV#In_case_of_trouble|below]] for a installation hints for the 'spaces in names' problem that seem to affect Windows users more often than other platforms.
  
is the replacement for [[fmtutil]]
+
* '''Debian''': Mark IV suggests the tex-gyre package ([[TeX Gyre]]), not distributed currently under Debian due to licensing issues (see [[Debian installation]] )
  
 +
=== Special notes (TL 2010): ===
  
== Filename and Configuration databases ==
+
* '''[[MacTeX]]''': You need to [[Running_Mark_IV#Updating_ConTeXt|update]] ConTeXt first.
  
=== Introduction ===
+
== Quickstart ==
  
When you run
+
If you have trouble following these instructions, please direct you questions or complaints on the [http://www.ntg.nl/mailman/listinfo/ntg-context ntg-context mailing list], '''not''' the tex-live list.
  
  context myfile
+
Just for the extremely impatient unix user that knows how to deal with a web2c-based TeX distribution and has a simple, private (single-user) installation, this is the only absolutely required extra knowledge:
  
both the script 'context' and the Mark IV code inside the format have to find various files. To make this happen as fast as possible, a file name database is used. There is also some configuration information and some generated font data is cached because extracting the OpenType information is a relatively lengthy and memory-consuming process.
+
  luatools --generate        # 2010 and earlier
 +
or
 +
  mtxrun --generate          # 2011
  
All of this data stored inside the 'texmf cache'. To keep this cache up to date, you have to (re)run
+
is the Mark IV replacement for mktexlsr.
  
   luatools --generate
+
   context --make
  
whenever you install or remove files from your tex installation that have to be found by Mark IV, like after updating context itself, or installing extra font packages from [[TeX Live]] or [[CTAN]]. System-wide fonts don't count, these are always re-discovered at runtime. {{todo| check that}}
+
is the Mark IV replacement for fmtutil.
  
=== Cache location ===
+
  context myfile.tex
  
When
+
typesets a file.
  
  luatools --generate
+
Everybody else should read on.
  
is run, it creates the file name database. To prevent duplication of settings as much as possible, it parses [[texmf.cnf]] as the source for discovering at what directory trees need to be indexed in the database. This ensures that while Mark IV does not actually use kpathsea, it is still compatible with how kpathsea handles things.
+
'''A note for windows users: (tl2008/2009)''' the luatex executable does not accept filenames with spaces in their name. This means that if you are on windows you will at least have to set up a texmfcnf.lua as explained below, and you likely will even have to reinstall texlive, because the default texlive installation directory itself contains spaces. [[Running_Mark_IV#File_not_found | Details]].
  
But there is a catch: luatools has to decide where on the harddisk the file name database should be stored. This needs extra care, because a runtime command like 'context myfile' does not look at texmf.cnf.
+
== Generating and updating formats ==
  
The internal logic used for finding the cache location is as follows:
+
Before first use and after each update to either the [[context]] distribution or the [[luatex]] executable you will have to regenerate the formats. The manual command for that is:
  
* If there is an environment variable named TEXMFCACHE, this will point to the desired harddisk location
+
  context --make cont-en
* Otherwise, if the web2c configuration directory contains a file named texmfcnf.lua, this file is read and used to discover the desired location. '''this is the recommended approach, see below'''
 
* If that doesn't work either, a list of 'likely' environment variables is tested, and the first variable that is set is assumed to point to the correct location.
 
* As a last resort, the local directory is used.
 
  
 +
This will generate a new Mark IV format with the English interface.  Without an explicit format argument, it will generate four formats: Mark IV cont-en, Matk IV cont-nl, luatex+mptopdf, and luatex+plain.
  
For completeness: the list of tested environment variables is currently:
+
Starting with 2011, formats are automatically (re)generated by `context myfile.tex` when it believes it is necessary to do so. Specifically: assuming the format already exists, automatic regeneration takes places whenever the ConTeXt version of the luatex version do not match the ones in the pre-existing format. Manual patches of the core files still require manual regeneration of the formats.
  
  TMPDIR, TEMPDIR, TMP, TEMP, HOME, HOMEPATH
+
The generated formats are stored inside the texmf cache.
  
but there have been quite some variations on this in the past.
+
== Experimental speedup ==
  
=== texmfcnf.lua ===
+
In addition to standard [[luatex]] we provide [[luajittex]] which uses just in time compilation to speed up processing. The speedup is mostly due to the faster virtual machine. The exact speedup depends on the tex/lua ratio and we measured speedups between 10% and 40%. You force this engine with 
  
Files named texmfcnf.lua can appear anywhere on the search path where there can also be a [[texmf.cnf]] present. Multiple files can be present. They are read sequentially using the same logic as for [[texmf.ncf]]: earlier files fix values that cannot be overruled by later ones. All texmfcnf.lua files are read before all texmf.cnf files, and values defined in texmfcnf.lua cannot be overruled by texmf.cnf files.
+
  context --jit ....
  
The format of texmfcnf.lua is pretty simple: it is a Lua script that has to return a single table with keys for the variables it wants to define. Mark IV will interpret specific keys in the table and do something special (unknown keys are ignored).
+
Alternatively you can put
  
Besides the 'normal' texmf.cnf keys, the most important key is TEXMFCACHE. Here is a simple one-line example texmfcnf.lua to set up the texmf cache to be inside your private texlive 2008 directory:
+
<texcode>
 +
% engine=luajittex
 +
</texcode>
  
  return { TEXMFCACHE =  '$HOME/.texlive2008/texmf-var' }
+
on top of your main document source.  
  
Note: Windows users should replace $HOME with $USERPROFILE. Both unix and windows users of [[TeX Live]] 2008 and users of Mark IV beta's released after Jul 10, 2008 can use ~ instead.
+
== Using fonts ==
 
 
== Generating and updating formats ==
 
 
 
Before first use and after each update to either the [[context]] distribution or the [[luatex]] executable you will have to regenerate the formats. The command for that is:
 
 
 
  context --make cont-en
 
  
This will generate a new Mark IV format with the English interface.  Without an explicit format argument, it will generate four formats: Mark IV cont-en, Matk IV cont-nl, luatex+mptopdf, and luatex+plain.
+
The <code>OSFONTDIR</code> environment variable should contain the paths where the system fonts must be looked for.
  
The generated formats are stored inside the texmf cache.
+
See [[Fonts in LuaTex#Getting access to the system fonts]]
  
== Using fonts ==
+
== Downloaded files ==
  
Context Mark IV can use non-tex fonts (in OpenType, TrueType or Type1 format) directly. The first time that you run
+
If you have the program 'curl' installed, then your input file can contain e.g.
 
 
  context myfile.tex
 
  
on a file that uses such non-traditional fonts (which is probably always), the actual fonts themselves have to be loaded in luatex's embedded section of fontforge to extract their metric information. This removes the need for tfm and vf files on your disk, but it is also a somewhat lengthy and often memory-hungry process.  
+
<texcode>
 +
\externalfigure
 +
  [http://www.pragma-ade.com/show-gra.pdf]
 +
</texcode>
  
That is why the results of that internal preprocessing are also saved inside the texmf cache. Future runs using the same font (not just this file, but all files using that particular font) can then use the cached version of the metrics instead.
+
and the referenced file will be downloaded automatically and placed in the 'texmf cache' if is not found in the cache already.
  
=== Font name database ===
+
== In case of trouble ==
  
Because Luatex does not know about the names of fonts all by itself,  Mark IV uses a database of 'known font names'
+
=== File not found (TL 2008/2009) ===
to allow you to specify fonts by their internal name instead of the name of the disk file. The mtxrun script named
 
'fonts' maintains this database. At any time, you can run
 
  
   mtxrun --script fonts --list
+
* You may get errors like
 +
   I can't find file `C:/Program.tex'.
 +
This is likely caused by the luatex executable not handling filename quoting properly yet. If filenames with spaces work at all, you are just lucky.
  
to get the list of currently known fonts.  
+
This problem is most likely to happen on Windows. There two steps to deal with it:
 +
* You must put the main texmf tree in path without spaces. If you're installing Mark IV via TeXLive 2008, then simply change the top installation path at install time.
 +
* If you're running XP or Server 2003 (and not Vista or Server 2008), you must also set the TeX user home and the Lua cache to paths without spaces. <code>C:/tlhome</code> is a good candidate.
 +
** In <code>texmf.cnf</code> set TEXMFHOME to <code>C:/tlhome/$USERNAME</code> or to something similar of your liking. Using $USERNAME in the path will still give a per-user directory, just not under "Documents and Settings". This assumes that the user name doesn't have spaces of course...
 +
** As mentioned above, create/edit <code>texmfcnf.lua</code> and put a line like <code>return { TEXMFCACHE =  'C:/tlhome/$USERNAME/.texlive2008/texmf-var' }</code>, which sets each user's cache inside his space-free TeX home path.
  
  mtxrun --script fonts --help
+
== Cache internals ==
  
will show some other options.
+
(this paragraph is for developers and power-users only)
  
Previously,
+
If you look at the folder that TEXMFCACHE points to, you will see a folder named luatex-cache. Inside that, there is the folder context. Inside the context folder, there usually is a single subfolder with a hexadecimal name. This is the md5sum of the name of the 'tex root' location (to be precise, it is the md5sum of the full path expansion of the context lua equivalent of the TEXMFCNF variable).
  
  mtxrun --script fonts --reload
+
Below this, there are currently four folders:
  
was always needed if you wanted to use new fonts by their font name (as opposed to their file names).  
+
* '''curl''' contains downloaded files. The url is converted to something that can be represented  on disk by replacing problematic characters by dashes
 +
* '''fonts''' contains the file data/names.tma, this is the 'font name database' (and the derived names.tmc, tmc files are byte-compiled lua code for faster loading). There are also subdirectories containing the preprocessed metrics for non-tex fonts, one folder for each font type.
 +
* '''formats''' contains the Mark IV format file
 +
* '''trees''' has a small set of files with hexadecimal names, one for each of the top-level constituents of TEXMF (containing the filename database for that tree), and one for each of the found texmf.cnf files (containing in preprocessed form all the variables defined by that file). The filenames are md5sums of the contained directory or filename.
  
In the latest beta (juli 1008) versions of Mark IV this is no longer the case. In these new versions, the reload command is run automatically whenever a named font cannot be found.
 
  
Nowadays, you only have to do the reload manually when you have installed updates to already installed fonts that cause the font and/or file names to change. This normally only happens for fonts that are still in development like [[TeX Gyre]] and [[Latin Modern]].
+
[[Category:Installation]]

Latest revision as of 22:36, 13 January 2013

Introduction

ConTeXt Mark IV does not use the kpathsea library to find files and configuration settings. It follows that running mktexlsr or fmtutil has no effect. This page explains the details of setting up and maintaining Mark IV.

History

The most important original motivation for this page was that the TeX Live 2008 edition contained Mark IV, but it was only copied to the installed system, no post-install setup took place at all. For the Tex Live 2008 release, the post-install setup had to be done manually. TeX Live 2009 did not come with a usable Mark IV at all because the luatex on the 2009 editions was too old compared to the ConTeXt version. Instructions on how to deal with TeX Live 2009 were needed then. TeX Live 2010 has a working Context Mark IV out of the box: you can just run 'context' directly. At most, you could run luatools --generate; context --make to deal with any tlmgr updates you may have performed since installing TeX Live.

In TeX Live 2011, not even that is needed and you only have to run mtxrun --generate when files are installed manually outside of tlmgr. If you are running tl 2011, this page is just background information, there should be no need to take actual actions.

Special notes (TL 2008/2009):

  • Windows: Running Mark IV under Windows is the same as on Unix, but see below for a installation hints for the 'spaces in names' problem that seem to affect Windows users more often than other platforms.
  • Debian: Mark IV suggests the tex-gyre package (TeX Gyre), not distributed currently under Debian due to licensing issues (see Debian installation )

Special notes (TL 2010):

Quickstart

If you have trouble following these instructions, please direct you questions or complaints on the ntg-context mailing list, not the tex-live list.

Just for the extremely impatient unix user that knows how to deal with a web2c-based TeX distribution and has a simple, private (single-user) installation, this is the only absolutely required extra knowledge:

 luatools --generate        # 2010 and earlier

or

 mtxrun --generate          # 2011

is the Mark IV replacement for mktexlsr.

 context --make

is the Mark IV replacement for fmtutil.

 context myfile.tex

typesets a file.

Everybody else should read on.

A note for windows users: (tl2008/2009) the luatex executable does not accept filenames with spaces in their name. This means that if you are on windows you will at least have to set up a texmfcnf.lua as explained below, and you likely will even have to reinstall texlive, because the default texlive installation directory itself contains spaces. Details.

Generating and updating formats

Before first use and after each update to either the context distribution or the luatex executable you will have to regenerate the formats. The manual command for that is:

 context --make cont-en

This will generate a new Mark IV format with the English interface. Without an explicit format argument, it will generate four formats: Mark IV cont-en, Matk IV cont-nl, luatex+mptopdf, and luatex+plain.

Starting with 2011, formats are automatically (re)generated by context myfile.tex when it believes it is necessary to do so. Specifically: assuming the format already exists, automatic regeneration takes places whenever the ConTeXt version of the luatex version do not match the ones in the pre-existing format. Manual patches of the core files still require manual regeneration of the formats.

The generated formats are stored inside the texmf cache.

Experimental speedup

In addition to standard luatex we provide luajittex which uses just in time compilation to speed up processing. The speedup is mostly due to the faster virtual machine. The exact speedup depends on the tex/lua ratio and we measured speedups between 10% and 40%. You force this engine with

 context --jit ....

Alternatively you can put

% engine=luajittex

on top of your main document source.

Using fonts

The OSFONTDIR environment variable should contain the paths where the system fonts must be looked for.

See Fonts in LuaTex#Getting access to the system fonts

Downloaded files

If you have the program 'curl' installed, then your input file can contain e.g.

\externalfigure
  [http://www.pragma-ade.com/show-gra.pdf]

and the referenced file will be downloaded automatically and placed in the 'texmf cache' if is not found in the cache already.

In case of trouble

File not found (TL 2008/2009)

  • You may get errors like
 I can't find file `C:/Program.tex'.

This is likely caused by the luatex executable not handling filename quoting properly yet. If filenames with spaces work at all, you are just lucky.

This problem is most likely to happen on Windows. There two steps to deal with it:

  • You must put the main texmf tree in path without spaces. If you're installing Mark IV via TeXLive 2008, then simply change the top installation path at install time.
  • If you're running XP or Server 2003 (and not Vista or Server 2008), you must also set the TeX user home and the Lua cache to paths without spaces. C:/tlhome is a good candidate.
    • In texmf.cnf set TEXMFHOME to C:/tlhome/$USERNAME or to something similar of your liking. Using $USERNAME in the path will still give a per-user directory, just not under "Documents and Settings". This assumes that the user name doesn't have spaces of course...
    • As mentioned above, create/edit texmfcnf.lua and put a line like return { TEXMFCACHE = 'C:/tlhome/$USERNAME/.texlive2008/texmf-var' }, which sets each user's cache inside his space-free TeX home path.

Cache internals

(this paragraph is for developers and power-users only)

If you look at the folder that TEXMFCACHE points to, you will see a folder named luatex-cache. Inside that, there is the folder context. Inside the context folder, there usually is a single subfolder with a hexadecimal name. This is the md5sum of the name of the 'tex root' location (to be precise, it is the md5sum of the full path expansion of the context lua equivalent of the TEXMFCNF variable).

Below this, there are currently four folders:

  • curl contains downloaded files. The url is converted to something that can be represented on disk by replacing problematic characters by dashes
  • fonts contains the file data/names.tma, this is the 'font name database' (and the derived names.tmc, tmc files are byte-compiled lua code for faster loading). There are also subdirectories containing the preprocessed metrics for non-tex fonts, one folder for each font type.
  • formats contains the Mark IV format file
  • trees has a small set of files with hexadecimal names, one for each of the top-level constituents of TEXMF (containing the filename database for that tree), and one for each of the found texmf.cnf files (containing in preprocessed form all the variables defined by that file). The filenames are md5sums of the contained directory or filename.