pm39 cmdref revamp proposal

"Towards A Revamped ConTeXt Reference Manual"

The Short Version

The ConTeXt command reference could be more complete, with full descriptions of options and settings, examples, and relations ships between commands.

One way to accomplish this is to develop the ConTeXt command reference as document set using the Darwin Information Typing Architecture (DITA) specification. This provides a well-structured way to produce and maintain the documentation for the command interface, and can be used as a platform to produce many output formats.

For an example of DITA sources:

https://github.com/pmazaitis/context_command_reference

For examples of output:

https://paul.mazaitis.org/ctx_comref/

The Problem

In learning ConTeXt, one thing I run into a lot is uncertainty about what the settings do; either I'm not sure what any of the possible values mean in the key=value pairs, or I'm not even sure what all of those are.

I'd like to have a resource that is a complete description of the ConTeXt command set: I'd like to have one, central document where there is a place for all of this information to go.

Goals for this Project

Personal Goals

By the end of this, I'd like to:

• Have a command reference with descriptions for what all of the options and settings do.
• Have the command reference be in a format that I can easily transform into a Documentation Set for Dash.app (https://kapeli.com/dash).

General Goals

The command reference should be adaptable for different requirements: like the current interface files, it should still be able to drive automatic content on the wiki, as well as be available as a unified PDF.

The ConTeXt interface has a lot of inheritance in it. One nice feature of this is that documentation editors only need to update one piece of information, and it automatically applies everywhere. Users, though, shouldn't have to chase down relevant documentation for a command through the manual: all of the options for a command should be available in one place.

The command reference should be complete: it should be able to support a description of every command, including syntax and setups.

Other nice features should be a MWE for each command, and how that command relates to other commands.

Non-Goals

This project is focused on the command reference, and doesn't make any claims about other documentation (although, hopefully, it will be better able to support them).

Questions for This Project

• Will this help?
• Will this work?
• Is there something I'm missing?

Darwin Information Typing Architecture

The Darwin Information Typing Architecture (DITA) is an OASIS-backed, XML-based standard for developing topic-based documentation,

https://www.xml.com/articles/2017/01/19/what-dita/

DITA supports topic-based authoring. For purposes of this manual, we can consider each command a topic, a self-contained and complete description of what the command does, it's syntax, what setups it accepts, etc.

With DITA, topics can then be compiled together using maps. Maps are arbitrary external expressions of hierarchy, and multiple maps can share the same topic. For example, there could be a map with the complete command reference, and another that leave out the internal, developer-facing commands. These maps could be used to produce two documents, but rely on the same command topic files as sources, and editing a topic file for a particular command will ensure that it's updated for both documents when they are rebuilt.

DITA also supports pulling content from one topic to another. For the command set, DITA can pull options and settings from one command topic into one of more other topics, and have that material included directly in the former topics when they are compiled. This preserves the inheritance and single sourcing of setups (excellent for the for the document maintainers) as well as making sure all relevant content is available everywhere it's needed (excellent for end users).

A Hand-Built Prototype

I've put together a hand-built prototype with a handful of commands to act as examples:

https://github.com/pmazaitis/context_command_reference_prototype

This is meant to be a small scale, easily digestible prototype to show how DITA might be used to develop a comprehensive documentation set.

Output Products

From the hand-built prototype, The DITA Open Toolkit (dita-ot) can generate lots of different formats out of the box, and use plugins for others. Here are some examples, all generated from the above DITA document set.

HTML5

dita-ot can generate an HTML version of the documentation set:

dita --input=context_command_reference.ditamap --format=html5

...as hosted here:

https://paul.mazaitis.org/ctx_comref/prototype-html5/

This is without any CSS, etc. - just the structure.

XSL-FO PDF

dita-ot can generate an XSL-FO version of the documentation set:

dita --input=context_command_reference.ditamap --format=pdf

...as hosted here:

https://paul.mazaitis.org/ctx_comref/prototype-pdf/context_command_reference.pdf

This is pretty straightforward FO. (I imagine ConTeXt could do a nicer job...)

Normalized XML

dita-ot can generate a normalized version of the documentation set:

dita --input=context_command_reference.ditamap --format=dita

...as available here:

https://paul.mazaitis.org/ctx_comref/prototype-normalized-topics/

This transformation does all of the processing work on individual topics (coping content, resolving lints, etc.) but keeps the content as XML files for further processing.

Web Help

I also use the Oxygen XML Editor, which comes with several other transformations. One transformation of the documentation set that might be interesting as an example of what is possible is their Web Help implementation, hosted here:

https://paul.mazaitis.org/ctx_comref/prototype-webhelp-responsive/

Other Available Transformations

There are many other transformations available, and it is possible to create new ones.

A Full Command Set

Dita Sources

A full DITA rendering of the ConTeXt command interface is tracked here:

https://github.com/pmazaitis/context_command_reference

These files are generated by a Python script from the command interface files distributed with ConTeXt. The Script is available here:

https://github.com/pmazaitis/context-interface2dita

...with a noted lack of documentation as of yet.

This documentation set is tested to build to PDF, HTML5 and Normalized DITA.

Topic Level Features

Short Description

One of the first elements in a topic is the shortdesc element:

https://github.com/pmazaitis/context_command_reference_prototype/blob/master/en/commands/b/r_command_blank.dita#L5-L7

This element holds a brief description of the topic.

This is a quirk of DITA, and can be difficult to write, but is an important requirement for cross-linking references.

Metadata

The metadata for the topic is contained in the prolog element:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/b/r_command_blank.dita#L8-L19

We have a few interesting features here.

The audience element expresses the intended audience for this topic. This can be used to indicate a level of expertise expected for the audience of this particular command:

• user: command is meant for the general ConTeXt user base
• programmer: command is meant for folks who create their own ConTeXt commands
• implementor: command is meant for internal use in ConTeXt development

There is some flexibility here with the labels.

The category element describes one or more categories that this command is part of. A command topic can have multiple category elements.

The keywords element can support one or more keyword elements, to tag commands that fall into more than one area.

The prodinfo element contains metadata about how this command topic related to the entire project. The prodname element generally links the command to ConTeXt, while the vrmlist allows tracking of version (whatever makes sense) and release (which might be useful for production/experimental commands, etc.).

The vrmlist can contain multiple vrm elements, and could be used to support commands across mkii, mkiv, etc.

These elements can be used for conditional processing. For example, we could instruct the DITA processor to only include topics that are user-focused, etc.

In addition to the formal metadata, the prolog also supports two other pieces of information: the source element, and the critdates element.

The source element may be useful here to describe a full (non-rooted?) path to the source file in the standard ConTeXt distribution.

The critdates element can contain created and revised elements to capture creating, modification, and expiry dates. This element can be used progamatically to identify documentation that is scheduled for an edit.

Syntax

The main body of the topic (the refbody element) starts with a discussion of syntax:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/v/r_command_vspacing.dita#L20-L39

This includes a description of the syntax phrase, as well as a table of all of the arguments, with a quick description of each.

Options

Example:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/v/r_command_vspacing.dita#L20-L39

If the command accepts options (single keywords, identified as 'keyword' in the interface files) as part of its setups, they are presented in a table in the body of the topic. The options table presents each option, and a brief description of what it does. The options section also can support notes, more tables, etc.

Commands can inherit option content from other topics, as in:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/b/r_command_blank.dita#L50-L53

When the final document is produced, this inheritance directive will copy the source content into the topic, so it's available to users in both places immediately.

Settings

Example:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/s/r_command_setupbuffer.dita#L47-L118

If the commands accepts settings (key-value pairs, identified as 'parameter' in the interface files) as part of its setups, they are presented as a grouped series of tables in the body of the topic; each key has a table of available values for the key and their descriptions.

Commands can inherit sets of settings content from other topics, as in:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/d/r_command_definebuffer.dita#L58-L71

Commands can also inherit an option set as the values of a setting key:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/s/r_command_setuptyping.dita#L660-L664

When the final document is produced, this inheritance directive will copy the source content into the topic, so it's available to users in both places immediately.

Notes

A notes section can be included:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/v/r_command_vspacing.dita#L251-L345

Individual notes can be included in various sections with the note element: use of a notes section can also be useful for general documentation that doesn't fit elsewhere in the topic.

This is not meant to grow to take the place of more comprehensive documentation: the notes in a given topic should be focused only on the command detailed in that topic.

Examples

A goal is to have one minimal working example for every command. For example:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/b/r_command_bold.dita#L43-L52

It's fairly straightforward to pull the MWE out of the topic; one piece of future work is adding automatic builds of MWEs back into the manual as image assets, if requested in the topic (using the 'otherprops' attribute). To be reasonable, this should be handled entirely automatically by the build system.

(This also presents the possibility of the manual being used as a test or coverage suite, but that's outside the scope of this project.)

A command can have more than one example, but should always have at least one.

Related Topics

The related topics that appear in the final documentation do not come from the topic sources, but are generated by the DITA processor based on relationship descriptions maintained elsewhere.

For this document set, the relationships are described with a reltable in the relations map:

https://github.com/pmazaitis/context_command_reference_prototype/blob/master/en/relations.ditamap

This allows for arbitrary relationships between commands, and the relationships can be ignored if the commands aren't included in a given product, surpassing confusing and/or broken links in the output.

Related Links

Each command topic can also have an explicit related links section:

https://github.com/pmazaitis/context_command_reference_prototype/blob/936e331726515175160190a6b383f7735105c8cb/en/commands/i/r_command_input.dita#L65-L69

This can also be used to pointing the user at some other external, relevant part of the ConTeXt documentation system (a manual, or a My Way, etc.).

System Level Features

Using a Map to Establish a Hierarchy

The topics themselves know nothing about the hierarchy into which they're placed for a product; this is handled with a ditamap file, which establishes inclusion, hierarchy, order, and grouping:

https://github.com/pmazaitis/context_command_reference_prototype/blob/master/en/context_command_reference.ditamap

Map files can link to other map files helping reduce the complexity of maintenance.

A map file should be maintained for each product; one map file for the comprehensive reference, another map file for an appendix to a particular manual, etc. In this manner, topics can be reused across multiple products, are there is no worry about a particular piece of command documentation getting out of sync.

Using Keys to Reference Other Content

A core part of how DITA transforms and transcodes content from topic to topic is through the use of keys; DITA uses these (arbitrary) labels to know where to find specific pieces of content. These keys are maintained in their own map:

https://github.com/pmazaitis/context_command_reference_prototype/blob/master/en/inheritance.ditamap

Using a Map to Establish Relationships Between Topics

For this document set, the relationships are described with a reltable in the relations map:

https://github.com/pmazaitis/context_command_reference_prototype/blob/master/en/relations.ditamap

This is a simple two column table that describes commands (column one) and the commands related to that command (column two).

Multiple Maps for Multiple Products

At the moment, the distribution supports three maps of command topics:

• The full command set
• User-facing commands
• System commands

...each drawing on the same pool of command topics to build the final product.

Maps could also be made to support command glossaries for manuals, etc.

Keeping Track of What Needs To Be Worked On

Several elements in the topics now have the rev attribute. This can be used to keep track of what elements in the command set are still boilerplate by incrementing the attribute value after edit.

Output Products

The Normalized DITA version of the current effort is here:

https://paul.mazaitis.org/ctx_comref/ctx_com_ref-ditanorm-20200622_002.zip

...presented as a zip file (to browse though a smaller example, the prototype is still available online).

This one is interesting to me, in that it might be useful as a basis for inclusion and formatting in both ConTeXt documents (through the XML interface) and/or the Wiki (if current methods could be adopted).

Other available formats:

HTML5 (No CSS, Zipped): https://paul.mazaitis.org/ctx_comref/ctx_com_ref-html5-20200622_002.zip

PDF (XSL-FO, Full Command Set): https://paul.mazaitis.org/ctx_comref/ctx_com_ref-pdf-20200622_002.pdf

There are also maps available to build a reference manual with only system commands, and a reference manual with only user commands. A PDF version of the latter:

https://paul.mazaitis.org/ctx_comref/ctx_user_ref-pdf-20200622_002.pdf

Much Yet to Do

The documentation still needs to be completed and edited, etc., but I think this could be a viable way to systematically document more of the interface's depth and detail.

Pros and Cons

Pros

DITA is an open specification, and comes with an open source tool chain.

One set of documentation set can produce multiple documents and formats.

Documentation needs only be edited in one place.

ConTeXt can handle XML inputs, so there are intriguing possibilities for nicer representations.

Cons

DITA is a complicated specification, and requires conformant XML to work.

DITA is strict (but that's how it works).

The DITA toolchain is complicated, and has a number of dependancies (Java, etc.).

Current Work: Transforming the Current Interface Files

Working out a way to manage variants in the documentation set.

Future Work

Filling In the Documentation

This documentation set offers a place to document syntax, options, and settings; there's a lot to document, even with re-use.

Editing the Manual

The manual should be correct! This, too, is a lot of editing work.

Maintaining the Documentation

Tricky!

One way this might work is to use the wiki: use a system similar to the current one to auto-generate wiki pages for each command, and then let users fill in notes or corrections on the rest of the page...

Automatically Processing Examples

It's fairly simple to pull out the examples from the topics; slightly more complicated is adapting the dita-ot build system (based on Ant) to automatically pull the examples, run them through ConTeXt, and then link them to the parent document as an inlined graphic.

Using Examples as a Test Suite

Another possibility is to pull examples from topics (either all, or a flagged subset) and use them as a test suite.

Future Work That's Really Hard

Normalizing Signatures

Part of what makes this difficult at the moment is that settings (key=value pairs) do not have a one-to-one mapping with signatures.

The setting foo may have these options for one command:

foo=(up|down)

...but may have a different set of valid values for another command:

foo=(up|down|left|right)

If all of the settings keys had one and only one signature, this would go far toward unifying and harmonizing the documentation (and would allow for really nifty content re-use in the documentation!).

The downside, of course, is that this breaks the command interface in decidedly non-trial ways.

Thanks!

Any and all comment welcome!