% language=us runpath=texruns:manuals/luametatex

\environment luametatex-style

\startdocument[title=Callbacks]

\startsubject[title=Contents]
    \setups[sectioncontents]
\stopsubject

\startsection[title={Introduction}]

Right from the start of the \LUATEX\ project callbacks were the way to extend the
engine. At various places in processing the document source and typesetting the
text the engine checks if there is a callback set and if so, calls out to \LUA.
Here we collect the various callbacks. For examples you can consult the \CONTEXT\
code base.

The \type {callback} library has functions that register, find and list
callbacks. Callbacks are \LUA\ functions that are called in well defined places.
There are two kinds of callbacks: those that mix with existing functionality, and
those that (when enabled) replace functionality. In most cases the second
category is expected to behave similar to the built in functionality because in a
next step specific data is expected. For instance, you can replace the
hyphenation routine. The function gets a list that can be hyphenated (or not).
The final list should be valid and is (normally) used for constructing a
paragraph. Another function can replace the ligature builder and|/|or kern
routine. Doing something else is possible but in the end might not give the user
the expected outcome.

In order for a callback to kick in you need register it. This can be permanent or
temporarily. One can use names as well as id's:

\starttyping[option=LUA]
function callback.register (
    <t:string>   name,
    <t:function> action | <t:false> | <t:nil>
)
    return <t:integer> -- id
end

function callback.register (
    <t:integer>  id,
    <t:function> action | <t:false> | <t:nil>
)
    return <t:integer> -- id
end
\stoptyping

Here the \type {name} is a predefined callback name as discussed in following
sections. The function returns the internal \type {id} of the callback when
registration succeeds or \type {nil} if the callback could not be registered.
Callback assignments are always global. You can use the \type {nil} or \type
{false} instead of a function for clearing the callback. \LUAMETATEX\
internalizes the callback function in such a way that it does not matter if you
redefine a function accidentally.

There are two ways to figure out what callbacks are available. The first one
returns a hash with the names as key and the boolean value indicating of the
callback is known.

\starttyping[option=LUA]
function callback.list ( )
    return <t:hash> -- active
end
\stoptyping

The list of available callbacks is:
\startluacode
    local t = callback.names()
    local n = #t
    for i=1,n do
        if i == n then
            context(" and ")
        elseif i > 1 then
            context(", ")
        end
        context.typ(t[i])
    end
\stopluacode .

\starttyping[option=LUA]
function callback.names ( )
    return <t:table> -- available
end
\stoptyping

This list is in order of id but that order might change which us why using names might
be better. A quick way to get the id is:

\starttyping[option=LUA]
function callback.getindex ( <t:string> name )
    return <t:integer> -- id
end
\stoptyping

If the callback is not set, \type {find} returns \type {nil}. The \type {known}
function can be used to check if a callback is supported. That function can
actually be used but changing it will of course not affect the callback that uses
it.

\starttyping[option=LUA]
function callback.find (name) return
    <t:function> | <t:nil>
end
\stoptyping

\starttyping[option=LUA]
if callback.known("foo") then
    -- do what is needed
end
\stoptyping

If you use \CONTEXT\ you should use the interfaces that it provides. Some of the
built-in \type {callback} functions are overloaded anyway.

You can use the \type {callback.getoptionvalues} and \type {callback.getstatevalues}
functions to get tables of some of the possible properties. The options are just for
diagnostics: Currently we have only a few:

\starttabulate[||||]
\FL
\NC value  \NC bit          \NC meaning \NC \NR
\ML
\NC direct \NC \type {0x01} \NC operates on a direct node \NC \NR
\NC trace  \NC \type {0x02} \NC us used for tracing \NC \NR
\LL
\stoptabulate

A callback has a state bitset where the following bits can be set. This feature
is very much related to the way \CONTEXT\ wants to protect itself against users
messing up the many callbacks it needs to function properly.

\starttabulate[||||]
\FL
\NC value       \NC bit          \NC meaning \NC \NR
\ML
\NC set         \NC \type {0x01} \NC can be used \NC \NR
\NC disabled    \NC \type {0x02} \NC temporarily disabled \NC \NR
\NC frozen      \NC \type {0x04} \NC can never be set or changed again \NC \NR
\NC private     \NC \type {0x08} \NC don't return the function \NC \NR
\NC touched     \NC \type {0x10} \NC set (or unset) by the user \NC \NR
\NC tracing     \NC \type {0x20} \NC only called when tracing \NC \NR
\NC selective   \NC \type {0x40} \NC only called when there is a need \NC \NR
\NC fundamental \NC \type {0x80} \NC need to be set and kick in on demand \NC \NR
\LL
\stoptabulate

Both have a setter and getter. In the case op states, some set stated can inhibit
changes.

\starttyping[option=LUA]
function callback.setstate (
    <t:string> name | <t:integer> id,
    <t:integer> state
)
    -- no return value
end

function callback.getstate (
    <t:string> name | <t:integer> id
)
    return <t:integer> -- state
end

function callback.setoptions (
    <t:string> name | <t:integer> id,
    <t:integer> option
)
    -- no return value
end

function callback.getoptions (
    <t:string> name | <t:integer> id
)
    return <t:integer> -- options
end
\stoptyping

You can emulate a callback with:

\starttyping[option=LUA]
function callback.testonly ( )
    return <t:boolean> -- success
end
\stoptyping

but you need to register the \type {test_only} callback for that to work. The set
function gets the state as first argument. When the function crashes, the \typ
{callback.testonly} call will return \type {false}, otherwise you get back \type
{true}.

% We also have callback.usage() that we keep around but it's also in the status
% library so there is no real need to discuss uit here.

\stopsection

\startsection[title=Files]

\startsubsection[title=find_log_file]

This is one of the callbacks that has to be set in order for the engine to work
at all. If it's not set, the run will abort with a message.

\starttyping[option=LUA]
function (
    <t:string> askedname
)
    return <t:string> foundname
end
\stoptyping

\stopsubsection

\startsubsection[title=find_format_file]

A format file is an efficient memory dump of the (in our case \CONTEXT) macro
package. In \LUATEX\ it can have a mix of \TEX and \LUA\ code but one should be
aware that storing the \LUA\ state is not up to the engine.

\starttyping[option=LUA]
function (<t:string> askedname)
    return <t:string> foundname
end
\stoptyping

A format file can be read from any valid location but is always written in the
current directory. When written the number of bytes for each section is reported.
When read all kind of checks take place in order to intercept corruption or
incompatibilities. Contrary to \LUATEX, the \LUAMETATEX\ is not (zip) compressed
so, in spite of more aggressive compression of data otherwise the file is a bit
larger. \footnote {This is not that true for \CONTEXT\ because it is more
efficient than \MKIV\ so the uncompressed \MKIV\ format file is much larger than
\MKXL, although the later grows as we preload more functionality.}

\stopsubsection

\startsubsection[title=open_data_file]

This callback function gets a filename passed. The return value is either the boolean
value false or a table with two functions. A mandate \type {reader} function fill be
called once for each new line to be read, the optional \type {close} function will
be called once \LUATEX\ is done with the file.

\starttyping[option=LUA]
function (
    <t:string> filename
)
    return <table> {
        <function> reader(<table> environment) end,
        <function> close (<table> environment) end,
    }
end
\stoptyping

\LUAMETATEX\ never looks at the rest of the table, so we can use it to store
additional per|-|file data. Both the callback functions will receive the table as
their only argument.

\stopsubsection

\startsubsection[title=start_file]

This callback replaces the code that \LUAMETATEX\ prints when a file is opened like
\type {(filename} for regular files. The category is a number:

\starttyping[option=LUA]
function (
    <t:integer> category,
    <t:string>  filename
)
    -- no return values
end
\stoptyping

The following categories can occur:

\starttabulate[|c|l|]
% \DB value  \BC meaning \NC \NR
% \TB
\FL
\BC value  \BC meaning \NC \NR
\ML
\NC 1 \NC a normal data file, like a \TEX\ source \NC \NR
\NC 2 \NC a font map coupling font names to resources \NC \NR
\NC 3 \NC an image file (\type {png}, \type {pdf}, etc) \NC \NR
\NC 4 \NC an embedded font subset \NC \NR
\NC 5 \NC a fully embedded font \NC \NR
\LL
\stoptabulate

\stopsubsection

\startsubsection[title=stop_file]

This callback replaces the code that \LUAMETATEX\ prints when a file is closed like
the \type {)} for regular files.

\starttyping[option=LUA]
function (
    <t:integer> category
)
    -- no return values
end
\stoptyping

\stopsubsection

\stopsection

\startsection[title=Running]

\startsubsection[title=process_jobname]

This callback allows you to change the jobname given by \type {\jobname} in \TEX\
and \type {tex.jobname} in \LUA. It does not affect the internal job name or the
name of the output or log files.

\starttyping[option=LUA]
function (
    <t:string> jobname
)
    return <t:string> adjusted_jobname
end
\stoptyping

The only argument is the actual job name; you should not use \type {tex.jobname}
inside this function or infinite recursion may occur. If you return \type {nil},
\LUAMETATEX\ will pretend your callback never happened. This callback does not
replace any internal code.

\stopsubsection

\startsubsection[title=pre_dump]

This function is called just before dumping to a format file starts. It does not
replace any code and there are neither arguments nor return values. It can be
used to do some cleanup and other housekeeping.

\starttyping[option=LUA]
function (
    -- no arguments
)
    -- no return values
end
\stoptyping

\stopsubsection

\startsubsection[title=start_run]

\starttyping[option=LUA]
function(
    -- no arguments
)
    -- no return values
end
\stoptyping

This callback replaces the code that prints \LUATEX's banner. Note that for
successful use, this callback has to be set in the \LUA\ initialization script,
otherwise it will be seen only after the run has already started.

\stopsubsection

\startsubsection[title=stop_run]

\starttyping[option=LUA]
function(
    -- no arguments
)
    -- no return values
end
\stoptyping

This callback replaces the code that prints \LUATEX's statistics and \quote
{output written to} messages. The engine can still do housekeeping and therefore
you should not rely on this hook for post-processing the \PDF\ or log file.

\stopsubsection

\startsubsection[title=intercept_tex_error]

This callback is run from inside the \TEX\ error function, and the idea is to
allow you to do some extra reporting on top of what \TEX\ already does, for
instance recovering. You may find some of the values in the \type {status} table
useful. The \TEX\ related callback gets two arguments: the current processing
mode and a boolean indicating if there was a runaway argument.

\starttyping[option=LUA]
function (
    -- no arguments
)
    -- no return values
end
\stoptyping

When \TEX\ was written, when computers were slower, running a batch hob could
involve a terminal, and startup-time mattered, it made sense to enter an edit
mode and try to fix the error, for instance by inserting in or deleting from the
input. In \LUAMETATEX\ we don't have these features. A quit, fix, restart cycle
makes more sense that is what we do in \CONTEXT.

\stopsubsection

\startsubsection[title=intercept_lua_error]

This callback is similar to the one discussed in the previous section but for
\LUA. Of course we should end up in a recoverable state for this to work well so
in practice you could as well just quit the run.

\starttyping[option=LUA]
function (
    -- no arguments
)
    -- no return values
end
\stoptyping

\stopsubsection

\startsubsection[title=show_error_message]

This callback replaces the code that prints the error message. As mentioned before
it probably makes sense to quit the run after reporting.

\starttyping[option=LUA]
function (
    -- no arguments
)
    -- no return values
end
\stoptyping

\stopsubsection

\startsubsection[title=show_warning_message]

This callback replaces the code that prints a (non fatal) warning message.
Contrary to an error one will likely continue the run.

\starttyping[option=LUA]
function (
    -- no arguments
)
    -- no return values
end
\stoptyping

\stopsubsection

\startsubsection[title=wrapup_run]

This callback is called after the \PDF\ and log files are closed. Use it at your own
risk because after all you need to make sure that the user is not surprised by for
instance bad result.

\starttyping[option=LUA]
function (
    -- no arguments
)
    -- no return values
end
\stoptyping

\stopsubsection

\startsubsection[title=handle_overload]

One characteristic of \TEX\ is that you have quite some control over what a
control sequence triggers. For instance, \type {\hbox} normally starts a
horizontal box but a user can redefine this primitive as macro to do whatever is
required. This means that when other macros use this primitive their behavior
will change. One way out of this is using aliases, for instance:

\starttyping
\normalsetbox0\normalhbox{test}
\normalifdim\normalwd0>10pt \normalbox0 \normalfi
\stoptyping

But even these normal aliases can be redefined. Of course you can use special
characters like \type {_} in names but once you start doing this:

\starttyping
\p_setbox0\p_hbox{test}
\p_ifdim\p_wd0>10pt \p_box0 \p_fi
\stoptyping

you should wonder if you still offer the user \TEX\ as a programming language.
It's not the route that \CONTEXT\ takes.

In \LUAMETATEX\ every macro (including primitives) can be flagged and that happens
with so called prefixes. Traditional \TEX\ offers:

\starttyping
\global\def\foo{...}
\long  \def\foo{...} % no-op
\outer \def\foo{...} % no-op
\stoptyping

The \type {\long} and \type {\outer} made sense at that time but are no-ops in
\LUAMETATEX: every macro can take \type {\par} equivalents as arguments and can
be \ defined at every level. The \ETEX\ extensions introduced this prefix:

\starttyping
\protected\def\foo{...}
\stoptyping

which prevents expansion unless the value is really expected (needed). The
\LUAMETATEX\ engine added:

\starttyping
\semiprotected\def\foo{...}
\stoptyping

but when eventually I see no reason to use it in \CONTEXT\ it might be dropped. A
special prefix is:

\starttyping
\constant\def\foo{...}
\stoptyping

This effectively is equivalent to \type {\edef} but signals that in some
scenarios (like an \type {\csname} equivalent situation) no expansion and
checking has to happen which improves performance.

These two prefixes are just signals to \LUA\ driven functionality:

\starttyping
\deferred  \foo
\immediate \foo
\stoptyping

The prefixes do nothing except when \type {\foo} are \LUA\ calls that can use
this information to adapt behavior. Because we have no backend the macro package
has to come up with equivalents for e.g.\ \type {\write} than can be immediate or
deferred (default) operations.

Another prefix relates to alignments:

\starttyping
\noaligned\protected\def\foo{...}
\stoptyping

Which makes a macro accepted between alignment rows where otherwise protected
macros will trigger an error due to look ahead.

A definition with \type {\def} or \type {\gdef} can take arguments and these can be
made optional with:

\starttyping
\def\tolerant[#1]{...}
\stoptyping

but there are more features related to tolerant:

\starttyping
\def\tolerant[#1]#*[#2]{...}
\stoptyping

that are discusses in low level manuals. Users can define macros that are
reported (in tracing) as if they were primitives:

\starttyping
\untraced\protected\def\foo{...}
\stoptyping

The prefixes \type {\constrained} and \type {\retained} relate to register values
being saved and restored in groups. The \type {\inherited} is used in for
instance math spacing assignments where we need dynamic binding to for instance
\type {\muskip} registers (instead of values).

Although not related to the callback discussed here we mentioned these prefixes because
they belong to the \type {prefixed_cmd} operator|/|operand pair. So to come back to
users being able to use primitives instead of funny unreadable aliases. It's good to
keep in  mind that one can combine prefixes like the following:

\starttyping
\frozen    \foo{...}
\immutable \foo{...}
\instance  \foo{...}
\mutable   \foo{...}
\overloaded\foo{...}
\permanent \foo{...}
\stoptyping

so this is valid too:

\starttyping
\global\permanent\untraced\tolerant\protected\def\foo[#1]#*[#2]{...}
\stoptyping

So what do these prefixes do? It depends on the value of an internal
integer \type {\overloadmode} where the following values have meaning:

\starttabulate[|c|l|c|c|c|c|c|]
% \DB   \BC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
% \TB
\FL
\BC   \BC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
\ML
\NC 1 \NC warning \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
\NC 2 \NC error   \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
\NC 3 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
\NC 4 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
\NC 5 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
\NC 6 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
\LL
\stoptabulate

The \type {\enforced} prefix can be used to bypass this mechanism:

\starttyping
\permanent\protected\def\foo{...}

\protected\def\oof{\enforced\def\foo{...}}
\stoptyping

But only in so called quote {ini} mode, that is when the format file is created. In order to
save work we also have:

\starttyping
\aliased\let\foo\relax
\stoptyping

This makes \type {\foo} a copy (or more precise, a reference) including all
flags, so in this case it will be flagged a a primitive which is \type
{\permanent} too. You cannot define primitives yourself but when reported in a
trace you see it being a primitive indeed.

Of course this all means that one has to define basically all relevant macros
with a combination of prefixes and that happens to be the case in \CONTEXT, which
in the end makes this callback a rather \CONTEXT\ specific one.

\starttyping[option=LUA]
function (
    <t:boolean> error,
    <t:integer> overload,
    <t:string>  csname,
    <t:integer> flags
)
    -- no return values
end
\stoptyping

\stopsubsection

\stopsection

\startsection[title=Fonts]

\startsubsection[title=define_font]

The engine has no font loader but it does need some information about the glyphs
that are uses like width, height and depth, possibly italic correction, kerns,
and ligatures. And for math some more information is needed. Keep in mind that
for instance italic correction is something specific for \TEX\ and that kerns and
ligatures only are needed when you leave them to the engine. For modern \OPENTYPE\ fonts
we let \LUA\ deal with this.

\starttyping[option=LUA]
function (
    <t:string>  name,
    <t:integer> size
)
    return <t:integer> id
end
\stoptyping

The string \type {name} is the filename part of the font specification, as given
by the user, for instance when \type {\font} is used for defining an instance.
The number \type {size} is a bit special:

\startitemize[packed]
\startitem
    If it is positive, it specifies an \quote{at size} in scaled points.
\stopitem
\startitem
    If it is negative, its absolute value represents a \quote {scaled} setting
    relative to the design size of the font.
\stopitem
\stopitemize

The font can be defined with \type {font.define} which returns a font identifier
that can be returned in the callback. Contrary to \LUATEX, in \LUAMETATEX\
we only accept a number.

The internal structure of the \type {font} table that is passed to \type
{font.define} is explained elsewhere but there can be much more in that table.
Likely the macro package will keep the passes table around for other usage, for
instance for usage in the backend.

Setting this callback to \type {false} is pointless because it will prevent font
loading completely because without fonts there is little to do for the engine.

\stopsubsection

\startsubsection[title=quality_font]

When you use font expansion you will normally pass the glyph specific expansion and
compression values along with the dimensions. However, this can be delayed. When we
use par passes (or otherwise set one of the adjust parameters) and a font has not
yet bee setup for expansion this callback will kick in but only once per font.

\starttyping[option=LUA]
function (
    <t:integer> id
)
    -- no return values
end
\stoptyping

The function can set additional parameters in the font and pass them to \TEX\ using
helpers from the font library.

\stopsubsection

\stopsection

\startsection[title=Typesetting]

\startsubsection[title=pre_output]

This callback is called when \TEX\ is ready to start boxing the box 255 for \prm
{output}. The callback does not replace any internal code.

\starttyping[option=LUA]
function (
    <t:node>    head,
    <t:string>  groupcode,
    <t:integer> size,
    <t:string>  packtype,
    <t:integer> maxdepth,
    <t:integer> direction
)
    return <t:node> newhead
end
\stoptyping

\stopsubsection

\startsubsection[title=buildpage]

This callback is called whenever \LUAMETATEX\ is ready to move stuff to the main
vertical list. You can use this callback to do specialized manipulation of the
page building stage like imposition or column balancing.

\starttyping[option=LUA]
function (
    <t:string> extrainfo
)
    -- no return values
end
\stoptyping

The string \type {extrainfo} gives some additional information about what \TEX's
state is with respect to the \quote {current page}. The possible values for this
callback are:

\starttabulate[|l|p|]
% \DB value                 \BC explanation                                 \NC \NR
% \TB
\FL
\BC value                 \BC explanation                                 \NC \NR
\ML
\NC \type{alignment}      \NC a (partial) alignment is being added        \NC \NR
\NC \type{after_output}   \NC an output routine has just finished         \NC \NR
\NC \type{new_graf}       \NC the beginning of a new paragraph            \NC \NR
\NC \type{vmode_par}      \NC \type {\par} was found in vertical mode     \NC \NR
\NC \type{hmode_par}      \NC \type {\par} was found in horizontal mode   \NC \NR
\NC \type{insert}         \NC an insert is added                          \NC \NR
\NC \type{penalty}        \NC a penalty (in vertical mode)                \NC \NR
\NC \type{before_display} \NC immediately before a display starts         \NC \NR
\NC \type{after_display}  \NC a display is finished                       \NC \NR
\NC \type{end}            \NC \LUAMETATEX\ is terminating (it's all over) \NC \NR
\LL
\stoptabulate


\stopsubsection

\startsubsection[title=hpack]

This callback is called when \TEX\ is ready to start boxing some horizontal mode
material. Math items and line boxes are ignored at the moment. The callback does
not replace any internal code.

\starttyping[option=LUA]
function (
    <t:node>    head,
    <t:string>  groupcode,
    <t:integer> size,
    <t:string>  packtype
    <t:integer> direction,
    <t:node>    attributelist
)
    return <t:node> newhead
end
\stoptyping

The \type {packtype} is either \type {additional} or \type {exactly}. If \type
{additional}, then the \type {size} is a \type {\hbox spread ...} argument. If
\type {exactly}, then the \type {size} is a \type {\hbox to ...}. In both cases,
the number is in scaled points.

\stopsubsection

\startsubsection[title=vpack]

This callback is called when \TEX\ is ready to start boxing some vertical mode
material. Math displays are ignored at the moment. The callback does not replace
any internal code.

This function is very similar to \type {hpack}. Besides the fact that it is
called at different moments, there is an extra variable that matches \TEX's \type
{\maxdepth} setting.

\starttyping[option=LUA]
function (
    <t:node>    head,
    <t:string>  groupcode,
    <t:integer> size,
    <t:string>  packtype,
    <t:integer> maxdepth,
    <t:integer> direction,
    <t:node>    attributelist
)
    return <t:node> newhead
end
\stoptyping

\stopsubsection

\startsubsection[title=hyphenate]

This callback is supposed to insert discretionary nodes in the node list it
receives.

\starttyping[option=LUA]
function (
    <t:node> head,
    <t:node> tail
)
    -- no return values
end
\stoptyping

Setting this callback to \type {false} will prevent the internal discretionary
insertion pass.

\stopsubsection

\startsubsection[title=ligaturing]

This callback, which expects no return values, has to apply ligaturing to the
node list it receives.

\starttyping[option=LUA]
function (
    <t:node> head,
    <t:node> tail
)
    -- no return values
end
\stoptyping

You don't have to worry about return values because the \type {head} node that is
passed on to the callback is guaranteed not to be a glyph_node (if need be, a
temporary node will be prepended), and therefore it cannot be affected by the
mutations that take place. After the callback, the internal value of the \quote
{tail of the list} will be recalculated.

The \type {next} of \type {head} is guaranteed to be non-nil. The \type {next} of
\type {tail} is guaranteed to be nil, and therefore the second callback argument
can often be ignored. It is provided for orthogonality, and because it can
sometimes be handy when special processing has to take place.

Setting this callback to \type {false} will prevent the internal ligature
creation pass. You must not ruin the node list. For instance, the head normally
is a local par node, and the tail a glue. Messing too much can push \LUATEX\ into
panic mode.

\stopsubsection

\startsubsection[title=kerning]

This callback has to apply kerning between the nodes in the node list it
receives. See \type {ligaturing} for calling conventions.

\starttyping[option=LUA]
function (
    <t:node> head,
    <t:node> tail
)
    -- no return values
end
\stoptyping

Setting this callback to \type {false} will prevent the internal kern insertion
pass. You must not ruin the node list. For instance, the head normally is a local
par node, and the tail a glue. Messing too much can push \LUATEX\ into panic
mode.

\stopsubsection

\startsubsection[title=glyph_run]

When set this callback is triggered when \TEX\ normally handles the ligaturing
and kerning. In \LUATEX\ you use the \typ {hpack} and \typ {pre_linebreak}
callbacks for that (where each passes different arguments). This callback doesn't
get triggered when there are no glyphs (in \LUATEX\ this optimization is
controlled by a a variable).

\starttyping[option=LUA]
function (
    <t:node>    head,
    <t:string>  groupcode,
    <t:integer> direction
)
    return <t:node> newhead
end
\stoptyping

The traditional \TEX\ font processing is bypassed so you need to take care of that
with the helpers. (For the moment we keep the ligaturing and kerning callbacks but
they are kind of obsolete.)

\stopsubsection

\startsubsection[title=pre_linebreak]

This callback is called just before \LUATEX\ starts converting a list of nodes
into a stack of \type {\hbox}es, after the addition of \type {\parfillskip}. The
callback does not replace any internal code.

\starttyping[option=LUA]
function (
    <t:node>   head,
    <t:string> groupcode
)
    return <t:node> newhead
end
\stoptyping

The string called \type {groupcode} identifies the nodelist's context within
\TEX's processing. The range of possibilities is given in the table below, but
not all of those can actually appear here, some are for the \type {hpack} and
\type {vpack} callbacks.

\starttabulate[|l|p|]
% \DB value                \BC explanation                                     \NC \NR
% \TB
\FL
\BC value                \BC explanation                                     \NC \NR
\ML
\NC \type{<empty>}       \NC main vertical list                              \NC \NR
\NC \type{hbox}          \NC \type {\hbox} in horizontal mode                \NC \NR
\NC \type{adjusted_hbox} \NC \type {\hbox} in vertical mode                  \NC \NR
\NC \type{vbox}          \NC \type {\vbox}                                   \NC \NR
\NC \type{vtop}          \NC \type {\vtop}                                   \NC \NR
\NC \type{align}         \NC \type {\halign} or \type {\valign}              \NC \NR
\NC \type{disc}          \NC discretionaries                                 \NC \NR
\NC \type{insert}        \NC packaging an insert                             \NC \NR
\NC \type{vcenter}       \NC \type {\vcenter}                                \NC \NR
\NC \type{local_box}     \NC \type {\localleftbox} or \type {\localrightbox} \NC \NR
\NC \type{split_off}     \NC top of a \type {\vsplit}                        \NC \NR
\NC \type{split_keep}    \NC remainder of a \type {\vsplit}                  \NC \NR
\NC \type{align_set}     \NC alignment cell                                  \NC \NR
\NC \type{fin_row}       \NC alignment row                                   \NC \NR
\LL
\stoptabulate

As for all the callbacks that deal with nodes, the return value can be one of
three things:

\startitemize
\startitem
    boolean \type {true} signals successful processing
\stopitem
\startitem
    \type {<t:node>} signals that the \quote {head} node should be replaced by the
    returned node
\stopitem
\startitem
    boolean \type {false} signals that the \quote {head} node list should be
    ignored and flushed from memory
\stopitem
\stopitemize

\stopsubsection

\startsubsection[title=linebreak]

This callback replaces \LUATEX's line breaking algorithm. The callback does not
replace any internal code.

\starttyping[option=LUA]
function (
    <t:node>    head,
    <t:boolean> is_display
)
    return <t:node> newhead
end
\stoptyping

The returned node is the head of the list that will be added to the main vertical
list, the boolean argument is true if this paragraph is interrupted by a
following math display.

If you return something that is not a \type {<t:node>}, \LUATEX\ will apply the
internal linebreak algorithm on the list that starts at \type {<head>}.
Otherwise, the \type {<t:node>} you return is supposed to be the head of a list of
nodes that are all allowed in vertical mode, and at least one of those has to
represent an \type {\hbox}. Failure to do so will result in a fatal error.

Setting this callback to \type {false} is possible, but dangerous, because it is
possible you will end up in an unfixable \quote {deadcycles loop}.

\stopsubsection

\startsubsection[title=post_linebreak]

This callback is called just after \LUATEX\ has converted a list of nodes into a
stack of \type {\hbox}es.

\starttyping[option=LUA]
function (
    <t:node>   head,
    <t:string> groupcode
)
    return <t:node> newhead
end
\stoptyping

\stopsubsection

\startsubsection[title=append_to_vlist]

This callback is called whenever \LUATEX\ adds a box to a vertical list (the
\type {mirrored} argument is obsolete):

\starttyping[option=LUA]
function (
    <t:node>    box,
    <t:string>  locationcode,
    <t:integer> prevdepth
)
    return <t:node> list [, <t:integer> prevdepth [, <t:boolean> checkdepth ] ]
end
\stoptyping

It is ok to return nothing or \type {nil} in which case you also need to flush
the box or deal with it yourself. The prevdepth is also optional. Locations are
\type {box}, \type {alignment}, \type {equation}, \type {equation_number} and
\type {post_linebreak}. When the third argument returned is \type {true} the
normal prevdepth correction will be applied, based on the first node.

\stopsubsection

\startsubsection[title=alignment]

This is an experimental callback that when set is called several times during the
construction of an alignment. The context values are available in \typ
{tex.getalignmentcontextvalues()}.

\starttyping[option=LUA]
function (
    <t:node>   head,
    <t:string> context,
    <t:node>   attributes,
    <t:node>   preamble
)
    -- no return values
end
\stoptyping

There are no sanity checks so if a user messes up the passed node lists the results
can be unpredictable and, as with other node related callbacks, crash the engine.

\stopsubsection

\startsubsection[title=local_box]

Local boxes are a somewhat tricky and error prone feature so use this callback
with care because the paragraph is easily messed up. A line can have a left,
right and middle box where the middle one has no width. This callback does not
replace any internal code. The callback gets quite some parameters passed:

\starttyping[option=LUA]
function (
    <t:node>    linebox,
    <t:node>    leftbox,
    <t:node>    rightbox,
    <t:node>    middlebox,
    <t:integer> linenumber,
    <t:integer> leftskip,
    <t:integer> rightskip,
    <t:integer> lefthang,
    <t:integer> righthang,
    <t:integer> indentation,
    <t:integer> parinitleftskip,
    <t:integer> parinitrightskip,
    <t:integer> parfillleftskip,
    <t:integer> parfillrightskip,
    <t:integer> overshoot
)
    -- no return values
end
\stoptyping

This is an experimental callback that will be tested in different \CONTEXT\
mechanisms before it will be declared stable.

\stopsubsection

\startsubsection[title=packed_vbox]

After the \type {vpack} callback (see previous section) is triggered the box get
packed and after that this callback can be configured to kick in.

\starttyping[option=LUA]
function (
    <t:node>   head,
    <t:string> groupcode
)
    return <t:node> newhead
end
\stoptyping

\stopsubsection

\startsubsection[title=handle_uleader]

The \typ {\uleaders} command inserts a user leader into the list. When a list get packed
and has such leaders, a run over the list happens after packing so that it can be
finalized.

\starttyping[option=LUA]
function (
    <t:node>    head,
    <t:string>  context,
    <t:integer> index,
    <t:node>    box,
    <t:integer> location
)
    return <t:node> head
end
\stoptyping

\stopsubsection

\startsubsection[title=italic_correction]

The concept of italic correction is very much related to traditional \TEX\ fonts.
At least in 2024 it is absent from \OPENTYPE\ although it has some meaning in
\OPENTYPE\ math. In \TEX\ this correction is normally inserted by \type {\/}
although in \LUAMETATEX\ we also have \typ {\explicititaliccorrection} as well as
\typ {\forcedleftcorrection} and \typ {\forcedrightcorrection}.

When this callback is enabled it gets triggered when one of left or right
correction commands is given and the returned kern is then used as correction.

\starttyping[option=LUA]
function (
    <t:node>    glyph,
    <t:integer> kern,
    <t:integer> subtype,
)
    return <t:integer> kern
end
\stoptyping

\stopsubsection

\startsubsection[title=insert_par]

Each paragraph starts with a local par node that keeps track of for instance
the direction. You can hook a callback into the creator:

\starttyping[option=LUA]
function (
    <t:node>   par,
    <t:string> location
)
    -- no return values
end
\stoptyping

There is no return value and you should make sure that the node stays valid
as otherwise \TEX\ can get confused.

\stopsubsection

\startsubsection[title=append_line]

Every time a line is added this callback is triggered, when set. migrated
material and adjusts also qualify as such and the detail relates to the adjust
index.

\starttyping[option=LUA]
function (
    <t:node>    head,
    <t:node>    tail,
    <t:string>  context,
    <t:integer> detail
)
    return <t:node> newhead
end
\stoptyping

A list of possible context values can be queried with \typ
{tex.getappendlinecontextvalues()}.

\stopsubsection

\startsubsection[title=insert_distance]

This callback is called when the page builder adds an insert. There is not much
control over this mechanism but this callback permits some last minute
manipulations of the spacing before an insert, something that might be handy when
for instance multiple inserts (types) are appended in a row.

\starttyping[option=LUA]
function (
    <t:integer> class,
    <t:integer> order
)
    return <t:integer> register
end
\stoptyping

The return value is a number indicating the skip register to use for the
prepended spacing. This permits for instance a different top space (when \type
{class} equals one) and intermediate space (when \type {class} is larger than
one). Of course you can mess with the insert box but you need to make sure that
\LUATEX\ is happy afterwards.

\stopsubsection

\startsubsection[title=begin_paragraph]

Every time a paragraph starts this callback, when configured, will kick in:

\starttyping[option=LUA]
function (
    <t:boolean> invmode,
    <t:boolean> indented,
    <t:string>  context
)
    return <t:boolean> indented
end
\stoptyping

There are many places where a new paragraph can be triggered:

\startfourrows
\getbuffer[engine:syntax:parcontextcodes]
\stopfourrows

\stopsubsection

\startsubsection[title=paragraph_context]

When the return value of this callback is \typ {false} the paragraph related
settings, when they have been updated, will not be updated.

\starttyping[option=LUA]
function (
    <t:string> context
)
    return <t:boolean> ignore
end
\stoptyping

\stopsubsection

\startsubsection[title=missing_character]

This callback is triggered when a character node is created and the font doesn't
have the requested character.

\starttyping[option=LUA]
function (
    <t:integer> location,
    <t:node>    glyph,
    <t:integer> font,
    <t:integer> character
)
    -- no return value
end
\stoptyping

When \prm {tracinglostchars} is set to a positive value a message goes to the log
and a value larger than one also makes it show up non the terminal. In the
callback, the location is one of:

\startfourrows
\getbuffer[engine:syntax:missingcharactervalues]
\stopfourrows

\stopsubsection

\startsubsection[title=process_character]

This callback is experimental and gets called when a glyph node is created and
the callback field in a character is set.

\starttyping[option=LUA]
function (
    <t:integer> font,
    <t:integer> character
)
    -- no return value
end
\stoptyping

\stopsubsection

\startsubsection[title=tail_append]

%         This callback is called when \LUATEX\ adds contents to list:

%         \starttyping[option=LUA]
%         function (
%             <t:string> extrainfo
%         )
%         end
%         \stoptyping

%         The string reports the group code. From this you can deduce from
%         what list you can give a treat.

%         \starttabulate[|l|p|]
%         \DB value             \BC explanation                                  \NC \NR
%         \TB
%         \NC \type{pre_box}    \NC interline material is being added            \NC \NR
%         \NC \type{pre_adjust} \NC \type {\vadjust} material is being added     \NC \NR
%         \NC \type{box}        \NC a typeset box is being added (always called) \NC \NR
%         \NC \type{adjust}     \NC \type {\vadjust} material is being added     \NC \NR
%         \LL
%         \stoptabulate

\stopsubsection

\stopsection

\startsection[title=Tracing]

\startsubsection[title=hpack_quality]

This callback can be used to intercept the overfull messages that can result from
packing a horizontal list (as happens in the par builder). The function takes a
few arguments:

\starttyping[option=LUA]
function (
    <t:string>  incident,
    <t:integer> detail,
    <t:node>    head,
    <t:integer> first,
    <t:integer> last
 )
    return <t:node> whatever
end
\stoptyping

The incident is one of \type {overfull}, \type {underfull}, \type {loose} or
\type {tight}. The detail is either the amount of overflow in case of \type
{overfull}, or the badness otherwise. The head is the list that is constructed
(when protrusion or expansion is enabled, this is an intermediate list).
Optionally you can return a node, for instance an overfull rule indicator. That
node will be appended to the list (just like \TEX's own rule would).

\stopsubsection

\startsubsection[title=vpack_quality]

This callback can be used to intercept the overfull messages that can result from
packing a vertical list (as happens in the page builder). The function takes a
few arguments:

\starttyping[option=LUA]
function (
    <t:string>  incident,
    <t:integer> detail,
    <t:node>    head,
    <t:integer> first,
    <t:integer> last
)
    -- no return values
end
\stoptyping

The incident is one of \type {overfull}, \type {underfull}, \type {loose} or
\type {tight}. The detail is either the amount of overflow in case of \type
{overfull}, or the badness otherwise. The head is the list that is constructed.

\stopsubsection

\startsubsection[title=line_break]

This callback is actually a set of callbacks that has to be deals with as a
whole. The main reason why we have this callback is that we wanted to be able to
see what the par builder is doing, especially when we implement multiple
paragraph building passes. This makes the callback pretty much a rather \CONTEXT\
specific one.

{\em We can also consider fetching the passive and active lists because we now keep
much more info around.}

\starttyping[option=LUA]
function(
    <t:integer> context,
    <t:integer> checks,
    ...
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function initialize (
    <t:integer> context,
    <t:integer> checks,
    <t:integer> subpasses
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function start (
    <t:integer> context,
    <t:integer> checks,
    <t:integer> pass,
    <t:integer> subpass,
    <t:integer> classes,
    <t:integer> decent
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function stop (
    <t:integer> context,
    <t:integer> checks,
    <t:integer> demerits
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function collect (
    <t:integer> context,
    <t:integer> checks
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function line (
    <t:integer> context,
    <t:integer> checks,
    <t:integer> box,
    <t:integer> badness,
    <t:integer> overshoot,
    <t:integer> shrink,
    <t:integer> stretch,
    <t:integer> line,
    <t:integer> serial
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function delete (
    <t:integer> context,
    <t:integer> checks,
    <t:integer> serial
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function wrapup (
    <t:integer> context,
    <t:integer> checks,
    <t:integer> demerits,
    <t:integer> looseness
)
    -- no return values
end
\stoptyping
\starttyping[option=LUA]
function check (
    <t:integer> context,
    <t:integer> checks,
    <t:integer> pass,
    <t:integer> subpass,
    <t:integer> serial,
    <t:integer> prevserial,
    <t:integer> linenumber,
    <t:integer> nodetype,
    <t:integer> fitness,,
    <t:integer> demerits,
    <t:integer> classes,
    <t:integer> badness,
    <t:integer> demerits,
    <t:node>    breakpoint,
    <t:integer> short,
    <t:integer> glue,
    <t:integer> linewidth
)
    return <t:integer> demerits  -- optional
end
\stoptyping

\starttyping[option=LUA]
function list (
    <t:integer> context,
    <t:integer> checks,
    <t:integer> serial
)
    -- no return values
end
\stoptyping

Every one of these gets a \type {context} and \type {checks} passes. Possible
contexts are:

\startfourrows
\getbuffer[engine:syntax:linebreakcontextcodes]
\stopfourrows

The \type {checks} parameters is the value of \type {\linebreakchecks} which
makes it possible to plug in actions depending on that number. To give an idea if
what gets called, this is what you get when typesetting \type {tufte.tex}:
initialize, start, report, delete, delete, stop, start, report, report, delete,
report, report, report, delete, delete, report, report, report, delete, report,
delete, delete, report, report, report, delete, report, delete, delete, report,
report, delete, report, report, delete, delete, delete, report, delete, report,
delete, delete, report, report, report, delete, delete, report, delete, report,
report, delete, report, delete, delete, delete, report, report, delete, report,
report, delete, report, delete, report, delete, report, delete, delete, report,
report, report, report, delete, delete, delete, delete, delete, delete, delete,
delete, delete, report, stop, collect, list, list, list, list, list, list, list,
list, list, line, line, line, line, line, line, line, line, line, wrapup.

\stopsubsection

\startsubsection[title=show_build]

You can trace (and even influence) the page builder with this callback. It comes in
several variants that are called during the process. Callbacks like these assume that
one knows what is going on in the engine.

\starttyping[option=LUA]
function initialize (
    <t:integer> context
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function step (
    <t:integer> context,
    <t:node>    current,
    <t:integer> pagegoal,
    <t:integer> pagetotal
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function check (
    <t:integer> context,
    <t:node>    current,
    <t:boolean> moveon,
    <t:boolean> fireup,
    <t:integer> badness,
    <t:integer> costs,
    <t:integer> penalty
)
    return <t:boolean> moveon, <t:boolean> fireup
end
\stoptyping

\starttyping[option=LUA]
function skip (
    <t:integer> context,
    <t:node>    current,
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function move (
    <t:integer> context,
        <t:node>    current,
        <t:integer> lastheight,
        <t:integer> lastdepth,
        <t:integer> laststretch,
        <t:integer> lastshrink,
        <t:boolean> hasstretch
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function fireup (
    <t:integer> context,
    <t:node> current
)
    -- no return values
end
\stoptyping

\starttyping[option=LUA]
function wrapup (
    <t:integer> context
)
    -- no return values
end
\stoptyping

\stopsubsection

\startsubsection[title=show_whatsit]

Because we only have a generic whatsit it is up to the macro package to provide
details when tracing them.

\starttyping[option=LUA]
function (
    <t:node>    whatsit,
    <t:integer> indentation,
    <t:integer> tracinglevel,
    <t:integer> currentlevel,
    <t:integer> inputlevel
)
    -- no return value
end
\stoptyping

Here \type {indentation} tells how many periods are to be typeset if you want to
be compatible with the rest of tracing. The \type {tracinglevel} indicates if the
current level and|/|or input level are shown cf. \type {\tracinglevels}. Of
course one is free to show whatever in whatever way suits the whatsit best.

\stopsubsection

\startsubsection[title=linebreak_quality]

\starttyping[option=LUA]
function (
    <t:node>    par,
    <t:integer> id,
    <t:integer> pass,
    <t:integer> subpass,
    <t:integer> subpasses,
    <t:integer> state,
    <t:integer> overfull,
    <t:integer> underfull,
    <t:integer> verdict,
    <t:integer> classified,
    <t:integer> line
)
    return <t:node> result
end
\stoptyping

\stopsubsection

\startsubsection[title=show_loners]

In spite of widow, club, broken and shaping penalties we can have single lines in the
result. When set, this callback replaces the output that normally \typ {\tracingloners}
produces.

\starttyping[option=LUA]
function (
    <t:integer> options,
    <t:integer> penalty
)
    return <t:node> result
end
\stoptyping

The options are those set on the encountered penalty:

\startthreerows
\getbuffer[engine:syntax:penaltyoptioncodes]
\stopthreerows

\stopsubsection

\startsubsection[title=get_attribute]

Because attributes are abstract pairs of indices and values the reported
properties makes not much sense and are very macro package (and user) dependent.
This callback permits more verbose reporting by the engine when tracing is
enabled.

\starttyping[option=LUA]
function (
    <t:integer> index,
    <t:integer> value
)
    return <t:string>, <t:string>
end
\stoptyping

\stopsubsection

\startsubsection[title=get_noad_class]

We have built|-|in math classes but there can also be user defined ones. This
callback can be used to report more meaningful strings instead of numbers when
tracing.

\starttyping[option=LUA]
function (
    <t:integer> class
)
    return <t:string>
end
\stoptyping

\stopsubsection

\startsubsection[title=get_math_dictionary]
    todo
\stopsubsection

\startsubsection[title=show_lua_call]

When the engine traces something that involves a \LUA\ call it makes sense to report something
more meaningful than just that. This callback can be used provide a meaningful string (like the
name of a function).

\starttyping[option=LUA]
function (
    <t:string>  name,
    <t:integer> index
)
    return <t:string>
end
\stoptyping

\stopsubsection

\startsubsection[title=trace_memory]

When the engine starts all kind of memory is pre|-|allocated> depending on the
configuration more gets allocated when a category runs out of memory. The
\LUAMETATEX\ engine is more dynamic than \LUATEX. If this callback is set it will
get called as follows:

\starttyping[option=LUA]
function (
    <t:string>  category,
    <t:boolean> success
)
    -- no return value
end
\stoptyping

The boolean indicates if the allocation has been successful. One can best quit
the run when this one is \type {false} which the engine is likely to do that
anyway, be in in a less graceful way that you might like.

\stopsubsection

\startsubsection[title=paragraph_pass]

{\em This callback is not yet stable.}

\stopsubsection

\stopsection

\startsection[title=Math]

\startsubsection[title=mlist_to_hlist]

This callback replaces \LUATEX's math list to node list conversion algorithm.

\starttyping[option=LUA]
function (
    <t:node>    head,
    <t:string>  display_type,
    <t:boolean> need_penalties
)
    return <t:node> newhead
end
\stoptyping

The returned node is the head of the list that will be added to the vertical or
horizontal list, the string argument is either \quote {text} or \quote {display}
depending on the current math mode, the boolean argument is \type {true} if
penalties have to be inserted in this list, \type {false} otherwise.

Setting this callback to \type {false} is bad, it will almost certainly result in
an endless loop.

\stopsubsection

\startsubsection[title=math_rule]

In math rules are used for fractions, radicals and accents. In the case of
radicals rules mix with glyphs to build the symbol. In \CONTEXT\ we can enable an
alternate approach that uses glyphs instead of rules so that we can have more
consistent shapes, for instance with slopes or non square endings. This callback
takes care of that.

\starttyping[option=LUA]
function (
    <t:integer> subtype,
    <t:integer> font,
    <t:integer> width,
    <t:integer> height,
    <t:node>    attributes
)
    return <t:node> rule
end
\stoptyping

\stopsubsection

\startsubsection[title=make_extensible]

Like \typ {math_rule} this callback is used to construct nicer extensibles in \CONTEXT\
math support. It can optionally be followed by \typ {register_extensible}.

\starttyping[option=LUA]
function (
    <t:node>    extensible,
    <t:integer> fnt,
    <t:integer> chr,
    <t:integer> size,
    <t:integer> width,
    <t:integer> height,
    <t:integer> depth,
    <t:integer> linewidth,
    <t:integer> axis,
    <t:integer> exheight,
    <t:integer> emwidth
)
    return <t:node> -- boxed extensible
end
\stoptyping

\stopsubsection

\startsubsection[title=register_extensible]

This callback is a possible follow up on \typ {make_extensible} and it can be
used to share pre-build extensibles or package them otherwise (for instance as
\TYPETHREE\ glyph).

\starttyping[option=LUA]
function (
    <t:integer> fnt,
    <t:integer> chr,
    <t:integer> size,
    <t:node>    attributes,
    <t:node>    extensible
)
    return <t:node> -- boxed
end
\stoptyping

\stopsubsection

\startsubsection[title=balance]

This callback is comparable with the \type {line_break} callback. We use it for
tracing in \CONTEXT\ during development (as well as for documentation).

\stopsubsection

\startsubsection[title=balance_insert]

This is callback kicks in every time an insert is seen when balancing.

\starttyping[option=LUA]
function (
    <t:node>    current,
    <t:integer> callback,
    <t:integer> insert_index,
    <t:integer> insert_identifier
)
    -- no return value
end
\stoptyping

\stopsubsection

\startsubsection[title=balance_boundary]

When balancing, this is callback kicks in every time a node resulting from \prm
{balanceboundary} is seen.

\starttyping[option=LUA]
function (
    <t:integer> boundary_data,
    <t:integer> boundary_reserved,
    <t:integer> shape_identifier,
    <t:integer> shape_slot
)
    return
        <t:integer>, -- action
        <t:integer>, -- penalty
        <t:integer>  -- extra
end
\stoptyping

What happens after the callback returns control to \TEX\ depends on the first
return value:

\showenginevalues{getbalancecallbackvalues}

This is an experimental feature. In due time there will be a bit more explanation
here.

\stopsubsection

\stopsection

\stopdocument

% timestamp october 2024, after running into Anna von Hausswolff's "All Thoughts
% Fly" live at Basilica di Santa Maria dei Servi 29.11.2021
