% language=us runpath=texruns:manuals/musings

\startcomponent musings-kerns

\environment musings-style

\startchapter[title={Missing kerns}]

\startsubject[title=Introduction]

When a \TEX\ user posts a math related font question on SE there is a good
chance that it catches our eyes. In this case the question
\footnote {\hyphenatedurl {https://tex.stackexchange.com/q/692788/52406}}
was about the lack of kerning in \OPENTYPE\ math usage.
The person (signature \quote{dmaxwell}) showed these examples

\startplacefloat
    [intermezzo]
    [location=nonumber]
    \startcombination[nx=2,ny=1,distance=2em]
        {\externalfigure[musings-kerns-Q-pdftex][scale=3000]}{pdftex}
        {\externalfigure[musings-kerns-Q-luatex][scale=3000]}{luatex}
    \stopcombination
\stopplacefloat

and asked if it is possible to use kern pairs also in \OPENTYPE math.

Before we dive into the details we show how we can solve it in \CONTEXT\ lmtx.
We do in fact have support for kern pairs, and one way to set them up is via
the goodie file system. The goodie file for Latin Modern Math is \typ
{modern-math.lfg}. There are plenty of tweaks, and we can add an instance
of the \quotation{kernpairs} tweak.

\starttyping
{
    tweak = "kernpairs",
    list = {
        [0x1D451] = { -- italic d
            [0x1D453] = -.2, -- italic f
        },
    },
},
\stoptyping

This moves the italic f to the left (the minus sign), and the amount is
20\percent\ of the width of the italic d. With a similar entry for the
combination of italic W and the comma, we get

\startplacefloat
    [intermezzo]
    [location=nonumber]
    \externalfigure[musings-kerns-A-luametatex.pdf][scale=3000]
\stopplacefloat

The kerning here is not as aggressive as the (rather famous) $df$ kerning
in Computer modern. When we came to discuss this issue, we realized that
there were a bit more for us to do, and more to say on the subject that also
could interest others, so keep on reading.

\stopsubject

\startsubject[title=Kerns in \OPENTYPE\ math]

Scanning the \OPENTYPE\ math font specification
\footnote {\hyphenatedurl {https://learn.microsoft.com/en-us/typography/opentype/spec/math}}
for kerning, one might become surprised. Somewhere at the beginning one can
read the following

\startnarrower
    Layout of math formulas is quite different from regular text layout that
    is done using tables such as GSUB and GPOS. Regular text layout mainly
    deals with a line of text, often formatted with a single font. In this
    situation, actions such as contextual substitution or kerning can be done
    with access to the complete context of the line of text, and the rules
    can be expressed in terms of known glyph sequences. Math layout is quite
    different from this.
\stopnarrower

Reading this, one gets the impression that there is no support for kerning in
math, at least not pairwise between characters. But further down, one finds
that there is indeed support for kerning in math, but the only kerning that
is described is what we call staircase kerning, between base characters and
sub-, super- and prescripts. They explain it like this,

\startnarrower
    For any given glyph, different values can be specified for four corner
    positions — top-right, to-left [{\em sic}], etc. — allowing for different
    kerning adjuments according to whether the glyph occurs as a subscript, a
    superscript, a base being kerned with a subscript, or a base being kerned
    with a superscript.
\stopnarrower

and then follows a rather long and technical description of how this is meant
to work.

As \quote{dmaxwell}, asking the question, noticed, this lack of kerning of
math fonts might be a mistake. This holds in particular for fonts like Latin
Modern Math, that originate from a math fonts that indeed have pairwise kerning.

Even if not specified by Microsoft, \TEX\ engines can implement additional
support. This was done for example in \LUATEX\ and \LUAMETATEX. In the latter,
there is indeed support for the built in staircase kerns, corner kerns, as well
as pairwise kerns. In fact, the staircase kerns has proven to not be too useful,
so they are automatically transformed into corner kerns (that also work between
for example a base character and a superscript, as in the example below).

\startbuffer
\dm {\left( \frac {a}{b} \right)^2}
\stopbuffer

\startlinecorrection
\scale[s=2]{\strut\showglyphs\showfontkerns\switchtobodyfont[modern]\inlinebuffer}
\stoplinecorrection

In the rest of this article we will focus on the pairwise kerns.

\stopsubject

\startsubject[title=Kerns in traditional math]

Traditional \TEX\ font handling involves \TFM\ files. These files provide
properties of glyphs. For each glyph it specifies a width, height, depth and
italic correction, but does that in a compact way: choosing from a limited set.
For the discussion here this is not relevant, apart maybe from the observation
that the combination of width and italic correction in math is kind of special.
It also explains why there can be differences in metrics between \OPENTYPE\ fonts
and its ancestors, just because in \OPENTYPE\ we can be more granular. Optionally
a character can have a pointer to a larger size, an extensible recipe, for
instance for delimiters. Also optional is ligature and kern information. For
efficiency reasons these properties are packed into a table with flags in the
entries indicating what we're dealing with.

You can use the \type {tftopl} program to see what is actually in a \TFM\ file,
and here is such an entry for the \im {N}:

\starttyping
(CHARACTER C N
    (CHARWD R 0.8034725)
    (CHARHT R 0.683332)
    (CHARIC R 0.109027)
    (COMMENT
        (KRN C = R -0.083334)
        (KRN C = R -0.027779)
        (KRN C ; R -0.055555)
        (KRN C : R -0.055555)
        (KRN O 177 R 0.083336)
    )
)
\stoptyping

If we look at the \LUA\ representation that we have in \CONTEXT\ we get this:

\startluacode
    local t = fonts.handlers.tfm.readers.loadtfm(resolvers.findfile("cmmi10.tfm"))
    buffers.assign("temp",table.serialize(t.glyphs[utf.byte("N")]))
\stopluacode

\typebuffer[temp]

Here we are only interested in the \type {kerns} sub table and when you compare both
listings you will notice that the first one has one entry more and we admit that this
caught us by surprise. We were interested in these values because we wanted to add them
to the Latin Modern Math goodie file, so what do we have to choose?

When we load this font in \CONTEXT, we get a message:

\starttyping
duplicate between index 0x4E and 0x3D: -0.83334pt -> -0.27779pt
duplicate between index 0x58 and 0x3D: -0.83334pt -> -0.27779pt
\stoptyping

The left characters are the \type {N} and \type {X}, the right one is the \type
{/}, which is actually one that we are less interested in because we have
fraction spacing etc. But that doesn't mean that we wanted to explore it a
bit more. The next code shows some variants, we wrap into an ord to make sure
that no inter|-|atom spacing kicks in.

\startbuffer
$\mathord{N\kern 0pt      /}$
$\mathord{N\kern-0.83334pt/}$
$\mathord{N\kern-0.27779pt/}$
\stopbuffer

\typebuffer

\startlinecorrection
\scale[width=\textwidth]{\showglyphs\getbuffer}
\stoplinecorrection

One can argue that the second variant looks best and that is indeed the one that
Don Knuth expects the engine to use. When (in traditional mode) a kern is needed
the engine will run over the characters property array and when it hits the kern
asked for it quits scanning. That also means that when we use a \LUA\ approach,
where we use a hash table instead, we should now overwrite already set entries
when the \TFM\ table gets parsed and its data gets stored. Just to be sure we
checked all engines, and all behave as expected with respect to applying the
first kern.

A natural follow up question is \quotation {where does this duplicate entry come
from}. For that we need to go back to the source. Just for the record, the Latin
Modern fonts as well as its Computer Modern predecessors behave the same here.

\starttyping
ligtable "N":"X": slash kern -1.5u#, % $N:X:$
 "C":"T": slash kern -.5u#, comma kern -u#, period kern -u#, % $C:T:$
 "B":"E":"G":"O":"Q":"R":oct"174": % $B:E:G:O:Q:R:\jmath:$
 "l":"p":"q":"t":"w":oct"002":oct"004": % $l:p:q:t:w:\it\Theta:\Xi:$
 oct"006":oct"010":oct"012":oct"014": % $\it\Sigma:\Phi:\Omega:\beta:$
 oct"020":oct"022":oct"032":oct"036": % $\zeta:\theta:\rho:\phi:$
 oct"042":oct"043":oct"045": % $\varepsilon:\vartheta:\varrho:$
 oct"046":oct"047":oct"100": % $\varsigma:\varphi:\partial:$
 skewchar kern 3skew#;
\stoptyping

This blob of \METAFONT\ code can be found in \type {mathit.mf} and in order to
understand it better we can reformat it a bit:

\starttyping
ligtable
    "N":"X":
        slash kern -1.5u#,
    "C":"T":
        slash kern -.5u#,
        comma kern -u#,
        period kern -u#,
    "B":"E":"G":"O":"Q":"R":oct"174":
    "l":"p":"q":"t":"w":oct"002":oct"004":
    oct"006":oct"010":oct"012":oct"014":
    oct"020":oct"022":oct"032":oct"036":
    oct"042":oct"043":oct"045":
    oct"046":oct"047":oct"100":
        skewchar kern 3skew#;
\stoptyping

The second \type {slash} entry will result in the extra entry in the \TFM\ file but
as mentioned it will be ignored by \TEX. One can argue that \type {tftopl} should
warn for this duplicate. This is the only entry that saves on the skewchar definition,
so it could have been like this:

\starttyping
ligtable
    "N":"X":
        slash kern -1.5u#,
        comma kern -u#,
        period kern -u#,
        skewchar kern 3skew#;
ligtable
    "C":"T":
        slash kern -.5u#,
        comma kern -u#,
        period kern -u#,
    "B":"E":"G":"O":"Q":"R":oct"174":
    "l":"p":"q":"t":"w":oct"002":oct"004":
    oct"006":oct"010":oct"012":oct"014":
    oct"020":oct"022":oct"032":oct"036":
    oct"042":oct"043":oct"045":
    oct"046":oct"047":oct"100":
        skewchar kern 3skew#;
\stoptyping

Which then gives just one kern pair for \type {N} and \type {X} with \type {/},
not that anyone ever sees this as a real problem. Of course we were curious if
other files in the collection have these duplicate entries and there is indeed
one other case, the upright text specification (including slanted and bold but
not italic) has this:

\startluacode
    local t = fonts.handlers.tfm.readers.loadtfm(resolvers.findfile("cmr10.tfm"))
    buffers.assign("temp",table.serialize(t.glyphs[utf.byte("N")]))
\stopluacode

\starttyping
duplicate between index 0x6B and 0x61: -0.55556pt -> -0.27779pt
duplicate between index 0x76 and 0x61: -0.55556pt -> -0.27779pt
\stoptyping

\startbuffer
k\kern 0pt      a
k\kern-0.55556pta
k\kern-0.27779pta
\stopbuffer

Here the second kern is actually better, so one can wonder what the intended
kerning was. It might be the reason why the Dutch word \quote {kaas} comes
out so badly in a traditional setup.

\startlinecorrection
\scale[width=\textwidth]{\showglyphs\getbuffer}
\stoplinecorrection

It anyhow demonstrates that one should be careful with the order of definitions,
and adhere to the specification that the first kern wins. This is actually not
that different from the way \OPENTYPE\ works, although, especially in the early
days, one could find fonts that were set up with the assumption that he last one
wins or that kerns accumulate. Of course much can get unnoticed on a relatively
low resolution output medium.

\stopsubject

\startsubject[title=How to proceed?]

We now have to decide how to proceed in the perspective of \CONTEXT\ \LMTX, where
we follow a somewhat different approach to math spacing. As discussed elsewhere,
we already rid ourselves of italic correction as a means to anchor subscripts: we
went natural width with bottom kerns.

We also have, as shown at the beginning of this article, a dedicated mechanism
for math kerns that compensates for the lack of such feature in \OPENTYPE\ math.

Because an \OPENTYPE\ math font is a hybrid font that has upright, italic and
bold shapes in one font, kerning can be more extensive than in a traditional
assembly. Now, if only the fonts had a kern table, we could just use that one but
the fonts lack such a table. For Latin Modern we can add definitions taken from
the eight bit fonts but for instance Pagella and Bonum need their own. We use this
example:

\startbuffer
$df + dj + dl+ ab + if + ij$
\stopbuffer

\typebuffer

In untweaked Latin Modern we get this

\startlinecorrection
\scale[s=\ifmode{tugboat}1.5\else2\fi]{\strut\showglyphs\showfontkerns\switchtobodyfont[modern-nt]\inlinebuffer}
\stoplinecorrection

and when tweak we get:

\startlinecorrection
\scale[s=\ifmode{tugboat}1.5\else2\fi]{\strut\showglyphs\showfontkerns\switchtobodyfont[modern]\inlinebuffer}
\stoplinecorrection

The \TEX Gyre Pagella and Bonum math fonts give:

\startlinecorrection
\scale[s=\ifmode{tugboat}1.5\else2\fi]{\strut\showglyphs\showfontkerns\switchtobodyfont[pagella-nt]\inlinebuffer}
\stoplinecorrection

\startlinecorrection
\scale[s=\ifmode{tugboat}1.5\else2\fi]{\strut\showglyphs\showfontkerns\switchtobodyfont[bonum-nt]\inlinebuffer}
\stoplinecorrection

However, in the tweaked Bonum we get this:

\startlinecorrection
\scale[s=\ifmode{tugboat}1.5\else2\fi]{\strut\showglyphs\showfontkerns\switchtobodyfont[bonum]\inlinebuffer}
\stoplinecorrection

We consider it an interesting coincidence that we actually already had added math
kerns to the goodie file that do something similar to the type \type {df} and \type
{dj}. We also admit that the choice for what to kern is mostly determined by what
one encounters and we're sure that this is also the case the limited set of kerns
in math italic characters of Latin Modern: there is a reason why \type {dj} gets
a kern and \type {ij} doesn't.

But the main reason why we have these kerns in Bonum is not that we wanted to get
better kerning but that because we compensate the excessive \quote {sticking out
at the left} that we compensate has to be partially decompensated in some cases.
But we anyway had to translate the kerns into a fraction of the width (in our
case the width plus discarded italic correction) because that is what we use in
the goodie files where we need to avoid hard coded dimensions.

This is how Pagella comes out tweaked:

\startlinecorrection
\scale[s=\ifmode{tugboat}1.5\else2\fi]{\strut\showglyphs\showfontkerns\switchtobodyfont[pagella]\inlinebuffer}
\stoplinecorrection

So where can we get the kerns from? Playing a bit with values from the text font
alphabets give some cue where we should kern but the amounts can't be used: text
and math characters sit differently in their bounding boxes. Using side bearing
can help but is also not optimal, So, after half a day playing with this we gave
up on that. Instead we now use a feature that starts from the values that we find
in Computer Modern set, assuming that italic shapes are similar. These are the
most important candidates that get kerning with periods and commas:

\noindent\im{
    \setmathspacing \mathordinarycode \mathordinarycode \allmathstyles
        \ifmode{tugboat}.3\fi\thickmuskip
    C F H J K M N P S T U V W X Y Z
    f j r
    {\tf\Gamma \Pi \Upsilon \Psi}
    \delta \nu \sigma\tau
}

In the Computer Modern fonts the skew character is used for anchoring accents and
in \OPENTYPE\ that is done using the \typ {topanchor} property. So, we can
discard these. The same is true for the slash: either we set kerns for all
characters that matter, or we don't. For instance Latin Modern has a kern after a
slash before \im {A}, \im {M}, \im {N}, \im {Y}, \im {Z} and \im {\Delta}, but
not before e.g.\ \im {V} and \im{W}. Also, if we compare the values for italic
correction in Latin Modern Math with Computer Modern, we notice subtle
differences in the width|-|italic ratio: $0.367$ versus $0.381$ so we cannot be
too picky. For now have commented the values but they can easily become a
feature.

\stopsubject

\startsubject[title=(Optional) side note]

It is no secret that eight bit \PDFTEX\ beats the \UNICODE\ engines when it comes
to processing (\UTF8) input and rendering text. Among the \CONTEXT\ font loader
processing mode, {\em base} mode comes close to the traditional approach, while
{\em node} mode is slower because all is done in \LUA. Node modes is needed when
we go beyond simple pairwise kerns and simple (non contextual) ligatures.

In \LUA\ we usually store data in a hashed table and that is also true for kerns
in fonts that get loaded. When in node mode, we keep all at the \LUA\ end but in
base mode we pass the kerns to \TEX. On previous pages we indicated that
properties like kerns get stored as a sequential lookup. Because \LUA\ randomizes
the hash using a different seed each run. The order in which kerns end up in
\TEX\ is also random. The way we store information today (like in these \LUA\
hashes) was not an option in the early days of computing which makes \TEX\ even
more of a masterpiece of efficiency.

If we use \type {texnansi} encoding we have kern pairs between \type {va} and
\type {vo} and in Latin Modern the first one is the first entry and the second
one the sixteenth. So does that have some effect on runtime? A somewhat over the
top test using these combinations shows that in \PDFTEX\ that stored the kerns in
original order indeed the \type {va} kerns takes less time to resolve. The
further down the list the kern is, the more time it will cost. This is comparable
with for instance access to path points in \METAPOST: the assumption is that
paths are short when this is needed as looping over a 5000 point path involved
quite a bit traversing. This has never been an issue for \TEX\ because fonts are
limited to 256 glyphs which means that of the many accented variants of vowels
only a few end up in a specific encoding. Of course the more random order in the
\LUA\ hash variant makes for a constant look up time in node mode and a more
averaging time in base mode. A side effect is that one can add one own additional
kerns in node mode.

Isn't it amazing to what observations a simple question on SE bring up? There is
one more thing worth mentioning. The lack of these kerns went unnoticed for quite
a while. This can indicate a that either users don't care, or just don't consider
it needed, or have adapted by adding manuals kerns. It definitely has not
resulted in it being seen as a necessity in \OPENTYPE\ math. It is kind of
interesting that a power user was told about this he remarked that checking the
period of comma after a capital in math mode was part of some quality assurance
process because it could indicate that authors messed the spacing (in \MATHML\
and of \ASCIIMATH\ input). So, adding these kerns already paid back. It also shows
that \TEX\ can adapt very well to whatever circumstances it has to deal with.

\stopsubject

\stopchapter

\stopcomponent

% % engine=pdftex
%
% \starttext
%
% % \definefontfeature[myfeature][default][mode=base]
% % \definedfont[Serif*myfeature]
%
% \language0
%
% \hfuzz \maxdimen
% \hbadness 10000
% \newcount\foo
% \foo=20000
% \loop
% \setbox0\vbox{
% vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava
% vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava
% vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava
% vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava
% vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava
% vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava
% vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava
% vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava vavavava
%
% % vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo
% % vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo
% % vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo
% % vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo
% % vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo
% % vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo
% % vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo
% % vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo vovovovo
%         \par
%     }
% \advance \foo -1
% \ifnum\foo>0 \repeat
% done
%
% \hsize 1mm vovovovo
%
% \hsize 1mm vavavava
%
% \stoptext
%
% va 3.974
% vo 2.732
