% language=us runpath=texruns:manuals/lowlevel

\environment lowlevel-style

\startdocument
  [title=characters,
   color=middlered]

\startsectionlevel[title=Introduction]

This explanation is part of the low level manuals because in practice users will
not have to deal with these matters in \MKIV\ and even less in \LMTX. You can
skip to the last section for commands.

\stopsectionlevel

\startsectionlevel[title=History]

If we travel back in time to when \TEX\ was written we end up in eight bit
character universe. In fact, the first versions assumed seven bits, but for
comfortable use with languages other than English that was not sufficient.
Support for eight bits permits the usage of so called code pages as supported by
operating systems. Although \ASCII\ input became kind of the standard soon
afterwards, the engine can be set up for different encodings. This is not only
true for \TEX, but for many of its companions, like \METAFONT\ and therefore
\METAPOST. \footnote {This remapping to an internal representation (e.g. ebcdic)
is not present in \LUATEX\ where we assume \UTF8 to be the input encoding. The
\METAPOST\ library that comes with \LUATEX\ still has that code but in
\LUAMETATEX\ it's gone. There one can set up the machinery to be \UTF8 aware
too.}

Core components of a \TEX\ engine are hyphenation of words, applying
inter|-|character kerns and build ligatures. In traditional \TEX\ engines those
processes are interwoven into the par builder but in \LUATEX\ these are separate
stages. The original approach is the reason that there is a relation between the
input encoding and the font encoding: the character in the input is the slot used
in a reference to a glyph. When producing the final result (e.g.\ \PDF) there can
also be a mapping to an index in a font resource.

\starttyping
input A [tex ->] font slot A [backend ->] glyph index A
\stoptyping

The mapping that \TEX\ does is normally one|-|to|-|one but an input character can
undergo some transformation. For instance a character beyond \ASCII\ 126 can be
made active and expand to some character number that then becomes the font slot.
So, it is the expansion (or meaning) of a character that end up as numeric
reference in the glyph node. Virtual fonts can introduce yet another remapping
but that's only visible in the backend.

Actually, in \LUATEX\ the same happens but in practice there is no need to go
active because (at least in \CONTEXT) we assume a \UNICODE\ path so there the
font slot is the \UNICODE\ got from the \UTF8 input.

In the eight bit universe macro packages (have to) provide all kind of means to
deal with (in the perspective of English) special characters. For instance, \type
{\"a} would put a diaeresis on top of the a or even better, refer to a character
in the encoding that the chosen font provides. Because there are some limitations
of what can go in an eight bit font, and because in different countries the used
\TEX\ fonts evolved kind of independent, we ended up with quite some different
variants of fonts. It was only with the Latin Modern project that this became
better. Interesting is that when we consider the fact that such a font has often
also hardly used symbols (like registered or copyright) coming up with an
encoding vector that covers most (latin based) European languages (scripts) is
not impossible \footnote {And indeed in the Latin Modern project we came up with
one but it was already to late for it to become popular.} Special symbols could
simply go into a dedicated font, also because these are always accessed via a
macro so who cares about the input. It never happened.

Keep in mind that when \UTF8 is used with eight bit engines, \CONTEXT\ will
convert sequences of characters into a slot in a font (depending on the font
encoding used which itself depends on the coverage needed). For this every first
(possible) byte of a multibyte \UTF\ sequence is an active character, which is no
big deal because these are outside the \ASCII\ range. Normal \ASCII\ characters
are single byte \UTF\ sequences and fall through without treatment.

Anyway, in \CONTEXT\ \MKII\ we dealt with this by supporting mixed encodings,
depending on the (local) language, referencing the relevant font. It permits
users to enter the text in their preferred input encoding and also get the words
properly hyphenated. But we can leave these \MKII\ details behind.

\stopsectionlevel

\startsectionlevel[title=The heritage]

In \MKIV\ we got rid of input and font encodings, although one can still load
files in a specific code page. \footnote {I'm not sure if users ever depend on an
input encoding different from \UTF8.} We also kept the means to enter special
characters, if only because text editors seldom support(ed) a wide range of
visual editing of those. This is why we still have

\starttyping[option=TEX]
\"u \^a \v{s} \AE \ij \eacute \oslash
\stoptyping

and many more. The ones with one character names are rather common in the \TEX\
community but it is definitely a weird mix of symbols. The next two are kind of
outdated: in these days you delegate that to the font handler, where turning them
into \quote {single} character references depends on what the font offers, how it
is set up with respect to (for instance) ligatures, and even might depend on
language or script.

The ones with the long names partly are tradition, but as we have a lot of them,
in \MKII\ they actually serve a purpose. These verbose names are used in the so
called encoding vectors and are part of the \UTF\ expansion vectors. They are
also used in labels so that we have a good indication if what goes in there:
remember that in those times editors often didn't show characters, unless the
font for display had them, or the operating system somehow provided them from
another font. These verbose names are used for latin, greek and cyrillic and for
some other scripts and symbols. They take up quite a bit of hash space and the
format file. \footnote {In \MKII\ we have an abstract front|-|end with respect to
encodings and also an abstract backend with respect to supported drivers but both
approaches no longer make sense today.}

\stopsectionlevel

\startsectionlevel[title=The \LMTX\ approach]

In the process of tagging all (public) macros in \LMTX\ (which happened in
2020|-|2021) I wondered if we should keep these one character macros, the
references to special characters and the verbose ones. When asked on the mailing
list it became clear that users still expect the short ones to be present, often
just because old \BIBTEX\ files are used that might need them. However, in \MKIV\
and \LMTX\ we load \BIBTEX\ files in a way that turn these special character
references into proper \UTF8 input so it makes a weak argument. Anyway, although
they could go, for now we keep them because users expect them. However, in \LMTX\
the implementation is somewhat different now, a bit more efficient in terms of
hash and memory, potentially a bit less efficient in runtime, but no one will
notice that.

A new command has been introduced, the very short \type {\chr}.

\startbuffer
\chr {à} \chr {á} \chr {ä}
\chr {`a} \chr {'a} \chr {"a}
\chr {a acute} \chr {a grave} \chr {a umlaut}
\chr {aacute}  \chr {agrave}  \chr {aumlaut}
\stopbuffer

\typebuffer[option=TEX]

In the first line the composed character using two characters, a base and a so
called mark. Actually, one doesn't have to use \type {\chr} in that case because
\CONTEXT\ does already collapse characters for you. The second line looks like
the shortcuts \type {\`}, \type {\'} and \type {\"}. The third and fourth lines
could eventually replace the more symbolic long names, if we feel the need. Watch
out: in \UNICODE\ input the marks come {\em after}.

\startlines \getbuffer \stoplines

Currently the repertoire is somewhat limited but it can be easily be extended. It
all depends on user needs (doing Greek and Cyrillic for instance). The reason why
we actually save code deep down is that the helpers for this have always been
there. \footnote {So if needed I can port this approach back to \MKIV, but for
now we keep it as is because we then have a reference.}

The \type {\"} commands are now just aliases to more verbose and less hackery
looking macros:

\starttabulate[|||||]
    \NC \type {\withgrave}        \NC \withgrave       {a} \NC \type {\`} \NC \`{a} \NC \NR
    \NC \type {\withacute}        \NC \withacute       {a} \NC \type {\'} \NC \'{a} \NC \NR
    \NC \type {\withcircumflex}   \NC \withcircumflex  {a} \NC \type {\^} \NC \^{a} \NC \NR
    \NC \type {\withtilde}        \NC \withtilde       {a} \NC \type {\~} \NC \~{a} \NC \NR
    \NC \type {\withmacron}       \NC \withmacron      {a} \NC \type {\=} \NC \={a} \NC \NR
    \NC \type {\withbreve}        \NC \withbreve       {e} \NC \type {\u} \NC \u{e} \NC \NR
    \NC \type {\withdotaccent}    \NC \withdot         {c} \NC \type {\.} \NC \.{c} \NC \NR
    \NC \type {\withdiaeresis}    \NC \withdieresis    {e} \NC \type {\"} \NC \"{e} \NC \NR
    \NC \type {\withring}         \NC \withring        {u} \NC \type {\r} \NC \r{u} \NC \NR
    \NC \type {\withhungarumlaut} \NC \withhungarumlaut{u} \NC \type {\H} \NC \H{u} \NC \NR
    \NC \type {\withcaron}        \NC \withcaron       {e} \NC \type {\v} \NC \v{e} \NC \NR
    \NC \type {\withcedilla}      \NC \withcedilla     {e} \NC \type {\c} \NC \c{e} \NC \NR
    \NC \type {\withogonek}       \NC \withogonek      {e} \NC \type {\k} \NC \k{e} \NC \NR
\stoptabulate

Not all fonts have these special characters. Most natural is to have them
available as precomposed single glyphs, but it can be that they are just two
shapes with the marks anchored to the base. It can even be that the font somehow
overlays them, assuming (roughly) equal widths. The \type {compose} font feature
in \CONTEXT\ normally can handle most well.

An occasional ugly rendering doesn't matter that much: better have something than
nothing. But when it's the main language (script) that needs them you'd better
look for a font that handles them. When in doubt, in \CONTEXT\ you can enable
checking:

\starttabulate[|l|l|]
    \BC command                           \BC equivalent to \NC \NR
    \NC \type {\checkmissingcharacters}   \NC \type{\enabletrackers[fonts.missing]} \NC \NR
    \NC \type {\removemissingcharacters}  \NC \type{\enabletrackers[fonts.missing=remove]} \NC \NR
    \NC \type {\replacemissingcharacters} \NC \type{\enabletrackers[fonts.missing=replace]} \NC \NR
    \NC \type {\handlemissingcharacters}  \NC \type{\enabletrackers[fonts.missing={decompose,replace}]} \NC \NR
\stoptabulate

The decompose variant will try to turn a composed character into its components
so that at least you get something. If that fails it will inject a replacement
symbol that stands out so that you can check it. The console also mentions
missing glyphs. You don't need to enable this by default \footnote {There is some
overhead involved here.} but you might occasionally do it when you use a font for
the first time.

In \LMTX\ this mechanism has been upgraded so that replacements follow the shape
and are actually real characters. The decomposition has not yet been ported back
to \MKIV.

The full list of commands can be queried when a tracing module is loaded:

\startbuffer
\usemodule[s][characters-combinations]

\showcharactercombinations
\stopbuffer

\typebuffer

We get this list:

\getbuffer

Some combinations are special for \CONTEXT\ because \UNICODE\ doesn't specify
decomposition for all composed characters.

\stopsectionlevel

\startsectionlevel[title={spaces}]

The engine has no real concept of a space. When the input has one it is turned
into a glue, likely with some stretch and shrink. When \typ {\nospaces} is set to
one, no glue will be inserted. A value of two will inject a zero width glue. When
set to three a glyph will be inserted with the character code set by \typ
{\spacechar}.

\startbuffer
\nospaces\plusthree
\spacechar\underscoreasciicode
\hccode\underscoreasciicode\underscoreasciicode
Where are the spaces?
\stopbuffer

\typebuffer

The \type {\hccode} tells the machinery that the underscore is a valid word
separator (think compound words).

\start \getbuffer \stop

\stopsectionlevel

\startsectionlevel[title=Casing]

We started by mentioning that \TEX\ was primarily made and used for typesetting
English although quickly other Latin scripts were supported. As long as a
language uses a script that fits nicely in a few hundred characters we're okay.
Of course mixing scripts and languages is more demanding, but Latin, Cyrillic and
Greek were no problem.

A characteristic of western languages and these scripts is that there is a mix of
upper and lowercase characters. Now, in most cases the input has the right usage
of casing so there is little for \TEX\ to worry about. However, in the
traditional approach every character has a so called \type {\lccode} and \type
{\uccode}. These are used to map from upper- to lowercase or reverse and the
lowercase code, when set, is also a signal that controls hyphenation. In \LUATEX\
and \LUAMETATEX\ that role is taken over by the \type {\hccode}.

Good old \TEX\ has two primitives that deal with case swapping.

\startbuffer
\lowercase{Don't get fooled!}
\uppercase{Don't get fooled!}
\stopbuffer

\typebuffer[option=TEX]

This does what one expects:

\startlines
\getbuffer
\stoplines

So, what do you expect here?

\startbuffer
\edef\fooled{\lowercase{Don't get fooled!}}\meaning\fooled
\edef\fooled{\uppercase{Don't get fooled!}}\meaning\fooled
\stopbuffer

\typebuffer[option=TEX]

\startlines
\getbuffer
\stoplines

And here?

\startbuffer
\lowercase{\edef\fooled{Don't get fooled!}}\meaning\fooled
\uppercase{\edef\fooled{Don't get fooled!}}\meaning\fooled
\stopbuffer

\typebuffer[option=TEX]

\startlines
\getbuffer
\stoplines

These primitives are non-expandable but act on a token list when interpretation
takes place.

\startbuffer
\begingroup
    \lowercase{\edefcsname fooled\endcsname{Guess}}
    \meaningfull\fooled
    \meaningfull\FOOLED
\endgroup
\begingroup
    \uppercase{\edefcsname fooled\endcsname{Guess}}
    \meaningfull\fooled
    \meaningfull\FOOLED
\endgroup
\stopbuffer

\typebuffer[option=TEX]

\startlines
\getbuffer
\stoplines

It will be clear that using these primitives in a situation where you want
expansion you really need to make sure that in getting there no side effects take
place. Instead of these primitives, that we keep around for compatibility
reasons, you can better use the converters that \CONTEXT\ provides. Even these
are probably of little used when typesetting is your main focus, because when
case mapping of various kind is needed there are high level mechanism for just
that.

\startbuffer
\edef\fooled{\utflower{Don't get fooled!}}\meaning\fooled
\edef\fooled{\utfupper{Don't get fooled!}}\meaning\fooled
\stopbuffer

\typebuffer[option=TEX]

\startlines
\getbuffer
\stoplines

It will be clear that in an \UNICODE\ environment there is more than an English
alphabet involved. The primitive case mapping as well as the \CONTEXT\ variants
use of course \UNICODE\ specified mapping but the last two macros are just
wrappers around a lot more character related properties.

In \MKII\ we supported various encodings and each came with a case mapping. We
gradually moved to \UTF8 and for instance hyphenation patterns came in that
format so that they could easily be shared. \footnote {This was reason for Arthur
and Mojca to start the hyphenation project that resulted in \UTF\ patterns being
that standard now. It's a nice example of prototyping and exploring in \CONTEXT\
and applying it to a wider audience.}

When we started in 2005 with a follow up a major change in approach to dealing
with characters between \MKII\ and \MKIV\ was that we could get rid of, if really
needed, remapping at the \TEX\ end. We delegate much to \LUA\ which also has the
advantage that we operate on the text as intended, possibly resulting from
generated input. This means that right from the start we had support for
\UNICODE, although of course that standard also evolved over time. Some was
driven by specific user need: for the oriental \TEX\ project we needed
bidirectional interpretation, for some scripts we needed (de)composition, we
needed sorting, tracing, special purpose conversions, etc. \footnote {If
manipulation of character properties in the \TEX\ language is your hobby, maybe
\MKII\ is a better choice as it pretty much belongs to that period.}

Most of this is rather stable and as it has been present from the start and it
hasn't been touched much, although we do keep up with \UNICODE. But one has to
keep in mind that much is about control and user input and demand so we can
deviate from what is considered \quote {standard}: we target typesetting, not
programming and text manipulation! We keep most (not all) information in the
\type {char-def.lua} file and actually although it gets adapted to \UNICODE\
updates (for which of course we use a script) it hasn't changed much, apart from
an update with respect to math classes and such when we upgraded the math engine.
It can be an interesting read.

That said: if users want some more explanations or low level examples, just let
us know and we will add them here (or to the low level manual that makes sense).

% amcode     active math
% hccode     hyphenation control
% hmcode     math discretionary
% lccode     lower case
% uccode     upper case
% sfcode     space factor
% cccode     classification
% catcode    command category
% delcode    math delimiter
% mathcode   math character

\stopsectionlevel

\startsectionlevel[title=Specials]

Traditionally \TEX\ has a few specially interpreted characters. Most noticeably
the backslash starts a control sequence. Any programming languages has some
escape and this the one that \TEX\ uses. Of course any character can be an escape
character.

The dollar is well known for bounding inline math. A pair of dollars wraps
display math. In \CONTEXT\ you can use the dollars but double ones act as single ones
and trigger inline display style math. In practice one will use \type {\im {...}} and \type {\dm {...}}
and for display math \typ {\startformula} and \typ {\stopformula}.

The ampersand is, unlike in other macro packages, just that and not a table cell
separator. The circumflex character is also just that. However, in math upto four
are used for super scripts (pre, post and index) while the underscore(s) handle
the subscripts. The single quote is in math signaling a prime (multiple primes
are supported).

The end of line character(s) are in fact spaces unless configured to be obeyed,
what can be the case when you use \typ {\startlines} and \typ {\stoplines} or
display verbatim.

The tilde is normally active and equivalent to a \type {\nbsp} character. In
\CONTEXT\ it as an \type {\amcode} set so that it will never expand in for
instance an \type {\edef} like situation. The hash mark is used to signal a
parameter in a macro body and is followed by a (hexadecimal) digit. They get
duplicated in nested macro definitions. The percent sign starts comment that then
runs till the end of the line.

Hyphens are also special, as they can become a discretionary (hyphenation point),
or when entered double or triple can become an en- of emdash, either or not with
similar hyphenation behavior (think compound words).

Quotation and explanation marks and some punctuation like periods, commas, colons
and semicolons can get different spacing depending on the \type {\sfcode}. Here
\quote {french spacing} is the buzzword.

In \CONTEXT\ the vertical bar has always been an active character used for
compound word marks and other word bounding characters. They come in pairs where
the second one is a delimiter of a possible argument. In math it behaves as
normal bar, although bars in math are sort special too. In fact, all characters
in math get a treatment depending on what they represent. More about that
can be read in the math manual.

The various spacing characters in \UNICODE\ are interpreted but normally can't be
seen in an editor. Instead one can use commands. {\em todo: list them here}

Of course all can be changed to ones need but be careful and aware of interferences,
especially when loading code that assumes otherwise.

\stopsectionlevel

\stopdocument

% on an old machine, so consider them just relative measures
%
% mkiv  lmtx
%
% 0.012 0.009 % faster core code
% 0.028 0.036 % different io code path
% 0.055 0.043 % different io code path / faster core code
% 0.156 0.129 % more efficient resolving
% 0.153 0.119 % more efficient resolving
%
% \ifdefined\withdieresis\else\let\withdieresis\"\fi % for mkiv
%
% \setbox0\hpack{\testfeatureonce{100000}{ü}}                \par \elapsedtime \par % direct
% \setbox0\hpack{\testfeatureonce{100000}{ü}}                \par \elapsedtime \par % composed (input)
% \setbox0\hpack{\testfeatureonce{100000}{u{}̈}}              \par \elapsedtime \par % overlay
% \setbox0\hpack{\testfeatureonce{100000}{\withdieresis{u}}} \par \elapsedtime \par % official also \"u
% \setbox0\hpack{\testfeatureonce{100000}{\" u}}             \par \elapsedtime \par % alias of previous

