Utext: Rich Unicode Documents

The primary purpose of Utext is to generate the coolest output, so the main challenge is just achieving any of the suggestions above. But as an optional feature, mostly independent of the foregoing, I’ll discuss a special-purpose Utext format.

Fancy Unicode tricks do not, in principle, require a new document format beyond the classic .txt file. One can simply store the source for a Utext in an ordinary Markdown text file with a .md extension, and ‘compile’ to a .txt file.

But it is convenient to define a new format (especially if Utext winds up adding extensions or syntax which make no sense to a Markdown compiler), and since we are using strictly UTF-8, we can do something interesting and combine a fancy Unicode text file with a text file into a single file format.

Each line of a compiled utext is prefixed with an invisible special character (eg. ASCII’s Start of Text control character) which indicates it is a compiled line. A compiled line is human-readable UTF-8 text line. (120 characters would be a good maximum width for all devices; but a Utext could be rendered at multiple widths so as to support other devices, like smartphones which usually a comfortable width of ~40 characters.) But a compiled line is not meant to be human-editable, because it uses all available Unicode tricks for rich formatting, like ASCII art. For example, a source line may use italics or bolds, which normally cannot be displayed in pure UTF-8 text without using terminal control codes or relying on markup conventions like HTML/Markdown, but the compiled version abuses Unicode to map each italic letter onto the math symbols—thereby providing ‘italics’ and ‘bolds’ in pure UTF-8 text rendering in any text viewer.

Since both compiled & source lines are UTF-8 text by definition, they can be stored in the same file. Utexts can be compiled ‘in-place’ and a text editor could easily provide a ‘WYSIWYG’ view of a utext being edited, and this interoperates with version-control systems. A standard utext provides the full compiled version first, followed by the source. This is most convenient for the reader, who can read a document in its pretty compiled form, and then skip the source. (It also helps solve accessibility issues: if a 40-column smartphone rendition is unacceptable, then one can provide the source with some HTML like a <pre style="white-space: pre-wrap"></pre> wrapper; ASCII/ANSI art is inherently screen-reader hostile, but a screen-reader could simply skip compiled lines for the human-readable source text, and so a Utext would be far more accessible than a hand-written equivalent, which lacks any formal structure or documentation of intent.)

And because compiled vs source lines can be matched by a per-line regexp, a literate utext can be viewed easily as an incremental streamed—simply grep for the compiled lines. This means one can do command line stunts like create an IRC client/server with nice formatting simply by telnet: a client opens a telnet connection and pipes in the source text they type, sending it to the server; they open a second telnet connection, which downloads an (incrementally streaming) literate utext, filtering out the source lines with grep; and the server simply pipes each client line of text through the Utext compiler and appends to the file that each client is indefinitely downloading. (This can be done to some degree with regular IRC, but not with formatting.)

When web developers meet to boast about their web page speeds, the web pages in question inevitably devolve into fancy text files—there is no way to be faster than to ask the browser to do as little as possible, after all, given that ultimately a web page is fundamentally a document, and everything else is tacked on afterwards. Considering how much Utext can do, one might ask, can you make a Utext web page? You can of course simply put a bunch of utext files on a web server and serve them as a UTF-8 text file, no problem. But can we do better? Can we make a ‘HTML Utext’ where, say, a programmer could seriously contemplate making a personal website out of which people might actually use? (Nothing on the level of a Gwern.net would be possible, but could we hope to beat, let’s say, Dan Luu’s website?)

A Utext is just a text. What is the essence of a hyper text? What separates a hypertext from a regular text like a printed book or a handwritten papyrus scroll?

The missing link. I would say that the essence of a hypertext is not supporting some typographical feature like italics, and nor is it having citations or references of some sort like page numbers (or even ISBN/ DOI IDs); if that was enough, then we’ve had “hypertext” for centuries or millennia. Nor is it the concept of a “link” referring in some way to where to physically get another work—even at the Library of Alexandria, there was at least one well-developed indexing system you could refer to a scroll by its genre/type and then the first letter of the author, and possibly the physical location in a bin; and an ISBN often implies that the legal deposit Library of Congress (which controls ISBNs) would have a copy in Washington DC. And nor is it containing Internet URLs: if someone printed out a web page and it included a list of the raw URLs appended at the end, you wouldn’t regard it as all that different from a regular book with its bibliography of references. Likewise, a utext which contained a bunch of URLs in it, as text, however informally marked up with < > or ( ) or whatever, is still not a hypertext.

Active, not passive. Rather, the essence of a hypertext is that the links are active, and will do something: usually, load the referenced document and display it. (Although not always: it could instead download the document, open it in a new ‘tab’, or add it to a sidebar as a kind of queue/bookmarks like Symbolics Concordia, or any number of things.)

So, what Utext is missing is some way to provide active links, true hyperlinks. That’s the only important thing it is missing compared to HTML. Everything else, from multimedia to JavaScript, is optional—plenty of hypertext systems have lacked them, and were usable.

There is no way to implement clickable active links in pure Unicode, as the viewing software must in some way execute code. But this hints that we only need one small, tiny feature from HTML: its <a> link. So… why not create utexts which are utexts except where they switch to HTML and use a <a> link?

As previously mentioned, we can display UTF-8 text literally in a browser using the HTML <pre> tag, which stands for ‘pre-formatted’—exactly the sort of text we want to display here.

So, a minimal HTML Utext would be to define a normal HTML document, except that all features or tags not strictly required to be a valid HTML document are forbidden, and all that is allowed in the <body> of the document is <pre> and <a> tags. That is, the <pre> tags are used to display normal Utext content, and whenever a hyperlink is required, they are closed with </pre>, then an <a> link is defined, which can only contain 1 <pre> with Utext contents (which will be a clickable hyperlink).

<html>
    <body>
<pre>Normal Utext content here, but then one wants to link to
some URL like</pre> <a href="https://example.com/"><pre>Example.com</pre></a> <pre>to demonstrate how
linking works in HTML Utext.</pre>
    </body>
</html>

Thus, the author can write in the normal Markdown-esque Utext source language, and freely hyperlink away as usual (eg. generating a hyperlinked table of contents where each ‘section’ is a different <pre> ID), and it will compile down to a HTML document which nevertheless looks exactly like a utext—except for the blue underlined links throughout.

This is the minimum viable ‘HTML Utext’.

(And also, to be good web citizens, we should permit the <head> to contain <meta> and <link> fields.)

Because we can use CSS, we can pull in web-fonts (such as 8-bit fonts) to ensure much more exact rendering of the utexts; and we can indulge in retro hacker esthetics like setting green-on-black. (A key feature, we can all agree.)

However, we are unable to indulge the temptation to go beyond global stylization (like setting background/foreground), because the <pre> tags are opaque and block any further HTML/CSS creep. Should a user find themselves fighting that, and trying to crack open the black-boxes to do colors or syntax-highlighting or even (gasp) make it interactive via JS, or do something where the utext would break fundamentally if rendered to a terminal—then it’s time for them to stop using Utext and start using real HTML.

The one immediate catch with this simple approach is that <pre> tags are ‘block’ elements and <a> are ‘inline’, and so we cannot quite intersperse them as freely as that, and get the expected layout. The above example would actually render each of the 3 elements as their own paragraph-like block, more like this:

Normal Utext content here, but then one wants to link to
some URL like
Example.com
to demonstrate how
linking works in HTML Utext.

This might be viable as a Gopher/Gemini-like highlighting of links, but I like inline links, and we can make it render as expected by adding 1 line of CSS:

<style>pre { display: inline; }</style>

If we ever want to render it like the former version, then we simply write that explicitly.

A second flaw with this is that in 2024, it is no longer possible to pretend mobile users don’t exist. Mobile is a requirement; anything which breaks on mobile is too esoteric to take seriously.

We are able to support mobile by the brute-force approach of rendering a new utext for each desired screen-width. This allows a tradeoff between responsiveness and pre-computation/filesize: the more widths we generate a tailored utext for, the better the experience, but the longer the compilation & larger the filesize.

But can we deliver the different versions as appropriate? Yes, if we are willing to spend a little more CSS, or complicate the server itself:

• <pre>/<a> tags support the usual HTML things like classes & IDs; so we can support mobile by rendering our utexts for multiple widths, assigning a class/ID to each one, and then add a media-query CSS declaration to switch at various screen widths.

• We could also have separate websites/domains for mobile vs desktop, or we could set our HTTP server to do user-agent sniffing and try to detect what the device is and send it an appropriately-compiled utext. (Direct content negotiation for screen width is now unsupported.)

  This saves bandwidth and simplifies the delivered file, but is risky (user-agent sniffing is notoriously error-prone) and complicated and can be hard to debug. So this would be my least-preferred option.

So a complete HTML Utext might look like this:

<!DOCTYPE html>
<html lang="en-us">
  <head>
    <meta charset="utf-8">
    ...

    <style>pre { display: inline; }
    @media all and (max-width: 649px) {
        .wide { display: none; }
        }
    @media all and (min-width: 650px) {
        .narrow { display: none; }
        }
    </style>
  </head>

  <body>
<pre class="narrow">Normal Utext content here, but then one wants to link to
some URL like </pre><a class="narrow" href="https://example.com/"><pre>Example.com</pre></a><pre class="narrow"> to demonstrate how
linking works in HTML Utext.</pre>

<pre class="wide">Normal Utext content here, but then one wants to link to some URL like</pre> <a class="wide" href="https://example.com/"><pre>Example.com</pre></a>
<pre class="wide">to demonstrate how linking works in HTML Utext.</pre>
  </body>
</html>

This elegantly gives us a ‘web page’ which is: still lightning-fast & small, supports the full range of Unicode display tricks in Utext, has clickable links, will display well on the devices that make up 99% of the market, is standards-compliant, has metadata hidden from the reader, can be readily turned into pure plain Utext with the usual HTML → text tools (eg. elinks -dump)…

All in all, HTML Utext is something I can actually imagine using for a personal homepage or project without the esoteric document format being the real reason for it.