Approaches to robust realtime text rendering in threejs (and WebGL in general)

[bunnybones1]

Hello,

I'm part of a small team (Corban Brook and myself) working on a module for robust just-in-time text rendering in threejs. We're at an experimental stage of development. The module source is here https://github.com/horizon-games/three-text-renderer
three-text-renderer uses the harfbuzz WASM build.

Another similar module is part of the troika framework. This module is published on npm and has a pretty big user-base. The source is here: https://github.com/protectwise/troika/tree/master/packages/troika-three-text
troika-three-text uses Typr.js, but it doesn't use the harfbuzz-wasm initialization (due to wasm size concerns).

I think it would be valuable to discuss technical goals and design choices to ultimately create a module that renders text quickly and with as small of a footprint as possible.

My personal involvement centers around the graphics pipeline that turns glyph shapes into SDF or MSDF atlases via some JS and GLSL. I've seen the SDF algorithm implemented purely in js/canvas but I strongly feel that there are big performance advantages to doing the work on the GPU, and not to mention, the result need to reside on the GPU anyways. The SDF/MSDF algorithms therefore need to be adapted to GLSL, and we have done some promising work in this area.

Henrik Nyman has actually demonstrated an OpenGL solution to this, and when we chatted a couple months ago, it sounded like OpenGL 2 was all that was needed (and hopefully no fancy extensions). https://github.com/nyyManni/msdfgl

One more thing I want to mention is why the focus is on threejs. This module's design is imagined to work in service of an engine like threejs or babylonjs (or any others not mentioned). This is for two reasons:

1. it's just meant to fill a missing piece in otherwise fantastic libraries
2. it would be ill-advised to work directly with WebGL, since the WebGL state is usually managed by the graphics library for performance reasons.

Though threejs is my engine of choice because of my experience with it, anything discussed applies to any library or framework that manages WebGL.

Anyways, my hope is that next time we need to render text in WebGL, whatever the language, whatever the font, we'll be able to, precise, beautiful and fast.

I will try to consolidate the design choices made in three-text-renderer into a UML diagram soon. I'm particularly interested in when/how the work can be best done for the smoothest end-user experience (avoid jank, webworkers are great except on iOS, expect unexpected text, etc.)

[bunnybones1]

Here is a diagram of the basic architecture behind https://github.com/horizon-games/three-text-renderer.

https://docs.google.com/drawings/d/1cicoJ3UzIERzDEakrRgg2WBn3AOQZ2s_NYoq6U2ssTw/edit (in case you want to modify a copy)
The design is based on running on a single thread (no workers) and a single WebGL context managed by the main 3D engine.
Currently this approach would run smooth but text that is not prepared "behind the scenes" would technically "animate" by characters showing up as they're processed.

Rendering glyphs to atlas could mean any method, including SDF and MSDF. These are a heavy topic themselves, which I look forward to expanding.

[lojjic]

Nice diagram! Troika-three-text follows a pretty similar pattern, with the main difference being that the font parsing, layout solving, and glyph generation is all done async in a web worker.

One small correction to the original post: it's not currently true that troika-three-text uses Harfbuzz -- it does use Typr.js, but an older JS-only version prior to the Harfbuzz integration which lacks some advanced shaping features. While I'd love to get those features, I'm not yet comfortable including a Harfbuzz WASM build due to its large size and the resulting impact on downstream users.

Anything that can be done to reduce the WASM file size would be helpful in making me reconsider.

[lojjic]

Here's what each currently contributes to my bundle:

• Opentype.js: 103k minified, 31.5k gzipped (I've customized a build to exclude a bunch of unused code for stuff like hinting)
• Typr.js: adds 42k minified, 12.1k gzipped

The rest of the bundle is ~46k minified, ~18k gzipped (maybe less with treeshaking)

My own personal bias is toward enabling folks building websites who want to sprinkle in some 3D (ala many react-three-fiber users), enabling lightweight WebXR experiences, etc. -- while being conscious of load times. For many of these sites an extra 180k is a 50-percent-plus increase in bundle size.

I think both areas of focus are equally valid and they obviously drive us to different decisions. Honestly I think it's pretty cool to have both approaches in the community. 😃

[lojjic]

Come to think of it, I wouldn't necessarily have to update Typr.js to make use of Harfbuzz-wasm. I could add a parallel code path that loads/uses Harfbuzz if requested and falls back to the old Typr (or Opentype even) for the rest. I'll definitely look into that.

[bunnybones1]

Makes sense. Not to mention, the load/initialization of the wasm itself is definitely an extra complication.
My only concern is that a typesetting library that doesn't provide maximum correctness for all languages could alienate users. This discrimination is not intentional ofcourse; some languages require only simpler algorithms to typeset, and simpler solutions result in faster/smaller libraries.

I found this issue on Opentype.js opentypejs/opentype.js#372 that shows me that this is something they'll eventually have to support or atleast address.

This has been an interesting topic for us at work too, for over a year. As long as our WebGL work uses latin typesetting, we can use a simpler typesetting solution, but language support is ultimately a marketing (or a philosophical) decision.

Having said that, Harfbuzz or something similar, opens up some really interesting possibilities, like having truly world-wide users seeing their native language correctly rendered, interacting with each other without a shared language, or community-sourcing our language files (knowing their text editor and our experience will look the same).

On a more philosophical note: the inclusion of as much of the trueness and variety of human typesetting seems like a noble use of the vast quantities of bytes and cycles afforded to us by today's computers.

Anyways, I don't meant to rant, but this stuff gets me really excited. Typesetting really matters. Typesetting in WebGL? Maybe not quite as essential, but I wonder who out there is just not making/enjoying WebGL content because their language doesn't render correctly.

Anyways, I totally agree that a variety of solutions in the community is great.

[behdad]

I'm not yet comfortable including a Harfbuzz WASM build due to its large size and the resulting impact on downstream users.

Anything that can be done to reduce the WASM file size would be helpful in making me reconsider.

What size are you uncomfortable about? What would be an acceptable size?

[lojjic]

What size are you uncomfortable about? What would be an acceptable size?

Oh, that's a tough one because "acceptable" isn't really up to me, it's up to each of the downstream users of my tool. As I mentioned above many or most of these are folks serving up websites with a bit of extra WebGL fanciness thrown in, where load time matters. The entirety of three.js, even without treeshaking, tops out at 142K gzipped. So I feel like it's a pretty big ask to force an extra 180K on them just for some text rendering.

Anyway, like I said I think I have a good path forward where I can keep my bundle as light as possible by default, and let users opt in to more advanced text layout by providing a hb.wasm url, where they're fully aware of the size tradeoff. This makes the wasm's size much less of a pressing concern for me.