Unicode text converter

Unicode gives every character a number, its code point. The styled letters live in the Mathematical Alphanumeric Symbols block, added in Unicode 3.1 in 2001: just under 1,000 characters for maths, where a bold 𝐯 and a plain v can stand for different things. Every style is stored in alphabetical order, capitals first, so this reference is all a converter needs:

To get any letter, add its place in the alphabet to the starting point: bold C is U+1D400 plus 2, or U+1D402. The block holds 13 Latin alphabets in all, the six above plus bold italic, bold script, bold Fraktur and four sans-serif styles, with Greek letters and five sets of digits alongside. Twenty-four letters have no slot of their own (one italic, eleven script, five Fraktur and seven double-struck), because they were encoded earlier as letterlike symbols between U+2100 and U+214F, and a converter fills those in from there. Anything without a styled twin, such as punctuation, emoji and accented letters, passes through unchanged.

A plain English letter takes one byte in UTF-8, the encoding most of the web uses. Every letter in the maths block sits above U+FFFF, so it takes four bytes in UTF-8 and two code units, a surrogate pair, in UTF-16, the encoding used inside JavaScript, Java and Windows. The sample on this page shows the difference:

That's why converted text can hit a length limit early. A form that counts bytes or UTF-16 units fills up two to four times faster, X counts each styled letter as two of its 280, and an older MySQL column declared as “utf8” stores at most three bytes per character, so it can't keep these letters at all; it needs utf8mb4.

Converted letters pass some tests for “is this a letter?” and fail others, which catches out search boxes, sign-up forms and scripts that handle what people type. Here is what JavaScript does with 𝐓𝐞𝐱𝐭:

Python behaves much the same: str.lower() leaves them alone, and sorted() puts them last. The safe pattern in any language is to keep what the person typed for display, and compare, search and deduplicate on a normalized copy.

Every letter in the maths block has a compatibility mapping, tagged <font>, back to its plain letter. Unicode normalization form NFKC applies those mappings, so a single function call undoes the conversion, gaps included: ℎ becomes h and ℝ becomes R.

• JavaScript: text.normalize('NFKC')
• Python: unicodedata.normalize('NFKC', text)
• Java: Normalizer.normalize(text, Normalizer.Form.NFKC)
• Swift: text.precomposedStringWithCompatibilityMapping

NFKC is blunt, though. It also turns fi into fi, ² into 2 and ½ into 1⁄2, so run it over the styled part rather than a whole document full of footnotes and fractions. Styles built from other alphabets, such as small caps, upside down text and faux Cyrillic, have no mapping back and have to be retyped.

Tip: To see what a string is made of, list its code points. In a browser console, [...text].map(c => c.codePointAt(0).toString(16)) prints each character's number in hex; values from 1d400 to 1d7ff are maths letters.