First release.
tidycjk is a tidy toolkit for Chinese, Japanese and
Korean text: script and language classification, display width, width
normalisation, and a pluggable word-segmentation engine that the caller
names, as verbs that return tibbles. The package itself has no compiled
code, bundles no data, and makes no network requests.
cjk_summary(data, col) reports n_docs,
n_with_cjk, prop_with_cjk and
mean_ratio for a text column.cjk_char_counts(data, col) returns one row per distinct
CJK character with its code point, script, Unicode block and count.cjk_segment(x, engine) splits CJK text into words, and
cjk_tokens(data, col, engine) is the tidy version,
returning one row per token.cjk_segmenters() lists what is available and
register_cjk_segmenter() adds an engine: any function of
(x, ...) returning a list of character vectors.engine has no
default. jiebaR
was the obvious candidate and was archived from CRAN on 2025-05-01, so
it cannot be a dependency of a CRAN package.
?cjk_segmenters shows the four lines that register it once
you have installed it from source."character": one token per
CJK character, with runs of non-CJK text split on whitespace. It is
character tokenisation rather than word segmentation and says so on its
help page. Making it the silent default would have handed character
tokens to callers asking for words, which is the mistake this package
exists to avoid, so the choice is explicit instead.has_cjk(), cjk_script() and
cjk_ratio() classify and measure.cjk_detect_language() infers the language from the
scripts present.cjk_width(), cjk_pad() and
cjk_truncate() work in terminal columns rather than
characters.to_halfwidth() and to_fullwidth()
normalise width variants.cjk_blocks() exports the Unicode block table the
package is built on, covering every unified ideograph block through
Extension I as well as the phonetic scripts, CJK punctuation and the
width variants.Language detection returns NA rather than
guessing. Japanese written without kana is not distinguishable
from Chinese by script alone, so a Han-only string gets NA.
cjk_detect_language(x, han_only = "chinese") opts into the
guess and keeps the assumption visible in the calling code.
Width is delegated to stringi.
cjk_width() and cjk_pad() wrap
stringi::stri_width() and stringi::stri_pad(),
which read the live Unicode tables in ICU, the Unicode Consortium’s C
library. A hand-maintained range table would go stale at every Unicode
release, and the version commonly copied around is already wrong for
tens of thousands of assigned code points: it stops below the
supplementary planes, so every ideograph in Extensions B through I comes
out one column instead of two. cjk_truncate() has no
stringi equivalent and is implemented here.
Normalisation is surgical.
to_halfwidth() maps fullwidth ASCII, the ideographic space
and halfwidth katakana, and touches nothing else. NFKC
additionally rewrites ligatures, superscripts, Roman numerals, circled
numbers, the no-break space and the CJK compatibility ideographs, which
is almost never wanted.
Voiced marks are composed by default. Halfwidth
katakana writes a voiced syllable as two code points;
compose = TRUE folds them into the single precomposed
character, so "ｶﾞ" becomes "ガ". Every pair in
the composition table agrees with Unicode NFC, including the five that
break the base-plus-one rule.
Ties never depend on the locale.
cjk_script() and cjk_char_counts() break ties
by first appearance rather than by collation order.
data-raw/unihan.R downloads and parses it,
but no character data is hand-written or bundled yet. pinyin and hanyupinyin
are on CRAN today.