BPE in Haskell

[BobMcDear]

Hello,

I have created a port of minbpe in Haskell, minbpe-hs, that provides the same functionalities as minbpe minus GPT4Tokenizer. Thanks to the inherently recursive structure of BPE, it can be rendered quite nicely in functional languages, and I hope those who are struggling to apprehend the workings of this algorithm can benefit from studying its Haskell implementation.

The Wikipedia example can be reproduced using minbpe-hs as follows.

{-# LANGUAGE OverloadedStrings #-}

import BPE.Base
import BPE.Basic

main :: IO ()
main = do
    let (merges, vocab) = trainTokenizer (256 + 3) "aaabdaaabac"
    putStrLn $ show $ encode merges "aaabdaaabac"
    putStrLn $ show $ decode vocab [258, 100, 258, 97, 99]
    saveMergesAndVocab "toy" merges vocab

Would it be all right if I submit a PR to add this to the list of community extensions?

Thank you,
Borna