SentencePiece model training

[versae]

After taking a look at the vocabularies of the SentencePiece models referenced in the example T5X gin files, and after reading SeqIO's vocabularies.py code, there are a few things that I think might be important when creating a tokenizer for training T5X models that I haven't found anywhere.

1. Are <pad>, </s>, and <unk> (with ids 0, 1, and 2) the only symbols used in the existing tokenizers for training T5X models? Why is <s> not included?
2. Are these models unigram or BPE? Does it matter?
3. Can the 100 extra ids be added regardless of the SentencePiece model used by just passing in the extra_ids param?
4. Should always NUM_EMBEDDINGS be round up to the next multiple of 128 of vobab_size + 100 when training in a TPU?

[adarob]

These question really apply to the original T5 models (https://arxiv.org/abs/1910.10683), as you are not required to use any specific vocabulary with T5X.

1. <s> was not used in the original T5 paper.
2. Neither. See https://blog.floydhub.com/tokenization-nlp/#sentencepiece for details.
3. Yes.
4. This is recommended for efficiency reasons.

[versae]

Thanks for the answers and the links, @adarob!

On the SentencePiece tokenizer (pasting here for reference):

Which I interpret as Google having their own closed BPE implementation compatible with the SentencePiece format.

(Kudo, 2018)

[adarob]

The implementation is here: https://github.com/google/sentencepiece.

[versae]

Got it. Then it's BPE as implemented in Google's SentencePiece library.