H-Nets

#

H-Nets are

• an extension of LLMs (they predict the next $thing)
• a dynamic tokenizer (they define token chunks on-the-fly)
• recursively defined (you can do nested tokenization)

Hence, “Dynamic Chunking for End-to-End Hierarchical Sequence Modeling”.

I think H-Nets make an awful lot of sense, so I’ve done lots of research with them.