#openmachine
the MHA projection dimension is larger than the embedding dimension, the memory can be reduced by a factor of 32 for the T5-11B model for example.
See https://github.com/OpenMachine-ai/transformer-tricks for code and more transformer tricks, and [4/5 of https://arxiv.org/abs/2503.05840v1]
March 11, 2025 at 6:41 AM