Sparsetral [MODEL RELEASE]

Introducing Sparsetral, a sparse MoE model made from the dense model mistral. For more information on the theory, here is the original paper (Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks). Here is the original repo that goes with the paper (original repo) and the here is the forked repo with sparsetral (mistral) integration

👉 Get it here: forked repo @SERP AI

We also forked unsloth and vLLM for efficient training and inferencing. Sparsetral on vLLM has been tested to work on a 4090 at bf16 precision, 4096 max_model_len, and 64 max_num_seqs.

Here is the model on huggingface. - Note this is v2. v1 was trained with (only listing changes from v2) (64 adapter dim, 32 effective batch size, slim-orca dataset)

Up next is evaluations, then DPO (or CPO) + possibly adding activation beacons after for extended context length

Training

• 8x A6000s
• Forked version of unsloth for efficient training
• Sequence Length: 4096
• Effective batch size: 128
• Learning Rate: 2e-5 with linear decay
• Epochs: 1
• Dataset: OpenHermes-2.5
• Base model trained with QLoRA (rank 64, alpha 16) and MoE adapters/routers trained in bf16
• Num Experts: 16
• Top K: 4
• Adapter Dim: 512

If you need any help or have any questions don't hesitate to comment!