lyogavin/airllm

compression: supported options: 4bit, 8bit for 4-bit or 8-bit block-wise quantization, or by default None for no compression
profiling_mode: supported options: True to output time consumptions or by default False
layer_shards_saving_path: optionally another path to save the split model
hf_token: huggingface token can be provided here if downloading gated models like: meta-llama/Llama-2-7b-hf
prefetching: prefetching to overlap the model loading and compute. By default, turned on. For now, only AirLLMLlama2 supports this.
delete_original: if you don't have too much disk space, you can set delete_original to true to delete the original downloaded hugging face model, only keep the transformed one to save half of the disk space.

Maintainer? Show your league in your README

Card: your league's emblem, from Copper to Legendary

[![bestofgithub](https://bestofgithub.dev/badge/lyogavin/airllm.svg?style=card)](https://bestofgithub.dev/repo/lyogavin/airllm)

Flat: fits beside your other badges

[![bestofgithub](https://bestofgithub.dev/badge/lyogavin/airllm.svg)](https://bestofgithub.dev/repo/lyogavin/airllm)

Loading README…

AI Agents Recommendation: