![]() ![]() Model = om_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True) To do so, you first need to install Flash Attention: We recommend using Flash Attention 2 if your GPU allows for it. You can apply additional speed and memory improvements to Distil-Whisper which we cover in the following. Fast Inference from Transformers via Speculative Decoding paper by Leviathan et.Assisted Generation: a new direction toward low-latency text generation blog post by Joao Gante.Speculative decoding for 2x faster Whisper inference blog post by Sanchit Gandhi.In existing speech recognition pipelines.įor more details on speculative decoding, refer to the following resources: This makes it the perfect replacement for Whisper large-v2 In VRAM memory usage while mathematically ensuring the same results. Note: speculative decoding should be on average 2x faster than using "only" Whisper large-v2 at a mere 8% increase For this example, we'll also install □ Datasets to load a toyįrom datasets import load_dataset dataset = load_dataset( "hf-internal-testing/librispeech_asr_dummy", "clean", split = "validation") Install the latest version of the Transformers library. Usageĭistil-Whisper is supported in Hugging Face □ Transformers from version 4.35 onwards. We will soon update the repository with multilingual checkpoints when ready! 1. If you are interested in distilling Whisper in your language, check out the provided training code. We are working with the community to distill Whisper on other languages. Note: Distil-Whisper is currently only available for English speech recognition. It is only 166M parameters, while performing within 3% WER of Whisper large-v2. Such as on-device or mobile applications, the distil-small.en is a great choice, since ![]() Distil-Whisper is a distilled version of Whisper that is 6 times faster, 49% smaller, and performs within 1% wordĮrror rate (WER) on out-of-distribution evaluation sets: Modelįor applications where latency and accuracy are important, we recommend the distil-medium.en ![]()
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |