Create Habana_Validated_Models.md to list all the models validated (#…

…1778)

README.md

@@ -325,6 +325,8 @@ After training your model, feel free to submit it to the Intel [leaderboard](htt
 to evaluate, score, and rank open-source LLMs that have been pre-trained or fine-tuned on Intel Hardwares. Models submitted to the leaderboard will be evaluated on
 the Intel Developer Cloud. The evaluation platform consists of Gaudi Accelerators and Xeon CPUs running benchmarks from the Eleuther AI Language Model Evaluation Harness.
 
+The list of validated models through continuous integration tests is posted [here](https://github.com/huggingface/optimum-habana/tree/main/tests/Habana_Validated_Models.md)
+
 ## Development
 
 Check the [contributor guide](https://github.com/huggingface/optimum/blob/main/CONTRIBUTING.md) for instructions.

tests/Habana_Validated_Models.md

@@ -0,0 +1,136 @@
+# The following table lists the models validated through integration tests.
+
+|File|Model|Precision|1x/8x|
+|:----|:----|:----|:----|
+|tests/test_examples.py::TextClassificationExampleTester|bert-large-uncased-whole-word-masking|mixed-precision(bf16)|single_card|
+|tests/test_examples.py::QuestionAnsweringExampleTester|roberta-large|mixed-precision(bf16)|single_card|
+|tests/test_examples.py::EagerModeCausalLanguageModelingExampleTester|google/gemma-2b-it|mixed-precision(bf16)|single_card|
+|tests/test_examples.py::CausalLanguageModelingExampleTester|google/gemma-2b-it|mixed-precision(bf16)|single_card|
+|tests/test_examples.py::ImageClassificationExampleTester|microsoft/swin-base-patch4-window7-224-in22k|mixed-precision(bf16)|single_card|
+|tests/test_examples.py::ImageClassificationExampleTester|google/vit-base-patch16-224-in21k|mixed-precision(bf16)|single_card|
+|tests/test_examples.py::ProteinFoldingExampleTester|facebook/esmfold_v1|mixed-precision(bf16)|single_card|
+|tests/test_examples.py::ProteinFoldingExampleTester2|mila-intel.ProtST-esm1b|mixed-precision(bf16)|single_card|
+|tests/test_examples.py::CausalLanguageModelingLORAExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|single_card|
+|tests/test_peft_inference.py::TestGaudiPeftTextGeneration|huggyllama/llama-7b-prompt-tuning|mixed-precision(bf16)|single_card|
+|tests/test_peft_inference.py::TestGaudiPeftTextGeneration|huggyllama/llama-7b-prefix-tuning|mixed-precision(bf16)|single_card|
+|tests/test_peft_inference.py::TestGaudiPeftTextGeneration|huggyllama/llama-7b-p-tuning|mixed-precision(bf16)|single_card|
+|tests/test_peft_inference.py::TestGaudiPeftTextGeneration|huggyllama/llama-7b-llama-adapter|mixed-precision(bf16)|single_card|
+|tests/test_pipeline.py::TestGaudiPipeline::test_image_to_text|Salesforce/blip-image-captioning-base|mixed-precision(bf16)|single_card|
+|tests/test_pipeline.py::TestGaudiPipeline::test_image_to_text|nlpconnect/vit-gpt2-image-captioning|mixed-precision(bf16)|single_card|
+|tests/test_pipeline.py::TestGaudiPipeline::test_text_to_speech|microsoft/speecht5_tts-16000|mixed-precision(bf16)|single_card|
+|tests/test_pipeline.py::TestGaudiPipeline::test_text_to_speech|facebook/hf-seamless-m4t-medium-16000|mixed-precision(bf16)|single_card|
+|tests/test_pipeline.py::TestGaudiPipeline::test_text_to_speech|facebook/mms-tts-eng-16000|mixed-precision(bf16)|single_card|
+|tests/test_fsdp_examples.py|bert-base-uncased|mixed-precision(bf16)|multi_card|
+|tests/test_fsdp_examples.py|meta-llama/Llama-2-7b-hf|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardVisionLanguageExampleTester|./clip-roberta|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardTextClassificationExampleTester|bert-large-uncased-whole-word-masking|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardQuestionAnsweringExampleTester|roberta-large|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingExampleTester|google/gemma-2b-it|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardImageClassificationExampleTester|microsoft/swin-base-patch4-window7-224-in22k|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardImageClassificationExampleTester|google/vit-base-patch16-224-in21k|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardMaskedLanguageModelingExampleTester|roberta-large|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardAudioClassificationExampleTester|MIT/ast-finetuned-speech-commands-v2|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardAudioClassificationExampleTester|facebook/wav2vec2-base|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardVisionLanguageExampleTester|./clip-roberta|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingLORAExampleTester2|tiiuae/falcon-40b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingLORAExampleTester2|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingLORAExampleTester|tiiuae/falcon-40b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingLORAExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardBridgetowerExampleTester|BridgeTower/bridgetower-large-itm-mlm-itc|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardSeq2SeqSpeechRecognitionExampleTester|openai/whisper-small|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingLORAFSDPCompileExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardSFTExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardSFTChatExampleTester|Qwen/Qwen2-7B|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardSFTChatPeftExampleTester|Qwen/Qwen2-7B|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardDPOExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardRewardExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardPPOExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardProteinFoldingClassificationTester|mila-intel/protst-esm1b-for-sequential-classification|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingPromptTuningExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingPrefixTuningExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingPTuningExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingLlamaAdapterExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingLoRAFP8ExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardImageToTextModelingLoRAExampleTester|meta-llama/Llama-3.2-11B-Vision-Instruct|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardImageToTextModelingLoRAExampleTester|HuggingFaceM4/idefics2-8b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingVeraExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingLnExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingIA3ExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::MultiCardCausalLanguageModelingAdaloraExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_examples.py::DeepSpeedTextClassificationExampleTester|meta-llama/LlamaGuard-7b|mixed-precision(bf16)|deepspeed|
+|tests/test_examples.py::DeepspeedCausalLanguageModelingExampleTester|codellama/CodeLlama-13b-Instruct-hf|mixed-precision(bf16)|deepspeed|
+|tests/test_examples.py::DeepspeedCausalLanguageModelingExampleTester|THUDM/chatglm3-6b|mixed-precision(bf16)|deepspeed|
+|tests/test_examples.py::DeepspeedCausalLanguageModelingExampleTester|google/gemma-2b-it|mixed-precision(bf16)|deepspeed|
+|tests/test_examples.py::DeepspeedCausalLanguageModelingExampleTester|EleutherAI/gpt-neox-20b|mixed-precision(bf16)|deepspeed|
+|tests/test_examples.py::DeepspeedCausalLanguageModelingExampleTester|gpt2-xl|mixed-precision(bf16)|deepspeed|
+|tests/test_examples.py::DeepspeedSummarizationExampleTester|google/flan-t5-xxl|mixed-precision(bf16)|deepspeed|
+|tests/test_examples.py::DeepspeedSFTExampleTester|Qwen/Qwen2-72B|mixed-precision(bf16)|deepspeed|
+|tests/test_examples.py::MultiCardCausalLanguageModelingLoRACPExampleTester|huggyllama/llama-7b|mixed-precision(bf16)|multi_card|
+|tests/test_image_to_text_example.py|llava-hf/llava-v1.6-mistral-7b-hf|mixed-precision(bf16)|single_card|
+|tests/test_image_to_text_example.py|llava-hf/llava-v1.6-vicuna-7b-hf|mixed-precision(bf16)|single_card|
+|tests/test_image_to_text_example.py|llava-hf/llava-v1.6-vicuna-13b-hf|mixed-precision(bf16)|single_card|
+|tests/test_image_to_text_example.py|google/paligemma-3b-mix-224|mixed-precision(bf16)|single_card|
+|tests/test_image_to_text_example.py|HuggingFaceM4/idefics2-8b|mixed-precision(bf16)|single_card|
+|tests/test_image_to_text_example.py|meta-llama/Llama-3.2-11B-Vision-Instruct|mixed-precision(bf16)|single_card|
+|tests/test_image_to_text_example.py|tiiuae/falcon-11B-vlm|mixed-precision(bf16)|single_card|
+|tests/test_image_to_text_example.py|Qwen/Qwen2-VL-2B-Instruct|mixed-precision(bf16)|single_card|
+|tests/test_image_to_text_example.py|Qwen/Qwen2-VL-7B-Instruct|mixed-precision(bf16)|single_card|
+|tests/test_image_to_text_example.py|llava-hf/llava-v1.6-mistral-7b-hf|fp8|single_card|
+|tests/test_image_to_text_example.py|llava-hf/llava-v1.6-vicuna-7b-hf|fp8|single_card|
+|tests/test_image_to_text_example.py|llava-hf/llava-v1.6-vicuna-13b-hf|fp8|single_card|
+|tests/test_openclip_vqa.py|laion/CLIP-ViT-g-14-laion2B-s12B-b42K|mixed-precision(bf16)|single_card|
+|tests/test_openclip_vqa.py|microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|bigscience/bloomz-7b1|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|gpt2-xl|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|EleutherAI/gpt-j-6b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|EleutherAI/gpt-neox-20b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|meta-llama/Llama-2-7b-hf|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|tiiuae/falcon-40b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|bigcode/starcoder|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|Salesforce/codegen2-1B|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|mosaicml/mpt-30b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|mistralai/Mistral-7B-v0.1|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|mistralai/Mixtral-8x7B-v0.1|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|microsoft/phi-2|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|meta-llama/Meta-Llama-3-8B|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|meta-llama/Llama-2-7b-hf|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|stabilityai/stablelm-2-12b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|codellama/CodeLlama-34b-hf|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|bigcode/starcoder2-3b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|adept/persimmon-8b-base|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|google/gemma-7b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|google/gemma-2-9b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|google/gemma-2-27b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|state-spaces/mamba-130m-hf|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|Qwen/Qwen2-7B|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|Qwen/Qwen1.5-MoE-A2.7B|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|tiiuae/falcon-mamba-7b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|openbmb/MiniCPM3-4B|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|baichuan-inc/Baichuan2-7B-Chat|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|baichuan-inc/Baichuan2-13B-Chat|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|deepseek-ai/DeepSeek-V2-Lite|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|THUDM/chatglm3-6b|mixed-precision(bf16)|single_card|
+|tests/test_text_generation_example.py|tiiuae/falcon-180B|fp8|multi_card|
+|tests/test_text_generation_example.py|meta-llama/Llama-2-7b-hf|fp8|single_card|
+|tests/test_text_generation_example.py|meta-llama/Llama-2-70b-hf|fp8|multi_card|
+|tests/test_text_generation_example.py|mistralai/Mistral-7B-Instruct-v0.2|fp8|single_card|
+|tests/test_text_generation_example.py|mistralai/Mixtral-8x7B-v0.1|fp8|multi_card|
+|tests/test_text_generation_example.py|microsoft/phi-2|fp8|single_card|
+|tests/test_text_generation_example.py|TheBloke/Llama-2-7b-Chat-GPTQ|mixed-precision(bf16)|gptq|
+|tests/test_text_generation_example.py|TheBloke/Llama-2-7b-Chat-AWQ|mixed-precision(bf16)|awq|
+|tests/test_text_generation_example.py|bigscience/bloomz|mixed-precision(bf16)|deepspeed|
+|tests/test_text_generation_example.py|meta-llama/Meta-Llama-3-70B-Instruct|mixed-precision(bf16)|deepspeed|
+|tests/test_text_generation_example.py|facebook/opt-66b|mixed-precision(bf16)|deepspeed|
+|tests/test_text_generation_example.py|google/gemma-2-9b|mixed-precision(bf16)|deepspeed|
+|tests/test_text_generation_example.py|google/gemma-2-27b|mixed-precision(bf16)|deepspeed|
+|tests/test_text_generation_example.py|meta-llama/Llama-2-7b-hf|mixed-precision(bf16)|torch.compile|
+|tests/test_text_generation_example.py|meta-llama/Llama-2-7b-hf|mixed-precision(bf16)|distributed_tp|
+|tests/test_text_generation_example.py|gpt2-xl|mixed-precision(bf16)|contrastive_search|
+|tests/test_text_generation_example.py|Qwen/Qwen2-7b-Instruct|mixed-precision(bf16)|beam_search|
+|tests/test_encoder_decoder.py|facebook/bart-large-cnn|mixed-precision(bf16)|single_card|
+|tests/test_encoder_decoder.py|t5-3b|mixed-precision(bf16)|single_card|
+|tests/test_encoder_decoder.py|t5-small|mixed-precision(bf16)|single_card|
+|tests/test_cmnrl.py|train_samples_mnrl0-train_samples_cmnrl0|mixed-precision(bf16)|single_card|
+|tests/test_sentence_transformers.py|sentence-transformers/all-mpnet-base-v2|mixed-precision(bf16)|single_card|
+|tests/test_custom_file_input.py|bigcode/starcoder|mixed-precision(bf16)|single_card|
+|tests/test_diffusers.py|CompVis/stable-diffusion-v1-4|mixed-precision(bf16)|single_card|