Has anyone able to run ChatDoctor?

[tomsnunes]

I recently came across with ChatDoctorrepository, it looks very interesting but I haven't been able to generate working ggml templates so far to be able to run via llama.cpp (master).

Got this error while trying to export to .pth:
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. The tokenizer class you load from this checkpoint is 'LLaMATokenizer'. The class this function is called from is 'LlamaTokenizer'.

There was no error while running the convertion from .pth to .ggml or while running the quantization to get ggml-model-q4_0.bin, but I can't execute the ggml-model-q4_0.bin model from llama.cpp. Got the following message:

.\bin\Release\main.exe -m C:\.ai\.models\chatdoctor\ggml-model-q4_0.bin main: seed = 1680125564 llama_model_load: loading model from 'C:\.ai\.models\chatdoctor\ggml-model-q4_0.bin' - please wait ... llama_model_load: n_vocab = 32000 llama_model_load: n_ctx = 512 llama_model_load: n_embd = 4096 llama_model_load: n_mult = 256 llama_model_load: n_head = 32 llama_model_load: n_layer = 32 llama_model_load: n_rot = 128 llama_model_load: f16 = 2 llama_model_load: n_ff = 11008 llama_model_load: n_parts = 1 llama_model_load: type = 1 llama_model_load: ggml ctx size = 4273.34 MB llama_model_load: mem required = 6065.34 MB (+ 1026.00 MB per state) llama_model_load: loading model part 1/1 from 'C:\.ai\.models\chatdoctor\ggml-model-q4_0.bin' llama_model_load: llama_model_load: tensor 'tok_embeddings.weight' has wrong size in model file llama_init_from_file: failed to load model main: error: failed to load model 'C:\.ai\.models\chatdoctor\ggml-model-q4_0.bin'

Don't know if this a issue. I guess I'm doing something wrong while exporting to pth.

Until now I was following and trying to adapt the following process, so don't know if I should open a issue or even if such model is compatible with llama.cpp at moment.

Any help will be appreciated.

[Topping1]

I remember vaguely that the whole LLaMATokenizer vs LlamaTokenizer was because you need to update your transformers python library to the latest version.

[tomsnunes]

After I reinstalled the packages in requirements.txt it works. I had to use the last version of export_state_dict_checkpoint.py of Alpaca-Lora repository.
Thank you!

[Topping1]

try also editing the tokenizer_config.json

{
  "bos_token": "",
  "eos_token": "",
  "model_max_length": 512,
  "padding_side": "right",
  "special_tokens_map_file": "./weights/llama-7b/special_tokens_map.json",
  "tokenizer_class": "LLaMATokenizer",
  "unk_token": ""
}

change LLaMATokenizer to LlamaTokenizer. That might work.

[xor2003]

@tomsnunes Where have you downloaded the model files?

[tomsnunes]

There a form to request access to the checkpoint data here in the project repository, but there is too a filelist.txt (https://github.com/Kent0n-Li/ChatDoctor/blob/main/filelist.txt) that contains the same file content but its is crypted so you must have the Llamma weights to be able to decrypt it. I use the decrypt.py script from https://github.com/pointnetwork/point-alpaca and it works.

[xor2003]

I did like you suggested. Could you also tell how to convert or rename from pytorch_model-00001-of-00003.bin to .pth? Thanks

[xor2003]

Here the answer: #303

[schue]

https://huggingface.co/TheBloke/medalpaca-13B-GGML

[aseok]

Where to find ggml compatible datasets/how to convert them?