Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move tokenizer info into pte for et #1510

Merged
merged 1 commit into from
Mar 14, 2025

Conversation

silverguo
Copy link
Contributor

@silverguo silverguo commented Mar 13, 2025

For #1484

$python3 torchchat.py export llama3.1 --quantize torchchat/quant_config/mobile.json --output-pte-path exportedModels/llama3.1.pte
....
The methods are:  {'forward'}
The config methods are:  {'tokenizer_type'}
....
$torchchat/utils/scripts/build_native.sh et
....
$cmake-out/et_run exportedModels/llama3.1.pte -z `python3 torchchat.py where llama3.1`/tokenizer.model -i "Once upon a time"
....
Once upon a time, in a land far, far away, there was a small village nestled between two great mountains. In this peaceful village, a wise and gentle soul named Sophia lived. Sophia was renowned for her extraordinary kindness, compassion, and ability to understand and communicate with all living creatures.
Her kindness was not limited to humans; she had a special gift for understanding and talking to animals. The villagers would often joke that Sophia had an "animal whisperer" role in her village. Her deep understanding and gentle demeanor had earned the trust of the animals, who would come from far and wide to seek her counsel and wisdom.
Sophia

achieved tok/s: 20.268114
....

Worked without passing -l.

Copy link

pytorch-bot bot commented Mar 13, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1510

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 3 Pending

As of commit 56f5f3e with merge base 3c7e839 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Mar 13, 2025
@silverguo silverguo force-pushed the et_tokenizer_in_pte branch from 4e2a42a to 56f5f3e Compare March 13, 2025 23:13
@silverguo silverguo marked this pull request as ready for review March 13, 2025 23:14
@silverguo silverguo requested a review from Jack-Khuu March 13, 2025 23:14
@Jack-Khuu Jack-Khuu merged commit eb73f34 into pytorch:main Mar 14, 2025
72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants