Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move tokenizer information into pte to reduce ExecuTorch runner args #1484

Closed
Jack-Khuu opened this issue Jan 30, 2025 · 0 comments
Closed
Assignees
Labels
actionable Items in the backlog waiting for an appropriate impl/fix enhancement New feature or request ExecuTorch Issues related to ExecuTorch installation, export, or build. Mobile uses separate tags good first issue Good for newcomers triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@Jack-Khuu
Copy link
Contributor

🚀 The feature, motivation and pitch

After an ExecuTorch model is exported to a pte, tokenization information must be passed in as an arg (-l <#>) to the runner. This can be avoided by writing this information into the pte file itself since the tokenizer is known at export time (sentencepiece => 2, tiktoken =>3). Tokenization information can be stored during export as a constant_method.

For example: https://github.com/pytorch/torchchat?tab=readme-ov-file#deploy-and-run-on-android

cmake-out/et_run llama3.1.pte -z `python3 torchchat.py where llama3.1`/tokenizer.model -l 3 -i "Once upon a time"

Task:

  1. Update ExecuTorch exporting to save tokenization information in the pte artifact
  2. Update the ExecuTorch runner to read the newly saved metadata

For a similar optimization made for aoti: #1159.
See #1439 for conversation/more context

Alternatives

Continue to pass tokenizer arguments to the runner

Additional context

No response

RFC (Optional)

No response

@Jack-Khuu Jack-Khuu added actionable Items in the backlog waiting for an appropriate impl/fix enhancement New feature or request ExecuTorch Issues related to ExecuTorch installation, export, or build. Mobile uses separate tags good first issue Good for newcomers triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Jan 30, 2025
@Jack-Khuu Jack-Khuu moved this from To triage to Ready in [torchchat] Looking for Contributors Feb 21, 2025
@Jack-Khuu Jack-Khuu moved this from Ready to Backlog in [torchchat] Looking for Contributors Feb 21, 2025
@Jack-Khuu Jack-Khuu moved this from Backlog to Ready in [torchchat] Looking for Contributors Feb 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
actionable Items in the backlog waiting for an appropriate impl/fix enhancement New feature or request ExecuTorch Issues related to ExecuTorch installation, export, or build. Mobile uses separate tags good first issue Good for newcomers triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
Development

No branches or pull requests

2 participants