Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maximum content length is limited #64

Closed
wizd opened this issue Nov 4, 2023 · 1 comment
Closed

Maximum content length is limited #64

wizd opened this issue Nov 4, 2023 · 1 comment
Labels
good first issue Good for newcomers

Comments

@wizd
Copy link

wizd commented Nov 4, 2023

I've noticed that when I try to translate a long story, the output gets truncated. How can I increase the length of the translated text?

@winstxnhdw
Copy link
Owner

winstxnhdw commented Nov 4, 2023

You can set the max_input_length kwargs for translate_iterable.

https://opennmt.net/CTranslate2/python/ctranslate2.Translator.html#ctranslate2.Translator.translate_iterable

But if you are going to be translating that much, it would be best for you to convert them into batches < 1024 tokens and then translate them in parallel.

You can look at my test to see how you can pass multiple batches to the API. In general, the API checks for newlines, splits and sends the entire batch to translate_iterable.

@winstxnhdw winstxnhdw pinned this issue Dec 3, 2023
@winstxnhdw winstxnhdw added the good first issue Good for newcomers label Dec 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants