Prevent to block entire event loop when creating multiple sessions #1221
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
How I discovered?
While monitoring the TikTokApi session creation process, I noticed that the actual sleep duration varied based on the number of concurrent sessions, even with the same
sleep_after
value.For instance, with
sleep_after=3
:This linear scaling of total sleep time with the number of sessions indicated that the sessions weren't being created concurrently as intended. Further investigation revealed that this was causing timeout issues when the cumulative sleep time exceeded the browser's connection timeout limit. This issue particularly affects users who want to scrape data in parallel.
Causes
The root cause is the use of blocking
time.sleep()
within theasync __create_session()
function. This creates several issues:sleep_after * number_of_sessions
For example, when using asyncio.gather() to create multiple sessions:
Even though asyncio.gather() is designed for concurrent execution, the blocking time.sleep() prevents true concurrency, causing each session to wait for the previous one's sleep to complete.
The fix is to replace
time.sleep()
withawait asyncio.sleep()
, allowing true concurrent session creation and proper async execution. (Also puttingtime.sleep()
is banned in many linters!)Tested working on my end :)