Skip to content

Commit

Permalink
Update known_issues.rst
Browse files Browse the repository at this point in the history
  • Loading branch information
@ivknv
ivknv committed Jan 30, 2023
1 parent 09712cc commit 5d9f78c
Show file tree
Hide file tree
Showing 2 changed files with 68 additions and 142 deletions.
65 changes: 23 additions & 42 deletions docs/known_issues.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,51 +4,32 @@ Known Issues
Very Slow Upload of Certain Types of Files
##########################################

For whatever reason, files with specific extensions take much longer time to upload.
Here are some known extensions with this problem:

1. :code:`.db`
2. :code:`.mp4`
3. :code:`.avi`
4. :code:`.3gp`
5. :code:`.rar`
6. :code:`.zip`
7. :code:`.gz`
8. :code:`.tgz`
9. :code:`.dat`
10. ...and more.
.. note::

The following information may become outdated at some point in the future.

Yandex.Disk's REST API limits upload speeds up to 128 KiB/s for certain MIME types of files.
More specifically, throttling takes place based on value of :code:`media_type`
(see :any:`yadisk_async.YaDisk.get_meta`).
It appears there are 3 types of media types that are throttled:

1) :code:`data` (.db, .dat, etc.)
2) :code:`compressed` (.zip, .gz, .tgz, .rar, .etc)
3) :code:`video` (.3gp, .mp4, .avi, etc.)

This behavior of throttling is predetermined at the moment of requesting an
upload link (with :any:`yadisk_async.YaDisk.get_upload_link`). The content of the
uploaded file does not matter.

The reason why this problem cannot be observed when uploading through the
official website, is that this throttling does not apply to internal services
(the Yandex.Disk website uses an intermediate internal API to obtain upload links).

While it is not clear what the purpose of this throttling is, it is certain at
this point that this is not a bug.

The only known workaround is to upload files with changed filename extensions (or without them entirely).
For example, if you want to upload a file named "my_database.db", you can initially
upload it under the name "my_database.some_other_extension" and then rename it back
to "my_database.db". This approach has some obvious downsides but at least it
works.

Low Upload Speed on Windows
###########################

.. _http.client: https://docs.python.org/3/library/http.client.html
.. _urllib3: https://pypi.org/project/urllib3/
.. _eventlet: https://pypi.org/project/eventlet
.. _yadisk-async: https://pypi.org/project/yadisk-async
.. _aiohttp: https://pypi.org/project/aiohttp
.. _requests: https://pypi.org/project/requests

If you experience low upload speeds on Windows, the reason might be due to
Python's standard library internally using :code:`select()` to wait for sockets.
There are several ways around it:

1) Monkey-patching `http.client`_ and `urllib3`_ to use bigger :code:`blocksize`.
See `this comment <https://github.com/urllib3/urllib3/issues/1394#issuecomment-954044006>`_ for more details.
2) Monkey-patching through a library like `eventlet`_.
3) Using `yadisk-async`_ instead. It uses `aiohttp`_ instead of `requests`_.
4) Uploading files to direct links (obtained through :any:`yadisk_async.YaDisk.get_upload_link()`) using
a different library (such as `aiohttp`_).

Upload Timeout on Large Files
#############################

When uploading large files (over a couple of GB in size) you may experience
timeout errors after the full upload. This might be caused by Yandex.Disk computing
hash sums or doing some other operations. The bigger the file, the bigger the
timeouts may need to be set.
145 changes: 45 additions & 100 deletions docs/locales/ru/LC_MESSAGES/known_issues.po
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: YaDisk 1.2.19\n"
"Project-Id-Version: YaDisk 1.3.5\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2023-01-28 02:48+0500\n"
"POT-Creation-Date: 2023-01-30 18:45+0500\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <[email protected]>\n"
Expand All @@ -25,56 +25,64 @@ msgstr "Известные проблемы"
msgid "Very Slow Upload of Certain Types of Files"
msgstr "Очень медленная загрузка некоторых типов файлов на Яндекс.Диск"

#: ../../known_issues.rst:7
msgid ""
"For whatever reason, files with specific extensions take much longer time"
" to upload. Here are some known extensions with this problem:"
msgstr ""
"По неизвестной причине файлы с определенными расширениями в имени "
"загружаются на Яндекс.Диск гораздо медленнее. Вот некоторые из известных "
"проблемных расширений файлов:"

#: ../../known_issues.rst:10
msgid ":code:`.db`"
msgstr ""
#: ../../known_issues.rst:9
msgid "The following information may become outdated at some point in the future."
msgstr "Данная информация может в дальнейшем будущем оказаться устаревшей."

#: ../../known_issues.rst:11
msgid ":code:`.mp4`"
msgstr ""

#: ../../known_issues.rst:12
msgid ":code:`.avi`"
msgstr ""

#: ../../known_issues.rst:13
msgid ":code:`.3gp`"
msgid ""
"Yandex.Disk's REST API limits upload speeds up to 128 KiB/s for certain "
"MIME types of files. More specifically, throttling takes place based on "
"value of :code:`media_type` (see :any:`yadisk_async.YaDisk.get_meta`). It"
" appears there are 3 types of media types that are throttled:"
msgstr ""
"REST API Яндекс.Диск ограничивает скорость загрузки файлов на Диск до 128"
" KiB/s для определенных MIME типов файлов. Если быть точнее, троттлинг "
"осуществляется в зависимости от значения :code:`media_type` (см. "
":any:`yadisk_async.YaDisk.get_meta`). Судя по всему ограничение скорости "
"действует на 3 типа файлов (media type):"

#: ../../known_issues.rst:14
msgid ":code:`.rar`"
#: ../../known_issues.rst:16
msgid ":code:`data` (.db, .dat, etc.)"
msgstr ""

#: ../../known_issues.rst:15
msgid ":code:`.zip`"
#: ../../known_issues.rst:17
msgid ":code:`compressed` (.zip, .gz, .tgz, .rar, .etc)"
msgstr ""

#: ../../known_issues.rst:16
msgid ":code:`.gz`"
#: ../../known_issues.rst:18
msgid ":code:`video` (.3gp, .mp4, .avi, etc.)"
msgstr ""

#: ../../known_issues.rst:17
msgid ":code:`.tgz`"
#: ../../known_issues.rst:20
msgid ""
"This behavior of throttling is predetermined at the moment of requesting "
"an upload link (with :any:`yadisk_async.YaDisk.get_upload_link`). The "
"content of the uploaded file does not matter."
msgstr ""
"Ограничение скорости предопределяется в момент получения ссылки для "
"загрузки файла на диск (см. :any:`yadisk_async.YaDisk.get_upload_link`). "
"Содержимое загружаемого файла не имеет значения."

#: ../../known_issues.rst:18
msgid ":code:`.dat`"
#: ../../known_issues.rst:24
msgid ""
"The reason why this problem cannot be observed when uploading through the"
" official website, is that this throttling does not apply to internal "
"services (the Yandex.Disk website uses an intermediate internal API to "
"obtain upload links)."
msgstr ""
"Причина, по которой эта проблема не наблюдается при попытке загрузить "
"файл через оффициальный сайт, заключается в том, что ограничение скорости"
" не применяется для внутренних сервисов (сайт Яндекс.Диска использует "
"промежуточный внутренний API для получения ссылок)."

#: ../../known_issues.rst:19
msgid "...and more."
msgstr "...и другие."
#: ../../known_issues.rst:28
msgid ""
"While it is not clear what the purpose of this throttling is, it is "
"certain at this point that this is not a bug."
msgstr "Хотя и не понятно, в чем смысл такого ограничения, это точно не баг."

#: ../../known_issues.rst:21
#: ../../known_issues.rst:31
msgid ""
"The only known workaround is to upload files with changed filename "
"extensions (or without them entirely). For example, if you want to upload"
Expand All @@ -89,66 +97,3 @@ msgstr ""
"загрузить его под именем \"my_database.some_other_extension\" и после "
"загрузки переименовать обратно в \"my_database.db\". У такого подхода "
"есть очевидные недостатки, но по крайней мере он работает."

#: ../../known_issues.rst:28
msgid "Low Upload Speed on Windows"
msgstr "Низкая скорость загрузки файлов на Диск под Windows"

#: ../../known_issues.rst:37
msgid ""
"If you experience low upload speeds on Windows, the reason might be due "
"to Python's standard library internally using :code:`select()` to wait "
"for sockets. There are several ways around it:"
msgstr ""
"Если вы столкнулись с низкой скоростью загрузки файлов на Диск под "
"Windows, то причиной может стандартная библиотека Python, которая внутри "
"использует :code:`select()` для ожидания сокетов. В таком случае "
"существует несколько путей обхода данной проблемы:"

#: ../../known_issues.rst:41
msgid ""
"Monkey-patching `http.client`_ and `urllib3`_ to use bigger "
":code:`blocksize`. See `this comment "
"<https://github.com/urllib3/urllib3/issues/1394#issuecomment-954044006>`_"
" for more details."
msgstr ""
"Monkey-patch `http.client`_ и `urllib3`_, чтобы увеличить "
":code:`blocksize`. См. `этот комментарий "
"<https://github.com/urllib3/urllib3/issues/1394#issuecomment-954044006>`_."

#: ../../known_issues.rst:43
msgid "Monkey-patching through a library like `eventlet`_."
msgstr "Monkey-patch через библиотеку `eventlet`_."

#: ../../known_issues.rst:44
msgid "Using `yadisk-async`_ instead. It uses `aiohttp`_ instead of `requests`_."
msgstr ""
"Использование `yadisk-async`_. Эта версия использует `aiohttp`_ вместо "
"`requests`_."

#: ../../known_issues.rst:45
msgid ""
"Uploading files to direct links (obtained through "
":any:`yadisk_async.YaDisk.get_upload_link()`) using a different library "
"(such as `aiohttp`_)."
msgstr ""
"Загрузка файлов по прямым ссылкам (полученным через "
":any:`yadisk_async.YaDisk.get_upload_link()`), используя другую библиотеку "
"(например, `aiohttp`_)."

#: ../../known_issues.rst:49
msgid "Upload Timeout on Large Files"
msgstr "Таймауты при загрузки больших файлов на Диск"

#: ../../known_issues.rst:51
msgid ""
"When uploading large files (over a couple of GB in size) you may "
"experience timeout errors after the full upload. This might be caused by "
"Yandex.Disk computing hash sums or doing some other operations. The "
"bigger the file, the bigger the timeouts may need to be set."
msgstr ""
"При загрузке больших файлов на Диск (несколько ГБ и более) вы можете "
"столкнуться с таймаутами после загрузки содержимого файла на Диск. "
"Возможно, это связано с вычислением хэш сумм и выполнением каких-то "
"других операций. Чем больше файл, тем больше следует поставить таймаут."

0 comments on commit 5d9f78c

Please sign in to comment.