Skip to content

Commit

Permalink
Publish (#2)
Browse files Browse the repository at this point in the history
* wip

* wip

* wip

* wip

* wip

* wip

* wip
  • Loading branch information
will-ockmore authored Jan 26, 2025
1 parent aa7868f commit c370a1b
Show file tree
Hide file tree
Showing 16 changed files with 1,444 additions and 231 deletions.
34 changes: 34 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Publish

on:
push:
tags:
- "*"

jobs:
publish:
name: "Publish release"
runs-on: "ubuntu-latest"

environment:
name: deploy

steps:
- uses: "actions/checkout@v4"
- uses: "actions/setup-python@v5"
with:
python-version: 3.9

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
python-version: 3.9
uv-version: "0.5.22"

- name: "Build package & docs"
run: "scripts/build"

- name: "Publish to PyPI & deploy docs"
run: "scripts/publish"
env:
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_TOKEN }}
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,6 @@ venv.bak/
.mypy_cache/
.dmypy.json
dmypy.json

# Editor configuration
.vscode/
1 change: 1 addition & 0 deletions .python-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3.9
87 changes: 85 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# httpx-retries
# HTTPX Retries


<p>
<a href="https://github.com/will-ockmore/httpx-retry/actions">
Expand All @@ -10,4 +11,86 @@
<!-- </a> -->
</p>

<em>A modern retry layer for HTTPX.</em
<!-- badges-end -->

<em>A retry layer for HTTPX.</em>


---

HTTPX Retries is a full implementation of request retry policies for HTTPX.

It's very common to deal with **flaky** and **unreliable** APIs. When requests fail, your program needs to be able
to retry them.

---

Install HTTPX Retries using pip:

``` bash
pip install httpx-retries
```

---

To get started, add the transport to your client:

``` python
import httpx
from httpx_retries import RetryTransport

with httpx.Client(transport=RetryTransport()) as client:
response = client.get("https://example.com")
```

For async usage:

``` python
async with httpx.AsyncClient(transport=RetryTransport()) as client:
response = await client.get("https://example.com")
```

If you want to use a specific retry strategy, provide a `Retry` configuration:

``` python
from httpx_retries import Retry

retry = Retry(total=5, backoff_factor=0.5)
transport = RetryTransport(retry=retry)

with httpx.Client(transport=transport) as client:
response = client.get("https://example.com")
```

## Features

HTTPX Retries builds on the patterns users will expect from `urllib` and `requests`. The typical approach has been
to use [urllib3's Retry](https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.Retry)
utility to configure a retry policy. The equivalent code to match the above example using
[requests](https://requests.readthedocs.io/en/latest/) is:

``` python
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(total=5, backoff_factor=0.5)
adapter = HTTPAdapter(max_retries=retry)

with requests.Session() as session:
session.mount("http://", adapter)
session.mount("https://", adapter)
response = session.get("https://example.com")
```

To reduce boilerplate, this package includes a transport that works with both sync and async HTTPX clients, so you don't have to explicitly define policies for simple use cases.

HTTPX adds support for asynchronous requests, so the package exposes a new retry utility. To make it easy to migrate, the API surface is almost identical, with a few main differences:

- `total` is the only parameter used to configure the number of retries.
- `asleep` is an async implementation of `sleep`.
- `backoff_strategy` can be overridden to customize backoff behavior.
- Some options that are not strictly retry-related are not included (`raise_on_status`, `raise_on_redirect`)

## Acknowledgements

This package builds on the great work done on [HTTPX](https://www.python-httpx.org/), [urllib3](https://urllib3.readthedocs.io/en/stable/) and [requests](https://requests.readthedocs.io/en/latest/).
2 changes: 2 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,5 @@
- RetryTransport
- AsyncRetryTransport
- Retry
filters:
- "!^_"
45 changes: 32 additions & 13 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@

HTTPX Retries is a full implementation of request retry policies for HTTPX.

It's very common to deal with **flaky** and **unreliable** APIs. When requests fail, applications need to be able
to resend them.
It's very common to deal with **flaky** and **unreliable** APIs. When requests fail, your program needs to be able
to retry them.

---

Expand All @@ -31,13 +31,28 @@ pip install httpx-retries

---

To get started, define your retry strategy and add the transport to your client.
To get started, add the transport to your client:

``` python
import httpx
from httpx_retries import Retry, RetryTransport
from httpx_retries import RetryTransport

retry = Retry(total=5, backoff_factor=0.5, respect_retry_after_header=False)
with httpx.Client(transport=RetryTransport()) as client:
response = client.get("https://example.com")
```

For async usage:
``` python
async with httpx.AsyncClient(transport=RetryTransport()) as client:
response = await client.get("https://example.com")
```

If you want to use a specific retry strategy, provide a [Retry][httpx_retries.Retry] configuration:

``` python
from httpx_retries import Retry

retry = Retry(total=5, backoff_factor=0.5)
transport = RetryTransport(retry=retry)

with httpx.Client(transport=transport) as client:
Expand All @@ -48,8 +63,8 @@ with httpx.Client(transport=transport) as client:

HTTPX Retries builds on the patterns users will expect from `urllib` and `requests`. The typical approach has been
to use [urllib3's Retry](https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.Retry)
utility to configure a retry policy. For example, with [requests](https://requests.readthedocs.io/en/latest/) the above
code becomes
utility to configure a retry policy. The equivalent code to match the above example using
[requests](https://requests.readthedocs.io/en/latest/) is:

``` python
from requests.adapters import HTTPAdapter
Expand All @@ -64,14 +79,18 @@ with requests.Session() as session:
response = session.get("https://example.com")
```

To reduce boilerplate, this package includes custom transports
([RetryTransport][httpx_retries.RetryTransport] and [AsyncRetryTransport][httpx_retries.AsyncRetryTransport]), so
you don't have to to explicitly define policies for simple use cases.
To reduce boilerplate, this package includes a transport that works with both sync and async HTTPX clients, so you don't have to explicitly define policies for simple use cases.

As HTTPX adds support for asynchronous requests, the package exposes a new retry
utility ([Retry][httpx_retries.Retry]). To make it easy to migrate, the API surface is almost identical, with a few main
differences:
HTTPX adds support for asynchronous requests, so the package exposes a new retry utility ([Retry][httpx_retries.Retry]). To make it easy to migrate, the API surface is almost identical, with a few main differences:

- `total` is the only parameter used to configure the number of retries.
- [asleep][httpx_retries.Retry.asleep] is an async implementation of [sleep][httpx_retries.Retry.sleep].
- [backoff_strategy][httpx_retries.Retry.backoff_strategy] can be overridden to customize backoff behavior.
- Some options that are not strictly retry-related are not included (`raise_on_status`, `raise_on_redirect`)

!!! note
For more information, visit the [API reference](./api.md).

## Acknowledgements

This package builds on the great work done on [HTTPX](https://www.python-httpx.org/), [urllib3](https://urllib3.readthedocs.io/en/stable/) and [requests](https://requests.readthedocs.io/en/latest/).
4 changes: 2 additions & 2 deletions httpx_retries/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from .retry import Retry
from .transport import AsyncRetryTransport, RetryTransport
from .transport import RetryTransport

__all__ = ["Retry", "RetryTransport", "AsyncRetryTransport"]
__all__ = ["Retry", "RetryTransport"]
65 changes: 31 additions & 34 deletions httpx_retries/retry.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,29 +39,19 @@ class Retry:
For complex use cases, you can override the `backoff_strategy` method.
```python
class CustomRetry(Retry):
def backoff_strategy(self) -> float:
# Custom backoff logic here
if self.attempts_made == 3:
return 1.0
return super().backoff_strategy()
```
Args:
total (int, optional): The maximum number of times to retry a request before giving up. Defaults to 10.
max_backoff_wait (float, optional): The maximum time to wait between retries in seconds. Defaults to 120.
total (int, optional): The maximum number of times to retry a request before giving up.
max_backoff_wait (float, optional): The maximum time in seconds to wait between retries.
backoff_factor (float, optional): The factor by which the wait time increases with each retry attempt.
Defaults to 0.
respect_retry_after_header (bool, optional): Whether to respect the Retry-After header in HTTP responses
when deciding how long to wait before retrying. Defaults to True.
when deciding how long to wait before retrying.
allowed_methods (Iterable[http.HTTPMethod, str], optional): The HTTP methods that can be retried. Defaults to
["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE"].
status_forcelist (Iterable[http.HTTPStatus, int], optional): The HTTP status codes that can be retried.
Defaults to [429, 502, 503, 504].
backoff_jitter (float, optional): The amount of jitter to add to the backoff time. Defaults to 1 (full jitter).
attempts_made (int, optional): The number of attempts already made. Defaults to 0.
backoff_jitter (float, optional): The amount of jitter to add to the backoff time, between 0 and 1.
Defaults to 1 (full jitter).
attempts_made (int, optional): The number of attempts already made.
"""

RETRYABLE_METHODS: Final[frozenset[HTTPMethod]] = frozenset(
Expand All @@ -75,20 +65,16 @@ def backoff_strategy(self) -> float:
HTTPStatus.GATEWAY_TIMEOUT,
]
)
DEFAULT_MAX_BACKOFF_WAIT: Final[float] = 120.0
DEFAULT_TOTAL_RETRIES: Final[int] = 10
DEFAULT_BACKOFF_FACTOR: Final[float] = 0.0
DEFAULT_BACKOFF_JITTER: Final[float] = 1.0

def __init__(
self,
total: int = DEFAULT_TOTAL_RETRIES,
total: int = 10,
allowed_methods: Optional[Iterable[Union[HTTPMethod, str]]] = None,
status_forcelist: Optional[Iterable[Union[HTTPStatus, int]]] = None,
backoff_factor: float = DEFAULT_BACKOFF_FACTOR,
backoff_factor: float = 0.0,
respect_retry_after_header: bool = True,
max_backoff_wait: float = DEFAULT_MAX_BACKOFF_WAIT,
backoff_jitter: float = DEFAULT_BACKOFF_JITTER,
max_backoff_wait: float = 120.0,
backoff_jitter: float = 1.0,
attempts_made: int = 0,
) -> None:
"""Initialize a new Retry instance."""
Expand All @@ -103,27 +89,27 @@ def __init__(
if attempts_made < 0:
raise ValueError("attempts_made must be non-negative")

self.max_attempts = total
self.total = total
self.backoff_factor = backoff_factor
self.respect_retry_after_header = respect_retry_after_header
self.max_backoff_wait = max_backoff_wait
self.backoff_jitter = backoff_jitter
self.attempts_made = attempts_made

self.retryable_methods = frozenset(
self.allowed_methods = frozenset(
HTTPMethod(method.upper()) for method in (allowed_methods or self.RETRYABLE_METHODS)
)
self.retry_status_codes = frozenset(
self.status_forcelist = frozenset(
HTTPStatus(int(code)) for code in (status_forcelist or self.RETRYABLE_STATUS_CODES)
)

def is_retryable_method(self, method: str) -> bool:
"""Check if a method is retryable."""
return HTTPMethod(method.upper()) in self.retryable_methods
return HTTPMethod(method.upper()) in self.allowed_methods

def is_retryable_status_code(self, status_code: int) -> bool:
"""Check if a status code is retryable."""
return HTTPStatus(status_code) in self.retry_status_codes
return HTTPStatus(status_code) in self.status_forcelist

def is_retry(self, method: str, status_code: int, has_retry_after: bool) -> bool:
"""
Expand All @@ -132,15 +118,15 @@ def is_retry(self, method: str, status_code: int, has_retry_after: bool) -> bool
This functions identically to urllib3's `Retry.is_retry` method.
"""
return (
self.max_attempts > 0
self.total > 0
and self.is_retryable_method(method)
and self.is_retryable_status_code(status_code)
and not has_retry_after
)

def is_exhausted(self) -> bool:
"""Check if the retry attempts have been exhausted."""
return self.attempts_made >= self.max_attempts
return self.attempts_made >= self.total

def parse_retry_after(self, retry_after: str) -> float:
"""
Expand Down Expand Up @@ -173,6 +159,17 @@ def backoff_strategy(self) -> float:
"""
Calculate the backoff time based on the number of attempts.
For complex use cases, you can override this method to implement a custom backoff strategy.
```python
class CustomRetry(Retry):
def backoff_strategy(self) -> float:
if self.attempts_made == 3:
return 1.0
return super().backoff_strategy()
```
Returns:
The calculated backoff time in seconds, capped by max_backoff_wait.
"""
Expand Down Expand Up @@ -213,12 +210,12 @@ async def asleep(self, response: httpx.Response) -> None:
def increment(self) -> "Retry":
"""Return a new Retry instance with the attempt count incremented."""
return Retry(
total=self.max_attempts,
total=self.total,
max_backoff_wait=self.max_backoff_wait,
backoff_factor=self.backoff_factor,
respect_retry_after_header=self.respect_retry_after_header,
allowed_methods=self.retryable_methods,
status_forcelist=self.retry_status_codes,
allowed_methods=self.allowed_methods,
status_forcelist=self.status_forcelist,
backoff_jitter=self.backoff_jitter,
attempts_made=self.attempts_made + 1,
)
Loading

0 comments on commit c370a1b

Please sign in to comment.