Adds user-agent to the request, if it doesn't exist in the headers.

[Geosynopsis]

Related Issue(s):

Description:
Some servers block the request without user-agent. This PR adds user-agent to headers if it doesn't already exist (through headers override).

PR Checklist:

• pre-commit hooks pass locally
• Tests pass (run scripts/test)
• Documentation has been updated to reflect changes, if applicable
• This PR maintains or improves overall codebase code coverage.
• Changes are added to the CHANGELOG. See the docs for information about adding to the changelog.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (cc71fc3) 91.12% compared to head (2781b24) 91.13%.

❗ Current head 2781b24 differs from pull request most recent head 6cf86b0. Consider uploading reports for the commit 6cf86b0 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1292      +/-   ##
==========================================
+ Coverage   91.12%   91.13%   +0.01%     
==========================================
  Files          51       51              
  Lines        6994     7002       +8     
  Branches     1001     1002       +1     
==========================================
+ Hits         6373     6381       +8     
  Misses        444      444              
  Partials      177      177              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gadomski]

@Geosynopsis is this change necessary? Working from current main, it appears as though urllib is sending a user agent already.

Client code:

from pystac.stac_io import DefaultStacIO

stac_io = DefaultStacIO()
stac_io.read_text("http://localhost:7822/catalog.json")

To test:

$ nc -l 7822 & python test_1292.py && fg
[2] 48657
GET /catalog.json HTTP/1.1
Accept-Encoding: identity
Host: localhost:7822
User-Agent: Python-urllib/3.11
Connection: close

-- >8 --

Note the User-Agent: Python-urllib/3.11 in the request.

[Geosynopsis]

@gadomski It seems to be the issue with the URL with https behind cloudflare. Could you give it a try for this https://www.cloudflare.com/en-gb/. You should get 403 Forbidden without the User-agent.

[gadomski]

It seems to be the issue with the URL with https behind cloudflare.

That shouldn't be a pystac issue then -- sounds like the User-Agent header might be getting stripped on the way through? It would be very surprising if urllib didn't send User-Agent for https requests but did for http requests.

[Geosynopsis]

It seems to be the issue with the URL with https behind cloudflare.

That shouldn't be a pystac issue then -- sounds like the User-Agent header might be getting stripped on the way through? It would be very surprising if urllib didn't send User-Agent for https requests but did for http requests.

Yeah, you're right it shouldn't be pystac issue. I'll close the pr. Either the User-Agent is stripped or the Cloudflare blocks Python-urllib agent.