check_library_properties fails with 403 errors for all URLs that are on arduino.cc

[SpenceKonde]

This used to work, as of the past few days, i'm failing library properties test every time. Between this and the fact that checklinks is blocked by microchip, I have to manually check around 100 URLs for EVERY RELEASE OF EVERY CORE. Is there any way to fix this? If not, we need to get rid of the tests that are failing with forbidden errors because the sites (you know, from organizations that benefit from a vibrant Arduino community) have BLOCKED US FROM CHECKING THE LINKS

I am considering simply removing every link to microchip.com because I have no way to verify that it's valid, but even a single bad link ,makes people think you're a total loser and that your work isn't worth further examination. But now not even Arduino allows us to check their website?! What the hell is this? This is destroying my ability to release a quality core.

[per1234]

Hi @SpenceKonde. Thanks for your report.

As someone who maintains automated link checking infrastructure in quite a few projects, I understand the frustration of spurious failures.

The script is running as intended. You can see the command used for the link check here:

arduino-ci-script/arduino-ci-script.sh

Line 1886 in 9f37f6b

urlStatus=$(curl --location --request GET --output /dev/null --silent --head --write-out '%{http_code}' "$urlValue")

If you run the command directly, you will get the same result. Here we have an example of a spurious failure:

https://github.com/SpenceKonde/ATTinyCore/actions/runs/5339900593/jobs/9679095711

ERROR: /home/runner/work/ATTinyCore/ATTinyCore/avr/libraries/Wire/library.properties: url value http://www.arduino.cc/en/Reference/Wire returned error status 403.

The command the script ran:

$ curl --location --request GET --output /dev/null --silent --head --write-out '%{http_code}' "http://www.arduino.cc/en/Reference/Wire"

'403'

In this case, there was a simple workaround, which was to change the http scheme to https:

$ curl --location --request GET --output /dev/null --silent --head --write-out '%{http_code}' "http://www.arduino.cc/en/Reference/Wire"

'302'

(the 302 HTTP response code indicates there is a redirect for the URL, and is not treated as a failure by the script)

Or better yet, updating the URL to final target:

curl --location --request GET --output /dev/null --silent --head --write-out '%{http_code}' "https://www.arduino.cc/reference/en/language/functions/communication/wire/"

'200'

I submitted PRs to update all these outdated URLs in your platforms, which should serve as an effective mitigation of the problem:

• Update redirecting URLs in bundled library metadata SpenceKonde/ATTinyCore#793
• Update redirecting URLs in bundled library metadata SpenceKonde/megaTinyCore#987
• Update redirecting URLs in bundled library metadata SpenceKonde/DxCore#458

Even though there was a simple workaround in this case, it is true that it is possible for there to be other cases where there is no way to workaround a spurious failure by adjusting the URL. In this case, it is useful to have a sophisticated configuration system for the link check that allows the user to skip checks on certain URL patterns, customize which response codes are treated as failures, etc.

An example of such a system is that provided by the excellent markdown-link-check tool:

https://github.com/tcort/markdown-link-check#config-file-format

The library.properties validation capability of this script is now redundant to the far superior Arduino Lint tool:

https://arduino.github.io/arduino-lint/dev/rules/library/#dead-url-lp042

I have no plans to invest any time into further development of the script features that are provided by Arduino Lint so such a configuration system will not be added to this script. If we receive reports that Arduino Lint users are experiencing real spurious link check failures then work in that codebase to mitigate the problem would be appropriate.

I will leave the issue open to document this known limitation of the script even though it will not be resolved. If users of this script are affected by unavoidable spurious link check failures, they can make the necessary modifications to the script, which might be as simple as adjusting the regular expression here:

arduino-ci-script/arduino-ci-script.sh

Line 1887 in 9f37f6b

local errorStatusRegex='^[045]'