Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail crawl if custom behavior cannot be fetched #797

Closed
tw4l opened this issue Mar 26, 2025 · 0 comments · Fixed by #799
Closed

Fail crawl if custom behavior cannot be fetched #797

tw4l opened this issue Mar 26, 2025 · 0 comments · Fixed by #799
Assignees
@tw4l

Comments

@tw4l
Copy link
Member

tw4l commented Mar 26, 2025 •
edited
Loading

Currently, the crawler will log a warning and continue without the custom behavior if a custom behavior cannot be loaded from the specified URL or Git repository. This can result in unexpected crawl results. Instead, the crawler should log an error and exit if a custom behavior cannot be loaded.

We should do something similar for custom behaviors specified via a local filepath, namely fail if if no custom behaviors are collected or the specified filepath doesn't exist.
@tw4l tw4l self-assigned this Mar 26, 2025
@tw4l tw4l moved this from Triage to Implementing in Webrecorder Projects Mar 26, 2025
@tw4l tw4l mentioned this issue Mar 26, 2025
Merged
@tw4l tw4l changed the title Fail crawl if custom behavior cannot be fetched from URL/Git repo Fail crawl if custom behavior cannot be fetched Mar 26, 2025
@tw4l tw4l moved this from Implementing to In Review in Webrecorder Projects Mar 26, 2025
ikreymer pushed a commit that referenced this issue Apr 1, 2025
@tw4l
Fail crawl with fatal message if custom behavior isn't loaded (#799)
5fedde6 
Fixes #797 

The crawler will now exit with a fatal log message and exit code 17 if:

- A Git repository specified with `--customBehavior` cannot be cloned
successfully (new)
- A custom behavior file at a URL specified with `--customBehavior` is
not fetched successfully (new)
- No custom behaviors are collected at a local filepath specified with
`--customBehavior`, or if an error is thrown while attempting to collect
files from a nonexistent path (new)
- Any custom behaviors collected fail `Browser.checkScript` validation
(existing behavior)

Tests have also been added accordingly.
@github-project-automation github-project-automation bot moved this from In Review to Done! in Webrecorder Projects Apr 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tw4l tw4l

Labels
None yet
Projects
Webrecorder Projects
Status: Done!
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fail crawl with fatal message if custom behavior isn't loaded webrecorder/browsertrix-crawler
1 participant
@tw4l