You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If I did not miss anything, PFERD as of now has no feature to test / try out what files a crawler would download (and how it would save those) without actually downloading them. I would find this very helpful so while creating my configuration file, I can re-assure (to some extend) that PFERD does not fill up my drive with a bunch of files I did not account for.
The text was updated successfully, but these errors were encountered:
After a successful crawl, you can use --debug-transforms to test changes to your transforms. While this is not a full dry run, it can help in some cases.
Thats good to know, but I have already seen that.
I was more interested into using that dry-run feature before downloading a whole new course and especially to avoid producing unnecessary load & using up too much storage when e.g. the course contains unexpectedly big files or (which I have not seen yet) just in case PFERD misclassifies video files and downloads then anyway while having video=no set.
If I did not miss anything, PFERD as of now has no feature to test / try out what files a crawler would download (and how it would save those) without actually downloading them. I would find this very helpful so while creating my configuration file, I can re-assure (to some extend) that PFERD does not fill up my drive with a bunch of files I did not account for.
The text was updated successfully, but these errors were encountered: