Prevent specifying video as short due to web-scraping abuse

[Benjamin-Loison]

As requested on Discord.

Related to #11.

[Benjamin-Loison]

grep -ri 'isRedirection('

Output:

common.php:    function isRedirection($url)
videos.php:                'available' => !isRedirection("https://www.youtube.com/shorts/$id")

[Benjamin-Loison]

grep -r 'getHeadersFromOpts'

Output:

common.php:    function getHeadersFromOpts($url, $opts)
common.php:		$http_response_header = getHeadersFromOpts($url, $opts);

[Benjamin-Loison]

grep -ri 'redirect' --exclude-dir={.git,tools}

Output:

common.php:    function getRedirection($url)
common.php:    function getJSONFromHTML($url, $opts = [], $scriptVariable = '', $prefix = 'var ', $forceLanguage = false, $verifiesChannelRedirection = false)
common.php:        if($verifiesChannelRedirection)
common.php:            $redirectedToChannelIdPath = 'onResponseReceivedActions/0/navigateAction/endpoint/browseEndpoint/browseId';
common.php:            if(doesPathExist($json, $redirectedToChannelIdPath))
common.php:                $redirectedToChannelId = getValue($json, $redirectedToChannelIdPath);
common.php:                $url = preg_replace('/[\w\-_]{24}/', $redirectedToChannelId, $url);
common.php:                // Does a redirection of redirection for a channel exist?
common.php:                return getJSONFromHTML($url, $opts, $scriptVariable, $prefix, $forceLanguage, $verifiesChannelRedirection);
common.php:                if (str_starts_with($url, 'https://www.youtube.com/redirect?')) {
channels.php:            $result = getJSONFromHTML("https://www.youtube.com/channel/$id", forceLanguage: true, verifiesChannelRedirection: true);
channels.php:            $result = getJSONFromHTML("https://www.youtube.com/channel/$id", forceLanguage: true, verifiesChannelRedirection: true);
channels.php:                $result = getJSONFromHTML("https://www.youtube.com/channel/$id/shorts", forceLanguage: true, verifiesChannelRedirection: true);
channels.php:                $result = getJSONFromHTML("https://www.youtube.com/channel/$id/community", forceLanguage: true, verifiesChannelRedirection: true);
channels.php:                $result = getJSONFromHTML("https://www.youtube.com/channel/$id/channels", forceLanguage: true, verifiesChannelRedirection: true);
channels.php:            $result = getJSONFromHTML("https://www.youtube.com/channel/$id/about", forceLanguage: true, verifiesChannelRedirection: true);
channels.php:            $result = getJSONFromHTML("https://www.youtube.com/channel/$id", forceLanguage: true, verifiesChannelRedirection: true);
channels.php:            $result = getJSONFromHTML("https://www.youtube.com/channel/$id", verifiesChannelRedirection: true);
channels.php:                $result = getJSONFromHTML("https://www.youtube.com/channel/$id/playlists", forceLanguage: true, verifiesChannelRedirection: true);
.htaccess:Redirect /matrix https://matrix.to/#/#youtube-operational-api:matrix.org
.htaccess:Redirect /discord https://discord.gg/pDzafhGWzf
.htaccess:Redirect /code https://github.com/Benjamin-Loison/YouTube-operational-API
.htaccess:Redirect /host-your-own-instance https://github.com/Benjamin-Loison/YouTube-operational-API/blob/main/README.md#install-your-own-instance-of-the-api
.htaccess:Redirect /issues https://github.com/Benjamin-Loison/YouTube-operational-API/issues
videos.php:                'available' => getRedirection("https://www.youtube.com/shorts/$id")

[Benjamin-Loison]

die(json_encode($http_response_header));

[Benjamin-Loison]

Bash script:

VIDEO_IDS=(
    ydPkyvWtmg4
    NiXD4xVJM5Y
)
for videoId in "${VIDEO_IDS[@]}"
do
    echo -n "$videoId: "
    curl -s "http://localhost/YouTube-operational-API/videos?part=short&id=$videoId" | jq .items[0].short.available
done

Output:

ydPkyvWtmg4: true
NiXD4xVJM5Y: false

[Benjamin-Loison]

Knowing that https://www.youtube.com/shorts/VIDEO_ID redirects to https://www.youtube.com/watch?v=VIDEO_ID only if it is not a short.

Is detected as spamming \ Video type	Video	Short
False	Redirected to https://www.youtube.com/watch?v=VIDEO_ID	Not redirected
True	Redirected to an arbitrary URL	Redirected to an arbitrary URL

We want available to be:

Is detected as spamming \ Video type	Video	Short
False	false	true
True	die	die

[Benjamin-Loison]

So the point is more about verifying the redirected URL to be as expected if any. So could refine web-scraping detected detection to not just rely on HTTP codes.

[Benjamin-Loison]

diff:

diff --git a/common.php b/common.php
index ef9d2f4..67d771a 100644
--- a/common.php
+++ b/common.php
@@ -84,7 +84,7 @@
         return [$result, $http_response_header];
     }
 
-    function isRedirection($url)
+    function getRedirection($url)
     {
         $opts = [
             'http' => [
@@ -97,7 +97,10 @@
         if (in_array($code, HTTP_CODES_DETECTED_AS_SENDING_UNUSUAL_TRAFFIC)) {
             detectedAsSendingUnusualTraffic();
         }
-        return $code == 303;
+        if ($code == 303) {
+            return $http_response_header['Location'];
+        }
+        return null;
     }
 
     function getRemote($url, $opts = [], $verifyTrafficIfForbidden = true)
diff --git a/videos.php b/videos.php
index 4328c1c..c8b3e53 100644
--- a/videos.php
+++ b/videos.php
@@ -171,7 +172,7 @@
 
         if ($options['short']) {
             $short = [
-                'available' => !isRedirection("https://www.youtube.com/shorts/$id")
+                'available' => getRedirection("https://www.youtube.com/shorts/$id") === null,
             ];
             $item['short'] = $short;
         }

[Benjamin-Loison]

Fix #330: Change `isRedirection` to `getRedirection`