Bugs in "Disable restrictions for this tab"

Bug reports and enhancement requests
Post Reply
skriptimaahinen
Senior Member
Posts: 158
Joined: Wed Jan 10, 2018 7:37 am

Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen » Mon Sep 02, 2019 7:37 am

Ended up taking a closer look at how NoScript handles the "Disable restrictions for this tab".

Turns out the "cookie method" has couple of bugs and problems. Testing on https://www.theguardian.com shows several CSP errors in the log. These appear to be due to the following reasons:

In RequestGuard.js, LastListener for onHeadersReceived does not check for isEnforced, sometimes resulting in "Safety net" kicking in.

In staticNS.js, wrappedJSObject.checkNoScriptUnrestricted sometimes fails. Didn't take closer look but I gues it's because the reason stated in the codes comment.

There is also third reason why the cookie might be missing (besides non-http subframe or race-condition). If the request is fetched from cache only, writing the cookie in the header does not result in the cookie being set for the page. This can happen if Cache-Control is set to cache-only. In that case no validation request is sent, only webRequest formed is the one with fromCache=true and the problem occurs. (For the first load everything is ok, but for some reason some subframes (ads, trackers, etc.) tend to get reloaded quite often from the cache.)

Doubt that anyone really misses any trackers or ads, but since fixing this does not appear to be that straightforward (simplest fix to mess with the Cache-Control most likely would yield unwanted consequences), I decided to see if I could find any alternative for the cookie method.

First I ended up revisiting the old "tabs.executeScript on filterResponseData" and I'm pretty sure I managed to make it reliable (see below).

However, this method is not capable of handling cases where no webRequest is formed. For example ftp, file, about:blank and Service Worker pages.

So I ended up taking another look and this (monstrosity) (see below) is what came out of it. That should work anywhere you can get contentscript to run, but do let me know if I overlooked something.

Do note that this is tested only on FF68. No idea of Tor-browser or Chrome compatibility. Hope it proves useful though.
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

skriptimaahinen
Senior Member
Posts: 158
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen » Mon Sep 02, 2019 7:38 am

Was going to upload the code to GitHub, but it turns out they had added some security measures and now my account is so secure that I can't log in anymore. Sigh. Pasting the code below because do not feel like creating a new account now.
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

skriptimaahinen
Senior Member
Posts: 158
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen » Mon Sep 02, 2019 7:48 am

WebRequestFilterSuspendTest

manifest.json

Code: Select all

{

    "manifest_version": 2,
    "name": "WebrequestFilterSuspendTest",
    "version": "1.0",

    "description": "Tests WebRequest suspending to execute script.",

    "applications": {
        "gecko": {
            "id": "webrequestfiltersuspendtest@example.com",
            "strict_min_version": "60.0"
        }
    },

    "content_scripts": [
        {
          "run_at": "document_start",
          "matches": ["<all_urls>"],
          "match_about_blank": true,
          "all_frames": true,
          "js": ["contentscript.js"]
        }
    ],

    "permissions": [
        "webRequest",
        "webRequestBlocking",
        "<all_urls>"
    ],

    "background": {
        "scripts": ["bg.js"]
    }

}
bg.js

Code: Select all

function onHeadersReceived(e) {
    if (e.type === "main_frame" || e.type === "sub_frame") {

        // SUMMARY:

        // We start by running executeScript on filter.onstart, then proceed to
        // store data from filter.ondata until both our contentscript and
        // filter.onstop have executed, after which we write out the data and
        // let the page load.

        // Tried also suspending the filter on onstart and disconnecting after
        // the script got executed, but that led to some rare issues with the
        // window.onload not firing for the page and even more rare issues of
        // the page not appearing at all.

        // Tried to fix the above by resuming in onExecuted rather than disconnecting
        // and then writing the data in ondata and closing in onstop. That got
        // rid of the onload problem, but still was missing the entire page sometimes.
        // For some reason the ondata was not firing even though there were no
        // errors from the filter. 

        // Current iteration does not use suspend and everything seems reliable
        // so far.

        let filter = browser.webRequest.filterResponseData(e.requestId);
        let bufferArray = [];
        let scriptExecuted = false;

        function writeData() {
            for (let i = 0; i < bufferArray.length; i++) {
                filter.write(bufferArray[i]);
            }
            filter.close();
        }

        filter.onerror = event => {
            console.log(`Error: ${filter.error}`);
        }

        filter.onstop = event => {
            if (scriptExecuted) writeData();
        }

        filter.ondata = event => {
            bufferArray.push(event.data);
        }

        filter.onstart = event => {
            // Unfortunately we are always running bit early. So early that even
            // documentElement is not available, so we can't modify DOM yet.
            // However, we can reliably set variables and access them from
            // registered content scripts.
            let script = {
                runAt: "document_start",
                matchAboutBlank: true,
                allFrames: true,
                code: `var data_time = (new Date).getTime();`
            }

            function onScriptExecuted(result) {
                if (filter.status === "finishedtransferringdata") writeData();
                else scriptExecuted = true;
            }

            let retries = 1;
            function onScriptError(error) {
                // Only error I ever see here is the "No matching message handler"-error,
                // that happens if the script gets executed too early.
                // Allowing only one retry as that seems to always be enough.
                console.log("Error: "+error);
                if (retries > 0) {
                    retries--;
                    executeScript();
                } else {
                    // Give up and let page load. Fallback to failsafe.
                    onScriptExecuted("failed");
                }
            }

            function executeScript() {
                let executing = browser.tabs.executeScript(e.tabId, script);
                executing.then(onScriptExecuted, onScriptError);
            }

            executeScript();
        }
    }
    return {responseHeaders: e.responseHeaders};
}

browser.webRequest.onHeadersReceived.addListener(
    onHeadersReceived,
    {urls: ["<all_urls>"]},
    ["blocking", "responseHeaders"]
);
contentscript.js

Code: Select all

if (data_time) {
    let delta = (new Date).getTime() - data_time;
    console.log("Found data! Time difference is (ms): "+delta);
}
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

skriptimaahinen
Senior Member
Posts: 158
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen » Mon Sep 02, 2019 7:51 am

SyncMsgExample

manifest.json

Code: Select all

{

    "manifest_version": 2,
    "name": "SyncMsgExample",
    "version": "1.0",
    "description": "Testing communication with background page through synchronous XHR and filterResponseData.",

    "applications": {
        "gecko": {
            "id": "syncmsgexample@example.com",
            "strict_min_version": "60.0"
        }
    },

    "permissions": [
        "contextualIdentities",
        "webRequest",
        "webRequestBlocking",
        "<all_urls>"
    ],

    "background": {
        "scripts": ["background.js"]
    }

}
background.js

Code: Select all

// Secret that is embedded to the xhr url so that we can ensure it is sent by us.
const uuid = "123456789";

async function register() {
    return await browser.contentScripts.register({
        "js": [{code: `var uuid = $uuid`}, {file: "script.js"}], // ensure uuid is available for the contentscript
        "matches": ["<all_urls>"],
        "matchAboutBlank": true,
        "allFrames": true,
        "runAt": "document_start"
    });
}
const registered = register();

var tabIdArray = [];

// Extracts helper id from the url.
function getHelperId(request) {
    let arr = request.url.split("/");
    return arr[arr.length-2]
}

// Returns: tabId from tabIdArray if helperId matches or tabId from request if no match is found.
// Stores tabId in the tabIdArray.
// Cleans up tabIdArray. (Should not be necessary on normal operation though.)
function checkTabId(request) {
    let retVal = request.tabId;
    let time = (new Date).getTime();
    let helperId = getHelperId(request);

    for (let i = tabIdArray.length - 1; i >= 0; --i) {
        let [id, tabId, timeStamp] = tabIdArray[i];
        if (helperId === id) {
            tabIdArray.splice(i, 1);
            retVal = tabId;
            continue;
        }
        if (time > timeStamp + 20000) { // if older than 20s (placeholder value, adjust based on user experience)
            tabIdArray.splice(i, 1);
            continue;
        }
    }
    if (request.tabId > -1) {
        tabIdArray.push([helperId, request.tabId, time]);
    }
    return retVal;
}

function onBeforeRequest(request) {
    if (request.type === "xmlhttprequest" && request.url.endsWith(uuid)) {
        let tabId = request.tabId; // case http(s) page
        if (request.url.startsWith("https://invalid.example")) { // case ftp, file or about:blank
            tabId = checkTabId(request);
            if (request.tabId > -1) return {cancel: true}; // for 1st XHR. CORS would cause filter to fail anyway.
        }
        let filter = browser.webRequest.filterResponseData(request.requestId);
        filter.onstart = event => {
            let encoder = new TextEncoder();
            filter.write(encoder.encode(`{"tabId": "$tabId"}`)); // Payload to contentscript goes here.
            filter.close();
        }        
        // Just pointing it to somewhere that's supported but not necessarily allowed.
        // We just need it to cause request response that filterResponseData can overwrite.
        return {redirectUrl: "resource://"};
    }
}

browser.webRequest.onBeforeRequest.addListener(
    onBeforeRequest,
    {urls: ["<all_urls>"]},
    ["blocking"]
);
script.js

Code: Select all

// SUMMARY:

// To fetch info synchronously from background, we use synchronous XMLHttpRequest
// that is redirected to internal uri but the response is crafted with filterResponseData.

// helperId links the two XHR requests below, so that they don't mix up with
// requests from other tabs.
const helperId = Math.floor(Math.random() * 1000000);

// Need to use "content" here to get the real tabId but unfortunately this
// forces pages CORS rules on us.
let request = new content.XMLHttpRequest();
// Using same origin to avoid CORS error (for http(s)).
let url = document.location.origin;
// In case ftp, file or about:blank. Will fail due to CORS,
// but these protocols would fail anyhow for various reasons.
if (!document.location.protocol.startsWith("http")) url = "https://invalid.example";
// (Gets intercepted and never reaches the server.)
request.open("GET", url+"/"+helperId+"/"+uuid, false);
try { request.send(null); }
catch (e) {} // Throws DOMException:NetworkError every time. We just ignore it.
let response = request.responseText;
console.log(response);

// For http(s) request we have our data in the response already. However, if we
// are on ftp, file or about:blank page, there is no response and we need to
// make another round-trip to background to fetch the tabId we just stored there.

if (!response) {
    // Using XMLHttpRequest with content script privileges to avoid page CORS
    // rules so filterResponseData wont fail while using bogus url.
    let request = new XMLHttpRequest(); // tabId is always -1 (bug?)
    // (Also intercepted and never goes anywhere.)
    request.open('GET', "https://invalid.example/"+helperId+"/"+uuid, false);
    try { request.send(null); }
    catch (e) {}
    console.log(request.responseText);
}
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

User avatar
Giorgio Maone
Site Admin
Posts: 8701
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: Bugs in "Disable restrictions for this tab"

Post by Giorgio Maone » Mon Sep 02, 2019 8:21 pm

Thank you very much, that's very interesting.
Actually using asyncrhonous XHR was something I was keeping as a last resort since my first attempts to a Chromium porting, and my original idea was trying to redirect to a data: URI, since filterResponseData() is Firefox-only.
Did you try something like that or already know it wouldn't work?
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0

skriptimaahinen
Senior Member
Posts: 158
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen » Tue Sep 03, 2019 6:42 am

Couldn't remember if I had tried data-uris, so I made a quick test.

XMLHttpRequest accepts them fine but there appears to be no way to modify the request or redirect it to another data- or resource-uri. Filter has no effect and neither onBeforeSendHeaders or onHeadersReceived is fired.

However, redirecting (non-data) request to data-uri does work for XMLHttpRequest, but not for content.XMLHttpRequest. This means that it is possible to do away with the filterResponseData, but every request, including http(s), would need to go through the two XMLHttpRequests. I guess this is valid approach for Chrome if there is no alternative for the filterResponseData.
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

skriptimaahinen
Senior Member
Posts: 158
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen » Thu Sep 05, 2019 10:20 am

Seems this was a short lived hack. That is, if Mozilla does as it says and implements the Cross-origin communication part of Manifest v3. Which means that there is no using bogus url with XHR due to CORS restrictions even in contentscript context and thus no way to get response on ftp, file or about:blank pages. :(
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0

User avatar
Giorgio Maone
Site Admin
Posts: 8701
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: Bugs in "Disable restrictions for this tab"

Post by Giorgio Maone » Thu Sep 05, 2019 11:33 am

It really looks we need synchronous messaging initiated from content (just like in the e10s API), as I advocated for years (since the Chrome extensions API was initially designed).
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0

Post Reply