Bugs in "Disable restrictions for this tab"

Bug reports and enhancement requests
Post Reply
skriptimaahinen
Master Bug Buster
Posts: 244
Joined: Wed Jan 10, 2018 7:37 am

Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen »

Ended up taking a closer look at how NoScript handles the "Disable restrictions for this tab".

Turns out the "cookie method" has couple of bugs and problems. Testing on https://www.theguardian.com shows several CSP errors in the log. These appear to be due to the following reasons:

In RequestGuard.js, LastListener for onHeadersReceived does not check for isEnforced, sometimes resulting in "Safety net" kicking in.

In staticNS.js, wrappedJSObject.checkNoScriptUnrestricted sometimes fails. Didn't take closer look but I gues it's because the reason stated in the codes comment.

There is also third reason why the cookie might be missing (besides non-http subframe or race-condition). If the request is fetched from cache only, writing the cookie in the header does not result in the cookie being set for the page. This can happen if Cache-Control is set to cache-only. In that case no validation request is sent, only webRequest formed is the one with fromCache=true and the problem occurs. (For the first load everything is ok, but for some reason some subframes (ads, trackers, etc.) tend to get reloaded quite often from the cache.)

Doubt that anyone really misses any trackers or ads, but since fixing this does not appear to be that straightforward (simplest fix to mess with the Cache-Control most likely would yield unwanted consequences), I decided to see if I could find any alternative for the cookie method.

First I ended up revisiting the old "tabs.executeScript on filterResponseData" and I'm pretty sure I managed to make it reliable (see below).

However, this method is not capable of handling cases where no webRequest is formed. For example ftp, file, about:blank and Service Worker pages.

So I ended up taking another look and this (monstrosity) (see below) is what came out of it. That should work anywhere you can get contentscript to run, but do let me know if I overlooked something.

Do note that this is tested only on FF68. No idea of Tor-browser or Chrome compatibility. Hope it proves useful though.
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
skriptimaahinen
Master Bug Buster
Posts: 244
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen »

Was going to upload the code to GitHub, but it turns out they had added some security measures and now my account is so secure that I can't log in anymore. Sigh. Pasting the code below because do not feel like creating a new account now.
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
skriptimaahinen
Master Bug Buster
Posts: 244
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen »

WebRequestFilterSuspendTest

manifest.json

Code: Select all

{

    "manifest_version": 2,
    "name": "WebrequestFilterSuspendTest",
    "version": "1.0",

    "description": "Tests WebRequest suspending to execute script.",

    "applications": {
        "gecko": {
            "id": "webrequestfiltersuspendtest@example.com",
            "strict_min_version": "60.0"
        }
    },

    "content_scripts": [
        {
          "run_at": "document_start",
          "matches": ["<all_urls>"],
          "match_about_blank": true,
          "all_frames": true,
          "js": ["contentscript.js"]
        }
    ],

    "permissions": [
        "webRequest",
        "webRequestBlocking",
        "<all_urls>"
    ],

    "background": {
        "scripts": ["bg.js"]
    }

}
bg.js

Code: Select all

function onHeadersReceived(e) {
    if (e.type === "main_frame" || e.type === "sub_frame") {

        // SUMMARY:

        // We start by running executeScript on filter.onstart, then proceed to
        // store data from filter.ondata until both our contentscript and
        // filter.onstop have executed, after which we write out the data and
        // let the page load.

        // Tried also suspending the filter on onstart and disconnecting after
        // the script got executed, but that led to some rare issues with the
        // window.onload not firing for the page and even more rare issues of
        // the page not appearing at all.

        // Tried to fix the above by resuming in onExecuted rather than disconnecting
        // and then writing the data in ondata and closing in onstop. That got
        // rid of the onload problem, but still was missing the entire page sometimes.
        // For some reason the ondata was not firing even though there were no
        // errors from the filter. 

        // Current iteration does not use suspend and everything seems reliable
        // so far.

        let filter = browser.webRequest.filterResponseData(e.requestId);
        let bufferArray = [];
        let scriptExecuted = false;

        function writeData() {
            for (let i = 0; i < bufferArray.length; i++) {
                filter.write(bufferArray[i]);
            }
            filter.close();
        }

        filter.onerror = event => {
            console.log(`Error: ${filter.error}`);
        }

        filter.onstop = event => {
            if (scriptExecuted) writeData();
        }

        filter.ondata = event => {
            bufferArray.push(event.data);
        }

        filter.onstart = event => {
            // Unfortunately we are always running bit early. So early that even
            // documentElement is not available, so we can't modify DOM yet.
            // However, we can reliably set variables and access them from
            // registered content scripts.
            let script = {
                runAt: "document_start",
                matchAboutBlank: true,
                allFrames: true,
                code: `var data_time = (new Date).getTime();`
            }

            function onScriptExecuted(result) {
                if (filter.status === "finishedtransferringdata") writeData();
                else scriptExecuted = true;
            }

            let retries = 1;
            function onScriptError(error) {
                // Only error I ever see here is the "No matching message handler"-error,
                // that happens if the script gets executed too early.
                // Allowing only one retry as that seems to always be enough.
                console.log("Error: "+error);
                if (retries > 0) {
                    retries--;
                    executeScript();
                } else {
                    // Give up and let page load. Fallback to failsafe.
                    onScriptExecuted("failed");
                }
            }

            function executeScript() {
                let executing = browser.tabs.executeScript(e.tabId, script);
                executing.then(onScriptExecuted, onScriptError);
            }

            executeScript();
        }
    }
    return {responseHeaders: e.responseHeaders};
}

browser.webRequest.onHeadersReceived.addListener(
    onHeadersReceived,
    {urls: ["<all_urls>"]},
    ["blocking", "responseHeaders"]
);
contentscript.js

Code: Select all

if (data_time) {
    let delta = (new Date).getTime() - data_time;
    console.log("Found data! Time difference is (ms): "+delta);
}
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
skriptimaahinen
Master Bug Buster
Posts: 244
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen »

SyncMsgExample

manifest.json

Code: Select all

{

    "manifest_version": 2,
    "name": "SyncMsgExample",
    "version": "1.0",
    "description": "Testing communication with background page through synchronous XHR and filterResponseData.",

    "applications": {
        "gecko": {
            "id": "syncmsgexample@example.com",
            "strict_min_version": "60.0"
        }
    },

    "permissions": [
        "contextualIdentities",
        "webRequest",
        "webRequestBlocking",
        "<all_urls>"
    ],

    "background": {
        "scripts": ["background.js"]
    }

}
background.js

Code: Select all

// Secret that is embedded to the xhr url so that we can ensure it is sent by us.
const uuid = "123456789";

async function register() {
    return await browser.contentScripts.register({
        "js": [{code: `var uuid = $uuid`}, {file: "script.js"}], // ensure uuid is available for the contentscript
        "matches": ["<all_urls>"],
        "matchAboutBlank": true,
        "allFrames": true,
        "runAt": "document_start"
    });
}
const registered = register();

var tabIdArray = [];

// Extracts helper id from the url.
function getHelperId(request) {
    let arr = request.url.split("/");
    return arr[arr.length-2]
}

// Returns: tabId from tabIdArray if helperId matches or tabId from request if no match is found.
// Stores tabId in the tabIdArray.
// Cleans up tabIdArray. (Should not be necessary on normal operation though.)
function checkTabId(request) {
    let retVal = request.tabId;
    let time = (new Date).getTime();
    let helperId = getHelperId(request);

    for (let i = tabIdArray.length - 1; i >= 0; --i) {
        let [id, tabId, timeStamp] = tabIdArray[i];
        if (helperId === id) {
            tabIdArray.splice(i, 1);
            retVal = tabId;
            continue;
        }
        if (time > timeStamp + 20000) { // if older than 20s (placeholder value, adjust based on user experience)
            tabIdArray.splice(i, 1);
            continue;
        }
    }
    if (request.tabId > -1) {
        tabIdArray.push([helperId, request.tabId, time]);
    }
    return retVal;
}

function onBeforeRequest(request) {
    if (request.type === "xmlhttprequest" && request.url.endsWith(uuid)) {
        let tabId = request.tabId; // case http(s) page
        if (request.url.startsWith("https://invalid.example")) { // case ftp, file or about:blank
            tabId = checkTabId(request);
            if (request.tabId > -1) return {cancel: true}; // for 1st XHR. CORS would cause filter to fail anyway.
        }
        let filter = browser.webRequest.filterResponseData(request.requestId);
        filter.onstart = event => {
            let encoder = new TextEncoder();
            filter.write(encoder.encode(`{"tabId": "$tabId"}`)); // Payload to contentscript goes here.
            filter.close();
        }        
        // Just pointing it to somewhere that's supported but not necessarily allowed.
        // We just need it to cause request response that filterResponseData can overwrite.
        return {redirectUrl: "resource://"};
    }
}

browser.webRequest.onBeforeRequest.addListener(
    onBeforeRequest,
    {urls: ["<all_urls>"]},
    ["blocking"]
);
script.js

Code: Select all

// SUMMARY:

// To fetch info synchronously from background, we use synchronous XMLHttpRequest
// that is redirected to internal uri but the response is crafted with filterResponseData.

// helperId links the two XHR requests below, so that they don't mix up with
// requests from other tabs.
const helperId = Math.floor(Math.random() * 1000000);

// Need to use "content" here to get the real tabId but unfortunately this
// forces pages CORS rules on us.
let request = new content.XMLHttpRequest();
// Using same origin to avoid CORS error (for http(s)).
let url = document.location.origin;
// In case ftp, file or about:blank. Will fail due to CORS,
// but these protocols would fail anyhow for various reasons.
if (!document.location.protocol.startsWith("http")) url = "https://invalid.example";
// (Gets intercepted and never reaches the server.)
request.open("GET", url+"/"+helperId+"/"+uuid, false);
try { request.send(null); }
catch (e) {} // Throws DOMException:NetworkError every time. We just ignore it.
let response = request.responseText;
console.log(response);

// For http(s) request we have our data in the response already. However, if we
// are on ftp, file or about:blank page, there is no response and we need to
// make another round-trip to background to fetch the tabId we just stored there.

if (!response) {
    // Using XMLHttpRequest with content script privileges to avoid page CORS
    // rules so filterResponseData wont fail while using bogus url.
    let request = new XMLHttpRequest(); // tabId is always -1 (bug?)
    // (Also intercepted and never goes anywhere.)
    request.open('GET', "https://invalid.example/"+helperId+"/"+uuid, false);
    try { request.send(null); }
    catch (e) {}
    console.log(request.responseText);
}
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: Bugs in "Disable restrictions for this tab"

Post by Giorgio Maone »

Thank you very much, that's very interesting.
Actually using asyncrhonous XHR was something I was keeping as a last resort since my first attempts to a Chromium porting, and my original idea was trying to redirect to a data: URI, since filterResponseData() is Firefox-only.
Did you try something like that or already know it wouldn't work?
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0
skriptimaahinen
Master Bug Buster
Posts: 244
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen »

Couldn't remember if I had tried data-uris, so I made a quick test.

XMLHttpRequest accepts them fine but there appears to be no way to modify the request or redirect it to another data- or resource-uri. Filter has no effect and neither onBeforeSendHeaders or onHeadersReceived is fired.

However, redirecting (non-data) request to data-uri does work for XMLHttpRequest, but not for content.XMLHttpRequest. This means that it is possible to do away with the filterResponseData, but every request, including http(s), would need to go through the two XMLHttpRequests. I guess this is valid approach for Chrome if there is no alternative for the filterResponseData.
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
skriptimaahinen
Master Bug Buster
Posts: 244
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen »

Seems this was a short lived hack. That is, if Mozilla does as it says and implements the Cross-origin communication part of Manifest v3. Which means that there is no using bogus url with XHR due to CORS restrictions even in contentscript context and thus no way to get response on ftp, file or about:blank pages. :(
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: Bugs in "Disable restrictions for this tab"

Post by Giorgio Maone »

It really looks we need synchronous messaging initiated from content (just like in the e10s API), as I advocated for years (since the Chrome extensions API was initially designed).
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: Bugs in "Disable restrictions for this tab"

Post by Giorgio Maone »

Please check latest dev build:

v 11.0.4rc2
=============================================================
x Recursive webgl context monkeypatching across same origin
windows (thanks skriptimaahinen for concept and patch)
x Replaced cookie-based hacks with synchronous messaging
(currently shimmed) to retrieve fallback and
per-tab restriction policies

x Work-around for Chromium not supporting frameAncestors
in webRequest
x Block CSP violation reports requests synchronously,
x before they fail on .invalid DNS resolution, on Chromium

As you can notice I wrapped our XHR-based hack (which works much better on Chromium than on Firefox because their "privileged" XHR actually sends out the tab id, requiring just one request) in a POC shim for a browser.runtime.sendSyncMessage / browser.runtime.onSyncMessage API proposal.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0
skriptimaahinen
Master Bug Buster
Posts: 244
Joined: Wed Jan 10, 2018 7:37 am

Re: Bugs in "Disable restrictions for this tab"

Post by skriptimaahinen »

So looks like I overlooked some cases (such as the session restore) with the file- and ftp-protocol. Also found it interesting that using the async call like that actually worked. So it appears that the xmlhttprequest is not quite so synchronous in every case after all.

Armed with that knowledge, here is the latest iteration to fix the problems:

Code: Select all

// SUMMARY:
// First we setup regular async call for our messaging needs.
// Secondly we setup mutationobserver to wait for the head.
// If the head is created before our async call completes, we start
// a XHR loop to block the DOM parsing until our async call is done.

// This seems to work as although the XHR request is synchronous, it releases
// the execution so that async calls (e.g. our background call) get chance to
// run. The DOM blocking is necessary to prevent DOM from completing and the
// onload event from firing before we know if we can allow the event listeners
// to run.

// Sometimes for file-protocol (did not see this with http(s) or ftp) injecting
// the CSP in the mutationobserver (even without the XHR loop) would not block
// the scripts in the head (did block scripts in body though. bug?).

// So as a backup I use beforescriptexecute and the same XHR loop to block the
// script execution too. Hopefully this is not issue with Chrome. I noticed that
// someone had made a polyfill beforescriptexecute for Chrome using
// mutationobserver, so the chance is good.

let response; // from background
let done = false; // stop condition for the DOM blocking XHR loop
let block = true; // whether we decide to block scripts or not

function handleResponse(message) {
    response = message;
    block = response.block;
    observer.disconnect();
    // The listener should not be removed at this point if we wish to block
    // as each script in the head is in queue waiting to execute.
    if (!block) document.removeEventListener("beforescriptexecute", beforescriptexecute, true);  
    if (block) insertCSP(); // etc.
    done = true; // stop loop
}

// Regular async call to background.
browser.runtime.sendMessage({}).then(handleResponse);

// The mutationobserver waits for the head.
// Sometimes the async call completes before this happens, but often not.
function callback(mutationList, observer) {
    mutationList.forEach((mutation) => {
        mutation.addedNodes.forEach((node) => {
            if (node.nodeName.toLowerCase() === "head") {
                while (!done) {
                    // Both priviledged and unpriviledged XHR calls work,
                    // though priviledged does not produce warnings.
                    // We expect no answer and simply block it in the background.
                    let request = new XMLHttpRequest();
                    request.open('GET', "https://invalid.example/", false);
                    try { request.send(null); }
                    catch (e) {}
                }
                // In case something goes very wrong and we get no response
                // from the async call, here is our chance for fallback.
                // (Might want additional stop condition for that.)
                // Never seen the loop do more that one XHR call, except
                // when introducing artificial lag to the async call.
                if (!response) insertCSP(); // fallback
            }
        });
    });
}

let targetNode = document.documentElement;
let observerOptions = { childList: true }
let observer = new MutationObserver(callback);
observer.observe(targetNode, observerOptions);

function beforescriptexecute(e) {
    while (!done) {
        let request = new XMLHttpRequest();
        request.open('GET', "https://invalid.example/", false);
        try { request.send(null); }
        catch (e) {}
    }
    if (block) e.preventDefault();
}

document.addEventListener("beforescriptexecute", beforescriptexecute, true);
Hopefully I didn't miss anything this time, but let me know if you notice any problems.

In case you want to test out the problem with the file-protocol, it seems to happen mainly when session-restoring, using arrow-buttons to navigate back to the page or duplicating the tab. Have not seen it happen while normally opening or reloading the page.
Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: Bugs in "Disable restrictions for this tab"

Post by Giorgio Maone »

skriptimaahinen wrote: Thu Oct 17, 2019 8:30 am So looks like I overlooked some cases (such as the session restore) with the file- and ftp-protocol.
I've incorporated some of these ideas in the SyncMessage shim. Could you check latest dev build (rc12) which seems to work great both on Firefox and Chromium, and PM me if you find any specific trouble?
Thank you so much!
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0
User avatar
therube
Ambassador
Posts: 7924
Joined: Thu Mar 19, 2009 4:17 pm
Location: Maryland USA

Re: Bugs in "Disable restrictions for this tab"

Post by therube »

(Did you mean to link to github?)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.19) Gecko/20110420 SeaMonkey/2.0.14 Pinball NoScript FlashGot AdblockPlus
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.5
Post Reply