[RESOLVED] Referring URL problem with NoScript and Google

[Darkoshi]

To recreate:
- Open a new Firefox window
- Select Google as the search engine in the Search Bar
- Enter a search string such as "abc" in the Search Bar and press Enter.
- The Google page displays with search results. The page URL looks like this:
http://www.google.com/search?q=abc&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a

- Over-type the original search string in the search box at the top of the Google page with a new search string such as "xyz" (don't enter it in the Firefox Search bar) and press Enter.

- Extra data gets added to the end of the page URL, and the Google page refreshes with new search results.
The extra data starts with a hash sign. That might be related to the problem.

- Open one of the result pages. Right-click and select "View Page Info" to check the referring URL

With NoScript enabled, the referring URL matches the URL example above and includes the original search string (it shouldn't include it).
With NoScript disabled, the referring URL does not include the original search string.

I've updated to the latest development build of NoScript (v2.3.6.rc1), but the problem persists.
I've tried disabling all other add-ons except for NoScript. The problem persists. (But if I disable NoScript, the problem goes away).
I've tried changing various NoScript Options, including a test to allow scripts globally, but the problem has persisted.

I've tested on another computer which did not originally have NoScript installed. It did not have the problem to begin with, but after installing NoScript (with the default options), the problem occurs on it too.

I do not have any of the add-ons which the general troubleshooting post says are buggy.
My Error Console does not show any errors, only warnings and messages.

[Tom T.]

To recreate:
- Open a new Firefox window
- Select Google as the search engine in the Search Bar
- Enter a search string such as "abc" in the Search Bar and press Enter.
- The Google page displays with search results. The page URL looks like this:

Code: Select all

http://www.google.com/search?q=abc&ie=u ... =firefox-a

Not for me. There is more info after the firefox-a, but it's too long for the bar. So you have to move the cursor until you can see the end of the string.

- Over-type the original search string in the search box at the top of the Google page with a new search string such as "xyz" (don't enter it in the Firefox Search bar) and press Enter.

- Extra data gets added to the end of the page URL,

Not for me; it matches the length and number of parameters as the first search. (con't)

and the Google page refreshes with new search results.
The extra data starts with a hash sign. That might be related to the problem.

I don't get a hash sign. (con't).

- Open one of the result pages. Right-click and select "View Page Info" to check the referring URL

With NoScript enabled, the referring URL matches the URL example above and includes the original search string (it shouldn't include it).

Not for me, probably because I use the RefControl add-on, and I don't allow scripting or cookies from Google.

There may be a bug here, but RefControl is a fast, lightweight fix, apparently. Or maybe just disabling cookies and scripting from Google would do it.

JOC, why would one *normally* do this -- overwrite a search result string in the address bar, rather than simply put the new search term in the Search box?

One other factor that may contribute to inability to reproduce this is an ABE rule developed in response to another user's desire to sanitize Google, while still allowing their recaptchas and their Maps:

Code: Select all

# Allow all Google recaptcha and Maps, but sandbox all www.google.com.*
Site ^https?://www\.google\.com/recaptcha/*
Accept
Site ^https?://www\.google\.com/*
Sandbox

Error console then shows:

Code: Select all

[ABE] <^https?://www\.google\.com/* > Sandbox on {GET http://www.google.com/images/nav_logo_hp2.png <<< http://www.google.com/search?q=abc&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a&gbv=1&sei=PmluT7T7L-Hi0QH84un3Bg - 3}
USER rule:
Site ^https?://www\.google\.com/*
Sandbox
______________
[ABE] <^https?://www\.google\.com/* > Sandbox on {GET http://www.google.com/search?q=xyz&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a&gbv=1&sei=PmluT7T7L-Hi0QH84un3Bg <<< chrome://browser/content/browser.xul - 6}
USER rule:
Site ^https?://www\.google\.com/*
Sandbox

With this many lockdowns, I'd have to disable them one-by-one to try to reproduce, but would rather not. All of the above are good ideas for privacy.

If you find the specific suggestion that fixes that exact issue, please do let us know.

[Darkoshi]

By reading this post: viewtopic.php?f=10&t=7758
I decided to try turning off Google Instant.
When I turn Google Instant off, then the problem does not happen. However that isn't a good solution for me, because the Google settings always get reset, as my Firefox settings are to delete cookies when I close Firefox.

Google.com is in my whitelist... I may try removing it. I'd have to see if that would cause me problems with too many pages, or not.

Not for me. There is more info after the firefox-a, but it's too long for the bar. So you have to move the cursor until you can see the end of the string.

For me, there is nothing after the firefox-a when I initially search from the Search Bar (or from the context menu "Search Google for...").

JOC, why would one *normally* do this -- overwrite a search result string in the address bar, rather than simply put the new search term in the Search box?

I don't overwrite the string in the address bar - I overwrite it in the Search Box on Google's page. That might have been unclear due to my statement about my not overtyping it in the *Firefox* search bar.

In fact, I don't usually even use the Firefox search bar, but I do often select text on a page, and use the context menu action "Search Google for..". That has the same results as doing the search from the search bar.

I'll try the RefControl addon. That seems simple enough. Thanks!

[Tom T.]

Google.com is in my whitelist... I may try removing it. I'd have to see if that would cause me problems with too many pages, or not.

The only effect that I have on Google.com is that I get a page saying that if the results don't show in a few seconds, click here.

I was able to remove that by removing the ABE rule -- the new page loads automatically -- *but still with no Google scripting allowed".
So you may be able to do that.

I'll try the RefControl addon. That seems simple enough. Thanks!

I'm betting that it will fix this -- and preserve your privacy in many, many more cases.

Please let us know if it indeed solves your test case, thanks.

ETA: There are various cookie-management add-ons out there that will allow you to mark certain sites or certain cookies as always to be preserved, even as the remainder are deleted.

However, this does help Google to build a search history and dossier on you, so it's a privacy trade-off. Since you seem to be privacy-conscious -- not wanting the referring string and site to show -- might want to ponder whether keeping Google cookies is a good idea, even if some extra feature is lost. Just a thought.

I use a more privacy-aware search engine, https://duckduckgo.com/html, but the results may not be as comprehensive. Might be worth a try, and there is a search plug-in for it.

[Darkoshi]

After more research, I've discovered the following.

If google.com is not included in NoScript's whitelist, then the problem does not occur. The referring URL that gets passed has the correct search term.
However, various other functionality is lost on the Google pages. Therefore this is not my preferred solution. (Actually, I've come to the conclusion that the problem isn't a big deal. The first time I noticed the problem, it shocked me and did seem a big deal, because the original search term that was mistakenly sent to another site included personal data that I had searched on earlier in the day. But in general, that isn't likely to happen, and even if it does, nothing is likely to come of it.) I'm not opposed to the referring URL being passed, as long as it has my most recent search term rather than some prior search term.

The RefControl add-on lets you configure or block the referring URL that gets sent TO a specific site, but it doesn't let you configure the URL that gets sent FROM a specific site (like google.com). It does let you block referring URLs for ALL sites, but apparently that can also be configured via the network.http.sendRefererHeader entry on Firefox's about:config page.

Using the site "https://www.google.com" prevents all referring URLs being sent, except for ad links.
Using the site "https://encrypted.google.com" prevents referring URLs being sent, even for ad links.

If I configure NoScript to force HTTPS on *.google.com, then referring URLs are not sent, even for ad links.
For certain pages to work, the following also need to be configured in the "Never Force" box:
www.google.com/imgres
translate.google.com

I was planning to switch over to using the encrypted version of Google anyway, so the latter option is what I'm going to do.
I also use DuckDuckGo as one of my main search engines, and I like it a lot. But as you mentioned, sometimes the results aren't as comprehensive, which is why I also still use Google. I also recently found out about "ixquick" which is another privacy-minded search engine.

[Tom T.]

The RefControl add-on lets you configure or block the referring URL that gets sent TO a specific site, but it doesn't let you configure the URL that gets sent FROM a specific site (like google.com).

... Which only matters if eavesdroppers are watching the packets flow across the Net, correct? Still, worthwhile. I just sent a feature request to the developer of RefControl; will see what happens.

I was planning to switch over to using the encrypted version of Google anyway, so the latter option is what I'm going to do.

The creepy thing about allowing script at Google is their geolocation detector, totally separate from Fx's. Disable in Fx, and Google script still shows your location. If we're concerned about privacy... I'd rather have the inconvenience of no auto-complete that put up with that. IMHO. YMMV.

I also use DuckDuckGo as one of my main search engines, and I like it a lot. But as you mentioned, sometimes the results aren't as comprehensive, which is why I also still use Google. I also recently found out about "ixquick" which is another privacy-minded search engine.

Thanks. I was using Scroogle for so long that by the time they were forced out of business, I couldn't remember that name.

[Giorgio Maone]

This is probably due to NoScript removing the tracking redirector from Google search results, no matter if scripts are allowed or blocked (Google uses two distinct methods to the same effect).

If google.com is not allowed, you don't get the original search in the referrer because the page actually refreshes on new searches, rather than just performing an AJAX call but keeping the same URL modulo the hash part (the string after the '#' character, whose changes don't cause a page reload).

If NoScript is disabled, the tracking redirector, which is an interstitial blank page performing a meta-refresh, causes the original referrer to be lost.

To disprove this theory of mine, just temporarily set your noscript.surrogate.enabled about:config preference to false.

[Darkoshi]

To disprove this theory of mine, just temporarily set your noscript.surrogate.enabled about:config preference to false.

Yes indeed, that seems to be it. With that set to false, the problem doesn't happen.

[Tom T.]

This is probably due to NoScript removing the tracking redirector from Google search results,.... If NoScript is disabled, the tracking redirector, which is an interstitial blank page performing a meta-refresh, causes the original referrer to be lost.

What, exactly, were you referring to by "This"? Something OP said, or something I said?

I'm confused. *Disabling* NS causes the original referrer to be lost? Enabling NS keeps the tracking redirector?

To disprove this theory of mine, just temporarily set your noscript.surrogate.enabled about:config preference to false.

Making sure IIUC before testing: "disproves" your theory, or "proves" it?

Sorry, bad Monday (aren't they all?)

(FWIW, my ABE rule to sandbox all Google except for recaptcha and maps causes the intermediate page to stick until i click here. Small price to pay, IMHO, but is that related to this discussion?)

ETA: I cross-posted with the post above.

[Giorgio Maone]

This is probably due to NoScript removing the tracking redirector from Google search results,.... If NoScript is disabled, the tracking redirector, which is an interstitial blank page performing a meta-refresh, causes the original referrer to be lost.

What, exactly, were you referring to by "This"? Something OP said, or something I said?

The "problem" originally reported.

I'm confused. *Disabling* NS causes the original referrer to be lost?

If scripts are disabled, each search has its own page with its specific URL, and that's what is sent as a referer: hence the original URL, i.e. the one which you opened Google from the browser's search box, is lost as soon as you make your first search from the [b]page's[/url] search box, because the page gets reloaded to a new URL with a different query string.
If scripts are enabled, even if you make further searches the page is never reloaded because Google uses AJAX to refresh its content and just changes the hash (the part of the URL after the "#" character, which is never sent to the server and whose changes don't cause any new HTTP request) just to make the search URL bookmarkable.

Enabling NS keeps the tracking redirector?

Nope, but it enables AJAX and, since the redirector is skipped, the referer includes the original because the URL's leftmost part (on the left of the "#" character) never changes across searches.

Maybe I can tweak the surrogate to make the request referer-less if scripts are enabled, but I'm not sure it's worth the effort.
BTW, I'm a long time user of this add-on, which I use to block referer sending everywhere except those crappy image hosting websites which break otherwise.

[Tom T.]

@ Giorgio:
Thanks for clarifying all.

BTW, I'm a long time user of this add-on, which I use to block referer sending everywhere except those crappy image hosting websites which break otherwise.

I have it because of your recommendation to me a few years ago.
A few posts above:

- Open one of the result pages. Right-click and select "View Page Info" to check the referring URL. With NoScript enabled, the referring URL matches the URL example above and includes the original search string (it shouldn't include it).

Not for me, probably because I use the RefControl add-on, ...