twitter links don't work - escaped_fragment issue

Post by **GµårÐïåñ** » Fri Nov 04, 2011 7:46 pm

Just for the record, I use Wireshark as part of my weekly if not daily network tasks and did use it to observe your issue to reproduce since you mentioned it. I also didn't forget about the proxy, I didn't mention it directly but I did test it with 2 proxies (1 in germany and 1 in the u.s.) and it didn't result in the issue either. Both Tom and I have have tested it as-is with no issue, and I went further with proxies to find no issue either. So it might be related to a way your proxy handles callbacks and resolutions.

That's why I said, even if we presume that NS is somehow "causing" the issue, it might only be compounding something that is done via the proxy, as I didn't have issues with either proxy and my configuration as-is and everything fetched fine. So my final say on this is that, "maybe" NS is causing something in the resolution but it might be dependent/or at least contributed to by something in the proxy too as I didn't encounter it with proxy either.

sourcejedi · Post by **sourcejedi** » Sat Nov 05, 2011 11:19 am

GµårÐïåñ, If you're still picking at this & hoping to narrow down the differences with me, then I could use some more assurances. If you don't have time to sift through the details, that's fine, but you're not going to be able to help with this

. It was pretty frustrating reproducing this myself, even though I knew it had been a persistent problem for a long time.

I tested the link you provided (http://twitter.com/#!/PostDeskUK) and it went there, didn't forward, change url or anything and it loaded just fine

If you're staying at <http://twitter.com/#!/PostDeskUK>, and seeing recent posts from "PostDeskUK" (or _any_ text that says "PostDeskUK"), then either

1) you must have one or more scripts whitelisted. I'd expect you'd have to have twitter.com whitelisted, otherwise NoScript would apply the escaped_fragment transformation, and you'd see a different URL.

OR

2) you don't have twitter.com whitelisted; NoScript applies the escaped_fragment URL transformation and fetches the non-JS version of the page for you, BUT - very strangely - you're not seeing the transformed URL in the location bar.

If you're seeing #2, please say so. That'd be an even weirder issue!

If you've been talking about #1, then that's the expected behaviour. I only have this issue because I don't whitelist twitter. Please use the instructions I provided. Re-posted below for your convenience.

Note: When I see Firefox fetching "completely the wrong URL", that's the *first* URL it fetches.

Environment

Me, I'm on Linux; Firefox is 7.0.1 from the Ubuntu repos; the proxy is Polipo from Ubuntu repos.

Steps

0. Configure firefox to use a proxy.
1. Make sure twitter.com is not whitelisted.
2. Clear all cookies from twitter.com. This is absolutely necessary to see the issue at a user level. If you've had twitter.com working without javascript, it remembers that, and sends different pages at some point, which bypass the user-visible error.
3. Visit http://twitter.com/#!/PostDeskUK.

Results

There are two possible results from visiting http://twitter.com/#!/PostDeskUK with javascript disabled.

i) If you don't have the escaped_fragment transformation for some reason, you'll stay at that URL, and see <twitter.com> with no PostDeskUK-specific content. That is, the URL "doesn't work". If this should happen with an up-to-date version of NoScript, which claims to support escaped_fragment, then you've found some other bug I'm not hitting, which needs reporting

.

ii) If you are seeing the escaped_fragment transformation, then you should see an updated URL in the location bar. What is that URL? And since you're looking at wireshark, does Firefox go through any redirects to get there? and if so, what are the redirected URLs?

Reproducing this without the Polipo proxy

This sequence doesn't depend on proxy behaviour. You can easily reproduce the "fetches one URL, shows a different URL in the location bar", using netcat -l <port> as a fake proxy. (And a single fake response, like this -

<html>completely fake http 0.9 proxy</html>
ctrl-D, or however you trigger an EOF on your platform (DOS used ctrl-z).

Nor is it triggered by previous Polipo responses which have been cached by Firefox. It still happens with Polipo if I clear Firefox's cache (and offline storage) immediately before I follow the instructions above.

Post by **Giorgio Maone** » Sat Nov 05, 2011 11:22 pm

Investigating, thanks.

Post by **GµårÐïåñ** » Mon Nov 07, 2011 11:56 pm

I am just attempting to help you, I have no stake in this either way other than to provide perspective. I believe that you are encountering issues, I am not disputing it, just saying that I can't see it happening as I have tried to reproduce this, so make sure we are clear there that I am not trying to prove you wrong, quite the contrary, because I believe you are experiencing an issue and I have been where you are with everyone telling me all is well until a while later suddenly it gets reported again and everyone is onboard, I am going with the benefit of doubt mentality here. So that being said let me address some of the things you mentioned.

1. I cannot confirm honestly whether at the time I was in situation one or two, since I don't normally allow twitter I assume I was in number two. However, just for the record, on another machine I tried ensuring condition 1 is not true and it went without any scripting allowed from: http://twitter.com/#!/PostDeskUK to http://twitter.com/PostDeskUK?_escaped_ ... PostDeskUK in the URL but displayed the content just fine.

With this as the latest post:

themightykai PS Big Al MP is behind me and looks raging! #SDLP11 @PostDeskUK 7:02 AM Nov 5th via Twitter for Android Retweeted by PostDeskUK and 1 other

2. This last test was done on a windows machine, not Linux but I can further test it in Linux when I get back into town and have access to a testing machine.

3. I tried with proxies and got the same results as #1 above.

Not sure if that totally helps or not, but I wanted to reply to you now instead of waiting another week until I get back. Hope it helps at some level and now that Giorgio is investigating, chances are he will find something before I do by the time I get back. Take care and good luck.

sourcejedi · Post by **sourcejedi** » Tue Nov 08, 2011 12:35 am

> 3. I tried with proxies and got the same results as #1 above.

Heh! Ok, thanks.

I'll try profile-switching and see what happens. You can see why I reacted badly. But it was a valid query, I know how I can do it, & I should have tried it when you asked.

sourcejedi · Post by **sourcejedi** » Tue Nov 08, 2011 1:16 am

Temporarily moving ~/.mozilla gave me a fresh profile. I installed NoScript. My proxy settings were inherited from environment variables. I had exactly the same problem.

So it doesn't seem to depend on my specific profile. It might be OS-specific though

.

Post by **GµårÐïåñ** » Tue Nov 08, 2011 7:55 pm

Ok, that changes things possibly, not sure. If the proxy is declared at OS level it might do something different than being declared for Fx only. I tested my proxies INSIDE fx, not system level. So when I get back, I can try them on the system level and see if I can reproduce your issue. How is that? At least you trying it this way yielding one other piece of information that normally wouldn't matter but in this case could, so will consider it.

Here is the test case I am going to create just for testing this:

1. Linux platform/os (which distribution shouldn't really matter but I will be testing with Ubuntu and/or Fedora and/or BSD, depending on which is more readily available for testing) - current tests were done using windows which was all that was available to me on the road.

2. OS/system level proxy (not just within Fx) - also windows networking component handle things "slightly" differently, so that might add some unnecessary noise, so will do it on linux to avoid any introduction of differences, if any.

3. Fx version 7.x (if not let me know and I will try to install a rollback version) but I think since you suspect NS, it will only depend on NS version and not so much the browser, so the 7.x should be fine, but I will be using the latest dev version of NS for all testing, just so we are on the same

4. Link you provided

Did I miss anything? Want to add anything to the test case environment? I will have port capture and wireshark running at the choke point to observe and see if I see anything that might explain it.

sourcejedi · Post by **sourcejedi** » Tue Nov 08, 2011 8:41 pm

GµårÐïåñ wrote:Ok, that changes things possibly, not sure. If the proxy is declared at OS level it might do something different than being declared for Fx only.

I don't think that's it. My default profile has the proxy configured manually. (No overwhelming reason; I guess I'm just a control freak

.

I don't have any positive expectation that it's OS-specific; I'm just echoing what you were saying that it didn't happen on Windows. Quite the opposite - this seems to a bug in URL manipulation in Firefox+NoScript, and I wouldn't expect any of Firefox's URL manipulation code to be platform-specific.

Since you're curious, here's where I set the global environment variables. I had to grep my home directory; I couldn't remember where they were:

$ cat .pam_environment
http_proxy=http://localhost:8123/
no_proxy=localhost

I don't know whether Firefox picks up GNOME proxy settings. I guess it _should_ do, given the system-installed addons you see in stock Ubuntu.

I don't believe it's ever picked up on my proxy settings from KDE, and I don't have any KDE-specific Firefox addons.

1. Linux platform/os (which distribution shouldn't really matter

Ubuntu might be more convenient. I don't know if Fedora have an official Polipo package.

Did I miss anything?

Sigh. Not that I noticed! Of course, that's what I thought the first two times I tried to report this. Could be worse; at least reproducing this bug doesn't rely on unpredictable feline behaviour.

Post by **GµårÐïåñ** » Tue Nov 08, 2011 9:11 pm

Ok, done, I will take all that into consideration. I didn't think it would matter to be OS level either, but since I only tested in Fx and you were seeing something I didn't, I figured two possible things, 1) it could be a general os level issue, or 2) more specifically a difference between windows/linux issue, and 3) to a lesser extent distribution package handling issue, not sure.

So I will use Ubuntu, will use the package you used, will set the environment variables accordingly and setup my test environment as closely as I can, so that I can give as best of a feedback as can be managed without actually sitting at your terminal. I will keep you posted but as I said before, I am out of town, on the road for treatment and I won't be back home until Friday at the earliest, so I will have to do it then. So please be patient.

I am hoping Giorgio's investigation yields results sooner but I know he is traveling as well and has his plate full, so it might be delayed for him too. It just so happened you caught us both out of town and running in a situation that might make testing more difficult. So I will provide feedback as soon as I can. If you need to tweak the test case or add anything else, please do so in the meantime, so when we start the testing, we can take them all into consideration.

I told you I was trying to help you, how many do you know on the road will take this much time to try and help someone?

We'll figure this out, just be patient ok?

sourcejedi · Post by **sourcejedi** » Thu Nov 17, 2011 2:41 pm

THIS BOARD EATS MY DRAFT MESSAGES ON TIMEOUT AND I HATE IT

sourcejedi · Post by **sourcejedi** » Thu Nov 17, 2011 2:47 pm

Guardian still couldn't reproduce this (reported in PM), so we conferred and I resorted to sending him a VM.

<http://dl.dropbox.com/u/49925445/tc.vmdk>. If anyone else downloads it, please let me know. DropBox seem very friendly, but they do have bandwidth caps.

It's a 100Mb VMWare format disk image, created with QEMU. (I believe users of the free VMWare player would also need to hack together a VMX file - google is your friend). The system is Tiny Core Linux (hence the size), with Firefox 8 from Mozilla.org. There's no icon, just start a terminal and run "firefox/firefox". NoScript is installed for you, but you will need to reset the proxy settings. It shows the same symptoms...

...just like Firefox on my other Linux boxes (which are all Ubuntu, older and newer versions, albeit similar software configurations, x86_64 as well as i386. Tested with clean firefox profiles). So it puzzles both of us as to why it didn't happen in Guardian's fresh installs. (It's not an issue with my UK locale - the Tiny Core image has the default locale, which seems to be US, judging from the keyboard layout).

There's a polipo package in the VM, if one wishes to test it that way. You have to run "polipo" manually, and it doesn't include the directory for the disk cache. I've just checked, and it still reproduces the problem with Twitter. However, the exact behaviour might somehow depend on where you connect the VM to the internet. (I also had some problems getting the network working - try rebooting the image, or "sudo /etc/init.d/services/dhcp restart", or "sudo ifconfig up eth0" (!)).

To reproduce (with or without the VM) without network access as a variable, see "Reproducing this without the Polipo proxy" above. Except that in Tiny Core linux, you want "nc" instead of "netcat". Also, you need "nc -l -p 8000" instead of "netcat -l 8000". On other systems, you may want to check the documentation for your version of netcat.

Post by **GµårÐïåñ** » Thu Nov 17, 2011 10:20 pm

Thank you, downloading it. See PM for further comments.

sourcejedi · Post by **sourcejedi** » Sat Nov 19, 2011 7:11 pm

I think I've found it. NoScript changes the URI to use _escaped_fragment_ here:

RequestWatchdog.js:152
channel.URI.spec = abeReq.destination = newURL;

But "The URI corresponding to the channel. Its value is immutable. Read only."

and the C code relies on that:

http://mxr.mozilla.org/mozilla-central/ ... el.cpp#541
requestURI = &mSpec;

mSpec must be a private copy of channel.URI.spec. I guess it gets set before NoScript changes the URI. And this line is specific to the proxy case. Which is consistent with my own observations, although it doesn't explain why Guardian couldn't reproduce it.

From what I can find, there's no hook to change request URIs. It does seem a slightly suspicious thing to do. Ideally you'd want to do it earlier, at the point where the URI is taken from the address bar / clicked link...

1. RedirectRemover suggests it's possible to intercept link clicks and frame loading (though NoScript doesn't need all the code; it doesn't need to force the "ugly URL" into the "status bar", or the "copy link location" option on the context menu).

2. StackOverflow thinks intercepting the location bar is doable, but undocumented.

3. I'm not sure whether it's possible to catch assignments to window.location, calls to window.open(), assignments to the src attribute of frames... and there's most likely more stuff I'm overlooking.

sourcejedi · Post by **sourcejedi** » Wed Nov 23, 2011 11:40 pm

This is not working right. it won't show anything and the browser simply says the proxy rejected the connection, so any other way you want to share that I can test this to actually see what you are seeing. I was thinking of doing it my own way but i want to stay as close to your actions as possible to avoid me getting by something or doing something differently to work that yours didn't or don't. So let me know.

...

I actually didn't get anything and unfortunately the HTTP live headers is not installed and won't show up under extensions, so not sure what happened there, NoScript is there though. Anyway, just let me know how to replicate this YOUR way and we can keep at it until we see what's up and I can talk to Giorgio about it.

Impressive.

<sanity-check status="failing">
Dropbox: Version History of 'tc.vmdk' - 'tc.vmdk' has only one version / All files up to date

md5sum Dropbox/Public/tc.vmdk
26fda35d0ce05f7a290e228def6cec24 Dropbox/Public/tc.vmdk

<steps>
0. Copy tc.vmdk to a working location.
1. Boot the VM.
2. Click terminal icon (second from the left).
3. Type "firefox/firefox" and hit ENTER
4. Drag the firefox window to the top of the screen, so you can see the icons again.
5. Start a second terminal window.
6. Type "nc -l -p 8000"
7. Click in firefox location bar, enter "http://twitter.com/#!/PostDeskUK" and hit ENTER
=> Request appears in second terminal window
=> Firefox spins, waiting for a response
8. Click in second terminal window
9 Type "HTTP IS FAKE" and hit ENTER.
10. press Ctrl+C
=> Netcat dies with "punt!"
=> Firefox shows content; location bar updates, but is inconsistent with the request displayed in the terminal window.
11. Post screenshot in this thread

.
</steps>
</sanity-check>

Re: HTTP Live Headers:
For me, it shows up under the Extensions page in about:addons, but it does say it's disabled (with an immediate link to restart firefox, after which it will be enabled). Yes, that's what it said when I started Firefox from the image above. I have no idea why that could happen. I have even less idea how it would spontaneously disappear from exactly the same image.

Re: Proxy refusing connection:
The most likely cause is that you're typing "nc -l 8000" instead of "nc -l -p 8000". Netcat/busybox won't warn you about this! It'll just sit there doing nothing sensible.

You can tell whether netcat is listening on the right port by running "netstat -eetl" in another terminal. You should see one line only, with the port number 8000. If you forgot the "-p" option to netcat, you'll see a random port number like "44676" under "Local Address".

You can test whether netcat is accepting connections on the right port by - connecting to it using netcat in another terminal. "netcat localhost 8000". Lines (terminated by ENTER) typed in one terminal window should then show up in the other. Killing either of them with ctrl+C will close the connection, and cause the other one to terminate.

You should obviously check the Firefox proxy configuration; it should say localhost on port 8000 (not 192.168.0.16 on port 8123).

Ideally you'd use "strace -f -e trace=network" to check Firefox's network activity. You'd have to install strace using the AppBrowser (6th icon, counting from left to right) - hit "Connect" to get the package list, select the package, and then "Go" to install. Unfortunately Firefox uses non-blocking IO. You would at least see the connect() call, which will show you the IP address and port number. However connect() will return immediately with EINPROGRESS, instead of waiting to return success / an informative failure code.

sourcejedi · Post by **sourcejedi** » Wed Nov 23, 2011 11:45 pm

Resulting screenshot: https://picasaweb.google.com/lh/photo/_ ... Jy0liipFm0

InformAction Forums

twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

[found] twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue

Re: twitter links don't work - escaped_fragment issue