twitter links don't work - escaped_fragment issue

[sourcejedi]

[WORKAROUND FOUND, still looks like a bug to me though]

I clicked on this link:

Code: Select all

http://twitter.com/#!/PostDeskUK/status/124811515495841792

Bad news: Twitter is not in my whitelist, so would just take me to twitter.com (i.e. the root / index page / home page of Twitter), which is useless.

Good news: NoScript is supposed to support Google's escaped_fragment protocol, which addresses this problem.

NoScript rewrites the URL in my location bar, so that I see this:

Code: Select all

http://twitter.com/?_escaped_fragment_=/PostDeskUK/status/124811515495841792#!/PostDeskUK/status/124811515495841792

... which doesn't work ... but this does -

Code: Select all

http://twitter.com/?_escaped_fragment_=/PostDeskUK/status/124811515495841792

Which doesn't even make sense... ah, I see the problem is I had googleapis.com whitelisted.

But still, why is noscript including the fragment?

It's not a valid HTML id, so it doesn't help there. The only thing that it can affect are whitelisted scripts. Which doesn't seem like a good idea to me - what's the logic behind this? It can't possibly be part of Google's "standard", can it?

[Tom T.]

[WORKAROUND FOUND, still looks like a bug to me though]

I clicked on this link:

Code: Select all

http://twitter.com/#!/PostDeskUK/status/124811515495841792

Bad news: Twitter is not in my whitelist, so would just take me to twitter.com (i.e. the root / index page / home page of Twitter), which is useless.

Good news: NoScript is supposed to support Google's escaped_fragment protocol, which addresses this problem.

NoScript rewrites the URL in my location bar, so that I see this:

Code: Select all

http://twitter.com/?_escaped_fragment_=/PostDeskUK/status/124811515495841792#!/PostDeskUK/status/124811515495841792

Sorry that it took a week for this to be looked at. I'm not LInux-friendly, but on Win XP, it went like this (no Twitter or Google in my whitelist):

Your first link took me to a text-only navigation page, but with a URL somewhat different from your second link:

Code: Select all

http://twitter.com/PostDeskUK/statuses/124811515495841792?_escaped_fragment_=/PostDeskUK/statuses/124811515495841792#!/PostDeskUK/statuses/124811515495841792

When I temp-allowed http dot twitter.com, various a*.twimg.com ( i. e., a1.twimg, a0, etc.), and api.twitter.com, the page displayed what seemed to be "normal":

@PostDeskUK PostDesk
UK visitors to PostDesk.com may be experiencing issues viewing the site - we expect these to be resolved in the next few hours

and with this URL:

Code: Select all

http://twitter.com/#!/PostDeskUK/statuses/124811515495841792

which is very close to where you wanted to be in the first link. ("statuses" vs. "status")

It may be different because I'm accessing it from the US, and it's a UK Error message. But the "escaped fragment" part of the URL disappeared when the needed scripts were allowed.

I'll ask Giorgio to look at this. Being in the EU, I'm sure he can arrange a UK proxy, and also on Linux, to see if he can reproduce this and/or find any actual problem, including the esccaped_fragment issue -- bearing in mind that the message received was that Twitter users in the UK might have problems. And you did.

[sourcejedi]

Sorry, I can't reproduce this using my instructions either . If we're lucky, this is just me being stupid. It didn't feel like a nasty non-deterministic bug. At least not once I realized it only happened with the #!-style twitter URLs.

All I can tell you is what the "doesn't work" page looked like. It was the same as when you visit a twitter #! URL with twitter.com enabled, but twimg.com disabled. Or if you disable both, and copy and paste the #! URL manually into the address bar (which NoScript doesn't touch).

One possibility is that that I had Twitter's javascript working, but I wasn't patient enough. Twitter can take 30 seconds to load. (!!!) I run Firefox on a netbook in fullscreen mode, and sometimes I don't notice the throbber.

Secondly, in mentioning "others", I was thinking of LifeHacker (and other Gawker sites). But none of their pages - either articles or homepage - seem to work with javascript disabled. The homepage URLs don't include #!. So that's their fault; it's not a NoScript problem.

[Tom T.]

I think what you said is that there is no NoScript issue here, so I'll mark this as Invalid (not a NS bug).

The thread won't be locked or anything, so if you do find a reproducible error, please feel free to post back with full details, including any Error messages in Tools > Error Console. Otherwise, glad that it's all good, and we'll close the issue unless otherwise advised.

Cheers,
Tom T.

[sourcejedi]

HAH! I found the non-deterministic bit!

To start from scratch, it's not enough to make sure twitter.com is blacklisted. You also have to clear all your twitter cookies. (Alt->Edit_Preferences, Privacy, Show Cookies, Search:twitter, select all the resulting cookies and click [Remove Cookie]).

_Then_ I can reproduce it as follows:

1. Open http://twitter.com/#!/PostDeskUK (either clicking or copy+paste should work the same).

=> is redirected to http://twitter.com/?_escaped_fragment_= ... PostDeskUK, but this doesn't work.

2. BUT, it does work if you then hit F5 - or copy+paste the URL into a different tab - OR (as I did before) click in the location bar, remove the fragment to force a reload, and hit enter.

I think all that leaves as the culprit is Referer? In which case twitter is lying when it says "Vary: Accept-Encoding".

Can you reproduce this now?

[Tom T.]

To start from scratch, it's not enough to make sure twitter.com is blacklisted. You also have to clear all your twitter cookies. (Alt->Edit_Preferences, Privacy, Show Cookies, Search:twitter, select all the resulting cookies and click [Remove Cookie]).

Fortunately, my default settings include blacklisting twitter and blocking their cookies, not being a user.

_Then_ I can reproduce it as follows:

1. Open http://twitter.com/#!/PostDeskUK (either clicking or copy+paste should work the same).

=> is redirected to http://twitter.com/?_escaped_fragment_= ... PostDeskUK, but this doesn't work.

Not quite. Again, I get a different redirect, to

Code: Select all

http://twitter.com/PostDeskUK?_escaped_fragment_=/PostDeskUK#!/PostDeskUK

which is slightly different from yours, but again, that might be due to accessing a UK-specific site from the US.

The resultant page actually "works", in that it contains navigation links, login boxes, lists of recent posts, etc. The graphics are missing and the layout is not styled. Allowing temporary requests from twitter to twimg in RequestPolicy made the page work perfectly for me, "as-is". No reload, no new tab, no removing fragment. And I can return it to the unstylish, but apparently effective, page by revoking the temp-allow in Request Policy.

This is on my preferred default of Fx 3.6.23. Will be back in a few minutes to see if there's a difference with F7, but I don't expect one.

[Tom T.]

Same results on Fx 7.01, except that RP was more finely-grained in separating a0.tiwmg.com, a1, a2, a3... when all a*.twimg.com requests were TA, it worked.

Not being a user, I don't know what the page was *supposed* to look like, but TA-ing script from twitter and twimg did produce the right-side panel with sign-up boxes, stats, etc. -- and made the escaped_fragment stuff go away in the address bar. IOW, it worked perfectly, AFAICT.

We could do screenshots if necessary, but I'm still not seeing the problem here. The first link works after RP is allowing the image feed, and the whole site works when its necessary scripts are allowed.

[sourcejedi]

Forum ate my post .

I found one last condition: my local HTTP proxy (Polipo). Without that, it works as you described. So I asked Wireshark and Live Headers what the hell was going on. Hopefully this should be enough information to reproduce & debug this now. (Polipo is advertised as being easy to install - there's even a Windows installer - but I think the same thing will happen with any HTTP Proxy).

The HTTP Live Headers extension for firefox claims that Firefox is requesting

GET http://twitter.com/?_escaped_fragment_=/PostDeskUK HTTP/1.1


But this is a lie. I ran Wireshark on the localhost interface (i.e. intercepting the traffic to Polipo), and it shows

GET http://twitter.com/ HTTP/1.1

which completely explains the difference between the two URLs we're seeing.

So, I think there's a bug in how NoScript generates escaped_fragment URLs, which means that it doesn't work when a proxy is used.

[sourcejedi]

Ah. "Completely explains" once you know that the URL

http://twitter.com/?_escaped_fragment_=/PostDeskUK

serves a 301 redirect to

http://twitter.com/PostDeskUK

[GµårÐïåñ]

I tested the link you provided (http://twitter.com/#!/PostDeskUK) and it went there, didn't forward, change url or anything and it loaded just fine. Just had to allow the connection between twitter and twimg in RP but otherwise, no issue and everything loaded fine. Not sure why you are having this issue but do you have any other addons that might be interfering? Have you tried with a clean profile and NS to see if this occurs, this way you might be able to just eliminate us as the problem. I have tons of security configurations and I had no issue loading this link and as I said, it never changed, jumped or redirected to anything and no reload was necessary to make it work. So hope this helps.

[sourcejedi]

> Not sure why you are having this issue

As I say, it only happens with a proxy.

Install Polipo, or any other HTTP proxy. Configure Firefox to use it.

Make sure twitter.com is not in your whitelist.

Clear all your twitter cookies

Then you'll be able to reproduce it by clicking on the link http://twitter.com/#!/PostDeskUK.

[al_9x]

@Tom T. - there are several real issues with _escaped_fragment_, you should remove the invalid tag

1. when the url is transformed, the hash fragment is kept, http://host/#!frag -> http://host/?_escaped_fragment_=frag#!frag
   this is probably only a cosmetic issue, but it creates confusion, so hash should be removed, per the spec
2. twitter 301s http://twitter.com/?_escaped_fragment_=/PostDeskUK to http://twitter.com/PostDeskUK, but Fx (presumably under NS influence?) then requests http://twitter.com/PostDeskUK?_escaped_ ... PostDeskUK. I don't think 301 calls for that, the bottom line is, the final url for http://twitter.com/#!PostDeskUK should be http://twitter.com/PostDeskUK not http://twitter.com/PostDeskUK?_escaped_ ... PostDeskUK
3. urls with an existing search string (http://host/?a=b#!c) are not transformed, per the spec they should be.
4. when Fx is configured to use an http proxy (Polipo, Fiddler), http://twitter.com/#!/PostDeskUK is not actually transformed (though the UI shows a transformed URL) and is passed to the proxy as http://twitter.com/

[sourcejedi]

I'd be very happy to see those individual issues looked at. But I hope the proxy issue isn't forgotten. The $SUBJECT I started this thread about - clicking on links to twitter posts and not being able to read anything - only happens when I use a HTTP proxy.

I believe my last update proved that this was due to a bug in NoScript. (I.e. not a bug in the proxy, a deficiency of twitter.com, or a flaw/grey area in some specification). When/if you're able to reproduce it with my updated instructions, then check out the HTTP requests in Wireshark: you'll see Firefox fetching completely the wrong URL.

[al_9x]

I'd be very happy to see those individual issues looked at. But I hope the proxy issue isn't forgotten.

I've confirmed the proxy problem, will add it to the list.

[Tom T.]

@Tom T. -

1. when the url is transformed, the hash fragment is kept, http://host/#!frag -> http://host/?_escaped_fragment_=frag#!frag
   this is probably only a cosmetic issue, but it creates confusion, so hash should be removed, per the spec

Fair enough.

[*] twitter 301s http://twitter.com/?_escaped_fragment_=/PostDeskUK to http://twitter.com/PostDeskUK, but Fx (presumably under NS influence?) then requests http://twitter.com/PostDeskUK?_escaped_ ... PostDeskUK. I don't think 301 calls for that, the bottom line is, the final url for http://twitter.com/#!PostDeskUK should be http://twitter.com/PostDeskUK not http://twitter.com/PostDeskUK?_escaped_ ... PostDeskUK

As you said, Giorgio should determine whether this is "presumably under NS influence?" vs. a Fx bug, but yes, it's worth a look to see if it's NS.

[*] urls with an existing search string (http://host/?a=b#!c) are not transformed, per the spec they should be.
[*] when Fx is configured to use an http proxy, http://twitter.com/#!/PostDeskUK is not actually transformed (though the UI shows a transformed URL) and is passed to the proxy as http://twitter.com/[/list]

Not a user of proxies, so thanks for that information and for the info on Google code specs -- as said, I try to avoid Google.

Generally, I don't find it surprising that if you blacklist the scripts and cookies that a site needs, you're going to break something.
So when neither Guardian nor I had any problem using the site (I didn't know that a proxy was needed to reproduce), it didn't seem an issue.

there are several real issues with _escaped_fragment_, you should remove the invalid tag

Done, and will PM Giorgio, if you haven't already notified him or written the corrective code. Thanks for diving through these multiply-murky waters to find the possible issues.