BBC iplayer wants to run amazon scripts by direct IP address

[Richard (MQ)]

Hi,

I've noticed for a while that accessing BBC iplayer (from within the UK) causes noscript to flag attempts to run scripts from one of a range of IP addresses (not URLs) all falling in the range 52.0.0.0 to 52.31.255.255 (i.e. 52.0.0.0/11) - whois reports these as belonging to Amazon. Quite why they want to hide behind IP addresses is another issue, but is there any way to block scriopts from this entire range in noscript? Blocking them one at a time works, but next time a different address is used requiring a new block to be set.

I tried adding "52.0.0.0/11" to about:config -> noscript.untusted but it is silently ignored; maybe because https://noscript.net/features suggests

Subnet matching - an address with a partial numeric IPv4 IP will match all the subnet. You must specify at least the 2 leftmost bytes, e.g. 192.168 or 10.0.0.

but this is hard to use with the wide range required. I also tried variations on "52.0.0.0 - 52.31.255.255" in this setting, also silently ignored.

And yes, I realise that blocking the whole of 52.0.0.0/11 may well stop me using amazon's other services - but I don't want to to use them!

Of course it would be better still to get the beeb to stop trying to spawn these scripts anyway, but I doubt I will get any joy there...

Thanks in advance
Richard

[DJ-Leith]

I know I'm not answering your question directly; I'm just thinking of another approach that might be easier to implement.

I speculate that the BBC might use (hire) resources from Amazon (lots of companies do - Amazon Web Services etc).
Are these required to use the iPlayer?

I don't know, but I fear that they might be.

Some people use a 'separate Profile just for the BBC'.

Using Release, ESR, Beta, Aurora, and/or Nightly together.
http://forums.mozillazine.org/viewtopic ... &t=2821799

Is a good source (lots of helpful links) of how to have several Profiles, and run them from shortcuts, at the same time.

DJ-Leith

[DJ-Leith]

I advocate and use 'specific Profiles for specific uses / sites'.
Examples include a Profile 'just used for Bank' and a separate one 'just for webmail'
and a third one 'just for making online purchases' (general looking at 'things to buy'
is done in yet another Profile).

All my Profiles have NoScript (NS) to block Javascript (and other active content) and
RequestPolicy Continued (RPC) to block all outgoing requests from the 'site I appear to be on',
as seen in the 'URL bar', to other sites.

These 'other sites' might be to Content Delivery Networks (CDNs)
but they might be to Advertising Networks.

RPC is controlling 'all outgoing Requests', which might 'collect an image', 'collect a script',
'collect an advert' etc
FROM the 'site you appear to be on' TO another domain.


I live in the UK and most of the active commentators here do not. I'll try and see
what I 'need to allow' to use the iPlayer.

I think the BBC block non UK IP Addresses from some of the content that is paid for by
the BBC Licence Fee (that UK residents pay to watch TV).

DJ-Leith

[DJ-Leith]

1 of 3

I cloned a test Profile.

It has several Extensions including:

NoScript 2.6.9.35rc2 (NS)
RequestPolicy Continued 1.0.beta10 (RPC)
https://requestpolicycontinued.github.io/

This version of RPC is 'not signed by Mozilla', see
NoScript Not A Signed Add On (Yet)?
viewtopic.php?f=8&t=21126
for information and links about signing.

There is an older version of RPC at AMO (that is signed).
https://addons.mozilla.org/en-US/firefo ... continued/

Both NS and RPC have very short Whitelists, NO BBC domains on either Whitelist.

RPC set to "Block requests by default."
but to "Allow requests to the same domain (www.example.com -> static.example.com)."

Using Firefox Developer Edition [Fx 42.0a2 (2015-08-15)]
I have 'switched off e10s' (in about:preferences).

First, went to the BBC Homepage
http://www.bbc.co.uk/

Use RPC to "Temporary allow" (TA)

Code: Select all

*.bbc.co.uk to *.bbci.co.uk

Opened http://www.bbc.co.uk/iplayer in a new Tab.

Use NS to TA both bbc.co.uk and bbci.co.uk (I think I'm going to have to allow JavaScript at the BBC).

Use the 'search box', top right (that has "Search BBC iPlayer") and search for "Atlantic".

This finds

http://www.bbc.co.uk/iplayer/search?q=A ... on%20Earth
(3 episodes - so far have been put on the iPlayer).

There are some NS Blocked Objects.
TA FONT@http://static.bbc.co.uk

(icon fonts are used on this page)

Click the 'play' (for the 3rd episode) now at:
http://www.bbc.co.uk/iplayer/episode/p0 ... en-to-hell

There is a 'flash video' here,
use the NS Blocked Objects Menu to
TA

Code: Select all

shockwave-flash@http://emp.bbci.co.uk ... /StandardMediaPlayer ...

continued ...

[DJ-Leith]

2 of 3

[Even in code tags the full object, in the post above, is being blocked by the spam filter]

That did not 'play the video', instead I had a 'black rectangle' (where the video would be shown).
I forced a reload without cache Ctrl+Shift+R.

Then I used the NS Blocked Objects Menu to
TA

Code: Select all

*@http://emp.bbc.co.uk

Then allowed the shockwave-flash blocked object.
This time I could see the documentary.

I did not see any attempts to go to any site other than

Code: Select all

*.bbc.co.uk to *.bbci.co.uk

So, perhaps there is an issue with my RPC not
blocking the outgoing connections from the BBC to 52.x.x.x

... range of IP addresses (not URLs) all falling in the range 52.0.0.0 to 52.31.255.255 ...

continued ...

[DJ-Leith]

3 of 3

I do have some 'unusual preferences'
(you can use "about:config" to change the "prefs.js").

Here are some, from the Profile I used to make this test of the iPlayer.

(spam filter is blocking code tags again)

Code: Select all

user_pref("media.peerconnection.enabled", false);
user_pref("network.cookie.cookieBehavior", 1);
user_pref("network.cookie.lifetimePolicy", 2);
user_pref("network.cookie.prefsMigrated", true);

Code: Select all

user_pref("network.dns.disableIPv6", true);
user_pref("network.dns.disablePrefetch", true);
user_pref("network.dns.disablePrefetchFromHTTPS", true);
user_pref("network.http.speculative-parallel-limit", 0);

Code: Select all

user_pref("network.predictor.enabled", false);
user_pref("network.prefetch-next", false);

I'll keep this 'BBC iPlayer Profile' for a few days and run more tests.

So far, I've yet to see the 'traffic to 52.0.0.0' but I may be missing something.

DJ-Leith

[barbaz]

"Block scripts"? It's already happening if it shows up in the NS menu as "Allow [...]" and/or "Temporarily allow [...]".

If that's not enough for you, and if you don't want to try to add each of the 52.0 through 52.31 entries to about:config > noscript.untrusted (or if that doesn't help), then DJ-Leith's posts detail the best solution.

Also, to note. Given that DJ-Leith didn't see these IP addresses you're seeing, check your system for malware? (see General Troubleshooting Instructions #2 for some suggestions how)
Amazon could be Amazon AWS and/or Cloudfront, which are Amazon-provided services that host content for other entities, some of which could possibly (at least in theory) be malicious. Please try a reverse DNS lookup on these IP addresses next time you see them, to see if it's one of those Amazon services (subdomain of [s3.]amazonaws.com or cloudfront.net).

[Richard (MQ)]

Thanks all for the responses.

In the light of these comments, I should add some more information:

1. Platform is Seamonkey 2.33.1 running on opensuse Linux 13.2 - also visible on latest Firefox in same platform.

2. NS is certainly blocking these scripts, I wondered why they were being requested. I don't consider NS to be failing in this respect, but I wondered if I could blacklist them all by a suitable entry in the "noscript.untrusted" field (avoiding the notification each time).

3. I just realised that these issues only present when accessing live streaming radio via a specific method - from the pages that give live score updates. I saw it on footy and cricket; neither are currently live but the golf is - http://www.bbc.co.uk/sport/live/golf/33674538 is a good example. I see these scripts being blocked even before I click on "live coverage".

Maybe with this extra information others will be able to replicate my report?

Thanks
Richard

[DJ-Leith]

All,
I have other 'unusual preferences' (that I choose to use to improve my Privacy and Security).

A good source of ideas, and documentation of the settings are here:
https://github.com/pyllyukko/user.js/
Just my personal opinion.


Richard,

I still think it is easier to 'block requests to domains'
[that you don't want to make connections to] using an Extension,
like RequestPolicy, that is designed to do this - vs creating lots of 'blocking in NoScript'.

3. I just realised that these issues only present when accessing live streaming radio via a
specific method - from the pages that give live score updates.

So, how do you expect the BBC to give you 'live scores' if you don't allow JavaScript?

Using my test 'BBC iPlayer Profile' (that I setup - see above)
I did the following:

1.1 Went to the BBC Homepage
http://www.bbc.co.uk/

Use RPC to "Temporary allow" (TA)

Code: Select all

*.bbc.co.uk to *.bbci.co.uk

1.2 Opened http://www.bbc.co.uk/iplayer in a new Tab.

This time, do NOT use NS to TA both bbc.co.uk and bbci.co.uk

1.3 Open another new Tab and pasted your example URL:
> http://www.bbc.co.uk/sport/live/golf/33674538 is a good example.

This became:
http://www.bbc.co.uk/sport/live/golf/33 ... iant=nonjs
Notice, the BBC have changed the URL to end "?live_variant=nonjs"

I don't hear anything. It looks like a transcript of two people talking about Golf.
It is a very simple screen, mainly text.
It could be thought of as 'sub titles' (it might be sub titles, I don't know).
If you were 'listening on a radio' (built into your cell phone) it could be the
commentators' speech.

There is a green "Update" button.
If I open this (in a new Tab so that I get a new URL to copy for this documentation)

I get to
http://www.bbc.co.uk/sport/live/golf/33 ... =1#lts-top

The text is the 'same as before'.

Live Reporting

By Ben Dirs

All times stated are UK

Update <== this is the 'green Update button'

Posted at 00:35 <== I speculate the time the live Golf broadcast finished (about 19 hours ago).

Righto, I think that might be time to say goodnight. Well done Jason Day, no longer the best player never to win
a major championship. And nobody handed it to him, he went out and won it.

Looks like the Golf (US PGA) is finished [not really a surprise].

N.B.
So far the only domains allowed (remember controlled by RPC) are:

Code: Select all

bbc.co.uk to make requests itself and its sub-domains *.bbc.co.uk and *.bbc.co.uk to *.bbci.co.uk

Close the browser (to throw away all TA permissions, cache and cookies etc).

continued ...

[DJ-Leith]

Still using my test 'BBC iPlayer Profile' (that I setup - see above)
I did the following:

2.1 Went to the BBC Homepage
http://www.bbc.co.uk/

Use RPC to "Temporary allow" (TA)

Code: Select all

*.bbc.co.uk to *.bbci.co.uk

2.2 Opened http://www.bbc.co.uk/iplayer in a new Tab.

This time, use NS to TA both bbc.co.uk and bbci.co.uk
(so now JavaScript is allowed at the BBC).

2.3 Open another new Tab and pasted your example URL:
> http://www.bbc.co.uk/sport/live/golf/33674538 is a good example.


This time, the URL did NOT change. The Page has lots of images.
Below the 'main picture' (which was NOT shown on the 'text only no JS page' - above)
there is the 'transcript like text'.

There is also a 'Red Flag', from RPC, indicating that 'the page we are on
is trying to make requests to other domains'.

These are (domain, number of outgoing requests from this page [1 in these cases]):

Code: Select all

go-mpulse.net 1
chartbeat.com 1
edigitalsurvey.com 1
52.16.106.38 1
bbcimg.co.uk 1

requests are blocked.

Also, 92 requests to bbci.co.uk are allowed (at this point).

So, the questions are:
1. Do you trust the BBC (with these outgoing connections)?
2. Do you trust OTHER websites to make connections to the domains (listed above)?

With RPC you can choose to allow or TA the BBC but NOT others to make
the connections (that are not obvious) while you are on a site.

So, there are connections [one in this case] to 52.16.106.38 from this 'live Golf page'.

Richard,
I don't know 'how many of the blocked requests' you will need to allow
to hear the 'live sport'.

Can I recommend that you try
* a special BBC Profile
* use RPC (or another blocking Extension e.g. uBlock Origin) to control the 'outgoing requests' on that profile.

uBlock Blocking mode
https://github.com/gorhill/uBlock/wiki/Blocking-mode

Then you will have more control.
From my reading of your posts you can 'get what you want from the BBC'
WITHOUT the requests to 52.x.x.x

However, I speculate, you may have to allow some of the other domains (in the code block - above).

I'll let you do some experiments / tests.

Please report your findings to help others.

DJ-Leith

[Richard (MQ)]

Thanks DJ-Leith

We have got slightly off track here - my issues were around the BBC running scripts by direct IP address rather than a public URL, and whether NS can deny them en-bloc. It seems that the answer to the second question is negative, though maybe uBlock can provide such a feature. As for the first, no-one other than me seems to find it odd that the beeb are semi-secretly running amazon scripts. I don't like or trust amazon and see no reason ever to allow their scripts, especially on unrelated sites such as BBC iplayer. No, I don't buy from them!

From my reading of your posts you can 'get what you want from the BBC' WITHOUT the requests to 52.x.x.x

Yes, this is correct - whatever those scripts do, blocking them does not seem to have any negative effects.

Maybe the best approach is to try to capture one of them and analyse it, though I think that may be beyond my ability.

go-mpulse.net 1
chartbeat.com 1
edigitalsurvey.com 1
52.16.106.38 1
bbcimg.co.uk 1

I block the first 3 on your list with NS, and the last one is useful for some pages on www.bbc.co.uk so I allow it.

I will look into installing uBlock as it seems to be a more comprehensive browser add-in, though as I'm on Linux there are other ways to block IP ranges within the OS.

Thanks for your interest
Richard (MQ)

[DJ-Leith]

Richard,
you are welcome, I enjoy a puzzle.

1 of 2

Three additional points.

1. Extension for blocking outgoing requests.

The advantage of an Extension to 'block requests' is the UI is easy to use and, with RPC, it is easy to see
'which domains are blocked'. I'm much less familiar with uBlock.

With RPC it blocks 'all requests' to the 'other domain', not just scripts.

I'll use an example of a story from ghacks.
Neither RPC nor NoScript has ghacks on the Whitelist (on the Profile I'm using for this example).

RPC in strict mode ("Block requests by default" 'ticked' and "Allow requests to the same domain" 'unticked').

Fix for installing unsigned add-ons in Firefox Dev and Nightly
http://www.ghacks.net/2015/08/04/fix-fo ... d-nightly/

I can read most of the 'story' without JS.
I can even read the comments (often NOT the case if the comments are 'hosted elsewhere').

However there are some images, that are hosted at cdn.ghacks.net
RPC has a 'Red Flag', indicating blocked requests.
There is also a 'place holder' (provided by RPC) for the blocked image
e.g.

Code: Select all

 ... Nightly prevented the installation of the unverified add-on <snip>
href="http://cdn.ghacks.net/wp-content/uploads/2015/08/firefox-prevent-installation-unsigned-addon.jpg" 

This helps me 'decide what to do'.
I decide to 'let ghacks send requests to the CDN, to collect the picture'.

So I use RPC to TA

Code: Select all

*.ghacks.net to *.ghacks.net

(requests TA TO same domain and sub-domains of ghacks ONLY from ghacks) the pictures are shown. Still no JS is
allowed. I didn't allow JS on this visit, I did see 'everything I wanted to see'.

On many sites (not ghacks) there a dozens of domains that RPC blocks (I'm very happy NOT to see all the adverts).

I find the combination of NoScript (for active content) and RPC (requests to other domains) works well.

Richard, my "noscript.untrusted", on all my Profiles, is EMPTY because RPC is 'blocking before NS even has a
chance to assess if the content is active'. NS never even sees it, unless I went to the site or I allowed site A
to send requests to Site B. Of course, if NS never saw it neither did I (LOTS of Adverts are blocked) and
pages load faster too.


2. I hope you saw the advantage, especially when testing, of having a separate Profile.
Anything you 'allow' is confined to that Profile (either TA or added to the Whitelist).

As I post this, I'm running 4 profiles: two Fx 40.0.2 and two Fx 42.0a2.
Only one is 'logged in' to forums.informaction.com. I keep one profile 'just for making posts',
and I keep the History on that Profile (to quickly find the posts). I don't make many posts.

continued ...

[DJ-Leith]

2 of 2

3. Now, your initial question: about why the BBC might use 52.x.x.x in some situations and why nobody
seems to have noticed the connections to 52.x.x.x

I don't know. Here are my thoughts, they include several 'wild speculations' - they are just ideas, could well be
totally wrong.

3.1 The BBC is a huge website, I would not be surprised if there are many people who provide content to the BBC
who also have no idea about 'how the WHOLE thing works'.

3.2 You noticed that 52.x.x.x was only used for 'live sport'.
WILD speculation - the BBC 'book an Amazon resource' just for the US PGA. They only want it for the event (and
say 8 days before and one week after). While they test their 'event micro site' they hardcode the IP addresses.
When it is live they don't need to allow DNS replication. As soon as they don't need it they stop using it.

3.3 Perhaps, to continue the WILD speculation, the Sports Division have a pool of these addresses that they 'swap
in to cope with demand'. There could be any number of reasons.

3.4 Here is another WILD speculation. The BBC have partners for 'live events' (e.g. local broadcasters). They
allow these partners 'short term and specific access' just to one of the BBC's 'hired 52.x.x.x addresses' - just
for the event.

3.5 WHAT is on the 52.x.x.x? I've no idea I've not looked. You have 'blocked the active content', using NoScript,
and still 'enjoyed what the BBC was offering'. In my posts (above) I never allowed RPC to let my 'BBC iPlayer
Profile' even make the request to 52.16.106.38. If you use a blocker (e.g. a Host file or an Extension) do you still
'get what you want' when you block ALL requests?

3.6 Why did nobody report (or notice) this before?
I doubt that they were looking. It is easy to see (passive and active requests) with RPC. With NoScript
you tend to only be shown what 'NoScript is blocking' (fonts, JavaScript, Flash etc) - the requests to active
content. Again, from my post above - RPC only saw the potential request to 52.16.106.38 AFTER I ran JavaScript
at the site. The 'non js' version did not 'trigger a RPC Red Flag'.

Notice, I've been quite neutral about the purpose of the 52.x.x.x
At this stage I've no idea if it is anything to be concerned about. Like you, I don't KNOW what it is.
I do think, because your NoScript detected it, that it is 'active content'.

I do expect, for reasons of 'paying lots of money for rights to cover Sports' that the BBC do restrict 'who can
see the content'. They might 'block non UK IP Addresses' for some content (because another broadcaster has
the rights).

DJ-Leith

[barbaz]

Did anyone even notice my prior post in this thread?

Please try a reverse DNS lookup on these IP addresses next time you see them, to see if it's one of those Amazon services (subdomain of [s3.]amazonaws.com or cloudfront.net).

So I went ahead and did the reverse DNS lookup... and guess what -

Code: Select all

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45801
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;38.106.16.52.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
38.106.16.52.in-addr.arpa. 300	IN	PTR	ec2-52-16-106-38.eu-west-1.compute.amazonaws.com.

- it's Amazon AWS.
So we have no idea what it is, and short of examining the content that's being requested from it we can't find out very readily. Suffice it to say that the fact it's an Amazon IP doesn't mean much, all we know (without examining the content it's trying to serve) is that it's a CDN for some site.

If the site works fine without it Allowed, in my experience most likely it's a tracker or ad server of some sort. But again, one would have to look at the content it's serving to be sure.

[Richard (MQ)]

Thanks Barbaz,

As I stated in my original posting, I used whois for much the same result:

Code: Select all

$ whois 52.16.106.38     
#
# ARIN WHOIS data and services ...
...
NetRange:       52.0.0.0 - 52.31.255.255
CIDR:           52.0.0.0/11
...
Organization:   Amazon Technologies Inc. (AT-88-Z)
...
OrgTechName:   Amazon EC2 Network Operations
...
OrgNOCName:   Amazon AWS Network Operations

I don't think that there's much doubt that it is registered to Amazon.

Cheers
Richard