[Feature request]Automatically updating ABEs

[access2godzilla]

Could there be some method in which ABEs can be updated automatically without user interaction?

This could be useful in some cases. For example, I convert the hpHosts file into an ABE rule and make it available somewhere, and since that'll require updates...

[Thrawn]

That's what Adblock Plus is for.

[access2godzilla]

That's what Adblock Plus isn't for.

1. You can deny a javascript file from loading using the ^$script suffix, but what happens when the script is embedded within the page?
2. Adblock Plus has considerably slower performancewith regexes (although I don't know why – regexes should be faster since it's parsed by the browser) and with longer lists.

BTW, there may be other uses of ABE auto-updates – but that's what came to my mind (and I might actually do it if Noscript adds this feature – it will help newbies run in scripts globally allowed mode, while having reasonable security.)

[Thrawn]

1. You can deny a javascript file from loading using the ^$script suffix, but what happens when the script is embedded within the page?

I'm not sure what relevance this has to ABE? It doesn't involve a HTTP request, so ABE isn't involved.

2. Adblock Plus has considerably slower performancewith regexes (although I don't know why – regexes should be faster since it's parsed by the browser) and with longer lists.

Have you done any benchmarks of ABP against ABE with regexes and long lists? If so, and if ABE is much faster, then please contact the ABP people and suggest that they review the ABE code to get some ideas.

[access2godzilla]

I'm not sure what relevance this has to ABE? It doesn't involve a HTTP request, so ABE isn't involved.

Assuming scripts globally allowed:
So this is to say that I can't write an ABE rule to block an embedded script running on example.com (when I'm on example.com)?

Have you done any benchmarks of ABP against ABE with regexes and long lists? If so, and if ABE is much faster, then please contact the ABP people and suggest that they review the ABE code to get some ideas.

Logically regexes should be faster (it's being parsed by FF itself), but ABP people and filter list maintainers mention everywhere that regex is slow so don't use it . Requesting Wladimir & Co. is useless - I'll not go into the reasons, but happily state them if anyone asks.

I will repeat the statement I made in my previous post, though:
Making a blocklist is only one of the purposes that came to my mind. Obviously there could be other varied purposes - a company may want to push a ruleset for their company's computers for protection, someone may make a ruleset that helps enforce privacy - and that would require automatic updating.

[Thrawn]

I'm not sure what relevance this has to ABE? It doesn't involve a HTTP request, so ABE isn't involved.

Assuming scripts globally allowed:
So this is to say that I can't write an ABE rule to block an embedded script running on example.com (when I'm on example.com)?

Well, I suppose you could try using the Sandbox keyword...but if you mistrust the site so much that you want to block scripts included inline in the page, then why are you whitelisting it? Is there a particular site where you want to do this? You might get further with surrogate scripts.

Have you done any benchmarks of ABP against ABE with regexes and long lists? If so, and if ABE is much faster, then please contact the ABP people and suggest that they review the ABE code to get some ideas.

Logically regexes should be faster (it's being parsed by FF itself), but ABP people and filter list maintainers mention everywhere that regex is slow so don't use it . Requesting Wladimir & Co. is useless - I'll not go into the reasons, but happily state them if anyone asks.

Well, regex is slower than simple string matching and wildcards. It is faster than techniques like actual parsing of request contents etc.

I would think that ABE and ABP would be pretty similar in their ability to use simple string matching or regexes, and similar in performance with each one. As mentioned previously, if that's not the case, then it would be something to suggest to the ABP developers. And if they're not responsive...well, Giorgio is busy too...

I will repeat the statement I made in my previous post, though:
Making a blocklist is only one of the purposes that came to my mind. Obviously there could be other varied purposes - a company may want to push a ruleset for their company's computers for protection, someone may make a ruleset that helps enforce privacy - and that would require automatic updating.

That, on the other hand, already exists (in a limited form) . If you check the 'Options-Advanced-ABE-Allow sites to push their own rulesets' box, then NoScript will look for a rules.abe file on sites you visit, and will apply what it finds there - but only if it doesn't conflict with your own rules.

[Thrawn]

By the way, I did begin to compile just such a list, at http://forums.informaction.com/viewtopi ... =23&t=8734

But I wouldn't recommend just subscribing to it and getting automatic updates. It's very much something that you should tailor to your own needs. Some people want to restrict Facebook so it works only while they're actually browsing on Facebook; others want to share things while browsing around. Some want to use all of Yahoo's services, while others (like Tom T and myself) want to restrict it to only what is essential for Yahoo Mail. Some people need exceptions to the built-in rule, to cater for services they use; others don't. Just use the collected rules as a guideline.

And if you develop any ABE rules that work well for you, please feel free to submit them there! Particularly site-specific ones, carefully tailored to the behavior of a popular site, like eBay, Amazon, banks, etc. If nothing else, I guarantee you'll learn more about ABE and how you could make it work better for you.

[access2godzilla]

Well, I suppose you could try using the Sandbox keyword...but if you mistrust the site so much that you want to block scripts included inline in the page, then why are you whitelisting it? Is there a particular site where you want to do this? You might get further with surrogate scripts.

Thanks for the info! (I'll have to read the spec again to get a clear idea, I suppose.)
BTW, it is not a question of whitelisting. My intention is to help newer users to run in scripts globally allowed mode but without the risk of getting infected by a malicious webpage, and I want to build a blocklist that can be updated automatically.

That, on the other hand, already exists (in a limited form)

I know, but it still makes it a manual process - visit the site each and every time to get the updated rules. Some kind of automatic process would be better.

BTW, I don't know why this idea is being refused, but if it is:

• Busy - Giorgio doesn't have to do it immediately. He can take his own time to implement it.
• Breaks spec/standard - commented update command (e.g #__updateURI <url> , #__expires <time> etc.)
• Security issue - The user will have to be sufficiently warned in some dialog while installing the ruleset.

[Thrawn]

BTW, it is not a question of whitelisting. My intention is to help newer users to run in scripts globally allowed mode but without the risk of getting infected by a malicious webpage, and I want to build a blocklist that can be updated automatically.

Adblock Plus has an anti-malware subscription list, which is probably the kind of thing you're after.

BTW, I don't know why this idea is being refused,

Well, Giorgio has already discussed the idea of having subscriptions for the general whitelist/blacklist; you can search the forum for 'subscription' to find details. The same principles likely apply here.

It might happen eventually, but the key problem is trust. Who do you trust, who do you trust to make decisions about who to trust, who do you hold accountable if trust is broken, etc. And NoScript's usual philosophy is that those are individual choices, with Giorgio merely providing his best effort at sensible defaults. He's not keen - or available - to take full responsibility for vetting a whitelist, or blacklist, and without a central provider, it just hasn't been vital enough to take priority over other projects like NSA, NoScript 3.x for the desktop, etc.

In the case of ABE rules, there are extra complications, because your rules have to match your usage of the site, and everyone's usage may be different, especially for sites (like Facebook and Google) that get imported all over the place. And those are the most important ones to have rules for, because they need to be protected against tampering. So probably a list like the abovementioned thread, suggesting rules but allowing you to pick and choose, is the best compromise.

[access2godzilla]

Please note that I've been asking for an auto-update mechanism in this thread, and not for some method of blocking malicious domains in scripts globally allowed mode. Having said that,

Adblock Plus has an anti-malware subscription list, which is probably the kind of thing you're after.

• ABP provide an equivalent of "Sandbox" of ABE due to limitations in ABP (but the limitation is practical since it's an ad blocker).
• The list is not optimised, with listing of all known malicious domains (with a large number of inactive ones) rather than a listing of URL patterns of exploit kits. (No use talking to the list maintainers about this.)
• Even optimising the whole list into a regex is useless, because ABP gives priority to wildcards rather than support only regexes. (And processing a regex should be much faster, since FF does it natively by using regex.h or similar)

It might happen eventually, but the key problem is trust. Who do you trust, who do you trust to make decisions about who to trust, who do you hold accountable if trust is broken, etc.

I'm asking for an update mechanism only. The issue of trust is a seperate one. If an user installs an ABE that does the wrong things, they are at fault. All NS can do is show a warning about bad things happening if the wrong ruleset is installed.
This could be said of browser extensions/software too. If they install the wrong extension, user's fault he screwed himself, didn't update the AV/plugins etc. Does that mean OSes don't allow running external programs or browsers don't provide a framework for extensions?
In comparison to browser extensions/software, though, the ABE cannot be used to facilitate something more malicious than what would have been possible without NS not being installed, so I don't see any support issues, nor any issues of "trust".

[Giorgio's] not keen - or available - to take full responsibility for vetting a whitelist, or blacklist, and without a central provider[.]

Why should Giorgio have to take any responsibility for an ABE? If I'm the one maintaining the list, I'll take the responsibilty, and I'll be the central provider.
And if a particular user asks for support regarding other's rulesets, tell them to go to the ruleset maintainer.

In the case of ABE rules, there are extra complications, because your rules have to match your usage of the site, and everyone's usage may be different[.]

I accept that, but that's not any reason for not having the auto update mechanism. If you don't like our rulesets, don't use it. Simple.

So probably a list like the abovementioned thread, suggesting rules but allowing you to pick and choose, is the best compromise.

Not everyone checks the threads. And do note that I'm asking for the AUTO-UPDATE MECHANISM for ABEs.

Anyway, it would be quite nice to have the auto update. If you can ask Giorgio to consider the update mechanism (not the rulesets, I have never asked for a ruleset), I'll be grateful.

[Thrawn]

ABP provide an equivalent of "Sandbox" of ABE due to limitations in ABP (but the limitation is practical since it's an ad blocker).

I was under the impression that ABP is able to completely block cross-site requests. That's as much as ABE would do.

The list is not optimised, with listing of all known malicious domains (with a large number of inactive ones) rather than a listing of URL patterns of exploit kits. (No use talking to the list maintainers about this.)

Maintaining the list is one of the problems with any blacklist-based solution.

Even optimising the whole list into a regex is useless, because ABP gives priority to wildcards rather than support only regexes. (And processing a regex should be much faster, since FF does it natively by using regex.h or similar)

That would be one nasty regex...worse than Giorgio's tracking-script regex.

I'm asking for an update mechanism only. The issue of trust is a seperate one. If an user installs an ABE that does the wrong things, they are at fault. All NS can do is show a warning about bad things happening if the wrong ruleset is installed.

It's not so simple as that. If someone writes their own ABE rules, and they get it wrong, then they are at fault, but if they subscribe to a list of rules maintained by a third party, and if that third party gets it wrong, then the third party is likely to bear some of the responsibility.

In comparison to browser extensions/software, though, the ABE cannot be used to facilitate something more malicious than what would have been possible without NS not being installed, so I don't see any support issues, nor any issues of "trust".

If someone relies on the subscription, rather than doing all the work themselves, then trust is happening.

Why should Giorgio have to take any responsibility for an ABE? If I'm the one maintaining the list, I'll take the responsibilty, and I'll be the central provider.

Yes, that would work, but since it would only affect a tiny percentage of NoScript users, it hasn't pushed this up the priority list. If you would really like to maintain a list for all NS users, feel free to talk to Giorgio about doing so.

Anyway, it would be quite nice to have the auto update. If you can ask Giorgio to consider the update mechanism (not the rulesets, I have never asked for a ruleset), I'll be grateful.

I hope this doesn't bother you, but my personal leaning is that ABE rules are never going to be so universally applicable that a subscription is the right model for them. I think the best that can be done is to provide a good interface, like the one in RequestPolicy. If I ever get time, etc, available to keep working on that.

[access2godzilla]

I was under the impression that ABP is able to completely block cross-site requests. That's as much as ABE would do.

You seem to have contradicted your own self. http://forums.informaction.com/viewtopi ... 607#p51255:

Well, I suppose you could try using the Sandbox keyword...

That would be one nasty regex...worse than Giorgio's tracking-script regex.

So what? The maintainability of the rulesets is the maintainers' problem, and that isn't an excuse for not implementing the auto-update.

Yes, that would work, but since it would only affect a tiny percentage of NoScript users, it hasn't pushed this up the priority list. If you would really like to maintain a list for all NS users, feel free to talk to Giorgio about doing so.

I'd like to maintain one, but that requires the auto-update mechanism, which isn't there (and which I'm requesting in this thread)!

I hope this doesn't bother you, but my personal leaning is that ABE rules are never going to be so universally applicable that a subscription is the right model for them. I think the best that can be done is to provide a good interface, like the one in RequestPolicy.

If you are talking about my usage case (blocking malicious domains), then a subscription model is perfect. New users can have a very decent amount of protection this way: scripts are globally allowed, but malicious domains are blocked.
For other usage cases, it is often quite good to have a subscription model. Sure there might be some bloat, but it makes users comfortable.

[Thrawn]

I was under the impression that ABP is able to completely block cross-site requests. That's as much as ABE would do.

You seem to have contradicted your own self. http://forums.informaction.com/viewtopi ... 607#p51255:

Well, I suppose you could try using the Sandbox keyword...

Oh, did you mean a Sandbox equivalent for inline content? I was talking about cross-site requests, where either ABE or ABP can completely block requests.

Actually element hiding is not really similar to Sandbox, it just makes things invisible. Sandbox actually disables scripts, I believe. But getting it to selectively apply to the pages you want to disable scripts on could be a pain. If you're that committed, go ahead - but bear in mind that Sandbox has been less thoroughly tested than the other keywords.

The maintainability of the rulesets is the maintainers' problem, and that isn't an excuse for not implementing the auto-update.

Step back right there. Excuse? Giorgio doesn't need excuses for anything. He implements what he considers to be the highest priority for his userbase, when he has time to do it, and since he's unpaid, he's accountable to no-one.

However, back on topic: the maintainability of the rulesets would, I think, make this feature less popular with potential maintainers, which makes it lower-priority.

Yes, that would work, but since it would only affect a tiny percentage of NoScript users, it hasn't pushed this up the priority list. If you would really like to maintain a list for all NS users, feel free to talk to Giorgio about doing so.

I'd like to maintain one, but that requires the auto-update mechanism, which isn't there (and which I'm requesting in this thread)!

If you're offering to maintain a list for all NoScript users, then you could send Giorgio a private message to see whether that would bump the priority of this feature.

I hope this doesn't bother you, but my personal leaning is that ABE rules are never going to be so universally applicable that a subscription is the right model for them. I think the best that can be done is to provide a good interface, like the one in RequestPolicy.

If you are talking about my usage case (blocking malicious domains), then a subscription model is perfect. New users can have a very decent amount of protection this way: scripts are globally allowed, but malicious domains are blocked.

But there are already hosts files, ABP anti-malware subscription, etc...ABE can do the same job, but was not really designed for it. Why not use the right tool for the job?

For other usage cases, it is often quite good to have a subscription model. Sure there might be some bloat, but it makes users comfortable.

Yeah, but ABE is not really a tool to use without understanding what you're doing, especially without a graphical interface, so it can be hard to tell when something is being blocked, and what it is.

ABE was designed, first and foremost, to prevent Cross-Site Request Forgery. You're supposed to write rules to protect the sites you regularly visit. It's up to Giorgio, but I really think that subscriptions belong with other tools, not ABE.

[access2godzilla]

Oh, did you mean a Sandbox equivalent for inline content? I was talking about cross-site requests, where either ABE or ABP can completely block requests.
Actually element hiding is not really similar to Sandbox, it just makes things invisible. Sandbox actually disables scripts.

That's exactly what I want to do - block inline (as well as external) scripts. The shellcode might be embedded into inline JS, blocks malicious redirects and other nastiness).

Suppose I write a rule like this:

Code: Select all

Site (.*(?:\/.*\/(?:([a-z]{2,}[_-])+.*|\.[a-z]+)|:8080\/forum\/links\/[a-z]+)\.php)
Sandbox

Will this help me to block all (inline and external) active content on the URL which matches this regex?

However, back on topic: the maintainability of the rulesets would, I think, make this feature less popular with potential maintainers, which makes it lower-priority.

Why do you think maintainability would be a problem?

If you're offering to maintain a list for all NoScript users, then you could send Giorgio a private message to see whether that would bump the priority of this feature.

First of all, is PMing the admin an acceptable practice on this forum?

But there are already hosts files, ABP anti-malware subscription, etc...ABE can do the same job, but was not really designed for it. Why not use the right tool for the job?

If you're asking for the right tool, it's an IDS/firewall/AV. Hosts files and ABP lists aren't the right tool, if you ask me.

Yeah, but ABE is not really a tool to use without understanding what you're doing, especially without a graphical interface, so it can be hard to tell when something is being blocked, and what it is.

Which calls for another feature request - optional logging of ABE actions to the error console.