Need feedback about my FlashGot-like addon

[pouriap19]

TL;DR: I'm creating an addon that lets you download links with external download managers. It's possible that other FlashGot functionality could also be added to it. But it will take a lot of time and effort to make so I want to know if people want it or not.

Migrating to Firefox Quantum was very difficult for me because I just couldn't live without some of my addons, including FlashGot. So after recently migrating to it I decided to re-create those essential addons that I couldn't live without. I started with FlashGot because it was very handy and ended up with something that was quite satisfying for my needs. (i.e. downloading links with Internet Download Manager).

But I know there are a lot of people out there like me who want FlashGot back, so I decided to go a little bit further and see if I can improve the addon to work with other download managers. I have used FlashGot.exe from the old FlashGot adddon to communicate with download managers. The result is Download Grab, an addon that intercepts Firefox's built-in download and lets you choose how you want to download a file. Other FlashGot functionality like downloading all links and gallery making, etc. could possibly be added to the addon in the future.

However this will take a lot of time and effort. So before I spend any more time on this I would like to have some input from you guys to see if it's even worth the try.

- Do you think the addon is even good?
- Do you think a lot of people are going to use it?
- Do you see any major obstacles or any other reasons that I should not invest time and effort in this?
Or any other thought you might have.

About the addon:
- The current version is for demonstration only and not a production release. There are things that are broken, there are a lot of logs, there is clutter, UI inconsistency etc.
- The user has to install a native client. There's no way around it.
- Because of the way it is currently detecting downloads there's no guarantee that it will intercept 100% of downloads.
- Currently only works on windows.
- Only FlashGot functionality that is currently implemented is downloading with external download manager.
- Flashgot.exe is quite outdated so I'm not sure how many download managers are detected successfully. Internet Download Manager and FlashGet are detected but the new version of Free Download Manager is not. I haven't tested other managers. (This is something that can be fixed and not related to Quantum limitations).

Screenshots: https://imgur.com/a/sjCI0Vb

The native client: https://www.dropbox.com/s/mpdvovpizd46a ... p.exe?dl=1
It installs a customized version of https://github.com/andy-portmen/native-client that works only with Download Grab. This application is needed for Download Grab to be able to communicate with download managers installed on the system. This is a requirement of Firefox Quantum.
(If you have the previous version of the native client installed, remove it using uninstall.bat and install the new one)

The addon file: https://www.dropbox.com/s/9gf5zobxu7iyb ... b.xpi?dl=1
It's unsigned so you will need FF developer edition to test it. You should go to 'about:config' and change xpinstall.signatures.required to false. This lets you install addons that are not signed by Mozilla.

[barbaz]

Moving to Web Tech

[therube]

(When messing with FF) I used (don't recall offhand) downloader (webextension) - to grab a link, copied to clipboard (through the webextension), then I'd manually paste the link into my download manager.


Webextensions suck. Period.
UI/UX are horrendous (in most cases).
Are so limited (like not being able to specify a download location...)
Not having shortcut keys...

I rarely download through a browser itself.
Almost always through my download manager.


If I'm in a "foreign" browser, or in a Profile that does not have FlashGot installed, I'll at times not download - until those conditions are rectified.


(No time to look, presently, but I'll get to it...)

So your extension is doing what, it becomes an interface between a link & a download manager (IDM, in this case)?

So it is not "detecting" anything, is that correct? There needs to be a link that you click on, or a link that pops up - something that "generates" a (normal) download dialog box. And then you hack into that box, offering IDM as an alternative?


Would it be useful (in that manner)? Yes.
A link is passed off to (download manager), download manager handles the download, ensures the download is successful & determines where the file ends up.
That is all good.


What it does not do is to "detect" links (or streams), so like if you wanted to download from Youtube, or similar, a place where there is no link (or otherwise) to initiate a download (hence, download dialog) then nothing is seen. OK, it's not made to deal with that situation. (Nice if it would, but OK.)


Webextensions suck.
IMO, if one wanted to develop something that sucks, on a platform that generally sucks, one (FF) with less & less appeal (& users?) as time goes on... one needs to be determined, & needs to understand that (as happened before - with Mozilla), that tomorrow, they may just pull the rug out from under you.


So, will you have users who might use your extension (if that is a concern of yours)? Hmm.


Presumably it would be portable to Chrome?
Or maybe not? Does Chrome support node.js? (Will FF support node.js - when you wake up tomorrow?)


If I were a FF user, yes, I'd take a look at it.
Yes, if links were more easily sent to my download manager, yes it would be worth it.


Is it FlashGot? No, not in a complete sense.
Where FlashGot (for me) is integral in my browsing, makes things very convenient, one tool that handles most all of my downloading needs.


And this, this would be one part. It would be worthwhile (to me).
(Worthwhile to others, to other FF users? Given what is expected of a current FF user [a mindless drone]...?)


And just to confirm, while I (use) FF Nightly, in FF Release, node.js is available to them also?
(And as you've mentioned, that signing joke, severely limits you - at this point, to only #dev versions of FF.)

[pouriap19]

(When messing with FF) I used (don't recall offhand) downloader (webextension) - to grab a link, copied to clipboard (through the webextension), then I'd manually paste the link into my download manager.

Sometimes that doesn't work because some request headers are missing. Namely cookies. Also I'm lazy. Which is why I developed this addon.

I rarely download through a browser itself.
Almost always through my download manager.

Same.

So your extension is doing what, it becomes an interface between a link & a download manager (IDM, in this case)?

Currently yes. Like how FlashGot gave you the options to choose to download with browser or with a DM.

So it is not "detecting" anything, is that correct? There needs to be a link that you click on, or a link that pops up - something that "generates" a (normal) download dialog box. And then you hack into that box, offering IDM as an alternative?

Yes and no. There's no way for webextensions to "hack into" UI. You can't change anything about UI. What my addon does is it looks at the HTTP request and response to guess if this is a download. And then shows its own download dialog instead of FF download dialog.

What it does not do is to "detect" links (or streams), so like if you wanted to download from Youtube, or similar, a place where there is no link (or otherwise) to initiate a download (hence, download dialog) then nothing is seen. OK, it's not made to deal with that situation. (Nice if it would, but OK.)

That's actually possible with WebExtensions because (afaik) they have full access to the page DOM. So things like downloading selected links or making galleries that FlashGot supported should be possible to remake. I don't know about streams like YouTube but I'm pretty sure those should also be possible. I have not delved into any of these yet because I don't personally need them. Hence making this thread to see if I should further develop the addon.

IMO, if one wanted to develop something that sucks, on a platform that generally sucks, one (FF) with less & less appeal (& users?) as time goes on... one needs to be determined, & needs to understand that (as happened before - with Mozilla), that tomorrow, they may just pull the rug out from under you.

Good point.

Presumably it would be portable to Chrome?

I don't like Chrome. Actually if possible I would like to take steps to make sure it is not portable to Chrome.
(But I might change my mind in the future).

And just to confirm, while I (use) FF Nightly, in FF Release, node.js is available to them also?

node.js is in the native client that you have to install to be able to use the addon. It's a portable version and is in the zip file I provided. It doesn't have anything to do with the browser.

[pouriap19]

No takers?

I've added the ability to download links in the page too:



I'm pretty sure the ability to sniff media streams in YouTube and other sites is also possible with WebExtensions because you have access to all the requests. DownloadGrab already grabs static media like mp3/mp4 files that are playable in <video> and <audio> tags.

Tho the core of the addon is FlashGot.exe which communicates with all the download managers. Giorgio must have put a lot of time into making it and I think it's such a waste to just let it rot.

[therube]

xpinstall.signatures.required;false is required.

If (Net Transport) is not already running, the first attempt to grab only opens the download manager.
Second attempt to grab, works.

The need to highlight a link, say for a .txt file that would otherwise open in the browser window is highly awkward.

(Oh, current Nightly have screen draw issues [on my end] that are a true PITA.)

FlashGot Link (type of action, sort of like, aka Grab Selection) requires that (above noted) highlight of the link.

Does it work? Yes.
Does it make downloading with FF more convenient? Yes.
Is it (nonetheless) awkward? Yes. (I.e., it's a webextension .)

Node...
Oh, that gives me great pause. As in I have no idea what it is, so trust is a very big factor. (Not to mention that it is, physically, big.)
Stuff is dropped into %appdata%/Local. Oh, that gives me great pause. (I tend not to keep or use anything there. Generally, what I use tends to be self-contained, or at the least in places where I want it.)
(Windows) Registry Entries. Probably no big deal, but again, I don't care for it.

[pouriap19]

xpinstall.signatures.required;false is required.

Right. Will edit the original post to mention that for people who want to test.

If (Net Transport) is not already running, the first attempt to grab only opens the download manager.
Second attempt to grab, works.

It's because FlashGot.exe is outdated. FlashGet 3 does weird things too. Shouldn't be a problem if FlashGot.exe is updated.

The need to highlight a link, say for a .txt file that would otherwise open in the browser window is highly awkward.

I'm not sure what you mean. Like when you click on a link for a .txt file and it opens in the browser?

(Oh, current Nightly have screen draw issues [on my end] that are a true PITA.)

I've tested it on FF73 and FF56 with no issues. Probably a Nightly thing?
Doesn't matter tho this is like an alpha demo.

FlashGot Link (type of action, sort of like, aka Grab Selection) requires that (above noted) highlight of the link.

Oh right. Will add context menu item for downloading a single link.

Does it work? Yes.
Does it make downloading with FF more convenient? Yes.
Is it (nonetheless) awkward? Yes. (I.e., it's a webextension .)

I have no love for WebExtensions either specially because they took TabScope and TabMixPlus from me. But aside from the native client that has to be installed separately, I think FlashGot is actually one of those addons that could be re-created with WebExtensions.

Node...
Oh, that gives me great pause. As in I have no idea what it is, so trust is a very big factor.

FlashGot and other "legacy" addons could always interact with the operating system and do things. It was just hidden from the users. This native client business that we have to do with webexts is just more explicit.
But I understand that this can not be explained to the user and as a result a lot of people will be reluctant to use the addon. But there's nothing we can do about that aside from making the source code public.

Not to mention that it is, physically, big.

Both size and the trust issues may be alleviated if we could have a native c++ executable instead of node . It would be way smaller (KBs probably) and it would be only one file instead of several scripts. Unfortunately I do not have the skills to do that, but if this project takes off I'm sure someone will chip in and make a .exe for us. It's even possible to integrate the native client into FlashGot.exe. Then we would only have one single .exe that does everything.
This is actually what I have in mind for the final release. I don't like this node business either. But it has one advantage: the code is in plain text and people can study it so more trust.
In either case the source code of the addon will be put on github.

Stuff is dropped into %appdata%/Local. Oh, that gives me great pause. (I tend not to keep or use anything there. Generally, what I use tends to be self-contained, or at the least in places where I want it.)
(Windows) Registry Entries. Probably no big deal, but again, I don't care for it.

An installer could be created to make everything more smooth. It would ask the user where they want to put the files. It would instill more trust too. The installer can take care of adding the registry entries as well behind the scene. (Those entries are needed to tell Firefox where our native client is).
I personally don't like anything that installs itself in my appdata folder either (or doesn't ask me where I want to install it).

Finally, thanks a lot for taking the time to test the addon. Much appreciated.

[therube]

> If (download manager) is not already running, the first attempt to grab only opens the download manager.

It's because FlashGot.exe is outdated.

Works in Legacy browsers .

Like when you click on a link for a .txt file and it opens in the browser?

If a link is normally handled by the browser itself, such as a .txt link (with proper mime-type), it will open in the browser window.
So to Grab it, you need to highlight (at least part of the link itself) for Grab to offer then Grab -> Grab selection.
(IOW, hovering the link does not give you a Grab -> Selection. Though you did mention... ^.)

(Oh, current Nightly have screen draw issues [on my end] that are a true PITA.)

I set everything "webrender" to 'false' & that seems to have "fixed" things.

I think FlashGot is actually one of those addons that could be re-created with WebExtensions.

Give Giorgio a nudge .

It's even possible to integrate the native client into FlashGot.exe

I wouldn't mess with FlashGot.exe - only because it is a known entity, (literally) unchanged in almost a decade .

[maxoku]

It's great idea and I would like really to use it, but it doesn't work in Waterfox Current.
I'm getting an error after installing that download grab could not be initialized.

[pouriap19]

New update:
- Context menu option for downloading a single link has been added.
- The native client is now an installer and also smaller in size (~5.6Mb)
- Waterfox issue is fixed

Works in Legacy browsers .

For me FlashGot does the same thing as DLGrab, i.e. only opens the manager the first time you download. Tested with:
Nettransport 2.96(x86)
Firefox 56(x86)
Windows 7(x64)
Flashgot 1.5.6.14
Maybe your FlashGot is on a different system with an older version of NetTransport?

It's great idea and I would like really to use it, but it doesn't work in Waterfox Current.
I'm getting an error after installing that download grab could not be initialized.

It's fixed now. Thank you for reporting this.
There's an installer available now. Remove the previous native client and use the installer, things should work.

[maxoku]

Can't download the native client. My antivirus claims it's a trojan. Could you fix that?
I guess the installer is a problem, so maybe post it normally without installer?

[pouriap19]

Can't download the native client. My antivirus claims it's a trojan. Could you fix that?
I guess the installer is a problem, so maybe post it normally without installer?

Ah crap. Sorry about that.
Yes it was the installer. I'm using another tool for creating the installer now. But it's still detected as malware by a few AVs.
Currently these AVs give false positive (according to VirusTotal.com):
- SecureAge APEX
- McAfee-GW-Edition
- Sophos ML
- BitDefenderTheta
- Cylance

It's just the five of them and I think they aren't very popular either so I guess we're ok for now. In the meantime I'll see what I can do to prevent these false positives.

[maxoku]

Installer generally is a good idea, but rather for the final version. As for testing it's best to keep it the most simple as it can be for quickly access.
In the meantime you'll figure out how to make the installer to not be falsly detected as a virus, but testing itself can't wait.

BTW I use AVG (rather popular) and it's very annoying and grumpy when detecting anything. Even if I add something as an exception it still doesn't allow me to open it. Or maybe the Windows itself is blocking it after seeing that an antivirus detected a virus in a file? I don't know.

[pouriap19]

Installer generally is a good idea, but rather for the final version. As for testing it's best to keep it the most simple as it can be for quickly access.
In the meantime you'll figure out how to make the installer to not be falsly detected as a virus, but testing itself can't wait.

BTW I use AVG (rather popular) and it's very annoying and grumpy when detecting anything. Even if I add something as an exception it still doesn't allow me to open it. Or maybe the Windows itself is blocking it after seeing that an antivirus detected a virus in a file? I don't know.

I thought maybe even in test people would be more inclined if an installer was available since it's more visual and familiar. But maybe you're right.
BTW I've already updated the installer and there should be no problem with AVG. Only the five AV I listed above detect it as malware. Did you try the new version of the installer? (The link is the same link i provided before).

Edit: Now that I read my previous post I think I didn't make it obvious that I have fixed the installer

[maxoku]

Okay, now it works.

First thing it tries to download script or font files while loading some pages which is strange. Doesn't do it when I exclude common web files though. I've reported them, so maybe you'll be able to fix it. On some pages though it opened empty windows with a reset slider (?), so I couldn't report them. I've made a screenshot, but I can't put it here.

Nevertheless it works great and how it suppose to. It grabs the links and send to download manager just like FlashGot.
Does option 'Override Firefox's download dialog cause download grab dialog window to appear? Cuz if yes then it does it even if the option is disabled.

I have some suggestions:
1. Add a 'Grab Image' in contex menu when right click on an image. Maybe also with a video.
2. Grab selection shouldn't appear in context menu when only a simple text is selected. If not possible then maybe option hide this item (maybe along with Grab All) but leaving Grab Link.
3. Could FlashGot.exe be permanently running like Process Spawner? That could make it work faster and smoother.
4. Toolbar button could show list of files to those grabbed in the current tab and clear after refreshing/loading new page. It might be an option to choose how it would work.
5. It would be nice if it could grab medias. It does it now in that toolbar button but it's not perfect.
First it doesn't notify that it grabbed something.
Second it requires clicking two times to download it.
Third it doesn't show the size of the file on the list.
6. Or instead to change anything it could utilize (if possible) other add-ons that grab them to use Download Grab instead of the browser.
I use two add-ons to grab videos. 'Video Downloader Prime' and 'Save Video As'.
'Video Downloader Prime' is nice cuz it shows resolution of videos and allows to remove single files from the list.
But it also show empty files and doesn't grab all videos. 'Save Video As' doesn't have these problems, but also those advantages above.

It's practically perfect as it is, so if you don't like any of my suggestions just ignore them.
I'd like to thank you for creating it. I was really getting nuts without properly downloading files.
I think a lot of people will use it if they have an external download manager.
It detects old FDM 3 and GetRight.

PS: You could use Waterfox Current, it allows legacy add-ons though they need to be rewritten (not to webextension, but normally).
What other add-ons are you rewriting? I might be interested in them as well.
I couldn't live e.g. without legacy Stylish to modify the browser's UI. Good it was rewritten. Now I miss only TabMixPlus.


Edit1: It seems that it can't override the dialog window in one case. When mp4 videos are suppose to be opened and played in the browser. I've changed in about:config 'media.play-stand-alone' to false and videos aren't being opened. Browser asks now what to do with it, but Download Grab isn't appearing before that.
Edit2: For some reason it also can't do it with txt files it seems. (correction: Exclude common web files prevents detecting txt files, weird.)
Edit3: It has problems with Polish letters when the dialog window appears. Browser decodes them properly.
Edit4: It sometimes doesn't recognize the file's name (unknown) while the browser see it properly.
Edit5: It sometimes sends file's name to comments instead to file name. Though FlashGot does the same, so probably it can't be fixed.
I've sent you some links in private message to test it.


Three more suggestions:
1. Change 'Firefox' to 'Browser' in dialog window.
2. Allow to choose which file extensions exclude.
3. Allow to choose (if possible) which file extensions automatically will be send to download manager without asking.