Need feedback about my FlashGot-like addon

General discussion about web technology.
TAO
Posts: 1
Joined: Wed May 20, 2020 10:23 am

Re: Need feedback about my FlashGot-like addon

Post by TAO »

Really interesting, I've tried it and it's a good start but needs more work to be done.
I keep my eye on this project.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0
pouriap19
Posts: 17
Joined: Wed Apr 29, 2020 6:21 pm

New update

Post by pouriap19 »

Hey it's me.
I've made some improvements and attempted to fix the issues that maxoku had found:

Changes in v0.5.4a:
Major changes in download detection policies. Problems with falsely detecting downloads should be minimized now.
Fixed a problem with detecting download file names.
Added option to specify which file extensions should be grabbed.
Added option to specify which file extensions should be ignored.
Added option to specify which file extensions should be automatically sent to default DM.
Fixed a problem with the report button.

Known problems:
If a file's name is sent to the browser using the "Content-disposition" header and that name has UTF-8 characters in it then the name will be shown as gibberish. This is a browser problem and I can't do anything about it. But it's possible I'll be able to work around it with a hack.

Things that will be fixed in the future:
- When a text with no links is selected the 'Grab links' option still appears
- Add option for to grab image/video when right clicking on them
- FlashGot.exe will eventually be merged with the native host application, so there will be no need for node.exe and its scripts. Just FlashGot.exe
- Media will be sniffed in the page and grabbed. When media is grabbed a notification icon will appear in the address bar. Currently media that is explicitly loaded like <video> and <audio> are grabbed.
- Show number of grabbed items on toolbar button
- Option to show only grabbed items for the current page
- Files grabbed in private browsing will be only visible in private browsing windows and will be cleared when private browsing is over

Note:
If DLGrab tried to download something that it was not supposed to, for example a script that should be loaded into the page, then first kindly report it using the report button at the bottom of the download dialog, then choose "Continue with -> Browser" which will cause this file to be loaded as normal without the intervention of DLGrab.

Download/Update link is available in my original post.

maxoku wrote: Sat May 16, 2020 9:16 pm Or instead to change anything it could utilize (if possible) other add-ons that grab them to use Download Grab instead of the browser.
It's possible. But the authors of those addons have to add this functionality to their addons.
What other add-ons are you rewriting? I might be interested in them as well.
Currently none. TabmixPlus is not possible to recreate with the current API. But some of its features might be. I might give it a try in the future but currently I'm occupied with Download Grab.
It seems that it can't override the dialog window in one case. When mp4 videos are suppose to be opened and played in the browser.
This is a feature. Some people might want to view these files in the browser instead of downloading them. I've added an option for this now that you can disable if you want.
For some reason it also can't do it with txt files it seems.
Same as above. Plain text files, PDF files and media files are by default opened in browser. If you want to be prompted for download use the option mentioned above.

(Sorry for the long wait. I've been trying to install Visual Studio for the past week and I'm still unsuccessful. Piece of garbage. :x )
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0
User avatar
maxoku
Posts: 16
Joined: Tue Sep 17, 2019 6:12 pm
Location: Poland

Re: Need feedback about my FlashGot-like addon

Post by maxoku »

Great, seems everything you mentioned works properly now. Thanks 👍
I'll be waiting for more improvements to test them anytime.
Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
User avatar
therube
Ambassador
Posts: 7924
Joined: Thu Mar 19, 2009 4:17 pm
Location: Maryland USA

Re: Need feedback about my FlashGot-like addon

Post by therube »

So node.exe (node.js, & DLGrabspawner.exe) persist in the FF process space.

There have been odd times, on opening FF, that I'd get a "message" (yeah, don't ask what it said), & I'd have to restart FF to get DLG back working.


It works. It's nice. It allows using my download manager. It's better then anything native from FF.
But... it's not enough, because there is no "detection".
So unless there is a link you can select, or a download dialog request it can hook in to...


Now, don't get me wrong, when I say, "not enough", it is not enough to drag FF from the scourge (isn't that what trump calls it) that it is, to anything remotely like, FF. IOW, am I going to switch from SeaMonkey, & NoScript & FlashGot (& everything else that "just works") to FF any time soon? LOL.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.19) Gecko/20110420 SeaMonkey/2.0.14 Pinball NoScript FlashGot AdblockPlus
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0 SeaMonkey/2.53.4
User avatar
therube
Ambassador
Posts: 7924
Joined: Thu Mar 19, 2009 4:17 pm
Location: Maryland USA

Re: Need feedback about my FlashGot-like addon

Post by therube »

Maybe if DLG had a 'Paste from clipboard' (&/or 'Grab from clipboard')... that would be a benefit.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.19) Gecko/20110420 SeaMonkey/2.0.14 Pinball NoScript FlashGot AdblockPlus
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0 SeaMonkey/2.53.4
User avatar
therube
Ambassador
Posts: 7924
Joined: Thu Mar 19, 2009 4:17 pm
Location: Maryland USA

Re: Need feedback about my FlashGot-like addon

Post by therube »

Oh, I'm running this setup, sandboxed, Sandboxie.

SBIE2205 Service not implemented: ConsoleInit (C00000D4)
a node.exe "console" window opened (empty, black)

Download Grab
ERROR: ERROR: No download managers found on the system

(A browser restart should "fix" this.)
Yes, but maybe ? with a DLG disable/enable toggling too.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.19) Gecko/20110420 SeaMonkey/2.0.14 Pinball NoScript FlashGot AdblockPlus
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0 SeaMonkey/2.53.4
User avatar
therube
Ambassador
Posts: 7924
Joined: Thu Mar 19, 2009 4:17 pm
Location: Maryland USA

Re: Need feedback about my FlashGot-like addon

Post by therube »

i don't know if this is related to DLG, but...

running FF sandboxed, Sandboxie
i've been keeping FF open, running, in the sandbox
i update (Nightly) FF (outside the sandbox) - even though the instance running sandboxed has not been restarted
the sandboxed instance (source if you will) stems from the non-sandboxed FF
(iow, FF is extracted, outside the sandbox, & from there, "firefox.exe" is run, sandboxed - so the sandboxed "version" is the unsandboxed "version", only told to run, sandboxed [heh, did that confuse you ;-)])

anyhow, been running like this for a while now...
computer goes to sleep, whenever, wakes up & i continue on...

yesterday (or day before ?), existing instance of FF, i go to open a link in a new window
window opens, but not the link
type a url in, hit return & nothing, Ctrl+W to close the window, nothing
the close 'X' box does close the window
attempt to open in new tab, no dice

FF is fubar'd

Quit
restart & carry on...

get to today, same situation, FF not working as expected

don't know if this is quirk in FF, FF/Sandboxie, FF/Sandboxie & updates, DLG/node.exe...

so, just throwing this out there...

(& the only reason i happen to be purposely keeping FF open, is because i was on one of those "never ending" pages, & while i can restore the page, it doesn't restore it's position - in the never ending, so all i can do is to try to find where i had left off... which since this has now occurred, twice, i'm going to have to do anyhow. will find another site to peruse - that is not never ending...)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.19) Gecko/20110420 SeaMonkey/2.0.14 Pinball NoScript FlashGot AdblockPlus
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0 SeaMonkey/2.53.4
User avatar
therube
Ambassador
Posts: 7924
Joined: Thu Mar 19, 2009 4:17 pm
Location: Maryland USA

Re: Need feedback about my FlashGot-like addon

Post by therube »

What do you mean by "detection"?
Only that FlashGot & yes, I know this isn't FG, in addition to "feeding" links to a download manager, could also "detect" links (i.e., FlashGot Media), which this doesn't do at all (which I understand). But as I look at it, FG did both. With this, & more pointing out that while this does some, the other part, detection, there would be another tool needed to accomplish that.
not enough
Well, I can certainly judge "not enough" - for me :-).
Even NoScript - until it was able to work with bookmarklets (which it could not do, early on), was not enough - for me.
(And I did not use it - until it did work with bookmarklets.)

Paste from(/Grab from) clipboard, with the thought that if I have to use another utility to "detect" a link, if that other utility can copy the link to the clipboard, & if DLG can grab that link (clipboard copy), then in that vein DLG has "detected" the link (so then able to be passed to download manager).


Now, don't get me wrong, this works, & it is a help.


But from my perspective (& I'll broaden scope), Quantum is limited in what it does (which is the real problem).
And extensions, & users, & developers have to live with - worse. Worse then they had.
To me, that just doesn't make sense :roll:.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.19) Gecko/20110420 SeaMonkey/2.0.14 Pinball NoScript FlashGot AdblockPlus
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0 SeaMonkey/2.53.4
User avatar
maxoku
Posts: 16
Joined: Tue Sep 17, 2019 6:12 pm
Location: Poland

Re: Need feedback about my FlashGot-like addon

Post by maxoku »

therube wrote: Tue Jun 02, 2020 3:05 amin addition to "feeding" links to a download manager, could also "detect" links (i.e., FlashGot Media), which this doesn't do at all
It has a toolbar button that grabs many files to its list. Also medias, maybe not all of them, but still. As I use it now it detects all mp4 video files that suppose to. It's still alpha, so it's not perfect and fully comfortable to use. Generally I use Video Downloader Prime add-on that grabs links and I right click them to grab it with DLG. It still has its issues, but it's better than nothing. What add-ons do you use for media grabbing?
Well, I can certainly judge "not enough" - for me
Still don't know what you need besides the media.
Paste from(/Grab from) clipboard, with the thought that if I have to use another utility to "detect" a link, if that other utility can copy the link to the clipboard, & if DLG can grab that link (clipboard copy), then in that vein DLG has "detected" the link (so then able to be passed to download manager).
Doesn't your download manager have such function to grab links from clipboard? Does FlashGot has it?

BTW update your DLG, it's in version 0.6.2a now. It's much better than before.
Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
pouriap19
Posts: 17
Joined: Wed Apr 29, 2020 6:21 pm

Re: Need feedback about my FlashGot-like addon

Post by pouriap19 »

@therube
The problems you're facing seems to be from node running in the background. Apparently your sandbox doesn't like console windows (that's what Googling that error code suggested), and node is console window.
Currently the addon's ability to detect file downloads seems to be alright. The next step is to update flashgot.exe to work with recent versions of download managers. It already works with a lot of them but I want to add new download managers and update/remove the ones that do not work anymore.
I've contacted Giorgio about some help with building flashgot.exe, and hopefully we'll have a brand new flashgot.exe not long from now. This more might or might not help with your issues.
After that I'll go back to the addon and add the ability to grab media.
Media grabbing will be added to DLG. And it will be good. I promise. :geek:
After that I'll go back to the native host and get rid of node.js for good. Everything will be standalone executables eventually.
This is the general roadmap I have in mind for this project.

Also I'm going to be away for a couple days so there won't be any updates from me in that time.

Cheers.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0
User avatar
therube
Ambassador
Posts: 7924
Joined: Thu Mar 19, 2009 4:17 pm
Location: Maryland USA

Re: Need feedback about my FlashGot-like addon

Post by therube »

DLG is unable to download from "firefox".
Oh, the irony ;-).

URL: https://send.firefox.com/download/8cbe1 ... I0OzMFMuEg
(I have no idea how long send.firefox.com download links are valid for, much less how long that Mozilla "service" will exist.)
And, to top it off, the service fails in SeaMonkey (web browser), either returning an (invalidly reported) 404, or SeaMonkey generates non-stop Request's, with the service eventually "banning" you, then returning 429 Too Many Requests :twisted:.
Edit. Turns out this is an issue with current beta versions of SeaMonkey (2.53.4beta).
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.19) Gecko/20110420 SeaMonkey/2.0.14 Pinball NoScript FlashGot AdblockPlus
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0 SeaMonkey/2.53.4
User avatar
maxoku
Posts: 16
Joined: Tue Sep 17, 2019 6:12 pm
Location: Poland

Re: Need feedback about my FlashGot-like addon

Post by maxoku »

Looks like Firefox Send is a download manager itself like mega. DLG shouldn't launch on those downloads, cuz no download manager would be able to do anything with them.
I don't really know how exactly that works, but seems those web download managers download files to the cache and then they can be only saved to the hard drive by the browser while it's its cache. I remember that in the past I wanted to save a file from mega with FlashGot. Nothing happened and I lost the file. Even the button of DownThemAll OneClick! is inactive, cuz even if it's a browser download manager it can't do anything whith such download.

So summarizing it's like it suppose to be. Btw you should give a smaller file for testing. :lol:
Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0
pouriap19
Posts: 17
Joined: Wed Apr 29, 2020 6:21 pm

Re: Need feedback about my FlashGot-like addon

Post by pouriap19 »

therube wrote: Mon Jun 15, 2020 11:29 am DLG is unable to download from "firefox".
Oh, the irony ;-).
Certain parts of addons do not work in "special pages" for "security". These special pages are some Mozilla and Firefox domains + pages of other addons.
It's because of the way Firefox Send works. See answer to barbaz below.
Last edited by pouriap19 on Tue Jun 16, 2020 4:43 pm, edited 2 times in total.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0
pouriap19
Posts: 17
Joined: Wed Apr 29, 2020 6:21 pm

version 0.11.0a released

Post by pouriap19 »

Guys I've release a new version of the native host and a new version of the addon.
For downloading and more info go to DLG's github page: https://github.com/pouriap/Firefox-Down ... g/v0.11.0a
I've also created a separate github repository for FlashGot.exe in case you want to build for yourself or inspect the source code : https://github.com/pouriap/FlashGot-For-DownloadGrab

This isn't really a good time for uploading a release because I will not be available until Saturday so if there's any bugs you'll have to wait until I'm back. :P
But I haven't posted any update in a while so I wanted to have some activity so you won't think I'm slacking away. I've been working very hard on this.

Also, I have good news and bad news.
Good news is the next thing I'm going to start working on is grabbing media. (yay!)
The bad news is due to the nature of streaming protocols DLG will not be able to download stream media (YouTube and any other major video site) with download managers. Instead we will have to use an internal downloader that will come with the native host. This is because all major video sites nowadays use protocols that split up the video into small parts, and the audio is also split apart, so the program will have to download all these parts and then glue them together. Download managers obviously cannot do this.
So media will be grabbed, but with DLG's internal downloader not with a download manager.

Have fun.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0
barbaz
Senior Member
Posts: 10841
Joined: Sat Aug 03, 2013 5:45 pm

Re: Need feedback about my FlashGot-like addon

Post by barbaz »

pouriap19 wrote: Tue Jun 16, 2020 9:24 am
therube wrote: Mon Jun 15, 2020 11:29 am DLG is unable to download from "firefox".
Oh, the irony ;-).
Certain parts of addons do not work in "special pages" for "security". These special pages are some Mozilla and Firefox domains
So does it work if you set about:config > privacy.resistFingerprinting.block_mozAddonManager to true (you might need to create this boolean pref) and/or blank out about:config > extensions.webextensions.restrictedDomains ?
*Always* check the changelogs BEFORE updating that important software!
-
Post Reply