Page 2 of 3
Re: POST data includes headers
Posted: Sun Mar 15, 2015 6:32 pm
by MoSal
Only 'a=b' is passed with both POST and RAWPOST here.
Re: POST data includes headers
Posted: Mon Mar 16, 2015 12:08 am
by flashgot.user
Are you testing with an HTML form or with an XMLHttpRequest? Because the latter doesn't contain headers in its upload stream.
Anyway, I'll need your log produced by this version:
Code: Select all
http://rghost.net/private/6jlN6NpwD/dfcbeb58cade67e99a69c0e64e317057
, don't merge this one yet. It does some debug logging, and I'm still not sure it should be like this.
#1. The [RAWPOST] placeholder was "random", because:
1) HTML form POSTs contain headers before the POST data.
2) POSTs made with XMLHttpRequest don't - they contain just the POST data (whatever you pass to xhr.send()).
This version does the same as the browser does: inject the headers into XHR POSTs' data so that it has the same format as HTML form POSTs: "Content-Type: ...\r\nContent-Length: ...\r\n\r\n<POST data>". I mean, it injects them into the placeholder, not into the actual channel.uploadStream.
#2. It also converts multipart data to urlencoded so that those who use [POST] and can't handle non-urlencoded data (and that includes flashgot.exe) will work with multipart data without any modifications. The problem is, this conversion will cause discrepancy between [POST] and [RAWPOST]:
1) [RAWPOST] contains original multipart data along with its original Content-Type and Content-Lengh headers.
2) [POST] contains urlencoded data and [HEADERS] contains Content-Type and Content-Length for [POST], not [RAWPOST].
So those who use [RAWPOST] must remember to ignore these 2 headers in [HEADERS].
Here's a sample multipart POST dump that demonstrates that. The test form is the same as in my
previous post, only with enctype="multipart/form-data".
Output:
Code: Select all
argc=9
[0]: len=49: 'C:\hiawatha-9.2.UNP\hiawatha-9.2\wwwroot\test.exe'
00000000: 43 3a 5c 68 69 61 77 61 74 68 61 2d 39 2e 32 2e C:\hiawatha-9.2.
00000010: 55 4e 50 5c 68 69 61 77 61 74 68 61 2d 39 2e 32 UNP\hiawatha-9.2
00000020: 5c 77 77 77 72 6f 6f 74 5c 74 65 73 74 2e 65 78 \wwwroot\test.ex
00000030: 65 e
[1]: len=4: '-url'
00000000: 2d 75 72 6c -url
[2]: len=25: 'http://localhost/test.php'
00000000: 68 74 74 70 3a 2f 2f 6c 6f 63 61 6c 68 6f 73 74 http://localhost
00000010: 2f 74 65 73 74 2e 70 68 70 /test.php
[3]: len=5: '-post'
00000000: 2d 70 6f 73 74 -post
[4]: len=3: 'a=b'
00000000: 61 3d 62 a=b
[5]: len=8: '-rawpost'
00000000: 2d 72 61 77 70 6f 73 74 -rawpost
[6]: len=240: 'Content-Type: multipart/form-data; boundary=---------------------------54912754623
Content-Length: 133
-----------------------------54912754623
Content-Disposition: form-data; name="a"
b
-----------------------------54912754623--
'
00000000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 6d 75 Content-Type: mu
00000010: 6c 74 69 70 61 72 74 2f 66 6f 72 6d 2d 64 61 74 ltipart/form-dat
00000020: 61 3b 20 62 6f 75 6e 64 61 72 79 3d 2d 2d 2d 2d a; boundary=----
00000030: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
00000040: 2d 2d 2d 2d 2d 2d 2d 35 34 39 31 32 37 35 34 36 -------549127546
00000050: 32 33 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 23..Content-Leng
00000060: 74 68 3a 20 31 33 33 0d 0a 0d 0a 2d 2d 2d 2d 2d th: 133....-----
00000070: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
00000080: 2d 2d 2d 2d 2d 2d 2d 2d 35 34 39 31 32 37 35 34 --------54912754
00000090: 36 32 33 0d 0a 43 6f 6e 74 65 6e 74 2d 44 69 73 623..Content-Dis
000000a0: 70 6f 73 69 74 69 6f 6e 3a 20 66 6f 72 6d 2d 64 position: form-d
000000b0: 61 74 61 3b 20 6e 61 6d 65 3d 22 61 22 0d 0a 0d ata; name="a"...
000000c0: 0a 62 0d 0a 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d .b..------------
000000d0: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
000000e0: 2d 35 34 39 31 32 37 35 34 36 32 33 2d 2d 0d 0a -54912754623--..
[7]: len=8: '-headers'
00000000: 2d 68 65 61 64 65 72 73 -headers
[8]: len=68: 'content-type: application/x-www-form-urlencoded
content-length: 3
'
00000000: 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 20 61 70 content-type: ap
00000010: 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 77 77 2d plication/x-www-
00000020: 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 65 64 0d form-urlencoded.
00000030: 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a .content-length:
00000040: 20 33 0d 0a 3..
Re: POST data includes headers
Posted: Mon Mar 16, 2015 9:39 am
by MoSal
The logs triggered anti-spam protection. So, I pasted them elsewhere:
http://ix.io/gVc
Re: POST data includes headers
Posted: Wed Mar 18, 2015 11:24 am
by flashgot.user
Weird. It's extracted fine, but then gets lost somewhere, and I have no idea where or why. I added more logging, can you try it and show the log?
Also, try [HEADERS] - it's also multiline, maybe this has something to do with it.
Code: Select all
http://rghost.net/private/7YmxFH6d7/68e989df63e16898cb7b280e23fb392f
Here's my log, just in case you were wondering if this thing works at all.
Command line arguments:
Code: Select all
[;;;URL;;;] [-u ;;;UA;;;] [-e ;;;REFERER;;;] [-D ;;;FOLDER;;;] [-o ;;;FNAME;;;] [-K ;;;CFILE;;;] [-p ;;;RAWPOST;;;] [--header=HEADER] [-h HEADERS] -r -VVV
Log:
Code: Select all
2015-03-18T11:01:46.623Z Extracting post data...
2015-03-18T11:01:46.623Z res.rawPostData (len=73): 'Content-Type: application/x-www-form-urlencoded
Content-Length: 3
a=b'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Host', value='localhost'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='User-Agent', value='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Accept', value='text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Accept-Encoding', value='gzip, deflate'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='DNT', value='1'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Referer', value='http://localhost/test.html'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Connection', value='keep-alive'
2015-03-18T11:01:46.623Z origPostContentType='null'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Content-Type', value='application/x-www-form-urlencoded'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Content-Length', value='3'
2015-03-18T11:01:46.623Z extractPostData: OUT:
2015-03-18T11:01:46.623Z res.rawPostData: len=73: 'Content-Type: application/x-www-form-urlencoded
Content-Length: 3
a=b'
2015-03-18T11:01:46.623Z res.postData: len=3: 'a=b'
2015-03-18T11:01:46.623Z res.extraHeaders: [object Object]
2015-03-18T11:01:46.623Z 'content-type: application/x-www-form-urlencoded'
2015-03-18T11:01:46.624Z 'content-length: 3'
2015-03-18T11:01:48.808Z Preprocessing done in ms9
2015-03-18T11:01:48.810Z Starting dispatch
2015-03-18T11:01:50.212Z FlashGotDMCust::performJob: l.rawPostData: len=73: 'Content-Type: application/x-www-form-urlencoded
Content-Length: 3
a=b'
2015-03-18T11:01:50.212Z FlashGotDMCust::performJob: l.extraHeaders: [object Object]
2015-03-18T11:01:50.212Z FlashGotDMCust::performJob: extraHeaders: len=68: 'content-type: application/x-www-form-urlencoded
content-length: 3
'
2015-03-18T11:01:50.234Z Running C:\hiawatha-9.2.UNP\hiawatha-9.2\wwwroot\test.exe ;;;http://localhost/test.php ;;; -u ;;;Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25 ;;; -e ;;;http://localhost/test.html ;;; -D ;;;C:\Users\пк\Downloads\1 ;;; -o ;;;test_php.mp3 ;;; -K ;;;C:\Users\7349~1\AppData\Local\Temp\flashgot.1e3apr2v.default\cookies ;;; -p ;;;Content-Type: application/x-www-form-urlencoded
Content-Length: 3
a=b ;;; --header=content-type: application/x-www-form-urlencoded --header=content-length: 3 -h content-type: application/x-www-form-urlencoded
content-length: 3
-r -VVV -- async
2015-03-18T11:01:50.247Z Native execution time 13
2015-03-18T11:01:50.289Z Dispatch done in ms1478
Total processing time: ms1489
Re: POST data includes headers
Posted: Wed Mar 18, 2015 4:41 pm
by MoSal
Tested with a fresh profile.
http://ix.io/gXG
Re: POST data includes headers
Posted: Wed Mar 18, 2015 8:36 pm
by flashgot.user
Looks like you have autostart enabled, and I don't. I couldn't make autostart work, so you'll have to test it for me.
Code: Select all
http://rghost.net/private/84h9sZlGg/2c16726a4d25cf97648b586a03f4c407
Re: POST data includes headers
Posted: Thu Mar 19, 2015 12:01 pm
by MoSal
Okay. This one worked.
Thank you.
http://ix.io/gYx
Is this the correct behaviour of HEADER?
More importantly, the space after each placeholder is gone. This should be documented in the Changelog if this code is pulled by upstream. Some shell/batch scripts used to parse the arguments will need to be adjusted to the change.
Re: POST data includes headers
Posted: Thu Mar 19, 2015 3:25 pm
by flashgot.user
Yes, [HEADER] is a "flattened" alternative to [HEADERS]. It's for those who want one header per argument, e.g. aria2.
As for the "missing" space, I guess it's time to fork for me.
This version comes with all the extra spaces you'll ever need. But without the [HEADER] placeholder because it's incompatible with idiocy.
http://rghost.net/private/6mdRrrB4F/d1423c9503676a6204a4318da3bf1075
Re: POST data includes headers
Posted: Thu Mar 19, 2015 4:20 pm
by MoSal
Thank you for all your help.
The added spaces annoyed me too. But I think Giorgio will opt to keep them for backwards-compatibility.
Edit: Or maybe add a space after each ']' instead of after the placeholder and hope no one's script will break.
Re: POST data includes headers
Posted: Thu Mar 19, 2015 5:25 pm
by flashgot.user
Meh, forgot to remove some debug logging leftovers. This version should be clean:
Code: Select all
http://rghost.net/private/8wcv4XgJL/bdc170463cdb1e32b1bc41e19b88171a
Re: POST data includes headers
Posted: Thu Mar 19, 2015 9:31 pm
by Giorgio Maone
I don't care about spaces, I'd rather see them removed.
Can I have a clean version with [HEADER] and without extra spaces to be merged for good?
Thanks
Re: POST data includes headers
Posted: Tue Mar 24, 2015 2:46 pm
by flashgot.user
Did you read
this thread? It only looks like an extra space because runNative uses args.join(" ") for logging, but in fact, the second ";;;" (the one after the placeholder name) went to a different argument because it was captured in m[5]. And the reason it went "missing" is that I used the r3 regexp which captures the second ";;;" in m[4].
And while we're at removing extra spaces, winEscHack, in its current form, is unnecessary. Because for normal exe files, you don't need to do anything special with arguments because nsIProcess does everything for you, i.e. quotes arguments that contain whitespace and escapes double quotes with a backslash. And for batch scripts, there are a couple more things to do besides quoting arguments that contain ;&=. For one, commas also work as argument separators. :P Then ^%!&|<> - they must be escaped with "^", the first 3 at the very least. Then double quotes - they must be doubled. Finally, no amount of sanitizing will protect you from a buggy script that uses "%"-style variables ("%var%") when delayed expansion is on.
So, what this version does:
1) Uses the r3 regexp instead of r1.
2) Unconditionally uses winEscHackBatch for batch files, i.e. there's no pref to disable it.
3) For normal exe files, winEscHack is false by default.
4) New pref
flashgot.intercept.extra_request_headers - a comma-separated case-insensitive list of request header names to extract from requests. Default is "origin, /^x-./". The /.../ syntax is used to specify a JavaScript regular expression, i.e. /^x-./ matches everything that starts with "x-" or "X-", e.g. "X-Requested-With".
Code: Select all
http://rghost.net/private/8jrbfYyfb/13cbb284880cca59176da43965d69ed6
Re: POST data includes headers
Posted: Mon Apr 27, 2015 3:31 pm
by MoSal
I hope a working RAWPOST will be included in an official release someday.
Re: POST data includes headers
Posted: Sat Sep 12, 2015 7:33 pm
by MoSal
MoSal wrote:I hope a working RAWPOST will be included in an official release someday.
So...
Is flashgot with working RAWPOST ever going to be available in AMO?
Re: POST data includes headers
Posted: Tue Sep 15, 2015 6:42 pm
by flashgot.user
MoSal wrote:Is flashgot with working RAWPOST ever going to be available in AMO?
In case you've lost the version I uploaded before, I can reupload it. And if you haven't, why do you even care about AMO?