POST data includes headers

Bug reports and enhancement requests
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

Only 'a=b' is passed with both POST and RAWPOST here.
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

Are you testing with an HTML form or with an XMLHttpRequest? Because the latter doesn't contain headers in its upload stream.
Anyway, I'll need your log produced by this version:

Code: Select all

http://rghost.net/private/6jlN6NpwD/dfcbeb58cade67e99a69c0e64e317057
Giorgio, don't merge this one yet. It does some debug logging, and I'm still not sure it should be like this.
#1. The [RAWPOST] placeholder was "random", because:
1) HTML form POSTs contain headers before the POST data.
2) POSTs made with XMLHttpRequest don't - they contain just the POST data (whatever you pass to xhr.send()).
This version does the same as the browser does: inject the headers into XHR POSTs' data so that it has the same format as HTML form POSTs: "Content-Type: ...\r\nContent-Length: ...\r\n\r\n<POST data>". I mean, it injects them into the placeholder, not into the actual channel.uploadStream.

#2. It also converts multipart data to urlencoded so that those who use [POST] and can't handle non-urlencoded data (and that includes flashgot.exe) will work with multipart data without any modifications. The problem is, this conversion will cause discrepancy between [POST] and [RAWPOST]:
1) [RAWPOST] contains original multipart data along with its original Content-Type and Content-Lengh headers.
2) [POST] contains urlencoded data and [HEADERS] contains Content-Type and Content-Length for [POST], not [RAWPOST].
So those who use [RAWPOST] must remember to ignore these 2 headers in [HEADERS].
Here's a sample multipart POST dump that demonstrates that. The test form is the same as in my previous post, only with enctype="multipart/form-data".
Output:

Code: Select all

argc=9
  [0]: len=49: 'C:\hiawatha-9.2.UNP\hiawatha-9.2\wwwroot\test.exe'
00000000: 43 3a 5c 68 69 61 77 61  74 68 61 2d 39 2e 32 2e  C:\hiawatha-9.2.
00000010: 55 4e 50 5c 68 69 61 77  61 74 68 61 2d 39 2e 32  UNP\hiawatha-9.2
00000020: 5c 77 77 77 72 6f 6f 74  5c 74 65 73 74 2e 65 78  \wwwroot\test.ex
00000030: 65                                                e
  [1]: len=4: '-url'
00000000: 2d 75 72 6c                                       -url
  [2]: len=25: 'http://localhost/test.php'
00000000: 68 74 74 70 3a 2f 2f 6c  6f 63 61 6c 68 6f 73 74  http://localhost
00000010: 2f 74 65 73 74 2e 70 68  70                       /test.php
  [3]: len=5: '-post'
00000000: 2d 70 6f 73 74                                    -post
  [4]: len=3: 'a=b'
00000000: 61 3d 62                                          a=b
  [5]: len=8: '-rawpost'
00000000: 2d 72 61 77 70 6f 73 74                           -rawpost
  [6]: len=240: 'Content-Type: multipart/form-data; boundary=---------------------------54912754623
Content-Length: 133

-----------------------------54912754623
Content-Disposition: form-data; name="a"

b
-----------------------------54912754623--
'
00000000: 43 6f 6e 74 65 6e 74 2d  54 79 70 65 3a 20 6d 75  Content-Type: mu
00000010: 6c 74 69 70 61 72 74 2f  66 6f 72 6d 2d 64 61 74  ltipart/form-dat
00000020: 61 3b 20 62 6f 75 6e 64  61 72 79 3d 2d 2d 2d 2d  a; boundary=----
00000030: 2d 2d 2d 2d 2d 2d 2d 2d  2d 2d 2d 2d 2d 2d 2d 2d  ----------------
00000040: 2d 2d 2d 2d 2d 2d 2d 35  34 39 31 32 37 35 34 36  -------549127546
00000050: 32 33 0d 0a 43 6f 6e 74  65 6e 74 2d 4c 65 6e 67  23..Content-Leng
00000060: 74 68 3a 20 31 33 33 0d  0a 0d 0a 2d 2d 2d 2d 2d  th: 133....-----
00000070: 2d 2d 2d 2d 2d 2d 2d 2d  2d 2d 2d 2d 2d 2d 2d 2d  ----------------
00000080: 2d 2d 2d 2d 2d 2d 2d 2d  35 34 39 31 32 37 35 34  --------54912754
00000090: 36 32 33 0d 0a 43 6f 6e  74 65 6e 74 2d 44 69 73  623..Content-Dis
000000a0: 70 6f 73 69 74 69 6f 6e  3a 20 66 6f 72 6d 2d 64  position: form-d
000000b0: 61 74 61 3b 20 6e 61 6d  65 3d 22 61 22 0d 0a 0d  ata; name="a"...
000000c0: 0a 62 0d 0a 2d 2d 2d 2d  2d 2d 2d 2d 2d 2d 2d 2d  .b..------------
000000d0: 2d 2d 2d 2d 2d 2d 2d 2d  2d 2d 2d 2d 2d 2d 2d 2d  ----------------
000000e0: 2d 35 34 39 31 32 37 35  34 36 32 33 2d 2d 0d 0a  -54912754623--..
  [7]: len=8: '-headers'
00000000: 2d 68 65 61 64 65 72 73                           -headers
  [8]: len=68: 'content-type: application/x-www-form-urlencoded
content-length: 3
'
00000000: 63 6f 6e 74 65 6e 74 2d  74 79 70 65 3a 20 61 70  content-type: ap
00000010: 70 6c 69 63 61 74 69 6f  6e 2f 78 2d 77 77 77 2d  plication/x-www-
00000020: 66 6f 72 6d 2d 75 72 6c  65 6e 63 6f 64 65 64 0d  form-urlencoded.
00000030: 0a 63 6f 6e 74 65 6e 74  2d 6c 65 6e 67 74 68 3a  .content-length:
00000040: 20 33 0d 0a                                        3..
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

The logs triggered anti-spam protection. So, I pasted them elsewhere:
http://ix.io/gVc
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

Weird. It's extracted fine, but then gets lost somewhere, and I have no idea where or why. I added more logging, can you try it and show the log?
Also, try [HEADERS] - it's also multiline, maybe this has something to do with it.

Code: Select all

http://rghost.net/private/7YmxFH6d7/68e989df63e16898cb7b280e23fb392f
Here's my log, just in case you were wondering if this thing works at all.
Command line arguments:

Code: Select all

[;;;URL;;;] [-u ;;;UA;;;] [-e ;;;REFERER;;;] [-D ;;;FOLDER;;;] [-o ;;;FNAME;;;] [-K ;;;CFILE;;;] [-p ;;;RAWPOST;;;] [--header=HEADER] [-h HEADERS] -r -VVV
Log:

Code: Select all

2015-03-18T11:01:46.623Z Extracting post data...
2015-03-18T11:01:46.623Z res.rawPostData (len=73): 'Content-Type: application/x-www-form-urlencoded
Content-Length: 3

a=b'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Host', value='localhost'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='User-Agent', value='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Accept', value='text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Accept-Encoding', value='gzip, deflate'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='DNT', value='1'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Referer', value='http://localhost/test.html'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Connection', value='keep-alive'
2015-03-18T11:01:46.623Z origPostContentType='null'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Content-Type', value='application/x-www-form-urlencoded'
2015-03-18T11:01:46.623Z headerVisitor::visitHeader: name='Content-Length', value='3'
2015-03-18T11:01:46.623Z extractPostData: OUT:
2015-03-18T11:01:46.623Z   res.rawPostData:  len=73: 'Content-Type: application/x-www-form-urlencoded
Content-Length: 3

a=b'
2015-03-18T11:01:46.623Z   res.postData:  len=3: 'a=b'
2015-03-18T11:01:46.623Z   res.extraHeaders: [object Object]
2015-03-18T11:01:46.623Z     'content-type: application/x-www-form-urlencoded'
2015-03-18T11:01:46.624Z     'content-length: 3'
2015-03-18T11:01:48.808Z Preprocessing done in ms9
2015-03-18T11:01:48.810Z Starting dispatch
2015-03-18T11:01:50.212Z FlashGotDMCust::performJob: l.rawPostData:  len=73: 'Content-Type: application/x-www-form-urlencoded
Content-Length: 3

a=b'
2015-03-18T11:01:50.212Z FlashGotDMCust::performJob: l.extraHeaders: [object Object]
2015-03-18T11:01:50.212Z FlashGotDMCust::performJob: extraHeaders: len=68: 'content-type: application/x-www-form-urlencoded
content-length: 3
'
2015-03-18T11:01:50.234Z Running C:\hiawatha-9.2.UNP\hiawatha-9.2\wwwroot\test.exe ;;;http://localhost/test.php ;;; -u ;;;Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25 ;;; -e ;;;http://localhost/test.html ;;; -D ;;;C:\Users\пк\Downloads\1 ;;; -o ;;;test_php.mp3 ;;; -K ;;;C:\Users\7349~1\AppData\Local\Temp\flashgot.1e3apr2v.default\cookies ;;; -p ;;;Content-Type: application/x-www-form-urlencoded
Content-Length: 3

a=b ;;; --header=content-type: application/x-www-form-urlencoded --header=content-length: 3 -h content-type: application/x-www-form-urlencoded
content-length: 3
 -r -VVV -- async
2015-03-18T11:01:50.247Z Native execution time 13
2015-03-18T11:01:50.289Z Dispatch done in ms1478
Total processing time: ms1489
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

Tested with a fresh profile.
http://ix.io/gXG
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

Looks like you have autostart enabled, and I don't. I couldn't make autostart work, so you'll have to test it for me.

Code: Select all

http://rghost.net/private/84h9sZlGg/2c16726a4d25cf97648b586a03f4c407
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

Okay. This one worked.
Thank you.

http://ix.io/gYx

Is this the correct behaviour of HEADER?

More importantly, the space after each placeholder is gone. This should be documented in the Changelog if this code is pulled by upstream. Some shell/batch scripts used to parse the arguments will need to be adjusted to the change.
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

Yes, [HEADER] is a "flattened" alternative to [HEADERS]. It's for those who want one header per argument, e.g. aria2.
As for the "missing" space, I guess it's time to fork for me.
This version comes with all the extra spaces you'll ever need. But without the [HEADER] placeholder because it's incompatible with idiocy.
http://rghost.net/private/6mdRrrB4F/d1423c9503676a6204a4318da3bf1075
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

Thank you for all your help.

The added spaces annoyed me too. But I think Giorgio will opt to keep them for backwards-compatibility.

Edit: Or maybe add a space after each ']' instead of after the placeholder and hope no one's script will break.
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

Meh, forgot to remove some debug logging leftovers. This version should be clean:

Code: Select all

http://rghost.net/private/8wcv4XgJL/bdc170463cdb1e32b1bc41e19b88171a
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: POST data includes headers

Post by Giorgio Maone »

I don't care about spaces, I'd rather see them removed.
Can I have a clean version with [HEADER] and without extra spaces to be merged for good?
Thanks :)
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

Did you read this thread? It only looks like an extra space because runNative uses args.join(" ") for logging, but in fact, the second ";;;" (the one after the placeholder name) went to a different argument because it was captured in m[5]. And the reason it went "missing" is that I used the r3 regexp which captures the second ";;;" in m[4].

And while we're at removing extra spaces, winEscHack, in its current form, is unnecessary. Because for normal exe files, you don't need to do anything special with arguments because nsIProcess does everything for you, i.e. quotes arguments that contain whitespace and escapes double quotes with a backslash. And for batch scripts, there are a couple more things to do besides quoting arguments that contain ;&=. For one, commas also work as argument separators. :P Then ^%!&|<> - they must be escaped with "^", the first 3 at the very least. Then double quotes - they must be doubled. Finally, no amount of sanitizing will protect you from a buggy script that uses "%"-style variables ("%var%") when delayed expansion is on.

So, what this version does:
1) Uses the r3 regexp instead of r1.
2) Unconditionally uses winEscHackBatch for batch files, i.e. there's no pref to disable it.
3) For normal exe files, winEscHack is false by default.
4) New pref flashgot.intercept.extra_request_headers - a comma-separated case-insensitive list of request header names to extract from requests. Default is "origin, /^x-./". The /.../ syntax is used to specify a JavaScript regular expression, i.e. /^x-./ matches everything that starts with "x-" or "X-", e.g. "X-Requested-With".

Code: Select all

http://rghost.net/private/8jrbfYyfb/13cbb284880cca59176da43965d69ed6
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

I hope a working RAWPOST will be included in an official release someday.
Mozilla/5.0 (X11; Linux x86_64; rv:39.0) Gecko/20100101 Firefox/39.0
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

MoSal wrote:I hope a working RAWPOST will be included in an official release someday.
So...
Is flashgot with working RAWPOST ever going to be available in AMO?
Mozilla/5.0 (X11; Linux x86_64; rv:41.0) Gecko/20100101 Firefox/41.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

MoSal wrote:Is flashgot with working RAWPOST ever going to be available in AMO?
In case you've lost the version I uploaded before, I can reupload it. And if you haven't, why do you even care about AMO?
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
Post Reply