POST data includes headers

Bug reports and enhancement requests
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

POST data includes headers

Post by MoSal »

Hello.

FlashGot passes with POST data the headers:

Code: Select all

Content-Type: application/x-www-form-urlencoded
Content-Length: <POST data length>
Is there a way to pass the POST data alone to a downloader?

I can add logic in my downloader to skip the headers. But I was hoping for a cleaner solution.
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

Of course there is a cleaner solution: file a bug against whatever thing (e.g. a Flash plugin) puts headers to POST data. FlashGot just grabs a channel's upload stream which is supposed to contain just the POST data without any headers.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

flashgot.user wrote:Of course there is a cleaner solution: file a bug against whatever thing (e.g. a Flash plugin) puts headers to POST data. FlashGot just grabs a channel's upload stream which is supposed to contain just the POST data without any headers.
To clarify, I added my still-unpublished downloader manually to FlashGot for testing. The POST placeholder provided by FlashGot included the headers when I tested a typical case using an HTML submit button. There are no third parties in action here.

I thought those headers were provided to help dumb downloaders incapable of setting those headers correctly on their own.

Are you saying that there is something wrong? Those headers are not usually a part of the POST placeholder?
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

If it's Content-Type and Content-Length, and only these 2, then yes, they're injected by the browser, and FlashGot passes them untouched to you. But I guess that just breaks every downloader because no one expects POST data in such format. Besides, I forgot to modify flashgot.exe to account for that change, so it was broken anyway. So here we go, this version drops the headers again:

Code: Select all

http://rghost.net/private/72gGHl5R4/552e8ef7b269a14e4936c72eed791505
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

flashgot.user wrote:If it's Content-Type and Content-Length, and only these 2, then yes, they're injected by the browser, and FlashGot passes them untouched to you. But I guess that just breaks every downloader because no one expects POST data in such format. Besides, I forgot to modify flashgot.exe to account for that change, so it was broken anyway. So here we go, this version drops the headers again:

Code: Select all

http://rghost.net/private/72gGHl5R4/552e8ef7b269a14e4936c72eed791505
Thank you.
I hope the next stable version will be released soon.
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

But you do realise that this approach is still broken? I mean, multipart POSTs won't work at all (without you (the downloader) parsing the POST data and adding the correct Content-Type header), and even for a url-encoded POST you may need the charset.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

flashgot.user wrote:But you do realise that this approach is still broken? I mean, multipart POSTs won't work at all (without you (the downloader) parsing the POST data and adding the correct Content-Type header), and even for a url-encoded POST you may need the charset.
Now that you mention it. A list of reference examples for different POST scenarios would be very useful.

I don't know about other downloaders. But the one I'm working on is based on libcurl which provides interfaces for almost any scenario imaginable. But I'm struggling to find non-trivial use-cases to test with.
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

Well, for a multipart POST scenario, just set enctype="multipart/form-data" for your test form. You (the downloader) still get the POST data, but you will need to parse it, i.e. to look for the presence of "\r\n" in it, and if it's there, get the boundary (the first line of the POST data) and set it in the Content-Type header.
As for a url-encoded POST, I think it's a more rare, yet not impossible scenario. Things can go wrong in case of a framework (Java applets/JSP, etc.) that does all the decoding for you, using the charset specified by the client (browser, downloader, whatever). And if its default charset doesn't match the POST charset, you (the server script) might end up with garbage.
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

I just realised that I forgot to add a placeholder for extra headers. This is where those POST-related headers could also go.
Giorgio, what do you think?
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: POST data includes headers

Post by Giorgio Maone »

flashgot.user wrote:I just realised that I forgot to add a placeholder for extra headers. This is where those POST-related headers could also go.
Giorgio, what do you think?
That was exactly my thinking, thank you.
Since this seems to be a bug introduced by some recent Firefox change, I'd keep [POST] as the backward-compatible "dummy" headerless name-value pairs format which assumes application/x-www-form-urlencoded, while we could introduce a [RAWPOST] or [UPLOAD] placeholder which would stand for the uploaded data as it is, including its preamble headers, in case some client wants to consume more exotic payloads such as multipart or raw XML/JSON.
Are you going to make another patch :)?
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

Well, here we go, 2 new placeholders:
[RAWPOST] - POST data with all the POST-related headers injected by the browser: "Content-Type: ... \r\nContent-Length: ... \r\n\r\n<POST data>".
[HEADERS] - extra request headers, each terminated with "\r\n", e.g. "X-tra-header: foo\r\n".

Code: Select all

http://rghost.net/private/7S6fZJWgc/9d50750d9038f2dfeb03ce6af183c48b
For now, the support is limited to custom downloader managers only (FlashGotDMCust), i.e. those that were manually added using the Add button. Because 1) I have no idea which downloader from the default supported downloader list supports custom request headers and POST data format other than url-encoded, 2) I'm still not sure what would be the best way to modify flashgot.exe, and thinking straight while your head is trying to kill you is not what I'm famous for. :)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: POST data includes headers

Post by Giorgio Maone »

flashgot.user wrote:I have no idea which downloader from the default supported downloader list supports custom request headers and POST data format other than url-encoded
None, AFAIK, hence we're golden as we are I guess.
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
User avatar
Giorgio Maone
Site Admin
Posts: 9454
Joined: Wed Mar 18, 2009 11:22 pm
Location: Palermo - Italy
Contact:

Re: POST data includes headers

Post by Giorgio Maone »

Please check latest development build 1.5.6.11rc1, thanks.
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
MoSal
Posts: 13
Joined: Wed Mar 11, 2015 10:48 am

Re: POST data includes headers

Post by MoSal »

Unfortunately, the new placeholders do not pass the intended data in my setup. [HEADERS] is always empty (never passed). And [RAWPOST] is the same as [POST]!
Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
flashgot.user
Junior Member
Posts: 27
Joined: Thu Sep 11, 2014 9:47 am

Re: POST data includes headers

Post by flashgot.user »

[HEADERS] are supposed to always be empty because they weren't supposed to contain any POST-related headers. They were only used with grooveshark.com, and now they're useless because they finally fixed that loophole.

[RAWPOST] works for me. My command line is:

Code: Select all

-url [URL] [-post POST] [-rawpost RAWPOST] [-headers HEADERS]
The program just dumps its arguments to a file. Here's the output:

Code: Select all

argc=7
  [0]: len=49: 'C:\hiawatha-9.2.UNP\hiawatha-9.2\wwwroot\test.exe'
00000000: 43 3a 5c 68 69 61 77 61  74 68 61 2d 39 2e 32 2e  C:\hiawatha-9.2.
00000010: 55 4e 50 5c 68 69 61 77  61 74 68 61 2d 39 2e 32  UNP\hiawatha-9.2
00000020: 5c 77 77 77 72 6f 6f 74  5c 74 65 73 74 2e 65 78  \wwwroot\test.ex
00000030: 65                                                e
  [1]: len=4: '-url'
00000000: 2d 75 72 6c                                       -url
  [2]: len=25: 'http://localhost/test.php'
00000000: 68 74 74 70 3a 2f 2f 6c  6f 63 61 6c 68 6f 73 74  http://localhost
00000010: 2f 74 65 73 74 2e 70 68  70                       /test.php
  [3]: len=5: '-post'
00000000: 2d 70 6f 73 74                                    -post
  [4]: len=3: 'a=b'
00000000: 61 3d 62                                          a=b
  [5]: len=8: '-rawpost'
00000000: 2d 72 61 77 70 6f 73 74                           -rawpost
  [6]: len=73: 'Content-Type: application/x-www-form-urlencoded
Content-Length: 3

a=b'
00000000: 43 6f 6e 74 65 6e 74 2d  54 79 70 65 3a 20 61 70  Content-Type: ap
00000010: 70 6c 69 63 61 74 69 6f  6e 2f 78 2d 77 77 77 2d  plication/x-www-
00000020: 66 6f 72 6d 2d 75 72 6c  65 6e 63 6f 64 65 64 0d  form-urlencoded.
00000030: 0a 43 6f 6e 74 65 6e 74  2d 4c 65 6e 67 74 68 3a  .Content-Length:
00000040: 20 33 0d 0a 0d 0a 61 3d  62                        3....a=b
Here's my test form:

Code: Select all

<form method="POST" action="test.php">
<input type="text" name="a" value="b" />
<input type="submit" />
</form>
Here's test.php:

Code: Select all

<?php
http_response_code(200);
header("Content-Type: audio/mpeg");
?>
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 SeaMonkey/2.25
Post Reply