InformAction Forums

Posted: **Wed Mar 11, 2015 11:05 am**

Hello.

FlashGot passes with POST data the headers:

Content-Type: application/x-www-form-urlencoded
Content-Length: <POST data length>

Is there a way to pass the POST data alone to a downloader?

I can add logic in my downloader to skip the headers. But I was hoping for a cleaner solution.

Posted: **Thu Mar 12, 2015 6:12 am**

Of course there is a cleaner solution: file a bug against whatever thing (e.g. a Flash plugin) puts headers to POST data. FlashGot just grabs a channel's upload stream which is supposed to contain just the POST data without any headers.

Posted: **Thu Mar 12, 2015 6:06 pm**

flashgot.user wrote:Of course there is a cleaner solution: file a bug against whatever thing (e.g. a Flash plugin) puts headers to POST data. FlashGot just grabs a channel's upload stream which is supposed to contain just the POST data without any headers.

To clarify, I added my still-unpublished downloader manually to FlashGot for testing. The POST placeholder provided by FlashGot included the headers when I tested a typical case using an HTML submit button. There are no third parties in action here.

I thought those headers were provided to help dumb downloaders incapable of setting those headers correctly on their own.

Are you saying that there is something wrong? Those headers are not usually a part of the POST placeholder?

Posted: **Fri Mar 13, 2015 2:20 am**

If it's Content-Type and Content-Length, and only these 2, then yes, they're injected by the browser, and FlashGot passes them untouched to you. But I guess that just breaks every downloader because no one expects POST data in such format. Besides, I forgot to modify flashgot.exe to account for that change, so it was broken anyway. So here we go, this version drops the headers again:

Code: Select all

http://rghost.net/private/72gGHl5R4/552e8ef7b269a14e4936c72eed791505

Posted: **Fri Mar 13, 2015 3:19 pm**

flashgot.user wrote:If it's Content-Type and Content-Length, and only these 2, then yes, they're injected by the browser, and FlashGot passes them untouched to you. But I guess that just breaks every downloader because no one expects POST data in such format. Besides, I forgot to modify flashgot.exe to account for that change, so it was broken anyway. So here we go, this version drops the headers again:
Code: Select all
http://rghost.net/private/72gGHl5R4/552e8ef7b269a14e4936c72eed791505

Thank you.
I hope the next stable version will be released soon.

Posted: **Fri Mar 13, 2015 4:16 pm**

But you do realise that this approach is still broken? I mean, multipart POSTs won't work at all (without you (the downloader) parsing the POST data and adding the correct Content-Type header), and even for a url-encoded POST you may need the charset.

Posted: **Fri Mar 13, 2015 5:35 pm**

flashgot.user wrote:But you do realise that this approach is still broken? I mean, multipart POSTs won't work at all (without you (the downloader) parsing the POST data and adding the correct Content-Type header), and even for a url-encoded POST you may need the charset.

Now that you mention it. A list of reference examples for different POST scenarios would be very useful.

I don't know about other downloaders. But the one I'm working on is based on libcurl which provides interfaces for almost any scenario imaginable. But I'm struggling to find non-trivial use-cases to test with.

Posted: **Fri Mar 13, 2015 5:52 pm**

Well, for a multipart POST scenario, just set enctype="multipart/form-data" for your test form. You (the downloader) still get the POST data, but you will need to parse it, i.e. to look for the presence of "\r\n" in it, and if it's there, get the boundary (the first line of the POST data) and set it in the Content-Type header.
As for a url-encoded POST, I think it's a more rare, yet not impossible scenario. Things can go wrong in case of a framework (Java applets/JSP, etc.) that does all the decoding for you, using the charset specified by the client (browser, downloader, whatever). And if its default charset doesn't match the POST charset, you (the server script) might end up with garbage.

Posted: **Fri Mar 13, 2015 6:58 pm**

I just realised that I forgot to add a placeholder for extra headers. This is where those POST-related headers could also go.
Giorgio, what do you think?

Posted: **Fri Mar 13, 2015 7:26 pm**

flashgot.user wrote:I just realised that I forgot to add a placeholder for extra headers. This is where those POST-related headers could also go.
Giorgio, what do you think?

That was exactly my thinking, thank you.
Since this seems to be a bug introduced by some recent Firefox change, I'd keep [POST] as the backward-compatible "dummy" headerless name-value pairs format which assumes application/x-www-form-urlencoded, while we could introduce a [RAWPOST] or [UPLOAD] placeholder which would stand for the uploaded data as it is, including its preamble headers, in case some client wants to consume more exotic payloads such as multipart or raw XML/JSON.
Are you going to make another patch

?

Posted: **Fri Mar 13, 2015 10:57 pm**

Well, here we go, 2 new placeholders:
[RAWPOST] - POST data with all the POST-related headers injected by the browser: "Content-Type: ... \r\nContent-Length: ... \r\n\r\n<POST data>".
[HEADERS] - extra request headers, each terminated with "\r\n", e.g. "X-tra-header: foo\r\n".

Code: Select all

http://rghost.net/private/7S6fZJWgc/9d50750d9038f2dfeb03ce6af183c48b

For now, the support is limited to custom downloader managers only (FlashGotDMCust), i.e. those that were manually added using the Add button. Because 1) I have no idea which downloader from the default supported downloader list supports custom request headers and POST data format other than url-encoded, 2) I'm still not sure what would be the best way to modify flashgot.exe, and thinking straight while your head is trying to kill you is not what I'm famous for. :)

Posted: **Fri Mar 13, 2015 11:51 pm**

flashgot.user wrote:I have no idea which downloader from the default supported downloader list supports custom request headers and POST data format other than url-encoded

None, AFAIK, hence we're golden as we are I guess.

Posted: **Sat Mar 14, 2015 1:54 pm**

Please check latest development build 1.5.6.11rc1, thanks.

Posted: **Sun Mar 15, 2015 3:10 pm**

Unfortunately, the new placeholders do not pass the intended data in my setup. [HEADERS] is always empty (never passed). And [RAWPOST] is the same as [POST]!

Posted: **Sun Mar 15, 2015 4:41 pm**

[HEADERS] are supposed to always be empty because they weren't supposed to contain any POST-related headers. They were only used with grooveshark.com, and now they're useless because they finally fixed that loophole.

[RAWPOST] works for me. My command line is:

Code: Select all

-url [URL] [-post POST] [-rawpost RAWPOST] [-headers HEADERS]

The program just dumps its arguments to a file. Here's the output:

Code: Select all

argc=7
  [0]: len=49: 'C:\hiawatha-9.2.UNP\hiawatha-9.2\wwwroot\test.exe'
00000000: 43 3a 5c 68 69 61 77 61  74 68 61 2d 39 2e 32 2e  C:\hiawatha-9.2.
00000010: 55 4e 50 5c 68 69 61 77  61 74 68 61 2d 39 2e 32  UNP\hiawatha-9.2
00000020: 5c 77 77 77 72 6f 6f 74  5c 74 65 73 74 2e 65 78  \wwwroot\test.ex
00000030: 65                                                e
  [1]: len=4: '-url'
00000000: 2d 75 72 6c                                       -url
  [2]: len=25: 'http://localhost/test.php'
00000000: 68 74 74 70 3a 2f 2f 6c  6f 63 61 6c 68 6f 73 74  http://localhost
00000010: 2f 74 65 73 74 2e 70 68  70                       /test.php
  [3]: len=5: '-post'
00000000: 2d 70 6f 73 74                                    -post
  [4]: len=3: 'a=b'
00000000: 61 3d 62                                          a=b
  [5]: len=8: '-rawpost'
00000000: 2d 72 61 77 70 6f 73 74                           -rawpost
  [6]: len=73: 'Content-Type: application/x-www-form-urlencoded
Content-Length: 3

a=b'
00000000: 43 6f 6e 74 65 6e 74 2d  54 79 70 65 3a 20 61 70  Content-Type: ap
00000010: 70 6c 69 63 61 74 69 6f  6e 2f 78 2d 77 77 77 2d  plication/x-www-
00000020: 66 6f 72 6d 2d 75 72 6c  65 6e 63 6f 64 65 64 0d  form-urlencoded.
00000030: 0a 43 6f 6e 74 65 6e 74  2d 4c 65 6e 67 74 68 3a  .Content-Length:
00000040: 20 33 0d 0a 0d 0a 61 3d  62                        3....a=b

Here's my test form:

Code: Select all

<form method="POST" action="test.php">
<input type="text" name="a" value="b" />
<input type="submit" />
</form>

Here's test.php:

Code: Select all

<?php
http_response_code(200);
header("Content-Type: audio/mpeg");
?>

InformAction Forums

POST data includes headers

POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers

Re: POST data includes headers