HTTP::Cookie won't store sent cookie

[Richard Lawrence]

Hi all,

My script requests [url]http://foo.bar.com/[/url] with code that looks a little
like this:

my $ua = LWP::UserAgent->new;
my $cookie_jar = HTTP::Cookies->new(file => $cookie_path);
$cookie_jar->load($cookie_path);
$ua->cookie_jar($cookie_jar);

my $req = HTTP::Request->new(GET => "http://foo.bar.com/");
$cookie_jar->add_cookie_header($req);

# Make request
my $res = $ua->request($req);

# HTML back
if ($res->is_success)
{
$cookie_jar->extract_cookies($res);
$cookie_jar->save();
}

This works great, however the site sends back this:

Set-Cookie: name=fred; domain=.bar.com; path=/

which for some reason doesn't get saved in the cookie jar.

I'm not sure if this is because the set-cookie header is badly formed
or non-standard but since Firefox and IE are both happy with it I
really need to make my code happy with it.

Have I done something wrong or is there a way to get this to work?

Many thanks in advance,

Richard

[Gunnar Hjalmarsson]

Richard Lawrence wrote:

> My script requests [url]http://foo.bar.com/[/url] with code that looks a little
> like this:

<code snipped>

> This works great, however the site sends back this:
>
> Set-Cookie: name=fred; domain=.bar.com; path=/
>
> which for some reason doesn't get saved in the cookie jar.

Have you possibly finished the printing of CGI headers prematurely? If
you don't understand what I mean by that, please post a *short* but
*complete* script that illustrates the issue.

--
Gunnar Hjalmarsson
Email: [url]http://www.gunnar.cc/cgi-bin/contact.pl[/url]

[Richard Lawrence]

Gunnar Hjalmarsson wrote:

> Richard Lawrence wrote:

> > My script requests [url]http://foo.bar.com/[/url] with code that looks a

little

> > like this:

>
> <code snipped>
>

> > This works great, however the site sends back this:
> >
> > Set-Cookie: name=fred; domain=.bar.com; path=/
> >
> > which for some reason doesn't get saved in the cookie jar.

>
> Have you possibly finished the printing of CGI headers prematurely?

If

> you don't understand what I mean by that, please post a *short* but
> *complete* script that illustrates the issue.

You're right in the fact that I don't understand :) Also following a
bit more investigation, it doesn't look like what I originally thought
was the problem isn't the problem.

Here is a script that demonstrates the code although sadly I have to
mask the site and the details sent. It's worth noting I have no control
over the site I connect to and that the one I use below is different
from the original one as it's easier to show the problem:

#!/usr/bin/perl

use strict;
use warnings;

use LWP::UserAgent;
use HTTP::Cookies;
use HTTP::Request;

my $received_html;

# Note: horrible wrapping caused by groups-google.com

&get_page("http://www.somedomain.com/process_logon.jsp",
"referring_page=sms.jsp&email_address=my%40email%2 Eaddress&password=mypassword",
"http://www.somedomain.com/process_logon.jsp");

&get_page("http://www.somedomain.com/sms.jsp", "",
"http://www.somedomain.com/process_logon.jsp");

sub get_page
{
my ($url, $data, $referer) = @_;
my $cookie_path = "./cookies";

# If $data exists then it's POST, otherwise GET

# Create the user agent

my $ua = LWP::UserAgent->new(env_proxy => 1);
$ua->agent("Mozilla/4.0");
push @{ $ua->requests_redirectable }, 'POST';

# Create the cookie jar

my $cookie_jar = HTTP::Cookies->new(file => $cookie_path);
$cookie_jar->load($cookie_path);
$ua->cookie_jar($cookie_jar);

# If we have a data= section, then this is a POST

my $req;

if ($data)
{
$req = HTTP::Request->new(POST => $url);
$req->content_type('application/x-www-form-urlencoded');
$req->content($data);
}
else
{
$req = HTTP::Request->new(GET => $url);
}

$req->header('Accept' => 'text/*');

# Add a referer if required

$req->referer($referer) if ($referer);

# Add cookies

$cookie_jar->add_cookie_header($req);

print "-- Start of sent headers --\n";
print $req->as_string();
print "\n-- End of sent headers --\n";

# Connect!

my $res = $ua->request($req);

$received_html = "";

die "stop: Error sending request - " . $res->status_line . "\n" if
(!$res->is_success);

# Since we're here, we know that it went ok

print "-- Start of received headers and HTML --\n";
print $res->as_string . "\n";
print "-- End of received headers and HTML --\n\n";

$received_html = $res->content;

# Extract cookies

$cookie_jar->extract_cookies($res);
$cookie_jar->save();
}

Here is the output I get:

-- Start of sent headers --
POST [url]http://www.somedomain.com/process_logon.jsp[/url]
Accept: text/*
Referer: [url]http://www.somedomain.com/process_logon.jsp[/url]
Content-Type: application/x-www-form-urlencoded

referring_page=sms.jsp&email_address=my%40email%2E address&password=mypassword

-- End of sent headers --
-- Start of received headers and HTML --
HTTP/1.1 200 OK
Cache-Control: no-cache="set-cookie,set-cookie2"
Connection: close
Date: Fri, 18 Mar 2005 10:07:10 GMT
Via: 1.1 [url]www.somedomain.com[/url]
Server: Microsoft-IIS/5.0
Content-Length: 103
Content-Type: text/html; charset=ISO-8859-1
Expires: Thu, 01 Dec 1994 16:00:00 GMT
Client-Date: Fri, 18 Mar 2005 10:07:22 GMT
Client-Peer: 10.10.142.190:80
Client-Response-Num: 1
Refresh: 2;url="sms.jsp"
Set-Cookie: registered=1
Set-Cookie: jsessionid=2380421411141442392;path=/
X-Powered-By: ASP.NET

[html snipped]
-- End of received headers and HTML --

As you can see here, two cookies have been set however in the next
send...

-- Start of sent headers --
GET http://www.somedomain.com/sms.jsp
Accept: text/*
Referer: http://www.somedomain.com/process_logon.jsp


-- End of sent headers --

These aren't sent back to the server.

Many thanks for any suggestions!

Richard

[Gunnar Hjalmarsson]

Richard Lawrence wrote:

> Gunnar Hjalmarsson wrote:

>> Richard Lawrence wrote:

>>> My script requests [url]http://foo.bar.com/[/url] with code that looks a little
>>> like this:

>>
>> <code snipped>
>>

>>> This works great, however the site sends back this:
>>>
>>> Set-Cookie: name=fred; domain=.bar.com; path=/
>>>
>>> which for some reason doesn't get saved in the cookie jar.

>>
>> Have you possibly finished the printing of CGI headers prematurely? If
>> you don't understand what I mean by that, please post a *short* but
>> *complete* script that illustrates the issue.

>
> You're right in the fact that I don't understand :)

Well, I obviously didn't read your question carefully enough, sorry.
Please disregard that remark.

A cookie without a valid expires parameter is not saved to file by
default. That makes perfect sense, since such a cookie is just a session
cookie which automatically expires at end of session. It gets not saved
to disk by browsers either.

Use the "ignore_discard" parameter to still have it saved to file:

my $cookie_jar =
HTTP::Cookies->new(file => $cookie_path, ignore_discard => 1);

--
Gunnar Hjalmarsson
Email: [url]http://www.gunnar.cc/cgi-bin/contact.pl[/url]

[Richard Lawrence]

Gunnar Hjalmarsson wrote:

> Well, I obviously didn't read your question carefully enough, sorry.
> Please disregard that remark.

Not to worry :)

> A cookie without a valid expires parameter is not saved to file by
> default. That makes perfect sense, since such a cookie is just a

session

> cookie which automatically expires at end of session. It gets not

saved

> to disk by browsers either.
>
> Use the "ignore_discard" parameter to still have it saved to file:
>
> my $cookie_jar =
> HTTP::Cookies->new(file => $cookie_path, ignore_discard => 1);

Ahhh, I didn't realise I needed that because I failed to spot it was a
session cookie. Works like a charm now.

Thanks very much!

Richard