473,320 Members | 1,817 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Not getting cookies in LWP

I'm trying to access a site with data that needs to be paged through, one
page at a time. It won't allow back buttons and you have to use the menu
links to get through. In other words, it is run through CGI (ASP, I think)
and must be able to keep track of sessions.

I have a simple program I've written using Perl and LWP. I tried a totally
innocuous site (TVLand.com) that I found had cookies. When I read in 1
page from this site, I got a cookie and it showed up in lwpcookies.txt.
Whenever I try this other site (which requires a password and account, so I
can't list it here, according to registration agreement -- and no, it isn't
pr0n!), I find that there is a line in the header to set a cookie in both
web pages. The only difference is that the one where the cookie isn't
storied doesn't list a domain name in the cookie line. Here's the header
lines with the cookie info (1st is TVLand, 2nd is private site):

Set-Cookie: JSESSIONID=M1LHYKX2DVXRMCQBAFML3UQ; domain=.tvland.com; path=/
Set-Cookie: JSESSIONID=0000CSPVV3Q5TXU2BUPIIDEWOCY:ulnfn1uq;Pa th=/

my program (listed below) prints out that I have a cookie. (Whenever I try
the site with Galeon, I can read a cookie for the site, too), but no cookie
shows up in lwpcookie.txt (the cookie file) at all.

Am I doing something wrong? I need to be sure that the cookie is persistant
in my program, but it never shows up in the cookie file. Is there a reason
for that, or am I doing something wrong? What can I do to make sure the
cookie from the 2nd site is stored for later -- and also read back when
needed by other pages?

Thanks!

Hal
--------------------------------------------------------------------
Program listing:

use LWP::UserAgent;
use HTTP::Cookies;

our $domain = "tvland.com";
our $locmenu = "schedule";
our $ua = LWP::UserAgent->new;
$ua->agent("Mozilla/4.0");
$ua->agent("MSIE/6.0");
$ua->cookie_jar(HTTP::Cookies->new(file =>"lwpcookies.txt", autosave =>
1));
$url = "HTTP://".$domain."/".$locmenu;
print "Url: $url\n";
$req = HTTP::Request->new(GET => $url);
$req->content_type("application/x-www-form-urlencoded");
$req->header('Accept' => 'text/html');
$res = $ua->request($req);
print "Cookie: ".$res->status_line."\n";
$page = $res->as_string;
# print "Page: $page\n";
Jul 19 '05 #1
1 6778
Hal Vaughan wrote:
I'm trying to access a site with data that needs to be paged through, one
page at a time. It won't allow back buttons and you have to use the menu
links to get through. In other words, it is run through CGI (ASP, I think)
and must be able to keep track of sessions.

I have a simple program I've written using Perl and LWP. I tried a totally
innocuous site (TVLand.com) that I found had cookies. When I read in 1
page from this site, I got a cookie and it showed up in lwpcookies.txt.
Whenever I try this other site (which requires a password and account, so I
can't list it here, according to registration agreement -- and no, it isn't
pr0n!), I find that there is a line in the header to set a cookie in both
web pages. The only difference is that the one where the cookie isn't
storied doesn't list a domain name in the cookie line. Here's the header
lines with the cookie info (1st is TVLand, 2nd is private site):

Set-Cookie: JSESSIONID=M1LHYKX2DVXRMCQBAFML3UQ; domain=.tvland.com; path=/
Set-Cookie: JSESSIONID=0000CSPVV3Q5TXU2BUPIIDEWOCY:ulnfn1uq;Pa th=/

my program (listed below) prints out that I have a cookie. (Whenever I try
the site with Galeon, I can read a cookie for the site, too), but no cookie
shows up in lwpcookie.txt (the cookie file) at all.

Am I doing something wrong? I need to be sure that the cookie is persistant
in my program, but it never shows up in the cookie file. Is there a reason
for that, or am I doing something wrong? What can I do to make sure the
cookie from the 2nd site is stored for later -- and also read back when
needed by other pages?

Thanks!

Hal
--------------------------------------------------------------------
Program listing:

use LWP::UserAgent;
use HTTP::Cookies;

our $domain = "tvland.com";
our $locmenu = "schedule";
our $ua = LWP::UserAgent->new;
$ua->agent("Mozilla/4.0");
$ua->agent("MSIE/6.0");
$ua->cookie_jar(HTTP::Cookies->new(file =>"lwpcookies.txt", autosave =>
1));
$url = "HTTP://".$domain."/".$locmenu;
print "Url: $url\n";
$req = HTTP::Request->new(GET => $url);
$req->content_type("application/x-www-form-urlencoded");
$req->header('Accept' => 'text/html');
$res = $ua->request($req);
print "Cookie: ".$res->status_line."\n";
$page = $res->as_string;
# print "Page: $page\n";


$ua = LWP::UserAgent->new( cookie_jar =>HTTP::Cookies->new( file =>
'/tmp/cookies.txt', autosave => 1, ignore_discard => 1 ));

try ignore_discard => 1
Jul 19 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Steve Mauldin | last post by:
This is a standard ASP application that has several pages at the root withthe global.asa. I set a session variable session("accountid") = "123456" within an asp page and then response.redirect to...
1
by: Rajan Vijayaraghavan | last post by:
Hi, I have a question with the JavaScript and CGI. This is the scenario. Say, a user is coming to page 'X'. The page 'X' is completely generated by a cgi bin program. The cgi-bin program is in C....
1
by: Mike | last post by:
I got the code below from an earlier post but I can't get it to work (I get an error on the "for (i=0; i<a.length; i++)" line) Anyone have code that works for cookies with keys? > Anyone...
3
by: Beryl Small | last post by:
Hi, I have a third party software that needs to send information to an .aspx page for processing to communicate with an SQL database. The software sends the information something like this: ...
3
by: rss | last post by:
SUMMARY: ========== I am unable to pass along a simple Cookie obtained from a HttpWebRequest call (Machine A's ASP.NET app) so that another Web Server (Machine B) recongnizes the cookie. I...
2
by: Prasad Dannani | last post by:
Hi, I want to know the browser settings like Image Enabled, Cookie Enabled etc., For Cookies and Script Request object has some properties but after disabling certain features also we are...
4
by: Vlad Dogaru | last post by:
Hello, I am trying to use cookies and Python to create a simple login example. But I am very disoriented at the existence of two cookie libraries, namely Cookie and cookielib. I have seen...
2
by: jpgonzalezgarcia | last post by:
hi. i am trying to make a script that will extract info from a protected page on ebay, which you can view only by logging in, which requires SSL 443. 1. get the url 2. get the login screen 3....
5
by: Zeba | last post by:
Hi ! I want to know the cookie value that was set in the previous page. Initially in my page in project B I used HttpCookie cookie = Request.Cookies.Get(somefile.COOKIE_NAME); where...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.