code to fetch data from online source

[cobra2341]

Hi, first post.
How I'm trying to do is get data like
<a href="http://www.minorleaguebaseball.com/milb/stats/stats.jsp? sid=milb&t=g_log&gid=2007_08_18_colaaa_tolaaa_1">t his</a>
(only the log) to download automactically. Here is a
<a href="http://www.minorleaguebaseball.com/milb/stats/stats.jsp? t=t_sch&cid=445">Schedule </a> that links to all the logs.
I read that perl can 'spider' data. Is there a way to get Perl to fetch all the text that I am wanting?
Thanks

[numberwhun]

Hello, and welcome to the scripts!

Sure, Perl can spider data, but you have to do some coding. You may want to start reading up on LWP if you are going to be grabbing information from web pages.

Regards,

Jeff

[cobra2341]

Thanks, but I guess I am really bad at perl. I am trying to run the LWP program put nothing happens. What am I doing wrong?

[numberwhun]

I truly do not know. It seems you have decided to keep your source code a secret from everyone. Why don't you post your code and any errors that you are seeing.

Regards,

Jeff

[cobra2341]

I'll go step-by-step what I am doing.
First I am opening the Command line.
Then I type perl myprog.pl
I get the error 'Can't open perl script 'myprog.pl': No such file
I have no idea what to do

[KevinADC]

Write a program that does what you want, name it "myprog.pl", save it to the hard drive. Then try running it from the command line.

[cobra2341]

so I don't save it in the Perl bin?

[KevinADC]

Where ever you normally save your perl programs is where you would save it. If that is in the perl/bin folder that is fine.

[numberwhun]

Ok, I would say that at this point, you really need to find yourself a good Perl tutorial.

My suggestion would be to go to your local Barnes & Noble and pick up a copy of Learning Perl. You need something like that to teach you the basics of the language.

Regards,

Jeff

[cobra2341]

Is there any good online reads. I found some, but when I have questions, there's no one to anwser my questions. That's why I came here

[Kelicula]

Yeah some people aren't very tolerant here. It seems that they are only interested in answering questions that "they" find interesting. Not much concern for the person who's posting's needs.

Anyway, when you open the command line, are you in the directory that the script is located in? You have to get there first then type perl myprog.pl. Also make sure to use the "include warnings" switch (-w) like this:
perl -w myprog.pl that will enable the warnings to output to your screen.

I found a really good perl tutorial online here.
http://www.cbkihong.com/download/perltut.pdf

I suggest saving this to your comp, and slowly taking the time to read it, making sure to do the examples yourself as you go.

Good luck!

[cobra2341]

Thanks alot. I think this is a great site and I have gotten alot more responds than I have before.
One question, What do you mean by are you in the directory that the script is located in?
Thanks alot

[Kelicula]

Well I guess first I should say, what system are you running? I was reffering to ActiveSate perl for windows. I open the DOS prompt. Then of course I have to cd into the directory that my scripts are in and then run them. I asume it is the same for other systems. You have to be in the "working directory" of the script to run it. in linux/unix type pwd (print working directory) to see what directory you're in. then cd (change directory) ect...

hope were on the same page. ;)

[cobra2341]

I read the guide you gave me and I understand I must do cd: first.
Thanks alot

[numberwhun]

Yeah some people aren't very tolerant here. It seems that they are only interested in answering questions that "they" find interesting. Not much concern for the person who's posting's needs.

Be that as you may think it, but its not a matter of being tolerant. This is a forum for learning to code/script in Perl. You don't look at a computer one day, having never touched one and say, I think I'll learn that today. Some semblance of computer knowledge (ie: knowing what it means to be in the same directory as your script) is actually implied. We aren't trying to be rude and crude, but we aren't out to teach everyone how to turn their computer on. If that were the case, this would be the Misc. Questions or even the Windows forum.

As I mentioned in my earlier posting, there are a number of great Perl tutorials out there. One that I liked was tizag.com. It was well put together and understandable. Another think that you could do is to go to the perldoc page and start reading all of the resources there. The FAQ's will be especially helpful once you get the basics down, but I would start reading those and get up to speed.

We are definitely wanting to help and assist every person who comes in the door, but when one says, "It doesn't work, Why?" and that is all they give you, you have to prompt them for more information. We aren't mind reader!

Regards,

Jeff

[cobra2341]

Sorry.
Ok, I read on LWP and it doesn't seem to be what I want. What I am wanting to do is have perl download the text from the site so I can import it to a database.
Thanks

[numberwhun]

Well, any way you look at it, what you are going to be downloading is the web page itself, which includes all of the HTML. Now, you could use LWP::Simple 's get function to fetch the web page, but you are going to have to parse out the text with a regex, or go so far as to use a module that helps to parse HTML into its elements (I would look one up, but am exhausted and on the way to bed at the moment).

Regards,

Jeff

[KevinADC]

Yeah some people aren't very tolerant here. It seems that they are only interested in answering questions that "they" find interesting. Not much concern for the person who's posting's needs.

And what is wrong with answering only questions a person finds interesting? You're welcome to participate here, but that attitude is totally misguided.

Personally, I have no concern at all for the person posting the question. I answer questions because I want to, for my own reasons, not for anyone elses reasons or for anyone elses "needs". It's all for my own selfish benefit. If it were not for my benefit, I would never answer one single question on this or any other forum. And because it is for my own selfish benefit, I am very prolific in this regards. I have helped literally thousands of people over the years on forums in a variety of disciplines, but these days it's 99% perl that interests me. If you do something like this for the other persons benefit or based on their needs, you will help a few people at best.

If it were not for my own benefit why would I continue? I'm not here to make some sort of sacrifice of myself and become the willing servant of other peoples needs. THEY COME HERE TO FULFILL MY NEEDS (upper-case for emphasis only). A concept you may not understand. I get the most benefit out of the whole deal, and they get some benefit as a sort of accidental side affect, another concept you may find hard to wrap your head around.

I give all questions the exact same priority and attention: none. If I want to answer a question, I will. If it iterests me, all the better chance I will answer it. If it is asked intelligently, all the better chance of being answered. If it shows real effort by the person asking, all the better it will be answered.

But as I said, I do it all for my own benefit, for my own selfish reasons, nobody elses reasons and nobody elses benefit. Reasons that are none of yours or anyone elses business unless I choose to share my reasons with them.

If you or anyone else answers questions on forums because of the benefit the other person receives, or based on the other persons needs, I hope you get a warm and fuzzy feeling all over. My opinion is that anyone that answers questions for those reasons is a clueless i-d-i-o-t or just too young to know better.

That same philosophy applies for nearly everything else in life. First and foremost: do something that will benefit you (as long as it does not cause harm to others that you can reasonably be expected to know about) and everyone else will benefit, either directly or indirectly.

Regards,
Kevin

[KevinADC]

Sorry.
Ok, I read on LWP and it doesn't seem to be what I want. What I am wanting to do is have perl download the text from the site so I can import it to a database.
Thanks

See if the site has an RSS feed.

[cobra2341]

No the site doesn't have an rss feed.

[numberwhun]

And what is wrong with answering only questions a person finds interesting? You're welcome to participate here, but that attitude is totally misguided.

I have to, in addition to my earlier statement, second Kevin.

Since dedecating 99% of my time to Perl, I have spent a lot of time in this forum answering questions. More than once, I have found myself researching a module on cpan just to figure out why someone's code wasn't working.

Sure, I got the script working as expected, but the biggest benefit was actually to me. I learned a new Perl module and a few other tricks at the same time. Alsdo, more than once has either Kevin or Miller threw in their .02 and I learned a whole new way to do something.

In your tunnel-visioned view, you see us as rude and not caring when we are actually asking for the person's code for a reason; they want to learn and usd writing your script or doing your homework will not benefit you at all.

Besides, you thing we are rude? Try going over to perlmonks.org and posting questions without the necessasry pertinent information. You will either be sent a link to the posting guidelines, or told to revamp your question to be more informative and measningful.

Regards,

Jeff

[KevinADC]

No the site doesn't have an rss feed.

Rats! That would have been too easy. ;)

[cobra2341]

ok, so what do I need to learn in order to do what I want?
Thanks

[docdiesel]

Hi cobra,

the question "what do I need to learn" forces another question: what do you know so far? Do you already have experience in programming, maybe in another language like C or PHP? Then you'll need to learn Perl, otherwise you'll need to learn programming.

I assume that the latter one is the case. Let's have a look at some simple code:

#!/usr/bin/perl
# top line should be the path to your perl interpreter
# on Unix systems; I think you won't need it on windows
 
# 1) define basic settings and modules to be used 1st.
use strict;
use LWP::UserAgent;
 
### 2) then set your definitions/constants
 
# use "my varname" for first definition
#-- URL to the web page
my $myurl = 'http://www.xxxsite.com/change/this';
 
#-- 3) define misc. variables
my $myagent;
my $myrequest;
my $myresult;
 
 
### 4) then comes the code
 
#-- create an http user agent
$myagent = new LWP::UserAgent;
 
 
#-- Prepare & send request
$myrequest = new HTTP::Request GET => $myurl;
$myresult  = $myagent->request($myrequest);
 
#-- check if everything went fine
#-- if yes, write to stdout
#-- if no, die & write error msg
if( $myresult->is_success )
{
  print $myrequest->content;
} 
else
{
  die "HTTP request failed.";
}
 

Btw., do you know about Perl modules and how to handle them? Anyway, I agree to that point that you should look for some tutorials like at www.tizag.com or similar.

Regards,

Bernd

[cobra2341]

I am doing MySQL right now, and was going to use this data for a database. I really didn't want to learn alot about perl, I just wanted to use it to get this data. I have no idea what that code did. I need the data to actually download so I can use it for a database, not just say success or die. Thanks

[numberwhun]

While I do understand your situation, this however, is not a code writing service, it is a learning forum. Granted, there are people here who will happily take your $$$ and write you the script that you are looking for, but I doubt you will find anyone to just write it or finish it for you.

Regards,

Jeff

[cobra2341]

I understand. I am going to finsh learning MySQL then I'll try some Perl. Thanks for all the help

[KevinADC]

I understand you are not wanting to learn perl, but if you had taken maybe 15 minutes to read the LWP::UserAgent documentation I'm sure you could have figured it out.

[docdiesel]

Hi,

I am doing MySQL right now, and was going to use this data for a database. I really didn't want to learn alot about perl, I just wanted to use it to get this data. I have no idea what that code did. I need the data to actually download so I can use it for a database, not just say success or die. Thanks

well, it was a simple example which you could have based a script with your needs on. Nevermind, can't always be a success.

If that data to be downloaded is right in the format to import into mysql by using its command line interface, you may try wget also (see http://en.wikipedia.org/wiki/Wget ).

Write a batch to wget' the data and run mysql cmd line on the downloaded file subsequently. For that mysql is something you know I guess you won't need help there.

Otherwise, have fun learning perl.

Regards,

Bernd

[cobra2341]

I did read the LWP and I didn't think it was what I needed. Thanks

[cobra2341]

ok i used LWP and downloaded the page. But all the text on the page is missing

[Kelicula]

And what is wrong with answering only questions a person finds interesting? You're welcome to participate here, but that attitude is totally misguided.

Personally, I have no concern at all for the person posting the question. I answer questions because I want to, for my own reasons, not for anyone elses reasons or for anyone elses "needs". It's all for my own selfish benefit. If it were not for my benefit, I would never answer one single question on this or any other forum. And because it is for my own selfish benefit, I am very prolific in this regards. I have helped literally thousands of people over the years on forums in a variety of disciplines, but these days it's 99% perl that interests me. If you do something like this for the other persons benefit or based on their needs, you will help a few people at best.

If it were not for my own benefit why would I continue? I'm not here to make some sort of sacrifice of myself and become the willing servant of other peoples needs. THEY COME HERE TO FULFILL MY NEEDS (upper-case for emphasis only). A concept you may not understand. I get the most benefit out of the whole deal, and they get some benefit as a sort of accidental side affect, another concept you may find hard to wrap your head around.

I give all questions the exact same priority and attention: none. If I want to answer a question, I will. If it iterests me, all the better chance I will answer it. If it is asked intelligently, all the better chance of being answered. If it shows real effort by the person asking, all the better it will be answered.

But as I said, I do it all for my own benefit, for my own selfish reasons, nobody elses reasons and nobody elses benefit. Reasons that are none of yours or anyone elses business unless I choose to share my reasons with them.

If you or anyone else answers questions on forums because of the benefit the other person receives, or based on the other persons needs, I hope you get a warm and fuzzy feeling all over. My opinion is that anyone that answers questions for those reasons is a clueless i-d-i-o-t or just too young to know better.

That same philosophy applies for nearly everything else in life. First and foremost: do something that will benefit you (as long as it does not cause harm to others that you can reasonably be expected to know about) and everyone else will benefit, either directly or indirectly.

Regards,
Kevin

Understood.
I apologize, I was very frustrated that night, and somehow thought that I would motivate a more positive attitude with my post (or reply).
I obviously was blinded by anger/frustration, because it had the opposite effect, and served to work against my needs. You're right, I'm here only for my needs as well. In the same manner you described.

I hope I haven't permanently angered members of this forum, I do enjoy it here.
You guys have been extremely helpful.

I will "watch my mouth" as they say in the future.
(and my thoughts really)

Sincere apology!

[numberwhun]

Understood.
I apologize, I was very frustrated that night, and somehow thought that I would motivate a more positive attitude with my post (or reply).
I obviously was blinded by anger/frustration, because it had the opposite effect, and served to work against my needs. You're right, I'm here only for my needs as well. In the same manner you described.

I hope I haven't permanently angered members of this forum, I do enjoy it here.
You guys have been extremely helpful.

I will "watch my mouth" as they say in the future.
(and my thoughts really)

Sincere apology!

That is the problem with letting your fingers to the thinking instead of doing what the brain and its thought process tell them. One must always be in control of ones responses at all times, especially when first impressions tend to leave a lasting mark.

Speaking only for me, appology accepted.

Regards,

Jeff