469,306 Members | 1,930 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,306 developers. It's quick & easy.

code to fetch data from online source

Hi, first post.
How I'm trying to do is get data like
<a href="http://www.minorleaguebaseball.com/milb/stats/stats.jsp? sid=milb&t=g_log&gid=2007_08_18_colaaa_tolaaa_1">t his</a>
(only the log) to download automactically. Here is a
<a href="http://www.minorleaguebaseball.com/milb/stats/stats.jsp? t=t_sch&cid=445">Schedule </a> that links to all the logs.
I read that perl can 'spider' data. Is there a way to get Perl to fetch all the text that I am wanting?
Thanks
Aug 20 '07 #1
32 2132
numberwhun
3,503 Expert Mod 2GB
Hello, and welcome to the scripts!

Sure, Perl can spider data, but you have to do some coding. You may want to start reading up on LWP if you are going to be grabbing information from web pages.

Regards,

Jeff
Aug 20 '07 #2
Thanks, but I guess I am really bad at perl. I am trying to run the LWP program put nothing happens. What am I doing wrong?
Aug 20 '07 #3
numberwhun
3,503 Expert Mod 2GB
I truly do not know. It seems you have decided to keep your source code a secret from everyone. Why don't you post your code and any errors that you are seeing.

Regards,

Jeff
Aug 20 '07 #4
I'll go step-by-step what I am doing.
First I am opening the Command line.
Then I type perl myprog.pl
I get the error 'Can't open perl script 'myprog.pl': No such file
I have no idea what to do
Aug 20 '07 #5
KevinADC
4,059 Expert 2GB
Write a program that does what you want, name it "myprog.pl", save it to the hard drive. Then try running it from the command line.
Aug 20 '07 #6
so I don't save it in the Perl bin?
Aug 20 '07 #7
KevinADC
4,059 Expert 2GB
Where ever you normally save your perl programs is where you would save it. If that is in the perl/bin folder that is fine.
Aug 20 '07 #8
numberwhun
3,503 Expert Mod 2GB
Ok, I would say that at this point, you really need to find yourself a good Perl tutorial.

My suggestion would be to go to your local Barnes & Noble and pick up a copy of Learning Perl. You need something like that to teach you the basics of the language.

Regards,

Jeff
Aug 20 '07 #9
Is there any good online reads. I found some, but when I have questions, there's no one to anwser my questions. That's why I came here
Aug 21 '07 #10
Kelicula
176 Expert 100+
Yeah some people aren't very tolerant here. It seems that they are only interested in answering questions that "they" find interesting. Not much concern for the person who's posting's needs.

Anyway, when you open the command line, are you in the directory that the script is located in? You have to get there first then type perl myprog.pl. Also make sure to use the "include warnings" switch (-w) like this:
perl -w myprog.pl that will enable the warnings to output to your screen.

I found a really good perl tutorial online here.
http://www.cbkihong.com/download/perltut.pdf

I suggest saving this to your comp, and slowly taking the time to read it, making sure to do the examples yourself as you go.

Good luck!
Aug 21 '07 #11
Thanks alot. I think this is a great site and I have gotten alot more responds than I have before.
One question, What do you mean by are you in the directory that the script is located in?
Thanks alot
Aug 21 '07 #12
Kelicula
176 Expert 100+
Well I guess first I should say, what system are you running? I was reffering to ActiveSate perl for windows. I open the DOS prompt. Then of course I have to cd into the directory that my scripts are in and then run them. I asume it is the same for other systems. You have to be in the "working directory" of the script to run it. in linux/unix type pwd (print working directory) to see what directory you're in. then cd (change directory) ect...

hope were on the same page. ;)
Aug 21 '07 #13
I read the guide you gave me and I understand I must do cd: first.
Thanks alot
Aug 21 '07 #14
numberwhun
3,503 Expert Mod 2GB
Yeah some people aren't very tolerant here. It seems that they are only interested in answering questions that "they" find interesting. Not much concern for the person who's posting's needs.
Be that as you may think it, but its not a matter of being tolerant. This is a forum for learning to code/script in Perl. You don't look at a computer one day, having never touched one and say, I think I'll learn that today. Some semblance of computer knowledge (ie: knowing what it means to be in the same directory as your script) is actually implied. We aren't trying to be rude and crude, but we aren't out to teach everyone how to turn their computer on. If that were the case, this would be the Misc. Questions or even the Windows forum.

As I mentioned in my earlier posting, there are a number of great Perl tutorials out there. One that I liked was tizag.com. It was well put together and understandable. Another think that you could do is to go to the perldoc page and start reading all of the resources there. The FAQ's will be especially helpful once you get the basics down, but I would start reading those and get up to speed.

We are definitely wanting to help and assist every person who comes in the door, but when one says, "It doesn't work, Why?" and that is all they give you, you have to prompt them for more information. We aren't mind reader!

Regards,

Jeff
Aug 21 '07 #15
Sorry.
Ok, I read on LWP and it doesn't seem to be what I want. What I am wanting to do is have perl download the text from the site so I can import it to a database.
Thanks
Aug 21 '07 #16
numberwhun
3,503 Expert Mod 2GB
Well, any way you look at it, what you are going to be downloading is the web page itself, which includes all of the HTML. Now, you could use LWP::Simple 's get function to fetch the web page, but you are going to have to parse out the text with a regex, or go so far as to use a module that helps to parse HTML into its elements (I would look one up, but am exhausted and on the way to bed at the moment).

Regards,

Jeff
Aug 21 '07 #17
KevinADC
4,059 Expert 2GB
Yeah some people aren't very tolerant here. It seems that they are only interested in answering questions that "they" find interesting. Not much concern for the person who's posting's needs.
And what is wrong with answering only questions a person finds interesting? You're welcome to participate here, but that attitude is totally misguided.

Personally, I have no concern at all for the person posting the question. I answer questions because I want to, for my own reasons, not for anyone elses reasons or for anyone elses "needs". It's all for my own selfish benefit. If it were not for my benefit, I would never answer one single question on this or any other forum. And because it is for my own selfish benefit, I am very prolific in this regards. I have helped literally thousands of people over the years on forums in a variety of disciplines, but these days it's 99% perl that interests me. If you do something like this for the other persons benefit or based on their needs, you will help a few people at best.

If it were not for my own benefit why would I continue? I'm not here to make some sort of sacrifice of myself and become the willing servant of other peoples needs. THEY COME HERE TO FULFILL MY NEEDS (upper-case for emphasis only). A concept you may not understand. I get the most benefit out of the whole deal, and they get some benefit as a sort of accidental side affect, another concept you may find hard to wrap your head around.

I give all questions the exact same priority and attention: none. If I want to answer a question, I will. If it iterests me, all the better chance I will answer it. If it is asked intelligently, all the better chance of being answered. If it shows real effort by the person asking, all the better it will be answered.

But as I said, I do it all for my own benefit, for my own selfish reasons, nobody elses reasons and nobody elses benefit. Reasons that are none of yours or anyone elses business unless I choose to share my reasons with them.

If you or anyone else answers questions on forums because of the benefit the other person receives, or based on the other persons needs, I hope you get a warm and fuzzy feeling all over. My opinion is that anyone that answers questions for those reasons is a clueless i-d-i-o-t or just too young to know better.

That same philosophy applies for nearly everything else in life. First and foremost: do something that will benefit you (as long as it does not cause harm to others that you can reasonably be expected to know about) and everyone else will benefit, either directly or indirectly.

Regards,
Kevin
Aug 21 '07 #18
KevinADC
4,059 Expert 2GB
Sorry.
Ok, I read on LWP and it doesn't seem to be what I want. What I am wanting to do is have perl download the text from the site so I can import it to a database.
Thanks
See if the site has an RSS feed.
Aug 21 '07 #19
No the site doesn't have an rss feed.
Aug 21 '07 #20
numberwhun
3,503 Expert Mod 2GB
And what is wrong with answering only questions a person finds interesting? You're welcome to participate here, but that attitude is totally misguided.
I have to, in addition to my earlier statement, second Kevin.

Since dedecating 99% of my time to Perl, I have spent a lot of time in this forum answering questions. More than once, I have found myself researching a module on cpan just to figure out why someone's code wasn't working.

Sure, I got the script working as expected, but the biggest benefit was actually to me. I learned a new Perl module and a few other tricks at the same time. Alsdo, more than once has either Kevin or Miller threw in their .02 and I learned a whole new way to do something.

In your tunnel-visioned view, you see us as rude and not caring when we are actually asking for the person's code for a reason; they want to learn and usd writing your script or doing your homework will not benefit you at all.

Besides, you thing we are rude? Try going over to perlmonks.org and posting questions without the necessasry pertinent information. You will either be sent a link to the posting guidelines, or told to revamp your question to be more informative and measningful.

Regards,

Jeff
Aug 21 '07 #21
KevinADC
4,059 Expert 2GB
No the site doesn't have an rss feed.

Rats! That would have been too easy. ;)
Aug 21 '07 #22
ok, so what do I need to learn in order to do what I want?
Thanks
Aug 21 '07 #23
docdiesel
297 Expert 100+
Hi cobra,

the question "what do I need to learn" forces another question: what do you know so far? Do you already have experience in programming, maybe in another language like C or PHP? Then you'll need to learn Perl, otherwise you'll need to learn programming.

I assume that the latter one is the case. Let's have a look at some simple code:

Expand|Select|Wrap|Line Numbers
  1. #!/usr/bin/perl
  2. # top line should be the path to your perl interpreter
  3. # on Unix systems; I think you won't need it on windows
  4.  
  5. # 1) define basic settings and modules to be used 1st.
  6. use strict;
  7. use LWP::UserAgent;
  8.  
  9. ### 2) then set your definitions/constants
  10.  
  11. # use "my varname" for first definition
  12. #-- URL to the web page
  13. my $myurl = 'http://www.xxxsite.com/change/this';
  14.  
  15. #-- 3) define misc. variables
  16. my $myagent;
  17. my $myrequest;
  18. my $myresult;
  19.  
  20.  
  21. ### 4) then comes the code
  22.  
  23. #-- create an http user agent
  24. $myagent = new LWP::UserAgent;
  25.  
  26.  
  27. #-- Prepare & send request
  28. $myrequest = new HTTP::Request GET => $myurl;
  29. $myresult  = $myagent->request($myrequest);
  30.  
  31. #-- check if everything went fine
  32. #-- if yes, write to stdout
  33. #-- if no, die & write error msg
  34. if( $myresult->is_success )
  35. {
  36.   print $myrequest->content;
  37. else
  38. {
  39.   die "HTTP request failed.";
  40. }
  41.  
Btw., do you know about Perl modules and how to handle them? Anyway, I agree to that point that you should look for some tutorials like at www.tizag.com or similar.

Regards,

Bernd
Aug 21 '07 #24
I am doing MySQL right now, and was going to use this data for a database. I really didn't want to learn alot about perl, I just wanted to use it to get this data. I have no idea what that code did. I need the data to actually download so I can use it for a database, not just say success or die. Thanks
Aug 22 '07 #25
numberwhun
3,503 Expert Mod 2GB
While I do understand your situation, this however, is not a code writing service, it is a learning forum. Granted, there are people here who will happily take your $$$ and write you the script that you are looking for, but I doubt you will find anyone to just write it or finish it for you.

Regards,

Jeff
Aug 22 '07 #26
I understand. I am going to finsh learning MySQL then I'll try some Perl. Thanks for all the help
Aug 23 '07 #27
KevinADC
4,059 Expert 2GB
I understand you are not wanting to learn perl, but if you had taken maybe 15 minutes to read the LWP::UserAgent documentation I'm sure you could have figured it out.
Aug 23 '07 #28
docdiesel
297 Expert 100+
Hi,

I am doing MySQL right now, and was going to use this data for a database. I really didn't want to learn alot about perl, I just wanted to use it to get this data. I have no idea what that code did. I need the data to actually download so I can use it for a database, not just say success or die. Thanks
well, it was a simple example which you could have based a script with your needs on. Nevermind, can't always be a success.

If that data to be downloaded is right in the format to import into mysql by using its command line interface, you may try wget also (see http://en.wikipedia.org/wiki/Wget ).

Write a batch to wget' the data and run mysql cmd line on the downloaded file subsequently. For that mysql is something you know I guess you won't need help there.

Otherwise, have fun learning perl.

Regards,

Bernd
Aug 23 '07 #29
I did read the LWP and I didn't think it was what I needed. Thanks
Aug 23 '07 #30
ok i used LWP and downloaded the page. But all the text on the page is missing
Aug 23 '07 #31
Kelicula
176 Expert 100+
And what is wrong with answering only questions a person finds interesting? You're welcome to participate here, but that attitude is totally misguided.

Personally, I have no concern at all for the person posting the question. I answer questions because I want to, for my own reasons, not for anyone elses reasons or for anyone elses "needs". It's all for my own selfish benefit. If it were not for my benefit, I would never answer one single question on this or any other forum. And because it is for my own selfish benefit, I am very prolific in this regards. I have helped literally thousands of people over the years on forums in a variety of disciplines, but these days it's 99% perl that interests me. If you do something like this for the other persons benefit or based on their needs, you will help a few people at best.

If it were not for my own benefit why would I continue? I'm not here to make some sort of sacrifice of myself and become the willing servant of other peoples needs. THEY COME HERE TO FULFILL MY NEEDS (upper-case for emphasis only). A concept you may not understand. I get the most benefit out of the whole deal, and they get some benefit as a sort of accidental side affect, another concept you may find hard to wrap your head around.

I give all questions the exact same priority and attention: none. If I want to answer a question, I will. If it iterests me, all the better chance I will answer it. If it is asked intelligently, all the better chance of being answered. If it shows real effort by the person asking, all the better it will be answered.

But as I said, I do it all for my own benefit, for my own selfish reasons, nobody elses reasons and nobody elses benefit. Reasons that are none of yours or anyone elses business unless I choose to share my reasons with them.

If you or anyone else answers questions on forums because of the benefit the other person receives, or based on the other persons needs, I hope you get a warm and fuzzy feeling all over. My opinion is that anyone that answers questions for those reasons is a clueless i-d-i-o-t or just too young to know better.

That same philosophy applies for nearly everything else in life. First and foremost: do something that will benefit you (as long as it does not cause harm to others that you can reasonably be expected to know about) and everyone else will benefit, either directly or indirectly.

Regards,
Kevin

Understood.
I apologize, I was very frustrated that night, and somehow thought that I would motivate a more positive attitude with my post (or reply).
I obviously was blinded by anger/frustration, because it had the opposite effect, and served to work against my needs. You're right, I'm here only for my needs as well. In the same manner you described.

I hope I haven't permanently angered members of this forum, I do enjoy it here.
You guys have been extremely helpful.

I will "watch my mouth" as they say in the future.
(and my thoughts really)

Sincere apology!
Oct 5 '07 #32
numberwhun
3,503 Expert Mod 2GB
Understood.
I apologize, I was very frustrated that night, and somehow thought that I would motivate a more positive attitude with my post (or reply).
I obviously was blinded by anger/frustration, because it had the opposite effect, and served to work against my needs. You're right, I'm here only for my needs as well. In the same manner you described.

I hope I haven't permanently angered members of this forum, I do enjoy it here.
You guys have been extremely helpful.

I will "watch my mouth" as they say in the future.
(and my thoughts really)

Sincere apology!
That is the problem with letting your fingers to the thinking instead of doing what the brain and its thought process tell them. One must always be in control of ones responses at all times, especially when first impressions tend to leave a lasting mark.

Speaking only for me, appology accepted.

Regards,

Jeff
Oct 5 '07 #33

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

2 posts views Thread by Michel Esber | last post: by
232 posts views Thread by robert maas, see http://tinyurl.com/uh3t | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
1 post views Thread by Geralt96 | last post: by
reply views Thread by harlem98 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.