GOOGLE NEWS PARSER - Visual Basic .NET

Krakatioison

I realized I am an idiot. Whole saturday wasted on this crazy regex crap, I
can't even look at it anymore...uff
Honestly, I am even willing to pay to someone who is able to solve this.
Please help, I really used all my knowledge on this and I cannot get it to
work.

Let me explain:

I need to parse, URL , URL TEXT and DESCRIPTION TEXT from GOOGLE NEWS.

Every headline in google news has the same structure and it looks like this:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q =http://www.canada.com/sports/st
ory.html%3Fid%3 D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a> 
Canada.c om -15 mi nutes ago 
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. 

go to lets say: http://news.google.com/news/en/us/sports.html and look up
the source if needed.
So, this is my parser (function) for grabbing links. And it works (thanks
God) and I am able to extract with this function all 20 major headlines URL
links:

Public Function ParseLinks(ByVa l HTML As String) As ArrayList
Dim objRegEx As System.Text.Reg ularExpressions .Regex

Dim objMatch As System.Text.Reg ularExpressions .Match

Dim arrLinks As New System.Collecti ons.ArrayList

objRegEx = New System.Text.Reg ularExpressions .Regex("(?:y
[hH][rR][eE][fF]\s*=)(?:[\s""']*)(?!#|[Mm]ailto|[lL]ocation.|[jJ]avascript|.
*css|.*this\.)( .*?)(?:[\s>""'])",
System.Text.Reg ularExpressions .RegexOptions.I gnoreCase Or
System.Text.Reg ularExpressions .RegexOptions.C ompiled)

objMatch = objRegEx.Match( HTML)

While objMatch.Succes s

Dim strMatch As String

strMatch = objMatch.Groups (1).ToString

arrLinks.Add(st rMatch)

objMatch = objMatch.NextMa tch()

End While

Return arrLinks

End Function

Now as you probably guessed already, my problem is that I am simply not able
to write the same function for extraction of URL TEXT and DESCRIPTION TEXT.

please help

K.

Nov 20 '05 #1

Subscribe Reply

2568

Jared

Krakatioison,
What do you mean by url, url test and description text. show me which
portions you need.
Jared

"Krakatiois on" <Kr**********@h uh.com> wrote in message
news:41******** **@Usenet.com.. .

I realized I am an idiot. Whole saturday wasted on this crazy regex crap, I
can't even look at it anymore...uff
Honestly, I am even willing to pay to someone who is able to solve this.
Please help, I really used all my knowledge on this and I cannot get it to
work.

Let me explain:

I need to parse, URL , URL TEXT and DESCRIPTION TEXT from GOOGLE NEWS.

Every headline in google news has the same structure and it looks like
this:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q =http://www.canada.com/sports/st
ory.html%3Fid%3 D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a> 
Canada.c om -15 mi nutes ago 
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. 

go to lets say: http://news.google.com/news/en/us/sports.html and look up
the source if needed.
So, this is my parser (function) for grabbing links. And it works (thanks
God) and I am able to extract with this function all 20 major headlines
URL
links:

Public Function ParseLinks(ByVa l HTML As String) As ArrayList
Dim objRegEx As System.Text.Reg ularExpressions .Regex

Dim objMatch As System.Text.Reg ularExpressions .Match

Dim arrLinks As New System.Collecti ons.ArrayList

objRegEx = New System.Text.Reg ularExpressions .Regex("(?:y
[hH][rR][eE][fF]\s*=)(?:[\s""']*)(?!#|[Mm]ailto|[lL]ocation.|[jJ]avascript|.
*css|.*this\.)( .*?)(?:[\s>""'])",
System.Text.Reg ularExpressions .RegexOptions.I gnoreCase Or
System.Text.Reg ularExpressions .RegexOptions.C ompiled)

objMatch = objRegEx.Match( HTML)

While objMatch.Succes s

Dim strMatch As String

strMatch = objMatch.Groups (1).ToString

arrLinks.Add(st rMatch)

objMatch = objMatch.NextMa tch()

End While

Return arrLinks

End Function

Now as you probably guessed already, my problem is that I am simply not
able
to write the same function for extraction of URL TEXT and DESCRIPTION
TEXT.

please help

K.

Nov 20 '05 #2

Krakatioison

Hi Jared,
so when you look at this google html:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q =http://www.canada.com/sports/st
ory.html%3Fid%3 D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a> 
Canada.c om -15 mi nutes ago 
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. 

This is what I need to get:

LINK:
http://news.google.com/url?ntc=05SA0...D-01CEBC350782
LINKS TEXT: Everyone expected Ullrich to show up - instead it was teammate
Kloden
DESCRIPTION: BESANCON, France (AP) - Look for a German cyclist to be on the
Tour de France podium Sunday - just not the one most people expected.

That is what I need.
My parser gets the LINK.

But I need to get LINKS TEXT and DESCRIPTION.

God bless you men If you can extract it from the code.

K.

Nov 20 '05 #3

Krakatioison

Guy's,
I'll pay us$50 to person who can write the regex command.
Looks like there is no other way.
I spent so much of my own time trying to write it without any success.
k.

Nov 20 '05 #4

Ken Tucker [MVP]

Hi,

Why dont you use a news rss instead.

http://www.wired.com/news/rss

http://news.yahoo.com/rss

Google RSS

http://googlenews.74d.com/

http://www.google.com/newsalerts?q=news+rss&hl=en

Parse Web page

http://www.developerfusion.com/show/4415/
RSS Reader

http://www.philweber.com/articles/easy_rss_in_vbnet.htm

Ken
---------------------
"Krakatiois on" <Kr**********@h uh.com> wrote in message
news:41******** **@Usenet.com.. .
I realized I am an idiot. Whole saturday wasted on this crazy regex crap, I
can't even look at it anymore...uff
Honestly, I am even willing to pay to someone who is able to solve this.
Please help, I really used all my knowledge on this and I cannot get it to
work.

Let me explain:

I need to parse, URL , URL TEXT and DESCRIPTION TEXT from GOOGLE NEWS.

Every headline in google news has the same structure and it looks like this:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q =http://www.canada.com/sports/st
ory.html%3Fid%3 D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a> 
Canada.c om -15 mi nutes ago 
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. 

go to lets say: http://news.google.com/news/en/us/sports.html and look up
the source if needed.
So, this is my parser (function) for grabbing links. And it works (thanks
God) and I am able to extract with this function all 20 major headlines URL
links:

Public Function ParseLinks(ByVa l HTML As String) As ArrayList
Dim objRegEx As System.Text.Reg ularExpressions .Regex

Dim objMatch As System.Text.Reg ularExpressions .Match

Dim arrLinks As New System.Collecti ons.ArrayList

objRegEx = New System.Text.Reg ularExpressions .Regex("(?:y
[hH][rR][eE][fF]\s*=)(?:[\s""']*)(?!#|[Mm]ailto|[lL]ocation.|[jJ]avascript|.
*css|.*this\.)( .*?)(?:[\s>""'])",
System.Text.Reg ularExpressions .RegexOptions.I gnoreCase Or
System.Text.Reg ularExpressions .RegexOptions.C ompiled)

objMatch = objRegEx.Match( HTML)

While objMatch.Succes s

Dim strMatch As String

strMatch = objMatch.Groups (1).ToString

arrLinks.Add(st rMatch)

objMatch = objMatch.NextMa tch()

End While

Return arrLinks

End Function

Now as you probably guessed already, my problem is that I am simply not able
to write the same function for extraction of URL TEXT and DESCRIPTION TEXT.

please help

K.

Nov 20 '05 #5

Krakatioison

Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html
This is the html code:

<a class="y"
href=http://www.LINK.com>LI NK TEXT</a> 
Canada.c om -15 mi nutes ago LI
NK DESCRIPTION 

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.

Nov 20 '05 #6

Ken Tucker [MVP]

Hi,

http://www.developerfusion.com/show/4415/

Ken
---------------------
"Krakatiois on" <Kr**********@h uh.com> wrote in message
news:41******** @Usenet.com...
Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html
This is the html code:

<a class="y"
href=http://www.LINK.com>LI NK TEXT</a> 
Canada.c om -15 mi nutes ago LI
NK DESCRIPTION 

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.

Nov 20 '05 #7

Krakatioison

Ken,
I already tried it.
Their code is not build to extract from google.... it doesn't work.
k.
"Ken Tucker [MVP]" <vb***@bellsout h.net> wrote in message
news:Oi******** ******@TK2MSFTN GP12.phx.gbl...

Hi,

http://www.developerfusion.com/show/4415/

Ken
---------------------
"Krakatiois on" <Kr**********@h uh.com> wrote in message
news:41******** @Usenet.com...
Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html
This is the html code:

<a class="y"
href=http://www.LINK.com>LI NK TEXT</a> 
Canada.c om -15 mi nutes ago LI NK DESCRIPTION 

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.

Nov 20 '05 #8

Ken Tucker [MVP]

Hi,

Here is a start.

Dim wc As New System.Net.WebC lient

Dim sr As New System.IO.Strea mReader(wc.Open Read("http://news.google.com/"))

Dim strHtml As String

Dim regLink As New
System.Text.Reg ularExpressions .Regex("\""(?[^\""]*)\""")

Dim regTitle As New System.Text.Reg ularExpressions .Regex(">(.*?)\ <")

Dim regHref As New System.Text.Reg ularExpressions .Regex("\<a
href=""(.*?)""\ >(.*?)\<\/a\>")

Dim m As System.Text.Reg ularExpressions .Match

strHtml = sr.ReadToEnd

Try

For Each m In regHref.Matches (strHtml)

Dim mLink As System.Text.Reg ularExpressions .Match

For Each mLink In regLink.Matches (m.ToString())

Trace.WriteLine (String.Format( "Link {0}", mLink.ToString) )

Next

For Each mLink In regTitle.Matche s(m.ToString())

Dim strTitle As String = mLink.ToString

strTitle = strTitle.Replac e(">", "")

strTitle = strTitle.Replac e("<", "")

Trace.WriteLine (String.Format( "Title {0}", strTitle))

Next

Next

Catch

End Try

sr.Close()

wc.Dispose()

Ken

----------------------------

"Krakatiois on" <Kr**********@h uh.com> wrote in message
news:41******** @Usenet.com...
Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html
This is the html code:

<a class="y"
href=http://www.LINK.com>LI NK TEXT</a> 
Canada.c om -15 mi nutes ago LI
NK DESCRIPTION 

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.

Nov 20 '05 #9

James R

Krakatioison

<snip>

All I need is one line of code for regex to parse LINK, >LINK TEXT,

DESCRIPTION out of this html

</snip>

I think you are asking too much of Regex. Besides, even if you manage to
find the correct command, think of the next person who has to maintain
the code, including yourself. The following works, and is easier to
read. It will not be as fast, but you are only dealing with 20 links.

Public Function ParseGoogle(ByV al GoogleStr As String, ByRef URL As
String, ByRef URLText As String, ByRef Description As String) As Boolean
Dim HTTP_pos As Integer
Dim UrlEnd_pos As Integer
Dim UrlTextEnd_pos As Integer
Dim DscrStart_pos As Integer
Dim Result As Boolean

Result = True

HTTP_pos = InStr(1, GoogleStr, "http", CompareMethod.T ext)
UrlEnd_pos = InStr(1, GoogleStr, ">", CompareMethod.T ext)
UrlTextEnd_pos = InStr(1, GoogleStr, "</a>", CompareMethod.T ext)
DscrStart_pos = InStr(1, GoogleStr, " ", CompareMethod.T ext)
DscrStart_pos = InStr(DscrStart _pos + 1, GoogleStr, " ",
CompareMethod.T ext)

URL = Mid$(GoogleStr, HTTP_pos, UrlEnd_pos - HTTP_pos - 1)
URLText = Mid$(GoogleStr, UrlEnd_pos + 1, UrlTextEnd_pos -
UrlEnd_pos - 1)
Description = Mid$(GoogleStr, DscrStart_pos + 4, Len(GoogleStr) -
DscrStart_pos - 8)

ParseGoogle = Result

End Function

Hope this helps.

James

*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

Nov 20 '05 #10

Similar topics

1796

News feed webpage ?

by: John | last post by:

Hi everyone ! I am a newbie to this newsgroup. I would appreciate if someone can help me to solve this problem. OK, I am currently writing a webpage in PHP that will get RSS file(XML file) (news feed) and parse it using PHP and display articles' link and description. I want everytime a user click on one article's link, it take the user to another page which was written by me (for example, "display.php?aid=43232") (I don't want user...

PHP

9614

Choosing the right parser for parsing C headers

by: Jean de Largentaye | last post by:

Hi, I need to parse a subset of C (a header file), and generate some unit tests for the functions listed in it. I thus need to parse the code, then rewrite function calls with wrong parameters. What I call "shaking the broken tree" :) I chose to make my UT-generator in Python 2.4. However, I am now encountering problems in choosing the right parser for the job. I struggle in choosing between the inappropriate, the out-of-date, the...

Python

3125

Where to look for source of HTML::Parser

by: Himanshu Garg | last post by:

Hello, I am trying to pinpoint an apparent bug in HTML::Parser. The encoding of the text seems to change incorrectly if the locale isn't set properly. However Parser.pm in the directory (/usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi/HTML/) doesn't seem to contain the "real" parsing statements.

Perl

2068

application/xhtml+xml not recognized by Google

by: Andreas Prilop | last post by:

It seems that Google is unable to "recognize" application/xhtml+xml: http://google.com/search?q=www.unics.uni-hannover.de/nhtcapri/ruby-annotation.x.html "File Format: Unrecognized" Then follow the link "View as HTML" to http://google.com/search?q=cache:www.unics.uni-hannover.de/nhtcapri/ruby-annotation.x.html and look at the source text (as compared with the original)! Are they mad?

HTML / CSS

3375

Google Search results

by: No One | last post by:

Has anyone implemented a parser for Google search result's XML in Jave or EJB?

Java

3225

import parser does not import parser.py in same dir on win

by: Joel Hedlund | last post by:

Hi! I have a possibly dumb question about imports. I've written two python modules: parser.py ------------------------------------ class Parser(object): "my parser" ------------------------------------

Python

16418

Linq; expression parser?

by: Marc Gravell | last post by:

In Linq, you can apparently get a meaningful body from and expression's .ToString(); random question - does anybody know if linq also includes a parser? It just seemed it might be a handy way to write a safe but easy implementation (i.e. no codedom) for an IBindingListView.Filter (by compiling to a Predicate<T>). Anybody know if this is possible at all? Marc

C# / C Sharp

2240

Neophyte having trouble Installing XML::Parser module on OS X

by: UncleRic | last post by:

Environment: Mac OS X (10.4.10) on MacBook Pro I'm a Perl Neophyte. I've downloaded the XML::Parser module and am attempting to install it in my working directory (referenced via PERL5LIB env): PERL5LIB=/Users/Ric/Library/Perl/ ls XML-Parser-2.34/ XML-Parser-2.34.tar

Perl

11720

Convert Google Finance data to Array

by: NitinSawant | last post by:

Hello, I'm trying to extract data from Google finance csv file using javasript. http://finance.google.com/finance/historical?q=GOOG&startdate=May+1%2C+2009&enddate=May+5%2C+2009&start=225&num=25&output=csv however the csv file has 6 columns (Date,Open,High,Low,Close,Volume), and i want only two columns (Date,Close) in array from it.

Javascript

8832

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...

Windows Server

9562

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...

C / C++

9386

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...

Online Marketing

9333

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...

Windows Server

9254

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...

General

6799

Access Europe - Using VBA to create a class based on a table - Wed 1 May

by: isladogs | last post by:

The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...

Microsoft Access / VBA

4608

Trying to create a lan-to-lan vpn between two differents networks

by: TSSRALBI | last post by:

Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...

Networking - Hardware / Configuration

4879

Windows Forms - .Net 8.0

by: adsilva | last post by:

A Windows Forms form does not have the event Unload, like VB6. What one acts like?

Visual Basic .NET

3319

transfer the data from one system to another through ip address

by: 6302768590 | last post by:

Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

C# / C Sharp