By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,569 Members | 1,615 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,569 IT Pros & Developers. It's quick & easy.

GOOGLE NEWS PARSER

P: n/a
I realized I am an idiot. Whole saturday wasted on this crazy regex crap, I
can't even look at it anymore...uff
Honestly, I am even willing to pay to someone who is able to solve this.
Please help, I really used all my knowledge on this and I cannot get it to
work.

Let me explain:

I need to parse, URL , URL TEXT and DESCRIPTION TEXT from GOOGLE NEWS.

Every headline in google news has the same structure and it looks like this:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q=http://www.canada.com/sports/st
ory.html%3Fid%3D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. <br>

go to lets say: http://news.google.com/news/en/us/sports.html and look up
the source if needed.
So, this is my parser (function) for grabbing links. And it works (thanks
God) and I am able to extract with this function all 20 major headlines URL
links:

Public Function ParseLinks(ByVal HTML As String) As ArrayList
Dim objRegEx As System.Text.RegularExpressions.Regex

Dim objMatch As System.Text.RegularExpressions.Match

Dim arrLinks As New System.Collections.ArrayList

objRegEx = New System.Text.RegularExpressions.Regex("(?:y
[hH][rR][eE][fF]\s*=)(?:[\s""']*)(?!#|[Mm]ailto|[lL]ocation.|[jJ]avascript|.
*css|.*this\.)(.*?)(?:[\s>""'])",
System.Text.RegularExpressions.RegexOptions.Ignore Case Or
System.Text.RegularExpressions.RegexOptions.Compil ed)

objMatch = objRegEx.Match(HTML)

While objMatch.Success

Dim strMatch As String

strMatch = objMatch.Groups(1).ToString

arrLinks.Add(strMatch)

objMatch = objMatch.NextMatch()

End While

Return arrLinks

End Function

Now as you probably guessed already, my problem is that I am simply not able
to write the same function for extraction of URL TEXT and DESCRIPTION TEXT.

please help

K.
Nov 20 '05 #1
Share this Question
Share on Google+
10 Replies


P: n/a
Krakatioison,
What do you mean by url, url test and description text. show me which
portions you need.
Jared

"Krakatioison" <Kr**********@huh.com> wrote in message
news:41**********@Usenet.com...
I realized I am an idiot. Whole saturday wasted on this crazy regex crap, I
can't even look at it anymore...uff
Honestly, I am even willing to pay to someone who is able to solve this.
Please help, I really used all my knowledge on this and I cannot get it to
work.

Let me explain:

I need to parse, URL , URL TEXT and DESCRIPTION TEXT from GOOGLE NEWS.

Every headline in google news has the same structure and it looks like
this:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q=http://www.canada.com/sports/st
ory.html%3Fid%3D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. <br>

go to lets say: http://news.google.com/news/en/us/sports.html and look up
the source if needed.
So, this is my parser (function) for grabbing links. And it works (thanks
God) and I am able to extract with this function all 20 major headlines
URL
links:

Public Function ParseLinks(ByVal HTML As String) As ArrayList
Dim objRegEx As System.Text.RegularExpressions.Regex

Dim objMatch As System.Text.RegularExpressions.Match

Dim arrLinks As New System.Collections.ArrayList

objRegEx = New System.Text.RegularExpressions.Regex("(?:y
[hH][rR][eE][fF]\s*=)(?:[\s""']*)(?!#|[Mm]ailto|[lL]ocation.|[jJ]avascript|.
*css|.*this\.)(.*?)(?:[\s>""'])",
System.Text.RegularExpressions.RegexOptions.Ignore Case Or
System.Text.RegularExpressions.RegexOptions.Compil ed)

objMatch = objRegEx.Match(HTML)

While objMatch.Success

Dim strMatch As String

strMatch = objMatch.Groups(1).ToString

arrLinks.Add(strMatch)

objMatch = objMatch.NextMatch()

End While

Return arrLinks

End Function

Now as you probably guessed already, my problem is that I am simply not
able
to write the same function for extraction of URL TEXT and DESCRIPTION
TEXT.

please help

K.

Nov 20 '05 #2

P: n/a
Hi Jared,
so when you look at this google html:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q=http://www.canada.com/sports/st
ory.html%3Fid%3D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. <br>

This is what I need to get:

LINK:
http://news.google.com/url?ntc=05SA0...D-01CEBC350782
LINKS TEXT: Everyone expected Ullrich to show up - instead it was teammate
Kloden
DESCRIPTION: BESANCON, France (AP) - Look for a German cyclist to be on the
Tour de France podium Sunday - just not the one most people expected.

That is what I need.
My parser gets the LINK.

But I need to get LINKS TEXT and DESCRIPTION.

God bless you men If you can extract it from the code.

K.
Nov 20 '05 #3

P: n/a
Guy's,
I'll pay us$50 to person who can write the regex command.
Looks like there is no other way.
I spent so much of my own time trying to write it without any success.
k.
Nov 20 '05 #4

P: n/a
Hi,

Why dont you use a news rss instead.

http://www.wired.com/news/rss

http://news.yahoo.com/rss

Google RSS

http://googlenews.74d.com/

http://www.google.com/newsalerts?q=news+rss&hl=en

Parse Web page

http://www.developerfusion.com/show/4415/
RSS Reader

http://www.philweber.com/articles/easy_rss_in_vbnet.htm

Ken
---------------------
"Krakatioison" <Kr**********@huh.com> wrote in message
news:41**********@Usenet.com...
I realized I am an idiot. Whole saturday wasted on this crazy regex crap, I
can't even look at it anymore...uff
Honestly, I am even willing to pay to someone who is able to solve this.
Please help, I really used all my knowledge on this and I cannot get it to
work.

Let me explain:

I need to parse, URL , URL TEXT and DESCRIPTION TEXT from GOOGLE NEWS.

Every headline in google news has the same structure and it looks like this:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q=http://www.canada.com/sports/st
ory.html%3Fid%3D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. <br>

go to lets say: http://news.google.com/news/en/us/sports.html and look up
the source if needed.
So, this is my parser (function) for grabbing links. And it works (thanks
God) and I am able to extract with this function all 20 major headlines URL
links:

Public Function ParseLinks(ByVal HTML As String) As ArrayList
Dim objRegEx As System.Text.RegularExpressions.Regex

Dim objMatch As System.Text.RegularExpressions.Match

Dim arrLinks As New System.Collections.ArrayList

objRegEx = New System.Text.RegularExpressions.Regex("(?:y
[hH][rR][eE][fF]\s*=)(?:[\s""']*)(?!#|[Mm]ailto|[lL]ocation.|[jJ]avascript|.
*css|.*this\.)(.*?)(?:[\s>""'])",
System.Text.RegularExpressions.RegexOptions.Ignore Case Or
System.Text.RegularExpressions.RegexOptions.Compil ed)

objMatch = objRegEx.Match(HTML)

While objMatch.Success

Dim strMatch As String

strMatch = objMatch.Groups(1).ToString

arrLinks.Add(strMatch)

objMatch = objMatch.NextMatch()

End While

Return arrLinks

End Function

Now as you probably guessed already, my problem is that I am simply not able
to write the same function for extraction of URL TEXT and DESCRIPTION TEXT.

please help

K.

Nov 20 '05 #5

P: n/a
Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html
This is the html code:

<a class="y"
href=http://www.LINK.com>LINK TEXT</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>LI
NK DESCRIPTION <br>

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.
Nov 20 '05 #6

P: n/a
Hi,

http://www.developerfusion.com/show/4415/

Ken
---------------------
"Krakatioison" <Kr**********@huh.com> wrote in message
news:41********@Usenet.com...
Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html
This is the html code:

<a class="y"
href=http://www.LINK.com>LINK TEXT</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>LI
NK DESCRIPTION <br>

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.

Nov 20 '05 #7

P: n/a
Ken,
I already tried it.
Their code is not build to extract from google.... it doesn't work.
k.
"Ken Tucker [MVP]" <vb***@bellsouth.net> wrote in message
news:Oi**************@TK2MSFTNGP12.phx.gbl...
Hi,

http://www.developerfusion.com/show/4415/

Ken
---------------------
"Krakatioison" <Kr**********@huh.com> wrote in message
news:41********@Usenet.com...
Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html
This is the html code:

<a class="y"
href=http://www.LINK.com>LINK TEXT</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>LI NK DESCRIPTION <br>

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.

Nov 20 '05 #8

P: n/a
Hi,

Here is a start.

Dim wc As New System.Net.WebClient

Dim sr As New System.IO.StreamReader(wc.OpenRead("http://news.google.com/"))

Dim strHtml As String

Dim regLink As New
System.Text.RegularExpressions.Regex("\""(?<url>[^\""]*)\""")

Dim regTitle As New System.Text.RegularExpressions.Regex(">(.*?)\<")

Dim regHref As New System.Text.RegularExpressions.Regex("\<a
href=""(.*?)""\>(.*?)\<\/a\>")

Dim m As System.Text.RegularExpressions.Match

strHtml = sr.ReadToEnd

Try

For Each m In regHref.Matches(strHtml)

Dim mLink As System.Text.RegularExpressions.Match

For Each mLink In regLink.Matches(m.ToString())

Trace.WriteLine(String.Format("Link {0}", mLink.ToString))

Next

For Each mLink In regTitle.Matches(m.ToString())

Dim strTitle As String = mLink.ToString

strTitle = strTitle.Replace(">", "")

strTitle = strTitle.Replace("<", "")

Trace.WriteLine(String.Format("Title {0}", strTitle))

Next

Next

Catch

End Try

sr.Close()

wc.Dispose()

Ken

----------------------------

"Krakatioison" <Kr**********@huh.com> wrote in message
news:41********@Usenet.com...
Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html
This is the html code:

<a class="y"
href=http://www.LINK.com>LINK TEXT</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>LI
NK DESCRIPTION <br>

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.

Nov 20 '05 #9

P: n/a
Krakatioison

<snip>
All I need is one line of code for regex to parse LINK, >LINK TEXT,

DESCRIPTION out of this html

</snip>

I think you are asking too much of Regex. Besides, even if you manage to
find the correct command, think of the next person who has to maintain
the code, including yourself. The following works, and is easier to
read. It will not be as fast, but you are only dealing with 20 links.

Public Function ParseGoogle(ByVal GoogleStr As String, ByRef URL As
String, ByRef URLText As String, ByRef Description As String) As Boolean
Dim HTTP_pos As Integer
Dim UrlEnd_pos As Integer
Dim UrlTextEnd_pos As Integer
Dim DscrStart_pos As Integer
Dim Result As Boolean

Result = True

HTTP_pos = InStr(1, GoogleStr, "http", CompareMethod.Text)
UrlEnd_pos = InStr(1, GoogleStr, ">", CompareMethod.Text)
UrlTextEnd_pos = InStr(1, GoogleStr, "</a>", CompareMethod.Text)
DscrStart_pos = InStr(1, GoogleStr, "<br>", CompareMethod.Text)
DscrStart_pos = InStr(DscrStart_pos + 1, GoogleStr, "<br>",
CompareMethod.Text)

URL = Mid$(GoogleStr, HTTP_pos, UrlEnd_pos - HTTP_pos - 1)
URLText = Mid$(GoogleStr, UrlEnd_pos + 1, UrlTextEnd_pos -
UrlEnd_pos - 1)
Description = Mid$(GoogleStr, DscrStart_pos + 4, Len(GoogleStr) -
DscrStart_pos - 8)

ParseGoogle = Result

End Function

Hope this helps.

James

*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!
Nov 20 '05 #10

P: n/a
not what I was looking for, but I figured it out now...so thanks anyway

"James R" <an*******@devdex.com> wrote in message
news:O4**************@TK2MSFTNGP12.phx.gbl...
Krakatioison

<snip>
All I need is one line of code for regex to parse LINK, >LINK TEXT,

DESCRIPTION out of this html

</snip>

I think you are asking too much of Regex. Besides, even if you manage to
find the correct command, think of the next person who has to maintain
the code, including yourself. The following works, and is easier to
read. It will not be as fast, but you are only dealing with 20 links.

Public Function ParseGoogle(ByVal GoogleStr As String, ByRef URL As
String, ByRef URLText As String, ByRef Description As String) As Boolean
Dim HTTP_pos As Integer
Dim UrlEnd_pos As Integer
Dim UrlTextEnd_pos As Integer
Dim DscrStart_pos As Integer
Dim Result As Boolean

Result = True

HTTP_pos = InStr(1, GoogleStr, "http", CompareMethod.Text)
UrlEnd_pos = InStr(1, GoogleStr, ">", CompareMethod.Text)
UrlTextEnd_pos = InStr(1, GoogleStr, "</a>", CompareMethod.Text)
DscrStart_pos = InStr(1, GoogleStr, "<br>", CompareMethod.Text)
DscrStart_pos = InStr(DscrStart_pos + 1, GoogleStr, "<br>",
CompareMethod.Text)

URL = Mid$(GoogleStr, HTTP_pos, UrlEnd_pos - HTTP_pos - 1)
URLText = Mid$(GoogleStr, UrlEnd_pos + 1, UrlTextEnd_pos -
UrlEnd_pos - 1)
Description = Mid$(GoogleStr, DscrStart_pos + 4, Len(GoogleStr) -
DscrStart_pos - 8)

ParseGoogle = Result

End Function

Hope this helps.

James

*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!


Posted Via Usenet.com Premium Usenet Newsgroup Services
----------------------------------------------------------
** SPEED ** RETENTION ** COMPLETION ** ANONYMITY **
----------------------------------------------------------
http://www.usenet.com
Nov 20 '05 #11

This discussion thread is closed

Replies have been disabled for this discussion.