Regular Expression Matches

[Pete Davis]

I'm using regular expressions to extract some data and some links from some
web pages. I download the page and then I want to get a list of certain
links.

For building regular expressions, I use an app call The Regulator, which
makes it pretty easy to build and test regular expressions.

As a warning, I'm real weak with regular expressions. Let's say my regular
expression is:

(href=)(?<caturl>.*)(class=title>\[&nbsp;&nbsp;&nbsp;)

Now, using The Regulator and giving it the source for a particular web page,
I get 8 matches.

According to the regulator, the options it's using are:

Multiline, ignore case, ignore whitespace

In my own code, I'm doing:

Regex indexRegex = new Regex(categoryListRegex,
RegexOptions.Multiline |
RegexOptions.IgnorePatternWhitespace |
RegexOptions.IgnoreCase);
MatchCollection indexMatches = indexRegex.Matches(pageText);

This only returns one match in indexMatches with the same page that I'm
giving The Regulator. It seems that no matter what combination of regex
options I use, I'm only getting one match.

Why is that? How do I get all 8 matches?

Thanks.

pete

[Kevin Spencer]

Hi Pete,

You need to escape the '<' and '>' characters in your Regurlar Expression.
These are used in some flavors of Regular Expression language to indicate a
named group. If the first (<caturl>) is a group name, name both groups or
neither.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:1e******************************@giganews.com ...

I'm using regular expressions to extract some data and some links from
some web pages. I download the page and then I want to get a list of
certain links.

For building regular expressions, I use an app call The Regulator, which
makes it pretty easy to build and test regular expressions.

As a warning, I'm real weak with regular expressions. Let's say my regular
expression is:

(href=)(?<caturl>.*)(class=title>\[&nbsp;&nbsp;&nbsp;)

Now, using The Regulator and giving it the source for a particular web
page, I get 8 matches.

According to the regulator, the options it's using are:

Multiline, ignore case, ignore whitespace

In my own code, I'm doing:

Regex indexRegex = new Regex(categoryListRegex,
RegexOptions.Multiline |
RegexOptions.IgnorePatternWhitespace |
RegexOptions.IgnoreCase);
MatchCollection indexMatches = indexRegex.Matches(pageText);

This only returns one match in indexMatches with the same page that I'm
giving The Regulator. It seems that no matter what combination of regex
options I use, I'm only getting one match.

Why is that? How do I get all 8 matches?

Thanks.

pete

[Pete Davis]

Well, there are 3 groups. <caturl> is a group name. The other 2 are unnamed.
Why do they need to be named if <caturl> is named? I'm not interested in the
other groups. I'm simply using them as "delimiters" for lack of a better
word.

I've modified the expression to look like this:

(href=)(?<caturl>.*)(class=title.*\[&nbsp;&nbsp;&nbsp;)

This gives the exact same results.

The escaping stuff gets a little confusing because the regular expressions
are actually stored in an XML file, so they get escaped for that.

In the XML file that looks like:

(href=)(?&lt;caturl&gt;.*)(class=title.*\[&amp;nbsp;&amp;nbsp;&amp;nbsp;)

This still isn't returning multiple results. Just the last match. I don't
think the < was the problem.

Pete

"Kevin Spencer" <ke***@DIESPAMMERSDIEtakempis.com> wrote in message
news:%2****************@TK2MSFTNGP11.phx.gbl...

Hi Pete,

You need to escape the '<' and '>' characters in your Regurlar Expression.
These are used in some flavors of Regular Expression language to indicate
a named group. If the first (<caturl>) is a group name, name both groups
or neither.

--
HTH,

Kevin Spencer
Microsoft MVP
.Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:1e******************************@giganews.com ...

I'm using regular expressions to extract some data and some links from
some web pages. I download the page and then I want to get a list of
certain links.

For building regular expressions, I use an app call The Regulator, which
makes it pretty easy to build and test regular expressions.

As a warning, I'm real weak with regular expressions. Let's say my
regular expression is:

(href=)(?<caturl>.*)(class=title>\[&nbsp;&nbsp;&nbsp;)

Now, using The Regulator and giving it the source for a particular web
page, I get 8 matches.

According to the regulator, the options it's using are:

Multiline, ignore case, ignore whitespace

In my own code, I'm doing:

Regex indexRegex = new Regex(categoryListRegex,
RegexOptions.Multiline |
RegexOptions.IgnorePatternWhitespace |
RegexOptions.IgnoreCase);
MatchCollection indexMatches = indexRegex.Matches(pageText);

This only returns one match in indexMatches with the same page that I'm
giving The Regulator. It seems that no matter what combination of regex
options I use, I'm only getting one match.

Why is that? How do I get all 8 matches?

Thanks.

pete

[Pete Davis]

I found the solution to the problem, so now if someone could explain why,
I'd appreciate that:

The solution is to replace carriage returns with line feeds.

If I do that, I get all 8 results instead of the 1 result I was getting.

Now, can anyone tell me WHY? And aren't there other regex options for
dealing with that without having to change the source text?

Thanks.

Pete

[Kevin Spencer]

> The solution is to replace carriage returns with line feeds.

I didn't see any carriage returns or line feeds in your regular expression.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:pK********************@giganews.com...

I found the solution to the problem, so now if someone could explain why,
I'd appreciate that:

The solution is to replace carriage returns with line feeds.

If I do that, I get all 8 results instead of the 1 result I was getting.

Now, can anyone tell me WHY? And aren't there other regex options for
dealing with that without having to change the source text?

Thanks.

Pete

[Pete Davis]

The Carriage Returns and Linefeeds are in the web page I'm scanning, not the
regex.

Pete
"Kevin Spencer" <ke***@DIESPAMMERSDIEtakempis.com> wrote in message
news:u4*************@tk2msftngp13.phx.gbl...

The solution is to replace carriage returns with line feeds.

I didn't see any carriage returns or line feeds in your regular
expression.

--
HTH,

Kevin Spencer
Microsoft MVP
.Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:pK********************@giganews.com...

I found the solution to the problem, so now if someone could explain why,
I'd appreciate that:

The solution is to replace carriage returns with line feeds.

If I do that, I get all 8 results instead of the 1 result I was getting.

Now, can anyone tell me WHY? And aren't there other regex options for
dealing with that without having to change the source text?

Thanks.

Pete

[Kevin Spencer]

Well, Pete, I'm not sure, but I do know that Microsoft and Unix text
documents have a distinct difference in terms of line feeds. The Microsoft
text document uses CR/LF (\r\n), while the Unix model uses only LF(\n). Does
that tell you anything?

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:Re******************************@giganews.com ...

The Carriage Returns and Linefeeds are in the web page I'm scanning, not
the regex.

Pete
"Kevin Spencer" <ke***@DIESPAMMERSDIEtakempis.com> wrote in message
news:u4*************@tk2msftngp13.phx.gbl...

The solution is to replace carriage returns with line feeds.

I didn't see any carriage returns or line feeds in your regular
expression.

--
HTH,

Kevin Spencer
Microsoft MVP
.Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:pK********************@giganews.com...

I found the solution to the problem, so now if someone could explain why,
I'd appreciate that:

The solution is to replace carriage returns with line feeds.

If I do that, I get all 8 results instead of the 1 result I was getting.

Now, can anyone tell me WHY? And aren't there other regex options for
dealing with that without having to change the source text?

Thanks.

Pete

[Pete Davis]

Kevin,

Thanks for the help. The Unix/Windows difference on crlf vs. lf I'm aware
of, but that's not really the issue.

The Regex engine in .NET, I would assume, is more geared towards windows
documents if anything, but maybe not.

There are Regex options for singleline and multiline to help determine how
the parser handles the data, but it's unclear to me how to get all matches
from my document without replacing the carriage returns. Nothing I've tried
has worked. I suspect there are changes I can make to the regular
expressions themselves to handle it, but I don't know how to do that.

Pete
"Kevin Spencer" <ke***@DIESPAMMERSDIEtakempis.com> wrote in message
news:Os****************@TK2MSFTNGP12.phx.gbl...

Well, Pete, I'm not sure, but I do know that Microsoft and Unix text
documents have a distinct difference in terms of line feeds. The Microsoft
text document uses CR/LF (\r\n), while the Unix model uses only LF(\n).
Does that tell you anything?

--
HTH,

Kevin Spencer
Microsoft MVP
.Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:Re******************************@giganews.com ...

The Carriage Returns and Linefeeds are in the web page I'm scanning, not
the regex.

Pete
"Kevin Spencer" <ke***@DIESPAMMERSDIEtakempis.com> wrote in message
news:u4*************@tk2msftngp13.phx.gbl...

The solution is to replace carriage returns with line feeds.

I didn't see any carriage returns or line feeds in your regular
expression.

--
HTH,

Kevin Spencer
Microsoft MVP
.Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:pK********************@giganews.com...
I found the solution to the problem, so now if someone could explain
why, I'd appreciate that:

The solution is to replace carriage returns with line feeds.

If I do that, I get all 8 results instead of the 1 result I was
getting.

Now, can anyone tell me WHY? And aren't there other regex options for
dealing with that without having to change the source text?

Thanks.

Pete

[Larry Lard]

Pete Davis wrote:

I'm using regular expressions to extract some data and some links from some
web pages. I download the page and then I want to get a list of certain
links.

Completely not answering your actual question, but maybe this will save
you a lot of bother: the free HtmlAgilityPack will convert an (even
malformed) HTML document into a nice XML tree which makes getting
content out much much easier.

--
Larry Lard
Replies to group please

[Kevin Spencer]

Hi Pete,

Here are some Regular Expressions that may help with this sort of thing. I
tend to use these rather than the options in the Regex engine:

(?i) Turn of case-sensitivity for the remainder
of the regular expression.
(?s) Turn on "dot matches now line" for the
remainder of the regular expression.
(?m) Caret and dollar sign match after and before
new lines for the remainder of the regular expression.
(?i-sm) Tunrs on the "i" and "m" options for the remainder
of the regular expression, and turn off "s"
(?i-sm:regex) Turns on the "i" and "m" options
for the regular expression inside the parentheses,
and turn off "s"

Here is the Microsoft .Net Framework Regular Expressions reference:

http://msdn.microsoft.com/library/de...geElements.asp

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:aa********************@giganews.com...

Kevin,

Thanks for the help. The Unix/Windows difference on crlf vs. lf I'm aware
of, but that's not really the issue.

The Regex engine in .NET, I would assume, is more geared towards windows
documents if anything, but maybe not.

There are Regex options for singleline and multiline to help determine how
the parser handles the data, but it's unclear to me how to get all matches
from my document without replacing the carriage returns. Nothing I've
tried has worked. I suspect there are changes I can make to the regular
expressions themselves to handle it, but I don't know how to do that.

Pete
"Kevin Spencer" <ke***@DIESPAMMERSDIEtakempis.com> wrote in message
news:Os****************@TK2MSFTNGP12.phx.gbl...

Well, Pete, I'm not sure, but I do know that Microsoft and Unix text
documents have a distinct difference in terms of line feeds. The
Microsoft text document uses CR/LF (\r\n), while the Unix model uses only
LF(\n). Does that tell you anything?

--
HTH,

Kevin Spencer
Microsoft MVP
.Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:Re******************************@giganews.com ...

The Carriage Returns and Linefeeds are in the web page I'm scanning, not
the regex.

Pete
"Kevin Spencer" <ke***@DIESPAMMERSDIEtakempis.com> wrote in message
news:u4*************@tk2msftngp13.phx.gbl...
> The solution is to replace carriage returns with line feeds.

I didn't see any carriage returns or line feeds in your regular
expression.

--
HTH,

Kevin Spencer
Microsoft MVP
.Net Developer
To a tea you esteem
a hurting back as a wallet.
"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:pK********************@giganews.com...
>I found the solution to the problem, so now if someone could explain
>why, I'd appreciate that:
>
> The solution is to replace carriage returns with line feeds.
>
> If I do that, I get all 8 results instead of the 1 result I was
> getting.
>
> Now, can anyone tell me WHY? And aren't there other regex options for
> dealing with that without having to change the source text?
>
> Thanks.
>
> Pete
>