Loopy over loops

[GeorgeAtkins]

Using VB.NET in VS 2003.
This should be a simple routine, but it has me flummoxed.
I have to compare strings in two text files:
FILE1(srar) consists of lines of book titles.
FILE (srac) consists of multi-line “book records�, separated by blank rows.
The number of rows in each “book record� varies, as in this diagram:

1xxxxxx
xxxxxx
TitleRow
xxxxxx

2xxxxxx
TitleRow
xxxxxx
xxxxxx
xxxxxx
All I’m trying to do is to first read a title in File1, then read through
the entire File2 file, one “book record� at a time, looking for a matching
title. After other processing (not relevant here) and all book records have
been searched, go to the next File1 title and read all the book records in
File2 again. In short: typical looping within a loop.

The problem is that I cannot get it to work correctly! I know the issue
involves the blank rows and how the streamreader works. Here are core
excerpts of my code. Any enlightenment is greatly appreciated!

Dim RowCntr, ARBldr, ARCntr, MARCRowCntr, z, StartPos, TabPos As Long
Dim C1, C2, BkTitle, ArTitle, Arline, Acline, ArWtr, AcWtr, arbk As String
Dim srAR As System.IO.StreamReader = New StreamReader(FILE1.txt,
Encoding.GetEncoding(1252))
Dim srAC As System.IO.StreamReader = New StreamReader(FILE2.txt,
Encoding.GetEncoding(1252))

' Loop through the AR file (FILE1), reading each book title
For ARCntr = 0 To RdrArray.GetUpperBound(1)
MarcCntr = 1 ' initialize marc record counter variable
‘ Read through FILE2, one “book record� at a time.
Do
RowCntr = 1 ‘ reset variable for next book record row counter
' Now, the Inner loop supposed to read all lines for a single book
record.
Do
Acline = srAC.ReadLine
LibList.Add(Acline) ' new field row
If Acline=�� Then ‘ found the blank row
LibList.Add(vbCr) ' new field row
RowCntr += 1 ' set value of RowCntr
End If
Loop Until srAC.Peek = -1
Loop Until srAC.Peek = -1 ' of first DO. Get another book record
Next ARCntr ' of the original FOR loop. Get another book title

I hope this is enough information to work with. I'll reply with more info if
necessary.
Thanks again for any help!

George

[Rod Gill]

Hi,

Can't give you code (I'm learning vb.net myself!) but I would:

Read the second file into memory ignoring all blank rows and ignoring any
non-title rows if possible.
I would save possible title rows into a collection you define to hold all
the data you need
Then you can search the list letting the code behind the collection do all
the searching for you. It should also run a lot faster.

--

Rod Gill
"GeorgeAtkins" <Ge**********@discussions.microsoft.com> wrote in message
news:83**********************************@microsof t.com...

Using VB.NET in VS 2003.
This should be a simple routine, but it has me flummoxed.
I have to compare strings in two text files:
FILE1(srar) consists of lines of book titles.
FILE (srac) consists of multi-line "book records", separated by blank
rows.
The number of rows in each "book record" varies, as in this diagram:

1xxxxxx
xxxxxx
TitleRow
xxxxxx

2xxxxxx
TitleRow
xxxxxx
xxxxxx
xxxxxx
All I'm trying to do is to first read a title in File1, then read through
the entire File2 file, one "book record" at a time, looking for a matching
title. After other processing (not relevant here) and all book records
have
been searched, go to the next File1 title and read all the book records in
File2 again. In short: typical looping within a loop.

The problem is that I cannot get it to work correctly! I know the issue
involves the blank rows and how the streamreader works. Here are core
excerpts of my code. Any enlightenment is greatly appreciated!

Dim RowCntr, ARBldr, ARCntr, MARCRowCntr, z, StartPos, TabPos As Long
Dim C1, C2, BkTitle, ArTitle, Arline, Acline, ArWtr, AcWtr, arbk As String
Dim srAR As System.IO.StreamReader = New StreamReader(FILE1.txt,
Encoding.GetEncoding(1252))
Dim srAC As System.IO.StreamReader = New StreamReader(FILE2.txt,
Encoding.GetEncoding(1252))

' Loop through the AR file (FILE1), reading each book title
For ARCntr = 0 To RdrArray.GetUpperBound(1)
MarcCntr = 1 ' initialize marc record counter variable
' Read through FILE2, one "book record" at a time.
Do
RowCntr = 1 ' reset variable for next book record row counter
' Now, the Inner loop supposed to read all lines for a single book
record.
Do
Acline = srAC.ReadLine
LibList.Add(Acline) ' new field row
If Acline="" Then ' found the blank row
LibList.Add(vbCr) ' new field row
RowCntr += 1 ' set value of RowCntr
End If
Loop Until srAC.Peek = -1
Loop Until srAC.Peek = -1 ' of first DO. Get another book record
Next ARCntr ' of the original FOR loop. Get another book title

I hope this is enough information to work with. I'll reply with more info
if
necessary.
Thanks again for any help!

George

[GeorgeAtkins]

Thanks for the reply, Rod.
Unfortunately, there are reasons I cannot do this (otherwise, it would have
been a no-brainer). And I probably didn't go into sufficient detail about
this, so I apologize:

1. These "book records" must be written back out to a text file to be
imported into a database; hence, removing blank rows, extracting only certain
rows, etc., will do no good.
2. As for the blank rows, they must remain, as they are the legal
"separators" between the records. Otherwise, the import process (which also
involves a file format change that we don't need to talk about here) will
fail. However, I have tried stopping the loop when it hits a blank row and
programmatically putting the blank row back in before it is written out to a
file.

And the reason I'm trying to read these records one at a time is that I'm
dealing with tens of thousands of them and I fear running out of memory if I
try to read them all at one time. Remember each "record" contains numerous
rows of data.

In fact, after I finish processing a single record, I clear the array and
read in another record to process.

Perhaps another way for me to ask t his question is this:

With the outside For loop, looking through the AR book title files, what is
the most effective way for me to run an inner loop to read through the book
records file, one "book record" of rows at a time?

If anything else comes to mind, don't hesitate to respond! Thanks for your
ideas and time, Rod!

George
"Rod Gill" wrote:

Hi,

Can't give you code (I'm learning vb.net myself!) but I would:

Read the second file into memory ignoring all blank rows and ignoring any
non-title rows if possible.
I would save possible title rows into a collection you define to hold all
the data you need
Then you can search the list letting the code behind the collection do all
the searching for you. It should also run a lot faster.

--

Rod Gill
"GeorgeAtkins" <Ge**********@discussions.microsoft.com> wrote in message
news:83**********************************@microsof t.com...

Using VB.NET in VS 2003.
This should be a simple routine, but it has me flummoxed.
I have to compare strings in two text files:
FILE1(srar) consists of lines of book titles.
FILE (srac) consists of multi-line "book records", separated by blank
rows.
The number of rows in each "book record" varies, as in this diagram:

1xxxxxx
xxxxxx
TitleRow
xxxxxx

2xxxxxx
TitleRow
xxxxxx
xxxxxx
xxxxxx
All I'm trying to do is to first read a title in File1, then read through
the entire File2 file, one "book record" at a time, looking for a matching
title. After other processing (not relevant here) and all book records
have
been searched, go to the next File1 title and read all the book records in
File2 again. In short: typical looping within a loop.

The problem is that I cannot get it to work correctly! I know the issue
involves the blank rows and how the streamreader works. Here are core
excerpts of my code. Any enlightenment is greatly appreciated!

Dim RowCntr, ARBldr, ARCntr, MARCRowCntr, z, StartPos, TabPos As Long
Dim C1, C2, BkTitle, ArTitle, Arline, Acline, ArWtr, AcWtr, arbk As String
Dim srAR As System.IO.StreamReader = New StreamReader(FILE1.txt,
Encoding.GetEncoding(1252))
Dim srAC As System.IO.StreamReader = New StreamReader(FILE2.txt,
Encoding.GetEncoding(1252))

' Loop through the AR file (FILE1), reading each book title
For ARCntr = 0 To RdrArray.GetUpperBound(1)
MarcCntr = 1 ' initialize marc record counter variable
' Read through FILE2, one "book record" at a time.
Do
RowCntr = 1 ' reset variable for next book record row counter
' Now, the Inner loop supposed to read all lines for a single book
record.
Do
Acline = srAC.ReadLine
LibList.Add(Acline) ' new field row
If Acline="" Then ' found the blank row
LibList.Add(vbCr) ' new field row
RowCntr += 1 ' set value of RowCntr
End If
Loop Until srAC.Peek = -1
Loop Until srAC.Peek = -1 ' of first DO. Get another book record
Next ARCntr ' of the original FOR loop. Get another book title

I hope this is enough information to work with. I'll reply with more info
if
necessary.
Thanks again for any help!

George

[Nick Malik [Microsoft]]

You logic is strange.

Best to do this as psuedo-code first. Then convert to the language. I also
suggest that you pull out a seperate function to get the group of records
from the second file and return only the book title.

Open File1
Read a record from File1
Do while not EOF(File1)
FoundIt = false
Open File 2
BookTitle = GetBookTitle(File2)
Do while not EOF(File2)
if (File1.Record == BookTitle) then
FoundIt = true
break out of inner loop
end if
BookTitle = GetBookTitle(File2)
Loop
if (FoundIt) then
' do your processing
end if
Close File 2
Read a record from File 1
Loop
The inner call: GetBookTitle is pretty simple at this point:
Function GetBookTitle(file File2) as string
GetBookTitle = blank ' make sure you return a valid value
read a line from File2
do wile line is not blank
save line to array or other object
if line is the title
GetBookTitle = line read
end if
read a line from File2
loop
End Function

Hopefully this will clear up the log jam.

BTW: if you are searching for items from the first set in the second set,
why not just load both of them into database tables and then do a simple
join? It's a LOT more efficient and would go much quicker!

You mention also that these files can get big. This algorithm, while it
will solve the immediate problem, will present you with a new one: it is not
efficient at all. You will reread the entire large file 2 for each record
in File 2. I don't know how many records are in File 1 or File 2, but if
you are afraid of running out of memory, I'd mention that this method will
have you running out of processor time and will put an amazing strain on the
garbage collector!

If you must do this kind of processing, is there any way you can presort
both files? If so, you won't have to read each file from the beginning.
You can read from file1 until you pass where the first record would occur
alphabetically. Then, if you don't fine record 1, you move forward to
record2 and read forward from there. each file is read once. This is
considerably more efficient. (It's how we used to do this in the old days,
where every cycle on a computer was counted and charged back to the person
running the job.

The best approach is still to load both table in a db and do a join.

HTH,
--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.
--
"GeorgeAtkins" <Ge**********@discussions.microsoft.com> wrote in message
news:83**********************************@microsof t.com...

Using VB.NET in VS 2003.
This should be a simple routine, but it has me flummoxed.
I have to compare strings in two text files:
FILE1(srar) consists of lines of book titles.
FILE (srac) consists of multi-line "book records", separated by blank
rows.
The number of rows in each "book record" varies, as in this diagram:

1xxxxxx
xxxxxx
TitleRow
xxxxxx

2xxxxxx
TitleRow
xxxxxx
xxxxxx
xxxxxx
All I'm trying to do is to first read a title in File1, then read through
the entire File2 file, one "book record" at a time, looking for a matching
title. After other processing (not relevant here) and all book records
have
been searched, go to the next File1 title and read all the book records in
File2 again. In short: typical looping within a loop.

The problem is that I cannot get it to work correctly! I know the issue
involves the blank rows and how the streamreader works. Here are core
excerpts of my code. Any enlightenment is greatly appreciated!

Dim RowCntr, ARBldr, ARCntr, MARCRowCntr, z, StartPos, TabPos As Long
Dim C1, C2, BkTitle, ArTitle, Arline, Acline, ArWtr, AcWtr, arbk As String
Dim srAR As System.IO.StreamReader = New StreamReader(FILE1.txt,
Encoding.GetEncoding(1252))
Dim srAC As System.IO.StreamReader = New StreamReader(FILE2.txt,
Encoding.GetEncoding(1252))

' Loop through the AR file (FILE1), reading each book title
For ARCntr = 0 To RdrArray.GetUpperBound(1)
MarcCntr = 1 ' initialize marc record counter variable
' Read through FILE2, one "book record" at a time.
Do
RowCntr = 1 ' reset variable for next book record row counter
' Now, the Inner loop supposed to read all lines for a single book
record.
Do
Acline = srAC.ReadLine
LibList.Add(Acline) ' new field row
If Acline="" Then ' found the blank row
LibList.Add(vbCr) ' new field row
RowCntr += 1 ' set value of RowCntr
End If
Loop Until srAC.Peek = -1
Loop Until srAC.Peek = -1 ' of first DO. Get another book record
Next ARCntr ' of the original FOR loop. Get another book title

I hope this is enough information to work with. I'll reply with more info
if
necessary.
Thanks again for any help!

George