Split large text file by number of lines?

[ivan.perak]

Hello,

im a beginner in VB.NET... The thing i would like to do is as it
follows....

I have a text file (list of names, every name to the next line) which
is about 350000 lines long. I would like to split it and create a new
file at every lets say 20000 lines... so, the directory output would
have to be something like this:

File1: 1-20000 lines of the original file
File2: 20001-40000 lines of the original file
File3: 40001-60000 lines of the original file

etc.

Can it be done simply? one form with field to enter the number of
lines, button to load a text file and a "Start" button...

Thanks in advance

[Stephany Young]

Yes.

Read the source file line by line

Write each line to the target file

After each nth line, close the target file and open a new one (with a
different name of course).
<iv********@gma il.comwrote in message
news:11******** **************@ a75g2000cwd.goo glegroups.com.. .

Hello,

im a beginner in VB.NET... The thing i would like to do is as it
follows....

I have a text file (list of names, every name to the next line) which
is about 350000 lines long. I would like to split it and create a new
file at every lets say 20000 lines... so, the directory output would
have to be something like this:

File1: 1-20000 lines of the original file
File2: 20001-40000 lines of the original file
File3: 40001-60000 lines of the original file

etc.

Can it be done simply? one form with field to enter the number of
lines, button to load a text file and a "Start" button...

Thanks in advance

[Michael M.]

This code I have writen works but it takes some time to complete(about 50
seconds for a 1 mb text file)

Mabye beter to do a "readall" and then use the SPLIT(str, vbcrlf) function
anyway this should do it

add a textbox and a button. This is created in vb.net 2005 (the free
version from microsoft)

Private Sub Button1_Click(B yVal sender As System.Object, ByVal e As
System.EventArg s) Handles Button1.Click

TextSplitter()

End Sub
Sub TextSplitter()

' open the source fle and read assign it to a stream

Dim AsciiStreamRead er As IO.StreamReader =
IO.File.OpenTex t("C:\HugeSourc eTextFile1.txt" )

Dim sb As New Text.StringBuil der

Dim LineCounter As Integer = 0

Dim FileNumber As Integer = 1

Dim bProcessWinMsg As Boolean = 0

Me.Text = "processng file... "

While AsciiStreamRead er.EndOfStream = False

bProcessWinMsg += 1

If bProcessWinMsg Then Application.DoE vents()

sb.Append(Ascii StreamReader.Re adLine() & vbCrLf)

If LineCounter = CInt(TextBox1.T ext) Or AsciiStreamRead er.EndOfStream = True
Then

' Writes the data stored in the stringBuiler(sb ) and then closes the file

IO.File.WriteAl lText("C:\" & "File " & FileNumber & ".txt", sb.ToString,
Encoding.ASCII)

' Reset the line count, clear the sb string and increment the file number

LineCounter = 0

sb.Length = 0

FileNumber += 1

End If

LineCounter += 1

End While

Me.Text = "Complete: created " & FileNumber & " files"

End Sub

"Stephany Young" <noone@localhos twrote in message
news:%2******** ********@TK2MSF TNGP05.phx.gbl. ..

Yes.

Read the source file line by line

Write each line to the target file

After each nth line, close the target file and open a new one (with a
different name of course).
<iv********@gma il.comwrote in message
news:11******** **************@ a75g2000cwd.goo glegroups.com.. .

>Hello,

im a beginner in VB.NET... The thing i would like to do is as it
follows....

I have a text file (list of names, every name to the next line) which
is about 350000 lines long. I would like to split it and create a new
file at every lets say 20000 lines... so, the directory output would
have to be something like this:

File1: 1-20000 lines of the original file
File2: 20001-40000 lines of the original file
File3: 40001-60000 lines of the original file

etc.

Can it be done simply? one form with field to enter the number of
lines, button to load a text file and a "Start" button...

Thanks in advance

[Armin Zingler]

"Michael M." <no****@mike.co mschrieb

This code I have writen works but it takes some time to
complete(about 50 seconds for a 1 mb text file)

Mabye beter to do a "readall" and then use the SPLIT(str, vbcrlf)
function anyway this should do it

add a textbox and a button. This is created in vb.net 2005 (the
free version from microsoft)

Suggestion (untested):

Sub TextSplitter()

Dim fsIN, fsOut As IO.FileStream
Dim sr As IO.StreamReader
Dim sw As IO.StreamWriter
Dim OutCount As Integer

fsIN = New IO.FileStream( _
"infile.txt ", IO.FileMode.Ope n, IO.FileAccess.R ead _
)

sr = New IO.StreamReader (fsIN, System.Text.Enc oding.Default)

Do
Dim Line As String
Dim LineCount As Integer

Line = sr.ReadLine()
If Line Is Nothing Then Exit Do

If fsOut Is Nothing Then
OutCount += 1

fsOut = New IO.FileStream( _
"outfile" & OutCount & ".txt", _
IO.FileMode.Cre ateNew, IO.FileAccess.W rite _
)

sw = New IO.StreamWriter (fsOut, System.Text.Enc oding.Default)
LineCount = 0
End If

sw.WriteLine(Li ne)
LineCount += 1

If LineCount = 20000 Then
sw.Close()
fsOut = Nothing
End If
Loop

If fsOut IsNot Nothing Then
sw.Close()
End If

fsIN.Close()

End Sub
Be aware that Encoding.Ascii supports only 7 bit characters.
Armin

[Tom Leylan]

I'm going to opt for an OOP solution which isn't quite so dependent upon all
the inputs being fixed. Personally I've learned that "specs change" and
planning for change saves the client money which makes for a happy client.

So try the other solutions out and then try this one. Do note that you can
change the input file name, set the output file names, set the line count
(indpendently per file) and it can produce more (or fewer) than 3 files by
calling the Copy() method as many times as you want.

Personally I'd add some error handling before I tried to sell it and I might
add a methodology to indicate the end of the input file was reached. While
it should cause no harm it seems pointless to keep calling Copy() if the
end-of-file was already reached.

The reason there is only 30 lines indicated in my example is that's all I
wanted to type into my test.

Tom

Dim oCopier As Copier = New Copier()

With oCopier
.Open("infile.t xt")
.Copy("file1.tx t", 10)
.Copy("file2.tx t", 10)
.Copy("file3.tx t", 10)
.Close()
End With
Public Class Copier
Inherits Object

Private fs As IO.FileStream
Private sr As IO.StreamReader

Public Sub Open(ByVal file As String)
fs = New IO.FileStream(f ile, IO.FileMode.Ope n, IO.FileAccess.R ead)
sr = New IO.StreamReader (fs, System.Text.Enc oding.Default)
End Sub

Public Sub Close()
sr.Close()
End Sub

Public Sub Copy(ByVal file As String, ByVal max As Int32)

Dim fs As IO.FileStream = New IO.FileStream(f ile,
IO.FileMode.Cre ateNew, IO.FileAccess.W rite)
Dim sw As IO.StreamWriter = New IO.StreamWriter (fs,
System.Text.Enc oding.Default)

Dim input As String
Dim count As Int32 = 0

Dim processing As Boolean = True
While processing

input = sr.ReadLine()
count += 1

processing = ((input IsNot Nothing) AndAlso count < max)

If processing Then
sw.WriteLine(in put)
End If

End While

sw.Close()

End Sub

End Class


<iv********@gma il.comwrote in message
news:11******** **************@ a75g2000cwd.goo glegroups.com.. .

Hello,

im a beginner in VB.NET... The thing i would like to do is as it
follows....

I have a text file (list of names, every name to the next line) which
is about 350000 lines long. I would like to split it and create a new
file at every lets say 20000 lines... so, the directory output would
have to be something like this:

File1: 1-20000 lines of the original file
File2: 20001-40000 lines of the original file
File3: 40001-60000 lines of the original file

etc.

Can it be done simply? one form with field to enter the number of
lines, button to load a text file and a "Start" button...

Thanks in advance

[PFC Sadr]

you know they make partitioning in databases right?

keep everything in one table and then you can use rank; or you can
filter, search-- anything you want to do.

and it doesn't matter if you have 20k records or 4m...

I mean seriously; why reinvent the wheel? Did I mention why reinvent
the wheel?

On Feb 21, 1:58 pm, ivan.pe...@gmai l.com wrote:

Hello,

im a beginner in VB.NET... The thing i would like to do is as it
follows....

I have a text file (list of names, every name to the next line) which
is about 350000 lines long. I would like to split it and create a new
file at every lets say 20000 lines... so, the directory output would
have to be something like this:

File1: 1-20000 lines of the original file
File2: 20001-40000 lines of the original file
File3: 40001-60000 lines of the original file

etc.

Can it be done simply? one form with field to enter the number of
lines, button to load a text file and a "Start" button...

Thanks in advance

[peraklo]

Armin Zingler je napisao/la:

"Michael M." <no****@mike.co mschrieb

This code I have writen works but it takes some time to
complete(about 50 seconds for a 1 mb text file)

Mabye beter to do a "readall" and then use the SPLIT(str, vbcrlf)
function anyway this should do it

add a textbox and a button. This is created in vb.net 2005 (the
free version from microsoft)

Suggestion (untested):

Sub TextSplitter()

Dim fsIN, fsOut As IO.FileStream
Dim sr As IO.StreamReader
Dim sw As IO.StreamWriter
Dim OutCount As Integer

fsIN = New IO.FileStream( _
"infile.txt ", IO.FileMode.Ope n, IO.FileAccess.R ead _
)

sr = New IO.StreamReader (fsIN, System.Text.Enc oding.Default)

Do
Dim Line As String
Dim LineCount As Integer

Line = sr.ReadLine()
If Line Is Nothing Then Exit Do

If fsOut Is Nothing Then
OutCount += 1

fsOut = New IO.FileStream( _
"outfile" & OutCount & ".txt", _
IO.FileMode.Cre ateNew, IO.FileAccess.W rite _
)

sw = New IO.StreamWriter (fsOut, System.Text.Enc oding.Default)
LineCount = 0
End If

sw.WriteLine(Li ne)
LineCount += 1

If LineCount = 20000 Then
sw.Close()
fsOut = Nothing
End If
Loop

If fsOut IsNot Nothing Then
sw.Close()
End If

fsIN.Close()

End Sub
Be aware that Encoding.Ascii supports only 7 bit characters.
Armin

thanks man, this code does exactly what i need, and pretty fast....