By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
431,990 Members | 1,741 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 431,990 IT Pros & Developers. It's quick & easy.

Quick way to transfer huge data from one file to another.

P: n/a
Hi All,

I want to read data from one file and write that into another file.

Which is a way to do it quickly in terms of processing time?
Data to be transfer is huge one. E.g. 2147483458 bytes.
So I can't use array as temporary data type.
Since looping takes too much time, Is there any way to directly use
StreamReader and StreamWriter in conjunction.

Any help will be appreciated.

Thanks and Regards
Sakharam Phapale
Nov 21 '05 #1
Share this Question
Share on Google+
9 Replies


P: n/a

"Sakharam Phapale" <sp******@annetsite.com> wrote

I want to read data from one file and write that into another file.
Is the Copy method in the File class out of the question?

Since looping takes too much time, Is there any way to directly use
StreamReader and StreamWriter in conjunction.


To do what? Do you need to process the data? Then yeah, use
the streams. If not, then copy it outright....

LFS
Nov 21 '05 #2

P: n/a
Sakharam,

I can not understand direct from, your message if you mean a complete file
to a new or a part.

In the first situation, than why not just filecopy

http://msdn.microsoft.com/library/de...ctfilecopy.asp

I hope this helps?

Cor

"Sakharam Phapale" <sp******@annetsite.com>
I want to read data from one file and write that into another file.

Which is a way to do it quickly in terms of processing time?
Data to be transfer is huge one. E.g. 2147483458 bytes.
So I can't use array as temporary data type.
Since looping takes too much time, Is there any way to directly use
StreamReader and StreamWriter in conjunction.

Any help will be appreciated.

Thanks and Regards
Sakharam Phapale

Nov 21 '05 #3

P: n/a
Hi Cor,

I want to transfer given part of file.
From byte no to byte no.

Thanks and Regards
Sakharam Phapale
"Cor Ligthert" <no************@planet.nl> wrote in message
news:%2****************@TK2MSFTNGP11.phx.gbl...
Sakharam,

I can not understand direct from, your message if you mean a complete file
to a new or a part.

In the first situation, than why not just filecopy

http://msdn.microsoft.com/library/de...ctfilecopy.asp
I hope this helps?

Cor

"Sakharam Phapale" <sp******@annetsite.com>
I want to read data from one file and write that into another file.

Which is a way to do it quickly in terms of processing time?
Data to be transfer is huge one. E.g. 2147483458 bytes.
So I can't use array as temporary data type.
Since looping takes too much time, Is there any way to directly use
StreamReader and StreamWriter in conjunction.

Any help will be appreciated.

Thanks and Regards
Sakharam Phapale


Nov 21 '05 #4

P: n/a
Sakharam,

When you do not use a kind of database (even a very classic one which
adresses bytes direct on disk) you and it is something you can append. Than
the only thing (as Larry already wrote I saw now) is using the streamreader
and the streamwriter (There are more methods which do the same, however see
that yourself on this page, all this pages have direct links to other
methods)

http://msdn.microsoft.com/library/de...classtopic.asp

That can by reading first everything in memory or doing it line by line,
both methods can be used with the classes above.

I hope this helps?

Cor
"Sakharam Phapale" <sp******@annetsite.com>
...
Hi Cor,

I want to transfer given part of file.
From byte no to byte no.

Thanks and Regards
Sakharam Phapale
"Cor Ligthert" <no************@planet.nl> wrote in message
news:%2****************@TK2MSFTNGP11.phx.gbl...
Sakharam,

I can not understand direct from, your message if you mean a complete
file
to a new or a part.

In the first situation, than why not just filecopy

http://msdn.microsoft.com/library/de...ctfilecopy.asp

I hope this helps?

Cor

"Sakharam Phapale" <sp******@annetsite.com>
> I want to read data from one file and write that into another file.
>
> Which is a way to do it quickly in terms of processing time?
>
>
> Data to be transfer is huge one. E.g. 2147483458 bytes.
> So I can't use array as temporary data type.
> Since looping takes too much time, Is there any way to directly use
> StreamReader and StreamWriter in conjunction.
>
> Any help will be appreciated.
>
> Thanks and Regards
> Sakharam Phapale
>
>



Nov 21 '05 #5

P: n/a

Hi Cor,

First of all, thanks for reply.

I have used following two ways to transfer part of data from one
file to another.
If I used first option, performance is very slow.

Since, I am using big files, If I use second option i.e. using
array, I get error of out of memory because size of data is very big.

So, How can I improve the spped of transfering data?
Is there any other way around, or any suggetions then please let
me know.

Thanks and Regards
Sakharam Phapale

1) Using for Loop

Dim fsReader As FileStream
Dim br As BinaryReader
Dim fsWriter As FileStream
Dim bw As BinaryWriter

fsReader = New FileStream(TxaFileName, FileMode.Open,
FileAccess.Read)
br = New BinaryReader(fsReader)

fsWriter = New FileStream(m_strTxtFile, FileMode.CreateNew,
FileAccess.Write)
bw = New BinaryWriter(fsWriter)

Dim myByte As Byte

br.BaseStream.Seek(lngFromByte, SeekOrigin.Begin)
bw.BaseStream.Seek(0, SeekOrigin.Begin)

Do While br.BaseStream.Position <> lngFileLen
Try
myByte = br.ReadByte
Catch
Exit Do
End Try
bw.Write(value:=myByte)
Loop

bw.Close()
fsWrite.Close()
br.Close()
fsRead.Close()
2) Using buffer (array)
Dim arrCopiedWaveData(lngDataSize - 1)

Dim fsRead As New FileStream(TxaFileName, FileMode.Open,,
FileAccess.Read)
Dim br As New BinaryReader(fsRead)

br.BaseStream.Seek(lngFromByte, SeekOrigin.Begin)
br.Read(arrCopiedWaveData, 0, arrCopiedWaveData.Length)

bw.BaseStream.Seek(0, SeekOrigin.Begin)
bw.Write(arrCopiedWaveData, 0, arrCopiedWaveData.Length)

bw.Close()
fsWrite.Close()
br.Close()
fsRead.Close()


"Cor Ligthert" <no************@planet.nl> wrote in message
news:ei**************@TK2MSFTNGP10.phx.gbl...
Sakharam,

When you do not use a kind of database (even a very classic one which
adresses bytes direct on disk) you and it is something you can append. Than the only thing (as Larry already wrote I saw now) is using the streamreader and the streamwriter (There are more methods which do the same, however see that yourself on this page, all this pages have direct links to other
methods)

http://msdn.microsoft.com/library/de...classtopic.asp
That can by reading first everything in memory or doing it line by line,
both methods can be used with the classes above.

I hope this helps?

Cor
"Sakharam Phapale" <sp******@annetsite.com>
..
Hi Cor,

I want to transfer given part of file.
From byte no to byte no.

Thanks and Regards
Sakharam Phapale
"Cor Ligthert" <no************@planet.nl> wrote in message
news:%2****************@TK2MSFTNGP11.phx.gbl...
Sakharam,

I can not understand direct from, your message if you mean a complete
file
to a new or a part.

In the first situation, than why not just filecopy

http://msdn.microsoft.com/library/de...ctfilecopy.asp

I hope this helps?

Cor

"Sakharam Phapale" <sp******@annetsite.com>

> I want to read data from one file and write that into another file.
>
> Which is a way to do it quickly in terms of processing time?
>
>
> Data to be transfer is huge one. E.g. 2147483458 bytes.
> So I can't use array as temporary data type.
> Since looping takes too much time, Is there any way to directly use
> StreamReader and StreamWriter in conjunction.
>
> Any help will be appreciated.
>
> Thanks and Regards
> Sakharam Phapale
>
>



Nov 21 '05 #6

P: n/a

"Sakharam Phapale" <sp******@annetsite.com> wrote
So, How can I improve the spped of transfering data?
Is there any other way around, or any suggetions then please let me know. Dim myByte As Byte

Do While br.BaseStream.Position <> lngFileLen
Try
myByte = br.ReadByte
Catch
Exit Do
End Try
bw.Write(value:=myByte)
Loop


Instead of reading 1 byte at a time, use ReadBytes to read 4096 or 8192 bytes
at a time. (Using multiples of disk cluster size increases performance over other
odd amounts (like 1000, or 5000 etc...))

Pseudo code:

' Open streams ...
' Seek to position ...

size = 8192
Try
Do While lngFileLen > 0
If lngFileLen < size then size = lngFileLen
bw.Write(br.ReadBytes(size))
lngFileLen -= size
Loop
Catch
' Report error ...
Finally
' Close streams...
End Try
Nov 21 '05 #7

P: n/a
Thanks Larry, Cor for your kind help.

Regards
Sakharam Phapale
"Larry Serflaten" <se*******@usinternet.com> wrote in message
news:%2****************@TK2MSFTNGP15.phx.gbl...

"Sakharam Phapale" <sp******@annetsite.com> wrote
So, How can I improve the spped of transfering data?
Is there any other way around, or any suggetions then please
let me know.
Dim myByte As Byte

Do While br.BaseStream.Position <> lngFileLen
Try
myByte = br.ReadByte
Catch
Exit Do
End Try
bw.Write(value:=myByte)
Loop
Instead of reading 1 byte at a time, use ReadBytes to read 4096 or 8192

bytes at a time. (Using multiples of disk cluster size increases performance over other odd amounts (like 1000, or 5000 etc...))

Pseudo code:

' Open streams ...
' Seek to position ...

size = 8192
Try
Do While lngFileLen > 0
If lngFileLen < size then size = lngFileLen
bw.Write(br.ReadBytes(size))
lngFileLen -= size
Loop
Catch
' Report error ...
Finally
' Close streams...
End Try

Nov 21 '05 #8

P: n/a
Sakharam,
In addition to the other comments about using a buffer to copy chunks of the
file at a time.

You may want to review the asynchronous IO sample from MSDN.

http://msdn.microsoft.com/library/de...nousfileio.asp

As asynchronous IO can, in some cases, be quicker then normal synchronous
IO.

Hope this helps
Jay

"Sakharam Phapale" <sp******@annetsite.com> wrote in message
news:%2****************@TK2MSFTNGP11.phx.gbl...
Hi All,

I want to read data from one file and write that into another file.

Which is a way to do it quickly in terms of processing time?
Data to be transfer is huge one. E.g. 2147483458 bytes.
So I can't use array as temporary data type.
Since looping takes too much time, Is there any way to directly use
StreamReader and StreamWriter in conjunction.

Any help will be appreciated.

Thanks and Regards
Sakharam Phapale

Nov 21 '05 #9

P: n/a
On 2004-11-26, Sakharam Phapale <sp******@annetsite.com> wrote:
Hi Cor,

I want to transfer given part of file.
From byte no to byte no.

Thanks and Regards
Sakharam Phapale


Out of curiosity... Are you saying that you want to transfer a chunk of
the file with a given byte postion to a given byte position? If so,
are these positions fixed, or known before hand?

If so, you could simply seek to the start position, read the chunk, and
write it to the new file. Of course, I may be totally misunderstanding
your requirements :)

--
Tom Shelton [MVP]
Nov 21 '05 #10

This discussion thread is closed

Replies have been disabled for this discussion.