I tried to follow this advice, but I still run out of memory.
http://bytes.com/topic/visual-basic-net/answers/363011-md5-large-files
Basically, I am reading all the bytes of a 1 gb disk and I want to try and do a md5 hash on it. However, when following this example, my cryptostream gets too large a System out of memory exception.
Is there another way to do the md5 hash over some much data?
I am currently reading it into 10 million bytes arrays at a time. Can I hash the first array, and then use it as a seed for the next array, and so one, like you might do to a CRC32 calculation?
Any help would be appreciated.
2 2925 Rabbit 12,516
Recognized Expert Moderator MVP
Can you post your code?
You won't be able to feed the result of one hash into another because of the preprocessing that is done for MD5 hashes. However, you can do the hash append yourself. You will not get an officially correct MD5 hash but as long as you hash using the same function, the essence is the same.
Also, I found this on google. - Imports System.Security.Cryptography
-
Imports System.Text
-
Imports System.IO
-
-
Module RijndaelSample
-
-
Sub Main()
-
Try
-
' Create a new Rijndael object to generate a key
-
' and initialization vector (IV).
-
Dim RijndaelAlg As Rijndael = Rijndael.Create
-
-
' Create a string to encrypt.
-
Dim sData As String = "Here is some data to encrypt."
-
Dim FileName As String = "CText.txt"
-
-
' Encrypt text to a file using the file name, key, and IV.
-
EncryptTextToFile(sData, FileName, RijndaelAlg.Key, RijndaelAlg.IV)
-
-
' Decrypt the text from a file using the file name, key, and IV.
-
Dim Final As String = DecryptTextFromFile(FileName, RijndaelAlg.Key, RijndaelAlg.IV)
-
-
' Display the decrypted string to the console.
-
Console.WriteLine(Final)
-
Catch e As Exception
-
Console.WriteLine(e.Message)
-
End Try
-
-
Console.ReadLine()
-
-
End Sub
-
-
-
Sub EncryptTextToFile(ByVal Data As String, ByVal FileName As String, ByVal Key() As Byte, ByVal IV() As Byte)
-
Try
-
' Create or open the specified file.
-
Dim fStream As FileStream = File.Open(FileName, FileMode.OpenOrCreate)
-
-
' Create a new Rijndael object.
-
Dim RijndaelAlg As Rijndael = Rijndael.Create
-
-
' Create a CryptoStream using the FileStream
-
' and the passed key and initialization vector (IV).
-
Dim cStream As New CryptoStream(fStream, _
-
RijndaelAlg.CreateEncryptor(Key, IV), _
-
CryptoStreamMode.Write)
-
-
' Create a StreamWriter using the CryptoStream.
-
Dim sWriter As New StreamWriter(cStream)
-
-
Try
-
-
' Write the data to the stream
-
' to encrypt it.
-
sWriter.WriteLine(Data)
-
Catch e As Exception
-
-
Console.WriteLine("An error occurred: {0}", e.Message)
-
-
Finally
-
-
' Close the streams and
-
' close the file.
-
sWriter.Close()
-
cStream.Close()
-
fStream.Close()
-
-
End Try
-
Catch e As CryptographicException
-
Console.WriteLine("A Cryptographic error occurred: {0}", e.Message)
-
Catch e As UnauthorizedAccessException
-
Console.WriteLine("A file error occurred: {0}", e.Message)
-
End Try
-
End Sub
-
-
-
Function DecryptTextFromFile(ByVal FileName As String, ByVal Key() As Byte, ByVal IV() As Byte) As String
-
Try
-
' Create or open the specified file.
-
Dim fStream As FileStream = File.Open(FileName, FileMode.OpenOrCreate)
-
-
' Create a new Rijndael object.
-
Dim RijndaelAlg As Rijndael = Rijndael.Create
-
-
' Create a CryptoStream using the FileStream
-
' and the passed key and initialization vector (IV).
-
Dim cStream As New CryptoStream(fStream, _
-
RijndaelAlg.CreateDecryptor(Key, IV), _
-
CryptoStreamMode.Read)
-
-
' Create a StreamReader using the CryptoStream.
-
Dim sReader As New StreamReader(cStream)
-
-
' Read the data from the stream
-
' to decrypt it.
-
Dim val As String = Nothing
-
-
Try
-
-
val = sReader.ReadLine()
-
-
Catch e As Exception
-
Console.WriteLine("An Cerror occurred: {0}", e.Message)
-
Finally
-
' Close the streams and
-
' close the file.
-
sReader.Close()
-
cStream.Close()
-
fStream.Close()
-
-
-
End Try
-
-
' Return the string.
-
Return val
-
-
Catch e As CryptographicException
-
Console.WriteLine("A Cryptographic error occurred: {0}", e.Message)
-
Return Nothing
-
Catch e As UnauthorizedAccessException
-
Console.WriteLine("A file error occurred: {0}", e.Message)
-
Return Nothing
-
End Try
-
End Function
-
End Module
If push comes to shove and nothing else works, and you must have an official MD5 hash, you can implement the algorithm yourself.
Thanks for your reply, but I think I found a better way of doing it. This is what I did if anyone is interested based on msdn.
This snippet assume 10 total large blocks of date to hash together. Readsize is number of elements in array, i.e. the length.
Using the following as a basis, I was able to computer a md5 hash over 1gb of data.
Dim md5Hash As MD5CryptoServic eProvider = New MD5CryptoServic eProvider
dim arrRead() as byte
dim arrHash() as byte
for i = 1 to 9
arrRead = ReadDataIntoDat aBlock(i)
md5Hash.Transfo rmBlock(arrRead , 0, ReadSize, arrRead, 0)
next i
'For the last, in this case, 10th block use
'TransformFinal Block.
arrRead = ReadDataIntoDat aBlock(10)
arrRead = md5Hash.Transfo rmFinalBlock(ar rRead, 0, ReadSize)
arrHash = md5Hash.Hash
'arrHash now contains 16 byte hash
Sign in to post your reply or Sign up for a free account.
Similar topics |
by: Ice Man |
last post by:
Hi All,
I will need to send bewteen 2 asp pages very large amounts of data
what is the best way to send it and to read it?
for ex. I know this method:
For i = 1 To Request.QueryString.Count - 1
Response.write(Request.QueryString.Key(i))
Next
|
by: steve |
last post by:
Hi, I have researched but have not found a good solution to this
problem.
I am importing large amounts of data (over 50 Meg) into a new mysql db
that I set up. I use
>mysql dbname < importfile.txt
But I keep getting timeouts and errors due to the data being too
large. I know that since if I break the imported data into multiple
chuncks (by importing a few tables at a time) then everything works.
|
by: Mike |
last post by:
This is a general question on the best way to import a large amount of data
to a MS-SQL DB.
I can have the data in just about any format I need to, I just don't know
how to import the data. I some experience with SQL but not much.
There is about 1500 to 2000 lines of data. I am looking for the best way to
get this amount of data in on a monthly basis.
Any help is greatly thanked!!
|
by: Bart |
last post by:
Dear all,
I would like to encrypt a large amount of data by using public/private keys,
but I read on MSDN:
"Symmetric encryption is performed on streams and is therefore useful to
encrypt large amounts of data. Asymmetric encryption is performed on a small
number of bytes and is therefore only useful for small amounts of data."
There is not possibility to do it? I have tried to encrypt a 300kB file by
RSA Algorithm, but I received...
|
by: loretta |
last post by:
I have data within an xml tag that is being truncated when being read
into a javascript variable. On Firefox, I am only getting up to 4096
characters. On IE, I am getting 31324 characters. I can view the xml
source, all the data is there. I am using javascript function
getElementsByTagName to read the data from the tag and then the
firstChild and nodeValue notations to get the data. I can't find any
references to xml size limits, but I am...
| |
by: eggie5 |
last post by:
OK, so I'm trying to get a large amount of text back to my ASP script
on the server. My large amount of text is the source to a web page, and
I want to preserve the formatting on it, ie the indenting. Before any
of this the only way I got stuff back to the server is with query
strings. Now I'm pretty sure there's other ways to get data back to the
sever besides query stings but I'm having trouble finding them. Can
somebody point me in the...
|
by: Pja |
last post by:
hi..
i need some help here..
how do i display large amount of data in vb6.0 in a table form that can be scrolled and to include print functions..
i am using vb6.0 as my front end and ms sql server 2000 as my back end..
any help would be appreciated..
thank you :)
|
by: Jack |
last post by:
I need to process large amount of data. The data structure fits well
in a dictionary but the amount is large - close to or more than the size
of physical memory. I wonder what will happen if I try to load the data
into a dictionary. Will Python use swap memory or will it fail?
Thanks.
|
by: AnishAbs |
last post by:
Hi ,
I have got a senario in which large amount of data should be copied from MS excel to MS SQL server. which is the best option to do so. Because when I use recordsets the process is very slow and it effects the performance. Any one please suggest me.
Thanks
Anish
|
by: Ale |
last post by:
Hi all,
how can I execute a large amount of DDL Queries on Access 2000?
I've a DDl/SQL Script file which defines a data structure with CREATE
TABLE, ALTER TABLE UPDATE TABLE, etc. etc.
I know how to execute this script on MySQL, MS Sql Server, but don't
know how execute it on access 2000 :(
Thank you all.
Bye
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
|
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |