How can you compute hash on large amount of data?

[Bobby Price]

I tried to follow this advice, but I still run out of memory.

http://bytes.com/topic/visual-basic-net/answers/363011-md5-large-files

Basically, I am reading all the bytes of a 1 gb disk and I want to try and do a md5 hash on it. However, when following this example, my cryptostream gets too large a System out of memory exception.

Is there another way to do the md5 hash over some much data?

I am currently reading it into 10 million bytes arrays at a time. Can I hash the first array, and then use it as a seed for the next array, and so one, like you might do to a CRC32 calculation?

Any help would be appreciated.

[Rabbit]

Can you post your code?

You won't be able to feed the result of one hash into another because of the preprocessing that is done for MD5 hashes. However, you can do the hash append yourself. You will not get an officially correct MD5 hash but as long as you hash using the same function, the essence is the same.

Also, I found this on google.

Imports System.Security.Cryptography
Imports System.Text
Imports System.IO
 
Module RijndaelSample
 
    Sub Main()
        Try
            ' Create a new Rijndael object to generate a key
            ' and initialization vector (IV).
            Dim RijndaelAlg As Rijndael = Rijndael.Create
 
            ' Create a string to encrypt.
            Dim sData As String = "Here is some data to encrypt."
            Dim FileName As String = "CText.txt"
 
            ' Encrypt text to a file using the file name, key, and IV.
            EncryptTextToFile(sData, FileName, RijndaelAlg.Key, RijndaelAlg.IV)
 
            ' Decrypt the text from a file using the file name, key, and IV.
            Dim Final As String = DecryptTextFromFile(FileName, RijndaelAlg.Key, RijndaelAlg.IV)
 
            ' Display the decrypted string to the console.
            Console.WriteLine(Final)
        Catch e As Exception
            Console.WriteLine(e.Message)
        End Try
 
        Console.ReadLine()
 
    End Sub
 
 
    Sub EncryptTextToFile(ByVal Data As String, ByVal FileName As String, ByVal Key() As Byte, ByVal IV() As Byte)
        Try
            ' Create or open the specified file.
            Dim fStream As FileStream = File.Open(FileName, FileMode.OpenOrCreate)
 
            ' Create a new Rijndael object.
            Dim RijndaelAlg As Rijndael = Rijndael.Create
 
            ' Create a CryptoStream using the FileStream 
            ' and the passed key and initialization vector (IV).
            Dim cStream As New CryptoStream(fStream, _
                                           RijndaelAlg.CreateEncryptor(Key, IV), _
                                           CryptoStreamMode.Write)
 
            ' Create a StreamWriter using the CryptoStream.
            Dim sWriter As New StreamWriter(cStream)
 
            Try
 
                ' Write the data to the stream 
                ' to encrypt it.
                sWriter.WriteLine(Data)
            Catch e As Exception
 
                Console.WriteLine("An error occurred: {0}", e.Message)
 
            Finally
 
                ' Close the streams and
                ' close the file.
                sWriter.Close()
                cStream.Close()
                fStream.Close()
 
            End Try
        Catch e As CryptographicException
            Console.WriteLine("A Cryptographic error occurred: {0}", e.Message)
        Catch e As UnauthorizedAccessException
            Console.WriteLine("A file error occurred: {0}", e.Message)
        End Try
    End Sub
 
 
    Function DecryptTextFromFile(ByVal FileName As String, ByVal Key() As Byte, ByVal IV() As Byte) As String
        Try
            ' Create or open the specified file. 
            Dim fStream As FileStream = File.Open(FileName, FileMode.OpenOrCreate)
 
            ' Create a new Rijndael object.
            Dim RijndaelAlg As Rijndael = Rijndael.Create
 
            ' Create a CryptoStream using the FileStream 
            ' and the passed key and initialization vector (IV).
            Dim cStream As New CryptoStream(fStream, _
                                            RijndaelAlg.CreateDecryptor(Key, IV), _
                                            CryptoStreamMode.Read)
 
            ' Create a StreamReader using the CryptoStream.
            Dim sReader As New StreamReader(cStream)
 
            ' Read the data from the stream 
            ' to decrypt it.
            Dim val As String = Nothing
 
            Try
 
                val = sReader.ReadLine()
 
            Catch e As Exception
                Console.WriteLine("An Cerror occurred: {0}", e.Message)
            Finally
                ' Close the streams and
                ' close the file.
                sReader.Close()
                cStream.Close()
                fStream.Close()
 
 
            End Try
 
            ' Return the string. 
            Return val
 
        Catch e As CryptographicException
            Console.WriteLine("A Cryptographic error occurred: {0}", e.Message)
            Return Nothing
        Catch e As UnauthorizedAccessException
            Console.WriteLine("A file error occurred: {0}", e.Message)
            Return Nothing
        End Try
    End Function
End Module

If push comes to shove and nothing else works, and you must have an official MD5 hash, you can implement the algorithm yourself.

[Bobby Price]

Thanks for your reply, but I think I found a better way of doing it. This is what I did if anyone is interested based on msdn.

This snippet assume 10 total large blocks of date to hash together. Readsize is number of elements in array, i.e. the length.

Using the following as a basis, I was able to computer a md5 hash over 1gb of data.


Dim md5Hash As MD5CryptoServiceProvider = New MD5CryptoServiceProvider

dim arrRead() as byte
dim arrHash() as byte

for i = 1 to 9
arrRead = ReadDataIntoDataBlock(i)
md5Hash.TransformBlock(arrRead, 0, ReadSize, arrRead, 0)
next i


'For the last, in this case, 10th block use
'TransformFinalBlock.

arrRead = ReadDataIntoDataBlock(10)
arrRead = md5Hash.TransformFinalBlock(arrRead, 0, ReadSize)
arrHash = md5Hash.Hash


'arrHash now contains 16 byte hash