calculate an unique id of a file

timor.super

Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice

Oct 17 '08 #1

Hans Kesting

ti*********@gmail.com pretended :

Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice

I don't think you can guarantee that you can identify file uniquely by
some hash code.
Say you calculate a hash of a single byte. This can hold 256 distinct
values, so by the time you encode file #257 you *will* have found a
duplicate hashcode. For a hash of a 32-bit integer, that amount is in
the billions while larger sizes go to astronomical numbers, but still
you cannot guarantee that there will be no duplicates.

As for speed, large files will require more time, as every byte has to
be read and processed.

Hans Kesting

Oct 17 '08 #2

Peter Morris

Based only on the data or the filename too?

If you have the same filename, same size, and same hashcode it is likely it
is the same file. However, you don't say whether or not the filename is
important. If it isn't a factor then I would probably make the ID based on

FileSize/Hash1/Hash2

Where Hash1 and Hash2 are the results of two different hash algorithms.
Pete
====
http://mrpmorris.blogspot.com
http://www.capableobjects.com

Oct 17 '08 #3

rossum

On Fri, 17 Oct 2008 05:23:05 -0700 (PDT), ti*********@gmail.com wrote:

>Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice

No hash function can guarantee uniqueness; a CRC32 will have a
collision probability of between 1 and 1 in 2^32. The 1 is for cases
where you should have used a cryptographically secure hash function,
i.e. where there is someone deliberately trying to break your system
or the Data Protection law requires a reasonable level of security.

For non-cryptographic use go for a CRC with sufficient size to reduce
the collision probability to an acceptably low level.

For cryptographic purposes use SHA-256 or SHA-512. The MD series are
broken and SHA-384 just calculates a SHA-512 result and truncates it
so you might as well go for SHA-512.

rossum

Oct 17 '08 #4

timor.super

On 17 oct, 19:18, rossum <rossu...@coldmail.comwrote:

On Fri, 17 Oct 2008 05:23:05 -0700 (PDT), timor.su...@gmail.com wrote:

Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice

No hash function can guarantee uniqueness; a CRC32 will have a
collision probability of between 1 and 1 in 2^32. *The 1 is for cases
where you should have used a cryptographically secure hash function,
i.e. where there is someone deliberately trying to break your system
or the Data Protection law requires a reasonable level of security.

For non-cryptographic use go for a CRC with sufficient size to reduce
the collision probability to an acceptably low level.

For cryptographic purposes use SHA-256 or SHA-512. *The MD series are
broken and SHA-384 just calculates a SHA-512 result and truncates it
so you might as well go for SHA-512.

rossum

Thanks for your answer,
I think I don't need SHA.

In fact, I have to know if a file has been modified between to access.
Then, if I use a crc64, it should be enough to know that file has been
modified. Ins't it ?

Do you think this class is good ? http://damieng.com/blog/2007/11/19/c...4-in-c-and-net

Best regards

Oct 17 '08 #5

Peter Duniho

On Fri, 17 Oct 2008 13:39:16 -0700, <ti*********@gmail.comwrote:

[...]
In fact, I have to know if a file has been modified between to access.
Then, if I use a crc64, it should be enough to know that file has been
modified. Ins't it ?

That depends on what your criteria is. If the CRC is different, then
yes...you know for sure that the file has been modified. But it's
possible for the CRC to be the same even though the file has changed. If
it's the same, there is a small but non-zero possibility that the file has
been changed but in a way that maps to the same CRC.

If you need to know with 100% certainty whether the file is changed, then
you have to keep a copy of it and compare byte-by-byte.

For many applications, a CRC plus a feature to allow the user to force
notification of a change is sufficient. Though, for that matter, for many
applications simply using the "Modified" timestamp provided by the OS is
sufficient. It really depends on your reliability requirements.

Pete

Oct 17 '08 #6

=?ISO-8859-1?Q?Arne_Vajh=F8j?=

ti*********@gmail.com wrote:

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Any CRC or Hash should do.

Hash will have less probability of collisions than CRC.

If you need to guarantee no collisions, then you can not use
any of them.

If you can live with a small probability of collisions, then
everything is possible.

The important questions is then whether you need to worry about
files with identical start but different end. Because if not then
you could speed up the process a lot by only checksumming
up to like 100 KB of data.

Arne

Oct 18 '08 #7