By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,905 Members | 879 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,905 IT Pros & Developers. It's quick & easy.

calculate an unique id of a file

P: n/a
Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice
Oct 17 '08 #1
Share this Question
Share on Google+
6 Replies


P: n/a
ti*********@gmail.com pretended :
Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice
I don't think you can guarantee that you can identify file uniquely by
some hash code.
Say you calculate a hash of a single byte. This can hold 256 distinct
values, so by the time you encode file #257 you *will* have found a
duplicate hashcode. For a hash of a 32-bit integer, that amount is in
the billions while larger sizes go to astronomical numbers, but still
you cannot guarantee that there will be no duplicates.

As for speed, large files will require more time, as every byte has to
be read and processed.

Hans Kesting
Oct 17 '08 #2

P: n/a
Based only on the data or the filename too?

If you have the same filename, same size, and same hashcode it is likely it
is the same file. However, you don't say whether or not the filename is
important. If it isn't a factor then I would probably make the ID based on

FileSize/Hash1/Hash2

Where Hash1 and Hash2 are the results of two different hash algorithms.
Pete
====
http://mrpmorris.blogspot.com
http://www.capableobjects.com

Oct 17 '08 #3

P: n/a
On Fri, 17 Oct 2008 05:23:05 -0700 (PDT), ti*********@gmail.com wrote:
>Hi group,

I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?

Thanks for any advice
No hash function can guarantee uniqueness; a CRC32 will have a
collision probability of between 1 and 1 in 2^32. The 1 is for cases
where you should have used a cryptographically secure hash function,
i.e. where there is someone deliberately trying to break your system
or the Data Protection law requires a reasonable level of security.

For non-cryptographic use go for a CRC with sufficient size to reduce
the collision probability to an acceptably low level.

For cryptographic purposes use SHA-256 or SHA-512. The MD series are
broken and SHA-384 just calculates a SHA-512 result and truncates it
so you might as well go for SHA-512.

rossum

Oct 17 '08 #4

P: n/a
On 17 oct, 19:18, rossum <rossu...@coldmail.comwrote:
On Fri, 17 Oct 2008 05:23:05 -0700 (PDT), timor.su...@gmail.com wrote:
Hi group,
I need a way of calculating an unique id for a file.
I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions
What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?
Thanks for any advice

No hash function can guarantee uniqueness; a CRC32 will have a
collision probability of between 1 and 1 in 2^32. *The 1 is for cases
where you should have used a cryptographically secure hash function,
i.e. where there is someone deliberately trying to break your system
or the Data Protection law requires a reasonable level of security.

For non-cryptographic use go for a CRC with sufficient size to reduce
the collision probability to an acceptably low level.

For cryptographic purposes use SHA-256 or SHA-512. *The MD series are
broken and SHA-384 just calculates a SHA-512 result and truncates it
so you might as well go for SHA-512.

rossum
Thanks for your answer,
I think I don't need SHA.

In fact, I have to know if a file has been modified between to access.
Then, if I use a crc64, it should be enough to know that file has been
modified. Ins't it ?

Do you think this class is good ? http://damieng.com/blog/2007/11/19/c...4-in-c-and-net

Best regards
Oct 17 '08 #5

P: n/a
On Fri, 17 Oct 2008 13:39:16 -0700, <ti*********@gmail.comwrote:
[...]
In fact, I have to know if a file has been modified between to access.
Then, if I use a crc64, it should be enough to know that file has been
modified. Ins't it ?
That depends on what your criteria is. If the CRC is different, then
yes...you know for sure that the file has been modified. But it's
possible for the CRC to be the same even though the file has changed. If
it's the same, there is a small but non-zero possibility that the file has
been changed but in a way that maps to the same CRC.

If you need to know with 100% certainty whether the file is changed, then
you have to keep a copy of it and compare byte-by-byte.

For many applications, a CRC plus a feature to allow the user to force
notification of a change is sufficient. Though, for that matter, for many
applications simply using the "Modified" timestamp provided by the OS is
sufficient. It really depends on your reliability requirements.

Pete
Oct 17 '08 #6

P: n/a
ti*********@gmail.com wrote:
I need a way of calculating an unique id for a file.

I've seen things like Crc32, 64, checksum .... there's a list here :
http://en.wikipedia.org/wiki/List_of_hash_functions

What is the best option for me ? I have to identify larges files and
small files. I need something fast if possible. How can I be sure that
it is unique ?
Any CRC or Hash should do.

Hash will have less probability of collisions than CRC.

If you need to guarantee no collisions, then you can not use
any of them.

If you can live with a small probability of collisions, then
everything is possible.

The important questions is then whether you need to worry about
files with identical start but different end. Because if not then
you could speed up the process a lot by only checksumming
up to like 100 KB of data.

Arne
Oct 18 '08 #7

This discussion thread is closed

Replies have been disabled for this discussion.