[runes]

You have sets in Python 2.3x and 2.4x. I don't know if they can handle

your amounts of data, but i guess so.

The original poster (OP) spoke about 10**13 numbers, if I remember

correctly (I did not keep the original message). It all depends on the

real sparsity of the spare sets! :-) If not sparse enough, this might be

a bit inordinate for standard Python sets on most machines, also given

that each Python integer uses a lot more than 4 bytes...

On the other hand, the OP also wrote that the numbers he handles are

grouped in big and compact intervals, separated by big unpopulated

holes. So maybe his data could be represented in Python as a

user-written type hiding a sorted list of intervals, for which it should

be relatively easy to write set functions. That might make the problem

tractable enough, with reasonably speedy processing.

As last resort, the wanted sets could be represented as sorted files of

numbers. Slower, but set operations would also be easy to write.

--

François Pinard

http://pinard.progiciels-bpi.ca