How to sort very large arrays?

[kj]

I'm downloading some very large tables from a remote site. I want
to sort these tables in a particular way before saving them to
disk. In the past I found that the most efficient way to do this
was to piggy-back on Unix's highly optimized sort command. So,
from within a Perl script, I'd create a pipe handle through sort
and then just print the data through that handle:

open my $out, "|$sort -t '\t' -k1,1 -k2,2 -u $out_file" or die $!;
print $out $_ for @data;

But that's distinctly Perlish, and I'm wondering what's the "Python
Way" to do this.

TIA!

kynn

--
NOTE: In my address everything before the first period is backwards;
and the last period, and everything after it, should be discarded.

[Dan Stromberg]

On Fri, 13 Jun 2008 17:54:32 +0000, kj wrote:

I'm downloading some very large tables from a remote site. I want to
sort these tables in a particular way before saving them to disk. In
the past I found that the most efficient way to do this was to
piggy-back on Unix's highly optimized sort command. So, from within a
Perl script, I'd create a pipe handle through sort and then just print
the data through that handle:

open my $out, "|$sort -t '\t' -k1,1 -k2,2 -u $out_file" or die $!;
print $out $_ for @data;

But that's distinctly Perlish, and I'm wondering what's the "Python Way"
to do this.

TIA!

kynn

os.system and os.popen are much like what you'd find in C.

The subprocess module is more specific to python, and is a little more
complicated but more powerful.

[Terry Reedy]

"kj" <so***@987jk.com.invalidwrote in message
news:g2**********@reader2.panix.com...
| I'm downloading some very large tables from a remote site. I want
| to sort these tables in a particular way before saving them to
| disk. In the past I found that the most efficient way to do this
| was to piggy-back on Unix's highly optimized sort command. So,

If the tables can fit in memory as a list of key,text tuples and if they
have some of the non-random structure exploited by Python's current
list.sort (only documented, as far as I know, either in the source or test
code, not sure), then you might consider that. Otherwise, use the system
sort.

[rent]

On Jun 14, 1:54 am, kj <[e-mail]:

I'm downloading some very large tables from a remote site. I want
to sort these tables in a particular way before saving them to
disk. In the past I found that the most efficient way to do this
was to piggy-back on Unix's highly optimized sort command. So,
from within a Perl script, I'd create a pipe handle through sort
and then just print the data through that handle:

This is a python clone of your code from a python rookie :)

from os import popen

p = popen("sort -t '\t' -k1,1 -k2,2 -u %s" % out_file)
for line in data:
print >p, line

there is no "die $!" here, I think it is good to let python
throw the exception to your console

>
open my $out, "|$sort -t '\t' -k1,1 -k2,2 -u $out_file" or die $!;
print $out $_ for @data;

But that's distinctly Perlish, and I'm wondering what's the "Python
Way" to do this.

TIA!

kynn

--
NOTE: In my address everything before the first period is backwards;
and the last period, and everything after it, should be discarded.

[rent]

On Jun 14, 1:54 am, kj <[e-mail]:

I'm downloading some very large tables from a remote site. I want
to sort these tables in a particular way before saving them to
disk. In the past I found that the most efficient way to do this
was to piggy-back on Unix's highly optimized sort command. So,
from within a Perl script, I'd create a pipe handle through sort
and then just print the data through that handle:

This is a python clone of your code from a python rookie :)

from os import popen

p = popen("sort -t '\t' -k1,1 -k2,2 -u %s" % out_file)
for line in data:
print >p, line

there is no "die $!" here, I think it is good to let python
throw the exception to your console

>
open my $out, "|$sort -t '\t' -k1,1 -k2,2 -u $out_file" or die $!;
print $out $_ for @data;

But that's distinctly Perlish, and I'm wondering what's the "Python
Way" to do this.

TIA!

kynn

--
NOTE: In my address everything before the first period is backwards;
and the last period, and everything after it, should be discarded.