Barry Margolin <barmar@alum.mit.edu> writes:[color=blue]
> In article <1137104386.676900.158730@g44g2000cwa.googlegroups .com>,
> "ulyses" <ulyses@autograf.pl> wrote:[color=green]
>> David Schwartz napisal(a):[color=darkred]
>> > "ulyses" <ulyses@autograf.pl> wrote in message
>> > news:1136824220.477573.98320@g44g2000cwa.googlegro ups.com...
>> >
>> > > Let's assume I have following file:
>> > >
>> > > 2938929384902491233.....
>> > > 923949919199191919112....
>> > >
>> > > File contains INTs only. What is more they are huge. For example first
>> > > row in file may contain integer which size is 50MB and the second 30MB.
>> > > Now we come to my problem. Is there possibility to swap this rows
>> > > without using system memory (preferably in Unix/Linux)? Is there any
>> > > function in C to do this?
>> >
>> > It's not clear what you mean by "without using system memory". Why
>> > wouldn't you want to use memory if this results in faster operation?
>> >[/color]
>>
>> I thouhg about something that would be some kind of low level function
>> that would modify pointers not real data, the same way as we do with
>> pointer to data in memory, e.g.:
>> int a*, b*, temp*;
>> ...
>> //the swap
>> temp = a;
>> a = b;
>> b = temp;
>>
>> And the data was swapped without moving it in memory. I asked if there
>> is such functionality that would enable me to do something like that
>> but with rows in file. Swap them without moving them on disk.[/color]
>
> No, because the information in files is not organized by lines, it's
> just sequences of bytes organized by disk blocks. If you wanted to
> rearrange the blocks it would theoretically be possible to update the
> block pointers in the inode. However, there's no API for this, so you
> would have to do it by writing a new ioctl or device driver, or by
> accessing the disk device directly (and you'd need to unmount the
> filesystem first, to avoid conflicts with the kernel's in-memory copies
> of inodes).
>
> But to rearrange lines you have to read the file into memory, search for
> the newline characters, then write the lines back out to the file.[/color]
If you want to rearrange the lines so you can read the rearranged file
with ordinary stdio calls, you'll need to physically re-write the file
(unless you can manage to do some nasty low-level file system stuff).
But if you want to be able to access the lines in some specified order
other than their physical order in the file, you can just create a
separate index. Do one pass over the file, creating an index of the
position of the start of each line (using ftell() or fgetpos()). Once
you have the index, you can use fseek() or fsetpos() to jump directly
to the beginning of any line you want.
Reading the file in reverse order is likely to be less efficient than
if you had physically reversed the file; the performance tradeoff
depends on how often you read it.
--
Keith Thompson (The_Other_Keith)
kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.
