FileInputStream is too slow

[Nepomuk]

Hi!

I'm trying to transfer files and to do so must, of course, read them. Now, my reference File has a size of 4.240.821 Bytes and I need something between 9 and 14 seconds to simply read it and save the data to arrays.

When however I transfer the file via FTP, everything is done in about 1 second - reading, transferring and writing.

I've written the following code, to find out, how much time is needed with different sizes of arrays:

import java.io.IOException;
import java.io.FileInputStream;
import java.io.File;
 
public class FileReadSpeed {
    public static void main(String[] args) {
        int howMany = 8;
        int howOften = 5;
        long[][] results = new long[howMany][howOften];
        int size = 64;
 
        try
        {
            File file = new File("E:\\temp\\file.tar.gz");
 
            for(int i=8;i<8+howMany;i++)
            {
                size *= 2;
                for(int j=0; j<howOften; j++)
                {
                    System.gc();
                    FileInputStream fiStream = new FileInputStream(file);
                    long startTime = System.currentTimeMillis();
                    byte[] data = new byte[size];
                    long amount = file.length() / size;
                    if(file.length() % size > 0) amount++;
                    for(int k=0; k<amount; k++)
                    {
                        for(int l=0; l<data.length; l++)
                        {
                            if((k == amount-1) && (l == file.length() % size)) break;
                            data[l] = (byte) fiStream.read();
                        }
                    }
                    long finishTime = System.currentTimeMillis();
                    long timeTaken = finishTime - startTime;
                    results[i-8][j] = timeTaken;
                    System.out.println("Recorded (" + (i-8) + " | " + j + ") with size = " + size + " Bytes");
                }
            }
        }
        catch(IOException ioe)
        {
            System.err.println(ioe);
        }
 
        System.out.println();
 
        size = 64;
        for(int i=0; i<results.length; i++)
        {
            size *= 2;
            System.out.print(size + ":\t");
            for(int j=0; j<results[i].length; j++)
            {
                System.out.print(results[i][j] + "\t");
            }
            double tmp = 0.;
            for(int j=0; j<results[i].length; j++)
            {
                tmp += results[i][j];
            }
            System.out.println("-> " + tmp/results[0].length);
        }
    }
}
 

Does anyone know, if there are implementations of FileInputStream or similar, which are faster? (Maybe with reduced functionality?)

Greetings,
Nepomuk

[JosAH]

You're reading a single byte every time; that is dead slow; you can read entire
byte arrays in one sweep, use that instead (check the FileInputStream API).
A byte array size of, say, 4096 is a typical buffer size.

kind regards,

Jos

[Nepomuk]

You're reading a single byte every time; that is dead slow; you can read entire
byte arrays in one sweep, use that instead (check the FileInputStream API).
A byte array size of, say, 4096 is a typical buffer size.

kind regards,

Jos

OK, I've tried that, but somehow I've made a mistake and can't find it... it should be directly before my eyes I guess, but maybe someone can help.

Here's my changed for-loop:

for(int k=0; k<amount; k++)
{
    int offset = (int)(k*size);
    System.out.println("offset:\t" + offset + "\nsize:\t" + size);
    fiStream.read(data,offset,size);
}
 

I also changed the settings a little:

int howMany = 4;
int howOften = 5;
int size = 512;
 

And I get a IndexOutOfBoundsException in the line, which does the reading, when the output is as follows:

offset:    0
size:    1024
offset:    1024
size:    1024
java.lang.IndexOutOfBoundsException...
 

It seems to me, that the offset is wrong somehow, but I don't understand, what is wrong...

Greetings,
Nepomuk

[JosAH]

Just do a:

int nofBytesRead= fiStream.read(data);
 

and it will attempt to read as many bytes as possible in the 'data' byte array.
All in one sweep that is.

kind regards,

Jos

[Nepomuk]

Just do a:

int nofBytesRead= fiStream.read(data);
 

and it will attempt to read as many bytes as possible in the 'data' byte array.
All in one sweep that is.

kind regards,

Jos

The Problem with that is, that my program should be able to send files of any size and with a file, which is over 100MB big, that won't work (I have tried it).

So, what mistake did I make with the offset version?

Greetings,
Nepomuk

[Nepomuk]

OK, I've found the error - I thought offset was the offset for the file, not for the array. It does work much faster now.

Thank you very much!

Greetings,
Nepomuk

[JosAH]

OK, I've found the error - I thought offset was the offset for the file, not for the array. It does work much faster now.

Thank you very much!

Greetings,
Nepomuk

Told you so ;-) Simply create a byte array of 64K or so, keep on reading into that
array, write the array until the read() call returns -1. The previous read might
return a value less than the size of the array so you should always use that
value.

kind regards,

Jos