Reading Binary Image Data

[eddiefisher41]

Hey Guys.
I'm looking for some info on how to read binary image data. Basicly RAW image files are strings of bits with say 2 bytes per pixel. What i need to do is read the value of each pixel in turn and store then in another variable, px1, pix2 etc. I'm running an iterative algorithm on 2 pixels at a time then moving to the next two pixels.
I am having alot of problems however reading the data. from the net and various books i see the readline method however this looks for new line charicters. I need something that will read the value of a certain number of bits from the file.

for example to read the first pixel data which is 16bits long i need something that will just read those 16bits and return its value.
Thanks.
Ed

[bvdet]

This will read two bytes at a time:

>>> print [f.read(2) for _ in range(12)]
['96', '80', '07', '91', '37', '84', '41', '71', '15', '25', '18', '99']
>>> 

This was done on a text file. You should look at PIL link

[bartonc]

This will read two bytes at a time:

>>> print [f.read(2) for _ in range(12)]
['96', '80', '07', '91', '37', '84', '41', '71', '15', '25', '18', '99']
>>> 

This was done on a text file. You should look at PIL link

I imagine that you opened that text file in binary mode...

f = open('somefile.txt', 'b')

first.

[bartonc]

Hey Guys.
I'm looking for some info on how to read binary image data. Basicly RAW image files are strings of bits with say 2 bytes per pixel. What i need to do is read the value of each pixel in turn and store then in another variable, px1, pix2 etc. I'm running an iterative algorithm on 2 pixels at a time then moving to the next two pixels.
I am having alot of problems however reading the data. from the net and various books i see the readline method however this looks for new line charicters. I need something that will read the value of a certain number of bits from the file.

for example to read the first pixel data which is 16bits long i need something that will just read those 16bits and return its value.
Thanks.
Ed

I'm guessing that you know the inner workings of the image format that you are working on and can separate the format information (header) from the actual pixels of the image (data).

For working on large data sets, SciPy arrays have some features that native python lists lack.

[eddiefisher41]

Thanks guys.
Will investigate those options once i have time.
The RAW format i am working with is literally just raw data, it doesnt have a header file or start and stop bytes. The pixels have a bitdepth of 2bytes however the actual image data is only 10bits out of those 16, the rest are parity and padding bits.

Edit:
Ok just looking at the f.read(2) code above. The method returns the value of those two bytes as a string. The returned value is a hex value but is a string representation. How do i convert this string value into the actual hex value.
For example:

>>> f = open(r'C:\EFisher_ST_project07\RAW test images\724_day_mac_AV_on.raw', 'rb')
>>> pixel = f.read(2)
>>> pixel
'\xff\x00'
>>> type(pixel)
<type 'str'>

Cheers
Ed

[bvdet]

Thanks guys.
Will investigate those options once i have time.
The RAW format i am working with is literally just raw data, it doesnt have a header file or start and stop bytes. The pixels have a bitdepth of 2bytes however the actual image data is only 10bits out of those 16, the rest are parity and padding bits.

Edit:
Ok just looking at the f.read(2) code above. The method returns the value of those two bytes as a string. The returned value is a hex value but is a string representation. How do i convert this string value into the actual hex value.
For example:

>>> f = open(r'C:\EFisher_ST_project07\RAW test images\724_day_mac_AV_on.raw', 'rb')
>>> pixel = f.read(2)
>>> pixel
'\xff\x00'
>>> type(pixel)
<type 'str'>

Cheers
Ed

Maybe this will help:

import re
patt = re.compile(r'(?<=\\x)[a-f0-9]+')
s = repr('\xff\x00')
 
for hexNum in patt.findall(s):
    print int(hexNum, 16)

Output:
>>> 255
0
>>>
>>> patt.findall(s)
['ff', '00']
>>>

[bartonc]

Thanks guys.
Will investigate those options once i have time.
The RAW format i am working with is literally just raw data, it doesnt have a header file or start and stop bytes. The pixels have a bitdepth of 2bytes however the actual image data is only 10bits out of those 16, the rest are parity and padding bits.

Edit:
Ok just looking at the f.read(2) code above. The method returns the value of those two bytes as a string. The returned value is a hex value but is a string representation. How do i convert this string value into the actual hex value.
For example:

>>> f = open(r'C:\EFisher_ST_project07\RAW test images\724_day_mac_AV_on.raw', 'rb')
>>> pixel = f.read(2)
>>> pixel
'\xff\x00'
>>> type(pixel)
<type 'str'>

Cheers
Ed

I've never played with the struct module before, but here's what I found:
>>> import struct
>>> a = '\xff\xff'
>>> b = struct.unpack('BB', a)
>>> b
(255, 255)
>>> b = struct.unpack('H', a)
>>> b
(65535,)
>>>

[bartonc]

I've never played with the struct module before, but here's what I found:
>>> import struct
>>> a = '\xff\xff'
>>> b = struct.unpack('BB', a)
>>> b
(255, 255)
>>> b = struct.unpack('H', a)
>>> b
(65535,)
>>>

How about this:

>>> a = a * 8
>>> a
'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\ xff\xff\xff\xff'
>>> import array
>>> b = array.array('H',a)
>>> b
array('H', [65535, 65535, 65535, 65535, 65535, 65535, 65535, 65535])
>>>

[bartonc]

How about this:

>>> a = a * 8
>>> a
'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\ xff\xff\xff\xff'
>>> import array
>>> b = array.array('H',a)
>>> b
array('H', [65535, 65535, 65535, 65535, 65535, 65535, 65535, 65535])
>>>

Or better yet, use the array module to read the data directly:

fromfile( f, n)

Read n items (as machine values) from the file object f and append them to the end of the array. If less than n items are available, EOFError is raised, but the items that were available are still inserted into the array. f must be a real built-in file object; something else with a read() method won't do.

[eddiefisher41]

Cheers guys, will give these ideas a try and see how they turn out.
Thanks.
Ed

[eddiefisher41]

Hey Guys. I managed to do it, thats for your help.
Here is what i came up with:

def GetPxValue(openfile, bitdepth): #can iterate this part to get the next pixel
    read_bytes = operator.div(bitdepth, 8)  
    px_value = openfile.read(read_bytes)    
    if (bitdepth == 8):
        unpack_code = 'B'
    if (bitdepth == 16):
        unpack_code = 'BB'          
    if (bitdepth == 32):
        unpack_code = 'BBBB'
    hex_value = struct.unpack(unpack_code, px_value)  
    msbyte = Dec2Bin(hex_value[0])
    if (bitdepth == 16):
        lsbyte = Dec2Bin(hex_value[1])      
        for i in range(8):          
            msbyte.append(lsbyte[i])
    if (bitdepth == 32):
        byte2 = Dec2Bin(hex_value[1])
        byte3 = Dec2Bin(hex_value[2])       
        lsbyte = Dec2Bin(hex_value[3])
        for i in range(8):
            msbyte.append(byte2[i])
        for i in range(8):
            msbyte.append(byte3[i])
        for i in range(8):
            msbyte.append(lsbyte[i]) 
    return msbyte 
#return the whole bitdepth long pixel value as a list ready for filtering
 

Note: Dec2Bin is a decimal to binary converter that returns a binary value as a list. the list is necessary for filtering in the next stage.
Cheers.
Ed

[bartonc]

Hey Guys. I managed to do it, thats for your help.
Here is what i came up with:

def GetPxValue(openfile, bitdepth): #can iterate this part to get the next pixel
    read_bytes = operator.div(bitdepth, 8)  
    px_value = openfile.read(read_bytes)    
    if (bitdepth == 8):
        unpack_code = 'B'
    if (bitdepth == 16):
        unpack_code = 'BB'          
    if (bitdepth == 32):
        unpack_code = 'BBBB'
    hex_value = struct.unpack(unpack_code, px_value)  
    msbyte = Dec2Bin(hex_value[0])
    if (bitdepth == 16):
        lsbyte = Dec2Bin(hex_value[1])      
        for i in range(8):          
            msbyte.append(lsbyte[i])
    if (bitdepth == 32):
        byte2 = Dec2Bin(hex_value[1])
        byte3 = Dec2Bin(hex_value[2])       
        lsbyte = Dec2Bin(hex_value[3])
        for i in range(8):
            msbyte.append(byte2[i])
        for i in range(8):
            msbyte.append(byte3[i])
        for i in range(8):
            msbyte.append(lsbyte[i]) 
    return msbyte 
#return the whole bitdepth long pixel value as a list ready for filtering
 

Note: Dec2Bin is a decimal to binary converter that returns a binary value as a list. the list is necessary for filtering in the next stage.
Cheers.
Ed

Hi Ed. It's great when members post the solutions that they have come up with. I've added CODE tags to your post. Instructions on how to do this are on the right hand side of the page while you are posting or replying.

I felt sure that you would go with the array thing when I discovered it. I think that you will get a performance boost be reading larger chunks and working on slices of the array. The cool thing about python is that you may use one approach to get things working, then, with a little tweaking, make it work really well.

Have fun,
Barton