473,228 Members | 1,664 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,228 software developers and data experts.

large image db delemia

I have a camera system (Axis) which stores JPG via FTP 1-10fps. There
is also a motion jpg live stream.

I am trying to store these images either in JPG or in video format so
they can be reviewed at a later date. I would need to be able to pull
a date-time range from the list.

This means 86,400 - 2,160,000 images (around 110Kb a piece) per camera
a day, depending on speed 1fps - 10fps. There will probably be 1-5
cameras typically. Perhaps more in some rare cases.

I have considered storing via FTP process to a file system, then
creating directories by day/hour to group the images. Then store the
filename and timestamp in a searchable (by timestamp) table in mySQL.
This would allow to read the image from disk and have it searchable by
timestamp.

I still am unsure how to display the images, one option is to do meta
refresh 1-10 times a second, but i think this will be a burden on the
client. There should only be 1-2 clients accessing this data and it
will be via local lan most likely.

Concerns I have:

1. Would storing the images in the database directly be fast enough
(by passes the need to worry about file locations and directory
sizes).

2. How many files can you store in a directory before it becomes to
slow to work with (50,000? 1 million?)

3. How do you display these jpgs as a smooth animation without hogging
bandwidth and client cpu to display a slideshow.

4. Was considering storing into a movie format, but how do you search
by time and what method use to store as movie.

Any ideas or insight on this problem would be greatly appreciated.

Jul 16 '05 #1
4 2895
On Tue, 16 Sep 2003 09:39:19 GMT, Jason Murry <me@isp.com> wrote:
I have a camera system (Axis) which stores JPG via FTP 1-10fps. There
is also a motion jpg live stream.

I am trying to store these images either in JPG or in video format so
they can be reviewed at a later date. I would need to be able to pull
a date-time range from the list.

This means 86,400 - 2,160,000 images (around 110Kb a piece) per camera
a day, depending on speed 1fps - 10fps. There will probably be 1-5
cameras typically. Perhaps more in some rare cases.
That's quite a bit of data... between ~3 terabytes and ~400 terabytes per
year, between the minimum and maximum numbers you've listed.

Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)

2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb
I have considered storing via FTP process to a file system, then
creating directories by day/hour to group the images. Then store the
filename and timestamp in a searchable (by timestamp) table in mySQL.
This would allow to read the image from disk and have it searchable by
timestamp.

I still am unsure how to display the images, one option is to do meta
refresh 1-10 times a second, but i think this will be a burden on the
client. There should only be 1-2 clients accessing this data and it
will be via local lan most likely.

Concerns I have:

1. Would storing the images in the database directly be fast enough
(by passes the need to worry about file locations and directory
sizes).
With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.

But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.
2. How many files can you store in a directory before it becomes to
slow to work with (50,000? 1 million?)
Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.

Splitting it by date has the advantage you can just archive a directory or set
of directories for older sections that you don't immediately need, when you
start running out of disk space, rather than having to hunt for old files.
3. How do you display these jpgs as a smooth animation without hogging
bandwidth and client cpu to display a slideshow.
Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.
4. Was considering storing into a movie format, but how do you search
by time and what method use to store as movie.


This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).

To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.

--
Andy Hassall (an**@andyh.co.uk) icq(5747695) (http://www.andyh.co.uk)
Space: disk usage analysis tool (http://www.andyhsoftware.co.uk/space)
Jul 16 '05 #2
Hello, thanks for the response.

That's quite a bit of data... between ~3 terabytes and ~400 terabytes per
year, between the minimum and maximum numbers you've listed.

Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)

2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb
Here is my math, per camera.

About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.

Per Picture Size 109,000

Seconds in a day 86,400
Seconds in an hour 3,600

Images / day @ 1fps 86,400
Images / day @ 10fps 864,000

Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23

With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.
But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.
With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead. Storing the DB on a raw
petition would remove the file system overhead.

Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.

I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process

Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.

I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.

This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).

The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.

To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.


In motion JPG as I found out, all frames are key frames.

The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.

If you look at Milestone (a company who works with this problem with
Axis cameras) they claim to use a high performance proprietary db to
store the images. As an Oracle DBA, I know I can do this with Oracle,
but Oracle is way out of the bugdget for this project. Since the
images are not being searched or anything, there is really no reason
to store them in the db.

We trying to find a way to solve this problem so we can offer this
with the cameras for recording functionality.
Jul 16 '05 #3
On Tue, 16 Sep 2003 20:36:49 GMT, Jason Murry <me@isp.com> wrote:
Hello, thanks for the response.
Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)
Here is my math, per camera.

About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.

Per Picture Size 109,000

Seconds in a day 86,400
Seconds in an hour 3,600

Images / day @ 1fps 86,400
Images / day @ 10fps 864,000

Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23


OK - not sure where the 2,160,000 images came from in the first example - but
you also said 5 cameras; so 45 - 450 Gb a day? Still on the big side - largest
MySQL database I can find mentioned is 1Tb.
With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.

But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.


With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead.


MySQL does have transaction support nowadays, can have logging turned on, and
even with those turned off there's still an overhead getting large data in and
out, compared with a filesystem.
Storing the DB on a raw petition would remove the file system overhead.


Didn't know MySQL had support for that.
Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.

I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process


Isn't that just a matter of moving the files, rather than copying?
Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.


I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.


If you're doing any processing on this much data, forget Java or PHP, you'll
be needing something a bit quicker, e.g. C or C++.
This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).


The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.


Don't know anything off-the-shelf; you mention VB, so you could be using
DirectShow or the older Video for Windows with a MJPEG codec.

Or it could be as simple as working out whatever protocol it's using to send
the stream (HTTP?) and just saving it straight to disc without worrying too
much about processing it at the time.
To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.


In motion JPG as I found out, all frames are key frames.

The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.


Most likely - and then you might be pushing it unless you've got quite a bit
of hardware to throw at it. If the stream's coming in as MJPEG, then the best
way has got to be to just save that stream as it comes in (since you won't have
enough time to process it much), and then possibly decode to frames on demand,
or just stream back the video.

I don't think splitting and storing as individual frames is going to be
practical.

What are these cameras likely to be recording? Lots of movement? If there can
be 'quiet' periods or areas in the picture you're pouring away disk space with
individual frames.

--
Andy Hassall (an**@andyh.co.uk) icq(5747695) (http://www.andyh.co.uk)
Space: disk usage analysis tool (http://www.andyhsoftware.co.uk/space)
Jul 16 '05 #4

"Jason Murry" <me@isp.com> wrote in message
news:ef********************************@4ax.com...
Hello, thanks for the response.

That's quite a bit of data... between ~3 terabytes and ~400 terabytes peryear, between the minimum and maximum numbers you've listed.

Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes inthe wrong place somewhere!)

2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb


Here is my math, per camera.

About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.

Per Picture Size 109,000

Seconds in a day 86,400
Seconds in an hour 3,600

Images / day @ 1fps 86,400
Images / day @ 10fps 864,000

Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23

With the data volumes above? You'd need a pretty serious database. Storingimages in the database gives you transactional control over changes, and letsyou back up the database in one go, rather than worrying about matching backupsof the filesystem to the index in the database.

But it does add overhead to inserting and retreiving the data, which could bereduced by caching the images in the filesystem, at least for repeated reads.You mention you're using MySQL - forget storing in the database then, for thissort of volume.


With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead. Storing the DB on a raw
petition would remove the file system overhead.

Depends entirely on the filesystem. Tens of thousands is probably where youstart thinking seriously about splitting it up.


I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process

Send it as video; you mentioned motion jpeg earlier, that would seem to makesense.


I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.

This could have large advantages in terms of space; since you save a lot onthe compression as it'll be storing differences between frames. Again, motionjpeg has advantages here, as it can store the video with lossless compression(lossless compared to the original jpeg frames, anyway). Or you can run it withlossy compression to save even more space; in which case you could considerother video compression schemes. If you can stream it in as motion jpeg in thefirst place, that could cut down on the processing power you'd need (a lot).


The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.

To play from time X to time Y, it'd just be a matter of finding the videofile(s) spanning that period, and extracting the frames. It'll probably have toseek back a bit to the previous keyframe, to get to the frame at time X if itdoesn't lie exactly on a keyframe.


In motion JPG as I found out, all frames are key frames.

The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.

If you look at Milestone (a company who works with this problem with
Axis cameras) they claim to use a high performance proprietary db to
store the images. As an Oracle DBA, I know I can do this with Oracle,
but Oracle is way out of the bugdget for this project. Since the
images are not being searched or anything, there is really no reason
to store them in the db.

We trying to find a way to solve this problem so we can offer this
with the cameras for recording functionality.


Since you're open to the idea of movies, one possibility is to arrange each
video in to five minute slots or something - You can create the jpgs in to a
single directory and then use something like Real Players Helix or whatever
to convert the jpgs in to a streaming format. I believe they will handle
jpgs though I've not tried it - I used the basic version to convert aload of
avi and mpeg movies in to a streaming format (I did the conversion on a
windoze machine but know they supply the package for linux)... The basic
version does not permit batch processing however I was able to create DOS
batch files that permitted me to perform the conversion.

I'd be pretty sure that you could re-create something similar for jpg images
on a linux box with a cron job... and the streaming format of the images
should help save some disk space too...
Jul 16 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: CDB | last post by:
Hello I was running the Image Browser example (from the JAI book by Lawrence Rodrigues) with a very large image, specifically one of the size 12,000 x 6,000 pixels, and the image does not render...
1
by: delong | last post by:
Hi I am trying to display a large image on the form and make the form scrollable. My image is about 4200 x 7000 pixel. private void Form1_Paint(object sender,...
3
by: DOK | last post by:
I'm creating an ad for eBay and want to keep the page condensed. I have 6 images - different views of an antique lap steel guitar - that need to show superb detail. I have cropped the images to...
11
by: CSN | last post by:
Is it possible to iterate over an array in plpgsql? Something like: function insert_stuff (rel_ids int) .... foreach rel_ids as id insert into table (rel_id, val) values (id, 5);
12
by: Sharon | last post by:
I’m wrote a small DLL that used the FreeImage.DLL (that can be found at http://www.codeproject.com/bitmap/graphicsuite.asp). I also wrote a small console application in C++ (unmanaged) that uses...
5
by: Louis LeBlanc | last post by:
Hey folks. I'm new to the list, and not quite what you'd call a DB Guru, so please be patient with me. I'm afraid the lead up here is a bit verbose . . . I am working on an application that...
1
by: Stedak | last post by:
I have the following class I use to save Tiff's. The problem I have with it is that the final size of the images are very large. If we scan directly to a file the final tiff may be 600-900 kb.s but...
0
by: pankajinfoedge | last post by:
hi friends, i m pankaj.I am save JPEG map in MS sql server database in Image datatype formate. and show image in my application in convert image datatype to byte and save in memorystream .and...
13
by: sevenz24 | last post by:
So i have my images set up like this : http://cgi.ebay.com/Tippmann-98-paintBall-Marker-Gun-Paint-Ball_W0QQitemZ250274334261QQihZ015QQcategoryZ47248QQssPageNameZWDVWQQrdZ1QQcmdZViewItem Scroll...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.