I have a camera system (Axis) which stores JPG via FTP 1-10fps. There
is also a motion jpg live stream.
I am trying to store these images either in JPG or in video format so
they can be reviewed at a later date. I would need to be able to pull
a date-time range from the list.
This means 86,400 - 2,160,000 images (around 110Kb a piece) per camera
a day, depending on speed 1fps - 10fps. There will probably be 1-5
cameras typically. Perhaps more in some rare cases.
I have considered storing via FTP process to a file system, then
creating directories by day/hour to group the images. Then store the
filename and timestamp in a searchable (by timestamp) table in mySQL.
This would allow to read the image from disk and have it searchable by
timestamp.
I still am unsure how to display the images, one option is to do meta
refresh 1-10 times a second, but i think this will be a burden on the
client. There should only be 1-2 clients accessing this data and it
will be via local lan most likely.
Concerns I have:
1. Would storing the images in the database directly be fast enough
(by passes the need to worry about file locations and directory
sizes).
2. How many files can you store in a directory before it becomes to
slow to work with (50,000? 1 million?)
3. How do you display these jpgs as a smooth animation without hogging
bandwidth and client cpu to display a slideshow.
4. Was considering storing into a movie format, but how do you search
by time and what method use to store as movie.
Any ideas or insight on this problem would be greatly appreciated.
4  2905 
On Tue, 16 Sep 2003 09:39:19 GMT, Jason Murry <me@isp.com> wrote: I have a camera system (Axis) which stores JPG via FTP 1-10fps. There
is also a motion jpg live stream.
I am trying to store these images either in JPG or in video format so
they can be reviewed at a later date. I would need to be able to pull
a date-time range from the list.
This means 86,400 - 2,160,000 images (around 110Kb a piece) per camera
a day, depending on speed 1fps - 10fps. There will probably be 1-5
cameras typically. Perhaps more in some rare cases.
That's quite a bit of data... between ~3 terabytes and ~400 terabytes per
year, between the minimum and maximum numbers you've listed.
Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)
2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb
I have considered storing via FTP process to a file system, then
creating directories by day/hour to group the images. Then store the
filename and timestamp in a searchable (by timestamp) table in mySQL.
This would allow to read the image from disk and have it searchable by
timestamp.
I still am unsure how to display the images, one option is to do meta
refresh 1-10 times a second, but i think this will be a burden on the
client. There should only be 1-2 clients accessing this data and it
will be via local lan most likely.
Concerns I have:
1. Would storing the images in the database directly be fast enough
(by passes the need to worry about file locations and directory
sizes).
With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.
But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.
2. How many files can you store in a directory before it becomes to
slow to work with (50,000? 1 million?)
Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.
Splitting it by date has the advantage you can just archive a directory or set
of directories for older sections that you don't immediately need, when you
start running out of disk space, rather than having to hunt for old files.
3. How do you display these jpgs as a smooth animation without hogging
bandwidth and client cpu to display a slideshow.
Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.
4. Was considering storing into a movie format, but how do you search
by time and what method use to store as movie.
This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).
To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.
--
Andy Hassall (an**@andyh.co.uk) icq(5747695) ( http://www.andyh.co.uk)
Space: disk usage analysis tool ( http://www.andyhsoftware.co.uk/space)
Hello, thanks for the response. That's quite a bit of data... between ~3 terabytes and ~400 terabytes per
year, between the minimum and maximum numbers you've listed.
Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)
2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb
Here is my math, per camera.
About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.
Per Picture Size 109,000
Seconds in a day 86,400
Seconds in an hour 3,600
Images / day @ 1fps 86,400
Images / day @ 10fps 864,000
Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23
With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.
But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.
With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead. Storing the DB on a raw
petition would remove the file system overhead.
Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.
I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process
Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.
I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.
This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).
The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.
To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.
In motion JPG as I found out, all frames are key frames.
The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.
If you look at Milestone (a company who works with this problem with
Axis cameras) they claim to use a high performance proprietary db to
store the images. As an Oracle DBA, I know I can do this with Oracle,
but Oracle is way out of the bugdget for this project. Since the
images are not being searched or anything, there is really no reason
to store them in the db.
We trying to find a way to solve this problem so we can offer this
with the cameras for recording functionality.
On Tue, 16 Sep 2003 20:36:49 GMT, Jason Murry <me@isp.com> wrote: Hello, thanks for the response.
Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)
Here is my math, per camera.
About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.
Per Picture Size 109,000
Seconds in a day 86,400
Seconds in an hour 3,600
Images / day @ 1fps 86,400
Images / day @ 10fps 864,000
Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23
OK - not sure where the 2,160,000 images came from in the first example - but
you also said 5 cameras; so 45 - 450 Gb a day? Still on the big side - largest
MySQL database I can find mentioned is 1Tb.
With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.
But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.
With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead.
MySQL does have transaction support nowadays, can have logging turned on, and
even with those turned off there's still an overhead getting large data in and
out, compared with a filesystem.
Storing the DB on a raw petition would remove the file system overhead.
Didn't know MySQL had support for that. Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.
I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process
Isn't that just a matter of moving the files, rather than copying? Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.
I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.
If you're doing any processing on this much data, forget Java or PHP, you'll
be needing something a bit quicker, e.g. C or C++. This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).
The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.
Don't know anything off-the-shelf; you mention VB, so you could be using
DirectShow or the older Video for Windows with a MJPEG codec.
Or it could be as simple as working out whatever protocol it's using to send
the stream (HTTP?) and just saving it straight to disc without worrying too
much about processing it at the time. To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.
In motion JPG as I found out, all frames are key frames.
The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.
Most likely - and then you might be pushing it unless you've got quite a bit
of hardware to throw at it. If the stream's coming in as MJPEG, then the best
way has got to be to just save that stream as it comes in (since you won't have
enough time to process it much), and then possibly decode to frames on demand,
or just stream back the video.
I don't think splitting and storing as individual frames is going to be
practical.
What are these cameras likely to be recording? Lots of movement? If there can
be 'quiet' periods or areas in the picture you're pouring away disk space with
individual frames.
--
Andy Hassall (an**@andyh.co.uk) icq(5747695) ( http://www.andyh.co.uk)
Space: disk usage analysis tool ( http://www.andyhsoftware.co.uk/space)
"Jason Murry" <me@isp.com> wrote in message
news:ef********************************@4ax.com... Hello, thanks for the response.
That's quite a bit of data... between ~3 terabytes and ~400 terabytes
peryear, between the minimum and maximum numbers you've listed.
Ouch... over a terabyte a day worst case? (I hope I've put a load of
zeroes inthe wrong place somewhere!)
2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb
Here is my math, per camera.
About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.
Per Picture Size 109,000
Seconds in a day 86,400
Seconds in an hour 3,600
Images / day @ 1fps 86,400
Images / day @ 10fps 864,000
Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23
With the data volumes above? You'd need a pretty serious database.
Storingimages in the database gives you transactional control over changes, and
letsyou back up the database in one go, rather than worrying about matching
backupsof the filesystem to the index in the database.
But it does add overhead to inserting and retreiving the data, which
could bereduced by caching the images in the filesystem, at least for repeated
reads.You mention you're using MySQL - forget storing in the database then, for
thissort of volume.
With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead. Storing the DB on a raw
petition would remove the file system overhead.
Depends entirely on the filesystem. Tens of thousands is probably where
youstart thinking seriously about splitting it up.
I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process
Send it as video; you mentioned motion jpeg earlier, that would seem to
makesense.
I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.
This could have large advantages in terms of space; since you save a lot
onthe compression as it'll be storing differences between frames. Again,
motionjpeg has advantages here, as it can store the video with lossless
compression(lossless compared to the original jpeg frames, anyway). Or you can run
it withlossy compression to save even more space; in which case you could
considerother video compression schemes. If you can stream it in as motion jpeg
in thefirst place, that could cut down on the processing power you'd need (a
lot).
The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.
To play from time X to time Y, it'd just be a matter of finding the
videofile(s) spanning that period, and extracting the frames. It'll probably
have toseek back a bit to the previous keyframe, to get to the frame at time X
if itdoesn't lie exactly on a keyframe.
In motion JPG as I found out, all frames are key frames.
The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.
If you look at Milestone (a company who works with this problem with
Axis cameras) they claim to use a high performance proprietary db to
store the images. As an Oracle DBA, I know I can do this with Oracle,
but Oracle is way out of the bugdget for this project. Since the
images are not being searched or anything, there is really no reason
to store them in the db.
We trying to find a way to solve this problem so we can offer this
with the cameras for recording functionality.
Since you're open to the idea of movies, one possibility is to arrange each
video in to five minute slots or something - You can create the jpgs in to a
single directory and then use something like Real Players Helix or whatever
to convert the jpgs in to a streaming format. I believe they will handle
jpgs though I've not tried it - I used the basic version to convert aload of
avi and mpeg movies in to a streaming format (I did the conversion on a
windoze machine but know they supply the package for linux)... The basic
version does not permit batch processing however I was able to create DOS
batch files that permitted me to perform the conversion.
I'd be pretty sure that you could re-create something similar for jpg images
on a linux box with a cron job... and the streaming format of the images
should help save some disk space too... This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: CDB |
last post by:
Hello I was running the Image Browser example (from the JAI book
by Lawrence Rodrigues) with a very large image, specifically one of the
size 12,000 x 6,000 pixels, and the image does not render...
|
by: delong |
last post by:
Hi
I am trying to display a large image on the form and make the form
scrollable.
My image is about 4200 x 7000 pixel.
private void Form1_Paint(object sender,...
|
by: DOK |
last post by:
I'm creating an ad for eBay and want to keep the page condensed. I
have 6 images - different views of an antique lap steel guitar - that
need to show superb detail. I have cropped the images to...
|
by: CSN |
last post by:
Is it possible to iterate over an array in plpgsql?
Something like:
function insert_stuff (rel_ids int)
....
foreach rel_ids as id
insert into table (rel_id, val)
values (id, 5);
|
by: Sharon |
last post by:
I’m wrote a small DLL that used the FreeImage.DLL (that can be found at
http://www.codeproject.com/bitmap/graphicsuite.asp).
I also wrote a small console application in C++ (unmanaged) that uses...
|
by: Louis LeBlanc |
last post by:
Hey folks. I'm new to the list, and not quite what you'd call a DB
Guru, so please be patient with me. I'm afraid the lead up here is a
bit verbose . . .
I am working on an application that...
|
by: Stedak |
last post by:
I have the following class I use to save Tiff's. The problem I have with it
is that the final size of the images are very large. If we scan directly to a
file the final tiff may be 600-900 kb.s but...
|
by: pankajinfoedge |
last post by:
hi friends,
i m pankaj.I am save JPEG map in MS sql server database in Image datatype formate.
and show image in my application in convert image datatype to byte and save in memorystream .and...
|
by: sevenz24 |
last post by:
So i have my images set up like this : http://cgi.ebay.com/Tippmann-98-paintBall-Marker-Gun-Paint-Ball_W0QQitemZ250274334261QQihZ015QQcategoryZ47248QQssPageNameZWDVWQQrdZ1QQcmdZViewItem
Scroll...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
| |
