I have a camera system (Axis) which stores JPG via FTP 1-10fps. There
is also a motion jpg live stream.
I am trying to store these images either in JPG or in video format so
they can be reviewed at a later date. I would need to be able to pull
a date-time range from the list.
This means 86,400 - 2,160,000 images (around 110Kb a piece) per camera
a day, depending on speed 1fps - 10fps. There will probably be 1-5
cameras typically. Perhaps more in some rare cases.
I have considered storing via FTP process to a file system, then
creating directories by day/hour to group the images. Then store the
filename and timestamp in a searchable (by timestamp) table in mySQL.
This would allow to read the image from disk and have it searchable by
timestamp.
I still am unsure how to display the images, one option is to do meta
refresh 1-10 times a second, but i think this will be a burden on the
client. There should only be 1-2 clients accessing this data and it
will be via local lan most likely.
Concerns I have:
1. Would storing the images in the database directly be fast enough
(by passes the need to worry about file locations and directory
sizes).
2. How many files can you store in a directory before it becomes to
slow to work with (50,000? 1 million?)
3. How do you display these jpgs as a smooth animation without hogging
bandwidth and client cpu to display a slideshow.
4. Was considering storing into a movie format, but how do you search
by time and what method use to store as movie.
Any ideas or insight on this problem would be greatly appreciated.
4  2930 
On Tue, 16 Sep 2003 09:39:19 GMT, Jason Murry <me@isp.com> wrote: I have a camera system (Axis) which stores JPG via FTP 1-10fps. There
is also a motion jpg live stream.
I am trying to store these images either in JPG or in video format so
they can be reviewed at a later date. I would need to be able to pull
a date-time range from the list.
This means 86,400 - 2,160,000 images (around 110Kb a piece) per camera
a day, depending on speed 1fps - 10fps. There will probably be 1-5
cameras typically. Perhaps more in some rare cases.
That's quite a bit of data... between ~3 terabytes and ~400 terabytes per
year, between the minimum and maximum numbers you've listed.
Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)
2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb
I have considered storing via FTP process to a file system, then
creating directories by day/hour to group the images. Then store the
filename and timestamp in a searchable (by timestamp) table in mySQL.
This would allow to read the image from disk and have it searchable by
timestamp.
I still am unsure how to display the images, one option is to do meta
refresh 1-10 times a second, but i think this will be a burden on the
client. There should only be 1-2 clients accessing this data and it
will be via local lan most likely.
Concerns I have:
1. Would storing the images in the database directly be fast enough
(by passes the need to worry about file locations and directory
sizes).
With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.
But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.
2. How many files can you store in a directory before it becomes to
slow to work with (50,000? 1 million?)
Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.
Splitting it by date has the advantage you can just archive a directory or set
of directories for older sections that you don't immediately need, when you
start running out of disk space, rather than having to hunt for old files.
3. How do you display these jpgs as a smooth animation without hogging
bandwidth and client cpu to display a slideshow.
Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.
4. Was considering storing into a movie format, but how do you search
by time and what method use to store as movie.
This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).
To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.
--
Andy Hassall (an**@andyh.co. uk) icq(5747695) ( http://www.andyh.co.uk)
Space: disk usage analysis tool ( http://www.andyhsoftware.co.uk/space)
Hello, thanks for the response. That's quite a bit of data... between ~3 terabytes and ~400 terabytes per
year, between the minimum and maximum numbers you've listed.
Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)
2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb
Here is my math, per camera.
About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.
Per Picture Size 109,000
Seconds in a day 86,400
Seconds in an hour 3,600
Images / day @ 1fps 86,400
Images / day @ 10fps 864,000
Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23
With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.
But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.
With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead. Storing the DB on a raw
petition would remove the file system overhead.
Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.
I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process
Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.
I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.
This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).
The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.
To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.
In motion JPG as I found out, all frames are key frames.
The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.
If you look at Milestone (a company who works with this problem with
Axis cameras) they claim to use a high performance proprietary db to
store the images. As an Oracle DBA, I know I can do this with Oracle,
but Oracle is way out of the bugdget for this project. Since the
images are not being searched or anything, there is really no reason
to store them in the db.
We trying to find a way to solve this problem so we can offer this
with the cameras for recording functionality.
On Tue, 16 Sep 2003 20:36:49 GMT, Jason Murry <me@isp.com> wrote: Hello, thanks for the response.
Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)
Here is my math, per camera.
About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.
Per Picture Size 109,000
Seconds in a day 86,400
Seconds in an hour 3,600
Images / day @ 1fps 86,400
Images / day @ 10fps 864,000
Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23
OK - not sure where the 2,160,000 images came from in the first example - but
you also said 5 cameras; so 45 - 450 Gb a day? Still on the big side - largest
MySQL database I can find mentioned is 1Tb.
With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.
But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.
With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead.
MySQL does have transaction support nowadays, can have logging turned on, and
even with those turned off there's still an overhead getting large data in and
out, compared with a filesystem.
Storing the DB on a raw petition would remove the file system overhead.
Didn't know MySQL had support for that. Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.
I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process
Isn't that just a matter of moving the files, rather than copying? Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.
I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.
If you're doing any processing on this much data, forget Java or PHP, you'll
be needing something a bit quicker, e.g. C or C++. This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).
The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.
Don't know anything off-the-shelf; you mention VB, so you could be using
DirectShow or the older Video for Windows with a MJPEG codec.
Or it could be as simple as working out whatever protocol it's using to send
the stream (HTTP?) and just saving it straight to disc without worrying too
much about processing it at the time. To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.
In motion JPG as I found out, all frames are key frames.
The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.
Most likely - and then you might be pushing it unless you've got quite a bit
of hardware to throw at it. If the stream's coming in as MJPEG, then the best
way has got to be to just save that stream as it comes in (since you won't have
enough time to process it much), and then possibly decode to frames on demand,
or just stream back the video.
I don't think splitting and storing as individual frames is going to be
practical.
What are these cameras likely to be recording? Lots of movement? If there can
be 'quiet' periods or areas in the picture you're pouring away disk space with
individual frames.
--
Andy Hassall (an**@andyh.co. uk) icq(5747695) ( http://www.andyh.co.uk)
Space: disk usage analysis tool ( http://www.andyhsoftware.co.uk/space)
"Jason Murry" <me@isp.com> wrote in message
news:ef******** *************** *********@4ax.c om... Hello, thanks for the response.
That's quite a bit of data... between ~3 terabytes and ~400 terabytes
peryear, between the minimum and maximum numbers you've listed.
Ouch... over a terabyte a day worst case? (I hope I've put a load of
zeroes inthe wrong place somewhere!)
2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb
Here is my math, per camera.
About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.
Per Picture Size 109,000
Seconds in a day 86,400
Seconds in an hour 3,600
Images / day @ 1fps 86,400
Images / day @ 10fps 864,000
Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23
With the data volumes above? You'd need a pretty serious database.
Storingimages in the database gives you transactional control over changes, and
letsyou back up the database in one go, rather than worrying about matching
backupsof the filesystem to the index in the database.
But it does add overhead to inserting and retreiving the data, which
could bereduced by caching the images in the filesystem, at least for repeated
reads.You mention you're using MySQL - forget storing in the database then, for
thissort of volume.
With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead. Storing the DB on a raw
petition would remove the file system overhead.
Depends entirely on the filesystem. Tens of thousands is probably where
youstart thinking seriously about splitting it up.
I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process
Send it as video; you mentioned motion jpeg earlier, that would seem to
makesense.
I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.
This could have large advantages in terms of space; since you save a lot
onthe compression as it'll be storing differences between frames. Again,
motionjpeg has advantages here, as it can store the video with lossless
compression(lossless compared to the original jpeg frames, anyway). Or you can run
it withlossy compression to save even more space; in which case you could
considerother video compression schemes. If you can stream it in as motion jpeg
in thefirst place, that could cut down on the processing power you'd need (a
lot).
The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.
To play from time X to time Y, it'd just be a matter of finding the
videofile(s) spanning that period, and extracting the frames. It'll probably
have toseek back a bit to the previous keyframe, to get to the frame at time X
if itdoesn't lie exactly on a keyframe.
In motion JPG as I found out, all frames are key frames.
The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.
If you look at Milestone (a company who works with this problem with
Axis cameras) they claim to use a high performance proprietary db to
store the images. As an Oracle DBA, I know I can do this with Oracle,
but Oracle is way out of the bugdget for this project. Since the
images are not being searched or anything, there is really no reason
to store them in the db.
We trying to find a way to solve this problem so we can offer this
with the cameras for recording functionality.
Since you're open to the idea of movies, one possibility is to arrange each
video in to five minute slots or something - You can create the jpgs in to a
single directory and then use something like Real Players Helix or whatever
to convert the jpgs in to a streaming format. I believe they will handle
jpgs though I've not tried it - I used the basic version to convert aload of
avi and mpeg movies in to a streaming format (I did the conversion on a
windoze machine but know they supply the package for linux)... The basic
version does not permit batch processing however I was able to create DOS
batch files that permitted me to perform the conversion.
I'd be pretty sure that you could re-create something similar for jpg images
on a linux box with a cron job... and the streaming format of the images
should help save some disk space too... This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: CDB |
last post by:
Hello I was running the Image Browser example (from the JAI book
by Lawrence Rodrigues) with a very large image, specifically one of the
size 12,000 x 6,000 pixels, and the image does not render inside the
view port when using the original size display mode, the image appears
to be loaded in memory, and the application appears to be interacting.
When I use scaled or to-fit display mode everything shows inside the
view port. Do you know...
|
by: delong |
last post by:
Hi
I am trying to display a large image on the form and make the form
scrollable.
My image is about 4200 x 7000 pixel.
private void Form1_Paint(object sender, System.Windows.Forms.PaintEventArgs
e)
{
Graphics g=e.Graphics;
|
by: DOK |
last post by:
I'm creating an ad for eBay and want to keep the page condensed. I
have 6 images - different views of an antique lap steel guitar - that
need to show superb detail. I have cropped the images to take up a
full webpage width. However, I don't want to put all six images on
the page. I thought I could use icons as the trigger for displaying
the images one by one. I cropped the six images down to small (but
identifiable) icons.
Is there...
|
by: CSN |
last post by:
Is it possible to iterate over an array in plpgsql?
Something like:
function insert_stuff (rel_ids int)
....
foreach rel_ids as id
insert into table (rel_id, val)
values (id, 5);
|
by: Sharon |
last post by:
I’m wrote a small DLL that used the FreeImage.DLL (that can be found at
http://www.codeproject.com/bitmap/graphicsuite.asp).
I also wrote a small console application in C++ (unmanaged) that uses the
DLL above.
Now the application, together with the above DLL’s is successfully loading a
TIF image file (62992 x 113386 Pixels, Huffman RLE compression, 3200 x 3200
DPI resolution, binary colored (1 Bit Per Pixel), file on disk size 43.08...
| | |
by: Louis LeBlanc |
last post by:
Hey folks. I'm new to the list, and not quite what you'd call a DB
Guru, so please be patient with me. I'm afraid the lead up here is a
bit verbose . . .
I am working on an application that uses very high volume DB
transactions - in the order of tens of millions per day . . .
Anyway, the current database which will remain nameless, but begins
with O and rymes with debacle (sorta), has a problem with high volume
work when it comes to...
|
by: Stedak |
last post by:
I have the following class I use to save Tiff's. The problem I have with it
is that the final size of the images are very large. If we scan directly to a
file the final tiff may be 600-900 kb.s but with this code it is often
4000-5000 kb.s. What am I missing?
public class EmrTiff : IDisposable
{
private string fileName;
private ArrayList imageContainer = null;
|
by: pankajinfoedge |
last post by:
hi friends,
i m pankaj.I am save JPEG map in MS sql server database in Image datatype formate.
and show image in my application in convert image datatype to byte and save in memorystream .and show in picturebox control. this run fine .But when zoom in the map. them my application is work good for small image size(Just like 500KB).but my image size is 2.5 MB them in zoom time my application zoom more large time or hange the system.
please...
|
by: sevenz24 |
last post by:
So i have my images set up like this : http://cgi.ebay.com/Tippmann-98-paintBall-Marker-Gun-Paint-Ball_W0QQitemZ250274334261QQihZ015QQcategoryZ47248QQssPageNameZWDVWQQrdZ1QQcmdZViewItem
Scroll down to the bottom and you see several images, One large one and then several smaller ones, when you click on the smaller ones it places a larger version of that picture in place of where the other larger picture was, well i use this method on my...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
| | |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| | |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
