large image db delemia

[Jason Murry]

I have a camera system (Axis) which stores JPG via FTP 1-10fps. There
is also a motion jpg live stream.

I am trying to store these images either in JPG or in video format so
they can be reviewed at a later date. I would need to be able to pull
a date-time range from the list.

This means 86,400 - 2,160,000 images (around 110Kb a piece) per camera
a day, depending on speed 1fps - 10fps. There will probably be 1-5
cameras typically. Perhaps more in some rare cases.

I have considered storing via FTP process to a file system, then
creating directories by day/hour to group the images. Then store the
filename and timestamp in a searchable (by timestamp) table in mySQL.
This would allow to read the image from disk and have it searchable by
timestamp.

I still am unsure how to display the images, one option is to do meta
refresh 1-10 times a second, but i think this will be a burden on the
client. There should only be 1-2 clients accessing this data and it
will be via local lan most likely.

Concerns I have:

1. Would storing the images in the database directly be fast enough
(by passes the need to worry about file locations and directory
sizes).

2. How many files can you store in a directory before it becomes to
slow to work with (50,000? 1 million?)

3. How do you display these jpgs as a smooth animation without hogging
bandwidth and client cpu to display a slideshow.

4. Was considering storing into a movie format, but how do you search
by time and what method use to store as movie.

Any ideas or insight on this problem would be greatly appreciated.

[Andy Hassall]

On Tue, 16 Sep 2003 09:39:19 GMT, Jason Murry <me@isp.com> wrote:

>I have a camera system (Axis) which stores JPG via FTP 1-10fps. There
>is also a motion jpg live stream.
>
>I am trying to store these images either in JPG or in video format so
>they can be reviewed at a later date. I would need to be able to pull
>a date-time range from the list.
>
>This means 86,400 - 2,160,000 images (around 110Kb a piece) per camera
>a day, depending on speed 1fps - 10fps. There will probably be 1-5
>cameras typically. Perhaps more in some rare cases.

That's quite a bit of data... between ~3 terabytes and ~400 terabytes per
year, between the minimum and maximum numbers you've listed.

Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
the wrong place somewhere!)

2160000 * 110000 * 5 * 365 = 433620000000000
/ 1099511627776 (1Tb)
= 394.37 Tb

>I have considered storing via FTP process to a file system, then
>creating directories by day/hour to group the images. Then store the
>filename and timestamp in a searchable (by timestamp) table in mySQL.
>This would allow to read the image from disk and have it searchable by
>timestamp.
>
>I still am unsure how to display the images, one option is to do meta
>refresh 1-10 times a second, but i think this will be a burden on the
>client. There should only be 1-2 clients accessing this data and it
>will be via local lan most likely.
>
>Concerns I have:
>
>1. Would storing the images in the database directly be fast enough
>(by passes the need to worry about file locations and directory
>sizes).

With the data volumes above? You'd need a pretty serious database. Storing
images in the database gives you transactional control over changes, and lets
you back up the database in one go, rather than worrying about matching backups
of the filesystem to the index in the database.

But it does add overhead to inserting and retreiving the data, which could be
reduced by caching the images in the filesystem, at least for repeated reads.
You mention you're using MySQL - forget storing in the database then, for this
sort of volume.

>2. How many files can you store in a directory before it becomes to
>slow to work with (50,000? 1 million?)

Depends entirely on the filesystem. Tens of thousands is probably where you
start thinking seriously about splitting it up.

Splitting it by date has the advantage you can just archive a directory or set
of directories for older sections that you don't immediately need, when you
start running out of disk space, rather than having to hunt for old files.

>3. How do you display these jpgs as a smooth animation without hogging
>bandwidth and client cpu to display a slideshow.

Send it as video; you mentioned motion jpeg earlier, that would seem to make
sense.

>4. Was considering storing into a movie format, but how do you search
>by time and what method use to store as movie.

This could have large advantages in terms of space; since you save a lot on
the compression as it'll be storing differences between frames. Again, motion
jpeg has advantages here, as it can store the video with lossless compression
(lossless compared to the original jpeg frames, anyway). Or you can run it with
lossy compression to save even more space; in which case you could consider
other video compression schemes. If you can stream it in as motion jpeg in the
first place, that could cut down on the processing power you'd need (a lot).

To play from time X to time Y, it'd just be a matter of finding the video
file(s) spanning that period, and extracting the frames. It'll probably have to
seek back a bit to the previous keyframe, to get to the frame at time X if it
doesn't lie exactly on a keyframe.

--
Andy Hassall (andy@andyh.co.uk) icq(5747695) ([url]http://www.andyh.co.uk[/url])
Space: disk usage analysis tool ([url]http://www.andyhsoftware.co.uk/space[/url])

[Jason Murry]

Hello, thanks for the response.

> That's quite a bit of data... between ~3 terabytes and ~400 terabytes per
>year, between the minimum and maximum numbers you've listed.
>
> Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
>the wrong place somewhere!)
>
>2160000 * 110000 * 5 * 365 = 433620000000000
> / 1099511627776 (1Tb)
> = 394.37 Tb

Here is my math, per camera.

About 9Gb/day per camera, I was thinking 1fps would be about what i
would be recording at.

Per Picture Size 109,000

Seconds in a day 86,400
Seconds in an hour 3,600

Images / day @ 1fps 86,400
Images / day @ 10fps 864,000

Size (MB) / day @ 1fps 8,981.32
Size (MB) / day @ 10fps 89,813.23

> With the data volumes above? You'd need a pretty serious database. Storing
>images in the database gives you transactional control over changes, and lets
>you back up the database in one go, rather than worrying about matching backups
>of the filesystem to the index in the database.
>

> But it does add overhead to inserting and retreiving the data, which could be
>reduced by caching the images in the filesystem, at least for repeated reads.
>You mention you're using MySQL - forget storing in the database then, for this
>sort of volume.

With mySQL since you do not have transaction support, there is no
logging and thus no transaction overhead. Storing the DB on a raw
petition would remove the file system overhead.

> Depends entirely on the filesystem. Tens of thousands is probably where you
>start thinking seriously about splitting it up.
>

I was thinking about the same, some where around 50k I was expecting
to have problems, if I split off by directories I can avoid this
limit, but I would also add an additional process of coping the
images, which would double th bandwith used by the ftp process

> Send it as video; you mentioned motion jpeg earlier, that would seem to make
>sense.
>

I would love to know how to do this, I don't know. I was thinking
about using PHP which I have used alot in the past, but I have never
worked with dynamic video formats and such. I did find a java class
in the JMF that takes JPG and makes it quicktime, but i never got it
working correctly. And I am not sure if dynamic video encoding would
be fast enough with java.

>
> This could have large advantages in terms of space; since you save a lot on
>the compression as it'll be storing differences between frames. Again, motion
>jpeg has advantages here, as it can store the video with lossless compression
>(lossless compared to the original jpeg frames, anyway). Or you can run it with
>lossy compression to save even more space; in which case you could consider
>other video compression schemes. If you can stream it in as motion jpeg in the
>first place, that could cut down on the processing power you'd need (a lot).
>

The camera has a Motion JPG Stream, but I can't find any way to use
this directly. I tried righting a vb program to open a connection and
try to read the stream but I was left with string of jpgs.

> To play from time X to time Y, it'd just be a matter of finding the video
>file(s) spanning that period, and extracting the frames. It'll probably have to
>seek back a bit to the previous keyframe, to get to the frame at time X if it
>doesn't lie exactly on a keyframe.

In motion JPG as I found out, all frames are key frames.

The problem with video, is how much work will it be to do real time
encoding of the jpgs into the video files. If the files coming in
around the clock, and have to move the files into directory structure
and encode a video, this is alot of work. I think it would need to be
C to be fast enough.

If you look at Milestone (a company who works with this problem with
Axis cameras) they claim to use a high performance proprietary db to
store the images. As an Oracle DBA, I know I can do this with Oracle,
but Oracle is way out of the bugdget for this project. Since the
images are not being searched or anything, there is really no reason
to store them in the db.

We trying to find a way to solve this problem so we can offer this
with the cameras for recording functionality.

[Andy Hassall]

On Tue, 16 Sep 2003 20:36:49 GMT, Jason Murry <me@isp.com> wrote:

>Hello, thanks for the response.
>

>> Ouch... over a terabyte a day worst case? (I hope I've put a load of zeroes in
>>the wrong place somewhere!)

>
>Here is my math, per camera.
>
>About 9Gb/day per camera, I was thinking 1fps would be about what i
>would be recording at.
>
>Per Picture Size 109,000
>
>Seconds in a day 86,400
>Seconds in an hour 3,600
>
>Images / day @ 1fps 86,400
>Images / day @ 10fps 864,000
>
>Size (MB) / day @ 1fps 8,981.32
>Size (MB) / day @ 10fps 89,813.23

OK - not sure where the 2,160,000 images came from in the first example - but
you also said 5 cameras; so 45 - 450 Gb a day? Still on the big side - largest
MySQL database I can find mentioned is 1Tb.

>> With the data volumes above? You'd need a pretty serious database. Storing
>>images in the database gives you transactional control over changes, and lets
>>you back up the database in one go, rather than worrying about matching backups
>>of the filesystem to the index in the database.
>>
>> But it does add overhead to inserting and retreiving the data, which could be
>>reduced by caching the images in the filesystem, at least for repeated reads.
>>You mention you're using MySQL - forget storing in the database then, for this
>>sort of volume.

>
>With mySQL since you do not have transaction support, there is no
>logging and thus no transaction overhead.

MySQL does have transaction support nowadays, can have logging turned on, and
even with those turned off there's still an overhead getting large data in and
out, compared with a filesystem.

>Storing the DB on a raw petition would remove the file system overhead.

Didn't know MySQL had support for that.

>> Depends entirely on the filesystem. Tens of thousands is probably where you
>>start thinking seriously about splitting it up.
>>

>I was thinking about the same, some where around 50k I was expecting
>to have problems, if I split off by directories I can avoid this
>limit, but I would also add an additional process of coping the
>images, which would double th bandwith used by the ftp process

Isn't that just a matter of moving the files, rather than copying?

>> Send it as video; you mentioned motion jpeg earlier, that would seem to make
>>sense.

>
>I would love to know how to do this, I don't know. I was thinking
>about using PHP which I have used alot in the past, but I have never
>worked with dynamic video formats and such. I did find a java class
>in the JMF that takes JPG and makes it quicktime, but i never got it
>working correctly. And I am not sure if dynamic video encoding would
>be fast enough with java.

If you're doing any processing on this much data, forget Java or PHP, you'll
be needing something a bit quicker, e.g. C or C++.

>> This could have large advantages in terms of space; since you save a lot on
>>the compression as it'll be storing differences between frames. Again, motion
>>jpeg has advantages here, as it can store the video with lossless compression
>>(lossless compared to the original jpeg frames, anyway). Or you can run it with
>>lossy compression to save even more space; in which case you could consider
>>other video compression schemes. If you can stream it in as motion jpeg in the
>>first place, that could cut down on the processing power you'd need (a lot).

>
>The camera has a Motion JPG Stream, but I can't find any way to use
>this directly. I tried righting a vb program to open a connection and
>try to read the stream but I was left with string of jpgs.

Don't know anything off-the-shelf; you mention VB, so you could be using
DirectShow or the older Video for Windows with a MJPEG codec.

Or it could be as simple as working out whatever protocol it's using to send
the stream (HTTP?) and just saving it straight to disc without worrying too
much about processing it at the time.

>> To play from time X to time Y, it'd just be a matter of finding the video
>>file(s) spanning that period, and extracting the frames. It'll probably have to
>>seek back a bit to the previous keyframe, to get to the frame at time X if it
>>doesn't lie exactly on a keyframe.

>
>In motion JPG as I found out, all frames are key frames.
>
>The problem with video, is how much work will it be to do real time
>encoding of the jpgs into the video files. If the files coming in
>around the clock, and have to move the files into directory structure
>and encode a video, this is alot of work. I think it would need to be
>C to be fast enough.

Most likely - and then you might be pushing it unless you've got quite a bit
of hardware to throw at it. If the stream's coming in as MJPEG, then the best
way has got to be to just save that stream as it comes in (since you won't have
enough time to process it much), and then possibly decode to frames on demand,
or just stream back the video.

I don't think splitting and storing as individual frames is going to be
practical.

What are these cameras likely to be recording? Lots of movement? If there can
be 'quiet' periods or areas in the picture you're pouring away disk space with
individual frames.

--
Andy Hassall (andy@andyh.co.uk) icq(5747695) ([url]http://www.andyh.co.uk[/url])
Space: disk usage analysis tool ([url]http://www.andyhsoftware.co.uk/space[/url])

[Randell D.]

"Jason Murry" <me@isp.com> wrote in message
news:efsemvgmagfa1bebb06ig8k3hs571s6rlh@4ax.com...

> Hello, thanks for the response.
>
>

> > That's quite a bit of data... between ~3 terabytes and ~400 terabytes

per

> >year, between the minimum and maximum numbers you've listed.
> >
> > Ouch... over a terabyte a day worst case? (I hope I've put a load of

zeroes in

> >the wrong place somewhere!)
> >
> >2160000 * 110000 * 5 * 365 = 433620000000000
> > / 1099511627776 (1Tb)
> > = 394.37 Tb

>
> Here is my math, per camera.
>
> About 9Gb/day per camera, I was thinking 1fps would be about what i
> would be recording at.
>
> Per Picture Size 109,000
>
> Seconds in a day 86,400
> Seconds in an hour 3,600
>
> Images / day @ 1fps 86,400
> Images / day @ 10fps 864,000
>
> Size (MB) / day @ 1fps 8,981.32
> Size (MB) / day @ 10fps 89,813.23
>
>

> > With the data volumes above? You'd need a pretty serious database.

Storing

> >images in the database gives you transactional control over changes, and

lets

> >you back up the database in one go, rather than worrying about matching

backups

> >of the filesystem to the index in the database.
> >

>

> > But it does add overhead to inserting and retreiving the data, which

could be

> >reduced by caching the images in the filesystem, at least for repeated

reads.

> >You mention you're using MySQL - forget storing in the database then, for

this

> >sort of volume.

>
> With mySQL since you do not have transaction support, there is no
> logging and thus no transaction overhead. Storing the DB on a raw
> petition would remove the file system overhead.
>
>

> > Depends entirely on the filesystem. Tens of thousands is probably where

you

> >start thinking seriously about splitting it up.
> >

>
> I was thinking about the same, some where around 50k I was expecting
> to have problems, if I split off by directories I can avoid this
> limit, but I would also add an additional process of coping the
> images, which would double th bandwith used by the ftp process
>
>

> > Send it as video; you mentioned motion jpeg earlier, that would seem to

make

> >sense.
> >

>
> I would love to know how to do this, I don't know. I was thinking
> about using PHP which I have used alot in the past, but I have never
> worked with dynamic video formats and such. I did find a java class
> in the JMF that takes JPG and makes it quicktime, but i never got it
> working correctly. And I am not sure if dynamic video encoding would
> be fast enough with java.
>

> >
> > This could have large advantages in terms of space; since you save a lot

on

> >the compression as it'll be storing differences between frames. Again,

motion

> >jpeg has advantages here, as it can store the video with lossless

compression

> >(lossless compared to the original jpeg frames, anyway). Or you can run

it with

> >lossy compression to save even more space; in which case you could

consider

> >other video compression schemes. If you can stream it in as motion jpeg

in the

> >first place, that could cut down on the processing power you'd need (a

lot).

> >

>
> The camera has a Motion JPG Stream, but I can't find any way to use
> this directly. I tried righting a vb program to open a connection and
> try to read the stream but I was left with string of jpgs.
>
>

> > To play from time X to time Y, it'd just be a matter of finding the

video

> >file(s) spanning that period, and extracting the frames. It'll probably

have to

> >seek back a bit to the previous keyframe, to get to the frame at time X

if it

> >doesn't lie exactly on a keyframe.

>
> In motion JPG as I found out, all frames are key frames.
>
> The problem with video, is how much work will it be to do real time
> encoding of the jpgs into the video files. If the files coming in
> around the clock, and have to move the files into directory structure
> and encode a video, this is alot of work. I think it would need to be
> C to be fast enough.
>
> If you look at Milestone (a company who works with this problem with
> Axis cameras) they claim to use a high performance proprietary db to
> store the images. As an Oracle DBA, I know I can do this with Oracle,
> but Oracle is way out of the bugdget for this project. Since the
> images are not being searched or anything, there is really no reason
> to store them in the db.
>
> We trying to find a way to solve this problem so we can offer this
> with the cameras for recording functionality.

Since you're open to the idea of movies, one possibility is to arrange each
video in to five minute slots or something - You can create the jpgs in to a
single directory and then use something like Real Players Helix or whatever
to convert the jpgs in to a streaming format. I believe they will handle
jpgs though I've not tried it - I used the basic version to convert aload of
avi and mpeg movies in to a streaming format (I did the conversion on a
windoze machine but know they supply the package for linux)... The basic
version does not permit batch processing however I was able to create DOS
batch files that permitted me to perform the conversion.

I'd be pretty sure that you could re-create something similar for jpg images
on a linux box with a cron job... and the streaming format of the images
should help save some disk space too...

[Jason Murry]

Yes, there will be alot of dead time, in fact a majority of it should
be dead time. Thats where converting to video should prove to be very
efficient compression-wise.

Ideally, most cameras should be recording still frames, unless
something happens.

Some clients may have something different, but this should be the
norm.

Ahh, I placed too many 0's on my first post thats why you had such a
high number compared to mine hehehe.