By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
438,196 Members | 973 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 438,196 IT Pros & Developers. It's quick & easy.

data handling issue

P: n/a
Hi all,
I'd like to have you opinion about the following performance/design
issue:

a logging system is generating a lot of records every our,(about
500k/day)
I need to store them in a huge Oracle RAC system
a Java web application connects to the data and executes queries on
them, mainly on last entered set of records
queries on informations stored more then 3 months ago are quite rare

as first guess I managed to store them in a table indexed by a
primary key, let's say in the table 'LOGS'
the problem is that table is going to grows very big,

have you got 'best practice' ideas to handle the situation in a better
way?

I thought about moving data priodically in a 'LOGS_ARCHIVE' table ,
for example using a Oracle job
is it necessary?
PS I can't use Oracle standard file archiving support, I need data to
stay on tables

thanks in advance
jc
Jul 19 '05 #1
Share this Question
Share on Google+
13 Replies


P: n/a

"curwen" <jo**********@despammed.com> wrote in message
news:cc**************************@posting.google.c om...
Hi all,
I'd like to have you opinion about the following performance/design
issue:

a logging system is generating a lot of records every our,(about
500k/day)
I need to store them in a huge Oracle RAC system
a Java web application connects to the data and executes queries on
them, mainly on last entered set of records
queries on informations stored more then 3 months ago are quite rare

as first guess I managed to store them in a table indexed by a
primary key, let's say in the table 'LOGS'
the problem is that table is going to grows very big,

have you got 'best practice' ideas to handle the situation in a better
way?

I thought about moving data priodically in a 'LOGS_ARCHIVE' table ,
for example using a Oracle job
is it necessary?
PS I can't use Oracle standard file archiving support, I need data to
stay on tables

thanks in advance
jc

I always find it strange that we see requests like this. Sort of along the
lines of "I am building this huge Oracle database and I don't know much
about Oracle but..." Wouldn't one get people on the project who have some
knowledge and expertise on Oracle? It just strikes me as very odd.
Jim
Jul 19 '05 #2

P: n/a
"Jim Kennedy" <ke****************************@attbi.net> wrote in message news:<%Y3%b.56084$Xp.266401@attbi_s54>...
[]
I always find it strange that we see requests like this. Sort of along the
lines of "I am building this huge Oracle database and I don't know much
about Oracle but..." Wouldn't one get people on the project who have some
knowledge and expertise on Oracle? It just strikes me as very odd.
Jim


My guess is the manager gets the cheapest programmer available and
keeps asking him to take over responsibilities of the more
knowledgable folks as they are laid off. Of course then the manager
hears about outsourcing to India at even lower rates and the pattern
begins again. 8^)

But seriously, I'm as puzzled as you are.
Ed
Jul 19 '05 #3

P: n/a
> > jc
I always find it strange that we see requests like this. Sort of along the
lines of "I am building this huge Oracle database and I don't know much
about Oracle but..." Wouldn't one get people on the project who have some
knowledge and expertise on Oracle? It just strikes me as very odd.
Jim

My experience in computer science is limited, I only have a First
Class Masters degree in Electronics Engineering, but I'm confident
with hard working and studying I can become 'good' as you

plus, I'll be much more professional answering in Usenet

thanks anyway
jc
Jul 19 '05 #4

P: n/a
ed********@magicinterface.com (Ed prochak) wrote in message news:<4b**************************@posting.google. com>...
"Jim Kennedy" <ke****************************@attbi.net> wrote in message

My guess is the manager gets the cheapest programmer available and
keeps asking him to take over responsibilities of the more
knowledgable folks as they are laid off. Of course then the manager
hears about outsourcing to India at even lower rates and the pattern
begins again. 8^)

But seriously, I'm as puzzled as you are.
Ed


My experience in computer science is limited, I only have a First
Class Masters degree in Electronics Engineering, but I'm confident
with hard working and studying I can become 'good' as you

plus, I'll be much more professional answering in Usenet

thanks anyway
jc
Jul 19 '05 #5

P: n/a

"curwen" <jo**********@despammed.com> wrote in message
news:cc**************************@posting.google.c om...
jc

I always find it strange that we see requests like this. Sort of along the lines of "I am building this huge Oracle database and I don't know much
about Oracle but..." Wouldn't one get people on the project who have some knowledge and expertise on Oracle? It just strikes me as very odd.
Jim

My experience in computer science is limited, I only have a First
Class Masters degree in Electronics Engineering, but I'm confident
with hard working and studying I can become 'good' as you

plus, I'll be much more professional answering in Usenet

thanks anyway
jc

I am sure you are a hard working good person. My comment has nothing to do
with your personality, but rather why people do this. (more the fault of
the manager from not getting experienced people for a task that needs an
experienced person) It is sort of like I've been hired to design a Boeing
747 and I have experience building paper airplanes. There is quite a gulf
there in needed experience.
Jim
Jul 19 '05 #6

P: n/a
Jim,
I do have some experience, maybe this is the most complex project I
have dealt with, but we have enough time and resources to improve the
skills if required..
anyway the project is not so mission critical, even if a lot of data
are expected, and from a general point of view the factors are quite
calibrated

anyway we used range partitioning to handle the issue

best regards
jc
"Jim Kennedy" <ke****************************@attbi.net> wrote in message news:<HgI%b.29833$AL.531756@attbi_s03>...
"curwen" <jo**********@despammed.com> wrote in message
news:cc**************************@posting.google.c om...
> jc
I always find it strange that we see requests like this. Sort of along the lines of "I am building this huge Oracle database and I don't know much
about Oracle but..." Wouldn't one get people on the project who have some knowledge and expertise on Oracle? It just strikes me as very odd.
Jim

My experience in computer science is limited, I only have a First
Class Masters degree in Electronics Engineering, but I'm confident
with hard working and studying I can become 'good' as you

plus, I'll be much more professional answering in Usenet

thanks anyway
jc

I am sure you are a hard working good person. My comment has nothing to do
with your personality, but rather why people do this. (more the fault of
the manager from not getting experienced people for a task that needs an
experienced person) It is sort of like I've been hired to design a Boeing
747 and I have experience building paper airplanes. There is quite a gulf
there in needed experience.
Jim

Jul 19 '05 #7

P: n/a
Hi

Engineer from india here :-)..Well there are many different ways to
ask a question and depending on the way its asked you would get many
different responses :-).

Your problem can it be understood this way you want to better manage
archival data that keeps on growing but is huge and you need it
online.

Well one possible solution would be to dig into the oracle Docs and
look into partitioning.Which is divide and conquer approcah for
managing data .Specifically look at range partitioning.

well if you want to be sure that usenet advice really works for you
you should take a hint and start doing test for yourself.You are smart
hmm and you will take every advice offered here and test it out.

And if you are too lazy well indian engineer here would welcome you
with open arms ..remember outsourcing is just a arms length away. :-))
regards
Hrishy

P.S:Please keep outsourcing and other bull shit away from this news
group folks.
I hate it this forum is meant for oracle and technical discussions.

jo**********@despammed.com (curwen) wrote in message news:<cc**************************@posting.google. com>...
Jim,
I do have some experience, maybe this is the most complex project I
have dealt with, but we have enough time and resources to improve the
skills if required..
anyway the project is not so mission critical, even if a lot of data
are expected, and from a general point of view the factors are quite
calibrated

anyway we used range partitioning to handle the issue

best regards
jc
"Jim Kennedy" <ke****************************@attbi.net> wrote in message news:<HgI%b.29833$AL.531756@attbi_s03>...
"curwen" <jo**********@despammed.com> wrote in message
news:cc**************************@posting.google.c om...
> > jc
> I always find it strange that we see requests like this. Sort of along the > lines of "I am building this huge Oracle database and I don't know much
> about Oracle but..." Wouldn't one get people on the project who have some > knowledge and expertise on Oracle? It just strikes me as very odd.
> Jim
My experience in computer science is limited, I only have a First
Class Masters degree in Electronics Engineering, but I'm confident
with hard working and studying I can become 'good' as you

plus, I'll be much more professional answering in Usenet

thanks anyway
jc

I am sure you are a hard working good person. My comment has nothing to do
with your personality, but rather why people do this. (more the fault of
the manager from not getting experienced people for a task that needs an
experienced person) It is sort of like I've been hired to design a Boeing
747 and I have experience building paper airplanes. There is quite a gulf
there in needed experience.
Jim

Jul 19 '05 #8

P: n/a

"hrishy" <hr*****@yahoo.co.uk> wrote in message
news:4e**************************@posting.google.c om...
Hi

Engineer from india here :-)..Well there are many different ways to
ask a question and depending on the way its asked you would get many
different responses :-).

Your problem can it be understood this way you want to better manage
archival data that keeps on growing but is huge and you need it
online.

Well one possible solution would be to dig into the oracle Docs and
look into partitioning.Which is divide and conquer approcah for
managing data .Specifically look at range partitioning.

well if you want to be sure that usenet advice really works for you
you should take a hint and start doing test for yourself.You are smart
hmm and you will take every advice offered here and test it out.

And if you are too lazy well indian engineer here would welcome you
with open arms ..remember outsourcing is just a arms length away. :-))
regards
Hrishy

P.S:Please keep outsourcing and other bull shit away from this news
group folks.
I hate it this forum is meant for oracle and technical discussions.
I've seen a lot of Indian engineers do this very thing. I've also seen
non-Indian engineers do this type of thing. I don't think it is a
nationality or country of origin thing.
Jim
jo**********@despammed.com (curwen) wrote in message

news:<cc**************************@posting.google. com>...
Jim,
I do have some experience, maybe this is the most complex project I
have dealt with, but we have enough time and resources to improve the
skills if required..
anyway the project is not so mission critical, even if a lot of data
are expected, and from a general point of view the factors are quite
calibrated

anyway we used range partitioning to handle the issue

best regards
jc
"Jim Kennedy" <ke****************************@attbi.net> wrote in message news:<HgI%b.29833$AL.531756@attbi_s03>...
"curwen" <jo**********@despammed.com> wrote in message
news:cc**************************@posting.google.c om...
> > > jc
> > I always find it strange that we see requests like this. Sort of along
the
> > lines of "I am building this huge Oracle database and I don't know
much > > about Oracle but..." Wouldn't one get people on the project who have some
> > knowledge and expertise on Oracle? It just strikes me as very

odd. > > Jim
>
>
> My experience in computer science is limited, I only have a First
> Class Masters degree in Electronics Engineering, but I'm confident
> with hard working and studying I can become 'good' as you
>
> plus, I'll be much more professional answering in Usenet
>
> thanks anyway
> jc
I am sure you are a hard working good person. My comment has nothing to do with your personality, but rather why people do this. (more the fault of the manager from not getting experienced people for a task that needs an experienced person) It is sort of like I've been hired to design a Boeing 747 and I have experience building paper airplanes. There is quite a gulf there in needed experience.
Jim

Jul 19 '05 #9

P: n/a
> Engineer from india here :-)..Well there are many different ways to
ask a question and depending on the way its asked you would get many
different responses :-).
And if you are too lazy well indian engineer here would welcome you
with open arms ..remember outsourcing is just a arms length away. :-))
regards
Hrishy

P.S:Please keep outsourcing and other bull shit away from this news
group folks.
I hate it this forum is meant for oracle and technical discussions.


I find it very weird:
you just raise a plain technical question and so many etherogeneous
and *OT* topics come out

a simple answer like: 'try use partitioning'
would have been too complex, isn't

PS even silence is a good option sometimes
Jul 19 '05 #10

P: n/a
jo**********@despammed.com (curwen) wrote in message news:<cc**************************@posting.google. com>...
ed********@magicinterface.com (Ed prochak) wrote in message news:<4b**************************@posting.google. com>...
"Jim Kennedy" <ke****************************@attbi.net> wrote in message

My guess is the manager gets the cheapest programmer available and
keeps asking him to take over responsibilities of the more
knowledgable folks as they are laid off. Of course then the manager
hears about outsourcing to India at even lower rates and the pattern
begins again. 8^)

But seriously, I'm as puzzled as you are.
Ed


My experience in computer science is limited, I only have a First
Class Masters degree in Electronics Engineering, but I'm confident
with hard working and studying I can become 'good' as you

plus, I'll be much more professional answering in Usenet

thanks anyway
jc


The point is that there is little to offer in way of answering your
original question. And the little bit of information you did provide
sounds so much like the many
just-out-of-college-web_guru-now-dba-on-Oracle posts that we have to
wonder. Now with your additional information, I really wonder why the
manager hired you when there are experienced Oracle folks around. Try
rereading my post, I was making a comment about the hiring manager,
not you.

But since you brough it up. If you are the DBA, yet you cannot find
out how to add space to the databasebase as needed, or what partioning
of tables can get you, or any of a dozen other DBA tasks I have to
wonder.

You never mentioned ORACLE version, hosting platform. You only hint
about size (500k records per DAY is not necessarily BIG).
(Boy I've been in a grouchy mood recently.)

Finally, FYI, this group is not really supported. The official groups
are
comp.databases.oracle.server
comp.databases.oracle.misc
comp.databases.oracle.tools
comp.databases.oracle.marketplace

Some servers still carry this group, but you'll get wider support
posting in the appropriate one of those four.

Sorry you took offense. Rephrase your question and you'll get a better
response.

good luck.
Ed
Jul 19 '05 #11

P: n/a
hr*****@yahoo.co.uk (hrishy) wrote in message news:<4e**************************@posting.google. com>...
Hi

Engineer from india here :-)..Well there are many different ways to
ask a question and depending on the way its asked you would get many
different responses :-).

Your problem can it be understood this way you want to better manage
archival data that keeps on growing but is huge and you need it
online.

Well one possible solution would be to dig into the oracle Docs and
look into partitioning.Which is divide and conquer approcah for
managing data .Specifically look at range partitioning.

well if you want to be sure that usenet advice really works for you
you should take a hint and start doing test for yourself.You are smart
hmm and you will take every advice offered here and test it out.

And if you are too lazy well indian engineer here would welcome you
with open arms ..remember outsourcing is just a arms length away. :-))
regards
Hrishy
Some good advice above.
P.S:Please keep outsourcing and other bull shit away from this news
group folks.
I hate it this forum is meant for oracle and technical discussions.

Given that THIS group (comp.databases.oracle) is really defunct, there
shouldn't be discussion here at all.

But being that some servers, (GOOGLE among them) still permits
posting to this group, some of the regulars in the 4 real ORACLE
groups browse here to lend a bit of help.

And if there is any doubt, my point had nothing to do with outsourcing
to India or anywhere else. It DID have to do with the management
culture of lower costs at all costs. IOW managers looking to hire an
ORACLE DBA, instead hire a tech school programmer, because they are
"cheaper". Whether they are the best candidate is not considered. And
switching to MySQL only because its free, opensource, not considering
whether it will work in that application either.

Now I wonder why I bothered to put that smiley on my post.
Jul 19 '05 #12

P: n/a

"Ed prochak" <ed********@magicinterface.com> wrote in message
news:4b**************************@posting.google.c om...
| hr*****@yahoo.co.uk (hrishy) wrote in message
news:<4e**************************@posting.google. com>...
| > Hi
| >
| > Engineer from india here :-)..Well there are many different ways to
| > ask a question and depending on the way its asked you would get many
| > different responses :-).
| >
| > Your problem can it be understood this way you want to better manage
| > archival data that keeps on growing but is huge and you need it
| > online.
| >
| > Well one possible solution would be to dig into the oracle Docs and
| > look into partitioning.Which is divide and conquer approcah for
| > managing data .Specifically look at range partitioning.
| >
| > well if you want to be sure that usenet advice really works for you
| > you should take a hint and start doing test for yourself.You are smart
| > hmm and you will take every advice offered here and test it out.
| >
| > And if you are too lazy well indian engineer here would welcome you
| > with open arms ..remember outsourcing is just a arms length away. :-))
| >
| >
| > regards
| > Hrishy
|
| Some good advice above.
| >
| > P.S:Please keep outsourcing and other bull shit away from this news
| > group folks.
| > I hate it this forum is meant for oracle and technical discussions.
| >
| Given that THIS group (comp.databases.oracle) is really defunct, there
| shouldn't be discussion here at all.
|
| But being that some servers, (GOOGLE among them) still permits
| posting to this group, some of the regulars in the 4 real ORACLE
| groups browse here to lend a bit of help.
|
| And if there is any doubt, my point had nothing to do with outsourcing
| to India or anywhere else. It DID have to do with the management
| culture of lower costs at all costs. IOW managers looking to hire an
| ORACLE DBA, instead hire a tech school programmer, because they are
| "cheaper". Whether they are the best candidate is not considered. And
| switching to MySQL only because its free, opensource, not considering
| whether it will work in that application either.
|
| Now I wonder why I bothered to put that smiley on my post.

ed,

you fail to realize that this management approach has been tried and proven
in the medical fields by many successful HMOs ;-)

;-{ mcs
Jul 19 '05 #13

P: n/a
Hi All

I think i should be joining the management..;-)

regards
Hrishy

"Mark C. Stock" <mcstockX@Xenquery .com> wrote in message news:<s4********************@comcast.com>...
"Ed prochak" <ed********@magicinterface.com> wrote in message
news:4b**************************@posting.google.c om...
| hr*****@yahoo.co.uk (hrishy) wrote in message
news:<4e**************************@posting.google. com>...
| > Hi
| >
| > Engineer from india here :-)..Well there are many different ways to
| > ask a question and depending on the way its asked you would get many
| > different responses :-).
| >
| > Your problem can it be understood this way you want to better manage
| > archival data that keeps on growing but is huge and you need it
| > online.
| >
| > Well one possible solution would be to dig into the oracle Docs and
| > look into partitioning.Which is divide and conquer approcah for
| > managing data .Specifically look at range partitioning.
| >
| > well if you want to be sure that usenet advice really works for you
| > you should take a hint and start doing test for yourself.You are smart
| > hmm and you will take every advice offered here and test it out.
| >
| > And if you are too lazy well indian engineer here would welcome you
| > with open arms ..remember outsourcing is just a arms length away. :-))
| >
| >
| > regards
| > Hrishy
|
| Some good advice above.
| >
| > P.S:Please keep outsourcing and other bull shit away from this news
| > group folks.
| > I hate it this forum is meant for oracle and technical discussions.
| >
| Given that THIS group (comp.databases.oracle) is really defunct, there
| shouldn't be discussion here at all.
|
| But being that some servers, (GOOGLE among them) still permits
| posting to this group, some of the regulars in the 4 real ORACLE
| groups browse here to lend a bit of help.
|
| And if there is any doubt, my point had nothing to do with outsourcing
| to India or anywhere else. It DID have to do with the management
| culture of lower costs at all costs. IOW managers looking to hire an
| ORACLE DBA, instead hire a tech school programmer, because they are
| "cheaper". Whether they are the best candidate is not considered. And
| switching to MySQL only because its free, opensource, not considering
| whether it will work in that application either.
|
| Now I wonder why I bothered to put that smiley on my post.

ed,

you fail to realize that this management approach has been tried and proven
in the medical fields by many successful HMOs ;-)

;-{ mcs

Jul 19 '05 #14

This discussion thread is closed

Replies have been disabled for this discussion.