WebSphere - Enterprise App

[James]

We are writing a web based application which primarily will serve
information to run an ivr. Based on our current IVR getting 1000000 hits a
week, this is going to be heavy duty. We have very little time, we have
data retrieval classes and a struts based MVC. I'm contemplating a
CallManager to hold call variables critical for tracking and writing. Since
there are sever small tables it reads from, I am loading those into memory
with a ServletContextListener when the server first comes up. These are
read only and they are stored in a hashmap using String[] objects. We also
potentially bring back a lot of data. What are the basic guidelines for
keeping the session under control so we can handle a decent amount of
concurrent calls. Is the saving of thousands of table reads worth the table
load into the context? The size of the tables is not bad, although one 10
col table has over 2000 rows, mostly single or double character fields.
Any thoughts from the pros would be appreciated. we are using struts 1.1
within our companies framework. I really have little Java help so a tip or
two would be terrific.

JT

[Yoyoma_2]

I haven't worked with websphere but here are my 0.02$

For my part when i worked on a J2EE project, we were running mysql.
Even with 3000 queries it was still pretty fast (because the queries
were local).

You said you were getting 1M hits a weak, what's your peak load? how is
it distributed? What is your projected maximum concurrent load?

Also, you mentioned you used MVC, did you ever think of using other
design patterns within your applications? Could using a Command pattern
with a write queue probably help you instead of this handling of
concurrent calls. Thread synchronization is very, very costly. If you
can elliminate aspects of your application that are concurrent onto even
a queue-based system, then it can go a long way.

Also I read that "We have very little time", "Our current IVR is getting
1M hits a week" and "I have little java help". At 1M hits a week, lets
use a gross horrible estimate of 1M/7days = 14000 hits/day (your actual
distribution, even on a weekday basis is probably what you want to go
on). If you are down for even 1 day you would loose 14 000 connections
(probably more). I would b e *VERY* carefull how you desing this if
this is the load conditions you are working on. Maby even ask a
Software Engineer (an actual Engineer, or a CS with a specialty in
realtime systems) to go and help you because to me, this sounds like a
project in trouble. If not now then soon when you need to go back and
patch this code, and try migrate it when 300 calls are coming in. If
you get it out there and there are bugs, your maintenance is going to be
very, very hard. Unless it is expected that there can be some downtime
but i would be very weary about that. If your current load is 1M/wk,
would next years be 1.5M/wk? How will this project grow? How will you
deal with this in a year?

Did you consider doing load-balancing across an array of servers?

Notes:
1) Usually probability distributions of telephone loads are poisson
distributions, according to studies.

2) Anywhere you can save synchronizing, load time, function calling,
moving from O(n) better datastructures. This type of stuff can really
improve your load.

3) Run your program through a profiler with a variety of test loads.
Try to optimize the sections that the profiler returns as being the most
costly.

I would need more information on what you are trying to do (sorry its
late :) ) to help you any further, hope it helps.

James wrote:

We are writing a web based application which primarily will serve
information to run an ivr. Based on our current IVR getting 1000000 hits a
week, this is going to be heavy duty. We have very little time, we have
data retrieval classes and a struts based MVC. I'm contemplating a
CallManager to hold call variables critical for tracking and writing. Since
there are sever small tables it reads from, I am loading those into memory
with a ServletContextListener when the server first comes up. These are
read only and they are stored in a hashmap using String[] objects. We also
potentially bring back a lot of data. What are the basic guidelines for
keeping the session under control so we can handle a decent amount of
concurrent calls. Is the saving of thousands of table reads worth the table
load into the context? The size of the tables is not bad, although one 10
col table has over 2000 rows, mostly single or double character fields.
Any thoughts from the pros would be appreciated. we are using struts 1.1
within our companies framework. I really have little Java help so a tip or
two would be terrific.

JT

[James]

Thanks as you raise very good points. The current load is handled by 9 AIX
boxes. I have worked with web applications, but this is the biggest by far.
So far our design is taking shape well. You see, there are vendors involved
also as the site will be hosted off site by a company that can provide
plenty of horsepower. Given our time frames I am tying into the corporate
framework which we have to do or the application would not be approved.
That features the struts framework as the basis for your application. I'm
not a huge fan and have seen struts abused. I did design the original
system when I was pretty green myself about 3 years ago. Since then I have
gotten Java certified and gotten a Masters in CS so I am growing more
confident as I have discarded some ideas and held onto others. Our backend
will hit the mainframe for legacy data. We have done this with the existing
app and we had already ported that to the framework I mentioned. The idea
of having a web side goes back to the original design and was mine as I
wanted the testers to have a tool and the guys were hard working and thirsty
for work like that. Anyway, we are using different design patterns. I
have introduced business delegate and session facade. No state will be held
in the business layer or mainframe access pieces. I have asked the
mainframe to reduce data delivered to us, as we use to get 18,000 bytes of
data most of which was not used but was parsed from a huge string. Reducing
those items that will never be spoken or printed actually reduced that call
to 1,000 bytes. I use a singleton that is bound to the session to manage
the call and I am trying to limit the state held by holding keys rather then
big amounts of data. Our server does offer good web tier support and I feel
we are on the right track. I have thought of commands and essentially calls
to the business will be commands in spirit although we may turn on the
command portion of our inherited framework. Configuration is difficult but
we have gradually turned on the services. I am not an EJB fan but was still
considering an asynchronous capability for certain background functions.
MDB's were an early consideration, now I don't know if we have to introduce
that layer. The app will also feature a visible side that I am planning to
enable via a mode setting so that our testing philosophy from the past is
maintained. Does anything here sound off base to you? Sounds like you have
done Enterprise apps so your comments are welcome. Oh, if anything sounds
good, let me know that to! Thanks.
"Yoyoma_2" <Yoyoma_2@[at-]Hotmail.com> wrote in message
news:fdy3c.757031$X%5.663997@pd7tw2no...

I haven't worked with websphere but here are my 0.02$

For my part when i worked on a J2EE project, we were running mysql.
Even with 3000 queries it was still pretty fast (because the queries
were local).

You said you were getting 1M hits a weak, what's your peak load? how is
it distributed? What is your projected maximum concurrent load?

Also, you mentioned you used MVC, did you ever think of using other
design patterns within your applications? Could using a Command pattern
with a write queue probably help you instead of this handling of
concurrent calls. Thread synchronization is very, very costly. If you
can elliminate aspects of your application that are concurrent onto even
a queue-based system, then it can go a long way.

Also I read that "We have very little time", "Our current IVR is getting
1M hits a week" and "I have little java help". At 1M hits a week, lets
use a gross horrible estimate of 1M/7days = 14000 hits/day (your actual
distribution, even on a weekday basis is probably what you want to go
on). If you are down for even 1 day you would loose 14 000 connections
(probably more). I would b e *VERY* carefull how you desing this if
this is the load conditions you are working on. Maby even ask a
Software Engineer (an actual Engineer, or a CS with a specialty in
realtime systems) to go and help you because to me, this sounds like a
project in trouble. If not now then soon when you need to go back and
patch this code, and try migrate it when 300 calls are coming in. If
you get it out there and there are bugs, your maintenance is going to be
very, very hard. Unless it is expected that there can be some downtime
but i would be very weary about that. If your current load is 1M/wk,
would next years be 1.5M/wk? How will this project grow? How will you
deal with this in a year?

Did you consider doing load-balancing across an array of servers?

Notes:
1) Usually probability distributions of telephone loads are poisson
distributions, according to studies.

2) Anywhere you can save synchronizing, load time, function calling,
moving from O(n) better datastructures. This type of stuff can really
improve your load.

3) Run your program through a profiler with a variety of test loads.
Try to optimize the sections that the profiler returns as being the most
costly.

I would need more information on what you are trying to do (sorry its
late :) ) to help you any further, hope it helps.

James wrote:

We are writing a web based application which primarily will serve
information to run an ivr. Based on our current IVR getting 1000000 hits a week, this is going to be heavy duty. We have very little time, we have
data retrieval classes and a struts based MVC. I'm contemplating a
CallManager to hold call variables critical for tracking and writing. Since there are sever small tables it reads from, I am loading those into memory with a ServletContextListener when the server first comes up. These are
read only and they are stored in a hashmap using String[] objects. We also potentially bring back a lot of data. What are the basic guidelines for
keeping the session under control so we can handle a decent amount of
concurrent calls. Is the saving of thousands of table reads worth the table load into the context? The size of the tables is not bad, although one 10 col table has over 2000 rows, mostly single or double character fields.
Any thoughts from the pros would be appreciated. we are using struts 1.1 within our companies framework. I really have little Java help so a tip or two would be terrific.

JT

[Yoyoma_2]

James wrote:

Thanks as you raise very good points. The current load is handled by 9 AIX
boxes. I have worked with web applications, but this is the biggest by far.
So it seems but those are alwaise the most fun!
So far our design is taking shape well. You see, there are vendors involved
also as the site will be hosted off site by a company that can provide
plenty of horsepower. Given our time frames I am tying into the corporate
framework which we have to do or the application would not be approved.
That's good, sounds like there is good corporate, segregated control
over the QA process.
That features the struts framework as the basis for your application. I'm
not a huge fan and have seen struts abused. I did design the original
system when I was pretty green myself about 3 years ago. Since then I have
gotten Java certified and gotten a Masters in CS so I am growing more
confident as I have discarded some ideas and held onto others.
Also good. From the original e-mail i thought you were just a
programmer graduating from the 2 year programming course in college or
something hehe. With a Masters in CS, i have no doubt you did (or have
a feeling of) the stats for the loads of the system. My fear in a
project like this would be abnormal peak loads that could break the
system. For example, I used to work for a major bank and on sept. 11,
everyone tried to get to their accounts to see, i guess, if their money
was still there. In those kinds of situations, we were happy we
designed for the emergency or the "probable" fact-of-life. Cohereantly,
we were also very happy of the security measures we put on the system
:). When you read "Enterprise Applications" like this it basically
means engineering. Or if you are from the states, something more than
just simple applications programming.

The integration of the system is the key, how your system will behave
on a variety of loads is very important. AT the very least do some
traffic loads throughout the night. Note any exception, make a timing
page to view response time and check every X seconds. It might help you
sniff out some race conditions that might have been missed in a no or
"minimal" load condition.

Again this is my 0.02c. I haven't done much J2EE and VIR software but i
have done plenty of enterprise and mission-critical software. That's
why i'me back in school now taking an engineering degree. Here in
canada, we have "software engineering" where we can certify software
like a civil engineer certifies a bridge. If you are in canada its a
good idea to go have it stamped by an engineer. At least you get the
liability shield from it :)
Our backend
will hit the mainframe for legacy data.
Do you expect any delays from doing that at such a high load? Can your
mainframe hold the augmentation in load for the next year or two? Its
bad when you design for now, but your requests climb in a linear
fashion, so then 2 years down the road your mainframe legacy data can't
take the load from the newer systems.
We have done this with the existing
app and we had already ported that to the framework I mentioned. The idea
of having a web side goes back to the original design and was mine as I
wanted the testers to have a tool and the guys were hard working and thirsty
for work like that. Anyway, we are using different design patterns. I
have introduced business delegate and session facade.
Those are good ideas. Though design patterns slow down the software
considerably, it does help alot in stability and maintainability. Like
i alwaise say "I'de rather pay 2000$ extra for a better server, then
spend 2M on trying to upgrade a dying software project"
No state will be held
in the business layer or mainframe access pieces. I have asked the
mainframe to reduce data delivered to us, as we use to get 18,000 bytes of
data most of which was not used but was parsed from a huge string. Reducing
those items that will never be spoken or printed actually reduced that call
to 1,000 bytes.
Parsing is a cpu-extensive task. Any way you can organize a
communication protocol between the two systems? Reduce the bandwidth =
less data = less data to parse. Maby format the data in such a way tha
tit might be easyer for your side to parse it. Depends if the system is
faster at generating the data then you of parsing it, of course. If its
one big XML, that can be slow. If you can bring it down to a code or
stuctured field type of data, it could go a long way. Parsing by state
machine might be faster for you in this case than using StringTokenizer
or something like that. You could parse and set the data in 1-pass and
not 2. That could help with load.

oh did i mention java -O ? :)
I use a singleton that is bound to the session to manage
the call and I am trying to limit the state held by holding keys rather then
big amounts of data. Our server does offer good web tier support and I feel
we are on the right track. I have thought of commands and essentially calls
to the business will be commands in spirit although we may turn on the
command portion of our inherited framework. Configuration is difficult but
we have gradually turned on the services. I am not an EJB fan but was still
considering an asynchronous capability for certain background functions.
EJB's rely on RMI if i remember correctly. RMI relies on you
intrinsically trusting an untrusted source == Not good for an enterprise
app :) Also RMI relies on serialization that is a huge slowdown in your
case.
MDB's were an early consideration, now I don't know if we have to introduce
that layer. The app will also feature a visible side that I am planning to
enable via a mode setting so that our testing philosophy from the past is
maintained.
I haven't worked on MDB's, we made our own technologies to do that kind
of stuff. This was around 2 years ago too so some stuff was still
pretty immature.

App visibility for testing is ok. Maby test on a seperate set of data?
The exposed "test" site could run the same code but with different
database connections so that if you bring down the site by load testing
or by finding a bug/feature, then you won't bring down the production
site.

Unit Testing is ok when you test the production site. But i would be
weary of doing any type of extensive testing on a "live" site.

Make sure your security for this "mode" setting is air-tight. It sounds
like the first place a cracker would try to get into the system from.
ESPECIALLY since testers alwaise like to leave stuff "opened" for
convenience.
Does anything here sound off base to you? Sounds like you have
done Enterprise apps so your comments are welcome. Oh, if anything sounds
good, let me know that to! Thanks.
Nothing sounds too off base but remember legally I have to tell you to
go see a certified professional about this kind of stuff. Thanks for
your reply, its nice to hear that my long, long, long replies sometimes
get read :)

"Yoyoma_2" <Yoyoma_2@[at-]Hotmail.com> wrote in message
news:fdy3c.757031$X%5.663997@pd7tw2no...

I haven't worked with websphere but here are my 0.02$

For my part when i worked on a J2EE project, we were running mysql.
Even with 3000 queries it was still pretty fast (because the queries
were local).

You said you were getting 1M hits a weak, what's your peak load? how is
it distributed? What is your projected maximum concurrent load?

Also, you mentioned you used MVC, did you ever think of using other
design patterns within your applications? Could using a Command pattern
with a write queue probably help you instead of this handling of
concurrent calls. Thread synchronization is very, very costly. If you
can elliminate aspects of your application that are concurrent onto even
a queue-based system, then it can go a long way.

Also I read that "We have very little time", "Our current IVR is getting
1M hits a week" and "I have little java help". At 1M hits a week, lets
use a gross horrible estimate of 1M/7days = 14000 hits/day (your actual
distribution, even on a weekday basis is probably what you want to go
on). If you are down for even 1 day you would loose 14 000 connections
(probably more). I would b e *VERY* carefull how you desing this if
this is the load conditions you are working on. Maby even ask a
Software Engineer (an actual Engineer, or a CS with a specialty in
realtime systems) to go and help you because to me, this sounds like a
project in trouble. If not now then soon when you need to go back and
patch this code, and try migrate it when 300 calls are coming in. If
you get it out there and there are bugs, your maintenance is going to be
very, very hard. Unless it is expected that there can be some downtime
but i would be very weary about that. If your current load is 1M/wk,
would next years be 1.5M/wk? How will this project grow? How will you
deal with this in a year?

Did you consider doing load-balancing across an array of servers?

Notes:
1) Usually probability distributions of telephone loads are poisson
distributions, according to studies.

2) Anywhere you can save synchronizing, load time, function calling,
moving from O(n) better datastructures. This type of stuff can really
improve your load.

3) Run your program through a profiler with a variety of test loads.
Try to optimize the sections that the profiler returns as being the most
costly.

I would need more information on what you are trying to do (sorry its
late :) ) to help you any further, hope it helps.

James wrote:

We are writing a web based application which primarily will serve
information to run an ivr. Based on our current IVR getting 1000000
hits a
week, this is going to be heavy duty. We have very little time, we have
data retrieval classes and a struts based MVC. I'm contemplating a
CallManager to hold call variables critical for tracking and writing.
Since
there are sever small tables it reads from, I am loading those into
memory
with a ServletContextListener when the server first comes up. These are
read only and they are stored in a hashmap using String[] objects. We
also
potentially bring back a lot of data. What are the basic guidelines for
keeping the session under control so we can handle a decent amount of
concurrent calls. Is the saving of thousands of table reads worth the
table
load into the context? The size of the tables is not bad, although one
10
col table has over 2000 rows, mostly single or double character fields.
Any thoughts from the pros would be appreciated. we are using struts
1.1
within our companies framework. I really have little Java help so a tip
or
two would be terrific.

JT