Application Center Test - Any good?

[Anthony Abbot]

Hi

I have a requirement to use the Application Center Test
(part of Visual Studio.Net Enterprise architect) program,
to simulate loading on my Web servers. Does anyone know
if it is any good? I have gotten it to work and it seems
to do the job.

I have a requirement to run multiple tests from many
computers simultaneously. The manual seems to hint that
there may be an 'enterpricse version' that does this.

I also need to call the system via COM - I can get the
COM objects to create, and a test to load OK, but get a
permission error trying to run a test. Is there any
secret to doing this?

Kind Regards

Anthony

[Skip Sailors]

Anthony Abbot wrote:

Hi

I have a requirement to use the Application Center Test
... Does anyone know if it is any good? I have gotten
it to work and it seems to do the job. I was on a team that spent a year developing capacity tests on a rather
complex application using ACT. After a year of poor results we invested in
Mercury Interactive Load Runner. IME, in capacity planning, you get what
you pay for. ACT is a start at a capacity-planning solution. Activities
can be recorded and then played back. A single machine can simulate the
activity of several machines.

Now, after you have that, you need a few more features that ACT lacks.
First, you need a way to manage parameterization. Since the base is
VBScript or JScript, it's easy enough to splice in a variable to a stream,
all you have to do is program it. So when you have say, 100 different
points that you want to parameterize, all you have to do is write code that
splices in values at 100 different points. If the parameterization is
context-sensitive; dependent on application state, all you have to do is
develop a state machine for the client to manage those dependencies, and a
parser to read the response from the application. To write a state machine
to manage such, all you need to do is understand the application under test
to fine details so that for any intersting input you can predict the form
of output you should parse... Remember to code for unexpected behavior.
Be a master of exception handling and error trapping.

Then you can begin to develop the reporting functionality. ACT reports are
a start. The reports are in an XML format. The schema that the reports
use is one that requires nothing more that there be a root element named, I
think, <report>. There is an implied schema that I am sure will be easy
for you to figure out. If you are interested in trends across tests, or
comparisons that are not present in the reporting tool, or generating
documents rather than using the interface provided, all you have to do is
write an XSLT or something that will read reports generated by ACT and
transform them to what you want. You might even consider a mecahnism to
move report data to an RDBMS for more flexibility. An XSL transform can
output SQL as easily as it can output HTML. All you have to do is develop
a relational schema that adequately captures the information that ACT can
generate. All you need to do there is find a good DBE and then a
reasonable DBA to manage that resource.

I hope you see my point. Using ACT you will invest as much effort writing
the testing framework as developing the application under test, if you
aren't careful. Much of the work you do will be developing an application
test framework, not testing the application, and most organizations will
become impatient waiting for numbers. Since there will be such a large
body of code behind the tests, clients can as easily suspect the quality of
the test, you will need a QA process for the tests that rivals the QA
process for the application.

It may be that when you look at the price of a Load Runner virtual user
(some $hundreds per) it is enough to make the organization balk. And I am
not suggesting the LR is a perfect application. It is still a lot of work
to make good capacity plans even with good tools. IME, the price of a
mature tool with good support, like LR, versus the price of ACT (free) and
the cost of developing the infrastructure for results (at least a year of
development and the cost of meeting requirements for high-quality test
software) there's not as much difference as it might appear at first.

I have a requirement to run multiple tests from many
computers simultaneously. The manual seems to hint that
there may be an 'enterpricse version' that does this.
One of the reasons we kept with ACT so long was we also saw this hint. When
we pressed M$ for a date on this alleged version, we found that the team
who built ACT was scattered over the width and breadth of the campus, that
plans for furhter development of the product were on hold indefinitely, and
that support for even the existing single-machine version was below our
expectations.
I also need to call the system via COM - I can get the
COM objects to create, and a test to load OK, but get a
permission error trying to run a test. Is there any
secret to doing this?

It's been awhile since I tried ACT and COM, and I only kind of remember, you
have to know the account that ACT is running and adjust that account to
have the rights to execute, not the human logged on to the machine.

HTH
--
An HWND is a terrible thing to waste.

[Skip Sailors]

Anthony Abbot wrote:

Hi

I have a requirement to use the Application Center Test
... Does anyone know if it is any good? I have gotten
it to work and it seems to do the job. I was on a team that spent a year developing capacity tests on a rather
complex application using ACT. After a year of poor results we invested in
Mercury Interactive Load Runner. IME, in capacity planning, you get what
you pay for. ACT is a start at a capacity-planning solution. Activities
can be recorded and then played back. A single machine can simulate the
activity of several machines.

Now, after you have that, you need a few more features that ACT lacks.
First, you need a way to manage parameterization. Since the base is
VBScript or JScript, it's easy enough to splice in a variable to a stream,
all you have to do is program it. So when you have say, 100 different
points that you want to parameterize, all you have to do is write code that
splices in values at 100 different points. If the parameterization is
context-sensitive; dependent on application state, all you have to do is
develop a state machine for the client to manage those dependencies, and a
parser to read the response from the application. To write a state machine
to manage such, all you need to do is understand the application under test
to fine details so that for any intersting input you can predict the form
of output you should parse... Remember to code for unexpected behavior.
Be a master of exception handling and error trapping.

Then you can begin to develop the reporting functionality. ACT reports are
a start. The reports are in an XML format. The schema that the reports
use is one that requires nothing more that there be a root element named, I
think, <report>. There is an implied schema that I am sure will be easy
for you to figure out. If you are interested in trends across tests, or
comparisons that are not present in the reporting tool, or generating
documents rather than using the interface provided, all you have to do is
write an XSLT or something that will read reports generated by ACT and
transform them to what you want. You might even consider a mecahnism to
move report data to an RDBMS for more flexibility. An XSL transform can
output SQL as easily as it can output HTML. All you have to do is develop
a relational schema that adequately captures the information that ACT can
generate. All you need to do there is find a good DBE and then a
reasonable DBA to manage that resource.

I hope you see my point. Using ACT you will invest as much effort writing
the testing framework as developing the application under test, if you
aren't careful. Much of the work you do will be developing an application
test framework, not testing the application, and most organizations will
become impatient waiting for numbers. Since there will be such a large
body of code behind the tests, clients can as easily suspect the quality of
the test, you will need a QA process for the tests that rivals the QA
process for the application.

It may be that when you look at the price of a Load Runner virtual user
(some $hundreds per) it is enough to make the organization balk. And I am
not suggesting the LR is a perfect application. It is still a lot of work
to make good capacity plans even with good tools. IME, the price of a
mature tool with good support, like LR, versus the price of ACT (free) and
the cost of developing the infrastructure for results (at least a year of
development and the cost of meeting requirements for high-quality test
software) there's not as much difference as it might appear at first.

I have a requirement to run multiple tests from many
computers simultaneously. The manual seems to hint that
there may be an 'enterpricse version' that does this.
One of the reasons we kept with ACT so long was we also saw this hint. When
we pressed M$ for a date on this alleged version, we found that the team
who built ACT was scattered over the width and breadth of the campus, that
plans for furhter development of the product were on hold indefinitely, and
that support for even the existing single-machine version was below our
expectations.
I also need to call the system via COM - I can get the
COM objects to create, and a test to load OK, but get a
permission error trying to run a test. Is there any
secret to doing this?

It's been awhile since I tried ACT and COM, and I only kind of remember, you
have to know the account that ACT is running and adjust that account to
have the rights to execute, not the human logged on to the machine.

HTH
--
An HWND is a terrible thing to waste.