large data transfer between 2 processes

hi,

fist the context: I have a web server which query a mySql database. but
as the number of parralel queries increase, the server slows down too
much.

I got 2 ideas, one of which is to run N deamons which effectively
execute requests. the PHP uses message queuing(1) to queue queries
request to the daemons. but the answer may be quite large, larger than
messages the message queueing service can handle.

I thought to open back a communication link between the current daemon
and the waiting PHP to tranfer/process the result, using the message
queueing service to send back info regarding the opened link.

I thought to open a pipe between the current daemon and the PHP. is
this possible? and if so, how? if not, is an IP link to localhost be as
fast as a pipe?

TIA,
Pierre.

(1) http://www.phpfreaks.com/phpmanual/p...get-queue.html

re: large data transfer between 2 processes

pierre.bru@gmail.com wrote:[color=blue]
> hi,
>
> fist the context: I have a web server which query a mySql database. but
> as the number of parralel queries increase, the server slows down too
> much.
>
> I got 2 ideas, one of which is to run N deamons which effectively
> execute requests. the PHP uses message queuing(1) to queue queries
> request to the daemons. but the answer may be quite large, larger than
> messages the message queueing service can handle.
>
> I thought to open back a communication link between the current daemon
> and the waiting PHP to tranfer/process the result, using the message
> queueing service to send back info regarding the opened link.
>
> I thought to open a pipe between the current daemon and the PHP. is
> this possible? and if so, how? if not, is an IP link to localhost be as
> fast as a pipe?
>
> TIA,
> Pierre.
>
> (1) http://www.phpfreaks.com/phpmanual/p...get-queue.html
>[/color]

Pierre,

How is that going to help you? You're just adding another layer to an
already slow process.

First of all you need to determine why the slowdown occurs. Is it lack
of CPU cycles? Insufficient real memory? Disk access time?
Insufficient buffers/cache? Or any of a number of other things.

Once you determine *why* the slowdown occurs, you can work on that
problem. It may be tuning parameters, a faster CPU, multiple physical
disks, more memory, or even another machine to run your MySQL queries.

But just trying things without understanding the cause of the slowdown
is a very hit-and-miss way of "fixing" it.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

re: large data transfer between 2 processes

pierre.bru@gmail.com wrote:
[color=blue]
> hi,
>
> fist the context: I have a web server which query a mySql database. but
> as the number of parralel queries increase, the server slows down too
> much.
>[/color]

Not a PHP question.

MySQL doesn't do parallel queries. Queries are processed in a queue.

Your suggestion - to run more code with more layers of abstraction is more
likely to slow your machine down. Are you sure that it is purely down to
the size of the returned dataset? IME most DB performance issues can be
fixed by tweaking your schema. What does EXPLAIN say for your queries? Are
you logging slow queries?

I've never been a great fan of BLOBs in DBMS - filesystems are much more
capable of handling them.

Assuming that these avenues have been exhausted, you might think about
setting up a MySQL5 parallel cluster with partitioned data (not a
replicated cluster).

HTH

C.

re: large data transfer between 2 processes

And, I think you have to optimize your code ;-) and databases.



"Jerry Stuckle" <jstucklex@attglobal.net> ???????/???????? ? ????????
?????????: news:tOGdnZUtjbRVBqTeRVn-gQ@comcast.com...[color=blue]
> pierre.bru@gmail.com wrote:[color=green]
>> hi,
>>
>> fist the context: I have a web server which query a mySql database. but
>> as the number of parralel queries increase, the server slows down too
>> much.
>>
>> I got 2 ideas, one of which is to run N deamons which effectively
>> execute requests. the PHP uses message queuing(1) to queue queries
>> request to the daemons. but the answer may be quite large, larger than
>> messages the message queueing service can handle.
>>
>> I thought to open back a communication link between the current daemon
>> and the waiting PHP to tranfer/process the result, using the message
>> queueing service to send back info regarding the opened link.
>>
>> I thought to open a pipe between the current daemon and the PHP. is
>> this possible? and if so, how? if not, is an IP link to localhost be as
>> fast as a pipe?
>>
>> TIA,
>> Pierre.
>>
>> (1) http://www.phpfreaks.com/phpmanual/p...get-queue.html
>>[/color]
>
> Pierre,
>
> How is that going to help you? You're just adding another layer to an
> already slow process.
>
> First of all you need to determine why the slowdown occurs. Is it lack of
> CPU cycles? Insufficient real memory? Disk access time? Insufficient
> buffers/cache? Or any of a number of other things.
>
> Once you determine *why* the slowdown occurs, you can work on that
> problem. It may be tuning parameters, a faster CPU, multiple physical
> disks, more memory, or even another machine to run your MySQL queries.
>
> But just trying things without understanding the cause of the slowdown is
> a very hit-and-miss way of "fixing" it.
>
> --
> ==================
> Remove the "x" from my email address
> Jerry Stuckle
> JDS Computer Training Corp.
> jstucklex@attglobal.net
> ==================[/color]

re: large data transfer between 2 processes

> MySQL doesn't do parallel queries. Queries are processed in a queue.

oh! I didn't know that :-/

you do mean that if my web site receive let say 100 simultaneaous hits,
each one running a PHP scrit doing one or more mySQL queries, these
queries will be queued and done in sequence ?

where can find more info on this ?

BTW, the web server runs on a linux box and the DB on another box.

TIA,
Pierre.