Graham Dumpleton wrote:
>.......
In that blog, Guido says:
"""Concurrency: It seems we're now happily out exploring here. I'm
looking forward to benchmarks showing that PP or similar (or
dissimilar!) solutions actually provide a performance gain. Another
route I'd like to see explored is integrating one such solution into
an existing web framework (or perhaps as WSGI middleware) so that web
applications have an easy way out without redesigning their
architecture."""
........
Well I have been trying out a system very like PP. For various reasons I prefer
to write my own scheduler based on more traditional code, but it's very easy to
get SSH based workers going. Agreed it's a noddy system, but I find the
following timings for the pp callback demo with start=1, end=200000000.
All the workers are similar to a pp local worker, but some are controlled via
ssh and hence have to be initialised by an argument like -c"exec input()" which
downloads some bootstrap code; I find that an easy way to utilize a server which
already has ssh.
The local workers are clearly dominant, but even though app1,app3,app4 are not
in the same country they can still be used to provide useful work.
I suspect most people won't want to use either ssh or socket based workers when
64 cpu machines arrive. The worker farm topology is also a bit wrong for
massively parallel stuff.
################################################## ##############
Starting scheduler with 1 workers(1 local)
Partial sum is 0.69314718306 | diff = -2.50006693125e-009
Job execution statistics:
job count | % of all jobs | job time sum | time per job | worker
128 | 100.00 | 59.7190 | 0.466555 | local:3676
128 | 100.00 | 59.7190 | 0.466555 | all local
128 | 100.00 | 59.7190 | 0.466555 | total
Time elapsed since scheduler creation 59.7820000648
Real average time 0.467046875507
Starting scheduler with 2 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006693125e-009
Job execution statistics:
job count | % of all jobs | job time sum | time per job | worker
64 | 50.00 | 31.5300 | 0.492656 | local:324
64 | 50.00 | 31.2190 | 0.487797 | local:3684
128 | 100.00 | 62.7490 | 0.490227 | all local
128 | 100.00 | 62.7490 | 0.490227 | total
Time elapsed since scheduler creation 31.5460000038
Real average time 0.24645312503
Starting scheduler with 3 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006704228e-009
Job execution statistics:
job count | % of all jobs | job time sum | time per job | worker
53 | 41.41 | 25.6580 | 0.484113 | local:2392
52 | 40.63 | 25.5150 | 0.490673 | local:416
23 | 17.97 | 24.2500 | 1.054348 | ssh:RLUK_robin:20457
105 | 82.03 | 51.1730 | 0.487362 | all local
128 | 100.00 | 75.4230 | 0.589242 | total
Time elapsed since scheduler creation 25.7349998951
Real average time 0.20105468668
Starting scheduler with 4 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006704228e-009
Job execution statistics:
job count | % of all jobs | job time sum | time per job | worker
46 | 35.94 | 22.2200 | 0.483043 | local:2328
46 | 35.94 | 22.4670 | 0.488413 | local:3580
21 | 16.41 | 22.3150 | 1.062619 | ssh:RLUK_robin:20500
15 | 11.72 | 21.9730 | 1.464867 | ssh:app1:3549
92 | 71.88 | 44.6870 | 0.485728 | all local
128 | 100.00 | 88.9750 | 0.695117 | total
Time elapsed since scheduler creation 23.0320000648
Real average time 0.179937500507
Starting scheduler with 5 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006704228e-009
Job execution statistics:
job count | % of all jobs | job time sum | time per job | worker
40 | 31.25 | 19.3600 | 0.484000 | local:1376
41 | 32.03 | 19.7030 | 0.480561 | local:3608
18 | 14.06 | 19.1090 | 1.061611 | ssh:RLUK_robin:20524
14 | 10.94 | 19.1110 | 1.365071 | ssh:app1:3596
15 | 11.72 | 19.0950 | 1.273000 | ssh:app3:44316
81 | 63.28 | 39.0630 | 0.482259 | all local
128 | 100.00 | 96.3780 | 0.752953 | total
Time elapsed since server creation 19.9850001335
Real average time 0.156132813543
Starting scheduler with 7 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006693125e-009
Job execution statistics:
job count | % of all jobs | job time sum | time per job | worker
31 | 24.22 | 14.9540 | 0.482387 | local:2588
31 | 24.22 | 15.0630 | 0.485903 | local:472
13 | 10.16 | 14.0790 | 1.083000 | ssh:RLUK_robin:20548
9 | 7.03 | 14.7680 | 1.640889 | ssh:app1:3626
11 | 8.59 | 14.3260 | 1.302364 | ssh:app3:44344
15 | 11.72 | 14.2990 | 0.953267 | ssh:app4:61396
18 | 14.06 | 14.4050 | 0.800278 | ssh:app5:1258
62 | 48.44 | 30.0170 | 0.484145 | all local
128 | 100.00 | 101.8940 | 0.796047 | total
Time elapsed since scheduler creation 15.2030000687
Real average time 0.118773438036
################################################## ##############
--
Robin Becker