Queue module and Python Documentation Rant

[Bart Nessux]

How can one view the contents of a queue? I'd like to verify that the
queue has the same number of objects as the list that has been put into
it. Also, should one think of a queue as static or dynamic (on the put
side)? Do you put stuff in it once and then get from it until it's empty
or do you put stuff in it a bit at a time? Or, can you do both? Also,
does the queue actually get smaller as the threads feed from it or does
it stay the same size and remember what stuff has been gotten from it
already? Is this an appropriate way to create a queue of 100,000 items?

url_queue = Queue.Queue(0)
for url in urls:
url_queue.put(u rl)

<rant>

I ask all of these questions because I find the Python documentation
lacking, to say the least. Python is a most excellent language, but the
docs are horrible! IMO, one can learn more about less sane languages
such as Perl and PHP simply because their documentation is so well
done... PHP especially. Don't get me wrong, I love Python, just hate
it's documentation. For a lang that prides itself on ease of use and
newbie-friendliness, Python should be ashamed of its module documentation

</rant>

Bart

[Roger Binns]

Bart Nessux wrote:

How can one view the contents of a queue?
You are over thinking the whole thing :-)

Especially note that the Queue is intended to be used in threaded
programming (multi-producer, multi-consumer) which the doc clearly
states.
I'd like to verify that the queue has the same number of objects
as the list that has been put into it.
There is a qsize() method clearly documented. What is wrong with
that?
Also, should one think of a queue as static or dynamic (on the put
side)?
The constructor documentation clearly describes how it behaves.
You can limit the number of items in the Queue at which point
new puts will block, or you can have the default behaviour which
allows unlimited items in the queue.
Do you put stuff in it once and then get from it until it's empty
or do you put stuff in it a bit at a time?
You put items into the Queue and get items out. Each item is an
object of any type. The Queue does not split up items. For example
if you place a string into the queue, you cannot retrieve just part
of the string first and another part later. You only get the whole
string out.
Also,
does the queue actually get smaller as the threads feed from it or does
it stay the same size and remember what stuff has been gotten from it
already?
It is a classic queue in the computer science sense. Objects can be added
(via the put method) and taken off (by the get method). The size is the
number of objects in the queue. The order is FIFO (first in first out).
Contrast with a stack which is LIFO (last in is the first out).

The "staying the same size" part of your comment usually refers to
another data type known as a circular buffer (or circular queue).

I would strongly recommend getting a copy of the Python Cookbook
and reading it. The contents are also available online but the
dead tree version is a better read.
Is this an appropriate way to create a queue of 100,000 items?

url_queue = Queue.Queue(0)
for url in urls:
url_queue.put(u rl)
Yes. However in most code you are unlikely to do that. In general
you will have some code producing items, and other code consuming
them. Another thread in this group had an example. One piece of
code was producing IP addresses to scan, and a seperate piece of
code (for example a worker thread) would take one work item and
do the scanning.

Consequently the number of items would be small since you would expect
the rate at which items are produced to be a similar order of magnitude
to the rate at which they are consumed.
I ask all of these questions because I find the Python documentation
lacking, to say the least.
The documentation for this class is actually short and sweet and I
don't think it could be improved. How would you improve it?

It is however missing an example which would make things a lot clearer
especially for people who aren't used to standard programming idioms.
The tutorial is a little brief but does cover data structures:
http://docs.python.org/tut/tut.html
PHP especially.
Pretty much anyone who has done PHP in anger raves about their docs (me
included). Not only is the meta-data really good (eg which version the
item was introduced in etc), but the user contributions are what makes
the big difference. Also as far as I can tell, every single page includes
an example.

There was a discussion here about doing the Python docs in a similar fashion
(especially the user annotations) a few weeks ago. As far as I could tell,
the results were that anyone who wanted to submit changes had to do it via
SourceForge bugs and wait a fair while (months) for the changes to appear,
or construct a wiki that is seperate from the docs.
Python should be ashamed of its module documentation

The documentation is fine. It is pretty standard for docs constructed
"cathedral" style. I do wish it was more bazaar style like the PHP
ones, but I don't have the time or inclination to help put the infrastructure
in place. Do you?

Roger

[Bart Nessux]

Roger Binns wrote:

There is a qsize() method clearly documented. What is wrong with
that?
Where??? The documentation for the module named Queue says nothing at
all about this...

http://docs.python.org/lib/module-Queue.html
The [Python] documentation is fine.

The docs are useless. One needs a Master's in CS just to read the module
index. In general, it's far too abstract. Many have no examples for
actual *usage* at all. Read the section about classes in the tutorial...
how long can you stay awake? It's so abstract and scholarly that it
boarders on being useless.

CS is applied math. However, Python does not think of itself as being
applied, but theoretical. This is good if you're a geek with a Ph.D, but
not if you're the average sys-admin attempting to solve an every-day
problem.

Python has a great syntax for beginners and advanced users alike. One
can do sequential, procedural and OO programming with it. Also, it can
be applied to most any problem on most any platform. However, the docs
are an exercise in futility. They remind me of the circumlocution that I
read about in law school... constantly talking around things instead of
about things.

This is good if everyone is on the same high-level of knowledge, but not
for a programming language (think applied approach to problem solving)
that has tremendous potential and is touted as user friendly for
non-programmers. No wonder so many people use Perl. As bad as Perl is,
it does not have a scholarly opinion of itself and is thus approachable
and usable.

Forgive me, I'm frustrated.

Bart

[Bart Nessux]

Bart Nessux wrote:

Roger Binns wrote:

There is a qsize() method clearly documented. What is wrong with
that?

Where??? The documentation for the module named Queue says nothing at
all about this...

http://docs.python.org/lib/module-Queue.html

You're right... at the bottom of the page, there is a subsection.
qsize() is there. In frustration, I overlooked it.

[Roger Binns]

Bart Nessux wrote:

Where??? The documentation for the module named Queue says nothing at
all about this...

http://docs.python.org/lib/module-Queue.html

You're right... at the bottom of the page, there is a subsection.
qsize() is there. In frustration, I overlooked it.

In this particular case it is because the module name and the class
within are same name. For most modules, they differ. The top page
contains an overview of the module and then the subsections/next
button take you through each of the classes provided.

Although even that style is followed in some places such as mailbox
http://docs.python.org/lib/module-mailbox.html it isn't followed in
others such as the textwrap module
http://docs.python.org/lib/module-textwrap.html

Unfortunately I don't see what you could have done to spot the
following page other than looking at the bottom. Even if you
use the CHM docs, they only point to the front page (even
for the "Queue (class)" index entry) rather than the actual
doc page for the Queue class (ie the following page).

Roger

[Roel Schroeven]

Roger Binns wrote:

Pretty much anyone who has done PHP in anger raves about their docs (me
included). Not only is the meta-data really good (eg which version the
item was introduced in etc), but the user contributions are what makes
the big difference. Also as far as I can tell, every single page includes
an example.

Well, not everybody: I don't like the PHP documentation at all. Part of
the problem is precisely the user comments: they are unstructured by
their very nature, and you always have to read them all because it's
very much possible to miss important information on special cases etc.
if you don't. And often they contradict each other, or are written by
newbies who feel it is their duty to share their clumsy re-invention of
the wheel to the whole community.

What should be done is regularly incorporate the useful information from
the user comments in the documentation itself and then remove the user
comments. That way the useful information is collected in one place,
while it is now scattered over the documentation proper on one side and
the different user comments on the other side.

I also find it very confusing and impractical that the user comments are
sorted from newest to oldest: very often new comments are reactions to
old comments. That requires me to scroll down-up-down-up-down-up, very
frustrating. It's like top-posters on Usenet.

Actually, I like the Python documentation much better. IMO it is both
more comprehensive and does a better job of explaining the underlying
concepts. For instance, I had absolutely no trouble finding the
documentation of the Queue module and understanding its purpose and usage.

--
"Codito ergo sum"
Roel Schroeven

[Paramjit Oberoi]

> The documentation is fine. It is pretty standard for docs constructed

"cathedral" style. I do wish it was more bazaar style like the PHP
ones, but I don't have the time or inclination to help put the infrastructure
in place. Do you?

I personally really like Python documentation, and very rarely have any
trouble with it. However, I think it could be improved by having a
consistent policy regarding examples. Some modules do have examples, but
many don't.

I think a policy that requires all modules to have examples, and a
consistent place for them in the documentation, would be a big help. It
doesn't have to be a sudden overhaul of everything; but a clear policy
will hopefully lead to examples slowly being added to all parts of the
docs in a year or two.

Regardless, in my experience, the Python documentation is much better that
most other documentation I have had to use. But then, I have a masters in
computer science, so I can actually understand the module index <wink>.

-param

[Bart Nessux]

Paramjit Oberoi wrote:

The documentation is fine. It is pretty standard for docs constructed
"cathedral" style. I do wish it was more bazaar style like the PHP
ones, but I don't have the time or inclination to help put the infrastructure
in place. Do you?

I personally really like Python documentation, and very rarely have any
trouble with it. However, I think it could be improved by having a
consistent policy regarding examples. Some modules do have examples, but
many don't.

I think a policy that requires all modules to have examples, and a
consistent place for them in the documentation, would be a big help.

I agree. It'd be nice to see examples based on how one intends to use
the language. For example, demonstrate how one might use a module in a
sequential manner for basic sys-admin scripting purposes, then show how
the same module might be used procedurally with functions to do
something larger and more complex, and finally give an example of the
most complex scenario (object oriented) in which someone like Param
might want to use a module. IMO, this would cover all the bases.

Python appeals to such a vast user base. Why not make the docs useful to
everyone regardless of their knowledge of programming concepts?

Also, the docs aren't that bad. I was just frustrated. Once I cooled
down and read harder, I figured it out.

[Peter Hansen]

Roger Binns wrote:

Unfortunately I don't see what you could have done to spot the
following page other than looking at the bottom.

Reading the docs more often would help. There are many different
pages that have such subsection links at the bottom, so one
should be used to it.

For the Queue module the main page is so small it really ought
to be hard to miss!

On the other hand, it would be nice if the name "Queue" in
the class Queue constructor above had a link to the Queue Objects
page as well. No doubt contributions to change the script that
generates the docs would be welcome. :-)

-Peter

[Peter Hansen]

Bart Nessux wrote:

Python appeals to such a vast user base. Why not make the docs useful to
everyone regardless of their knowledge of programming concepts?

Why not indeed? How much time do you have to contribute to the
improvements? (If the answer is "none right now", then you've
just answered your own question of course.)

-Peter