By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
448,799 Members | 1,356 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 448,799 IT Pros & Developers. It's quick & easy.

Queue module and Python Documentation Rant

P: n/a
How can one view the contents of a queue? I'd like to verify that the
queue has the same number of objects as the list that has been put into
it. Also, should one think of a queue as static or dynamic (on the put
side)? Do you put stuff in it once and then get from it until it's empty
or do you put stuff in it a bit at a time? Or, can you do both? Also,
does the queue actually get smaller as the threads feed from it or does
it stay the same size and remember what stuff has been gotten from it
already? Is this an appropriate way to create a queue of 100,000 items?

url_queue = Queue.Queue(0)
for url in urls:
url_queue.put(url)

<rant>

I ask all of these questions because I find the Python documentation
lacking, to say the least. Python is a most excellent language, but the
docs are horrible! IMO, one can learn more about less sane languages
such as Perl and PHP simply because their documentation is so well
done... PHP especially. Don't get me wrong, I love Python, just hate
it's documentation. For a lang that prides itself on ease of use and
newbie-friendliness, Python should be ashamed of its module documentation

</rant>

Bart
Jul 18 '05 #1
Share this Question
Share on Google+
17 Replies


P: n/a
Bart Nessux wrote:
How can one view the contents of a queue?
You are over thinking the whole thing :-)

Especially note that the Queue is intended to be used in threaded
programming (multi-producer, multi-consumer) which the doc clearly
states.
I'd like to verify that the queue has the same number of objects
as the list that has been put into it.
There is a qsize() method clearly documented. What is wrong with
that?
Also, should one think of a queue as static or dynamic (on the put
side)?
The constructor documentation clearly describes how it behaves.
You can limit the number of items in the Queue at which point
new puts will block, or you can have the default behaviour which
allows unlimited items in the queue.
Do you put stuff in it once and then get from it until it's empty
or do you put stuff in it a bit at a time?
You put items into the Queue and get items out. Each item is an
object of any type. The Queue does not split up items. For example
if you place a string into the queue, you cannot retrieve just part
of the string first and another part later. You only get the whole
string out.
Also,
does the queue actually get smaller as the threads feed from it or does
it stay the same size and remember what stuff has been gotten from it
already?
It is a classic queue in the computer science sense. Objects can be added
(via the put method) and taken off (by the get method). The size is the
number of objects in the queue. The order is FIFO (first in first out).
Contrast with a stack which is LIFO (last in is the first out).

The "staying the same size" part of your comment usually refers to
another data type known as a circular buffer (or circular queue).

I would strongly recommend getting a copy of the Python Cookbook
and reading it. The contents are also available online but the
dead tree version is a better read.
Is this an appropriate way to create a queue of 100,000 items?

url_queue = Queue.Queue(0)
for url in urls:
url_queue.put(url)
Yes. However in most code you are unlikely to do that. In general
you will have some code producing items, and other code consuming
them. Another thread in this group had an example. One piece of
code was producing IP addresses to scan, and a seperate piece of
code (for example a worker thread) would take one work item and
do the scanning.

Consequently the number of items would be small since you would expect
the rate at which items are produced to be a similar order of magnitude
to the rate at which they are consumed.
I ask all of these questions because I find the Python documentation
lacking, to say the least.
The documentation for this class is actually short and sweet and I
don't think it could be improved. How would you improve it?

It is however missing an example which would make things a lot clearer
especially for people who aren't used to standard programming idioms.
The tutorial is a little brief but does cover data structures:
http://docs.python.org/tut/tut.html
PHP especially.
Pretty much anyone who has done PHP in anger raves about their docs (me
included). Not only is the meta-data really good (eg which version the
item was introduced in etc), but the user contributions are what makes
the big difference. Also as far as I can tell, every single page includes
an example.

There was a discussion here about doing the Python docs in a similar fashion
(especially the user annotations) a few weeks ago. As far as I could tell,
the results were that anyone who wanted to submit changes had to do it via
SourceForge bugs and wait a fair while (months) for the changes to appear,
or construct a wiki that is seperate from the docs.
Python should be ashamed of its module documentation


The documentation is fine. It is pretty standard for docs constructed
"cathedral" style. I do wish it was more bazaar style like the PHP
ones, but I don't have the time or inclination to help put the infrastructure
in place. Do you?

Roger

Jul 18 '05 #2

P: n/a
Roger Binns wrote:
There is a qsize() method clearly documented. What is wrong with
that?
Where??? The documentation for the module named Queue says nothing at
all about this...

http://docs.python.org/lib/module-Queue.html
The [Python] documentation is fine.


The docs are useless. One needs a Master's in CS just to read the module
index. In general, it's far too abstract. Many have no examples for
actual *usage* at all. Read the section about classes in the tutorial...
how long can you stay awake? It's so abstract and scholarly that it
boarders on being useless.

CS is applied math. However, Python does not think of itself as being
applied, but theoretical. This is good if you're a geek with a Ph.D, but
not if you're the average sys-admin attempting to solve an every-day
problem.

Python has a great syntax for beginners and advanced users alike. One
can do sequential, procedural and OO programming with it. Also, it can
be applied to most any problem on most any platform. However, the docs
are an exercise in futility. They remind me of the circumlocution that I
read about in law school... constantly talking around things instead of
about things.

This is good if everyone is on the same high-level of knowledge, but not
for a programming language (think applied approach to problem solving)
that has tremendous potential and is touted as user friendly for
non-programmers. No wonder so many people use Perl. As bad as Perl is,
it does not have a scholarly opinion of itself and is thus approachable
and usable.

Forgive me, I'm frustrated.

Bart

Jul 18 '05 #3

P: n/a
Bart Nessux wrote:
Roger Binns wrote:
There is a qsize() method clearly documented. What is wrong with
that?

Where??? The documentation for the module named Queue says nothing at
all about this...

http://docs.python.org/lib/module-Queue.html


You're right... at the bottom of the page, there is a subsection.
qsize() is there. In frustration, I overlooked it.
Jul 18 '05 #4

P: n/a
Bart Nessux wrote:
Where??? The documentation for the module named Queue says nothing at
all about this...

http://docs.python.org/lib/module-Queue.html


You're right... at the bottom of the page, there is a subsection.
qsize() is there. In frustration, I overlooked it.


In this particular case it is because the module name and the class
within are same name. For most modules, they differ. The top page
contains an overview of the module and then the subsections/next
button take you through each of the classes provided.

Although even that style is followed in some places such as mailbox
http://docs.python.org/lib/module-mailbox.html it isn't followed in
others such as the textwrap module
http://docs.python.org/lib/module-textwrap.html

Unfortunately I don't see what you could have done to spot the
following page other than looking at the bottom. Even if you
use the CHM docs, they only point to the front page (even
for the "Queue (class)" index entry) rather than the actual
doc page for the Queue class (ie the following page).

Roger
Jul 18 '05 #5

P: n/a
Roger Binns wrote:
Pretty much anyone who has done PHP in anger raves about their docs (me
included). Not only is the meta-data really good (eg which version the
item was introduced in etc), but the user contributions are what makes
the big difference. Also as far as I can tell, every single page includes
an example.


Well, not everybody: I don't like the PHP documentation at all. Part of
the problem is precisely the user comments: they are unstructured by
their very nature, and you always have to read them all because it's
very much possible to miss important information on special cases etc.
if you don't. And often they contradict each other, or are written by
newbies who feel it is their duty to share their clumsy re-invention of
the wheel to the whole community.

What should be done is regularly incorporate the useful information from
the user comments in the documentation itself and then remove the user
comments. That way the useful information is collected in one place,
while it is now scattered over the documentation proper on one side and
the different user comments on the other side.

I also find it very confusing and impractical that the user comments are
sorted from newest to oldest: very often new comments are reactions to
old comments. That requires me to scroll down-up-down-up-down-up, very
frustrating. It's like top-posters on Usenet.

Actually, I like the Python documentation much better. IMO it is both
more comprehensive and does a better job of explaining the underlying
concepts. For instance, I had absolutely no trouble finding the
documentation of the Queue module and understanding its purpose and usage.

--
"Codito ergo sum"
Roel Schroeven
Jul 18 '05 #6

P: n/a
> The documentation is fine. It is pretty standard for docs constructed
"cathedral" style. I do wish it was more bazaar style like the PHP
ones, but I don't have the time or inclination to help put the infrastructure
in place. Do you?


I personally really like Python documentation, and very rarely have any
trouble with it. However, I think it could be improved by having a
consistent policy regarding examples. Some modules do have examples, but
many don't.

I think a policy that requires all modules to have examples, and a
consistent place for them in the documentation, would be a big help. It
doesn't have to be a sudden overhaul of everything; but a clear policy
will hopefully lead to examples slowly being added to all parts of the
docs in a year or two.

Regardless, in my experience, the Python documentation is much better that
most other documentation I have had to use. But then, I have a masters in
computer science, so I can actually understand the module index <wink>.

-param
Jul 18 '05 #7

P: n/a
Paramjit Oberoi wrote:
The documentation is fine. It is pretty standard for docs constructed
"cathedral" style. I do wish it was more bazaar style like the PHP
ones, but I don't have the time or inclination to help put the infrastructure
in place. Do you?

I personally really like Python documentation, and very rarely have any
trouble with it. However, I think it could be improved by having a
consistent policy regarding examples. Some modules do have examples, but
many don't.

I think a policy that requires all modules to have examples, and a
consistent place for them in the documentation, would be a big help.


I agree. It'd be nice to see examples based on how one intends to use
the language. For example, demonstrate how one might use a module in a
sequential manner for basic sys-admin scripting purposes, then show how
the same module might be used procedurally with functions to do
something larger and more complex, and finally give an example of the
most complex scenario (object oriented) in which someone like Param
might want to use a module. IMO, this would cover all the bases.

Python appeals to such a vast user base. Why not make the docs useful to
everyone regardless of their knowledge of programming concepts?

Also, the docs aren't that bad. I was just frustrated. Once I cooled
down and read harder, I figured it out.
Jul 18 '05 #8

P: n/a
Roger Binns wrote:
Unfortunately I don't see what you could have done to spot the
following page other than looking at the bottom.


Reading the docs more often would help. There are many different
pages that have such subsection links at the bottom, so one
should be used to it.

For the Queue module the main page is so small it really ought
to be hard to miss!

On the other hand, it would be nice if the name "Queue" in
the class Queue constructor above had a link to the Queue Objects
page as well. No doubt contributions to change the script that
generates the docs would be welcome. :-)

-Peter
Jul 18 '05 #9

P: n/a
Bart Nessux wrote:
Python appeals to such a vast user base. Why not make the docs useful to
everyone regardless of their knowledge of programming concepts?


Why not indeed? How much time do you have to contribute to the
improvements? (If the answer is "none right now", then you've
just answered your own question of course.)

-Peter
Jul 18 '05 #10

P: n/a
Roel Schroeven wrote:
Well, not everybody: I don't like the PHP documentation at all. Part of
the problem is precisely the user comments:
You don't have to read them :-) They are clearly distinguished from the
documentation itself.
if you don't. And often they contradict each other, or are written by
newbies who feel it is their duty to share their clumsy re-invention of
the wheel to the whole community.
To be honest I would rather have that and ignore them, than have nothing.
(That is a the cathedral vs the bazaar style situation).
What should be done is regularly incorporate the useful information from
the user comments in the documentation itself and then remove the user
comments.
I believe they do that, although I don't think it is done often enough.
I also find it very confusing and impractical that the user comments are
sorted from newest to oldest:


Agreed.

Roger
Jul 18 '05 #11

P: n/a
Peter Hansen <pe***@engcorp.com> schreef:
Reading the docs more often would help. There are many different
pages that have such subsection links at the bottom, so one
should be used to it.


True, but OTOH adding a level of detail to the contents page of the library
reference would be good for newcomers & less experienced Python users...

--
JanC

"Be strict when sending and tolerant when receiving."
RFC 1958 - Architectural Principles of the Internet - section 3.9
Jul 18 '05 #12

P: n/a
Roger Binns wrote:
Roel Schroeven wrote:
Well, not everybody: I don't like the PHP documentation at all. Part of
the problem is precisely the user comments:

You don't have to read them :-) They are clearly distinguished from the
documentation itself.


Yes, of course, but sometimes one or a few of them contain essential
information (that really ought to be in the documentation itself), and
there's no way to knowing except reading them all.

But it doesn't matter that much, since I'm not doing any PHP development
anymore, and most people who do seem to like the docs. Just wanted to
let the world know about the minority opinion :)

--
"Codito ergo sum"
Roel Schroeven
Jul 18 '05 #13

P: n/a
Bart Nessux wrote:
e:
There is a qsize() method clearly documented. What is wrong with
that?


Where??? The documentation for the module named Queue says nothing at
all about this...


it's the *first* item in the section that describes how Queue objects
work:

...

The public methods are:

qsize()
Return the approximate size of the queue.

...

what's wrong with you?

</F>


Jul 18 '05 #14

P: n/a
I think a policy that requires all modules to have examples, and a
consistent place for them in the documentation, would be a big help.


Bart> I agree. It'd be nice to see examples based on how one intends to
Bart> use the language.

People are always welcome to submit patches for the documentation.

Skip

Jul 18 '05 #15

P: n/a

Peter> Roger Binns wrote:
Unfortunately I don't see what you could have done to spot the
following page other than looking at the bottom.


Peter> Reading the docs more often would help. There are many different
Peter> pages that have such subsection links at the bottom, so one
Peter> should be used to it.

It's perhaps worth noting that there are basically two types of module doc
structures. Let's call them the Fred way (such as the docs for the Queue
module) and the Greg way (such as the docs for the optparse module).

There's nothing wrong with the way Greg wrote his docs, but they seem much
less like reference material to me and more like a normal user's manual,
emphasizing the important stuff but without being 100% comprehensive. That
makes it more challenging than I think it ought to be to answer a simple
question like, "what methods do OptionParser objects implement?" You can
fall back to pydoc but that requires that most/all methods have doc strings
to be useful. I have a particularly hard time finding information in the
distutils section of the libref manual. I should probably pay more
attention to the various submodule docs.

Skip

Jul 18 '05 #16

P: n/a
if you don't. And often they contradict each other, or are written by
newbies who feel it is their duty to share their clumsy re-invention
of the wheel to the whole community.


Roger> To be honest I would rather have that and ignore them, than have
Roger> nothing. (That is a the cathedral vs the bazaar style
Roger> situation).

If you are yourself a newbie how do you distinguish the wheat from the
chaff?

Skip

Jul 18 '05 #17

P: n/a
Skip Montanaro wrote:
If you are yourself a newbie how do you distinguish the wheat from the
chaff?


The same way they do it when reading this group :-)

Providing the documentation clearly distinguishes the "official" content
from the user contributed comments I don't see a problem. Sure some
idiot could put in clueless misleading drivel, but the same is true
of wikis, mailing lists and the php doc. It has not been a major problem
for any them.

Roger
Jul 18 '05 #18

This discussion thread is closed

Replies have been disabled for this discussion.