472,328 Members | 1,271 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,328 software developers and data experts.

Threads and Progress Bar

Hi,

I have a small application, written in Python, that uses threads.
The application uses function foo() to download files from the web. As it reads
data from the web server, it runs a progress bar by calling an install of a
progress bar class.

When using threads, I get the problem that the progress bar gets over-written by
the download progress of files from other threads.

I believe my change has to go into the progress bar class to make it thread
aware.

Are they any docs/suggestions on how to implement progress bars along with
threads ?

Thanks,
Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

Sep 1 '06 #1
3 8493
Dennis Lee Bieber on Friday 01 Sep 2006 23:04 wrote:
Well... first off -- some minimal code would be of use...
I was scared that people might feel that I'm asking for a ready-made
solution. :-)

Here's the code.

The is the progress bar code.
progressbar.py
class progressBar:
def __init__(self, minValue = 0, maxValue = 10, totalWidth=12):
self.progBar = "[]" # This holds the progress bar string
self.min = minValue
self.max = maxValue
self.span = maxValue - minValue
self.width = totalWidth
self.amount = 0 # When amount == max, we are 100% done
self.updateAmount(0) # Build progress bar string

def updateAmount(self, newAmount = 0):
if newAmount < self.min: newAmount = self.min
if newAmount self.max: newAmount = self.max
self.amount = newAmount

# Figure out the new percent done, round to an integer
diffFromMin = float(self.amount - self.min)
percentDone = (diffFromMin / float(self.span)) * 100.0
percentDone = round(percentDone)
percentDone = int(percentDone)

# Figure out how many hash bars the percentage should be
allFull = self.width - 2
numHashes = (percentDone / 100.0) * allFull
numHashes = int(round(numHashes))

# build a progress bar with hashes and spaces
self.progBar = "[" + '#'*numHashes + ' '*(allFull-numHashes) + "]"

# figure out where to put the percentage, roughly centered
percentPlace = (len(self.progBar) / 2) - len(str(percentDone))
percentString = str(percentDone) + "%"

# slice the percentage into the bar
self.progBar = self.progBar[0:percentPlace] + percentString +
self.progBar[percentPlace+len(percentString):] \
+ " " + str(newAmount/1024) + "KB of " + str(self.max/1024) + "KB"

def __str__(self):
return str(self.progBar)

def myReportHook(count, blockSize, totalSize):
import sys
global prog
prog = ""

if prog == "":
prog = progressBar(0,totalSize,50)
prog.updateAmount(count*blockSize)
sys.stdout.write (str(prog))
sys.stdout.write ("\r")
#print count * (blockSize/1024) , "kb of " , (totalSize/1024) , "kb
downloaded.\n"
Here's the function, download_from_web() which calls the progress bar:
main.py
def download_from_web(sUrl, sFile, sSourceDir, checksum):

try:
block_size = 4096
i = 0
counter = 0

os.chdir(sSourceDir)
temp = urllib2.urlopen(sUrl)
headers = temp.info()
size = int(headers['Content-Length'])
data = open(sFile,'wb')

log.msg("Downloading %s\n" % (sFile))
while i < size:
data.write (temp.read(block_size))
i += block_size
counter += 1
progressbar.myReportHook(counter, block_size, size)
print "\n"
data.close()
temp.close()
And since I later implemented threads, multiple threads call download_from_web()
concurrently, which in effect calls progress bar, thus I get a progress bar
which continuously keeps getting overwritten. :-)

Here's the code where multiple threads execute:

try:
lRawData = open(uri, 'r').readlines()
except IOError, (errno, strerror):
log.err("%s %s\n" % (errno, strerror))
errfunc(errno, '')
#INFO: Mac OS is having issues with Python Threading.
# Use the conventional model for Mac OS
if sys.platform == 'darwin':
log.verbose("Running on Mac OS. Python doesn't have proper support
for Threads on Mac OS X.\n")
log.verbose("Running in the conventional non-threaded way.\n")
for each_single_item in lRawData:
(sUrl, sFile, download_size, checksum) =
stripper(each_single_item)

if download_from_web(sUrl, sFile, sSourceDir, None) != True:
#sys.stderr.write("%s not downloaded from %s\n" % (sFile,
sUrl))
#sys.stderr.write("%s failed\n\n" % (sFile))
variables.errlist.append(sFile)
pass
else:
if zip_bool:
compress_the_file(zip_type_file, sFile, sSourceDir)
os.remove(sFile) # Remove it because we don't need the
file once it is zipped.
else:
#INFO: Thread Support
if variables.options.num_of_threads 1:
log.msg("WARNING: Threads is still in alpha stage. It's better
to use just a single thread at the moment.\n")
log.warn("Threads is still in alpha stage. It's better to use
just a single thread at the moment.\n")

NUMTHREADS = variables.options.num_of_threads
name = threading.currentThread().getName()
ziplock = threading.Lock()

def run(request, response, func=download_from_web):
'''Get items from the request Queue, process them
with func(), put the results along with the
Thread's name into the response Queue.

Stop running once an item is None.'''

while 1:
item = request.get()
if item is None:
break
(sUrl, sFile, download_size, checksum) = stripper(item)
response.put((name, sUrl, sFile, func(sUrl, sFile,
sSourceDir, None)))

# This will take care of making sure that if downloaded,
they are zipped
(thread_name, Url, File, exit_status) = responseQueue.get()
if exit_status == True:
if zip_bool:
ziplock.acquire()
try:
compress_the_file(zip_type_file, File,
sSourceDir)
os.remove(File) # Remove it because we don't
need the file once it is zipped.
finally:
ziplock.release()
else:
variables.errlist.append(File)
pass

# Create two Queues for the requests and responses
requestQueue = Queue.Queue()
responseQueue = Queue.Queue()

# Pool of NUMTHREADS Threads that run run().
thread_pool = [
threading.Thread(
target=run,
args=(requestQueue, responseQueue)
)
for i in range(NUMTHREADS)
]

# Start the threads.
for t in thread_pool: t.start()

# Queue up the requests.
for item in lRawData: requestQueue.put(item)

# Shut down the threads after all requests end.
# (Put one None "sentinel" for each thread.)
for t in thread_pool: requestQueue.put(None)

# Don't end the program prematurely.
#
# (Note that because Queue.get() is blocking by
# defualt this isn't strictly necessary. But if
# you were, say, handling responses in another
# thread, you'd want something like this in your
# main thread.)
for t in thread_pool: t.join()

Second... It sounds like you only created one progress bar, and each
thread is referencing that single bar. I'd suspect you need to create a
bar for EACH thread you create, and tell the thread which bar to update.
Yes, you're correct. That's what I'm also suspecting. I tried to do some minor
changes but couldn't succeed.
Request you to, if you reply with code, give a little explanation so that I can
understand and learn from it.

Thanks,
Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

Sep 1 '06 #2
Ritesh Raj Sarraf wrote:
[...]
Here's the function, download_from_web() which calls the progress bar:
main.py
def download_from_web(sUrl, sFile, sSourceDir, checksum):
[...]
temp = urllib2.urlopen(sUrl)
headers = temp.info()
size = int(headers['Content-Length'])
data = open(sFile,'wb')
Incidentally, not all HTTP responses with bodies have
a 'Content-Length' header.
--
--Bryan
Sep 2 '06 #3
On Fri, 01 Sep 2006 21:43:46 +0530
Ritesh Raj Sarraf <rr*@researchut.comwrote:
Hi,

I have a small application, written in Python, that uses threads.
The application uses function foo() to download files from the web. As itreads
data from the web server, it runs a progress bar by calling an install ofa
progress bar class.

When using threads, I get the problem that the progress bar gets over-written by
the download progress of files from other threads.

I believe my change has to go into the progress bar class to make it thread
aware.
You need some kind of lock. Look at the module threading. There
is a class called "Lock".

PS: If you use pygtk: I switched from using threads to idle_add and (*). This is much easier
and you don't need any locking.

# (*)
while gtk.events_pending():
gtk.main_iteration()

--
Thomas GŁttler, http://www.tbz-pariv.de/
Bernsdorfer Str. 210-212, 09126 Chemnitz, Tel.: 0371/5347-917
Sep 15 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: rollasoc | last post by:
Hi, Having a slight problem with how to pass parameters to functions between threads. I have a form with a progress bar and a description...
3
by: Jacob | last post by:
I'm working on a class that needs to be called from a windows form, do it's work, and then, show progress back to the main form. I'm well aware...
7
by: Mr. Mountain | last post by:
In the following code I simulate work being done on different threads by sleeping a couple methods for about 40 ms. However, some of these methods...
12
by: Grant | last post by:
I am having great difficulty understanding this and any code samples I find online are kilometres long and complicated to understand...Please could...
4
by: MSDousti | last post by:
Hi I have written a VB .NET app, which uses several threads. I thought that when the user closes the main window (when MainForm.closed event...
10
by: Brad | last post by:
I need to learn about threads and threading. I currently have a process that is pulling 10 to 12 thousand records from the AS400 and formatting the...
6
by: DaTurk | last post by:
Here's a quick question, say I have a class with a main routine, we'll just call it main. Main is on the "main" thread. Now lets have main...
10
by: David | last post by:
I am trying to get this straight in my head so I can implement. I wrote a small utility some time ago in c# that utilized a handful of threads....
0
by: PC GROUP | last post by:
C#, VS2005, THREADS Ok. I have to ways to achieve (almost) the same thing. What do i want to do? Control the progress of an operation in a...
0
by: tammygombez | last post by:
Hey everyone! I've been researching gaming laptops lately, and I must say, they can get pretty expensive. However, I've come across some great...
0
better678
by: better678 | last post by:
Question: Discuss your understanding of the Java platform. Is the statement "Java is interpreted" correct? Answer: Java is an object-oriented...
0
by: CD Tom | last post by:
This happens in runtime 2013 and 2016. When a report is run and then closed a toolbar shows up and the only way to get it to go away is to right...
0
by: CD Tom | last post by:
This only shows up in access runtime. When a user select a report from my report menu when they close the report they get a menu I've called Add-ins...
0
by: Naresh1 | last post by:
What is WebLogic Admin Training? WebLogic Admin Training is a specialized program designed to equip individuals with the skills and knowledge...
0
jalbright99669
by: jalbright99669 | last post by:
Am having a bit of a time with URL Rewrite. I need to incorporate http to https redirect with a reverse proxy. I have the URL Rewrite rules made...
0
by: antdb | last post by:
Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was...
1
by: Matthew3360 | last post by:
Hi, I have a python app that i want to be able to get variables from a php page on my webserver. My python app is on my computer. How would I make it...
0
by: AndyPSV | last post by:
HOW CAN I CREATE AN AI with an .executable file that would suck all files in the folder and on my computerHOW CAN I CREATE AN AI with an .executable...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.