By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,345 Members | 1,783 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,345 IT Pros & Developers. It's quick & easy.

Copies of Code

P: 5
Has anyone tried to run several copies of the same python code in the background? Let me explain. What I need to do is open 50 or so different folders and do operations on the stuff inside each folder. It works fine in serial, open folder1, do work, open folder2, do work etc. But this takes a long time, order of 3-4 hours. So I was trying to get it to run concurrently.

The way I was planning on doing this was by calling each instance of the python code from a bash script. Here it is for four folders.

Expand|Select|Wrap|Line Numbers
  1. #!/bin/sh
  2. for i in $(seq 0 3);
  3. do
  4.         python calculate.py $i  &
  5. done
  6. wait
  7. exit 0
  8.  
The script takes the number as an argument and uses that number to open the appropriate folder.

Now here is the problem. When I run this script, folder0 is worked on correctly. The other 3 folders are not. Here is some output.

Expand|Select|Wrap|Line Numbers
  1. -bash-3.1$ ./test.sh 
  2. Engage the System
  3. Engage the System
  4. Engage the System
  5. Engage the System
  6. reading `/tmp/shared/sqb/folder0/snapshot_000' ...
  7. Error: can't open file.
  8. allocating memory...done
  9. Error: can't open file.
  10. Error: can't open file.
  11. reordering...done.
  12. space for particle ID freed
  13. DONE
  14. DONE
  15. DONE
  16. .
  17. .
  18. folder0 is being worked on... 
  19. .
  20. .
  21. DONE
  22.  
I assume that the "Error: can't open file" is a python error. Has anyone tried to do something like this before?

Thanks for all your help
Jan 22 '09 #1
Share this Question
Share on Google+
1 Reply


kaarthikeyapreyan
100+
P: 107
Check if the files that the python script manipulates are present in the corresponding folders the script is expected to work fine in that case e.g:

Dir structure
readir.py
test0/testfile0.py
test1/testfile1.py
test.sh
Expand|Select|Wrap|Line Numbers
  1. #!/bin/sh
  2. for i in $(seq 0 1);
  3. do
  4.    python readir.py $i  &
  5. done
  6. wait
  7. exit 0
  8.  
The python code would just print the contents of the directory
Expand|Select|Wrap|Line Numbers
  1. import os
  2. import sys
  3.  
  4. exec "os.chdir('test%s'%sys.argv[1])"
  5. print os.listdir('.')
The script works fine for me
o/p
['testfile1.py']
['testfile0.py']
Jan 23 '09 #2

Post your reply

Sign in to post your reply or Sign up for a free account.