473,757 Members | 2,066 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Python, subprocess, dump, gzip and Cron

Hi,

I'm having a bit of trouble with a python script I wrote, though I'm not
sure if it's related directly to python, or one of the other software
packages...

The situation is that I'm trying to create a system backup script that
creates an image of the system, filters the output though gzip, and then
uploads the data (via ftp) to a remote site.

The problem is that when I run the script from the command line, it
works as I expect it, but when it is run by cron I only get a 20 byte
file where the compressed image should be... does anyone have any idea
as to why this might be happening? Code follows

<code>

#!/usr/bin/python

from subprocess import PIPE, Popen
from ftplib import FTP

host = 'box'

filename = '%s.img.gz' % host
ftp_host = '192.168.1.250'
ftpuser, ftppass = 'admin', 'admin'
dest_dir = '/share/%s' % host

dump = Popen('dump 0uaf - /',shell=True,st dout=PIPE)
gzip = Popen('gzip',sh ell=True,stdin= dump.stdout,std out=PIPE)

ftp = FTP(ftp_host)
ftp.login(ftpus er,ftppass)
ftp.cwd(dest_di r)
ftp.storbinary( 'STOR %s' % filename,gzip.s tdout)
ftp.quit()

print "Image '%s' created" % filename

</code>

I appreciate all feedback. Thanks in advance.
Jun 27 '08 #1
4 3959
TT
On Jun 10, 2:37*pm, Aidan <awe...@gmail.c omwrote:
Hi,

I'm having a bit of trouble with a python script I wrote, though I'm not
sure if it's related directly to python, or one of the other software
packages...

The situation is that I'm trying to create a system backup script that
creates an image of the system, filters the output though gzip, and then
uploads the data (via ftp) to a remote site.

The problem is that when I run the script from the command line, it
works as I expect it, but when it is run by cron I only get a 20 byte
file where the compressed image should be... *does anyone have any idea
as to why this might be happening? *Code follows

<code>

#!/usr/bin/python

from subprocess import PIPE, Popen
from ftplib import FTP

host = 'box'

filename = '%s.img.gz' % host
ftp_host = '192.168.1.250'
ftpuser, ftppass = 'admin', 'admin'
dest_dir = '/share/%s' % host

dump = Popen('dump 0uaf - /',shell=True,st dout=PIPE)
gzip = Popen('gzip',sh ell=True,stdin= dump.stdout,std out=PIPE)

ftp = FTP(ftp_host)
ftp.login(ftpus er,ftppass)
ftp.cwd(dest_di r)
ftp.storbinary( 'STOR %s' % filename,gzip.s tdout)
ftp.quit()

print "Image '%s' created" % filename

</code>

I appreciate all feedback. *Thanks in advance.
it's possible that the cron doesn't have the environment variables you
have, especially $PATH. So the script failed to find the command it
need to create the image.
Jun 27 '08 #2
TT wrote:
On Jun 10, 2:37 pm, Aidan <awe...@gmail.c omwrote:
>Hi,

I'm having a bit of trouble with a python script I wrote, though I'm not
sure if it's related directly to python, or one of the other software
packages...

The situation is that I'm trying to create a system backup script that
creates an image of the system, filters the output though gzip, and then
uploads the data (via ftp) to a remote site.

The problem is that when I run the script from the command line, it
works as I expect it, but when it is run by cron I only get a 20 byte
file where the compressed image should be... does anyone have any idea
as to why this might be happening? Code follows

<code>

#!/usr/bin/python

from subprocess import PIPE, Popen
from ftplib import FTP

host = 'box'

filename = '%s.img.gz' % host
ftp_host = '192.168.1.250'
ftpuser, ftppass = 'admin', 'admin'
dest_dir = '/share/%s' % host

dump = Popen('dump 0uaf - /',shell=True,st dout=PIPE)
gzip = Popen('gzip',sh ell=True,stdin= dump.stdout,std out=PIPE)

ftp = FTP(ftp_host)
ftp.login(ftpu ser,ftppass)
ftp.cwd(dest_d ir)
ftp.storbinary ('STOR %s' % filename,gzip.s tdout)
ftp.quit()

print "Image '%s' created" % filename

</code>

I appreciate all feedback. Thanks in advance.

it's possible that the cron doesn't have the environment variables you
have, especially $PATH. So the script failed to find the command it
need to create the image.
*fore head slap*

Of course... adding the full path to both those utilities on the Popen
lines seems to have fixed it.

Thank you very much for your assistance.
Jun 27 '08 #3
Aidan <aw****@gmail.c omat Dienstag 10 Juni 2008 07:21:
TT wrote:
>On Jun 10, 2:37 pm, Aidan <awe...@gmail.c omwrote:
>>Hi,

I'm having a bit of trouble with a python script I wrote, though I'm not
sure if it's related directly to python, or one of the other software
packages...

The situation is that I'm trying to create a system backup script that
creates an image of the system, filters the output though gzip, and then
uploads the data (via ftp) to a remote site.

The problem is that when I run the script from the command line, it
works as I expect it, but when it is run by cron I only get a 20 byte
file where the compressed image should be... does anyone have any idea
as to why this might be happening? Code follows

<code>

#!/usr/bin/python

from subprocess import PIPE, Popen
from ftplib import FTP

host = 'box'

filename = '%s.img.gz' % host
ftp_host = '192.168.1.250'
ftpuser, ftppass = 'admin', 'admin'
dest_dir = '/share/%s' % host

dump = Popen('dump 0uaf - /',shell=True,st dout=PIPE)
You should avoid the use of ``shell=True`` here and use a argument list
instead:

dump = Popen(['dump', '0uaf', '-', '/'], stdout=PIPE)

This results in an exception thrown if the executable doesn't exist. This
exception can be caught and handle for instance with the logging module.
>>gzip = Popen('gzip',sh ell=True,stdin= dump.stdout,std out=PIPE)
Same here, but why don't you use the gzip functionality from the standard
library?

--
Freedom is always the freedom of dissenters.
(Rosa Luxemburg)
Jun 27 '08 #4
Sebastian "lunar" Wiesner wrote:
Aidan <aw****@gmail.c omat Dienstag 10 Juni 2008 07:21:
>TT wrote:
>>On Jun 10, 2:37 pm, Aidan <awe...@gmail.c omwrote:
Hi,

I'm having a bit of trouble with a python script I wrote, though I'm not
sure if it's related directly to python, or one of the other software
packages.. .

The situation is that I'm trying to create a system backup script that
creates an image of the system, filters the output though gzip, and then
uploads the data (via ftp) to a remote site.

The problem is that when I run the script from the command line, it
works as I expect it, but when it is run by cron I only get a 20 byte
file where the compressed image should be... does anyone have any idea
as to why this might be happening? Code follows

<code>

#!/usr/bin/python

from subprocess import PIPE, Popen
from ftplib import FTP

host = 'box'

filename = '%s.img.gz' % host
ftp_host = '192.168.1.250'
ftpuser, ftppass = 'admin', 'admin'
dest_dir = '/share/%s' % host

dump = Popen('dump 0uaf - /',shell=True,st dout=PIPE)
You should avoid the use of ``shell=True`` here and use a argument list
instead:

dump = Popen(['dump', '0uaf', '-', '/'], stdout=PIPE)

This results in an exception thrown if the executable doesn't exist. This
exception can be caught and handle for instance with the logging module.
thanks. That exception certainly would have helped me...
>>>gzip = Popen('gzip',sh ell=True,stdin= dump.stdout,std out=PIPE)

Same here, but why don't you use the gzip functionality from the standard
library?
is there a way I can create a gzip file-like object which can read the
output from the dump subprocess, and has a read method which outputs the
compressed data, which will not write to disk first? With the above
code python doesn't have to write the system image data to disk at all,
which helps when there is not enough disk space to hold an intermediate
image (at least, that is my understanding of it...).

I had a look at the gzip module, but eventually just fell back to using
the stdin and stdout of a gzip subprocess. I'd be interested to know
how it could be done using the python standard lib though.
Jun 27 '08 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
2165
by: David Abrahams | last post by:
I started having some weird problems with Python recently; they're so weird that I can't begin to explain them. All I can do is describe the symptoms and hope someone else has a clue. So here goes: FreeBSD 4.2, Python 2.2.2. I have a nightly cron job that downloads the boost cvs tarball from SourceForge and bunzip2s it. For about a year everything worked with no problems. About a month ago the download started getting truncated...
25
7753
by: Xah Lee | last post by:
Python Doc Problem Example: gzip Xah Lee, 20050831 Today i need to use Python to compress/decompress gzip files. Since i've read the official Python tutorial 8 months ago, have spent 30 minutes with Python 3 times a week since, have 14 years of computing experience, 8 years in mathematical computing and 4 years in unix admin and perl, i have quickly found the official doc: http://python.org/doc/2.4.1/lib/module-gzip.html
14
7141
by: Bill | last post by:
I've written a small program that, in part, reads in a file and parses it. Sometimes, the file is gzipped. The code that I use to get the file object is like so: if filename.endswith(".gz"): file = GzipFile(filename) else: file = open(filename) Then I parse the contents of the file in the usual way (for line in
0
361
by: Kurt B. Kaiser | last post by:
Patch / Bug Summary ___________________ Patches : 378 open ( +3) / 3298 closed (+34) / 3676 total (+37) Bugs : 886 open (-24) / 5926 closed (+75) / 6812 total (+51) RFE : 224 open ( +7) / 227 closed ( +7) / 451 total (+14) New / Reopened Patches ______________________
1
2711
by: symbioid | last post by:
Hello, I'm working on a project, and VMware has problems with suspending the virtual machine. We are accessing the machine through samba. However, when I suspend the VM, it stops the Samba service. The solution we hit upon was to run a script that checks, say, once a minute to determine whether the machine was active or suspended. If it's been longer than a minute, we restart the samba service.
1
5090
by: Steven Bethard | last post by:
I'm having trouble using the subprocess module on Windows when my command line includes special characters like "&" (ampersand):: .... stdout=subprocess.PIPE, .... stderr=subprocess.PIPE) "'y' is not recognized as an internal or external command,\r\noperable program or batch file.\r\n" As you can see, Windows is interpreting that "&" as separating two commands, instead of being part of the single argument...
4
7559
by: Stephen Cattaneo | last post by:
Hello all, I am attempting to execute an automated test (written in Python) via cron. I have to check the HOSTNAME variable as part of the test, oddly under cron the HOSTNAME environment variable is not in the os.environ dictionary. I know that cron runs in a subshell that does not have all of the normally set environment variables. HOSTNAME is not one of those variables, it is set even in cron's subshell. Why doesn't python get...
0
3477
by: Cameron Simpson | last post by:
On 17Aug2008 21:25, John Nagle <nagle@animats.comwrote: Because $HOSTNAME is a bash specific variable, set by bash but NOT EXPORTED! Like $0 and a bunch of other "private" variables, subprocesses do not inherit this value. From "man bash": Shell Variables The following variables are set by the shell:
0
9487
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10069
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
9884
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9735
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7285
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6556
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5168
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5324
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3828
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.