mirroring files and data via http

[Steve Potter]

I'm working on a project to create a central administration interface
for several websites located on different physical servers.

You can think of the websites as a blog type application. My
administration application will be used to create new blog posts with
associated media (images, etc..)

So I am thinking to setting up a script on each of the sites to "phone
home" once per day and download any new posts and media files.

I am thinking of transmitting the post data in an xml format that
could then be decoded an recorded into the database on the slave
site. Are there any easy ways to encode a python dictionary to and
from xml?

For the media files I am thinking that the administration interface
would also provide an xml file with a list of all of the media files
required along with an http path to retrieve them and a checksum of
some sort to verify that they were downloaded correctly.

Basically I am trying to figure out if anything already exists to
perform some of these functions or if I have to make them from
scratch. I know none of it should be too hard, but I hate to re-
invent the wheel.

Thanks,

Steven Potter

[Matt Nordhoff]

Steve Potter wrote:

I'm working on a project to create a central administration interface
for several websites located on different physical servers.

You can think of the websites as a blog type application. My
administration application will be used to create new blog posts with
associated media (images, etc..)

So I am thinking to setting up a script on each of the sites to "phone
home" once per day and download any new posts and media files.

I am thinking of transmitting the post data in an xml format that
could then be decoded an recorded into the database on the slave
site. Are there any easy ways to encode a python dictionary to and
from xml?

For the media files I am thinking that the administration interface
would also provide an xml file with a list of all of the media files
required along with an http path to retrieve them and a checksum of
some sort to verify that they were downloaded correctly.

Basically I am trying to figure out if anything already exists to
perform some of these functions or if I have to make them from
scratch. I know none of it should be too hard, but I hate to re-
invent the wheel.

Thanks,

Steven Potter

It sounds like JSON would be perfect for this. For working with it in
Python, the simplejson module is popular:

<http://pypi.python.org/pypi/simplejson>
--

[Steve Potter]

On Jul 6, 8:19 pm, Matt Nordhoff <[e-mail]:

Steve Potter wrote:

I'm working on a project to create a central administration interface
for several websites located on different physical servers.

You can think of the websites as a blog type application. My
administration application will be used to create new blog posts with
associated media (images, etc..)

So I am thinking to setting up a script on each of the sites to "phone
home" once per day and download any new posts and media files.

I am thinking of transmitting the post data in an xml format that
could then be decoded an recorded into the database on the slave
site. Are there any easy ways to encode a python dictionary to and
from xml?

For the media files I am thinking that the administration interface
would also provide an xml file with a list of all of the media files
required along with an http path to retrieve them and a checksum of
some sort to verify that they were downloaded correctly.

Basically I am trying to figure out if anything already exists to
perform some of these functions or if I have to make them from
scratch. I know none of it should be too hard, but I hate to re-
invent the wheel.

Thanks,

Steven Potter

It sounds like JSON would be perfect for this. For working with it in
Python, the simplejson module is popular:

<http://pypi.python.org/pypi/simplejson>
--

Matt,

You are correct JSON would be much easier to deal with than xml. That
takes care of several of the problems.

Now it really just comes down to the file transfer from one server to
the other and a verification of some sort that the file transfer was
not corrupt. Is there a python module that takes care of file
downloads and verification?

Thanks,

Steve

[Matt Nordhoff]

Steve Potter wrote:

On Jul 6, 8:19 pm, Matt Nordhoff <[e-mail]:

>Steve Potter wrote:

>>I'm working on a project to create a central administration interface
for several websites located on different physical servers.
You can think of the websites as a blog type application. My
administration application will be used to create new blog posts with
associated media (images, etc..)
So I am thinking to setting up a script on each of the sites to "phone
home" once per day and download any new posts and media files.
I am thinking of transmitting the post data in an xml format that
could then be decoded an recorded into the database on the slave
site. Are there any easy ways to encode a python dictionary to and
from xml?
For the media files I am thinking that the administration interface
would also provide an xml file with a list of all of the media files
required along with an http path to retrieve them and a checksum of
some sort to verify that they were downloaded correctly.
Basically I am trying to figure out if anything already exists to
perform some of these functions or if I have to make them from
scratch. I know none of it should be too hard, but I hate to re-
invent the wheel.
Thanks,
Steven Potter

It sounds like JSON would be perfect for this. For working with it in
Python, the simplejson module is popular:

<http://pypi.python.org/pypi/simplejson>
--

Matt,

You are correct JSON would be much easier to deal with than xml. That
takes care of several of the problems.

Now it really just comes down to the file transfer from one server to
the other and a verification of some sort that the file transfer was
not corrupt. Is there a python module that takes care of file
downloads and verification?

Thanks,

Steve

Not automatically, but this should be easy to do yourself. You could
generate a simple JSON or plain text file listing URLs and hashes, and
then use urllib or urllib2 to download them, and md5, sha1 or (in Python
2.5) hashlib to verify them. All of those modules are in the standard
library.
--