I'm working on writing a program that will synchronize one database with
another. For the source database, we can just use the python sybase API;
that's nice and normal.
For the target database, unfortunately, the only interface we have access
to, is http+html based. There's same javascript involved too, but
hopefully we won't have to interact with that.
So, I've got Basic AUTH going with http, but now I'm faced with the
following questions, due to the fact that I need to pull some lists out of
HTML, and then make some changes via POST or so, again over HTTP:
1) Would I be better off just regex'ing the html I'm getting back? (I
suppose this depends on the complexity of the html received, eh?)
2) Would I be better off feeding the HTML into an HTML parser, and then
traversing that datastructure (is that really how it works?)?
3) When I retrieve stuff over http, it's clear that the web server is
sending some kind of odd gibberish, which the python urllib2 API is
passing on to me. In a packet trace, it looks like:
Date: Wed, 10 Nov 2004 01:09:47 GMT^M
Server: Apache/1.3.29 (Unix) (Red-Hat/Linux) mod_perl/1.23^M
Keep-Alive: timeout=15, max=98^M
Connection: Keep-Alive^M
Transfer-Encoding: chunked^M
Content-Type: text/html^M
^M
ef1^M
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">^M
<html>^M
<head>^M
....and so on. It seems to me that "ef1" should not be there. Is that
true? What -is- that nonsense? It's not the same string every time, and
it doesn't show up in a web browser.
Thanks!
7  2549 
> For the target database, unfortunately, the only interface we have access to, is http+html based. There's same javascript involved too, but
hopefully we won't have to interact with that.
Argl. That sounds like royal pain in somewhere...
So, I've got Basic AUTH going with http, but now I'm faced with the
following questions, due to the fact that I need to pull some lists out of
HTML, and then make some changes via POST or so, again over HTTP:
1) Would I be better off just regex'ing the html I'm getting back? (I
suppose this depends on the complexity of the html received, eh?)
2) Would I be better off feeding the HTML into an HTML parser, and then
traversing that datastructure (is that really how it works?)?
I personally would certainly go that way - the best thing IMHO would be to
make a dom-tree out of the html you then can work on with xpath. 4suite
might be good for that. While this seems a bit overengineered at first,
using xpath allows for pretty strong queries against your dom-tree so even
larger changes in the "interface" can be coped with. And writing htmlparser
based class isn't hard, either.
3) When I retrieve stuff over http, it's clear that the web server is
sending some kind of odd gibberish, which the python urllib2 API is
passing on to me. In a packet trace, it looks like:
Date: Wed, 10 Nov 2004 01:09:47 GMT^M
Server: Apache/1.3.29 (Unix) (Red-Hat/Linux) mod_perl/1.23^M
Keep-Alive: timeout=15, max=98^M
Connection: Keep-Alive^M
Transfer-Encoding: chunked^M
Content-Type: text/html^M
^M
ef1^M
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">^M
<html>^M
<head>^M
...and so on. It seems to me that "ef1" should not be there. Is that
true? What -is- that nonsense? It's not the same string every time, and
it doesn't show up in a web browser.
A webserver serving http doesn't care what you return - remember that http
can also be used to transfer binary data.
So the problem is not the apache, but whoever wrote that webapplication. Is
it cgi based?
For the ignoring part: Webbrowser tend to be very relaxed about html
documents format, otherwise a lot of the web would be "unsurfable". So I'm
not to astonished that they ignore that leading crap.
--
Regards,
Diez B. Roggisch
Forget about the last part - I just read Jp Calderone's reply after writing
mine - interesting. I've never heard of "chunked" encoding before.
--
Regards,
Diez B. Roggisch
On Wed, 10 Nov 2004 12:26:04 +0100, Diez B. Roggisch wrote: So, I've got Basic AUTH going with http, but now I'm faced with the
following questions, due to the fact that I need to pull some lists out of
HTML, and then make some changes via POST or so, again over HTTP:
1) Would I be better off just regex'ing the html I'm getting back? (I
suppose this depends on the complexity of the html received, eh?)
2) Would I be better off feeding the HTML into an HTML parser, and then
traversing that datastructure (is that really how it works?)?
I personally would certainly go that way - the best thing IMHO would be to
make a dom-tree out of the html you then can work on with xpath. 4suite
might be good for that. While this seems a bit overengineered at first,
using xpath allows for pretty strong queries against your dom-tree so even
larger changes in the "interface" can be coped with. And writing htmlparser
based class isn't hard, either.
This sounds interesting.
But if I use an XML parser to parse HTML instead of a dedicated HTML
parser, will I still get smart handling of unpaired tags? I'm not sure we
can count on getting 100% properly formed HTML...
> But if I use an XML parser to parse HTML instead of a dedicated HTML parser, will I still get smart handling of unpaired tags? I'm not sure we
can count on getting 100% properly formed HTML...
There should be html2dom parsers - after all, extending htmlparser to
generate dom shouldn't be to hard.
Googling turns up tidy - so you may want to feed your html through it
before: http://www.xml.com/pub/a/2004/09/08/pyxml.html
--
Regards,
Diez B. Roggisch
In article <pa****************************@dcs.nac.uci.edu> ,
Dan Stromberg <st******@dcs.nac.uci.edu> wrote: I'm working on writing a program that will synchronize one database with
another. For the source database, we can just use the python sybase API;
that's nice and normal.
[...]
1) Would I be better off just regex'ing the html I'm getting back? (I
suppose this depends on the complexity of the html received, eh?)
2) Would I be better off feeding the HTML into an HTML parser, and then
traversing that datastructure (is that really how it works?)?
I recommend you look at BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/
It is very forgiving of the typical affronts HTML writers put into their
code.
-M
--
Michael J. Fromberger | Lecturer, Dept. of Computer Science http://www.dartmouth.edu/~sting/ | Dartmouth College, Hanover, NH, USA
On Thu, 11 Nov 2004 15:05:27 -0500, Michael J. Fromberger wrote: In article <pa****************************@dcs.nac.uci.edu> ,
Dan Stromberg <st******@dcs.nac.uci.edu> wrote:
I'm working on writing a program that will synchronize one database with
another. For the source database, we can just use the python sybase API;
that's nice and normal.
[...]
1) Would I be better off just regex'ing the html I'm getting back? (I
suppose this depends on the complexity of the html received, eh?)
2) Would I be better off feeding the HTML into an HTML parser, and then
traversing that datastructure (is that really how it works?)?
I recommend you look at BeautifulSoup:
http://www.crummy.com/software/BeautifulSoup/
It is very forgiving of the typical affronts HTML writers put into their
code.
BeautifulSoup looks great.
Regrettably, I'm getting:
Traceback (most recent call last):
File "./netreo.py", line 130, in ?
soup.feed(html)
File "/Dcs/staff/strombrg/netreo/lib/BeautifulSoup.py", line 308, in feed
SGMLParser.feed(self, text)
File "/Web/lib/python2.3/sgmllib.py", line 94, in feed
self.rawdata = self.rawdata + data
TypeError: cannot concatenate 'str' and 'list' objects
....upon feeding some html into the "feed" method.
This is with python 2.3.4, on an RHEL 3 system.
Am I perhaps using a version of python that is too recent?
In article <pa****************************@dcs.nac.uci.edu> ,
Dan Stromberg <st******@dcs.nac.uci.edu> wrote: On Thu, 11 Nov 2004 15:05:27 -0500, Michael J. Fromberger wrote:
In article <pa****************************@dcs.nac.uci.edu> ,
Dan Stromberg <st******@dcs.nac.uci.edu> wrote: [...]
1) Would I be better off just regex'ing the html I'm getting back? (I
suppose this depends on the complexity of the html received, eh?)
2) Would I be better off feeding the HTML into an HTML parser, and then
traversing that datastructure (is that really how it works?)?
I recommend you look at BeautifulSoup:
http://www.crummy.com/software/BeautifulSoup/
It is very forgiving of the typical affronts HTML writers put into their
code.
BeautifulSoup looks great.
Regrettably, I'm getting:
Traceback (most recent call last):
File "./netreo.py", line 130, in ?
soup.feed(html)
File "/Dcs/staff/strombrg/netreo/lib/BeautifulSoup.py", line 308, in feed
SGMLParser.feed(self, text)
File "/Web/lib/python2.3/sgmllib.py", line 94, in feed
self.rawdata = self.rawdata + data
TypeError: cannot concatenate 'str' and 'list' objects
...upon feeding some html into the "feed" method.
This is with python 2.3.4, on an RHEL 3 system.
Am I perhaps using a version of python that is too recent?
Are you feeding a string to your parser, or a list? It wants a string;
it looks like maybe you're giving it a list.
-M
--
Michael J. Fromberger | Lecturer, Dept. of Computer Science http://www.dartmouth.edu/~sting/ | Dartmouth College, Hanover, NH, USA This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Jonas Galvez |
last post by:
Hi,
I'm trying to parse some binary data with regexes. It works well in
the latest Python build, but I need to run this on Python 1.5.2. The
binary data has a pattern like this:
...
|
by: daveyand |
last post by:
Hey guys,
I've spent abit of time searching th web and also this group and cant
seem to find a solution to my issue.
Basically i want to get the current url, and then replace http:// with...
|
by: Masa Ito |
last post by:
I am trying to capture the contents of a function with Regex. I am using
Expresso to test (nice - thanks for the great tool UltraPico!). I can
handle my own with single line regex's (I think).. ...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
| |
