I'm trying to write a web client script in python to log onto a web
page and pull some information off of it. The page has quite a few
behind the scenes http things going on that are making it difficult to
write the python script. I think if I could see the raw http data
that comes to my browser life would be much easier. Does anybody know
of programs that would run alongside my browser and show the http data
that is send back and forth?
I've looked at plugins like httpwatch and iewatch. httpwatch looks
good but it costs quite a bit of money, and the trial is not fully
functional. iewatch has a free trial, but they don't show cookie
info, which is important.
Does anybody know a good way of view the raw http?
Thanks for your help,
Ben 8 2231
"ben" <sa********@yah oo.com> wrote in message
news:56******** *************** ***@posting.goo gle.com... I'm trying to write a web client script in python to log onto a web page and pull some information off of it. The page has quite a few behind the scenes http things going on that are making it difficult to write the python script. I think if I could see the raw http data that comes to my browser life would be much easier. Does anybody know of programs that would run alongside my browser and show the http data that is send back and forth?
I've looked at plugins like httpwatch and iewatch. httpwatch looks good but it costs quite a bit of money, and the trial is not fully functional. iewatch has a free trial, but they don't show cookie info, which is important.
Does anybody know a good way of view the raw http?
Thanks for your help,
Ben
Try HTTPLook at www.httpsniffer.com . We've used it some on our current
project, to monitor SOAP message contents. I don't know about cookie
viewing, but it's free for 15-day (full featured) trial, less than $30 if
you want to keep it.
-- Paul
In article <56************ **************@ posting.google. com>, ben wrote: I'm trying to write a web client script in python to log onto a web page and pull some information off of it. The page has quite a few behind the scenes http things going on that are making it difficult to write the python script. I think if I could see the raw http data that comes to my browser life would be much easier. Does anybody know of programs that would run alongside my browser and show the http data that is send back and forth?
I've looked at plugins like httpwatch and iewatch. httpwatch looks good but it costs quite a bit of money, and the trial is not fully functional. iewatch has a free trial, but they don't show cookie info, which is important.
from urllib import urlopen
r = urlopen("http://www.python.org/")
print r.headers
You can also take a look at mozilla firefox and download the excellent
webdeveloper extension. It also allows you to view headers and lots of other
stuff.
Hth,
PterK
--
Peter van Kampen
pterk -- at -- datatailors.com sa********@yaho o.com (ben) writes: Does anybody know a good way of view the raw http?
I always just use a proxy server for that. www.junkbuster.org has a
free and convenient one that has options for logging the http stream. sa********@yaho o.com (ben) wrote in message news:<56******* *************** ****@posting.go ogle.com>... I'm trying to write a web client script in python to log onto a web page and pull some information off of it. The page has quite a few behind the scenes http things going on that are making it difficult to write the python script. I think if I could see the raw http data that comes to my browser life would be much easier. Does anybody know of programs that would run alongside my browser and show the http data that is send back and forth?
I've looked at plugins like httpwatch and iewatch. httpwatch looks good but it costs quite a bit of money, and the trial is not fully functional. iewatch has a free trial, but they don't show cookie info, which is important.
Does anybody know a good way of view the raw http?
Thanks for your help,
Ben
TCPwatch should do what you want. It's a proxy serverso you'll have to
point your browser at that - it shows headers and page content
etc.....
If you're handling cookies then ClientCookie (or cookielib) could well
do what you want without you having to worry about it (if you're using
python why do you need a browser - why can't you just print the
headers ?)
Regards,
Fuzzy http://www.voidspace.org.uk/atlantib...thonutils.html sa********@yaho o.com (ben) wrote in message news:<56******* *************** ****@posting.go ogle.com>... I'm trying to write a web client script in python to log onto a web page and pull some information off of it. The page has quite a few behind the scenes http things going on that are making it difficult to write the python script. I think if I could see the raw http data that comes to my browser life would be much easier. Does anybody know of programs that would run alongside my browser and show the http data that is send back and forth?
I've looked at plugins like httpwatch and iewatch. httpwatch looks good but it costs quite a bit of money, and the trial is not fully functional. iewatch has a free trial, but they don't show cookie info, which is important.
Does anybody know a good way of view the raw http?
Thanks for your help,
Ben
I downloaded a "sniffer" in Python. Let me know if you are interested
give www.debugbar.com a bash
the trial version is fully functional and it is cheap sa********@yaho o.com (ben) writes: I'm trying to write a web client script in python to log onto a web page and pull some information off of it. The page has quite a few behind the scenes http things going on that are making it difficult to write the python script. I think if I could see the raw http data that comes to my browser life would be much easier. Does anybody know of programs that would run alongside my browser and show the http data that is send back and forth?
I've looked at plugins like httpwatch and iewatch. httpwatch looks good but it costs quite a bit of money, and the trial is not fully functional. iewatch has a free trial, but they don't show cookie info, which is important.
Does anybody know a good way of view the raw http?
[...]
from http://wwwsearch.sourceforge.net/bits/GeneralFAQ.html :
| I want to see what my web browser is doing, but standard network
| sniffers like ethereal or netcat (nc) don't work for HTTPS. How do I
| sniff HTTPS traffic?
|
| Three good options:
|
| Mozilla plugin: livehttpheaders .
| ieHTTPHeaders does the same for MSIE.
| Use lynx -trace, and filter out the junk with a script.
|
| I'm told you can also use a proxy like proxomitron (never tried it
| myself). There's also a commercial MSIE plugin.
URLs: http://www.ethereal.com/ http://livehttpheaders.mozdev.org/ http://www.blunck.info/iehttpheaders.html http://lynx.browser.org/ http://www.proxomitron.info/ http://www.simtec.ltd.uk/
There are plugins that show what session & permanent cookies your
browser has stored, too. Both Mozilla and Konqueror (& Safari and
Opera too, I guess?) do this as standard, but for IE: http://www.codeproject.com/shell/coo...0&select=21591
John
Ben,
I am using iewatch (iewatch.com) quite a bit and it works great in
showing me the cookie information, both for request cookies and
response cookies (look for the Cookie and Set-Cookie headers). There
is a fully functional 21 day trial version available and it only costs
a fraction of what httpwatch is charging.
Michael sa********@yaho o.com (ben) wrote in message news:<56******* *************** ****@posting.go ogle.com>... I'm trying to write a web client script in python to log onto a web page and pull some information off of it. The page has quite a few behind the scenes http things going on that are making it difficult to write the python script. I think if I could see the raw http data that comes to my browser life would be much easier. Does anybody know of programs that would run alongside my browser and show the http data that is send back and forth?
I've looked at plugins like httpwatch and iewatch. httpwatch looks good but it costs quite a bit of money, and the trial is not fully functional. iewatch has a free trial, but they don't show cookie info, which is important.
Does anybody know a good way of view the raw http?
Thanks for your help,
Ben This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: lori3laz |
last post by:
How do you disable the right click>view source option on web pages so people
can't view your coding and copy it? What's the html I need to include in my
website to utilize this feature?
Thank you.
|
by: Tomas Vera |
last post by:
Hello All,
I'm posting this for the benefit of the search engines.
I've been developing web apps on my computer for about 18 months using VS2002.
This morning, I was showing a colleague how to set up a WebApp project. All went
well. I compiled the project then opened a browser and tried to navigate to the
page and I got the message listed below.
I did some newsgroup browsing and found that since I had both v1.0.3705 and
|
by: Brad |
last post by:
I have an intranet app that has just started sporadically getting the
following error
"The viewstate is invalid for this page and might be corrupted."
By sproadic I mean 3-4 times during the past two days out of 100's of hits.
The error just started yesterday and this app has been running for quite
some time without this error and has not been updated just before or after
the error started.
Reseaching this error I've checked the...
|
by: Paolo Pignatelli |
last post by:
I have an aspx code behind page that goes something like this in the HTML
view:
<asp:HyperLink id=HyperLink1 runat="server"
NavigateUrl='<%#"mailto:" &
DataBinder.Eval(Container.DataItem,"StoreEmail") & "&Subject=" &
DataBinder.Eval(Container.DataItem,"ProductName")
....
|
by: Zack Sessions |
last post by:
Has anyone tried to create a SQL7 view using the CREATE VIEW command
and ADO.NET? If so, is there a trick in trapping a SQL error when
trying to create the view? I have a VB.NET app that, amoung other
things, can create views based on existing tables and a table of
column name equivalents. If I accidently introduce a column name
equivalent that is the same as another column, I end up with a CREATE
VIEW SQL statement that is in error, if...
| |
by: Jacob Donajkowski |
last post by:
Once the user logs in I want to have the users switch from the Roster
View to the Profile View and enter their profile infomation and save
it. Then the next time they login and go to the Profile View I want
the form populated from there profile on the sql server. The code to
save the profile works fine. But when the user logs back in they data
doesn't load back to the form. The multiview is located inside the
LoginView's Logged-In View ....
|
by: CBidwill |
last post by:
Hello, I'm a ASP.NET developer with a little experience in IIS but I'm having difficulties getting a little ASP.NET application I wrote up on my home intranet.
Here's the details:
- I can view the site fine where the application is hosted by browsing http://<machine name>/<application virtual directory>
- When going to http://<machine name>/<application virtual directory> on a remote computer, I get paged cannot be displayed.
- From the...
|
by: =?Utf-8?B?UmljaA==?= |
last post by:
I am using a web browser control in my program for bringing up files/folders
in Windows Explorer on a given user's workstatino -- searching special
files/folders on the company server. This works fine -- I call a default url
to the special folder on the server and up comes Windows Explorer on the
User's workstation and displays all the designated folders/files. My problem
is that the default view of Windows explorer when it comes up is in...
|
by: Arno R |
last post by:
Hi all.
When I need to search for pictures, I always have too choose thumbnail-view manually.
Is it possible to open the common dialog in thumbnail-view programmatically?
Example ??
At the moment I am using Ken Getz' code. at http://www.mvps.org/access/api/api0001.htm
Thanks
Arno R
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
| |
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |