473,230 Members | 1,511 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,230 software developers and data experts.

Curl - resume http download - Apache log files. gz

I am writing a Php script to run on my home PC (Windows) that downloads
an Apache access log file and inserts new entries into a database.. The
only way I can access these log files is through a cPanel URL that
delivers a gzipped copy of the log file with all entires for the current
month (up to the current date and time).

I want to run this script once a day (at night) and would like to
download the gzipped file by resuming from where I left off the day
before (instead of always downloading the entire gzipped file).

I have tried using CURL with the option - curl_setopt($ch,
CURLOPT_RESUME_FROM, $prev_offset) - but that always restarts the
download from the beginning. ($prev_offset is the previous days file size).

I'm not sure what the structure of a gzip file is, so I don't even know
if it is possible to resume a gzip file download when the file has been
rebuilt in the mean time (cPanel creates a gzipped file with the same
name from the raw log files every time you access that page in cPanel).
My guess is that this is probably not possible. ... But .... I am
trying to make the resume work to find out for myself and I can not make
the resume work.

The bottom line is that I am trying to avoid downloading the entire
gzipped log file every time I need to download it.

As an alternate, I have been successful using gzopen, and then I can
gzseek to the previous offset, but I am unable to use a secure URL
(https) with the Zlib functions (gzopen) with my Windows version of Php
(the username and password have to be included in the cPanel URL to
access the log file, so I'd like to use SSL.)

(I have started another thread to figure out how to get a Php Windows
binary with https and ftps as registered streams (built with
--with-openssl in the ./configure command), but so far I have gotten no
replies).

Question: When using gzread, where does the decompression take place -
on the server where the file is located, or on my PC after the
compressed data is downloaded? Does gzseek work on a compressed file,
or must it be uncompressed on the server first? The point being; when I
use gzopen and gzread, am I downloading compressed or uncompressed data
from the server where the log files are located?

I can use either method, but want to meet two requirements.
1. I do not want to download redundant (old) data every day (I want to
resume where I left off)
... and ...
2. I want to send the request for the log file (containing a username
and password) using SSL.

I would appreciate any help or insight.

--
*****************************
Chuck Anderson • Boulder, CO
http://www.CycleTourist.com
Integrity is obvious.
The lack of it is common.
*****************************
Oct 11 '05 #1
0 3016

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Haluk Durmus | last post by:
Hello I checked out openssl,mm,apr,apr-util,apache 2,curl,libxml and php from cvs. php couse an ERROR I did the following steps:
3
by: Chris Fortune | last post by:
# uname -a Linux stargate.mxc-online.net 2.4.20-021stab022.2.777-smp #1 SMP Wed Jul 28 17:12:37 MSD 2004 i686 i686 i386 GNU/Linux I recompiled PHP with mcrypt, openssl, and curl phpinfo():...
6
by: benji | last post by:
I have set up a system to download datafeeds in pain text or zipped. The download part of this system uses the curl extension to download the files. All was well when I tested it with various...
0
by: axlq | last post by:
While trying to learn the ins and outs of the php CURL library, I decided to write a php script that posts a form on the Chicago Board of Options (CBOE) web site, which returns an ASCII text file. ...
4
by: BinnyVA | last post by:
Hi, I am using PHP 5.1.2 with curl enabled. But whenever I try to use curl to fetch a url, it fails - 'curl_exec()' returns nothing. But if I try to execute the same file in CLI - like 'php...
4
by: Terry | last post by:
I'm using curl to invoke a php script on the same site/server. It works great, but if I call it again while it's still running, nothing happens. Why? Can that be fixed? Why use curl? To make...
3
by: Matthias Leopold | last post by:
hi i've got problems getting php4 scripts which use curl to work on RHEL4 (Nahant Update 4) and CentOS 4.4. when the script is accessed the browser "hangs", after a couple of minutes i'm...
0
by: xerc | last post by:
I am trying to create a generic function I can call to download all files from a single remote FTP directory -- using CURL. I want to multi-thread it, but need to get the single thread functionality...
0
by: Floris | last post by:
I'm using libcurl to download some content from another webpage. For reasons previously unknown to me all calls are very slow to the point of usual timeout. Now i found curl_getinfo() and retrieved...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, youll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.