I have a large file that I would like to transform and then feed to a
function (psycopg2 copy_from) that expects a file-like object (needs
read and readline methods).
I have a class like so:
class GeneInfo():
def __init__(self):
#urllib.urlretr ieve('ftp://ftp.ncbi.nih.go v/gene/DATA/
gene_info.gz',"/tmp/gene_info.gz")
self.fh = gzip.open("/tmp/gene_info.gz")
self.fh.readlin e() #deal with header line
def _read(self,n=1) :
for line in self.fh:
if line=='':
break
line=line.strip ()
line=re.sub("\t-","\t",line )
rowvals = line.split("\t" )
yield "\t".join([rowvals[i] for i in
[0,1,2,3,6,7,8,9 ,10,11,12,14]]) + "\n"
def readline(self,n =1):
return self._read().ne xt()
def read(self,n=1):
return self._read().ne xt()
def close(self):
self.fh.close()
and I use it like so:
a=GeneInfo()
cur.copy_from(a ,"gene_info" )
a.close()
It works well except that the end of file is not caught by copy_from.
I get errors like:
psycopg2.extens ions.QueryCance ledError: COPY from stdin failed: error
during .read() call
CONTEXT: COPY gene_info, line 1000: ""
for a 1000 line test file. Any ideas what is going on?
Thanks,
Sean 6 3735
Sean Davis schrieb:
I have a large file that I would like to transform and then feed to a
function (psycopg2 copy_from) that expects a file-like object (needs
read and readline methods).
I have a class like so:
class GeneInfo():
def __init__(self):
#urllib.urlretr ieve('ftp://ftp.ncbi.nih.go v/gene/DATA/
gene_info.gz',"/tmp/gene_info.gz")
self.fh = gzip.open("/tmp/gene_info.gz")
self.fh.readlin e() #deal with header line
def _read(self,n=1) :
for line in self.fh:
if line=='':
break
line=line.strip ()
line=re.sub("\t-","\t",line )
rowvals = line.split("\t" )
yield "\t".join([rowvals[i] for i in
[0,1,2,3,6,7,8,9 ,10,11,12,14]]) + "\n"
def readline(self,n =1):
return self._read().ne xt()
def read(self,n=1):
return self._read().ne xt()
def close(self):
self.fh.close()
and I use it like so:
a=GeneInfo()
cur.copy_from(a ,"gene_info" )
a.close()
It works well except that the end of file is not caught by copy_from.
I get errors like:
psycopg2.extens ions.QueryCance ledError: COPY from stdin failed: error
during .read() call
CONTEXT: COPY gene_info, line 1000: ""
for a 1000 line test file. Any ideas what is going on?
I'm a bit lost why the above actually works - as _read() appears to be
re-created instead of re-used for each invocation, and thus can't work IMHO.
Anyway, I think the real problem is that you don't follow the
readline-protocol. it returns "" if there is no more line to read,
instead you raise a StopIteration
Diez
On Sep 10, 6:59*pm, Sean Davis <seand...@gmail .comwrote:
I have a large file that I would like to transform and then feed to a
function (psycopg2 copy_from) that expects a file-like object (needs
read and readline methods).
I have a class like so:
class GeneInfo():
* * def __init__(self):
* * * * #urllib.urlretr ieve('ftp://ftp.ncbi.nih.go v/gene/DATA/
gene_info.gz',"/tmp/gene_info.gz")
* * * * self.fh = gzip.open("/tmp/gene_info.gz")
* * * * self.fh.readlin e() #deal with header line
* * def _read(self,n=1) :
* * * * for line in self.fh:
* * * * * * if line=='':
* * * * * * * * break
* * * * * * line=line.strip ()
* * * * * * line=re.sub("\t-","\t",line )
* * * * * * rowvals = line.split("\t" )
* * * * * * yield "\t".join([rowvals[i] for i in
[0,1,2,3,6,7,8,9 ,10,11,12,14]]) + "\n"
* * def readline(self,n =1):
* * * * return self._read().ne xt()
* * def read(self,n=1):
* * * * return self._read().ne xt()
Each time readline() and read() call self._read() they are creating a
new generator. They then get one value from the newly-created
generator and then discard that generator. What you should do is
create the generator in __init__ and then use it in readline() and
read().
* * def close(self):
* * * * self.fh.close()
and I use it like so:
a=GeneInfo()
cur.copy_from(a ,"gene_info" )
a.close()
It works well except that the end of file is not caught by copy_from.
I get errors like:
psycopg2.extens ions.QueryCance ledError: COPY from stdin failed: error
during .read() call
CONTEXT: *COPY gene_info, line 1000: ""
for a 1000 line test file. *Any ideas what is going on?
I wonder whether it's expecting readline() and read() to return an
empty string at the end of the file instead of raising StopIteration.
On Sep 10, 10:52*pm, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
Sean Davis schrieb:
I have a large file that I would like to transform and then feed to a
function (psycopg2 copy_from) that expects a file-like object (needs
read and readline methods).
I have a class like so:
class GeneInfo():
* * def __init__(self):
* * * * #urllib.urlretr ieve('ftp://ftp.ncbi.nih.go v/gene/DATA/
gene_info.gz',"/tmp/gene_info.gz")
* * * * self.fh = gzip.open("/tmp/gene_info.gz")
* * * * self.fh.readlin e() #deal with header line
* * def _read(self,n=1) :
* * * * for line in self.fh:
* * * * * * if line=='':
* * * * * * * * break
* * * * * * line=line.strip ()
* * * * * * line=re.sub("\t-","\t",line )
* * * * * * rowvals = line.split("\t" )
* * * * * * yield "\t".join([rowvals[i] for i in
[0,1,2,3,6,7,8,9 ,10,11,12,14]]) + "\n"
* * def readline(self,n =1):
* * * * return self._read().ne xt()
* * def read(self,n=1):
* * * * return self._read().ne xt()
* * def close(self):
* * * * self.fh.close()
and I use it like so:
a=GeneInfo()
cur.copy_from(a ,"gene_info" )
a.close()
It works well except that the end of file is not caught by copy_from.
I get errors like:
psycopg2.extens ions.QueryCance ledError: COPY from stdin failed: error
during .read() call
CONTEXT: *COPY gene_info, line 1000: ""
for a 1000 line test file. *Any ideas what is going on?
I'm a bit lost why the above actually works - as _read() appears to be
re-created instead of re-used for each invocation, and thus can't work IMHO.
Each generator that's created reads a single line from the file
(self.fh), yields the result, and is then discarded; none of the
individual generator read more than one line from the file.
Anyway, I think the real problem is that you don't follow the
readline-protocol. it returns "" if there is no more line to read,
instead you raise a StopIteration
Diez
On Sep 11, 8:01*am, MRAB <goo...@mrabarn ett.plus.comwro te:
On Sep 10, 6:59*pm, Sean Davis <seand...@gmail .comwrote:
I have a large file that I would like to transform and then feed to a
function (psycopg2 copy_from) that expects a file-like object (needs
read and readline methods).
I have a class like so:
class GeneInfo():
* * def __init__(self):
* * * * #urllib.urlretr ieve('ftp://ftp.ncbi.nih.go v/gene/DATA/
gene_info.gz',"/tmp/gene_info.gz")
* * * * self.fh = gzip.open("/tmp/gene_info.gz")
* * * * self.fh.readlin e() #deal with header line
* * def _read(self,n=1) :
* * * * for line in self.fh:
* * * * * * if line=='':
* * * * * * * * break
* * * * * * line=line.strip ()
* * * * * * line=re.sub("\t-","\t",line )
* * * * * * rowvals = line.split("\t" )
* * * * * * yield "\t".join([rowvals[i] for i in
[0,1,2,3,6,7,8,9 ,10,11,12,14]]) + "\n"
* * def readline(self,n =1):
* * * * return self._read().ne xt()
* * def read(self,n=1):
* * * * return self._read().ne xt()
Each time readline() and read() call self._read() they are creating a
new generator. They then get one value from the newly-created
generator and then discard that generator. What you should do is
create the generator in __init__ and then use it in readline() and
read().
* * def close(self):
* * * * self.fh.close()
and I use it like so:
a=GeneInfo()
cur.copy_from(a ,"gene_info" )
a.close()
It works well except that the end of file is not caught by copy_from.
I get errors like:
psycopg2.extens ions.QueryCance ledError: COPY from stdin failed: error
during .read() call
CONTEXT: *COPY gene_info, line 1000: ""
for a 1000 line test file. *Any ideas what is going on?
I wonder whether it's expecting readline() and read() to return an
empty string at the end of the file instead of raising StopIteration.
Don't wonder; ReadTheFantasti cManual:
read( [size])
.... An empty string is returned when EOF is encountered
immediately. ...
readline( [size])
... An empty string is returned only when EOF is encountered
immediately.
On Sep 10, 7:54*pm, John Machin <sjmac...@lexic on.netwrote:
On Sep 11, 8:01*am, MRAB <goo...@mrabarn ett.plus.comwro te:
On Sep 10, 6:59*pm, Sean Davis <seand...@gmail .comwrote:
I have a large file that I would like to transform and then feed to a
function (psycopg2 copy_from) that expects a file-like object (needs
read and readline methods).
I have a class like so:
class GeneInfo():
* * def __init__(self):
* * * * #urllib.urlretr ieve('ftp://ftp.ncbi.nih.go v/gene/DATA/
gene_info.gz',"/tmp/gene_info.gz")
* * * * self.fh = gzip.open("/tmp/gene_info.gz")
* * * * self.fh.readlin e() #deal with header line
* * def _read(self,n=1) :
* * * * for line in self.fh:
* * * * * * if line=='':
* * * * * * * * break
* * * * * * line=line.strip ()
* * * * * * line=re.sub("\t-","\t",line )
* * * * * * rowvals = line.split("\t" )
* * * * * * yield "\t".join([rowvals[i] for i in
[0,1,2,3,6,7,8,9 ,10,11,12,14]]) + "\n"
* * def readline(self,n =1):
* * * * return self._read().ne xt()
* * def read(self,n=1):
* * * * return self._read().ne xt()
Each time readline() and read() call self._read() they are creating a
new generator. They then get one value from the newly-created
generator and then discard that generator. What you should do is
create the generator in __init__ and then use it in readline() and
read().
* * def close(self):
* * * * self.fh.close()
and I use it like so:
a=GeneInfo()
cur.copy_from(a ,"gene_info" )
a.close()
It works well except that the end of file is not caught by copy_from.
I get errors like:
psycopg2.extens ions.QueryCance ledError: COPY from stdin failed: error
during .read() call
CONTEXT: *COPY gene_info, line 1000: ""
for a 1000 line test file. *Any ideas what is going on?
I wonder whether it's expecting readline() and read() to return an
empty string at the end of the file instead of raising StopIteration.
Don't wonder; ReadTheFantasti cManual:
read( [size])
... An empty string is returned when EOF is encountered
immediately. ...
readline( [size])
*... An empty string is returned only when EOF is encountered
immediately.
Thanks. This was indeed my problem--not reading the manual closely
enough.
And the points about the iterator being re-instantiated were also
right on point. Interestingly, in this case, the code was working
because read() and readline() were still returning the next line each
time since the file handle was being read one line at a time.
Sean
On Sep 11, 9:23*am, Sean Davis <seand...@gmail .comwrote:
On Sep 10, 7:54*pm, John Machin <sjmac...@lexic on.netwrote:
On Sep 11, 8:01*am, MRAB <goo...@mrabarn ett.plus.comwro te:
On Sep 10, 6:59*pm, Sean Davis <seand...@gmail .comwrote:
I have a large file that I would like to transform and then feed toa
function (psycopg2 copy_from) that expects a file-like object (needs
read and readline methods).
I have a class like so:
class GeneInfo():
* * def __init__(self):
* * * * #urllib.urlretr ieve('ftp://ftp.ncbi.nih.go v/gene/DATA/
gene_info.gz',"/tmp/gene_info.gz")
* * * * self.fh = gzip.open("/tmp/gene_info.gz")
* * * * self.fh.readlin e() #deal with header line
* * def _read(self,n=1) :
* * * * for line in self.fh:
* * * * * * if line=='':
* * * * * * * * break
* * * * * * line=line.strip ()
* * * * * * line=re.sub("\t-","\t",line )
* * * * * * rowvals = line.split("\t" )
* * * * * * yield "\t".join([rowvals[i] for i in
[0,1,2,3,6,7,8,9 ,10,11,12,14]]) + "\n"
* * def readline(self,n =1):
* * * * return self._read().ne xt()
* * def read(self,n=1):
* * * * return self._read().ne xt()
Each time readline() and read() call self._read() they are creating a
new generator. They then get one value from the newly-created
generator and then discard that generator. What you should do is
create the generator in __init__ and then use it in readline() and
read().
* * def close(self):
* * * * self.fh.close()
and I use it like so:
a=GeneInfo()
cur.copy_from(a ,"gene_info" )
a.close()
It works well except that the end of file is not caught by copy_from.
I get errors like:
psycopg2.extens ions.QueryCance ledError: COPY from stdin failed: error
during .read() call
CONTEXT: *COPY gene_info, line 1000: ""
for a 1000 line test file. *Any ideas what is going on?
I wonder whether it's expecting readline() and read() to return an
empty string at the end of the file instead of raising StopIteration.
Don't wonder; ReadTheFantasti cManual:
read( [size])
... An empty string is returned when EOF is encountered
immediately. ...
readline( [size])
*... An empty string is returned only when EOF is encountered
immediately.
Thanks. *This was indeed my problem--not reading the manual closely
enough.
And the points about the iterator being re-instantiated were also
right on point. *Interestingly, in this case, the code was working
because read() and readline() were still returning the next line each
time since the file handle was being read one line at a time.
After further thought, do you actually need a generator? read() and
readline() could just call _read(), which would read a line from the
file and return the result or an empty string. Or the processing could
be done in readline() and read() just could call readline(). This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Jochen Daum |
last post by:
Hi,
I have to emulate a "file upload" to a Java Servlet which is done with
the class URLConnection.
The java source basically does
URLConnection conn = theurl.openConnection("https://somesite");
w = conn.getOutputStream();
w.write(...)
|
by: Kamilche |
last post by:
"""
Emulating Python inheritance manually.
By loading it from disk at run time, you can create new custom types
without programmer intervention, and reload them on demand,
without breaking anything. The only time programmer intervention
is required, is when new functions are added.
It works well for data... I can use 'chicken.color' to access that
attribute, but not for functions. I can't say 'chicken.printme()',
|
by: G.Esmeijer |
last post by:
Friends,
Want to read a textfile with characters that go above 7F ( ascii > 128) and
put the results line by line in a string to process it further
I used a streamreader to do so but the readline() methods in someway skipps
the characters > 127 .
I used the binary reader as well, same result.
string regel;
StreamReader sr = new StreamReader ( ImportFile, System.Yext.Unicode.UTF8);
|
by: Tim Bücker |
last post by:
Hello.
Is there a way to read a specified line in a textfile?
Something like TextReader.ReadTextLine(4);
It seems very odd to use
reader.ReadLine();
reader.ReadLine();
reader.ReadLine();
string wishedLine = reader.ReadLine();
|
by: Paulers |
last post by:
Hello,
I need to emulate keyboard strokes from a console application. The
console application monitors a textfile and when something is matched
in a text file I need the matched string outputted to the keyboard as
if someone was typing on the keyboard. can someone help me locate the
correct vb.net function to use?
Thanks!
| |
by: Prasanta |
last post by:
Hello,
Please cnay one can tell me how to read mail as formatted.... i have made
some code using that able to read but not able to serialize..... so am i need
to parse the HTML, or is there any class using that i can directly get the
full info ablt the mail, like TO,From,Subject,Body,Attachments,
Below is the code which i develop,
using System;
|
by: Ray |
last post by:
Hello,
What do you think about emulating private variables for a class this
way?
function Something() {
var private;
Something.prototype.getPrivate = function() { return private; }
Something.prototype.setPrivate = function(newPrivate) { private =
newPrivate; }
|
by: newsaboutgod |
last post by:
I think VB.NET drives some people crazy because some simple VB6 things
seem so hard.
Here is some VB6 code:
'Write CSV File
open "c:\test.csv" for output as #1
write#1, "1","2","3","4","5"
write#1, "1","2","3","4","5"
write#1, "1","2","3","4","5"
|
by: Horacius ReX |
last post by:
Hi, I have a text file like this;
1 -33.453579
2 -148.487125
3 -195.067172
4 -115.958374
5 -100.597841
6 -121.566441
7 -121.025381
8 -132.103507
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
| |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |