Hi,
R language has very high-level IO functions, its read.table can read a
total .csv file and recogonize the types of each column. write.table can
do the reverse.
R's MySQL interface has high-level functions, too, e.g. dbWriteTable can
automatically build a MySQL table and write a table of R data
into it.
Is there any python packages do similar things?
-York 8 2175
York
Short answer: yes
We use python and R at work, and in general you will find python syntax a
little cleaner for functionality they have in common. R is better for
some of the more hard-wired stats stuff, though.
On Mon, 19 Sep 2005 20:04:37 +0200, York <yo*******@gmai l.com> wrote: Hi,
R language has very high-level IO functions, its read.table can read a total .csv file and recogonize the types of each column. write.table can do the reverse.
R's MySQL interface has high-level functions, too, e.g. dbWriteTable can automatically build a MySQL table and write a table of R data into it.
Is there any python packages do similar things?
-York
While it may "attempt" to recognize the types, it in fact cannot
be more correct than the programmer. Example:
data="""0X1E04 111"""
That "looks" lile a hex and an int. But wait. What if it is
instead two strings?
In Python you can easily write a class with a interator that can
read the data from the file/table and return the PROPER data types
as lists, tuples, or dictionaries that are easy to manipulate.
-Larry Bates
York wrote: Hi,
R language has very high-level IO functions, its read.table can read a total .csv file and recogonize the types of each column. write.table can do the reverse.
R's MySQL interface has high-level functions, too, e.g. dbWriteTable can automatically build a MySQL table and write a table of R data into it.
Is there any python packages do similar things?
-York
Caleb Hattingh wrote: York
Short answer: yes
Brilliant! and what are they?
We use python and R at work, and in general you will find python syntax a little cleaner for functionality they have in common. R is better for some of the more hard-wired stats stuff, though.
I love python. However, as a biologist, I like some high-levels
functions in R. I don't want to spend my time on parse a data file. Then
in my python script, I call R to read data file and write them into an
MySQL table. If python can do this easily, I don't need R at all.
Cheers,
-York On Mon, 19 Sep 2005 20:04:37 +0200, York <yo*******@gmai l.com> wrote:
Hi,
R language has very high-level IO functions, its read.table can read a total .csv file and recogonize the types of each column. write.table can do the reverse.
R's MySQL interface has high-level functions, too, e.g. dbWriteTable can automatically build a MySQL table and write a table of R data into it.
Is there any python packages do similar things?
-York
Your are right, a program cannot be smarter than its programmer. However
I need a program to parse any table-format data files offered by user. R
offer such a function, I hope python such a function too.
-York While it may "attempt" to recognize the types, it in fact cannot be more correct than the programmer. Example:
data="""0X1E04 111"""
That "looks" lile a hex and an int. But wait. What if it is instead two strings?
In Python you can easily write a class with a interator that can read the data from the file/table and return the PROPER data types as lists, tuples, or dictionaries that are easy to manipulate.
-Larry Bates
York wrote:
Hi,
R language has very high-level IO functions, its read.table can read a total .csv file and recogonize the types of each column. write.table can do the reverse.
R's MySQL interface has high-level functions, too, e.g. dbWriteTable can automatically build a MySQL table and write a table of R data into it.
Is there any python packages do similar things?
-York
York a écrit :
(snip) I love python. However, as a biologist, I like some high-levels functions in R. I don't want to spend my time on parse a data file.
http://www.python.org/doc/current/lib/module-csv.html
Then in my python script, I call R to read data file and write them into an MySQL table. If python can do this easily, I don't need R at all.
So you don't need R at all.
It's so easy (using csv module), no need to build in.
You can wrap in a class if you want to make even easier.
Same can be done for tables from SQL database.
import csv
fp=open(r'C:\te st.txt', 'r')
#
# test.txt contains:
#
# "record","value 1","value2"
# "1","2","3"
# "2","4","5"
# "3","6","7"
table=csv.DictR eader(fp)
for record in table:
#
# Record is a dictionary with keys as fieldnames
# and values of the data in each record
#
print "record #=%s, value1=%s, value2=%s" % \
(record['record'],record['value1'],record['value2'])
fp.close()
-Larry Bates
York wrote: Your are right, a program cannot be smarter than its programmer. However I need a program to parse any table-format data files offered by user. R offer such a function, I hope python such a function too.
-York
While it may "attempt" to recognize the types, it in fact cannot be more correct than the programmer. Example:
data="""0X1E04 111"""
That "looks" lile a hex and an int. But wait. What if it is instead two strings?
In Python you can easily write a class with a interator that can read the data from the file/table and return the PROPER data types as lists, tuples, or dictionaries that are easy to manipulate.
-Larry Bates
York wrote:
Hi,
R language has very high-level IO functions, its read.table can read a total .csv file and recogonize the types of each column. write.table can do the reverse.
R's MySQL interface has high-level functions, too, e.g. dbWriteTable can automatically build a MySQL table and write a table of R data into it.
Is there any python packages do similar things?
-York
On Mon, 19 Sep 2005, Bruno Desthuilliers wrote: York a écrit : (snip)
I love python. However, as a biologist, I like some high-levels functions in R. I don't want to spend my time on parse a data file.
http://www.python.org/doc/current/lib/module-csv.html
Then in my python script, I call R to read data file and write them into an MySQL table. If python can do this easily, I don't need R at all.
So you don't need R at all.
Did you even read the OP's post? Specifically, this bit:
R language has very high-level IO functions, its read.table can read a
total .csv file and recogonize the types of each column.
^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^ ^^^^^^^^^
Python's csv module gives you tuples of strings; it makes no effort to
recognise the types of the data. AFAIK, python doesn't have any IO
facilities like this.
Larry's point that automagical type detection is risky because it can make
mistakes is a good one, but that doesn't mean that magic is useless - on
the contrary, for the majority of cases, it works fine, and is extremely
convenient.
The good news is that it's reasonably easy to write such a function: you
just need a function 'type_convert' which takes a string and returns an
object of the right type; then you can do:
import csv
def read_table(f):
for row in csv.reader(f):
yield map(type_conver t, row)
This is a very, very rough cut - it doesn't do comment stripping, skipping
blank lines, dealing with the presence of a header line or the use of
different separators, etc, but all that's pretty easy to add. Also, note
that this returns an iterator rather than a list; use list(read_table (f))
if you want an actual list, or change the implementation of the function.
type_convert is itself fairly simple:
def _bool(s): # helper method for booleans
s = s.lower()
if (s == "true"): return True
elif (s == "false"): return False
else: raise ValueError, s
types = (int, float, complex, _bool, str)
def type_convert(s) :
for type in types:
try:
return type(s)
except ValueError:
pass
raise ValueError, s
This whole thing isn't quite as sophisticated as R's table.convert; R
reads the whole table in, then tries to find a type for each column which
will fit all the values in that column, whereas i do each cell
individually. Again, it wouldn't be too hard to do this the other way
round.
Anyway, hope this helps. Bear in mind that there are python bindings for
the R engine, so you could just use R's version of read.table in python.
tom
--
Don't trust the laws of men. Trust the laws of mathematics.
Thank you, Tom.
-York
Tom Anderson wrote: On Mon, 19 Sep 2005, Bruno Desthuilliers wrote:
York a écrit : (snip)
I love python. However, as a biologist, I like some high-levels functions in R. I don't want to spend my time on parse a data file. http://www.python.org/doc/current/lib/module-csv.html
Then in my python script, I call R to read data file and write them into an MySQL table. If python can do this easily, I don't need R at all.
So you don't need R at all.
Did you even read the OP's post? Specifically, this bit:
R language has very high-level IO functions, its read.table can read a total .csv file and recogonize the types of each column. ^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^ ^^^^^^^^^
Python's csv module gives you tuples of strings; it makes no effort to recognise the types of the data. AFAIK, python doesn't have any IO facilities like this.
Larry's point that automagical type detection is risky because it can make mistakes is a good one, but that doesn't mean that magic is useless - on the contrary, for the majority of cases, it works fine, and is extremely convenient.
The good news is that it's reasonably easy to write such a function: you just need a function 'type_convert' which takes a string and returns an object of the right type; then you can do:
import csv
def read_table(f): for row in csv.reader(f): yield map(type_conver t, row)
This is a very, very rough cut - it doesn't do comment stripping, skipping blank lines, dealing with the presence of a header line or the use of different separators, etc, but all that's pretty easy to add. Also, note that this returns an iterator rather than a list; use list(read_table (f)) if you want an actual list, or change the implementation of the function.
type_convert is itself fairly simple:
def _bool(s): # helper method for booleans s = s.lower() if (s == "true"): return True elif (s == "false"): return False else: raise ValueError, s
types = (int, float, complex, _bool, str)
def type_convert(s) : for type in types: try: return type(s) except ValueError: pass raise ValueError, s
This whole thing isn't quite as sophisticated as R's table.convert; R reads the whole table in, then tries to find a type for each column which will fit all the values in that column, whereas i do each cell individually. Again, it wouldn't be too hard to do this the other way round.
Anyway, hope this helps. Bear in mind that there are python bindings for the R engine, so you could just use R's version of read.table in python.
tom This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Steven O. |
last post by:
I am seeking some kind of tool that I can use for GUI prototyping. I
know how to use Visual Basic, but since a lot of software is being
coded in Java or C++, I'd like to learn a Java or C++ -based tool.
Back when I took my Java and C++ classes (two or three years ago), the
available tools -- at least the ones I could find -- were still not as
easy, not as "drag-and-drop", as Visual Basic. Has that changed?
Is there some software out...
|
by: jfixsen |
last post by:
Hello!
Oracle 9.2.0.4
SunOS pchi-db01 5.8 Generic_108528-19 sun4u sparc
SUNW,Ultra-EnterpriseSystem = SunOS
Node = pchi-db01
Release = 5.8
KernelID = Generic_108528-19
Machine = sun4u
BusType = <unknown>
|
by: Lorenzo J. Lucchini |
last post by:
My code contains this declaration:
: typedef union {
: word Word;
: struct {
: byte Low;
: byte High;
: } Bytes;
: } reg;
|
by: Durgesh Sharma |
last post by:
Hi All,
Pleas help me .I am a starter as far as C Language is concerned .
How can i Right Trim all the white spaces of a very long (2000 chars)
Charecter string ( from the Right Side ) ? or how can i make a fast
Right Trim Function in c,using Binary search kind of fast algorithm ?
Offcourse...I can use the classical approach too.
like :
Start from the right most charecter of the string to the left of the
|
by: qushui_chen |
last post by:
I store the MSSqlServer is Nvarchar(unicode),
the store data as "我就是我文本",
How can i decode in C#?
| |
by: johnb41 |
last post by:
In my app, I need to open up a multipage tiff file, and also display
it's thumbnail images IN HIGH QUALITY. (High Quality meaning
anti-aliased, and looking good; not rough and pixely) The thumbnail
images are displayed in a ListView control. (I go through each page of
the file, create a thumbnail of it, and put it into an ImageList. Then
i hook that imagelist up to the ListView.)
It works fine, but it is VERY slow. Creating and...
|
by: dembla |
last post by:
Hey Frnds can anyone help me in this
i need a program in 'c'
PROGRAM to print NxN Matrix
9 1 8 1 2 3
2 7 3 as 4 5 6
6 4 5 7 8 9
in sorted form
|
by: rdudejr |
last post by:
Hi all,
Ive got a database approx 350 GB in which Im getting very high Time
waited for prefetch. This is directly out of the snapshot for the db
(these are for the entire database I assume as I pulled it out of get
snapshot for all on {dbname})
Total buffer pool read time (milliseconds) = 45660639
Total buffer pool write time (milliseconds)= 42128058
Total elapsed asynchronous read time = 33856320
|
by: adamscybot |
last post by:
Here is the site in question:
http://www.sws.vxcomputers.com/h2k/
Basically, on the left, you'll see a roster (them images) and you have to click the "Counter-Strike" link for it to load the images/names. However, I overcome this by executing what is usually the OnClick event on the OnLoad event instead. This works great in firefox - meaning there is no need to click the link to see the
proper images and not the filler images - it just...
|
by: Zach |
last post by:
After having taken a looong break from VB (last used 6.0), I started
getting back into programming again, this time with VS 2005. I began
reading up on VB.NET from various sources (books, internet, etc.) and
found out about the CLR and how all of the .NET languages access it, the
major difference being the syntax and structure of the individual
languages. What I'm wondering, since VB.NET is obviously easier to
learn/use than C#.NET and...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
| |
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |