473,242 Members | 1,837 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,242 software developers and data experts.

Feeding data into MySQLdb LOAD DATA from Python

Is it possible to feed data into a LOAD DATA command in MySQL without
writing out the data to a file? It's possible to do this using the
MySQL command line and a UNIX FIFO, but that's kind of clunky.
I have to load a few gigabytes of data, and using INSERT takes
a whole day for each update.

And can this be done portably across UNIX and Windows? Thanks.

John Nagle
Nov 14 '07 #1
1 1761
John Nagle wrote:
Is it possible to feed data into a LOAD DATA command in MySQL without
writing out the data to a file? It's possible to do this using the
MySQL command line and a UNIX FIFO, but that's kind of clunky.
I have to load a few gigabytes of data, and using INSERT takes
a whole day for each update.

And can this be done portably across UNIX and Windows? Thanks.

John Nagle
Where is the data coming from? Two suggestions:

1) There is a MySQL option to not commit every insert, sorry I can't remember
what it is, but it REALLY speeds things up (especially if you have lots of
keys). Or drop all the keys load, the data, and recreate the keys. Sounds odd,
but it is a recommended method.
2) If data is coming from a file now, use LOAD DATA directly, don't go through
Python. You may even find writing to file and then loading it is very fast. I
recommend tab delimited as that seems to work well.

-Larry
Nov 14 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Dave Harrison | last post by:
Hi all, got a problem combinging mx and MySQLdb, when I build and install both for my Python2.1 install on a Solaris 9 box I can import mx fine, but importing MySQLdb causing python to core dump. ...
5
by: Chris Stromberger | last post by:
When issuing updates in mysql (in the console window), mysql will tell you if any rows matched and how many rows were updated (see below). I know how to get number of rows udpated using MySQLdb,...
1
by: Derek Fountain | last post by:
I was trying to use MySQLdb to connect to a database. All is OK, except I can't figure out how to get the details of an error. Suppose I try to connect to a non existant server, or with the wrong...
2
by: Tim Williams | last post by:
I'm trying to write a simple python program to access a MySQL database. I'm having a problem with using MySQLdb to get the results of a SQL command in a cursor. Sometimes the cursor.execute works,...
1
by: Steve | last post by:
Darwin steve.local 8.3.0 Darwin Kernel Version 8.3.0: Mon Oct 3 20:04:04 PDT 2005; root:xnu-792.6.22.obj~2/RELEASE_PPC Power Macintosh powerpc MacOSX 10.4.3 mysql Ver 14.7 Distrib 4.1.14, for...
11
by: Fred | last post by:
I hope someone can help me with the below problem... Thanks, Fred My enviroment: -------------------------- Slackware Linux 10.2 Python 2.4.2 MySql version 4.1.14
29
by: dyork | last post by:
When getting data from a database using the dbapi and an SQL query, how do you in general round trip the data? Especially date-time? An SQL datetime column translates nicely into a Python...
1
by: Sjoerd | last post by:
Hello! When I try to import the MySQLdb lib python generates an error: Traceback (most recent call last): File "<pyshell#0>", line 1, in <module> import MySQLdb File...
3
by: Lawrence D'Oliveiro | last post by:
In message <mailman.109.1216158151.922.python-list@python.org>, Cyril Bazin wrote: What's the error message? This should be in Apache's error_log file.
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.