472,799 Members | 1,390 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,799 software developers and data experts.

mysql bulk insert from python

hi

my news carrier didn't have any mysql specific news groups - so
apologize in advance if this isn't exactly the proper group for these
question(s) ...

anyway - i'm trying to figure out how to perform a bulk insert of data
from a text file into a table in a mysql database from the python
programming language. i have lots of experience with ms sql server and
there we used bcp and then dts packages, but i can't seem to find the
analogous method for mysql - and would appreciate any tips/insight the
group can offer.

fwiw - the data files contain on the order of 100,000 + records
(sometimes approx. 1 million), so anything but a 'bulk' solution
probably isn't practical.

also - i found some reference to a 'Loader' python module, but it
seems to related to MaxDB and SAP. i'm not sure what that is - and
would really prefer a more generic solution.

thanks
dave

Jul 18 '05 #1
1 6701
Dave Ekhaus wrote:
hi

my news carrier didn't have any mysql specific news groups - so
apologize in advance if this isn't exactly the proper group for these
question(s) ...

anyway - i'm trying to figure out how to perform a bulk insert of
data from a text file into a table in a mysql database from the python
programming language. i have lots of experience with ms sql server and
there we used bcp and then dts packages, but i can't seem to find the
analogous method for mysql - and would appreciate any tips/insight the
group can offer.

fwiw - the data files contain on the order of 100,000 + records
(sometimes approx. 1 million), so anything but a 'bulk' solution
probably isn't practical.

also - i found some reference to a 'Loader' python module, but it
seems to related to MaxDB and SAP. i'm not sure what that is - and
would really prefer a more generic solution.

thanks
dave

The MySQL command LOAD DATA INFILE
(http://dev.mysql.com/doc/mysql/en/LOAD_DATA.html) might be what you are
looking for, no need for python here, but you can call this command from
any python script that uses the MySQLDB module
(http://sourceforge.net/projects/mysql-python)
HTH,
Wolfram

PS: Maybe news.gmane.org has some mysql-lists that you can try
Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: duikboot | last post by:
Hi all, I'm trying to export a view tables from a Oracle database to a Mysql database. I create insert statements (they look alright), but it all goes wrong when I try to execute them in Mysql,...
11
by: grumfish | last post by:
I'm trying to add a row to a MySQL table using insert. Here is the code: connection = MySQLdb.connect(host="localhost", user="root", passwd="pw", db="japanese") cursor = connection.cursor()...
2
by: php newbie | last post by:
Hello, I am trying to load a simple tab-delimited data file to SQL Server. I created a format file to go with it, since the data file differs from the destination table in number of columns. ...
7
by: iqbal | last post by:
Hi all, We have an application through which we are bulk inserting rows into a view. The definition of the view is such that it selects columns from a table on a remote server. I have added the...
6
by: pk | last post by:
Sorry for the piece-by-piece nature of this post, I moved it from a dormant group to this one and it was 3 separate posts in the other group. Anyway... I'm trying to bulk insert a text file of...
16
by: Philip Boonzaaier | last post by:
I want to be able to generate SQL statements that will go through a list of data, effectively row by row, enquire on the database if this exists in the selected table- If it exists, then the colums...
1
by: jlee | last post by:
I'm pretty much a newbie on mysql, and I need some help. I am running mysql Ver 12.22 Distrib 4.0.24, for portbld-freebsd5.4 (i386) on a server hosting an active website. The site's developer...
1
by: The|Godfather | last post by:
Hello, I have a "slight" problem with the mySQL C API. I want to bind vectors for output/input prepared statements. However, the C API does NOT support such data binding. Is there ANY other way to...
1
by: Ike | last post by:
Recently, I began using a different MySQL verver (i.e. different machine as well as different version#, going from 4.12a to 4.1.9 max). The following query used to work: select firstname,...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 2 August 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
0
by: erikbower65 | last post by:
Using CodiumAI's pr-agent is simple and powerful. Follow these steps: 1. Install CodiumAI CLI: Ensure Node.js is installed, then run 'npm install -g codiumai' in the terminal. 2. Connect to...
0
linyimin
by: linyimin | last post by:
Spring Startup Analyzer generates an interactive Spring application startup report that lets you understand what contributes to the application startup time and helps to optimize it. Support for...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Sept 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
14
DJRhino1175
by: DJRhino1175 | last post by:
When I run this code I get an error, its Run-time error# 424 Object required...This is my first attempt at doing something like this. I test the entire code and it worked until I added this - If...
0
by: Rina0 | last post by:
I am looking for a Python code to find the longest common subsequence of two strings. I found this blog post that describes the length of longest common subsequence problem and provides a solution in...
5
by: DJRhino | last post by:
Private Sub CboDrawingID_BeforeUpdate(Cancel As Integer) If = 310029923 Or 310030138 Or 310030152 Or 310030346 Or 310030348 Or _ 310030356 Or 310030359 Or 310030362 Or...
0
by: lllomh | last post by:
Define the method first this.state = { buttonBackgroundColor: 'green', isBlinking: false, // A new status is added to identify whether the button is blinking or not } autoStart=()=>{
0
by: Mushico | last post by:
How to calculate date of retirement from date of birth

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.