472,805 Members | 3,592 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,805 software developers and data experts.

How to speed up the import of 400,000+ records

jj
It's taking forever to upload 400,000 records to the database through
access/odbc, and I've tried phpMyAdmin's interface but it seems to timeout
during import of a CSV file. Is there a better way to import 400,000+
records?

Jul 19 '05 #1
7 9212
jj
Thanks that gave me an idea. I'm importing all the records to my local
MySQL database and will create an extract of that which will format it like
you said. Then take that .sql file and import it onto the server. We'll
see...
"Aggro" <sp**********@yahoo.com> wrote in message
news:X9***************@read3.inet.fi...
jj wrote:
It's taking forever to upload 400,000 records to the database through
access/odbc, and I've tried phpMyAdmin's interface but it seems to timeout during import of a CSV file. Is there a better way to import 400,000+
records?


If it is possible for you to get the data into sql command-form like this:

-----filename.txt--------
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
-----filename.txt--------

You can insert it into database using mysql console like this:

c:\mysql\bin\mysql -u username databasename < filename.txt

Jul 19 '05 #2
Don't listen to the previous guy.

Use LOAD DATA CONCURRENT INFILE

Loading of several megs of data will take only a few seconds.

On Fri, 19 Dec 2003 19:30:04 GMT, "jj" <jj@test.net> wrote:
Thanks that gave me an idea. I'm importing all the records to my local
MySQL database and will create an extract of that which will format it like
you said. Then take that .sql file and import it onto the server. We'll
see...
"Aggro" <sp**********@yahoo.com> wrote in message
news:X9***************@read3.inet.fi...
jj wrote:
> It's taking forever to upload 400,000 records to the database through
> access/odbc, and I've tried phpMyAdmin's interface but it seems totimeout > during import of a CSV file. Is there a better way to import 400,000+
> records?


If it is possible for you to get the data into sql command-form like this:

-----filename.txt--------
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
-----filename.txt--------

You can insert it into database using mysql console like this:

c:\mysql\bin\mysql -u username databasename < filename.txt


Jul 19 '05 #3
jj wrote:
It's taking forever to upload 400,000 records to the database through
access/odbc, and I've tried phpMyAdmin's interface but it seems to timeout
during import of a CSV file. Is there a better way to import 400,000+
records?


If it is possible for you to get the data into sql command-form like this:

-----filename.txt--------
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
-----filename.txt--------

You can insert it into database using mysql console like this:

c:\mysql\bin\mysql -u username databasename < filename.txt

Jul 19 '05 #4
jj
from Access via MyODBC for MySQL to my local MySQL server it took only a
minute or so. But through ODBC to a remote server, it takes hours then
finally dies. We have a high speed connection. Weird...
<us******@tampabay.rr.com> wrote in message
news:av********************************@4ax.com...
Don't listen to the previous guy.

Use LOAD DATA CONCURRENT INFILE

Loading of several megs of data will take only a few seconds.

On Fri, 19 Dec 2003 19:30:04 GMT, "jj" <jj@test.net> wrote:
Thanks that gave me an idea. I'm importing all the records to my local
MySQL database and will create an extract of that which will format it likeyou said. Then take that .sql file and import it onto the server. We'll
see...
"Aggro" <sp**********@yahoo.com> wrote in message
news:X9***************@read3.inet.fi...
jj wrote:

> It's taking forever to upload 400,000 records to the database through
> access/odbc, and I've tried phpMyAdmin's interface but it seems to

timeout
> during import of a CSV file. Is there a better way to import 400,000+ > records?

If it is possible for you to get the data into sql command-form like this:
-----filename.txt--------
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
-----filename.txt--------

You can insert it into database using mysql console like this:

c:\mysql\bin\mysql -u username databasename < filename.txt

Jul 19 '05 #5
jj
Thanks that gave me an idea. I'm importing all the records to my local
MySQL database and will create an extract of that which will format it like
you said. Then take that .sql file and import it onto the server. We'll
see...
"Aggro" <sp**********@yahoo.com> wrote in message
news:X9***************@read3.inet.fi...
jj wrote:
It's taking forever to upload 400,000 records to the database through
access/odbc, and I've tried phpMyAdmin's interface but it seems to timeout during import of a CSV file. Is there a better way to import 400,000+
records?


If it is possible for you to get the data into sql command-form like this:

-----filename.txt--------
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
-----filename.txt--------

You can insert it into database using mysql console like this:

c:\mysql\bin\mysql -u username databasename < filename.txt

Jul 19 '05 #6
Don't listen to the previous guy.

Use LOAD DATA CONCURRENT INFILE

Loading of several megs of data will take only a few seconds.

On Fri, 19 Dec 2003 19:30:04 GMT, "jj" <jj@test.net> wrote:
Thanks that gave me an idea. I'm importing all the records to my local
MySQL database and will create an extract of that which will format it like
you said. Then take that .sql file and import it onto the server. We'll
see...
"Aggro" <sp**********@yahoo.com> wrote in message
news:X9***************@read3.inet.fi...
jj wrote:
> It's taking forever to upload 400,000 records to the database through
> access/odbc, and I've tried phpMyAdmin's interface but it seems totimeout > during import of a CSV file. Is there a better way to import 400,000+
> records?


If it is possible for you to get the data into sql command-form like this:

-----filename.txt--------
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
-----filename.txt--------

You can insert it into database using mysql console like this:

c:\mysql\bin\mysql -u username databasename < filename.txt


Jul 19 '05 #7
jj
from Access via MyODBC for MySQL to my local MySQL server it took only a
minute or so. But through ODBC to a remote server, it takes hours then
finally dies. We have a high speed connection. Weird...
<us******@tampabay.rr.com> wrote in message
news:av********************************@4ax.com...
Don't listen to the previous guy.

Use LOAD DATA CONCURRENT INFILE

Loading of several megs of data will take only a few seconds.

On Fri, 19 Dec 2003 19:30:04 GMT, "jj" <jj@test.net> wrote:
Thanks that gave me an idea. I'm importing all the records to my local
MySQL database and will create an extract of that which will format it likeyou said. Then take that .sql file and import it onto the server. We'll
see...
"Aggro" <sp**********@yahoo.com> wrote in message
news:X9***************@read3.inet.fi...
jj wrote:

> It's taking forever to upload 400,000 records to the database through
> access/odbc, and I've tried phpMyAdmin's interface but it seems to

timeout
> during import of a CSV file. Is there a better way to import 400,000+ > records?

If it is possible for you to get the data into sql command-form like this:
-----filename.txt--------
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
insert into tablename(column1,column2) values(xx,yy);
-----filename.txt--------

You can insert it into database using mysql console like this:

c:\mysql\bin\mysql -u username databasename < filename.txt

Jul 19 '05 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

14
by: Bulba! | last post by:
One of the posters inspired me to do profiling on my newbie script (pasted below). After measurements I have found that the speed of Python, at least in the area where my script works, is...
17
by: Shailesh Humbad | last post by:
I just posted an article I wrote called ASP Speed Tricks. It covers techniques to optimize output of database data in HTML, for both simple tables and complex tables. More advanced ASP authors...
0
by: Creigh Shank | last post by:
Using an Apache/PHP/MySQL/Linux (Redhat 8.0) solution, PHPList, to create an e-mailing list for our 5.6 million book club members. Unfortunately, the import speed for importing records (at record...
1
by: jj | last post by:
It's taking forever to upload 400,000 records to the database through access/odbc, and I've tried phpMyAdmin's interface but it seems to timeout during import of a CSV file. Is there a better way...
13
by: David Mitchell | last post by:
I use the above function in queries for a number of forms and reports. The reports take approx 20 seconds to open. There are only 100 product id's in tblProducts. My concern is that the time will...
3
by: Reddy | last post by:
The sql query for my datagrid returns 100, 000 records. But the datagrid should display 20 records per page. I am using datagrid paging, but it is taking too much time for the page to load. Is...
11
by: Sezai YILMAZ | last post by:
Hello I need high throughput while inserting into PostgreSQL. Because of that I did some PostgreSQL insert performance tests. ------------------------------------------------------------ --...
9
by: Paul | last post by:
I have a process that I want to speed up. It first was written in Microsoft Access then converted to VB.NET. I would like to hear some suggestions on how to speed it up. The process is to match...
8
by: SaltyBoat | last post by:
Needing to import and parse data from a large PDF file into an Access 2002 table: I start by converted the PDF file to a html file. Then I read this html text file, line by line, into a table...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 2 August 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
0
by: erikbower65 | last post by:
Using CodiumAI's pr-agent is simple and powerful. Follow these steps: 1. Install CodiumAI CLI: Ensure Node.js is installed, then run 'npm install -g codiumai' in the terminal. 2. Connect to...
0
by: erikbower65 | last post by:
Here's a concise step-by-step guide for manually installing IntelliJ IDEA: 1. Download: Visit the official JetBrains website and download the IntelliJ IDEA Community or Ultimate edition based on...
0
by: kcodez | last post by:
As a H5 game development enthusiast, I recently wrote a very interesting little game - Toy Claw ((http://claw.kjeek.com/))。Here I will summarize and share the development experience here, and hope it...
0
by: Taofi | last post by:
I try to insert a new record but the error message says the number of query names and destination fields are not the same This are my field names ID, Budgeted, Actual, Status and Differences ...
0
by: Rina0 | last post by:
I am looking for a Python code to find the longest common subsequence of two strings. I found this blog post that describes the length of longest common subsequence problem and provides a solution in...
5
by: DJRhino | last post by:
Private Sub CboDrawingID_BeforeUpdate(Cancel As Integer) If = 310029923 Or 310030138 Or 310030152 Or 310030346 Or 310030348 Or _ 310030356 Or 310030359 Or 310030362 Or...
0
by: lllomh | last post by:
Define the method first this.state = { buttonBackgroundColor: 'green', isBlinking: false, // A new status is added to identify whether the button is blinking or not } autoStart=()=>{
2
by: DJRhino | last post by:
Was curious if anyone else was having this same issue or not.... I was just Up/Down graded to windows 11 and now my access combo boxes are not acting right. With win 10 I could start typing...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.