473,320 Members | 2,104 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Inserting results of a recordset in Chunks

I use ADO.Recordsets constantly to query data from external sources
and append the results into tables in Access databases. A number of
the queries I deal with bring back in excess of 50,000 records and as
many as 700,000 records. It takes hours to sometimes to load the
results of one query using this method. I would rather avoid using the
"make table" method of appending a pass-through query to accomplish
this. I'm looking for a faster, more efficient way to append the
results of the recordsets into tables rather than appending each
record one at a time.

Is it possible to append the entire result of the SQL at one time or
in larger chunks to speed the process up? Or does someone have an
alternative solution that is more efficient?
Nov 13 '05 #1
2 2491
On 11 Aug 2004 03:57:34 -0700, vb*@cox-internet.com (Chuck Reed)
wrote:

That is exactly what MakeTable or Append queries do. At least
logically. For example if you have a local table tblHere and a remote
table tblThere with the same fields layout, you only write:
DoCmd.RunSQL "Insert into tblHere Select * from tblThere"
The database engine will use the most economic way available to
satisfy that request.

Alternatives include:
* Leave the data where it is, link to those tables, and use them
directly.
* Try SQL Server as your main platform, and use BCP or DTS.

-Tom.

I use ADO.Recordsets constantly to query data from external sources
and append the results into tables in Access databases. A number of
the queries I deal with bring back in excess of 50,000 records and as
many as 700,000 records. It takes hours to sometimes to load the
results of one query using this method. I would rather avoid using the
"make table" method of appending a pass-through query to accomplish
this. I'm looking for a faster, more efficient way to append the
results of the recordsets into tables rather than appending each
record one at a time.

Is it possible to append the entire result of the SQL at one time or
in larger chunks to speed the process up? Or does someone have an
alternative solution that is more efficient?


Nov 13 '05 #2
I did leave some important information out of my question. Do to the
size of database I'm querying against, I'm using ODBCDirect to a
teradata platform to retrieve the particular records that I'm needing.
Since I do not have access to SQL Server, it looks like my only
alternative is to create a pass-through query dynamically via VBA and
then use a Make Table or Append query to insert the records to get the
optimum speed in loading the tables with the results of the ODBCDirect
query. If I don't get any other suggestions that I can implement, I
probably will have to pursue the Make Table/Append option.

Tom van Stiphout <no*************@cox.net> wrote in message news:<6g********************************@4ax.com>. ..
On 11 Aug 2004 03:57:34 -0700, vb*@cox-internet.com (Chuck Reed)
wrote:

That is exactly what MakeTable or Append queries do. At least
logically. For example if you have a local table tblHere and a remote
table tblThere with the same fields layout, you only write:
DoCmd.RunSQL "Insert into tblHere Select * from tblThere"
The database engine will use the most economic way available to
satisfy that request.

Alternatives include:
* Leave the data where it is, link to those tables, and use them
directly.
* Try SQL Server as your main platform, and use BCP or DTS.

-Tom.

I use ADO.Recordsets constantly to query data from external sources
and append the results into tables in Access databases. A number of
the queries I deal with bring back in excess of 50,000 records and as
many as 700,000 records. It takes hours to sometimes to load the
results of one query using this method. I would rather avoid using the
"make table" method of appending a pass-through query to accomplish
this. I'm looking for a faster, more efficient way to append the
results of the recordsets into tables rather than appending each
record one at a time.

Is it possible to append the entire result of the SQL at one time or
in larger chunks to speed the process up? Or does someone have an
alternative solution that is more efficient?

Nov 13 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Guy Erez | last post by:
Hi, I'm running queries with MySql 4.0.17 that return thousands of records. Because I need to present them in GUI, I returieve the results in chunks using LIMIT, for example - get first 100,...
2
by: Josh Strickland | last post by:
I am attempting to create an Access database which uses forms to enter data. The issue I am having is returning the query results from the Stored Procedure back in to the Access Form. ...
9
by: Dom Boyce | last post by:
Hi First up, I am using MS Access 2002. I have a database which records analyst rating changes for a list of companies on a daily basis. Unfortunately, the database has been set up (by my...
4
by: James | last post by:
Why does my LDAP query from a C# console app limit its results to 1000? When I run the same query from a vb script I get over 6000 results. I have tried to set the SearchRequest.SizeLimit to a...
10
by: Roger Withnell | last post by:
I'm using ASP, VBScript and SQL Server. I'm also using UTF-8 character set and so my codepage is 65001 and SQL Server datatype nvarchar. I can insert unicode characters correctly into the...
1
by: Phlazar | last post by:
I have created a 'Intelligent' search stored procedure that accepts two input parameters SearchFor and SearchIn and an OUTPUT parameter @Result Basically it starts by looking for a match for the...
5
by: Henrik | last post by:
The problem is (using MS Access 2003) I am unable to retrieve long strings (255 chars) from calculated fields through a recordset. The data takes the trip in three phases: 1. A custom public...
2
by: wallconor | last post by:
Hi, I am having a problem using Dreamweaver CS3 standard recordset paging behavior. It doesn’t seem to work when I pass parameter values from a FORM on my search page, to the recordset on my...
7
by: pcauchon | last post by:
When I try to run this code on my machine (iMac with MacOS 10.5), I get very strange results. I am using this compiler: i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5465) The first...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.