473,699 Members | 3,159 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

COPY error handling

Hi, I'm not sure if this is the correct group for this question, but
I'll post it hoping that it is.
I'm loading several ~15 million row files into a table using the COPY
command. Apparently one of the rows, about 6 million in, has an invalid
entry. This is causing the COPY command to fail, so my question is this:
Is there any way to skip invalid rows? Or send them to a separate log
file etc to go through later?
I've gone through the docs, but I didn't see anything specific to this.
Any information, links, or hints are greatly appreciated.

Cheers!

Bricklen
Nov 23 '05 #1
4 1630
On Fri, Jun 04, 2004 at 14:11:19 +0000,
Bricklen <br**********@y ahoo.comz> wrote:
Hi, I'm not sure if this is the correct group for this question, but
I'll post it hoping that it is.
I'm loading several ~15 million row files into a table using the COPY
command. Apparently one of the rows, about 6 million in, has an invalid
entry. This is causing the COPY command to fail, so my question is this:
Is there any way to skip invalid rows? Or send them to a separate log
file etc to go through later?
I've gone through the docs, but I didn't see anything specific to this.
Any information, links, or hints are greatly appreciated.


Currently there isn't a builtin way to do this. You can pass the data
through a filter script that removes rows that are not in a proper format.

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Nov 23 '05 #2
On Fri, Jun 04, 2004 at 14:11:19 +0000,
Bricklen <br**********@y ahoo.comz> wrote:
Hi, I'm not sure if this is the correct group for this question, but
I'll post it hoping that it is.
I'm loading several ~15 million row files into a table using the COPY
command. Apparently one of the rows, about 6 million in, has an invalid
entry. This is causing the COPY command to fail, so my question is this:
Is there any way to skip invalid rows? Or send them to a separate log
file etc to go through later?
I've gone through the docs, but I didn't see anything specific to this.
Any information, links, or hints are greatly appreciated.


Currently there isn't a builtin way to do this. You can pass the data
through a filter script that removes rows that are not in a proper format.

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Nov 23 '05 #3
Bruno Wolff III wrote:
On Fri, Jun 04, 2004 at 14:11:19 +0000,
Bricklen <br**********@y ahoo.comz> wrote:
Hi, I'm not sure if this is the correct group for this question, but
I'll post it hoping that it is.
I'm loading several ~15 million row files into a table using the COPY
command. Apparently one of the rows, about 6 million in, has an invalid
entry. This is causing the COPY command to fail, so my question is this:
Is there any way to skip invalid rows? Or send them to a separate log
file etc to go through later?
I've gone through the docs, but I didn't see anything specific to this.
Any information, links, or hints are greatly appreciated.

Currently there isn't a builtin way to do this. You can pass the data
through a filter script that removes rows that are not in a proper format.

Thanks for the clarification!

Cheers,

Bricklen
Nov 23 '05 #4
Bruno Wolff III wrote:
On Fri, Jun 04, 2004 at 14:11:19 +0000,
Bricklen <br**********@y ahoo.comz> wrote:
Hi, I'm not sure if this is the correct group for this question, but
I'll post it hoping that it is.
I'm loading several ~15 million row files into a table using the COPY
command. Apparently one of the rows, about 6 million in, has an invalid
entry. This is causing the COPY command to fail, so my question is this:
Is there any way to skip invalid rows? Or send them to a separate log
file etc to go through later?
I've gone through the docs, but I didn't see anything specific to this.
Any information, links, or hints are greatly appreciated.

Currently there isn't a builtin way to do this. You can pass the data
through a filter script that removes rows that are not in a proper format.

Thanks for the clarification!

Cheers,

Bricklen
Nov 23 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
3267
by: WSeeger | last post by:
When creating a new class, is it encouraged to always include error handling routines within your LET and GET procedures? It's seems that most text books never seem to include much about error handling within classes. Just hoping to hear some programmer's thoughts on error handling.
1
1861
by: Rich | last post by:
Hi, I have a query regarding VC6 and its handling of templated copy constructors. Here goes: Take a look at the following code sample... template<class _Ty, size_t t_uiSize = 10 > class my_template
6
8456
by: Squirrel | last post by:
I have a command button on a subform to delete a record. The only statement in the subroutine is: DoCmd.RunCommand acCmdDeleteRecord The subform's recordsource is "select * from tblVisit order by VisitDt" I'm getting this error message: Errno is 2465. Err.description is "Can't find field '|' referred to in your expression"
13
4478
by: Thelma Lubkin | last post by:
I use code extensively; I probably overuse it. But I've been using error trapping very sparingly, and now I've been trapped by that. A form that works for me on the system I'm using, apparently runs into problems on the system where it will actually be used, and since I used so little error-trapping it dies very ungracefully. I will of course try to fix whatever is causing the error and add error-trapping to the functions where the...
21
4408
by: Anthony England | last post by:
Everyone knows that global variables get re-set in an mdb when an un-handled error is encountered, but it seems that this also happens when the variable is defined as private at form-level. So if "global variables get re-set" doesn't tell the whole story, then what does? ***please note*** I'm not looking for a solution - I'm looking for a more detailed description of what happens when an un-handled error occurs - possibly with help file...
12
2676
by: Andrew Schepler | last post by:
When compiled with Visual C++ .NET 2003 (only), the program below aborts as though no matching catch clause is present. If the copy constructor of A is made public, it successfully catches the exception instead. If I step into __CxxThrowException in debug/assembly mode, it looks like the pThrowInfo parameter created by the compiler excluded typeid(A) from pCatchableTypeArray. It would be reasonable to fail to catch A by value because of the...
3
2854
by: Stefan Johansson | last post by:
Hi all I'am moving from Visual Foxpro and have a question regarding "best practice" error handling in vb .net. In VFP I have always used a "central" error handling object in order to have a easy and reusable way of handling all errors in a program. The VB 6 coding examples I have seen there has always been error handling code in each program module.
19
4189
by: Angus | last post by:
I have a socket class CTestClientSocket which I am using to simulate load testing. I create multiple instances of the client like this: for (int i = 0; i < 5; i++) { CTestClientSocket* pTemp = new CTestClientSocket(this, ip, port); pTemp->Connect(); m_collClients.push_back(pTemp);
0
1011
by: subramanian100in | last post by:
Consider the following program: #include <cstdlib> #include <iostream> using namespace std; class Test { public:
9
3295
by: MrDeej | last post by:
Hello guys! We have an SQL server which sometimes makes timeouts and connection errors. And we have an function witch writes and updates data in 2 tables on this server. When the SQL server error appears it, in 99%, of the cases, works if we just press the play button in VBA debug. Therefor we have maked an error handling which just tryes again. However, as this error handling is difficult to test because of maybe 1 or 2 errors a day, we...
0
8706
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8633
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9199
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9055
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8902
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7787
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4642
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3076
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2366
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.