By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
464,386 Members | 1,156 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 464,386 IT Pros & Developers. It's quick & easy.

TPC H data

P: n/a
Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 23 '05 #1
Share this Question
Share on Google+
6 Replies

P: n/a
Shalu Gupta wrote:
Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.


Shalu,

I loaded the largest TPC-H table (lineitem, roughly 6 million rows) the
other day into a completely untuned 7.5devel PostgreSQL instance running
on RH 9, and it didn't raise a sweat. I delayed creating the indexes
until after the load. Data load took roughly 10 minutes, index creation
took a further 35 minutes (there are 13 of them).

HTH. (I'm just down the road from NCSU, would be happy to help out)

cheers

andrew

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 23 '05 #2

P: n/a
Shalu Gupta wrote:
Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.


Shalu,

I loaded the largest TPC-H table (lineitem, roughly 6 million rows) the
other day into a completely untuned 7.5devel PostgreSQL instance running
on RH 9, and it didn't raise a sweat. I delayed creating the indexes
until after the load. Data load took roughly 10 minutes, index creation
took a further 35 minutes (there are 13 of them).

HTH. (I'm just down the road from NCSU, would be happy to help out)

cheers

andrew

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 23 '05 #3

P: n/a
On Wed, 21 Apr 2004, Shalu Gupta wrote:
Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.


what method(s) are you using to load the data?
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

Nov 23 '05 #4

P: n/a
On Wed, 21 Apr 2004, Shalu Gupta wrote:
Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.


what method(s) are you using to load the data?
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

Nov 23 '05 #5

P: n/a
What scale factor TPC H are you importing?

additionally - might be worth giving the specs of the machine you are
doing this on.

(I seem to recall trying this with Pg 7.2 a while ago without this
issue, mind you - think I had ~1G of Ram and used the scale fact 1
dataset, i.e 1G)

regards

Mark
Shalu Gupta wrote:
Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 23 '05 #6

P: n/a
What scale factor TPC H are you importing?

additionally - might be worth giving the specs of the machine you are
doing this on.

(I seem to recall trying this with Pg 7.2 a while ago without this
issue, mind you - think I had ~1G of Ram and used the scale fact 1
dataset, i.e 1G)

regards

Mark
Shalu Gupta wrote:
Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 23 '05 #7

This discussion thread is closed

Replies have been disabled for this discussion.