TPC H data

[Shalu Gupta]

Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

[Andrew Dunstan]

Shalu Gupta wrote:

Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.

Shalu,

I loaded the largest TPC-H table (lineitem, roughly 6 million rows) the
other day into a completely untuned 7.5devel PostgreSQL instance running
on RH 9, and it didn't raise a sweat. I delayed creating the indexes
until after the load. Data load took roughly 10 minutes, index creation
took a further 35 minutes (there are 13 of them).

HTH. (I'm just down the road from NCSU, would be happy to help out)

cheers

andrew

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[Andrew Dunstan]

Shalu Gupta wrote:

Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.

Shalu,

I loaded the largest TPC-H table (lineitem, roughly 6 million rows) the
other day into a completely untuned 7.5devel PostgreSQL instance running
on RH 9, and it didn't raise a sweat. I delayed creating the indexes
until after the load. Data load took roughly 10 minutes, index creation
took a further 35 minutes (there are 13 of them).

HTH. (I'm just down the road from NCSU, would be happy to help out)

cheers

andrew

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

[scott.marlowe]

On Wed, 21 Apr 2004, Shalu Gupta wrote:

Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

what method(s) are you using to load the data?
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

[scott.marlowe]

On Wed, 21 Apr 2004, Shalu Gupta wrote:

Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

what method(s) are you using to load the data?
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

[Mark Kirkwood]

What scale factor TPC H are you importing?

additionally - might be worth giving the specs of the machine you are
doing this on.

(I seem to recall trying this with Pg 7.2 a while ago without this
issue, mind you - think I had ~1G of Ram and used the scale fact 1
dataset, i.e 1G)

regards

Mark
Shalu Gupta wrote:

Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[Mark Kirkwood]

What scale factor TPC H are you importing?

additionally - might be worth giving the specs of the machine you are
doing this on.

(I seem to recall trying this with Pg 7.2 a while ago without this
issue, mind you - think I had ~1G of Ram and used the scale fact 1
dataset, i.e 1G)

regards

Mark
Shalu Gupta wrote:

Hello,

We are trying to import the TPC-H data into postgresql using the COPY
command and for the larger files we get an error due to insufficient
memory space.

We are using a linux system with Postgresql-7.3.4

Is it that Postgresql cannot handle such large files or is there some
other possible reason.

Thanks
Shalu Gupta
NC State University.

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match