473,473 Members | 1,768 Online
Bytes | Software Development & Data Engineering Community
Create Post

Home Posts Topics Members FAQ

Generating a SQL Server population routine

Has some one come up with a similar type script that could be used in a
Postgresql database?
The script below was created for a SQLServer database.
Thx,
-Martin

++++++++++++++++++++++++++++++++++++++
http://searchdatabase.techtarget.com...913717,00.html

In the early stages of application design DBA or a developer creates a data
model. Unfortunately many models work very well with a handful of rows but
fail miserably when the application grows by leaps and bounds. This is why
it is important to populate your data model with data and stress test it
prior to making it available for users. Test data doesn't have to be
perfect; indeed, you can duplicate the same record, or a few different
records, to test the performance of your queries.

This article offers a script for generating INSERT statements for every
table in your database. The script is fairly simple -- it relies on three
system tables: sysusers, sysobjects and syscolumns. It generates an INSERT
statement duplicating the top row in your table. However, it can be easily
altered to fit your needs.

SET NOCOUNT ON
DECLARE @table VARCHAR(200),
@owner VARCHAR(100),
@sql VARCHAR(2000),
@sql1 VARCHAR(2000)

DECLARE @schema TABLE (
table_name VARCHAR(200),
column_name VARCHAR(200))

INSERT @schema
SELECT c.name + '.' + a.name, b.name FROM sysobjects a INNER JOIN
syscolumns b ON b.id = a.id
AND a.type = 'u'
AND a.name <> 'dtproperties'
INNER JOIN sysusers c ON c.uid = a.uid
ORDER BY a.name, b.colid

DECLARE table_cursor CURSOR FOR

SELECT DISTINCT table_name FROM @schema

OPEN table_cursor
FETCH NEXT FROM table_cursor INTO @table

WHILE @@FETCH_STATUS = 0
BEGIN

SELECT @sql1 = ''
SELECT @sql1 = @sql1 + ', '+ column_name FROM @schema
WHERE table_name = @table

SELECT @sql1 = SUBSTRING(@sql1, 3, LEN(@sql1)-2)
SELECT @sql = 'INSERT ' + @table + '( ' + @sql1 + ' ) ' + CHAR(10) + '
SELECT TOP 1 ' + @sql1 + ' FROM ' +
@table

SELECT @sql
FETCH NEXT FROM table_cursor INTO @table
END
CLOSE table_cursor
DEALLOCATE table_cursor

In the pubs database, the output will be similar to the following:

INSERT dbo.authors( au_id, au_lname, au_fname, phone, address, city, state,
zip, contract )
SELECT TOP 1 au_id, au_lname, au_fname, phone, address, city, state, zip,
contract FROM dbo.authors

INSERT dbo.discounts( discounttype, stor_id, lowqty, highqty, discount )
SELECT TOP 1 discounttype, stor_id, lowqty, highqty, discount FROM
dbo.discounts

INSERT dbo.employee( emp_id, fname, minit, lname, job_id, job_lvl, pub_id,
hire_date )
SELECT TOP 1 emp_id, fname, minit, lname, job_id, job_lvl, pub_id,
hire_date FROM dbo.employee


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #1
12 2461
Ma**********@dom.com wrote:
Has some one come up with a similar type script that could be used in a
Postgresql database? The script below was created for a SQLServer database.
Thx,
-Martin


I haven't. But I was wondering if a general purpose tuple-generating
function, which would be trivial to implement, might be worthwhile in
PostgreSQL or perhaps added to Joe Conway's tablefunc module.
Something like:

tuple_generator(integer)

which returns a set of numbers whose elements are the integer values
between 1 and the number supplied.

You could then create any number of pseudo-duplicates (can't violate
the candidate key, obviously) from a single-record table like so:

INSERT INTO employees (name, salary)
SELECT employees.name, employees.salary
FROM employees, tuple_generator(1000)
WHERE employees.employeeid = 1;

You could easily build a script to fill your database by querying
pg_class.relname and feeding the output to psql.

It would also be useful for handling sparse date and time data:

SELECT day_of_year,
(SELECT COALESCE(SUM(purchases.qty), 0)
FROM purchases
WHERE EXTRACT(doy FROM purchases.sale_date) = day_of_year)
FROM tuple_generator(366) AS day_of_year
ORDER BY day_of_year;

Mike Mascari
ma*****@mascari.com






---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #2
I think I have come across a bug in postgres 7.3.x. I noticed that when I
upgraded to 7.3 my code broke on certain sql statements. They still don't
work on the lastest stable release of 7.3.

Here is an example to illustrate my problem:

-- create the table with:
CREATE TABLE bugtest ( a int2 );

-- then do the following inserts:
insert into bugtest (a) select 1 union select 1; -- this one succeeds
insert into bugtest (a) select 1 union select '1'; -- this one also succeeds
insert into bugtest (a) select '1' union select 1; -- this one also succeeds
insert into bugtest (a) select '1' union select '1'; -- this one fails

The all succeed except the last one. It fails with the following error:

ERROR: column "a" is of type smallint but expression is of type text
You will need to rewrite or cast the expression

It seems to me that they should all succeed. At least postgres seems to
have no problem converting '<intvalue>' to <intvalue> anywhere else that I
can find. At least the last 3 inserts shoudl either all fail or all succeed
(IMHO).

Is this a bug? Has anyone else reported it? Is there a procedure I need to
follow to report it? Has it been fixed in 7.4?

On a sidenote I am doing the weird select union thing as a way to insert
many records at once without having to execute multiple queries. I first
started doing it on SQLServer and it was much, much faster than doing
separate inserts. Is there a better way to do it in postgres? I have
looked at the copy from command but I can't find any examples of how to use
it in php or how to specify the columns / column order that you are going to
use with php. Also what characters need to be excaped if I do this
(obviously new lines and tabs). And nulls are represented by \N.

Rick Gigger
---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #3
"Rick Gigger" <ri**@alpinenetworking.com> writes:
insert into bugtest (a) select '1' union select '1'; -- this one fails Is this a bug?


No. It's unfortunate perhaps, but it's not a bug. The UNION forces us
to make a decision about the output datatype of the UNION operation.
In your other cases the chosen datatype is integer, which can later be
cast to smallint for the insert, but in this case the chosen datatype is
text, which is not implicitly castable to smallint.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

Nov 12 '05 #4
What was it that changed in 7.3 that made this behavior change. (it worked
in 7.2)

Thanks,

Rick Gigger

----- Original Message -----
From: "Tom Lane" <tg*@sss.pgh.pa.us>
To: "Rick Gigger" <ri**@alpinenetworking.com>
Cc: <pg***********@postgresql.org>
Sent: Monday, October 06, 2003 1:04 PM
Subject: Re: [GENERAL] Possible bug on insert

"Rick Gigger" <ri**@alpinenetworking.com> writes:
insert into bugtest (a) select '1' union select '1'; -- this one fails

Is this a bug?


No. It's unfortunate perhaps, but it's not a bug. The UNION forces us
to make a decision about the output datatype of the UNION operation.
In your other cases the chosen datatype is integer, which can later be
cast to smallint for the insert, but in this case the chosen datatype is
text, which is not implicitly castable to smallint.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Nov 12 '05 #5
I guess then I will switch to use COPY "tablename (fieldlist)" FROM. Will
this end up being faster anyway.

In case anyone was wondering here is a good example of how to do it in php
from:

http://us4.php.net/manual/en/function.pg-put-line.php

<?php
$conn = pg_pconnect("dbname=foo");
pg_query($conn, "create table bar (a int4, b char(16), d float8)");
pg_query($conn, "copy bar from stdin");
pg_put_line($conn, "3\thello world\t4.5\n");
pg_put_line($conn, "4\tgoodbye world\t7.11\n");
pg_put_line($conn, "\\.\n");
pg_end_copy($conn);
?>

thanks,

Rick Gigger

----- Original Message -----
From: "Tom Lane" <tg*@sss.pgh.pa.us>
To: "Rick Gigger" <ri**@alpinenetworking.com>
Cc: <pg***********@postgresql.org>
Sent: Monday, October 06, 2003 1:04 PM
Subject: Re: [GENERAL] Possible bug on insert

"Rick Gigger" <ri**@alpinenetworking.com> writes:
insert into bugtest (a) select '1' union select '1'; -- this one fails

Is this a bug?


No. It's unfortunate perhaps, but it's not a bug. The UNION forces us
to make a decision about the output datatype of the UNION operation.
In your other cases the chosen datatype is integer, which can later be
cast to smallint for the insert, but in this case the chosen datatype is
text, which is not implicitly castable to smallint.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #6
"Rick Gigger" <ri**@alpinenetworking.com> writes:
What was it that changed in 7.3 that made this behavior change. (it worked
in 7.2)


7.2 allowed implicit casts from text to smallint, I think.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

Nov 12 '05 #7
Is there any possiblity of adding an option for compatibilityes sake (or
does one already exist) to revert to the old behavior. This has currently
kept me from upgrading beyond 7.2.4 thus far in production as it will break
all of my apps. I can slowly update them but many of them don't get changed
very often and I will have to undergo a testing cycle for each of them just
to maintain compatibility with postgres > 7.2.4. This is not something I
really want to do. I would much prefer to just upgrade and have my legasy
apps work without modification or testing.

Thanks,

Rick Gigger

----- Original Message -----
From: "Tom Lane" <tg*@sss.pgh.pa.us>
To: "Rick Gigger" <ri**@alpinenetworking.com>
Cc: <pg***********@postgresql.org>
Sent: Monday, October 06, 2003 2:26 PM
Subject: Re: [GENERAL] Possible bug on insert

"Rick Gigger" <ri**@alpinenetworking.com> writes:
What was it that changed in 7.3 that made this behavior change. (it worked in 7.2)


7.2 allowed implicit casts from text to smallint, I think.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Nov 12 '05 #8
Two questions:

1) how would I go about doing that
2) is there any change that doing that could break other things?

thanks,

Rick Gigger

----- Original Message -----
From: "Tom Lane" <tg*@sss.pgh.pa.us>
To: "Rick Gigger" <ri**@alpinenetworking.com>
Sent: Monday, October 06, 2003 2:57 PM
Subject: Re: [GENERAL] Possible bug on insert

Is there any possiblity of adding an option for compatibilityes sake (or
does one already exist) to revert to the old behavior.


If you're desperate you can mark that cast as implicit by changing its
entry in pg_cast. Not sure what side-effects you might see though.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ma*******@postgresql.org

Nov 12 '05 #9
>>>>> "RG" == Rick Gigger <ri**@alpinenetworking.com> writes:

RG> very often and I will have to undergo a testing cycle for each of them just
RG> to maintain compatibility with postgres > 7.2.4. This is not something I
RG> really want to do. I would much prefer to just upgrade and have my legasy
RG> apps work without modification or testing.

You can't have progress that way. To be fair, it *is* a major version
number change, and if you don't test your apps across major version
changes of *anything* you don't need to be in this business. No
offense ;-)

Anyhow, Bruce posted a patch to the 7.3 line to allow this behavior
back, as it was necessary for some apps. However, don't count on it
for 7.4...

--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Vivek Khera, Ph.D. Khera Communications, Inc.
Internet: kh***@kciLink.com Rockville, MD +1-240-453-8497
AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/

---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Nov 12 '05 #10
In article <3F**************@mascari.com>,
Mike Mascari <ma*****@mascari.com> writes:
Ma**********@dom.com wrote:
Has some one come up with a similar type script that could be used in a
Postgresql database? The script below was created for a SQLServer database.
Thx,
-Martin
I haven't. But I was wondering if a general purpose tuple-generating
function, which would be trivial to implement, might be worthwhile in
PostgreSQL or perhaps added to Joe Conway's tablefunc module.
Something like: tuple_generator(integer) which returns a set of numbers whose elements are the integer values
between 1 and the number supplied.


How about this?

CREATE OR REPLACE FUNCTION enum (INT) RETURNS SETOF INT AS '
DECLARE
numvals ALIAS FOR $1;
BEGIN
FOR currval IN 0 .. numvals - 1 LOOP
RETURN NEXT currval;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql';

CREATE OR REPLACE FUNCTION enum (INT, INT) RETURNS SETOF INT AS '
DECLARE
numvals ALIAS FOR $1;
minval ALIAS FOR $2;
BEGIN
FOR currval IN 0 .. numvals - 1 LOOP
RETURN NEXT minval + currval;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql';

CREATE OR REPLACE FUNCTION enum (INT, INT, INT) RETURNS SETOF INT AS '
DECLARE
numvals ALIAS FOR $1;
minval ALIAS FOR $2;
maxval ALIAS FOR $3;
BEGIN
FOR currval IN 0 .. numvals - 1 LOOP
RETURN NEXT currval % (maxval - minval + 1) + minval;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql';

Usage: SELECT * FROM enum (numvals [, minval [, maxval]])
Returns numvals consecutive numbers, beginning with 0 or minval
Wraps around to minval if maxval is reached
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ma*******@postgresql.org so that your
message can get through to the mailing list cleanly

Nov 12 '05 #11
In article <3F**************@mascari.com>,
Mike Mascari <ma*****@mascari.com> writes:
Ma**********@dom.com wrote:
Has some one come up with a similar type script that could be used
in a Postgresql database? The script below was created for a SQLServer database.
Thx,
-Martin

I haven't. But I was wondering if a general purpose tuple-generating
function, which would be trivial to implement, might be worthwhile in
PostgreSQL or perhaps added to Joe Conway's tablefunc module.
Something like:

tuple_generator(integer)

which returns a set of numbers whose elements are the integer values
between 1 and the number supplied.


How about this?

CREATE OR REPLACE FUNCTION enum (INT) RETURNS SETOF INT AS '
DECLARE
numvals ALIAS FOR $1;
BEGIN
FOR currval IN 0 .. numvals - 1 LOOP
RETURN NEXT currval;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql';

CREATE OR REPLACE FUNCTION enum (INT, INT) RETURNS SETOF INT AS '
DECLARE
numvals ALIAS FOR $1;
minval ALIAS FOR $2;
BEGIN
FOR currval IN 0 .. numvals - 1 LOOP
RETURN NEXT minval + currval;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql';

CREATE OR REPLACE FUNCTION enum (INT, INT, INT) RETURNS SETOF INT AS
' DECLARE
numvals ALIAS FOR $1;
minval ALIAS FOR $2;
maxval ALIAS FOR $3;
BEGIN
FOR currval IN 0 .. numvals - 1 LOOP
RETURN NEXT currval % (maxval - minval + 1) + minval;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql';

Usage: SELECT * FROM enum (numvals [, minval [, maxval]])
Returns numvals consecutive numbers, beginning with 0 or minval
Wraps around to minval if maxval is reached


Or a little different, with the over-loaded functions relying on the
original:

CREATE OR REPLACE FUNCTION public.enum(int4) RETURNS SETOF int4 AS '
DECLARE
numvals ALIAS FOR $1;
BEGIN
FOR currval IN 0 .. numvals - 1 LOOP
RETURN NEXT currval;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql' VOLATILE;
CREATE OR REPLACE FUNCTION public.enum(int4, int4) RETURNS SETOF int4 AS '
DECLARE
numvals ALIAS FOR $1;
minval ALIAS FOR $2;
currval RECORD;
BEGIN
FOR currval IN SELECT minval + enum AS enum FROM enum(numvals) LOOP
RETURN NEXT currval.enum;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql' VOLATILE;

CREATE OR REPLACE FUNCTION public.enum(int4, int4, int4) RETURNS SETOF
int4 AS ' DECLARE
numvals ALIAS FOR $1;
minval ALIAS FOR $2;
maxval ALIAS FOR $3;
currval RECORD;
/*
From: Harald Fuchs
Date: Wed, October 8, 2003 5:53
To: pg***********@postgresql.org

tuple_generator(integer)

which returns a set of numbers whose elements are the integer values
between 1 and the number supplied.

Usage: SELECT * FROM enum (numvals [, minval [, maxval]])
Returns numvals consecutive numbers, beginning with 0 or minval
Wraps around to minval if maxval is reached
*/

BEGIN
FOR currval IN SELECT * FROM enum(numvals, minval) LOOP
RETURN NEXT currval.enum % maxval;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql' VOLATILE;
But, as interesting as these look, what would you actually use them for?
~Berend Tober


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Nov 12 '05 #12
In article <64591.66.212.203.144.1065659357.squirrel@$HOSTNAM E>,
<bt****@seaworthysys.com> writes:
Or a little different, with the over-loaded functions relying on the
original: CREATE OR REPLACE FUNCTION public.enum(int4) RETURNS SETOF int4 AS '
DECLARE
numvals ALIAS FOR $1;
BEGIN
FOR currval IN 0 .. numvals - 1 LOOP
RETURN NEXT currval;
END LOOP;
RETURN;
END;
' LANGUAGE 'plpgsql' VOLATILE;
Why VOLATILE? Shouldn't that be IMMUTABLE?
(Sorry, but I'm a PostgreSQL newbie.)
But, as interesting as these look, what would you actually use them for?


SELECT extract (month FROM sdate) AS month,
count (*) AS monthly_sales
FROM sales
GROUP BY month
ORDER BY month;

gives you the monthly sales, but what if you would like a result row
also for months with nothing sold?

SELECT enum, count (sdate) AS monthly_sales
FROM enum (12, 1)
LEFT JOIN sales ON enum = extract (month FROM sdate)
GROUP BY enum
ORDER BY enum;

This would do the trick. Is there a more elegant solution?
---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

Nov 12 '05 #13

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Ron Hiler | last post by:
Hi guys, I've written a small bit of PHP code who's purpose is to detect my game server. If the game server is running, I want to display the Launcher and Client population (two parts of the...
0
by: RonHiler | last post by:
Hi guys, I run a game server (for a game I'm writing in development). I had a routine on my web site which detected if the server was up or down and displayed a icon, so that people could see...
21
by: Andreas Lobinger | last post by:
Aloha, i wanted to ask another problem, but as i started to build an example... How to generate (memory and time)-efficient a string containing random characters? I have never worked with...
1
by: Naser_geraminia | last post by:
Hi all. I have an keyword table by 2000000 rows that is an int(4) key and a varchar(50) Keyword(Unique index) in Sql Server 2000. After about 80 seconds of starting full population, speed of...
6
by: khanif | last post by:
Can someone provide me with an example with how to import data from an excel file into SQL Server with VB.NET? Thanks
0
by: Brian Henry | last post by:
I'm sure some of you ran into this already and wondered how to fix it. When you Have a solution that contains solution folders (the partially transparent folders that can hold groupings of...
14
by: neonman14 | last post by:
Hello I am in Intro to Java Programming and I am having problems with assignment. The Homework assignment is called Population. Population Write a program that will predict the size of a...
1
by: zamzoum | last post by:
hi i just wanted to ask u how can i solve the exception error "couldnot find installable ISAM" this exception occurs when i run this code : Dim excelConnection As...
1
by: tg | last post by:
http://img522.imageshack.us/img522/8647/scan10005ci7.jpg As a percentage of world inhabitants, the white population will plummet to a single digit (9.76%) by 2060 from a high-water mark of...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
1
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...
0
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The...
0
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated ...
0
muto222
php
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.