inputing, paging, sorting, a large text file

[JJ]

I have a need to input a large tab delimited text file, which I will parse
to check it has the expected columns, before allowing the user to submit it
to the database. The user may paste the file into a textbox, or upload it
(haven't decided yet).

The problem I have is that the text file consists of around 3000 lines, and
I want to display the formatted columns to the user before submitting it to
the database (perhaps allowing them to edit some fields if they want, before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so the
question is::

How can I page the data in the grid, without having to parse the text each
time - i.e. where can I 'store' the parsed text so that I use that during my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

[pbromberg]

On Jun 7, 11:56 am, "JJ" <[e-mail]:

I have a need to input a large tab delimited text file, which I will parse
to check it has the expected columns, before allowing the user to submit it
to the database. The user may paste the file into a textbox, or upload it
(haven't decided yet).

The problem I have is that the text file consists of around 3000 lines, and
I want to display the formatted columns to the user before submitting it to
the database (perhaps allowing them to edit some fields if they want, before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so the
question is::

How can I page the data in the grid, without having to parse the text each
time - i.e. where can I 'store' the parsed text so that I use that during my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

JJ,
What I'd consider doing is to read and parse the file into a
DataTable, and use this to bind to a pageable grid.
Peter

[JJ]

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless I'm
mistaken, that each time you change the page index, you need to rebind the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for such a
large datatable perhaps?

JJ

<pb*******@gmail.comwrote in message
news:11**********************@q75g2000hsh.googlegr oups.com...

On Jun 7, 11:56 am, "JJ" <[e-mail]:

>I have a need to input a large tab delimited text file, which I will
parse
to check it has the expected columns, before allowing the user to submit
it
to the database. The user may paste the file into a textbox, or upload it
(haven't decided yet).

The problem I have is that the text file consists of around 3000 lines,
and
I want to display the formatted columns to the user before submitting it
to
the database (perhaps allowing them to edit some fields if they want,
before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so the
question is::

How can I page the data in the grid, without having to parse the text
each
time - i.e. where can I 'store' the parsed text so that I use that during
my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

JJ,
What I'd consider doing is to read and parse the file into a
DataTable, and use this to bind to a pageable grid.
Peter

[Alexey Smirnov]

"JJ" <ab*@xyz.comwrote in message
news:uU**************@TK2MSFTNGP02.phx.gbl...

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless I'm
mistaken, that each time you change the page index, you need to rebind the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for such a
large datatable perhaps?

JJ

Why don't you save the data in the temporary table in the database? Once
user confirmed the changes you will copy all data to the real table.

[=?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=]

Store the DataTable in Session after you've first constructed it.
Get it back out of Session after changing the PageIndex and rebind:

grid.CurrrentPageIndex =e.newPageIndex;
DataTable myTable = (DataTable)Session["myDataTable"];
grid.DataSource=myTable;
grid.DataBind();

Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"JJ" wrote:

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless I'm
mistaken, that each time you change the page index, you need to rebind the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for such a
large datatable perhaps?

JJ

<pb*******@gmail.comwrote in message
news:11**********************@q75g2000hsh.googlegr oups.com...

On Jun 7, 11:56 am, "JJ" <[e-mail]:

I have a need to input a large tab delimited text file, which I will
parse
to check it has the expected columns, before allowing the user to submit
it
to the database. The user may paste the file into a textbox, or upload it
(haven't decided yet).

The problem I have is that the text file consists of around 3000 lines,
and
I want to display the formatted columns to the user before submitting it
to
the database (perhaps allowing them to edit some fields if they want,
before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so the
question is::

How can I page the data in the grid, without having to parse the text
each
time - i.e. where can I 'store' the parsed text so that I use that during
my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

JJ,
What I'd consider doing is to read and parse the file into a
DataTable, and use this to bind to a pageable grid.
Peter

[JJ]

Is there a limit to how large the datatable can be within a Session
variable? We're talking a 3000 line tab delimited text file here.

JJ

"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in
message news:63**********************************@microsof t.com...

Store the DataTable in Session after you've first constructed it.
Get it back out of Session after changing the PageIndex and rebind:

grid.CurrrentPageIndex =e.newPageIndex;
DataTable myTable = (DataTable)Session["myDataTable"];
grid.DataSource=myTable;
grid.DataBind();

Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"JJ" wrote:

>Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless I'm
mistaken, that each time you change the page index, you need to rebind
the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for such a
large datatable perhaps?

JJ

<pb*******@gmail.comwrote in message
news:11**********************@q75g2000hsh.googleg roups.com...

On Jun 7, 11:56 am, "JJ" <[e-mail]:
I have a need to input a large tab delimited text file, which I will
parse
to check it has the expected columns, before allowing the user to
submit
it
to the database. The user may paste the file into a textbox, or upload
it
(haven't decided yet).

The problem I have is that the text file consists of around 3000
lines,
and
I want to display the formatted columns to the user before submitting
it
to
the database (perhaps allowing them to edit some fields if they want,
before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so
the
question is::

How can I page the data in the grid, without having to parse the text
each
time - i.e. where can I 'store' the parsed text so that I use that
during
my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

JJ,
What I'd consider doing is to read and parse the file into a
DataTable, and use this to bind to a pageable grid.
Peter

[Alexey Smirnov]

On Jun 7, 9:51 pm, "JJ" <[e-mail]:

Is there a limit to how large the datatable can be within a Session
variable? We're talking a 3000 line tab delimited text file here.

It depends on the web server's memory. Storing large objects in a
user's session is maybe not always a good idea because of performance
but storing and reloading data from memory will be faster than any
other method.

[=?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=]

As Alexey said. A 3000 line text file (or the DataTable you get from
converting it) isn't going to bring down your server. But, if it is
user-specific and you have to use Session, and you have a lot of users, well
- you can do the math.
Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"JJ" wrote:

Is there a limit to how large the datatable can be within a Session
variable? We're talking a 3000 line tab delimited text file here.

JJ

"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in
message news:63**********************************@microsof t.com...

Store the DataTable in Session after you've first constructed it.
Get it back out of Session after changing the PageIndex and rebind:

grid.CurrrentPageIndex =e.newPageIndex;
DataTable myTable = (DataTable)Session["myDataTable"];
grid.DataSource=myTable;
grid.DataBind();

Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"JJ" wrote:

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless I'm
mistaken, that each time you change the page index, you need to rebind
the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for such a
large datatable perhaps?

JJ

<pb*******@gmail.comwrote in message
news:11**********************@q75g2000hsh.googlegr oups.com...
On Jun 7, 11:56 am, "JJ" <[e-mail]:
I have a need to input a large tab delimited text file, which I will
parse
to check it has the expected columns, before allowing the user to
submit
it
to the database. The user may paste the file into a textbox, or upload
it
(haven't decided yet).

The problem I have is that the text file consists of around 3000
lines,
and
I want to display the formatted columns to the user before submitting
it
to
the database (perhaps allowing them to edit some fields if they want,
before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so
the
question is::

How can I page the data in the grid, without having to parse the text
each
time - i.e. where can I 'store' the parsed text so that I use that
during
my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

JJ,
What I'd consider doing is to read and parse the file into a
DataTable, and use this to bind to a pageable grid.
Peter

[JJ]

Looks like storing it in a temp table in the database may be the way to go
then,
Thanks,
JJ

"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in
message news:FA**********************************@microsof t.com...

As Alexey said. A 3000 line text file (or the DataTable you get from
converting it) isn't going to bring down your server. But, if it is
user-specific and you have to use Session, and you have a lot of users,
well
- you can do the math.
Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"JJ" wrote:

>Is there a limit to how large the datatable can be within a Session
variable? We're talking a 3000 line tab delimited text file here.

JJ

"Peter Bromberg [C# MVP]" <pb*******@yahoo.yabbadabbadoo.comwrote in
message news:63**********************************@microsof t.com...

Store the DataTable in Session after you've first constructed it.
Get it back out of Session after changing the PageIndex and rebind:

grid.CurrrentPageIndex =e.newPageIndex;
DataTable myTable = (DataTable)Session["myDataTable"];
grid.DataSource=myTable;
grid.DataBind();

Peter
--
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short urls & more: http://ittyurl.net


"JJ" wrote:

Hi Peter,

Thats what I've done (used a dataTable that is). But I notice, unless
I'm
mistaken, that each time you change the page index, you need to rebind
the
grid to the datatable. I lose the contents of that datatable on each
refresh. To recreate it I need to parse the text again, which will be
a
lengthy process if the text file is 3000 lines.

I'm therefore at a loss as to how to keep hold of the large, formatted
datatable throughout postbacks.
Could I write the gridview as a user control and use viewstate for
such a
large datatable perhaps?

JJ

<pb*******@gmail.comwrote in message
news:11**********************@q75g2000hsh.googleg roups.com...
On Jun 7, 11:56 am, "JJ" <[e-mail]:
I have a need to input a large tab delimited text file, which I
will
parse
to check it has the expected columns, before allowing the user to
submit
it
to the database. The user may paste the file into a textbox, or
upload
it
(haven't decided yet).

The problem I have is that the text file consists of around 3000
lines,
and
I want to display the formatted columns to the user before
submitting
it
to
the database (perhaps allowing them to edit some fields if they
want,
before
clicking submit).
Clearly 3000 lines is too much to display on one gridview page - so
the
question is::

How can I page the data in the grid, without having to parse the
text
each
time - i.e. where can I 'store' the parsed text so that I use that
during
my
paging operations (prior to inserting it into the database)?

Thanks,
JJ

JJ,
What I'd consider doing is to read and parse the file into a
DataTable, and use this to bind to a pageable grid.
Peter

[Alexey Smirnov]

On Jun 7, 11:03 pm, "JJ" <[e-mail]:

Looks like storing it in a temp table in the database may be the way to go
then,
Thanks,
JJ

If you go this way, create a table with columns you need to get from
Excel and an additional column for SessionId (varchar 25). And use
that SessionId to identify data of the current user.

To clean table you can create a sql job and schedule to run... (in
this case you might also need to have a datetime column to know when
data were added)

[JJ]

Good point - I didn't think of that.
Thanks,
JJ
"Alexey Smirnov" <al************@gmail.comwrote in message
news:11**********************@n4g2000hsb.googlegro ups.com...

On Jun 7, 11:03 pm, "JJ" <[e-mail]:

>Looks like storing it in a temp table in the database may be the way to
go
then,
Thanks,
JJ

If you go this way, create a table with columns you need to get from
Excel and an additional column for SessionId (varchar 25). And use
that SessionId to identify data of the current user.

To clean table you can create a sql job and schedule to run... (in
this case you might also need to have a datetime column to know when
data were added)

[JJ]

It doesn't look like I'm able to access the sql agent to create a scheduled
job. Perhaps I could check for records which are 24 hours old when I do
another insert/delete etc and remove them then?

Thanks,
JJ
"Alexey Smirnov" <al************@gmail.comwrote in message
news:11**********************@n4g2000hsb.googlegro ups.com...

On Jun 7, 11:03 pm, "JJ" <[e-mail]:

>Looks like storing it in a temp table in the database may be the way to
go
then,
Thanks,
JJ

If you go this way, create a table with columns you need to get from
Excel and an additional column for SessionId (varchar 25). And use
that SessionId to identify data of the current user.

To clean table you can create a sql job and schedule to run... (in
this case you might also need to have a datetime column to know when
data were added)

[Alexey Smirnov]

On Jun 8, 11:46 am, "JJ" <[e-mail]:

It doesn't look like I'm able to access the sql agent to create a scheduled
job. Perhaps I could check for records which are 24 hours old when I do
another insert/delete etc and remove them then?

Yes, it could be a good idea. In general, you need to delete temporary
data when the user has been confirmed the changes and the data have
been transferred to a working table. But I think it could be a
situation where an user did imported his data to that temporary table,
and didn't confirmed... So, in this case you will need this kind of
cleaning function

[JJ]

Yes - exactly my thoughts. I intend to delete each record with the current
sessionID as it is inserted into the final database table, but, as this is
quiet a lot of data, and the user may close the browser just after adding to
the temp table and before committing to the final table, I think that before
I insert any data into the temp table, I should run some sql:

DELETE
Temp_Table
WHERE
DATEDIFF (hour , AddedDate , GetDate(); ) 2

to delete any records older than 2 hours.

Thanks,
JJ

"Alexey Smirnov" <al************@gmail.comwrote in message
news:11**********************@q69g2000hsb.googlegr oups.com...

On Jun 8, 11:46 am, "JJ" <[e-mail]:

>It doesn't look like I'm able to access the sql agent to create a
scheduled
job. Perhaps I could check for records which are 24 hours old when I do
another insert/delete etc and remove them then?

Yes, it could be a good idea. In general, you need to delete temporary
data when the user has been confirmed the changes and the data have
been transferred to a working table. But I think it could be a
situation where an user did imported his data to that temporary table,
and didn't confirmed... So, in this case you will need this kind of
cleaning function