473,569 Members | 2,895 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Flat File To XML Roundtrip

OK I'm on a steep learning curve with XML et.al. and need some advice.

I'm writing a B2B front door for a new application. I have multiple
data suppliers all sending various formats of flat files. Most of the
inbound files are CSV (but a couple are tab-delimitted and one is an
Excel spreadsheet). The goal is to read/process/validate each file and
to "transform" the data into a common, internal, file format which
will by sent on to a downstream application for final processing and
consumption. I'm using VB.Net 2003 as my development platform.

My novice approach to do this will be to:

1) Convert each inbound file to XML (hopfully using some
template-driven parser). I'd like to maintain x number of
file-descriptor templates but use common code to convert the flat
files.

2) Use XSLT to transform the new inbound XML file into the output
target file format

3) Load the XSLT-transformed file (now having the desired outbound XML
format) into a DataSet (for validation and processing)

4) Write/Convert the validated contents of the DataSet to the output
flat file.

My Question is...

Does this sound like a good approach? and are there any good how-to
examples which address these steps? I need to get the job done but I'm
also wanting to use this as a good training exercise in .Net and XML.

Thanks in advance for your input!
Nov 12 '05 #1
6 5561
In there user samples on www.gotdotnet.com you will find an XmlCsvReader
which may be of some help to you for step (1).

Converting the DataSet back to XML is easy, then from XML back to CSV can be
done with an XSL transform, which I think is included in teh XmlCsvReader
download.

So yes, this approach makes sense. A lot of folks are standardizing on XML
for this kind of processing. Hopefully eventually, you'll be able to
convince your business partner to just send you XML at some point in the
future, and then you're processing will get a bit more streamlined.
"Glenn Owens" <go****@nixonpe abody.com> wrote in message
news:bc******** *************** ***@posting.goo gle.com...
OK I'm on a steep learning curve with XML et.al. and need some advice.

I'm writing a B2B front door for a new application. I have multiple
data suppliers all sending various formats of flat files. Most of the
inbound files are CSV (but a couple are tab-delimitted and one is an
Excel spreadsheet). The goal is to read/process/validate each file and
to "transform" the data into a common, internal, file format which
will by sent on to a downstream application for final processing and
consumption. I'm using VB.Net 2003 as my development platform.

My novice approach to do this will be to:

1) Convert each inbound file to XML (hopfully using some
template-driven parser). I'd like to maintain x number of
file-descriptor templates but use common code to convert the flat
files.

2) Use XSLT to transform the new inbound XML file into the output
target file format

3) Load the XSLT-transformed file (now having the desired outbound XML
format) into a DataSet (for validation and processing)

4) Write/Convert the validated contents of the DataSet to the output
flat file.

My Question is...

Does this sound like a good approach? and are there any good how-to
examples which address these steps? I need to get the job done but I'm
also wanting to use this as a good training exercise in .Net and XML.

Thanks in advance for your input!

Nov 12 '05 #2
In there user samples on www.gotdotnet.com you will find an XmlCsvReader
which may be of some help to you for step (1).

Converting the DataSet back to XML is easy, then from XML back to CSV can be
done with an XSL transform, which I think is included in teh XmlCsvReader
download.

So yes, this approach makes sense. A lot of folks are standardizing on XML
for this kind of processing. Hopefully eventually, you'll be able to
convince your business partner to just send you XML at some point in the
future, and then you're processing will get a bit more streamlined.
"Glenn Owens" <go****@nixonpe abody.com> wrote in message
news:bc******** *************** ***@posting.goo gle.com...
OK I'm on a steep learning curve with XML et.al. and need some advice.

I'm writing a B2B front door for a new application. I have multiple
data suppliers all sending various formats of flat files. Most of the
inbound files are CSV (but a couple are tab-delimitted and one is an
Excel spreadsheet). The goal is to read/process/validate each file and
to "transform" the data into a common, internal, file format which
will by sent on to a downstream application for final processing and
consumption. I'm using VB.Net 2003 as my development platform.

My novice approach to do this will be to:

1) Convert each inbound file to XML (hopfully using some
template-driven parser). I'd like to maintain x number of
file-descriptor templates but use common code to convert the flat
files.

2) Use XSLT to transform the new inbound XML file into the output
target file format

3) Load the XSLT-transformed file (now having the desired outbound XML
format) into a DataSet (for validation and processing)

4) Write/Convert the validated contents of the DataSet to the output
flat file.

My Question is...

Does this sound like a good approach? and are there any good how-to
examples which address these steps? I need to get the job done but I'm
also wanting to use this as a good training exercise in .Net and XML.

Thanks in advance for your input!

Nov 12 '05 #3
"Chris Lovett" <chris@!nospam! .net> wrote in message news:<10******* ******@corp.sup ernews.com>...
In there user samples on www.gotdotnet.com you will find an XmlCsvReader
which may be of some help to you for step (1).

Converting the DataSet back to XML is easy, then from XML back to CSV can be
done with an XSL transform, which I think is included in teh XmlCsvReader
download.

So yes, this approach makes sense. A lot of folks are standardizing on XML
for this kind of processing. Hopefully eventually, you'll be able to
convince your business partner to just send you XML at some point in the
future, and then you're processing will get a bit more streamlined.

Chris, thanks for the response. I am currently looking at the
XmlCsvReader - it definitely looks like something that I could use.
Thanks for your hard work and pointing me in this direction.

Can it be modified? (it does appear to be copywritten)...

Is there a way to use an .XSD file to describe/interpret the column
tags instead of using the first line of the file to define the column
names (tags)? This would allow me to fully describe any of the input
files, using separate schemas, without needing to modify them before
the initial XML conversion. The added bonus would be to use the .XSD
files to validate the structure, datatype, cardinality, etc. - could
the XmlCsvReader descend from XmlValidatingRe ader?

Also, I came across a really good MSDN link article on customizing the
XmlReader:

http://msdn.microsoft.com/webservice...ustxmlread.asp

You are correct about the furture of our B2B. This is just the first
step before we "potentiall y" begin requesting each of our data vendors
to supply well-formed XML input streams (preferably via a web
service). But, we needed to get this part done reasonably quickly.

Thanks again,
Glenn
Nov 12 '05 #4
"Chris Lovett" <chris@!nospam! .net> wrote in message news:<10******* ******@corp.sup ernews.com>...
In there user samples on www.gotdotnet.com you will find an XmlCsvReader
which may be of some help to you for step (1).

Converting the DataSet back to XML is easy, then from XML back to CSV can be
done with an XSL transform, which I think is included in teh XmlCsvReader
download.

So yes, this approach makes sense. A lot of folks are standardizing on XML
for this kind of processing. Hopefully eventually, you'll be able to
convince your business partner to just send you XML at some point in the
future, and then you're processing will get a bit more streamlined.

Chris, thanks for the response. I am currently looking at the
XmlCsvReader - it definitely looks like something that I could use.
Thanks for your hard work and pointing me in this direction.

Can it be modified? (it does appear to be copywritten)...

Is there a way to use an .XSD file to describe/interpret the column
tags instead of using the first line of the file to define the column
names (tags)? This would allow me to fully describe any of the input
files, using separate schemas, without needing to modify them before
the initial XML conversion. The added bonus would be to use the .XSD
files to validate the structure, datatype, cardinality, etc. - could
the XmlCsvReader descend from XmlValidatingRe ader?

Also, I came across a really good MSDN link article on customizing the
XmlReader:

http://msdn.microsoft.com/webservice...ustxmlread.asp

You are correct about the furture of our B2B. This is just the first
step before we "potentiall y" begin requesting each of our data vendors
to supply well-formed XML input streams (preferably via a web
service). But, we needed to get this part done reasonably quickly.

Thanks again,
Glenn
Nov 12 '05 #5
The eula is very open. Feel free to use it in your product and make
whatever changes necessary.

You could read the XSD and then pass in the column names:

XmlCsvReader reader = new XmlCsvReader(ur i, nameTable);
reader.ColumnNa mes = new string[] { "foo", "bar", "etc" };
....

The problem is in .NET v1.1 you can't wrap this reader to the
XmlValidatingRe ader. But you could write out the XML, then read it using
XmlTextReader, and then the XmlValidatingRe ader can work over that.

"Glenn Owens" <go****@nixonpe abody.com> wrote in message
news:bc******** *************** ***@posting.goo gle.com...
"Chris Lovett" <chris@!nospam! .net> wrote in message news:<10******* ******@corp.sup ernews.com>...
In there user samples on www.gotdotnet.com you will find an XmlCsvReader
which may be of some help to you for step (1).

Converting the DataSet back to XML is easy, then from XML back to CSV can be done with an XSL transform, which I think is included in teh XmlCsvReader download.

So yes, this approach makes sense. A lot of folks are standardizing on XML for this kind of processing. Hopefully eventually, you'll be able to
convince your business partner to just send you XML at some point in the
future, and then you're processing will get a bit more streamlined.

Chris, thanks for the response. I am currently looking at the
XmlCsvReader - it definitely looks like something that I could use.
Thanks for your hard work and pointing me in this direction.

Can it be modified? (it does appear to be copywritten)...

Is there a way to use an .XSD file to describe/interpret the column
tags instead of using the first line of the file to define the column
names (tags)? This would allow me to fully describe any of the input
files, using separate schemas, without needing to modify them before
the initial XML conversion. The added bonus would be to use the .XSD
files to validate the structure, datatype, cardinality, etc. - could
the XmlCsvReader descend from XmlValidatingRe ader?

Also, I came across a really good MSDN link article on customizing the
XmlReader:

http://msdn.microsoft.com/webservice...ustxmlread.asp
You are correct about the furture of our B2B. This is just the first
step before we "potentiall y" begin requesting each of our data vendors
to supply well-formed XML input streams (preferably via a web
service). But, we needed to get this part done reasonably quickly.

Thanks again,
Glenn

Nov 12 '05 #6
The eula is very open. Feel free to use it in your product and make
whatever changes necessary.

You could read the XSD and then pass in the column names:

XmlCsvReader reader = new XmlCsvReader(ur i, nameTable);
reader.ColumnNa mes = new string[] { "foo", "bar", "etc" };
....

The problem is in .NET v1.1 you can't wrap this reader to the
XmlValidatingRe ader. But you could write out the XML, then read it using
XmlTextReader, and then the XmlValidatingRe ader can work over that.

"Glenn Owens" <go****@nixonpe abody.com> wrote in message
news:bc******** *************** ***@posting.goo gle.com...
"Chris Lovett" <chris@!nospam! .net> wrote in message news:<10******* ******@corp.sup ernews.com>...
In there user samples on www.gotdotnet.com you will find an XmlCsvReader
which may be of some help to you for step (1).

Converting the DataSet back to XML is easy, then from XML back to CSV can be done with an XSL transform, which I think is included in teh XmlCsvReader download.

So yes, this approach makes sense. A lot of folks are standardizing on XML for this kind of processing. Hopefully eventually, you'll be able to
convince your business partner to just send you XML at some point in the
future, and then you're processing will get a bit more streamlined.

Chris, thanks for the response. I am currently looking at the
XmlCsvReader - it definitely looks like something that I could use.
Thanks for your hard work and pointing me in this direction.

Can it be modified? (it does appear to be copywritten)...

Is there a way to use an .XSD file to describe/interpret the column
tags instead of using the first line of the file to define the column
names (tags)? This would allow me to fully describe any of the input
files, using separate schemas, without needing to modify them before
the initial XML conversion. The added bonus would be to use the .XSD
files to validate the structure, datatype, cardinality, etc. - could
the XmlCsvReader descend from XmlValidatingRe ader?

Also, I came across a really good MSDN link article on customizing the
XmlReader:

http://msdn.microsoft.com/webservice...ustxmlread.asp
You are correct about the furture of our B2B. This is just the first
step before we "potentiall y" begin requesting each of our data vendors
to supply well-formed XML input streams (preferably via a web
service). But, we needed to get this part done reasonably quickly.

Thanks again,
Glenn

Nov 12 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

13
3404
by: raykyoto | last post by:
Hi all, I'm sure this is a popular question that comes up every few months here. Indeed, I've looked at some of the past postings, but I would like to ask things differently. Basically, I'm using a flat file to storing data. I have to do this because mySQL is not installed on my web server, and I am not the root user. The amount of...
0
581
by: Glenn Owens | last post by:
OK I'm on a steep learning curve with XML et.al. and need some advice. I'm writing a B2B front door for a new application. I have multiple data suppliers all sending various formats of flat files. Most of the inbound files are CSV (but a couple are tab-delimitted and one is an Excel spreadsheet). The goal is to read/process/validate each...
22
2989
by: Daniel Billingsley | last post by:
Ok, I wanted to ask this separate from nospam's ridiculous thread in hopes it could get some honest attention. VB6 had a some simple and fast mechanisms for retrieving values from basic text files, which in turn could be simply and easily maintained with notepad. I understand the benefits of XML, really, but in the case of configuration...
2
1456
by: mawi | last post by:
Hi there, When removing page children controls created dynamically not in last-to-first order, the close button of the last control looses its event wiring, even though the handler is rewired on each postback. It needs one postback roundtrip to "get it back". Form has an "add panel" button. Using it I dynamically add 3 panels with a...
4
1708
by: Ben | last post by:
So, at my place of employment, we use a national standard to transmit data between certain applications. This standard consists of a fixed width, flat file 4500-some-odd chars wide that contain 375-some-odd columns (Not delimited). This standard is not changing anytime soon, so I attempting to write a library of some sort to read and write...
7
2601
by: Ryan Liu | last post by:
Hi, I want to backup database to a text file. There are binary columns. I read byte from the column and use ASCII encoding convert it to string and write to a text file. But there are bytes large than 7F, and I lost 80Hex, e.g. 91Hex will be 11Hex.
9
2182
by: FFMG | last post by:
In my site I have a config table, (MySQL), with about 30 entries; the data is loaded on every single page load. This is not the only call to the db, (we do a total of about 8 calls to the db). As with many configurations settings, once the options are set the values will not change much. So I thought it would be a good idea to move the data...
2
6121
by: murthydb2 | last post by:
Hi My requirement is that i have to write a stored procedure in db2 and that will be executed in a batch file . Any system error or validation error that occurs inside the db2 sp during the execution need to be written in external flat file ( Log file to know the error ) . Is there anything similar in db2 like UTL_FILE in oracle...
15
5249
by: lxyone | last post by:
Using a flat file containing table names, fields, values whats the best way of creating html pages? I want control over the html pages ie 1. layout 2. what data to show 3. what controls to show - text boxes, input boxes, buttons, hyperlinks ie the usual. The data is not obtained directly from a database.
0
7620
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7931
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8139
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
5515
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5234
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3659
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2120
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1230
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
959
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.