Re: Little direction please Python MySQL

[Ethan Furman]

len wrote:

Hi all;

[snip]

Here is my problem. I need to start doing this in the really world at
my company converting some older cobol system and data to python
programs and MySQL. I have gotten past packed decimal fields and
various other little tidbits. My problem is the data files aren't
little three of four field files but Customer File with 98 fields
etc. I understand building dictionaries and building with zip and I
have even seen a reference to using __setattr__ in an empty class but
I'm having a hard time moving past the little code snippts to real
code.

[snip]

Thanks Len

I've never had the (mis?)fortune to work with COBOL -- what are the
files like? Fixed format, or something like a dBase III style? I
presume also that you only need access to them in COBOL format long
enough to transfer them into MySQL -- true?

~ethan~

[len]

On Nov 13, 7:32*pm, Ethan Furman <et...@stonelea f.uswrote:

len wrote:

Hi all;

[snip]

Here is my problem. *I need to start doing this in the really world at
my company converting some older cobol system and data to python
programs and MySQL. *I have gotten past packed decimal fields and
various other little tidbits. *My problem is the data files aren't
little three of four field files but Customer File with 98 fields
etc. *I understand building dictionaries and building with zip and I
have even seen a reference to using __setattr__ in an empty class but
I'm having a hard time moving past the little code snippts to real
code.

[snip]

Thanks Len

I've never had the (mis?)fortune to work with COBOL -- what are the
files like? *Fixed format, or something like a dBase III style? *I
presume also that you only need access to them in COBOL format long
enough to transfer them into MySQL -- true?

~ethan~

Files are fixed format no field delimiters, fields are position and
length
records are terminated by newline. In cobol the read statement which
read
a record from the file automaticly mapped the date to the fieldnames
in
the cobol file definition.

In python you as the programmer have to do the mapping of data to
fieldnames
whether this is using list and numeric indexing (list[n]),
dictionaries
file['fieldname'] = value or attribute (self.fieldname = value through
some class).
Now in my case I literally have a couple of hundred files and each
file may have
20 or 30 fieldnames and in several cases 100 to 150 fields (customer
file alone
has 98). So as you can imagine standardize the mapping is a big deal
to me.

Now all of the sample code you find (understandably ) usually shows SQL
code
and python code manipulating 3 or 4 fields at the most and one 1 or 2
tables
at a time. In the real world I have programs that will need to work
on 5, 10, and
15 files at a time and 100's of fields. Basicly it is the difference
between
writing your jave, C++, or python program to complete your programming
language
assignment for your college class and then graduating and getting a
job and being
told to write the companies new CRM or ERP system. You can find
plenty of beginning
tutorial and code snippets or esotiric code using stuff for landing
the lunar lander
but where is the middle ground.

That is the stuff I'm looking for.

Please understand this is not a rant against SQL or python or their
communities
but at my own progress in these to become a competent programmer and
I'm sure
as every programmer in the world has experienced, it just never occurs
fast
enough.

Len

[Ethan Furman]

len wrote:

On Nov 13, 7:32 pm, Ethan Furman <et...@stonelea f.uswrote:

>>len wrote:

>>>Hi all;

[snip]

>>>Here is my problem. I need to start doing this in the really world at
my company converting some older cobol system and data to python
programs and MySQL. I have gotten past packed decimal fields and
various other little tidbits. My problem is the data files aren't
little three of four field files but Customer File with 98 fields
etc. I understand building dictionaries and building with zip and I
have even seen a reference to using __setattr__ in an empty class but
I'm having a hard time moving past the little code snippts to real
code.

[snip]

>>>Thanks Len

I've never had the (mis?)fortune to work with COBOL -- what are the
files like? Fixed format, or something like a dBase III style? I
presume also that you only need access to them in COBOL format long
enough to transfer them into MySQL -- true?

~ethan~

Files are fixed format no field delimiters, fields are position and
length
records are terminated by newline. In cobol the read statement which
read
a record from the file automaticly mapped the date to the fieldnames
in
the cobol file definition.

[snip]

>
Len

Are the cobol file definitions available in a file that can be parsed,
or are they buried in the source code?

What type of data is in the files? Integer, float, character, date, etc.

Once you have the data out, will you need access these same cobol files
in the future? (i.e. more data is being added to them that you will
need to migrate)

~ethan~

[len]

On Nov 15, 4:41*pm, Dennis Lee Bieber <[e-mail] com.comwrote:

On Sat, 15 Nov 2008 11:41:17 -0800, Ethan Furman <et...@stonelea f.us>
declaimed the following in comp.lang.pytho n:

len wrote:

* * * * <snip>

Files are fixed format no field delimiters, fields are position and
length
records are terminated by newline. *In cobol the read statement which
read
a record from the file automaticly mapped the date to the fieldnames
in
the cobol file definition.

* * * * Sounds like standard COBOL record definitions. Next factor would be
if they are text format (human readable) or COBOL binary format (and if
so, are they using comp-1 integers or COBOL standard packed decimal?)...
Given the mention of new-line termination, probably not binary (though
technically, COBOL's fixed width files probably don't even require a
new-line).

* * * * In either event, use of the struct module to break the input record
into a cluster of Python strings is probably useful, and may be more
efficient than a series of string slicing operations.

* * * * Also, if the conversion is from file direct to database, it is
likely safe to leave most of the fields in text format; since MySQLdb
passes everything as delimited strings in the INSERT statement -- which
convert from "123.5" to float("123.5") -123.5 only to have the
cursor.execute( ) convert it back to "123.5"

* * * * Exception: might want to convert date/time fields into Python
date/time objects and let MySQLdb handle conversion to/from MySQL
datetime formats.

Are the cobol file definitions available in a file that can be parsed,
or are they buried in the source code?

* * * * Hmmm, ever seen COBOL source? <G>

* * * * Nothing is buried in COBOL -- the data section should have *nicely
laid out record representations ... (it's been some time, so this is
pseudo-COBOL)

01 * * *MYRECORD
* * * * 03 * * *NAME * *PIC A(50)
* * * * 03 * * *DATE
* * * * * * * * 05 * * *MONTH * PIC 99
* * * * * * * * 05 * * *DAY * * * * * *PIC 99
* * * * * * * * 05 * * *YEAR * * * * * *PIC 9999
* * * * 03 * * *AGE * * PIC 999
* * * * 03 * * *ADDRESS
* * * * * * * * 05 STREET * * * PIC X(50)
* * * * * * * * 05 CITY * * * * PIC A(50)
* * * * * * * * 05 STATE * * * * * * * *PIC A(50)
* * * * * * * * 05 ZIP * * * * * * * * *PIC 99999-9999

What type of data is in the files? *Integer, float, character, date, etc.

* * * * If new-line terminated, likely all is human readable text-- see my
above comment re: numeric conversions and MySQL

Once you have the data out, will you need access these same cobol files
in the future? *(i.e. more data is being added to them that you will
need to migrate)

* * * * That is what I considered key also...

* * * * Best would be a one-time conversion -- once the new applications
have been checked out -- meaning the converter may be used multiple
times during development and testing of the new applications (to refresh
the development database with production data), but that in the end the
files become defunct and the new input process directly loads to the
production database.

* * * * No indication of what type of processes the existing COBOL
application is performing, but I can easily visualize a pre-database
processing style, using sorted input files, with parallel readings

read EMPLOYEE (with salary rate)
* * * * * * * * read TIMECARD (with hours)

while EMPLOYEE.ID < TIMECARD.ID
* * * * write EXCEPTION No timecard for EMPLOYEE
* * * * read EMPLOYEE
while TIMECARD.ID < EMPLOYEE.ID
* * * * write EXCEPTION No employee for TIMECARD
* * * * read TIMECARD

compute and write paycheck

repeat until EOF on both EMPLOYEE and TIMECARD

{side note: apologies for piggy-backing -- the original poster is using
an address that my filters are set to kill; as most of the spam on this
group has the same domain}
--
* * * * Wulfraed * * * *Dennis Lee Bieber * * * ** * * KD6MOG
* * * * [e-mail] om.com * * * * * * *wulfr...@besti aria.com
* * * * * * * * HTTP://wlfraed.home.netcom.com/
* * * * (Bestiaria Support Staff: * * * * * * * web-a...@bestiaria. com)
* * * * * * * * HTTP://www.bestiaria.com/

If anyone is interested I have just posted on the group under the
title
'Newbie code review of parsing program Please'

Len