By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,705 Members | 2,013 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,705 IT Pros & Developers. It's quick & easy.

Fixed-length text file to database script

P: n/a
Hi Guys,

I'm new to Python (mostly) and I'm wanting to use it for a new project
I'm faced with.

I have a machine (PLC) that is dumping its test results into a fixed-
length text file. I need to pull this data into a database (MySQL
most likely) so that I can access it with Crystal Reports to create
daily reports for my engineers.

I've been reading the Python manual for about a week now and I'm
learning a lot. Unfortunately, I was given a deadline today that I
cannot meet without a little help.

I need to know how to write a script that will DAILY pull this text
file into a MySQL database.

Can anyone show me how to do this?

Thanks

Stacey
Aug 13 '08 #1
Share this Question
Share on Google+
6 Replies


P: n/a
ss******@gmail.com wrote:
Hi Guys,

I'm new to Python (mostly) and I'm wanting to use it for a new project
I'm faced with.

I have a machine (PLC) that is dumping its test results into a fixed-
length text file. I need to pull this data into a database (MySQL
most likely) so that I can access it with Crystal Reports to create
daily reports for my engineers.

I've been reading the Python manual for about a week now and I'm
learning a lot. Unfortunately, I was given a deadline today that I
cannot meet without a little help.

I need to know how to write a script that will DAILY pull this text
file into a MySQL database.

Can anyone show me how to do this?

Thanks

Stacey
Just use the built in import SQL statement to import the information. You don't
really need a Python script. import can handle fixed field records (as well as
CSV, etc.).

-Larry
Aug 13 '08 #2

P: n/a
Larry Bates wrote:
ss******@gmail.com wrote:
>I have a machine (PLC) that is dumping its test results into a fixed-
length text file. I need to pull this data into a database (MySQL
most likely) so that I can access it with Crystal Reports to create
daily reports for my engineers.
[..]
I need to know how to write a script that will DAILY pull this text
file into a MySQL database.

Just use the built in import SQL statement to import the information.
You don't really need a Python script. import can handle fixed field
records (as well as CSV, etc.).
If the input data has to be pre-processed before storing it into the
database a Python script would be needed.

Just in case somebody needs a module for reading fixed-length files in
the spirit of module csv:

http://www.stroeder.com/pylib/fixedlenfields.py

For the MySQL part:
http://mysql-python.sourceforge.net/

Ciao, Michael.
Aug 14 '08 #3

P: n/a
Michael Ströder wrote:
Larry Bates wrote:
>ss******@gmail.com wrote:
>>I have a machine (PLC) that is dumping its test results into a fixed-
length text file. I need to pull this data into a database (MySQL
most likely) so that I can access it with Crystal Reports to create
daily reports for my engineers.
[..]
I need to know how to write a script that will DAILY pull this text
file into a MySQL database.

Just use the built in import SQL statement to import the information.
You don't really need a Python script. import can handle fixed field
records (as well as CSV, etc.).

If the input data has to be pre-processed before storing it into the
database a Python script would be needed.

Just in case somebody needs a module for reading fixed-length files in
the spirit of module csv:

http://www.stroeder.com/pylib/fixedlenfields.py

For the MySQL part:
http://mysql-python.sourceforge.net/

Ciao, Michael.
While you are correct, that is not what the OP asked. There is no reference to
processing data prior to insertion into MySQL database. Also the OP said they
had a 1 day deadline.

-Larry
Aug 14 '08 #4

P: n/a
I have a machine (PLC) that is dumping its test results into a fixed-
length text file.
While it has nothing to do with python, I found that creating a MySQL
table with the proper fixed length char() fields and using 'load data
infile' was the easiest way to deal with that sort of scenario. The
python script is the secondary part, that handles the normalization
and proper typing of the first table to the second, permanent storage
area. But in this case, the more advanced bits are the database and
SQL details, and python is just a very convenient way to build the SQL
statements and execute them.

I'm really not sure what the best way to deal with fixed length data
is in python. I might define a list with the field lengths and use a
string slicing to get the items.. as a first thought:

myfile = '/somewhere/somefile.txt'
sizes = [16,4,8,8,8]

fd = open(myfile,r)

for line in fd.readlines() :

idx1 = 0
for l in sizes :
Aug 14 '08 #5

P: n/a
Sorry, didn't get to finish my script. Have to figure out the deal
with gmail and the tab key someday.

myfile = '/somewhere/somefile.txt'
sizes = [16,4,8,8,8]

fd = open(myfile,r)

data = []
for line in fd.readlines() :
a = []
idx1 = 0
for l in sizes :
idx2 = idx1 + l
a.append(line[idx1:idx2])
idx1 += l
data.append(a)

fd.close()
print data

This isn't tested, and there are probably more elegant ways to do it,
but for quick and dirty I think it should work.
Aug 14 '08 #6

P: n/a
Larry Bates wrote:
While you are correct, that is not what the OP asked. There is no
reference to processing data prior to insertion into MySQL database.
Also the OP said they had a 1 day deadline.
Larry, having a bad day?

I'm confident that the OP is able to sort out *himself* what he needs.
Also the 1 day deadline would not be an obstacle. Would it for you?

Ciao, Michael.
Aug 14 '08 #7

This discussion thread is closed

Replies have been disabled for this discussion.