Hello-
I'm creating fixed-length record layouts for various record translations
I need to perform. I have been using a home-grown object,
"FixedLengthRec ord" for about 4 years now and am very happy with it.
Almost. The burr under my saddle occurs when I serialize the record.
Currently, I create an empty text field and for each record in my
database (all record layouts, data types, lengths, defaults, etc. are
held in an SQL server) I concatenate the field to the text segment. I
have been led to believe this is bad form because Python will copy the
entire segment each time I add a field. Up until now, it was not a big
deal because the segments had at most 20 fields. I have just been
handed a record layout that is almost 5000 bytes long with 350 fields in
it. Gulp!! Although I could let my object churn away on this bad boy,
I'd like to know if there is a more pythonic way to serialize the record.
One thought I had, which might lead to an addition to the language, was
to use the struct module. If I could feed the pack method a format
string then a tuple of values (instead of individual values), then I
could create the format string once, then pass it a tuple with the
values for that record. Just a thought.
So, gurus, what are your suggestions to tame this record? Are there
easier ways that I'm just not seeing?
Thanks,
--greg 2 5249
Greg Lindstrom wrote: [...] I concatenate the field to the text segment. I have been led to believe this is bad form because Python will copy the entire segment each time I add a field. Up until now, it was not a big deal because the segments had at most 20 fields. I have just been handed a record layout that is almost 5000 bytes long with 350 fields in it. Gulp!! Although I could let my object churn away on this bad boy, I'd like to know if there is a more pythonic way to serialize the record.
The standard Python way would be to read your fields into a list, and
then join() the list. Psuedocode-ish example:
values = []
for field in list_of_fields:
this_value = database.get(fi eld) # or however you want to
retrieve it
values.append(t his_value)
text = ''.join(values)
Of course, if you want to be able to reverse the process, then you will
need some kind of format definition. This could be an explicit list of
column numbers where fields break, or an implicit list in the form of a
struct module format string, or whatever. Using the struct module would
(I believe) be a reasonable way of getting data into/out of the
fixed-length block.
Jeff Shannon
Technician/Programmer
Credit International
On Wed, 08 Dec 2004 17:29:19 -0600, Greg Lindstrom wrote: One thought I had, which might lead to an addition to the language, was to use the struct module. If I could feed the pack method a format string then a tuple of values (instead of individual values), then I could create the format string once, then pass it a tuple with the values for that record. Just a thought.
The language extension has been around since the early days. It was
originally called 'apply'. But now is called '*'. import struct fmt = '8s6s2s' v = ('foo','bar','A A') struct.pack(fmt ,*v)
'foo\x00\x00\x0 0\x00\x00bar\x0 0\x00\x00AA'
--
Stuart D. Gathman <st****@bmsi.co m>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Adrian Parker |
last post by:
I'm using the code below in my project. When I print all of these fixed
length string variables, one per line, they strings in questions do not
properly pad with 0s. strQuantity prints as " 4". Six spaces than the
value of intQuantity. This is correct.
But all the others end up being string objects of only 6 characters long
(with the exception of strTotal). The left most positions of the string
object are being padded with one...
|
by: Greg Lindstrom |
last post by:
Hello-
I have created a class, FixedLengthRecord.py, that allows me to manipulate
fixed length records in the routines I write. I recently converted it to
read record layouts from an SQL server (it used to read config files) and I
am thinking of making another change to make my code cleaner .
The heart of the class is a dictionary named "fields" that stores the
current value, length, data type, and default value for the field. Each...
|
by: Todd_M |
last post by:
I was wondering what anyone might suggest as "best practice" patterns for
streaming out fixed formatted text files with C#? Let's say we get our data
in a dataset table and we need to iterate over thousands..potentially tens
of thousands of rows to create a properly formatted text file -- like an ACH
file, for example. In there you typically have a header, a body block of
detail rows and a trailer. (The trailers usually contain sums of the...
|
by: Jami Bradley |
last post by:
Hi,
I'm looking for an efficient way to do this, because I know it will be heavily used :-)
I have a fixed width string and I need to substitute a substring of characters with new values. I
can do this with 2 substring calls, but it will need to rebuild the string just to write a few
characters.
Here is the simple, but inefficient, version:
string s = "0123456789";
|
by: ecov |
last post by:
Is there any easy way to read into a dataset a file that was created
from a BCP command. The file is in a fixed length format. I would
also like to be able to write out a dataset to the same type of file
structure.
The obvious solution would be to use XML file processing however that
is not an option as I have a third party vendor that is not capable of
processing XML files.
| |
by: booksnore |
last post by:
I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million...
|
by: JimN1 |
last post by:
Hi,
I am looking for an example to convert a flat text file with variable length records to a flat file with fixed length records. Does anyone have an example?
|
by: htres |
last post by:
Hello,
I'm new to programming and C#, so please bear with my ignorance!
I need to extract jpeg images and header data from a binary file.
The binary file is formatted with several fixed length fields containing information about the jpeg image, followed by the jpeg itself, followed by more header data, another jpeg, etc...
Using a FileStream and BinaryReader I am able to read and store the metadata, because I know the length of the...
|
by: Neil Cerutti |
last post by:
Many of the file formats I have to work with are so-called
fixed-format records, where every line in the file is a record,
and every field in a record takes up a specific amount of space.
For example, one of my older Python programs contains the
following to create a fixed-format text record for a batch of new
students:
new = file("new.dat", "w")
if not new:
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
| |
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |