473,403 Members | 2,293 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,403 software developers and data experts.

data storage question...


Suppose you want to have a large number of items (as an array of struct)
wherein one field is "record-specific" and of variable length, yet not
violate standard C (C90 probably, since C99 isn't available on the platforms
in question) to get it there so that other modules can get the
data at run-time.

I could have the structure contain:

struct element_record foo_tag
{
... /* some number of conventional entries */
size_t blob_size;
unsigned char *blob;
}

Then, manually declare individually named arrays with the raw "blob"
data of varying sizes, and manually put them and their size into an
array of structures like the one above.

Or,

#define WORST_CASE 2048

struct element_record foo_tag
{
... /* some number of conventional entries */
size_t blob_size;
unsigned char blob[WORST_CASE];
}

and initialize them in place, or read it in from a data file, but this
winds up bloating the size of the binary (or memory usage) considerably.

For this application, there is a very wide range from smallest
to largest of this variable portion of the data, (from 2 bytes
up to a little under 2K) and several hundred instances.

This seems like it should be really obvious, but I haven't tripped over
this before and I'm hoping someone can point out what I'm missing without
using some compiler-specific extension.


Nov 14 '05 #1
1 1294
Randy Howard wrote:
Suppose you want to have a large number of items (as an array of struct)
wherein one field is "record-specific" and of variable length, yet not
violate standard C (C90 probably, since C99 isn't available on the platforms
in question) to get it there so that other modules can get the
data at run-time.
C99's "flexible array member" would not work anyhow,
because you can't make an array of FAM-containing structs.
(Think about it: How would array indexing work when the
elements all have different sizes?)
I could have the structure contain:

struct element_record foo_tag
ITYM `element_record' *or* `foo_tag', but not both.
{
... /* some number of conventional entries */
size_t blob_size;
unsigned char *blob;
}

Then, manually declare individually named arrays with the raw "blob"
data of varying sizes, and manually put them and their size into an
array of structures like the one above.

Or,

#define WORST_CASE 2048

struct element_record foo_tag
{
... /* some number of conventional entries */
size_t blob_size;
unsigned char blob[WORST_CASE];
}

and initialize them in place, or read it in from a data file, but this
winds up bloating the size of the binary (or memory usage) considerably.

For this application, there is a very wide range from smallest
to largest of this variable portion of the data, (from 2 bytes
up to a little under 2K) and several hundred instances.

This seems like it should be really obvious, but I haven't tripped over
this before and I'm hoping someone can point out what I'm missing without
using some compiler-specific extension.


Given the wide range of blob sizes, the first method is
probably preferable. You'd wind up with something like

static unsigned char blob1[] = { ... };
static unsigned char blob2[] = { ... };
...

struct element_record {
... conventional entries ...
size_t blob_size;
unsigned char *blob_data;
} blob_array[] = {
{ ..., sizeof blob1, blob1 },
{ ..., sizeof blob2, blob2 },
...
};

You could save a small amount of typing by using a
simple macro like

#define BLOB(name) sizeof name , name

in the initializer.

If you're willing to abandon the array in favor of some
kind of linked data structure, you could use flexible array
members (C99) or "the struct hack" (a widely-accepted abuse
of C90).

Finally, if you've really got a lot of this stuff you
may want to consider the merits of managing it in some other
way than by compiling it into the program. Read it from a
file or files distributed along with the program, perhaps.
There *may* be some (off-topic) advantages to putting the data
where it might be (off-topic) shared by (off-topic) other
programs, but there's no 100% reliable way to tell the compiler
to put your data in (off-topic) shareable memory. On the other
hand, there may be advantages in being able to distribute just
one new data file when something changes, instead of recompiling
and redistributing the entire monolithic program package.

--
Er*********@sun.com

Nov 14 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Lars Behrens | last post by:
Hi there! For a web project I need a little expert help. I don't have written much code yet, just been fiddling around a bit, testing and planning. The web site will have a submission page for...
8
by: Scott J. McCaughrin | last post by:
The following program compiles fine but elicits this message from the linker: "undefined reference to VarArray::funct" and thus fails. It seems to behave as if the static data-member:...
10
by: Zap | last post by:
Widespread opinion is that public data members are evil, because if you have to change the way the data is stored in your class you have to break the code accessing it, etc. After reading this...
6
by: kobu.selva | last post by:
I was recently part of a little debate on the issue of whether constants and string literals are considered "data objects" in C. I'm more confused now than before. I was always under the...
8
by: Art | last post by:
Hi folks, I'm writing a traditional desktop app using VB.NET and am stumbling over what seems like a very basic question: My app does not need to be connected to a server or another computer....
12
by: Chris Springer | last post by:
I'd like to get some feedback on the issue of storing data out to disk and where to store it. I've never been in a production environment in programming so you'll have to bear with me... My...
1
by: SQLMan_25 | last post by:
Hi All, As per BOL, XML data type can store up 2 GB of data. My question is when a row is inserted in a table, for its xml column, 2GB of space will be resered. In other words, how xml is...
6
by: cristizaharioiu | last post by:
Hello, I am beginner with db2 ( DB2 v9.1.0.1 running on RHEL 4); this is my first post. I have this error " SQL0968C The file system is full. SQLSTATE=57011"" in my instance configured with...
8
by: Markus | last post by:
Hello everyone. Recently I stumbled upon an interesting problem related to thread-parallel programming in C (and similarily C++). As an example assume a simple "buffer" array of size 8, e.g....
1
by: smileprince00 | last post by:
hi friends, i am new learner of this sql server 2008.my question is if my primary data file i.e .mdf is restricted to certain amount of memory like 100mb..i have some more secondary data files.i.e...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.