473,790 Members | 2,805 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

data storage question...


Suppose you want to have a large number of items (as an array of struct)
wherein one field is "record-specific" and of variable length, yet not
violate standard C (C90 probably, since C99 isn't available on the platforms
in question) to get it there so that other modules can get the
data at run-time.

I could have the structure contain:

struct element_record foo_tag
{
... /* some number of conventional entries */
size_t blob_size;
unsigned char *blob;
}

Then, manually declare individually named arrays with the raw "blob"
data of varying sizes, and manually put them and their size into an
array of structures like the one above.

Or,

#define WORST_CASE 2048

struct element_record foo_tag
{
... /* some number of conventional entries */
size_t blob_size;
unsigned char blob[WORST_CASE];
}

and initialize them in place, or read it in from a data file, but this
winds up bloating the size of the binary (or memory usage) considerably.

For this application, there is a very wide range from smallest
to largest of this variable portion of the data, (from 2 bytes
up to a little under 2K) and several hundred instances.

This seems like it should be really obvious, but I haven't tripped over
this before and I'm hoping someone can point out what I'm missing without
using some compiler-specific extension.


Nov 14 '05 #1
1 1310
Randy Howard wrote:
Suppose you want to have a large number of items (as an array of struct)
wherein one field is "record-specific" and of variable length, yet not
violate standard C (C90 probably, since C99 isn't available on the platforms
in question) to get it there so that other modules can get the
data at run-time.
C99's "flexible array member" would not work anyhow,
because you can't make an array of FAM-containing structs.
(Think about it: How would array indexing work when the
elements all have different sizes?)
I could have the structure contain:

struct element_record foo_tag
ITYM `element_record ' *or* `foo_tag', but not both.
{
... /* some number of conventional entries */
size_t blob_size;
unsigned char *blob;
}

Then, manually declare individually named arrays with the raw "blob"
data of varying sizes, and manually put them and their size into an
array of structures like the one above.

Or,

#define WORST_CASE 2048

struct element_record foo_tag
{
... /* some number of conventional entries */
size_t blob_size;
unsigned char blob[WORST_CASE];
}

and initialize them in place, or read it in from a data file, but this
winds up bloating the size of the binary (or memory usage) considerably.

For this application, there is a very wide range from smallest
to largest of this variable portion of the data, (from 2 bytes
up to a little under 2K) and several hundred instances.

This seems like it should be really obvious, but I haven't tripped over
this before and I'm hoping someone can point out what I'm missing without
using some compiler-specific extension.


Given the wide range of blob sizes, the first method is
probably preferable. You'd wind up with something like

static unsigned char blob1[] = { ... };
static unsigned char blob2[] = { ... };
...

struct element_record {
... conventional entries ...
size_t blob_size;
unsigned char *blob_data;
} blob_array[] = {
{ ..., sizeof blob1, blob1 },
{ ..., sizeof blob2, blob2 },
...
};

You could save a small amount of typing by using a
simple macro like

#define BLOB(name) sizeof name , name

in the initializer.

If you're willing to abandon the array in favor of some
kind of linked data structure, you could use flexible array
members (C99) or "the struct hack" (a widely-accepted abuse
of C90).

Finally, if you've really got a lot of this stuff you
may want to consider the merits of managing it in some other
way than by compiling it into the program. Read it from a
file or files distributed along with the program, perhaps.
There *may* be some (off-topic) advantages to putting the data
where it might be (off-topic) shared by (off-topic) other
programs, but there's no 100% reliable way to tell the compiler
to put your data in (off-topic) shareable memory. On the other
hand, there may be advantages in being able to distribute just
one new data file when something changes, instead of recompiling
and redistributing the entire monolithic program package.

--
Er*********@sun .com

Nov 14 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
3398
by: Lars Behrens | last post by:
Hi there! For a web project I need a little expert help. I don't have written much code yet, just been fiddling around a bit, testing and planning. The web site will have a submission page for attendants of a congress. In a form the user will submit name, mailadress etc. and I plan to store the data in a list of dictionaries: People =
8
4594
by: Scott J. McCaughrin | last post by:
The following program compiles fine but elicits this message from the linker: "undefined reference to VarArray::funct" and thus fails. It seems to behave as if the static data-member: VarArray::funct were an extern, but it is declared in the same file (q.v.). What is the remedy for this? =================
10
5790
by: Zap | last post by:
Widespread opinion is that public data members are evil, because if you have to change the way the data is stored in your class you have to break the code accessing it, etc. After reading this (also copied below for easier reference): http://groups.google.it/groups?hl=en&lr=&safe=off&selm=6beiuk%24cje%40netlab.cs.rpi.edu&rnum=95 I don't agree anymore.
6
2023
by: kobu.selva | last post by:
I was recently part of a little debate on the issue of whether constants and string literals are considered "data objects" in C. I'm more confused now than before. I was always under the understanding that only "named" storage areas(from the standard) were data objects. Since constants and string literals don't have lvalues, they can't be data objects. Yet, I was shown the first page of chapter 2 in K&R2, which states that variables...
8
3773
by: Art | last post by:
Hi folks, I'm writing a traditional desktop app using VB.NET and am stumbling over what seems like a very basic question: My app does not need to be connected to a server or another computer. It just runs locally. So what is the best way to store user-entered data? The app is like an address book, where users can enter contacts and save the data.
12
3767
by: Chris Springer | last post by:
I'd like to get some feedback on the issue of storing data out to disk and where to store it. I've never been in a production environment in programming so you'll have to bear with me... My question is about storing data in a database. Yes I understand that you can link to a database in your program and read and write to the database etc etc. Well, that's all find and dandy but what if the person you're writing the application for...
1
1687
by: SQLMan_25 | last post by:
Hi All, As per BOL, XML data type can store up 2 GB of data. My question is when a row is inserted in a table, for its xml column, 2GB of space will be resered. In other words, how xml is internally stored. Is storage allocation is similar to varchar(max) data type? Thanks in advance for everything.
6
5124
by: cristizaharioiu | last post by:
Hello, I am beginner with db2 ( DB2 v9.1.0.1 running on RHEL 4); this is my first post. I have this error " SQL0968C The file system is full. SQLSTATE=57011"" in my instance configured with AUTO STORAGE ON & AUTORESIZE (indeed, the file system was almost full) I tried to add new storage using "alter database XX add storage
8
1428
by: Markus | last post by:
Hello everyone. Recently I stumbled upon an interesting problem related to thread-parallel programming in C (and similarily C++). As an example assume a simple "buffer" array of size 8, e.g. with static lifetime and exteral linkage. One thread fills the buffer structure, and the other in some way evaluates its contents, e.g. do { synchronize_with_other_thread(); for (int i=0;i<8;++i) {
1
1337
by: smileprince00 | last post by:
hi friends, i am new learner of this sql server 2008.my question is if my primary data file i.e .mdf is restricted to certain amount of memory like 100mb..i have some more secondary data files.i.e .ndf.. wen ever the memory of .mdf is filled in restricted memory, the control storage will automatically switch on to .ndf files and there the data storage can takes place.. now my question is ...if that shifting of data storage to .ndf from .mdf...
0
9666
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9512
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10413
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10200
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9986
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9021
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7530
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5422
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
3
2909
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.