Dynamic C to C Data Transfer

[Tom]

I need my data generating C program on computer #1 to export small
amounts of data (one - 40 byte data structure) periodically (once per
minute) to a C program on computer #2.

I am considering having computer #1 to create two duplicate data files
sequentially. (file_1.dat & file_2.dat).

Then, Computer #2 tries to open file_2.dat and upon failure it opens
file_1.dat.

Due to how small these files are ... one of these two files should
always be "available" for Computer #2.

If either file can not be opened by Computer #1 (Computer #2 is
reading it) ... then that file is simply skipped on that output cycle.
If Computer #2 sees the same time stamp on the most recently read
file_2.dat ... then it reads file_1.dat.

I only know the basics of C programming and hope this can be resolved
within C easily. Both the timing and communication issues are a
challenge for me. Perhaps using a small ram disk would prevent
hammering away on a hard drive? Most likely there is a simple solution
and I am just ignorant of the best approach. Please share how you
would solve this problem. Thanks in advance. :)

[Randy Howard]

Tom wrote
(in article <u8************ *************** *****@4ax.com>) :

I need my data generating C program on computer #1 to export small
amounts of data (one - 40 byte data structure) periodically (once per
minute) to a C program on computer #2.

I am considering having computer #1 to create two duplicate data files
sequentially. (file_1.dat & file_2.dat).

Then, Computer #2 tries to open file_2.dat and upon failure it opens
file_1.dat.

Due to how small these files are ... one of these two files should
always be "available" for Computer #2.

It's not on topic for this newsgroup, but a sockets based (or
other network API) solution might make a lot more sense.

If either file can not be opened by Computer #1 (Computer #2 is
reading it) ... then that file is simply skipped on that output cycle.
If Computer #2 sees the same time stamp on the most recently read
file_2.dat ... then it reads file_1.dat.

You seem to be implying that you will ping pong between two
filenames so the app on the other end knows the file has been
updated. Why not simply have the consumer remove the file when
it is done with it, and the producer not write the next version
until it has been removed? Or just keep appending to the
existing file, and have the consumer do blocking reads?

I only know the basics of C programming and hope this can be resolved
within C easily. Both the timing and communication issues are a
challenge for me. Perhaps using a small ram disk would prevent
hammering away on a hard drive?

I don't see how 40 bytes once a minute will hammer on anything.
Odds are the OS fs cache will make it all invisible most of the
time anyway.

Most likely there is a simple solution
and I am just ignorant of the best approach. Please share how you
would solve this problem. Thanks in advance. :)

I would just open some sort of direct communication channel
between the two programs and send the data when needed and skip
the file twiddling.

--
Randy Howard (2reply remove FOOBAR)
"The power of accurate observation is called cynicism by those
who have not got it." - George Bernard Shaw

[goose]

Tom wrote:

I need my data generating C program on computer #1 to export small
amounts of data (one - 40 byte data structure) periodically (once per
minute) to a C program on computer #2.

<snipped>

I would ask this question in a group dedicated to your platform. Try
to find a group along the lines of comp.unix.progr ammer or
comp.windows.pr ogrammer or similar for your platform.

goose,

[Roland Pibinger]

On Sat, 16 Dec 2006 13:47:40 GMT, Tom wrote:

>I need my data generating C program on computer #1 to export small
amounts of data (one - 40 byte data structure) periodically (once per
minute) to a C program on computer #2.
I am considering having computer #1 to create two duplicate data files
sequentially . (file_1.dat & file_2.dat).
Then, Computer #2 tries to open file_2.dat and upon failure it opens
file_1.dat.
Due to how small these files are ... one of these two files should
always be "available" for Computer #2.

Can you explain the rationale behind that proceeding?

[Tom]

On Sat, 16 Dec 2006 15:33:40 GMT, rp*****@yahoo.c om (Roland Pibinger)
wrote:

>On Sat, 16 Dec 2006 13:47:40 GMT, Tom wrote:

>>I need my data generating C program on computer #1 to export small
amounts of data (one - 40 byte data structure) periodically (once per
minute) to a C program on computer #2.
I am considering having computer #1 to create two duplicate data files
sequentiall y. (file_1.dat & file_2.dat).
Then, Computer #2 tries to open file_2.dat and upon failure it opens
file_1.dat.
Due to how small these files are ... one of these two files should
always be "available" for Computer #2.

Can you explain the rationale behind that proceeding?

The scenario is: Several networked machines. Each performing stock
analysis on individual or a small group of financial instruments and
then passing buy/sell instructions to an order placing machine.

The central machine follows/enforces portfolio rules and the results
of the individual instrument analyses must be agglomerated ... but the
individual analyses is too complex and cumbersome for one personal
computer to perform the analysis for every instrument in the
portfolio. Thus I need to use some sort of parallel processing.

The rational is: I am a novice in data transfer methods and I don't
know where to begin. I am not a proficient or professional programmer
by any stretch. HA!! (Big joke.) However, I am better than average in
technical analysis, mathematics, and process design and optimization.

My hope is there is a best method for doing this within C so that I
don't have to venture outside the structured C approach that I like.
Sort of like a common file shared by several computers and programs at
once? Several programs being able to simultaneously have open and to
have write privileges to a single file is not possible? Even with the
usage of SEEK to the file end and flushing the buffer at each write
.... two programs might be writing at the same time. Thus the single
file approach seems to this rookie as impossible. However, each
program on the server computer(s) could write to a specific file set
up for each financial instrument and the central computer could open,
read, and then close these files at a specified timed interval? The
reason to write to two files and thus duplicating the task is an
attempt to avoid timing conflicts. I am looking for functionality and
simplicity over elegance. Although I find the simplest method that is
reliable to be very elegant in my eyes. This level of data transfer
between multiple programs running simultaneously is obviously new
ground for me. As powerful as C is ... there's got to be a best method
and I'd bet the farm (if I had one) that several of the many gurus in
this user group has the knowledge.

[Jens Thoms Toerring]

Tom <Th********@ear thlink.netwrote :

My hope is there is a best method for doing this within C so that I
don't have to venture outside the structured C approach that I like.
Sort of like a common file shared by several computers and programs at
once? Several programs being able to simultaneously have open and to
have write privileges to a single file is not possible? Even with the
usage of SEEK to the file end and flushing the buffer at each write
...

With all this you are already outside of what the C language comes
with. The C language doesn't deal with much more than a single pro-
gram running on a single machine. When you need communication be-
tween different programs already on the same machine then you must
resort to ways and means your environment allows you to use beside.
If you e.g. are on a machine where the system is POSIX compliant you
could e.g. use the socket interface to communicate between different
processes on the same or on different machines. But this is outside
of the realm of the C language and thus you have to ask these kinds
of questions in a group that deals with your environment - it's not
that people here are unwilling to help but you're asking questions
about baking bread at a carpenters shop and even though the carpen-
ter might know a lot about baking bread he usually will tell you to
go to the bakery next door where (hopefully) the real experts are;-)

Regards, Jens
--
\ Jens Thoms Toerring ___ [e-mail]
\______________ ____________ http://toerring.de

[Tom]

On 16 Dec 2006 18:49:33 GMT, [e-mail] (Jens Thoms Toerring)
wrote:

>Tom <Th********@ear thlink.netwrote :

>My hope is there is a best method for doing this within C so that I
don't have to venture outside the structured C approach that I like.
Sort of like a common file shared by several computers and programs at
once? Several programs being able to simultaneously have open and to
have write privileges to a single file is not possible? Even with the
usage of SEEK to the file end and flushing the buffer at each write
...

With all this you are already outside of what the C language comes
with. The C language doesn't deal with much more than a single pro-
gram running on a single machine. When you need communication be-
tween different programs already on the same machine then you must
resort to ways and means your environment allows you to use beside.
If you e.g. are on a machine where the system is POSIX compliant you
could e.g. use the socket interface to communicate between different
processes on the same or on different machines. But this is outside
of the realm of the C language and thus you have to ask these kinds
of questions in a group that deals with your environment - it's not
that people here are unwilling to help but you're asking questions
about baking bread at a carpenters shop and even though the carpen-
ter might know a lot about baking bread he usually will tell you to
go to the bakery next door where (hopefully) the real experts are;-)

Regards, Jens

Thank you Jens (and others) for directing me towards an operating
system approach. I suppose I must break new ground but I was hoping
for a clever solution without having to learn about sockets and
various other complexities.

Jens' explanation and comparison between bakers and carpenters is
excellent. I understand fully. Still ... I was hoping for an easy way
out.

I guess I will put away my Milwaukee Holeshooter, Porter-Cable
circular saw, and Delta table saw and go shopping for a nice
convection air oven and a Kitchen Aid stand mixer. At least the toys
are still electrical! :)

I am so impressed by the knowledge of some in this group that I can't
begin to blasphemy with the claim of "real" experts being elsewhere
.... but certainly I will find other experts on my journey. I hope they
are not offended after their efforts at educating me and I then tell
them they bake a great cake? Perhaps that baking will remain our
secret. ;) After all, real men construct things ... right? Funny how
both carpenters and bakers wear aprons!! Yeah, I have one too. :))

[Roland Pibinger]

On Sat, 16 Dec 2006 18:11:49 GMT, Tom wrote:

>The scenario is: Several networked machines. Each performing stock
analysis on individual or a small group of financial instruments and
then passing buy/sell instructions to an order placing machine.

....

>Sort of like a common file shared by several computers and programs at
once?

From that I suppose you machines have access to a common (LAN)
directory which they shall use to exchange data via files?

>Several programs being able to simultaneously have open and to
have write privileges to a single file is not possible?

That depends on the file system (better assume nothing).

>Even with the
usage of SEEK to the file end and flushing the buffer at each write
... two programs might be writing at the same time. Thus the single
file approach seems to this rookie as impossible.

Probably yes.

>However, each
program on the server computer(s) could write to a specific file set
up for each financial instrument and the central computer could open,
read, and then close these files at a specified timed interval? The
reason to write to two files and thus duplicating the task is an
attempt to avoid timing conflicts.

I don't quite understand that. But let's assume we have 2 computers
(pc1, pc2) that have to exchange data via a common directory and shall
not disturb each other. I would define a name and behavior convention:
In the specified common directory pc1 opens only the file with the
name 'to_pc1.dat' and pc2 opens only the file 'to_pc2.dat'. To
exchange data between pc1 and pc2 the following steps need to be
performed:
- pc1 copies the data to the common directory with a temporary file
name
- after copying is finished pc1 renames the file to 'to_pc2.dat'
- pc2 opens (only!) the file 'to_pc2.dat' and changes it
- afterwards pc2 renames the file to 'to_pc1.dat'
- and so on
The point is that pc1 and pc2 never concurrently access (not even
read) the same file and that the rename() function is atomic. You need
to implement some sort of polling to make it work and the approach
isn't very fast. But everything can be done in Standard C (but you
should e.g. look for the sleep() function for polling).

Hope that's relevant to your problem,
Roland Pibinger

[Kenny McCormack]

In article <45************ **@news.utanet. at>,
Roland Pibinger <rp*****@yahoo. comwrote:

>On Sat, 16 Dec 2006 18:11:49 GMT, Tom wrote:

>>The scenario is: Several networked machines. Each performing stock
analysis on individual or a small group of financial instruments and
then passing buy/sell instructions to an order placing machine.

Is the platform, by any chance, Windows?
If so, try using INI files. Works real well.

[CBFalconer]

Tom wrote:

rp*****@yahoo.c om (Roland Pibinger) wrote:

>Tom wrote:

>>I need my data generating C program on computer #1 to export
small amounts of data (one - 40 byte data structure) periodically
(once perminute) to a C program on computer #2.

I am considering having computer #1 to create two duplicate data
files sequentially. (file_1.dat & file_2.dat). Then, Computer #2
tries to open file_2.dat and upon failure it opens file_1.dat.

Due to how small these files are ... one of these two files
should always be "available" for Computer #2.

Can you explain the rationale behind that proceeding?

The scenario is: Several networked machines. Each performing stock
analysis on individual or a small group of financial instruments
and then passing buy/sell instructions to an order placing machine.

The central machine follows/enforces portfolio rules and the
results of the individual instrument analyses must be agglomerated
... but the individual analyses is too complex and cumbersome for
one personal computer to perform the analysis for every instrument
in the portfolio. Thus I need to use some sort of parallel
processing.

The rational is: I am a novice in data transfer methods and I
don't know where to begin. I am not a proficient or professional
programmer by any stretch. HA!! (Big joke.) However, I am better
than average in technical analysis, mathematics, and process
design and optimization.

Try getting and reading "The Little Book of Semaphores":

<http://greenteapress.c om/semaphores/>

--
Chuck F (cbfalconer at maineline dot net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net>