473,372 Members | 819 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,372 software developers and data experts.

communication between mini-modules

Hi,

I am looking for a class or any information which would allow me to
make communicate mini-module in c++. I have been looking

on the net for some examples of such implementation, but I did not
find any good one. This is why I am asking here if some

of you know where to find some information on the internet about such
implementation.

I explain you what I mean per communication between mini-modules.

I want to implement an application for example composed from modules
A,B,C,D and E.

Module A : simply reads the content of a file per n bytes
Module B : convert the binary content in integers numbers text
Module C : Insert html tags in text
Module D : Change text color
Module E : Display text on the screen

What I want to do is using a communication tool (a class) between
these modules, to create dinamicaly a chain which would

allow me to create any combinaision between these modules.

For e.g.

1: A -> B -> C -> D -> E (go back to A)
or
2: A -> B -> D -> C -> E (go back to A)

You can see that the program would be able to first insert the html
tags before to change the color (1), or do it the other

way around (2).

The really interesting part is that the order could be then
configurable and changed whenever a user need to insert a new

module without compliling a new program.

I have been searching extensively the internet, believing several
times that I found the solution.
I though CORBA was the solution, but it seems that even if it could do
the job, the kde team as abandonned it because it was

too slow.
I tryed communicating through the pipe and using stdin stdout, but
even if this works, that is a one way channel and does

not allow complex connection between modules.
I have tryed using soap but this is definitly to slow.
I have tryed using ip (using sockets) but this is not the best
solution if all the modules are on the same machine (though

this is quite good if the modules are on separate machines).

Thanks in advance,

Emmanuel Charruau
Jul 23 '05 #1
9 2709
shp
Hi!

I am in a similar situation, wanting to dynamically be able to chain up
different modules and have them communicate.

I have considered using the signals/slots paradigm for the
communication. Have you considered that Emmanuel?

Cheers,
Kim

Jul 23 '05 #2
"Emmanuel Charruau" <ec*******@yahoo.com> wrote in message
news:bb**************************@posting.google.c om...
I am looking for a class or any information which would allow me to
make communicate mini-module in c++. I have been looking
on the net for some examples of such implementation, but I did not
find any good one. This is why I am asking here if some
of you know where to find some information on the internet about
such implementation.

I explain you what I mean per communication between mini-modules. [...]
Which looks pretty much like standard unix pipes: you create
multiple separate programs that read from their standard input
and write to their standard output, and then a shell can be used
to 'pipe' data through a set of programs in any order you want.
The really interesting part is that the order could be then
configurable and changed whenever a user need to insert a new
module without compliling a new program.

I have been searching extensively the internet, believing several
times that I found the solution.
I though CORBA was the solution, but it seems that even if it could
do the job, the kde team as abandonned it because it was too slow.
I tryed communicating through the pipe and using stdin stdout, but
even if this works, that is a one way channel and does
not allow complex connection between modules. What type of connections are you looking for? Why is a one-way
channel not good enough?
I have tryed using soap but this is definitly to slow.
I have tryed using ip (using sockets) but this is not the best
solution if all the modules are on the same machine (though
this is quite good if the modules are on separate machines).


The key questions are:
- what are the kinds of messages you want to send between the
objects, and in what directions?
- how do you want to schedule the processing of the data?

For the latter, do you want a chunk of data provided as input to
be immediately processed all the way down the chain, or are you
looking for a somewhat more asynchronous processing?

I mean, in the simplest case you could just use a base class such as:

// base class for an object that receives data
class DataHandler
{
public:
virtual void process(std::string const& data)=0;
virtual ~DataHandler() {}
}

// base class for a data processing filter
class DataPipe : public DataHandler
{
public:
DataPipe(DataHandler& next) : next_(next) {}
virtual void process(std::string const& data)=0;
//NB: implementations shall call forward() to pass data further
protected:
void forward(std::string const& data) { next->process(data); }
private:
DataHandler& next_;
};

You may also replace the 'push' model above with a 'pull'
model, where the end-user of the data requests for the
next 'packet' from a (chained) data source.
Another approach would be to 'post' data packets to be processed
to a global queue of <data,object&> pairs, and use some kind of
scheduling (possibly distributing the processing to multiple
threads).

There are many related techniques, design patterns, and libraries
that can be used. Similar architectures are used for implementing
drivers, for multimedia frameworks (e.g. QuickTime), and many
other problem domains. You haven't provided a sufficient level
of detail to allow the selection of a specific approach.
Cheers,
Ivan

--
http://ivan.vecerina.com/contact/?subject=NG_POST <- email contact form
Brainbench MVP for C++ <> http://www.brainbench.com
Jul 23 '05 #3
Hi Thanks a lot fot your answer,
I answer your question in the text
Which looks pretty much like standard unix pipes: you create
multiple separate programs that read from their standard input
and write to their standard output, and then a shell can be used
to 'pipe' data through a set of programs in any order you want.
Exactly and this is one of the goal of the architecture. One of the
goal is like I wrote before is to be able to interconnect different
modules together in different orders, but an other one is to try to
keep the unix way of working, small parts but which works fine. This
way if we create a new module we can concentrate our testing mainly on
this module and not on th whole application.

The key questions are:
- what are the kinds of messages you want to send between the
objects, and in what directions?
- how do you want to schedule the processing of the data?
This will mainly be big bunch of data (1000 lines of text data in a
row) not so often, or very small amount of data (1 line of text to
filter) very often, this depending on the way I will find to
implement.
The application must analyse text lines with number in it (could be a
list of telephone call for example). And apply rules on the lines it
decided to filter (for example all the telephone call starting with
12345). The rule can be for example to add 0033 to the beginning of
the filtered number to tell it is a french number.

I could implement it easily in a monolityque way, but this would
really be interesting to do it per separate programs, to be able to be
very flexible to assemble the programs in different order.

I give you an example where it would be extremly useful. I have a
module A which read a text file line by line, A->B. The module B
filters all the numbers starting with 12345 then gives the lines
containing these numbers to C. C look if the number is a french
number. If it is a french number the text line is given to D and a
copy of the line is slighly modified and sent again to B, being
filtered again.
About the scheduling, I was thinking that a if we process the data
line by line each line would go at a time through the modules, and
wait that the next module is finished with it works to be pushed.
Maybe testing a "I am ready to process" flag.

For the latter, do you want a chunk of data provided as input to
be immediately processed all the way down the chain, or are you
looking for a somewhat more asynchronous processing?

I mean, in the simplest case you could just use a base class such as:

// base class for an object that receives data
class DataHandler
{
public:
virtual void process(std::string const& data)=0;
virtual ~DataHandler() {}
}
Here I think I understand, for every new module I can create a new
class derived from DataHandler and implement a specific virtual
function corresponding to this process. Which allows me to "plug" new
modules (even without recompiling the project if I recall the theory).

// base class for a data processing filter
class DataPipe : public DataHandler
{
public:
DataPipe(DataHandler& next) : next_(next) {}
virtual void process(std::string const& data)=0;
//NB: implementations shall call forward() to pass data further
protected:
void forward(std::string const& data) { next->process(data); }
private:
DataHandler& next_;
};
Ok it took me more time to understand this one.
I derive a class DataPipe from the base class DataHandler.
Then I initialise the variable next_ with the value given in
DataPipe(next). _next being in fact the next module (process) or its
pointer at least.
Then Forward executes the next function.
I have to try that definitly!


You may also replace the 'push' model above with a 'pull'
model, where the end-user of the data requests for the
next 'packet' from a (chained) data source.
Another approach would be to 'post' data packets to be processed
to a global queue of <data,object&> pairs, and use some kind of
scheduling (possibly distributing the processing to multiple
threads).

There are many related techniques, design patterns, and libraries
that can be used. Similar architectures are used for implementing
drivers, for multimedia frameworks (e.g. QuickTime), and many
other problem domains. You haven't provided a sufficient level
of detail to allow the selection of a specific approach.

Are the info I gave you enough to choose a specific design patterns,
and library?

Really thanks a lot for the answer you already gave me answer, it
already proves me that the realisation is possible. It is even better
to know that some technics are available somewhere.

Regards,

Emmanuel

Cheers,
Ivan

Jul 23 '05 #4
"Emmanuel Charruau" <ec*******@yahoo.com> wrote in message
news:bb**************************@posting.google.c om...
This will mainly be big bunch of data (1000 lines of text data in a
row) not so often, or very small amount of data (1 line of text to
filter) very often, this depending on the way I will find to
implement.
The application must analyse text lines with number in it (could be a
list of telephone call for example). And apply rules on the lines it
decided to filter (for example all the telephone call starting with
12345). The rule can be for example to add 0033 to the beginning of
the filtered number to tell it is a french number. .... About the scheduling, I was thinking that a if we process the data
line by line each line would go at a time through the modules, and
wait that the next module is finished with it works to be pushed.
Maybe testing a "I am ready to process" flag. .... Are the info I gave you enough to choose a specific design patterns,
and library? I think so - the lines can be processed independently, so this
should be a relatively simple case.
The example I provided should work. One potential inconvenience
of it is that the message is forwarded recursively (shouldn't matter,
really, can even be good for debugging/flexibility, but some would
argue that this is suboptimal for long chains of filters).

An alternative approach would be something like:
// base class for all data 'filters'
class Filter {
public:
// modifies line, returns false to cancel further processing.
virtual bool transform(std::string& line)=0;
virtual ~Filter() {}
};
class MultiFilter : public Filter
{
public:
MultiFilter() {}

// to add a 'filter' to the chain
void appendFilter( Filter& filter ) { filters_.push_back(&filter); }

// implementation of Filter interface:
bool tranform(std::string& line)
{
for( Filters::iterator scan = filters_.begin()
; scan != filters_.end() ; ++filters_ )
{
if( ! (**scan).transform(line) )
return false;
}
return true;
}
private:
typedef std::vector<Filter*> Filters;
Filters filters_;
}

void applyTransform( std::istream& source
, std::ostream& destination
, Filter& filter )
{
std::string line;
while( std::getline(source,line) )
{
if( filter.tranform(line) )
destination << line << '\n';
}
}

And the top-level code could be something like:
int main()
{
MultiFilter mf;
FilterA fa; // a subclass of Filter
mf.appendFilter( fa );
FilterB fb; // etc...
mf.appendFilter( fb );
applyTransform(std::cin, std::cout, mf);
}

Really thanks a lot for the answer you already gave me answer, it
already proves me that the realisation is possible. It is even better
to know that some technics are available somewhere.

From your description, it still looks like you could use sed or
a Perl script ;) but the base infrastructure is easy to implement
in C++.
Regards,
Ivan
--
http://ivan.vecerina.com/contact/?subject=NG_POST <- email contact form
Jul 23 '05 #5
Hi Ivan,

Thanks again for your answer. I will try to implement this solution
next week and I will tell you the result.

Emmanuel
Jul 23 '05 #6
Hi Kim,

I will first try Ivan solution, but this seems a good idea, and
knowing that I am using qt to program that would be an easy solution
to try.
Thanks for the hint.

Regards,

Emmanuel
sh*@elreki.net wrote in message news:<11**********************@l41g2000cwc.googleg roups.com>...
Hi!

I am in a similar situation, wanting to dynamically be able to chain up
different modules and have them communicate.

I have considered using the signals/slots paradigm for the
communication. Have you considered that Emmanuel?

Cheers,
Kim

Jul 23 '05 #7
Hi Ivan,

I implemented your solution and it works perfectly. I slightly
modifyed the project by replacing the loop scanning the vector by a
mapping letting the user choosing the filter execution the order he
needs.

What I will try now to do is to find how to make the differents
modules working at the same time. Let say if module A has already
given its informations to module B it can process some new information
at the same time module B is processing its data.

Regards,

Emmanuel
Jul 23 '05 #8
Hi,

I have been looking the web to find a solution to run multiple modules
(process) at the same time. I found out that I could use
multithreading, but I also have to manage scheduling the messages
communication between the modules.
What I would like to know is, does it exist a C++ library helping to
manage running several processes at the same time and scheduling some
messages communication between the modules. All that in the same C++
program? If not, can somebody suggests some books explaining how to
create a program allowing this?

Regards,

Emmanuel
Jul 23 '05 #9
"Emmanuel Charruau" <ec*******@yahoo.com> wrote in message
news:bb**************************@posting.google.c om...
(re)bonjour Emmanuel,
I have been looking the web to find a solution to run multiple modules
(process) at the same time. I found out that I could use
multithreading, but I also have to manage scheduling the messages
communication between the modules.
What I would like to know is, does it exist a C++ library helping to
manage running several processes at the same time and scheduling some
messages communication between the modules. All that in the same C++
program? If not, can somebody suggests some books explaining how to
create a program allowing this?


First of all, you need to consider the fact that unless the
program is running on a mutliprocessor/multicore or at least
hyperthreaded machine, it is unlikely to run faster (just)
because you use multithreading or multiprocessing.

Second, multi-threaded programming is notoriously difficult.
Especially in performance-critical applications!

The proper way to share the load between multiple threads
depends a lot on the specific of the application, i.e. the
kind of data source and data destination being used.
Ways to parallelize the processing of your chunks of data
you use include:
- Splitting the chain of filters in two, with a single
(serialized) queue of data chunks used for data exchange.
This would be related to a producer/consumer pattern.
What is difficult here is to find the right split point
so that both threads work at full capacity.
- Having two identical threads being feeding themselves
from a common data source, and sending results to the
same output. What has to be watched for in this case
is that all accesses to the source/destination need
to be serialized, and some extra logic may be required
if the data chunks must be written to the destination
in the same order as they were written in the source.
In either of the above cases, you could use more that
two threads if this is supported by your system.

Multithreaded programming in C++ requires you to rely
on platform-specific libraries, although some cross-platform
wrappers exist as well (such as boost::threads, see
http://www.boost.org/doc/html/threads.html).
Some pointers to references are provided at the bottom
of that page. I liked Schmidt's "Pattern-Oriented (System)
Architecture" (volume 2).
I hope this helps,
Ivan
--
http://ivan.vecerina.com/contact/?subject=NG_POST <- email contact form
Brainbench MVP for C++ <> http://www.brainbench.com
Jul 23 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Tim Reynolds | last post by:
I support a .Net application running on a SERVER accessing MF Db2 data. Occasionally, we have some type of connection problem that we have been unable to debug up to this point. We typically...
2
by: Daniel | last post by:
Hi , i was just wondering if it is possible to create a mini game in vb.net say for example like space invaders with a spacecraft coming across the top of the screen in a web form dropping...
0
by: tbatwork828 | last post by:
VS 2005. I have compiled my dlls/exes in Release mode and also setting Debug Info="full"under Project - Properties - Build - select "Release" under "Configuration" drop down - the Advanced button...
3
by: aurora | last post by:
This is an entry I just added to ASPN. It is a somewhat novel technique I have employed quite successfully in my code. I repost it here for more explosure and discussions. ...
7
by: RSH | last post by:
I am working through some design patterns, and one recurring theme for me is the need to be able to communicate between objects while promoting encapsulation and loose coupling between them. ...
2
by: oyster | last post by:
The following is my pure-python wxwidgets test. It runs and give a frame, but soon python comes up to say "unexpected parameter 'mini.py'" and I have to close it. I cannot find the reason. Can...
8
by: geert | last post by:
Hi all, I have a mac mini running maocosx 10.5 leopard I want to deploy a django project on. My backend is MySQL, and I have it running as a 64- bit app. Of course, apache2 is also running as...
5
by: =?Utf-8?B?R3VpbmVhcGln?= | last post by:
Hi, I just wrote a mini C# lab for myself, I think it may be useful for others, so I shared it on my blog. If you often need to write only serveral lines of code and don't want to waste time on...
1
by: =?Utf-8?B?VGhhZA==?= | last post by:
My Windows media player won't find the mini dvd - any help on how I can play these mini dvd's recorded on my camcorder in media player? I have media player 9. -- Thanks, Thad
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.