473,326 Members | 2,732 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

c source

I need some help in finding a program written in c that is large enough that
it takes
10-20 minutes to compile/link on an *average* machine.
Why? I am in a HPC class and I am thinking about trying to speed up
compiling of larger programs. At least not so large at first.

I have done a bunch of searching. I have found some apps but most of them
are too big and too complicated.
I know this is a bit off topic, apologies to the purists..
Thanks for any help.

Mike
Nov 13 '05 #1
8 1996
Hi

Michael Gaab wrote:
I need some help in finding a program written in c that is large enough
that it takes
10-20 minutes to compile/link on an *average* machine.
Why? I am in a HPC class and I am thinking about trying to speed up
compiling of larger programs. At least not so large at first.

I have done a bunch of searching. I have found some apps but most of them
are too big and too complicated.
I know this is a bit off topic, apologies to the purists..
Thanks for any help.

Mike


Something that takes 10-20 minues to compile needs to be complicated ... :-)
Nov 13 '05 #2
Michael Gaab wrote:
I need some help in finding a program written in c that is large enough that
it takes
10-20 minutes to compile/link on an *average* machine.
Why? I am in a HPC class and I am thinking about trying to speed up
compiling of larger programs. At least not so large at first.
Just a few questions and thoughts:

Should it be a single program file, or are multiple files ok? Are you
planning on timing the compilation-step proper, or also the
preprocessing, assembly, and linking steps?

Will this be just an exercise, or are you aiming for something that is
practically useful? In the latter case there should be examples already
out there :-)
I have done a bunch of searching. I have found some apps but most of them
are too big and too complicated.


Something that compiles for 10--20 minutes is almost bound to be either
big or complicated. One way of getting e.g. gcc to slow down (even with
not-so-big-or-not-so-complicated programs) is to enable function
inlining and set the inlining threshold to a very high value. This
worked wonders for the simple chess program I once wrote, it took ages
to compile. Of course, this is mainly because the compiler's data
structures blow up. You're essentially measuring near-worst-case cache
behaviour for memory/cpu throughput or even I/O throughput (swapping) in
cases like this.

Another way of producing high compile times is of course to generate
source code. Anyway, this would usually yield non-typical code compared
to real application code, so that is perhaps not suited for you. And to
get in the 10--20 minutes range, it may be better to write a program
generates a program that generates a program, or something like that :-)

Of course, in C++ this would be a much easier task since you can do
awful things with templates.

Best regards,

Sidney

Nov 13 '05 #3
On Sat, 15 Nov 2003 18:11:51 -0500, Michael Gaab wrote:
I need some help in finding a program written in c that is large enough
that it takes
10-20 minutes to compile/link on an *average* machine. Why? I am in a
HPC class and I am thinking about trying to speed up compiling of larger
programs. At least not so large at first.

I have done a bunch of searching. I have found some apps but most of
them are too big and too complicated.


Look at big open source projects like X, the Linux kernel, or Samba. All
of these are several hundred thousand lines of code. Of course anything
this size is going to be complicated. If you need something predictable
and ansi that you can try with different compilers and platforms then I
think your best bet will be to dynamically generate it. Think of 10
C constructs that can be "exploded" to be huge like huge static structure
initializers or large combinations of conditional statements that call
random functions and weave the whole thing together into one big 1MB set
of 10 source files. You probably only need it to compile for 2 minutes
and loop 10 times for startup costs to be statistically insignificant as
opposed to compiling something for 20 minutes.

Mike
Nov 13 '05 #4

"Sidney Cadot" <si****@jigsaw.nl> wrote in message
news:bp**********@news.tudelft.nl...
Michael Gaab wrote:
I need some help in finding a program written in c that is large enough that it takes
10-20 minutes to compile/link on an *average* machine.
Why? I am in a HPC class and I am thinking about trying to speed up
compiling of larger programs. At least not so large at first.
Just a few questions and thoughts:

Should it be a single program file, or are multiple files ok?


Multiple files.
Are you
planning on timing the compilation-step proper, or also the
preprocessing, assembly, and linking steps?
We were planning on parallelizing the compilation, e.g., gcc -c. I am not
a compiler expert, or for that matter an expert at anything. :-) Just keep
plugging along with what God gave me. Doing the best I can.
Will this be just an exercise, or are you aiming for something that is
practically useful? In the latter case there should be examples already
out there :-)
It is for a project in a parallel program class. Now that I have a better
understanding of HPC, I think I would have chosen a different thing to
parallel. But then on the other hand, it seems very practical if
complication/linking could be sped up on large projects. Not sure about
that assumption because I have never worked on any.
I have done a bunch of searching. I have found some apps but most of them are too big and too complicated.


Something that compiles for 10--20 minutes is almost bound to be either
big or complicated. One way of getting e.g. gcc to slow down (even with
not-so-big-or-not-so-complicated programs) is to enable function
inlining and set the inlining threshold to a very high value. This
worked wonders for the simple chess program I once wrote, it took ages
to compile.


Would you mind sharing the chess program? The issue in our class is whether
or not we can realize any speedup, not whether or not we have developed the
code ourselves or not.

I am not sure about inlining, I'll have to play around with that a bit.

Of course, this is mainly because the compiler's data structures blow up. You're essentially measuring near-worst-case cache
behaviour for memory/cpu throughput or even I/O throughput (swapping) in
cases like this.

Another way of producing high compile times is of course to generate
source code. Anyway, this would usually yield non-typical code compared
to real application code, so that is perhaps not suited for you.
Like I said, we are only interested in speedup. How do you generate source
code? A flag?
And to
get in the 10--20 minutes range, it may be better to write a program
generates a program that generates a program, or something like that :-)


How would you do that?

Thanks a lot for your help.
Mike
Nov 13 '05 #5

"Michael B Allen" <mb*****@ioplex.com> wrote in message
news:pa**********************************@ioplex.c om...
On Sat, 15 Nov 2003 18:11:51 -0500, Michael Gaab wrote:
I need some help in finding a program written in c that is large enough
that it takes
10-20 minutes to compile/link on an *average* machine. Why? I am in a
HPC class and I am thinking about trying to speed up compiling of larger
programs. At least not so large at first.

I have done a bunch of searching. I have found some apps but most of
them are too big and too complicated.
Look at big open source projects like X, the Linux kernel, or Samba.


I have and they are way too complicated for what I need. I am sure I could
do it, just don't have the time.
All
of these are several hundred thousand lines of code. Of course anything
this size is going to be complicated.
Yes, very complicated. More complicated than I had imagined.
If you need something predictable
and ansi that you can try with different compilers and platforms then I
think your best bet will be to dynamically generate it. Think of 10
C constructs that can be "exploded" to be huge like huge static structure
initializers or large combinations of conditional statements that call
random functions and weave the whole thing together into one big 1MB set
of 10 source files.
This is what I need. We have limited disc space on the cluster.
How elaborate would the functions need to be?
You probably only need it to compile for 2 minutes
and loop 10 times for startup costs to be statistically insignificant as
opposed to compiling something for 20 minutes.


That sounds reasonable.

Thanks for your help.
Mike
Nov 13 '05 #6

Hi Michael,
Should it be a single program file, or are multiple files ok? Multiple files.

Are you
planning on timing the compilation-step proper, or also the
preprocessing, assembly, and linking steps? We were planning on parallelizing the compilation, e.g., gcc -c. I am not
a compiler expert, or for that matter an expert at anything. :-) Just keep
plugging along with what God gave me. Doing the best I can.
If you're not both a compiler expert and a parallelism expert, I'd say
this is aiming way, way too high. I personally know a guy who did a PhD
in parallelization of parsers (which is just the first step of the
compiler) and this took him 5 years.

The compiler-proper has several stages: first parsing (building an
abstract syntax tree), and then a couple of stages that perform
transformations on the AST, then emitting code. Even the parallelization
of one stage would be a tremendous task.

Unless, of course, you consider flocking out the multiple C files over a
farm of compiling machines, then collecting the results for linking.
However, this would probably be too trivial.
Will this be just an exercise, or are you aiming for something that is
practically useful? In the latter case there should be examples already
out there :-) It is for a project in a parallel program class.
I would seriously urge you to reconsider doing this. You'll get nowhere
if you're not willing to spend a couple of years on this.
Now that I have a better
understanding of HPC, I think I would have chosen a different thing to
parallel. But then on the other hand, it seems very practical if
complication/linking could be sped up on large projects. Not sure about
that assumption because I have never worked on any.
Farming out compilations over many machines is practical, and has been
done, at the file level. Anything else, I'd say, is 'only' of
theoretical interest for the time being.
Something that compiles for 10--20 minutes is almost bound to be either
big or complicated. One way of getting e.g. gcc to slow down (even with
not-so-big-or-not-so-complicated programs) is to enable function
inlining and set the inlining threshold to a very high value. This
worked wonders for the simple chess program I once wrote, it took ages
to compile.


Would you mind sharing the chess program? The issue in our class is whether
or not we can realize any speedup, not whether or not we have developed the
code ourselves or not.


Sure, look at http://libchess.sourceforge.net. However it is not nearly
finished and not in a usable state at the moment. If you want I can send
you a small tarball with a version that is complete enough to solve
chess problems. A suggestion: this may be a better thing to parallelize
than a full compiler, at least it's a well-understood and limited problem.
I am not sure about inlining, I'll have to play around with that a bit.
in gcc, you'll have to use -finline-limit=<some number>. This worked
nicely on the chess example since it will also inline recursive function
invocations :-)
Like I said, we are only interested in speedup. How do you generate source
code? A flag?


With a program, which may be programmed in C.

e.g.

#include <stdio.h>
int main(int argc, char *argv[])
{
int howmany=0;
if (argc>1) sscanf(argv[1],"%d", &howmany);
puts("#include <stdio.h>");
puts("int main(void)\n{");
for(;howmany>=0;howmany--)
printf(" puts(\"%d bottles of beer on the wall.\");\n", howmany);
puts(" return 0;\n}");
return 0;
}
And to
get in the 10--20 minutes range, it may be better to write a program
generates a program that generates a program, or something like that :-)


How would you do that?


With difficulty. It would be rather perverse.

Best regards, Sidney

Nov 13 '05 #7

"Sidney Cadot" <si****@jigsaw.nl> wrote in message
news:bp**********@news.tudelft.nl...
Unless, of course, you consider flocking out the multiple C files over a
farm of compiling machines, then collecting the results for linking.
However, this would probably be too trivial.
Yes trivial with a small project but not so with a program like Linux.
I think we are going to break our project into two parts. Part one will be
to prove the concept. Part two will be to reverse engineer Linux using
Rational and start to develop a scheme to parallelize Linux. I said start.

Would you mind sharing the chess program? The issue in our class is whether or not we can realize any speedup, not whether or not we have developed the code ourselves or not.


Sure, look at http://libchess.sourceforge.net. However it is not nearly
finished and not in a usable state at the moment. If you want I can send
you a small tarball with a version that is complete enough to solve
chess problems. A suggestion: this may be a better thing to parallelize
than a full compiler, at least it's a well-understood and limited problem.


Too late in the semester to change now. With suggestions from you and one
other
fellow I was able to create a single .5 MB file that took over 3 minutes to
compile.
I'll take this file and duplicate it a number of times but also I will break
it up in such a
way as to simulate a real program with dependencies, etc.
I am not sure about inlining, I'll have to play around with that a bit.


in gcc, you'll have to use -finline-limit=<some number>. This worked
nicely on the chess example since it will also inline recursive function
invocations :-)


Thanks.
Like I said, we are only interested in speedup. How do you generate source code? A flag?


With a program, which may be programmed in C.


Oh, I miss understood.

With difficulty. It would be rather perverse.


I am sure.

Thanks again.
Mike
Nov 13 '05 #8
Michael Gaab wrote:
I need some help in finding a program written in c that is large enough that
it takes
10-20 minutes to compile/link on an *average* machine.
Why? I am in a HPC class and I am thinking about trying to speed up
compiling of larger programs. At least not so large at first.

I have done a bunch of searching. I have found some apps but most of them
are too big and too complicated.
I know this is a bit off topic, apologies to the purists..
Thanks for any help.

Mike

I can tell you right now, something that takes a long time to compile is
OpenSSL. That may be more complicated than you want though :)

Jeff

Nov 13 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
by: FISH | last post by:
Ever have one of those days when you're not sure if it's you who's gone mad, or the rest of the world? I have an Open Source project on SourceForge for communication with YSMG - Yahoo's IM...
30
by: Hallvard B Furuseth | last post by:
Now that the '-*- coding: <charset> -*-' feature has arrived, I'd like to see an addition: # -*- str7bit:True -*- After the source file has been converted to Unicode, cause a parse error if a...
1
by: M.E.Farmer | last post by:
Hello c.l.py!, I have just finished this and decided to share. PySourceColor is a module to convert Python source into colored html. Yes it has been done before, but I like this better:) You can...
0
by: Monica Ferrero | last post by:
Hi! I'm not sure if this is the most adequate mySQL list for this post. If not, please indicat me which one I should use... I'm using Tomcat 4.1.24 with Apache 2 and MySQL 4.0.13. I have the...
5
by: Pete Wason | last post by:
Hiall! I have a demo viewer page for javascript stuff that has three buttons "DEMO" "HTML" and "JSCR", and an IFRAME called 'viewer'. Initially, the IFRAME gets loaded with the actual demo...
2
by: Next | last post by:
Hello all, I have a windows service that was suppose to write some events into its own EventLog. I created the EventLog using the component on VS 2003 toolbar Added an installer for it. Set...
8
by: Alvo von Cossel I | last post by:
hey everybody, I have written a great browser but it is missing a feature (quite a lot actually, but forget about them for now). that feature just so happens to be the View > Source function....
1
by: johnlim20088 | last post by:
Hi, Currently I have 6 web projects located in Visual Source Safe 6.0, as usual, everytime I will open solution file located in my local computer, connected to source safe, then check out/check in...
66
by: Jon Skeet [C# MVP] | last post by:
I'm sure the net will be buzzing with this news fairly soon, but just in case anyone hasn't seen it yet: Microsoft are going to make the source code for the .NET framework (parts of it,...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.