473,713 Members | 2,781 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Please Stop It Rebuilding Everything!

Hi There,

We have a medium size project about 2000 source files and 700,000
lines of code, as you can imagine this takes some time to rebuild all.

Now our problem is that we have a code generator we have developed
that updates some of our classes, as an example lets say I update 50
files. When we do a build the ide builds nearly all of the source
files while we swear and then twiddle our thumbs for a couple of
hours.

Its almost as if it says "well alot of files have changed I will just
rebuild everything".

Does anyone know why this happens?

We are useing 2003 but 2002 did it as well.

As a side point I have noticed that the upcomeing version can use
multiple processors to build, will this work within one project as
then we could at least halve the time. We can build it on uour 24
processor irix box in about 10 minutes!

Any help would be much apreciated.

Well back to twidling my thumbs.

Cheers

Andy
Nov 17 '05 #1
8 1118
Andy Capon wrote:
Hi There,

We have a medium size project about 2000 source files and 700,000
lines of code, as you can imagine this takes some time to rebuild all.

Now our problem is that we have a code generator we have developed
that updates some of our classes, as an example lets say I update 50
files. When we do a build the ide builds nearly all of the source
files while we swear and then twiddle our thumbs for a couple of
hours.

Its almost as if it says "well alot of files have changed I will just
rebuild everything".

Does anyone know why this happens?
Have you analyzed the dependencies in your code to demonstrate that
significantly fewer files should be compiled? When you update classes with
the code generator, are unchanged files given a new timestamp? If so, that
could be a huge source of unnecessary re-work by the compiler.

Are you making proper use of precompiled headers? I'd expect a 700,000 line
program to build in 10 minutes tops, even on a single processor machine if
precompiled headers are used properly and consistently.
We are useing 2003 but 2002 did it as well.
I know there have been reports of VS always rebuilding everything, but I
don't know any of the details in those cases - hopefully someone else that
does know will reply as well.
As a side point I have noticed that the upcomeing version can use
multiple processors to build, will this work within one project as
then we could at least halve the time.


No, it won't help you. The MP support in Whidbey works at the project
level, so if you have 2000 source files in one project, they'll all be built
sequentially. If you can restructure your code into several projects
(libraries, I'd assume), then you could get some parallel compilation
benefit.

-cd
Nov 17 '05 #2
Hi Carl,

Thanks for your reply see remarks below:

"Carl Daniel [VC++ MVP]" wrote:
Andy Capon wrote:
Hi There,

We have a medium size project about 2000 source files and 700,000
lines of code, as you can imagine this takes some time to rebuild all.

Now our problem is that we have a code generator we have developed
that updates some of our classes, as an example lets say I update 50
files. When we do a build the ide builds nearly all of the source
files while we swear and then twiddle our thumbs for a couple of
hours.

Its almost as if it says "well alot of files have changed I will just
rebuild everything".

Does anyone know why this happens?
Have you analyzed the dependencies in your code to demonstrate that
significantly fewer files should be compiled? When you update classes with
the code generator, are unchanged files given a new timestamp? If so, that
could be a huge source of unnecessary re-work by the compiler.


I am definate of the dependencies affected a smaller amount of files,
unchanged files remain totally unchanged by the code-gernerator and are
read-only as they are in sourcesafe!

Are you making proper use of precompiled headers? I'd expect a 700,000 line
program to build in 10 minutes tops, even on a single processor machine if
precompiled headers are used properly and consistently.

We are useing precompiled headers but maybe not properly, could you direct
me to some info on how to use them properly! If we could get this to build in
10 minutes you would be a hero!

We are useing 2003 but 2002 did it as well.


I know there have been reports of VS always rebuilding everything, but I
don't know any of the details in those cases - hopefully someone else that
does know will reply as well.
As a side point I have noticed that the upcomeing version can use
multiple processors to build, will this work within one project as
then we could at least halve the time.


No, it won't help you. The MP support in Whidbey works at the project
level, so if you have 2000 source files in one project, they'll all be built
sequentially. If you can restructure your code into several projects
(libraries, I'd assume), then you could get some parallel compilation
benefit.


Well thats a shame, so not really any different to loading two versions of
the ide with different projects and building in both then!


-cd

Nov 17 '05 #3
Hi Andy,
Well thats a shame, so not really any different to loading two versions of
the ide with different projects and building in both then! No, it isn't since we do track dependencies between projects. For future
versions we will enable intra-process parallellism. We did work on
concurrent PDB access for this release and that was the biggest blocker.

However, the fastest way is going to stay using multiple proc support for
building individual projects. The concurrent access to teh PDB does incur
serialisation overhead that you otherwise avoid. This is the way we
structure all our builds at Microsoft.

For precompiled headers the few top level tips are. Do NOT use /YX, do use
/Yc/Yu. Do make sure you have as much as possible of shared dependencies in
a set of .pch files, do make sure you don't create one mongo .pch file idf
most of your .cpp files only depend on a small part of it.

Try compiling stable code into libraries and linking them in.

If you determine that there is an issue with dependency checking, we can
definitely track that down if you can provide a reasonable repro. If you
cannot, then you can call our product support services and work with them
(and get re-imbursed for the support incident if the cause is indeed a bug
in the product, if you don't already have free incidents from buying the
product or MSDN).

Ronald Laeremans
Visual C++ team

"Andy Capon" <Andy Ca***@discussio ns.microsoft.co m> wrote in message
news:0B******** *************** ***********@mic rosoft.com... Hi Carl,

Thanks for your reply see remarks below:

"Carl Daniel [VC++ MVP]" wrote:
Andy Capon wrote:
> Hi There,
>
> We have a medium size project about 2000 source files and 700,000
> lines of code, as you can imagine this takes some time to rebuild all.
>
> Now our problem is that we have a code generator we have developed
> that updates some of our classes, as an example lets say I update 50
> files. When we do a build the ide builds nearly all of the source
> files while we swear and then twiddle our thumbs for a couple of
> hours.
>
> Its almost as if it says "well alot of files have changed I will just
> rebuild everything".
>
> Does anyone know why this happens?


Have you analyzed the dependencies in your code to demonstrate that
significantly fewer files should be compiled? When you update classes
with
the code generator, are unchanged files given a new timestamp? If so,
that
could be a huge source of unnecessary re-work by the compiler.


I am definate of the dependencies affected a smaller amount of files,
unchanged files remain totally unchanged by the code-gernerator and are
read-only as they are in sourcesafe!

Are you making proper use of precompiled headers? I'd expect a 700,000
line
program to build in 10 minutes tops, even on a single processor machine
if
precompiled headers are used properly and consistently.


We are useing precompiled headers but maybe not properly, could you direct
me to some info on how to use them properly! If we could get this to build
in
10 minutes you would be a hero!

> We are useing 2003 but 2002 did it as well.


I know there have been reports of VS always rebuilding everything, but I
don't know any of the details in those cases - hopefully someone else
that
does know will reply as well.
> As a side point I have noticed that the upcomeing version can use
> multiple processors to build, will this work within one project as
> then we could at least halve the time.


No, it won't help you. The MP support in Whidbey works at the project
level, so if you have 2000 source files in one project, they'll all be
built
sequentially. If you can restructure your code into several projects
(libraries, I'd assume), then you could get some parallel compilation
benefit.


Well thats a shame, so not really any different to loading two versions of
the ide with different projects and building in both then!


-cd

Nov 17 '05 #4
"=?Utf-8?B?QW5keSBDYXB vbg==?=" <Andy Ca***@discussio ns.microsoft.co m>
wrote in news:0B******** *************** ***********@mic rosoft.com:
> Its almost as if it says "well alot of files have changed I will
> just rebuild everything".
> Does anyone know why this happens?


Have you analyzed the dependencies in your code to demonstrate that
significantly fewer files should be compiled? When you update
classes with the code generator, are unchanged files given a new
timestamp? If so, that could be a huge source of unnecessary re-work
by the compiler.


We have seen a lot of "always rebuild" problems throughout the years.
Two most common causes:
1. extra unneeded dependencies specified in source code
2. a bug in our dependency scanner for .rc files where we did not pre-
process the file and tried to get every single include (even those
#ifdef'ed out) and recompiling the .rc file if any one of them was not
found.

The first we can't really fix since it's the developer's source code
that needs to be fixed :) The second we fixed in VS2005 (and may even
have a hotfix for for earlier releases but I'm not certain).

Does your code belong to either category?

If not and you really suspect it is a bug in our dependency scanner,
then you should call product support to get a fix for it as Ronald
suggests in another response.

Thanks,
--
Tarek Madkour, Visual C++ Team
This posting is provided "AS IS" with no warranties, and confers
no rights.
Nov 17 '05 #5
Hello,

As the guys from MS have said, a 700,000 lines of code is actually not very
much at all. I can rebuild a much larger suite of projects from scratch in
about 40 minutes on a reasonable, but not super high-end, machine. Single
processor, hyper-threaded, RAID 1 drive (redundant). So you might simply
consider getting a machine that has a fast drive. I found that with this
RAID 1 setup, things became much, much faster. With RAID 0 (striping) it
may be even faster still. (Perhaps have I have RAID 0 and 1 backwards here,
I can never remenber, I have the redundant drive version).

However, you will get a huge increase in speed from using precompiled
headers correctly, as one of the posters mentioned. I think we probably got
a 50-1 improvement in time building some projects from using precompiled
headers.

We don't have any projects with 2000 source files (50-100 is largest for
us), so your use of precompiled headers may need to be slightly different
than ours, but I can give you a snapshot of how we use it.

In each project that warrents it, we create a file "precomp.h" into which we
place all of the windows and system headers (crt / stdc++ etc.) that are
reasonable for the project. We also place headers in there from our own
suite of libraries that are being used in the project being built. If
appropriate, some project local headers may also be placed there, but only
if they are never (or very very rarely) changing.

Create a file "precomp.cp p" that has one line of text - #include
"precomp.h" . Add this file to your project.

In every single one of your files being compiled (i.e. all of your ".cpp"
files), insert #include "precomp.h" as the very first non-comment text line.

Change your project settings so that "Use precompiled headers (/Yu)" is
selected as the default. Indicate that "precomp.h" is the header to use.

Change then the file specific setting for "precomp.cp p" to be "Create
precompiled header (/Yc)". Again specify "precomp.h" as the appropriate
header.

That should be it. It might be painful to inser that #include "precomp.h"
in every one of your source files, but it is a one time process. For a
project with 2000 files it may be appropriate to have a number of different
precompiled headers, but these .pch files are quite large and if the
compiler has to switch from using one to using another between different
translation units, that may slow it down because of disk access. Once the
..pch file is loaded and in the disk cache, it is very fast retrieving it,
which is a big win.

I would also though seriously consider breaking up your project into pieces,
as 2000 source files in one project seems excessive. If you look at large
library suites that people have out there (GDAL comes to mind) you will see
I think that mostly things are broken up into more managable sized
collections of files per project (.dll or .lib).

Anyway bottom line is I would look first to these other solutions than
assuming that the dependency walker of DevStudio is malfunctioning.
Whenever I have thought there was something going wrong with it in the past,
I have come to realize that there was some dependency I wasn't considering.

Good luck.

-Eric Twietmeyer

PS However, I do have a gripe about CSharp projects. They do always rebuild
as far as I can tell, there doesn't seem to be a notion for these projects
that all targets are up to date and don't need to be touched. They simply
always rebuild completely. Don't know why....

"Andy Capon" <ac****@sgi.com > wrote in message
news:f2******** *************** **@posting.goog le.com...
Hi There,

We have a medium size project about 2000 source files and 700,000
lines of code, as you can imagine this takes some time to rebuild all.

Now our problem is that we have a code generator we have developed
that updates some of our classes, as an example lets say I update 50
files. When we do a build the ide builds nearly all of the source
files while we swear and then twiddle our thumbs for a couple of
hours.

Its almost as if it says "well alot of files have changed I will just
rebuild everything".

Does anyone know why this happens?

We are useing 2003 but 2002 did it as well.

As a side point I have noticed that the upcomeing version can use
multiple processors to build, will this work within one project as
then we could at least halve the time. We can build it on uour 24
processor irix box in about 10 minutes!

Any help would be much apreciated.

Well back to twidling my thumbs.

Cheers

Andy

Nov 17 '05 #6
Eric Twietmeyer wrote:
PS However, I do have a gripe about CSharp projects. They do always
rebuild as far as I can tell, there doesn't seem to be a notion for
these projects that all targets are up to date and don't need to be
touched. They simply always rebuild completely. Don't know why....


Think about the dependency analysis when there are no .OBJ files...

I recall hearing/reading that the C# compiler does do some kind of partial
rebuild - But it seems that's either not the case, or it's not very
effective.

-cd
Nov 17 '05 #7
Hi everyone,

Thanks for your replies, and from what I see we don't seem to be doing
anything wrong.

First of all about total build times :

We use precompiled headers for all headers that do not change often,
basically windows.h, all the stl headers we use and some includes from us.

We use /Yu, and if anything in the pre-compiled header changes we manually
rebuild it by setting to /Yc rebuilding one file to create the precompiled
header and then switching back to /Yu to build the rest.

We have fast machines with raid 0 across three ultra320 15K disks and dual
xeon processors.
Now concerning problems with dependencies:

I can reproduce this problem here easily and it is not the .rc problem that
Tarek mentioned as it is a console app, also it is not uneeded dependencies
in the source code. Totally unrelated files are built!

Also lets say I regenerate 50 of our classes, mostly it will rebuild these
50 and only the other dependent classes lets say 75% of the time, the other
25% of the time it will rebuild nearly everything.

If I only regenerate 10 classes then it will only rebuild everything very
rairly, say 5% of the time.

If I regenerate a large number of classes say 1000 it will nearly always
rebuild everything.

So to us it looks like the more files that are changed the higher % chance
of it rebuilding everything.

I think I will take the advice of creating library projects to see if that
will help us.

Thanks for all your replies

Andy



"Andy Capon" wrote:
Hi There,

We have a medium size project about 2000 source files and 700,000
lines of code, as you can imagine this takes some time to rebuild all.

Now our problem is that we have a code generator we have developed
that updates some of our classes, as an example lets say I update 50
files. When we do a build the ide builds nearly all of the source
files while we swear and then twiddle our thumbs for a couple of
hours.

Its almost as if it says "well alot of files have changed I will just
rebuild everything".

Does anyone know why this happens?

We are useing 2003 but 2002 did it as well.

As a side point I have noticed that the upcomeing version can use
multiple processors to build, will this work within one project as
then we could at least halve the time. We can build it on uour 24
processor irix box in about 10 minutes!

Any help would be much apreciated.

Well back to twidling my thumbs.

Cheers

Andy

Nov 17 '05 #8
I would really encourage you to call MS support to help them diagnose this
issue and get you a fix if it turns out to be a product bug.

Trying this with the Beta of Visual C++ 2005 would also be a great data
point.

Thanks.

Ronald

"Andy Capon" <An*******@disc ussions.microso ft.com> wrote in message
news:69******** *************** ***********@mic rosoft.com...
Hi everyone,

Thanks for your replies, and from what I see we don't seem to be doing
anything wrong.

First of all about total build times :

We use precompiled headers for all headers that do not change often,
basically windows.h, all the stl headers we use and some includes from us.

We use /Yu, and if anything in the pre-compiled header changes we manually
rebuild it by setting to /Yc rebuilding one file to create the precompiled
header and then switching back to /Yu to build the rest.

We have fast machines with raid 0 across three ultra320 15K disks and dual
xeon processors.
Now concerning problems with dependencies:

I can reproduce this problem here easily and it is not the .rc problem
that
Tarek mentioned as it is a console app, also it is not uneeded
dependencies
in the source code. Totally unrelated files are built!

Also lets say I regenerate 50 of our classes, mostly it will rebuild these
50 and only the other dependent classes lets say 75% of the time, the
other
25% of the time it will rebuild nearly everything.

If I only regenerate 10 classes then it will only rebuild everything very
rairly, say 5% of the time.

If I regenerate a large number of classes say 1000 it will nearly always
rebuild everything.

So to us it looks like the more files that are changed the higher % chance
of it rebuilding everything.

I think I will take the advice of creating library projects to see if that
will help us.

Thanks for all your replies

Andy



"Andy Capon" wrote:
Hi There,

We have a medium size project about 2000 source files and 700,000
lines of code, as you can imagine this takes some time to rebuild all.

Now our problem is that we have a code generator we have developed
that updates some of our classes, as an example lets say I update 50
files. When we do a build the ide builds nearly all of the source
files while we swear and then twiddle our thumbs for a couple of
hours.

Its almost as if it says "well alot of files have changed I will just
rebuild everything".

Does anyone know why this happens?

We are useing 2003 but 2002 did it as well.

As a side point I have noticed that the upcomeing version can use
multiple processors to build, will this work within one project as
then we could at least halve the time. We can build it on uour 24
processor irix box in about 10 minutes!

Any help would be much apreciated.

Well back to twidling my thumbs.

Cheers

Andy

Nov 17 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

35
4540
by: wired | last post by:
Hi, I've just taught myself C++, so I haven't learnt much about style or the like from any single source, and I'm quite styleless as a result. But at the same time, I really want nice code and I go to great lengths to restructure my code just to look concise and make it more manageable. When I say this, I'm also referring to the way I write my functions. It seems to me sometimes that I shouldn't have many void functions accepting...
14
4220
by: KnockKnock | last post by:
Hi all, I have run into a bit of a problem. I recently setup mysql to run on a Debian Linux box I setup. I am by no means an expert on either linux or mysql. I know enough to just be dangerous. =) I had it set up so I could read and write to the database through my perl code, connect with phpmyadmin, etc. Not one to be happy I attempted to set up a pop3 mailserver .. well .. not to successful there so I abandoned that idea. Then to my...
0
2329
by: Mark Adams | last post by:
I really need some help with this. MySQL will not start on boot despite everything I've done to make sure that it is set to do so. When I start it as root from a terminal with "/usr/bin/mysqld_safe --skip-grant-tables &" I show several mysqld procs. and one mysqld_safe, but I can't connect: # ps -aux| grep mysql Warning: bad syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html root 3650 0.0 0.1 2276 664 ? S ...
1
363
by: Peter Steele | last post by:
Okay, I assume I'm missing something obvious here. I have created a simple service in C# that on starting spawns a thread to do some processing. The service can be stopped with a "net stop" command of course, but under some circumstances the service will decide to terminate itself. My OnStart looks something like this: protected override void OnStart(string args) { serviceThread = new Thread(new ThreadStart(ServiceThreadStart));...
0
1005
by: andrei | last post by:
Hi Group, Environment: Windows 2000 Professional, VS.NET 2003 The solution has 30 projects (one main asp.net application and the rest are dlls) We have this annoying behaviour with our ASP.NET application From time to time, when trying to build or rebuild or run the application, we experience VS crashes. VS starts by building the dependent dlls (which is perfectly OK) but when it
3
1214
by: Simon Harvey | last post by:
Hi everyone, I keep getting a problem with dropdownlist controls. It sounds really stupid, but my app is screwed as long as this keeps happening. It seems to spontaneously happen and then I cant fix it without rebuilding the whole page from scratch! Not Fun. :-( So whats happening is I have my dropdowns on the page. The user comes along and selects an option, but when the user submits the page,. its as if nothing
1
1137
by: tshad | last post by:
I don't use VS much for my websites. But I do have one that does use VS. I am trying to figure out what I can change that would not require a rebuild. I assume that if I make any changes to my aspx.vb file - I would have to rebuild. Can I make change to my .aspx file without rebuilding, such as change some of the html?
1
2177
by: Ahmed Yasser | last post by:
Hi all, i have a problem with the datagridview sorting, the problem is a bit complicated so i hope i can describe in the following steps: 1. i have a datagridview with two columns (LoginName,UserName) 2. the datagridview sorting is set to automatic, so when i click on the column header is sorts well. 3. i put in an event handler for the CellEndEdit Event, so whenever the user of the program changes the content of a cell in the LoginName...
0
8701
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9306
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9168
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9068
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9009
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7942
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6621
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4715
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3155
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.