Tools and Techniques for Managing Large Websites

[dub]

Hello web folks... I've been desigining web pages for 13 years using my
trusty text editor (UltraEdit) and in depth knowledge of HTML. I'm
truly a text editor ninja at this point. I am frequently using regular
expressions to search and replace in multiple files. I also use many
different copy buffers and know all the keyboard shortcuts like the
back of my hand. The point is... it's comfortable.

Unfortunately, it's become just too time consuming to manage many of
the larger sites I'm making. I'm aware that programs like Dreamweaver
offer template features and site management goodies. But I'm hesitant
to learn a whole new system until I'm sure that it's the best way to do
it.

Does anybody with experience managing large websites have advice on how
to make things a little less tedious?

Thanks,
Dub

[Adrienne]

Gazing into my crystal ball I observed "dub" <du*****@yahoo.com> writing in
news:11*********************@l41g2000cwc.googlegro ups.com:

Hello web folks... I've been desigining web pages for 13 years using my
trusty text editor (UltraEdit) and in depth knowledge of HTML. I'm
truly a text editor ninja at this point. I am frequently using regular
expressions to search and replace in multiple files. I also use many
different copy buffers and know all the keyboard shortcuts like the
back of my hand. The point is... it's comfortable.

Unfortunately, it's become just too time consuming to manage many of
the larger sites I'm making. I'm aware that programs like Dreamweaver
offer template features and site management goodies. But I'm hesitant
to learn a whole new system until I'm sure that it's the best way to do
it.

Does anybody with experience managing large websites have advice on how
to make things a little less tedious?

Thanks,
Dub

I use HTML-Kit <http://www.chami.com/html-kit/>. While it does not have
"site management", it does have the ability to store templates and
frequently used snippets. It also has a lot of plugins that you can pick
and choose depending on what language you are working with. It's a plain
text editor with syntax highlighting and auto complete for several
frequently used elements.

I also use TopStyle <http://www.bradsoft.com/topstyle/>, and it has a site
manangement feature. I don't use its HTML feature, but it's great for CSS.

--
Adrienne Boswell
http://www.cavalcade-of-coding.info
Please respond to the group so others can share

[Martin!]

dub wrote:

Hello web folks... I've been desigining web pages for 13 years using my
trusty text editor (UltraEdit) and in depth knowledge of HTML. I'm
truly a text editor ninja at this point. I am frequently using regular
expressions to search and replace in multiple files. I also use many
different copy buffers and know all the keyboard shortcuts like the
back of my hand. The point is... it's comfortable.

Unfortunately, it's become just too time consuming to manage many of
the larger sites I'm making. I'm aware that programs like Dreamweaver
offer template features and site management goodies. But I'm hesitant
to learn a whole new system until I'm sure that it's the best way to do
it.

Does anybody with experience managing large websites have advice on how
to make things a little less tedious?

Thanks,
Dub

use SSI and/or Apache_Handlers to have the server construct manye pages
using standard modules.

[Geoff M]

In article <11*********************@l41g2000cwc.googlegroups. com>,
du*****@yahoo.com says...

Hello web folks... I've been desigining web pages for 13 years using my
trusty text editor (UltraEdit) and in depth knowledge of HTML. I'm
truly a text editor ninja at this point. I am frequently using regular
expressions to search and replace in multiple files. I also use many
different copy buffers and know all the keyboard shortcuts like the
back of my hand. The point is... it's comfortable.

Unfortunately, it's become just too time consuming to manage many of
the larger sites I'm making.

SSI (or the include function of any dynamic code generator like PHP) is
your friend.

If you have a "bit of code" that currently needs to be amended in many
places, why not replace the code in all those spots with an include to a
single file which will then require editing only once.

Geoff M

[Jim Moe]

dub wrote:

Does anybody with experience managing large websites have advice on how
to make things a little less tedious?

Or use a server-side program like PHP, perl, ASP, etc. Use them to
create the repetitive parts (menus, links, headers, footers) and you just
have to edit the page's unique content.

--
jmm dash list (at) sohnen-moe (dot) com
(Remove .AXSPAMGN for email)

[Alan J. Flavell]

On Tue, 22 Mar 2005, Geoff M wrote:

SSI (or the include function of any dynamic code generator like PHP)
is your friend.
But it's no friend of cacheability, which can significantly impact
server load and impair a site's response if it's used unnecessarily
and without taking care over such issues.
If you have a "bit of code" that currently needs to be amended in
many places, why not replace the code in all those spots with an
include to a single file which will then require editing only once.

Yes, and then why not build a static site once out of that source, at
the time that you publish it to the server, instead of asking the
server to rebuild the same stuff every time that the page is
revisited?

Both approaches have plus and minus points. I'm not saying one is
right and the other wrong, but to give due thought to the issues
before choosing.

"XBitHack full" is one of my friends ;-)

[Ståle Sæbøe]

dub wrote:

Hello web folks... I've been desigining web pages for 13 years using my
trusty text editor (UltraEdit) and in depth knowledge of HTML. I'm
truly a text editor ninja at this point. I am frequently using regular
expressions to search and replace in multiple files. I also use many
different copy buffers and know all the keyboard shortcuts like the
back of my hand. The point is... it's comfortable.

Unfortunately, it's become just too time consuming to manage many of
the larger sites I'm making. I'm aware that programs like Dreamweaver
offer template features and site management goodies. But I'm hesitant
to learn a whole new system until I'm sure that it's the best way to do
it.

Does anybody with experience managing large websites have advice on how
to make things a little less tedious?

Thanks,
Dub

I tried a lot of them. Used CSE Validator and Topstyle a while. In the
end I wound up with notepad, but I am testing NoteTab light now and I
like it.

Stick with your text editor. Explore database connectivity and server
side scripting. PHP is pretty straight forward and you will get to use
your regex skills there a lot.

An example of a project I made this weekend. A fully automated pic
gallery. The whole thing consists of 2 files totalling around 4k. If I
copy a folder with images into the root of the application, the rest is
automatic (it even makes thumbnails for the images the first time the
folder is browsed).

http://80.202.168.171/sandbox/galleries/

Note: The sandbox version is completely stripped of styles and you
cannot see the fullsized pics (I have restriced upload bandwith). Adding
a nice design wrap and a css file will add another 2k to the total code
length.

I started with PHP 2 weeks ago.

[David Ross]

dub wrote:

Hello web folks... I've been desigining web pages for 13 years using my
trusty text editor (UltraEdit) and in depth knowledge of HTML. I'm
truly a text editor ninja at this point. I am frequently using regular
expressions to search and replace in multiple files. I also use many
different copy buffers and know all the keyboard shortcuts like the
back of my hand. The point is... it's comfortable.

Unfortunately, it's become just too time consuming to manage many of
the larger sites I'm making. I'm aware that programs like Dreamweaver
offer template features and site management goodies. But I'm hesitant
to learn a whole new system until I'm sure that it's the best way to do
it.

Does anybody with experience managing large websites have advice on how
to make things a little less tedious?

My personal Web site is over 280 pages. With graphics (very few of
any great size), I have over 9.7 MB on my ISP's Web server. I also
maintain a smaller Web site for a charity -- 32 pages. I too
compose HTML and CSS manually; Web pages just seem to flow out of
my fingers and into my PC.

First of all, I mirror both sites on my hard drive, with the same
directory structure they have on the Web server. This makes
testing easier. I can verify hyperlinks without having to upload
affected pages.

I created a template for each subdirectory in my directory
structure plus a "general" template. If I want to create a new
page, I copy the template file (a TXT file) from my templates
directory into a Web directory and rename it (as an HTML file).
The use of a template does not prevent me from altering its preset
contents, but it does help me to standardize the appearance of my
pages.

I use an FTP client to upload my pages to my ISP's server. The
client creates or updates a log file in the directory from which it
uploaded. I keep that log file as a marker: files with more
recent dates have not yet been uploaded since they were last
modified. All I have to do is view the directory sorted by date.

I only use SSI for actual scripts and for displaying the revision
dates of frequently updated pages. Scripts include "hit" counters
(visible and invisible), computation of days since some prior
event, and weeks to some future event.

I recently found that a true site map is an excellent tool. Mine
has every page listed according to a logical directory structure
(with some pages listed for a directory where they do not yet
reside but should). Each entry in the list shows the page's title
with a link to the page. This was created manually and is
maintained manually. The site map helps me locate a page directly
without having to navigate through several other pages to reach
it.

I also have a site index created automatically by a script that
scans my space on my ISP's server. This has my pages sorted first
by actual directory and then by file name. Again each entry is a
link. Important features are the file sizes and dates. (The
listing is constructed from the UNIX "ls -l" command.) The listing
is regenerated every time it's requested.

--

David E. Ross
<URL:http://www.rossde.com/>

I use Mozilla as my Web browser because I want a browser that
complies with Web standards. See <URL:http://www.mozilla.org/>.

[Lars Eighner]

In our last episode,
<11*********************@l41g2000cwc.googlegroups. com>, the
lovely and talented dub broadcast on
comp.infosystems.www.authoring.html:

Does anybody with experience managing large websites have
advice on how to make things a little less tedious?

Pick a preprocessor and learn to use it. Mine is horrible
(chpp), but I have learned how to use it - and the best software
is the kind you know how to use. Then get to know your local
make utility, whatever it is.

--
Lars Eighner ei*****@io.com http://www.larseighner.com/
Close your eyes and press escape three times.

[Pierre Goiffon]

Alan J. Flavell wrote:

SSI (or the include function of any dynamic code generator like PHP)
is your friend.
But it's no friend of cacheability

Why ?
It don't change a thing for client side caching, and server side a lots
of servers do implement a caching system in order to optimize SSI includes.
which can significantly impact
server load and impair a site's response if it's used unnecessarily
and without taking care over such issues.

I was very surprised to read this !

Well yes, of course a page with SSI is a little CPU consuming while a
totally static page isn't almost at all but... It's not like if you were
using great pieces of PHP in your pages, SSI was used for years and
servers do implement it very well nowdays. And anyway : CPU isn't the
biggest source of costs in a hosting service isn't it ?

[c.thornquist]

"Alan J. Flavell" <fl*****@ph.gla.ac.uk> wrote in message
news:Pi*******************************@ppepc56.ph. gla.ac.uk...

On Tue, 22 Mar 2005, Geoff M wrote:

SSI (or the include function of any dynamic code generator like PHP)
is your friend.

But it's no friend of cacheability, which can significantly impact
server load and impair a site's response if it's used unnecessarily
and without taking care over such issues.

If you have a "bit of code" that currently needs to be amended in
many places, why not replace the code in all those spots with an
include to a single file which will then require editing only once.

Yes, and then why not build a static site once out of that source, at
the time that you publish it to the server, instead of asking the
server to rebuild the same stuff every time that the page is
revisited?

Both approaches have plus and minus points. I'm not saying one is
right and the other wrong, but to give due thought to the issues
before choosing.

"XBitHack full" is one of my friends ;-)

Is that why so many sites using PHP are so slow?

I maintain a 1,500 page site using plain HTML (no separate stylesheets
even). It loads fast, but if I ever had to make changes throughout, I'd be
in trouble.

I'd like to learn more about "search & replace". I used it on a 250 page
site that I recently built, but only on my hard drive. Had to upload all the
pages after doing so. I downloaded applications that perform search &
replace on the server, but they were too complex for me to understand in a
pinch. Anyway, search & replace makes me nervous. Is there a simple,
reliable way to do it?

Thanks,

Carla

[Alan J. Flavell]

On Tue, 22 Mar 2005, c.thornquist wrote:

"Alan J. Flavell" <fl*****@ph.gla.ac.uk> wrote in message
news:Pi*******************************@ppepc56.ph. gla.ac.uk...

On Tue, 22 Mar 2005, Geoff M wrote:

SSI (or the include function of any dynamic code generator like PHP)
is your friend.

But it's no friend of cacheability, which can significantly impact
server load and impair a site's response if it's used unnecessarily
and without taking care over such issues.

Is that why so many sites using PHP are so slow?

Possibly. It's hard to say without specific examples. There are
different ways of configuring and using PHP. But modern CPUs are
usually well up to the amount of processing involved, so, if they also
have enough memory available for the task, then the server itself
isn't the limiting factor.

But the non-cacheability can make quite a difference, if the site has
not been programmed with care. Not only because of the network
latency in fetching fresh copies of the same thing over and over
again, but the very fact that they're being fetched more often than
necessary is also putting extra load on the origin server.

Mark Nottingham has a good tutorial on this general topic -
http://www.mnot.net/cache_docs/
and an online tool to help to assess sites for cacheability.

[David Ross]

Pierre Goiffon wrote:

Alan J. Flavell wrote:

SSI (or the include function of any dynamic code generator like PHP)
is your friend.

But it's no friend of cacheability

Why ?
It don't change a thing for client side caching, and server side a lots
of servers do implement a caching system in order to optimize SSI includes.

The problem is that an SSI "include" causes the server to alter the
HTML file just as it is being sent. This changes the time-stamp on
the file. Thus, a browser that causes a cached file to be compared
with the server file will always find the server file to be newer.
This results in the cached file being replaced. Therefore, caching
becomes useless; and the server and bandwidth are impacted by
always having to resend the file.

--

David E. Ross
<URL:http://www.rossde.com/>

I use Mozilla as my Web browser because I want a browser that
complies with Web standards. See <URL:http://www.mozilla.org/>.

[Jan Roland Eriksson]

On Tue, 22 Mar 2005 14:44:47 +0000, "Alan J. Flavell"
<fl*****@ph.gla.ac.uk> wrote:

On Tue, 22 Mar 2005, c.thornquist wrote:

"Alan J. Flavell" <fl*****@ph.gla.ac.uk> wrote in message
news:Pi*******************************@ppepc56.ph. gla.ac.uk...

> On Tue, 22 Mar 2005, Geoff M wrote:

[...] Is that why so many sites using PHP are so slow?

Possibly. It's hard to say without specific examples.

PHP used on an Apache based server is all Ok, I have never had a speed
problem in that environment.

Me thinks that "so many sites using PHP" are sourced out by some
misconfigured MS misnomer and that would count for an addition to the
already built in "feature" of slowness in MS server technology.

--
Rex

[Alan J. Flavell]

On Tue, 22 Mar 2005, Pierre Goiffon wrote:

Alan J. Flavell wrote:

SSI (or the include function of any dynamic code generator like
PHP) is your friend.
But it's no friend of cacheability

Why ?

I recommend Mark Nottingham's tutorial to get up to speed on the
main features, from which a detailed further discussion might develop.
It don't change a thing for client side caching,
That depends a great deal on how it's used. But those who have just
discovered it, tend to apply it with happy abandon, without a care for
the cacheability consequences.
and server side a lots of servers do implement a caching system in
order to optimize SSI includes.
Perhaps you'd point to some specifics?

which can significantly impact server load and impair a site's
response if it's used unnecessarily and without taking care over
such issues.

I was very surprised to read this !

Oh well...
Well yes, of course a page with SSI is a little CPU consuming
The CPU consumption is rarely an issue these days. To focus on that
one issue is to miss the point, IMHO.
It's not like if you were using great pieces of PHP in your pages,
As far as cacheability is concerned, it doesn't matter how much or how
little you use, if you use it at all; but /how/ you deal with the
cacheability implications of what you're doing.
SSI was used for years and servers do implement it very well
nowdays.
It still comes, by default, with no cache validators. Even if we use
XBitHack full, we'll only get a last-modified date - no ETag.
And anyway : CPU isn't the biggest source of costs in a
hosting service isn't it ?

CPU is not the issue. But if your server gets unnecessarily hammered
by repeat requests for documents that haven't really changed, then it
*does* have implications for server load (think: extra memory usage,
I/O bandwidth, network bandwidth, request latency, and even, to a
small extent, a modicum of CPU).

All of this is also affected in detail by how you configure the
facility, of course. But the broad-brush effects are the same in
principle *whenever* you emit documents which have no cacheability
validator(s) and thus appear to be fresh whenever they're queried.

Then think of it from the user's point of view. Especially if they
have a slow network link, or a ceiling on monthly transfer volume.

So I recommend http://www.mnot.net/cache_docs/

[Pierre Goiffon]

Alan J. Flavell wrote:

SSI (or the include function of any dynamic code generator like
PHP) is your friend.

But it's no friend of cacheability

Why ?

I recommend Mark Nottingham's tutorial

Very good document, thanks very mutch to point it out to me ! I read the
chapter 13 of RFC 2616 when I had to, a few years ago : I'm a developper
and most of the code I write and wrote generates dynamic web pages - so
it was one of the main point I had to learn with Web technologies. And
with a lack of practice, of course, I forgot some of what I learnt
reading the RFC. This document is easy to read, very clear, very
comprehensive !

It don't change a thing for client side caching,

That depends a great deal on how it's used.

All right, I see now I didn't totaly understand your first post. I was
assuming you said : with SSI there is no possible client side caching.
But now I understand you just warned on the caching issues you must care
when using SSI - and that it could be used unnecessarily. Damned, my
english is still "far from perfect" :)

You're right, and it's right for almost all the server side technologies
: almost all of them, with their default configuration, makes the server
send caching headers that are not suitable.

Sorry everyone for these useless posts :D Maybe it wouldn't be so
useless for some non native english speakers readers who have an as bad
english level as I have ;o)

[Stan Brown]

"Pierre Goiffon" wrote in comp.infosystems.www.authoring.html:

Sorry everyone for these useless posts :D Maybe it wouldn't be so
useless for some non native english speakers readers who have an as bad
english level as I have ;o)

Pas du tout inutiles! J'ai appris quelque choses de nouvelles en
lisant ce fil.(*)

En plus, il n'est pas nécessaire de vous excuser de votre anglais.
Nous tous, nous vous avons compris parfaitement. Et vous voyez
quelle salade on fait avec des tentatives en francais. :-)

(*) Is "fil" le mot juste for a thread of articles in a newsgroup?

--

Stan Brown, Oak Road Systems, Tompkins County, New York, USA
http://OakRoadSystems.com/

[Pierre Goiffon]

Stan Brown wrote:

En plus, il n'est pas nécessaire de vous excuser de votre anglais.
Nous tous, nous vous avons compris parfaitement. Et vous voyez
quelle salade on fait avec des tentatives en francais. :-)
:)
Thank you very, very mutch :))
(*) Is "fil" le mot juste for a thread of articles in a newsgroup?

Yes exactly ! We use to call it a "fil de discussion"

[Fuzzyman]

David Ross wrote:

Pierre Goiffon wrote:

Alan J. Flavell wrote:

>SSI (or the include function of any dynamic code generator like PHP)>is your friend.

But it's no friend of cacheability
Why ?
It don't change a thing for client side caching, and server side a lots of servers do implement a caching system in order to optimize SSI

includes.
The problem is that an SSI "include" causes the server to alter the
HTML file just as it is being sent. This changes the time-stamp on
the file. Thus, a browser that causes a cached file to be compared
with the server file will always find the server file to be newer.
This results in the cached file being replaced. Therefore, caching
becomes useless; and the server and bandwidth are impacted by
always having to resend the file.

That's not good news. Is it possible to configure the server not to do
this ?

I guess the answer will be server specific... but it's still not good
news :-(

I use SSI a lot.

Regards,

Fuzzy
http://www.voidspace.org.uk/python

--

David E. Ross
<URL:http://www.rossde.com/>

I use Mozilla as my Web browser because I want a browser that
complies with Web standards. See <URL:http://www.mozilla.org/>.

[Pierre Goiffon]

Fuzzyman wrote:

The problem is that an SSI "include" causes the server to alter the
HTML file just as it is being sent. This changes the time-stamp on
the file. Thus, a browser that causes a cached file to be compared
with the server file will always find the server file to be newer.
This results in the cached file being replaced. Therefore, caching
becomes useless; and the server and bandwidth are impacted by
always having to resend the file.

That's not good news. Is it possible to configure the server not to do
this ?

Almost everytime yes, of course ! As you said, it depends on what
implementation you're using.

But first, you should read the article whose url was given by Alan J
Flavell just a few messages above. Then read your server's documentation
in order to send the appropriates headers. This could be said for all
server side technologies.

[dub]

Hi people, thanks for taking interest in this thread. I absolutely
appreciate all the great advice you have to offer.

As I said, I'm trying to figure out what direction to head in while
learning new coding techniques. Here's a few new questions to narrow
things down:

My number 1 question is: are you all advising me to continue with my
text editor (ultraedit) because you are biased text editor nerds (jus
like me) who are diehard and still use usenet?

Assuming I do continue down the path of fancy text editing:

- Why Should I use PHP vs PERL or ASP?
- Are there any coding environments (programs) that you recommend?
- Could you please provide more info on how to create a streamlined
wonderful page serving environment (links too please)?

Assuming I decide to throw my text editor out the window:

- Should I use Dreamweaver / Homesite / Coldfusion?
- What about Flash?
- What about the Dreamweaver Template and Server Management Features?
- What about other WYSWYG software?

Thanks everyone
Dub

[Ståle Sæbøe]

dub wrote:

My number 1 question is: are you all advising me to continue with my
text editor (ultraedit) because you are biased text editor nerds (jus
like me) who are diehard and still use usenet? After a few years of trying different editors I find that it is more
efficient to just stay with a simple text editor. Maybe it is because I
no longer need templates to get started, and because it is more
practical to do everything in one application.
Assuming I do continue down the path of fancy text editing:

- Why Should I use PHP vs PERL or ASP? PHP is available for all platforms and it is pretty easy to use.
ASP(now ASPX)/.NET offer a lot more functions built-in, and has certain
nice features when it comes to the windows platforms. - Could you please provide more info on how to create a streamlined
wonderful page serving environment (links too please)? Personally I like the database driven model. You store all metadata in a
database and make lookups for every page. The disadvantage as mentioned
is that it is near uncacheable. However, a combination where you
preprocess the most frequently visited locations could help.
Assuming I decide to throw my text editor out the window:

- Should I use Dreamweaver / Homesite / Coldfusion? In my experience these platform make use of a lot of proprietary
technologies making it difficult to change platform later. - What about Flash? Flash is all about presentation and can be part of any web strategy - What about the Dreamweaver Template and Server Management Features? Cannot say I used them much. - What about other WYSWYG software?

Never met one I liked ;)