Detecting JS enabled/capable browser

[Ryan Stewart]

I'm working on a project to collect web application usage statistics. What
are the recommended ways of detecting whether a browser is JavaScript
enabled and/or capable? Obviously I can write a script to invoke something
on the server, and if it works, then it works. Is there a better way? I'm
looking for the least intrusive way of doing it, from a web application
point of view. i.e. I'd like to be able to drop this into an existing
application without altering anything already there. It would be nice if
there were just a "JavaScript-Enabled" HTTP header.

[Andrew Thompson]

On Tue, 28 Dec 2004 00:00:28 -0600, Ryan Stewart wrote:

I am also curious as to what about HTTP makes you
think that statistics gathering is pointless.

Speaking from my own experience, I might add the following points..

- Is a reported figure of '3/10,000 IE 4' users to be trusted?
Or is it that the site crashes IE 4 on the entry page, and
IE 4 users get no further (and spend no *money*) on the site.

- Browser UA strings are notoriously inaccurate.

- Caching of the page between server and potential clients
might result in the page not even being fetched from the server
for some, or some thousands, of potential visitors.

- Any statistic collected from XYZ.com tells you not
'what % of internet surfers..', but instead 'what % of
internet surfers who visited xyz.com..' a much less useful
statistic for other web hosts.

Those are some of the most major and obvious pitfalls of
collecting data re visitors. Experts on the topic would
probably be able to add a half dozen more technical reasons
and a baker's dozen of 'lies, damn lies and statistics' reasons.

OTOH, I myself am interested in setting up just such a 'testing
machine' as you have been discussing across various groups, even
with the potential for misinformation and misuse of the information.

I was thinking of..
- feature detection for JS, I also wanted to attempt to use JS
in efforts to determine what pixel size is '100%' in browsers,
more to prove that you cannot second guess, and should use '100%'.
- CSS import hacks (possible user input required)
- testing for browser recognition of common media types
- Java support (both version and browser functionality,
particularly showDocument and whether the browser/VM
combo. steals the focus)

Of course, with you with your teeth into it and me having
put my plans on the back burner, it sounds as though you will
beat me to the punch!

In answer to your specific question, you might experiment
with this..
- Test the browser for met-refresh redirects.
- If successful, go to a page that has a refresh for 10 seconds
- ..but also has a JS that redirects immediately to
'YesWeHaveJSTha tWillRedirectAP age.jsp'

You should have an answer (for visitors to your site who
had 'meta-refresh' enabled and the page did not crash their
browser or got bored and opted out..) to your specific question.

Good luck with it. I'm outta' here..
(Ducks and runs from this the possible rotten fruit
hurled in his general direction)

--
Andrew Thompson
http://www.PhySci.org/codes/ Web & IT Help
http://www.PhySci.org/ Open-source software suite
http://www.1point1C.org/ Science & Technology
http://www.LensEscapes.com/ Images that escape the mundane

[Ryan Stewart]

"Andrew Thompson" <Se********@www .invalid> wrote in message
news:1g******** *************** *****@40tude.ne t...

Speaking from my own experience, I might add the following points..

- Is a reported figure of '3/10,000 IE 4' users to be trusted?
Or is it that the site crashes IE 4 on the entry page, and
IE 4 users get no further (and spend no *money*) on the site.
I can guarantee 99.999999999% plus that nobody will hit this (where I'm
planning to use it) with IE 4 or any outdated browser. Even if someone does,
and the site crashes in it, we are likely to hear about it to fix it, so
long as the user has a reason for using such a browser.
- Browser UA strings are notoriously inaccurate.
I plan to match them nearly character for character and collect unrecognized
ones for research.
- Caching of the page between server and potential clients
might result in the page not even being fetched from the server
for some, or some thousands, of potential visitors.
Thanks for reminding me of that. I'll plan for it.
- Any statistic collected from XYZ.com tells you not
'what % of internet surfers..', but instead 'what % of
internet surfers who visited xyz.com..' a much less useful
statistic for other web hosts.
Except that most users that I will monitor are logging in to secure
applications. Therefore it can be done on a per user basis if I desire.
Aside from that, it could be done on a per session basis for less accurate
results and possibly with IP matching if one wanted to get ambitious.
OTOH, I myself am interested in setting up just such a 'testing
machine' as you have been discussing across various groups, even
with the potential for misinformation and misuse of the information.
OT: How are you planning on writing it? I'm thinking about a JAR that can be
dropped into a webapp along with maybe a redirecting index.jsp.
I was thinking of..
- feature detection for JS, I also wanted to attempt to use JS
in efforts to determine what pixel size is '100%' in browsers,
more to prove that you cannot second guess, and should use '100%'.
- CSS import hacks (possible user input required)
- testing for browser recognition of common media types
- Java support (both version and browser functionality,
particularly showDocument and whether the browser/VM
combo. steals the focus)
Hmm, interesting ideas.
Of course, with you with your teeth into it and me having
put my plans on the back burner, it sounds as though you will
beat me to the punch!
We'll see. It's just a hobby project atm :)
In answer to your specific question, you might experiment
with this..
- Test the browser for met-refresh redirects.
- If successful, go to a page that has a refresh for 10 seconds
- ..but also has a JS that redirects immediately to
'YesWeHaveJSTha tWillRedirectAP age.jsp'

I've considered something along these lines.

[...]
Thanks for the thoughtful reply.

[Dr John Stockton]

JRS: In article <D6************ ********@texas. net>, dated Tue, 28 Dec
2004 00:00:28, seen in news:comp.lang. javascript, Ryan Stewart
<zz********@gSP AMo.com> posted :

"Richard Cornford" <Ri*****@litote s.demon.co.uk> wrote in message
news:cq******* ************@ne ws.demon.co.uk. ..

Richard wrote:

Ryan Stewart wrote:
I'm working on a project to collect web application usage
statistics. What are the recommended ways of detecting
whether a browser is JavaScript enabled and/or capable? ...

The recommended approach is to arrange to not have to care. That avoids
the consequences of the aspects of HTTP that make any form of statistics
gathering flawed to the point of being useless, the impossibility of
implementing any mechanism for determining that client-side scripting is
available without an omniscient knowledge of all scriptable web
browsers, and the fact that such information is next to useless anyway.

I always arrange to not have to care. I'm simply curious and would like to
have some numbers to show. I would guess that about 95% of the users that I
code for are using IE 6. Of the rest, most probably use NN 7. That's
something I'm going to find out, and I'd also just like to know how many of
them have turned off JS. I'm guessing that percentage will be very low--in
the neighborhood of 2%. I am also curious as to what about HTTP makes you
think that statistics gathering is pointless.

If you really do not care whether your script is enabled or not, then
that can only mean that your scripting adds no value. In that case,
writing script is a waste of your time and a waste of your employer's
time, and delivering script is a waste of Internet and users' resources.
But it may serve to inflate someone's ego.

If it is possible to present a useful page, section, paragraph, ... on
the subject without using script, then one should arrange to present in
that manner if script is not available.

If it is not possible to do anything useful without script, then the
user should be informed that there is a prerequisite for a useful
experience, called "javascript enabled", and that he is advised to seek
a system with that property.

If the scriptless experience is not as useful as the scripted one, then
the user should also be told; he may prefer to seek the better
experience, and it may be easy enough to find it. He may just have
forgotten to re-enable scripts!
One can, of course, take a commercial decision that as 98% of users have
script enabled there's no need to bother about the others; why invest
resources on 2% of possible customers? That is the attitude that was,
in the past, taken towards the physically-disabled - enough customers
can climb the steps into out shop, no point in bothering about the rest.
One thing that one can, AIUI, learn by reading javascript books, web
pages, and newsgroups is that a recommended approach, even a commonly-
recommended approach, is not necessarily an optimum, or even a sensible,
one.

--
© John Stockton, Surrey, UK. ?@merlyn.demon. co.uk Turnpike v4.00 IE 4 ©
<URL:http://www.jibbering.c om/faq/> JL/RC: FAQ of news:comp.lang. javascript
<URL:http://www.merlyn.demo n.co.uk/js-index.htm> jscr maths, dates, sources.
<URL:http://www.merlyn.demo n.co.uk/> TP/BP/Delphi/jscr/&c, FAQ items, links.

[Ryan Stewart]

"Dr John Stockton" <sp**@merlyn.de mon.co.uk> wrote in message
news:G4******** ******@merlyn.d emon.co.uk...

If you really do not care whether your script is enabled or not, then
that can only mean that your scripting adds no value.

[...]

In general, what you propose is essentially what I do. That is what I meant
by saying I arrange not to care. However, that's not relevant to this
question. I'm not trying to find out whether "my script" will run or not. I
am not writing a script. I'm writing a code library to gather information
about my customer base. Part of my reason for doing so is so that when my
coworkers ask why we should write web applications that don't rely on
JavaScript for proper functionality, I can pull up the logs and say,
"Because x% of users/hits on our other applications were browsers with
scripting disabled." I'm guessing x will be a very low number, so I won't
really be able to use this argument, but that will be nice to know as well.

[Jim Ley]

On Tue, 28 Dec 2004 12:37:02 -0600, "Ryan Stewart"
<zz********@gSP AMo.com> wrote:

"Andrew Thompson" <Se********@www .invalid> wrote in message
news:1g******* *************** ******@40tude.n et...

Speaking from my own experience, I might add the following points..

- Is a reported figure of '3/10,000 IE 4' users to be trusted?
Or is it that the site crashes IE 4 on the entry page, and
IE 4 users get no further (and spend no *money*) on the site.
I can guarantee 99.999999999% plus that nobody will hit this (where I'm
planning to use it) with IE 4 or any outdated browser. Even if someone does,
and the site crashes in it, we are likely to hear about it to fix it, so
long as the user has a reason for using such a browser.

- Browser UA strings are notoriously inaccurate.

I plan to match them nearly character for character and collect unrecognized
ones for research.

Then well over 50% of all your visitors will be using unrecognised
ones, character for character is possible, but with IE, plugins and
UA's built in IE recording all sorts of things in the UA string then
these will be changing daily. Then there's mozilla of course which
has many families and people using daily builds, and more...

- Caching of the page between server and potential clients
might result in the page not even being fetched from the server
for some, or some thousands, of potential visitors.

Thanks for reminding me of that. I'll plan for it.

but how are you going to effect your collection of statistics bases on
this (it's not unreasonable to assume differently capable UA's have
different caching behaviour, we know for a fact IE in "check each
time" and mozilla in "check each time" cache settings exhibit
different behaviour with IE well over-represented in any requests.

- Test the browser for met-refresh redirects.
- If successful, go to a page that has a refresh for 10 seconds
- ..but also has a JS that redirects immediately to
'YesWeHaveJSTha tWillRedirectAP age.jsp'

I've considered something along these lines.

so what happens when script is enabled but meta is disabled, or meta
is on a prompt basis, or javascript is on a prompt basis - all of
which are possible in Internet explorer, let alone other user agents.

Jim.

[Jim Ley]

On Tue, 28 Dec 2004 13:02:09 +0000, Dr John Stockton
<sp**@merlyn.de mon.co.uk> wrote:

If you really do not care whether your script is enabled or not, then
that can only mean that your scripting adds no value. In that case,
writing script is a waste of your time and a waste of your employer's
time, and delivering script is a waste of Internet and users' resources.
If 90% get the enhanced behaviour from the script, then it's an
extremely effective use of resources, if 10% don't then so what, it's
not hurt them, but neither has the lack of site functionality the
alternative approach yields.
If it is not possible to do anything useful without script, then the
user should be informed that there is a prerequisite for a useful
experience, called "javascript enabled", and that he is advised to seek
a system with that property.
You've got to assume users are thick, 'cos generally they are (or
rather they're probably just lazy, there's other places to
buy/visit/etc.) and they won't understand what you're saying and will
just leave. Script can always be done just to add value, sometimes
incredible value, so it's well worth doing it. There's almost never a
sound reason to just use script other than the ego or laziness of the
author.
That is the attitude that was,
in the past, taken towards the physically-disabled - enough customers
can climb the steps into out shop, no point in bothering about the rest.

Except of course, that was a mistaken approach, which has led to
significant repair costs from all such businesses that took that
approach, when it would've been much cheaper to do it properly
correctly in the first place.

Jim.

[Gus Richter]

Ryan Stewart wrote:

I'm working on a project to collect web application usage statistics. What
are the recommended ways of detecting whether a browser is JavaScript
enabled and/or capable? Obviously I can write a script to invoke something
on the server, and if it works, then it works. Is there a better way? I'm
looking for the least intrusive way of doing it, from a web application
point of view. i.e. I'd like to be able to drop this into an existing
application without altering anything already there. It would be nice if
there were just a "JavaScript-Enabled" HTTP header.

You may find these to be useful:
http://www.faqts.com/knowledge_base/...id/3459/fid/47
http://www.faqts.com/knowledge_base/...aid/323/fid/47

--
Gus

[Andrew Thompson]

On Tue, 28 Dec 2004 20:00:13 GMT, Jim Ley wrote:

On Tue, 28 Dec 2004 12:37:02 -0600, "Ryan Stewart"
<zz********@gSP AMo.com> wrote:

"Andrew Thompson" <Se********@www .invalid> wrote in message
news:1g****** *************** *******@40tude. net... ...

- Browser UA strings are notoriously inaccurate.

I plan to match them nearly character for character and collect unrecognized
ones for research.

Then well over 50% of all your visitors will be using unrecognised
ones, character for character is possible, but with IE, plugins and
UA's built in IE recording all sorts of things in the UA string then
these will be changing daily. Then there's mozilla of course which
has many families and people using daily builds, and more...

I am not sure I understand the finer intricacies of what you wrote,
Jim, so I thought I'd leave it but add. Ryan.. what about the UA's
that claim (character for character) to be IE?

The original intent of browser string spoofing was to get around the
idiotic web-designers who attempt to make a 'Netscape only/IE only' site.

In my plan, the only reason to collect UA strings was to cross-match
them against the information discovered through the CSS import hacks
(which reveal the core rendering engine of an UA) and a slew of JS
feature detection (also to help narrow down the UA), and thereby get
some gauge of the level of browser string spoofing.

- Test the browser for met-refresh redirects.
- If successful, go to a page that has a refresh for 10 seconds
- ..but also has a JS that redirects immediately to
'YesWeHaveJSTha tWillRedirectAP age.jsp'

I've considered something along these lines.

so what happens when script is enabled but meta is disabled, or meta
is on a prompt basis, or javascript is on a prompt basis - all of
which are possible in Internet explorer, let alone other user agents.

Interesting points. Ryan wants to do this as a silent app.
(which will be much more difficult) but the testing facility
I planned would ask for active user involvement as a prerequisite,
so it might be handy to forewarn them of the prompts and ask
them if prompts were encountered. (Yes, yes, ..I know! 'The User',
the single least reliable source of information available)

--
Andrew Thompson
http://www.PhySci.org/codes/ Web & IT Help
http://www.PhySci.org/ Open-source software suite
http://www.1point1C.org/ Science & Technology
http://www.LensEscapes.com/ Images that escape the mundane

[Richard Cornford]

Ryan Stewart wrote:

Richard Cornford wrote:

Ryan Stewart wrote:
I'm working on a project to collect web
application usage statistics. ...

<snip> The recommended approach is to arrange to not have to
care. That avoids the consequences of the aspects of
HTTP that make any form of statistics gathering flawed
to the point of being useless, ...

<snip> I always arrange to not have to care. I'm simply curious
and would like to have some numbers to show.
The desire to have numbers to show is completely understandable, but
numbers are not particularly useful in themselves. You need to know what
those numbers really mean, and be in a position to explain their meaning
to others.
I would guess that about 95% of the users that I code
for are using IE 6. Of the rest, most probably use NN 7.
That's something I'm going to find out,
I don't think you will find that out, unless you are in a position to
tell me which 3 of the web browsers I have installed on this computer
send the User Agent header:-

Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)

- by default (literally), and distinguish between them (and there is a
good chance that you have never even heard of the two that are not IE
6).
and I'd also just like to know how many of them
have turned off JS.
The usual proposal is to have some JS execute in a way that will send
evidence of its action back to the server. But a client with JS enabled
cannot guarantee the success of that action as it will be at least in
part dependent on the DOM provided by the browser, and will be subject
to language implementation bugs (ECMA standardisation has gone a long
way towards removing language implementation bugs but there are still
some). So any test script is virtually guaranteed to be incapable of
performing the required actions on at least some clients and so will
result in a false indication of a javascript incapable client.

But knowing a client's willingness to execute javascript remains
pointless, given javascript capable clients with such limited DOMs that
simple form validation is about the limit of their capabilities.
I'm guessing that percentage will be very low--in the
neighborhood of 2%. I am also curious as to what about
HTTP makes you think that statistics gathering is
pointless.

Caching is the main factor. Clients cache and organisations like ISPs
operate large-scale caches. HTTP is designed to encourage caching so
many request-response transactions will never arrive at the server(s)
for any given site, instead being handled by an intervening cache.
Clients often have user (and administrator) configurable cache settings
and exhibit different behaviour even with superficially similar cache
settings. And the other caches mean that the customers of one ISP may
have considerably more requests served from a cache than the customers
of another. The exact behaviour and characteristics of these caches is
impossible to gauge.

It has been proposed that this caching is already essential as the extra
load imposed on the network of having to route each and every
request/response to/from the actual server responsible would result in
traffic too heavy to be handled.

Obviously, with only a proportion of requests actually arriving at the
server, any statistics gathered on the server cannot be complete. And
the extent to which they may be representative of total requests cannot
be determined (or even estimated) because the information needed to do
that is never available.

This, and the many other factors that a well worded google search would
highlight, leave me thinking that any effort directed at the gathering
of web statistics would be wasted.

The consequences of individuals placing any trust in web statistics
(Think: Hans Christian Anderson's "The Emperor's New Clothes") produces
a feed-back effect that contributes to the meaninglessness of those
statistics. You often encounter people asserting that it is OK to build
IE only JS dependant web sites because their server logs show only a
tiny percentage of visitors not using IE with JS enabled. But given an
IE only, JS dependent web site, is it surprising that users who do not
fall into that category do not hand around on such a site producing log
entries? It becomes a chicken and egg situation where the statistics are
being used to justify the design decisions, while the statistics are
heavily biased by the consequences of the decisions that they were used
to justify.

In the end I think that web statistics are used to re-enforce prejudices
or justify decisions that were going to be made anyway, and so long as
those statistics broadly conform to the expectations of the interested
parities then they will not be (sufficiently) questioned and their
(in)accuracy won't make much difference. In which case I can see no
reason for going to any effort to gather and analyse any data, instead
it would just be a matter of finding out what the
expectations/prejudices of the interested parties were and generating
numbers that broadly conform with them. A set of meaningless numbers
derived with baseless assumptions is not really different from a set of
numbers directly based on assumptions and prejudices.

Richard.

[Richard Cornford]

Ryan Stewart wrote:

Richard Cornford wrote:

I don't think you will find that out, unless you are in a position to
tell me which 3 of the web browsers I have installed on this computer
send the User Agent header:-

Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)

- by default (literally), and distinguish between them (and there is
a good chance that you have never even heard of the two that are not
IE 6).
No, can't do that. I'd imagine the two "others" are
something somebody coded independently

Independently of whom, Microsoft?
and didn't bother coming up with their own UA string
since the IE one worked just fine.
They are both the produces of commercial software houses, and currently
under active development. The spoofing IE's UA string 100% is probably
deliberate as it is not unusual for less skilled/knowledgeable web
developers to attempt to actively exclude UAs that are not IE from their
web site, or provide a limited (and often out of date) alternative for
UAs they don't recognise, regardless of the actual capabilities of those
UAs. If admitting to be a browser that those developers have never heard
of results in the user getting an inferior experience then it is
obviously better for those browsers to clam to be a browser that has
come to the attention of web developers. Another feed-back effect that
diminishes the usefulness of web statistics.
But it's fairly irrelevant to my
situation, and completely so to this post.

<snip>

If you want to know what percentage of IE users disable javascript then
not being able to tell IE from browsers claming to be IE represents a
significant limitation.

And if you are going to assume that those browsers represent an
insignificant contribution to your figures then you are applying your
own prejudices to your task. You have no evidence that your prejudice is
valid (because all web statistics suffer exactly the same problem) and
as a result you eventual figures will be no more significant than
figures you make up directly based on those prejudices. In which case
you may as well fabricate the figures and save yourself the trouble and
expense of gathering and analysing any data.

Richard.