UTF-8 garbage characters

[lawrence]

I'd love to ask why this page is not rendering correctly in Safari on
a Macintosh but I suspect someone will tell me to validate the page
first. Nevertheless, if anyone sees an obvious reason that I'm
missing, I'd like to know. It looks like a missing div tag but I
can't see one.

http://www.krubner.com/
Let's move on to a question that might be answerable. If i copy and
paste non-UTF-8 characters to the page, and then send out a UTF-8
charset UTF-8 header then I'll get the garbage characters that I'm
seeing?

[Arne]

*lawrence* skrev 2004-10-01 09:34:

I'd love to ask why this page is not rendering correctly in Safari on
a Macintosh but I suspect someone will tell me to validate the page
first. Nevertheless, if anyone sees an obvious reason that I'm
missing, I'd like to know. It looks like a missing div tag but I
can't see one.

http://www.krubner.com/
Let's move on to a question that might be answerable. If i copy and
paste non-UTF-8 characters to the page, and then send out a UTF-8
charset UTF-8 header then I'll get the garbage characters that I'm
seeing?

If you set charset UTF-8 in the header, you also have to save the
document as UTF-8 before uploading it. I belive it's saved in i default
charset, probably iso-8859-1

--
/Arne

[lawrence]

Arne <ar**@luras.n u> wrote in message news:<SH******* **********@news b.telia.net>...

*lawrence* skrev 2004-10-01 09:34:

I'd love to ask why this page is not rendering correctly in Safari on
a Macintosh but I suspect someone will tell me to validate the page
first. Nevertheless, if anyone sees an obvious reason that I'm
missing, I'd like to know. It looks like a missing div tag but I
can't see one.

http://www.krubner.com/
Let's move on to a question that might be answerable. If i copy and
paste non-UTF-8 characters to the page, and then send out a UTF-8
charset UTF-8 header then I'll get the garbage characters that I'm
seeing?

If you set charset UTF-8 in the header, you also have to save the
document as UTF-8 before uploading it. I belive it's saved in i default
charset, probably iso-8859-1

Since sending out UTF-8 headers, I've started to get a lot of garbage
characters on the page. This despit hitting it with encode_UTF-8 in
PHP. The problem is I don't know what charset the text is in a head of
time, so I can't do any kind of proper conversion on the text. The web
is too heterogenous. I wonder how Blogger and TypePad tackle this
problem? Is it just luck that so few of their users end up with
garbage characters? Or is it because they standardize on a different
charset?

[Arne]

*lawrence* skrev 2004-10-02 02:13:

Arne <ar**@luras.n u> wrote in message news:<SH******* **********@news b.telia.net>...

*lawrence* skrev 2004-10-01 09:34:

> I'd love to ask why this page is not rendering correctly in Safari on
> a Macintosh but I suspect someone will tell me to validate the page
> first. Nevertheless, if anyone sees an obvious reason that I'm
> missing, I'd like to know. It looks like a missing div tag but I
> can't see one.
>
> http://www.krubner.com/
>
>
> Let's move on to a question that might be answerable. If i copy and
> paste non-UTF-8 characters to the page, and then send out a UTF-8
> charset UTF-8 header then I'll get the garbage characters that I'm
> seeing?

If you set charset UTF-8 in the header, you also have to save the
document as UTF-8 before uploading it. I belive it's saved in i default
charset, probably iso-8859-1

Since sending out UTF-8 headers, I've started to get a lot of garbage
characters on the page. This despit hitting it with encode_UTF-8 in
PHP. The problem is I don't know what charset the text is in a head of
time, so I can't do any kind of proper conversion on the text. The web
is too heterogenous. I wonder how Blogger and TypePad tackle this
problem? Is it just luck that so few of their users end up with
garbage characters? Or is it because they standardize on a different
charset?

What editor do you use for building and editing the pages?

In the editor I use, I can choose whatever encoding I want to use, and
on saving the file is saved in that encoding. If I open a file I can
change the encoding the same way, with just a click on a button in the
editors toolbar. In Windows XP you can also choose encoding for files
when saving them in Notebook.

Because I save the files in UTF-8 I don't manually need to do any
conversion in the text, when writing it. My guess is that Blogger and
TypePad is done the same way, the posts that the user write is saved as
UTF-8 and since that is a unicod, all kind of characters and languages
can be used.

--
/Arne

[lawrence]

Arne <ar**@luras.n u> wrote in message news:<%9******* **********@news b.telia.net>...

Since sending out UTF-8 headers, I've started to get a lot of garbage
characters on the page. This despit hitting it with encode_UTF-8 in
PHP. The problem is I don't know what charset the text is in a head of
time, so I can't do any kind of proper conversion on the text. The web
is too heterogenous. I wonder how Blogger and TypePad tackle this
problem? Is it just luck that so few of their users end up with
garbage characters? Or is it because they standardize on a different
charset?
What editor do you use for building and editing the pages?

Most people use Microsoft Internet Explorer to build pages on their
weblogs. That is, they log in and type some text in a TEXTAREA and hit
"Post" and then there words appear as a new page on the web.

The problem arises when they copy and paste from other places. The
biggest problems, I think, arise when they copy text off a webpage
that uses another web page encoding.

You can check see for yourself here:

http://www.publicdomainsoftware.org/designer/

You'll need a username and password to login. Use these:

username: designer
password: designer123

Link for login at bottom.

Because I save the files in UTF-8 I don't manually need to do any
conversion in the text, when writing it. My guess is that Blogger and
TypePad is done the same way, the posts that the user write is saved as
UTF-8 and since that is a unicod, all kind of characters and languages
can be used.

I guess that is the theory that now needs to be tested. My suspicion
is the opposite of yours - they are using something other than UTF-8,
which is why garbage characters appear if you copy something from them
and paste it into a web page that is using UTF-8.

[Daniel R. Tobias]

lk******@geocit ies.com (lawrence) wrote in message news:<da******* *************** ****@posting.go ogle.com>...

Arne <ar**@luras.n u> wrote in message news:<%9******* **********@news b.telia.net>...
Most people use Microsoft Internet Explorer to build pages on their
weblogs. That is, they log in and type some text in a TEXTAREA and hit
"Post" and then there words appear as a new page on the web.

I would have thought that bloggers would be the sort to be clueful
enough to use decent browsers, like Mozilla, instead.

--
Dan

[Arne]

*Daniel R. Tobias* skrev 2004-10-05 22:49:

lk******@geocit ies.com (lawrence) wrote in message news:<da******* *************** ****@posting.go ogle.com>...

Arne <ar**@luras.n u> wrote in message news:<%9******* **********@news b.telia.net>...
Most people use Microsoft Internet Explorer to build pages on their
weblogs. That is, they log in and type some text in a TEXTAREA and hit
"Post" and then there words appear as a new page on the web.

I would have thought that bloggers would be the sort to be clueful
enough to use decent browsers, like Mozilla, instead.

Ok. I see noe what Lawrence's problem is. The browser used don't matter
here. When testing the blog I used som words from my native language
(Swedish) and writed this:

The problem with charset UTF-8 on pages with forms for e.g. guestbooks,
formmail and bloggs is that writing in a non-english language can give
garbage characters from the letters that is not represented in the
english language. That's because what is writed in the text box don't
get encoded, as text done with HTML editors does.
So the best may be to use an other charset (is ISO-8859-1 the best?) for
pages with forms and textbox. ISO-8859-1 cover a lot of languages with
"strange" characters, but far from all, so the problem may not totally
be solved.

You can write the message in a separate software (sutch as Notebook) and
then encode foreign letters to entities. As an example the letter "å"
(latin small letter a with ring above) to &aring; but then, will the
&aring; look like "å" &aring; or on the page after posting?

--
/Arne

[Pierre Goiffon]

"Arne" <ar**@luras.n u> a écrit dans le message de
news:4t******** *************@n ewsc.telia.net

The problem with charset UTF-8 on pages with forms for e.g.
guestbooks, formmail and bloggs is that writing in a non-english
language can give garbage characters from the letters that is not
represented in the english language. That's because what is writed in
the text box don't get encoded, as text done with HTML editors does.

I really can't understand your post. A server that sends a form to a client
with the appropriate charset headers should get in return all the users
input encoded in that charset. If the form is sent with a UTF-8 header, you
should get all the characters encoded in UTF-8. And so user could input any
character included in Unicode.

[Alan J. Flavell]

On Wed, 6 Oct 2004, Pierre Goiffon wrote:

"Arne" <ar**@luras.n u> a écrit dans le message de
news:4t******** *************@n ewsc.telia.net

The problem with charset UTF-8 on pages with forms for e.g.
guestbooks, formmail and bloggs is that writing in a non-english
language can give garbage characters from the letters that is not
represented in the english language. That's because what is writed in
the text box don't get encoded, as text done with HTML editors does.
I really can't understand your post. A server that sends a form to a client
with the appropriate charset headers should get in return all the users
input encoded in that charset.

"should" sounds right, but it isn't always going to work, depending on
all kinds of browser bugs and oddities, to say nothing of the
uncertainty of what should happen with method GET (which /officially/
only supports us-ascii).
If the form is sent with a UTF-8 header, you should get all the
characters encoded in UTF-8. And so user could input any character
included in Unicode.

On the other hand, the combination of user errors, browser bugs and
plain malice means that your server *could* get presented with all
kinds of rubbish: you might not be able to do anything useful with it,
but you better program defensively to ensure it won't do you any harm.
(E.g you better validate any byte-sequences that are supposed to be
utf-8 to make sure they really are - the unicode spec effectively
mandates such validation, and failure to do so could be rated as a
security exposure).

I haven't worked this page over in a while, but it gives a general
flavour of what I found, reviewed against what the specifications say
(and - also important - what they /don't/ say):
http://ppewww.ph.gla.ac.uk/~flavell/...form-i18n.html

h.t.h

[Andreas Prilop]

On Wed, 6 Oct 2004, Pierre Goiffon wrote:

A server that sends a form to a client
with the appropriate charset headers should get in return all the users
input encoded in that charset. If the form is sent with a UTF-8 header, you
should get all the characters encoded in UTF-8. And so user could input any
character included in Unicode.

Try
<http://google.com/search?q=%EA%E5 %F1%DC%F4%E9%EF .%ED&ie=ISO-8859-7&oe=UTF-8>
with Netscape 4.x.

--
Top-posting.
What's the most irritating thing on Usenet?