Bug: Export templates and UTF8BOM

[svenne]

This is about export templates and the charset encoding UTF8BOM, PVD v0.9.9.21:
It seems to me that something goes wrong when exporting with:
encoding="UTF8BOM"
Every template using UTF8BOM generates three characters (the byte order mark) at the beginning of the exported file. These should be invisible, but every program I tested does not interpret them. They are displayed instead. Special characters within the text are broken. As an example: "ü" will be "ü". Notepad (unicode capable, WinXP) reports the files to be ANSI-encoded instead of UTF-8. Different files (not generated with PVD) saved as UTF8 with and without byte order mark are handled correctly by Notepad.

[buah]

Another unicode issue?

[rick.ca]

Every template using UTF8BOM generates three characters (the byte order mark) at the beginning of the exported file.

So that's why I was getting "?»?" at the beginning of my files. I'm glad I'm not alone.

[nostra]

Strange, but really all apps I tried did interpreted the BOM.
You are using Windows 7 as well, Rick, where do you see these characters?

[buah]

Rick, where do you see these characters?

I believe in html files edited with notepad, or some other "pad". I can confirm appearance of those characters unrelated to PVD but with earlier unicode issues in html files exported from "WhereIsIt?"

[nostra]

Unfortunately I can't reproduce the problem neither with notepad nor with Notepad++

[rick.ca]

You are using Windows 7 as well, Rick, where do you see these characters?

Windows 7. If, for example, I export using the "plain list" template and open the result in Notepad++, the first three characters displayed are ?»?.

[nostra]

Hmm, I do not know what am I doing wrong, but it does not work for me

[Ivek23]

Every template using UTF8BOM generates three characters (the byte order mark) at the beginning of the exported file.

So that's why I was getting "?»?" at the beginning of my files. I'm glad I'm not alone.

With me is the same as using in XP Pro SP 3.

[svenne]

Unfortunately I can't reproduce the problem

Just a guess: perhaps something is different with your database that has an effect on this issue? Did you try it with a newly generated database, too?

[nostra]

Unfortunately I can't reproduce the problem

Just a guess: perhaps something is different with your database that has an effect on this issue? Did you try it with a newly generated database, too?

I have tried it with a new database with the same result. I think I must try it on a Win XP machiene...

[mgpw4me@yahoo.com]

Vista also shows the header, regardless of the app (ie. notepad++).

[svenne]

Ok, to narrow it down... the problem seems to be: all files exported by PVD with utf8bom (at least on my computer with WinXP) start with the following three bytes:
3F BB 3F
Displayed as ISO 8851-1 (or ANSI) this is: ?»?

But the correct BOM for UTF8 should be:
EF BB BF
Displayed as ISO 8851-1 (or ANSI) this is: 
This one works fine and is recognized as UTF8.

I attached a zip containing a text file exported with PVD (template: "Plain list.ptm") using UTF8BOM (called "file as exported by PVD.txt") and a second file where I (manually) changed the first three bytes to EF BB BF ("same file with BOM corrected manually.txt")...

Plain list.ptm:

Code: [Select]

%OPTIONS%
filter="Text Files|*.txt"
encoding="UTF8BOM"
%OPTIONS%{%value=203}. {%value=title} / {%value=origtitle} ({%value=year}) [{%value=genre}]
Quite often programs change characters to "?" when the character that is exported does not fit the encoding. Might be a hint, but I'm wondering if it makes any sense, because ï and ¿ are both present in the ANSI character table.


[attachment deleted by admin]

[daddydave]

I have this problem as well and was able to reproduce svenne's troubleshooting. Is the fix on the to-do list for Version 1?

[nostra]

A fix will be available soon