perl.unicode...Re: utf8::valid and x14_000 - x1F_0000 by Juerd Waalboer- March 12, 2008 Chris Hall skribis 2008-03-12 13:20 (+0000):> >> OK. In the meantime IMHO chr(n) should be handling utf8 and has no> >> business worrying about things which UTF-8 or UCS think aren&39;t> >> characters.> >It should do Unicode, not any specific byte encoding, like UTF-8.> IMHO chr(n) should do characters, which may be interpreted as per> Unicode, but may not.> When I said utf8 I was following the (sloppy) convention that utf8 means> how Perl handles...http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3145.html Re: utf8::valid and x14_000 - x1F_0000 by Chris Hall- March 12, 2008 On Tue, 11 Mar 2008 Juerd Waalboer wrote>Chris Hall skribis 2008-03-11 21:09 (+0000):>> OK. In the meantime IMHO chr(n) should be handling utf8 and has no>> business worrying about things which UTF-8 or UCS think aren&39;t>> characters.>It should do Unicode, not any specific byte encoding, like UTF-8.IMHO chr(n) should do characters, which may be interpreted as perUnicode, but may not.When I said utf8 I was following the (sloppy) convention that utf8 meanshow Perl...http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3144.html Re: utf8::valid and x14_000 - x1F_0000 by Juerd Waalboer- March 11, 2008 Chris Hall skribis 2008-03-11 21:09 (+0000):> OK. In the meantime IMHO chr(n) should be handling utf8 and has no > business worrying about things which UTF-8 or UCS think aren&39;t > characters.It should do Unicode, not any specific byte encoding, like UTF-8.Internally, a byte encoding is needed. As a programmer I don&39;t want tobe bothered with such implementation details.> Note that chr(n) is whingeing about 0xFFFE, which Encode::endecode> (UTF-8) are happy with. Unicode...http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3143.html Re: utf8::valid and x14_000 - x1F_0000 by Chris Hall- March 11, 2008 On Tue, 11 Mar 2008 you wrote>Chris Hall skribis 2008-03-11 18:48 (+0000):>> I&39;m comfortable with the notion that perl characters are unsigned>> integers that overlap UCS, and happen to be held internally as a>> superset of UTF-8.>> I wonder if perl is completely comfortable.>It isn&39;t. There are some very unfortunate "features".>> chr(n) throws various runtime warnings where &39;n&39; isn&39;t kosher UCS, and>> "xh...h" throws..http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3142.html Re: utf8::valid and x14_000 - x1F_0000 by Juerd Waalboer- March 11, 2008 Chris Hall skribis 2008-03-11 18:48 (+0000):> I&39;m comfortable with the notion that perl characters are unsigned> integers that overlap UCS, and happen to be held internally as a> superset of UTF-8.> I wonder if perl is completely comfortable.It isn&39;t. There are some very unfortunate "features".> chr(n) throws various runtime warnings where &39;n&39; isn&39;t kosher UCS, and> "xh...h" throws the same ones at compile time.> (...)I&39;m not sure I see.http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3141.html Decode, byte codes ASCII & ISO-8859 and HTMLCREF or XMLCREF by Chris Hall- March 11, 2008 Having tried: $o = Encode::decode(&39;ascii&39;, "abc x80 xFF", FB_HTMLCREF) $o = Encode::decode(&39;ISO-8859-7&39;, "abc xFF", FB_HTMLCREF) $o = Encode::decode(&39;ascii&39;, "abc x80 xFF", FB_XMLCREF) $o = Encode::decode(&39;ISO-8859-7&39;, "abc xFF", FB_XMLCREF)(0xFF is not a valid character value in ISO-8859-7.)I find that they produce neither &128; and &255; nor &x80; and &xFF, but exactly the same as: $o.http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3140.html Re: utf8::valid and x14_000 - x1F_0000 by Chris Hall- March 11, 2008 On Tue, 11 Mar 2008 you wrote>Chris Hall skribis 2008-03-11 13:30 (+0000):>> I suggest utf8::valid() is broken.>> my $s = chr($c) ;>> my $v = utf8::valid($s) 1 : 0 ;>Agreed. utf8::valid(chr $foo) should ALWAYS return true. (Please note>that utf8::valid tests the internal consistency of a string - on the>outside, it has little to do with UTF8.)I&39;m comfortable with the notion that perl characters are unsignedintegers that overlap UCS, and happen to be...http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3139.html Re: utf8::valid and x14_000 - x1F_0000 by Juerd Waalboer- March 11, 2008 Chris Hall skribis 2008-03-11 13:30 (+0000):> I suggest utf8::valid() is broken.> my $s = chr($c) ;> my $v = utf8::valid($s) 1 : 0 ;Agreed. utf8::valid(chr $foo) should ALWAYS return true. (Please notethat utf8::valid tests the internal consistency of a string - on theoutside, it has little to do with UTF8.)Could you please report this bug with perlbug-- Met vriendelijke groet, Kind regards, Korajn salutojn, Juerd Waalboer: Perl hacker <juerd.nl> ...http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3138.html utf8::valid and x14_000 - x1F_0000 by Chris Hall- March 11, 2008 It appears that utf8::valid() disagrees with Encode::encode(&39;utf8&39;, ...)do not agree for characters 0x14_0000 - 0x1F_0000.I suggest utf8::valid() is broken.The following: use strict ; use Encode qw(FB_QUIET LEAVE_SRC) ; printf "Perl v%vd & Encode %sn", $V, $Encode::VERSION ; my $c = 0xFFFF ; while ($c < 0x8000_0000) my $s = chr($c) ; my $v = utf8::valid($s) 1 : 0 ; my $o = Encode::encode(&39;utf8&39;, $s, FB_QUIET() LEAVE_SRC()) ; my $r = $o 1 : 0 ; .http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3137.html |