lenzocs.cmu.edu: Re: Unicode Consortium Membership by Simon Cozens- February 21, 2001 Just a note, for anyone who&39;s interested, that I&39;m thinking about getting Perlmembership in the Unicode Consortium. I&39;d ideally like to be able to get usfull voting rights, but that is prohibitively expensive.Simon----- Forwarded message from "Kevin A. Lenzo" <lenzocs.cmu.edu> -----From: "Kevin A. Lenzo" <lenzocs.cmu.edu>Subject: Re: Unicode Consortium MembershipTo: Simon Cozens <simonnetthink.co.uk>Cc: yas-talkyapc.org,...http://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg28.html Re: string encoding by Tom Lord- February 17, 2001 On the subject of Unicode string processing...I&39;m not a perl internals hacker and more of a passive reader of theselists than an active contributor.With that caveat, may I humbly point out a design document for what I think is a clean C library supporting the use of mixedencoding forms. I&39;ve prototyped this design and believe it to bepractical. My advice to implementors is don&39;t try doing it withoutsimultaneously writing comprehensive unit tests for every function --the number of...http://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg27.html Re: string encoding by Simon Cozens- February 16, 2001 On Fri, Feb 16, 2001 at 04:51:14PM -0800, Hong Zhang wrote:> Yes and no. You can use for eq(), but not for cmp(). On little endian> machine, the memcmp() will first compare the least significant byte, not> most.We&39;d use a custom cmp in the vtable in that case anyway.-- <evilPetey> I often think I&39;d get better throughput yelling at the modem.http://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg26.html Re: string encoding by Dan Sugalski- February 16, 2001 At 06:47 PM 2162001 -0800, Hong Zhang wrote:>I like to wrap up my argument.>>I recommend to use UTF-8 as the sole string encoding.>If we end up with multiple encodings, there is absolutely>no point for this argument.Um, I hate to point this out, but perl isn&39;t going to have a single string encoding. I thought you knew that. Dan--------------------------------------"it&39;s like this"-------------------Dan Sugalski even samuraidansidhe.org.http://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg25.html Re: string encoding by Hong Zhang- February 16, 2001 I like to wrap up my argument.I recommend to use UTF-8 as the sole string encoding.If we end up with multiple encodings, there is absolutelyno point for this argument.Benefits of UTF-8 is more compact, less encoding conversion,more friendly to C API. UTF-16 is variable length encodingtoo, if considering the surrogates. UTF-32 is way too big.The main disadvantage of UTF-8 is O(n) random access, which Ipersonally believe is not very important, since most textprocessing require linear scan of...http://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg24.html Re: string encoding by Hong Zhang- February 16, 2001 > > I think you already mixed the codepoint vc character. What you will getis> > 10th codepoint, not 10th character.>> I think you&39;re confused. Codepoints are characters. Combining charactersare> taken care of as per the RFC.If you define that way, I can agree with it. Since you still have to handlecombining character in different place, you will not save much overall.> I&39;m talking about UTF16. You&39;re talking about UTF32.> Try talking about what I&39;m...http://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg23.html Re: string encoding by Hong Zhang- February 16, 2001 > On Fri, Feb 16, 2001 at 02:39:10PM -0800, Hong Zhang wrote:> > But you can not use memcmp() to compare binary order of two UTF-32> > strings on little endian machines, even both strings are using> > the same endian.>> Yes, you can.Yes and no. You can use for eq(), but not for cmp(). On little endianmachine, the memcmp() will first compare the least significant byte, notmost.Honghttp://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg22.html Re: string encoding by Simon Cozens- February 16, 2001 On Fri, Feb 16, 2001 at 02:25:59PM -0800, Hong Zhang wrote:> I think you already mixed the codepoint vc character. What you will get is> 10th codepoint, not 10th character.I think you&39;re confused. Codepoints are characters. Combining characters aretaken care of as per the RFC.> The UTF-32 has its problems too, such as cache locality, memory footprint,> encoding conversion.I&39;m talking about UTF16. You&39;re talking about UTF32.Try talking about what I&39;m talking about.> I..http://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg21.html Re: string encoding by Simon Cozens- February 16, 2001 On Fri, Feb 16, 2001 at 02:39:10PM -0800, Hong Zhang wrote:> But you can not use memcmp() to compare binary order of two UTF-32> strings on little endian machines, even both strings are using> the same endian.Yes, you can.> BTW, with UTF-8, you never worry about endian issue.cough. I know this.-- "If that makes any sense to you, you have a big problem." -- C. Durance, Computer Science 234http://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg20.html Re: string encoding by Hong Zhang- February 16, 2001 > > I have already given the counter argument. The codepoint position isuseless> > in many cases. They should be deprecated.>> Uh That doesn&39;t make sense. Codepoint position is exactly what people> expect when they use substr. When I say>> $a = substr($b,10);>> I want the 10th character. If I get the 10th byte, and we&39;re using UTF-8> as you suggest, I might be cutting into the middle of a character, leaving> the resulting string malformed....http://www.nntp.perl.org/group/perl.perl6.internals.unicode/2001/02/msg17.html |