perl.i18n...Re: Stripping out Unicode combining characters (diacritics) - by Brad Baxter- May 8, 2008 Just to throw this out there: you may be interested in Text::Unidecode(http:search.cpan.orgsburkeText-Unidecode-0.04) if your ultimategoal is to try to represent a unicode character with its closest ascii(or perhaps I should say, "romanized") equivalent.-- BradOn Wed, May 7, 2008 at 9:51 AM, Doran, Michael D <doranuta.edu> wrote:> I received a number of helpful suggestions and solutions. The approach I> decided to adopt in my larger script is to &39;decode&39; all the...http://www.nntp.perl.org/group/perl.i18n/2008/05/msg213.html RE: Stripping out Unicode combining characters (diacritics) - by Doran, Michael D- May 7, 2008 I received a number of helpful suggestions and solutions. The approach I decided to adopt in my larger script is to &39;decode&39; all the incoming form input as UTF-8 as well as the input from the database that I&39;ll be matching the form input against. This seems to allow the &39;pM&39; syntax to work as expected in a Perl regexp. In my test.cgi script for form input it would like like this: !usrlocalbinperl use strict; use CGI; use Encode; my $query = CGI::new(); my $search_term =...http://www.nntp.perl.org/group/perl.i18n/2008/05/msg212.html Re: Stripping out Unicode combining characters (diacritics) by David Kaufman- May 7, 2008 Hi Michael,"Doran, Michael D" <doranuta.edu> wrote:> I&39;m trying to strip out combining diacritics from some form input using > this code:> ...> $sans_diacritics = spMg;I do it like this:use Encode;use Unicode::Normalize qw(normalize);my $ascii = encode(&39;ascii&39;, normalize(&39;KD&39;, $utf8), sub $_0=&39;&39; );http://www.nntp.perl.org/group/perl.i18n/2008/05/msg211.html RE: Stripping out Unicode combining characters (diacritics) by Doran, Michael D- May 6, 2008 Hi Leif,> This is what I do. You can try that.> See if it helps:> > Encode::_utf8_on($str); <<<> $str = spMg;That works! I will gladly buy the beers Leif, should we ever meet in person.> I mean - have you for instance tried running your cgi scripts > in tainted mode (-T)No, I do not run my CGI scripts in tainted mode (although I realize that I probably should). Thanks (once again) for your help.-- Michael Michael Doran, Systems Librarian University of Texas at..http://www.nntp.perl.org/group/perl.i18n/2008/05/msg210.html Re: Stripping out Unicode combining characters (diacritics) by Leif Andersson- May 6, 2008 I&39;ve been doing it like Mike R suggested for quite some while.But some characters do not map nicely into this scheme.So you may want to manually take care of stuff like german eszet, ligature oe etc, etc.sx00dfssg;sx0152Oeg;sx0153oeg;...to be continued...Leif======================================Leif Andersson, Systems LibrarianStockholm University LibrarySE-106 91 StockholmSWEDENPhone : +46 8 162769Mobile: +46 70 6904281-----Ursprungligt meddelande-----Från: Doran, Michael D...http://www.nntp.perl.org/group/perl.i18n/2008/05/msg209.html Re: Stripping out Unicode combining characters (diacritics) by Mike Rylander- May 6, 2008 On Mon, May 5, 2008 at 8:26 PM, Doran, Michael D <doranuta.edu> wrote:snip>> I&39;m pulling my hair out on this... so any help would be appreciated. If there&39;s any other info I can provide, let me know.>You&39;ll want to transform the text to NFD format (nominally, basecharacters plus combining marks) instead of NFC (precombinedcharacters) using Unicode::Normalize: use Unicode::Normalize; my $text = NFD($original); $text = spM+go;Hope that helps.-- Mike Rylander VP,...http://www.nntp.perl.org/group/perl.i18n/2008/05/msg208.html RE: Stripping out Unicode combining characters (diacritics) by Doran, Michael D- May 5, 2008 Hi Mike,I appreciate the quick reply. I am familiar with the Unicode::Normalize module (and will also be using that), but I left it out of this question because it&39;s not relevant to the problem I&39;m currently trying to solve. The text I&39;m trying to strip diacritics out of does not have precomposed accented characters.-- Michael Michael Doran, Systems Librarian University of Texas at Arlington 817-272-5326 office 817-688-1926 cell doranuta.edu http:rocky.uta.edudoran-----Original...http://www.nntp.perl.org/group/perl.i18n/2008/05/msg207.html Stripping out Unicode combining characters (diacritics) by Doran, Michael D- May 5, 2008 I&39;m trying to strip out combining diacritics from some form input using this code: <head> <META http-equiv="Content-Type" content="texthtml; charset=UTF-8"> <head> <body> <form action="test.cgi" accept-charset="UTF-8" method="get"> <input type="text" name="text" value="" size="10"> <input type="submit" value="submit"> ...http://www.nntp.perl.org/group/perl.i18n/2008/05/msg206.html |