status of utf-8.el, etc [Re: Several serious problems]

[Kenichi Handa <handa@m17n.org>]

In article <rzqd6rin893.fsf@albion.dl.ac.uk>, Dave Love <d.love@dl.ac.uk> writes:
> Richard Stallman <rms@gnu.org> writes:
>>      For instance, the RC version of mule-utf-8 doesn't translate
>>      cyrillic-iso8859-5, and the Cyrillic coding systems don't translate
>>      mule-unicode-0100-24ff.
>>  
>>  We could consider adding that support in RC.  Is it a safe change?

> It won't break anything if done correctly, but I don't remember how
> much of a change it is relative to the 21.2 code and I don't know who
> might have been testing it, if anyone.

I noticed some combinations of unify-8859-on-encoding-mode,
utf-8-fragment-on-decoding, and utf-8-translate-cjk doesn't
work in HEAD.  So, I made a fairly comprehensive testsuite
for testing them (attached at the tail).

As the testsuite revealed several bugs, before working on
RC, I decided to fix them in HEAD at first.  I've finished
these:

(1) Fixing the following bugs.

(1-1) unify-8859-on-encoding-mode can't be turned off
safely.  For instance, then, iso-latin-1 can't encode
Latin-1 chars.

(1-2) utf-8-translate-cjk can never be turned off once
turned on.

(1-3) When utf-8-fragment-on-decoding is non-nil, utf-16-*
doesn't encode CJK chars correctly even if
utf-8-translate-cjk is non-nil.

(1-4) encode-char/decode-char don't reflect utf-8-translate-cjk.


(2) Renaming tables/variables.   We should have cleaner
    names before people starting to use it.

(2-1) As utf-8-fragment-on-decoding and utf-8-translate-cjk are
also applicable to utf-16, I cut off "-8" from them.

(2-2) Make translation table names and their body
char-tables different to avoid confusion.

The result is as follows:

(2-2-1) Translation-tables and translation-hash-tables (not variable)

old					new
---					---
ucs-mule-to-mule-unicode		utf-translation-table-for-encode
  (mule-utf-8/16 use it for encoding)

utf-translation-table-for-decode	utf-translation-table-for-decode
  (mule-utf-8/16 use it for decoding)

utf-8-subst-rev-table			utf-subst-table-for-encode
  (mule-utf-8/16 use it for encoding)

utf-8-subst-table			utf-subst-table-for-decode
  (mule-utf-8/16 use it for decoding)

(2-2-2) Mapping tables (variables) populating above.

old					new
---					---
ucs-mule-to-mule-unicode		ucs-mule-to-mule-unicode
  (this populates utf-translation-table-for-encode
   when unify-8859-on-encoding-mode is non-nil)

utf-8-subst-table			ucs-unicode-to-mule-cjk
  (this populates utf-subst-table-for-decode
   when utf-translate-cjk is non-nil)

utf-8-subst-rev-table			ucs-mule-cjk-to-unicode
  (this populates utf-subst-table-for-encode
   when utf-translate-cjk is non-nil)

utf-8-fragmentation-table		utf-fragmentation-table
  (this populates utf-translation-table-for-decode
   when utf-fragment-on-decoding is non-nil)

--not_exist--				utf-defragmentation-table
  (this populates utf-translation-table-for-encode
   when unify-8859-on-encoding-mode is nil
   and utf-fragment-on-decoding is non-nil)

utf-8-translation-table-for-decode	--deleted--


Don't you have better ideas for these names?   If not, I'll
install the changes soon.

---
Ken'ichi HANDA
handa@etl.go.jp

Two files: utf-test.el and result.txt.

result.txt is the result of loading utf-text.el, running M-x
utf-testsuite RET, and viewing the variable
utf-testsuite-result in the current Emacs.  After my
modification, all elements are `t'.

begin 664 temp.tar.gz
M'XL(`'9=D3T``^U:6W/;-A9V9O=%>M@_D!>LIU-3K<DEJ:OMZ>ZJJ:?U)K%G
M7'?[X&0F%`7)M"G2(4$G_O=[#@#>Q(LOL;M*!\<C&I=SP\&'`Q!2PA8ZHS$S
MJ+_U7&1:ICD:#;9,TS:MTG\DVQH/H0R/@3W$#N"W+7.P1<QG\ZA`2<R<B)"M
M"R>8.RU\=_5_I71P0/3O=.*&<R]8[A,O#G7;M&U]//,8[^IVM3E=W$",?(=Y
M@6[I[@54M)5S145Q1W2`Z&0RW-,M\J]W?=/L]=8$[8<++B-*KZIBHCD5&U?$
M+KWX,T!L4I6\=*Z=@,943UG(8$+Z?9#EPFX8Q(PD<DG$B<=`D$=&CV]C1E=Q
MEY!SC)&,!51)7K7+U3VL9K[J$X@HMKBWD>?[GEMJ9!>.IS,O'MDF5C,_BSRK
MQ*<Z.C<IU:R1/J/O6T8P=Y@#$C#3&`0]<%945.-KGVN&HAQ/)P@"$@2,!`S^
M,T88U!G4&=09*S';'<,PB@U[HE(9,1JN&3,T9P.&,DU<_?):E--!RB(?H:C^
M]NI70@.8$S&C_YC3K`P,.YJ6@G2J+R/G!J6`M"K0/K\R>[)S&^&SW>DPG(>F
M3\H<>'ZG`P_2\EE7O,ZR_7)W]N^7VH_;[:7'F-P$7AQV_YV-0[]72(%_/(+/
M>$O$ZUY"UIY]UXS!DBA,OJV?ZHZ;L'I(V&5(/&:6GQP^7QLO!F+XSK8'VS6(
M_V;7(DSB^\YR!1DOSIY,9W]@UJW(A^E:A];T3FA9Q2A9]4EA.DT30'-I7<U]
M5UB3S5FFO[FT22![3"*RZL#ST&CG*>K+U5E[_<<#H(R]/?TPB<+Z;6Z(R)L.
MGC_,]F#P9TV`,/,#&S`$GV%38F-O\@317J[@B3RIVDE_-&A(;@]0E@),'.2F
M_O6%(_57#]Z%Q+8ALW7_K+@A#ELCV#,+.27UDT_)MQ_SZ9'E^K/37[Y`R9Y5
M2$?WDY2B`BC9Z?[H1&HLGO<Y5D0JVBBL/&R:[,'_'5<PU[99WLY*DZ7]/9\L
M6:['RE]?/%*#9=K#!J0TBLIL<O!R]PR;";ZAAH$#[Y3;;P[/S@Y/R>L3\GIZ
M)*#$WX-?G^AI@VP1+XI?-X8V`F^XG0&&QF;3L0BF[T<GG\KV<@59?WOQ9#K[
MP\FP82][@+)T+\NNA%[]Y[4^E2::;H,V"`B;P`M@')D"D/=]VQ@,`&-P9.K+
MLPWVU_ZO(.CX`4(`85N>=YIY>^GE'MX,EB_&(AHG/D,^4+A]*FKA@K`+"FEJ
MM7*">5G"Z!ZQG9@XY(:Z+(S(.7GUR_14-_73PU]_>W-&#,,0+6_2EO=&M]S@
MQ5Q]E!L#Y5ZP)`O0)UAKK)S\='3\<]%.VO2V:&BMJ=&4,,,MBJ'B#2<1-YR9
M%J*]91>$^G1%`R[]H>U>=*=7]?KMR4^')9]YP[C@,6\X?B)_O8#WKL(Y)<<&
M(<>HT#2,L=$]PW;J!"@&>GF/$Y,9]<-/^S#W1PMX=PJ7.-_'9-#C@KLD";S%
MK<Z/+V&@\UM''#$W`!P`&[!R`C:C3UY,=SDH/8Q!`.S8NZ[9SC1#)!>1L\30
MHFY^BXG./4JM553+(B>(X3T0)NCRZI[ZUL)3F95BJ-@^CS).#;EVXIC.29RX
M+HWC1>+[M^B<N0]3%$4``_^6S"F#`G"!$AY"9^93@H[G/.)"=RY6]2QAY%,4
M!DLNS#O(S'&O4+-UM^:BN-2+DC#.?4#(7=*`L$1.-=8!TPL2ARLJL\**!S1D
MA.U""'G,$+,.P\Z(<OC/DF4*13<!:P'K'JX<-S:V,0^!`:VT<O1KLI/=<8O$
MID7T8^)%-.U($U@2E+,1T9!?\P)&(\=EW@V7UWS*B`;/P,)RL(157/T^`'1V
ML-=NX"FO[IX\[VLQ91_KDZCXED6N?6X<=Q`I-@^9MZ(Q^,J[9/Y.706ALKAF
M?8]\L%RX#G35`<OUACU,.,B2&;G,%4`[F.'":.6R;&<B]6>GA$P%7^.3S$\/
MEAM"5IN%23"7,S:I7<+<HMQC`2@M.:3`^4.VFKEAR$"F</W@@/P>1E?$B=`P
M!Q7B"]"8N'%N'_7#(O\=EG$"ZY(E42"EV04@EBL-`UC$D$(I[T;XAHN%P>/3
MENBL7A8%%W2'*QV"J<->ZN$"B<F'-NENIR/'N,MC^,^:<<(6C3NZY/R@"<YB
MH!O"+"0:.01#^TRU>F??S[M2PBV[5>HJ^5,5:G'$RAW)YB)#*JX'W9M_E@N+
MKVKV':PJBHLR@(6=<=1G@12#Z]^5:BX^4($%IP`PG5:*;7Q0O,$NM1<5QRSB
M#]Q=4'/):*S['B!6<R*Z$%]/EK,CXVE9B(G%7,XEME26C5JXH7T/>0$F,'52
MV))FVC(=\3+^?%<0*B&G\&5+:)D%]A`^:Q@OZ-72KR>E7C%P#(++\ZB03'<V
ME,RTI%]FEB33WJ)TFG-38?0R!Y%P9T57'T&(R!!GR.2=L,$X?N:R-)&QD#*;
M])7W8C_O$MY=Y_9[63]2R2W9"B(2,5AEL@1)ZH.Y4]Q+.?R\..^OG!1PE\^[
MZT\+HMLL#`HM68V64I;F8\4:(Z;WBF=&SF))T_Q-*FULF1I\#<GC7UI$8@<3
MZ*W#L,1#NM+A/"B1(A*(Z$X74&6E\&-`<;'<"?T<^`)!HK$5E0)/^3?W(O8[
ML(W)X7.&'YIX:C`J$9HOFY2YS%M@;L:L"&H^;+,H2[*TQ9M8<9GD^+(*:;PX
MYW<,/!?(:H6I?_#$RY3869]YKK+N]S^"PV"?V?/]QJC]]U_F<#@:I[__&@T'
MIOC]EZE^__5'4!V>X#WFQO$3_I8;7X2?`O'Z9W2[ZHY$W9&H.Q)U1_)<=R3=
M[G\Q\>QWS\_/UV^DF?A[#YZ>,U+ZV_@V:#S?&&>PK2VXC^E3`U0#?*8!?DF[
M&M0?.:@_+12Q)!PL/&5/I1V?K7TPOMI7,46*%"E2I$B1(D6*%"E2I$B1(D6*
7%"E2I$B1(D6*%"GZ(OH?U(],I@!0````
`
end