Re: [Emacs-diffs] master c4782ea: Improve and extend filepos-to-bufferpos

Re: [Emacs-diffs] master c4782ea: Improve and extend filepos-to-bufferpos

[Stefan Monnier]

> +            (if (<= byte eol-offset)
> +                (setq pos (point-min))
> +              (setq pos (point-max))))

Aka (setq pos (if (<= byte eol-offset) (point-min) (point-max)))

>    (let ((eol (coding-system-eol-type coding-system))
>          (type (coding-system-type coding-system))
> +        (base (coding-system-base coding-system))
>          (pm (save-restriction (widen) (point-min))))
> +    (and (eq type 'utf-8-emacs)
> +         (setq type 'utf-8))

(coding-system-type 'utf-8-emacs) returns `utf-8', so how/when can
`type' be `utf-8-emacs'?

> +    (and (eq type 'utf-8)
> +         ;; Any post-read/pre-write conversions mean it's not really UTF-8.
> +         (not (null (coding-system-get coding-system :pos-read-conversion)))
> +         (setq type 'not-utf-8))

I guess this also applies for latin-N and utf-16, IOW for any value of
`type', right?

> +    (and (not (eq type 'utf-8))
> +         (eq quality 'exact)
> +         (setq type 'use-exact))

IIUC this makes us use the slow exact code for latin-N.  Why is it needed?

> +      (`utf-16
> +       ;; Account for BOM, which is always 2 bytes in UTF-16.
> +       (setq byte (- byte 2))

Should that only be done for utf1-16B-with-signature?

> +       ;; In approximate mode, assume all characters are within the
> +       ;; BMP, i.e. take up 2 bytes.
> +       (setq byte (/ byte 2))
> +       (if (= eol 1)
> +           (filepos-to-bufferpos--dos (+ pm byte) #'byte-to-position)
> +         (byte-to-position (+ pm byte))))

Shouldn't this use `identity' rather than `byte-to-position'?


        Stefan

Re: [Emacs-diffs] master c4782ea: Improve and extend filepos-to-bufferpos

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Thu, 18 Jun 2015 23:17:28 -0400
> 
> > +            (if (<= byte eol-offset)
> > +                (setq pos (point-min))
> > +              (setq pos (point-max))))
> 
> Aka (setq pos (if (<= byte eol-offset) (point-min) (point-max)))

Yes, but my code is clearer, IMO.

> >    (let ((eol (coding-system-eol-type coding-system))
> >          (type (coding-system-type coding-system))
> > +        (base (coding-system-base coding-system))
> >          (pm (save-restriction (widen) (point-min))))
> > +    (and (eq type 'utf-8-emacs)
> > +         (setq type 'utf-8))
> 
> (coding-system-type 'utf-8-emacs) returns `utf-8', so how/when can
> `type' be `utf-8-emacs'?

Never.  I guess I got confused with coding-system-base.

> 
> > +    (and (eq type 'utf-8)
> > +         ;; Any post-read/pre-write conversions mean it's not really UTF-8.
> > +         (not (null (coding-system-get coding-system :pos-read-conversion)))
> > +         (setq type 'not-utf-8))
> 
> I guess this also applies for latin-N and utf-16, IOW for any value of
> `type', right?

Not really, no.  UTF-8 is special here, in that we believe we know how
to compute the byte position exactly, which is not true when there are
conversions.  Some profoundly non-UTF-8 encodings have that type, but
then apply conversions that make them something very different.

> > +    (and (not (eq type 'utf-8))
> > +         (eq quality 'exact)
> > +         (setq type 'use-exact))
> 
> IIUC this makes us use the slow exact code for latin-N.

Only if they ask for 'exact'.

> Why is it needed?

They asked for it, didn't they?

A more important problem is that we handle type before accuracy, so
when exact is requested, the type checks should be bypassed, except
with UTF-8.

> > +      (`utf-16
> > +       ;; Account for BOM, which is always 2 bytes in UTF-16.
> > +       (setq byte (- byte 2))
> 
> Should that only be done for utf1-16B-with-signature?

Do we have a UTF-16 encoding without a signature?

> > +       ;; In approximate mode, assume all characters are within the
> > +       ;; BMP, i.e. take up 2 bytes.
> > +       (setq byte (/ byte 2))
> > +       (if (= eol 1)
> > +           (filepos-to-bufferpos--dos (+ pm byte) #'byte-to-position)
> > +         (byte-to-position (+ pm byte))))
> 
> Shouldn't this use `identity' rather than `byte-to-position'?

This code tested OK for me, feel free to change if you have a test
that fails.

Re: [Emacs-diffs] master c4782ea: Improve and extend filepos-to-bufferpos

[Eli Zaretskii]

I've fixed in commit a2bb6c7 a few of the issues you pointed out:

> Date: Fri, 19 Jun 2015 09:59:38 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > >    (let ((eol (coding-system-eol-type coding-system))
> > >          (type (coding-system-type coding-system))
> > > +        (base (coding-system-base coding-system))
> > >          (pm (save-restriction (widen) (point-min))))
> > > +    (and (eq type 'utf-8-emacs)
> > > +         (setq type 'utf-8))
> > 
> > (coding-system-type 'utf-8-emacs) returns `utf-8', so how/when can
> > `type' be `utf-8-emacs'?
> 
> Never.  I guess I got confused with coding-system-base.

This one.

> > > +    (and (not (eq type 'utf-8))
> > > +         (eq quality 'exact)
> > > +         (setq type 'use-exact))
> > 
> > IIUC this makes us use the slow exact code for latin-N.
> 
> Only if they ask for 'exact'.
> 
> > Why is it needed?

And this one.

> > > +      (`utf-16
> > > +       ;; Account for BOM, which is always 2 bytes in UTF-16.
> > > +       (setq byte (- byte 2))
> > 
> > Should that only be done for utf1-16B-with-signature?
> 
> Do we have a UTF-16 encoding without a signature?

And this one.  (Yes, we do have such systems, I just missed them when
I reviewed all the definitions.)

> > > +       ;; In approximate mode, assume all characters are within the
> > > +       ;; BMP, i.e. take up 2 bytes.
> > > +       (setq byte (/ byte 2))
> > > +       (if (= eol 1)
> > > +           (filepos-to-bufferpos--dos (+ pm byte) #'byte-to-position)
> > > +         (byte-to-position (+ pm byte))))
> > 
> > Shouldn't this use `identity' rather than `byte-to-position'?
> 
> This code tested OK for me, feel free to change if you have a test
> that fails.

And this one; I believe you are right here, and my testing was
probably limited to ASCII-only files.

Thanks.