bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Drew Adams]

Some particular coding systems are mentioned in the Emacs manual,
presumably because they are important or instructive.  Please
consider adding these terms to the manual's index.
 
`no-conversion', `raw-text', `emacs-internal',
`iso-latin-1-unix', `iso-latin-1-dos', `iso-latin-1-mac',
`undecided-unix', `undecided-dos', `undecided-mac'.

In GNU Emacs 24.0.50.1 (i386-mingw-nt5.1.2600)
 of 2011-06-13 on 3249CTO
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (4.5) --no-opt --cflags
-Ic:/build/include'
 

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Lars Magne Ingebrigtsen]

"Drew Adams" <drew.adams@oracle.com> writes:

> Some particular coding systems are mentioned in the Emacs manual,
> presumably because they are important or instructive.  Please
> consider adding these terms to the manual's index.
>
> `no-conversion', `raw-text', `emacs-internal',
> `iso-latin-1-unix', `iso-latin-1-dos', `iso-latin-1-mac',
> `undecided-unix', `undecided-dos', `undecided-mac'.

Some of these were already in the index, but I've added the remaining
ones now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Drew Adams]

> > Some particular coding systems are mentioned in the Emacs manual,
> > presumably because they are important or instructive.  Please
> > consider adding these terms to the manual's index.
> >
> > `no-conversion', `raw-text', `emacs-internal',
> > `iso-latin-1-unix', `iso-latin-1-dos', `iso-latin-1-mac',
> > `undecided-unix', `undecided-dos', `undecided-mac'.
> 
> Some of these were already in the index, 

I don't see any of them.  `i' followed by a prefix of any of those names shows
[No match], in the latest Windows build at least:

In GNU Emacs 24.0.50.1 (i386-mingw-nt5.1.2600)
 of 2011-06-27 on 3249CTO
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (4.5) --no-opt --cflags
-Ic:/build/include'

And searching each of the indexes for each of those terms, I find none of them.

Maybe you were thinking of the `undecided, coding system' entry in the Concept
Index?  That's close, but it is not any of those I mentioned.

> but I've added the remaining ones now.

Thanks.

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Lars Magne Ingebrigtsen]

"Drew Adams" <drew.adams@oracle.com> writes:

>> > `no-conversion', `raw-text', `emacs-internal',
>> > `iso-latin-1-unix', `iso-latin-1-dos', `iso-latin-1-mac',
>> > `undecided-unix', `undecided-dos', `undecided-mac'.
>> 
>> Some of these were already in the index, 
>
> I don't see any of them.  `i' followed by a prefix of any of those
> names shows [No match], in the latest Windows build at least:

[...]

> Maybe you were thinking of the `undecided, coding system' entry in the
> Concept Index?  That's close, but it is not any of those I mentioned.

Yes, that's the one I meant.  And instead of adding all the
`iso-latin-1-*' variations, I just added "`iso-latin-1', coding system".

I think that should be sufficient for people who want to find these
things in the index.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Drew Adams]

> > Maybe you were thinking of the `undecided, coding system' 
> > entry in the Concept Index?  That's close, but it is not
> > any of those I mentioned.
> 
> Yes, that's the one I meant.  And instead of adding all the
> `iso-latin-1-*' variations, I just added "`iso-latin-1', 
> coding system".
> 
> I think that should be sufficient for people who want to find these
> things in the index.

I disagree.  These are real, implementation, user-visible, runtime names.  This
is just like indexing command names or variable names or package names.  The
exact name should appear in the index.

Consider, for instance, the use case that brought this to my attention:

I have updated various Emacs `describe-...' commands so that a link to their
coverage in the manuals is added to the *Help* buffer.  This is based on finding
the term in question in the manual's index.  The match naturally should be an
exact match against index entries.

See http://lists.gnu.org/archive/html/emacs-devel/2011-06/msg00368.html.

With a proper fix to this bug as I requested, these particular, literal names
would appear in the index, and I could then add `describe-coding-system' to my
updates.  A user looking at the output of `describe-coding-system' for one of
these coding systems could click the link to get to its doc in the manual.

This is what I have in my TODO list:

;; ADD THIS ONE to help-fns+.el ONLY IF BUG #8900 is fixed.

;; REPLACE ORIGINAL in `mule-diag.el'
;;
;; Call `Info-make-manuals-xref' to create a cross-ref link to manuals.
;;
;;;###autoload
(when (> emacs-major-version 23); Before Emacs 24 no coding systems are indexed.
  (defun describe-coding-system (coding-system)
    "Display information about CODING-SYSTEM."
    (interactive "zDescribe coding system (default current choices): ")
    (require 'mule-diag)
    (if (null coding-system)
        (describe-current-coding-system)
      (help-setup-xref (list #'describe-coding-system coding-system)
                       (called-interactively-p 'interactive))
      (with-output-to-temp-buffer (help-buffer)
        (print-coding-system-briefly coding-system 'doc-string)
        (let ((type (coding-system-type coding-system))
              ;; Fixme: use this
              (extra-spec (coding-system-plist coding-system)))
          (princ "Type: ")
          (princ type)
          (cond ((eq type 'undecided)
                 (princ " (do automatic conversion)"))
                ((eq type 'utf-8)
                 (princ " (UTF-8: Emacs internal multibyte form)"))
                ((eq type 'utf-16)
                 ;; (princ " (UTF-16)")
                 )
                ((eq type 'shift-jis)
                 (princ " (Shift-JIS, MS-KANJI)"))
                ((eq type 'iso-2022)
                 (princ " (variant of ISO-2022)\n")
                 (princ "Initial designations:\n")
                 (print-designation (coding-system-get coding-system
                                                       :designation))

                 (when (coding-system-get coding-system :flags)
                   (princ "Other specifications: \n  ")
                   (apply #'print-list
                          (coding-system-get coding-system :flags))))
                ((eq type 'charset)
                 (princ " (charset)"))
                ((eq type 'ccl)
                 (princ " (do conversion by CCL program)"))
                ((eq type 'raw-text)
                 (princ " (text with random binary characters)"))
                ((eq type 'emacs-mule)
                 (princ " (Emacs 21 internal encoding)"))
                ((eq type 'big5))
                (t (princ ": invalid coding-system.")))
          (princ "\nEOL type: ")
          (let ((eol-type (coding-system-eol-type coding-system)))
            (cond ((vectorp eol-type)
                   (princ "Automatic selection from:\n\t")
                   (princ eol-type)
                   (princ "\n"))
                  ((or (null eol-type) (eq eol-type 0)) (princ "LF\n"))
                  ((eq eol-type 1) (princ "CRLF\n"))
                  ((eq eol-type 2) (princ "CR\n"))
                  (t (princ "invalid\n")))))
        (let ((postread (coding-system-get coding-system
:post-read-conversion)))
          (when postread
            (princ "After decoding text normally,")
            (princ " perform post-conversion using the function: ")
            (princ "\n  ")
            (princ postread)
            (princ "\n")))
        (let ((prewrite (coding-system-get coding-system
:pre-write-conversion)))
          (when prewrite
            (princ "Before encoding text normally,")
            (princ " perform pre-conversion using the function: ")
            (princ "\n  ")
            (princ prewrite)
            (princ "\n")))
        (with-current-buffer standard-output
          (let ((charsets (coding-system-charset-list coding-system)))
            (when (and (not (eq (coding-system-base coding-system) 'raw-text))
                       charsets)
              (cond
                ((eq charsets 'iso-2022)
                 (insert "This coding system can encode all ISO 2022
charsets."))
                ((eq charsets 'emacs-mule)
                 (insert "This coding system can encode all emacs-mule
charsets."""))
                (t
                 (insert "This coding system encodes the following charsets:\n
")
                 (while charsets
                   (insert " " (symbol-name (car charsets)))
                   (search-backward (symbol-name (car charsets)))
                   (help-xref-button 0 'help-character-set (car charsets))
                   (goto-char (point-max))
                   (setq charsets (cdr charsets)))))))
          (when (boundp 'Info-virtual-files) ; Emacs 23.2+
            (Info-make-manuals-xref coding-system)))))))

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 01 Jul 2011 16:28:28 +0200
> Cc: 8900@debbugs.gnu.org
> 
> "Drew Adams" <drew.adams@oracle.com> writes:
> 
> > Some particular coding systems are mentioned in the Emacs manual,
> > presumably because they are important or instructive.  Please
> > consider adding these terms to the manual's index.
> >
> > `no-conversion', `raw-text', `emacs-internal',
> > `iso-latin-1-unix', `iso-latin-1-dos', `iso-latin-1-mac',
> > `undecided-unix', `undecided-dos', `undecided-mac'.
> 
> Some of these were already in the index, but I've added the remaining
> ones now.

FWIW, I think you put the additional index entries in wrong places.
The index entry should be near the text that describes the concept or
the symbol, not merely mentions it.

For example, this:

  +@cindex @code{iso-latin-1}, coding system
     These variant coding systems are omitted from the
   @code{list-coding-systems} display for brevity, since they are entirely
   predictable.  For example, the coding system @code{iso-latin-1} has

is wrong, because the text does not describe the iso-latin-1 encoding
in any way.

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Fri, 1 Jul 2011 08:54:44 -0700
> Cc: 8900@debbugs.gnu.org
> 
> > I think that should be sufficient for people who want to find these
> > things in the index.
> 
> I disagree.  These are real, implementation, user-visible, runtime names.  This
> is just like indexing command names or variable names or package names.  The
> exact name should appear in the index.

Not true, at least not until we have a detailed documentation of each
encoding there.  The absolute majority of coding-systems is not
documented in the manual, so there's no real place to put the index
entries.  I'm not sure there's something intelligent to tell about
these encodings in the manual, either.

> Consider, for instance, the use case that brought this to my attention:

You cannot assume that every symbol appears in the manual.  So this
feature can never work reliably, only ad-hoc.  You should be prepared
for the situation where the manual doesn't have this in its index, and
handle that gracefully.

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Drew Adams]

> > I disagree.  These are real, implementation, user-visible, 
> > runtime names.  This is just like indexing command names
> > or variable names or package names.  The exact name should
> > appear in the index.
> 
> Not true, at least not until we have a detailed documentation of each
> encoding there.  The absolute majority of coding-systems is not
> documented in the manual, so there's no real place to put the index
> entries.  I'm not sure there's something intelligent to tell about
> these encodings in the manual, either.

Doesn't matter that we don't have detailed doc about these.  Certainly doesn't
matter that we don't have detailed doc about *each* coding system.

The fact is that the manual provides some information about these particular
coding systems.  They should be indexed so users can easily find that
information.

An index does not refer only to "detailed documentation" about terms.  It refers
to terms that we think a user is likely to look for in the book.  It is often
the case that a term is indexed that is only mentioned in the book.

What is important is whether a user might want to look it up, not how much the
book goes into detail about it.  The book might even state in some context "this
book does not cover XYZ", and `XYZ' might still be appropriate as an index entry
- precisely to get you to that information, however negative and incomplete.

> > Consider, for instance, the use case that brought this to 
> > my attention:
> 
> You cannot assume that every symbol appears in the manual.

What makes you think I make such an assumption?  Far from it.

You have knee-jerk repeated that several times over the past - it seems to be
your mantra whenever indexing comes up.  Just a straw man - no one here is
assuming any such thing.

Certainly it is _not_ the case that every symbol _should_ appear in the manual.
Far from it.  The vast majority of symbols should _not_.

In any case, this is not about whether some given term should appear in the
manual.  This is about whether some particular terms that _do_ appear in the
manual should be indexed.

> So this feature can never work reliably, only ad-hoc.

Wrong.  "This feature" is to provide a link in *Help* (in this case, for
`describe-coding-system XYZ') to search the manuals for a corresponding index
entry.  That works 100% reliably.

"This feature" also includes an option for those who prefer that a link to
search the manuals not be added unless there are in fact index entries for the
subject term in the manuals to be searched.  And that option also lets users
choose which manuals are to be searched.

But regardless of the option value, clicking the `manuals' link will always
correctly search the specified manuals and produce a virtual index if the
subject term is found.  100% reliable.

> You should be prepared for the situation where the manual
> doesn't have this in its index, and handle that gracefully.

See above.  And see the emacs-devel thread where the feature was described.  Or
see the source code that implements it - either the submitted patch or the cited
library, help-fns+.el.

And independently of this feature, a user can _already_ create a virtual index
(which is what this feature creates automatically when you click the `manuals'
link).  Doing that should find a particular coding system that is discussed in
the manual.

If the manual had nothing to say about these coding systems then they would not
be there and would not be indexed.

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Eli Zaretskii]

> Date: Fri, 01 Jul 2011 20:45:08 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 8900@debbugs.gnu.org
> 
> FWIW, I think you put the additional index entries in wrong places.
> The index entry should be near the text that describes the concept or
> the symbol, not merely mentions it.
> 
> For example, this:
> 
>   +@cindex @code{iso-latin-1}, coding system
>      These variant coding systems are omitted from the
>    @code{list-coding-systems} display for brevity, since they are entirely
>    predictable.  For example, the coding system @code{iso-latin-1} has
> 
> is wrong, because the text does not describe the iso-latin-1 encoding
> in any way.

I fixed that now.

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <larsi@gnus.org>, <8900@debbugs.gnu.org>
> Date: Fri, 1 Jul 2011 11:17:23 -0700
> 
> > > I disagree.  These are real, implementation, user-visible, 
> > > runtime names.  This is just like indexing command names
> > > or variable names or package names.  The exact name should
> > > appear in the index.
> > 
> > Not true, at least not until we have a detailed documentation of each
> > encoding there.  The absolute majority of coding-systems is not
> > documented in the manual, so there's no real place to put the index
> > entries.  I'm not sure there's something intelligent to tell about
> > these encodings in the manual, either.
> 
> Doesn't matter that we don't have detailed doc about these.  Certainly doesn't
> matter that we don't have detailed doc about *each* coding system.

Well, in that case, we will have to disagree.  Having index entries
about something that isn't described is a Bad Thing.

> The fact is that the manual provides some information about these particular
> coding systems.  They should be indexed so users can easily find that
> information.

But there's no information to find.

> An index does not refer only to "detailed documentation" about terms.  It refers
> to terms that we think a user is likely to look for in the book.  It is often
> the case that a term is indexed that is only mentioned in the book.

Not in my book.

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Drew Adams]

> > Doesn't matter that we don't have detailed doc about these. 
> > Certainly doesn't matter that we don't have detailed doc
> > about *each* coding system.
> 
> Well, in that case, we will have to disagree.

You need detailed doc about *each* coding system before you will index *any* of
them?

With that logic we should remove all entries from our `Variable', `Command', and
`Key' indexes, since we certainly do not provide documentation - let alone
"detailed documentation" - about all of them.

> Having index entries about something that isn't described
> is a Bad Thing.

"Described" is vague here, so it's hard to judge.  But you clarified earlier
that only "detailed documentation" counts for you.

We have similar index entries in the manuals.  Looking only at the first few
entries of the `Variable' index, check `blink-cursor-alist' and `auto-coding-*',
for instance.  There is no "detailed documentation" describing these variables.
There is only a general, cursory mention of what they are for.  And these are
not outliers/alone in this respect.

That is not a bug.  We still want to index these variables, IMO.  Not because we
necessarily provide their "detailed documentation" in this manual (we do _not_),
but because:

a. we say something about them that could be useful to a user (otherwise we
wouldn't mention them!), and

b. we think a user might want to look them up.

Those, not simply the degree of detail provided, are proper, user-oriented
criteria for indexing.

(Of course, if such a variable also had a more complete, detailed documentation
elsewhere in the manual, then that location would likely be indexed - either in
addition to or instead of the current location, depending on the context and the
specific content.)

I think those two criteria apply to the coding-system entries I submitted this
bug about.  You can disagree about that.  But I would hope that you would agree
about such criteria, whether or not you agree that they are satisfied in these
particular cases.

> > The fact is that the manual provides some information about 
> > these particular coding systems.  They should be indexed so
> > users can easily find that information.
> 
> But there's no information to find.

Yes there is.  About the same degree of information as for our "descriptions" of
variables `blink-cursor-alist' and `auto-coding-alist' - more, in fact.  We say
what the undecided-* coding systems do wrt end-of-line conversion, and we
mention their aliases.

> > An index does not refer only to "detailed documentation" 
> > about terms.  It refers to terms that we think a user is
> > likely to look for in the book.  It is often the case that
> > a term is indexed that is only mentioned in the book.
> > What is important is whether a user might want to look it
> > up, not how much the book goes into detail about it. 
> 
> Not in my book.

Whatever.  I do indexing for a living.  But it's `your book'.

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <larsi@gnus.org>, <8900@debbugs.gnu.org>
> Date: Fri, 1 Jul 2011 14:00:01 -0700
> 
> You need detailed doc about *each* coding system before you will index *any* of
> them?

I never said that.  Some of them _are_ indexed: raw-text,
no-conversion, and emacs-internal.  That's because the manual says
something specific about them, unlike the others.

> > Having index entries about something that isn't described
> > is a Bad Thing.
> 
> "Described" is vague here, so it's hard to judge.

Not really.  The following is _not_ a description:

	  For example, the coding system @code{iso-latin-1} has
  variants @code{iso-latin-1-unix}, @code{iso-latin-1-dos} and
  @code{iso-latin-1-mac}.

This mentions iso-latin-1-*, but says nothing at all about them.  The
whole sentence is an explanation of the -dos, -unix, -mac variants of
each coding-system, and uses iso-latin-1-* as an example.  And there's
no other reference to iso-latin-1-* anywhere in the manual.

How hard is it to judge that these symbols are not documented in the
manual?  I say it's obvious.  A reader who wants to know something
about iso-latin-1 will not be wiser after reading this.

> I do indexing for a living.

Then I pity your readers.

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> is wrong, because the text does not describe the iso-latin-1 encoding
>> in any way.
>
> I fixed that now.

Great; thanks.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Drew Adams]

> > You need detailed doc about *each* coding system before you 
> > will index *any* of them?
> 
> I never said that.

Sure seems that way to me.  You said:

E> not until we have a detailed documentation of each
E> encoding there.

But if you've relaxed about that, fine.

> Some of them _are_ indexed: raw-text, no-conversion, and
> emacs-internal.  That's because the manual says
> something specific about them, unlike the others.

No.  _None_ of those are indexed in the build I reported on.  Perhaps you are
seeing them now because Lars added them in response to this bug report?  As I
said earlier, and to which Lars agreed:

> L> Some of these were already in the index, 
> 
D> I don't see any of them.  `i' followed by a prefix of any of 
D> those names shows [No match]...  And searching each of the
D> indexes for each of those terms, I find none of them.

> The following is _not_ a description:
>   For example, the coding system @code{iso-latin-1} has
>   variants @code{iso-latin-1-unix}...

I agree about that.  The same cannot be said however for the case I cited:

D> We say what the undecided-* coding systems do wrt end-of-line
D> conversion, and we mention their aliases.

bug#8900: 24.0.50; please index mentioned coding systems in Emacs manual

[Eli Zaretskii]

> Cc: <larsi@gnus.org>, <8900@debbugs.gnu.org>
> Date: Sat, 2 Jul 2011 07:37:07 -0700
> 
> > > You need detailed doc about *each* coding system before you 
> > > will index *any* of them?
> > 
> > I never said that.
> 
> Sure seems that way to me.  You said:
> 
> E> not until we have a detailed documentation of each
> E> encoding there.

There's no "before I will index *any* of them" here, is there?

> > Some of them _are_ indexed: raw-text, no-conversion, and
> > emacs-internal.  That's because the manual says
> > something specific about them, unlike the others.
> 
> No.  _None_ of those are indexed in the build I reported on.

  http://bzr.savannah.gnu.org/lh/emacs/trunk/annotate/head:/doc/emacs/mule.texi

> Perhaps you are seeing them now because Lars added them in response
> to this bug report?

"Are" does mean "now".  Yes, Lars added index entries, and I moved
them to their correct places.

> I agree about that.  The same cannot be said however for the case I cited:
> 
> D> We say what the undecided-* coding systems do wrt end-of-line
> D> conversion, and we mention their aliases.

Your bug report was about these:

  `no-conversion', `raw-text', `emacs-internal',
  `iso-latin-1-unix', `iso-latin-1-dos', `iso-latin-1-mac',
  `undecided-unix', `undecided-dos', `undecided-mac'

Of these, the iso-latin-1-* are not described in the manual, and thus
have no index entries.  All the others are indexed in the repository.

You are welcome.