recognizing a file by scanning it

recognizing a file by scanning it

[Thien-Thi Nguyen]

Some time back (within the last half-year or so) there was discussion
about Emacs being able to recognize file types by scanning content
rather than (or in addition to) using name-based heuristics.

One model for such a capability is the external command file(1), which
takes as its data a magic(5) file containing (possibly-chained) rules
specifying where and what to look for in the target file in order to
make a match, and additionally what to display on match.

For example, here is a fragment of ~/.magic ("|"-prefixed):

|# Emacs 18 - this is always correct, but not very magical.
|0      string  \012(         Emacs v18 byte-compiled Lisp data
|# Emacs 19+ - ver. recognition added by Ian Springer
|# Also applies to XEmacs 19+ .elc files; could tell them apart if we had regexp
|# support or similar - Chris Chittleborough <cchittleborough@yahoo.com.au>
|0      string  ;ELC
|>4     byte    >19
|>4     byte    <32           Emacs/XEmacs v%d byte-compiled Lisp data

I have written a Scheme program to translate this into sexps amenable
to both Scheme and Emacs Lisp `read'.  To continue the example:

|(0 0 string (= . "\n(") "Emacs v18 byte-compiled Lisp data")
|(0 0 string (= . ";ELC") "")
|(1 4 byte (> 19) "")
|(1 4 byte (< 32) "Emacs/XEmacs v%d byte-compiled Lisp data")

(See <http://www.gnuvola.org/data/> for the complete translation.)

The Scheme program also mimics basic file(1) functionality; it can
recognize an unknown bag of bytes using the rules in either the original
magic(5) format or the translated-to-sexps variant, displaying output
indistinguishable (for the most part) from that of "file -n -N".

|$ ls="src/temacs etc/images/info.pbm lisp/startup.el lisp/startup.elc"
|$ for f in $ls ; do file -n -N $f ; ttn-do magic $f ; done
|src/temacs: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.4.1, dynamically linked (uses shared libs), for GNU/Linux 2.4.1, not stripped
|src/temacs: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV)
|etc/images/info.pbm: Netpbm PBM "rawbits" image data
|etc/images/info.pbm: Netpbm PBM "rawbits" image data
|lisp/startup.el: Lisp/Scheme program text
|lisp/startup.el: Lisp/Scheme program text
|lisp/startup.elc: Emacs/XEmacs v23 byte-compiled Lisp data
|lisp/startup.elc: Emacs/XEmacs v23 byte-compiled Lisp data

Although it lacks advanced file(1) functionality (integrated ELF
grokking, charset guesstimation, fancy printf(3) output, etc), i
consider it complete enough to be a good starting point for a port to
Emacs Lisp.  (Indeed, Emacs is much nicer for implementing such features
as charset guesstimation.)

But before continuing, i would like to discover if anyone else is
working on something similar, to avoid (more?) duplicate effort.

thi

recognizing a file by scanning it

[Stephen J. Turnbull]

Thien-Thi Nguyen writes:

 > But before continuing, i would like to discover if anyone else is
 > working on something similar, to avoid (more?) duplicate effort.

file(1) contains a rather complete set of such heuristics and has been
stable for a long time.  AFAIK libmagic is free and GPL-compatible.

Also, it might be nice if you could call this facility from coding
systems (after all, what else is a BOM but file magic?)  This kind of
thing really doesn't need to be in Lisp, and might benefit from being
in C.

Re: recognizing a file by scanning it

[Thien-Thi Nguyen]

() "Stephen J. Turnbull" <stephen@xemacs.org>
() Mon, 28 Apr 2008 06:06:42 +0900

   file(1) contains a rather complete set of such heuristics and
   has been stable for a long time.  AFAIK libmagic is free and
   GPL-compatible.

Yes, i believe so.  My concern w/ using libmagic would be
portability, and handling the case where libmagic is not
available.  I envision the feature to be non-optional.

   Also, it might be nice if you could call this facility from
   coding systems (after all, what else is a BOM but file magic?)
   This kind of thing really doesn't need to be in Lisp, and might
   benefit from being in C.

Certainly, in C it can be much faster.  

thi

Re: recognizing a file by scanning it

[Stefan Monnier]

>    file(1) contains a rather complete set of such heuristics and
>    has been stable for a long time.  AFAIK libmagic is free and
>    GPL-compatible.

> Yes, i believe so.  My concern w/ using libmagic would be
> portability, and handling the case where libmagic is not
> available.  I envision the feature to be non-optional.

Would could like we do for `ls': use libmagic if it's available and
fallback on an Elisp replacement if not.  The Elisp replacement would
need to work even in the absence of /etc/magic.

The higher-level functionality also needs adjusting: we want to bring
magic-mode-alist and auto-mode-alist to the same level (rather than give
absolute priority to one of the two).  Basically, try them both, each
one can return a set/list of potential modes, then try and figure out
which mode to use (I've done something similar in doc-view-mode).
Hopefully this can be used to disambiguate the .arc archives from the
.arc Lisp files as well.

>    Also, it might be nice if you could call this facility from
>    coding systems (after all, what else is a BOM but file magic?)
>    This kind of thing really doesn't need to be in Lisp, and might
>    benefit from being in C.

> Certainly, in C it can be much faster.  

If it's used just to choose the major-mode, speed is a non-issue.


        Stefan

Re: recognizing a file by scanning it

[Jason Rumney]

Thien-Thi Nguyen wrote:
> But before continuing, i would like to discover if anyone else is
> working on something similar, to avoid (more?) duplicate effort.
>   

magic-mode-alist was added early in the development of Emacs 22. All 
entries were removed from it shortly before Emacs 22.1 was released due 
to security issues that were raised during pretest.

Re: recognizing a file by scanning it

[Chong Yidong]

Jason Rumney <jasonr@gnu.org> writes:

> Thien-Thi Nguyen wrote:
>> But before continuing, i would like to discover if anyone else is
>> working on something similar, to avoid (more?) duplicate effort.
>>   
>
> magic-mode-alist was added early in the development of Emacs 22. All
> entries were removed from it shortly before Emacs 22.1 was released
> due to security issues that were raised during pretest.

I don't recall that there was any security issues.  Rather, these
entries were moved to magic-fallback-mode-alist, so that they have a
lower priority than filename autodetection.

Re: recognizing a file by scanning it

[Jason Rumney]

Chong Yidong wrote:
> Jason Rumney <jasonr@gnu.org> writes:
>
>   
>> Thien-Thi Nguyen wrote:
>>     
>>> But before continuing, i would like to discover if anyone else is
>>> working on something similar, to avoid (more?) duplicate effort.
>>>   
>>>       
>> magic-mode-alist was added early in the development of Emacs 22. All
>> entries were removed from it shortly before Emacs 22.1 was released
>> due to security issues that were raised during pretest.
>>     
>
> I don't recall that there was any security issues.  Rather, these
> entries were moved to magic-fallback-mode-alist, so that they have a
> lower priority than filename autodetection.
>   

The image entries were all dropped completely because of the supposed 
security issues.

Re: recognizing a file by scanning it

[Thien-Thi Nguyen]

() Jason Rumney <jasonr@gnu.org>
() Mon, 28 Apr 2008 00:28:03 +0100

   The image entries were all dropped completely because of the
   supposed security issues.

Searching gmane, i see a huge list of postings w/ the subject
"Image mode".  Is that what you are referring to?

A brief read (handful of pages) gives me the impression that the
issue there is the security of image libraries (libpng, etc).  Am
i missing something?

thi

Re: recognizing a file by scanning it

[Jason Rumney]

Thien-Thi Nguyen wrote:
> () Jason Rumney <jasonr@gnu.org>
> () Mon, 28 Apr 2008 00:28:03 +0100
>
>    The image entries were all dropped completely because of the
>    supposed security issues.
>
> Searching gmane, i see a huge list of postings w/ the subject
> "Image mode".  Is that what you are referring to?
>
> A brief read (handful of pages) gives me the impression that the
> issue there is the security of image libraries (libpng, etc).  Am
> i missing something?
>   

Yes, that was the issue. Rereading that thread and others on the 
subject, it seems that RMS's solution was different than I remembered - 
he made image-mode prompt the user by default in all cases, and it was 
later restored when Lars Magnusson convinced him that no other 
application was this paranoid.

Re: recognizing a file by scanning it

[Reiner Steib]

On Mon, Apr 28 2008, Jason Rumney wrote:

> he made image-mode prompt the user by default in all cases, and it
> was later restored when Lars Magnusson convinced him that no other
> application was this paranoid.

s/Lars Magnusson/Lars Magne Ingebrigtsen/ ;-)

It's not only paranoid, but a security risk itself...

,----[ <http://article.gmane.org/gmane.emacs.devel/66004> ]
| Warning users about something that's almost certainly not dangerous is
| a huge security risk in itself, because you're inuring the users to
| warnings.  The user will answer "Yeah, whatever" when being bothered
| with these things, and then when Emacs asks the user "Are you sure you
| wish to do an rm -rf?" (or whatever the genuinely dangerous thing it
| is), they won't bother to read the warning. 
`----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: recognizing a file by scanning it

[Jason Rumney]

Reiner Steib wrote:
> s/Lars Magnusson/Lars Magne Ingebrigtsen/ ;-)
>   

Sorry Lars.