bug#8417: hexlify-buffer

bug#8417: hexlify-buffer

[Don Harter]

From: Don Harter <drh@>
To: bug-gnu-emacs@gnu.org
Subject: 23.1; hexlify-buffer
--text follows this line--

Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs@gnu.org mailing list,
and to the gnu.emacs.bug news group.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

I had loaded a file reverse_geocode.sql
I type esc then x and issued the command hexlify-buffer after using tab to
find the proper completion for it.
The result tha I got was that the file disappeared and I could not undo
anything. Here is the result:
"-hex: No such file or directory"

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
`bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/usr/share/emacs/23.1/etc/DEBUG for instructions.


In GNU Emacs 23.1.1 (x86_64-suse-linux-gnu, GTK+ Version 2.20.1)
of 2010-07-05 on build35
Windowing system distributor `The X.Org Foundation', version 11.0.10800000
configured using `configure '--with-pop' '--without-hesiod' 
'--with-kerberos' '--with-kerberos5' '--with-xim' '--prefix=/usr' 
'--mandir=/usr/share/man' '--infodir=/usr/share/info' 
'--datadir=/usr/share' '--localstatedir=/var' 
'--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--with-x' 
'--with-sound' '--with-sync-input' '--with-xpm' '--with-jpeg' 
'--with-tiff' '--with-gif' '--with-png' '--with-rsvg' '--with-dbus' 
'--without-gpm' '--with-x-toolkit=gtk' '--x-includes=/usr/include' 
'--x-libraries=/usr/lib64:/usr/share/X11' '--with-xft' '--with-libotf' 
'--with-m17n-flt' '--build=x86_64-suse-linux' 
'build_alias=x86_64-suse-linux' 'CC=gcc-4.3' 'CFLAGS=-fmessage-length=0 
-O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables 
-fasynchronous-unwind-tables -g -D_GNU_SOURCE -std=gnu89 -pipe 
-Wno-pointer-sign -Wno-unused-variable -Wno-unused-label 
-Wno-unprototyped-calls -fno-optimize-sibling-calls 
-DSYSTEM_PURESIZE_EXTRA=55000 -DSITELOAD_PURESIZE_EXTRA=10000 ' 
'LDFLAGS=-Wl,-O2 -Wl,--hash-size=65521''

Important settings:
value of $LC_ALL: nil
value of $LC_COLLATE: nil
value of $LC_CTYPE: nil
value of $LC_MESSAGES: nil
value of $LC_MONETARY: nil
value of $LC_NUMERIC: nil
value of $LC_TIME: nil
value of $LANG: en_US.UTF-8
value of $XMODIFIERS: @im=local
locale-coding-system: utf-8-unix
default-enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
show-paren-mode: t
tooltip-mode: t
tool-bar-mode: t
mouse-wheel-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
blink-cursor-mode: t
global-auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
column-number-mode: t
line-number-mode: t
transient-mark-mode: t

Recent input:
C-x 1 <escape> x h e x <tab> <tab> i f y - b u f f
e r <return> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<menu-bar> <help-menu> <send-emacs-bug-report> h e
x l i f y - b u f f e r <return> <help-echo> <help-echo>
<help-echo> <menu-bar> <buffer> C-@ <help-echo> <help-echo>
<down-mouse-1> <mouse-movement> <mouse-movement> <mouse-movement>
<mouse-movement> <mouse-movement> <mouse-movement>
<mouse-movement> <mouse-movement> <mouse-movement>
<mouse-movement> <drag-mouse-1> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <menu-bar>
<edit> <copy> <help-echo> <help-echo> <help-echo> <menu-bar>
<buffer> C-c <help-echo> <help-echo> <menu-bar> <buffer>
C-b <help-echo> <menu-bar> <buffer> C-c <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <menu-bar> <buffer> C-@ <help-echo> <menu-bar>
<buffer> C-a <help-echo> <menu-bar> <buffer> C-b <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<menu-bar> <help-menu> <send-emacs-bug-report>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
reverse_geocode.sql has auto save data; consider M-x recover-this-file
uim.el: starting uim-el-helper-agent... done
uim.el: starting uim-el-agent... done
Invalid image size (see `max-image-size') [6 times]
Making completion list...
Invalid image size (see `max-image-size')
Buffer reverse_geocode.sql has shrunk a lot; auto save disabled in that 
buffer until next real save
Auto-saving...
Invalid image size (see `max-image-size') [7 times]

bug#8417: hexlify-buffer

[Eli Zaretskii]

> Date: Sun, 03 Apr 2011 10:22:43 -0500
> From: Don Harter <donharter@comcast.net>
> Cc: 
> 
> I had loaded a file reverse_geocode.sql
> I type esc then x and issued the command hexlify-buffer after using tab to
> find the proper completion for it.
> The result tha I got was that the file disappeared and I could not undo
> anything. Here is the result:
> "-hex: No such file or directory"

I cannot reproduce this.  Do you have a program named hexl in some
directory that is mentioned in the exec-path variable?

bug#8417: hexlify-buffer

[Eli Zaretskii]

> Date: Sun, 03 Apr 2011 14:26:52 -0500
> From: Don Harter <donharter@comcast.net>
> 
> root:~>locate hexl
> /usr/lib/emacs/23.1/x86_64-suse-linux/hexl
> /usr/lib/xemacs/21.5.29.b604d235f028/x86_64-suse-linux/hexl

You should be using the first one, but from the error message, viz.

> >> "-hex: No such file or directory"

it seems like you are using the second, or anyway some `hexl' program
that does not understand the -hex switch that Emacs passes to it.
Please show the value of your Emacs's exec-path, with "M-: exec-path RET".

bug#8417: hexlify-buffer

[Eli Zaretskii]

> Date: Sun, 03 Apr 2011 14:33:32 -0500
> From: Don Harter <donharter@comcast.net>
> 
>     I am attaching some strace files.  Perhaps it is a permission
>     problem.

No, it looks like an encoding problem:

  stat("/usr/lib/emacs/23.1/x86_64-suse-linux/hexl", {st_mode=S_IFREG|0755, st_size=10680, ...}) = 0
  access("/usr/lib/emacs/23.1/x86_64-suse-linux/hexl", X_OK) = 0
  pipe([11, 12])                          = 0
  vfork()                                 = 2327
  close(12)                               = 0
  close(10)                               = 0
  read(11, "\357\273\277-hex: No such file or directo"..., 16384) = 35
            ^^^^^^^^^^^^^^^^

Does this happen in "emacs -Q"?

bug#8417: hexlify-buffer

[Eli Zaretskii]

> Date: Sun, 03 Apr 2011 19:23:19 -0500
> From: Don Harter <donharter@comcast.net>
> 
> The same thing happens when I do "emacs -Q ".

How about "LC_ALL=C emacs -Q", does it happen even then?

bug#8417: hexlify-buffer

[Eli Zaretskii]

[Please don't remove the bug tracker address from the CC list.]

> Date: Mon, 04 Apr 2011 02:16:31 -0500
> From: Don Harter <donharter@comcast.net>
> 
> Yes the same thing happens.
> 
> 
> On 04/03/2011 09:59 PM, Eli Zaretskii wrote:
> >> Date: Sun, 03 Apr 2011 19:23:19 -0500
> >> From: Don Harter<donharter@comcast.net>
> >>
> >> The same thing happens when I do "emacs -Q ".
> > How about "LC_ALL=C emacs -Q", does it happen even then?

Does this happen with other files as well, or just with this one?

bug#8417: hexlify-buffer

[Eli Zaretskii]

> Date: Mon, 04 Apr 2011 07:50:19 -0400
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 8417@debbugs.gnu.org
> 
> 
> [Please don't remove the bug tracker address from the CC list.]
> 
> > Date: Mon, 04 Apr 2011 02:16:31 -0500
> > From: Don Harter <donharter@comcast.net>
> > 
> > Yes the same thing happens.
> > 
> > 
> > On 04/03/2011 09:59 PM, Eli Zaretskii wrote:
> > >> Date: Sun, 03 Apr 2011 19:23:19 -0500
> > >> From: Don Harter<donharter@comcast.net>
> > >>
> > >> The same thing happens when I do "emacs -Q ".
> > > How about "LC_ALL=C emacs -Q", does it happen even then?
> 
> Does this happen with other files as well, or just with this one?

Forget it.  I can see the problem myself, and will debug it here.

bug#8417: hexlify-buffer

[Eli Zaretskii]

> Date: Mon, 04 Apr 2011 07:50:19 -0400
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 8417@debbugs.gnu.org
> 
> [Please don't remove the bug tracker address from the CC list.]
> 
> > Date: Mon, 04 Apr 2011 02:16:31 -0500
> > From: Don Harter <donharter@comcast.net>
> > 
> > Yes the same thing happens.
> > 
> > 
> > On 04/03/2011 09:59 PM, Eli Zaretskii wrote:
> > >> Date: Sun, 03 Apr 2011 19:23:19 -0500
> > >> From: Don Harter<donharter@comcast.net>
> > >>
> > >> The same thing happens when I do "emacs -Q ".
> > > How about "LC_ALL=C emacs -Q", does it happen even then?
> 
> Does this happen with other files as well, or just with this one?

Okay, I see the problem now.  It is triggered because this particular
file is encoded in UTF-8 with signature (BOM).  This triggers a bug in
Emacs 23.1, which is fixed in Emacs 23.2 and later (the latest release
is version 23.3).

So the solution for you would be to upgrade to a newer Emacs.  Failing
that, you can work around the problem by typing this in Emacs, before
you invoke hexlify-buffer:

  C-x RET f utf-8 RET C-x C-s

This will recode the file in UTF-8, but without the BOM, and then the
bug in hexl will not be triggered.

I'm closing this bug.  Thanks for all your patience in following up.

bug#8417: attached file

[Don Harter]

[-- Attachment #1: Type: text/html, Size: 334 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: reverse_geocode.sql --]
[-- Type: text/x-sql; name="reverse_geocode.sql", Size: 5274 bytes --]

--$Id: reverse_geocode.sql 6823 2011-02-15 13:49:06Z robe $
-- 
-- 
-- Copyright (C) 2011 Regina Obe and Leo Hsu (Paragon Corporation)
--
-- This function given a point try to determine the approximate street address (norm_addy form)
-- and array of cross streets, as well as interpolated points along the streets
--
ALTER DATABASE geocoder SET search_path TO tiger,public;
CREATE OR REPLACE FUNCTION public.reverse_geocode(
    IN pt geometry,
    IN include_strnum_range boolean,
    OUT intpt geometry[],
    OUT addy NORM_ADDY[],
    OUT street varchar[]
) RETURNS RECORD
AS $_$
DECLARE
  var_redge RECORD;
  var_states text[];
  var_addy NORM_ADDY;
  var_strnum varchar;
  var_nstrnum numeric(10);
  var_primary_line geometry := NULL;
  var_primary_dist numeric(10,2) ;
  var_pt geometry;
BEGIN
  IF pt IS NULL THEN
    RETURN;
  ELSE
    IF ST_SRID(pt) = 4269 THEN
        var_pt := pt;
    ELSE
        var_pt := ST_Transform(pt, 4269);
    END IF;
  END IF;
  -- Determine state tables to check 
  -- this is needed to take advantage of constraint exclusion
  var_states := ARRAY(SELECT statefp FROM state WHERE ST_Intersects(the_geom, var_pt) );
  IF array_upper(var_states, 1) IS NULL THEN
  -- We don't have any data for this state
  	RETURN;
  END IF;
  
  -- Find the street edges that this point is closest to with tolerance of 0.005 but only consider the edge if the point is contained in the right or left face
  -- Then order addresses by proximity to road
  FOR var_redge IN
        SELECT * 
        FROM (SELECT DISTINCT ON(fullname)  foo.fullname, foo.stusps, foo.zip, 
               (SELECT z.place FROM zip_state_loc AS z WHERE z.zip = foo.zip and z.statefp = foo.statefp LIMIT 1) As place, foo.center_pt,
              side, to_number(fromhn, '999999') As fromhn, to_number(tohn, '999999') As tohn, ST_GeometryN(ST_Multi(line),1) As line, foo.dist
        FROM 
          (SELECT e.the_geom As line, e.fullname, a.zip, s.stusps, ST_ClosestPoint(e.the_geom, var_pt) As center_pt, e.statefp, a.side, a.fromhn, a.tohn, ST_Distance_Sphere(e.the_geom, var_pt) As dist
          		FROM edges AS e INNER JOIN state As s ON (e.statefp = s.statefp AND s.statefp = ANY(var_states) )
          			INNER JOIN faces As fl ON (e.tfidl = fl.tfid AND e.statefp = fl.statefp)
          			INNER JOIN faces As fr ON (e.tfidr = fr.tfid AND e.statefp = fr.statefp)
          			INNER JOIN addr As a ON ( e.tlid = a.tlid AND e.statefp = a.statefp AND  
          			   ( ( ST_Covers(fl.the_geom, var_pt) AND a.side = 'L') OR ( ST_Covers(fr.the_geom, var_pt) AND a.side = 'R' ) ) )
          			-- INNER JOIN zip_state_loc As z ON (a.statefp =  z.statefp AND a.zip = z.zip) /** really slow with this join **/
          		WHERE e.statefp = ANY(var_states) AND a.statefp = ANY(var_states) AND ST_DWithin(e.the_geom, var_pt, 0.005)
          		ORDER BY ST_Distance_Sphere(e.the_geom, var_pt) LIMIT 4) As foo 
          		WHERE dist < 150 --less than 150 m
          		ORDER BY foo.fullname, foo.dist) As f ORDER BY f.dist LOOP
       IF var_primary_line IS NULL THEN --this is the first time in the loop and our primary guess
            var_primary_line := var_redge.line;
            var_primary_dist := var_redge.dist;
       END IF;
       -- We only consider other edges as matches if they intersect our primary edge -- that would mean we are at a corner place
       IF ST_Intersects(var_redge.line, var_primary_line) THEN
           intpt := array_append(intpt,var_redge.center_pt); 
           IF var_redge.fullname IS NOT NULL THEN
                street := array_append(street, (CASE WHEN include_strnum_range THEN COALESCE(var_redge.fromhn::varchar, '')::varchar || ' - ' || COALESCE(var_redge.tohn::varchar,'')::varchar || ' '::varchar  ELSE '' END::varchar ||  var_redge.fullname::varchar)::varchar);
                --interploate the number -- note that if fromhn > tohn we will be subtracting which is what we want
                -- We only consider differential distances are reeally close from our primary pt
                IF var_redge.dist < var_primary_dist*1.1 THEN 
                	var_nstrnum := (var_redge.fromhn + ST_Line_Locate_Point(var_redge.line, var_pt)*(var_redge.tohn - var_redge.fromhn))::numeric(10);
                	-- The odd even street number side of street rule
                	IF (var_nstrnum  % 2)  != (var_redge.tohn % 2) THEN
                		var_nstrnum := CASE WHEN var_nstrnum + 1 NOT BETWEEN var_redge.fromhn AND var_redge.tohn THEN var_nstrnum - 1 ELSE var_nstrnum + 1 END;
                	END IF;
                    var_strnum := var_nstrnum::varchar;
                    var_addy := normalize_address( COALESCE(var_strnum::varchar || ' ', '') || var_redge.fullname || ', ' || var_redge.place || ', ' || var_redge.stusps || ' ' || var_redge.zip);
                    addy := array_append(addy, var_addy);
                END IF;
           END IF;
        END IF;
  END LOOP;
          		
  RETURN;   
END;
$_$ LANGUAGE plpgsql STABLE;

CREATE OR REPLACE FUNCTION public.reverse_geocode(IN pt geometry, OUT intpt geometry[],
    OUT addy NORM_ADDY[],
    OUT street varchar[]) RETURNS RECORD
AS 
$$
-- default to not including street range in cross streets
SELECT reverse_geocode($1,false);
$$
language sql STABLE;