bug#13065: Bug in x-file-dialog with GetOpenFileName

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sun, 13 Jan 2013 18:35:41 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 13065@debbugs.gnu.org, duyanning@gmail.com
> 
> > From: Jason Rumney <jasonr@gnu.org>
> > Cc: duyanning@gmail.com,  13065@debbugs.gnu.org
> > Date: Sun, 13 Jan 2013 20:52:25 +0800
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > >> Is it not just a case of doing the same as we did previously?
> > >> 
> > >>   /* If we compile with _WIN32_WINNT set to 0x0400 (for NT4
> > >>      compatibility) we end up with the old file dialogs. Define a big
> > >>      enough struct for the new dialog to trick GetOpenFileName into
> > >>      giving us the new dialogs on newer versions of Windows.  */
> > >
> > > Sorry, I don't understand: we still do what the above comment
> > > describes.
> > 
> > I meant maybe we need to increase the size of the struct again to fool
> > Windows 7 into thinking we are compiling against a more recent version
> > of the SDK.
> 
> Thanks, will try that some time soon.

I spent some more time tinkering with this.

First, the new Windows 7 dialogs are not merely an extension of the
old ones, they are an entirely different implementation, via COM
objects and a C++ interface.  They also don't accept a structure
similar to OPENFILENAME.  So the trick to expand the structure is not
going to help us.

> >  c) The filter string uses some "*|*" magic in the directory filter,
> >     which doesn't seem to be documented anywhere.  What does it do?
> 
> Nothing.  It has to be different than the *.* that is used for normal
> file selection, but couldn't be simply * IIRC. The magic is in the
> callback function and the code that strips away the filename when the
> dialog call has finished if the "Directories" option is selected.
> 
> But in this report, the Directory filter was never chosen, so I don't
> think it is related.  You could try removing the Directories portion of
> the filter string and the callback to be sure.

Tried without the Directories portion, didn't help.

I tried to attack this from a different angle.  I compiled a simple
stand-alone program (source below) that just opens the file selector
dialog and prints the selected name after you click Open.  This
program works fine both on XP and on Windows 7, and shows a dialog
that is almost exactly the same as what Emacs displays -- the only
difference, which only shows on Windows 7, is in the slightly
different visual appearance of the "Open" and "Cancel" buttons (on XP
the dialog is identical to what Emacs shows, including the buttons).

(To show dialogs like Emacs does, compile the program with the
"-D_WIN32_WINNT=0x0501" switch, otherwise you get old-style dialogs.)

Then I examined the Emacs sources for any differences between how we
set up the dialog and this sample program, and modified the Emacs
source to exactly match the sample program, right down to use of
ZeroMemory instead of memset (ridiculous, I know; but I was
desperate).  Nothing helped: the same code in Emacs still won't close
the dialog and return the file name, unless I select an existing
file.  By contrast, the sample program below will return any file
name, whether or not that file exists, in any directory.

Maybe I'm blind and missed some difference that I didn't eliminate.
You are welcome to try that experiment and see if you succeed where I
failed.

Failing that, I'm beginning to think that the problem is elsewhere in
Emacs.  But what could that be?  Is it possible that one of our
message pumps somehow interferes with the dialog interaction, e.g., by
intercepting some of the messages that were supposed to wind up in our
hook procedure?  Could it be due to the fact that our main message
pump uses Unicode APIs, while the file selector uses ANSI APIs?
Something else?

I'm stumped.

Here's the program to try:

#include <stdio.h>
#include <windows.h>

static UINT_PTR CALLBACK
file_dialog_callback (HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam)
{
  return 0;
}

int
main (void)
{
  OPENFILENAMEA ofn;       // common dialog box structure
  char szFile[260];       // buffer for file name

  // Initialize OPENFILENAME
  ZeroMemory(&ofn, sizeof(ofn));
  ofn.lStructSize = sizeof(ofn);
  ofn.hwndOwner = NULL;
  ofn.lpstrFile = szFile;
  // Set lpstrFile[0] to '\0' so that GetOpenFileName does not 
  // use the contents of szFile to initialize itself.
  ofn.lpstrFile[0] = '\0';
  ofn.nMaxFile = sizeof(szFile);
  ofn.lpstrFilter = "All\0*.*\0Text\0*.TXT\0";
  ofn.nFilterIndex = 1;
  ofn.lpstrFileTitle = NULL;
  ofn.nMaxFileTitle = 0;
  ofn.lpstrInitialDir = NULL;
  ofn.Flags = (OFN_HIDEREADONLY | OFN_NOCHANGEDIR | OFN_EXPLORER | OFN_ENABLEHOOK);
  ofn.lpfnHook = file_dialog_callback;

  // Display the Open dialog box.

  if (GetOpenFileNameA(&ofn)==TRUE)
    printf ("GetOpenFile: '%s'\n", ofn.lpstrFile);
  else
    printf ("Failure: 0x%x\n", (unsigned)CommDlgExtendedError ());

  return 0;
}