Temp files from testing are permanent...

Temp files from testing are permanent...

[Achim Gratz]

I've started to always test each build.  It seems that the temporary
files from testing (they end up in /tmp) are not actually removed after
testing and continue to pile up.  Is there an option that I might have
missed that prevents these files from being removed or should cleaning
those files be added to the Makefile?  Im starting the test suite with

/usr/local/bin/emacs -batch -u myself -L lisp/ \
--eval '(defconst org-release "7.8.03-Test")' \
-l testing/org-test.el \
-eval "(setq org-confirm-babel-evaluate nil)" \
-f org-test-run-batch-tests


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Temp files from testing are permanent...

[Olaf Meeuwissen]

Achim Gratz <Stromeko@nexgo.de> writes:

> I've started to always test each build.  It seems that the temporary
> files from testing (they end up in /tmp) are not actually removed after
> testing and continue to pile up.  Is there an option that I might have
> missed that prevents these files from being removed or should cleaning
> those files be added to the Makefile?  Im starting the test suite with

If running `make check` (or similar) creates these files, `make clean`
(or similar) should clean them up.

See [[info:automake-1.11#Clean]] for details.

> /usr/local/bin/emacs -batch -u myself -L lisp/ \
> --eval '(defconst org-release "7.8.03-Test")' \
> -l testing/org-test.el \
> -eval "(setq org-confirm-babel-evaluate nil)" \
> -f org-test-run-batch-tests

Hope this helps,
-- 
Olaf Meeuwissen, LPIC-2           FLOSS Engineer -- AVASYS CORPORATION
FSF Associate Member #1962               Help support software freedom
                 http://www.fsf.org/jf?referrer=1962

Re: Temp files from testing are permanent...

[Achim Gratz]

Olaf Meeuwissen <olaf.meeuwissen@avasys.jp> writes:
> If running `make check` (or similar) creates these files, `make clean`
> (or similar) should clean them up.

I was asking that question to decide whether I do need to extend my
Makefile fork to handle the cleanup or if the testsuite needs to be
called differently to avoid leaving these stale files.  If you'd take a
look, you can see that the files are obviously made with a function that
ensures they're unique, most likely make-temp-file.  If so, it would
seem logical that the testsuite should remove them after each test
(since it is the only instance to know the complete filename).  It would
also seem logical that for debugging purposes one could leave the files
around.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Temp files from testing are permanent...

[Brian Wightman]

On Wed, Feb 15, 2012 at 11:11 AM, Achim Gratz <Stromeko@nexgo.de> wrote:
> Olaf Meeuwissen <olaf.meeuwissen@avasys.jp> writes:
>> If running `make check` (or similar) creates these files, `make clean`
>> (or similar) should clean them up.
>
> I was asking that question to decide whether I do need to extend my
> Makefile fork to handle the cleanup or if the testsuite needs to be
> called differently to avoid leaving these stale files.  ...  It would
> also seem logical that for debugging purposes one could leave the files
> around.

If created with make-temp-file, and created in the system-configured
$TMPDIR directory, I would urge (wearing my sysadmin hat) that they
get purged at the end of the test run, unless told to do otherwise.
If created in the build/test directory, then either make clean or
immediate purge seems reasonable.

Files created in the system $TMPDIR are not meant (caution: purist
view) to remain beyond the execution of the program (or set of
programs).  Some OS variants are set up to purge the $TMPDIR on
reboot, login, or at other times, and some even store it in a virtual
memory backed filesystem.  When performing as a sysadmin, finding that
an application has littered a (usually) limited system resource such
as the system $TMPDIR with files that are no longer useful is a minor
irritant at best, to a crash-inducing resource consumer at worst.

Just my $0.02 if you are taking donations.

Brian

Re: Temp files from testing are permanent...

[Achim Gratz]

Brian Wightman <MidLifeXis@wightmanfam.org> writes:
> Files created in the system $TMPDIR are not meant (caution: purist
> view) to remain beyond the execution of the program (or set of
> programs).

You are preaching to the choir... :-)

I'm still looking for some wisdom regarding the actual testsuite setup
in orgmode, though.  We have org-test-with-temp-text-in-file in
org-test.el, which supposedly cleans up after itself.  Other tests
aren't as careful, specifically files with the following prefixes: awk-,
C-bin-, C-src-, emacs, fortran-bin-, fortan-src-, maxima-, octave-,
org-test (seems to be an HTML export from a temporary org file that does
get removed), python-, scor and sh-.

Barring any setup or options I might have missed, it looks like I will
have to do two things: first, set TMPDIR to a newly created directory
during the test runs and second, remove that directory after the test
(with an option to keep it for inspection and removing only with
"clean").


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Temp files from testing are permanent...

[Achim Gratz]

Achim Gratz <Stromeko@nexgo.de> writes:
> Barring any setup or options I might have missed, it looks like I will
> have to do two things: first, set TMPDIR to a newly created directory
> during the test runs and second, remove that directory after the test
> (with an option to keep it for inspection and removing only with
> "clean").

Done and pushed to my Makefile fork on repo.or.cz.

Temporary test files automatically vaporize when the test suite
successfully runs and are left for inspection otherwise.  If you need to
keep them regardless of the test result, define TEST_NO_AUTOCLEAN
non-empty.  A new target cleantest can be used to remove the temporary
test directory and all its content whenever necessary, this is also done
when doing a "cleanall".  The location of the temporary test directory
follows $TMPDIR and can be overriden in local.mk via variable testdir.
TMPDIR, if not already present in the environment, will default to /tmp,
which incidentally should resolve a bug in my tricky build environment
at work.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Temp files from testing are permanent...

[Olaf Meeuwissen]

Achim Gratz <Stromeko@nexgo.de> writes:

> Olaf Meeuwissen <olaf.meeuwissen@avasys.jp> writes:
>> If running `make check` (or similar) creates these files, `make clean`
>> (or similar) should clean them up.
>
> I was asking that question to decide whether I do need to extend my
> Makefile fork to handle the cleanup or if the testsuite needs to be
> called differently to avoid leaving these stale files.  If you'd take a
> look, you can see that the files are obviously made with a function that
> ensures they're unique, most likely make-temp-file.  If so, it would
> seem logical that the testsuite should remove them after each test
> (since it is the only instance to know the complete filename).  It would
> also seem logical that for debugging purposes one could leave the files
> around.

Successful tests can clean up after themselves but failed tests should
not so you can debug.  The decision to remove these files should be left
to whoever runs the test suite.  That implies that even successful tests
don't really have to bother cleaning up after themselves.

I normally instruct my tests to create files in a dedicated directory
with a fixed name somewhere below $(top_builddir).  Then a `make clean`
can just remove the whole directory.

If you use `make check` to run the test suite, you can easily set TMPDIR
via the TESTS_ENVIRONMENT Makefile variable (assuming that that variable
is taken into account when making unique file names).  Something like

  TESTS_ENVIRONMENT = TMPDIR=$(builddir)/test-outputs

and then add a clean-local target like

  clean-local:
  	-rm -rf $(builddir)/test-outputs

See [[info:automake-1.11#Simple%20Tests][info:automake-1.11#Simple
Tests]] and [[info:automake-1.11#Extending]] for details.

Hope this helps,
-- 
Olaf Meeuwissen, LPIC-2           FLOSS Engineer -- AVASYS CORPORATION
FSF Associate Member #1962               Help support software freedom
                 http://www.fsf.org/jf?referrer=1962

Re: Temp files from testing are permanent...

[Achim Gratz]

Olaf Meeuwissen <olaf.meeuwissen@avasys.jp> writes:
> Successful tests can clean up after themselves but failed tests should
> not so you can debug.  The decision to remove these files should be left
> to whoever runs the test suite.  That implies that even successful tests
> don't really have to bother cleaning up after themselves.

As hinted at in another post yesterday, I did browse the test suite code
briefly.  The tests try to clean up after themselves, sort-of — the
trouble is that they leave tangled files and results produced during
suceesful tests.

> If you use `make check` to run the test suite, you can easily set TMPDIR
> via the TESTS_ENVIRONMENT Makefile variable (assuming that that variable
> is taken into account when making unique file names).  Something like

Yes, that's about the same conclusion I also reached.  Not sure when
I'll have time to implement and test it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Temp files from testing are permanent...

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Olaf Meeuwissen <olaf.meeuwissen@avasys.jp> writes:
>> Successful tests can clean up after themselves but failed tests should
>> not so you can debug.  The decision to remove these files should be left
>> to whoever runs the test suite.  That implies that even successful tests
>> don't really have to bother cleaning up after themselves.
>
> As hinted at in another post yesterday, I did browse the test suite code
> briefly.  The tests try to clean up after themselves, sort-of

Sorry to come late to this discussion.  The tests should clean up all
temporary files and buffers left during their execution.  There has been
some work on this in the last month or two so hopefully the situation
has improved.

> — the trouble is that they leave tangled files and results produced
> during suceesful tests.
>

It will be up to the authors of individual tests to remove tangled and
exported files.  Ideally we can patch each test to clean up after
itself.  Perhaps we should provide a test macro which accepts a list of
file names and optionally removes them if the test exists successfully.
e.g.,

(org-test-with-cleanup '("exported.html" "tangled.sh" etc...)
  ...test body...)

>
>> If you use `make check` to run the test suite, you can easily set TMPDIR
>> via the TESTS_ENVIRONMENT Makefile variable (assuming that that variable
>> is taken into account when making unique file names).  Something like
>
> Yes, that's about the same conclusion I also reached.  Not sure when
> I'll have time to implement and test it.
>

Even if the above is implemented this sounds like a good safeguard.

Thanks,

>
>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Temp files from testing are permanent...

[Achim Gratz]

Eric Schulte <eric.schulte@gmx.com> writes:
> It will be up to the authors of individual tests to remove tangled and
> exported files.  Ideally we can patch each test to clean up after
> itself.  Perhaps we should provide a test macro which accepts a list of
> file names and optionally removes them if the test exists successfully.
> e.g.,
>
> (org-test-with-cleanup '("exported.html" "tangled.sh" etc...)
>   ...test body...)

Currently those files would have inpredictable names AFAICS, which would
defeat that purpose.  I'm not sure where to do this, but it occurs to me
that it would be easier if those files only had a unique prefix that
didn't change for each invocation of the test suite (a timestamp would
be OK, so it is easier to see which files you're looking at).  If the
files always had the same name, you would have to make sure that there
was no collision, either with existing files or results from earlier
tests, so that would be no good.  If it is easier to always have the
same names for the files, one could just make the test directory name
unique.  I think it is useful for debugging to be able to keep a few
generation of tests around.

> Even if the above is implemented this sounds like a good safeguard.

Safeguards are in place in my Makefile fork.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Temp files from testing are permanent...

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte <eric.schulte@gmx.com> writes:
>> It will be up to the authors of individual tests to remove tangled and
>> exported files.  Ideally we can patch each test to clean up after
>> itself.  Perhaps we should provide a test macro which accepts a list of
>> file names and optionally removes them if the test exists successfully.
>> e.g.,
>>
>> (org-test-with-cleanup '("exported.html" "tangled.sh" etc...)
>>   ...test body...)
>
> Currently those files would have inpredictable names AFAICS, which would
> defeat that purpose.

I should have been more clear.  I'm thinking that this would be a macro
used /within/ unit tests so that testers could specify what files will
be created (test writers should be able to predict the file names
created by their tests) and then the macro will handle cleanup.  Here's
one implementation of such a macro

  (defmacro org-test-with-cleanup (files &rest body)
    "If BODY executes successfully delete FILES and return results of BODY."
    (declare (indent 1))
    `(let ((result (progn ,@body)))
       (mapc (lambda (file) (when (file-exists-p file) (delete-file file))) files)
       result))

> I'm not sure where to do this, but it occurs to me that it would be
> easier if those files only had a unique prefix that didn't change for
> each invocation of the test suite (a timestamp would be OK, so it is
> easier to see which files you're looking at).  If the files always had
> the same name, you would have to make sure that there was no
> collision, either with existing files or results from earlier tests,
> so that would be no good.  If it is easier to always have the same
> names for the files, one could just make the test directory name
> unique.  I think it is useful for debugging to be able to keep a few
> generation of tests around.
>

I do like the idea of a single directory in which all output files may
be collected.  The only potential downside I see for this is that files
will be generated both from within org files in the testing/examples
directory as well as temporary files.

>
>> Even if the above is implemented this sounds like a good safeguard.
>
> Safeguards are in place in my Makefile fork.
>
>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Temp files from testing are permanent...

[Achim Gratz]

Eric Schulte <eric.schulte@gmx.com> writes:
> I should have been more clear.  I'm thinking that this would be a macro
> used /within/ unit tests so that testers could specify what files will
> be created (test writers should be able to predict the file names
> created by their tests) and then the macro will handle cleanup.

I'm sitting on the fence with this.  Running the tests from the Makefile
it would probably be more difficult to ensure that one could keep the
files when the tests were trying to clean up after themselves (as some
option would need to be injected into the test invocation and/or a
different test command would need to be called).

> I do like the idea of a single directory in which all output files may
> be collected.  The only potential downside I see for this is that files
> will be generated both from within org files in the testing/examples
> directory as well as temporary files.

The temporary files could be in a sub-directory... or each test (group)
could have their own sub-directory.  Whatever the organisation, there
should be a single directory which, if recursively removed, gets rid of
all files created by the test run.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Temp files from testing are permanent...

[Olaf Meeuwissen]

FTR, I'm just commenting based on experience with a testing harness for
a completely unrelated piece of software.

Achim Gratz <Stromeko@nexgo.de> writes:

> Eric Schulte <eric.schulte@gmx.com> writes:
>> I should have been more clear.  I'm thinking that this would be a macro
>> used /within/ unit tests so that testers could specify what files will
>> be created (test writers should be able to predict the file names
>> created by their tests) and then the macro will handle cleanup.

Test writers can predict/choose the file names created by their tests
but they cannot predict the file names creates by other test writers'
tests (or their own tests written two weeks ago ;-).  Unless there is
some naming policy that is strictly adhered to, the chance of collisions
remains.  One approach that has worked for me is to have tests create
their outputs in a directory named after the test.  So if I have a test
implemented in test.el, it would create all outputs in test.out/, for
example.  As I put all my tests and their inputs in a tests/ directory
(and nothing else), I can just `rm -rf *.out` there to clean up.

> I'm sitting on the fence with this.  Running the tests from the Makefile
> it would probably be more difficult to ensure that one could keep the
> files when the tests were trying to clean up after themselves (as some
> option would need to be injected into the test invocation and/or a
> different test command would need to be called).

Personally, I prefer to have `make clean` take care of cleaning up my
test outputs.  I control when it gets run, can move valuable stuff out
of the way before doing so and none of the tests have to bother with
conditionalized clean up.

>> I do like the idea of a single directory in which all output files may
>> be collected.  The only potential downside I see for this is that files
>> will be generated both from within org files in the testing/examples
>> directory as well as temporary files.
>
> The temporary files could be in a sub-directory... or each test (group)
> could have their own sub-directory.  Whatever the organisation, there
> should be a single directory which, if recursively removed, gets rid of
> all files created by the test run.

See above.

Hope this helps,
-- 
Olaf Meeuwissen, LPIC-2           FLOSS Engineer -- AVASYS CORPORATION
FSF Associate Member #1962               Help support software freedom
                 http://www.fsf.org/jf?referrer=1962