propose deprecation of generalized-vector-*

propose deprecation of generalized-vector-*

[Daniel Llorens]

Hello,

today I filed a bug on generalized-vector->list

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12465

I remember similar bugs in the past, and I'm thinking that these functions are redundant since we have array-ref, array->list, and so on, which also work on strings, uniform vectors, etc.

The only generalized-vector-? function that doesn't have a direct array-? correspondence is generalized-vector-length. However, even for arrays of rank > 1 it is often convenient to have a function such as

(array-length a) = (car (array-dimensions a))

or maybe

(array-length a) = (fold * 1 (array-dimensions a))

Personally I'd favor the first as there's nothing to compute, but either would work to replace generalized-vector-length.

Possible objections:

— generalized-vector-? may be marginally faster than array-?

Both need to do a bunch of type checks and dispatching, and generalized-vector-? needs to check rank anyway, so I'm guessing this doesn't matter. We could do some benchmarking.

— generalized-vector-? acts as an extra check on rank

array-ref will flag rank errors, so this objection applies only to -length and ->list. I do not think this is important because it's usually clear from the code whether you're dealing with arrays of rank > 1 or not. But on this I'd like to hear from users of generalized-vector-?.

TLDR, I propose to remove the generalized-vector-? functions since the array-? set can replace them.

Regards,

	Daniel

Re: propose deprecation of generalized-vector-*

[Peter TB Brett]

Daniel Llorens <daniel.llorens@bluewin.ch> writes:

> The only generalized-vector-? function that doesn't have a direct
> array-? correspondence is generalized-vector-length. However, even for
> arrays of rank > 1 it is often convenient to have a function such as
>
> (array-length a) = (car (array-dimensions a))
>
> or maybe
>
> (array-length a) = (fold * 1 (array-dimensions a))
>
> Personally I'd favor the first as there's nothing to compute, but
> either would work to replace generalized-vector-length.

It seems to me that array-length should return the first non-unity
dimension. This is the approach taken by e.g. MATLAB's length()
function. It would give it a distinct utility compared to
array-dimensions (which is analogous to MATLAB's size() function).

WDYT?

                        Peter

-- 
Peter Brett <peter@peter-b.co.uk>
Remote Sensing Research Group
Surrey Space Centre

Re: propose deprecation of generalized-vector-*

[Daniel Llorens]

On Sep 19, 2012, at 18:00, guile-devel-request@gnu.org wrote:

> Date: Wed, 19 Sep 2012 13:02:25 +0100
> From: Peter TB Brett <peter@peter-b.co.uk>
> To: guile-devel@gnu.org
> Subject: Re: propose deprecation of generalized-vector-*

 ...

> It seems to me that array-length should return the first non-unity
> dimension. This is the approach taken by e.g. MATLAB's length()
> function. It would give it a distinct utility compared to
> array-dimensions (which is analogous to MATLAB's size() function).
> 
> WDYT?
> 
>                        Peter

That is not exactly what length() does in Matlab:

> >> help length
>  LENGTH   Length of vector.
>     LENGTH(X) returns the length of vector X.  It is equivalent
>     to MAX(SIZE(X)) for non-empty arrays and 0 for empty ones.

The notion of rank in Matlab is rather sui generis. There are no rank 1 objects as such, only row or column vectors. Even size(0) gives [1 1]. So there's constant confusion when you want a simple vector X, because Matlab gives you a column or a row and you need to know which one you've got, even though the things you want to do with X don't depend on that at all. That leads to superfluous use of (:) and .'   .

The above definition of length() is meant to paper over the row/column distinction. Octave says:

> octave:1> help length
> `length' is a built-in function
> 
>  -- Built-in Function:  length (A)
>      Return the `length' of the object A.  For matrix objects, the
>      length is the number of rows or columns, whichever is greater (this
>      odd definition is used for compatibility with MATLAB).

Guile is strict about rank, so this problem doesn't exist. Wouldn't you agree?

----

Going back to the two definitions I proposed, and after giving it some thought, I favor the first

  (array-length a) = (car (array-dimensions a))

more strongly than before, for these reasons:

1. Utility. I do the equivalent of (car (array-dimensions a)) much more often than either (fold * 1 (array-dimensions a)) or what Matlab length() does. An array is often also a list of objects, e.g. a list of n points in R^m is an [n m] shape array, or a list of n transformation matrices is an [n m m] shape array [*].

Of course this is because of the way I use arrays in my own code. I'd be interested in reading about what other people do.
 
2. Efficiency. (car (array-dimensions a)) deserves a shortcut, since it's a waste to construct a list only to take its car. You can say that of the other axes, but the first axis is used more often.

[*] This is very common in J and K, the descendants of APL. In fact K's 'arrays' don't even need to be rectangular. But I think J can be a good model for Guile arrays. J has an operator '#' (tally) which is basically what I propose for array-length. The only difference is that in J (# scalar) gives 1, and this seems irregular. It maybe better to make (array-length #0(0)) an error.

Here's an implementation with this behavior.

    SCM array_length(SCM a)
    {
        scm_t_array_handle h;
        scm_array_get_handle(a, &h);
        if (scm_array_handle_rank(&h)==0) {
            scm_array_handle_release(&h);
            scm_error_scm(scm_from_locale_symbol("out-of-range"), SCM_BOOL_F,
                          scm_from_locale_string("no items for rank 0 array"), SCM_EOL, a);
        }
        scm_t_array_dim const * dims = scm_array_handle_dims(&h);
        SCM l = scm_from_size_t(dims->ubnd-dims->lbnd+1);
        scm_array_handle_release(&h);
        return l;
    }

Re: propose deprecation of generalized-vector-*

[Ludovic Courtès]

Hi!

Daniel Llorens <daniel.llorens@bluewin.ch> skribis:

> today I filed a bug on generalized-vector->list
>
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12465
>
> I remember similar bugs in the past, and I'm thinking that these functions are redundant since we have array-ref, array->list, and so on, which also work on strings, uniform vectors, etc.

Agreed.

> The only generalized-vector-? function that doesn't have a direct array-? correspondence is generalized-vector-length. However, even for arrays of rank > 1 it is often convenient to have a function such as
>
> (array-length a) = (car (array-dimensions a))
>
> or maybe
>
> (array-length a) = (fold * 1 (array-dimensions a))
>
> Personally I'd favor the first as there's nothing to compute, but either would work to replace generalized-vector-length.

Yes.  That procedure would only make sense for one-dimensional arrays
anyway.  It could just as well throw an error when passed a
multi-dimensional array, no?

[...]

> TLDR, I propose to remove the generalized-vector-? functions since the array-? set can replace them.

I think I agree, but I’d like to hear what Andy thinks, since he’s done
a major overhaul of this part recently (and actually, thanks to this,
generalized-vectors.c is very small.)

Thanks,
Ludo’.

Re: propose deprecation of generalized-vector-*

[Daniel Llorens]

>   5. Re: propose deprecation of generalized-vector-*
>      (Ludovic =?utf-8?Q?Court=C3=A8s?=)

> Yes.  That procedure would only make sense for one-dimensional arrays
> anyway.  It could just as well throw an error when passed a
> multi-dimensional array, no?

I think that there should be a function that returns the leading dimension of an array for arrays of any rank.

I argued for this here

http://lists.gnu.org/archive/html/guile-devel/2012-09/msg00065.html

Extending array-length to have this behavior is really the simplest option.

Regards

	Daniel

Re: propose deprecation of generalized-vector-*

[Ludovic Courtès]

Hi,

Daniel Llorens <daniel.llorens@bluewin.ch> skribis:

>>   5. Re: propose deprecation of generalized-vector-*
>>      (Ludovic =?utf-8?Q?Court=C3=A8s?=)
>
>> Yes.  That procedure would only make sense for one-dimensional arrays
>> anyway.  It could just as well throw an error when passed a
>> multi-dimensional array, no?
>
> I think that there should be a function that returns the leading dimension of an array for arrays of any rank.
>
> I argued for this here
>
> http://lists.gnu.org/archive/html/guile-devel/2012-09/msg00065.html
>
> Extending array-length to have this behavior is really the simplest option.

Right, makes sense.  Let’s see what Andy thinks.

(And sorry for the ~2-month delay!)

Ludo’.

Re: propose deprecation of generalized-vector-*

[Andy Wingo]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

Hi,

Sorry for the long delay.

Deprecating the generalized-vector functions sounds mostly sensible to
me, and the proposed semantics of array-length sound fine.  Attached is
a first patch in that direction.

However, before going further, some thoughts.

Firstly, array-set! has a different interface from
generalized-vector-set!; a shame to change already sensible uses of
generalized-vector-set!.  But I am OK with that.

I think it's a useful simplification, mentally, to be able to treat
generic one-dimensional indexed sequences as vectors instead of arrays.
The feedback that you get from documentation and the editor is nicer
when you don't have to deal with variable arity.  Though I like removing
needless code, it seems to me that this abstraction does have some minor
benefit.

I always wondered why vector-ref and vector-set! didn't do what
generalized-vector-ref and generalized-vector-set! did.  I mean, they
are primitive generics.  Might it make sense to allow vector-ref to
operate as generalized-vector-ref did?  I really don't know, myself...

Finally, if we are doing this, then we should also deprecate the
pseudo-polymorphic uniform-vector-ref set of functions as well.

What do you think?

Andy


[-- Attachment #2: 0001-deprecate-generalized-vectors-in-favor-of-arrays.patch --]
[-- Type: text/x-diff, Size: 35650 bytes --]

From 8a9b22e6ecbd394a276b04383f4cde9c17481c49 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@pobox.com>
Date: Mon, 21 Jan 2013 17:04:09 +0100
Subject: [PATCH] deprecate generalized vectors in favor of arrays

* libguile/generalized-arrays.h:
* libguile/generalized-arrays.c (scm_c_array_length):
  (scm_array_length): New functions.

* module/ice-9/deprecated.scm:
* libguile/generalized-vectors.c:
* libguile/generalized-vectors.h:
* libguile/deprecated.h:
* libguile/deprecated.c (scm_generalized_vector_p)
  (scm_generalized_vector_length, scm_generalized_vector_ref)
  (scm_generalized_vector_set_x, scm_generalized_vector_to_list):
  Deprecate.

* libguile/uniform.c (scm_uniform_vector_to_list): Use
  scm_array_to_list.

* module/ice-9/boot-9.scm (case): Arrays are generalized vectors.

* module/srfi/srfi-4/gnu.scm (define-any->vector): Use the array
  functions instead of the generalized-vector functions.

* test-suite/tests/arrays.test: Remove generalized-vector->list test;
  covered by array->list test.

* test-suite/tests/bitvectors.test:
* test-suite/tests/bytevectors.test:
* test-suite/tests/srfi-4.test: Adapt to test using array interfaces
  instead of generalized-vector interfaces.

* doc/ref/api-compound.texi: Remove generalized vector docs.
* doc/ref/api-data.texi:
* doc/ref/srfi-modules.texi: Adapt.
---
 doc/ref/api-compound.texi         |   96 +++++--------------------------------
 doc/ref/api-data.texi             |   22 +++++----
 doc/ref/srfi-modules.texi         |    6 +--
 libguile/deprecated.c             |   84 +++++++++++++++++++++++++++++++-
 libguile/deprecated.h             |    8 ++++
 libguile/generalized-arrays.c     |   32 ++++++++++++-
 libguile/generalized-arrays.h     |    5 +-
 libguile/generalized-vectors.c    |   68 +-------------------------
 libguile/generalized-vectors.h    |    8 +---
 libguile/uniform.c                |    4 +-
 module/ice-9/boot-9.scm           |   10 ++--
 module/ice-9/deprecated.scm       |    9 +++-
 module/srfi/srfi-4/gnu.scm        |    8 ++--
 test-suite/tests/arrays.test      |   24 +---------
 test-suite/tests/bitvectors.test  |    3 +-
 test-suite/tests/bytevectors.test |   36 +++++++-------
 test-suite/tests/srfi-4.test      |   38 +++++++--------
 17 files changed, 212 insertions(+), 249 deletions(-)

diff --git a/doc/ref/api-compound.texi b/doc/ref/api-compound.texi
index be3d65f..9cd5468 100644
--- a/doc/ref/api-compound.texi
+++ b/doc/ref/api-compound.texi
@@ -1,7 +1,7 @@
 @c -*-texinfo-*-
 @c This is part of the GNU Guile Reference Manual.
 @c Copyright (C)  1996, 1997, 2000, 2001, 2002, 2003, 2004, 2005, 2006,
-@c   2007, 2009, 2010, 2011, 2012  Free Software Foundation, Inc.
+@c   2007, 2009, 2010, 2011, 2012, 2013  Free Software Foundation, Inc.
 @c See the file guile.texi for copying conditions.
 
 @node Compound Data Types
@@ -22,7 +22,6 @@ values can be looked up within them.
 * Lists::                       Special list functions supported by Guile.
 * Vectors::                     One-dimensional arrays of Scheme objects.
 * Bit Vectors::                 Vectors of bits.
-* Generalized Vectors::         Treating all vector-like things uniformly.
 * Arrays::                      Matrices, etc.
 * VLists::                      Vector-like lists.
 * Record Overview::             Walking through the maze of record APIs.
@@ -993,9 +992,8 @@ are displayed as a sequence of @code{0}s and @code{1}s prefixed by
 #*00000000
 @end example
 
-Bit vectors are also generalized vectors, @xref{Generalized
-Vectors}, and can thus be used with the array procedures, @xref{Arrays}.
-Bit vectors are the special case of one dimensional bit arrays.
+Bit vectors are the special case of one dimensional bit arrays, and can
+thus be used with the array procedures, @xref{Arrays}.
 
 @deffn {Scheme Procedure} bitvector? obj
 @deffnx {C Function} scm_bitvector_p (obj)
@@ -1163,74 +1161,6 @@ Like @code{scm_bitvector_elements}, but the pointer is good for reading
 and writing.
 @end deftypefn
 
-@node Generalized Vectors
-@subsection Generalized Vectors
-
-Guile has a number of data types that are generally vector-like:
-strings, uniform numeric vectors, bytevectors, bitvectors, and of course
-ordinary vectors of arbitrary Scheme values.  These types are disjoint:
-a Scheme value belongs to at most one of the five types listed above.
-
-If you want to gloss over this distinction and want to treat all four
-types with common code, you can use the procedures in this section.
-They work with the @emph{generalized vector} type, which is the union
-of the five vector-like types.
-
-@deffn {Scheme Procedure} generalized-vector? obj
-@deffnx {C Function} scm_generalized_vector_p (obj)
-Return @code{#t} if @var{obj} is a vector, bytevector, string,
-bitvector, or uniform numeric vector.
-@end deffn
-
-@deffn {Scheme Procedure} generalized-vector-length v
-@deffnx {C Function} scm_generalized_vector_length (v)
-Return the length of the generalized vector @var{v}.
-@end deffn
-
-@deffn {Scheme Procedure} generalized-vector-ref v idx
-@deffnx {C Function} scm_generalized_vector_ref (v, idx)
-Return the element at index @var{idx} of the
-generalized vector @var{v}.
-@end deffn
-
-@deffn {Scheme Procedure} generalized-vector-set! v idx val
-@deffnx {C Function} scm_generalized_vector_set_x (v, idx, val)
-Set the element at index @var{idx} of the
-generalized vector @var{v} to @var{val}.
-@end deffn
-
-@deffn {Scheme Procedure} generalized-vector->list v
-@deffnx {C Function} scm_generalized_vector_to_list (v)
-Return a new list whose elements are the elements of the
-generalized vector @var{v}.
-@end deffn
-
-@deftypefn {C Function} int scm_is_generalized_vector (SCM obj)
-Return @code{1} if @var{obj} is a vector, string,
-bitvector, or uniform numeric vector; else return @code{0}.
-@end deftypefn
-
-@deftypefn {C Function} size_t scm_c_generalized_vector_length (SCM v)
-Return the length of the generalized vector @var{v}.
-@end deftypefn
-
-@deftypefn {C Function} SCM scm_c_generalized_vector_ref (SCM v, size_t idx)
-Return the element at index @var{idx} of the generalized vector @var{v}.
-@end deftypefn
-
-@deftypefn {C Function} void scm_c_generalized_vector_set_x (SCM v, size_t idx, SCM val)
-Set the element at index @var{idx} of the generalized vector @var{v}
-to @var{val}.
-@end deftypefn
-
-@deftypefn {C Function} void scm_generalized_vector_get_handle (SCM v, scm_t_array_handle *handle)
-Like @code{scm_array_get_handle} but an error is signalled when @var{v}
-is not of rank one.  You can use @code{scm_array_handle_ref} and
-@code{scm_array_handle_set} to read and write the elements of @var{v},
-or you can use functions like @code{scm_array_handle_<foo>_elements} to
-deal with specific types of vectors.
-@end deftypefn
-
 @node Arrays
 @subsection Arrays
 @tpindex Arrays
@@ -1239,13 +1169,13 @@ deal with specific types of vectors.
 number of dimensions.  Each cell can be accessed in constant time by
 supplying an index for each dimension.
 
-In the current implementation, an array uses a generalized vector for
-the actual storage of its elements.  Any kind of generalized vector
-will do, so you can have arrays of uniform numeric values, arrays of
-characters, arrays of bits, and of course, arrays of arbitrary Scheme
-values.  For example, arrays with an underlying @code{c64vector} might
-be nice for digital signal processing, while arrays made from a
-@code{u8vector} might be used to hold gray-scale images.
+In the current implementation, an array uses a vector of some kind for
+the actual storage of its elements.  Any kind of vector will do, so you
+can have arrays of uniform numeric values, arrays of characters, arrays
+of bits, and of course, arrays of arbitrary Scheme values.  For example,
+arrays with an underlying @code{c64vector} might be nice for digital
+signal processing, while arrays made from a @code{u8vector} might be
+used to hold gray-scale images.
 
 The number of dimensions of an array is called its @dfn{rank}.  Thus,
 a matrix is an array of rank 2, while a vector has rank 1.  When
@@ -1267,9 +1197,9 @@ matrix with zero columns and 3 rows is different from a matrix with 3
 columns and zero rows, which again is different from a vector of
 length zero.
 
-Generalized vectors, such as strings, uniform numeric vectors,
-bytevectors, bit vectors and ordinary vectors, are the special case of
-one dimensional arrays.
+The array procedures are all polymorphic, treating strings, uniform
+numeric vectors, bytevectors, bit vectors and ordinary vectors as one
+dimensional arrays.
 
 @menu
 * Array Syntax::                
diff --git a/doc/ref/api-data.texi b/doc/ref/api-data.texi
index 21398f4..e74095e 100644
--- a/doc/ref/api-data.texi
+++ b/doc/ref/api-data.texi
@@ -4535,7 +4535,7 @@ R6RS (@pxref{R6RS I/O Ports}).
 * Bytevectors and Integer Lists::  Converting to/from an integer list.
 * Bytevectors as Floats::       Interpreting bytes as real numbers.
 * Bytevectors as Strings::      Interpreting bytes as Unicode strings.
-* Bytevectors as Generalized Vectors::  Guile extension to the bytevector API.
+* Bytevectors as Arrays::       Guile extension to the bytevector API.
 * Bytevectors as Uniform Vectors::  Bytevectors and SRFI-4.
 @end menu
 
@@ -4921,25 +4921,27 @@ or UTF-32-decoded contents of bytevector @var{utf}.  For UTF-16 and UTF-32,
 it defaults to big endian.
 @end deffn
 
-@node Bytevectors as Generalized Vectors
-@subsubsection Accessing Bytevectors with the Generalized Vector API
+@node Bytevectors as Arrays
+@subsubsection Accessing Bytevectors with the Array API
 
 As an extension to the R6RS, Guile allows bytevectors to be manipulated
-with the @dfn{generalized vector} procedures (@pxref{Generalized
-Vectors}).  This also allows bytevectors to be accessed using the
-generic @dfn{array} procedures (@pxref{Array Procedures}).  When using
-these APIs, bytes are accessed one at a time as 8-bit unsigned integers:
+with the @dfn{array} procedures (@pxref{Arrays}).  When using these
+APIs, bytes are accessed one at a time as 8-bit unsigned integers:
 
 @example
 (define bv #vu8(0 1 2 3))
 
-(generalized-vector? bv)
+(array? bv)
 @result{} #t
 
-(generalized-vector-ref bv 2)
+(array-rank bv)
+@result{} 1
+
+(array-ref bv 2)
 @result{} 2
 
-(generalized-vector-set! bv 2 77)
+;; Note the different argument order on array-set!.
+(array-set! bv 77 2)
 (array-ref bv 2)
 @result{} 77
 
diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi
index 70a49c8..dff8ca9 100644
--- a/doc/ref/srfi-modules.texi
+++ b/doc/ref/srfi-modules.texi
@@ -1,6 +1,6 @@
 @c -*-texinfo-*-
 @c This is part of the GNU Guile Reference Manual.
-@c Copyright (C)  1996, 1997, 2000, 2001, 2002, 2003, 2004, 2006, 2007, 2008, 2009, 2010, 2011, 2012
+@c Copyright (C)  1996, 1997, 2000, 2001, 2002, 2003, 2004, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013
 @c   Free Software Foundation, Inc.
 @c See the file guile.texi for copying conditions.
 
@@ -1770,8 +1770,8 @@ Like @code{scm_vector_writable_elements} (@pxref{Vector Accessing from
 C}), but returns a pointer to the elements of a uniform numeric vector.
 @end deftypefn
 
-Unless you really need to the limited generality of these functions, it is best
-to use the type-specific functions, or the generalized vector accessors.
+Unless you really need to the limited generality of these functions, it
+is best to use the type-specific functions, or the array accessors.
 
 @node SRFI-4 and Bytevectors
 @subsubsection SRFI-4 - Relation to bytevectors
diff --git a/libguile/deprecated.c b/libguile/deprecated.c
index f0211a5..b5e7cf3 100644
--- a/libguile/deprecated.c
+++ b/libguile/deprecated.c
@@ -2,7 +2,7 @@
    deprecate something, move it here when that is feasible.
 */
 
-/* Copyright (C) 2003, 2004, 2006, 2008, 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+/* Copyright (C) 2003, 2004, 2006, 2008, 2009, 2010, 2011, 2012, 2013 Free Software Foundation, Inc.
  *
  * This library is free software; you can redistribute it and/or
  * modify it under the terms of the GNU Lesser General Public License
@@ -2839,6 +2839,88 @@ SCM_DEFINE (scm_struct_vtable_tag, "struct-vtable-tag", 1, 0, 0,
 
 \f
 
+SCM_DEFINE (scm_generalized_vector_p, "generalized-vector?", 1, 0, 0,
+	    (SCM obj),
+	    "Return @code{#t} if @var{obj} is a vector, string,\n"
+	    "bitvector, or uniform numeric vector.")
+#define FUNC_NAME s_scm_generalized_vector_p
+{
+  scm_c_issue_deprecation_warning
+    ("generalized-vector? is deprecated.  Use array? and check the "
+     "array-rank instead.");
+  return scm_from_bool (scm_is_generalized_vector (obj));
+}
+#undef FUNC_NAME
+
+SCM_DEFINE (scm_generalized_vector_length, "generalized-vector-length", 1, 0, 0,
+	    (SCM v),
+	    "Return the length of the generalized vector @var{v}.")
+#define FUNC_NAME s_scm_generalized_vector_length
+{
+  scm_c_issue_deprecation_warning
+    ("generalized-vector-length is deprecated.  Use array-length instead.");
+  return scm_from_size_t (scm_c_generalized_vector_length (v));
+}
+#undef FUNC_NAME
+
+SCM_DEFINE (scm_generalized_vector_ref, "generalized-vector-ref", 2, 0, 0,
+	    (SCM v, SCM idx),
+	    "Return the element at index @var{idx} of the\n"
+	    "generalized vector @var{v}.")
+#define FUNC_NAME s_scm_generalized_vector_ref
+{
+  scm_c_issue_deprecation_warning
+    ("generalized-vector-ref is deprecated.  Use array-ref instead.");
+  return scm_c_generalized_vector_ref (v, scm_to_size_t (idx));
+}
+#undef FUNC_NAME
+
+SCM_DEFINE (scm_generalized_vector_set_x, "generalized-vector-set!", 3, 0, 0,
+	    (SCM v, SCM idx, SCM val),
+	    "Set the element at index @var{idx} of the\n"
+	    "generalized vector @var{v} to @var{val}.")
+#define FUNC_NAME s_scm_generalized_vector_set_x
+{
+  scm_c_issue_deprecation_warning
+    ("generalized-vector-set! is deprecated.  Use array-set! instead.  "
+     "Note the change in argument order!");
+  scm_c_generalized_vector_set_x (v, scm_to_size_t (idx), val);
+  return SCM_UNSPECIFIED;
+}
+#undef FUNC_NAME
+
+SCM_DEFINE (scm_generalized_vector_to_list, "generalized-vector->list", 1, 0, 0,
+	    (SCM v),
+	    "Return a new list whose elements are the elements of the\n"
+	    "generalized vector @var{v}.")
+#define FUNC_NAME s_scm_generalized_vector_to_list
+{
+  /* FIXME: This duplicates `array_to_list'.  */
+  SCM ret = SCM_EOL;
+  long inc;
+  ssize_t pos, i;
+  scm_t_array_handle h;
+
+  scm_c_issue_deprecation_warning
+    ("generalized-vector->list is deprecated.  Use array->list instead.");
+
+  scm_generalized_vector_get_handle (v, &h);
+
+  i = h.dims[0].ubnd - h.dims[0].lbnd + 1;
+  inc = h.dims[0].inc;
+  pos = (i - 1) * inc;
+
+  for (; i > 0; i--, pos -= inc)
+    ret = scm_cons (h.impl->vref (&h, h.base + pos), ret);
+
+  scm_array_handle_release (&h);
+  return ret;
+}
+#undef FUNC_NAME
+
+
+\f
+
 void
 scm_i_init_deprecated ()
 {
diff --git a/libguile/deprecated.h b/libguile/deprecated.h
index de85c6f..1812dd0 100644
--- a/libguile/deprecated.h
+++ b/libguile/deprecated.h
@@ -847,6 +847,14 @@ SCM_DEPRECATED SCM scm_struct_vtable_tag (SCM handle);
 
 \f
 
+SCM_DEPRECATED SCM scm_generalized_vector_p (SCM v);
+SCM_DEPRECATED SCM scm_generalized_vector_length (SCM v);
+SCM_DEPRECATED SCM scm_generalized_vector_ref (SCM v, SCM idx);
+SCM_DEPRECATED SCM scm_generalized_vector_set_x (SCM v, SCM idx, SCM val);
+SCM_DEPRECATED SCM scm_generalized_vector_to_list (SCM v);
+
+\f
+
 void scm_i_init_deprecated (void);
 
 #endif
diff --git a/libguile/generalized-arrays.c b/libguile/generalized-arrays.c
index 3a0ce25..11675d4 100644
--- a/libguile/generalized-arrays.c
+++ b/libguile/generalized-arrays.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 1995,1996,1997,1998,2000,2001,2002,2003,2004, 2005, 2006, 2009, 2010 Free Software Foundation, Inc.
+/* Copyright (C) 1995,1996,1997,1998,2000,2001,2002,2003,2004, 2005, 2006, 2009, 2010, 2013 Free Software Foundation, Inc.
  * 
  * This library is free software; you can redistribute it and/or
  * modify it under the terms of the GNU Lesser General Public License
@@ -107,6 +107,36 @@ SCM_DEFINE (scm_array_rank, "array-rank", 1, 0, 0,
 #undef FUNC_NAME
 
 
+size_t
+scm_c_array_length (SCM array)
+{
+  scm_t_array_handle handle;
+  size_t res;
+
+  scm_array_get_handle (array, &handle);
+  if (scm_array_handle_rank (&handle) < 1)
+    {
+      scm_array_handle_release (&handle);
+      scm_wrong_type_arg_msg (NULL, 0, array, "array of nonzero rank");
+    }
+  res = handle.dims[0].ubnd - handle.dims[0].lbnd + 1;
+  scm_array_handle_release (&handle);
+
+  return res;
+}
+
+SCM_DEFINE (scm_array_length, "array-length", 1, 0, 0, 
+           (SCM array),
+	    "Return the length of an array: the dimension of its first\n"
+            "dimension.  It is an error to ask for the length of an\n"
+            "array of rank 0.")
+#define FUNC_NAME s_scm_array_rank
+{
+  return scm_from_size_t (scm_c_array_length (array));
+}
+#undef FUNC_NAME
+
+
 SCM_DEFINE (scm_array_dimensions, "array-dimensions", 1, 0, 0, 
            (SCM ra),
 	    "@code{array-dimensions} is similar to @code{array-shape} but replaces\n"
diff --git a/libguile/generalized-arrays.h b/libguile/generalized-arrays.h
index 1f9b6ad..6860cfd 100644
--- a/libguile/generalized-arrays.h
+++ b/libguile/generalized-arrays.h
@@ -3,7 +3,7 @@
 #ifndef SCM_GENERALIZED_ARRAYS_H
 #define SCM_GENERALIZED_ARRAYS_H
 
-/* Copyright (C) 1995,1996,1997,1999,2000,2001, 2004, 2006, 2008, 2009 Free Software Foundation, Inc.
+/* Copyright (C) 1995,1996,1997,1999,2000,2001, 2004, 2006, 2008, 2009, 2013 Free Software Foundation, Inc.
  *
  * This library is free software; you can redistribute it and/or
  * modify it under the terms of the GNU Lesser General Public License
@@ -44,6 +44,9 @@ SCM_API SCM scm_typed_array_p (SCM v, SCM type);
 SCM_API size_t scm_c_array_rank (SCM ra);
 SCM_API SCM scm_array_rank (SCM ra);
 
+SCM_API size_t scm_c_array_length (SCM ra);
+SCM_API SCM scm_array_length (SCM ra);
+
 SCM_API SCM scm_array_dimensions (SCM ra);
 SCM_API SCM scm_array_type (SCM ra);
 SCM_API SCM scm_array_in_bounds_p (SCM v, SCM args);
diff --git a/libguile/generalized-vectors.c b/libguile/generalized-vectors.c
index 4da0e88..5e3e552 100644
--- a/libguile/generalized-vectors.c
+++ b/libguile/generalized-vectors.c
@@ -1,5 +1,5 @@
 /* Copyright (C) 1995, 1996, 1997, 1998, 2000, 2001, 2002, 2003, 2004,
- *   2005, 2006, 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+ *   2005, 2006, 2009, 2010, 2011, 2012, 2013 Free Software Foundation, Inc.
  *
  * This library is free software; you can redistribute it and/or
  * modify it under the terms of the GNU Lesser General Public License
@@ -83,16 +83,6 @@ scm_is_generalized_vector (SCM obj)
   return ret;
 }
 
-SCM_DEFINE (scm_generalized_vector_p, "generalized-vector?", 1, 0, 0,
-	    (SCM obj),
-	    "Return @code{#t} if @var{obj} is a vector, string,\n"
-	    "bitvector, or uniform numeric vector.")
-#define FUNC_NAME s_scm_generalized_vector_p
-{
-  return scm_from_bool (scm_is_generalized_vector (obj));
-}
-#undef FUNC_NAME
-
 #define SCM_VALIDATE_VECTOR_WITH_HANDLE(pos, val, handle)   \
   scm_generalized_vector_get_handle (val, handle)
    
@@ -119,15 +109,6 @@ scm_c_generalized_vector_length (SCM v)
   return ret;
 }
 
-SCM_DEFINE (scm_generalized_vector_length, "generalized-vector-length", 1, 0, 0,
-	    (SCM v),
-	    "Return the length of the generalized vector @var{v}.")
-#define FUNC_NAME s_scm_generalized_vector_length
-{
-  return scm_from_size_t (scm_c_generalized_vector_length (v));
-}
-#undef FUNC_NAME
-
 SCM
 scm_c_generalized_vector_ref (SCM v, size_t idx)
 {
@@ -141,16 +122,6 @@ scm_c_generalized_vector_ref (SCM v, size_t idx)
   return ret;
 }
 
-SCM_DEFINE (scm_generalized_vector_ref, "generalized-vector-ref", 2, 0, 0,
-	    (SCM v, SCM idx),
-	    "Return the element at index @var{idx} of the\n"
-	    "generalized vector @var{v}.")
-#define FUNC_NAME s_scm_generalized_vector_ref
-{
-  return scm_c_generalized_vector_ref (v, scm_to_size_t (idx));
-}
-#undef FUNC_NAME
-
 void
 scm_c_generalized_vector_set_x (SCM v, size_t idx, SCM val)
 {
@@ -162,43 +133,6 @@ scm_c_generalized_vector_set_x (SCM v, size_t idx, SCM val)
   scm_array_handle_release (&h);
 }
 
-SCM_DEFINE (scm_generalized_vector_set_x, "generalized-vector-set!", 3, 0, 0,
-	    (SCM v, SCM idx, SCM val),
-	    "Set the element at index @var{idx} of the\n"
-	    "generalized vector @var{v} to @var{val}.")
-#define FUNC_NAME s_scm_generalized_vector_set_x
-{
-  scm_c_generalized_vector_set_x (v, scm_to_size_t (idx), val);
-  return SCM_UNSPECIFIED;
-}
-#undef FUNC_NAME
-
-SCM_DEFINE (scm_generalized_vector_to_list, "generalized-vector->list", 1, 0, 0,
-	    (SCM v),
-	    "Return a new list whose elements are the elements of the\n"
-	    "generalized vector @var{v}.")
-#define FUNC_NAME s_scm_generalized_vector_to_list
-{
-  /* FIXME: This duplicates `array_to_list'.  */
-  SCM ret = SCM_EOL;
-  long inc;
-  ssize_t pos, i;
-  scm_t_array_handle h;
-
-  scm_generalized_vector_get_handle (v, &h);
-
-  i = h.dims[0].ubnd - h.dims[0].lbnd + 1;
-  inc = h.dims[0].inc;
-  pos = (i - 1) * inc;
-
-  for (; i > 0; i--, pos -= inc)
-    ret = scm_cons (h.impl->vref (&h, h.base + pos), ret);
-
-  scm_array_handle_release (&h);
-  return ret;
-}
-#undef FUNC_NAME
-
 void
 scm_init_generalized_vectors ()
 {
diff --git a/libguile/generalized-vectors.h b/libguile/generalized-vectors.h
index 71b58d2..e2acb98 100644
--- a/libguile/generalized-vectors.h
+++ b/libguile/generalized-vectors.h
@@ -3,7 +3,7 @@
 #ifndef SCM_GENERALIZED_VECTORS_H
 #define SCM_GENERALIZED_VECTORS_H
 
-/* Copyright (C) 1995,1996,1997,1999,2000,2001, 2004, 2006, 2008, 2009 Free Software Foundation, Inc.
+/* Copyright (C) 1995,1996,1997,1999,2000,2001, 2004, 2006, 2008, 2009, 2013 Free Software Foundation, Inc.
  *
  * This library is free software; you can redistribute it and/or
  * modify it under the terms of the GNU Lesser General Public License
@@ -30,12 +30,6 @@
 
 /* Generalized vectors */
 
-SCM_API SCM scm_generalized_vector_p (SCM v);
-SCM_API SCM scm_generalized_vector_length (SCM v);
-SCM_API SCM scm_generalized_vector_ref (SCM v, SCM idx);
-SCM_API SCM scm_generalized_vector_set_x (SCM v, SCM idx, SCM val);
-SCM_API SCM scm_generalized_vector_to_list (SCM v);
-
 SCM_API int scm_is_generalized_vector (SCM obj);
 SCM_API size_t scm_c_generalized_vector_length (SCM v);
 SCM_API SCM scm_c_generalized_vector_ref (SCM v, size_t idx);
diff --git a/libguile/uniform.c b/libguile/uniform.c
index d3ecb1b..a58242d 100644
--- a/libguile/uniform.c
+++ b/libguile/uniform.c
@@ -1,4 +1,4 @@
-/* Copyright (C) 1995,1996,1997,1998,2000,2001,2002,2003,2004, 2005, 2006, 2009, 2010 Free Software Foundation, Inc.
+/* Copyright (C) 1995,1996,1997,1998,2000,2001,2002,2003,2004, 2005, 2006, 2009, 2010, 2013 Free Software Foundation, Inc.
  * 
  * This library is free software; you can redistribute it and/or
  * modify it under the terms of the GNU Lesser General Public License
@@ -193,7 +193,7 @@ SCM_DEFINE (scm_uniform_vector_to_list, "uniform-vector->list", 1, 0, 0,
 {
   if (!scm_is_uniform_vector (uvec))
     scm_wrong_type_arg_msg (FUNC_NAME, SCM_ARG1, uvec, "uniform vector");
-  return scm_generalized_vector_to_list (uvec);
+  return scm_array_to_list (uvec);
 }
 #undef FUNC_NAME
 
diff --git a/module/ice-9/boot-9.scm b/module/ice-9/boot-9.scm
index a22ac8b..7936e28 100644
--- a/module/ice-9/boot-9.scm
+++ b/module/ice-9/boot-9.scm
@@ -532,12 +532,10 @@ If there is no handler at all, Guile prints an error and then exits."
                                          datum
                                          (syntax->datum clause)
                                          (syntax->datum whole-expr)))
-                                      (if (memv datum seen)
-                                          (warn-datum 'duplicate-case-datum))
-                                      (if (or (pair? datum)
-                                              (array? datum)
-                                              (generalized-vector? datum))
-                                          (warn-datum 'bad-case-datum))
+                                      (when (memv datum seen)
+                                        (warn-datum 'duplicate-case-datum))
+                                      (when (or (pair? datum) (array? datum))
+                                        (warn-datum 'bad-case-datum))
                                       (cons datum seen))
                                     seen
                                     (map syntax->datum #'(datums ...)))))
diff --git a/module/ice-9/deprecated.scm b/module/ice-9/deprecated.scm
index 3d40193..56b9c04 100644
--- a/module/ice-9/deprecated.scm
+++ b/module/ice-9/deprecated.scm
@@ -1,4 +1,4 @@
-;;;; Copyright (C) 2003, 2005, 2006, 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+;;;; Copyright (C) 2003, 2005, 2006, 2009, 2010, 2011, 2012, 2013 Free Software Foundation, Inc.
 ;;;;
 ;;;; This library is free software; you can redistribute it and/or
 ;;;; modify it under the terms of the GNU Lesser General Public
@@ -71,7 +71,12 @@
             process-define-module
             fluid-let-syntax
             set-system-module!
-            char-code-limit))
+            char-code-limit
+            generalized-vector?
+            generalized-vector-length
+            generalized-vector-ref
+            generalized-vector-set!
+            generalized-vector->list))
 
 
 ;;;; Deprecated definitions.
diff --git a/module/srfi/srfi-4/gnu.scm b/module/srfi/srfi-4/gnu.scm
index 39d6350..7f595d6 100644
--- a/module/srfi/srfi-4/gnu.scm
+++ b/module/srfi/srfi-4/gnu.scm
@@ -1,6 +1,6 @@
 ;;; Extensions to SRFI-4
 
-;; Copyright (C) 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+;; Copyright (C) 2009, 2010, 2011, 2012, 2013 Free Software Foundation, Inc.
 ;;
 ;; This library is free software; you can redistribute it and/or
 ;; modify it under the terms of the GNU Lesser General Public
@@ -101,14 +101,14 @@
               `(define (,(symbol-append 'any-> tag 'vector) obj)
                  (cond ((,(symbol-append tag 'vector?) obj) obj)
                        ((pair? obj) (,(symbol-append 'list-> tag 'vector) obj))
-                       ((generalized-vector? obj)
-                        (let* ((len (generalized-vector-length obj))
+                       ((and (array? obj) (eqv? 1 (array-rank obj)))
+                        (let* ((len (array-length obj))
                                (v (,(symbol-append 'make- tag 'vector) len)))
                           (let lp ((i 0))
                             (if (< i len)
                                 (begin
                                   (,(symbol-append tag 'vector-set!)
-                                   v i (generalized-vector-ref obj i))
+                                   v i (array-ref obj i))
                                   (lp (1+ i)))
                                 v))))
                        (else (scm-error 'wrong-type-arg #f "" '() (list obj))))))
diff --git a/test-suite/tests/arrays.test b/test-suite/tests/arrays.test
index f13b1a2..adb0b78 100644
--- a/test-suite/tests/arrays.test
+++ b/test-suite/tests/arrays.test
@@ -1,6 +1,6 @@
 ;;;; unif.test --- tests guile's uniform arrays     -*- scheme -*-
 ;;;;
-;;;; Copyright 2004, 2006, 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+;;;; Copyright 2004, 2006, 2009, 2010, 2011, 2012, 2013 Free Software Foundation, Inc.
 ;;;;
 ;;;; This library is free software; you can redistribute it and/or
 ;;;; modify it under the terms of the GNU Lesser General Public
@@ -228,26 +228,6 @@
       (array->list b))))
 
 ;;;
-;;; generalized-vector->list
-;;;
-
-(with-test-prefix "generalized-vector->list"
-  (pass-if-equal '(1 2 3) (generalized-vector->list #s16(1 2 3)))
-  (pass-if-equal '(1 2 3) (generalized-vector->list #(1 2 3)))
-  (pass-if-equal '()  (generalized-vector->list #()))
-
-  (pass-if-equal "http://bugs.gnu.org/12465 - ok"
-      '(3 4)
-    (let* ((a #2((1 2) (3 4)))
-           (b (make-shared-array a (lambda (j) (list 1 j)) 2)))
-      (generalized-vector->list b)))
-  (pass-if-equal "http://bugs.gnu.org/12465 - bad"
-      '(2 4)
-    (let* ((a #2((1 2) (3 4)))
-           (b (make-shared-array a (lambda (i) (list i 1)) 2)))
-      (generalized-vector->list b))))
-
-;;;
 ;;; array-fill!
 ;;;
 
@@ -649,6 +629,4 @@
     (pass-if (equal? (array-row array 1)
                      #u32(2 3)))
     (pass-if (equal? (array-ref (array-row array 1) 0)
-                     2))
-    (pass-if (equal? (generalized-vector-ref (array-row array 1) 0)
                      2))))
diff --git a/test-suite/tests/bitvectors.test b/test-suite/tests/bitvectors.test
index c16fb4d..4e32c61 100644
--- a/test-suite/tests/bitvectors.test
+++ b/test-suite/tests/bitvectors.test
@@ -1,6 +1,6 @@
 ;;;; bitvectors.test --- tests guile's bitvectors     -*- scheme -*-
 ;;;;
-;;;; Copyright 2010, 2011 Free Software Foundation, Inc.
+;;;; Copyright 2010, 2011, 2013 Free Software Foundation, Inc.
 ;;;;
 ;;;; This library is free software; you can redistribute it and/or
 ;;;; modify it under the terms of the GNU Lesser General Public
@@ -22,7 +22,6 @@
 
 (with-test-prefix "predicates"
   (pass-if (bitvector? #*1010101010))
-  (pass-if (generalized-vector? #*1010101010))
   (pass-if (uniform-vector? #*1010101010))
   (pass-if (array? #*1010101010)))
 
diff --git a/test-suite/tests/bytevectors.test b/test-suite/tests/bytevectors.test
index 4ba5012..67fc680 100644
--- a/test-suite/tests/bytevectors.test
+++ b/test-suite/tests/bytevectors.test
@@ -1,6 +1,6 @@
 ;;;; bytevectors.test --- R6RS bytevectors. -*- mode: scheme; coding: utf-8; -*-
 ;;;;
-;;;; Copyright (C) 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+;;;; Copyright (C) 2009, 2010, 2011, 2012, 2013 Free Software Foundation, Inc.
 ;;;; Ludovic Courtès
 ;;;;
 ;;;; This library is free software; you can redistribute it and/or
@@ -589,42 +589,42 @@
     (with-input-from-string "#vu8(0 256)" read)))
 
 \f
-(with-test-prefix "Generalized Vectors"
+(with-test-prefix "Arrays"
 
-  (pass-if "generalized-vector?"
-    (generalized-vector? #vu8(1 2 3)))
+  (pass-if "array?"
+    (array? #vu8(1 2 3)))
 
-  (pass-if "generalized-vector-length"
+  (pass-if "array-length"
     (equal? (iota 16)
-            (map generalized-vector-length
+            (map array-length
                  (map make-bytevector (iota 16)))))
 
-  (pass-if "generalized-vector-ref"
+  (pass-if "array-ref"
     (let ((bv #vu8(255 127)))
-      (and (= 255 (generalized-vector-ref bv 0))
-           (= 127 (generalized-vector-ref bv 1)))))
+      (and (= 255 (array-ref bv 0))
+           (= 127 (array-ref bv 1)))))
 
-  (pass-if-exception "generalized-vector-ref [index out-of-range]"
+  (pass-if-exception "array-ref [index out-of-range]"
     exception:out-of-range
     (let ((bv #vu8(1 2)))
-      (generalized-vector-ref bv 2)))
+      (array-ref bv 2)))
 
-  (pass-if "generalized-vector-set!"
+  (pass-if "array-set!"
     (let ((bv (make-bytevector 2)))
-      (generalized-vector-set! bv 0 255)
-      (generalized-vector-set! bv 1 77)
+      (array-set! bv 255 0)
+      (array-set! bv 77 1)
       (equal? '(255 77)
               (bytevector->u8-list bv))))
 
-  (pass-if-exception "generalized-vector-set! [index out-of-range]"
+  (pass-if-exception "array-set! [index out-of-range]"
     exception:out-of-range
     (let ((bv (make-bytevector 2)))
-      (generalized-vector-set! bv 2 0)))
+      (array-set! bv 0 2)))
 
-  (pass-if-exception "generalized-vector-set! [value out-of-range]"
+  (pass-if-exception "array-set! [value out-of-range]"
     exception:out-of-range
     (let ((bv (make-bytevector 2)))
-      (generalized-vector-set! bv 0 256)))
+      (array-set! bv 256 0)))
 
   (pass-if "array-type"
     (eq? 'vu8 (array-type #vu8())))
diff --git a/test-suite/tests/srfi-4.test b/test-suite/tests/srfi-4.test
index 033e39f..9b76c7a 100644
--- a/test-suite/tests/srfi-4.test
+++ b/test-suite/tests/srfi-4.test
@@ -1,7 +1,7 @@
 ;;;; srfi-4.test --- Test suite for Guile's SRFI-4 functions. -*- scheme -*-
 ;;;; Martin Grabmueller, 2001-06-26
 ;;;;
-;;;; Copyright (C) 2001, 2006, 2010, 2011 Free Software Foundation, Inc.
+;;;; Copyright (C) 2001, 2006, 2010, 2011, 2013 Free Software Foundation, Inc.
 ;;;; 
 ;;;; This library is free software; you can redistribute it and/or
 ;;;; modify it under the terms of the GNU Lesser General Public
@@ -438,24 +438,24 @@
   (pass-if "+inf.0, -inf.0, +nan.0 in c32vector"
     (c32vector? #c32(+inf.0 -inf.0 +nan.0)))
 
-  (pass-if "generalized-vector-ref"
+  (pass-if "array-ref"
     (let ((v (c32vector 1+1i)))
       (= (c32vector-ref v 0)
-         (generalized-vector-ref v 0))))
+         (array-ref v 0))))
 
-  (pass-if "generalized-vector-set!"
+  (pass-if "array-set!"
     (let ((x 1+1i)
           (v (c32vector 0)))
-      (generalized-vector-set! v 0 x)
-      (= x (generalized-vector-ref v 0))))
+      (array-set! v x 0)
+      (= x (array-ref v 0))))
 
-  (pass-if-exception "generalized-vector-ref, out-of-range"
+  (pass-if-exception "array-ref, out-of-range"
     exception:out-of-range
-    (generalized-vector-ref (c32vector 1.0) 1))
+    (array-ref (c32vector 1.0) 1))
 
-  (pass-if-exception "generalized-vector-set!, out-of-range"
+  (pass-if-exception "array-set!, out-of-range"
     exception:out-of-range
-    (generalized-vector-set! (c32vector 1.0) 1 2.0)))
+    (array-set! (c32vector 1.0) 2.0 1)))
 
 (with-test-prefix "c64 vectors"
 
@@ -497,24 +497,24 @@
   (pass-if "+inf.0, -inf.0, +nan.0 in c64vector"
     (c64vector? #c64(+inf.0 -inf.0 +nan.0)))
 
-  (pass-if "generalized-vector-ref"
+  (pass-if "array-ref"
     (let ((v (c64vector 1+1i)))
       (= (c64vector-ref v 0)
-         (generalized-vector-ref v 0))))
+         (array-ref v 0))))
 
-  (pass-if "generalized-vector-set!"
+  (pass-if "array-set!"
     (let ((x 1+1i)
           (v (c64vector 0)))
-      (generalized-vector-set! v 0 x)
-      (= x (generalized-vector-ref v 0))))
+      (array-set! v x 0)
+      (= x (array-ref v 0))))
 
-  (pass-if-exception "generalized-vector-ref, out-of-range"
+  (pass-if-exception "array-ref, out-of-range"
     exception:out-of-range
-    (generalized-vector-ref (c64vector 1.0) 1))
+    (array-ref (c64vector 1.0) 1))
 
-  (pass-if-exception "generalized-vector-set!, out-of-range"
+  (pass-if-exception "array-set!, out-of-range"
     exception:out-of-range
-    (generalized-vector-set! (c64vector 1.0) 1 2.0)))
+    (array-set! (c64vector 1.0) 2.0 1)))
 
 (with-test-prefix "accessing uniform vectors of different types"
 
-- 
1.7.10.4


[-- Attachment #3: Type: text/plain, Size: 25 bytes --]

-- 
http://wingolog.org/

Re: propose deprecation of generalized-vector-*

[Daniel Llorens]

[-- Attachment #1: Type: text/plain, Size: 3041 bytes --]


On Jan 21, 2013, at 17:11, Andy Wingo wrote:

> Hi,
> 
> Sorry for the long delay.
> 
> Deprecating the generalized-vector functions sounds mostly sensible to
> me, and the proposed semantics of array-length sound fine.  Attached is
> a first patch in that direction.

The changes look good to me. The patch attached applies over yours and is to document this function and a couple others in the manual.

> However, before going further, some thoughts.
> 
> Firstly, array-set! has a different interface from
> generalized-vector-set!; a shame to change already sensible uses of
> generalized-vector-set!.  But I am OK with that.

It's worse from C where one has to build a list explicitly. Maybe we should have scm_array_ref_1, scm_array_ref_2, etc. as it is done for some other functions taking rest lists. I can write a patch for that. But the argument order can't be helped.

> I think it's a useful simplification, mentally, to be able to treat
> generic one-dimensional indexed sequences as vectors instead of arrays.
> The feedback that you get from documentation and the editor is nicer
> when you don't have to deal with variable arity.  Though I like removing
> needless code, it seems to me that this abstraction does have some minor
> benefit.

I think this is a good argument for including the rank in a type system or for doing static analysis on rank, because the rank of arrays can often be determined statically. The compiler should know and this knowledge would help in debugging. But this is true for rank 2 or higher, not only for rank 1.

As an interface, I prefer uniformity. My inclination whenever I use arrays is to use only the array- functions and forget about the vector- functions.

> I always wondered why vector-ref and vector-set! didn't do what
> generalized-vector-ref and generalized-vector-set! did.  I mean, they
> are primitive generics.  Might it make sense to allow vector-ref to
> operate as generalized-vector-ref did?  I really don't know, myself...

I think this is fair for type polymorphism. It makes sense to let vector- work on uniform vectors because the implementation should be identical.

It's different for arrays because even for the 1D case you need a stride and an indirection (and in the current Guile, also bounds).

That is, I think that the main distinction is not between rank=1 and rank!=1 objects, but between ‘container’ and ‘views’ (if this makes sense). This is why generalized-vector- was bad. It was an if x, do X, if y, do Y kind of abstraction.

> Finally, if we are doing this, then we should also deprecate the
> pseudo-polymorphic uniform-vector-ref set of functions as well.

If vector-set / -ref / -length works on these, they are not needed, I agree. Some of the uniform-vector functions are still needed because they handle the element type.

So we would have array- / typed-array- functions for ‘views’ of any rank, and vector- / uniform-vector- functions for ‘containers’ of rank 1.

Does this make sense?



[-- Attachment #2: 0001-Array-documentation-fixes.patch --]
[-- Type: application/octet-stream, Size: 2993 bytes --]

From b096700697eeeefed31e4025ec5e30b85b7e1d13 Mon Sep 17 00:00:00 2001
From: Daniel Llorens <daniel.llorens@bluewin.ch>
Date: Tue, 22 Jan 2013 13:01:14 +0100
Subject: [PATCH] Array documentation fixes

* libguile/generalized-arrays.c: Fix wording of docstring for array-length.
* doc/ref/api-compund.texi:
  - Document scm_array_type(), scm_array_ref(), array-length,
    scm_array_length(), scm_c_array_length().
  - Fix wording of documentation for array-in-bounds?
---
 doc/ref/api-compound.texi     |   11 ++++++++++-
 libguile/generalized-arrays.c |    5 ++---
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/doc/ref/api-compound.texi b/doc/ref/api-compound.texi
index 9cd5468..b32cc91 100644
--- a/doc/ref/api-compound.texi
+++ b/doc/ref/api-compound.texi
@@ -1392,6 +1392,7 @@ as elements in the list.
 @end deffn
 
 @deffn {Scheme Procedure} array-type array
+@deffnx {C Function} scm_array_type (array)
 Return the type of @var{array}.  This is the `vectag' used for
 printing @var{array} (or @code{#t} for ordinary arrays) and can be
 used with @code{make-typed-array} to create an array of the same kind
@@ -1399,6 +1400,7 @@ as @var{array}.
 @end deffn
 
 @deffn {Scheme Procedure} array-ref array idx @dots{}
+@deffnx {C Function} scm_array_ref (array, idxlist)
 Return the element at @code{(idx @dots{})} in @var{array}.
 
 @example
@@ -1409,7 +1411,7 @@ Return the element at @code{(idx @dots{})} in @var{array}.
 
 @deffn {Scheme Procedure} array-in-bounds? array idx @dots{}
 @deffnx {C Function} scm_array_in_bounds_p (array, idxlist)
-Return @code{#t} if the given index would be acceptable to
+Return @code{#t} if the given indices would be acceptable to
 @code{array-ref}.
 
 @example
@@ -1450,6 +1452,13 @@ For example,
 @end example
 @end deffn
 
+@deffn {Scheme Procedure} array-length array
+@deffnx {C Function} scm_array_length (array)
+@deffnx {C Function} size_t scm_c_array_length (array)
+Return the length of an array: its first dimension. It is an error to
+ask for the length of an array of rank 0.
+@end deffn
+
 @deffn {Scheme Procedure} array-rank array
 @deffnx {C Function} scm_array_rank (array)
 Return the rank of @var{array}.
diff --git a/libguile/generalized-arrays.c b/libguile/generalized-arrays.c
index 706779e..f2e17e9 100644
--- a/libguile/generalized-arrays.c
+++ b/libguile/generalized-arrays.c
@@ -127,9 +127,8 @@ scm_c_array_length (SCM array)
 
 SCM_DEFINE (scm_array_length, "array-length", 1, 0, 0,
            (SCM array),
-	    "Return the length of an array: the dimension of its first\n"
-            "dimension.  It is an error to ask for the length of an\n"
-            "array of rank 0.")
+	    "Return the length of an array: its first dimension.\n"
+            "It is an error to ask for the length of an array of rank 0.")
 #define FUNC_NAME s_scm_array_rank
 {
   return scm_from_size_t (scm_c_array_length (array));
-- 
1.7.9.5

Re: propose deprecation of generalized-vector-*

[Daniel Llorens]

On Jan 22, 2013, at 15:31, Daniel Llorens wrote:

> On Jan 21, 2013, at 17:11, Andy Wingo wrote:

>> I always wondered why vector-ref and vector-set! didn't do what
>> generalized-vector-ref and generalized-vector-set! did.  I mean, they
>> are primitive generics.  Might it make sense to allow vector-ref to
>> operate as generalized-vector-ref did?  I really don't know, myself...
> 
> I think this is fair for type polymorphism. It makes sense to let vector- work on uniform vectors because the implementation should be identical.
> 
> It's different for arrays because even for the 1D case you need a stride and an indirection (and in the current Guile, also bounds).
> 
> That is, I think that the main distinction is not between rank=1 and rank!=1 objects, but between ‘container’ and ‘views’ (if this makes sense). This is why generalized-vector- was bad. It was an if x, do X, if y, do Y kind of abstraction.

To be a bit less vague…

The current implementation of scm_c_vector_ref() results in behavior such as

scheme@(guile-user)> (define (array-col A j) (make-shared-array A (lambda (i) (list i j)) (car (array-dimensions A))))
scheme@(guile-user)> (define A #2((1 2) (3 4)))
scheme@(guile-user)> (vector? (array-row A 0))
$1 = #f
scheme@(guile-user)> (vector-ref (array-col A 0) 0)
$2 = 1

I think vector-ref should probably fail in this case, i.e. the second branch of scm_c_vector_ref() should be gone.

On the other hand scm_c_uniform_vector_ref() goes through scm_c_generalized_vector_ref() which does handling of offset/stride/bounds. Instead, uniform vectors should be forced to have contiguous storage (no offset/stride/bounds) so that scm_c_vector_ref() is able to handle them and

scheme@(guile-user) [1]> (vector-ref #f64(1 2 3) 0)
<unnamed port>:32:0: In procedure #<procedure 10293ce80 at <current input>:32:0 ()>:
<unnamed port>:32:0: Wrong type argument in position 2: 0

works. This should be all right, because the only way to get non-contiguous arrays from Scheme (iiuc) is through make-shared-array —it's natural that the result should be handled with the array- functions.

The idea is to merge the implementations of SCM vectors and uniform vectors in the same way that

scheme@(guile-user)> (make-array 0 2 2)
$3 = #2((0 0) (0 0))

is equivalent to

scheme@(guile-user)> (make-typed-array #t 0 2 2)
$4 = #2((0 0) (0 0))

I'm not very conversant with the macrology in vectors.h or the exact way uniform vectors are implemented, but it seems that this should be possible.

Thoughts?

	- Daniel

Re: propose deprecation of generalized-vector-*

[Andy Wingo]

Hi Daniel,

Thanks for thinking about this.

On Tue 22 Jan 2013 19:31, Daniel Llorens <daniel.llorens@bluewin.ch> writes:

>> I think this is fair for type polymorphism. It makes sense to let
>> vector- work on uniform vectors because the implementation should be
>> identical.
>> 
>> It's different for arrays because even for the 1D case you need a
>> stride and an indirection (and in the current Guile, also bounds).

Handling stride and bounds is not a problem.  The generic array handle
interface lets us do this in a straightforward way.

>> That is, I think that the main distinction is not between rank=1 and
>> rank!=1 objects, but between ‘container’ and ‘views’ (if this makes
>> sense). This is why generalized-vector- was bad. It was an if x, do X,
>> if y, do Y kind of abstraction.

It was in 1.8.  In 2.0 it table-driven and extensible, which isn't a bad
thing at all IMO.

> To be a bit less vague…
>
> The current implementation of scm_c_vector_ref() results in behavior such as
>
> scheme@(guile-user)> (define (array-col A j) (make-shared-array A (lambda (i) (list i j)) (car (array-dimensions A))))
> scheme@(guile-user)> (define A #2((1 2) (3 4)))
> scheme@(guile-user)> (vector? (array-row A 0))
> $1 = #f
> scheme@(guile-user)> (vector-ref (array-col A 0) 0)
> $2 = 1

    scheme@(guile-user)> (vector? "foo")
    $1 = #f
    scheme@(guile-user)> (generalized-vector? "foo")
    $2 = #t
    scheme@(guile-user)> (vector-ref "foo" 1)
    <unnamed port>:3:0: Wrong type argument in position 2: 1
    scheme@(guile-user)> (generalized-vector-ref "foo" 1)
    $3 = #\o

The error message is incorrect, unfortunately... anyway.  I think this
highlights the reason why we can't make vector-ref generic: vector? and
string? should be disjoint predicates.  (Or should they?  Scheme has
real problems with polymorphism.  I don't know and am interested in
arguments.)

> I think vector-ref should probably fail in this case, i.e. the second
> branch of scm_c_vector_ref() should be gone.

My instinct would be to redefine vector as an interface of sorts:
vector-indexable.  Generic vector.  generalized-vector -- oh wait we're
back to the beginning!

> uniform vectors should be forced to have contiguous storage (no
> offset/stride/bounds) so that scm_c_vector_ref() is able to handle
> them

Not sure this is really what should happen.  The generalized array
interface from generalized-arrays can deal with this quite well, with no
need for a restriction.

> Thoughts?

I think the string case is vexing, and we really need to go back to
fundamentals.

What is a vector?

Possible answers:

  1. A vector is something that answers #t to vector?, which contains
  some number of storage slots accessible in a mostly-O(1) way, the
  number of slots is given by vector-length, and the slots can be
  accessed with vector-ref and vector-set!.

  2. A vector is a specific kind of object, implementing the interface
  described above, and disjoint from all other kinds of objects.

  3. A vector is a specific kind of object, as before, disjoint from all
  other kinds of objects defined in the R5RS.

  4. A vector is a specific kind of object, as before, disjoint from all
  other kinds of objects defined in the R6RS.

(1) defines vectors as an interface.

(2) defines vectors as a specific data structure.

(3) admits to a number of distinct types that may be vectors, of which
    one kind is defined by the R5RS.

(4) is like (3), but it precludes bytevectors from being vectors.

What do you think a vector is? :)

Andy
-- 
http://wingolog.org/

Re: propose deprecation of generalized-vector-*

[Daniel Llorens]

On Jan 22, 2013, at 21:52, Andy Wingo wrote:

Hello,

> Handling stride and bounds is not a problem.  The generic array handle
> interface lets us do this in a straightforward way.

Certainly, but in this case, a vector is just an array of rank 1. I guess I don't value that much having a specific interface just for rank 1 objects.

> It was in 1.8.  In 2.0 it table-driven and extensible, which isn't a bad
> thing at all IMO.

There is still some case jumping in evidence in scm_c_vector_ref() ...

I don't think of vectors and bytevectors and uniform vectors as distinct types. Uniform vectors are an optimization where the objects in the ‘vector’ share the type tag. That's why for me it makes perfect sense to have (vector? #u8(1)) -> #t and to allow (vector-ref #u8(1) 0). And the array/typed-array interface already works like this.

>    scheme@(guile-user)> (vector? "foo")
>    $1 = #f
>    scheme@(guile-user)> (generalized-vector? "foo")
>    $2 = #t
>    scheme@(guile-user)> (vector-ref "foo" 1)
>    <unnamed port>:3:0: Wrong type argument in position 2: 1
>    scheme@(guile-user)> (generalized-vector-ref "foo" 1)
>    $3 = #\o
> 
> The error message is incorrect, unfortunately... anyway.  I think this
> highlights the reason why we can't make vector-ref generic: vector? and
> string? should be disjoint predicates.  (Or should they?  Scheme has
> real problems with polymorphism.  I don't know and am interested in
> arguments.)

I can rationalize this because a string is (in principle) sufficiently different from a vector, maybe  because of Unicode (speaking over my head here). Not the case for numeric vectors or bytevectors, although the standards also disagree for the latter. 

> My instinct would be to redefine vector as an interface of sorts:
> vector-indexable.  Generic vector.  generalized-vector -- oh wait we're
> back to the beginning!

If I understand correctly, at this point it's a matter of names —we can't use use vector?, vector-ref, etc. for this generic interface because those names are captured by the standard.

> Not sure this is really what should happen.  The generalized array
> interface from generalized-arrays can deal with this quite well, with no
> need for a restriction.

I'm good with vector being shorthand for rank-1 array. Or even shorthand for type-#t rank-1 array. I was trying to carve an exclusive function for ‘vector’ that went beyond being shorthand for rank-1 array. It stands out that vector? is in fact making that distinction (vector? meaning both SCM elements *and* no stride/bounds). Maybe it shouldn't?

> I think the string case is vexing, and we really need to go back to
> fundamentals.
> 
> What is a vector?
> 
> Possible answers:
> 
>  1. A vector is something that answers #t to vector?, which contains
>  some number of storage slots accessible in a mostly-O(1) way, the
>  number of slots is given by vector-length, and the slots can be
>  accessed with vector-ref and vector-set!.
> 
>  2. A vector is a specific kind of object, implementing the interface
>  described above, and disjoint from all other kinds of objects.
> 
>  3. A vector is a specific kind of object, as before, disjoint from all
>  other kinds of objects defined in the R5RS.
> 
>  4. A vector is a specific kind of object, as before, disjoint from all
>  other kinds of objects defined in the R6RS.
> 
> (1) defines vectors as an interface.
> 
> (2) defines vectors as a specific data structure.
> 
> (3) admits to a number of distinct types that may be vectors, of which
>    one kind is defined by the R5RS.
> 
> (4) is like (3), but it precludes bytevectors from being vectors.
> 
> What do you think a vector is? :)

Something between 1 and 2…

If vector? has to be disjoint from bytevector?, presumably that's true also for f64vector?, etc. in which case we should

* keep generalized-vector as it is.
* restrict vector- to objects that satisfy vector?. vector? allows stride/bounds but not uniform type. This provides the disjointness.

The array interface seems more logical. Everything is array? and then things are typed-array? of specific types. I see myself not using the vector interface at all.

I don't know.

	- Daniel

Re: propose deprecation of generalized-vector-*

[Andy Wingo]

On Tue 22 Jan 2013 15:31, Daniel Llorens <daniel.llorens@bluewin.ch> writes:

> On Jan 21, 2013, at 17:11, Andy Wingo wrote:
>
> The patch attached applies over yours and is to document this function
> and a couple others in the manual.

Thanks, applied and pushed to wip-generalized-vectors.

> Maybe we should have scm_array_ref_1, scm_array_ref_2, etc. as it is
> done for some other functions taking rest lists. I can write a patch
> for that.

For C, that makes sense.  Something should be done for Scheme as well,
but it's not terribly urgent.  Perhaps make scm_array_ref not be bound
to "array-ref", and instead bind "array-ref" to some function that takes
two optional arguments and a rest argument.  A poor man's case-lambda...

> It makes sense to let vector- work on uniform vectors because the
> implementation should be identical.

FWIW this is not the case.  Vectors hold SCM objects, whereas uniform
vectors hold unpacked machine values.  It's possible to have a
one-dimensional slice of a uniform array also.

Now on to your next mail...

Andy
-- 
http://wingolog.org/

Re: propose deprecation of generalized-vector-*

[Andy Wingo]

Hi,

On Wed 23 Jan 2013 00:27, Daniel Llorens <daniel.llorens@bluewin.ch> writes:

> I guess I don't value that much having a specific interface just for
> rank 1 objects.

I don't care much either; I don't think I have ever used the generalized
vector routines.  If I wanted real polymorphism, I think I would also
want it over user types (GOOPS and record types) as well, and that's
another kettle of fish.

I'm now inclined to punt on any kind of general(ized) solution, and
leave it to a module to handle.

> The array interface seems more logical. Everything is array? and then
> things are typed-array? of specific types. I see myself not using the
> vector interface at all.

Yes, the array interface is consistent, easy to explain, and completely
subsumes the generalized-vector interface.  Let's recommend that for
now.

A guile with generalized arrays (as we have always had) and without
generalized vectors is better because it has the same power, fewer
concepts, fewer bugs, and less code.

I have pushed our patches to a new WIP branch, wip-generalized-vectors.
I'd like to leave it open for comments for a week or so before merging,
just in case someone hasn't had a chance to wade through all of my mails
over the last week.

Regards,

Andy
-- 
http://wingolog.org/

Re: propose deprecation of generalized-vector-*

[Daniel Llorens]

On Jan 23, 2013, at 10:06, Andy Wingo wrote:

> For C, that makes sense.  Something should be done for Scheme as well,
> but it's not terribly urgent.  Perhaps make scm_array_ref not be bound
> to "array-ref", and instead bind "array-ref" to some function that takes
> two optional arguments and a rest argument.  A poor man's case-lambda...

Just saying…

I have written a general rectangular selector for arrays as in APL.

It depends on having a prefix-selection operator. Here's an example from numpy:

In [1]: import numpy as np

In [2]: a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

In [3]: a
Out[3]:
array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9]])

In [4]: a[1]
Out[4]: array([4, 5, 6])

In [5]: a[1, 1]
Out[5]: 5

array-ref can be extended very simply to do that. It accumulates on the position as it is done now, but if the index list comes up short it makes a shared array with the remaining axes instead of giving a rank error. So it shouldn't be any slower than array_ref.

This cannot be done properly from user code with current Guile because scm_i_make_array() and friends are internal. The only option is make-shared-array. Now, this is a nice interface for general slicing, but it requires creating a closure that is only going to be used rank+1 times, plus a bunch of lists. Let's say that I want to iterate through the rows of a [100000 x 3] array. Moving from row to row is fundamentally just moving a pointer. make-shared-array is not a practical way to do it.

The extension of array-ref below isn't a real fix for this use case, because we're still creating a array descriptor for each iteration. But it's way faster than make-shared-array and it is a natural extension. I'm proposing a patch for this.

The only wart is what happens with arrays of rank 0. The function below doesn't return views of rank 0, but the element instead, just as array-ref does. The only advantage of returning a rank 0 view is that one could in principle modify the array through it, but they are a pain to deal with otherwise. My ‘solution’ is to treat the result of array-ref as read-only.

In APL/J there's no difference between an array of rank 0 and a scalar, so this problem doesn't exist. We may have a similar extension to (array-set! array obj . idxlist) where obj is an array of rank (rank(obj)-length(idxlist)) but an element if this gives 0. This would be just a wrapper over array-copy(array-ref) or the current array-set!.

Otherwise I think it should be possible to manipulate SCM_I_ARRAY_DIMS directly, at least from C. Unless there is a way already and I didn't realize, that would be great.

// Generalization of array-ref where the number of indices may be smaller than the rank of a.

    SCM from_(SCM a, SCM i_)
    {
        SCM i = i_;
        scm_t_array_handle ah;
        scm_array_get_handle(a, &ah);
        scm_t_array_dim * as = scm_array_handle_dims(&ah);
        int arank = scm_array_handle_rank(&ah);
        int k = arank;
        ssize_t pos = 0;
        for (; k>0 && scm_is_pair(i); --k, ++as, i=scm_cdr(i)) {
            ssize_t ik = scm_to_ssize_t(scm_car(i));
            if (ik<as->lbnd || ik>as->ubnd) {
                scm_array_handle_release(&ah);
                scm_error_scm(scm_from_locale_symbol("out-of-range"), SCM_BOOL_F,
                              scm_from_locale_string("indices out of range"),
                              SCM_EOL, i_);
            }
            pos += (ik-as->lbnd)*as->inc;
        }
        SCM o;
        if (k>0) {
            if (k==arank) {
                o = a;
            } else {
                o = scm_i_make_array(k);
                SCM_I_ARRAY_V(o) = SCM_I_ARRAY_V(a);
                SCM_I_ARRAY_BASE(o) = pos + SCM_I_ARRAY_BASE(a); // since arank>1.
                scm_t_array_dim * os = SCM_I_ARRAY_DIMS(o);
                for (; k>0; --k, ++as, ++os) {
                    os->ubnd = as->ubnd;
                    os->lbnd = as->lbnd;
                    os->inc = as->inc;
                }
            }
        } else if (scm_is_null(i)) {
            o = scm_array_handle_ref(&ah, pos); // these may be non-arrays.
        } else {
            scm_array_handle_release(&ah);
            scm_error_scm(scm_from_locale_symbol("out-of-range"), SCM_BOOL_F,
                          scm_from_locale_string("too many indices"), SCM_EOL, i_);
        }
        scm_array_handle_release(&ah);
        return o;
    }


> FWIW this is not the case.  Vectors hold SCM objects, whereas uniform
> vectors hold unpacked machine values.

Ok, I think I understand that. Still I don't see where the vref field for the srfi4 types is set. The conversions seem to be done in bytevector.c, can you point this out?

I've also noticed

scheme@(guile-user)> (f64vector-ref #s64(1 2 3) 0)
$1 = #.#
scheme@(guile-user)> (c64vector-ref #f64(1 2 3) 0)
$3 = 1.0+2.0i

(!)

Regards

	Daniel

Re: propose deprecation of generalized-vector-*

[Ludovic Courtès]

Hi!

Andy Wingo <wingo@pobox.com> skribis:

> What is a vector?
>
> Possible answers:
>
>   1. A vector is something that answers #t to vector?, which contains
>   some number of storage slots accessible in a mostly-O(1) way, the
>   number of slots is given by vector-length, and the slots can be
>   accessed with vector-ref and vector-set!.
>
>   2. A vector is a specific kind of object, implementing the interface
>   described above, and disjoint from all other kinds of objects.
>
>   3. A vector is a specific kind of object, as before, disjoint from all
>   other kinds of objects defined in the R5RS.
>
>   4. A vector is a specific kind of object, as before, disjoint from all
>   other kinds of objects defined in the R6RS.
>
> (1) defines vectors as an interface.
>
> (2) defines vectors as a specific data structure.
>
> (3) admits to a number of distinct types that may be vectors, of which
>     one kind is defined by the R5RS.
>
> (4) is like (3), but it precludes bytevectors from being vectors.

I would vote for (2).  Vectors are a specific data structure that has
always (?) been defined in the Scheme reports, so it ought to remain
disjoint IMO.

Guile’s arrays are closer to (1), but with a multi-dimensional
interface.

Ludo’.

Re: propose deprecation of generalized-vector-*

[Andy Wingo]

Hi Daniel,

On Wed 23 Jan 2013 10:06, Andy Wingo <wingo@pobox.com> writes:

> On Tue 22 Jan 2013 15:31, Daniel Llorens <daniel.llorens@bluewin.ch> writes:
>
>> On Jan 21, 2013, at 17:11, Andy Wingo wrote:
>>
>> The patch attached applies over yours and is to document this function
>> and a couple others in the manual.
>
> Thanks, applied and pushed to wip-generalized-vectors.

I'll merge this branch shortly, as there have been no objections.

>> Maybe we should have scm_array_ref_1, scm_array_ref_2, etc. as it is
>> done for some other functions taking rest lists. I can write a patch
>> for that.
>
> For C, that makes sense.  Something should be done for Scheme as well,
> but it's not terribly urgent.  Perhaps make scm_array_ref not be bound
> to "array-ref", and instead bind "array-ref" to some function that takes
> two optional arguments and a rest argument.  A poor man's case-lambda...

I have implemented this in stable-2.0.  It should speed up array
operations from Scheme significantly.

Andy
-- 
http://wingolog.org/

Re: propose deprecation of generalized-vector-*

[Andy Wingo]

Hi,

On Wed 23 Jan 2013 13:20, Daniel Llorens <daniel.llorens@bluewin.ch> writes:

> In [2]: a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
> In [4]: a[1]
> Out[4]: array([4, 5, 6])
> In [5]: a[1, 1]
> Out[5]: 5
>
> array-ref can be extended very simply to do that. It accumulates on the
> position as it is done now, but if the index list comes up short it
> makes a shared array with the remaining axes instead of giving a rank
> error. So it shouldn't be any slower than array_ref.

It could make sense, yes.  What do others think?  What happens for
array-set!?  Care to propose a patch?

>> FWIW this is not the case.  Vectors hold SCM objects, whereas uniform
>> vectors hold unpacked machine values.
>
> Ok, I think I understand that. Still I don't see where the vref field
> for the srfi4 types is set. The conversions seem to be done in
> bytevector.c, can you point this out?

Uniform vectors are just bytevectors.  The only difference between the
different kinds of uniform vectors is a "type" field indicating the
"natural" type of the object.  The type field is set when a bytevector
is created, and is used in the generic array-ref functionality, and in
the printer.

One consequence is that you can (s8vector-ref #u8(#xff) 0) => -1.

> I've also noticed
>
> scheme@(guile-user)> (f64vector-ref #s64(1 2 3) 0)
> $1 = #.#

Here you are interpreting an int64 as a double, which should work, but
this printed result is really bizarre and looks like a bug in our number
printer.  Mark? :)

> scheme@(guile-user)> (c64vector-ref #f64(1 2 3) 0)
> $3 = 1.0+2.0i

This is as expected.  You are aliasing a double vector as a double
complex vector.  These are purely bit-level conversions, not type
casting.

Andy
-- 
http://wingolog.org/

Re: propose deprecation of generalized-vector-*

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 1660 bytes --]

Hello,

On Mon, Feb 18, 2013 at 10:55 AM, Andy Wingo <wingo@pobox.com> wrote:

> Hi,
>
> On Wed 23 Jan 2013 13:20, Daniel Llorens <daniel.llorens@bluewin.ch>
> writes:
>
> > In [2]: a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
> > In [4]: a[1]
> > Out[4]: array([4, 5, 6])
> > In [5]: a[1, 1]
> > Out[5]: 5
> >
> > array-ref can be extended very simply to do that. It accumulates on the
> > position as it is done now, but if the index list comes up short it
> > makes a shared array with the remaining axes instead of giving a rank
> > error. So it shouldn't be any slower than array_ref.
>
> It could make sense, yes.  What do others think?  What happens for
> array-set!?  Care to propose a patch?


I haven't worked with the array functionality, so I might be missing
something, but I don't see why this is natural for array-ref. It breaks the
expectation that array-ref always returns an element of the array. It seems
to be that it might be better to have some other function that returns a
slice of the array, with a one-element array being a special case of its
result. (array-ref could even be implemented in terms of this other
function.)

I think that returning a slice instead of throwing an error would be
natural if we automatically mapped scalar operations over arrays. But we
don't, so an array really does have to be viewed as a very different type
than its components, so this change doesn't make sense to me.

I would be happy to be wrong here, but this just jumped out at me as
something that would be a surprising change in behavior, and possibly lead
to bugs. Does anyone have example code that shows why this makes sense?

Best,
Noah

[-- Attachment #2: Type: text/html, Size: 2325 bytes --]

Re: propose deprecation of generalized-vector-*

[Mike Gran]

From: Noah Lavine <noah.b.lavine@gmail.com>
>Hello,
>>On Wed 23 Jan 2013 13:20, Daniel Llorens <daniel.llorens@bluewin.ch> writes:
>>
>>> In [2]: a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
>>
>>> In [4]: a[1]
>>> Out[4]: array([4, 5, 6])
>>> In [5]: a[1, 1]
>>> Out[5]: 5
>>>
>>> array-ref can be extended very simply to do that. It accumulates on the
>>> position as it is done now, but if the index list comes up short it
>>> makes a shared array with the remaining axes instead of giving a rank
>>> error. So it shouldn't be any slower than array_ref.
>>
>>It could make sense, yes.  What do others think?  What happens for
>>array-set!?  Care to propose a patch?
>
>
> I haven't worked with the array functionality, so I might be missing
> something, but I don't see why this is natural for array-ref. 
 
One could imagine a Matlab-like syntax where array-ref has to have
the same number of indices as the underlying array, but, if an
index were replaced with a symbol, it would return a slice.
 
if np is [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
(array-ref np 1 1) -> 5
(array-ref np 1 :) -> [4, 5, 6]
(array-ref np : 1) -> [[2], [5], [8]]
 
Or maybe that's already in Scheme. I'll admit I've never done matrices
in scheme.
 
-Mike

Re: propose deprecation of generalized-vector-*

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

On Mon, Feb 18, 2013 at 11:25 AM, Mike Gran <spk121@yahoo.com> wrote:

> From: Noah Lavine <noah.b.lavine@gmail.com>
> > I haven't worked with the array functionality, so I might be missing
> > something, but I don't see why this is natural for array-ref.
>
> One could imagine a Matlab-like syntax where array-ref has to have
> the same number of indices as the underlying array, but, if an
> index were replaced with a symbol, it would return a slice.
>
> if np is [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
> (array-ref np 1 1) -> 5
> (array-ref np 1 :) -> [4, 5, 6]
> (array-ref np : 1) -> [[2], [5], [8]]
>

Yes, that would be cool. But I'm not arguing that we shouldn't have some
cool array-slicing functions - I'm just saying that they might belong in a
separate function, not in array-ref.


> Or maybe that's already in Scheme. I'll admit I've never done matrices
> in scheme.
>

As far as I know, there's no standard Scheme way to do arrays.

I haven't done arrays or matrices in Scheme either, so don't take what I
say too seriously. I would like to do it some time, but it'll probably only
happen if I need to do a lot of array programming for work some time. I
think the only way to get the interface right is to have someone write some
serious array-processing software in Guile, and then basically do whatever
they say. :-)

Noah

[-- Attachment #2: Type: text/html, Size: 2093 bytes --]

Re: propose deprecation of generalized-vector-*

[David Pirotte]

Heya,

> > One could imagine a Matlab-like syntax where array-ref has to have
> > the same number of indices as the underlying array, but, if an
> > index were replaced with a symbol, it would return a slice.
> >
> > if np is [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

	just in case it would [one day] matter, in mathlab and octave indices [are
	not offset] start from 1 to N

do we actually have a matrice calculus [offset based the better] lib or binding or
scheme package ? any pointer is welcome

David

Re: propose deprecation of generalized-vector-*

[Mike Gran]

> From: David Pirotte <david@altosw.be>

> do we actually have a matrice calculus [offset based the better] lib or binding 
> or
> scheme package ? any pointer is welcome

There once was a package called guile-num. You can find it here.

www.nongnu.org/guile-num

But, it would take some work to get it to run again.  It hasn't run
since, the 1.6 days, I think.

-Mike

Problems with 'number->string' (was Re: propose deprecation of generalized-vector-*)

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:

> On Wed 23 Jan 2013 13:20, Daniel Llorens <daniel.llorens@bluewin.ch> writes:
>
>> scheme@(guile-user)> (f64vector-ref #s64(1 2 3) 0)
>> $1 = #.#
>
> Here you are interpreting an int64 as a double, which should work, but
> this printed result is really bizarre and looks like a bug in our number
> printer.  Mark? :)

Yes, our number printer is seriously flawed and needs a rewrite.  It
prints subnormal[1] floats as "#.#", and even in typical cases often
fails to print enough digits to get the same number back when you read
it back in.

Note that this also affects compiled code involving numbers, because the
compiler serializes numbers using 'number->string'.  For example,
(* 1e-155 1e-155) returns #f at the REPL, because peval turns this into
a constant which happens to be a subnormal.  During assembly it serializes
this to "#.#", and then 'string->number' returns #f.

Also, 3.14159265358979323846264338327950288419716939937510582097494, if
compiled, fails to produce the float closest to pi.  (acos -1) works
properly, but only because this expression is not currently folded to a
constant by the compiler.

I've already started work on this (based on "Printing Floating-Point
Numbers Quickly and Accurately" by Burger and Dybvig) but got
distracted.

     Mark

[1] http://en.wikipedia.org/wiki/Subnormal_number

Re: propose deprecation of generalized-vector-*

[Daniel Hartwig]

On 19 February 2013 00:25, Mike Gran <spk121@yahoo.com> wrote:
> From: Noah Lavine <noah.b.lavine@gmail.com>
>>Hello,
>>>On Wed 23 Jan 2013 13:20, Daniel Llorens <daniel.llorens@bluewin.ch> writes:
>>>
>>>> In [2]: a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
>>>
>>>> In [4]: a[1]
>>>> Out[4]: array([4, 5, 6])
>>>> In [5]: a[1, 1]
>>>> Out[5]: 5
>>>>
>>>> array-ref can be extended very simply to do that. It accumulates on the
>>>> position as it is done now, but if the index list comes up short it
>>>> makes a shared array with the remaining axes instead of giving a rank
>>>> error. So it shouldn't be any slower than array_ref.
>>>
>>>It could make sense, yes.  What do others think?  What happens for
>>>array-set!?  Care to propose a patch?
>>
>>
>> I haven't worked with the array functionality, so I might be missing
>> something, but I don't see why this is natural for array-ref.

Right, these are my sentiments also (including the snipped part).
Relaxing array-ref in the proposed way means it could hide programming
errors, etc.  Particularly with typed arrays, where previously the
result will reliably be of the given type.

>
> One could imagine a Matlab-like syntax where array-ref has to have
> the same number of indices as the underlying array, but, if an
> index were replaced with a symbol, it would return a slice.
>
> if np is [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
> (array-ref np 1 1) -> 5
> (array-ref np 1 :) -> [4, 5, 6]
> (array-ref np : 1) -> [[2], [5], [8]]
>
> Or maybe that's already in Scheme. I'll admit I've never done matrices
> in scheme.

Yes, but please lets imagine this only and not multiply the purpose of
array-ref.  Other languages do that kind of thing a lot with a sort of
“guess what I really meant to do and do that instead”, which leads to
messy documentation, programming errors, and security issues
(Ruby-on-Rails?).

This is certainly a common enough usage of make-shared-array to
justify its own procedure with more helpful syntax.  Even I would make
this two procedures, array-slice and array-slice/shared, to decide
between new or shared array.

Regards

Re: propose deprecation of generalized-vector-*

[Daniel Llorens]

[-- Attachment #1: Type: text/plain, Size: 5809 bytes --]


On Feb 18, 2013, at 16:55, Andy Wingo wrote:

> It could make sense, yes.  What do others think?  What happens for
> array-set!?  Care to propose a patch?

Patch is attached. It looks a bit unwieldy because I am duplicating scm_array_handle_pos(), and I also had to fix the recent array_ref_1 optimization, but it shouldn't be slower for the old use. 

array-set! does what you would expect, obj must be an array of rank = the number of indices missing.

(define A #2((0 1) (2 3)))
(array-set! A #(7 8) 0)
A => #2((7 8) (2 3))

When all the indices are given, it works exactly as before, so one can still do stupid things like

(define A #2((0 1) (2 3)))
(array-set! A #0(9) 0 0)

and get

A => #2((#0(9) 1) (2 3))

and not

A => #2((9 1) (2 3)).

The lesson is to avoid rank-0 arrays except in the deep of libraries.

arrays.test fails in three places. Twice because I don't fail with the right exception 'wrong-num-indices', I don't know how to do that. Once because (array-set! rank-2 one index) doesn't fail on the missing index, but on the attempted copy. I haven't fixed these tests and I haven't added more.

---

It was interesting to read the other replies, thank you (plural). It was disappointing that nobody seems to be using arrays, but not very surprising. I use them a fair amount, but mostly for interfacing with C/C++. I do little with them in Guile since 1) the facilities are really not there or they are inconvenient to use, 2) when I need arrays I need speed and Guile isn't there either (yet!).

More than getting this patch accepted (as it is or with different function names), my concern is that what this patch does cannot be done on the user side except through make-shared-array, which is too slow. So I think there's a good argument that scm_i_make_array(), SCM_I_ARRAY_V(), SCM_I_ARRAY_BASE() and SCM_I_ARRAY_DIMS() should be published.

Now some optional motivation:

In other languages that have arrays, rank-0 arrays are not differentiated from the scalar inside. This is the right approach if you want to treat arrays of any rank as values. Then you can say that a rank-n array is

-------------------
a rank-n array of rank-0 cells
a rank-(n-1) array of rank-1 cells
...
a rank-1 array of rank-(n-1) cells
a rank-0 array of rank-n cells = a rank-n cell.

Figure 1.
-------------------

The extended versions of array-ref and array-set! allow you to use a rank-n array as any of these.

Old Guile used to have something called ‘enclosed arrays’, they are described in the 1.6 manual. This is a mechanism inspired by APL that allows one to split an array as in Figure 1, so that if you have a function that operates on rank-k arrays, you can array-map! it on arrays of any rank >= k.

Example:

(define A #((1 2) (3 4) (5 6))
(define (sqr x) (* x x))
(define (norm v) (+ (sqr (array-ref v 0)) (sqr (array-ref v 1))))
(define out #(0 0 0))
(array-map! out norm (enclosed-array A 1))

out => #(5 25 61), hopefully (I don't have Guile 1.6 here to try).

In later versions of APL (and most obviously in J) people realized that this is a bad way to go about things, since the reason you want to split A in rank-1 cells is ‘norm’. That is, rank-1 is a property of the function ‘norm’. And you don't need to do any conversions to iterate over the rank-1 cells of A; you only need to move a pointer on each iteration, but to do this array-map! should be aware of the rank of the procedure in each of its arguments.

I didn't want to mess with the C implementation of array-map! for what is only a proof of concept. One can do something passable in Guile by using the version of array-ref in the patch, since

(array-ref A 0) -> #(1 2), apply norm to this
(array-ref A 1) -> #(3 4), apply norm to this, etc.

I have a module that does this, not much more. I'll see if I can upload it somewhere, it could use a review.

Finally, to fuel the discussion of what kind of array operations Guile should provide, here is a list similar things that I have read about. I imagine there are many more. The software is all open source.

Scheme (!)
http://trac.sacrideo.us/wg/wiki/ArraysCowan
This is a proposal for R7RS large, very early-stages.

Python
http://www.scipy.org/Tentative_NumPy_Tutorial

Haskell
http://research.microsoft.com/en-us/um/people/simonpj/papers/ndp/rarrays.pdf
http://www.haskell.org/haskellwiki/Numeric_Haskell:_A_Repa_Tutorial

Perl
http://pdl.perl.org/

J
http://www.jsoftware.com/
http://www.jsoftware.com/help/jforc/contents.htm — I like this one a lot. There're also many classic articles on APL on the jsoftware site.

C++
http://blitz.sourceforge.net/
This one is dying for somebody to port it to C++11, but it does general rank (although compile-time only) and has clever features, e.g. TensorIndex.

There's also Matlab/Octave (bad language, I'm not a fan).

There's Fortran, for which I don't have a good link, but a lot of the array-compiler literature is about Fortran. There's SAC (http://www.sac-home.org/) as an example of a very ‘static’ array language. There're array languages targeting supercomputers, e.g. ZPL (http://www.cs.washington.edu/research/zpl/home/). 

Myself I have used Matlab way too much and numpy some. I have played with the J interpreter.

> Uniform vectors are just bytevectors.  The only difference between the
> different kinds of uniform vectors is a "type" field indicating the
> "natural" type of the object.  The type field is set when a bytevector
> is created, and is used in the generic array-ref functionality, and in
> the printer.
> 
> One consequence is that you can (s8vector-ref #u8(#xff) 0) => -1.

c64 <-> f64 is something that I have found useful myself.

Regards,

	Daniel

------------------


[-- Attachment #2: 0001-Extend-array-ref-array-set-to-work-with-cells-of-ran.patch --]
[-- Type: application/octet-stream, Size: 6874 bytes --]

From f85aaa7dec597b1d45830a3d346b55f3d3cefbff Mon Sep 17 00:00:00 2001
From: Daniel Llorens <daniel.llorens@bluewin.ch>
Date: Wed, 20 Feb 2013 22:44:06 +0100
Subject: [PATCH] Extend array-ref, array-set! to work with cells of rank > 0

* libguile/generalized-arrays.c
  - scm_array_ref: if the index list comes up short, return the cell at the
    position computed up to then.
  - scm_array_set_x: if the index list comes up short, use the cell at the
    position computed up to then as destination for o.
  - s_scm_i_array_ref: branch to the optimization only if rank of v matches.
  - s_scm_i_array_set_x: idem.
---
 libguile/generalized-arrays.c | 127 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 103 insertions(+), 24 deletions(-)

diff --git a/libguile/generalized-arrays.c b/libguile/generalized-arrays.c
index 9382e81..086019f 100644
--- a/libguile/generalized-arrays.c
+++ b/libguile/generalized-arrays.c
@@ -256,17 +256,49 @@ scm_c_array_ref_2 (SCM array, ssize_t idx0, ssize_t idx1)
   return res;
 }
 
-
 SCM
-scm_array_ref (SCM v, SCM args)
+scm_array_ref (SCM a, SCM args)
 {
-  scm_t_array_handle handle;
-  SCM res;
-
-  scm_array_get_handle (v, &handle);
-  res = scm_array_handle_ref (&handle, scm_array_handle_pos (&handle, args));
-  scm_array_handle_release (&handle);
-  return res;
+  int k, arank;
+  ssize_t pos = 0;
+  SCM i = args;
+  SCM o;
+  scm_t_array_handle ah;
+  scm_t_array_dim * as;
+  scm_array_get_handle (a, &ah);
+  as = scm_array_handle_dims (&ah);
+  k = arank = scm_array_handle_rank (&ah);
+  for (; k>0 && scm_is_pair (i); --k, ++as, i=scm_cdr (i)) {
+    ssize_t ik = scm_to_ssize_t (scm_car (i));
+    if (ik<as->lbnd || ik>as->ubnd) {
+      scm_array_handle_release (&ah);
+      scm_out_of_range (NULL, scm_list_2 (a, args));
+    }
+    pos += (ik-as->lbnd)*as->inc;
+  }
+  if (k>0) {
+    if (k==arank) {
+      o = a;
+    } else {
+      scm_t_array_dim * os;
+      o = scm_i_make_array (k);
+      SCM_I_ARRAY_V (o) = SCM_I_ARRAY_V (a);
+      SCM_I_ARRAY_BASE (o) = pos + SCM_I_ARRAY_BASE (a); /* since arank>1. */
+      os = SCM_I_ARRAY_DIMS (o);
+      for (; k>0; --k, ++as, ++os) {
+        os->ubnd = as->ubnd;
+        os->lbnd = as->lbnd;
+        os->inc = as->inc;
+      }
+    }
+  } else if (scm_is_null (i)) {
+    o = scm_array_handle_ref (&ah, pos); /* these may be non-arrays. */
+  } else {
+    scm_array_handle_release (&ah);
+    scm_misc_error (NULL, "too many indices", scm_list_2 (a, args));
+  }
+  scm_array_handle_release (&ah);
+  return o;
 }
 
 
@@ -295,29 +327,70 @@ scm_c_array_set_2_x (SCM array, SCM obj, ssize_t idx0, ssize_t idx1)
 
 
 SCM
-scm_array_set_x (SCM v, SCM obj, SCM args)
+scm_array_set_x (SCM a, SCM o, SCM args)
 {
-  scm_t_array_handle handle;
-
-  scm_array_get_handle (v, &handle);
-  scm_array_handle_set (&handle, scm_array_handle_pos (&handle, args), obj);
-  scm_array_handle_release (&handle);
+  int k, arank;
+  ssize_t pos = 0;
+  SCM i = args;
+  scm_t_array_handle ah;
+  scm_t_array_dim * as;
+  scm_array_get_handle (a, &ah);
+  as = scm_array_handle_dims (&ah);
+  k = arank = scm_array_handle_rank (&ah);
+  for (; k>0 && scm_is_pair (i); --k, ++as, i=scm_cdr (i)) {
+    ssize_t ik = scm_to_ssize_t (scm_car (i));
+    if (ik<as->lbnd || ik>as->ubnd) {
+      scm_array_handle_release (&ah);
+      scm_misc_error (NULL, "out of range", scm_list_2 (a, args));
+    }
+    pos += (ik-as->lbnd)*as->inc;
+  }
+  if (k>0) {
+    SCM ai;
+    if (k==arank) {
+      ai = a;
+    } else {
+      scm_t_array_dim * ais;
+      ai = scm_i_make_array (k);
+      SCM_I_ARRAY_V (ai) = SCM_I_ARRAY_V (a);
+      SCM_I_ARRAY_BASE (ai) = pos + SCM_I_ARRAY_BASE (a); /* since arank>1. */
+      ais = SCM_I_ARRAY_DIMS (ai);
+      for (; k>0; --k, ++as, ++ais) {
+        ais->ubnd = as->ubnd;
+        ais->lbnd = as->lbnd;
+        ais->inc = as->inc;
+      }
+    }
+    /* an error is still possible here if o and ai don't match. */
+    scm_array_copy_x (o, ai);
+  } else if (scm_is_null(i)) {
+    scm_array_handle_set (&ah, pos, o);
+  } else {
+    scm_array_handle_release (&ah);
+    scm_misc_error (NULL, "too many indices", scm_list_2 (a, args));
+  }
   return SCM_UNSPECIFIED;
 }
 
 
 SCM_DEFINE (scm_i_array_ref, "array-ref", 1, 2, 1,
             (SCM v, SCM idx0, SCM idx1, SCM idxN),
-	    "Return the element at the @code{(idx0, idx1, idxN...)}\n"
-            "position in array @var{v}.")
+	    "Return the rank-(n-k) cell at the @code{(idx0 idx1 .. idx(k-1))}\n"
+            "position in n-rank array @var{v}.")
 #define FUNC_NAME s_scm_i_array_ref
 {
   if (SCM_UNBNDP (idx0))
     return scm_array_ref (v, SCM_EOL);
   else if (SCM_UNBNDP (idx1))
-    return scm_c_array_ref_1 (v, scm_to_ssize_t (idx0));
+    if (scm_c_array_rank (v)==1)
+      return scm_c_array_ref_1 (v, scm_to_ssize_t (idx0));
+    else
+      return scm_array_ref (v, scm_cons (idx0, SCM_EOL));
   else if (scm_is_null (idxN))
-    return scm_c_array_ref_2 (v, scm_to_ssize_t (idx0), scm_to_ssize_t (idx1));
+    if (scm_c_array_rank (v)==2)
+      return scm_c_array_ref_2 (v, scm_to_ssize_t (idx0), scm_to_ssize_t (idx1));
+    else
+      return scm_array_ref (v, scm_cons (idx0, scm_cons (idx1, SCM_EOL)));
   else
     return scm_array_ref (v, scm_cons (idx0, scm_cons (idx1, idxN)));
 }
@@ -326,17 +399,23 @@ SCM_DEFINE (scm_i_array_ref, "array-ref", 1, 2, 1,
 
 SCM_DEFINE (scm_i_array_set_x, "array-set!", 2, 2, 1,
             (SCM v, SCM obj, SCM idx0, SCM idx1, SCM idxN),
-	    "Set the element at the @code{(idx0, idx1, idxN...)} position\n"
-	    "in the array @var{v} to @var{obj}.  The value returned by\n"
-            "@code{array-set!} is unspecified.")
+	    "Set the rank-(n-k) cell at the @code{(idx0 idx1 .. idx(k-1))}\n"
+            "position in the n-rank array @var{v} to the k-rank array\n"
+            "@var{obj}.  The value returned is unspecified.")
 #define FUNC_NAME s_scm_i_array_set_x
 {
   if (SCM_UNBNDP (idx0))
     scm_array_set_x (v, obj, SCM_EOL);
   else if (SCM_UNBNDP (idx1))
-    scm_c_array_set_1_x (v, obj, scm_to_ssize_t (idx0));
+    if (scm_c_array_rank (v)==1)
+      scm_c_array_set_1_x (v, obj, scm_to_ssize_t (idx0));
+    else
+      scm_array_set_x (v, obj, scm_cons (idx0, SCM_EOL));
   else if (scm_is_null (idxN))
-    scm_c_array_set_2_x (v, obj, scm_to_ssize_t (idx0), scm_to_ssize_t (idx1));
+    if (scm_c_array_rank (v)==2)
+      scm_c_array_set_2_x (v, obj, scm_to_ssize_t (idx0), scm_to_ssize_t (idx1));
+    else
+      scm_array_set_x (v, obj, scm_cons (idx0, scm_cons (idx1, SCM_EOL)));
   else
     scm_array_set_x (v, obj, scm_cons (idx0, scm_cons (idx1, idxN)));
 
-- 
1.8.1.1


[-- Attachment #3: Type: text/plain, Size: 2 bytes --]

Re: propose deprecation of generalized-vector-*

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 5440 bytes --]

Hello,


On Wed, Feb 20, 2013 at 8:13 PM, Daniel Llorens
<daniel.llorens@bluewin.ch>wrote:

>
> On Feb 18, 2013, at 16:55, Andy Wingo wrote:
>
> > It could make sense, yes.  What do others think?  What happens for
> > array-set!?  Care to propose a patch?
>
> Patch is attached. It looks a bit unwieldy because I am duplicating
> scm_array_handle_pos(), and I also had to fix the recent array_ref_1
> optimization, but it shouldn't be slower for the old use.
>

Thanks for working on this!


> ...
>
> ---
>
> It was interesting to read the other replies, thank you (plural). It was
> disappointing that nobody seems to be using arrays, but not very
> surprising. I use them a fair amount, but mostly for interfacing with
> C/C++. I do little with them in Guile since 1) the facilities are really
> not there or they are inconvenient to use, 2) when I need arrays I need
> speed and Guile isn't there either (yet!).
>

I agree about the speed issue, but I hope it will get better soon. The RTL
VM will fix some of it, and native compilation will fix more.


> More than getting this patch accepted (as it is or with different function
> names), my concern is that what this patch does cannot be done on the user
> side except through make-shared-array, which is too slow. So I think
> there's a good argument that scm_i_make_array(), SCM_I_ARRAY_V(),
> SCM_I_ARRAY_BASE() and SCM_I_ARRAY_DIMS() should be published.
>
> Now some optional motivation:
>
> In other languages that have arrays, rank-0 arrays are not differentiated
> from the scalar inside. This is the right approach if you want to treat
> arrays of any rank as values. Then you can say that a rank-n array is
>
> -------------------
> a rank-n array of rank-0 cells
> a rank-(n-1) array of rank-1 cells
> ...
> a rank-1 array of rank-(n-1) cells
> a rank-0 array of rank-n cells = a rank-n cell.
>
> Figure 1.
> -------------------
>
> The extended versions of array-ref and array-set! allow you to use a
> rank-n array as any of these.
>

I'm actually not very enthusiastic about this, not because you shouldn't be
able to do this, but because in order to enable the automatic de-ranking,
you have to have Guile assume which dimensions you want to map over. That's
how C, C++ and Fortran do it because that's how arrays are actually stored
in memory, so maybe that is the right way. It just seems too low-level for
me - I'd rather see an array-slice function that can split along any
dimensions.


> Old Guile used to have something called ‘enclosed arrays’, they are
> described in the 1.6 manual. This is a mechanism inspired by APL that
> allows one to split an array as in Figure 1, so that if you have a function
> that operates on rank-k arrays, you can array-map! it on arrays of any rank
> >= k.
>
> Example:
>
> (define A #((1 2) (3 4) (5 6))
> (define (sqr x) (* x x))
> (define (norm v) (+ (sqr (array-ref v 0)) (sqr (array-ref v 1))))
> (define out #(0 0 0))
> (array-map! out norm (enclosed-array A 1))
>
> out => #(5 25 61), hopefully (I don't have Guile 1.6 here to try).
>
> In later versions of APL (and most obviously in J) people realized that
> this is a bad way to go about things, since the reason you want to split A
> in rank-1 cells is ‘norm’. That is, rank-1 is a property of the function
> ‘norm’. And you don't need to do any conversions to iterate over the rank-1
> cells of A; you only need to move a pointer on each iteration, but to do
> this array-map! should be aware of the rank of the procedure in each of its
> arguments.
>

This gets at the heart of my issue with the array functionality. As far as
I can tell, in Guile, there is no way to figure out what the rank of a
function is. That's why you have to be explicit about what you're mapping
over.

I suppose the Common Lisp-y approach would be to make an object property
called 'rank', set it for all of the built-in arithmetic functions, and
maybe have some way to infer the rank of new functions That might be
interesting, but I'm skeptical.


> ...
>
> Finally, to fuel the discussion of what kind of array operations Guile
> should provide, here is a list similar things that I have read about. I
> imagine there are many more. The software is all open source.
>
> ...
>

Thanks a lot for starting the conversation. I would like to see Guile
provide enough array functionality for serious scientific computing, and it
sounds like you want the same thing. I don't really know what's missing
yet, though, because I haven't tried to write a program that would use it.

I think the idea of splitting arrays is great. My only concern is making it
part of array-ref. I still think that's a really bad idea, because it
introduces a new class of errors that are really easy to make -
accidentally getting an array when you expected whatever was inside the
array. I'm coming at this as a user of Matlab and Fortran. In those
languages, this isn't a problem, because operations automatically map over
arrays, so having an array where you expected a value doesn't lead to new
errors. But in Scheme, operations *don't* automatically map, so getting an
array could lead to an exception at some point later in a program when
really the error was that you didn't give enough indices to array-ref.

Other than that, I'm excited about having this.

Best,
Noah

[-- Attachment #2: Type: text/html, Size: 6680 bytes --]

Re: propose deprecation of generalized-vector-*

[Daniel Llorens]

On Feb 22, 2013, at 01:22, Noah Lavine wrote:

> I agree about the speed issue, but I hope it will get better soon. The RTL VM will fix some of it, and native compilation will fix more.

That's on Scheme, but there are also many optimization issues related to array operations. Temporaries, order of traversal, etc.

> I'm actually not very enthusiastic about this, not because you shouldn't be able to do this, but because in order to enable the automatic de-ranking, you have to have Guile assume which dimensions you want to map over. That's how C, C++ and Fortran do it because that's how arrays are actually stored in memory, so maybe that is the right way. It just seems too low-level for me - I'd rather see an array-slice function that can split along any dimensions.

enclosed-array also let you pick what axes you wanted for the cell. You needed to specify those axes every time. That feels /more/ low level to me.

The memory order of arrays in Guile is absolutely low level detail, especially since it can change at any time. However the ¿logical? order of the axes is not. It's simpler to define the looping operation so that the frame (the axes one loops over) consists of the axes that come first. It plays well with the rank extension / matching mechanism that I show at the end and with the view of an array as a list.

APL used to have an 'axis specifier' that was apparently a total mess. In numpy there are some functions with an 'axes' argument, but it's 'axis' for others and some don't have it.

I'm not opposed to being able to choose your axes in some legacy-friendly standard way, I just don't think it's a problem. You can always loop over other axes at no cost by transposing before and after. It's something that can be easily abstracted.

> This gets at the heart of my issue with the array functionality. As far as I can tell, in Guile, there is no way to figure out what the rank of a function is. That's why you have to be explicit about what you're mapping over.
> 
> I suppose the Common Lisp-y approach would be to make an object property called 'rank', set it for all of the built-in arithmetic functions, and maybe have some way to infer the rank of new functions That might be interesting, but I'm skeptical.

Exactly, this is what is needed. Then you can write array functions that can be extended for arguments of higher rank without the function itself having to deal with those extra axes that are none of its concern. Otherwise you need to give axis indications left and right. I've suffered this in numpy. This information belongs with the function.

> Thanks a lot for starting the conversation. I would like to see Guile provide enough array functionality for serious scientific computing, and it sounds like you want the same thing. I don't really know what's missing yet, though, because I haven't tried to write a program that would use it.

It's a problem, because one needs at the very least mapping and reductions to write any kind of numeric program. Guile has absolutely nothing for array reductions and the mapping is very low level.

> I think the idea of splitting arrays is great. My only concern is making it part of array-ref. I still think that's a really bad idea, because it introduces a new class of errors that are really easy to make - accidentally getting an array when you expected whatever was inside the array. I'm coming at this as a user of Matlab and Fortran. In those languages, this isn't a problem, because operations automatically map over arrays, so having an array where you expected a value doesn't lead to new errors. But in Scheme, operations *don't* automatically map, so getting an array could lead to an exception at some point later in a program when really the error was that you didn't give enough indices to array-ref.

In my experience the kind of rank errors you describe are unlikely to happen, because in most programs the ranks of arrays are static. It's a bit like function arity, the general case is important and must be supported, but most functions have fixed arity, and that reveals many optimization opportunities. If the rank of an array is known, then the rank of the array-ref result is also known. The Guile compiler seems to ignore all of this right now, but it probably shouldn't.

I've implemented the idea of assigning rank to functions and then extending these over arrays of higher rank. At this point I'm mostly interested in having the basic mechanism right, so the code is probably a bit rough.

I wrote some description of how it works in the README. Please have a look and let me know what you think. You can find it at:

	https://gitorious.org/guile-ploy

I also wanted to write a bit about passing arrays between C/C++ and Guile, but it's really a different matter, so maybe some other time. The problem here is that each library has its own calling convention and has different constraints on the kind of arrays it takes, so it's not something that the ffi can handle transparently.

Regards,
	
	Daniel

Re: propose deprecation of generalized-vector-*

[Nelson H. F. Beebe]

If guile introduces full-fledged support of arrays for numeric
computing, and for communicating with external libraries, such as the
currently-moribund Guile-numerics interface to GNU Scientific Library
(GSL), libsndfile, FFTW, and LAPACK

	http://www.nongnu.org/guile-num/

please do not forget, as higher-level language programmers too often
do, that array storage order matters a GREAT DEAL for efficiency.

Modern processors operate in registers up to 500x faster than on
(DRAM) memory, and cache (SRAM) is about 10x to 50x faster than DRAM.
Thus, if you use Fortran column-major order (first subscript varying
most rapidly), it can be MUCH faster to compute down the columns than
to compute across rows.  The C/C++ languages use row-major order (last
subscript varying most rapidly), so for them, the situation is
reversed: row traversals are preferred over column traversals.

Java really screwed things up by using an arrays-of-arrays-of-...
storage model, so MxN rectangular matrix data are spread over M or N
separate contiguous vectors allocated at arbitrary locations in
memory.  With that arrangement, and Java's requirement that array
accesses must be accompanied by mandatory bounds checks, it is
difficult to make good use of cache.  That serious deficiency is why
IBM introduced new contiguous-array classes for Java, and why the
(seemingly-now-defunct) Java Grande project was started:

	http://www.javagrande.org/
	http://en.wikipedia.org/wiki/Criticism_of_Java

The bounds-checking overhead can be reduced at a lower level: see

	Implicit array bounds checking on 64-bit architectures
	http://doi.acm.org/10.1145/1187976.1187982

For any scripting language that wishes to communicate array data with
external libraries written in other languages, it seems desirable to
offer support for remapping of (possibly-associative) arrays in the
scripting language to contiguous arrays in both row-major and
column-major order, with user-specified initializers for unset
elements.  That way, huge amounts of library code in Fortran, C, and
C++ become accessible.  The reverse mappings back from the external
library to the scripting language are also needed. If arrays are
handled with some generality in the latter, the library return might
not even require copying any numerical data, just updating some
metadata that record how array elements are to be found.

The GNU gawk developers are currently working on similar extensions to
the code world outside the scripting language; it would make sense for
both gawk and guile groups to be aware of the efforts of the other:

	https://lists.gnu.org/mailman/listinfo/gawk-devel

-------------------------------------------------------------------------------
- Nelson H. F. Beebe                    Tel: +1 801 581 5254                  -
- University of Utah                    FAX: +1 801 581 4148                  -
- Department of Mathematics, 110 LCB    Internet e-mail: beebe@math.utah.edu  -
- 155 S 1400 E RM 233                       beebe@acm.org  beebe@computer.org -
- Salt Lake City, UT 84112-0090, USA    URL: http://www.math.utah.edu/~beebe/ -
-------------------------------------------------------------------------------

Re: propose deprecation of generalized-vector-*

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 7354 bytes --]

Hello,


On Thu, Feb 28, 2013 at 2:10 PM, Daniel Llorens
<daniel.llorens@bluewin.ch>wrote:

>
> On Feb 22, 2013, at 01:22, Noah Lavine wrote:
>
> > I agree about the speed issue, but I hope it will get better soon. The
> RTL VM will fix some of it, and native compilation will fix more.
>
> That's on Scheme, but there are also many optimization issues related to
> array operations. Temporaries, order of traversal, etc.
>

Yes, you're right. This will be a long-term project.


> > I'm actually not very enthusiastic about this, not because you shouldn't
> be able to do this, but because in order to enable the automatic
> de-ranking, you have to have Guile assume which dimensions you want to map
> over. That's how C, C++ and Fortran do it because that's how arrays are
> actually stored in memory, so maybe that is the right way. It just seems
> too low-level for me - I'd rather see an array-slice function that can
> split along any dimensions.
>
> enclosed-array also let you pick what axes you wanted for the cell. You
> needed to specify those axes every time. That feels /more/ low level to me.
>
> The memory order of arrays in Guile is absolutely low level detail,
> especially since it can change at any time. However the ¿logical? order of
> the axes is not. It's simpler to define the looping operation so that the
> frame (the axes one loops over) consists of the axes that come first. It
> plays well with the rank extension / matching mechanism that I show at the
> end and with the view of an array as a list.
>

Yes, I think you've persuaded me that there is a "natural" order to axes.
There should still be an operator that splits in other ways, but I agree
that we can shortcut that in many cases.


> > This gets at the heart of my issue with the array functionality. As far
> as I can tell, in Guile, there is no way to figure out what the rank of a
> function is. That's why you have to be explicit about what you're mapping
> over.
> >
> > I suppose the Common Lisp-y approach would be to make an object property
> called 'rank', set it for all of the built-in arithmetic functions, and
> maybe have some way to infer the rank of new functions That might be
> interesting, but I'm skeptical.
>
> Exactly, this is what is needed. Then you can write array functions that
> can be extended for arguments of higher rank without the function itself
> having to deal with those extra axes that are none of its concern.
> Otherwise you need to give axis indications left and right. I've suffered
> this in numpy. This information belongs with the function.
>

(I'll reply to this below)


> > Thanks a lot for starting the conversation. I would like to see Guile
> provide enough array functionality for serious scientific computing, and it
> sounds like you want the same thing. I don't really know what's missing
> yet, though, because I haven't tried to write a program that would use it.
>
> It's a problem, because one needs at the very least mapping and reductions
> to write any kind of numeric program. Guile has absolutely nothing for
> array reductions and the mapping is very low level.
>

A slow array reduction is easy enough to add, but I'm guessing that's not
what we need. :-) Perhaps we should have some sort of (ice-9 array) module
where we put useful array functions.


> > I think the idea of splitting arrays is great. My only concern is making
> it part of array-ref. I still think that's a really bad idea, because it
> introduces a new class of errors that are really easy to make -
> accidentally getting an array when you expected whatever was inside the
> array. I'm coming at this as a user of Matlab and Fortran. In those
> languages, this isn't a problem, because operations automatically map over
> arrays, so having an array where you expected a value doesn't lead to new
> errors. But in Scheme, operations *don't* automatically map, so getting an
> array could lead to an exception at some point later in a program when
> really the error was that you didn't give enough indices to array-ref.
>
> In my experience the kind of rank errors you describe are unlikely to
> happen, because in most programs the ranks of arrays are static.


That's a good point. I'm still not convinced, because making array-ref do
this means that array-ref can return two fundamentally different types of
results - arrays and other objects. This is very different than most
functions. But I think this comes down to a more fundamental difference - I
still don't think that functions should automatically map over arrays, and
you do. If they did automatically map, then I would agree with you about
array-ref, because then arrays wouldn't be fundamentally different types
from the objects they contained.


> It's a bit like function arity, the general case is important and must be
> supported, but most functions have fixed arity, and that reveals many
> optimization opportunities. If the rank of an array is known, then the rank
> of the array-ref result is also known. The Guile compiler seems to ignore
> all of this right now, but it probably shouldn't.
>

I agree that the compiler should be better, but as one of the people
working on it, there are lots of things that it should do and presently
doesn't. I don't know when we'll get nice array optimizations.


> I've implemented the idea of assigning rank to functions and then
> extending these over arrays of higher rank. At this point I'm mostly
> interested in having the basic mechanism right, so the code is probably a
> bit rough.
>
> I wrote some description of how it works in the README. Please have a look
> and let me know what you think. You can find it at:
>
>         https://gitorious.org/guile-ploy
>

I read through your README. I still haven't looked at the code, but that
looks very cool! I would be excited to have a library like that in Guile -
but I think that this should be optional, and that not *every* function
should have rank information. This is because while it is fairly natural
for programs that involve a lot of array processing, I don't think it is as
natural for, for example, networking code or the web server. I really like
the ply function that lets you connect functions with rank information to
functions without it.


> I also wanted to write a bit about passing arrays between C/C++ and Guile,
> but it's really a different matter, so maybe some other time. The problem
> here is that each library has its own calling convention and has different
> constraints on the kind of arrays it takes, so it's not something that the
> ffi can handle transparently.
>

Yes, that seems like a big issue.

I definitely agree that we should provide enough primitive operators to
write fast array code in Guile. Your README says that certain functions
can't be implemented efficiently without access to the underlying array
representation - that should certainly change. However, I don't think that
we should make every function have rank information, when it's not really
used in most areas of programming. I think the library you've presented is
a great compromise, because it lets you put rank annotations on some
functions, but not all.

What do you think?

Best,
Noah

[-- Attachment #2: Type: text/html, Size: 9062 bytes --]

Re: propose deprecation of generalized-vector-*

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 8218 bytes --]

After reading through my email, I realize that I left out the most
important part - a reason *why* functions shouldn't map over arrays.

My reason is that this makes function application a more complicated
process. Right now, functions are very simple things in Scheme. If we add
automatic mapping behavior, the mental model becomes more complicated. It
seems cleaner to leave this in a module that users could load or not as
they choose.

However, I realize that this argument isn't very persuasive. I am open to
other reasons to make this work.

Best,
Noah



On Thu, Feb 28, 2013 at 9:42 PM, Noah Lavine <noah.b.lavine@gmail.com>wrote:

> Hello,
>
>
> On Thu, Feb 28, 2013 at 2:10 PM, Daniel Llorens <daniel.llorens@bluewin.ch
> > wrote:
>
>>
>> On Feb 22, 2013, at 01:22, Noah Lavine wrote:
>>
>> > I agree about the speed issue, but I hope it will get better soon. The
>> RTL VM will fix some of it, and native compilation will fix more.
>>
>> That's on Scheme, but there are also many optimization issues related to
>> array operations. Temporaries, order of traversal, etc.
>>
>
> Yes, you're right. This will be a long-term project.
>
>
>> > I'm actually not very enthusiastic about this, not because you
>> shouldn't be able to do this, but because in order to enable the automatic
>> de-ranking, you have to have Guile assume which dimensions you want to map
>> over. That's how C, C++ and Fortran do it because that's how arrays are
>> actually stored in memory, so maybe that is the right way. It just seems
>> too low-level for me - I'd rather see an array-slice function that can
>> split along any dimensions.
>>
>> enclosed-array also let you pick what axes you wanted for the cell. You
>> needed to specify those axes every time. That feels /more/ low level to me.
>>
>> The memory order of arrays in Guile is absolutely low level detail,
>> especially since it can change at any time. However the ¿logical? order of
>> the axes is not. It's simpler to define the looping operation so that the
>> frame (the axes one loops over) consists of the axes that come first. It
>> plays well with the rank extension / matching mechanism that I show at the
>> end and with the view of an array as a list.
>>
>
> Yes, I think you've persuaded me that there is a "natural" order to axes.
> There should still be an operator that splits in other ways, but I agree
> that we can shortcut that in many cases.
>
>
>> > This gets at the heart of my issue with the array functionality. As far
>> as I can tell, in Guile, there is no way to figure out what the rank of a
>> function is. That's why you have to be explicit about what you're mapping
>> over.
>> >
>> > I suppose the Common Lisp-y approach would be to make an object
>> property called 'rank', set it for all of the built-in arithmetic
>> functions, and maybe have some way to infer the rank of new functions That
>> might be interesting, but I'm skeptical.
>>
>> Exactly, this is what is needed. Then you can write array functions that
>> can be extended for arguments of higher rank without the function itself
>> having to deal with those extra axes that are none of its concern.
>> Otherwise you need to give axis indications left and right. I've suffered
>> this in numpy. This information belongs with the function.
>>
>
> (I'll reply to this below)
>
>
>> > Thanks a lot for starting the conversation. I would like to see Guile
>> provide enough array functionality for serious scientific computing, and it
>> sounds like you want the same thing. I don't really know what's missing
>> yet, though, because I haven't tried to write a program that would use it.
>>
>> It's a problem, because one needs at the very least mapping and
>> reductions to write any kind of numeric program. Guile has absolutely
>> nothing for array reductions and the mapping is very low level.
>>
>
> A slow array reduction is easy enough to add, but I'm guessing that's not
> what we need. :-) Perhaps we should have some sort of (ice-9 array) module
> where we put useful array functions.
>
>
>> > I think the idea of splitting arrays is great. My only concern is
>> making it part of array-ref. I still think that's a really bad idea,
>> because it introduces a new class of errors that are really easy to make -
>> accidentally getting an array when you expected whatever was inside the
>> array. I'm coming at this as a user of Matlab and Fortran. In those
>> languages, this isn't a problem, because operations automatically map over
>> arrays, so having an array where you expected a value doesn't lead to new
>> errors. But in Scheme, operations *don't* automatically map, so getting an
>> array could lead to an exception at some point later in a program when
>> really the error was that you didn't give enough indices to array-ref.
>>
>> In my experience the kind of rank errors you describe are unlikely to
>> happen, because in most programs the ranks of arrays are static.
>
>
> That's a good point. I'm still not convinced, because making array-ref do
> this means that array-ref can return two fundamentally different types of
> results - arrays and other objects. This is very different than most
> functions. But I think this comes down to a more fundamental difference - I
> still don't think that functions should automatically map over arrays, and
> you do. If they did automatically map, then I would agree with you about
> array-ref, because then arrays wouldn't be fundamentally different types
> from the objects they contained.
>
>
>> It's a bit like function arity, the general case is important and must be
>> supported, but most functions have fixed arity, and that reveals many
>> optimization opportunities. If the rank of an array is known, then the rank
>> of the array-ref result is also known. The Guile compiler seems to ignore
>> all of this right now, but it probably shouldn't.
>>
>
> I agree that the compiler should be better, but as one of the people
> working on it, there are lots of things that it should do and presently
> doesn't. I don't know when we'll get nice array optimizations.
>
>
>> I've implemented the idea of assigning rank to functions and then
>> extending these over arrays of higher rank. At this point I'm mostly
>> interested in having the basic mechanism right, so the code is probably a
>> bit rough.
>>
>> I wrote some description of how it works in the README. Please have a
>> look and let me know what you think. You can find it at:
>>
>>         https://gitorious.org/guile-ploy
>>
>
> I read through your README. I still haven't looked at the code, but that
> looks very cool! I would be excited to have a library like that in Guile -
> but I think that this should be optional, and that not *every* function
> should have rank information. This is because while it is fairly natural
> for programs that involve a lot of array processing, I don't think it is as
> natural for, for example, networking code or the web server. I really like
> the ply function that lets you connect functions with rank information to
> functions without it.
>
>
>> I also wanted to write a bit about passing arrays between C/C++ and
>> Guile, but it's really a different matter, so maybe some other time. The
>> problem here is that each library has its own calling convention and has
>> different constraints on the kind of arrays it takes, so it's not something
>> that the ffi can handle transparently.
>>
>
> Yes, that seems like a big issue.
>
> I definitely agree that we should provide enough primitive operators to
> write fast array code in Guile. Your README says that certain functions
> can't be implemented efficiently without access to the underlying array
> representation - that should certainly change. However, I don't think that
> we should make every function have rank information, when it's not really
> used in most areas of programming. I think the library you've presented is
> a great compromise, because it lets you put rank annotations on some
> functions, but not all.
>
> What do you think?
>
> Best,
> Noah
>

[-- Attachment #2: Type: text/html, Size: 10315 bytes --]

Re: propose deprecation of generalized-vector-*

[Daniel Llorens]

On Mar 1, 2013, at 03:42, Noah Lavine wrote:

> There should still be an operator that splits in other ways, but I agree that we can shortcut that in many cases.

One think I like about the frame / cell split is that you know that the result will have the same frame. So I imagine an operator

(do-with-frame array-op (arg0 2 1) (arg1 ...) ...)

that transposes 2 1 to  0 1 before the array op, and when the array op is done you know that the output shape will be 2 1 ... axes of the output cell ...  However you still can't transpose the frame back without knowing what the array op did to the cell. There're specific types of operations where this works because the frame would be gone (reductions, or any time the output rank is 0) or it would be unaffected (rank 1 to rank 1 operations, like 'shuffle this axis'), but not in general. That's one reason why the axes specification is ad hoc in those other languages. If array functions know their ranks, we could make this work automatically often enough, but sometimes you'd still have to say 'I don't know how to do that'.

> A slow array reduction is easy enough to add, but I'm guessing that's not what we need. :-) Perhaps we should have some sort of (ice-9 array) module where we put useful array functions.

That's a good idea. I have my own slow reduction and I imagine that the expedient vector->list / array->list and back is done way too often.

> think this comes down to a more fundamental difference - I still don't think that functions should automatically map over arrays, and you do. If they did automatically map, then I would agree with you about array-ref, because then arrays wouldn't be fundamentally different types from the objects they contained.

I actually agree here! I don't want regular scheme functions to have things done to them around their back, it would be another language. I can accept why you want array-ref to be strict. Indeed my approach tends to a confusion between a 2-array of 2-arrays and a 4-array. In guile-ploy you can see this in collapse-array ---if the verb doesn't provide an output shape, I make an assumption. I also banish 0-rank arrays.

The idea is to make special array-functions to get the array behavior. Like ufuncs in NumPy.

> I agree that the compiler should be better, but as one of the people working on it, there are lots of things that it should do and presently doesn't. I don't know when we'll get nice array optimizations.

Guile has gotten a lot better recently. I wish I had a clue about compilers though.

> library like that in Guile - but I think that this should be optional, and that not *every* function should have rank information. This is because while it is fairly natural for programs that involve a lot of array processing, I don't think it is as natural for, for example, networking code or the web server.

I totally agree with this.

> I really like the ply function that lets you connect functions with rank information to functions without it.

My idea was to define +., -., etc. as array versions of +, -, and so on. Then when the library was mature enough, (+. ...) would be transformed into (ply +. ...). However I've fallen into the vice of doing things on the spot, like (ply (make-verb + ...) ...). There's a lot of this in the test script.

> I definitely agree that we should provide enough primitive operators to write fast array code in Guile. Your README says that certain functions can't be implemented efficiently without access to the underlying array representation - that should certainly change.

I've thought about what could be provided on the Scheme side. The problem is that if you give strides / lbnd / ubnd directly, you still need to create a bunch of lists for any slicing, although maybe not as many as with make-shared-array. This is worse for ops that are going to be used inside loops a lot, like array-ref. The recent scm_c_array_ref_1 functions help, but it's just a specific case. All array operations are in C, maybe that's why they are opaque? I mean

scheme@(guile-user)> ,optimize (vector-ref #(1 2 3) 0)
$1 = 1
scheme@(guile-user)> ,optimize (array-ref #(1 2 3) 0)
$2 = (array-ref '#(1 2 3) 0)

> However, I don't think that we should make every function have rank information, when it's not really used in most areas of programming. I think the library you've presented is a great compromise, because it lets you put rank annotations on some functions, but not all.
> 
> What do you think?

Agreed, I'm sorry I wasn't clear in the other messages. This array extension should be a library. What I'd like from Guile is the facilities to allow it to be efficient.

Best regards,
	
	Daniel

Re: propose deprecation of generalized-vector-*

[Andy Wingo]

On Fri 01 Mar 2013 10:01, Daniel Llorens <daniel.llorens@bluewin.ch> writes:

> scheme@(guile-user)> ,optimize (vector-ref #(1 2 3) 0)
> $1 = 1
> scheme@(guile-user)> ,optimize (array-ref #(1 2 3) 0)
> $2 = (array-ref '#(1 2 3) 0)

File a bug for this case, this sort of thing is totally fixable :)

> What I'd like from Guile is the facilities to allow [array extensions]
> to be efficient.

Very much agreed; do keep pushing on this, and I think we should be able
to come to something good with time.

Andy
-- 
http://wingolog.org/

Re: propose deprecation of generalized-vector-*

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

Hello again,


On Fri, Mar 1, 2013 at 4:01 AM, Daniel Llorens <daniel.llorens@bluewin.ch>wrote:

> > think this comes down to a more fundamental difference - I still don't
> think that functions should automatically map over arrays, and you do. If
> they did automatically map, then I would agree with you about array-ref,
> because then arrays wouldn't be fundamentally different types from the
> objects they contained.
>
> I actually agree here! I don't want regular scheme functions to have
> things done to them around their back, it would be another language. I can
> accept why you want array-ref to be strict. Indeed my approach tends to a
> confusion between a 2-array of 2-arrays and a 4-array. In guile-ploy you
> can see this in collapse-array ---if the verb doesn't provide an output
> shape, I make an assumption. I also banish 0-rank arrays.
>

It seems that I misunderstood you then, and I apologize. I am very excited
about the library you are proposing, and I would be happy to help in any
way I can (as long as I have time...)!

(I'm snipping the rest of your message because it needs more thought than I
can give it right now.)


> Best regards,
>
>         Daniel
>
>
Best,
Noah

[-- Attachment #2: Type: text/html, Size: 1790 bytes --]

Re: propose deprecation of generalized-vector-*

[Aharon Robbins]

Hello All.

Nelson wrote:

> If guile introduces full-fledged support of arrays for numeric
> computing, and for communicating with external libraries, ....

About which I don't have much comment. He then wrote:

> The GNU gawk developers are currently working on similar extensions to
> the code world outside the scripting language; it would make sense for
> both gawk and guile groups to be aware of the efforts of the other:
>
> 	https://lists.gnu.org/mailman/listinfo/gawk-devel

I wanted to clarify that this is a private mailing list, and is likely
to remain so.  However, documentation of the API is available at

	http://www.skeeve.com/api-chapter.pdf

And of course, the texinfo source as well as all the code are available
in the gawk git repo on savannah, in the 'master' branch.

The above PDF has the full chapter (about 50 pages) explaining the API
and how to use it, including examples, and about 5 pages from an appendix
discussing the design decisions.

The API is intended for interfacing with awk programs, and not as a general
mechanism for any and all scripting languages.  Nonetheless, if it's useful
for the guile developers to see, that's great, and if anyone has comments
about it, they may be sent to bug-gawk@gnu.org.

Thanks,

Arnold

array operations

[Daniel Llorens]

On Mar 4, 2013, at 03:27, Noah Lavine wrote:

> (I'm snipping the rest of your message because it needs more thought than I can give it right now.)

That message contained a number of errors, the consequence of talking before walking, I suppose.

I have added a simple reduction operator to the library, enough to define an inner product operation:

(define* (dot + * A B #:key type)
  "dot + * A B
   Inner product between the last axis of A and the first of B."
  (let ((type (or type (array-type* A))))
    (ply/type type
              (w/rank (Jv (cut folda/type type (lambda (c a b) (+ c (* a b))) 0 <> <>) '_ '_)
                      1 '_)
              A B)))

There are three loops in here:

---the outer loop (ply) over the axes of A, save the last (whence the arg 1 to w/rank)
---the middle loop (folda) over the last of A and the first of B (the axes being reduced)
---the inner loop (ply) over the axes of B, save the first —--this is because folda applies its op (lambda (c a b) ...) as an array op.

If A & B are both rank 2, this is like 'ikj' in §1.1.11 of Golub & Van Loan.

For example:

(define a (i. 300 100))
(define b (i. 100 200))
(define af (array-copy 'f64 a))
(define bf (array-copy 'f64 b))
(define as (array-copy 's64 a))
(define bs (array-copy 's64 b))

> ,time (dot + * a b)
$19 = #
;; 4.836000s real time, 4.870000s run time.  0.350000s spent in GC.
> ,time (dot + * af bf)
$20 = #
;; 7.166000s real time, 7.210000s run time.  0.690000s spent in GC.
> ,time (dot + * as bs)
$21 = #
;; 10.218000s real time, 10.280000s run time.  0.240000s spent in GC.

So it's horribly slow, and the differences between types are very large. Still, this shows that it's spending a significant fraction of the time in  arithmetic, or at least not in loop overhead (I think?)

The profiler output for the type #t version is

> ,profile (dot + * a b)
%     cumulative   self             
time   seconds     seconds      name
 71.43      5.07      5.07  #<procedure 11d767ab0 at util/reduce.scm:87:47 (c a b)>
  7.14      1.01      0.51  map
  7.14      0.51      0.51  =
  7.14      0.51      0.51  %after-gc-thunk
  7.14      0.51      0.51  #<procedure 11c52ff90 at util/ploy.scm:165:16 i>
  0.00      7.10      0.00  #<procedure 1163083a0 at ice-9/top-repl.scm:66:5 ()>
  0.00      7.10      0.00  array-index-map!
  0.00      7.10      0.00  call-with-prompt
  0.00      7.10      0.00  folda/type
  0.00      7.10      0.00  apply-smob/1
  0.00      7.10      0.00  catch
  0.00      7.10      0.00  start-repl
  0.00      7.10      0.00  run-repl
  0.00      7.10      0.00  ply/type
  0.00      7.10      0.00  statprof
  0.00      7.10      0.00  #<procedure 1163085a0 at ice-9/top-repl.scm:31:6 (thunk)>
  0.00      7.10      0.00  #<procedure 1189d1d20 at util/ploy.scm:121:5 i>
  0.00      7.10      0.00  #<procedure 1189cf210 at statprof.scm:655:4 ()>
  0.00      5.07      0.00  array-map!
  0.00      1.52      0.00  nested-op-frames
  0.00      0.51      0.00  nested-match-frame
  0.00      0.51      0.00  array-map/frame!
  0.00      0.51      0.00  length=?
  0.00      0.51      0.00  make-shared-array
---
Sample count: 14
Total time: 7.1 seconds (0.22 seconds in GC)

The top entry is (lambda (c a b) ...) in the definition of dot above. The map comes from the definition of folda/type (the middle loop).

(define (folda/type type op_ z . a)
  (if (null? a)
    z
    (let ((op (if (verb? op_) op_ (make-verb op_ #f #f))))
      (let ((end (tally (car a))))
        (let loop ((i 0) (c z))
          (if (< i end)
; @todo factor out frame matching from repeated application of ply
            (loop (+ 1 i) (apply ply/type type op c (map (cut from_ <> i) a)))
            c))))))                                   
                                                      ^
   here -----------------------------------------------

There's no map in the inner loop since the op has rank 0, so it resolves to array-map!.

------------

As a reference point,

> ,time (blas-dgemm a b 0. 'no 'no )
$28 = #
;; 0.009000s real time, 0.000000s run time.  0.000000s spent in GC.
> ,time (blas-dgemm af bf 0. 'no 'no )
$29 = #
;; 0.002000s real time, 0.000000s run time.  0.000000s spent in GC.

is a few 1000 times faster. It takes more than 3 times longer to convert from scm to f64 than to multiply... lots of room for improvement.

------------


1) The variable rank/arity code is by necessity full of apply / map / rest args. However, in practice, it will be used most of the time with rank 1 / 2 / maybe 3. It's not acceptable to have  this apply-map business going on in an inner loop, but I don't want to write rank-specific or arity-specific variants by hand. I could try to use syntax-rules' ... for the fixed rank/arity functions and select with case-lambda, but that still requires that I write two versions of nearly every function in the library. Is there a way to avoid this? It occurred to me that the partial evaluator could help, but I'm not sure how. Or maybe somebody has a meta-macro that takes the version using ... and converts it to the variable rank/arity version or viceversa.

2) Iterating over an array is a pain. I had this issue before when mapping over subarrays, so I wrote an array-map/frame!. If I write an array-for-each/frame

(let ((c z))
  (apply array-for-each/frame f (lambda a (set! c (apply ply/type type op c a))) a)
  c)

at least this would remove the map-from in the scalar case. It's something... still, carrying c outside it's a bit ugly.

3) is it normal that the arithmetic is so slow? the code above generates tons of temporaries and has a ton of overhead and it still manages to spend most of its time in the + * op.

Regards,

	Daniel