Re: guile-user Digest, Vol 193, Issue 30

Re: guile-user Digest, Vol 193, Issue 30

[Zelphir Kaltstahl]

On 26.12.18 18:00, guile-user-request@gnu.org wrote:
> Hello,
>
>> Is there some library, that enables high performance matrix operations
>> or even n-dimensional array operations? I am thinking of something like
>> NumPy in the Python ecosystem. I think NumPy in turn also uses some
>> lower level thing to do what it does. I think OpenBLAS or MKL, depending
>> on the architecture. I wonder if there is any wrapper around OpenBLAS
>> for Guile or something similar.
>> ...
> There are quite a few libs you might look at before you start to code your own:
>
> 	https://notabug.org/lloda
> 		...
> 		guile-newra
> 		guile-ffi-cblas
> 		...
> 		guile-ploy
> 		...
>
> 	AIscm
> 	http://wedesoft.github.io/aiscm
>
> I also wrote a set of n-dim array math ops (including matrix ops) for Guile-CV:
>
> 	https://www.gnu.org/software/guile-cv/
> 		the manual is available online, math and matrix ops are described in
> 		the 'Processing' section
>
> 	mentioning Guile-CV because images are just a collection of n-dim arrays.
> 	(see 'Image Structure and Accessors' in the manual).
>
> 	in Guile-CV, all (almost all) arrays are f32vectors - see (srfi srfi-4) in
> 	the Guile manual - math and matrix ops are written in C (but memory
> 	allocation entirely kept in scheme). multi channel ops are multi-threaded
>
> 	some of these math and matrix ops also work on s32 and f64 (undocumented so
> 	far): it would not take much to actually fully generalize all guile-cv math
> 	and matrix ops for the full set of (srfi srfi-4 ) types.
>
> Cheers,
> David

Thanks David,

I've added these to my notes for further investigation : )

In SciPy and if I am not mistaken in the Python bindings for OpenCV they
also use NumPy ndarray to represent images and it feels quite natural,
so it makes sense to me to use n-dimensional arrays for this.

It seems the "uniform numeric vectors" of Guile are an implementation of
the vectors in SRFI-4 as per:
https://www.gnu.org/software/guile/manual/html_node/Uniform-Numeric-Vectors.html#Uniform-Numeric-Vectors.

So far I have a very naive but working matrix operation implementation
here (WIP):

https://gitlab.com/ZelphirKaltstahl/neural-network/blob/dev/matrix.scm

(I know, it's not something no one has ever seen before :D)

I think, once I have a good idea about how to use a fitting library or
how to interface with a library of another programming language, I
should be able to switch how the data abstraction procedures work and
should directly gain the speed advantage for that. It is possible I
should structure my code more, for example by putting the tests in a
separate file etc. Or I should add another layer of abstraction between
the matrix operations and the data abstraction layer, maybe some kind of
"package of data abstraction" that I give the program as a parameter or
something. For now all that it is, is a hacky thing, with which I am
trying to reimplement some Python tutorial for neural networks, trying
to use matrix multiplication at the right places, after understanding
what those places are.

Yesterday I tried to use my transpose procedure and it turns out I
sooner run out of RAM (8GB) than the thing slows down. Once I run out of
RAM of course my machine hangs and I need to try to squeeze in some
Ctrl+C etc. Transposing a 1000x1000 was no issue, but going up to
1000x10000 exploded in my face :D Well, transposing is not the most
computationally expensive operation, so not a good measure and as I read
in the NumPy internals, there are way cleverer ways to handle
transposing, compared to the naive way I implemented it.

Regards,

Zelphir

Re: guile-user Digest, Vol 193, Issue 30

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 3264 bytes --]

Hi Zelphir,

> > 	AIscm
> > 	http://wedesoft.github.io/aiscm

> Thanks David,

You are welcome!

> I've added these to my notes for further investigation : )

> In SciPy and if I am not mistaken in the Python bindings for OpenCV they
> also use NumPy ndarray to represent images and it feels quite natural,
> so it makes sense to me to use n-dimensional arrays for this.

Note that when I'm saying that images are a collection of n-dim arrays, I am
referring to the 'human representation' of things: all (fast) image library that I'm
aware of internally use homogeneous vectors to store image channels.

> It seems the "uniform numeric vectors" of Guile are an implementation of
> the vectors in SRFI-4 as per:
> https://www.gnu.org/software/guile/manual/html_node/Uniform-Numeric-Vectors.html#Uniform-Numeric-Vectors.

Afaict, in Guile, there is no other "uniform numeric vectors" implementation then
the one specified in (srfi srfi-4).

> So far I have a very naive but working matrix operation implementation
> here (WIP):

> https://gitlab.com/ZelphirKaltstahl/neural-network/blob/dev/matrix.scm

Hum, neural network ... you definitely  want to consider using AIscm instead, which
just added a binding to the tensor flow library:

	https://lists.gnu.org/archive/html/guile-user/2018-12/msg00070.html

	[ this email includes a detailed (well commented) example of a
	[ (very) famous (basic) example of NN training in the NN world ...

> I think, once I have a good idea about how to use a fitting library or
> how to interface with a library of another programming language, I
> should be able to switch ...

You should consider using one of the libs I 'linked' in my previous message, and
given your objective is NN, iiuc, AIscm (and Guile-CV - but it does not have a
NN lib binding yet) probably are your friends here ...

> Yesterday I tried to use my transpose procedure and it turns out I
> sooner run out of RAM (8GB) than the thing slows down ...
>...

Guile-CV is fast (as fast as, sometime a lot faster then InageJ [1], which, with
OpenCV, is a reference in the image processing and analysis world).  However, it
will never reach the performance offered by libs that uses GPU (such as tensorflow
just to name one).

Now, using Guile-CV, on my relatively slow laptop (i5-2450M CPU @ 2.50GHz × 4 - 6GB
RAM):

	scheme@(guile-user)> ,use (cv)

	scheme@(guile-user)> (im-make 1000 1000 1)
	$5 = (1000 1000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 # …)))

	scheme@(guile-user)> ,time (im-transpose $5)
	$6 = (1000 1000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 # …)))
	;; 0.342220s real time, 0.367022s run time.  0.040217s spent in GC.

	scheme@(guile-user)> (im-make 1000 10000 1)
	$7 = (1000 10000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 # …)))

	scheme@(guile-user)> ,time (im-transpose $7)
	$8 = (10000 1000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 # …)))
	;; 3.186683s real time, 3.374562s run time.  0.303187s spent in GC.

But I see that im-transpose still is pure scheme code, I'll write the low level
operation in C and post back ...

Cheers,
David

[1]	https://imagej.nih.gov/ij/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: guile-user Digest, Vol 193, Issue 30

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 2148 bytes --]

Hi Zelphir,

> ...
> Now, using Guile-CV, on my relatively slow laptop (i5-2450M CPU @ 2.50GHz × 4 - 6GB
> RAM):

> 	scheme@(guile-user)> ,use (cv)

> 	scheme@(guile-user)> (im-make 1000 1000 1)
> 	$5 = (1000 1000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
> 0.0 # …)))

> 	scheme@(guile-user)> ,time (im-transpose $5)
> 	$6 = (1000 1000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
> 0.0 # …))) ;; 0.342220s real time, 0.367022s run time.  0.040217s spent in GC.
> 
> 	scheme@(guile-user)> (im-make 1000 10000 1)
> 	$7 = (1000 10000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
> 0.0 # …)))
> 
> 	scheme@(guile-user)> ,time (im-transpose $7)
> 	$8 = (10000 1000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
> > 0.0 # …))) ;; 3.186683s real time, 3.374562s run time.  0.303187s spent in GC.
> 
> But I see that im-transpose still is pure scheme code, I'll write the low level
> operation in C and post back ...

I moved the im-transpose core functionality to libguile-cv, and now I get these
results (roughly 22x faster):

	scheme@(guile-user)> ,use (cv)
	scheme@(guile-user)> (im-make 1000 1000 1)
	$2 = (1000 1000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 # …)))
	scheme@(guile-user)> ,time (im-transpose $2)
	$3 = (1000 1000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 # …)))
	;; 0.027255s real time, 0.035129s run time.  0.014336s spent in GC.

	scheme@(guile-user)> (im-make 1000 10000 1)
	$4 = (1000 10000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 # …)))
	scheme@(guile-user)> ,time (im-transpose $4)
	$5 = (10000 1000 1 (#f32(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 # …)))
	;; 0.141708s real time, 0.147565s run time.  0.024564s spent in GC.

Note that if yo plan to 'play with' Guile-CV (and expect the above last figures):

	(a) you have to install form the source and check the devel branch (didn't
	merge to master yet and do not plan to release soon);

	(b) make sure you read the 'Configuring Guile for Guile-CV' before you even
	start ...

Cheers,
David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]