[ELPA] New package: csv2ledger

[ELPA] New package: csv2ledger

[Joost Kremers]

Hi,

I would like to offer a new package I wrote for inclusion to Non-GNU ELPA:

https://codeberg.org/joostkremers/csv2ledger

`csv2ledger` does what it says: it converts CSV files to Ledger[1] entries. Most
banks offer the option to download transactions as a CSV file. With
`csv2ledger`, these can be converted to entries suitable for Ledger.

There are already many solutions to do this conversion, not the least Ledger
itself, which provides a `convert` command. None of these are Emacs-based,
however, and none that I know of offer the customisability that I needed myself.

`csv2ledger` is extremely customisable: with a little Elisp scripting, it is
able to handle just about anything a bank could come up with in a CSV file.
Also, the user can customise the Ledger entry in any way they see fit.[2] At the
same time, though, I tried to keep the basic configuration simple and useful
enough that it can be used without any Elisp scripting.

I'm proposing this package for inclusion into Non-GNU ELPA, not GNU ELPA,
because it depends on a package that's only available on MELPA, `parse-csv`.[3] (If
that is a problem, please let me know. I'm assuming it's a problem for GNU ELPA,
but not for Non-GNU ELPA.)[4]

TIA,

Joost



Footnotes:
[1]  https://ledger-cli.org/

[2] In fact, it is so customisable that you could use it to convert CSV files to
     any format you like. JSON, YAML, whatever you fancy. Even XML if you're so
     inclined... :D

[3]  https://melpa.org/#/parse-csv

[4] If anyone has suggestions on how to get rid of that dependency, I'm all
     ears. `parse-csv` looks like a real parser, not the kind of regex-kludge
     that I would come up with... I've looked at `csv-mode`, which is on GNU
     ELPA (and which is also a dependency), but I don't think it can be used
     (easily) to parse CSV data.

-- 
Joost Kremers
Life has its moments

Re: [ELPA] New package: csv2ledger

[Philip Kaludercic]

Joost Kremers <joostkremers@fastmail.fm> writes:

> Hi,
>
> I would like to offer a new package I wrote for inclusion to Non-GNU ELPA:
>
> https://codeberg.org/joostkremers/csv2ledger

I don't have the time for a proper review of the package for now, but
that shouldn't block adding the package.  I'll try to come back to you
later with a few comments (e.g. getting rid of the dash dependency).

> `csv2ledger` does what it says: it converts CSV files to Ledger[1] entries. Most
> banks offer the option to download transactions as a CSV file. With
> `csv2ledger`, these can be converted to entries suitable for Ledger.
>
> There are already many solutions to do this conversion, not the least Ledger
> itself, which provides a `convert` command. None of these are Emacs-based,
> however, and none that I know of offer the customisability that I needed myself.

Can you elaborate on what you mean by "Emacs-based"?  Not everything one
does in Emacs has to be written in Emacs Lisp, I think it is quite fine
to invoke an external program as a kind of Unix filter and C-u M-| it on
some region.

> `csv2ledger` is extremely customisable: with a little Elisp scripting, it is
> able to handle just about anything a bank could come up with in a CSV file.
> Also, the user can customise the Ledger entry in any way they see fit.[2] At the
> same time, though, I tried to keep the basic configuration simple and useful
> enough that it can be used without any Elisp scripting.
>
> I'm proposing this package for inclusion into Non-GNU ELPA, not GNU ELPA,
> because it depends on a package that's only available on MELPA, `parse-csv`.[3] (If
> that is a problem, please let me know. I'm assuming it's a problem for GNU ELPA,
> but not for Non-GNU ELPA.)[4]

That is a blocker for both repositories.  All packages in NonGNU and GNU
ELPA should be installable OOTB, without having to configure third-party
repositories (especially if their website needs javascript to display
even the most basic information...).

A quick peak at the package reveals it to be more or less a single
function.  Can you check of the csv-mode on ELPA provides anything you
need?  It seems like something that could be replicated in that package.

> TIA,
>
> Joost
>
>
>
> Footnotes:
> [1]  https://ledger-cli.org/
>
> [2] In fact, it is so customisable that you could use it to convert CSV files to
>      any format you like. JSON, YAML, whatever you fancy. Even XML if you're so
>      inclined... :D
>
> [3]  https://melpa.org/#/parse-csv
>
> [4] If anyone has suggestions on how to get rid of that dependency, I'm all
>      ears. `parse-csv` looks like a real parser, not the kind of regex-kludge
>      that I would come up with... I've looked at `csv-mode`, which is on GNU
>      ELPA (and which is also a dependency), but I don't think it can be used
>      (easily) to parse CSV data.

-- 
	Philip Kaludercic on peregrine

Re: [ELPA] New package: csv2ledger

[Joost Kremers]

Hi Philip,

On Mon, May 20 2024, Philip Kaludercic wrote:
> Joost Kremers <joostkremers@fastmail.fm> writes:
>> https://codeberg.org/joostkremers/csv2ledger
>
> I don't have the time for a proper review of the package for now, but
> that shouldn't block adding the package.  I'll try to come back to you
> later with a few comments (e.g. getting rid of the dash dependency).

Is dash a real problem? I mean, it's on GNU ELPA, and I specifically made sure
to target the version there.

I can remove dash if you prefer, though. There are two dash functions I use that
don't seem to have a direct built-in equivalent (as far as I can tell):
`-compose` and `-zip-pair`. I'm sure I can deal with them.

>> There are already many solutions to do this conversion, not the least Ledger
>> itself, which provides a `convert` command. None of these are Emacs-based,
>> however, and none that I know of offer the customisability that I needed
>> myself.
>
> Can you elaborate on what you mean by "Emacs-based"?

Conveniently integrated into Emacs. You're right, of course, that any CLI-based
solution can be integrated into Emacs with some glue code. It's just that I'm
not aware that anyone has written such glue code. (Though I might have missed
something, of course).

>> I'm proposing this package for inclusion into Non-GNU ELPA, not GNU ELPA,
>> because it depends on a package that's only available on MELPA,
>> `parse-csv`.[3] (If that is a problem, please let me know. I'm assuming it's
>> a problem for GNU ELPA, but not for Non-GNU ELPA.)[4]

> That is a blocker for both repositories.

OK. Makes sense, actually.

> A quick peak at the package reveals it to be more or less a single
> function.  Can you check of the csv-mode on ELPA provides anything you
> need?  It seems like something that could be replicated in that package.

Yes, it's just one single function, `parse-csv-string`. I've taken a closer look
at csv-mode and just found the function `csv--collect-fields`, which seems to do
basically the same as `parse-csv-string`, but a) it's an internal function; and
b) it doesn't strip quotes from field values.

Neither seems unsolvable, I guess. I would be interested in your opinion,
though, as to what would be better: add a public function to csv-mode.el that
strips the quotes, or call `csv--collect-fields` from csv2ledger.el and strip
the quotes there?

-- 
Joost Kremers
Life has its moments

Re: [ELPA] New package: csv2ledger

[Philip Kaludercic]

Joost Kremers <joostkremers@fastmail.fm> writes:

> Hi Philip,
>
> On Mon, May 20 2024, Philip Kaludercic wrote:
>> Joost Kremers <joostkremers@fastmail.fm> writes:
>>> https://codeberg.org/joostkremers/csv2ledger
>>
>> I don't have the time for a proper review of the package for now, but
>> that shouldn't block adding the package.  I'll try to come back to you
>> later with a few comments (e.g. getting rid of the dash dependency).
>
> Is dash a real problem? I mean, it's on GNU ELPA, and I specifically made sure
> to target the version there.

I consider it an unnecessary dependency that introduces a different
style of programming that not everyone has to be familiar with.  So it
is not a "real" problem, but I consider it an annoyance that I'd
personally like to help get rid of.

> I can remove dash if you prefer, though. There are two dash functions I use that
> don't seem to have a direct built-in equivalent (as far as I can tell):
> `-compose` and `-zip-pair`. I'm sure I can deal with them.

-Compose seems like it is easy to replace, but it might be an idea to
add something to the core as well.  One should try to improve the output
of the byte-code though:

(disassemble (byte-compile (lambda (x) (funcall (-compose 'foo 'bar 'baz) x))))
byte code:
  doc:   ...
  args: (arg1)
0	constant  <byte-code-function>
      doc:   ...
      args: (&rest rest)
    0	    constant  apply
    1	    constant  baz
    2	    stack-ref 2
    3	    call      2
    4	    constant  (bar foo)
    5	    constant  0
    6	    constant  nil
    7	    dup	      
    8:1	    stack-ref 3
    9	    goto-if-nil 2
    12	    stack-ref 3
    13	    dup	      
    14	    cdr	      
    15	    stack-set 5
    17	    discardN-preserve-tos 2
    19	    car-safe  
    20	    stack-ref 1
    21	    dup	      
    22	    add1      
    23	    stack-set 3
    25	    stack-ref 1
    26	    stack-ref 5
    27	    call      1
    28	    stack-set 5
    30	    goto      1
    33:2    discardN  4
    35	    return    

1	stack-ref 1
2	call	  1
3	return	  

compared to the direct:

(disassemble (byte-compile (lambda (x) (foo (bar (baz x))))))
byte code:
  doc:   ...
  args: (arg1)
0	constant  foo
1	constant  bar
2	constant  baz
3	stack-ref 3
4	call	  1
5	call	  1
6	call	  1
7	return	  

-Zip-pair sounds like something that cl-loop could replicate.

>>> There are already many solutions to do this conversion, not the least Ledger
>>> itself, which provides a `convert` command. None of these are Emacs-based,
>>> however, and none that I know of offer the customisability that I needed
>>> myself.
>>
>> Can you elaborate on what you mean by "Emacs-based"?
>
> Conveniently integrated into Emacs. You're right, of course, that any CLI-based
> solution can be integrated into Emacs with some glue code. It's just that I'm
> not aware that anyone has written such glue code. (Though I might have missed
> something, of course).
>
>>> I'm proposing this package for inclusion into Non-GNU ELPA, not GNU ELPA,
>>> because it depends on a package that's only available on MELPA,
>>> `parse-csv`.[3] (If that is a problem, please let me know. I'm assuming it's
>>> a problem for GNU ELPA, but not for Non-GNU ELPA.)[4]
>
>> That is a blocker for both repositories.
>
> OK. Makes sense, actually.
>
>> A quick peak at the package reveals it to be more or less a single
>> function.  Can you check of the csv-mode on ELPA provides anything you
>> need?  It seems like something that could be replicated in that package.
>
> Yes, it's just one single function, `parse-csv-string`. I've taken a closer look
> at csv-mode and just found the function `csv--collect-fields`, which seems to do
> basically the same as `parse-csv-string`, but a) it's an internal function; and
> b) it doesn't strip quotes from field values.
>
> Neither seems unsolvable, I guess. I would be interested in your opinion,
> though, as to what would be better: add a public function to csv-mode.el that
> strips the quotes, or call `csv--collect-fields` from csv2ledger.el and strip
> the quotes there?

If you provide a patch for csv-mode, I'd be happy to apply it.  That
sounds like a useful functionality to have.

-- 
	Philip Kaludercic on peregrine

Re: [ELPA] New package: csv2ledger

[Philip Kaludercic]

[...]

>> I can remove dash if you prefer, though. There are two dash
>> functions I use that
>> don't seem to have a direct built-in equivalent (as far as I can tell):
>> `-compose` and `-zip-pair`. I'm sure I can deal with them.
>
> -Compose seems like it is easy to replace, but it might be an idea to
> add something to the core as well.  One should try to improve the output
> of the byte-code though:
>
> (disassemble (byte-compile (lambda (x) (funcall (-compose 'foo 'bar 'baz) x))))
> byte code:
>   doc:   ...
>   args: (arg1)
> 0	constant  <byte-code-function>
>       doc:   ...
>       args: (&rest rest)
>     0	    constant  apply
>     1	    constant  baz
>     2	    stack-ref 2
>     3	    call      2
>     4	    constant  (bar foo)
>     5	    constant  0
>     6	    constant  nil
>     7	    dup	      
>     8:1	    stack-ref 3
>     9	    goto-if-nil 2
>     12	    stack-ref 3
>     13	    dup	      
>     14	    cdr	      
>     15	    stack-set 5
>     17	    discardN-preserve-tos 2
>     19	    car-safe  
>     20	    stack-ref 1
>     21	    dup	      
>     22	    add1      
>     23	    stack-set 3
>     25	    stack-ref 1
>     26	    stack-ref 5
>     27	    call      1
>     28	    stack-set 5
>     30	    goto      1
>     33:2    discardN  4
>     35	    return    
>
> 1	stack-ref 1
> 2	call	  1
> 3	return	  
>
> compared to the direct:
>
> (disassemble (byte-compile (lambda (x) (foo (bar (baz x))))))
> byte code:
>   doc:   ...
>   args: (arg1)
> 0	constant  foo
> 1	constant  bar
> 2	constant  baz
> 3	stack-ref 3
> 4	call	  1
> 5	call	  1
> 6	call	  1
> 7	return	  

Correction, the first disassembly should be

(disassemble (byte-compile (-compose 'foo 'bar 'baz)))

byte code:
  doc:   ...
  args: (&rest rest)
0	constant  apply
1	constant  baz
2	stack-ref 2
3	call	  2
4	constant  (bar foo)
5	constant  0
6	constant  nil
7	dup	  
8:1	stack-ref 3
9	goto-if-nil 2
12	stack-ref 3
13	dup	  
14	cdr	  
15	stack-set 5
17	discardN-preserve-tos 2
19	car-safe  
20	stack-ref 1
21	dup	  
22	add1	  
23	stack-set 3
25	stack-ref 1
26	stack-ref 5
27	call	  1
28	stack-set 5
30	goto	  1
33:2	discardN  4
35	return	  

avoiding the unnecessary lambda.

It appears even a naive approach to implement the function oneself gives
better results:

(defun compose (&rest fns)
  (lambda (arg)
    (dotimes (fn (nreverse fns))
      (setq arg (funcall fn arg)))
    arg))

(disassemble (byte-compile (compose 'foo 'bar 'baz)))

byte code:
  doc:   ...
  args: (arg1)
0	constant  (foo bar baz)
1	nreverse  
2	constant  0
3:1	dup	  
4	stack-ref 2
5	lss	  
6	goto-if-nil 2
9	dup	  
10	stack-ref 3
11	call	  1
12	stack-set 3
14	add1	  
15	goto	  1
18:2	discardN  2
20	return	  

[...]

Re: [ELPA] New package: csv2ledger

[Joost Kremers]

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]



On 20 May 2024 14:18:24 Philip Kaludercic <philipk@posteo.net> wrote:

> -Compose seems like it is easy to replace, but it might be an idea to add 
> something to the core as well.

I'll leave that to more competent coders. 🙂 I only use `-compose` in one 
place, it shouldn't be difficult to work around it.

> -Zip-pair sounds like something that cl-loop could replicate.

Actually, I realised that `(seq-mapn #'cons list1 list2)` achieves the same 
result, so I've replaced it with that.

Of course that's just replicating the same style with built-ins, but I hope 
that's OK for you. (I have no doubt cl-loop can do it as well, but I'm less 
well-versed in it...)

> If you provide a patch for csv-mode, I'd be happy to apply it.  That sounds 
> like a useful functionality to have.

Sure, I'll see what I can come up with.

-- 
Joost Kremers
Life has its moments



[-- Attachment #2: Type: text/html, Size: 2387 bytes --]

Re: [ELPA] New package: csv2ledger

[Joost Kremers]

Hi Philip,

[snip suggested changes]

I've now removed the dependency on both dash and parse-csv, and the new version
of csv-mode has been installed.

I also made some other small changes, so I feel this is now ready to be
resubmitted. For ease of reference, here's the link to the code again:

https://codeberg.org/joostkremers/csv2ledger

Thanks

-- 
Joost Kremers
Life has its moments

Re: [ELPA] New package: csv2ledger

[Philip Kaludercic]

Joost Kremers <joostkremers@fastmail.fm> writes:

> Hi Philip,
>
> [snip suggested changes]
>
> I've now removed the dependency on both dash and parse-csv, and the new version
> of csv-mode has been installed.
>
> I also made some other small changes, so I feel this is now ready to be
> resubmitted. For ease of reference, here's the link to the code again:
>
> https://codeberg.org/joostkremers/csv2ledger

Done, should appear on the archive in a few hours.

Just one little remark, if submitting a package to NonGNU ELPA, please
mention that in the subject.  I almost added the package to elpa, but
double-checked because I recall your other packages being on NonGNU.

> Thanks

-- 
	Philip Kaludercic on peregrine

Re: [ELPA] New package: csv2ledger

[Joost Kremers]

On Sun, Jun 02 2024, Philip Kaludercic wrote:
>> https://codeberg.org/joostkremers/csv2ledger
>
> Done, should appear on the archive in a few hours.

Thanks!

> Just one little remark, if submitting a package to NonGNU ELPA, please
> mention that in the subject.  I almost added the package to elpa, but
> double-checked because I recall your other packages being on NonGNU.

Alright, will do. (It wouldn't have mattered much in this case, though. I wrote
the code and my copyright assignment is on file. But I'll keep it in mind in
case there's a next time.)

-- 
Joost Kremers
Life has its moments