Supported/Bundled CSV support?

Supported/Bundled CSV support?

[T.V Raman]

The simple/innocent looking "CSV File Format " AKA Comma Separated
Values, has a surprizingly high number of quirks with respect to
quoting etc.

Languages like Python work around this with a csv module that comes
bundled; Emacs has a couple of modules out on elpa/melpa all of which
look old.

Is it perhaps time that we brought one of those up to snuff (CSV was
finally officially defined a couple of years ago via an RFC).


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

Re: Supported/Bundled CSV support?

[Stefan Monnier]

> Languages like Python work around this with a csv module that comes
> bundled; Emacs has a couple of modules out on elpa/melpa all of which
> look old.

Don't know about Melpa, but in (Non)GNU ELPA, I can only find one such
package and it dates back to Dec 2021.


        Stefan

Re: Supported/Bundled CSV support?

[Ergus]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

Just an idea/question:

With the new Lars package vtable may be possible to support CSV or add a package over it?

I agree it is a bit annoying to depend of an external package for such a simple format and maybe the implementation may be simpler than expected as the format it pretty simple in general.

WDYT?






On March 14, 2022 11:30:43 PM GMT+01:00, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Languages like Python work around this with a csv module that comes
>> bundled; Emacs has a couple of modules out on elpa/melpa all of which
>> look old.
>
>Don't know about Melpa, but in (Non)GNU ELPA, I can only find one such
>package and it dates back to Dec 2021.
>
>
>        Stefan
>
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 1295 bytes --]

Re: Supported/Bundled CSV support?

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1683 bytes --]

Ergus <spacibba@aol.com> writes:


A couple more thoughts around why we may want built-in CSV support:

1. Over the last few years, emacs has incorporated built-in support for:

A. xml --- libxml
B. json --- libjs
C. We're now talking sqlite

Not having CSV builtin feels like ignoring what is in plain-sight.

I think having built-in CSV support that abstracts away the various
details of parsing the various CSV variants will allow the emacs Dev
community to focus on user solutions  e.g.:

A. Seamless data import/export
B. Support the processing of larger amounts of data via org-table and the like
C. Json support vastly sped up modules like lsp and eglot; I'd expect
   the same once we cross the chasm with respect to making CSV support
   a built-in detail that elisp developers can take for granted.
> Just an idea/question:
>
> With the new Lars package vtable may be possible to support CSV or add
> a package over it?
>
> I agree it is a bit annoying to depend of an external package for such
> a simple format and maybe the implementation may be simpler than
> expected as the format it pretty simple in general.
>
> WDYT?
>
> On March 14, 2022 11:30:43 PM GMT+01:00, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
>
>     Languages like Python work around this with a csv module that
>     comes
> bundled; Emacs has a couple of modules out on elpa/melpa all of which
> look old.
>
> Don't know about Melpa, but in (Non)GNU ELPA, I can only find one such
> package and it dates back to Dec 2021.
>
>         Stefan

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Supported/Bundled CSV support?

[Stefan Monnier]

> I think having built-in CSV support that abstracts away the various
> details of parsing the various CSV variants will allow the emacs Dev
> community to focus on user solutions  e.g.:
>
> A. Seamless data import/export
> B. Support the processing of larger amounts of data via org-table and the like
> C. Json support vastly sped up modules like lsp and eglot; I'd expect
>    the same once we cross the chasm with respect to making CSV support
>    a built-in detail that elisp developers can take for granted.

You might be right, but if so you're talking about a very different kind
of "CSV support" than the kind I'm familiar with (which is the support
to manually edit/browse a human-generated CSV file, as offered by
`csv-mode`).  You seem to be talking about code to efficiently parse
a whole CSV file into a Lisp representation (list of lists or better) or
to generate a CSV-formatted string from such Lisp representation.

Is that right?  In that case, indeed `csv-mode` is of no
help, basically.


        Stefan


>> Just an idea/question:
>>
>> With the new Lars package vtable may be possible to support CSV or add
>> a package over it?
>>
>> I agree it is a bit annoying to depend of an external package for such
>> a simple format and maybe the implementation may be simpler than
>> expected as the format it pretty simple in general.
>>
>> WDYT?
>>
>> On March 14, 2022 11:30:43 PM GMT+01:00, Stefan Monnier
>> <monnier@iro.umontreal.ca> wrote:
>>
>>     Languages like Python work around this with a csv module that
>>     comes
>> bundled; Emacs has a couple of modules out on elpa/melpa all of which
>> look old.
>>
>> Don't know about Melpa, but in (Non)GNU ELPA, I can only find one such
>> package and it dates back to Dec 2021.
>>
>>         Stefan

Re: Supported/Bundled CSV support?

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 2513 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:


Yes, you are correct.

There are modules like pcsv.el and  similarly named ones from the past
that do an approximate job -- when I looked into this approx 20 years
ago, I also ended up writing one of my own within Emacspeak --- in my
experience each of these hand-rolled csv parsers are differently broken
in their own way -- including the one in emacspeak, since as as I
pointed out in my earlier message, parsing various variants of CSV is
surprizingly hard.

>> I think having built-in CSV support that abstracts away the various
>> details of parsing the various CSV variants will allow the emacs Dev
>> community to focus on user solutions  e.g.:
>>
>> A. Seamless data import/export
>> B. Support the processing of larger amounts of data via org-table and the like
>> C. Json support vastly sped up modules like lsp and eglot; I'd expect
>>    the same once we cross the chasm with respect to making CSV support
>>    a built-in detail that elisp developers can take for granted.
>
> You might be right, but if so you're talking about a very different kind
> of "CSV support" than the kind I'm familiar with (which is the support
> to manually edit/browse a human-generated CSV file, as offered by
> `csv-mode`).  You seem to be talking about code to efficiently parse
> a whole CSV file into a Lisp representation (list of lists or better) or
> to generate a CSV-formatted string from such Lisp representation.
>
> Is that right?  In that case, indeed `csv-mode` is of no
> help, basically.
>
>
>         Stefan
>
>
>>> Just an idea/question:
>>>
>>> With the new Lars package vtable may be possible to support CSV or add
>>> a package over it?
>>>
>>> I agree it is a bit annoying to depend of an external package for such
>>> a simple format and maybe the implementation may be simpler than
>>> expected as the format it pretty simple in general.
>>>
>>> WDYT?
>>>
>>> On March 14, 2022 11:30:43 PM GMT+01:00, Stefan Monnier
>>> <monnier@iro.umontreal.ca> wrote:
>>>
>>>     Languages like Python work around this with a csv module that
>>>     comes
>>> bundled; Emacs has a couple of modules out on elpa/melpa all of which
>>> look old.
>>>
>>> Don't know about Melpa, but in (Non)GNU ELPA, I can only find one such
>>> package and it dates back to Dec 2021.
>>>
>>>         Stefan
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Supported/Bundled CSV support?

[Joost]

On Mon, 14 Mar 2022, at 23:30, Stefan Monnier wrote:
[csv-mode]
> Don't know about Melpa, but in (Non)GNU ELPA, I can only find one such
> package and it dates back to Dec 2021.

Speaking of which, I recently had an occasion for using this package and I was somewhat surprised that I did not find a way to specify the field separator on a per-file (or per-buffer) basis. The file I wanted to open used `;` as separator, and some of the fields hat commas in them. Adding `;` to `csv-separators` caused Emacs to recognize *both* `;` and `,` as field separators, meaning that the file wasn't displayed correctly.

I needed to set `csv-separators` to `(";")` to get Emacs to handle the file correctly. But if I make that permatent, I wouldn't be able to open any CSV files with commas as separators anymore.

Did I miss something, or is this indeed how `csv-mode` works? If so, I'm inclined to create a bug report, cause I reckon few (in any) CSV files use more than one separator character.





-- 
Joost Kremers
Life has its moments

Re: Supported/Bundled CSV support?

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 384 bytes --]

A slightly harder but more scalable in my experience  is to use
forms which has been bundled in emacs for a very, very long time, in
fact for so long that most people have forgotten it. 

See
https://emacspeak.blogspot.com/2020/06/viewing-data-records-as-forms-old-is.html

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Supported/Bundled CSV support?

[Stefan Monnier]

> I needed to set `csv-separators` to `(";")` to get Emacs to handle the file
> correctly. But if I make that permatent, I wouldn't be able to open any CSV
> files with commas as separators anymore.
>
> Did I miss something, or is this indeed how `csv-mode` works? If so, I'm
> inclined to create a bug report, cause I reckon few (in any) CSV files use
> more than one separator character.

Indeed, there's a missing functionality in the UI to do that.
It would make a lot of sense to have a command to change the (set
of) separators (which could even try and guess based on the buffer's
contents).

I basically only ever use `tsv-mode`, which is why I haven't improved
that part of the code.  Patches welcome.


        Stefan