How to develop a local fork of a package?

How to develop a local fork of a package?

[Alessandro Bertulli]

Hi all!

Pretty much the title.

I want to experiment a bit in modifying org-ref. I have installed it
using the package manager. Then I cloned the repository on my local
machine (in a directory "~/.emacs.d/git_packages/org-ref/"). Then I had
to instruct Emacs to use the local version, and not the
"~/.emacs.d/elpa/" one.

How can I do that? For now, my Google-fu lead to this:

(add-to-list 'load-path (expand-file-name "git_packages/org-ref/"))
(delete "/home/alessandro/.emacs.d/elpa/org-ref-20220830.1210" load-path)
(package-initialize)

But then, no matter which one of these I use

(require 'org-ref)
(load "~/.emacs.d/git_packages/org-ref/org-ref")
(use-package org-ref
  :ensure nil
  :load-path "~/.emacs.d/git_packages/org-ref/org-ref.el")

I always get errors.

- What am I doing wrong?
- Or maybe, this process is completely non idiomatic, and there is a
  better way?

Alessandro

Re: How to develop a local fork of a package?

[Emanuel Berg]

Alessandro Bertulli wrote:

> (add-to-list 'load-path (expand-file-name "git_packages/org-ref/"))
> (delete "/home/alessandro/.emacs.d/elpa/org-ref-20220830.1210" load-path)
> (package-initialize)
>
> But then, no matter which one of these I use
>
> (require 'org-ref)
> (load "~/.emacs.d/git_packages/org-ref/org-ref")
> (use-package org-ref
>   :ensure nil
>   :load-path "~/.emacs.d/git_packages/org-ref/org-ref.el")
>
> I always get errors

What errors?

-- 
underground experts united
https://dataswamp.org/~incal

Re: How to develop a local fork of a package?

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 2166 bytes --]

Emanuel Berg <incal@dataswamp.org> writes:

> Alessandro Bertulli wrote:
>
>> (add-to-list 'load-path (expand-file-name "git_packages/org-ref/"))

(expand-file-name "git_packages/org-ref/")?  Relative to what?  If the
directory is in your ".emacs.d", then it should be (add-to-list
'load-path (expand-file-name "git_packages/org-ref/"
user-emacs-directory)).

>> (delete "/home/alessandro/.emacs.d/elpa/org-ref-20220830.1210" load-path)
>> (package-initialize)

Why?  This would break whenever package.el updates org-ref.  And this
won't ever work because that directory is not in load-path before
(package-initialize), and after (package-initialize) org-ref will be on
load-path.  Your best option is to delete the package with M-x
package-delete.  And the "delete" call may not work.  You set load-path
to the return value of "delete", for example:

--8<---------------cut here---------------start------------->8---
(setq load-path
      (delete "/home/alessandro/.emacs.d/elpa/org-ref-20220830.1210"
              load-path))
--8<---------------cut here---------------end--------------->8---

>>
>> But then, no matter which one of these I use
>>
>> (require 'org-ref)

Unless org-ref is in load-path, this will always error.

>> (load "~/.emacs.d/git_packages/org-ref/org-ref")

Where is the ".el" suffix?  Did you mean (load
"~/.emacs.d/git_packages/org-ref/org-ref.el")?  Loading directly will
probably work for single file packages, but you should always add the
package directory to load-path.

>> (use-package org-ref
>>   :ensure nil
>>   :load-path "~/.emacs.d/git_packages/org-ref/org-ref.el")

I think the error is because you specify a file as load-path instead of
a directory.  I think the load-path should be probably
"~/.emacs.d/git_packages/org-ref/".  However, I have never used
:load-path keyword of use-package, so my assumptions may be wrong.

>>
>> I always get errors
>
> What errors?

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: How to develop a local fork of a package?

[Alessandro Bertulli]

Akib Azmain Turja <akib@disroot.org> writes:

>> Alessandro Bertulli wrote:
>>
>>> (add-to-list 'load-path (expand-file-name "git_packages/org-ref/"))
>
> (expand-file-name "git_packages/org-ref/")?  Relative to what?  If the
> directory is in your ".emacs.d", then it should be (add-to-list
> 'load-path (expand-file-name "git_packages/org-ref/"
> user-emacs-directory)).

Yes, but if not specified the user-emacs-directory is the default, so I
don't think that's a problem (will try tho, just in case).

>>> (delete "/home/alessandro/.emacs.d/elpa/org-ref-20220830.1210" load-path)
>>> (package-initialize)
>
> Why?  This would break whenever package.el updates org-ref.

Yes, it was just an hack to try with the current package.

> And this won't ever work because that directory is not in load-path
> before (package-initialize), and after (package-initialize) org-ref
> will be on load-path.

You're right, I didn't know the functioning of package-initialize. I'll
try and let you know.

> Your best option is to delete the package with M-x package-delete.

Aren't there any other methods? Like, if you want to make a PR, do you
uninstall the package every time?

> And the "delete" call may not work. You set load-path to the return
> value of "delete", for example:
>
> (setq load-path
>       (delete "/home/alessandro/.emacs.d/elpa/org-ref-20220830.1210"
>               load-path))

Well, it works for me actually. Also, delete uses side effects, if I
wanted to use setq, I should use remove, right?

>>>
>>> But then, no matter which one of these I use
>>>
>>> (require 'org-ref)
>
> Unless org-ref is in load-path, this will always error.

I suspect this is given by my wrong usage of package-initialize.

>>> (load "~/.emacs.d/git_packages/org-ref/org-ref")
>
> Where is the ".el" suffix?  Did you mean (load
> "~/.emacs.d/git_packages/org-ref/org-ref.el")?

Wait, shouldn't I omit the suffix on purpose? Looking at the docstring
of load: "First try FILE with .elc appended, then try with .el".

> Loading directly will probably work for single file packages, but you
> should always add the package directory to load-path.

Aside from the directory part, that file is the one containing the
(provide 'org-ref) call. Shouldn't loading that achieve the same result
of the (require 'org-ref) I normally use? Since the other code files are
loaded inside it.

>>> (use-package org-ref
>>>   :ensure nil
>>>   :load-path "~/.emacs.d/git_packages/org-ref/org-ref.el")
>
> I think the error is because you specify a file as load-path instead of
> a directory.  I think the load-path should be probably
> "~/.emacs.d/git_packages/org-ref/".  However, I have never used
> :load-path keyword of use-package, so my assumptions may be wrong.

If the above doesn't work, I'll try and I'll let you know. Thanks!

Alessandro

Re: How to develop a local fork of a package?

[Alessandro Bertulli]

It works! The trick was to first modify the load-path, then to call
(package-initialize). That way, calling (require 'org-ref) loaded the
git local version of the package.

Now the only point left is that, as you said, the delete call will break
when the package gets updated. I'll turn it into a regex version when I
have time.

Anyway, thanks to you and to Emanuel!

Alessandro

Re: How to develop a local fork of a package?

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 6488 bytes --]

Alessandro Bertulli <alessandro.bertulli96@gmail.com> writes:

> Akib Azmain Turja <akib@disroot.org> writes:
>
>>> Alessandro Bertulli wrote:
>>>
>>>> (add-to-list 'load-path (expand-file-name "git_packages/org-ref/"))
>>
>> (expand-file-name "git_packages/org-ref/")?  Relative to what?  If the
>> directory is in your ".emacs.d", then it should be (add-to-list
>> 'load-path (expand-file-name "git_packages/org-ref/"
>> user-emacs-directory)).
>
> Yes, but if not specified the user-emacs-directory is the default, so I
> don't think that's a problem (will try tho, just in case).

No, the default value is the value of default-directory variable.
However I don't know what the value of default-directory variable during
initialization.  There is a function "locate-user-emacs-file", which
will always expand the name relative to your .emacs.d.

>
>>>> (delete "/home/alessandro/.emacs.d/elpa/org-ref-20220830.1210" load-path)
>>>> (package-initialize)
>>
>> Why?  This would break whenever package.el updates org-ref.
>
> Yes, it was just an hack to try with the current package.
>
>> And this won't ever work because that directory is not in load-path
>> before (package-initialize), and after (package-initialize) org-ref
>> will be on load-path.
>
> You're right, I didn't know the functioning of package-initialize. I'll
> try and let you know.
>
>> Your best option is to delete the package with M-x package-delete.
>
> Aren't there any other methods? Like, if you want to make a PR, do you
> uninstall the package every time?

Actually I don't do PRs much, so much uninstalling packages is not a big
problem for me.  However, you may try to customize or set the variable
package-load-path like this:

--8<---------------cut here---------------start------------->8---
(setq package-load-path '((org-ref nil) all))
--8<---------------cut here---------------end--------------->8---

I think the above should work, but I haven't tested it.

>
>> And the "delete" call may not work. You set load-path to the return
>> value of "delete", for example:
>>
>> (setq load-path
>>       (delete "/home/alessandro/.emacs.d/elpa/org-ref-20220830.1210"
>>               load-path))
>
> Well, it works for me actually. Also, delete uses side effects, if I
> wanted to use setq, I should use remove, right?

No, you should use setq with delete because delete uses side-effects.
For example, if not don't use setq the following works as you expect:

--8<---------------cut here---------------start------------->8---
(let ((foo '(1 2 3)))
  (delete 2 foo)
  foo)
;; => '(1 3)
--8<---------------cut here---------------end--------------->8---

But the following doesn't:

--8<---------------cut here---------------start------------->8---
(let ((foo '(1 2 3)))
  (delete 1 foo)
  foo)
;; => '(1 2 3)
--8<---------------cut here---------------end--------------->8---

To understand why the above doesn't work, you need to understand how
lists work.  In Lisp, a list is a linked list made using cons cells.
For example, for the list '(1 2 3), it is actually '(1 . (2 . (3 .
nil))), or:

 foo
  |
  |
  |
+---+
|   | --- 1
+---+
|   | --- +---+
+---+     |   | --- 2
          +---+
          |   | --- +---+
          +---+     |   | --- 3
                    +---+
                    |   | --- nil
                    +---+

When you "delete" 2 from it:

 foo  return value
  |        |
  +--------+
  |
+---+
|   | --- 1
+---+
|   | ------+ 
+---+       |
            |
            +------ +---+
                    |   | --- 3
                    +---+
                    |   | --- nil
                    +---+

But when you "delete" 1:

 foo
  |
  |
  |    return value
+---+       |
|   | --- 1 |
+---+       |
|   | --- +---+
+---+     |   | --- 2
          +---+
          |   | --- +---+
          +---+     |   | --- 3
                    +---+
                    |   | --- nil
                    +---+

The variable foo still points to the old cons cell and therefore isn't
changed.  So to ensure the value is changed, you should use setq with
delete.  "remove" doesn't use side-effects, so it never modifies any
variable.

And that's why you should use the following:

--8<---------------cut here---------------start------------->8---
(setq load-path
      (delete "/home/alessandro/.emacs.d/elpa/org-ref-20220830.1210"
              load-path))
--8<---------------cut here---------------end--------------->8---

... because you can't be sure that load-path is really modified without
the setq.

>
>>>>
>>>> But then, no matter which one of these I use
>>>>
>>>> (require 'org-ref)
>>
>> Unless org-ref is in load-path, this will always error.
>
> I suspect this is given by my wrong usage of package-initialize.
>
>>>> (load "~/.emacs.d/git_packages/org-ref/org-ref")
>>
>> Where is the ".el" suffix?  Did you mean (load
>> "~/.emacs.d/git_packages/org-ref/org-ref.el")?
>
> Wait, shouldn't I omit the suffix on purpose? Looking at the docstring
> of load: "First try FILE with .elc appended, then try with .el".

Ah, I forgot that.  Sorry.  You're right.

>
>> Loading directly will probably work for single file packages, but you
>> should always add the package directory to load-path.
>
> Aside from the directory part, that file is the one containing the
> (provide 'org-ref) call. Shouldn't loading that achieve the same result
> of the (require 'org-ref) I normally use? Since the other code files are
> loaded inside it.

Unless there is any other file in the directory, it should work.  But if
there are other Emacs Lisp files, most likely the main file (that you're
trying to load) depends on them.

>
>>>> (use-package org-ref
>>>>   :ensure nil
>>>>   :load-path "~/.emacs.d/git_packages/org-ref/org-ref.el")
>>
>> I think the error is because you specify a file as load-path instead of
>> a directory.  I think the load-path should be probably
>> "~/.emacs.d/git_packages/org-ref/".  However, I have never used
>> :load-path keyword of use-package, so my assumptions may be wrong.
>
> If the above doesn't work, I'll try and I'll let you know. Thanks!
>
> Alessandro

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: How to develop a local fork of a package?

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

Alessandro Bertulli <alessandro.bertulli96@gmail.com> writes:

> It works! The trick was to first modify the load-path, then to call
> (package-initialize). That way, calling (require 'org-ref) loaded the
> git local version of the package.
>
> Now the only point left is that, as you said, the delete call will break
> when the package gets updated. I'll turn it into a regex version when I
> have time.

"cl-delete-if" and/or "cl-remove-if" function should help you in this
regard.

>
> Anyway, thanks to you and to Emanuel!
>
> Alessandro

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]