run shell command after publishing project

run shell command after publishing project

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

Hi everyone,
Feeling a little rude after a long absence in which I dropped all my
non-work commitments -- but still here to ask a question!

I'm setting up a new workflow using Kauhsal's ox-hugo.  I keep all my
course materials in a few org files & publish to hugo-flavoured markdown.
My source materials live in one git repo, and my website in another.  After
exporting any of my my courses to the local hugo directory, I'd like to run
a shell script that I'll keep in my org-files directory.

That script
- cds to the website repo directory, commits changes to the website master
branch, - runs hugo,
- switches to the "public" directory of compiled html pages, which has a
worktree checked out to  the  gh-pages branch,
- commits changes there as well, and then
- pushes both branches to github.

The script seems to work OK, and now I would like to run it every time I
export from the appropriate projects. Is there a good way for me to do
this? I guess a hook that only runs under certain conditions?

If I can get this to work, and then also auto export every time I commit
the org-files to master (maybe with a post-commit git hook of ~emacsclient
-e '(org-publish-project "course1"~ ?), then I will maybe be almost happy!

thanks as always,
matt

[-- Attachment #2: Type: text/html, Size: 1570 bytes --]

Re: run shell command after publishing project

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 2652 bytes --]

On Fri, Dec 8, 2017, 2:29 PM Matt Price <moptop99@gmail.com> wrote:

> Hi everyone,
> Feeling a little rude after a long absence in which I dropped all my
> non-work commitments -- but still here to ask a question!
>

Hello!

I'm setting up a new workflow using Kauhsal's ox-hugo.  I keep all my
> course materials in a few org files & publish to hugo-flavoured markdown.
>

Awesome! :D

  My source materials live in one git repo, and my website in another.
> After exporting any of my my courses to the local hugo directory, I'd like
> to run a shell script that I'll keep in my org-files directory.
>
> That script
> - cds to the website repo directory, commits changes to the website master
> branch, - runs hugo,
> - switches to the "public" directory of compiled html pages, which has a
> worktree checked out to  the  gh-pages branch,
> - commits changes there as well, and then
> - pushes both branches to github.
>
> The script seems to work OK, and now I would like to run it every time I
> export from the appropriate projects. Is there a good way for me to do
> this? I guess a hook that only runs under certain conditions?
>
> If I can get this to work, and then also auto export every time I commit
> the org-files to master (maybe with a post-commit git hook of ~emacsclient
> -e '(org-publish-project "course1"~ ?), then I will maybe be almost happy!
>

I haven't yet got to ox-publish to work with ox-hugo, because of the unique
flow for subtree-based exports where we want to export only subtrees with a
specific property (EXPORT_FILE_NAME).

But I was finally able to achieve something like that using a Makefile [1]
+ Netlify (or GitHub Pages/Travis CI or GitLab CI).

A very recent example (few days) is how I helped set up the use-package
website publishing flow.

- ox-hugo + Hugo using Makefile + GitHub Pages.

You *only* need to commit the use-package.org[2] to the GitHub repo, and
https://jwiegley.github.io/use-package/ updates in a minute or so. The
Travis CI simply calls "make doc". That takes care of:
- Installing dependencies if needed on the CI machine
- Exporting Org to Markdown using ox-hugo
- Running hugo
- Commiting published HTML to the gh-pages branch
- And the site gets deployed, just like that :)

PS: I publish the ox-hugo package website[3] the same way too, but using
Netlify (/which is the better than GitHub Pages or Gitlab CI IMO -- free
too/). See the footer of that site for the 1-file Org source.

[1]: https://github.com/jwiegley/use-package/blob/master/doc/Makefile

[2]: https://github.com/jwiegley/use-package/blob/master/use-package.org

[3]: https://ox-hugo.scripter.co/

-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 4413 bytes --]

Re: run shell command after publishing project

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 3883 bytes --]

This sounds fantastic, Kaushal. I'm struggling to follow, so wl lask some
questions inline.

On Fri, Dec 8, 2017 at 2:58 PM, Kaushal Modi <kaushal.modi@gmail.com> wrote:

> On Fri, Dec 8, 2017, 2:29 PM Matt Price <moptop99@gmail.com> wrote:
>
>> Hi everyone,
>> Feeling a little rude after a long absence in which I dropped all my
>> non-work commitments -- but still here to ask a question!
>>
>
> Hello!
>
> I'm setting up a new workflow using Kauhsal's ox-hugo.  I keep all my
>> course materials in a few org files & publish to hugo-flavoured markdown.
>>
>
> Awesome! :D
>
>   My source materials live in one git repo, and my website in another.
>> After exporting any of my my courses to the local hugo directory, I'd like
>> to run a shell script that I'll keep in my org-files directory.
>>
>> That script
>> - cds to the website repo directory, commits changes to the website
>> master branch, - runs hugo,
>> - switches to the "public" directory of compiled html pages, which has a
>> worktree checked out to  the  gh-pages branch,
>> - commits changes there as well, and then
>> - pushes both branches to github.
>>
>> The script seems to work OK, and now I would like to run it every time I
>> export from the appropriate projects. Is there a good way for me to do
>> this? I guess a hook that only runs under certain conditions?
>>
>> If I can get this to work, and then also auto export every time I commit
>> the org-files to master (maybe with a post-commit git hook of ~emacsclient
>> -e '(org-publish-project "course1"~ ?), then I will maybe be almost happy!
>>
>
> I haven't yet got to ox-publish to work with ox-hugo, because of the
> unique flow for subtree-based exports where we want to export only subtrees
> with a specific property (EXPORT_FILE_NAME).
>
> But I was finally able to achieve something like that using a Makefile [1]
> + Netlify (or GitHub Pages/Travis CI or GitLab CI).
>

ok, great.

>
> A very recent example (few days) is how I helped set up the use-package
> website publishing flow.
>
> - ox-hugo + Hugo using Makefile + GitHub Pages.
>
> You *only* need to commit the use-package.org[2] to the GitHub repo, and
> https://jwiegley.github.io/use-package/ updates in a minute or so. The
> Travis CI simply calls "make doc". That takes care of:
> - Installing dependencies if needed on the CI machine
> - Exporting Org to Markdown using ox-hugo
> - Running hugo
> - Commiting published HTML to the gh-pages branch
> - And the site gets deployed, just like that :)
>
> so,
- the doc/ directory contains the full Hugo source directory, with
config.toml, content/, layouts, etc., as well as the theme, which looks
like it's statically installed
- looks like maybe the hugo html ends up in public, but then it gets moved
somehow (can't quite figure out how)
- the ox-hugo files are kept somewhere I guess
- presumably any personal config has to be replicated somewhere on the web
where travis can access it
- is there any more setup that needs to be done on Travis?
- I have 4 main org files; 3 of them get copied per-header, and one as a
whole file.  Is that an issue?

This all looks really ocol; too bad it's a bit out of my range, I'd like to
understand it a little better!

To me the shell script idea feels easier, but I guess it's also a little
brittle


> PS: I publish the ox-hugo package website[3] the same way too, but using
> Netlify (/which is the better than GitHub Pages or Gitlab CI IMO -- free
> too/). See the footer of that site for the 1-file Org source.
>

Sticking with Github for now since all my course stuff is there, probably
easiest for my studnets ot have everything in the same place.


thank you!

>
> [1]: https://github.com/jwiegley/use-package/blob/master/doc/Makefile
>
> [2]: https://github.com/jwiegley/use-package/blob/master/use-package.org
>
> [3]: https://ox-hugo.scripter.co/
>
> --
>
> Kaushal Modi
>

[-- Attachment #2: Type: text/html, Size: 6760 bytes --]

Re: run shell command after publishing project

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 4497 bytes --]

On Fri, Dec 8, 2017 at 4:38 PM Matt Price <moptop99@gmail.com> wrote:

> This sounds fantastic, Kaushal. I'm struggling to follow, so wl lask some
> questions inline.
>

Sure, my answers are below.

- the doc/ directory contains the full Hugo source directory, with
> config.toml, content/, layouts, etc., as well as the theme, which looks
> like it's statically installed
>

Correct.


> - looks like maybe the hugo html ends up in public, but then it gets moved
> somehow (can't quite figure out how)
>

Travis CI deals with that. You can look at the use-package Travis setup
here[1]. The revelant portion for the site deployment is:

=====
before_deploy: cd doc && make doc
deploy:
  provider: pages
  skip_cleanup: true
  github_token: $GITHUB_TOKEN
  local_dir: public
  on:
    branch: master
    condition: $EMACS_VERSION = 26
=====

(I use emacs 26 specifically to build the site as it comes with Org 9.1.4..
so then I don't need to install Org on the Travis CI machine.)


> - the ox-hugo files are kept somewhere I guess
>

No. The latest ox-hugo version available on Melpa is used. That is done via
the same Makefile.. it loads a setup file I have: setup-ox-hugo.el[2].


> - presumably any personal config has to be replicated somewhere on the web
> where travis can access it
>

The same setup-ox-hugo.el has some minimal "sane settings" (Look at the
"(with-eval-after-load 'org" in there for example). You can also add your
personal setup to that as needed.


> - is there any more setup that needs to be done on Travis?
>

1. The .travis.yml file mentioned above.
2. Get your GITHUB_TOKEN (minimum permission level: public_repo) to assign
to Travis[3].
3. Set the GITHUB_TOKEN and HUGO_BASE_URL env variables in the Travis
settings for your repo on https://travis-ci.org/USER/REPO/settings.

- I have 4 main org files; 3 of them get copied per-header, and one as a
> whole file.  Is that an issue?
>

No. The latest ox-hugo version now has org-hugo-export-wim-to-md. "wim"
stands for What I Mean. See https://ox-hugo.scripter.co/#export-bindings or
the comment header in ox-hugo.el for details.

In setup-ox-hugo.el, I define this simple wrapper function:

(defun org-hugo-export-all-wim-to-md ()
  (org-hugo-export-wim-to-md :all-subtrees nil nil :noerror))

In the Makefile, that function is used in the "md1" rule.
And that rule is called in loop by the "md" rule:

md: $(org_files)
$(org_files):
    @$(MAKE) md1 ORG_FILE=$@ TIMEZONE=UTC # Use UTC/Universal time zone for
tests

So you only need to tweak the "org_files" rule which is a simple find
command to collect an array of all the Org files you want to export
(doesn't matter if they are file-based or subtree-based).

If there's any subtree that you don't want to export or any file that you
don't want to export in this process, tag the subtree with "foo" tag
(example) and the file with #+FILETAGS keyword set to "foo", and add the
same "foo" tag to org-export-exclude-tags in the setup-ox-hugo.el. More
info:
https://ox-hugo.scripter.co/doc/tags-and-categories/#marking-files-to-not-be-exported

This all looks really ocol; too bad it's a bit out of my range, I'd like to
> understand it a little better!
>

I hope above helps!

To me the shell script idea feels easier, but I guess it's also a little
> brittle
>
>
>> PS: I publish the ox-hugo package website[3] the same way too, but using
>> Netlify (/which is the better than GitHub Pages or Gitlab CI IMO -- free
>> too/). See the footer of that site for the 1-file Org source.
>>
>
> Sticking with Github for now since all my course stuff is there, probably
> easiest for my studnets ot have everything in the same place.
>

Netlify can be thought of just a CI machine. Your site can live in
Github/Gitlab/etc. repos, but Netlify will link to those. So the site
source will stay exactly where it is.. just that now Netlify will be
receptive to the git commit actions on your repo and auto-deploy the site
with each commit. For example: The ox-hugo doc site source is on GitHub,
but it is published using Netlify (https://ox-hugo.scripter.co).

Another thing: GitHub Pages does not support https on custom domains..
Netlify does.


> thank you!
>

You are welcome.

And finally, let's move all non-Org discussion off the Org mailing list.

[1]: https://github.com/jwiegley/use-package/blob/master/.travis.yml

[2]:
https://github.com/jwiegley/use-package/blob/master/doc/setup-ox-hugo.el

[3]: https://github.com/blog/1509-personal-api-tokens

-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 8592 bytes --]

Re: run shell command after publishing project

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

On Fri, Dec 8, 2017 at 5:19 PM, Kaushal Modi <kaushal.modi@gmail.com> wrote:

> On Fri, Dec 8, 2017 at 4:38 PM Matt Price <moptop99@gmail.com> wrote:
>
>> This sounds fantastic, Kaushal. I'm struggling to follow, so wl lask some
>> questions inline.
>>
>
> Sure, my answers are below.
>
> thank you!
>>
>
> You are welcome.
>
> And finally, let's move all non-Org discussion off the Org mailing list.
>
> I know this is a little off-topic, but if anyone is interested in a
slightly different, less elegant, solution, I just blogged about it here:
https://matt.hackinghistory.ca/2017/12/18/deploying-course-websites-with-git-hooks/
(mostly for my own records).

Ironically, my personal blog is I think my last remaining wordpress site --
I need to move it over to hugo soon as I can!

[-- Attachment #2: Type: text/html, Size: 1963 bytes --]

Re: run shell command after publishing project

[Rasmus]

Hi,

Matt Price <moptop99@gmail.com> writes:

> I'm setting up a new workflow using Kauhsal's ox-hugo.  I keep all my
> course materials in a few org files & publish to hugo-flavoured markdown.
> My source materials live in one git repo, and my website in another.  After
> exporting any of my my courses to the local hugo directory, I'd like to run
> a shell script that I'll keep in my org-files directory.

Do you mean commit or publish instead of export?  Otherwise you might have
to keep track of whether you are exporting to buffer as a buffer still has
a default-directory.  It would be easier to publish.

> That script
> - cds to the website repo directory, commits changes to the website master
> branch, - runs hugo,
> - switches to the "public" directory of compiled html pages, which has a
> worktree checked out to  the  gh-pages branch,
> - commits changes there as well, and then
> - pushes both branches to github.
>
> The script seems to work OK, and now I would like to run it every time I
> export from the appropriate projects. Is there a good way for me to do
> this? I guess a hook that only runs under certain conditions?

That is a sane approach with github pages, indeed.

I did something similar when I used github pages, but with ox-publish.  My
‘:preparation-function’ was

(defun project-prep-function ()
  (interactive)
  (or (zerop (shell-command (format "cd %s && git checkout master" project-main-dir)))
      (error (format "error checking out master in %s" project-main-dir))))

And my ‘:completion-function’ was as follows where ‘project-tmp-dir’ the
‘:publishing-directory’.

(defun project-completion-function ()
  (interactive)
  (let ((cd (format "cd %s && " project-main-dir)))
    (shell-command (concat cd "git commit -am \"auto-check-in\""))
    (or (zerop (shell-command (concat cd "git checkout gh-pages")))
        (error (format "error checking out gh-pages in %s" project-main-dir)))
    (or (zerop (shell-command (format "mv %s/*.html %s"
                                      project-tmp-dir project-main-dir)))
        (error "error moving files"))
    (or (zerop (shell-command
                (concat cd "git add *.html && git commit -am 'updated html'")))
        (error "error adding html files"))
    (or (zerop (shell-command
                (concat cd "git push origin gh-pages")))
        (warn "error pushing gh-pages"))
    (or (zerop (shell-command (concat cd "git checkout master")))
(error "error checking out master"))))

> If I can get this to work, and then also auto export every time I commit
> the org-files to master (maybe with a post-commit git hook of ~emacsclient
> -e '(org-publish-project "course1"~ ?), then I will maybe be almost happy!

If you use ox-publish you could auto-commit whenever you publish a
project.  But I guess you could also use a hook.  I use this to publish:

    emacs --batch --no-init-file --load publish.el --funcall org-publish-all

I do it via Gitlab CI, though, as I only want to keep track of source
files.

pages:
  script:
    - emacs --batch --no-init-file --load publish.el --funcall org-publish-all
  artifacts:
    paths:
      - public

Rasmus

-- 
Enough with the blah blah!