How do you store web pages for reference?

How do you store web pages for reference?

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

Hello,

I'm looking for a workflow that allows me to save a web page for
reference, ideally from Firefox. I know of org-protocol-capture-html
(https://github.com/alphapapa/org-protocol-capture-html), which is
perfect for pure-text pages, but I'm also looking for a solution for
images-heavy pages. I've tried to simply save the page to PDF, but it
does not preserve the links.

Do you have suggestions?

Thanks,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Monthly Athmospheric CO₂, Mauna Loa Obs. 2016-12: 404.48, 2015-12: 401.85

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]

Re: How do you store web pages for reference?

[Michael Welle]

Hello,

Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> Hello,
>
> I'm looking for a workflow that allows me to save a web page for
> reference, ideally from Firefox. I know of org-protocol-capture-html
> (https://github.com/alphapapa/org-protocol-capture-html), which is
> perfect for pure-text pages, but I'm also looking for a solution for
> images-heavy pages. I've tried to simply save the page to PDF, but it
> does not preserve the links.
>
> Do you have suggestions?
maybe Zotero is something you want to have a look at?

Regards
hmw

Re: How do you store web pages for reference?

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

Hello Michael,

On 2017-01-16 10:22, Michael Welle <mwe012008@gmx.net> writes:

>> I'm looking for a workflow that allows me to save a web page for
>> reference, ideally from Firefox. I know of org-protocol-capture-html
>> (https://github.com/alphapapa/org-protocol-capture-html), which is
>> perfect for pure-text pages, but I'm also looking for a solution for
>> images-heavy pages. I've tried to simply save the page to PDF, but it
>> does not preserve the links.
>>
>> Do you have suggestions?
> maybe Zotero is something you want to have a look at?

Thanks a lot for the suggestion. I thought Zotero was only for
bibliography, but I see it has a nice way to save web pages. I need to
see how much of a data silo it is.

Thanks again!

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Monthly Athmospheric CO₂, Mauna Loa Obs. 2016-12: 404.48, 2015-12: 401.85

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]

Re: How do you store web pages for reference?

[Michael Welle]

Hello,

Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> Hello Michael,
>
> On 2017-01-16 10:22, Michael Welle <mwe012008@gmx.net> writes:
>
>>> I'm looking for a workflow that allows me to save a web page for
>>> reference, ideally from Firefox. I know of org-protocol-capture-html
>>> (https://github.com/alphapapa/org-protocol-capture-html), which is
>>> perfect for pure-text pages, but I'm also looking for a solution for
>>> images-heavy pages. I've tried to simply save the page to PDF, but it
>>> does not preserve the links.
>>>
>>> Do you have suggestions?
>> maybe Zotero is something you want to have a look at?
>
> Thanks a lot for the suggestion. I thought Zotero was only for
> bibliography, but I see it has a nice way to save web pages. I need to
> see how much of a data silo it is.
yes, it can store web pages, index PDF files etc. There is some Org
integration, but I don't use that.

Regards
hmw

Re: How do you store web pages for reference?

[William Denton]

On 16 January 2017, Alan Schmitt wrote:

> Thanks a lot for the suggestion. I thought Zotero was only for
> bibliography, but I see it has a nice way to save web pages. I need to
> see how much of a data silo it is.

Zotero's wonderful, and with a Firefox add-on and an Emacs package you can 
easily search it and pull links into Org:

http://www.mkbehr.com/posts/a-research-workflow-with-zotero-and-org-mode/

Bill
-- 
William Denton :: Toronto, Canada :: https://www.miskatonic.org/
Caveat lector.

Re: How do you store web pages for reference?

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

Hello,

Thanks a lot for all the great answers. I'm leaning toward org-board at
the moment, but I've saved the research workflow article with zotero (in
org-board) to look at it in more details.

I'm very glad to see these solutions, as they will allow me to depend
less on DEVONthink, which is the last piece of software that keeps me on
macOS.

Thanks again,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Monthly Athmospheric CO₂, Mauna Loa Obs. 2016-12: 404.48, 2015-12: 401.85

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]

Re: How do you store web pages for reference?

[Charles A. Roelli]

Hi Alan,

On Mon, Jan 16 2017 at 09:48:38 am, Alan Schmitt wrote:

> I'm looking for a workflow that allows me to save a web page for
> reference, ideally from Firefox. I know of org-protocol-capture-html
> (https://github.com/alphapapa/org-protocol-capture-html), which is
> perfect for pure-text pages, but I'm also looking for a solution for
> images-heavy pages. I've tried to simply save the page to PDF, but it
> does not preserve the links.

You might also want to try org-board:

    https://github.com/scallywag/org-board

It offers archiving, diffing between archives, and anything `wget' can
do (see its manual for more details).  I haven't integrated it with
Firefox, but if you can

    a) Get the current URL from Firefox,
    b) Send it to Emacs,
    c) Open a dedicated web bookmark file buffer, create a heading for
       the URL, and run `org-board-archive',

then org-board would take care of archiving the site exactly as you see
it in the browser.

Cheers,
Charles

Re: How do you store web pages for reference?

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1443 bytes --]

Hi Charles,

On 2017-01-16 11:38, charles@aurox.ch (Charles A. Roelli) writes:

> On Mon, Jan 16 2017 at 09:48:38 am, Alan Schmitt wrote:
>
>> I'm looking for a workflow that allows me to save a web page for
>> reference, ideally from Firefox. I know of org-protocol-capture-html
>> (https://github.com/alphapapa/org-protocol-capture-html), which is
>> perfect for pure-text pages, but I'm also looking for a solution for
>> images-heavy pages. I've tried to simply save the page to PDF, but it
>> does not preserve the links.
>
> You might also want to try org-board:
>
>     https://github.com/scallywag/org-board
>
> It offers archiving, diffing between archives, and anything `wget' can
> do (see its manual for more details).  I haven't integrated it with
> Firefox, but if you can
>
>     a) Get the current URL from Firefox,
>     b) Send it to Emacs,
>     c) Open a dedicated web bookmark file buffer, create a heading for
>        the URL, and run `org-board-archive',
>
> then org-board would take care of archiving the site exactly as you see
> it in the browser.

This is great! Thanks a lot for the suggestion. I just gave it a try and
it works beautifully. (I haven't integrated it with Firefox, but I think
it should not be too difficult using org-protocol.)

Best,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Monthly Athmospheric CO₂, Mauna Loa Obs. 2016-12: 404.48, 2015-12: 401.85

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]

Re: How do you store web pages for reference?

[Karl Voit]

 On Mon, Jan 16 2017 at 09:48:38 am, Alan Schmitt wrote:

> I'm looking for a workflow that allows me to save a web page for
> reference, ideally from Firefox. I know of org-protocol-capture-html
> (https://github.com/alphapapa/org-protocol-capture-html), which is
> perfect for pure-text pages, but I'm also looking for a solution for
> images-heavy pages. I've tried to simply save the page to PDF, but it
> does not preserve the links.

I am using the Firefox plugin Shelve[1] which stores all of my web
pages visited. Those HTML files are written with an ISO time-stamp
in their file name. Therefore, my Memacs filename module (see sig)
is indexing all visited URLs and they appear on my agenda.

So I do have a direct link between my agenda and the HTML files of
all web pages I have visited.

[1] https://addons.mozilla.org/en-US/firefox/addon/shelve/

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: How do you store web pages for reference?

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1670 bytes --]

Hello Karl,

On 2017-01-16 15:43, Karl Voit <devnull@Karl-Voit.at> writes:

>  On Mon, Jan 16 2017 at 09:48:38 am, Alan Schmitt wrote:
>
>> I'm looking for a workflow that allows me to save a web page for
>> reference, ideally from Firefox. I know of org-protocol-capture-html
>> (https://github.com/alphapapa/org-protocol-capture-html), which is
>> perfect for pure-text pages, but I'm also looking for a solution for
>> images-heavy pages. I've tried to simply save the page to PDF, but it
>> does not preserve the links.
>
> I am using the Firefox plugin Shelve[1] which stores all of my web
> pages visited. Those HTML files are written with an ISO time-stamp
> in their file name. Therefore, my Memacs filename module (see sig)
> is indexing all visited URLs and they appear on my agenda.
>
> So I do have a direct link between my agenda and the HTML files of
> all web pages I have visited.
>
> [1] https://addons.mozilla.org/en-US/firefox/addon/shelve/

This plugin looks interesting, but it seems to rely on the existing
functionality of Firefox to save web pages. As I want to save a page
with its picture and CSS, I would need to choose “Web page, complete”,
but the FF documentation says “This choice allows you to view it as
originally shown with pictures, but it may not keep the HTML link
structure of the original page”, which worries me a little.

Do you only save the html or the pictures as well. If it's the latter,
have you had any issues about links not being preserved?

Thanks,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Monthly Athmospheric CO₂, Mauna Loa Obs. 2016-12: 404.48, 2015-12: 401.85

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]

Re: How do you store web pages for reference?

[Karl Voit]

Hi Alan,

* Alan Schmitt <alan.schmitt@polytechnique.org> wrote:

> On 2017-01-16 15:43, Karl Voit <devnull@Karl-Voit.at> writes:
>
>> I am using the Firefox plugin Shelve[1] which stores all of my web
>> pages visited. Those HTML files are written with an ISO time-stamp
>> in their file name. Therefore, my Memacs filename module (see sig)
>> is indexing all visited URLs and they appear on my agenda.
>>
>> So I do have a direct link between my agenda and the HTML files of
>> all web pages I have visited.
>>
>> [1] https://addons.mozilla.org/en-US/firefox/addon/shelve/
>
> This plugin looks interesting, but it seems to rely on the existing
> functionality of Firefox to save web pages. As I want to save a page
> with its picture and CSS, I would need to choose =E2=80=9CWeb page, complet=
> e=E2=80=9D,
> but the FF documentation says =E2=80=9CThis choice allows you to view it as
> originally shown with pictures, but it may not keep the HTML link
> structure of the original page=E2=80=9D, which worries me a little.

Well, this is a hard problem to do differently: when you save a web
page A which has an URL to B, do you want to end up with a local
copy of A that links to the local copy of B (which you might not
have at all) or an URL to online-B. The latter one is easy (no
change when downloading).

> Do you only save the html or the pictures as well. If it's the latter,
> have you had any issues about links not being preserved?

I save everything.

My settings (with self-translated terms from German):

Settings: MIME: Webpage, complete (HTML)

My default shelve:

Template: 
C:\Users\karl.voit\browser_history\%Y-%M_myhostname\%Y-%M-%DT%h.%m_%{host}_-_%{title}.html

MIME: Standard

This way, I end up with all web pages stored in my file system. When
I open an URL, the browser shows my local copy. Sometimes, included
stuff is not loaded correctly. All links point to their original
target (of course). So in case I want to stay local, I do not click
on any link in my local copy.

I mainly navigate through my agenda and its links: agenda -> local
copy -> back to agenda -> next local copy -> back to agenda -> ...

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: How do you store web pages for reference?

[Robert Horn]

There is also a Firefox plugin "ScrapBook X", which is a successor to
Scrapbook.  It can capture the web page alone (with links to outside
world) and allows you to select by depth or link additional pages
that are also to be captured.  (If you have infinite time and storage
with the right links you might attempt to capture the entire Internet.
Something like capture all pages to link depth 1000 comes to mind.)

I use it to capture a variety of things.  Each capture is stored in a
directory tree of html, css, etc. rooted at a time-date tag for when the
capture was performed.

I have not seen nor attempted to integrate it with org or any other
tools.  This is feasible in theory, since the file
<date-time>/index.html is a valid page starting point and links are been
rewritten as appropriate.  Something like "firefox
scrapbook-root/20170115205014/index.html" would be a proper reference.
The more the page content becomes active content like javascript, the
less likely that the page capture will save what you want, but that's
inherent with active content.

It would be nice to capture more metadata (like Zotero), but it only
preserves minimal metadata about the capture.

R Horn
rjhorn@alum.mit.edu

Re: How do you store web pages for reference?

[Scott Otterson]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

I use Evernote, which has handy annotation, tagging, and search functions.

The offline desktop client also has a "Copy URL" button, which puts the
Evernote server location where you stored the page into your clipboard.  I
paste that into an org-mode link.  If you're on Linux, you can copy the URL
from your browser.

Scott

[-- Attachment #2: Type: text/html, Size: 449 bytes --]

Re: How do you store web pages for reference?

[Bob Newell]

Scott Otterson <scotto@sharpleaf.org> writes:

> I use Evernote, which has handy annotation, tagging, and search

I use this also, but I wanted something working within Emacs similar to
the Evernote web-clipper. Org-board is very powerful but requires an
intermediate step of creating a headline with a URL property.

I wanted something fast, so I wrote an org-mode web clipper. It works
with both w3m and eww, making use of an org-capture template and a
little custom code. Images are captured if you've set up w3m or eww to
display them. CSS, etc, is obviously not captured.

I've put it here if it's of any interest to anyone. It's pretty much a
prototype but I use it daily. Emacs 25 required.

http://www.bobnewell.net/publish/35years/webclipper.html

-- 
Bob Newell
Honolulu, Hawai`i
* Via Gnus/BBDB/Org/Emacs/Linux *

Re: How do you store web pages for reference?

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

On 2017-03-16 09:04, Bob Newell <bobnewell@bobnewell.net> writes:

> Scott Otterson <scotto@sharpleaf.org> writes:
>
>> I use Evernote, which has handy annotation, tagging, and search
>
> I use this also, but I wanted something working within Emacs similar to
> the Evernote web-clipper. Org-board is very powerful but requires an
> intermediate step of creating a headline with a URL property.

I have this automated with org-capture and a hook:

#+BEGIN_SRC emacs-lisp
  (setq org-board-capture-file "my-org-board.org")

  (setq org-capture-templates
        `(...
          ("c" "capture through org protocol" entry
           (file+headline ,org-board-capture-file "Unsorted")
           "* %?%:description\n:PROPERTIES:\n:URL: %:link\n:END:\n\n Added %U")
          ...))

  (defun do-org-board-dl-hook ()
    (when (equal (buffer-name)
                 (concat "CAPTURE-" org-board-capture-file))
      (org-board-archive)))

  (add-hook 'org-capture-before-finalize-hook 'do-org-board-dl-hook)
#+END_SRC

Best,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Monthly Athmospheric CO₂, Mauna Loa Obs. 2017-02: 406.42, 2016-02: 404.04

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]

Re: How do you store web pages for reference?

[Adam Porter]

Thanks for sharing that, Bob.  You mentioned on that page that you
couldn't get it working with a capture template; here's what I use,
which does use a capture template.  Maybe this will be helpful:

#+BEGIN_SRC elisp
(defun ap/org-capture-w3m ()
    "Call org-capture with the `W' template.  Use this from within w3m.

Depends on Magnar Sveen's s.el package."
    (interactive)
    (if (use-region-p)
        (progn
          ;; Replace current kill with one wrapped in an org quote block
          (org-w3m-copy-for-org-mode)
          (kill-new (s-wrap (s-trim (current-kill 0 t)) "\n\n#+BEGIN_QUOTE\n" "\n#+END_QUOTE") t))
      ;; Prevent whatever happens to be in the kill-ring from being captured
      (kill-new ""))
    (org-capture nil "w3"))

;; Associated capture template
("w3" "w3m capture" entry
      (file "")
      "* %a :website:

%U %?%c")

#+END_SRC

Re: How do you store web pages for reference?

[Peter Salazar]

[-- Attachment #1: Type: text/plain, Size: 1084 bytes --]

By the way, you can definitely save a PDF with links if you prefer that. I
use DevonThink, and I use this snippet to save a PDF of a webpage, complete
with links.

javascript:window.location='x-devonthink://createPDF?title='+encodeURIComponent(document.title)+'&location='+encodeURIComponent(window.location)+'&referrer='+encodeURIComponent(document.referrer)+'&paginated=No&width='+window.innerWidth;

On Mon, Jan 16, 2017 at 3:48 AM, Alan Schmitt <
alan.schmitt@polytechnique.org> wrote:

> Hello,
>
> I'm looking for a workflow that allows me to save a web page for
> reference, ideally from Firefox. I know of org-protocol-capture-html
> (https://github.com/alphapapa/org-protocol-capture-html), which is
> perfect for pure-text pages, but I'm also looking for a solution for
> images-heavy pages. I've tried to simply save the page to PDF, but it
> does not preserve the links.
>
> Do you have suggestions?
>
> Thanks,
>
> Alan
>
> --
> OpenPGP Key ID : 040D0A3B4ED2E5C7
> Monthly Athmospheric CO₂, Mauna Loa Obs. 2016-12: 404.48, 2015-12: 401.85
>

[-- Attachment #2: Type: text/html, Size: 1669 bytes --]

Re: How do you store web pages for reference?

[Adonay Felipe Nogueira]

Note 1: You might need to load "org-bibtex".

Note 2: "org-bibtex" only accepts some properties to be exposed to
org-capture, it's not an exact parser like "bibtex" package. As a future
suggestion: I would like to suggest "org-bibtex" to be improved so as to
take all fields parsed by the "bibtex" package, but I don't know how to
do that yet. This would allow non-default BibTeX entries and fields such
as the "url" field (which is already supported), and other ones such as
"urlaccessdate" (this one is used by ABNTeX) and "note" to be made
available transparently through org-capture templates.

Note 3: Each time a reference entry is found in a refere file, such
entry should be removed from the references list, so as to avoid such
reference key being matched twice, even if there is a repeated reference
file. This is currently the only control mechanism that exists for this
hack.

Note 4: As an improvement, I still have to find a way to use an
org-capture template without being required to bind such template to a
key, as this can conflict with other templates or keys. I'm accepting
suggestions for such as long as I can figure out how it exactly works.

Note 5: I'm not a programmer, so I'm slow on picking up programming
concepts or languages' syntax.

Note 6: You must evaluate the first source code block once for the first
time the file is openned and everytime you adapt/change/modify the
source code block (for both cases: unless you preset the
"org-capture-templates" to reflect the same behavior in your overall
Emacs setup. For the last case: you must also make sure that there are
no old duplicates of such items in "org-capture-templates" otherwise
they might cause obsolete results or always raise an error).

Note 7: You must evaluate the second source code block
(import-references) once before exporting if either the "Reference" headline is
empty, if the references inside "References" headline is outdated, and
if the overall document contains a broken link (this one will already
cause an error that will stop exporting if broken-links #+OPTION is set
to nil).

Having read the notes above, I do the following:

# Begin of Org file
#+OPTIONS: d:(not "BibTeX_Importer")

:BibTeX_Importer:
#+BEGIN_SRC emacs-lisp :results silent
  (add-to-list 'org-capture-templates
	       '("B" "BibTeX references"
		 item (file+headline
		       reference-parent-file
		       "References")
		 "<<%:key>> %:author. *%:title*. %:year. %:annote. _[[%:url][Source]]_."
		 :immediate-finish t))
#+END_SRC

#+NAME: reference-files
- /home/adfeno/Publico/Referencias_BibTeX.bib

#+NAME: references
- Stallman-2015-Free_Software_and_Your_Freedom

#+NAME: import-references
#+BEGIN_SRC emacs-lisp :var reference-parent-file=(buffer-file-name) :var reference-files=reference-files :var referenced-entries=references
  (dolist (current-reference-file reference-files)
    (find-file (car current-reference-file))
    (let ((reference-file-buffer (buffer-name)))
      (dolist (current-entry referenced-entries)
	(set-buffer reference-file-buffer)
	(when (bibtex-find-entry (car current-entry))
	  (org-capture nil "B")
	  (setq referenced-entries (delete current-entry referenced-entries))))))
#+END_SRC
:END:

* References
# End of Org file

Respectfully, Adonay.
-- 
* https://libreplanet.org/wiki/User:Adfeno
* Palestrante e consultor sobre software livre (não confundir com
  gratis).
* "WhatsApp"? Ele não é livre, por isso não uso. Iguais a ele prefiro
  Ring, ou Tox. Quer outras formas de contato? Adicione o vCard que
  está no endereço acima aos teus contatos.
* Pretende me enviar arquivos .doc, .ppt, .cdr, ou .mp3? OK, eu
  aceito, mas não repasso. Entrego apenas em formatos favoráveis ao
  software livre. Favor entrar em contato em caso de dúvida.
* "People said I should accept the world. Bullshit! I don't accept the
  world."
                                                 --- Richard Stallman

Re: How do you store web pages for reference?

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Hello Peter,

On 2017-03-13 13:43, Peter Salazar <cycleofsong@gmail.com> writes:

> By the way, you can definitely save a PDF with links if you prefer that. I use DevonThink, and I use this snippet to save
> a PDF of a webpage, complete with links.
>
> javascript:window.location='x-devonthink://createPDF?title='+encodeURIComponent
> (document.title)+'&location='+encodeURIComponent(window.location)+'&referrer='+encodeURIComponent
> (document.referrer)+'&paginated=No&width='+window.innerWidth;

Thank you for the suggestion, I was doing something similar, but as I'm
planning to switch to Linux I need to stop using DevonThink.

I'm now using org-board (https://github.com/scallywag/org-board). It
works great and I even tweaked it to add the files to IPFS after
downloading them.

Best,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Monthly Athmospheric CO₂, Mauna Loa Obs. 2017-02: 406.42, 2016-02: 404.04

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]