We’re considering ways to effectively do hover previews for particular common external links (such as wikipedia and arxiv), but for the immediate future, all external links just provide a simple “url” hoveover.
I’ve been helping gwern develop exactly this feature for gwern.net (which now has hover previews for internal links, in-page links, all external links, and even locally hosted PDFs and other media), so I’d be happy to share how we did it, tips, etc. Feel free to find me or gwern on IRC to chat about this.
(It’s particularly easy to do this for Wikipedia, since they’ve got an excellent REST API.)
The basic idea behind the popups is that at the Markdown->HTML compile time, I do a lookup in a hashmap of (URL, (Title, Author, Date, DOI, Summary)), and if there is an entry, it gets inlined into the HTML as some quiet metadata; if there isn’t, various scripts are called to try to generate metadata, and if nothing works, it falls back to a headless web browser taking a screenshot and saving it to a file. Then at runtime in the user browser, some JS checks every link for the metadata and if it exists, and the user mouses-over, it popups a form with the metadata filled in. If there is no metadata, it tries to fetch SHA1(URL) from a gwern.net folder under the assumption there will be the fallback screenshot.
There are a lot of fiddly details in how exactly you scrape from Arxiv or Pubmed Central or use the WP REST API, unfortunately. So many links must be handled by hand-writing definitions.
Yeah, I was happy when I saw you/gwern were also doing this. We’ve looked a bit at the your implementation and may ping you about the details once we get to our own external links.
I’ve been helping gwern develop exactly this feature for gwern.net (which now has hover previews for internal links, in-page links, all external links, and even locally hosted PDFs and other media), so I’d be happy to share how we did it, tips, etc. Feel free to find me or gwern on IRC to chat about this.
(It’s particularly easy to do this for Wikipedia, since they’ve got an excellent REST API.)
The basic idea behind the popups is that at the Markdown->HTML compile time, I do a lookup in a hashmap of
(URL, (Title, Author, Date, DOI, Summary)), and if there is an entry, it gets inlined into the HTML as some quiet metadata; if there isn’t, various scripts are called to try to generate metadata, and if nothing works, it falls back to a headless web browser taking a screenshot and saving it to a file. Then at runtime in the user browser, some JS checks every link for the metadata and if it exists, and the user mouses-over, it popups a form with the metadata filled in. If there is no metadata, it tries to fetchSHA1(URL)from agwern.netfolder under the assumption there will be the fallback screenshot.There are a lot of fiddly details in how exactly you scrape from Arxiv or Pubmed Central or use the WP REST API, unfortunately. So many links must be handled by hand-writing definitions.
The relevant files in call order:
https://www.gwern.net/hakyll.hs
https://www.gwern.net/LinkMetadata.hs
https://www.gwern.net/linkAbstract.R
https://www.gwern.net/linkScreenshot.sh
https://www.gwern.net/static/js/popups.js
the generated/hand-written ‘database’ files:
https://www.gwern.net/static/metadata/auto.hs
https://www.gwern.net/static/metadata/custom.hs
https://www.gwern.net/static/metadata/preview.hs
Yeah, I was happy when I saw you/gwern were also doing this. We’ve looked a bit at the your implementation and may ping you about the details once we get to our own external links.