Create web page / section with list of publications using SasView. (Trac #483)

[ajj]

This will require some trolling of the literature...

Migrated from http://trac.sasview.org/ticket/483

{
    "status": "assigned",
    "changetime": "2019-03-23T14:13:50",
    "_ts": "2019-03-23 14:13:50.376216+00:00",
    "description": "This will require some trolling of the literature...",
    "reporter": "ajj",
    "cc": "",
    "resolution": "",
    "workpackage": "Support Infrastructure",
    "time": "2015-11-24T13:48:02",
    "component": "SasView",
    "summary": "Create web page / section with list of publications using SasView.",
    "priority": "critical",
    "keywords": "",
    "milestone": "Admin Tasks",
    "owner": "krzywon",
    "type": "task"
}

[butlerpd]

Trac update at 2015/12/19 18:44:58: butler changed workpackage from "SasView Bug Fixing" to "SasView Admin"

[smk78]

Trac update at 2015/12/23 15:40:36: smk78 commented:

See http://www.sasview.org/impact.html (note: this is not yet linked from the other web pages).

This list of publications was generated by searching the publications databases of the ACS and the RSC with the search terms "sasview" OR "sansview" OR "DMR-0520547" (the DANSE grant number). Results were collated, re-formatted, input into Excel, sorted, and the duplicates stripped out. The resulting worksheet was then sorted on the journal, output as text, re-formatted for html presentation, and pasted into this web page.

In the process there some very clearly 'suspicious' records (way too early for even DANSE, crazy journal, etc) were identified. These have been removed. ''However'', it is highly likely that the records remaining still contain some bogus entries. The only way to be certain is to go through them one-by-one! :-)

NOTE: The ACS/RSC search engines are very good in that they search all the text in ''both'' the manuscript AND the supplementary information. Capability that far exceeds what Web of Science can do!

[smk78]

Trac update at 2015/12/23 17:23:04: smk78 commented:

On Mathieu's advice have removed all records earlier than 2007 (bit of a no-brainer, really)

Mat adds that we should look carefully at those records that remain that have publication years of 2007, 2008 & 2009.

[ajj]

Trac update at 2017/04/04 21:58:25:

• ajj changed owner from "" to "smk78"
• ajj changed status from "new" to "assigned"

[butlerpd]

Trac update at 2019/02/27 17:48:34:

• butler commented:

Jeff Krzywon had an infrastructure to help with this for NIST SANS instruments. He started a githup repo (pub-tracker) to duplicate that but has since moved the NIST infrastructure to a different approach worked out by Brian Maranville. This should be done soon so we have a sustainable way of finding, entering, and storing the information.

• butler changed owner from "smk78" to "krzywon"
• butler changed workpackage from "SasView Admin" to "Support Infrastructure"

[butlerpd]

Trac update at 2019/03/23 14:13:50: butler changed priority from "major" to "critical"

[krzywon]

The site is up and running, but needs a few finishing touches to be 100% complete.

How it all works:

1. The website displaying the publications is live, but not linked to from sasview.org yet - http://www.sasview.org/publications/ which is generated using the publications.md file at https://github.com/SasView/sasview.github.io
2. The site is updated when the list of publications at https://www.zotero.org/groups/2309096/sasview is modified. Currently @ajj and I are the only two with write access to the zotero group.
3. The site is generated using source from https://github.com/krzywon/saspubs. instrument_zotero_feed.py generates a json file with all of the bibliographic information and then runs citeproc_to_html.py which converts the json to markdown using the format outlined in ./static/Sasview_linktitle.csl.
4. On the DANSE server, a job runs push_updates.sh every hour. The bash script assumes a single argument be the location of an SSH key file that is linked to a github account with write access to sasview.github.io. The bash script pulls the latest updates from saspubs and sasview.github.io, runs instrument_zotero_feed.py, commits any changes made to publications.md ("publications auto-update "), and then pushes.

Things left to do:

1. Create a github account dedicated to preforming automated tasks like this. This account should only have write access to the specific files it needs to update (sasview.github.io/publications.md is the only one currently). This account could be used for other automated tasks link online docs and creating news items after a release build, among others.
2. Create a DANSE server account dedicated to automated sasview tasks, and have this account run the job. The job should run more often than every hour.
3. Give more people access to the saspubs repo, or fork it into sasview.
4. Give more people write access to the zotero group. @butlerpd, @wpotrzebowski, @smk78 were my initial thoughts.
5. Sanitize the publications list. Many need verifying and some are lacking a DOI.
6. Once all of this is done, add a link the pubs site from the About menu on sasview.org.

[krzywon]

Updates to to the list of to dos:

1. I've created a generic github account, sasviewbot (not to be confused with sasview-bot) and given it access to krzywon/saspubs, but it still needs access to sasview.github.io.
2. I've created a generic DANSE server account, sasviewbot, that I will use once the github account can access both saspubs and the web repo.
3. I've sent invitations to @butlerpd, @wpotrzebowski, @smk78, and @mdoucet, but only @butlerpd has accepted so far.
4. I'm worked through most of the 2019 publications, but a few need updated DOIs. 2018 and 2019 should be done today.

[krzywon]

I've gone through all of the publications on the list and updated them in zotero. At this point, there are only a few administrative things to do, so I added a link to the page under About on sasview.org.

[smk78]

Ahh... I got some email purporting to be from you a week or so ago @krzywon but as I'd never heard of zotero I didn't know whether to trust it and wasn't going to click the link! I sent you an email but you never responded so I canned the invite! Ok, have just registered and applied to join the sasview group.

Maybe then someone can explain what we do with it... :-)

[krzywon]

What's this for, how does it all work, and what do I need to do?

Zotero is a free, easy-to-use tool to help you collect, organize, cite, and share research.
- zotero.com

We are using Zotero to track and collect papers that used SasView so we have a metric to show how much SasView is used. The NCNR moved to this system a year and a half ago and it's very easy to use.
Zotero client setup:
Download the client (available for all platforms), and install it. In the client preferences, on the general tab under file handling, de-select all automatic file attachments. The free version only offers a limited amount of storage space and I've run into this limit for the NCNR SANS publications with PDFs attached to everything (~1000 pubs right now).
Under the sync tab, link your account and select 'sync automatically' for automated two way updates.
Zotero usage:
Click the "add item(s) by identifier" button (looks like a magic wand) and paste in a DOI. You can add items by hand, but the Zotero system will automatically pull in all the bibliographic data, author info, etc with just the DOI.
Automated Tasks:
A script runs every five minutes on the DANSE server that will pull any updates from Zotero, update the publications page, and push the changes to the web site repo at sasview.github.io.

For more information, Zotero also has published a quick start guide.

[ajj]

I believe this is essentially complete - do we have a wiki page with instructions?

[krzywon]

I created a wiki page for managing the publications, but it is not finished or linked to yet. Once the page is complete, I will close this ticket and move on to SasView/saspubs#1
https://github.com/SasView/sasview/wiki/DevNotes_Processes_PublicationsWebPageInstructions

[krzywon]

The instructions page is now finished. I also linked to it from the web page instructions page.

Publication page editing instructions

[butlerpd]

Reviewing this task. I have the following comments:

• As with the web page instructions (which also need fixing in this regard) I think it is most important, somewhere at the top to explain the difference between adding pubs to the web page and changing HOW that page is displayed. The latter requiring actual coding in saspubs repo while the former means ONLY editing the database on DANSE using zotero.
• The biggest problem in zotero is the duplicates. I think something about that needs to be added to the usage notes? including dealing with pubs with a DOI but no pub date that later need updating.
• Probably worth re-iterating in the usage notes the need to TURN off auto downloading PDFs ... I think I added a bunch early on before I realized and figured out... still not sure I fixed in all my local instances of zotero ;-( we maybe should figure out how to go back and delete all those from the database.
• I think it is worth also mentioning the browser plugin that allows pulling in the info on pubs you are reading as this is when you verify that the pub is legit and you just want to add now.

[krzywon]

• As with the web page instructions (which also need fixing in this regard) I think it is most important, somewhere at the top to explain the difference between adding pubs to the web page and changing HOW that page is displayed. The latter requiring actual coding in saspubs repo while the former means ONLY editing the database on DANSE using zotero.

I'll add a section to the page and update this ticket once it's done.

• The biggest problem in zotero is the duplicates. I think something about that needs to be added to the usage notes? including dealing with pubs with a DOI but no pub date that later need updating.

I created a ticket in saspubs to handle this issue. I've talked to Brian Maranville about catching any changes to NCNR articles, and know how to do it already, but other instrument classes didn't want that feature. We want it in SasView and this repo is separate from the NCNR one. SasView/saspubs#2

• Probably worth re-iterating in the usage notes the need to TURN off auto downloading PDFs ... I think I added a bunch early on before I realized and figured out... still not sure I fixed in all my local instances of zotero ;-( we maybe should figure out how to go back and delete all those from the database.

Reiterated.

• I think it is worth also mentioning the browser plugin that allows pulling in the info on pubs you are reading as this is when you verify that the pub is legit and you just want to add now.

I added a mention to this

[krzywon]

I've added instructions on how saspubs works to the page.

[butlerpd]

a magic script would be good :-) but does it handle duplicates in general? we now have over 100 pubs a year (I still have not pulled those in because it takes some concentration :-) so I tend to bring things in in "bulk" but sometimes there are a couple of duplicates. It is hard to keep track so some tools to ensure that we aren't pulling in duplicates in the midst of a slew of new ones would be welcome clearly. Otherwise I've found that pulling in a set then using zotero "duplicates" to show all duplicates in the database then deleting the oldest ones of each duplicate works reasonabley IF you don't have auto synching on - as you say once they are on the website, hard to get rid of (or even notice that it is there) the duplicate.

[butlerpd]

Only thing I saw was that after turning auto Sync off the instructions never mention that once you have added your files you need to hit Sync again to make the additions live. I have added that point. If you agree we can move this to done.