Preliminary Results from EAGER Survey and Interviews

[nniiicc]

We distributed a survey to grant PI's across NSF directorates.

• Predicted whether the grant abstract would produce software
• Used this to determine which grant to email a PI about (e.g. for those with multiple grants)
• In total we sent about 34K surveys out, and with a high bounce rate we got about a 4.5% response rate
• Responses were of two varieties: If the grant did not produce software the PI responded "No Code"; if the software did produce software they were directed to a web-form -> demo: https://nniiicc.github.io/eager-survey/#/grant/test
• Below are results from the form responses and emails w/ "no code"

We offered survey respondents $50 to sign up for a 30 minute interview

• We did 13 interviews
• No significant findings to report
• High-level quotes https://docs.google.com/document/d/1dbhPPA9H6hAz4PA_lbK8T-WgtWLXqRDv1Gqh3FUAvOs/edit
• Interview questions https://docs.google.com/document/d/1nVbSFLYaXzS6HVqgJM0bKqGoHXfSCc5D4oOKHDzXNKU/edit

Number of parsed email responses: 1629

Number of app responses: 892
Number of email responses: 1627

Response rate: 4.59%


--------------------------------------------------------------------------------

Percentage of each code production response: 
	produced-and-all-available: 41.37% (369)
	produced-but-not-available: 38.0% (339)
	produced-and-some-available: 20.63% (184)

Total number of values provided (or selected) from repo gather: 1229

Percentage of responses that include at least one source host: 47.98% (428)

Percentage of each data gathering source:
	Continuous Textboxes: 91.68% (507)
	Github Repo Selection: 6.33% (35)
	Both: 1.99% (11)

Distribution of privacy of code responses:
	not-ready-for-public: 56.98% (298)
	utility: 36.9% (193)
	time: 33.84% (177)
	sensitive-data: 6.88% (36)
	other: 12.81% (67)
	intellectual-property: 16.83% (88)

Percentage of of each planned support response:
	none: 33.87% (211)
	some: 33.71% (210)
	all: 32.42% (202)

Distribution of support reasoning responses:
Did Plan To Support reasoning responses:
	research: 51.77% (321)
	used-by-others: 40.16% (249)
	teaching: 16.13% (100)
	other: 15.48% (96)
	funding: 9.19% (57)

Did Not Plan To Support reasoning responses:
	no-funding: 19.68% (122)
	no-time: 16.94% (105)
	no-research: 13.71% (85)
	no-use: 14.03% (87)
	no-teaching: 12.26% (76)
	no-credit: 14.19% (88)

Distribution of code users responses:
	field: 84.8% (212)
	lab: 68.8% (172)
	adopted: 40.8% (102)
	other: 4.8% (12)

--------------------------------------------------------------------------------

Stats from Abstract Prediction
	Precision: 62.22%
	Recall: 63.68%
	F1: 62.66%
Stats from Outcomes Prediction
	Precision: 62.64%
	Recall: 62.34%
	F1: 62.48%