Background

In 2015 we began creating an evaluation framework for understanding more about Public Lab's constituency and how we, as a non-profit, can better serve them. See https://publiclab.org/notes/Shannon/03-01-2016/creating-an-evaluation-framework-for-public-lab. Thanks to support from the Fund for Shared Insight's Listen4Good program and the Rita Allen Foundation, we have been able to do in-depth work collecting and analyzing data about one particular group of constituents: those who attend Public Lab in-person events. The idea was that the feedback from post-event surveys would help us understand who is coming to the events, why, what people's experiences are, what we could do better, and how people intend to use what they learned after the event. The feedback from these surveys will help us to shape our events and outreach.

Round One for the Public Lab Listen4Good project included 77 respondents who attended Public Lab events between July and December 20th, 2016. Attendees participated in events that ranged from the Regional Barnraising and Annual Barnraising, to local workshops such as mapping and spectrometer builds. The Round One report can be found here: https://publiclab.org/notes/stevie/02-09-2017/evaluation-listen4good-round-one-results Round Two

Data: Round two included 80 respondents who attended in-person events between April - November 2017: 78 from in-person events including the West Virginia Barnraising, the Annual Gulf Coast Barnraising, stormwater workshops in New Orleans, and air monitoring in Wisconsin; an additional 2 were submitted by online attendees of OpenHour, our monthly topical webinar.

Methods: The Listen4Good program includes six core questions, one of which is based on a framework called the "Net Promoter System" (NPS) which has long been used in customer engagement circles. This framework asks the question, on a scale of 1-10: 1 being not at all likely, and 10 being extremely likely, "How likely is it that you would recommend Public Lab to a friend or family member?" From this question, a score can be determined that can help you compare how your organization, or company, is doing relative to others who provide similar services. People who select anything between a 1-6 are considered detractors, this adds to a negative overall score for the organization. Those who score a 7-8 are considered passive and don't add or take away score, those who score a 9-10 are considered promoters and add positive points to your overall score. The total score subtracts the percent of detractors from the percent of promoters. We then added five additional custom questions more directly related to our mission for a total of eleven questions on the survey.

People responded to these questions through an online Survey Monkey link or on a printed paper survey which staff then entered into SurveyMonkey. Analysis was primarily done inside of SurveyMonkey’s platform which offers graphs and statistical analysis as well as benchmarks from across thousands of other organizations, as well as in Excel. Results -- Highlights

Participants at in-person events were 50% People of Color in round two (2017), up from 21% People of Color in round one (2016), meaning that Public Lab greatly improved demographic representation from last year’s results.
There was overwhelming feedback that the Public Lab community is a respectful community, as seen in the responses to the question: “how often do members of the Public Lab community treat you with respect?” 99% of responses were “always” (86%) and “mostly” (13%). One person choose “sometimes.”
Among those who said their reason for attending was “to join an organizing effort or campaign about this place, their NPS was a full +100.
Among those who said their reason for attending was for a “school trip,” meaning they didn’t actually choose to be there, their NPS was a -50.

Results -- data and analysis

Based on the demographic categories offered by the survey instrument, here is a table of how people who attended in-person events chose to self-identify their race and ethnicity: Response Percentage % # Respondents White 50% 37 Black or African American 32.43% 24 Hispanic or Latino 2.7% 2 Asian 6.76% 5 American Indian or Alaska Native 1.35% 1 Multiracial or multiethnic 2.7% 2 Some other race or ethnicity (please specify) 2.7% 2 -- # of respondents to this question (out of 80) 74

Participants at in-person events self-identified as 50% People of Color in round two (2017), up from 21% People of Color in round one (2016), meaning that Public Lab is getting closer to a representative distribution. Mission

We asked about two statements specific to our mission:

Participating in Public Lab helps me become more engaged in environmental science/monitoring.
Participating in Public Lab events provides me with tools that help me take action on my own environmental interests and concerns.

...which elicited high levels of agreement across the entire cohort: image description

89% somewhat or strongly agree that “Participating in Public Lab helps me become more engaged in environmental science/monitoring”
87% somewhat or strongly agree that “Participating in Public Lab events provides me with tools that help me take action on my own environmental interests and concerns”
None of the “Strongly disagree” / “Somewhat disagree” choices were from People of Color; one Person of Color chose “Neither disagree nor agree”

Respectful community

There was overwhelming feedback across the entire cohort that the Public Lab community is a respectful community, as seen in the responses to the question: “how often do members of the Public Lab community treat you with respect?”

image description Net Promoters Score

Net Promoters Score: As we understand it, the Listen4Good model is designed for service organizations that “directly meet people’s needs.” Public Lab is the inverse of a direct service agency in that we are a peer-to-peer community in which the staff and active members facilitate among people by bridging between domains of experience / expertise in order to address complex, real world issues.

For newcomers, those who have “never engaged before today” or have “engaged once or twice before, our NPS score is 43, or “good,” the same as it was in Round One.

Respondents were 52% female, and 48% male. Across all ages:

average female NPS is 65 (positive, and above the benchmark of 63)
average male NPS is 48 (positive, but below the benchmark of 63)

Our total NPS for Round 2 was 37 based on 42 promoters, 24 passives, 13 detractors, at a time when the benchmark NPS was 63. This is significantly lower than our very high score in Round One of 82, based on 64 promoters, 10 passives and 2 detractors, at a time when the benchmark was 50.

Looking more closely, the sharpest divide in NPS is associated with the reason someone attended the event:

Among those who said their reason for attending was “to join an organizing effort or campaign about this place, their NPS was a full 100.
Among those who said their reason for attending was for a “school trip,”, their NPS was a -50.

This shows that motivation is critical to how an event is experienced. We can only speculate if young people on a school trip -- who expressed lots of enthusiasm in person as noted by staff present on field trips -- might have seen a “6” as not quite that negative since its “above the midpoint” or “above average;” however NPS considers a 6 on a scale of 1-10 to be a “detractor.” Q10 What best describes your reasons for coming to this event? Stated motivation for attending # Respondents The event was recommended to me / I was invited: NPS +36 25 The topic interested me: NPS +78 18 I wanted to join an organizing effort or campaign about this issue: NPS +38 10 I want to collaborate with Public Lab / I am a partner / collaborator / I am from a co-hosting collaborating organization: NPS +50 6 I wanted to join an organizing effort or campaign about this place: NPS +100 5 The event was a school trip or field trip: NPS -50 4 Q2 What is Public Lab good at?

Respondents were asked what Public Lab is good at in an open ended question, visualized here as a word cloud:

image description

We then categorized the free text responses into groups with the following percentage breakdown:

40% collaboration (31 respondents) This included responses such as: “connecting people across boundaries,” distributing tasks, organizing, “working together as a TEAM.” Space to teach and learn.
26% environmental exploration (20 respondents) This included responses such as: “engaging people in exploring and understanding their environment,” “coming together for environmental changes,” “taking images and seeing a large space of land,” “ecological awareness,” “creative environmental justice,” “health and improvement of local communities in New Orleans”
21% accessibility (16 respondents) This included responses such as: “bringing a tool into the public grasp,” “making resources extremely available to everyone not just scientists and researchers,” “explaining scientific terms more easily for the layman,” “documentation,” “breaking down information into an understandable format,” “explaining,” “informing”
14% being open and inviting (11 respondents) This included responses such as: “facilitating open and welcoming conversations,” “creating inviting and friendly spaces for public participation,” “connecting with youth,” “creating a space where everyone can come in and be treated at an equal level no matter what their background”
9% data / science (7 respondents) This included responses such as: “making scientific observations easily and simply performed by the public,” “recording data,” “stats,” “technology”
4% focusing on "DIY / low-cost (3 respondents) This included responses such as: “low cost problem solving,” “utilizing low budget tools to effectively do things better,” “bringing good affordable solutions to real problems”

image description Q3 What could Public Lab do better?

Respondents were asked what could Public Lab do better in an open ended question, visualized here as a word cloud:

image description

Notably the largest category was “nothing” [that Public Lab could do better], which occasionally also included “encouragement”

24% nothing (14 respondents) This included responses such as: “I can't think of anything that Public Lab could do better;” “Their visit was all that I hoped it would be and BETTER;” “Everything was good;” “Nothing they are already great;” “Honestly everything like the planning and ideas are spot on;” “Just keep pushing until you find a way”

19% clarity / communication (11 respondents) This included responses such as: communicating what PL does through social media, describing/ celebrating successes and accomplishments; more excitement in presentation; effect/ public policy/perception, advertising, more publicity, hearing about more applications of (and in other fields) of PL type tools/ techniques

8 respondents each mentioned:

14% outreach (8 respondents) This included responses such as: [achieving] diversity and inclusion more reflective of the Gulf South, more visible presence internationally, “it's pretty white, which is unsettling from a justice perspective”, get into schools, libraries, etc., connecting to other non-profits
14% provide more resources (8 respondents) This included responses such as: improvise on current technology and cameras; photo quality; more DIY instead of kits; [computer] programming; Make tools more helpful; Try to solve even the smallest issues
14% structure (8 respondents) This included responses such as: offer more specific hands on certifiable workshops to establish authority, give members career building skills; Preparation for awesome field results; Planning things to do in case we aren't able to do a certain task

Smaller categories:

7% the website (4 respondents) This included responses such as: more user friendly and aesthetically looking site, I love all the diverse content on the site, but I wish it was more collected and organized to be less of a treasure hunt

3% accessibility (leveling) (2 respondents) This included responses such as: engage newcomers from all walks of life, make their program research more approachable for those not so into science

7% other (4 respondents) This included responses such as: not sure; too early to say; “fewer surveys, maybe?”; and staff favorite: “Public Lab to come visit after it stops raining”

image description Q4 How much of a positive difference has Public Lab made in your life?

image description Q5 Overall, how well has Public Lab met your needs?

image description Q12 How likely do you see yourself engaging in the following activities in the future?

image description