Public Lab Wiki documentation



Evaluation



Welcome! This is the home for all things related to evaluation at Public Lab. Many different feedback efforts are ongoing in different sectors and we try to coordinate our efforts to minimize survey fatigue or redundancy. @liz leads the evaluation team! Ask questions below to find out more.

Follow along with current work

See recent work related to evaluation here, or click on the boards below to see progress on:

1) our Snapshot Evaluation and Evaluation Framework (May 2015-May 2018) generously supported by the Rita Allen Foundation
2) "pulse" feedback collected at in-person events with the Listen For Good 5 questions plus 6 of our own custom questions
3) year-round ongoing evaluation processes

Trello Board
Trello Board
Trello Board

What are we measuring against?

LogicModel_headers.jpg

All evaluation is tracked against our Logic Model, and terms in Logic Model are defined in our Community Glossary.

Why we evaluate

The Public Lab community intentionally works together to create a place where collaboration thrives. We collaborate on collaboration. We seek to collectively and publicly understand how we ourselves work together, and the systems, conventions and structures which shape that cooperative practice.

To do this better, we need feedback loops that add to our self-awareness. The feedback we wish we could see includes additional stats about our community's activity, especially where there are gaps, for instance, community questions languishing unanswered, which can be heart-breaking when the topic is environmental health. We would also like to identify emerging topics in real time in order to better tune outreach; this helps us ensure that diversity stays high even as early adopters rush in.

As Chris Kelty famously wrote of his concept "recursive publics," "[we] are the builders and imaginers of this space." This theme stretches across the FLOSS community, and increasing our self-awareness will help us eliminate our collective blind spots. As FLOSS publics strive to broaden in diversity and inclusivity, careful monitoring of where onboarding processes fail is critical.

By watching channels and identifying people who connect with the community in one or more ways, we hope to become aware of the ways that people first connect with Public Lab, and what their second, third, etc steps may be. If there are not subsequent steps, what stopped people who had started to engage from participating further?


How are we measuring?

Community Surveys

Our Annual Community Survey is delivered over email lists and posted on the website. 2017_Public_Lab_Community_Survey_.pdf

We also survey segments of our community who are having shared experiences:


Stakeholder interviewing

A series of stakeholder interviews was done in 2017! You can read them here:

Title Author Updated Likes
Interview: Ramsey Sprague @stevie 3 months ago
Interview: Yvette Arellano @stevie about 1 year ago
Interview: Jim Gurley @stevie about 1 year ago
Interview: Nayamin Martinez and Gustavo Aguirre Jr. @stevie over 1 year ago
Interview: Karen Savage @stevie over 1 year ago
Interview series with Grassroots Community Organizers @stevie over 1 year ago
What fuels a movement? @stevie about 2 years ago


Online analytics

Statistics on community activity are publicly displayed at http://publiclab.org/stats.
Experiment with customizing your own queries of publiclab.org activity by adjusting the DD-MM-YYYY in the URL, for example → https://publiclab.org/stats/range/31-01-2016/31-12-2016/ !

Research into pathways through Public Lab's ecosystem is located at https://publiclab.org/first-contact.

The ever-growing Data Dictionary describes the datasets that are available for analysis. Created by @bsugar, maintained by @bsugar and @liz.

Topics include:

  • Conversational dynamics on mailing lists: 2017-07-12_mailing_list_activity.png

  • Rhythms of community activity on publiclab.org: Screen_Shot_2018-05-10_at_2.59.16_PM.png

User interface design

See the User Interface page for more on design work towards user interface and user interaction improvements. This is an area where many people are offering feedback!

Other interesting views of the Public Lab community over time


Questions

Title Author Updated Likes
evaluation notes: community segments -- not what you think! @liz 9 months ago
How can I make a tag graph visualization? @bsugar 11 months ago
Is anyone studying or writing about the rise in Codes of Conduct in open source projects? @warren 12 months ago
Help with a standard mini-evaluation for assessing software outreach efforts? @warren about 1 year ago

Ask a question  or help answer future questions


Related work

Title Author Updated Likes
How to create a Logic Model @liz about 1 month ago
Creating Public Lab's Logic Model @liz 2 months ago
Newark Barnraising & Crisis Convening evaluation results @liz 4 months ago
Evaluation: Listen4Good round two results @liz 5 months ago
evaluation notes: community segments -- not what you think! @liz 9 months ago
Convening diversity and inclusion initiatives across open source projects @liz 10 months ago
How can I make a tag graph visualization? @bsugar 11 months ago
How do we do user interface design work in a community process? @warren 11 months ago
Help with a standard mini-evaluation for assessing software outreach efforts? @warren about 1 year ago
Exporting GoogleGroups @bsugar over 1 year ago
Your input kindly requested on the 2017 Community Survey! @liz over 1 year ago
growing Github contributors @liz over 1 year ago
Evaluation: Listen4Good round one results @stevie almost 2 years ago
How useful is a map? @clauds almost 2 years ago
First draft of tag graph @liz about 2 years ago
Creating an evaluation framework for Public Lab @Shannon almost 3 years ago
Intended Purposes for Different Tools and Techniques @gretchengehrke about 3 years ago
2013 Barnraising evaluation summary @Shannon about 5 years ago
Ideas from the $35 Kickstarter backers on how they will use the spectrometer @Shannon almost 6 years ago
Measuring our community growth using web traffic @warren about 6 years ago
Show more


Older page content

From 2014 via @liz: brainstorming possible community metrics

From 2011 via @warren, interesting! Read on:

On this page we are in the process of summarizing and formulating our approach towards self-evaluation; as a community with strong principles, where we engage in open participation and advocacy in our partner communities, this process is not that of a typical researcher/participant nature. Rather, we seek to formulate an evaluative approach that takes into account:

  • multiple audiences - feedback for local communities, for ourselves, for institutions looking to adopt our data, for funding agencies, etc
  • reflexivity - we may work with local partners to formulate an evaluative strategy, and this may often include questionnaires, surveys, interviews which we take part in both as subjects and as investigators
  • outreach - by publishing evaluations in a variety of formats, we may employ diverse tactics to better understand and refine our work; its publication in diverse venues (journals, newspapers, white papers, video, public presentation, etc) offers us an opportunity to reach out to various fields (ecology, law, social science, technology, aid)
  • location - our evaluations should be situated in geographic communities, examining the effects of our tools and data production in collaboration with a specific group of residents

Goals

Good evaluative approaches could enable us to:

  • quantify our data and present it to scientific, government agencies for use in research, legal, and
  • provide rich feedback for field mappers (in the case of balloon mapping and other public scientists to improve their techniques
  • assess the effects of our work on local communities and situations of environmental (and other types of) conflict
  • involve local partners in the quantification and interpretation of our joint work
  • ...

Approaches

We're going to use a few different approaches in performing (self-)evaluation -- each has pros and cons, but we will attempt to meet the above goals in structuring them.

Approach A: Logbook questionnaire

The logbook is an idea for a Lulu.com printed book to bring on field mapping missions for balloon mapping.

Although this strategy can be reductive, compared to interviews, videos, etc, its standard approach yields data which we can graph, analyze and publish for public use. The results will be published here periodically. Any member of our community may use them for fundraising, outreach, or for example to print & carry to the beach to improve mapping technique.

Read more at the Logbook page.

A mini version of this questionnaire was used by Jen Hudon as part of her Grassroots Newark project and can be found here:

Approach B: Community Blog

The community blog represents a way for members of our community to ... critical as well as positive...

To contribute to the community blog, visit the Community Blog page

Approach C: Interviews

We're beginning a series of journalistic/narrative interviews with residents of the communities we work with. Read more at the interviews page.