Drafted in January 2016 by Gretchen Gehrke, Stevie Lewis, Liz Barry.

Why (the Situation): In order to confidently answer the questions we have about our environment, we want to learn how to structure our questions, use DIY tools, conduct experiments which make use of the scientific method, and understand the capabilities and limitations of our data in comparison to any existing data. This workshop will focus on the case study of the Oil Testing Kit

When: a 3.5 hour workshop in four sections, with a ten minute break after each section.

Where: a room with tables and chairs, with participants sitting in small groups.

What (the content): The basics of designing an effective experiment; transforming guesses into testable hypotheses; assessing precision and reproducibility; interpreting data based on data quality.

For What (Achievement Based Objectives):

By the end of this workshop, you will have:

• Met others attending the workshop
• Shared with each other what motivated you to to participate
• Written down your expectations for your time
• Discussed examples of scientific questions that interest you in small groups
• Read about the elements needed for good experimental design
• Noted the important points of designing a clear experiment
• Drafted your own questions and transform them into hypotheses
• Explored the concept of proof versus likelihood (facilitated discussion)
• Discussed the importance of precision, resolution, and accuracy in any data set
• Reflected upon how experimentation relates to your own interests/work

Notes for Facilitators

Materials Needed

• Markers, pens
• blank 8.5”x11” printer paper
• blank index cards
• blank large chart paper
• tape
• printed copies of this handout

Setting up the event

• Write out the 7 steps of the scientific method written out on index cards, one set per table in scattered order:
  • asking a question
  • gathering background information
  • developing a hypothesis
  • conducting an experiment to test the hypothesis
  • analyzing the experimental results
  • communicating the experimental results
  • retesting the hypothesis (or a new one, if necessary)

Workshop Schedule

1. Introduction
   1. Who is here today? (20 minutes)
   2. Introductions among tables (10 minutes)
   3. Everyone asks questions, but how?
   4. Step by step
2. Developing hypotheses
   1. From observing to questioning
   2. Evidence versus proof
   3. Let’s turn guesses into testable hypotheses
   4. Turn your own question into a hypothesis
3. Experiment design -- Testing your hypotheses
   1. Address the question
   2. Concept of using a known sample as a baseline for identifying unknowns
   3. Precision and resolution, Part 1
   4. Precision and resolution, Part 2
   5. Reproducibility
4. Wrap up / Congrats you made it!
   1. Relating all this to our real lives

Workshop Outline

1. Introduction

1.1 Who is here today?

As a whole group, take turns introducing yourself by saying your name, where you’re from, and your reason for coming to the workshop today.

1.2 What are we doing today?

Take a minute to read through the achievement based objectives on page 1 of your handout, and then each table should choose one person to read through the following out loud:

You may have heard people talk about the scientific method. The scientific method is a general guideline for the steps to take in order to answer a question that is based on an observation. Historically, the scientific method originated as a knowledge production technique, and it endures into the present day as a technique you can potentially use in your work, in your community, to help take control of a situation.

1.3 Everyone asks questions, but how?

Think of a situation in which have you wondered about something (how it works, why it exists, etc). How have you tried to answer your question? Think about the specific steps you have taken, and write them down.

Take five (5) minutes to make notes on your own paper, summarizing your individual thoughts as a five-step process. Share with your table. Next, each table will make a brief poster presentation back to the whole group. Finally, discuss your perceptions of how the EPA, or even law enforcement, goes about answering questions.

1.4 Step by step

Look at the index cards on your table, and notice that they are in no particular order.

There are seven basic steps listed for what is commonly understood as the “scientific method”. It is important to note that this is not a linear, one-time series of steps, but rather, it is cyclical. For example, a given hypothesis might require several different experiments to adequately test that hypothesis, or test results may indicate that the original hypothesis is false and you need to develop a new hypothesis.

Each step in the basic scientific method is important and challenging, and deserves special attention. In this workshop we are going to focus mostly on developing a hypothesis, and designing an experiment to test the hypothesis. During this series of workshops, you will conduct an experiment, and by the end, be able to analyze and communicate your results.

A couple of important points that we want to highlight that are crucial to scientific inquiry and experimental design are:

1. the concept of proof versus likelihood, and
2. the importance of knowing your analyses’ precision.

We’ll cover these topics in the following sections.

Take fifteen (15) minutes to work through the following prompt as a group:

In what order would you imagine proceeding through these steps? Place them in order, then discuss:

• Would an experiment be a straight path through these steps?
• Would you repeat certain parts, or occasionally take one out of order?
• Would you ever need to repeat the entire sequence?
• Why?

Return to what you wrote down during the previous activity (1.3) about what steps you took to answer your own question. Based on this new knowledge, name 2 or 3 things you might change about your original ideas.

Send one representative from your group to go up to the wall and tape your steps in order. Each group should line up their rows to facilitate comparison; discuss.

------ (10 minute break) ------

2. Developing Hypotheses

2.1 From observing to questioning

In this task, we are going to come up with research questions for our own observations. Choose one person at the table to read the following text out loud:

Before developing a hypothesis, the very first step in a scientific inquiry is asking a question. Scientific questions usually arise from observations. For example, we might observe that “The sky is blue,” which prompts us to wonder, “Why is the sky blue?” We might notice that “A lot of kids have asthma here,” which compels us to ask, “Why do so many kids have asthma in this neighborhood?” If we notice that “this substance looks like oil,” we ask, “Could this be oil?” These are all good questions based on an initial observation.

On a large piece of paper, draw a vertical line through the middle to make two columns, AKA a “T Chart”:

• Label the top of the left-hand column “Observations”
• Label the top of the right-hand column “Research Questions”

Individually, on sticky notes:

• Write observations that you have had about your yard, your neighborhood, your landscape, local industrial activity, etc. One per note.
• For each observation, take another sticky note & write a research question based on it.

After a few minutes, everyone can put up their sticky notes up on the “T chart” in the appropriate columns.

Review similarities and differences in the framing of research questions, and make a presentation about this to the whole group.

2.2 Evidence versus proof

Choose one person at the table to read the following text out loud:

A key difference between science and mathematics is that there is no such thing as absolute proof in science.

In mathematics, you can have a proven theorem because you are dealing with a closed system where all of the information is available and controlled, and the proof is final. As a result, there can be a binary, “yes or no”, “proven or disproven” set of logic in math.

In science, we don’t have the luxury of an absolute proof because not all of the information is known. We discover new relevant information constantly, and never deal with a truly closed circuit where all information is known. In science, knowledge is tentative, based on the information available, and we gather evidence that suggests a likelihood that something is true.

Everything in science is tentative, and is based on the best available evidence, but cannot be based on absolute proof. This is extremely important for our experimental design and how we talk about results.

Quick group check in: What is the difference between the kind of answers that are possible to arrive at for questions in the domain of math versus the domain of science?

Based on what we just read, we can begin designing our experiments with the understanding that any answers we reach through scientific research will be evidence-based but not an absolute truth. The basic sequence is to

1. Make an observation
2. Ask a related question
3. From that question, develop a hypothesis that can be tested.

What is a hypothesis? Your hypothesis is an educated guess about the answer to your question, but is different from a basic guess in two important ways: first, a hypothesis is based upon existing evidence (albeit a limited amount), and second, a hypothesis can be tested such that new evidence can be gathered that directly supports or refutes the hypothesis.

The steps for turning a question into a hypothesis are:

1. Starting with your observation, get as specific as possible (or as specific as you’d like to be, based on the scope of your research).
2. With your specific observation, brainstorm ideas that could be the cause of what you are observing. This is your “basic guess.”
3. Make sure you have some evidence, or there is existing information, that supports your guess. If there is not, then familiarize yourself with the information about the issue that is available, and adjust your guess/idea accordingly. This saves you time by helping you develop a more likely hypothesis.

To turn your guess into a real hypothesis, you must have a testable statement, with tests that can be observed and measured, and/or compared against a “known” value or entity. This often will be in the form of “X is more than/less than/similar to Y”, where X and Y can be observed and compared, or one of the two is already “known.”

Here is an example of turning a guess into a hypothesis:

Observation: The sidewalks on this tree-lined street are cracked and bumpy.

Get more specific

Revised Observation: The sidewalks on this tree-lined street are more cracked and bumpy than the sidewalks on the street without trees.

Good. Now guess why that is the case.

Guess: I think trees made sidewalks bumpy.

Make it more specific and testable

Hypothesis: I think tree roots grow under sidewalks, and as they get bigger, they can push up the cement sidewalks. Thus, sidewalks near bigger trees are likely to have more cracked and bumpy sidewalks than sidewalks on streets with small trees or without trees.

2.3 Let’s turn guesses into testable hypotheses

Individually, turn these guesses into testable hypotheses:

Guess: Cats like wet cat food.
Hypothesis: _______________________________

Guess: Breathing fumes from cleaning products is bad.
Hypothesis: _______________________________

Guess: Strip mining damages the environment.
Hypothesis: _______________________________

2.4 Turn your own question into a hypothesis

Each person take a look back at the observations you made in section 2.1.

• Chose one of your questions, transform it into a hypothesis, and write your hypothesis on a blank index card.
• In small groups, talk about how testing this hypothesis might or might not be helpful in your own work or pursuits.
• Consider kinds of tests are possible for this question. What tests or data are available? Which ones are used by environmental regulators?
• Write out variations on your question, like "in a worst case scenario, like a windy day" or "measure particulates" vs. "measure PM2.5" and try out "daily average" vs. "on a bad day".

------ (10 minute break) ------

3. Experimental Design – Testing your Hypothesis

3.1 Address the question

Individually, take five or ten minutes to read the following text:

The first step in designing your experiment is to make sure that it will allow you to address your hypothesis.

Your goal is to gather evidence that either directly supports or directly refutes your hypothesis, so the more specific your hypothesis is, the more tailored and efficient your experiment can and should be, and the more clearly it can answer your question. Broader questions and hypotheses can be very useful and provide a wealth of evidence, but do require more comprehensive investigations.

For example, with a broad hypothesis such as “I think tomato plants grow well in sunlight,” your experiment will have to encompass several varieties of tomato plants in several different potential growing conditions (e.g. different kinds of soil and watering patterns), and different sunlight exposures, and you would have to evaluate different aspects of growth (growth rate, fruit abundance, fruit quality, etc).

If your hypothesis were more specific, such as “I think beefsteak tomato plants growing in sandy soils under a variety of watering conditions grow faster in the sun than in the shade,” then your experiment need only include a few types of beefsteak tomato plants growing in one type of soil under either sunny or shady conditions, with variable watering patterns.

Take into consideration that the second experimental design would be inadequate to address the first hypothesis, while the first experimental design would be excessive to address the second hypothesis. The second experiment isolates one difference between the two scenarios -- it's a comparison -- which is an easier type of experiment.

As a group, discuss:

• Would it make sense for either of the two hypotheses written above to test eggplant growth in addition to tomato plant growth?
• Would it make sense to test growth under drought conditions?
• Would it make sense to test tomato plant growth in predominantly cloudy conditions?

Each person should individually write:

• one alternative hypothesis with experimental design that expands the scope of the scientific inquiry.
• one alternative hypothesis with experimental design that narrows the scope of the scientific inquiry.

As a group, go around and have each person can share how they expanded or narrowed the inquiry. Discuss what kind of answer you can expect from the modified question.

3.2 Concept of using a known sample as a baseline for identifying unknowns

Each table should choose one person to read through the following text out loud:

In many scientific inquiries, identification or classification of an unknown object or compound is accomplished through comparison with known compounds.

When designing an experiment to identify or classify unknown objects or compounds, you should choose relevant known samples against which to compare. “How will I know what’s relevant?”, you might ask. Well, if the question is “Is this mucky stuff actually oil?”, then obtain known samples of the oil carried on nearby train tracks, pipelines, etc, as well as commonly available consumer oils such as for vehicles or machinery. Once you have relevant known compounds available, you can assess your unknown compound through comparing similarities and differences to the “knowns”. You need to analyze known compounds to demonstrate that your method is valid and is able to correctly identify, classify, or quantify the known sample.

If your method can’t tell that oil is oil, your method is not valid (for using to identify anything else).

The assessment of whether a compound is similar enough to a known compound that you conclude they are in the same category, will depend on your method’s precision and resolution, which are discussed below.

As a group, discuss:

• When have you been faced with an unknown substance and gone about trying to figure out what it was?
• What other thoughts and ideas do these concepts bring up for you, especially related to unknown substances?

3.3 Precision and Resolution Part 1

Choose one person at the table to read the following text out loud:

In scientific experiments, we are assessing the likelihood of an assertion to be true, such as whether an unknown oily residue is in a certain class of oil. To do this, we have to know how well we know our own data. That knowledge depends on accuracy, precision, and resolution.

Each table should choose one person to read the following situation out loud:

mg = milligram, which is one thousandth of a gram;

L = liter.

Units of mg/L read as “milligrams per liter” are a measure of concentration.

You are interested in whether or not an industrial facility violated their permit by discharging more than 1.8 mg/L ammonia in their effluent.

• Let’s say your instrument measured 2 mg/L ammonia, but the resolution of your instrument is 1 mg/L, meaning that the only readings you can obtain are 0, 1, 2, 3… mg/L. Can you be certain that your measurement of 2 mg/L is definitely higher than 1.8 mg/L? Could the true value of your measurement be 1.6 mg/L and still read as 2 mg/L?

(pause for discussion)

• What if your tool could reliably measure 0.1 mg/L differences, and using it, you took three measurements which were 1.6, 2.2, and 1.8 mg/L. How well do you know what the “true” value is? (pause for discussion)
• What if your three measurements were 1.9, 1.8, and 1.9 mg/L? How well do you know what the “true” value of those measurements is? (pause for discussion)

Each table should choose another person to continue reading:

The example above illustrates that your ability to evaluate your own data depends on:

• knowing your instrument’s resolution (resolution means the smallest distinguishable difference from a given value)
• knowing your method’s precision (precision is the variability in values recorded for a given true value)
• knowing your method’s accuracy (accuracy means how close a measured value is to the true value, often based on the mean of several measurements)

The numbers or values that our equipment give us have a specific relationship to reality, and how well we can describe that relationship is dictated by resolution, precision, and accuracy. Describing your data’s accuracy*, precision, and resolution are part of** determining the probability that your original assertion is true.

*Note: that in the example above, it is assumed that your instrument was reading fairly accurate results because it had been appropriately calibrated. We will discuss calibration more in a future workshop.

**Note: There are several factors that will influence probability, and especially in larger data sets, more statistical analyses are needed to adequately describe the data. In this workshop we are focusing more conceptually rather than mathematically. If you want to do more statistics, here is a useful introductory guide for statistical analyses: http://www.robertniles.com/stats/.

3.4 Precision and Resolution Part 2

Everyone get a piece of paper and a writing implement, and get ready to draw!

Draw a few visual representations to describe what we learned above about accuracy, precision, and resolution. Draw a representation of measurements with:

• high precision but low accuracy
• low precision but high accuracy
• low precision and low accuracy
• high precision and high accuracy
• high precision but low resolution
• low precision but high resolution

Share your drawings with your group members and discuss them.

Next, as a table, refresh your memories about the ammonia concentrations measured in the effluent of the industrial facility (above).

Draw representations of the data from the example to explain whether or not you are able to determine if the discharge is above the 1.8 mg/L ammonia limit for each scenario listed (and remember to assume that your measurements are accurate).

3.5 Reproducibility

Each table should choose one person to read through the following out loud:

To demonstrate the validity of your experiment and your results, you must be able to show that you can achieve same results multiple times, and that another person could achieve the same results by following your protocol. This reproducibility of results is important for three reasons: 1. to help you assess your own precision, as discussed above, 2. to ensure that you provide enough information for other people to replicate your experiment and thus grow scientific knowledge and capacity, and 3. to demonstrate the validity of the data by asserting that it is reliably reproducible. In cases of scientific fraud that have occurred (thankfully rarely), they have most often been caught by another researcher attempting to reproduce an experiment’s results, and alerting people that the initial results must have been falsified. In Workshop 3, we will follow procedures to take each oil spectrum three times. By virtue of posting our oil spectra freely and openly to the Web, we are contributing to a massive body of experimental results, and both demonstrating and evaluating the reproducibility of those results.

Individually, and then as a group:

Think ahead towards future research projects you might have in mind. When do you think you might have to be aware of resolution, precision, and accuracy?

------ (10 minute break) ------

4. Wrap-up

Congrats, we made it!

4.1 Relating all this to our real lives

Each table should choose one person to read through the following out loud:

In our daily lives, we make observations constantly. Think about the observations you listed in section 2.1, and others that cross your mind:

• Have you ever noticed that grass pops up through cracks in the sidewalk in some places but not others?

• Have you ever been struck by an odd smell in a new house or new car?

Your take home assignment, should you choose to accept it, is to think of something that you have observed in an environment that matters to you. Write down your observation, and come up with a hypothesis that could be tested, or at least some baseline ideas that could become hypotheses after reading relevant background information. Remember that your hypothesis must be testable, and that the outcomes of your test will offer a likelihood of an answer, but not definitive proof. For the hypothesis that you create, define how precisely you are going to need to know your answer. Also define the purpose(s) you intend to use the data you collect for.

Congratulations! You have embarked upon the process of scientific inquiry!