Jump to content

Success in research: Difference between revisions

From IdeaWazaWiki
Lumo (talk | contribs)
m Lumo moved page Success in research to Comparing amateur and professional research: Use a title that reflects the article question
Lumo (talk | contribs)
Show how to plan and combine several studies using an AI and automation example
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
A large group of amateur researchers can make observations across many places, sort through more material than a small team can handle, and bring experience from outside a field. Professional researchers may bring specialized methods, equipment, or experience interpreting evidence. How could we test whether a large group of people with little formal training can produce more useful or original findings than a smaller team of trained researchers?
Success in [[Research|research]] means finding something useful and showing how you found it. You might establish when an event happened, collect observations that others can use, or narrow down which explanations fit the evidence. A small answer with clear evidence can give the next person a place to start.


Amateur and professional describe how people participate in research. An amateur astronomer may have decades of observing experience, and a paid researcher may be new to astronomy. A comparison should record relevant experience, formal education, and training for the task. It should also say how the teams used their time and resources.
Suppose you want to know why one street seems to flood more often. Start by finding out which floods you can document. A timeline gives you something to compare with rainfall records or changes to the storm drains.


== What counts as a successful result? ==
== Choose a question you can answer ==


Choose a measure that fits the question. A team building a public dataset might be judged by how much it covers, how accurate its records are, and whether someone else can use them. A team looking for new patterns might be judged by how many of its leads hold up under further investigation. A surprising observation can be easy to generate and expensive to check, so count the time spent checking rejected leads as well.
A good [[Research question|research question]] gives you a way to start looking for evidence. For the flooding project, choose one street, a range of years, and a definition of flooding. Do you mean water over the curb, water entering homes, or either? The definition affects which accounts belong in your timeline.


A finding that changes what a field teaches or how people solve a problem may take years to assess. A short experiment can track whether a finding is new to the published literature and whether an independent team can confirm it. It can also record whether another group uses the result later.
Make three decisions before you start.


== What Galaxy Zoo compared ==
# '''Where to look.''' Pick sources you can get to, such as digitized newspapers, dated photographs, or city records.
# '''What to make.''' Choose something you can finish and share. A table with one row per flood, a date, and a source is enough to get started.
# '''When to review.''' Set aside an hour to check how the search is going. If the newspaper archive has too many gaps, shorten the period or try city records. Write down what changed and why.


In the original Galaxy Zoo project, volunteers classified images of galaxies from the Sloan Digital Sky Survey. Multiple participants looked at each image. The project combined their classifications and compared the results with classifications made by professional astronomers for subsets of the galaxies. The researchers found that the results were consistent for the simple galaxy shapes that the project tracked. The study gives an example of volunteers and professionals contributing to different parts of the same research project. [https://arxiv.org/abs/0804.4483 Lintott and colleagues, Galaxy Zoo]
== Keep a record as you go ==


Agreement among volunteers is useful for finding images that need another look. It can also conceal a shared mistake. If most participants interpret a blurry spiral galaxy as an elliptical galaxy, a majority vote may preserve the mistake. Comparing classifications with an independent reference and examining disagreements can show which images or instructions are causing trouble.
Use a [[Research log|research log]] to save search terms, dates, page numbers, links, and notes on what you found. For a flood photo, record where you found it, who supplied the date, and how you recognized the street. A folder of photos and clippings is easier to use when each item has a note that ties it to a claim. This also makes [[Citing sources|citing your sources]] easier when you write.


== How to test the idea ==
== Check the most important claim ==


# '''Give teams the same research task.''' For example, ask volunteer, professional, and mixed teams to mark visible features in previously unseen images. Choose the images in advance, including clear and ambiguous cases. Give teams access to the same examples, tools, and task instructions.
Start with the claim your answer depends on. If two newspapers give the same date, see whether one copied the other's account or whether both drew on the same report. Look for a separate piece of evidence, such as a dated photograph or a road closure notice. Ask someone else to trace one or two entries from your timeline back to the original records. A fresh reader may catch a date you copied incorrectly or a location you assumed was obvious.
# '''Decide what the comparison is about.''' For a comparison of experience, give teams the same total number of work hours. To test whether a crowd can do more with a given budget, let teams vary in size and include training, coordination, and verification in their costs. Record both spending and participant time, including unpaid time.
# '''Set the measures before seeing the results.''' Have reviewers assess the markings without knowing which team made them. Use a reference set prepared independently and record how many features each team finds, misses, or marks by mistake. Allow reviewers to flag ambiguous images and report their uncertainty.
# '''Repeat the comparison.''' Try new images and recruit new teams. A result that holds up on several sets of images may still depend on the type of work. Try a second task, such as proposing an explanation for a pattern, and see whether the same arrangement works there.


For a classroom version, divide into small groups and pick a set of public images. Have each group label the same images independently. Compare individual labels, the group vote, and a reference prepared ahead of time. Then look at the cases where voting helped and the cases where asking for another opinion was more valuable.
Then state what the evidence supports. If you found three floods in the 1960s, give their dates and sources. If the archive is missing several years, say which ones and leave open how often the street flooded. Share the timeline and enough of your method for someone else to check or extend it. This is one way to put [[Open research|open research]] into practice.
 
== When a big question needs several studies ==
 
How could [[Artificial intelligence|AI]] and automation make goods and services more affordable while limiting surveillance and the concentration of wealth and power? Keep that question at the top of a shared plan. Start with a service you can compare across local governments, such as building permits.
 
# '''Decide what progress would look like.''' Track fees, wait times, and errors. Check separately for residents who need help applying or who apply offline, since an average can hide who is worse off. Look at which data the agency and vendor can collect or reuse, who can reverse a decision, and whether the agency can switch vendors and keep its records.
# '''Divide the work and agree on what to record.''' One team can compare service records across offices. Another can read public contracts and data policies. A third can interview residents and staff about appeals and how the rules work in practice. Use the same offices, time periods, and definitions. If records are unavailable, ask for summary figures or mark the gap in the shared log.
# '''Check what else might explain a change.''' Measure outcomes before and after adoption. If possible, compare the changes with those in a similar office that kept its previous process. Check whether the offices had similar trends before adoption and record changes in staffing or application volume. If those factors cannot be sorted out, report what changed and how uncertain its cause is.
# '''Bring the results back to the big question.''' Make a table with one row for each arrangement studied. Record who benefited, who gained control over data or decisions, and how strong the evidence is. A shorter wait alongside broader vendor access to residents' data is a mixed result. Look for offices that gained speed under tighter data rules. Where the sources disagree, record what would help resolve the difference.
 
A permit study can show which arrangements are worth testing elsewhere. To study abundance on a larger scale, repeat the approach where automation might change the cost of essentials or workers' pay. Explain which results appear across studies and which depend on the rules or conditions in one place.
 
== When working with others helps ==
 
Independent researchers pursue a project outside an institutional research appointment. They may bring years of experience to it. In '''citizen science''', members of the public volunteer to take part in scientific research. Professional researchers do research as part of their paid work. The same project can draw on all three. In a local flood study, one person might trace old street names, others might read newspapers, and someone with mapping experience might check where each flood happened. [https://www.noaa.gov/office-education/citizen-science NOAA's introduction to citizen science]
 
One example from astronomy is '''Galaxy Zoo'''. Volunteers looked at telescope images and identified galaxy shapes. Several volunteers examined each image. The team combined their judgments and checked a sample against classifications by professional astronomers. [https://arxiv.org/abs/0804.4483 The original Galaxy Zoo study]
 
== Try a small comparison ==
 
Could a group with little formal training produce a more useful result than a smaller, experienced team? Give both groups the same set of newspaper pages and ask them to find flood reports. Record the time they spend, including training and checking. Ask a third group to score the entries without knowing which team made them. Count well-supported entries, missed reports, and duplicates. If you want to see whether experience or group size accounts for a difference, repeat the task while changing one of those factors at a time.


==Discussion questions, essay ideas, and learning related AI prompt ideas==
==Discussion questions, essay ideas, and learning related AI prompt ideas==


* What kinds of research tasks benefit most from having more observers? What kinds depend on a person learning a difficult method over time?
* What would convince you that flooding became more frequent if newspaper coverage also increased?
* Write an essay on how you would compare a volunteer team with a professional team when one finishes faster and the other finds more errors. Explain which result matters most to the people who will use the work.
* Write a short essay about a project you could do independently and one that would benefit from a group. Explain how you would check the most important claim in each project.
* Ask an AI tool to propose a study comparing volunteer, professional, and mixed teams on a public image dataset. Have it specify the outcome and the costs for each team. Exchange designs with a classmate and see whether both of you could score the same sample result using the instructions.
* Give an AI tool a draft research question and ask it to suggest records to look for and a way to check the findings. Look up two of the suggested sources yourself and see whether they contain the information the tool expects.


==Readings from Wikipedia==
==Readings from Wikipedia==
Line 38: Line 58:
== Research readings ==
== Research readings ==


* [https://www.nationalacademies.org/read/25303/chapter/3 National Academies, Reproducibility and Replicability in Science]. The summary explains what can be learned by checking the original analysis and by repeating a study with new data.
* [https://www.nationalacademies.org/read/25303/chapter/3 National Academies, Reproducibility and Replicability in Science]. See the summary for ways to check an analysis and repeat a study with new data.
* [https://www.gov.uk/government/publications/the-magenta-book/guidance-on-the-impact-evaluation-of-ai-interventions-html HM Treasury, Guidance on the Impact Evaluation of AI Interventions]. See the sections on choosing a comparison and combining different sources of evidence.


== See also ==
== See also ==

Latest revision as of 22:24, 22 November 2026

Success in research means finding something useful and showing how you found it. You might establish when an event happened, collect observations that others can use, or narrow down which explanations fit the evidence. A small answer with clear evidence can give the next person a place to start.

Suppose you want to know why one street seems to flood more often. Start by finding out which floods you can document. A timeline gives you something to compare with rainfall records or changes to the storm drains.

Choose a question you can answer

A good research question gives you a way to start looking for evidence. For the flooding project, choose one street, a range of years, and a definition of flooding. Do you mean water over the curb, water entering homes, or either? The definition affects which accounts belong in your timeline.

Make three decisions before you start.

  1. Where to look. Pick sources you can get to, such as digitized newspapers, dated photographs, or city records.
  2. What to make. Choose something you can finish and share. A table with one row per flood, a date, and a source is enough to get started.
  3. When to review. Set aside an hour to check how the search is going. If the newspaper archive has too many gaps, shorten the period or try city records. Write down what changed and why.

Keep a record as you go

Use a research log to save search terms, dates, page numbers, links, and notes on what you found. For a flood photo, record where you found it, who supplied the date, and how you recognized the street. A folder of photos and clippings is easier to use when each item has a note that ties it to a claim. This also makes citing your sources easier when you write.

Check the most important claim

Start with the claim your answer depends on. If two newspapers give the same date, see whether one copied the other's account or whether both drew on the same report. Look for a separate piece of evidence, such as a dated photograph or a road closure notice. Ask someone else to trace one or two entries from your timeline back to the original records. A fresh reader may catch a date you copied incorrectly or a location you assumed was obvious.

Then state what the evidence supports. If you found three floods in the 1960s, give their dates and sources. If the archive is missing several years, say which ones and leave open how often the street flooded. Share the timeline and enough of your method for someone else to check or extend it. This is one way to put open research into practice.

When a big question needs several studies

How could AI and automation make goods and services more affordable while limiting surveillance and the concentration of wealth and power? Keep that question at the top of a shared plan. Start with a service you can compare across local governments, such as building permits.

  1. Decide what progress would look like. Track fees, wait times, and errors. Check separately for residents who need help applying or who apply offline, since an average can hide who is worse off. Look at which data the agency and vendor can collect or reuse, who can reverse a decision, and whether the agency can switch vendors and keep its records.
  2. Divide the work and agree on what to record. One team can compare service records across offices. Another can read public contracts and data policies. A third can interview residents and staff about appeals and how the rules work in practice. Use the same offices, time periods, and definitions. If records are unavailable, ask for summary figures or mark the gap in the shared log.
  3. Check what else might explain a change. Measure outcomes before and after adoption. If possible, compare the changes with those in a similar office that kept its previous process. Check whether the offices had similar trends before adoption and record changes in staffing or application volume. If those factors cannot be sorted out, report what changed and how uncertain its cause is.
  4. Bring the results back to the big question. Make a table with one row for each arrangement studied. Record who benefited, who gained control over data or decisions, and how strong the evidence is. A shorter wait alongside broader vendor access to residents' data is a mixed result. Look for offices that gained speed under tighter data rules. Where the sources disagree, record what would help resolve the difference.

A permit study can show which arrangements are worth testing elsewhere. To study abundance on a larger scale, repeat the approach where automation might change the cost of essentials or workers' pay. Explain which results appear across studies and which depend on the rules or conditions in one place.

When working with others helps

Independent researchers pursue a project outside an institutional research appointment. They may bring years of experience to it. In citizen science, members of the public volunteer to take part in scientific research. Professional researchers do research as part of their paid work. The same project can draw on all three. In a local flood study, one person might trace old street names, others might read newspapers, and someone with mapping experience might check where each flood happened. NOAA's introduction to citizen science

One example from astronomy is Galaxy Zoo. Volunteers looked at telescope images and identified galaxy shapes. Several volunteers examined each image. The team combined their judgments and checked a sample against classifications by professional astronomers. The original Galaxy Zoo study

Try a small comparison

Could a group with little formal training produce a more useful result than a smaller, experienced team? Give both groups the same set of newspaper pages and ask them to find flood reports. Record the time they spend, including training and checking. Ask a third group to score the entries without knowing which team made them. Count well-supported entries, missed reports, and duplicates. If you want to see whether experience or group size accounts for a difference, repeat the task while changing one of those factors at a time.

  • What would convince you that flooding became more frequent if newspaper coverage also increased?
  • Write a short essay about a project you could do independently and one that would benefit from a group. Explain how you would check the most important claim in each project.
  • Give an AI tool a draft research question and ask it to suggest records to look for and a way to check the findings. Look up two of the suggested sources yourself and see whether they contain the information the tool expects.

Readings from Wikipedia

Research readings

See also