ORIGINAL: Open Science Framework
Contributors: Elizabeth Iorns, William Gunn, Brian A. Nosek, Stuart Buck, Tim Errington, Johanna Cohoon, Fraser Elisabeth Tan, Tiffany L Bogich, Joseph Dureau, Sebastien Ballesteros, Erin Griner
Contributors: Elizabeth Iorns, William Gunn, Brian A. Nosek, Stuart Buck, Tim Errington, Johanna Cohoon, Fraser Elisabeth Tan, Tiffany L Bogich, Joseph Dureau, Sebastien Ballesteros, Erin Griner
Date Created:
10/8/2013 6:31 PM
| Last Updated:
2/7/2014 3:00 PM
Description: We are conducting a study to investigate the replicability of cancer biology studies. The top 50 most impactful cancer biology studies published between 2010-2012 are being replicated by the Science Exchange network.
Description: We are conducting a study to investigate the replicability of cancer biology studies. The top 50 most impactful cancer biology studies published between 2010-2012 are being replicated by the Science Exchange network.
Reproducibility of published results is a key component to scientific
progress.
The Reproducibility Project: Cancer Biology is an
open collaboration to independently replicate key results from 50 cancer
biology studies. This effort is an extension of the
Reproducibility Project: Psychology
and will add to the empirical examination of the reproducibility of published science.
We expect to learn about:
- The overall rate of reproducibility in a sample of the published cancer biology literature
- Obstacles that arise in conducting direct replications of original studies
- The feasibility and practical challenges of getting proper materials, methods, and instrumentation for a replication
- Predictors of replication success, such as the journal in which the original finding was published, the citation impact of the original report, the number of direct replications that have been published elsewhere, the transparency of materials and methods included with the publication, and adherence to publishing checklists and guidelines
Key experimental findings from each study are replicated by experts from the Science Exchange network according to best practices for replication established by the Center for Open Science, with impact of the replications tracked on Mendeley's research analytics platform.
The Reproducibility Project: Cancer Biology is advised by leaders on the issue of reproducibility. For a list and short biography of the advisors visit The Advisory Board page.
Visit the Project Progress page to get an idea of what stage all aspects of the RP:CB are at.
Currently we have filled our need for volunteers to help us extract the methodology, statistical information, and checklist adherence of the identified key experiments for each of the papers.
If you have any questions please contact tim@cos.io.
Purpose: This document is meant to outline a general process for coding the experimental protocols and statistical information of the key experiments to be replicated for each paper and to code the checklists for each paper.
- Template for coding studies - this is how the spreadsheets for recording all the information looks like before any stage.
- RPCB: Coding Study 1 & RPCB: Coding Study 7 - these are examples of how the spreadsheets will look when all filled in and additional tabs generated.
- RP:CB - Project Progress - this shows the status of each paper chosen for replication. Column L will be empty if the study needs help and I will fill in your initials when it is claimed by you.
You can also find additional information on the Reproducibility Project: Cancer Biologypage, were all experimental methods, data and results for each replication experiment will be posted as they are completed.
What stage is the coding for each paper at?
All experiment(s) have been reviewed and identified as either included (colored), excluded or include/exclude and are found in the ‘Figures’ tab of the spreadsheet for each paper. See Study 1 as an example.
The experiment(s) have been chosen because they are central to and embody the main conclusions of the original paper, which were drawn from multiple experiments. Not all experiments in a given paper will be replicated, nor will we pass judgement on the claims stated by the original authors. Experiments are selected because they are key to the overall conclusions compared to other experiments presented in the paper. If funds are available experiments identified as include/exclude will be evaluated for inclusion.
Do I get recognition for my contribution?Yes. Anyone who completes all of the coding listed below will earn authorship on that studies individual publication in PLoS ONE as well as the overall project publication. Your contribution will also be added to the project page for that given study on the Open Science Framework (OSF) as well as the overall Reproducibility Project: Cancer Biology project page which will identify you as an author to these projects. If you would prefer to remain anonymous, that is ok too.
What are the next steps of coding?As a quick reminder before describing the steps: if at any stage of the process you have a question or are unsure of how to proceed with the organization or presentation of the information, you should feel free to contact the Tim at tim@cos.io. This process will most likely require a constant dialogue to make the process easier for you and completed in a way that is the most beneficial for this project.
1) Identify one of the remaining papers for coding:a) Look at the Project Progress spreadsheet to identify which studies have not been claimed (column L (Coded by) will be empty for each available paper). Paper information can be found on the Studies spreadsheet as well as the Keywords spreadsheet which lists keywords associated with each paper.
b) Contact core team to claim study and have spreadsheet shared with you. Email Tim at tim@cos.io
2) Write up detailed experimental protocols for each key experiment:This step will allow us to have a detailed protocol to give to the labs which will be performing the replications. Each protocol you write for a given experiment will be given to a different lab to perform. Therefore each tab should have all the information necessary to perform that replication, or refer to other tabs for additional information if necessary. Only the necessary protocol tabs will be given to the researchers performing the experiment, not this entire spreadsheet.
The protocols also need to be written as you would for a protocol in lab (all the details in a step-by-step manner) and how methods papers are generally written. This might require some additional data searching (be sure to write in the ‘source of information’ column where you got the information from) and filling in missing details based on missing or unclear information (be sure to write in the ‘questions’ column the question that this needs to be confirmed/checked by the original authors and if necessary the ‘comments’ columns to explain your information (or lack of it)).
All experiment(s) have been reviewed and identified as either included (colored), excluded or include/exclude and are found in the ‘Figures’ tab of the spreadsheet for each paper. See Study 1 as an example.
The experiment(s) have been chosen because they are central to and embody the main conclusions of the original paper, which were drawn from multiple experiments. Not all experiments in a given paper will be replicated, nor will we pass judgement on the claims stated by the original authors. Experiments are selected because they are key to the overall conclusions compared to other experiments presented in the paper. If funds are available experiments identified as include/exclude will be evaluated for inclusion.
Do I get recognition for my contribution?Yes. Anyone who completes all of the coding listed below will earn authorship on that studies individual publication in PLoS ONE as well as the overall project publication. Your contribution will also be added to the project page for that given study on the Open Science Framework (OSF) as well as the overall Reproducibility Project: Cancer Biology project page which will identify you as an author to these projects. If you would prefer to remain anonymous, that is ok too.
What are the next steps of coding?As a quick reminder before describing the steps: if at any stage of the process you have a question or are unsure of how to proceed with the organization or presentation of the information, you should feel free to contact the Tim at tim@cos.io. This process will most likely require a constant dialogue to make the process easier for you and completed in a way that is the most beneficial for this project.
1) Identify one of the remaining papers for coding:a) Look at the Project Progress spreadsheet to identify which studies have not been claimed (column L (Coded by) will be empty for each available paper). Paper information can be found on the Studies spreadsheet as well as the Keywords spreadsheet which lists keywords associated with each paper.
b) Contact core team to claim study and have spreadsheet shared with you. Email Tim at tim@cos.io
2) Write up detailed experimental protocols for each key experiment:This step will allow us to have a detailed protocol to give to the labs which will be performing the replications. Each protocol you write for a given experiment will be given to a different lab to perform. Therefore each tab should have all the information necessary to perform that replication, or refer to other tabs for additional information if necessary. Only the necessary protocol tabs will be given to the researchers performing the experiment, not this entire spreadsheet.
The protocols also need to be written as you would for a protocol in lab (all the details in a step-by-step manner) and how methods papers are generally written. This might require some additional data searching (be sure to write in the ‘source of information’ column where you got the information from) and filling in missing details based on missing or unclear information (be sure to write in the ‘questions’ column the question that this needs to be confirmed/checked by the original authors and if necessary the ‘comments’ columns to explain your information (or lack of it)).
After you are done with this step we will share it with the original authors who will have the chance to check it and answer any questions you had during this process. But this step should be done as if they are not going to comment and you had to perform it with only the information in the paper/found on your own. Our material and methods section of papers should have all the information contained within them for someone else to replicate and shouldn’t need clarification or additional information, and the work you are doing is going to test if that is indeed what is occurring.
Finally, a last note. You should not contact the original authors of the paper during this process. We want to capture the attempt to write out the details from the original paper first before we contact the authors. The Center for Open Science will be facilitating the discussion with the authors after the spreadsheet has been filled out, with questions where necessary, and reviewed. We want the authors input to ensure we have the correct protocol and to provide any additional insight to ensure a fair replication attempt. What we need during this stage are detailed, succinct questions that you think we should ask the original authors to help clarify the experimental protocols and statistical information. Details are listed below about where to put those questions in the spreadsheet so we and eventually the original authors understand the question and the context of the question.
a) Read and review the original publication being replicated to understand the context of the chosen experiments.
b) Review key experiment(s) chosen for replication in “RPCB: Coding Study [X]” google doc shared with you. Determine if other related experiments (such as experiments found in the supplemental) should be included that may have been overlooked. Also feel free to suggest inclusion/exclusion of experiments.
c) For each key experiment, create a new tab (identified with the figure panel ID) for protocol coding. See the Template for detailed coding of replication papers. If two figure panels are related (ie the main figure shows the experiment and the supplemental figure shows the controls) and test the same hypothesis you can combine them, but ideally each one would have its own protocol tab.
To that tab:
d1) Add the hypothesis being tested.
d2) List all reagents identified from the original study that are required for the experiment, including, if necessary, comments to help explain additional technical details, questions, etc. You do not have to list supplier information here, that will be collated and included in a separate tab (see f below). Any additional information you find from another source outside of the original paper (ie references, supplier website, etc) be sure to include it (column E, source of information).
d3) List all controls included in the paper and any potential additional controls that were not included in the original paper that you think might be beneficial to include (you can change the color of the cell for these to make it clear). Additional controls should be noted in the ‘Figures’ tab (in the ‘Additional controls’ column) and also identified in that experiments protocol coding tab as additional. See Study 1 as an example. We will try to include additional controls, but that will be determined on a case-by-case basis. The goal is to replicate the original study the way it was done, however sometimes controls were not included and if it is not too difficult/expensive to include additional controls we might include them to ensure the replication experiment is able to be interpreted properly.
d4) Give a detailed protocol description of the experimental procedure extracted from the original study. List out each step as you would if you were going to perform it yourself, not a brief overview of the procedure, which is quite possibly what you are working with in the original paper. Comment on any additional details to clarify the protocol step and add any questions required from the author or that you have about that step (or missing step) of the protocol. See study 1 as an example. Any additional information you find from another source outside of the original paper (ie references, supplier website, etc) be sure to include it (column E, source of information).
d5) List all data that will be collected during the replication. This includes all raw data required to present the results similar to the original paper and any additional data that can easily be collected to measure a replication attempts progress (such as transfection efficiency, etc). All raw data will be collected and made accessible on that experiments project page on the OSF. See study 1 as an example.
e) If there are any general protocols applicable to multiple replication experiments, create a new tab for each one and code according to the same guidelines (where necessary) as described above. Reference the general protocol in each of the specific replication experiments it pertains to. See study 1 as an example. Only use this if the general protocol is very detailed in itself and is almost identical to each experiment it applies to, otherwise just copy the information from tab to tab.
f) Make a complete list of reagents required for all of the replication experiments in the ‘Complete reagent list’ tab. Add supplier, catalog number, sequence information if necessary, etc. Note where the information came from (the original paper, vs other sources) and highlight any areas that require information from the original authors. See study 1 as an example. Many reagents have already been identified here. Any additional information you find from another source outside of the original paper (ie references, supplier website, etc) be sure to include it (column E, source of information).
g) Summarize the required materials and additional information/clarification to be requested from the authors in the “Requests from authors” tab. See study 1 as an example.
h) Notify the core team of completion. Email Tim at tim@cos.io
i) Fill out the “Subjective analysis by coder” document that will be sent to you after you are done and comment on your opinion for the potential for replication for each experiment. (This is kept confidential and only filled out to account for any potential bias during the project)
3) Extract statistical information for power calculations:
This step will help extract the information necessary to complete the power calculations for determining the number of repeats needed to be performed to have at least 80% power. You will only have to fill out columns A, B, and C.
a) In “power calculation” tab list all key experiments to be replicated in the ‘Experiment ID’ column using the figure panel.
b) Comment on what type of replication is being performed: Quantitative (statistical information is presented in the original paper (and can be extracted)); Qualitative (only a representative image is presented (or statistical information can not be extracted)); In Silico (replication will be on the analysis of an existing data set).
c) Extract statistical information from paper. List a summary of all pertinent information extracted from the original paper, such as mean, s.d., s.e.m. and n. Note the appropriate comparisons. See study 1 as an example.
d) If necessary, add questions for the original authors in the ‘Requests for Authors’ tab for obtaining accurate numbers or more information that might allow qualitative replications to become quantitative. See study 1 as an example.
e) Notify the core team of completion. Email Tim at tim@cos.io
4) Go over all the listed guidelines and checklists:
This step will identify adherence to guidelines and checklists and the ability of a given experiment to replicate or not. See Study 1 as an example.
a) Comment on Red Flags, Nature’s reporting checklist and, if necessary, the ARRIVE guidelines in the ‘Checklists’ tab.
b) Notify the core team of completion. Email Tim at tim@cos.io
What’s next?
As mentioned above if funds are available experiments identified as include/exclude will be evaluated for inclusion. This will not be known until the included experiments are given an estimated cost from Science Exchange providers based on the experimental protocol coding and input from the original authors. If additional experiments are to be included, you would be asked to repeat this process for those additional replication experiments. Once the replication experiments are completed by the Science Exchange network, the data will be written up and published in PLOS ONE with you as a co-author.
Finally, a last note. You should not contact the original authors of the paper during this process. We want to capture the attempt to write out the details from the original paper first before we contact the authors. The Center for Open Science will be facilitating the discussion with the authors after the spreadsheet has been filled out, with questions where necessary, and reviewed. We want the authors input to ensure we have the correct protocol and to provide any additional insight to ensure a fair replication attempt. What we need during this stage are detailed, succinct questions that you think we should ask the original authors to help clarify the experimental protocols and statistical information. Details are listed below about where to put those questions in the spreadsheet so we and eventually the original authors understand the question and the context of the question.
a) Read and review the original publication being replicated to understand the context of the chosen experiments.
b) Review key experiment(s) chosen for replication in “RPCB: Coding Study [X]” google doc shared with you. Determine if other related experiments (such as experiments found in the supplemental) should be included that may have been overlooked. Also feel free to suggest inclusion/exclusion of experiments.
c) For each key experiment, create a new tab (identified with the figure panel ID) for protocol coding. See the Template for detailed coding of replication papers. If two figure panels are related (ie the main figure shows the experiment and the supplemental figure shows the controls) and test the same hypothesis you can combine them, but ideally each one would have its own protocol tab.
To that tab:
d1) Add the hypothesis being tested.
d2) List all reagents identified from the original study that are required for the experiment, including, if necessary, comments to help explain additional technical details, questions, etc. You do not have to list supplier information here, that will be collated and included in a separate tab (see f below). Any additional information you find from another source outside of the original paper (ie references, supplier website, etc) be sure to include it (column E, source of information).
d3) List all controls included in the paper and any potential additional controls that were not included in the original paper that you think might be beneficial to include (you can change the color of the cell for these to make it clear). Additional controls should be noted in the ‘Figures’ tab (in the ‘Additional controls’ column) and also identified in that experiments protocol coding tab as additional. See Study 1 as an example. We will try to include additional controls, but that will be determined on a case-by-case basis. The goal is to replicate the original study the way it was done, however sometimes controls were not included and if it is not too difficult/expensive to include additional controls we might include them to ensure the replication experiment is able to be interpreted properly.
d4) Give a detailed protocol description of the experimental procedure extracted from the original study. List out each step as you would if you were going to perform it yourself, not a brief overview of the procedure, which is quite possibly what you are working with in the original paper. Comment on any additional details to clarify the protocol step and add any questions required from the author or that you have about that step (or missing step) of the protocol. See study 1 as an example. Any additional information you find from another source outside of the original paper (ie references, supplier website, etc) be sure to include it (column E, source of information).
d5) List all data that will be collected during the replication. This includes all raw data required to present the results similar to the original paper and any additional data that can easily be collected to measure a replication attempts progress (such as transfection efficiency, etc). All raw data will be collected and made accessible on that experiments project page on the OSF. See study 1 as an example.
e) If there are any general protocols applicable to multiple replication experiments, create a new tab for each one and code according to the same guidelines (where necessary) as described above. Reference the general protocol in each of the specific replication experiments it pertains to. See study 1 as an example. Only use this if the general protocol is very detailed in itself and is almost identical to each experiment it applies to, otherwise just copy the information from tab to tab.
f) Make a complete list of reagents required for all of the replication experiments in the ‘Complete reagent list’ tab. Add supplier, catalog number, sequence information if necessary, etc. Note where the information came from (the original paper, vs other sources) and highlight any areas that require information from the original authors. See study 1 as an example. Many reagents have already been identified here. Any additional information you find from another source outside of the original paper (ie references, supplier website, etc) be sure to include it (column E, source of information).
g) Summarize the required materials and additional information/clarification to be requested from the authors in the “Requests from authors” tab. See study 1 as an example.
h) Notify the core team of completion. Email Tim at tim@cos.io
i) Fill out the “Subjective analysis by coder” document that will be sent to you after you are done and comment on your opinion for the potential for replication for each experiment. (This is kept confidential and only filled out to account for any potential bias during the project)
3) Extract statistical information for power calculations:
This step will help extract the information necessary to complete the power calculations for determining the number of repeats needed to be performed to have at least 80% power. You will only have to fill out columns A, B, and C.
a) In “power calculation” tab list all key experiments to be replicated in the ‘Experiment ID’ column using the figure panel.
b) Comment on what type of replication is being performed: Quantitative (statistical information is presented in the original paper (and can be extracted)); Qualitative (only a representative image is presented (or statistical information can not be extracted)); In Silico (replication will be on the analysis of an existing data set).
c) Extract statistical information from paper. List a summary of all pertinent information extracted from the original paper, such as mean, s.d., s.e.m. and n. Note the appropriate comparisons. See study 1 as an example.
d) If necessary, add questions for the original authors in the ‘Requests for Authors’ tab for obtaining accurate numbers or more information that might allow qualitative replications to become quantitative. See study 1 as an example.
e) Notify the core team of completion. Email Tim at tim@cos.io
4) Go over all the listed guidelines and checklists:
This step will identify adherence to guidelines and checklists and the ability of a given experiment to replicate or not. See Study 1 as an example.
a) Comment on Red Flags, Nature’s reporting checklist and, if necessary, the ARRIVE guidelines in the ‘Checklists’ tab.
b) Notify the core team of completion. Email Tim at tim@cos.io
What’s next?
As mentioned above if funds are available experiments identified as include/exclude will be evaluated for inclusion. This will not be known until the included experiments are given an estimated cost from Science Exchange providers based on the experimental protocol coding and input from the original authors. If additional experiments are to be included, you would be asked to repeat this process for those additional replication experiments. Once the replication experiments are completed by the Science Exchange network, the data will be written up and published in PLOS ONE with you as a co-author.
No hay comentarios:
Publicar un comentario
Nota: solo los miembros de este blog pueden publicar comentarios.