Checklist for Reviewers
1. Title: does the title adequately reflect the primary aim of the study?
- Does the abstract adequately report the primary findings of the study in terms of hard data results. For example, does the Abstract state that they the authors found significant findings (P<0.0X) = weak.
- Or (more appropriately), does the abstract state specific pre/post comparisons in terms of mean, SD, confidence intervals, etc. differences? = strong.
- Conclusions:Is the abstract’s conclusion justified by the data presented in the paper?
- Does the Introduction adequately present the intent of the study vs. being a review of the literature that may not apply to the intent of the study?
- Does the Introduction conclude with a specific study aim accompanied by a specific hypothesis driven intent?
Authors should always state their specific research hypothesis and identifying primary and secondary outcomes is even better. It is inappropriate to say that ‘we examined the effects of x-treatment on outcomes a, b, c, d, e….z and then build a paper about what shakes out.
The former is proper or reasoned science, while the latter is a shot-gun approach.
- Do the authors adequately describe the study design so that an independent group could replicate the trial?
- Is the trial randomized and blinded? If not, is there adequate justification for not being so.
- Is there adequate justification for the sample size used in the trial and how did the authors go about determining their sample?
- Are the methods and procedures to determine all outcomes adequately described so that others wishing to replicate the trial be able to do so assuming they had the appropriate skills to do so?
- Is the reviewer comfortable with statistics as a whole and, if not, should the reviewer ask for “statistical assistance” if they are not? Reviewers should be encouraged to ask for assistance, as this is a point where many things go wrong across multiple disciplines. This can be done right within their respective institution and doesn’t have to come back through the Journal.
- Are the statistical methods described adequate to assess the proposed study outcomes?
- Are statistical adjustments warranted within the analysis?
Example, if someone has a mixed cohort of men and women but not sufficient to detail gender responses, should gender be a covariate in the analysis?
- Are the results clearly presented?
- Are the data adequately detailed?
Example, the majority of reports should provide mean ± SD. When presenting mean or percent change those data should be accompanied by confidence intervals (90% or 95%).
It is incorrect to list change (percent or mean change) with SD or SEM. Also, the use of SEM or SE should be used sparingly unless justifiable. If there is doubt, the reviewer should ask the author to justify their use.
- Does the Table adequately reflect changes in the data from pre – post exam and interim markers when necessary.
- Does the Table contain an assessment of effect size or power if not detailed elsewhere? Will Hopkins “new view” is acceptable, particularly in this field (not 100% necessary but often helpful).
- Is the Table self-explanatory and detail how the data is presented. Mean ± SD, change, 95% CI’s etc
8. Figures: the same criteria as tables plus do the authors present figures with SEM or SE simply to make the error term look smaller.
9. Harms/Side effects/Drop-outs:
- Are Harms adequately addressed? If measuring safety via blood, how many people moved from within to outside of normal limits during study. (I think this is a case where the mean does not tell you everything.)
- Are drop-outs detailed relative to the reason or simply reported as a number. The reader should know why.
- Does the Discussion reflect the results an does it follow a logical progression to its conclusion or is it scattered and “all over the place?”
- Do the authors “take liberties” with their findings or try to make more of what isn’t there?
- Are Limitations to the study adequately discussed by the author or does the author try to skirt these issues?
- Is there a statement about Generalization and does that statement hold for the population examined? Do the authors try to make more of what is actually shown?
- Is the paper a sales job for a novel ingredient or is it a fair and balanced review of the product at hand?
- Do the authors Disclose financial relationships even if they are not directly applicable to the study at hand? (ISSN policy is noted here)
Exact wording used: Conflicts/Disclosures of Interest
In accordance with the guidelines on conflict of interest of ISSN, all authors and presenters are required to indicate if they have a relationship that, in the context of the subject matter and content of their submission, could be perceived as a real or apparent conflict of interest. This pertains to relationships with dietary supplement, pharmaceutical, food, beverage, or cosmetic ingredient and/or finished goods manufacturers and/or marketers, biomedical/analytical device manufacturers, contract manufacturers, contract laboratories, inventor/co-inventorship of a patent pending or issued, or with other corporations whose products or services are related in any way to the subject matter and content of the submission.
Disclosure is to include any relationship that may bias one's submission or which, if known, could give the perception of bias. Such relationships include, but are not limited to, employment by or receipt of research funds from an industrial concern, ownership of stock, membership on a standing advisory council, committee, or board of directors, authorship of a book of related title/topic, or being publicly associated with a company or its products or services. Other areas of real or perceived conflicts of interest could include receiving honoraria or consulting fees, receipt of gifts, gratuities, loans or special favors (including trips or speaker's fees), or receiving equipment in or out of the context of a research grant from such corporations or individuals representing such corporations.
If you have any conflicts of interest, please denote that in the acknowledgements.
- Is there a Funding Statement associated with the study?
The obvious is corporate or agency funding; however, if the study was funded by internal funds associated with the university, they should be disclosed as well. It adds to the weight of the paper as a means of presenting unfunded and often criticized scientific exploration.