Ranking in Science: relating meta-analysis to science
O#o van Nieuwenhuijze

Introduction

Presently many studies are published and presenting, addressing “Meta Analysis”.
The following relates these literature database studies to the databases, reviews, research papers and research.
It will become clear that literature study is very different, and  very remote, from actual research.
This poses the question of the relevance of such studies for society, and science.

It will become evident that Meta-Analysis publications are in part based on their (remote) relationship to scientific research.
In larger part they are based on the role of the researcher in the scientific community.
This necessitates us to discern the ranking of scientists in the scientific social society: workers, researchers, reviewers and analysts.
We will see that this ranking represents a transition away from subjective experience of findings in nature - via method studies, and comparative studies - to objectified analysis of ‘facts’ in print.

It will become evident that this ranking of social status is based on  academic organisation.
‘Heads of departments supervise Professors who supervise PhD Students who study the work done in the field of research’.
The social ranking - scales of salary - are implicitly applied to the ranking of the status of the publications also.
The nature of these publications are however very different; and the method of the ‘grading of papers’ can not be applied to the ‘assessing of research findings’.

Table 1 summarises the differences of salary rank, hierarchic status, publication position, and relevance of the paper.

Head of department   meta-analysis        statistics
Professor                  review                  data mining
PhD Student              research paper      methodological study
practitioner               findings                report

Setting

From a desktop computer, to a room, to a lab, to the work field, the settings of the study vary greatly; as does the relevance of the findings:

• Meta-analysis studies are generally done by someone in a room in a university uses a computer to log into a publications database, and looks for papers with the same topic. tries to find as many as possible, and then comes to odd conclusions.
  Most often they are studied by statisticians, without personal understanding of the field of study.
• Comparative studies are done in academic of research settings. They are literature studies which compare research reports for common denominators; most often they are studies by experts in the field.
  Very often these are professionals from the field of research, without practical involvement.
• Practical Research Papers are written by scientists in the field with active involvement; they describe what they experienced or what they verified from their colleagues.
• The practical work reports are generally done by people who do not have great interest in writing about their findings. The people who do the actual work (the real researchers) are very often too busy to describe this, or not very interested in writing.

Relevance

This means that the people who best know what happens 'on the job' rarely write all what happens; theoreticians - sometimes failed practicians - do this instead. They are sometimes to immersed in the field to know well about the state of the art of scientific research.
When their work is repeated in (academic) research setting, it is often done by people (students) who have little practical knowledge of what happens in the field.
Meta-studies are based on the theoretical reviews of the research descriptions made by (very few) people with 'hands on' experience.

It means that a statistician, generally without knowledge of the field, uses what reviews were found in a database, to assess studies about work in the field. I.e. a statistician rates academic reviews of theoretical patterns about practical work experience.
From, e.g. a doctor, to a PhD student, to a faculty professor, to a resident statistician, it is the statistician which claims status by reporting on disciplines.
The outcome however pertains to the reporting style of the review (the criteria and result which are mentioned), which is already an interpretation of the 'lab reports' of the actual findings.
The difference in involvement reflect in differences in academic status, and remoteness in relevance to the actual experiential findings.

Meta-analysis as a result rarely describes the actual findings.
Most of the discussions of meta-studies concern the suitability for comparative research.
Most of the Review Studies discuss the interpretability of the data which are published.
Most of the studies report on the way in which the experimentation was done and outlines the findings.

These are not only different types of science.
They are also different levels of academic status ranking.
The relevance of such studies are related to the rank of the researcher.
However the relevance of the studies is based on the findings in the field by the reporter.

The following table summarises the finding

Report  | practitioner  | reporter  | trying to understand new findings

Research | Paper | PhD student  | researcher trying to relate methods

Review  | Faculty Professor | critic  | interpretability of the data

Meta-study | Statistician  | 'analyst'  | suitability for comparison

Ranking

It will be evident that there is a large change from practice to theory, and from specific findings to an attempt to overall pattern recognition.
It will also be clear that there is a moving away from factual personal experience towards description based on form and protocol.

Many people are impressed by the rank of the (often) specialist statistician, and erroneously assume that this means that the outcome of the meta-study are relevant for the field; the following example illustrates that they are not. (The example summarises the method(ology) shown in 4+ publications by P Knipschild c.s.).

1. The newspaper heading said 'homeopathy/acupuncture disproved'.
2. The title of the paper said "no proof for homeopathy/acupuncture".
3. The abstract/summary mentioned "not enough suitable reviews could be found"
4. The paper described "the reviews were so different as that it was not possible to compare findings".

Such meta-study outcomes illustrate:

1. the papers are often intentionally written to mislead the public opinion.
2. the statistician is aware of own lack of knowledge of the field, but does not publish this.
3. the criteria for judging research reviews, replace the criteria for understanding the data/findings.
4. 'science' is presented as if the administration of the work of others, instead of discovering the unknown.

Conclusion

Social ranking of a scientist must not be confused with the scientific ranking of publications.
The methodology of the ranking of student papers is no basis for rating scientific research.
The aims of meta-studies are possibly best served by having computers with input-forms into which reports from the practitioners in the field can be filed and compared.
This will help more towards the possibility to compare findings and advance science: the real researchers can directly see the relevance of their own findings.