Measuring the worth of a research career – hard problem or hardest problem?

How not to measure scientific productivity
[Via Ars Technica]

Ever since funding agencies started demanding some way to measure a return on their investment, administrators have been coming up with ways to measure scientists’ performance. And scientists have been coming up with ways to maximize their value under any particular measuring scheme. In every generation of this game of performance measurement, scientists produce carefully thought-out articles on how the current system simply doesn’t capture scientists’ work accurately.

A recent Physical Review E paper is one of these, showing that the raw citation count doesn’t really capture a paper’s importance, nor the researcher’s true performance. Yeah, amazing isn’t it?

[More]

Does being one of 100 authors on a groundbreaking physics paper mean you are more productive that a single author working on an obscure insect? Does working in  a research area that permits dozens of short papers mean you are more productive than in a area which produces one massive paper?

Trying to determine which science is ‘better’ is like trying to determine who is more intelligent. Or in school where we now determine how a student is doing, not based on their actual aptitude but on their ability to take a standardized test.

Sure, we get what we measure but is what we measure actually relevant to what we want?  And, smart people being smart people, you can bet that they will game just about any proxy used, with less productive researchers finding ways to mask their work.

And there are many cases where the measures used penalized anyone who took an unusual or alternative path to their academic position. Simply because they were different. The statistics of these approaches are based on a ‘normal’ approach to getting the job. If one comes from a different field – or even took time off from research – one could get a low score.

I think that the old ways of structuring academic careers – and the incentives provided – will be rapidly changing in a complex world of massive collaborations. I’m not sure what it will look like but I am sure it will be interesting.