You
all know what I am saying below... I am not at all saying that we abandon citation numbers,
etc. I've just seen too many manipulations, gaming the system, misleading
claims, so that in the end I only ask: what is the contribution?, what is the
quality?, did they do the work? Yes I use Consumer Reports, and I tend to
ignore small differences in scores, and go for the higher number even though they tell me that differences of three points or less do not matter--just because it is easy, and the
consequences are minor. But here we are talking about something of greater
consequence.
Numbers in promotion packets
may even be helpful, but in general they are manipulated. I've probably read
1000 dossiers, and so have seen lots of numbers (claimed to be
statistics), only some of which are not faked one way or the other. In general,
very low or very high numbers are indicative, but they ought be checked against
your other evidence, and be treated with suspicion (this is my experience,
having been bamboozled at first more than once--if there is a university promotion committee, the
best part is that what you miss your colleague will discover). So I have read
about 10,000 reference letters, and something like 25%, at best, are
really helpful. I know of one distinguished scientist who checks his
h-index value each Friday--but he is at the very top of his field, very well
recognized and rewarded.
1. In
general, what matters in the end is your contribution to scholarship.
That is a substantive notion, and letters of reference and your personal
statement should indicate that. MIT's economics department, at least in 1980,
asked only that question. Numbers of publications, venues, citations, etc are
only secondary. I do appreciate the need for numbers and statistics (if they
are really statistics rather than numbers misrepresenting themselves as
statistics). And playing Moneyball has proved extraordinarily useful, revealing
what human judgment misses. Kahneman and Tversky have much to teach us.
As for numbers and rankings, it would be useful to have the most
elementary of measures of uncertainty attached to them. When rankings differ by
a tenth of a point, it surely matters for bragging, but not for actual
information. See # 5 below. The numbers we get from citation sources are
claimed to be complete samples, but in fact they are often polluted with junk.
What should be the errors assigned to them?
2. If
you are using numbers, and almost all citation "statistics" are just numbers
a. comparisons with a relevant cohort are useful
b. be sure they are not stuffed--
1. do most of the citations come from when someone was a postdoc with a
famous scholar--so that you compare your candidate with someone elsewhere with
terrific numbers, but in fact the high-number scholar's number come from
that postdoc period
2. is the source of the numbers reliable--Google overcounts, ISI does not count books but is the most studied by the sociologists of science
3. do they accord with what you know of the contribution?
3. Again
I understand the need for these numbers in rankings etc. Just be sure you are
getting what you are paying for. Of course, you may be willing to allow the
market to use these numbers and rankings to value your goods, but are you so
happy when your value goes way down?
4. Universities are fabulous at pumping themselves up--eg. Best in the West, heralding its new
rankings in someone's system etc,... Again, I want a university the
football team would be proud of (said by the coach at U. Oklahoma in 1922 to
the state legislature to get better support for the University)
5.
Trigger warning, PG or R rating: When I was a little boy, the New York Post
regularly gave the stats of women, beginning with the size of their brassiere.
Bigger was better, and Jayne Mansfield topped them all, as I recall. (Her
daughter, ?Marrissa Hargitay, stars in a current TV series.) Currently, men's
claims have entered the political realm, and historically, codpieces for men,
and in the 1980s socks stuffed in men's briefs, and pornographic movies of the
1970s and 80s featured such claims or visible evidence. I gather from women I
know, bigger may not be better, for it is how you use your instrument that
matters. In certain cities, a combination of silicone and exercise have a dramatic effect on the numbers and looks of women. In others, plastic surgery plays a large role. And I was told many times, that you marry someone who would be a good
mother or father to your children.
My point here is that attend to what matters, and use the numbers
(pretending to be statistics) to check your intuitions.
No comments:
Post a Comment