Thursday, December 18, 2014

ICD 203 and ICD 206: Improving analytical practice in arguments and sourcing.

http://www.dni.gov/files/documents/ICD/ICD%20203%20Analytic%20Standards%20pdf-unclassified.pdf 



I am reading T. Fingar, Reducing Uncertainty (Stanford, 2011), where he describes changes in analytical practices post the infamous Iraq WMD National Intelligence Estimate. He describes intelligence analysts as using their training from graduate school, but the agencies wanted to make more rigorous the arguments and sourcing of such estimates and the Presidential Daily Briefing. The links above are to memos on standards and sourcing. Some of you may find them of interest in your own scholarly work, giving quite explicit guidance for making better grounded arguments.

Tuesday, December 16, 2014

It's Up to Us, as Faculty Members, to Make Tenure and Promotion Judgments


Originality, Independence, and Publication. Home Run Papers

1. In some fields, it is possible for a graduate student to have their article published in a respectable journal, while in others that is very very rare.

2. Of course, jointly authored articles with their advisor/professor are more common and more likely.

3. As for Independence: While we might expect a junior faculty member coming up for tenure to have developed a research path that somewhat different than their teachers and advisors, it is rare if ever that happens with doctoral students. Doctoral students usually work within the purview of their advisor's field of interest and research program.

4. As for Originality: If you are working at the forefront of a field of research, you cannot help but be original. It's no big deal. If you are replicating earlier work since it was not sufficiently credible, or going over a long-standing problem, in general you will exhibit originality in the quality of your work.

5. While there is a demand for a publication or two by graduate students, in some fields, when they go looking for a job, I find that hard to believe, unless the field is not very deep. (But see #2 above.)

6. Home run papers: The baseball analogy is quite prevalent in economics and business, and perhaps in some other fields. I never hear about them in physics or mathematics, or the humanities, or most of the social sciences. 
          a. Make sure that the home-run paper is actually the responsibility of the candidate, rather than a joint production with more senior authors.
          b. Really significant work that has high impact usually demands some years of maturity and experience beyond the doctorate and the assistant professorship. 
          c. I don't know if the baseball analogy is appropriate for scholarship.

Citation Counting

If you are comparing citation counts for different scholars, keep in mind:

If someone has a very high citation account, make sure that their citations are not the result of working with a very prominent scholar, likely more senior, often at the beginning of their careers. They may well deserve the credit, but probably not.

If someone has begun publishing in year X, and the comparison person began to publish in X-10, be careful that the time to citation does not get in the way of your comparison. Some fields take some time for citations to accumulate. And of course, if you have been doing scholarship for 10 more years, you may well have accumulated lots more citations.

In any case, if there are many collaborators, and they are of equal strength as scholars, I am not sure how to count.

Thursday, December 11, 2014

The "Home Run" Article

When evaluating scholars who work in economics or business, there is often a demand for a "home run" article: published in the right place, highly cited, a significant advance. I have not seen such a demand in other social sciences or in the natural sciences. In book publishing fields, in general the first book, often used to make tenure decisions, is rarely so influential and since it takes a while for books to be appreciated the citation data won't be helpful.

I also note that this is a requirement that seems to be honored in the breach at most institutions.

I find it interesting that the requirement is put upon scholars perhaps 5-7 years beyond their doctoral work, where articles may take 1-2 years to get through the reviewing process.

Departments that tenure a small fraction of their faculty, perhaps hiring from outside at that level, might well have this requirement and honor it.

What is striking is the expectation that such junior inexperienced scholars should be making such strong contributions.  Or perhaps, what is called a "home-run" is actually in a ballpark with close-in fences? In any case, early salience is taken as a sign of long term excellence.

My other thought is that perhaps economics and related fields are comparatively less profound and deep than other fields, and so it is possible for a junior person to make such a contribution. It's surely quite rare in physics or mathematics, for example, although the strongest scholars do stand out fairly early.

I don't know.

Tenure and Promotion Committees: When You Believe They are Wrong

Departmental or school tenure and promotion committees review a dossier of letters of reference, a CV, a personal statement, and perhaps other material. They present those findings to the department as a whole for a vote. Perhaps you believe their interpretation of the dossier is incorrect, whether it be denial or a tenure/promotion.

1. If you are on the committee, you might file a dissenting report. Or you might argue later on, whether in the faculty meeting or in letters to the higher-ups.

a. It does you little good to argue against a particular letter writer, as such, since you chose the letter writers.

b. So, what you do is to make a list of the main points brought out in the letters and in the committee report. You then present your argument in terms of those points. No ad hominem of the committee or letter writers. Rather, you will be presenting your position on those points, with whatever authority you possess. If there is counter-evidence, of course you want to present it. What you are trying to do is to make people think twice.

c. Don't distort the letters or whatever else you are referring to. Selective quotation, insults to the letter writer, etc. are all likely to have your position weakened.

d. Say you are the strongest person in the field in your department. Your judgment might well be given greater weight, but still you are making an argument. Your opinion, as such, may well be important, but it is vital that your argument be fair, to the point. If you do not believe the candidate's methods are appropriate, say why....

2. If you are not on the committee, you are welcome to question the credibility of the letter writers. But you cannot be seen as biased or too selective. Credibility is questionable if the writer is too closely allied to the candidate, if methodological preferences seem to sway the letter writer, etc. Still you are likely to find yourself acting as in #1 above.

3. If you are in a higher-up position--chair, dean, university committee--you are welcome to act as in #2 if there is good evidence.  You may well have received a memo from a dissenting member of the faculty. But it is best to first read the dossier of that is presented by the committee, and then the later material.

4. What you are trying to do is to sway your colleagues. But you are also building into the record your dissent, and in so far as your dissent is fairly and accurately argued it will more likely be effective.

5. In the end, the university will survive.

If you believe the candidate should not have been tenured/promoted and the university decides otherwise, make sure the candidate gets the kind of mentoring that will make them more worthy of your good judgment.

If you believe the candidate should have been tenured/promoted but the university decides otherwise, help the candidate find a good position elsewhere. That you disagreed might well be confidential information, unless the dossier and related materials are to be public. That you believe the candidate is worthy is something you can share with the candidate as you help them go forward.

6. People often carry grudges for years, either because they have "lost" in this process. I am not talking about the candidate, rather the faculty and the committee. Again, the university will survive. And you have better things to do. If you don't, find another job.

7. As for legal remedies, I have no good advice.

Wednesday, December 10, 2014

Inside Black Boxes: Survey Research, Fieldwork, Ground Truth, and "Effective Field Theories"


Often a presentation starts with some sort of data set they are analyzing. Some of the time they have generated the data set on their own with systematic and effective survey work, some of the time it is from various governmental or health files. What is striking to me is that whatever these people are studying is in effect a black box, when in fact they could enter the box and find out better how it works. Fieldwork is a very different style of research. Some times it is referred to it as "anecdotal," while I believe that "ground truth" is more appropriate. You want to see if what there is inside these boxes makes sense in terms of the characteristic of the box. To my mind, I want to understand the mechanism in the black box, the way the system or the thinking or the behavior internal leads to the measured external variables--that data set and its analysis. 

Now if you are a physicist, the objects you may be dealing with are fully characterized by a very small number of characteristics. But if you want to learn more, about how those characteristics are "generated," you need to go to much higher energies and much smaller distances to see inside the object you are studying. Of course, it is black boxes all the way down, or at least pretty far down. You can't interview a proton. But you can hit it with other protons, electrons, gamma rays, ... and so discover lots about what is going on inside.

What physicists now say is that they have "effective field theories." They are "effective" because there is no claim that they are ultimately the true story, but they do account for what you see at a certain scale, and you expect that if you go to a finer scale you won't throw out the effective field theory at the less fine scale. The effective field theory that we use today is called the Standard Model, and at larger scales and smaller energies, there are other effective field theories. We don't have a good clue (in terms of empirical or experimental findings) of what will be the effective field theory at smaller scales, "beyond" the Standard Model, although there is a rich landscape of speculations--none of which have any empirical support yet.

Saturday, December 6, 2014

Blind Analysis of a Data Set

http://www.slac.stanford.edu/econf/C030908/papers/TUIT001.pdf 

is an article that discusses this in the context of particle physics.

Blind Analysis is a method used by particle physicists, who nowadays usually have very large data sets to analyze (typically terabytes and then some). They take a small fraction of the data set (say 1-3%), and develop their analysis methods (cuts in the data, ways of thinking there might be a signal, etc) using that small set to work it all out. What they are trying to do is to figure out how to see a small effect in a sea of noise and irrelevance (interactions other than the ones they are concerned with, which are very rare). When they are satisfied that they have done the best they can do, then and only then, do they run all the data through that "best" analysis method. With such a small test data set, they are unlikely to see anything very interesting so far. Rather, only after they put all the data through the system might there be any significant effects.

The basic idea is to prevent you from working over and changing your analysis method to get the result you might expect.  I guess this would be called data mining (although that is not the right term) or fudging.

I don't know if people then do some more analysis, revising their method, once they have had the blindfolds removed. 

Double-blind is the gold standard for medicine and biostatistics. I do not know if blind analysis is standard for social science statistical analysis. My impression is that people still try various regressors and various schemes to see if they can get good results. As I just said, I may well be wrong.

MK

Bupkis and Not-so-Bupkis in the US Budget

My colleague Ed Kleinbard sent me some of the CBO tables re the US Budget.

http://www.cbo.gov/sites/default/files/45229-UpdatedBudgetProjections_OneColumn.pdf

I am ashamed to say I have never spent much time looking at these numbers. They are a revelation, I believe.

1. Revenues from the corporate income tax is rather small compared to individual income taxes, and is in fact something like 1% of the Gross Domestic Product. You'd think that corporate titans would be more concerned with their own income taxes--but this shows how corporate-spirited they are

2. Discretionary Expenditures divide into domestic and defence, and they are roughly the same.

3. The various mandatory expenditures (Social Security, Medicare,...) dominate the discretionary ones.

...  More to follow.

MK

PS. BUPKIS is a Yiddish term for something so small it does not matter.

Friday, December 5, 2014

Iraqistan Dollar Numbers



The Iraqistan wars cost about $3+trillions, and the discounted cost of future benefits to veterans is a bit less than $1trillions (they peak 20-30 years out), and there is another $300billions of unreimbursed care provided by families and others. Linda Bilmes of Harvard's Kennedy school is my source. By the way, the US GDP is about $17trillions, and the US Budget is about 21% of that, with a difference between revenue and spending of a few of those percent. The national debt is about the same as the GDP. One of Bilmes' points is that the Iraqistan war was financed by borrowing, and we will pay interest on that debt (say 3% of $3trillions is $90billion, and the defense budget is about $500+billions) so decreasing our future military spending. Other places I look place defense spending at $1.2trillions--I'm sure I am missing something here. 

Experts point out how GDP has lots of problems as a measure, that there are big pools of money in the budget that are restricted (entitlements and the like) so that the unrestricted budget or discretionary spending (including defense) is rather smaller, that tax expenditures ("loopholes" and specific exclusions) are large, $1+trillions (about twice as large as non-defense discretionary spending).