In 1990, Michael Fix and I co-edited a book of articles about the use of a technique called ‘auditing’ to measure discrimination against ethnic minority groups when renting or purchasing a property. The publishing organisation, the Urban Institute, was a leading voice in this field, attracting top analysts doing ground-breaking work in the area to contribute chapters. Their draft articles were carefully reviewed and edited, and several thousand copies of the book printed. But when we sent advanced copies of the book to contributors, to our surprise, authors of one chapter told us they believed there to be serious methodological issues in another chapter.
Disagreements between authors and reviewers is fairly common, but it can be tricky to know how best to handle it. Here I talk about how we managed the auditing book disagreement and draw out lessons for others needing to resolve such disputes.
The dispute
Advanced copies of the book, Clear and Convincing Evidence: Measurement of Discrimination in America, were sent to contributors for their information and use in responding to potential inquiries following the book’s launch.
The authors of one chapter, James Heckman and Peter Siegelman, quickly sent a message saying that they believed there to be serious methodological issues in the chapter on the flagship 1989 study to measure housing discrimination. The study involved 3,800 audits in 25 metropolitan areas, selected to be representative of all US cities with a population of over 100,000.
Clearly this was a bombshell, particularly because James Heckman was a leading national econometrician (who in a few years would receive the Nobel Prize in Economic Sciences, with Daniel McFadden, for their contributions in developing modern econometrics).
The concept of auditing for discrimination is straightforward. Two individuals (auditors or testers) are matched for all relevant personal characteristics other than the one that is presumed to lead to discrimination, e.g. race, ethnicity or gender. Then they apply for a job, a housing unit, a mortgage, or begin to negotiate for goods or services. The results they achieve and the treatment they receive in the transaction are closely observed and documented separately by the auditors, and then analysed to determine if the outcomes reveal patterns of differential treatment on the basis of the trait studied and/or protected by antidiscrimination laws. +
How the dispute was handled
The Institute management’s first act after receiving the critical message was to halt the book’s distribution.
Senior managers, and Michael Fix and I were acutely aware of the reputational risk to the Institute of proceeding with distribution. We also knew that the author of the chapter in question, Professor John Yinger, who designed and executed the econometric analysis of the data from the 3,800 housing audits in the national study, had an excellent reputation and his analysis had previously undergone several rigorous reviews.
The decision was taken to hold a one-day discussion at the Institute. We had to wait a couple of months to fit the schedules of all essential participants.
It was decided to invite two other senior econometricians to provide external expert opinion: Arthur Goldberger, author of the first go-to book on ‘modern econometrics,’ and the Institute’s resident consulting econometrician. Michael Fix, I and a couple of other Institute staff also participated.
Two points about the discussion are noteworthy. First, the objective of the day was to have a full discussion of the issues and not necessarily to arrive at a final conclusion. Second, a constructive tone was established from the outset by the way in which people were greeted and the Institute’s President’s opening remarks. No one tried to ‘score points.’
At the end of the day, experts were asked to provide their views on the issues and suggested changes to be made within a few weeks.
A total agreement on the issues raised was not reached. Some design issues were such that experts could reasonably disagree about the best procedures. Still, the outcome, facilitated by Michael Fix and others, was satisfactory to all parties.
Heckman and Siegelman did a substantial rewrite of their chapter on the survey design and econometric issues and added a mathematical appendix, stimulated by their more complete understanding of Yinger’s analysis. Yinger made adjustments to his chapter and contributed a comment on some points in the Heckman-Siegelman chapter. The comment appeared as an annex to the chapter and is therefore readily available to specialised analysts interested in the dialogue.
The revised book was published in 1992 and remains an important contribution to auditing literature. The original press run was destroyed.
Lessons for others managing disputes
While this example has some specific elements, the underlying problem of disagreements between authors are reviewers is fairly common. And I think the example holds and illustrates four lessons that are applicable to resolving many such disputes.
- Don’t panic. It is natural for authors whose work quality is being challenged to be defensive and to want a quick resolution. Similarly, managers are likely to want a quick resolution too. This can lead to not bringing in other experts, or indeed the right experts if they are not immediately available. Pressure for fast resolution can also result in inferior decisions. Make a plan of what to do, based on the time and resources you have available. In our case, we already had the books printed, so the timing was far from ideal. But we were lucky that we did not have external time pressures, such as a contract deliverable deadline, so we could take the time needed to fully address the issues.
- Add other experts to the discussion. One really positive effect of external experts is to channel the discussion away from the author versus reviewer format (where the back-and-forth can easily become acrimonious, even with a manager trying to keep the discussion professional). In our case we were able to bring in another leading econometrician who was highly respected by both parties.
- Don’t force a quick decision. Bringing in other experts and giving everyone time to rethink the issues raised after an initial discussion both take time. But it’s worth it. Based on our experience, I suggest a three-staged approach. First, bring people together to just discuss the issues. Give them time to think and ask them to submit ideas for resolution. Bring people back together for a follow up discussion and to agree a way forward. Of course, sometimes there are hard external deadlines and if necessary, the time between steps can be shortened and internal experts brought in.
- In the end the issue must be resolved. In many cases the issue will be technical, i.e. did the regression analysis deal successfully with a particular issue, such as heteroskedasticity? But often the issue will not be so black-and-white. This will particularly be the case where policy recommendations are involved. Sometimes additional evidence, based on the relevant literature, can be advanced and the clarification results in an agreed change if the author’s position is supported. Compromise may be possible if both sides are willing. If this fails and neither side can be convinced or willing to compromise, ultimately the relevant manager must take a decision.
In sum, with some fore thought by the person overseeing the review (for example to carefully identify the right expert that is respected by both sides to be involved) serious conflicts can frequently be avoided and strong outcomes achieved.