Checklist: Step-by-Step Analysis of Competing Hypotheses

Below is an excerpt from Psychology of Intelligence Analysis in which the author discusses how to analyze competing hypotheses (emphasis added).

I have only included the introductory text of the chapter. To read the discussions for each step in the process for Analysis of Competing Hypotheses (ACH), please refer to the book in link above.

Chapter 8: Analysis of Competing Hypotheses

POIA2Analysis of competing hypotheses, sometimes abbreviated ACH, is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.

ACH is an eight-step procedure grounded in basic insights from cognitive psychology, decision analysis, and the scientific method. It is a surprisingly effective, proven process that helps analysts avoid common analytic pitfalls. Because of its thoroughness, it is particularly appropriate for controversial issues when analysts want to leave an audit trail to show what they considered and how they arrived at their judgment.

* * * * * * * * * * * * * * * * * * *

When working on difficult intelligence issues, analysts are, in effect, choosing among several alternative hypotheses. Which of several possible explanations is the correct one? Which of several possible outcomes is the most likely one? As previously noted, this book uses the term “hypothesis” in its broadest sense as a potential explanation or conclusion that is to be tested by collecting and presenting evidence.

Analysis of competing hypotheses (ACH) requires an analyst to explicitly identify all the reasonable alternatives and have them compete against each other for the analyst’s favor, rather than evaluating their plausibility one at a time.

The way most analysts go about their business is to pick out what they suspect intuitively is the most likely answer, then look at the available information from the point of view of whether or not it supports this answer. If the evidence seems to support the favorite hypothesis, analysts pat themselves on the back (“See, I knew it all along!”) and look no further. If it does not, they either reject the evidence as misleading or develop another hypothesis and go through the same procedure again.

Decision analysts call this a satisficing strategy. (See Chapter 4, Strategies for Analytical Judgment.) Satisficing means picking the first solution that seems satisfactory, rather than going through all the possibilities to identify the very best solution. There may be several seemingly satisfactory solutions, but there is only one best solution.

Chapter 4 discussed the weaknesses in this approach. The principal concern is that if analysts focus mainly on trying to confirm one hypothesis they think is probably true, they can easily be led astray by the fact that there is so much evidence to support their point of view. They fail to recognize that most of this evidence is also consistent with other explanations or conclusions, and that these other alternatives have not been refuted. Simultaneous evaluation of multiple, competing hypotheses is very difficult to do. To retain three to five or even seven hypotheses in working memory and note how each item of information fits into each hypothesis is beyond the mental capabilities of most people. It takes far greater mental agility than listing evidence supporting a single hypothesis that was pre-judged as the most likely answer. It can be accomplished, though, with the help of the simple procedures discussed here. The box below contains a step-by-step outline of the ACH process.

So, the analysis of competing hypotheses consists of eight steps in total, as shown in the image below. To read about each individual step, please refer to chapter eight in the book.

POIA2

Disclosure: I wrote this article myself, and it expresses my own opinions. I am not receiving compensation for it. I have no business relationship with any company or individual mentioned in this article. I have no positions in any stocks mentioned.

Mohnish Pabrai, Guy Spier & Michael Shearn on Investment Checklists

Let’s start with a few words from Mohnish Pabrai on the use of an investment checklist from the Value Conferences session (see links below).

“For example, typically when I run the checklist the first time when I’m looking at an investment — it’s actually the last thing I do before making an investment — it usually takes no more than 15-20 minutes, maybe 30 minutes max to run it, but the first time I run it, it actually pops up all sorts of questions to which I don’t know the answer. That’s like been the biggest value addition, which is there are these blind spots that I have completely ignored. Then I go back and research the business some more to get answers to those questions and sometimes that can take a week or longer. Then I rerun a second time. Now the second time when I rerun it, we’ve got all the questions answered and we can see the failure or possible failure points.”

—Mohnish Pabrai

Go here for full transcript of the session.

See here for original post at Value Conferences.

Disclosure: I wrote this article myself, and it expresses my own opinions. I am not receiving compensation for it. I have no business relationship with any company or individual mentioned in this article. I have no positions in any stocks mentioned.

Forbes Transcript: Mohnish Pabrai on Checklists

“No wise pilot, no matter how great his talent and experience, fails to use his checklist.”—Charlie Munger, Poor Charlie’s Almanack

“So you know, and all of these questions are not questions I created out of the blue. What I did is I looked at businesses where people had lost money.”—Mohnish Pabrai

Steve Forbes Interview With Mohnish Pabrai

A few years back, in 2010, Steve Forbes interviewed Mohnish Pabrai. This interview is about half an hour long, and among the topics discussed, checklists was one of them. The interview is available on YouTube, and there’s also a transcript available at Forbes’ – see here.

Below is an excerpt of the part of the discussion where Steve and Mohnish talk about checklists. Emphasis added by me.

“Make Checklists

Forbes: So in the first quarter of 2010, did you add any positions?

Pabrai: Yeah, actually, we did. We did find. In fact, there’s one I’m buying right now. But I found two businesses, but they’re anomalies. They were just, you know, businesses that had distress in them because of specific factors. And I think we’ll do very well on both of them. They’ll go nameless here. But no, I think, for example, in the fourth quarter of 2008 or the first quarter of 2009, you could have just thrown darts and done well. And that is definitely not the case today.

Forbes: And finally, telling you about mistakes, one of the things I guess an investor has to realize, they cannot control the universe. Delta Financial: You had done the homework, you fell and then events took it away from you.

FI1Pabrai: Well Delta Financial was a full loss for the firm, for the fund. We lost 100% of our investment. It was a company that went bankrupt. And we’ve learned a lot of lessons from Delta. And one of the lessons was that Delta was, in many ways, a very highly levered company and they were very dependent on a functioning securitization market. And when that market shut down, they were pretty much out of business. And they were caught flat-footed. And so there’s a number of lessons I’ve obviously learned from Delta.

It’s easier to learn the lessons when you don’t take the hits in your own portfolio. But when you take the hits in your own portfolio, those lessons stay with you for a long time.

Forbes: So that gets to, you’re a great fan of The Checklist Manifesto. And you now have checklists. You said one of the key things is mistakes, in terms of a checklist, so you don’t let your emotions get in the way of analyzing. What are some of the mistakes on your checklist now that you go through systematically, even if your gut says, ‘This is great. I want to do it.’

Pabrai: Yeah, so the checklist I have currently has about 80 items on it. And even though 80 sounds like a lot, it doesn’t take a long time.

It takes about 30 minutes to go through the checklist. What I do is when I’m starting a business, I go through my normal process of analyzing the business. When I’m fully done and I’m ready to pull the trigger, that’s when I take the business to the checklist. And I run it against the 80 items. And what happens the first time when I run it, there might be seven or eight questions that I don’t know the answer to, which is great, which what that means is, ‘Listen dummy, go find out the answer to these eight questions first.’ Which means I have more work to do. So I go off again to find those answers. When I have those answers, I come back and run the checklist again. And any business that I look at is going to have some items on which the checklist raises red flags. But the good news is that you’re looking in front of you with all your facilities at the range of things that could possibly cause a problem.

And when you look at that list, you can also compare it to how those factors correlate with the rest of your portfolio. And at that point, kind of, you have a go, no-go point, where you can say, ‘I’m comfortable with these risk factors here. I’m comfortable with probabilities. And I’ll go ahead with it.’ Or you can say, ‘I’m just going to take a pass.’

And one of the things that came out of running the checklist was I used to run a 10×10 portfolio, which is when I’d make a bet, it was typically 10% of assets. And after I incorporated the checklist and I started to see all the red flags, I changed my allocation. So the typical allocation now at Pabrai Funds is 5%. And we’ll go as low as 2%, if we are doing a basket bet.

And once in a blue moon, we’ll go up to 10%. In fact I haven’t done a 10% investment in a long time. And so the portfolio has become more names than it used to have. But since we started running the checklists, which is about 18 months ago, so far it’s a zero error rate. And in the last 18 months, it’s probably been the most prolific period of making investments for Pabrai Funds. We made a huge number of investments, more than any other period, any other 18-month period in our history. So with more activity so far, and it’s a very short period, we have a much lower error rate.

I know in the future we will make errors. But I know those errors, the rate of errors will be much lower. And this is key. The thing is that Warren says, ‘Rule No. 1: Don’t lose money. Rule No. 2: Don’t forget rule No. 1.’ OK, so the key to investing is downside protection. The upsides will take care of themselves. But you have to make sure that your losers are few and far between. And the checklist is very central to that.

Forbes: Can you give a couple of the things that are on your 80 [item] checklist?

C2Pabrai: Oh yeah, sure. The checklist was created, looking at my mistakes and other investors’ mistakes. So for example, there’s questions like, you know, ‘Can this business be decimated by low-cost competition from China or other low-cost countries?’ That’s a checklist question. Another question is, ‘Is this a win-win business for the entire ecosystem?’ So for example, if there’s some company doing, you know, high-interest credit cards and they make a lot of money, that’s not exactly, you know, helping society. So you might pass on that. Also, a liquor company or tobacco company, those can be great businesses, but in my book, I would just pass on those. Or a gambling business, and so on.

C2So the checklist will kind of focus you more toward playing center court rather than going to the edge of the court. And there’s a whole set of questions on leverage. For example, you know, how much leverage? What are the covenants? Is it recourse or non-recourse? There’s a whole bunch of questions on management, on management comp, on the interests of management. You know, just a whole–on their historical track records and so on. So there’s questions on unions, on collective bargaining.

C2So you know, and all of these questions are not questions I created out of the blue. What I did is I looked at businesses where people had lost money. I looked at Dexter Shoes, where Warren Buffett lost money. And he lost it to low-cost Chinese competition. So that led to the question. And I looked at CORT Furniture, which was a Charlie Munger investment. And that was an investment made at the peak of the dot-com boom, where they were doing a lot of office furniture rentals. And the question was, ‘Are you looking at normalized earnings or are you looking at boom earnings?’ And so that question came from there. So the checklist questions, I think, are very robust, because they’re based on real-world arrows people have taken in the back.”

OLYMPUS DIGITAL CAMERADisclosure: I wrote this article myself, and it expresses my own opinions. I am not receiving compensation for it. I have no business relationship with any company or individual mentioned in this article. I have no positions in any stocks mentioned.

Checklist: A List of Things…

Here comes some thoughts about checklists, and why it might be a good thing to use one, but also on how to build your own and from were to get your inspiration.

The Cambridge Dictionaries Online defines a checklist as “a list of things that you must think about, or that you must remember to do.” 

Actually, you don’t need to look up the word “checklist” in a dictionary to know its meaning. But I think that it’s always good to get a simple definition to read through and just look at.

Anyway, I didn’t really think a lot about checklists until I read the 1977 Letter to Shareholders written by Warren Buffett. In it Warren simply lays out the four things that Charlie and he look for when evaluating a business.

We select our marketable equity securities in much the same 
way we would evaluate a business for acquisition in its entirety.  
We want the business to be (1) one that we can understand, (2) 
with favorable long-term prospects, (3) operated by honest and 
competent people, and (4) available at a very attractive price.

These four criteria captures pretty much everything one should consider when evaluating a business. They may not look like much at first, but there is a lot of sublevels in each of them.

What then is the best way to learn about the four points and their content? A good start is to read all letters to shareholders that’s been published so far. You’ll find them at Berkshire Hathaway’s homepage. All of them are great reads and I recommend everyone to read them. 

These are some of the best business texts ever written for someone who wants to improve knowledge about business analysis and investing. And they’re available for free! 

In my next post I will write about two books I have read about checklists. Until then, get inspired by Mohnish Pabrai’s presentation The ChecklistMohnish Pabrai - The Checklist

Also, feel free to share your insights regarding checklist. Do you use an investment checklist? If you do, how do you use it and how did you compile it?