Two quite different ideas about probability have coexisted for more than a century. These probability approaches, which differ in several important ways, are as follows:
The frequentist view defines probability of some event in terms of the relative frequency with which the event tends to occur.
The Bayesian view defines probability in more subjective terms — as a measure of the strength of your belief regarding the true situation. (A less subjective formulation of Bayesian philosophy still assigns probabilities to the “population parameters” that define the true situation.)
Most statistical problems can be solved using either frequentist or Bayesian techniques, but the frequentist approach is much more widely used, and most of the statistical techniques in use today are based on the frequentist view of probability. This predominance is because the frequentist approach usually involves simpler calculations. Only recently have sufficiently powerful computers and sufficiently sophisticated software become available to allow real-world problems to be tackled within the Bayesian framework.
Here's how the frequentist and Bayesian views differ significantly:
Ways of reasoning: These two philosophies of probability apply different directions of reasoning. Frequentists think deductively: “If the true population looks like this, then my sample might look like this.” Bayesians think inductively: “My sample came out like this, so the true situation might be this.”
Ideas about what’s random: The two philosophies have different views of what is random. To the frequentist, the population parameters are fixed (but unknown), and the observed data is random, with sampling distributions that give the probabilities of observing various outcomes based on the values of certain population parameters. But in the Bayesian view, the observed data is fixed (after all, we know what we saw); it’s the population parameters that are random and have probability distribution functions associated with them based on the observed outcomes.
Terminology: Frequentists and Bayesians use different terminology. Frequentists never talk about the probability that a statement is true or the probability that the true value lies within some interval. And Bayesians never use terms like p value, significant, null hypothesis, or confidence interval, which sound so familiar to those statisticians brought up in the frequentist tradition; instead, they use strange terms like prior probability, noninformative priors, and credible intervals.
Usable information: Frequentists typically think of data from each experiment as a self-contained bundle of information, and they draw conclusions strictly from what’s in that set of data. Bayesians have a broader view of “usable information” — they typically start with some prior probabilities (preexisting beliefs about what the truth might be, perhaps based on previous experiments) and then blend in the results of their latest experiment to revise those probabilities (that is, to update their spread of belief about the true situation). These revised probabilities may become the prior probabilities in the analysis of their next experiment.