Sampling Bias

One of the more interesting research concepts that's been on my mind lately is sampling bias, or put simply, what happens when your sample doesn't represent your population?

This has been coming up a lot for me because of a few things:

• A good friend of mine that I have regular political conversations with has had vastly different experiences with the same group of people that I have had. I've found most liberals, feminists, and LGBT's to be rather moderate, rational (though not always informed), and highly capable of intellectual discussion. His experiences have been much more extreme.
• The whole #GamerGate thing (and really any heated online discussion) paints a lot of rather... 'colorful' pictures of both sides of the discussion, yet when I actually engage someone 1-on-1, I find things tend to (not always) be more moderate, rational, and level-headed.
• Since my job involves a degree of research design, it's a regular question I face professionally. 'who can we sample, and how are they different from those we can't?'

It's the first point that's really driven it home for me, as I'm more personally vested with my friend than I am with #GamerGate, or my job.

Side note: If you're not familiar with #GamerGate, Three of my more recent posts touch on different facets of the issue, here, here and here.

I'm really starting to wonder exactly how much of our understandings, impressions, assumptions, and whatnot is framed by these inconsistencies of experiences. Naturally each of us is limited in our personal sampling frame to our personal experiences, but as any good researcher would tell you, that's (at best) a step above a convenience sample, and is unlikely to be representative of the group as a whole. Yet we still form our opinions based on this bias sample.

So, I must wonder, if we're forming opinions based on our personal experiences as an override to something less bias, then how can anyone really trust their understandings of any group at all? This gets back to the idea of media bias, and the Orwellian nightmare. If the state controls what information we have, then that's creating a bias on our understanding of our environment and ability to come to broad-based insights.

Yet aren't we self-imposing the same kind of Orwellian nightmare on ourselves by functioning on personal experience? If our own personal experiences are as influential to how we perceive the world around us, and those experiences are based on bias sampling, then how can we help but form an improper concept of our world?

Going back to the real-world examples: My friend has had very negative experiences with individuals of a liberal persuasion (thankfully excluding myself), to the point where his perception has shifted from the belief that there's different objectives between the groups, to the suspicion that liberals pose a real threat to him and those with his perspectives.

My experiences have been radically different, with liberals and conservatives holding space for different views, debating each other, sometimes getting heated, but not being hateful or threatening (overall, there's always individual exceptions).

So, since both of our samples are quite bias (Mine mostly from the San Francisco, Boston, D.C. and Denver areas, and his mostly from South Carolina), where can an accurate picture of liberals be found? I'm not entirely sure, I can't say that my perspective is any more legitimate than his as I have no evidence, given that I have a rather limited knowledge of the full scope of the sampling biases involved.

Don't get me wrong, I *like* my perspective more, not just because it paints the group I associate with in a more positive light, but also because it's more positive in general. That's personal bias though, and doesn't make for objective generalizations.

Now, it's easy to take this concept to its logical conclusion and state that there's no way to get to an objective generalization of our perspectives. I would argue this stops short of a full analysis though, in that the goal is not necessarily to reach it, but to reach for it. It's similar logic used in the #GamerGate debate to support the idea that objective journalism is important (link here) as in doing so improves who we are and what we do. Complete objective generalizations is the perfect goal, reality gets in the way, and we still end up better off for the attempt than if we throw up our arms in nihilistic futility.

That may be me just being overly Nietzschian though (-:

Anyway, it's a food for thought.

Best thinking folks.

- Jason