BECG theblog

Measurement Matters (Part 1 of Many)

Have you ever participated in a survey or research study? It's hard to imagine that anyone would not have. Survey results cover the pages of most newspapers, Web sites, and just about any other medium. Most of them make me sputter in horror. Why? No, they're not scary. They are merely scarily bad - meaning that there are fatal flaws in the design that render the results DOA. Alternatively, the data are "analyzed" by people who don't know the difference between good and bad quality data, or do not realize that there is more to data analysis than knowing how to do the math.

Here's an example.

In December 2011, a study was published in the proceedings of the National Academy of Sciences, purporting to show that the sound made by valuable, old Stradivarius and Guarneri violins was no better than that produced by brand-new, stainless steel violins. Articles subsequently appeared first The New York Times ("In Classic vs. Modern Violins, Beauty Is in Ear of the Beholder") then in Ars Technica ("Old, million-dollar violins don't play better than the new models"). The story was picked up and featured everywhere from NPR to The DW Academie. 

There was just one small problem: the study was so badly done, and the math used so inappropriate for the tiny, non-random sample (21 people attending one conference), that there was no basis for drawing any conclusion(s) at all.

As I wrote to Ars Technica at the time:

.... the Big Mistake was to publish This Article under This Title (Old, million-dollar violins don't play better than the new models) without knowing anything about research design or inferential vs descriptive statistics. As a result, Ars Technica has shared with the public a piece of research that fails utterly to support the claim made in the title.

To wit:

The number of subjects in the study is: 21. This is too small a number of cases to merit computation of anything other than very simple descriptive statistics (e.g., "The number of people in the group is; their ages are," etc.). You may note that the authors made no attempt to generalize beyond these 21 individuals, and neither should you.

This is not to suggest that the investigator-authors did a fabulous job, either. For instance, I see no mention in the technical abstract (which I did download from the publisher's site) of 'inter-rater reliability,' which is a necessary computation in any research that asks people to rate and/or compare anything to anything else. Unless one has established, empirically, a reason to believe that, for each subject, there is consistency in ratings from one episode to another, one cannot tell whether or not one is receiving reliable reports - period. Moreover, the elaborate reporting of statistics like effect size, F, and p value may convey the appearance of probity. However, that appearance is highly misleading since the fundamental requirements for acceptable study design, implementation, and sample size have not been met.

All in all, this "research" does not merit coverage in your fine publication, and it certainly does not merit the conclusions drawn either by this story's author or by the original authors. 
 

In short, this kind of research is a classic case of fools rush in where angels fear to tread. There's a lot of rushing going on these days, so be careful to watch your step!

                                                                                               Stradivarius Violin

Behavioral Science, Empiricism, and Ingenuity

This morning's New York Times features a terrific article, Social Change's Age of Englightenment by David Bornstein. In the post, Bornstein shows how the growing realization that "we are not econs," combined with rigorous, disciplined collection/analysis/use of empirical data, can help us reframe intractible dilemmas in ways that allow breakthrough change.

Certainly, this matches my experience. I adopted the approach Bornstein describes when I worked in banking research. At some random point, it occurred to me that what we call 'money' is not a thing, but a symbol. Paper money, coins, and/or data flowing through an ACH system: none have inherent value. The value is found in the meaning we give them. They are no more or less than buckets of meaning. "But what is that meaning?" I asked myself. Although the field had no name at the time (late 1980's), it turned out that I had stumbled upon behavioral economics.

Bornstein does a terrific job of articulating what the approach entails, the tools we use, and the value that it can deliver. I'm grateful to him for his cogent description and compelling argument.

Into the Looking Glass: Trust, Virtual Trust, and Impostership

In honor of my upcoming attendance at the ISOC Philadelphia's session on Trust, I am reposting something I wrote on my previous blog, concerning trust and the Web (hint: trust is a big mistake if the counterparty is not trustworthy. My conclusion: to be worthy of another's trust is far more important).

Once, back in the ancient past (2002), AKMA commented on Brent Glass's question as to "why the companies that vociferously demand that we trust them are themselves the least trustworthy." My response follows:

Because they are the only ones stupid enough (in the world of emotional intelligence) to believe that trust is a commodity that can be bought and sold; that trustworthiness is a pose to strike in the service of competitive advantage rather than a stance in life. In this construction, trust is a product feature that encourages customers to do what you want them to do in spite of their better judgement or economic self-interest. Paradoxically, in urging us to trust them, they reveal an utter absence of understanding of the concept. This, in turn, lets us know they are entirely untrustworthy.

Three years later, amidst all the speeches, the debates, the attempts at definition, and the caterwauling about the cost of establishing trust online, I still believe this to be true. Indeed, it appears that the most vital question of all has received little or no attention. That question is: "Are you trustworthy?"

If trust is the willingness to place oneself in a position where one is to some extent vulnerable to another (whether economically, emotionally, or physically),trustworthiness is the willingness, ability, and commitment to honor that trust, through a combination of good will, competence, and follow-through.

In any successful trusting relationship, be it commercial or personal, the risk taken by the trustor is necessarily matched by the will and ability of the trusteeto deliver on the promise(s) they have made. Also inherent in any trusting relationship are an experience-based, mutual assumption of benevolence and a congruence of expectations. When these characteristics are present, risk is minimized and both parties benefit. The relationship in question could be either personal or professional, financial or emotional; the risk could be either large or small--but the equation does not vary.

The nature of the relationship notwithstanding, the 'contract' is that words and deeds will match. This is why, for instance, dramatic and sweeping commercial promises, when accompanied by lengthy, unreadably small caveats written in undecipherable legalese, engender suspicion....as well they should. Ultimately, exploitative aims reveal themselves in action: PR no matter how brilliant, can function only as a temporary camouflage. People can tell, instinctively, when the signs of impostership are there. Thus, David Isenberg recently asked: "Do you distrust "trusted computing" as much as I do?

Building trust is expensive. There is a good reason for this. Trust is precious. When trust is violated, the trustor typically pays a far higher price than does their MMoN (misfeasant, malfeasant, or nonfeasant) counterpart. Thus, if you or your organization are concerned with building others' trust but are not focusing with equal intensity on making sure you are trust-worthy, you are worrying about the wrong thing. This, in turn, suggests they are right to be suspicious.

technorati tags: virtual trust, trust online
permalink: posted by saraw at 9/02/2005 02:12:00 AM 
comments (2)

 

Tempus Fugit

Time Flies when you're having fun, or so they say.

As the global economy tanks and the fun fund implodes, does this imply that time slows down? Me thinks not. 

This morning, Ben Bernanke spoke to Congress's Joint Economic Committee concerning the outlook for the US economy. The unsurprising but discouraging conclusion: it's not good. 

I don't know about you, but I am not just watching the news about our lousy economy; I am experiencing it. I do not know a soul who isn't. From this vantage point it seems to me that our economic chieftains, governmental and corporate, as well as the media pundits, have got the thing all wrong.

Consumers not spending is not the problem. Consumers aren't spending because they do not have the money. Small businesses, which according to the Economic Census, make up 99% or more of all businesses,¹ cannot get credit from The Banks.

Technology illiteracy is not the problem. I can't tell you how many knowledge workers I know who can't find work and have either lost, or are on the verge of, losing everything. It's not just young people: it's everyone. People over fifty, who conscientiously undertook the slow, careful climb to greater job satisfaction, professional status, and income — suddenly find themselves out on the street, but with an expense structure that reflects their investment in their career(s). Cut back?! You can't lay off your children. It's great to fund infrastructure projects — heaven knows we need to upgrade — but that is not enough.

The problem is far bigger, and - I dare say — more sinister. Two years ago, I briefly held a consulting position at NTIA - the National Telecommunications and Information Agency. Setting aside the bizarre management climate of the place (my role was not one that required a security clearance, but I was prohibited from even speaking to any other employee at any level), I witnessed the following:

The Stimulus program (at least the part administered by Commerce) was a gigantic give-away from the gargantuan, wealthy US government (funded by taxpayers but controlled by monied interests), to its gargantuan, wealthy confrères in the ruling class. Inside deals with huge government contractors, state governments, and corporate astroturf institutions were the norm — so much so that they were assumed to be the only option. While I was not permitted to speak with anyone, lobbyists could sail in and corral the time of anyone at any level. It was highly educational, in the sense that all learning comes with pain.

It's no wonder that even the paltry "Stimulus" effort didn't work: the money merely exchanged hands within that small, coddled group otherwise known as "the 1%." To be clear, I completely, categorically reject everything the "Tea Party" stands for. That said, if the leaders of government and industry weren't busy adding (kevlar) layers to the cocoon of money and power that already encases them, they would have a clue about the nature and magnitude of the needed changes.

I don't want government to go away. I simply want it to exit the Matrix. 

¹ According to the Small Business Administration, small businesses comprise over 97% of all US businesses. The SBA's definition of small business is most closely calibrated to manufacturing — businesses with 500 - 1,000 or fewer employees fall within its definition. However, if one adds to that businesses with 20 or fewer employees and Non-employers - businesses where the owner is the sole employee as does the US Economic Census, the percentage climbs even higher. Who, then, are the "job creators?" They are the small enterprises who are being starved by the large ones.

Rhetorical Question

If you set out to write the "killer app," aren't you doomed from the start? Aren't most killer apps accidents that somebody had the imagination to put to as-yet-unforeseen uses?

The emperor has no clothes, but the empress is seriously over-dressed.

This phrase came to me in the shower (a veritable temple of insight) as I pondered the question of why I am having so much trouble writing about the relationship between connectivity and the economic meltdown of the day. I realized I have so much to say (and I candidly admit that it is of substance, as embarrassing as I find this admission to be) that I don't know where to start. Or how to start. Somehow all my drafts have an 'approachability quotient' in the same (exceptionally low) range as do those imposing stone edifices on Mount Rushmore.