Have you ever participated in a survey or research study? It's hard to imagine that anyone would not have. Survey results cover the pages of most newspapers, Web sites, and just about any other medium. Most of them make me sputter in horror. Why? No, they're not scary. They are merely scarily bad - meaning that there are fatal flaws in the design that render the results DOA. Alternatively, the data are "analyzed" by people who don't know the difference between good and bad quality data, or do not realize that there is more to data analysis than knowing how to do the math.
There was just one small problem: the study was so badly done, and the math used so inappropriate for the tiny, non-random sample (21 people attending one conference), that there was no basis for drawing any conclusion(s) at all.
As I wrote to Ars Technica at the time:
.... the Big Mistake was to publish This Article under This Title (Old, million-dollar violins don't play better than the new models) without knowing anything about research design or inferential vs descriptive statistics. As a result, Ars Technica has shared with the public a piece of research that fails utterly to support the claim made in the title.
The number of subjects in the study is: 21. This is too small a number of cases to merit computation of anything other than very simple descriptive statistics (e.g., "The number of people in the group is; their ages are," etc.). You may note that the authors made no attempt to generalize beyond these 21 individuals, and neither should you.
This is not to suggest that the investigator-authors did a fabulous job, either. For instance, I see no mention in the technical abstract (which I did download from the publisher's site) of 'inter-rater reliability,' which is a necessary computation in any research that asks people to rate and/or compare anything to anything else. Unless one has established, empirically, a reason to believe that, for each subject, there is consistency in ratings from one episode to another, one cannot tell whether or not one is receiving reliable reports - period. Moreover, the elaborate reporting of statistics like effect size, F, and p value may convey the appearance of probity. However, that appearance is highly misleading since the fundamental requirements for acceptable study design, implementation, and sample size have not been met.
All in all, this "research" does not merit coverage in your fine publication, and it certainly does not merit the conclusions drawn either by this story's author or by the original authors.
In short, this kind of research is a classic case of fools rush in where angels fear to tread. There's a lot of rushing going on these days, so be careful to watch your step!
Watching tonight's news about the bizarre, scary posturing of North Korea's latest monarchical communist, I was struck by two enduring truths:
The economists in Washington and the defense-information-industrial complex may wish to take note: this is not just about North Korea.
This morning's New York Times features a terrific article, Social Change's Age of Englightenment by David Bornstein. In the post, Bornstein shows how the growing realization that "we are not econs," combined with rigorous, disciplined collection/analysis/use of empirical data, can help us reframe intractible dilemmas in ways that allow breakthrough change.
Certainly, this matches my experience. I adopted the approach Bornstein describes when I worked in banking research. At some random point, it occurred to me that what we call 'money' is not a thing, but a symbol. Paper money, coins, and/or data flowing through an ACH system: none have inherent value. The value is found in the meaning we give them. They are no more or less than buckets of meaning. "But what is that meaning?" I asked myself. Although the field had no name at the time (late 1980's), it turned out that I had stumbled upon behavioral economics.
Bornstein does a terrific job of articulating what the approach entails, the tools we use, and the value that it can deliver. I'm grateful to him for his cogent description and compelling argument.
Le jour de la Bastille: the anniversary of the storming of the Bastille, which dealt a death blow to feudalism in France (& to quite a few individuals as well). Yet, like a virus, the feudal spirit seems to be coming back in the form of "legal" economic bondage.
Let's hope this time the revolution is a velvet one and that a Reign of Terror (la Terreur) does not arrive on its heels. People want to be connected, and they want to be genuinely free. It is always time for liberty, equality, and mutuality.
Vive la France!
Words of wisdom, from Chuck Schumer:
There are two tests in life, more important than any other test. On Monday morning, when you wake up, do you feel in the pit of your stomach you can’t wait to go to work? And when you’re ready to go home Friday afternoon, do you say, ‘I can’t wait to go home’? If you can say yes to both those tests, God has been good to you, don’t complain.
Watching the State of the Union address Tuesday night, I sometimes felt the urge to shout at the TV set. Although the President seems to want to do right by people, his repeated mentions of "responsible homeowners" told me that he does not understand borrowers' motivations, nor does he 'get' what underlies most late- or non-payment of debt. The reality is: If people do not make their mortgage payments, it is rarely because they are irresponsible.'
Obama is not alone in his lack of understanding. Both Robert Manning (2000) and David Graeber (2010) have noted that, in our society, to be in debt is to be judged morally wanting - even if that judgment has no basis in reality. Take a look at the facts - the main causes of consumer bankruptcy are:
Either one of these catastrophes would be bad enough to bring a famly down, but they are prone to co-occur since in the US, loss of employment typically translates into loss of health insurance. That these 'drive' non-payment, particularly for homeowners, was documented in As We Forgive Our Debtors (Sullivan, Warren & Westbrook, 2000) and has been corroborated by numerous rigorously-conducted studies by economists (see citations in the embedded links).
Even a decade ago, the harbingers of today's debt crisis were lurking in plain view. On a systemic level, things are now much worse. Yet, there are many answers to be found. They lie neglected because so many people can't get beyond their counter-factual moral judgments. Ironically, it is the moral judgment that causes the harm, and it is they who would punish the suffering that carry the disease.
It is the 148th anniversary of the date upon which President Abraham Lincoln delivered the Gettysburg Address:*
Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in liberty, and dedicated to the proposition that all men are created equal.
Now we are engaged in a great civil war, testing whether that nation, or any nation, so conceived and so dedicated, can long endure. We are met on a great battle-field of that war. We have come to dedicate a portion of that field, as a final resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this.
But, in a larger sense, we can not dedicate, we can not consecrate, we can not hallow this ground. The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to add or detract. The world will little note, nor long remember what we say here, but it can never forget what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us—that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion—that we here highly resolve that these dead shall not have died in vain—that this nation shall have a new birth of freedom—and that government of the people, by the people, for the people, shall not perish from the earth.
To me, this event - and these words - are sacred. Think about it:
" we here highly resolve that these dead shall not have died in vain—that this nation shall have a new birth of freedom—and that government of the people, by the people, for the people, shall not perish from the earth.
Never have they been more relevant.
In honor of my upcoming attendance at the ISOC Philadelphia's session on Trust, I am reposting something I wrote on my previous blog, concerning trust and the Web (hint: trust is a big mistake if the counterparty is not trustworthy. My conclusion: to be worthy of another's trust is far more important).
Time Flies when you're having fun, or so they say.
As the global economy tanks and the fun fund implodes, does this imply that time slows down? Me thinks not.
This morning, Ben Bernanke spoke to Congress's Joint Economic Committee concerning the outlook for the US economy. The unsurprising but discouraging conclusion: it's not good.
I don't know about you, but I am not just watching the news about our lousy economy; I am experiencing it. I do not know a soul who isn't. From this vantage point it seems to me that our economic chieftains, governmental and corporate, as well as the media pundits, have got the thing all wrong.
Consumers not spending is not the problem. Consumers aren't spending because they do not have the money. Small businesses, which according to the Economic Census, make up 99% or more of all businesses,1 cannot get credit from The Banks.
Technology illiteracy is not the problem. I can't tell you how many knowledge workers I know who can't find work and have either lost, or are on the verge of, losing everything. It's not just young people: it's everyone. People over fifty, who conscientiously undertook the slow, careful climb to greater job satisfaction, professional status, and income — suddenly find themselves out on the street, but with an expense structure that reflects their investment in their career(s). Cut back?! You can't lay off your children. It's great to fund infrastructure projects — heaven knows we need to upgrade — but that is not enough.
The problem is far bigger, and - I dare say — more sinister. Two years ago, I briefly held a consulting position at NTIA - the National Telecommunications and Information Agency. Setting aside the bizarre management climate of the place (my role was not one that required a security clearance, but I was prohibited from even speaking to any other employee at any level), I witnessed the following:
The Stimulus program (at least the part administered by Commerce) was a gigantic give-away from the gargantuan, wealthy US government (funded by taxpayers but controlled by monied interests), to its gargantuan, wealthy confrères in the ruling class. Inside deals with huge government contractors, state governments, and corporate astroturf institutions were the norm — so much so that they were assumed to be the only option. While I was not permitted to speak with anyone, lobbyists could sail in and corral the time of anyone at any level. It was highly educational, in the sense that all learning comes with pain.
It's no wonder that even the paltry "Stimulus" effort didn't work: the money merely exchanged hands within that small, coddled group otherwise known as "the 1%." To be clear, I completely, categorically reject everything the "Tea Party" stands for. That said, if the leaders of government and industry weren't busy adding (kevlar) layers to the cocoon of money and power that already encases them, they would have a clue about the nature and magnitude of the needed changes.
I don't want government to go away. I simply want it to exit the Matrix.
1 According to the Small Business Administration, small businesses comprise over 97% of all US businesses. The SBA's definition of small business is most closely calibrated to manufacturing — businesses with 500 - 1,000 or fewer employees fall within its definition. However, if one adds to that businesses with 20 or fewer employees and Non-employers - businesses where the owner is the sole employee as does the US Economic Census, the percentage climbs even higher. Who, then, are the "job creators?" They are the small enterprises who are being starved by the large ones.
As Frank Rich points out, censorship is alive and well in the United States. In the end, it matters not one whit whether the news is distorted and truncated by a frightened despot, or by "corporate gatekeepers." Censorship is censorship: it is anti-democratic, it harms economies, and it stunts the human spirit.
Unable to watch Al Jazeera English, and ravenous for comprehensive and sophisticated 24/7 television coverage of the Middle East otherwise unavailable on television, millions of Americans last week tracked down the network’s Internet stream on their computers. Such was the work-around required by the censorship practiced by America’s corporate gatekeepers. You’d almost think these news-starved Americans were Iron Curtain citizens clandestinely trying to pull in the jammed Voice of America signal in the 1950s — or Egyptians desperately seeking Al Jazeera after Mubarak disrupted its signal last week.
This spectacular blooper, shown by Fox News on July 27, 2009 is a Rorschach Test. It tells us plenty about the broadcaster's world view -- but it certainly does not tell us about the world.
The "value chain" is brought to fruition when global media conglomerates - newly permitted to extend their reach beyond national, state, and local borders - master the business of sponsoring political candidates. I can't possibly do justice to Paul Krugman's brilliant Fear and Favor, but I do hope you'll read it.
IQ, 'the score,' was never designed to measure intelligence. French psychologists, most notably Alfred Binet of Stanford-Binet fame, developed the test to assess the current level of functioning among people who were struggling with their schoolwork. Americans took this schema and twisted it beyond recognition, turning it into what it was never intended to be: a measure of 'intelligence'/'aptitude.' If nothing else, it is a superb measure of our national obsession with comparing ourselves to one another--the goal being to determine whether the other is "above" or "beneath" us. This is in part due to our historic legacy of having rejected explicit systems of social stratification based on kinship, such as a caste system (India is not the only place that has one) or monarchy.
The fact is that nobody knows what constitutes intelligence. It is one of the great mysteries of life. It is context-dependent, synchronicity-dependent, group-dependent, time and place-dependent and a bunch of other things that we can't possibly grasp because we are limited by our inability to accurately perceive that which we do not understand. The same is true of aptitude. We can grope around and hope we learn something, and we probably will. However, the certainty that we know how to assess the intelligence (read: worth) of our fellow humans or to predict how they will perform (or even to pretend we know what constitutes performance) is one of the surest signs of stupidity I can think of. BTW, everything I've said about intelligence applies to stupidity, too.
This is a wonderful TED talk by Steven Johnson, prolific author and inventive, provocative thinker.
To Johnson's talk, I would add only this: if you want to find the soul of innovation, you need to look outside the academy as well.
Per Plato, necessity is the mother of invention (1). (Frank Zappa and The Mothers of Invention were nothing if not innovative. Sample lyric: "Movin' to Montana soon. Gonna be a Dental Floss tycoon")
I've done several of my trademark academic, rigorously researched-type papers on this topic, and Johnson's comments resonate very much with what I learned. I also learned a few things that are not covered in Johnson's TED talk but are interesting, counter-intuitive, and useful.
So much baseless mythology surrounds the acts of innovation, invention, and their commercial counterpart--entrepreneurship. The first among these is that it is all about a brilliant individual having a personal epiphany. This is almost never true on both counts: it is rarely an individual phenomenon (although individuals, out of greed -- financial or narcissistic --or simply out of lack of insight, often do take credit for breakthroughs that have many parents). Also, breakthroughs are rarely brief, blinding and/or sudden. Sometimes thay are only recognized as such only in retrospect.
I titled one of the papers I wrote: The Opportunity Beyond the Obvious. This is because so often, breakthroughs (and their commercial counterparts) come via simple observation, often made by people who don't fit well into the consensus reality of the day (2). Benefiting from limited-to-no access to standing within conventional society, they thus enact the above-named principle (viz, necessity is the mother of invention), relying on instinct and experiential learning rather than on formal education (3).
If you set out to write the "killer app," aren't you doomed from the start? Aren't most killer apps accidents that somebody had the imagination to put to as-yet-unforeseen uses?
Tags: behavioral economics, creativity, economic psychology, innovation, invention, killer app, psychology, technology
I REALLY wish the people who hold the power to facilitate innovation writ large (as referenced in the article) knew some of these things. It would definitely make for greater effectiveness in "erasing our innovation deficit."
Tags: behavioral economics consulting group, conformity, connectivity, Eric Schmidt, geography, how-to, innovation, Locke, paul budde, sara wedeman, stereotyping, technology, trans sector thinking, tweakage
I was just reading an article about the Philadelphia city government buying the infrastructure for the now-defunct Philly Wifi network, when it occurred to me: we need a public option for Internet access, too. We the People have been thwarted at every turn in trying to bring fairness, equity, and transparency to the bloated, out-of-control telecommunications industry. Most of the time, they win. Among the major causes: legally or illegally, through campaign contributions, lobbying, and who-knows-what-else, our legislators and the political appointees they feed are in the employ of the dominant providers.
Sound familiar? All of a sudden, it hit me. This is just like the health care morass. Just change the names to those of the top health care insurance companies. The pattern is virtually identical. We are in a death-grip, held by corporate bodies bigger, fatter, and more powerful than ourselves. What started out as a simple, fair equation:
X builds something of value, Y buys it for (X's time+costs+a reasonable profit), both walk away happier and better off.
has been gutted to the point where Y exists primarily to feed X's limitless greed. Since Y (which stands for "you," by the way) depends on X for its livelihood, and X has guarded its flanks against all competitors with armies of lawyers, Y has no choice but to beg, whine, fuss, or roll over and play dead.
I don't like it a bit. We can do better.
We've done it in the past. The question is: what will it take to do better now? Nota bene: "now" means now. It does not mean "when and if someone lets us do it" or "wait until (fill in the blanks)". Lots of good, smart, hard-working people have been trying to bring about positive change. What can we learn from their successes and failures? What novel tools and strategies are sitting right under our noses, just waiting to be put to work?
But I digress. To reiterate: we need a public option for Internet connectivity. We need it yesterday. Granted, yesterday might be hard to achieve... but today's not over yet.
This phrase came to me in the shower (a veritable temple of insight) as I pondered the question of why I am having so much trouble writing about the relationship between connectivity and the economic meltdown of the day. I realized I have so much to say (and I candidly admit that it is of substance, as embarrassing as I find this admission to be) that I don't know where to start. Or how to start. Somehow all my drafts have an 'approachability quotient' in the same (exceptionally low) range as do those imposing stone edifices on Mount Rushmore.
This looks to me like a perspective dislocation emergency. Get help (AKA a grip), and (please) do so ASAP!
* Doubting that it's monolithic, moi.
This is a beginning stab at addressing my earlier question about what we can learn from the current economic crisis. It will probably take several posts to lay out my hypotheses and opinions, so please indulge me, and feel free to chime in (I mean it. Having a conversation with yourself is not what the web is made for and besides, I've got too many of those going on in my head already).
To begin, I want to pick up on two of the items I mentioned in that last post:
This is extremely common in all areas of life, but particularly so in the financial arena. Often, it evolved into a group delusion, akin to the psychiatric disorder, folie á plusieurs (madness of many). Nobody bothered to pay heed the few who checked the facts and thought for themselves. That which did not fit with consensus reality went unheard and unseen, because with delusional systems, inconsistencies with one's world view literally 'don't compute.' It's odd how history repeats itself. This is something Daniel Defoe (1660 - 1731) wrote about the South Sea Bubble of 1720:
Some in clandestine companies combine;
Erect new stocks to trade beyond the line;
With air and empty names beguile the town,
And raise new credits first, then cry 'em down;
Divide the empty nothing into shares,
And set the crowd together by the ears.
II. Ignorance of the assumptions and limitations that built into the mathematical models upon which high-stakes financial bets were placed.
Apparently, many of the problems that caused financial systems to implode were flaws in the financial models used to predict outcomes. These models include but are in no way limited to so-called 'neural networks' (Incidentally, 'neural networks' do not replicate the brain's computational actions in any known way. They are simply pared down, highly abstracted, regression equations).* The term "neural networks" smacks of marketeering: taking a form of statistical analysis that is in the public domain and can be done by anyone with the appropriate background, giving it a catchy new name, and making it sound far more special than it actually is.
Regression equations predict the likelihood of an outcome or outcomes, based on the input of a series of input variables (in other words, data about various characteristics that are hypothesized to be good predictors of that outcome. The highest possible adjusted R2 (this is also called the 'regression coefficient), indicating a 100% likelihood that the array of independent or input variables will produce a particular outcome, simply does not occur in nature. The reason is common sense: any outcome has multiple predictors, some of which are known, some of which are knowable, and some of which are neither. Moreover, every predictive model by definition comes along with an error term (this is a statistic that shows the average amount of error that one can expect) and a confidence interval (this shows the band within which 68.2%, 95.4%, 99.6, 99.8%, 99.9%, and on and on, of the true scores are likely to fall. Note that one can never reach 100%. It is an asymptotic curve, that is a curvilinear line that approaches nearer to the 'destination' (in this case, 100% of the scores falling within the confidence interval) but, though infinitely extended, would never meet it.
In fact, one of the central tenets of mathematical prediction is that no equation, no matter how perfect it is, can ever predict an individual outcome with 100% certainty. Moreover, there are several critical assumptions that must be met for the equation to be valid (regardless of whether it is statistically significant or not). An unique feature of the current situation is that, due to advances in computer and communications technology, it is now possible to run equations on truly massive data sets. This enables the mathematician to achieve higher and higher levels of statistical significance and power (meaning that the likelihood of getting a result that is way off by chance alone is greatly reduced). However, as the data sets get bigger, so does the likelihood of the highly improbable occurring, as Nassim Nicholas Taleb notes in his book, The Black Swan: the Impact of the Highly Improbable. Well, guess what? They did. In a sense, the financial community, in the throes of its folie á plusieurs, failed to take into account that their seemingly brilliant decisions were made on a foundation of infauxmation, that is something masquerading as highly credible information, but is distorted, inaccurate, presented without necessary caveats, or just plain wrong.
In Charles L.L.D. MacKay's 1820 book, Memoirs of Extraordinary Popular Delusions and the Madness of Crowds (worth reading based on the flamboyantly weird title alone, but also worth reading for its content), the author relays the following story:
An enthusiastic philosopher, of whose name we are not informed, had constructed a very satisfactory theory on some subject or other, and was not a little proud of it. "But the facts, my dear fellow," said his friend, "the facts do not agree with your theory."—"Don't they?" replied the philosopher, shrugging his shoulders, "then, tant pis pour les faits;"—so much the worse for the facts!
In short, financiers, government officials, and consumers in the throes of folie á plusieurs and 'armed' with infauxmation -- both amplified by speed, volume, and computing power -- constitute a marriage made in Hades.
Remember, this is just a beginning. There is much more to say and to discuss. For example, per George Santayana ("Those who forget the past are condemned to repeat it.”), there is the question of forgetting and of how we manage to get ourselves into these binds over and over again without, it seems, learning a thing. Another topic is the impact of elected (and selected) officials -- especially the creeping devastation that results when ideology dominates governance, crowding out the rule of law.
Oh yes, there is much more--so stay tuned!
* Statisticians, please forgive any simplifications I have made in the interest of increasing the comprehensibility of the concept's description.
These are just two of the quirks of the human psyche that I see expressing themselves in the current economic meltdown.
Others include: mistaking wishful thinking for reality, ignorance of the assumptions and limitations that are built into the mathematical models upon which high stakes financial were based, narcissism, and more.
My aim is neither to be cynical nor to depress you (or myself). There are good lessons here. What are they? As they say in Jamaica, 'soon come!'
* In the interest of transparency, I will candidly admit that I am a person.
I've been largely silent, consumed by the news of the day, but I have decided to break my silence, realizing that it was the sheer volume of my thoughts that was preventing me from speaking.
So often, I hear or read that computers are the problem. Our relatively newfound ability to locate and process mass quantities of data somehow 'caused' the current financial crisis. What is odd and funny to me (not in the ha-ha sense) is that the machines are not really the problem. It's that those using them -- by omission, commission, or both -- are unknowingly wielding tools about which they know so little. (I refer to the mathematical formulae, the hardware and the software necessary to process mass quantities of data, which comprise a key element of the deck of cards collapsing around us).
Please forgive what may seem like an oversimplification, but statistical analyses basically boil down to two types of methods:
Both of these, are inferential statistics. That is, they are statistics that result from calculations on a sample; if the results are dramatic enough that they are unlikely to be due to chance alone, one can infer that the result will hold true for the population as a whole, within a range of variation known as the confidence interval and limited by the likelihood of type 1 (false positive) or type 2 (false negative) errors. Any inferential statistic, by definition, contains something called an error term, because one is predicting something that applies to an entire population (be it human, financial, or otherwise) from a sample. Predictive models simply cannot predict a single case of anything. Note: In the case of a census, no inference is necessary because the population parameters are known.
Moreover, there are certain assumptions built into all of these models which, if violated, render the outcome invalid. My favorite is called: homocedasticity. This is a basic assumption of regression analysis and means that the variation of x scores around the regression (y) line falls within certain limits and is not scattered all over the place.
Several points about statistical analysis, inference, and prediction of outcomes:
The overarching point is that the machines are just doing the bidding of the people who run them. Any self-respecting statistician knows the above points to be true, but the statisticians have never been in charge. The people who run the show are the ones who hired the statisticians who used technology to perform calculations.
Whether the statisticians bowed to the wishes of their employers, or had themselves forgotten that no matter how perfect the strength of an association, the type one or type two error scores, or etc., no inference can ever predict any one specific outcome--or whether they were clear about the limits of prediction and were simply ignored by their employers--is immaterial. The point is it's easy to blame a machine, even for doing what you told it to. The last time I checked, machines weren't able to defend themselves.
Oh, by the way, there is no such thing as AI, unless one is referring to a certain extremely talented basketball player with a mind of his own. To think that a bunch of equations could ever mimic the complexity, the quirkiness, and the multidimensionality (not sure if this is a real word--if not, hope the meaning is clear) of the human mind -- which exists not just in the head but also in the finger, the small intestine, and etc.-- is surely delusional.
Truly, it is a modern day version of Pygmalion but with a less happy outcome. At least the original Pygmalion fell in love with the statue of a woman. After praying to Venus to bring his beloved statue to life and having his wish granted, the couple bore a son and a daughter. I think we now are seeing just how unappealing the offspring of a person and an algorithm can be.
(This, by the way, is an image created by The Internet Mapping Project, led by Bill Cheswick at Bell Labs and later at the Lumeta Corporation. It is an empirically generated map, which was constructed by repeatedly pinging all the nodes on the 'net.)
Some people say consultants "borrow your watch to tell you the time," and I suppose some do. However, the art and science of this fascinating field is found in how well you can recognize the patterns in the data you get from a vast array of sources, then take this back to the organization.........in ways that are useful, hearable, and actionable.
Here, I discuss the strange dynamic whereby companies simultaneously de-value the knowledge and skill of their own employees, while valuing the expertise of outside consultants, even as they resent them. One reason for this resentment is their fees, and I discuss the reasons why consultants appear to earn much higher daily wages when compared to employees who do similar work. The simple reason is that the 'input variables' and the way they are calculated differ, appropriately, from those used in calculating wages for full time employment.
In summary, the prevailing myth that most consultants pocket humongous fees compared to full-time employees of similar status is just that - a myth.
This is a conversation about debt, late payment and non-payment. Here, I talk about how looking at the issue differently - using an unbiased, investigative approach based in the behavioral sciences rather than jumping reflexively to more conventional (punitive) methods - led to sweeping changes in a client organization. These changes, in turn, produced dramatically favorable economic outcomes for the company.