The Genesis of Insight Generation
Professor John M. McCann
Fuqua School of Business
Insight generation was conceived in the early days of the Marketing
Workbench Laboratory, where the focus was on building prototype
expert systems for use in consumer packaged goods sales and marketing. As with any invention process, we went down a number of different paths during the 1986-1990 period. These paths involved many different expert system technologies and a multitude of marketing and sales problems. The research approach was experiential in nature: build an innovative system which solved a marketing or sales problem and learn new things via the process of building the system and presenting it to sales, marketing, and systems managers in a large number of companies. In the beginning, there were no clear visions of the underlying theme of the various innovations, other than the use of expert system technologies for producing the systems. The basic idea was to the find a problem, solve the problem, use introspection to arrive at the methodology being used to solve the problem, and then use an expert system shell to build a expert system which solved the problem.
Once a number of the systems had been built, we did a form of
meta-analysis to determine the features shared by the disparate
systems. This analysis led to a recognition that the early systems
were all writing short reports, in English, which summarized the
findings of the application of the expertise in the system. We
could see that our operative model of the user of the systems
was a reader ... someone who reads reports rather than analyze
This recognition led us to overtly apply a "writing model"
that evolved over time through discussions and statements that
are reflected in the following:
- If only the system were as easy to use as my word processor.
- If only I could simply compose a report in my word processor,
and then have an expert system come along behind me and correct
my work. If I were to write sales are up 4% this quarter,
I would like the system to check this fact and make corrections,
if necessary. For instance, it might look up the correct sales
figures in a database and then replace my sentence with sales
are down 1.3% this quarter.
- OK, I give up. I realize that we cannot write a system which
will check the facts underlying our writing as we write it. But
perhaps it does not have to be so smart. I bet that most (80-90%)
of the things which appear in most marketing reports are very
similar to paragraphs that have appeared in other marketing reports.
There are only so many things that one would want to say. Let's
try to make a system where one would write a new report by pulling
together segments of older reports. Once the user selected an
old segment, the system would re-write it for the user; for his/her
brand and time period.
- Hey, this is great. I can compose a new report from segments
of old reports.
- Although I am writing a report, I am actually doing analysis
at the same time. I tell the system what paragraph to write, it
does the required analysis and writes the report by re-writing
old material in light of the new analysis. Let's call this approach
analysis by composition to denote the fact that we are
doing analysis via the process of composing a report.
- It looks like we will need many small programs which do a
specific piece of analysis and then write a paragraph about its
findings. Let's call these programs analyzers.
- Once we have written a report, we know something new about
the brand and/or market. Wouldn't it be nice if the system could
monitor this new knowledge and let us know when it was no longer
true. Hey, we need monitors.
- OK, we have our system running and solving a problem we invented.
Let's ask our sponsoring companies to give use some real-world
problems. We can then see if our analysis by composition approach
is useful in writing prototypes which solve their problems.
- Do you know what we are doing here? We are building systems
that generate insights; let's call them insight generation
- We have been generating these insights "on demand"
by allowing the user to determine when new insights and reports
are generated. What happens if we put an agent in a loop and run
it for all items in all markets?
- I saw where there is a company using the phrase active
documents . That sort of applies to our work. But I like the
phrase dynamic documents. And their documents are not that data
oriented, as are ours. How about dynamic data documentsto
denote the fact that the document changes over time in response
- Each time we run an agent, we get new insights. And, a new
report is generated. Pretty soon, we are going to be inundated
with insights and reports. We need to be able to "know what
we know." We need an insight management system.
- Our analyzers and monitors are like Alan Kay's agents: soft
robots that live in the computer and look for things that they
know the user would be interested in. Let's call them agents.
We will have a lot of agents. What should we call the collection?
Some guys at Knowledge Systems Corporation have written a paper
about agents and agency. Let's see if their ideas are useful in
- I'm going to Japan to present these concepts and applications
to several companies. We need a bigger concept around which we
can organize all this stuff. How about the notion of a Marketing
These thoughts and snatches of conversations occurred over a four
year period. One can see how new ideas flowed from a yearning
to make the word processor the center of the analysis process
and involved the progressive/sequential process of building a
prototype system and then thinking broadly about it to find the
next step. This hyperbook takes the reader along the journey we
traveled. It presents our ideas, concepts, and prototypes so that
the reader can come away with a clear understanding of an evolutionary
path for marketing systems.