The Evolution of Marketing Systems
Professor John M. McCannFuqua School of Business
Duke University
Firms are spending millions of dollars on data and systems for use by their sales and marketing managers and professionals. There are many reasons for making these major expenditures:
These data and systems are particularly important in the consumer packaged goods industry. which was selected for study because 1) the industry is large and consumes considerable marketing resources, and 2) a data explosion has had an early impact on consumer marketing. In the late 1980s, approximately $300 billion dollars are spent on groceries annually in the USA. Large package goods manufacturers are spending $57 billion on brand marketing, with 39% on trade promotion, 28% for consumer promotion, and 33% for media advertising.[1] This spending level represents about 16% of the typical manufacturer's revenue, which is over three times the level of profits enjoyed by the typical firm. Even small importments in the efficiency and/or effectiveness of these resources could have a major impact on the profitability of the firms in the industy.
In this environment, the data and systems are used to answer specific questions, such as the following examples from Warner Lambert[2]:
An answer to one of these questions can be termed an insight , which can be defined as "a deep, thorough, or mature understanding." Data and systems are acquired to allow managers to gain insights ... to gain deep and thorough understandings of what is driving their businesses.
The next section describes 1)how managers currently use information systems to generate insights and 2) questions the wisdom of total reliance on managerial computer as a means of gaining insights from marketing data. Next, an evolutionary path is presented for a firm to use in moving from today's manual system to a more automated process. A military analogy is used to introduce the notion that basing marketing systems on a managerial computing framework leads to an improper focus because it turns managers into manufacturers; in this case, manufacturers of reports. The final sections of this chapter introduce the idea of a report factory and present an evolving architecture for marrying today's managerial computing systems with evolving insight generation systems.
Marketing data can be thought of as filling a cube ... a marketing datacube. In fact, when marketing data are purchased by a company in printed form, it is delivered by truck on a palette ... a physical cube. Marketing data refer to items in regions for multiple time periods, which also makes cube a convenient representation. In any cell of the cube, multiple measures can be thought to reside: volume, price, advertising, etc.
Firms have adopted two models for computerizing marketing data: data processing and information systems.
Data Processing Systems: The first attempt to computerize marketing data was with data processing systems, such as those that were prevalent in the 1950s and 1960s in most corporations. These systems were usually written in the COBOL language to run on the early mainframes in batch mode. Standard reports containing rows and columns of numbers printed on large format paper would be produced on a periodic basis and distributed to anyone in the firm who had a need for the data. The name "data processing" was a good description of the purpose of the systems: they took in raw data and processed it into a more meaningful form. They typically summarized the data by time period and/or geographical region and product line.
These systems were utilized by marketing firms to report their internal data, primarily orders and shipments. Outside data vendors used data processing systems to prepare printed copies of survey, store audit, and warehouse withdrawal data. Nielsen Marketing Research and SAMI would issue multiple volume sets of data on a periodic basis. It was common for brand managers to have two dozen or so notebooks of printed reports based upon Nielsen store audit data, one notebook for each Nielsen market. Additionally, the brand manager would have a stack of fan-fold paper that listed the time trend of shipments for each of his/her brand items, usually broken down by the firm's sales regions.
These types of systems are still in use in some firms, although on a limited basis due to the large volume of data. It is simply not practical to produce printed reports of all the possible view of the data that are needed by different people within the firm. A conversation with a marketing research manager in the mid-1980's illustrated the problem. The firm had a data library in a small room, and it had just switched from purchasing store audit data to buying scanner data. The manager said that the printed scanner data were delivered by truck to the firm's loading dock, where it had to be handled by a forklift truck. It completely filled the small library. About the time the data were moved into the library and organized, a new shipment arrived. The manager's comment: "I don't know what we are going to do because we are drowning in these data and our data processing systems cannot deal with it."
Management Information Systems: Management information systems (MIS) came into vogue in the 1970s with the advent of time-sharing computer systems. The earlier data processing systems only operated in the sequential, batch mode, which meant that only one program or "job" could be processed at one time. Operators would sequence the jobs in terms of a priority system, and process them one after the other. Time-sharing systems allowed multiple jobs to run at once, thus permitting many people to use the same computer at the same time. One outcome of this technology was the ability to place marketing data in a file or database and to then allow different users to gain access to the data in the same time frame. A brand manager could work at a computer terminal and request a brand topline report, at the same time that a sales planning analyst was using another terminal to produce an account review report.
Such information systems opened a new door for marketing managers because it allowed them to get the information when they needed it in the form that they needed. However, the early systems did not live up to this promise because they had restricted capabilities due to the lack of computing skills by marketing managers and analysts. These systems tended to allow the managers to produce and print a limited number of standard reports. For instance, they may contain only 10 or so report types: brand topline, region topline, brand trend, region trend, brand opportunity matrix, region opportunity matrix, etc. Managers would use a terminal to select the desired report type, enter several parameters that specified the brand, styles, flavors, time periods, and regions. Sometime later, a screen full of numbers would appear on the manager's terminal. Such systems were the dominant ones in use by marketing managers in the mid-1980s.[3]
The latter part of the 1980s saw an evolution in the power and ease of use of management information systems, along with emergence of two clear market leaders: the Express System with its Dataserver front-end and the Metaphor system that has been acquired and re-written as the Data Interpretation System. Ease of use was dramatically improved from earlier systems via the Dataserver's use of flexible pop-up menus and the iconic interface employed in the Data Interpretation System. These systems allow marketing managers to start using the system after a short training period, and provide a large degree of freedom in the types of reports that can be generated. It is common for firms to identify more than 100 report types that they can generate as needed.
By the early 1990s, almost all consumer packaged goods firms have acquired and made use of one of these two systems. Most brand groups could and would utilize these systems to produce standard reports as needed, and to do periodic ad hoc analyses. It was quite uncommon for a firm to develop its own marketing information system; these firms would buy a system rather than make one.
Managers use the MIS to generate "reports" that contain rows and columns of numbers. The following is a typical report that a manager might produce using an MIS.
Having produced this report, the manager can then do analysis;
can find the "news content" in the rows and columns
of numbers; can attach meaning to the information; can gain insights
from this information. The following would be such an analysis:
"I see that Magic Bean has lost share in Boston. And, the
share loss was due to recent loss of share for the 32 OZ containers."
The manager has generated insights from the information in the
report.
What does the manager then do with these insights? They either remain with the manager, or they
are communicated to others via memos, letters, reports, and/or presentations. Thus we
can view the managers job as both generating insights from information, and then sharing these insights via
communications. She uses her computing skills to convert data into information, uses her cognitive skills to convert
the information into insights, uses her composition and persuasion skills to produce an
effective communication, and then uses her diplomatic skills to persuade others.
The typical computer system may include dozens of of information displays like the one above, which can be produced by selecting the display type from a menu, and then using other menus to specify the brand, size, flavor, market, periods, and measures. Representative reports are brand toplines, geography toplines, event breakdowns, opportunity matrices, etc. The manager thus has an opportunity to examine any and all aspects of her business in varying levels of detail.
Is this approach the one for firms to use in gaining insights from sales and marketing data? Is it the "final solution?" It is clearly based upon the dominant model for computer systems that contain data that need to be accessed by more than one person: the management information system model. It would seem that this model is optimal for the problem of gaining insights from marketing data, given the large number of books, journals (e.g., MIS Quarterly), and academic degree programs in the MIS area. In fact, in most companies the organizational entity charged with managing the firm's computers is termed the Information Systems (IS) department or the Management Information Systems (MIS) department. The philosophy behind this approach is built around the idea that managers and analysts need access to information, and that the role of the IS department is to provide the databases, systems, and tools for such access. Once these facilities are in place, the only other job is to train the end users. Experience has proven, however, that such training is a never ending task which consumes far more resources than initially expected.[2]
The IS approach worked because it reduced the stacks of printouts and provided managers with a tool they could use to extract the information to accomplish their tasks. This worked because the data were not voluminous, and in some instances the data were actually scarce. However, the days of scarce data are over for many sales and marketing organizations. Peter Drucker has reached the following conclusion about the problems associated with viewing the new situation from the old perspective: "... people assume the more data, the more information -- which was a perfectly valid assumption yesterday when data were scarce, but leads to data overload and information blackout now that they are plentiful."[4]
Drucker introduces powerful phrases, data overload and information blackout, to describe today's situation. Do these terms apply to sales and marketing data? To answer this question, consider the size of marketing databases. A brand manager in a company such as Procter & Gamble purchases data about all the items in his or her product category, e.g., coffee. In the early 1980s, this manager would purchase data that were obtained from store audits from a firm such as Nielsen Marketing Research. The typical database would contain about 2 million numbers. In 1990, the manager would purchase data based upon UPC scanners, and the database might contain 2,000 million numbers.
In 1980, the data would be updated six times a year -- about every ninth week. The updates to the database would contain about 100,000 new numbers; numbers that the managers had to analyze to see what was happening to the brand. This analysis would take about one week, after which the managers could turn their attention away from analysis and spend the next eight weeks doing marketing. This was not a bad distribution of effort: one week of analysis to eight weeks of marketing.
By 1990, the 100,000 new numbers may have given way to 10 million new numbers, and they can arrive every week. If the new information systems allow the managers to do their work 100 times more efficiently, then they can still analyze the new numbers in one week. But, just as they finish with the analysis another 10 million new numbers arrive. As a marketing research director says: "There are just too many numbers and too many things that you can potentially do with them."[5] This situation was highlighted by a story of one marketing manager using an information system to print a report that broke his brand's sales down by regions and markets within region, and then by brand, size, flavor, etc. On the surface this appeared to be a reasonable request. However, the result was 1,000,000 pages of printed material ... one million pages of rows and columns of numbers.[6]
This situation has caused one manager of marketing data and systems to coin the phrase information ecology, "more information than the organization can process is a form of pollution; it is possible to have too much information."[7] Upon hearing this characterization of the situation, another manager of marketing information invented yet another descriptive phrase: infocarcinoma , "cancer of the information system; advanced stages lead to analysis paralysis."[8]
Both of these managers have been very involved in applying the traditional IS model to marketing and sales data, and they have found that this approach runs into a major barrier: the size of the databases and the inherent limitations of managers using information systems to generate insights from rows and columns of information, along with simple graphs. Individuals simply cannot devote the necessary time to generating all of the insights that are buried in the data. The human motor skills and information processing mechanisms cannot keep up with any where near the 10 million new numbers that can arrive every Monday morning.
This situation was forecast by McCann[3], who studied the use of computers and data in the mid-1980's in 12 consumer packaged goods firms and saw that the traditional MIS approach might not work due to the imbalance between the size of the databases and the time available for managers and analysts to run computer reports and then gain insights from studying the rows and columns of numbers in the reports. His recommended solution: use a knowledge-based system to capture the managers' and analysts' expertise and then let the knowledge-based system interact with the information system. The knowledge-based systems would run the reports, analyze the rows and columns of numbers, and send the results to the manager.
A knowledge-based system replaces the human in the use of the
information system to generate insights.
The knowledge-based system substitutes for the marketing manager; it absorbs rows and columns numbers from the MIS and uses them to produce documents such as memos, reports, or sales presentations.
The goal of the knowledge-based system is the generation of insights from the information which is produced by the information system from the data stored in the database management system. Such a knowledge-based system can be termed an Insight Generation System (IGS).
The key element in our conceptualization of an IGS is that it is a system and/or application which interacts with the firm's information system, rather than as a system which replaces the information system. As such, it can be viewed as the third phase in an evolution of marketing systems.
In this evolution model, firms first acquire data and put it into a database management system. Then managers and analysts are provided with a management information system, which the managers use to interpret the data. Managers are provided with personal computers that can access scanner databases through software such as the Data Interpretation System (DIS) and Data Server (DS). These software packages are used to first obtain data from a database and to then display the data in row and column format and/or as a business graph. These information systems allow the firm to move from data to information: the rows and columns of numbers and the graphs can be thought of as information. The raw numbers which were processed to produce these entities are data.
The basic philosophy underlying this approach to marketing computing is the philosophy which underlies the general area of Management Information Systems: managers and analysts will use the facilities of an information system to convert data into information. The manager then studies the information for the purpose of arriving at insights. That is, the MIS converts data into information, and the manager converts information into insights.
This philosophy is fine when certain conditions are met:
The last point needs clarification. A unique decision can be thought of as a decision which arises occasionally, and is not part of an on-going decision task. A decision class , on the other hand, is a set of similar decisions which one might make many times, either repeatedly through time or for many instances at the same point in time. An example will serve the clarify the difference. A decision to release a new product is clearly a unique decision which is only made occasionally. A decision to encourage a retail account to use displays or ad features to merchandise an item is a decision class because the firm or manager must make this decision for every retail account.
When one is faced with a decision class situation, the traditional approach to the data-to-information-to-insights problem becomes problematic. One either has to assign a number of analysts to the task, or train many managers to undertake the problem. In the example situation, a firm might provide an information system to all of its sales representatives and then train these people in the use of the decision for making the few decisions which involve their retail accounts. Another approach is to systematize the problem of insight generation with the use of analytical and/or expert system tools.
It has been our experience that most firms are in Phase I or II, and that it is possible to get stuck in that position ... to focus entirely on the data to information aspects of the situation. The conversion of data to information produces a way-station along the path to a higher goal: insights. However, experience shows that designers of a firm's marketing and sales systems often get way-layed while traveling down this path, and may confuse the way-station with the real destination. This section elaborates on this point and offers a new way to think about the role of marketing systems.
The reason for acquiring marketing data and deploying marketing systems is often stated as using information and information technology to gain a strategic, competitive advantage. But what does that mean for a firm which has masses of data about products, channels, consumers, and competitors?
To answer this question, firm's tend to form a committee of information systems and marketing research managers, perhaps augmented with marketing and sales managers. In almost all cases, the committee's solution is to build an information system for use by end users. This is an easy and obvious decision because it is the classical approach espoused by almost all information system professionals, managers, and professors. The basic idea is to put end-user tools in the hands of managers and analysts so that they can use their knowledge to gain competitive advantage. The information system is then a tool which these people use for their own purposes.
Using a military analogy, the IS professionals are the weapon vendors who provide the soldiers (marketing and sales managers) with a weapon that they can use to attack their enemies: consumers, competitors, and channels. So, information systems and PC tools such as spreadsheets are our tanks and rifles, with information being our bombs, rockets, and bullets, i.e., our projectiles. The user chooses which weapon to use with which bullets to attack which targets.
For instance, the manager may decide to attack the retail trade situation by gaining insights into where s/he has a problem or an opportunity to improve the brand's situation in the various accounts. S/he elects to use an on-line information system to obtain data about sales rates and distribution levels in each market for each of the brand/style/flavors. These data are then fed into another weapon, a spreadsheet, where transformations are made and a graph is produced. A table of information and the graph are saved in files, and the manager then selects another weapon (a graphics package) for annotating the graph. The table and the graph are then used as ammunition in yet another weapon, a word processor. The result of this combination of weapons and ammunition is a report which identifies those items which deserve more distribution in each of 50 markets.
What is the purpose of this report? What happens to it? More often than not, it is a weapon which the manager uses to gain corporate resources to attack the problem. These resources could be financial, in which case the manager could request increased trade support funds to be used to induce retailers to carry the items which deserve more distribution. Or, the resources could be time and energy, in which case the manager is requesting that the sales force allocate its time and energy to gaining distribution for these items. In such a case, the sales force would need to not undertake other activities in order to devote time and energy to gaining distribution for the items. For instance, the sales force may have to not work on other brands, or work on gaining distribution rather than shelf space or merchandising support.
There are several aspects of this view of the role of information systems which are important:
What are the weapons in the traditional approach? They seem to be the information, information systems, and PC tools. A manager assembles these weapons and fights a battle. But there are really multiple battles. The first one is the generation of the report.
A battle is raging within this manager. He is trying to use data
and a computer system to generate a report. He has to draw upon
knowledge about the data, the systems, analytical techniques,
and presentation techniques to produce a report that will have
the desired effect. This is not an easy battle for most managers
Perhaps this is the wrong battle. Perhaps this manager should not be exerting his/her precious time and energies to this task. Perhaps we have the wrong view of what is a weapon? Perhaps the weapon is the report, and not the tools used to generate the report.
If the report had been prepared in sufficient detail, it could become a battle plan and set of orders for the troops in the field ... for the field sales force. It could contain 1) statements about the importance of gaining new distribution, 2) an identification of the items which deserve more distribution, 3) an identification of the items that the sales force could attack (i.e., competitive items that the sales force could suggest for removal from retail shelves), and 4) presentation material for use in the sales calls. In that case, the report contains weapons for use with three targets: senior management, the sales force, and the retailers.
Using this weapon, the marketing manager can convince senior management that increased distribution for his/her brand is the appropriate target. Then, s/he can use the report to convince the sales force that they should spend their time on his/her brand. Finally, elements of the report can be used by the sales force to convince retailers that they should carry the items not currently on the shelves in their stores. In the first two instances, the report is used as a weapon to gain the support from senior management and the sales force in the battle to gain new distribution. In the third instance, the report is a weapon used by the sales force, once it has joined the battle, to attack the competitive items in order to gain shelf space for its own items.
If this is the correct view, then providing a manager with information systems and tools is analogous to turning him or her into a manufacturer. S/he must use the tools to produce the weapon ... to produce the report. If there are 50 such marketing managers in the firm, then there are 50 production processes going on. Worse yet, the report which identifies distribution opportunities is only one of dozens of similar reports which identify and analyze opportunities for media advertising, shelf space, display support, retail advertising features, competitive pricing, coupon activity, etc.
Does this information systems approach to getting value from marketing and sales data work? One way to assess the degree of success is to listen to how the targets of the reports describe the results. First, there are the comments of senior marketing managers, who make statements like the following:
Product managers are more effective because of the information technology, but the use of those data is still the same as it was 5 years ago. They don't do anything different on the computer than I did on my old calculator, they just do it faster. ... There has not been that 'great leap forward' from speeding up clerical tasks to really improving decision making. ... We will probably need some impetus, some creative light to think about how we make that leap because it's not going to come by itself. It's not going to happen just because they have PCs sitting on their desks. [9]
Then there are statements by managers of grocery retail firms and broker firms:
We now have Scan Track coming down the line. I don't think there is one out of 50 product managers who knows what Scan Track really does, what it measures, what it's used for, how to handle it, or how to bring it to a customer.[10]
These comments agree with many others we have encountered as we have worked on the problem of getting value from marketing and sales information systems. It is quite common to hear a senior manager comment that "millions of dollars have been spent on data and systems, yet we have not been able to get real value from these expenditures." Perhaps the reason lies in the fact that the approach has been oriented too much around the concept of end-user computing, which relies on the managers using the basic components (data, systems, and PC tools) to produce the weapons. Perhaps too many marketing and sales managers have been turned into producers of reports ... into factories. Perhaps we should adapt a new approach which focuses on factory automation, where the factory being automated is the production of marketing and sales weapons ... the reports.
The challenge is to build a factory which produces reports, with a report having sentences, paragraphs, tables, and graphs. Such a report presents evidence in the form of data tables and graphs, and then makes observations and draws conclusions using this evidence. The following would constitute a paragraph in a typical report.
The following chart shows the impact of a price cut, an advertising feature, and a retail display on the sales of Jiffy peanut butter in New Orleans.
How would one build an office factory for producing such sales and marketing reports? The approach involves the building of a sales and marketing report factory in the form of insight generations systems that evolve from, and are integrated with, the firm's existing marketing and sales management information systems.
Once a factory is built that generates insights, one has to deal with the output of the factory. One approach, which is obviously needed, is to the send the factory output to those who order it ... to those who can make use of it. So reports about Tide should go to the Tide brand manager; reports about San Francisco should go to the San Francisco sales manager; reports about Safeway should go to the Safeway account representative.
In addition, it might be advantageous to put these reports into a computer system so that the interested manager and/or analyst could retrieve them at will. This would permit the firm to "know what it knows" about a particular subject. "What do we know about Safeway?" "Let's look and see what insights have been generated about Safeway."
Thus another phase of the evolution is needed: the Insight Management Phase.
Another way to think about this evolution is in terms of what we call the DIIP model: Data to Information to Insights to Programs.
This model is based upon the idea that the primary job of marketing managers is the design and execution of marketing programs. Before the marketing manager can design marketing programs, s/he must understand market situations, and this understanding is obtained via insights. But in order to have insights, the manager must have information. And, before s/he can have information, there must be data. This simple model thus allows us to see that insight generation systems should not be developed in a vacuum, but should be integrated with the earlier systems. The next section describes an architecture for such system integration.
The Uniform Product Code (UPC) symbol appears on almost all items carried by supermarkets, thus providing the foundation for an extensive marketing database. These data allow manufacturers, wholesalers, brokers, and retailers to build databases which report on their own items as well as those of their competitors. As such, this industry is leading the way for similar advances in almost all consumer industries and most financial services. As similar data become available in other segments of the economy, the concepts, approaches and tools described in this book will see a wider audience.