The term data mining is just one of the several terms, including knowledge extraction, data archaeology, information harvesting software and even data dredging, that actually describe the concept of knowledge discovery in databases. Let us begin with a few basic facts to explain what data mining really means:
Many organizations, both private and government, have devoted a tremendous amount of resources to the construction and maintenance of large information databases over the recent decades, including the development of large scale data warehouses.
Frequently, the data cannot be analysed by standard statistical methods, either because there are numerous missing records or because the data is in the form of qualitative rather than quantitative measures.
In many cases, the information contained in these databases is undervalued and underutilized because the data cannot be easily accessed analysed.
Some databases have grown so large that even the system administrators do not always know what information might be represented or how relevant it might be to the questions on hand.
It would be beneficial for organizations to have a way to mine these large databases for important information or pattern that may be contained within.
There are a variety of data mining methodologies that may be used to analyse data sources in order to discovery new patterns and trends.
The idea behind data mining then is the nontrivial process of identifying valid, novel, potentially useful, and ultimately understanding patterns in data. On the other hand it automates the detection of relevant patterns in a database. In the example, above tracing brand movement across various segments, typically a data mining system helps the marketer detect any shifts that may be taking place before the situation becomes too bad. However, one has to keep in mind that data mining is not magic. Marketing researchers and statisticians have mined data looking for statistically significant patterns. Data mining tools today are both well established , statistical and computing techniques both of which help to build customer response models. Data mining and CRM allow users to analyse large databases to solve business decision problems.
Information technology has automated the mining process today, integrating with commercial data warehouses. Also, it should be kept in mind, that even though IT has been used for developing data mining systems, human experience and intuition in recognizing the difference between relevant and irrelevant correlation is still essential. For, data mining to impact a business, it needs to have relevance to the underlying business process. Data mining is part of a much larger series of steps that takes place between a company and its customers. The way in which data mining impacts a business depends on the business process and not the data mining process alone. A marketing manager’s job is to understand the market for his/her brand. With this understanding, comes the ability to interact with customers in this market using several channels. This includes a number of areas like direct marketing, advertising, telemarketing, sales promotion and so forth.
One has to keep in mind the fact that the results of data mining are different from other data driven business processes. In most standard interactions with customer data, nearly all of the results presented to the decision maker consist of information they were already aware of. For example, a sales information report showing the sales by product line and region is simple for the decision maker, to understand because he intuitively knows that this kind of information already exists in the database. If the company sells different products in different areas, district or city, there is no problem translating a display of this information into a relevant understanding of the business process.
Data mining on the other hand, extracts information from a database that the decision maker did not know existed. Relationships, between variables and customer behaviours that are non-intuitive are what one should find in the data mine. And since he does not know, beforehand, what the data mining process gas discovered it is a much bigger leap to the solution of a business problem.
Having discovered an unknown the challenge now for the marketer is to understand the phenomena and take a decision.
There are two parts to this problem:
- Presenting the output of the data mining process in a meaningful way, and
- allowing the marketer to interact with the output so that simple questions can be answered. Creative solutions, to the first part have recently been incorporated into a number of commercial data mining products. Response rates and (probably most importantly) financial indicators (for example, profit, cost, and return on investment)) give the marketer a sense of context that can quickly ground the results in reality.
Data mining can also help the marketer connect better with his/her customer. Customer relationship management (CRM) is one of the vital areas where data mining has been used successfully by institutions and firms like Citibank, HDFC, ICICI, Jet Airways, Sony India, Satyam Computer Services and Wipro Info tech.