Miv.t.u-tokyo.ac.jp

Department of Information and Communication EngineeringGraduate School of Information Science and TechnologyThe University of Tokyo7-3-1, Hongo, Bunkyo-ku, Tokyo 113-8656, Japanemail: {helmut,ishizuka}@miv.t.u-tokyo.ac.jp This paper investigates the methodological foundations of a new research fieldcalled chance discovery, which aims to detect future opportunities and risks.
By drawing on concepts from cybernetics and system theory, it is argued thatchance discovery best applies to open systems that are equipped with reg-ulatory and anticipatory mechanisms. Non-determinism, freedom (entropy)and open systems property are motivated as basic assumptions underlyingchance discovery. The prediction-explanation asymmetry and evaluation ofchance discovery models are discussed a fundamental problems of this field.
Several researchers within the Knowledge Discovery in Databases (KDD)community (e.g., Ohsawa [1.9]) questioned whether the methods of this re-search field are able to find what they call ‘chances’. Chances refer to phenom-ena that might have a (high) impact to the scientific (and human) society oran enterprise in the future. High impact is intended to have two complemen-tary readings: on the one hand it refers to opportunities, i.e., the possibilityto bring about desirable effects; on the other it refers to risks, i.e., possiblethreats to an enterprise or society. The notion of chance discovery has beencoined to cover both aspects. Finding future features is seen in contrast toprediction (e.g., in KDD), the scientific activity to derive phenomena thatappear at some future time point. By contrast, chance discovery explicitlyintegrates human initiative into the discovery process.
Procedurally, chance discovery can be seen as a two-step activity. The first step involves a actual discovery of a certain phenomenon. The secondstep suggests actions taken as a consequence of a designated phenomenon(chance), which is often called (chance) management and involves supportivemeasures in the case of opportunities as well as preventive measures in thecase of risks.
Although there might be some interesting interactions with the proba- bilistic notion of chance, this reading is not intended in chance discovery.
Likewise, chance discovery is not concerned with discovery by chance, suchas the discovery and isolation of penicillin by Alexander Fleming.
We will discuss the following topics. In the following section, the notion of open system is explicated in terms of cybernetics and system theory, and thepossibility of prediction is discussed for both nature and open systems. Thenext section discusses chance discovery in open systems. In particular, thenotion of ‘anticipation’ is introduced as a mechanism for chance discoveryand exemplified by examples. After that, we explicate notions underlyingthe possibility of chance discovery: uncertainty and freedom. In the followingsection, chance discovery is contrasted with KDD. Finally, we briefly discussand conclude the paper.
To clarify the application field of chance discovery, we draw a broad dis-tinction about the object of investigation: nature vs. open systems [1.12].
Whereas nature is governed by natural laws, open systems are typically mod-eled abstractly by cybernetics [1.1] and system theory [1.16]. Examples ofopen systems include ‘living’ systems such as human beings, scientific com-munities and companies, and artificial (or technical) systems, e.g., cars andpower plants. Both kinds can be described by the following system-theoretical(S1 − 2) and cybernetical (C1 − 2) features (Schurz [1.12]): S1 Open systems are physical ensembles placed into an environment signifi- cantly larger than themselves. There is a continuous exchange of energybetween system and environment. The environment may satisfy the sys-tem’s ‘needs’ (see C1) or ‘destroy’ the system (see C2).
S2 Open systems preserve a relative identity through time, called their dis- C1 The identity in time is abstractly governed by ideal states (or norm states) which the system tries to approximate, given its actual state.
C2 Regulatory mechanisms compensate disturbing influences of the envi- ronment, i.e., they continuously try to counteract influences that movethe system apart from its ideal state. If the external influences exceed a‘manageable’ range, the system is destroyed.
For our present discussion, the regulatory mechanisms of open systems areof central concern since they can actively interfere with the evolution of thesystem, by bringing about (an approximation of) the ideal state, or avoidthe destruction of the system. Later, we will introduce a new kind of mech-anism, called ‘anticipation’, that has the potential to significantly influencethe systems evolution and most closely corresponds to our notion of chancediscovery.
Nature is governed by the laws of physics, e.g., Newton’s second axiom (thetotal force law). Obviously, in the physics domain there is no way to influence 1. Methodological Considerations on Chance Discovery the natural laws. So even if we predict a phenomenon of high impact tosociety, such as a giant meteorite approaching the earth at high speed, all wecan do is to evacuate the area the meteorite is predicted to hit.
Since it is not possible to change the course of nature, chance discovery here means to take appropriate (supportive, preventive) measures to minimizedamage or maximize benefit.
Open systems are characterized by system laws. Schurz [1.12] argued thatwe are theoretically unable to determine the exact numerical values corre-sponding to system laws, because the systems are open and hence describedby nonlinear differential equations. In the extreme case, if external influencesexceed the manageable (or critical) range of the system, nonlinear dynamicsbecomes effective and leads to chaotic behavior. Due to the sensitivity ofopen systems to external influences, prediction is a difficult matter. Belowwe will argue that in open systems, the activity of regulatory mechanisms isof major importance, rather than prediction.
Let us first give an illustrative example. Enterprises (companies) can beviewed as open systems that consist of subsystems (branches, sections, andindividuals), and operate in an environment, the so-called ‘economic market’.
This environment typically satisfies the companies ‘needs’, e.g., customers de-mand the company’s products. Under unfortunate circumstances, the com-pany may run into the risk of being ‘destroyed’, e.g., by the appearance ofa strong competitor (cf. S1). In spite of that, companies preserve identitythrough time (cf. S2). A company constantly tries to approximate an idealstate where, for instance, increasing profits are made and the economic situa-tion of the company is stable. This is achieved by the company’s subsystemsthat perform certain functions, including good production and distribution,and marketing (cf. C1). A company is typically confronted with a multitudeof ‘disturbing’ influences in the form of, e.g., cheaper and better products ofother companies and changing customer needs. At this point, the regulatorymechanisms of the company come into force, e.g., to lower production costsby increasing the efficiency of the production cycle. It is well-known thatcompanies go bankrupt when a critical range is exceeded (cf. C2).
1.3.2 The Limits of Regulatory Mechanisms Regulatory mechanisms are the system’s means to approximate the system’sideal state. Those mechanisms are mainly active to compensate disturbing influences by reacting to them. Although regulatory mechanisms are usuallyable to guarantee the identity of an open system, they come into force only ifconfronted with ‘threats’ from the environment. For instance, if a company’ssales decrease, the CEO might decide to shrink the company, thereby makinga number of people unemployed.
In the next section we will argue that in addition to regulatory mecha- nisms, open systems need mechanisms of anticipation to cope with the com-plexities and influences of the environment.
In a recent report to the Club of Rome, Botkin et al. [1.2] introduce the term“anticipation” as a key feature of innovative learning that emphasizes humaninitiative. It is described as follows [1.2, p. 25]: [.] anticipation is not limited to simply encouraging desirable trendsand averting potentially catastrophic ones: it is also the “inventing”or creating of new alternatives where none existed before.
Anticipation is contrasted to prediction, since the former focuses on the cre-ation of possible and desirable futures, and plans to bring them about. Thenotion of anticipation shares the intuition of Alan Kay’s phrase “The bestway to predict the future is to invent the future”.
Promotion In philosophy of science, the term “self-fulfilling prophecy” de-scribes situations such as the following. Newspapers write articles about themorbidity of a bank institute. As a consequence, many customers of this in-stitute withdraw their money and other commitments. In effect, the bankinstitute gets into serious trouble. A recent ‘real’ example is the success ofthe so-called New Economy (internet and telecommunication related shares).
Since many people believed in its success, it became a great success (at leastfor some time).
Chance discovery as anticipation in this context means the promotion of a trend desired by New Economy companies. As a result of promotion,the desired trend could be effected. Similar forms of promotion are dailypractice in companies: certain products are advertised with the hope thatthey actually trigger a desire in customers. The detection of ‘latent’ customerdesires will be briefly discussed in the next section.
Collaboration In business there is a lot of talk about ‘mergers’. Collabo-rations are also seen in scientific research programs. We will briefly describethe field of Quantum Computation.
Deutsch [1.3] is reported to be the first to explicitly ask whether it is possi- ble to compute more efficiently on a quantum computer. For a long time, thispossible collaboration of quantum theory (physics) and artificial intelligence(computer science) remained a curiosity. However, there are already someindications of ‘killer applications’ for quantum theory. For instance, Spector 1. Methodological Considerations on Chance Discovery et al. [1.13] report on problems that take polynomial time on a quantumcomputer but exponential time on a classical computer.
In academics, possibilities for collaborations are ubiquitous, and some- times realized, e.g., in genome analysis, artificial intelligence and biology col-laborate. What might chance discovery as anticipation mean here? In partic-ular, how can we anticipate the success of a certain kind of collaboration? Wecannot provide a working methodology here. In the case of quantum com-putation, the chance was ‘discovered’ by Feynman [1.5] who observed thatclassical systems cannot effectively model quantum mechanical systems. Thisobservation suggests that computers based on the laws of quantum mechanics(instead of classical physics) could be used to efficiently model quantum me-chanical systems, and possibly even solve classical problems such as databasesearch in a highly efficient way.
Given that Quantum Computation will indeed be successful, how could we have known 10 years ago? One method would be to track the history of ‘con-jectures’ (ideas, observations) formulated by various insightful researchers,and evaluate their feasibility in the light of current knowledge in possiblyquite different research areas. The availability of huge amounts of informa-tion on the Web might facilitate such an endeavor.
1.4 Chance Discovery, Uncertainty, Freedom One of the tacit assumptions underlying chance discovery is that the futureis uncertain, and hence there is freedom to change is course of action. Forthe sake of argument, assume the opposite, i.e., the world history evolves de-terministically. Obviously, under this artificial assumption, chance discovery(in our sense) is not possible as there are no choices.
Following [1.15], we propose entropy as the measurement of freedom. Specif-ically, the measurement of freedom is phenomenologically rather than proce-durally oriented. The freedom of a set A of alternatives is measured by theentropy H of the actual chosen proportions, i.e., where log is to the base 2, pi ≥ 0 and if pi = 0 then 0 log 0 = 0. Accord-ingly, we may say that chances exist if there are (almost) evenly distributedalternatives. Consider the following situations (A) and (B).
(A) There are three sellers with (approximately) 30% market share.
(B) There are two sellers, one has 75%, the other has 25% market share.
Situation (A) has more freedom than situation (B), since a market withone dominant provider has low entropy. The more interesting notion here isfreedom of successive states for a number of time periods. For instance, amarket with 100% customer loyalty is not free.
Let us recall the aforementioned open system situation, that features a highdegree of uncertainty, and formulate it as a problem for chance discovery andchance management (CD&CM). In the following, M stands for a CD&CMmodel (or theory).
– Assume as given a model M that explains why a particular phenomenon X turned out to be a chance (opportunity or risk), as observed by its high(positive or negative) impact.
– Given a phenomenon of type X, can we employ M to predict high impact Of course, the notions of phenomenon of type and comparable warrant furtherexplication. In order to clarify the problem, consider the case of simple un-stable or chaotic systems that support explanations without predictive value.
Assume an ideal ball exactly on top of another ideal ball. Here, we cannotpredict in which direction the ball will roll down, but after it rolled down,we can explain it by an unmeasureably small disturbance in the direction inwhich the ball rolled down [1.11].
Thus, the ‘explanation vs. prediction’ problem raises the fundamental question about which systems support the predictive use of chance discoveryresults. Straightforward answers seem to be ruled out by the fact that humaninitiative is essential to take opportunities or avoid risks, and the complexityof systems such as the web or financial markets.
As a more realistic example, consider Ogawa’s [1.7] ILE (Information of Liability and Equity) measure that identifies risk factors that eventually leadto bankruptcy. Specifically, ILE explains bankruptcy. The crucial question,however, as in science is whether ILE can predict bankruptcy. If ILE haspredictive value, the impact of preventive measures can be proven. Given thetheoretical result about the infeasibility of prediction in open system, we areleft with a probabilistic notion of prediction.
A basic question about scientific theories is how they can be evaluated. Fol-lowing Popper [1.10], a theory is corraborated (or validated) if it predicts aphenomenon that is actually observed, while it is falsified when a phenomenonis observed that contradicts the observation. Note that a theory can never be 1. Methodological Considerations on Chance Discovery verified by a finite set of observations. The situation for CD&CD models iscomplicated for the following reason.
Triple-theory Problem Whether the discovery of a potential chance turnsinto a positive result is dependent on three factors: 1. The designated phenomenon was a ‘real’ chance, i.e., chance discovery is 2. The chosen measures were appropriate, i.e., chance management was suc- 3. The predictions about the world for the associated time span of CD&CD The triple-theory problem refers to the practical problem that in order tovalidate (or falsify) a CD&CM model, three sub-theories have to be successful.
If all of them are successful, observed by the positive result, the model iscorraborated. However, in the case of a negative result, we cannot simply saythat the designated phenomenon was no chance, because we either did notchoose appropriate (supportive or preventive) measures to bring about thepositive outcome or our predictions about the boundary conditions for thepositive outcome have been false.
From a methodological point of view, the triple-theory problem puts se- rious doubts whether we might be able to evaluate CD&CM models scientif-ically. Due to the very nature of the open systems, reproducibility of resultsis infeasible.
Fayyad et al. [1.4] characterize Knowledge Discovery in Databases (KDD) as [.] the nontrivial process of identifying valid, novel, potentially use-ful, and ultimately understandable patterns in data.
The discovery goal in KDD can be divided into a descriptive and a predictivepart. In description the system seeks for patterns (or models) in order topresent them to the user in an intelligible way; in prediction the system findspatterns so that the future behavior of some entity can be predicted. Thereexist a number of established (mostly statistical) data mining methods toachieve those goals, such as classification, regression, clustering, summariza-tion, dependency modeling, and change and deviation detection [1.4].
Chance discovery may use the knowledge extracted by data mining meth- ods to detect future features. For instance, by Web usage mining, i.e., theclustering of Web users based on their browsing activities, potential customergroups can be identified, and specifically addressed by companies. Here the in-terplay of data mining—describing correlations between users’ interests—andchance discovery—actively promoting a possibility—is of crucial importance.
One may ask whether, e.g., data mining already is a form of chance discovery. Our answer is “no”. Data mining can summarize or predict trends,but leaves out the rˆole of human interference. Anticipation as a mechanismof an open system, on the other hand, ‘matches’ a desired (or predicted)trend with the system’s goals (typically human ‘desires’) and accordinglytakes supportive or preventive measures.
Another way of contrasting Chance Discovery and KDD is as follows.
Whereas KDD tries to detect most likely trends in data, Chance Discoveryaims at finding data that do not match likely patterns but indicate interestingphenomena not yet exploited and bearing potential of future trends. However,currently there exist no serious analysis to distinguish those high-potentialphenomena from ‘noise’ in data. Basically, this means that exceptions can beequally informative as highly probable regularities. As an example, considerthe following. Humans that are infected with plasmodium vivax are verylikely to contract malaria. However, some people do not. In KDD terms,those people are ignored since they do fall under the likely case (contractingmalaria). It turned out that it is due to a special genetic constellation thatsome people have a strong protection against malaria. In Chance Discoveryterms, the explanation of those people’s resistance against malaria is a chancefor a significant scientific discovery.
In this paper, we explicate our take on a new research area called ‘Chance Dis-covery’. The notion of ‘open system’, as characterized in cybernetics and sys-tem theory, serves as a framework to embed the activity of Chance Discovery.
In particular, anticipation is introduced as a mechanism that may performthe rˆole of detecting chances in open systems. The anticipating mechanism isexplained in the context of promotion in New Economy and collaboration inthe Quantum Computation research programme. Chance Discovery is con-trasted to KDD and mutually beneficial aspects are explained. We identifyhuman initiative as a distinguishing feature of Chance Discovery (as opposedto KDD), e.g., to actively initiate and foster a trend by promotion or toactively explore the (practical) feasibility of a theoretical conjecture.
Unlike the practical methods for data mining, we only described a method- ology for Chance Discovery. A method for Chance Discovery might analyze‘success stories’, i.e., cases where features of high impact for the future weresuccessfully identified and accordingly promoted by human initiative. Thisretrospective analysis might be framed and processed by means of KeyGraph[1.8], a smart indexing method originally developed for information retrieval.
Recently, McBurney and Parsons [1.6] proposed principled methods to discover chances based on dialogue games. In the context of e-commercesystems, Stolze and Str¨obel [1.14] investigate interviews with buyers in order to identify their (implicit) needs. We believe that the theoretically foundedmethods will have the greatest impact on the field of Chance Discovery.
In this paper, we mainly focussed on the epistemological aspect of chance discovery. However, the discovery of potential opportunities and risks seemsto be intimately connected to questions about human values, what should bethe case and what should not be the case. Obviously, there are no opportu-nities or risks per se, they are only given with respect to certain values andassociated goals of humans. To give drastic example, the detection of a futureearthquake is not only a high risk for people living in a particular region, it isalso an opportunity for certain organizations to take advantage of the chaosfollowing the earthquake.
We would like to thank the reviewers for their sincere and valuable commentsand suggestions. This research is supported by a JSPS Research Grant (1999–2003) for the Future Program.
1.1 Ashby, W. R. (1964) An Introduction to Cybernetics. London.
1.2 Botkin, J. W.; Elmandjra, M.; and Malitza, M. (1998) No Limits To Learning.
Bridging The Human Gap. A Report to the Club of Rome. Pergamon Press.
1.3 Deutsch, D. (1985): Quantum theory, the Church-Turing principle and the uni- versal quantum computer. In Proceedings of the Royal Society of London, 97–117.
1.4 Fayyad, U.; Piatetsky-Shapiro, G.; and Smyth, P. (1996): Knowledge discovery and data mining: Towards a unifying framework. In Proceedings 2nd Interna-tional Conference on Knowledge Discovery and Data Mining (KDD-96).
1.5 Feynman, R. (1982) Simulating physics with computers. International Journal of Theoretical Physics 21:467–488.
1.6 McBurney, P.; Parsons, S. (2001): Chance discovery using dialectical argumen- tation. In Y. Ohsawa (ed.), Proceedings of the First International Workshop onChance Discovery, 37–45.
1.7 Ogawa, S. (2000): Building of trust evaluation model based on the failure pre- diction. In Y. Ohsawa (ed.), Workshop on Chance Discovery and Management.
In conjunction with KES’2000, Brighton, UK.
1.8 Ohsawa, Y.; Benson, N. E.; and Yachida, M. (1998): KeyGraph: automatic indexing by co-occurrence graph based on building construction metaphor. InProceedings Advanced Digital Library Conference (IEEE ADL-98), 12–18.
1.9 Ohsawa, Y., ed. (2000) Workshop on Chance Discovery and Management. In conjunction with the Forth International Conference on Knowledge-based In-telligent Engineering Systems and Allied Technologies (KES’2000). Brighton,UK: IEEE, Inc.
1.10 Popper, K. (1963) Conjectures and Refutations. Routledge and Keagan Paul.
1.11 Schurz, G. (1995): Scientific explanation: A critical survey. Foundations of 1.12 Schurz, G. (1999): Normic laws as system laws: Foundations of nonmonotonic reasoning. In Proceedings 4th Dutch-German Workshop on Nonmonotonic Rea-soning Techniques and Their Applications (DGNMR-99).
1.13 Spector, L.; Barnum, H.; Bernstein, H. J.; and Swamy, N. (1999): Quantum computation and AI. In Proceedings 16th National Conference on ArtificialIntelligence (AAAI-99). Invited Talk.
obel, M. (2001): Utility-based decision tree optimization: A framework for adaptive interviewing. In Proceedings 8th International Con-ference on User Modeling (UM-01), 105–116.
1.15 Suppes, P. (2000) Freedom and uncertainty. In Natke, H., and Ben-Haim, Y., eds., Uncertainty: Models and Measures, Mathematical Reasearch. AcademieVerlag. 69–83.
1.16 v. Bertalanffy, L. (1979) General System Theory. New York.

Source: http://www.miv.t.u-tokyo.ac.jp/papers/helmut-jsai01.pdf