On clusterization of "big data" streams

Abstract

Big Data refers to the rising flood of digital data from many different sources, including the sensors, digitizers, scanners, mobile phones, cameras, software-based tools, internet, and so on. "Big" and "diverse" are two important characteristics of Big Data. The diversity of the Big Data, such as text, geometry, image, video, or sound, also increases difficulties of big data processing. Coping with the "Big Data" problems requires a radical change in the philosophy of the organization of information processing. Primarily, the Big Data approach has to modify the underlying computational model in order to manage the uncertainty in the access to information items in a huge nebulous environment. As a result, the produced outcomes are directly influenced only by some active part of all information items, while the rest of the available information items just indirectly affect the choice of the active part. An analogous functionality exhibits the organization of the brain featuring the unconsciousness, and a characteristic similarity shows the retrieval process in Google. In this talk, we introduce a novel method for on-the-fly clusterization of amorphous data from diverse sources. The devised construction is based on the previously developed FuzzyFind Dictionary reversing the error-correction scheme of Golay Code. This clusterization involves processing of intensive continuous data streams that can be effectively implemented using multi-core pipelining with forced interrupts. The suggested clusterization is especially suitable for the Big Data computational model as it materializes the requirement of purposeful selection of information items in unsteady framework of cloud computing and stream processing. Furthermore, the uncertainties in relation to the considered method of clusterization are moderated due to the idea of the bounded rationality, an approach that does not require a complete exact knowledge for sensible decision-making.