OSC An Online Self-Configuring Big Data Framework for Optimization of QoS (TC-2020-02-0128.R1)

ABSTARCT :

Big-data frameworks such as MapReduce/Hadoop or Spark have many performance-critical configuration parameters which may interact with each other in a complex way. Their optimal values for an application on a given cluster are affected by not only the application itself but also its input data. This makes offline auto-configuration approaches hard to be used in practice because the input data of an application may change at each run. To address this issue, we propose an Online Self-Configuring (OSC) approach that automatically determines the optimal parameter values for a given application. OSC synergistically integrates three key techniques. First, OSC leverages ensemble learning to build a precise performance model for a given application. Second, it quantifies the parameter importance and interaction intensity between them to accelerate the genetic algorithm for searching optimal configuration parameters. Third, OSC supports an incremental modeling approach to achieve low overhead of the models for online needs. Our implementation of OSC atop MapReduce/Hadoop 2.6 improves performance by 60% on average and up to 120% compared with the state-of-the-art approach. Lastly, the performance benefit of an application running on OSC generally increases along with its input data size.

EXISTING SYSTEM :

? Existing cloud computing infrastructures could be utilized by aggregating and processing big data in powerful central data centers. ? Moreover, there are urgent needs for integrating new big data applications with existing Application Program Interfaces (API) such as Structured Query Language (SQL), and R language for statistical computing. ? The first category at the application level targets the studies that extend or optimize existing framework parameters and mechanisms such as optimizing jobs and data placements, and scheduling, in addition to developing benchmarks, traces, and simulators.

DISADVANTAGE :

? The expected service level should consider not only QoS guarantee, measured through different performance parameters (i.e., delay, packet delivery ratio, etc.), but also security and users’ privacy issues within different layers of the IoT architecture. ? We are capable of coping with this problem by implementing a new QBAIoT LL-Gw (i.e., coordinator) in order to extend the capacity of the IoT environment and respect the requirements of an increasing number of QoS classes corresponding to the subcribed iSLAs. ? Different issues are studied for guaranteeing the QoS. Thus, QUASIMODO presented a routing protocol for multi-hop routing under multi-constraints (energy, link reliability, delay).

PROPOSED SYSTEM :

• To process the big data generated by IoT devices, different solutions such as cloud and fog computing were proposed. • A dynamic algorithm that assigns data fragments according to nodes processing capacities was proposed to reduce data movement between over-utilized and under-utilized servers. • The study in proposed a Dependency Aware Locality for MapReduce (DALM) algorithm to process highly skewed and dependent input data. • In, a data replication-aware scheduling algorithm was proposed to reduce the network traffic and speculative executions. The proposed map jobs scheduler was implemented in two waves.

ADVANTAGE :

? In this research work, we aim to present the design details and performance evaluation of our proposed QBAIoT access method and its usage in an IoT architecture in order to satisfy the requirements of an e-health service according to a proposed iSLA. ? Therefore, new and well-adapted QoS models must be proposed to provide IoT services performance guarantees. ? Indeed, during congestion periods, IoT traffic performance will be affected and especially the performance of QoS constrained data streams. ? In order to avoid performance degradation when delivering critical data, an effective and optimized management of the available resources is necessary to guarantee a certain service level in the IoT environment.