Enhancing Sea Surface Temperature Data Using Hadoop-Based Neural Networks
Large-scale sea surface temperature (SST) data are crucial for analyzing vast amounts of information, but face challenges such as data scale, system load, and noise. A Hadoop-based Backpropagation Neural Network framework processes SST data efficiently using a Backpropagation algorithm. The system proposes a solution for filling missing data gaps in SST through artificial neural networks, outperforming traditional methods.
Uploaded on Sep 12, 2024 | 2 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Filling of data gaps in Sea Surface Temperature using Hadoop based Neural Networks Under The Guidance of Presented By B.Sangamithra S.Devikumari 13MT8202 Thursday, September 12, 2024
Contents Abstract Existing system Proposed system Data Processing With Hive Algorithm Neural network model Neural network steps Output Conclusion
Abstract The Large scale sea surface temperature(SST) data that are generated continuously by multiple sensors in daily communications are possessed high significance for analyzing the behaviors of huge amounts of data. However, the natural properties of satellite data present three non- trivial challenges: large data scale leads it difficult to keep both efficiency and accuracy; similar data increases the system load; and noise in the data set is also an important influence factor of the processing result and need to be worked efficiently with the neural networks on large data sets. Data is divided into separated segments, and learned by a same network structure. A Hadoop based framework called HBNN (i.e. Hadoop-based Back propagation Neural Network) is proposed to process forecast on large-scale SST data. It uses a Back propagation algorithm which provides greater efficiency and good scalability.
Existing system Traditional methods used for filling of data gaps are not effective for non-stationary and non-linear time series data . In case of missing data and when its location is random there is no solution is provided Proposed system The SST gap filling (missing data) problem can be solved through Hadoop based Neural Networks. We suggest an approach to solve the above mentioned problem i.e., recovery of missing data in time series using artificial neural networks.
Data Processing With Hive Hive is a database technology that can define databases and tables to analyze structured data. Hive is a data warehouse infrastructure tool to process structured data in Hadoop. It resides on top of Hadoop to summarize Big Data, and makes querying and analyzing easy. So here Hive database is used to format unstructured data
Hive Queries create table SST_DATAtable (col_value STRING); LOAD DATA INPATH '/home/hadoop/training/hive/SSTdata.csv' OVERWRITE INTO TABLE SST_DATA ; create table SSTdata (U1 string, U2 string, WV string, CLW string, RR string); insert overwrite table SSTdata SELECT regexp_extract(col_value, '^(?:([^,]*)\,?){1}', 1) U1 string, regexp_extract(col_value, '^(?:([^,]*)\,?){2}', 1) U2 string, regexp_extract(col_value, '^(?:([^,]*)\,?){3}', 1) WV string, regexp_extract(col_value, '^(?:([^,]*)\,?){4}', 1) CLW string, regexp_extract(col_value, '^(?:([^,]*)\,?){5}', 1) RR string, regexp_extract(col_value, '^(?:([^,]*)\,?){6}', 1) SST string from SST_DATAtable;
Back propagation Algorithm First apply to the inputs to the network and out the output At the start the neural network is assigned random weights for its connections. The output achieved by using the given input is compared to the target output. The weights are adjusted to reduce the different between the target and output to the minimum. This process is repeated until a low enough difference is achieved. This is a stopping condition known as the desired error. Anther stopping condition is the maximum number of training epochs Error calculations Error = Output(1-Output)(Target-Output) Change the weights. Let w1 be the new(trained) weight and w is initial weight w1= w+(Error* Output)
Steps for Neural Network Step1: We can start the neural network tool using the nntool command Step2: Select the import. It will display the window. Step3: Here brows the input and target datasets load from disk files.
Step5: Train the Perception Now click the Train tab. Specify the inputs and output by clicking the Training Info tab and selecting the inputs to trained input, and the targets to trained target. Click Train Network to train the Feed-forward backprob network. The following training results appear.
Conclusion The main objective of this study is to retrieve missing values in the measured sea surface temperature time series data.The proposed networks(HBNN) is well trained and tested in filling the SST data gaps with possible error estimation and correction. Our results show that the efficacy of the estimation procedure and thus the reliability of the estimated missing values are dependent on a number of factors. Hadoop based neural
Any Queries Thank You
