New Approaches in Learning Complex-Valued Neural Networks
VLADYSLAV KOTSOVSKY, ANATOLIY BATYUK, MAKSYM YURCHENKO
NEW APPROACHES IN THE LEARNING OF
COMPLEX-VALUED NEURAL NETWORKS
A MODEL OF COMPLEX-VALUED NEURON
2
A NETWORK ARCHITECTURE
3
NETWORK ERROR
4
ADAM OPTIMIZER
5
CALCULATION OF THE GRADIENT
6
CHOICE OF THE ACTIVATION FUNCTION
7
CHOICE OF THE ACTIVATION FUNCTION
8
SIMULATION
•
We applied the complex backpropagation algorithm for
solving the hard XOR problem in the case of 50
features.
•
We compared the performance of the real-valued and
complex-valued four-layer networks consisting of neurons with
sigmoidal and ReLU-like activation functions. We used 50-20-
10-1 topology in the real case and 25-20-10-1 topology in the
complex case. Each network was trained in on-line mode on
100 different learning samples. The number of epochs was
equal to 1000. We use
q
= 1 for complex rational sigmoid.
9
SIMULATION
The parameter
β
1
and
β
2
were initialized to 0.9 and 0.999,
respectively. The following presents average performance mea
sures in the case of learning using stochastic gradient descent
(SGD) and Adam optimizers, respectively.
10
CONCLUSIONS
•
Feed-forward complex-valued neural networks give a simple
and powerful architecture providing an adequate
approximation of nonlinear mappings. The new activation
functions and the complex modifications of effective modern
optimizers improve the quality of learning. Using the results of
simulation, we can conclude that the combination of CReLU
activation and Adam optimizer is preferable under conditions
of our experiment.
•
The new theoretical study and experiments are needed for
tuning better parameters in the learning framework and
establishing the behavior for such networks.
11
THANK YOU FOR YOUR TIME
This study explores innovative methods in training complex-valued neural networks, including a model of complex-valued neurons, network architecture, error analysis, Adam optimizer, gradient calculation, and activation function selection. Simulation results compare real-valued and complex-valued networks in solving the hard XOR problem, demonstrating improved performance with new activation functions and optimizers like CReLU and Adam. The conclusions highlight the efficacy of feed-forward complex-valued neural networks for nonlinear mappings and suggest further exploration to optimize learning frameworks.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
NEW APPROACHES IN THE LEARNING OF COMPLEX-VALUED NEURAL NETWORKS V L A D Y S L A V K O T S O V S K Y , A N A T O L I Y B A T Y U K , M A K S Y M Y U R C H E N K O
SIMULATION We applied the complex backpropagation algorithm for solving the hard XOR problem in the case of 50features. We compared the performance of the real-valued and complex-valued four-layer networks consisting of neurons with sigmoidal and ReLU-like activation functions. We used 50-20- 10-1 topology in the real case and 25-20-10-1 topology in the complex case. Each network was trained in on-line mode on 100 different learning samples. The number of epochs was equal to 1000. We use q = 1 for complex rational sigmoid. 9
SIMULATION The parameter 1 and 2 were initialized to 0.9 and 0.999, respectively. The following presents average performance mea- sures in the case of learning using stochastic gradient descent (SGD) and Adam optimizers, respectively. Average accuracy (in %) Activation function Training set SGD Test set SGD Training set Adam Test set Adam Real sigmoid 68.15 53.24 74.03 62.12 Splitted complex sigmoid 71.92 62.25 82.88 67.03 Complex rational sigmoid 73.01 62.19 85.35 67.76 CReLU 74.96 67.05 92.43 81.51 10
CONCLUSIONS Feed-forward complex-valued neural networks give a simple and powerful architecture approximation of nonlinear mappings. The new activation functions and the complex modifications of effective modern optimizers improve the quality of learning. Using the results of simulation, we can conclude that the combination of CReLU activation and Adam optimizer is preferable under conditions of our experiment. The new theoretical study and experiments are needed for tuning better parameters in the learning framework and establishing the behavior for such networks. providing an adequate 11
