论文标题
内部状态在经过普通语言培训的复发神经网络中的稳定性
Stability of Internal States in Recurrent Neural Networks Trained on Regular Languages
论文作者
论文摘要
我们提供了一项经验研究，旨在训练经过识别普通语言的复发性神经网络的稳定性。当将少量噪声引入激活函数时，复发层中的神经元倾向于饱和，以补偿可变性。在这个饱和的态度中，对网络激活的分析显示了一组类似于有限状态机器的离散状态的集群。我们表明，这些状态之间对输入符号的过渡是确定性和稳定的。网络显示任意长字符串的稳定行为，当将随机扰动应用于任何一个状态时，它们就能恢复，并且其演变会收敛到原始簇。该观察结果强化了网络作为有限自动机的解释，神经元或神经元组编码特定和有意义的输入模式。
We provide an empirical study of the stability of recurrent neural networks trained to recognize regular languages. When a small amount of noise is introduced into the activation function, the neurons in the recurrent layer tend to saturate in order to compensate the variability. In this saturated regime, analysis of the network activation shows a set of clusters that resemble discrete states in a finite state machine. We show that transitions between these states in response to input symbols are deterministic and stable. The networks display a stable behavior for arbitrarily long strings, and when random perturbations are applied to any of the states, they are able to recover and their evolution converges to the original clusters. This observation reinforces the interpretation of the networks as finite automata, with neurons or groups of neurons coding specific and meaningful input patterns.
