Reinforcement learning with one-shot memory

초록

In modern reinforcement learning to solve problems in virtual environments and realities including games, artificial neural networks are used as approximations. However, since the current artificial neural network is based on statistics, a large amount of data is required, so if there is no simulator, there are difficulties in using and applying it. Besides, making an accurate simulator is often a very expensive and hard problem in most domains we want to solve, and getting appropriate data and rewards from the environment is also not easy task, since they are mostly partially observable and relatively scarce. Due to these difficulties of acquiring sufficient data, there are certain limitations in utilizing artificial neural networks as function approximators in reinforcement learning. Therefore, by using the memory structure, which is case-based learning, a model that can perform fast learning in an environment with little data and sparse rewards. In the experiment, we challenged the Open AI Cartpole problem based on the existing policy gradient and memory. The advantage function that evaluates the gain, was implemented by transforming the one-shot learning memory structure. In subsequent experiments, the model showed poor performance on average due to large variance during training. However, by comparing the learning speed with other algorithms, it was confirmed that the top 10 and 5 scores obtained higher scores than other algorithms compared to the same episode within 100 small episodes. In conclusion, we found possibility that the method of using the memory structure can be effective for small data. And in the future, research on techniques to reduce the deviation of learning is needed. The model is expected to help the decision-making of artificial intelligence NPCs interacting with users.

키워드