Inspirations from Worms at NIPS2017
NIPS2017 is my first time to the conference. I would like to share some thoughts and inspirations from the Workshop on Worm's Neural Information Processing. Hopefully it would be useful for others' research.
C. Elegans is a worm species. What's special about it is that all of its neurons have been completely mapped out. Although the worm only has 302 neurons, it can perform most things animals do - navigating, mating, finding food etc. So the Workshop on Worm's Neural Information Processing tried to bridge between the neural science research on C. Elegans and machine learning. I personally had never heard about C. Elegans before the workshop. But the workshop turned out to be really interesting and inspiring. Below are some interesting points the great speakers came across that I found not entirely consistent with the current popular design of machine learning models, thus worth sharing.
Sensory layer does more than embedding
Professor Netta Cohen gave a talk about C. Elegans search for salt. She showed that the sensory layer in C. Elegans is adaptive to its experience. So the simple example was that if the worm is on salt, it will become less attracted to salt; if it is not around salt any more, it will become attracted by salt again. As a result, there seems to be recurrent computation between the internal sensory states. This kind of computation certainly is doing more than embedding. Indeed, they did some experiment on pothole detection using a simulated adaptive navigation [1].
Activation may be distributed to more than one neurons
Dr. William R. Schafer gave a talk about the effect of removing neurons on C. Elegans' muscle control. They found different types of mapping between neurons to the muscle movement. Some neuron is necessary to certain movement, while some other neuron is not individually necessary. It is not clear why the differences, but it is definitely worth looking into. However, I realize this is not inconsistent with MLP where the effect of removal of certain neuron on activation depends on the weight. Abstract of the paper on this topic can be found here [2].
Recurrent structure of neurons
Professor Radu Grosu presented his team's work on robot parking with worm inspired neural networks. There are two things make this neural network design really interesting. The first is that the network only has about 10 neurons. Secondly, each neuron is like a flip-flop. There are recurrent computations within groups of neurons and also from a single neuron direct back to itself. The network was trained using some genetic algorithms. But Ramin M. Hasani and Mathias Lechner (both are contributors to this work) mentioned they implemented this on tensorflow using some gradient method which increased the training speed significantly.
It seems the recurrent structure can really help with reducing the number of neurons. I believe the team will publish their work soon (hopefully the code as well). I can't wait to read more about this.
Holistic design in Biology
Another interesting issue pointed out during the final panel discussion is that the neurons and body of an organism are usually highly synchronized with each other. Maybe this is not something we need to concern if we just want to design a model that does not control a body. But if it is an intelligent system for a robot, it may be worth thinking about what constitutes this synergy and what's the benefit of it.
Conclusion
Although I don't think AI will be designed exactly the same way as any existing living organism, the work of nature can still give us a lot of inspirations. I remember hearing the argument that to achieve what human can do we should start small to achieve what a rat can do first. Thus, it may worth a while looking into C. Elegans first.
References
NIPS2017 是我第一次参加这个会议。我想分享一些来自线虫神经信息处理工作坊的想法和启发，希望对其他人的研究有所帮助。
秀丽隐杆线虫（C. Elegans）是一种线虫。它的特别之处在于，其所有神经元都已被完全绘制出来。尽管这种线虫只有 302 个神经元，它却能完成大多数动物能做的事情——导航、交配、寻找食物等。因此，线虫神经信息处理工作坊试图在 C. Elegans 的神经科学研究与机器学习之间架起桥梁。在此之前，我个人从未听说过 C. Elegans。但这场工作坊非常有趣且富有启发性。以下是几位优秀演讲者提出的一些有趣观点——我发现它们与当前流行的机器学习模型设计并不完全一致，因此值得分享。
感觉层不只是嵌入
Netta Cohen 教授做了一场关于 C. Elegans 寻找盐分的报告。她展示了 C. Elegans 的感觉层会根据经验进行自适应。一个简单的例子是：如果线虫处于有盐的环境中，它对盐的吸引力会减弱；如果它不再处于有盐的环境中，它又会重新被盐吸引。因此，内部感觉状态之间似乎存在循环计算。这种计算显然不只是在做嵌入。事实上，他们使用模拟的自适应导航进行了一些坑洼检测实验 [1]。
激活可能分布在多个神经元上
William R. Schafer 博士做了一场关于移除神经元对 C. Elegans 肌肉控制影响的报告。他们发现神经元与肌肉运动之间存在不同类型的映射。某些神经元对特定运动是必需的，而另一些神经元则不是单独必需的。目前尚不清楚差异的原因，但绝对值得深入研究。不过，我意识到这与 MLP 并不矛盾——移除某个神经元对激活的影响取决于权重。相关论文摘要见此处 [2]。
神经元的循环结构
Radu Grosu 教授展示了他团队用受线虫启发的神经网络进行机器人停车的研究。这种神经网络设计有两个非常有趣的特点。第一，网络只有大约 10 个神经元。第二，每个神经元类似于一个触发器（flip-flop）。神经元组内存在循环计算，单个神经元也可以直接反馈到自身。网络使用遗传算法进行训练。但 Ramin M. Hasani 和 Mathias Lechner（均为该工作的贡献者）提到，他们用 TensorFlow 实现了某种梯度方法，显著提高了训练速度。
循环结构似乎真的有助于减少神经元数量。我相信该团队很快会发表他们的工作（希望代码也会公开）。我迫不及待想阅读更多相关内容。
生物学中的整体设计
在最后的 panel 讨论中，另一个有趣的问题是：生物体的神经元与身体通常高度协同。如果我们只是想设计一个不控制身体的模型，这可能不是我们需要关心的问题。但如果是一个用于机器人的智能系统，也许值得思考这种协同是如何形成的，以及它有什么好处。
结论
虽然我不认为 AI 会完全按照现有生物体的方式设计，但自然界的作品仍然能给我们带来很多启发。我记得曾听到一种说法：要实现人类能做的事情，应该从小处着手，先实现老鼠能做的事情。因此，研究 C. Elegans 也许是值得的。
Comments