【编者按】上一期我们分享了一篇关于深度学习的文章，而深度学习最直接的对象就是神经网络。本次在深度学习的基础上更进一步，分享一篇麻省理工学院新闻网对于神经网络如何学习的最新研究成果的报道。这里的“彩票假说”也很有趣，如果有机会我们会在后续的文章中分享更具体的研究成果信息供探讨。
Neural Networks and the Lottery Ticket Hypothesis
神经网络与彩票假说
-05-07--3 MINUTES READ阅读约需3分钟
Deep neural networks, computer systems loosely modeled after the human brain, are found in the majority of AI-based products. As you wouldexpect, neural networks aren’t simple systems — they tend to be very large,require huge >
在大多数以人工智能为基础的产品中，人们都发现了模仿人脑松散建模的计算机系统--深度神经网络。如您所料，神经网络不是简单的系统，他们非常庞大，需求大量的数据集合和昂贵的设备，并且要耗费很多时间来训练。事实上，传统的神经网络会是很贵主儿。
Michael Carbin, an MIT Assistant Professor, and Jonathan Frankle, a PhD student and IPRI team member, responded to this issue in a paper titledThe Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.In this MIT CSAIL project, the researchers detail how these large neural netscontain smaller “subnetworks” that are up to 10 times smaller than the fullnetwork. The subnetworks can learn just as well, making equally precisepredictions sometimes faster than the full neural networks.This work may also have implications for transfer learning.
麻省理工学院助理教授Michael Carbin和一名博士研究生Jonathan Frankle及IPRI小组成员在其论文《彩票假设：寻找稀疏、可训练的神经网络》(The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks)中回应了该问题。在麻省理工学院这一CSAIL项目中，研究人员详细介绍了这些大型神经网络如何包含比完整网络小十倍的“子网络”。这些子网络也可以学习，有时甚至能比完整的神经网络更快地作出同样精确的预测。这一成果可能也会对迁移学习产生影响。
Image showing a neural network. Image in the public domain inWikimedia Commons.
图像显示了一个神经网络。来自于维基百科评论。
Understanding the Lottery Ticket Hypothesis
彩票假设
To explain how this works, the researchers comparetraditional deep learning methods to a lottery. In their analogy, traininglarge neural networks is akin to buying every possible lottery ticket toguarantee a win, even when only a few tickets are actually winners. However, training subnetworks would be like buying only the winning tickets. As Carbin states, the “remaining science is to figure how to identify the winning ticketswithout seeing the winning numbers first.”
为解释其工作原理，研究人员将传统的深度学习方法与彩票进行了比较。以此类推，训练大型神经网络购买所有可能的彩票以确保中奖，就算实际上只有几张彩票能中奖。但培训子网络就像只购买中奖彩票。如Carbin所说，“剩下的问题就是弄清楚如何在没看到中奖号码的前提下识别中奖彩票。”
Carbin and Frankle tested their lottery ticket hypothesis and the existence of subnetworks by performing a process called pruning, which involves eliminating unneeded connections from trained networks to fit them onlow-power devices. The pruned connections have the lowest network prioritization or weight.
Carbin和Frankle通过运行一个被称为“修剪”的程序来测试他们的彩票假设和子网络的存在，该过程涉及到从经训练的网络中消除不必要的链接，以使其适合低功耗设备。被修剪的链接的网络优先级和权重最低。
As Adam Conner-Simons states inMIT News, “[t]heir key innovation was the idea that connections that were pruned after the network was trained might never havebeen necessary at all.”
正如Adam Conner-Simons在《麻省理工新闻》(MIT News)中所说，“他们的主要创新在于网络训练后修剪链接可能完全没有必要这一想法。”
Researchers have not yet determined how to find subnetworks without building a full neural network and using pruning, but finding out how to move directly to the subnetwork step would save time and money. It wouldalso potentially enable individual programmers and not just large companies tocreate meaningful models. However, “[u]nderstanding the ‘lottery ticket hypothesis’ is likely to keep researchers busy for years to come,” notes Assistant Professor Daniel Roy inMIT News.
研究人员尚未确定如何不构建完整的神经网络且不修剪链接的情况下，找到子网络，但他们找到了怎样直接转移到子网络会更节省时间和金钱。这也可能使得单个程序员而不只是大型公司创建有意义的模型。据《麻省理工新闻》报道，助理教授DanielRoy指出“了解‘彩票假说’可能会使研究人员在未来几年忙碌起来。”
As for the researchers themselves, next on the agenda is determining why specific subnetworks are good at learning and how toefficiently find them.
对于研究人员自身而言，他们的下一步工作是研究子网络为什么善于学习，以及如何高效地找到他们。
原文出处Available athttps://internetpolicy.mit.edu/neural-networks-and-the-lottery-ticket-hypothesis/
