7. 2nd Generation Neural Networks多层感知机(Multi-layer Perceptron, MLP)
超过1层的hidden layers(正确输出未知的层)
BP算法 [Rumelhart et al., 1986]
Compute error signal;
Then, back-propagate error signal to get derivatives for learning7David E. Rumelhart,, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). "Learning representations by back-propagating errors". Nature 323 (6088): 533–536
8. Error BackpropagationW is the parameter of the network; J is the objective function
Feedforward operationBack error propagationDavid E. Rumelhart,, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). "Learning representations by back-propagating errors". Nature 323 (6088): 533–536Output layerHidden layersInput layerTarget values
9. 2nd Generation Neural Networks理论上多层好
两层权重即可逼近任何连续函数映射
遗憾的是,训练困难
It requires labeled training data
Almost all data is unlabeled.
The learning time does not scale well
It is very slow in networks with multiple hidden layers.
It can get stuck in poor local optima
These are often quite good, but for deep nets they are far from optimal.9
10. 1990-2006更流行…Specific methods for specific tasks
Hand-crafted features (SIFT, LBP, HOG)
ML methods
SVM
Kernel tricks
Boosting
AdaBoost
kNN
Decision tree10Kruger et al. TPAMI’13
11. A Breakthrough Back to 20062006年,通过分层的、无监督预训练,终于获得了训练深层网络结构的能力11
12. A Breakthrough Back to 2006Hinton, G. E., Osindero, S. and Teh, Y., A fast learning algorithm for deep belief nets. Neural Computation 18:1527-1554, 2006
Hinton, G. E. and Salakhutdinov, R. R. (2006) Reducing the dimensionality of data with neural networks. Science, Vol. 313. no. 5786, pp. 504 - 507, 28 July 2006
Yoshua Bengio, Pascal Lamblin, Dan Popovici and Hugo Larochelle, Greedy Layer-Wise Training of Deep Networks, Advances in Neural Information Processing Systems 19 (NIPS 2006)
Marc’Aurelio Ranzato, Christopher Poultney, Sumit Chopra and Yann LeCun. Efficient Learning of Sparse Representations with an Energy-Based Model, Advances in Neural Information Processing Systems (NIPS 2006)12
13. 其实是有例外的——CNN卷积神经网络CNN
K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics, vol. 36, pp. 193–202, 1980
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989
Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 199813
14. 其实是有例外的——CNNNeocognitron 198014K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics, vol. 36, pp. 193–202, 1980Local Connection
19. 2012年计算机视觉的巨大进步ImageNet物体分类任务上
物体分类任务:1000类,1,431,167幅图像
1919862006DBN
ScienceSpeech20112012RankNameError rates(TOP5)Description1U. Toronto0.153Deep learning2U. Tokyo0.261Hand-crafted features and learning models.
Bottleneck.3U. Oxford0.2704Xerox/INRIA0.271BP
20. ImageNet with Deep CNN方法:大规模CNN网络20A. Krizhevsky, L. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NIPS, 2012.
21. ImageNet with Deep CNN方法:大规模CNN网络
650K神经元, 60M参数
Trained with BP on GPU
使用了各种技巧+dropout
ReLU, Data augment, contrast normalization,...
被Google收编(Jan 2013)
Google+ Photo Tagging(2013.5)21A. Krizhevsky, L. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NIPS, 2012.
25. ImageNet物体分类(2014)GoogLeNet [CVPR2015]
22个卷积层
Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. CVPR 2015
2519862006DBN
ScienceSpeech20112012BP20142013
26. ImageNet物体分类(2010-2014)ImageNet Top 5 Error Rate上的持续进步
2619862006DBN
ScienceSpeech20112012BP20142013
29. DL有多热Deep Learning for Vision
602篇文章中,仅标题中出现Deep的就有87篇,出现Convolution的47篇,出现Neural的40篇,出现Network的51篇,Recurrent 7篇
Going deeper,优化,无监督、自主学习…
Fully Convolutional Network(for segmentation等)
Vision and Language(for看图说话, Google, Fei-fei, Microsoft, UCB)
RNN with LSTM(for 时序处理)
Deep Learning for X(detection, metric learning, attribute, hash,…)
29
30. 计算机视觉的重大进步Vision and Language(Google, Microsoft, UCB)——看图说话:Minsky 60年前布置的作业30Show and Tell: A Neural Image Caption Generator (a work from Google)From Captions to Visual Concepts and Back (a work from Microsoft)Long-term Recurrent Convolutional Networks for Visual Recognition and Description(a work from UTA/UML/UCB)
31. 人脸识别上的进步正确率95.17% [D.Chen, X. Cao, F. Wen, J. Sun, CVPR13]
正确率97.35% [Y.Taigman, M. Yang, M.Ranzato, L. Wolf, CVPR14]
正确率99.47% [Y. Sun, X. Wang, and X. Tang, CVPR14]
正确率99.63% [F. Schroff, D. Kalenichenko, and J. Philbin, CVPR15]
3119862006DBN
ScienceSpeech20112012Face20142015BP在LFW上,过去2年错误率从5%下降到0.5%
(错300对错30对)
32. 人脸识别上的进步Labeled Face in the Wild (LFW)
非限定条件下的人脸识别
数据来源于因特网
国外名人,Yahoo新闻
广为人知的测试模式
训练集:无限制
验证任务测试集
共6000图像对32Huang G B, Ramesh M, Berg T, et al. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report, University of Massachusetts, Amherst, 2007.
33. 人脸识别上的进步2014: DeepFace [1] (Facebook)
大数据:4K人,4.4M图像33[1] Taigman Y, Yang M, Ranzato M A, et al. Deepface: Closing the gap to human-level performance in face verification. CVPR, 2014.
[2] Sun Y, Wang X, Tang X. Deeply learned face representations are sparse, selective, and robust. arXiv preprint, 2014.
41. DL时代的视觉处理方法任务
人工设计F(部分学习F)
End-to-end地学习F(全步骤学习)
Representation learning
Feature learning
Nonlinear transform learning41离散类标签
(分类问题)连续向量
(回归/估计)Credit to Dr. Xiaogang Wang
42. DL时代的视觉处理方法42Collect dataPreprocessing 1Feature designClassifierEvaluationPreprocessing 2…Collect dataFeature transformFeature transform…ClassifierDeep neural networkEvaluationvs.Credit to Dr. Xiaogang Wang
45. Perceptron 45Frank Rosenblatt(1957), The Perceptron--a perceiving and recognizing automaton. Report 85-460-1, Cornell Aeronautical Laboratory.
46. Perceptron算法 46F. Rosenblatt. The perceptron: A probabilistic model for informationstorage and organization in the brain. Psychological Review, 65:386-408, 1958
47. Perceptron算法 47
48. 前馈神经网络的BP学习算法David E. Rumelhart,, Geoffrey E. Hinton, and Ronald J. Williams. (Oct.1986). "Learning representations by back-propagating errors". Nature 323 (6088): 533–536(单独slides)