GeneralGuidanceHung-yiLee李宏毅FrameworkofMLTrainingdata:Testingdata:(speaker)痛みを知れ了解痛苦吧SpeechRecognitionImageRecognitionSpeakerRecognitionMachineTranslationStep1:functionwithunknownStep2:definelossfromtrainingdataStep3:optimizationFrameworkofMLTraining:Trainingdata:Testingdata:UploadtoKagglelossontrainingdatalargesmallmodelbiasoptimizationmakeyourmodelcomplexNextLecturelossontestingdataoverfittingmismatchsmalllargeNotinHWs,exceptHW11makeyourmodelsimplermoretrainingdata(notinHWs)dataaugmentationtrade-offSplityourtrainingdataintotrainingsetandvalidationsetformodelselectionGeneralGuide•Themodelistoosimple.•Solution:redesignyourmodeltomakeitmoreflexibleMorefeaturesDeepLearning(moreneurons,layers)toosmall…ModelBiassmalllossfindaneedleinahaystack……butthereisnoneedlelossontrainingdatalargesmallmodelbiasoptimizationmakeyourmodelcomplexNextLecturelossontestingdataoverfittingmismatchsmalllargetrade-offSplityourtrainingdataintotrainingsetandvalidationsetformodelselectionGeneralGuideNotinHWs,exceptHW11makeyourmodelsimplermoretrainingdata(notinHWs)dataaugmentationOptimizationIssue•Largelossnotalwaysimplymodelbias.Thereisanotherpossibility…largeAneedleisinahaystack……Justcannotfindit.Aneedleisinahaystack……Justcannotfindit.toosmall…smalllossfindaneedleinahaystack……butthereisnoneedleOptimizationIssueModelBiasWhichone???ModelBiasv.s.OptimizationIssue•GainingtheinsightsfromcomparisonTestingDataOverfitting?TrainingDataOptimizationissueRef:http://arxiv.org/abs/1512.03385OptimizationIssue•Gainingtheinsightsfromcomparison•Startfromshallowernetworks(orothermodels),whichareeasiertooptimize.•Ifdeepernetworksdonotobtainsmallerlossontrainingdata,thenthereisoptimizationissue.•Solution:Morepowerfuloptimizationtechnology(nextlecture)Ref:http://arxiv.org/abs/1512.033851layer2layer3layer4layer5layer2017–20200.28k0.18k0.14k0.10k0.34klossontrainingdatalargesmallmodelbiasoptimizationmakeyourmodelcomplexNextLecturelossontestingdataoverfittingmismatchsmalllargetrade-off...
