准备数据

具体官网地址，请点击这里

ImageNet官网

训练数据集：ILSVRC_img_train.tar

验证数据集：ILSVRC_img_val.tar

数据解压

sudo tar –xvf ILSVRC_img_train.tar -C ./train

sudo tar –xvf ILSVRC_img_val.tar -C ./val

对于val数据集，解压以后是所有的验证集图片，共50000张，大约6.3G。

对于train数据集，解压后是1000个tar文件，每个tar文件表示1000类里的一个类，共138G，对于1000个子tar，需要再次解压，解压脚本unzip.sh如下

dir=/home/satisfie/imagenet/train #satisfie 是我的用户名for x in `ls *.tar`dofilename=`basename $x .tar` #注意空格mkdir $filenametar -xvf $x -C ./$filenamedone12345671234567

i7 6700K配合我的500G固态硬盘解压超快，到这原始数据就准备好了，分别放在

/home/satisfie/imagenet/train：里面有1000个文件夹，每个文件夹下为JPG图片

/home/satisfie/imagenet/val：里面有验证集的50000张图片

接下来下载标签等其他说明数据~~~

下载其他数据

进入大caffe根目录，执行/data/ilsvrc12/get_ilsvrc_aux.sh下载其他数据，包括

det_synset_words.txtsynset_words.txt— 1000个类别的文件夹名称及真是物体的名称，比如 “n01440764 tench Tinca tinca”,在训练中，这些都当做一个类别。synsets.txt— 1000个类别的文件夹名称,比如”n01440764”…train.txt— 1000个类别每张图片的名字及其标签，比如 “n01440764/n01440764_10026.JPEG 0” 共有1281167张图片val.txt— 同上，总共有50000张。比如“ILSVRC_val_00000001.JPEG 65”test.txt— 同上，为测试集合,总有100000张imagenet_mean.binaryproto— 模型图片的各个通道均值imagenet.bet.pickle

模型的训练

训练数据准备

由于转化为lmdb数据库格式需要耗费较大的空间，且不支持shuffle等操作，所以这里直接读取原图片，使用的类型是ImageData，具体看下面的prototxt

其中的train_new.txt中对每张图片的加上了绝对值路径，这样才能被读取。

使用sed命令即可，

sed 's/^/\/home\/satisfie\/imagenet\/val\/&/g' val.txt >val_new.txt11

VGG_train_val.prototxt

name: "VGG_ILSVRC_16_layers"layer {name: "data"type: "ImageData"include {phase: TRAIN}transform_param {#crop_size: 224mean_value: 104mean_value: 117mean_value: 123mirror: true}image_data_param {source: "/home/satisfie/imagenet/train_new.txt"batch_size: 8new_height: 224new_width: 224}top: "data"top: "label"}layer {name: "data"type: "ImageData"include {phase: TEST}transform_param {#crop_size: 224mean_value: 104mean_value: 117mean_value: 123mirror: false}image_data_param {source: "/home/satisfie/imagenet/val_new.txt"batch_size: 4new_height: 224new_width: 224}top: "data"top: "label"}layer {bottom: "data"top: "conv1_1"name: "conv1_1"type: "Convolution"convolution_param {num_output: 64pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv1_1"top: "conv1_1"name: "relu1_1"type: "ReLU"}layer {bottom: "conv1_1"top: "conv1_2"name: "conv1_2"type: "Convolution"convolution_param {num_output: 64pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv1_2"top: "conv1_2"name: "relu1_2"type: "ReLU"}layer {bottom: "conv1_2"top: "pool1"name: "pool1"type: "Pooling"pooling_param {pool: MAXkernel_size: 2stride: 2}}layer {bottom: "pool1"top: "conv2_1"name: "conv2_1"type: "Convolution"convolution_param {num_output: 128pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 0}param {lr_mult: 0}}layer {bottom: "conv2_1"top: "conv2_1"name: "relu2_1"type: "ReLU"}layer {bottom: "conv2_1"top: "conv2_2"name: "conv2_2"type: "Convolution"convolution_param {num_output: 128pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv2_2"top: "conv2_2"name: "relu2_2"type: "ReLU"}layer {bottom: "conv2_2"top: "pool2"name: "pool2"type: "Pooling"pooling_param {pool: MAXkernel_size: 2stride: 2}}layer {bottom: "pool2"top: "conv3_1"name: "conv3_1"type: "Convolution"convolution_param {num_output: 256pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv3_1"top: "conv3_1"name: "relu3_1"type: "ReLU"}layer {bottom: "conv3_1"top: "conv3_2"name: "conv3_2"type: "Convolution"convolution_param {num_output: 256pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv3_2"top: "conv3_2"name: "relu3_2"type: "ReLU"}layer {bottom: "conv3_2"top: "conv3_3"name: "conv3_3"type: "Convolution"convolution_param {num_output: 256pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv3_3"top: "conv3_3"name: "relu3_3"type: "ReLU"}layer {bottom: "conv3_3"top: "pool3"name: "pool3"type: "Pooling"pooling_param {pool: MAXkernel_size: 2stride: 2}}layer {bottom: "pool3"top: "conv4_1"name: "conv4_1"type: "Convolution"convolution_param {num_output: 512pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv4_1"top: "conv4_1"name: "relu4_1"type: "ReLU"}layer {bottom: "conv4_1"top: "conv4_2"name: "conv4_2"type: "Convolution"convolution_param {num_output: 512pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv4_2"top: "conv4_2"name: "relu4_2"type: "ReLU"}layer {bottom: "conv4_2"top: "conv4_3"name: "conv4_3"type: "Convolution"convolution_param {num_output: 512pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv4_3"top: "conv4_3"name: "relu4_3"type: "ReLU"}layer {bottom: "conv4_3"top: "pool4"name: "pool4"type: "Pooling"pooling_param {pool: MAXkernel_size: 2stride: 2}}layer {bottom: "pool4"top: "conv5_1"name: "conv5_1"type: "Convolution"convolution_param {num_output: 512pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv5_1"top: "conv5_1"name: "relu5_1"type: "ReLU"}layer {bottom: "conv5_1"top: "conv5_2"name: "conv5_2"type: "Convolution"convolution_param {num_output: 512pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv5_2"top: "conv5_2"name: "relu5_2"type: "ReLU"}layer {bottom: "conv5_2"top: "conv5_3"name: "conv5_3"type: "Convolution"convolution_param {num_output: 512pad: 1kernel_size: 3weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "conv5_3"top: "conv5_3"name: "relu5_3"type: "ReLU"}layer {bottom: "conv5_3"top: "pool5"name: "pool5"type: "Pooling"pooling_param {pool: MAXkernel_size: 2stride: 2}}layer {bottom: "pool5"top: "fc6"name: "fc6"type: "InnerProduct"inner_product_param {num_output: 4096weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "fc6"top: "fc6"name: "relu6"type: "ReLU"}layer {bottom: "fc6"top: "fc6"name: "drop6"type: "Dropout"dropout_param {dropout_ratio: 0.5}}layer {bottom: "fc6"top: "fc7"name: "fc7"type: "InnerProduct"inner_product_param {num_output: 4096weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {bottom: "fc7"top: "fc7"name: "relu7"type: "ReLU"}layer {bottom: "fc7"top: "fc7"name: "drop7"type: "Dropout"dropout_param {dropout_ratio: 0.5}}layer {name: "fc8"bottom: "fc7"top: "fc8"type: "InnerProduct"inner_product_param {num_output: 1000weight_filler {type: "xavier"}bias_filler {type: "constant"value: 0}}param {lr_mult: 1decay_mult :1}param {lr_mult: 2decay_mult: 0}}layer {name: "loss"type: "SoftmaxWithLoss"bottom: "fc8"bottom: "label"top: "loss/loss"}layer {name: "accuracy/top1"type: "Accuracy"bottom: "fc8"bottom: "label"top: "accuracy@1"include: { phase: TEST }accuracy_param {top_k: 1}}layer {name: "accuracy/top5"type: "Accuracy"bottom: "fc8"bottom: "label"top: "accuracy@5"include: { phase: TEST }accuracy_param {top_k: 5}}1234567891011121314151617181922232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811911122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200022032042052062072082092102112122132142152162172182192122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464512345678910111213141516171819222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119111221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992000220320420520620720820921021121221321421521621721821921222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645

solver.prototxt

net: "models/vgg/train_val.prototxt"test_iter: 10000test_interval: 40000test_initialization: falsedisplay: 200base_lr: 0.0001lr_policy: "step"stepsize: 320000gamma: 0.96max_iter: 10000000momentum: 0.9weight_decay: 0.0005snapshot: 800000snapshot_prefix: "models/vgg/vgg"solver_mode: GPU123456789101112131415123456789101112131415

finetuning

模型太大，试了下，在GTX980的4G显存下，batchsize只能设置为8或者16这么小。。。

大模型还是得服务器并行，直接在原有的模型上finetuning

VGG_ILSVRC_16_layers_deploy.prototxt

#!/usr/bin/env shset -eTOOLS=./build/toolsGLOG_logtostderr=0 GLOG_log_dir=models/vgg/Log/ \$TOOLS/caffe train \--solver=models/vgg/solver.prototxt \--weights models/vgg/VGG_ILSVRC_16_layers.caffemodel