CIFAR10 Model -Neural Network
Assignment-5
Created on May 29|Last edited on May 29
Comment
Model 1 & Model 2
- Model-1 SGD optimizer with learning rate = 0.001, momentum = 0.9 and cross-entropy as the loss function.
- Model-2 Adam optimizer with learning rate = 0.01 and cross-entropy as the loss function
meta
18m 19s
18m 10s
summary
gradients
graph_16conv1
bias
-
[-0.24363988637924194,-0.23799876868724823,-0.23235765099525452,-0.2267165333032608,-0.2210754156112671,-0.21543429791927338,-0.20979318022727966,-0.20415207743644714,-0.19851094484329224,-0.19286982715129852,-0.1872287094593048,-0.1815875917673111,-0.17594647407531738,-0.17030535638332367,-0.16466423869132996,-0.15902313590049744,-0.15338201820850372,-0.14774090051651,-0.1420997828245163,-0.13645866513252258,-0.13081754744052887,-0.12517642974853516,-0.11953531950712204,-0.11389420181512833,-0.10825308412313461,-0.1026119664311409,-0.09697084873914719,-0.09132973849773407,-0.08568862080574036,-0.08004750311374664,-0.07440638542175293,-0.06876526772975922,-0.0631241500377655,-0.05748303234577179,-0.051841914653778076,-0.04620080068707466,-0.04055968299508095,-0.034918565303087234,-0.02927745133638382,-0.023636333644390106,-0.017995215952396393,-0.012354099191725254,-0.006712982431054115,-0.0010718666017055511,0.004569251090288162,0.010210368782281876,0.01585148274898529,0.021492600440979004,0.027133718132972717,0.03277483582496643,0.038415953516960144,0.04405706748366356,0.04969818517565727,0.055339302867650986,0.0609804168343544,0.06662153452634811,0.07226265221834183,0.07790376991033554,0.08354488760232925,0.08918599784374237,0.09482711553573608,0.1004682332277298,0.10610935091972351,0.11175046861171722,0.11739158630371094]
-
[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1]
weight
-
[-0.14464111626148224,-0.1359540820121765,-0.12726706266403198,-0.11858003586530685,-0.10989300906658173,-0.1012059822678566,-0.09251895546913147,-0.08383192867040634,-0.07514490187168121,-0.06645787507295609,-0.05777084827423096,-0.04908382147550583,-0.0403967946767807,-0.03170976787805557,-0.023022741079330444,-0.014335714280605316,-0.005648687481880188,0.00303833931684494,0.011725366115570068,0.020412392914295197,0.029099419713020325,0.03778644651174545,0.04647347331047058,0.05516050010919571,0.06384752690792084,0.07253455370664597,0.0812215805053711,0.08990860730409622,0.09859563410282135,0.10728266090154648,0.1159696877002716,0.12465671449899673,0.13334375619888306,0.1420307755470276,0.1507178097963333,0.15940484404563904,0.16809186339378357,0.1767788827419281,0.18546591699123383,0.19415295124053955,0.20283997058868408,0.2115269899368286,0.22021402418613434,0.22890105843544006,0.2375880777835846,0.24627509713172913,0.25496214628219604,0.2636491656303406,0.2723361849784851,0.28102320432662964,0.28971022367477417,0.2983972728252411,0.3070842921733856,0.31577131152153015,0.32445836067199707,0.3331453800201416,0.34183239936828613,0.35051941871643066,0.3592064380645752,0.3678934872150421,0.37658050656318665,0.3852675259113312,0.3939545750617981,0.4026415944099426,0.41132861375808716]
-
[2,2,1,4,3,4,6,6,12,14,11,13,25,11,14,12,19,35,44,39,28,15,13,19,15,13,8,4,3,2,0,2,0,0,1,0,0,0,0,0,1,1,2,1,1,2,0,7,5,2,4,1,2,1,2,3,8,0,2,1,0,2,0,2]
graph_16conv2
bias
-
[-0.058601014316082,-0.05693819373846054,-0.05527536943554878,-0.05361254885792732,-0.05194972828030586,-0.050286903977394104,-0.048624083399772644,-0.046961262822151184,-0.045298442244529724,-0.043635621666908264,-0.041972797363996506,-0.040309976786375046,-0.038647156208753586,-0.03698433190584183,-0.03532151132822037,-0.03365869075059891,-0.03199587017297745,-0.03033304773271084,-0.02867022715508938,-0.02700740471482277,-0.02534458413720131,-0.0236817616969347,-0.02201893925666809,-0.02035611867904663,-0.018693294376134872,-0.017030471935868263,-0.015367651358246803,-0.013704828917980194,-0.012042008340358734,-0.010379185900092125,-0.00871636439114809,-0.007053542882204056,-0.005390722304582596,-0.0037279007956385612,-0.002065079053863883,-0.0004022573120892048,0.0012605641968548298,0.0029233861714601517,0.004586207680404186,0.006249029189348221,0.007911850698292255,0.00957467220723629,0.011237493716180325,0.012900315225124359,0.014563137665390968,0.016225960105657578,0.017888780683279037,0.019551601260900497,0.021214423701167107,0.022877246141433716,0.024540066719055176,0.026202889159321785,0.027865709736943245,0.029528532177209854,0.031191352754831314,0.03285417705774307,0.03451699763536453,0.03617981821298599,0.03784264251589775,0.03950546309351921,0.04116828367114067,0.04283110797405243,0.04449392855167389,0.04615674912929535,0.04781956970691681]
-
[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1]
weight
-
[-0.09354601055383682,-0.09070328623056412,-0.08786056190729141,-0.08501783758401871,-0.082175113260746,-0.0793323889374733,-0.07648966461420059,-0.07364694029092789,-0.07080421596765518,-0.06796149164438248,-0.06511876732110977,-0.06227604299783707,-0.05943331867456436,-0.056590594351291656,-0.05374787002801895,-0.050905145704746246,-0.04806242138147354,-0.045219697058200836,-0.04237697273492813,-0.039534248411655426,-0.03669152408838272,-0.033848799765110016,-0.03100607544183731,-0.028163351118564606,-0.0253206267952919,-0.022477902472019196,-0.01963517814874649,-0.016792453825473785,-0.01394972950220108,-0.011107005178928375,-0.00826428085565567,-0.005421556532382965,-0.00257883220911026,0.00026389211416244507,0.00310661643743515,0.005949340760707855,0.00879206508398056,0.011634789407253265,0.01447751373052597,0.017320238053798676,0.02016296237707138,0.023005686700344086,0.02584841102361679,0.028691135346889496,0.0315338596701622,0.034376583993434906,0.03721930831670761,0.040062032639980316,0.04290475696325302,0.045747481286525726,0.04859020560979843,0.051432929933071136,0.05427565425634384,0.05711837857961655,0.05996110290288925,0.06280382722616196,0.06564655154943466,0.06848927587270737,0.07133200019598007,0.07417472451925278,0.07701744884252548,0.07986017316579819,0.08270289748907089,0.0855456218123436,0.0883883461356163]
-
[2,1,3,0,1,1,1,3,6,6,10,5,4,7,9,14,16,11,13,7,18,16,22,14,21,12,23,15,15,28,26,17,21,20,29,30,19,30,23,30,33,25,37,34,32,23,23,25,20,27,25,10,11,6,2,3,5,3,2,1,0,0,3,1]
graph_16fc1
bias
-
[-0.010975795798003674,-0.01063584815710783,-0.010295899584889412,-0.009955951943993568,-0.00961600337177515,-0.009276055730879307,-0.008936107158660889,-0.008596159517765045,-0.008256211876869202,-0.007916264235973358,-0.00757631566375494,-0.0072363680228590965,-0.006896419450640678,-0.006556471809744835,-0.006216523703187704,-0.005876575596630573,-0.005536627024412155,-0.005196678917855024,-0.0048567308112978935,-0.00451678317040205,-0.004176834598183632,-0.0038368867244571447,-0.0034969388507306576,-0.0031569907441735268,-0.002817042637616396,-0.002477094531059265,-0.0021371464245021343,-0.0017971985507756472,-0.0014572504442185163,-0.0011173023376613855,-0.0007773544639348984,-0.00043740635737776756,-0.00009745731949806213,0.00024249072885140777,0.0005824387772008777,0.0009223867673426867,0.0012623348738998175,0.0016022829804569483,0.0019422308541834354,0.0022821789607405663,0.002622127067297697,0.002962075173854828,0.0033020232804119587,0.003641971154138446,0.003981919027864933,0.004321867600083351,0.004661815240979195,0.0050017633475363255,0.005341711454093456,0.005681659560650587,0.006021607667207718,0.006361555308103561,0.0067015038803219795,0.007041451521217823,0.007381399627774954,0.007721347734332085,0.008061295375227928,0.008401243016123772,0.00874119158834219,0.009081139229238033,0.009421087801456451,0.009761035442352295,0.010100983083248138,0.010440931655466557,0.010780880227684975]
-
[1,1,0,1,0,1,0,1,1,0,0,0,1,0,0,1,2,1,2,1,1,7,3,2,3,3,6,5,5,2,2,5,17,4,4,4,4,1,1,1,5,1,3,2,1,1,1,1,1,1,0,2,2,2,1,0,0,0,0,1,0,0,0,1]
weight
-
[-0.10869559645652771,-0.10565967112779617,-0.10262374579906464,-0.0995878130197525,-0.09655188769102097,-0.09351596236228943,-0.09048003703355789,-0.08744411170482635,-0.08440817892551422,-0.08137225359678268,-0.07833632826805115,-0.07530039548873901,-0.07226447016000748,-0.06922854483127594,-0.0661926195025444,-0.06315669417381287,-0.06012076884508133,-0.05708484351634979,-0.05404891446232796,-0.05101298913359642,-0.047977060079574585,-0.04494113475084305,-0.04190520942211151,-0.038869284093379974,-0.03583335503935814,-0.0327974297106266,-0.029761500656604767,-0.02672557532787323,-0.023689648136496544,-0.020653720945119858,-0.01761779561638832,-0.014581868425011635,-0.011545948684215546,-0.00851002149283886,-0.005474095232784748,-0.0024381689727306366,0.0005977582186460495,0.0036336854100227356,0.0066696107387542725,0.009705537930130959,0.012741465121507645,0.01577739231288433,0.018813319504261017,0.021849244832992554,0.02488517202436924,0.027921099215745926,0.030957024544477463,0.0339929535984993,0.037028878927230835,0.04006480425596237,0.04310073330998421,0.046136658638715744,0.04917258769273758,0.052208513021469116,0.05524443835020065,0.05828036367893219,0.061316292732954025,0.06435222178697586,0.0673881471157074,0.07042407244443893,0.07345999777317047,0.0764959305524826,0.07953184843063354,0.08256778120994568,0.08560370653867722]
-
[1,0,2,6,3,5,2,2,3,5,3,7,7,9,11,23,30,37,39,49,66,80,103,100,153,200,247,365,469,545,696,885,1103,1295,1395,3004,1183,1007,896,715,573,473,394,306,294,245,198,155,138,119,100,53,45,41,30,24,17,11,13,4,6,6,2,2]
graph_16fc2
bias
-
[-0.01997343823313713,-0.019448166713118553,-0.018922895193099976,-0.018397625535726547,-0.01787235401570797,-0.017347082495689392,-0.016821810975670815,-0.016296539455652237,-0.01577126979827881,-0.015245998278260231,-0.014720727689564228,-0.01419545616954565,-0.013670185580849648,-0.01314491406083107,-0.012619642540812492,-0.01209437195211649,-0.011569100432097912,-0.011043828912079334,-0.010518558323383331,-0.009993286803364754,-0.00946801621466875,-0.008942744694650173,-0.00841747410595417,-0.007892202585935593,-0.007366931531578302,-0.006841660477221012,-0.006316389422863722,-0.005791118368506432,-0.005265846848487854,-0.004740576259791851,-0.0042153047397732735,-0.003690033685415983,-0.003164762631058693,-0.0026394915767014027,-0.0021142205223441124,-0.0015889492351561785,-0.0010636781807988882,-0.0005384071264415979,-0.000013135839253664017,0.0005121352151036263,0.0010374062694609165,0.0015626773238182068,0.002087948378175497,0.002613219665363431,0.0031384907197207212,0.0036637617740780115,0.0041890330612659454,0.004714304115623236,0.005239575169980526,0.005764846224337816,0.0062901172786951065,0.006815388798713684,0.007340659387409687,0.007865930907428265,0.008391201496124268,0.008916473016142845,0.009441744536161423,0.00996701605618,0.010492286644876003,0.01101755816489458,0.011542828753590584,0.012068100273609161,0.012593371793627739,0.013118642382323742,0.013643912971019745]
-
[1,0,0,0,0,0,1,0,0,1,0,1,0,0,0,1,0,0,0,0,1,1,0,1,0,1,3,2,4,1,3,1,2,2,2,1,2,4,8,3,4,3,2,4,1,1,2,1,3,3,1,1,2,1,3,0,0,1,1,0,0,1,0,2]
weight
Group 2
2
From the above charts, we can see that as the number of epochs rises, the model gets well trained which increases accuracy and resulted in lowering down the Test Loss.
Here, we observed Test Accuracy increases up to 60% mark for model 1 and 48% for model 2. Here I able to take down Test loss has decreased from 200+ to 100 within in 60 epochs for model 1.
The upper left corner table indicates the time taken to execute each model and model parameters.
We can say model 2 is not able to perform as good as model 1 because of change in optimzer. We can expect that Adam optimizer was not able to reach global minima and hence not able to converge within limited number of epochs.
Model 3 & Model 4
(Squared Error Loss Functions Models)
- Model-3 SGD optimizer with learning rate = 0.001, momentum = 0.9 and squared error loss as the loss function.
- Model -4 Adam optimizer with learning rate = 0.01 and squared error loss as the loss function
Group 3
2
Here, we observe model-4 performed terribly badly. Model 4 is not able to train itself as it is trying to minimize squared loss error for class labels (Which doesn't make much sense) Model 3 were able to make some progress and reach to the test accuracy level of 40% within in 60 epochs. Though we can see that training loss is not minimizing and hence model failed to learn image parameters.
Resnet34 Model
- SGD optimizer, learning rate = 0.01 , momentum = 0.9 and cross-entropy as the loss function
Resnet
1
Here, we used pre-trained Resnet34 model. This model is trained on 34 layers comprising 21M parameters. We tried transfer learning and we able to see stark difference in results. We able to achieve Test Accuracy of 84%. As this is a deeper network it took a max time of 46 minute to train, in comparison other 4 models.
As this is mostly pretrained model, we observe training loss over the epochs is less in comparison to model 1. Model 1 started learning about image features from scratch, so change in trianing loss is higher.
Sample Predictions Samples
(Samples for all 5 models)
Resnet
5
As we know model 4 is very badly trained, we can see an error in predictions as well for image. Rest other models were able to predict this image as Class-9 correctly.
Add a comment