[DOLPHIN-94] Show both training/cross-validation error to check overfiting #95

dongjoon-hyun · 2015-08-27T23:07:05Z

This is the first draft described in #94 .
I will test and amend again after merging #88 .

dongjoon-hyun · 2015-08-28T04:58:46Z

Hi, @beomyeol . Could you review the design of this PR? I think you're the right person. I changed your Validator as ValidatorBase without asking your permission. :)

beomyeol · 2015-08-28T05:01:08Z

@dongjoon-hyun, I'll take a look at this.

beomyeol · 2015-08-28T05:58:47Z

@dongjoon-hyun, This looks good. I'm wondering if there is any chance to have difference between training validation and cross validation. If not, we don't need two separate TrainingValidator and CrossValidator classes, right?

dongjoon-hyun · 2015-08-28T06:10:39Z

Ur, I've heard that we need separate classes for REEF parameter injection.
@jsjason , did I misunderstand?

jsjason · 2015-08-28T07:01:07Z

@dongjoon-hyun is right, if we're going to use two separate classes. However, for this case I think we should just ~~include~~ stuff the two into a single Validator class. Having a inner abstract class doesn't seem so good.

dongjoon-hyun · 2015-08-28T07:31:28Z

Oh, I see.

dongjoon-hyun · 2015-08-30T01:15:55Z

Hi, @jsjason , @beomyeol . I merged into one. But, as a single Validator class, REEF complains like this again. How can I resolve this with this single Validator class? I thought it's a limitation of REEF. We should have two classes with same logics.

edu.snu.reef.dolphin.neuralnet.NeuralNetworkREEF.main main | REEF job completed: FAILED(org.apache.reef.tang.exceptions.ClassHierarchyException: Repeated constructor parameter detected.  Cannot inject constructor edu.snu.reef.dolphin.neuralnet.NeuralNetworkTask(edu.snu.reef.dolphin.core.DataParser,edu.snu.reef.dolphin.neuralnet.NeuralNetwork,int edu.snu.reef.dolphin.examples.ml.parameters.MaxIterations,edu.snu.reef.dolphin.neuralnet.NeuralNetworkTask$Validator,edu.snu.reef.dolphin.neuralnet.NeuralNetworkTask$Validator))

…iting

beomyeol · 2015-08-30T01:42:54Z

@dongjoon-hyun Oh, now I understand why you used two separate classes. I didn't know two arguments of the same classes cannot be injected. We can make a copy in the constructor of NeuralNetworkTask that needs public copy constructor of Validator. I think this may not look good since it need public copy constructor.
@jsjason @dongjoon-hyun I am okay to go back to have two separate classes If you are okay.

dongjoon-hyun · 2015-08-30T01:46:53Z

Thank you for your quick advice! @beomyeol .

@jsjason . Please give a guide with abstract class. @jsjason .
To avoid the duplication of logic, we need abstract class. But, if abstract class is not permitted, I will make two classes having the same logic.

jsjason · 2015-08-30T03:31:19Z

What I suggested was to create a Validator class that checks both training and validation data. It would have methods like getTrainingAccuracy and getValidationAccuracy. Yes, the logic gets repeated but the logic isn't so big, and you can create some private methods if the repetition gets too long.
Another way is to not use Tang injection: new Validator(). Even REEF uses new ...() when it needs more than one instance of the same class.
Yet another way is to declare some enum that separates training data and validation data, and receive it in Validator's methods.

Abstract classes are not preferred because having an abstract class means some classes share the same features, and this means it can be extracted to a separate class. The unshared part can be turned into an interface.

jsjason · 2015-08-30T03:34:55Z

You could even do this: declare a class that wraps two Validators, and receive an injection of that wrapper class. The Validators will have to be instantiated as new Validator(...) anyway.

dongjoon-hyun · 2015-08-30T03:41:07Z

Oh, thank you. @jsjason . I will apply the first suggestion!

jsjason · 2015-08-30T03:43:36Z

What about the second method of using new Validator()? All methods have their good points.

dongjoon-hyun · 2015-08-30T03:53:52Z

Okay. No problem. Thank you for quick response!

By the way, I have a question.
Escaping Tang means that we cat not store this in the Context in the future right?

bgchun · 2015-08-30T04:00:56Z

@dongjoon-hyun NamedObject will solve the issue discussed here; this is under way in REEF-31.

If you prefer to store Validators in Tang, you can use bindVolatile.

dongjoon-hyun · 2015-08-30T04:19:53Z

Thank you for your advice and pointer, @bgchun . REEF-31 looks like the best way. REEF 0.13 will have that? The fix version of REEF-31 is not assigned yet.

dongjoon-hyun · 2015-08-30T04:23:39Z

@jsjason . I hope I changed the code as your advice. Please correct me if I'm wrong.

By the way, the result shows us overfitting status.

~~ nnTask-0 | Iteration: 99
~~ nnTask-0 | Training Error: 0.014999986
~~ nnTask-0 | Cross Validation Error: 0.18
~~ nnTask-0 | # of validation inputs: 100

bgchun · 2015-08-30T04:24:42Z

My student's implementing it. I don't think it's ready for 0.13, which we
plan to do feature freeze at the end of September.

On Sun, Aug 30, 2015 at 1:19 PM, Dongjoon Hyun [email protected]
wrote:

Thank you for your advice and pointer, @bgchun https://github.com/bgchun
. REEF-31 looks like the best way. REEF 0.13 will have that? The fix
version of REEF-31 is not assigned yet.

—
Reply to this email directly or view it on GitHub
#95 (comment).

Byung-Gon Chun

jsjason · 2015-08-30T04:26:45Z

src/main/java/edu/snu/reef/dolphin/neuralnet/NeuralNetworkTask.java

-      LOG.log(Level.INFO, "# of validation inputs: {0}", String.valueOf(validator.getTotalNum()));
+      LOG.log(Level.INFO, "Training Error: {0}", String.valueOf(trainingValidator.getError()));
+      LOG.log(Level.INFO, "Cross Validation Error: {0}", String.valueOf(crossValidator.getError()));
+      LOG.log(Level.INFO, "# of validation inputs: {0}", String.valueOf(crossValidator.getTotalNum()));


We could use a line that shows us the number of training inputs, too.

You're right!

jsjason · 2015-08-30T04:28:55Z

@dongjoon-hyun Could the training error be too low because the number of training inputs is too small?

dongjoon-hyun · 2015-08-30T04:33:00Z

Right. The values means that current neural network configuration is much more complex and the data is too small. That's overfitting. But, this PR does not have anything to handle that.

beomyeol · 2015-08-30T04:37:59Z

@dongjoon-hyun Thank you for your experiment. To avoid overfitting, we can apply various techniques such as weight decaying and dropout later.

dongjoon-hyun · 2015-08-30T04:39:06Z

Sure, @beomyeol . And that's the reason why I'm implementing convolutional layers and pooling layers.

jsjason · 2015-08-30T04:45:26Z

@dongjoon-hyun The current changes look good for now. I don't think being unable to store Validators in Contexts would be a big problem yet. It'll depend on our design; if we restrict one network model to exist only in one task and not across several tasks, then it wouldn't make sense to have a Context-level Validator.

dongjoon-hyun · 2015-08-30T04:53:16Z

@jsjason , I totally agree with you.

I slightly worried and tried not to affect on #70 and #73 in Milestone 3.

Anyway, If you don't mind, please merge this~ :)

jsjason · 2015-08-30T04:53:20Z

My comment is a bit misleading. I meant that if we assume a certain model replica does not exist across more than one task in a given evaluator, then we wouldn't have to maintain a Context-level Validator. Partitioning a model across several evaluators is a different story.

dongjoon-hyun · 2015-08-30T04:54:27Z

Yep. I'm sure about that. It's my concern due to lack of knowledges. ;)

jsjason · 2015-08-30T04:55:30Z

I'm good. @bgchun and @beomyeol, are you okay with the changes too?

bgchun · 2015-08-30T04:58:35Z

+1

beomyeol · 2015-08-30T05:03:10Z

@jsjason I am good, too. Please merge this PR. :)

[DOLPHIN-94] Show both training/cross-validation error to check overfiting

dongjoon-hyun · 2015-08-30T05:11:14Z

Thank you!

Close #94 .

[DOLPHIN-94] Show both training/cross-validation error to check overf…

5d0031b

…iting

Remove Tang injection parts

48666ac

jsjason reviewed Aug 30, 2015
View reviewed changes

Print total number of training input

24b7518

jsjason added a commit that referenced this pull request Aug 30, 2015

Merge pull request #95 from dongjoon-hyun/DOLPHIN-94

194900d

[DOLPHIN-94] Show both training/cross-validation error to check overfiting

jsjason merged commit 194900d into snuspl:master Aug 30, 2015

jsjason deleted the DOLPHIN-94 branch August 30, 2015 05:07

dongjoon-hyun mentioned this pull request Aug 30, 2015

Show both training and cross-validation error rate together #94

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOLPHIN-94] Show both training/cross-validation error to check overfiting #95

[DOLPHIN-94] Show both training/cross-validation error to check overfiting #95

dongjoon-hyun commented Aug 27, 2015

dongjoon-hyun commented Aug 28, 2015

beomyeol commented Aug 28, 2015

beomyeol commented Aug 28, 2015

dongjoon-hyun commented Aug 28, 2015

jsjason commented Aug 28, 2015

dongjoon-hyun commented Aug 28, 2015

dongjoon-hyun commented Aug 30, 2015

beomyeol commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

bgchun commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

bgchun commented Aug 30, 2015

jsjason Aug 30, 2015

dongjoon-hyun Aug 30, 2015

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

beomyeol commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

bgchun commented Aug 30, 2015

beomyeol commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

[DOLPHIN-94] Show both training/cross-validation error to check overfiting #95

[DOLPHIN-94] Show both training/cross-validation error to check overfiting #95

Conversation

dongjoon-hyun commented Aug 27, 2015

dongjoon-hyun commented Aug 28, 2015

beomyeol commented Aug 28, 2015

beomyeol commented Aug 28, 2015

dongjoon-hyun commented Aug 28, 2015

jsjason commented Aug 28, 2015

dongjoon-hyun commented Aug 28, 2015

dongjoon-hyun commented Aug 30, 2015

beomyeol commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

bgchun commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

bgchun commented Aug 30, 2015

jsjason Aug 30, 2015

Choose a reason for hiding this comment

dongjoon-hyun Aug 30, 2015

Choose a reason for hiding this comment

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

beomyeol commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015

jsjason commented Aug 30, 2015

bgchun commented Aug 30, 2015

beomyeol commented Aug 30, 2015

dongjoon-hyun commented Aug 30, 2015