Added Supervised learning notes.

anonyknight · Jun 1, 2024 · fdabcda · fdabcda
1 parent bb6fc0a
commit fdabcda
Show file tree

Hide file tree

Showing 23 changed files with 127 additions and 4 deletions.
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/Computational-learning-theory.md b/docs/techs/OMSCS/CS7641-Machine-Learning/Computational-learning-theory.md
@@ -27,7 +27,7 @@ Learners need to eliminate as many as I can as a learner
 
 ### Steps
 
-1. Show what's irrelabant.
+1. Show what's irrelavant.
 2. Show what's relevant.
 
 ### Learn with Constrained Queries

diff --git a/...S/CS7641-Machine-Learning/RL-qlearning.md → ...ng/Reinforcement_Learning/RL-qlearning.md b/...S/CS7641-Machine-Learning/RL-qlearning.md → ...ng/Reinforcement_Learning/RL-qlearning.md
diff --git a/...techs/OMSCS/CS7641-Machine-Learning/RL.md → ...ine-Learning/Reinforcement_Learning/RL.md b/...techs/OMSCS/CS7641-Machine-Learning/RL.md → ...ine-Learning/Reinforcement_Learning/RL.md
@@ -9,7 +9,7 @@ tags: ['OMSCS', 'Algorithms', 'Machine-Learning']
 A problem:
 **The goal** is to choose an action in response to each data point.
 
-![RL Robot](./RL-Robot.png)
+![RL Robot](../images/RL-Robot.png)
 
 Robot: Sense Think Act cycle.
 

diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/Supervised_Learning/3.Neural_networks.md b/docs/techs/OMSCS/CS7641-Machine-Learning/Supervised_Learning/3.Neural_networks.md
@@ -0,0 +1,95 @@
+# Neural Networks
+
+Cell body -> Nevron -> Axon -> Synapses.
+
+Perceptron
+
+![NN](../images/SL3_Neural_Networks.png)
+
+## Artificial Neural Networks
+
+![NN](../images/SL3_ANN.png)
+
+![ANN](../images/SL 3 - Neural Networks Artificial Neural Networks.png)
+
+We need to pay attention to
+
+* Activation function
+* firing threshold
+
+### How powerful is perceptron unit?
+
+One activation function computes the half plane.
+
+![perceptron](../images/SL3_perceptron_Unit.png)
+
+What's the nice and short computing in the following?
+
+### Boolean: AND
+
+![perceptron_func](../images/SL3_perceptron_AND.png)
+
+### Boolean: OR
+
+![OR-Func](../images/SL3_perception_OR.png)
+
+### Boolean: Not
+
+![Unit-Not](../images/SL3_perception_NOT.png)
+
+### XOR as Perceptron Network
+
+![XOR](../images/SL3_perception_XOR.png)
+
+## Perceptron Training
+
+Given examples, find weights that map inputs to outputs.
+
+* Perceptron rule (**threshold**)
+* Gradient descent / delta rule (**un-thresholded**)
+
+### Perceptron rules
+
+Single Unit
+
+The halting problem for iterations.
+
+![perceptron-training](../images/SL3_perceptron_training.png)
+
+## Gradient Descent
+
+Avoid Non-linear separability issues.
+
+![Gradient-descent](../images/SL3_Gradient_descent.png)
+
+### Comparison of Learning rules
+
+#### Perceptron analysis
+
+guarantee to finite convergence only if linear separability
+$$
+\Delta W_i = \eta (y - y') x_i
+$$
+Eta = learning-rate
+
+y = target
+
+y' = output
+
+#### Gradient Descent analysis
+
+Calculus, robust, converge to local optimum
+
+$$
+\Delta W_i = \eta (y - a) x_i
+$$
+
+#### Comparing learning rules
+
+![comparison-learning-rules](../images/SL3_compariso_of_learning_rules.png)
+
+## Sigmoid - differentiable threshold
+
+![Sigmoid](../images/SL3_sigmoid.png)
+
+## Neural Network Sketch
diff --git a/...CS7641-Machine-Learning/decision_trees.md → ...ing/Supervised_Learning/decision_trees.md b/...CS7641-Machine-Learning/decision_trees.md → ...ing/Supervised_Learning/decision_trees.md
diff --git a/...1-Machine-Learning/supervised-learning.md → ...upervised_Learning/supervised-learning.md b/...1-Machine-Learning/supervised-learning.md → ...upervised_Learning/supervised-learning.md
@@ -1,6 +1,6 @@
 ---
 title: CS7641-SL
-summary: CS7641 Supervised Learning
+summary: CS7641 Supervised Learning types
 date: 'January 22, 2022'
 tags: ['OMSCS', 'Machine-Learning']
 ---
@@ -9,6 +9,31 @@ tags: ['OMSCS', 'Machine-Learning']
 
 Discrete labels vs continuous label
 
+### Regression
+
+Mapping continuous input to output.
+
+People's height **regresses** to the mean.
+
+* What prevent us from all being the same height?
+
+The noise from DNA combinations.
+
+The idea to find the regression to function approximation.
+
+## Linear regression
+
+### Find the best fit functions for data point
+
+Use calculus. Get assumption first, then find the optimal solution from calculus.
+
+* Historical facts
+* model selection and under/over fitting
+* Cross validation
+* Linear, polynomial regression
+* Best constant in terms of squared error: mean
+* Representation for regression.
+
 ## Classification Learning
 
 * **Instances**: (Input vectors of values)

diff --git a/...SCS/CS7641-Machine-Learning/supervised.md → ...earning/Supervised_Learning/supervised.md b/...SCS/CS7641-Machine-Learning/supervised.md → ...earning/Supervised_Learning/supervised.md
@@ -14,4 +14,3 @@ Give data set X, Y as the label, predict the Y' Label for the testing data set X
 * SVM
 * Artificial Neural Networks
 * Random Forests
-
diff --git a/...S/CS7641-Machine-Learning/unsupervised.md → ...ing/Unsupervised_Learning/unsupervised.md b/...S/CS7641-Machine-Learning/unsupervised.md → ...ing/Unsupervised_Learning/unsupervised.md
diff --git a/...MSCS/CS7641-Machine-Learning/RL-Robot.png → ...7641-Machine-Learning/images/RL-Robot.png b/...MSCS/CS7641-Machine-Learning/RL-Robot.png → ...7641-Machine-Learning/images/RL-Robot.png
diff --git a/...1-Machine-Learning/images/SL 3 - Neural Networks Artificial Neural Networks.png b/...1-Machine-Learning/images/SL 3 - Neural Networks Artificial Neural Networks.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_ANN.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_ANN.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_Gradient_descent.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_Gradient_descent.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_Neural_Networks.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_Neural_Networks.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_Neural_network_Sketch.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_Neural_network_Sketch.png
diff --git a/.../techs/OMSCS/CS7641-Machine-Learning/images/SL3_compariso_of_learning_rules.png b/.../techs/OMSCS/CS7641-Machine-Learning/images/SL3_compariso_of_learning_rules.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perception_NOT.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perception_NOT.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perception_OR.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perception_OR.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perception_XOR.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perception_XOR.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perceptron_AND.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perceptron_AND.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perceptron_Unit.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perceptron_Unit.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perceptron_training.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_perceptron_training.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_sigmoid.png b/docs/techs/OMSCS/CS7641-Machine-Learning/images/SL3_sigmoid.png
diff --git a/docs/techs/OMSCS/CS7641-Machine-Learning/index.md b/docs/techs/OMSCS/CS7641-Machine-Learning/index.md
@@ -15,6 +15,10 @@ tags: ['OMSCS', 'Algorithms', 'Machine-Learning']
 
 [Google AI Education](https://ai.google/education/)
 
+**Note**: The goal of Machine Learning is "Generalization".
+
+One meaning of "fold": A pen or enclosure in a field where livestock, esp. sheep, can be kept. (I don't think this meaning has anything to do with cross-validation, though.) Oh, wait, I found it: "consisting of so many parts or facets." So, *n-fold cross validation* means the data is in n parts. -ML
+
 ### Supervised Learning
 
 The data points contain X and labeled Y.
Original file line number	Diff line number	Diff line change
Expand Up		@@ -14,4 +14,3 @@ Give data set X, Y as the label, predict the Y' Label for the testing data set X
		* SVM
		* Artificial Neural Networks
		* Random Forests