Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Linear and Logistic Regression Parameters & Improve Documentation #8982

Merged
merged 7 commits into from
Jan 9, 2025
Merged

Update Linear and Logistic Regression Parameters & Improve Documentation #8982

Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
aef0607
Update Linear and Logistic Regression Parameters & Improve Documentation
rithin-pullela-aws Dec 23, 2024
cb33640
Merge branch 'main' into linear-regression
rithin-pullela-aws Dec 23, 2024
338d456
Merge branch 'main' into linear-regression
rithin-pullela-aws Dec 26, 2024
47d5745
Update _ml-commons-plugin/algorithms.md
rithin-pullela-aws Jan 2, 2025
7e821ed
Added Tribuo documentation links to optimizers, Objective types, and …
rithin-pullela-aws Jan 8, 2025
f45547a
Apply suggestions from code review
kolchfa-aws Jan 9, 2025
e8682ee
Merge branch 'main' into linear-regression
kolchfa-aws Jan 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 29 additions & 14 deletions _ml-commons-plugin/algorithms.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,20 +59,31 @@

## Linear regression

Linear regression maps the linear relationship between inputs and outputs. In ML Commons, the linear regression algorithm is adopted from the public machine learning library [Tribuo](https://tribuo.org/), which offers multidimensional linear regression models. The model supports the linear optimizer in training, including popular approaches like Linear Decay, SQRT_DECAY, [ADA](https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf), [ADAM](https://tribuo.org/learn/4.1/javadoc/org/tribuo/math/optimisers/Adam.html), and [RMS_DROP](https://tribuo.org/learn/4.1/javadoc/org/tribuo/math/optimisers/RMSProp.html).
Linear regression maps the linear relationship between inputs and outputs. In ML Commons, the linear regression algorithm is adopted from the public machine learning library [Tribuo](https://tribuo.org/), which offers multidimensional linear regression models. The model supports the linear optimizer in training, including popular approaches like Linear Decay, SQRT_DECAY, [ADA](https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf), [ADAM](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/Adam.html), and [RMS_PROP](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/RMSProp.html).

**Optimisers supported:** [SIMPLE_SGD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.html#:~:text=learning%20rate%20SGD.-,getSimpleSGD,-public%20static%C2%A0), [LINEAR_DECAY_SGD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.html#:~:text=linear%20decay%20SGD.-,getLinearDecaySGD,-public%20static%C2%A0), [SQRT_DECAY_SGD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.html#:~:text=sqrt%20decay%20SGD.-,getSqrtDecaySGD,-public%20static%C2%A0), [ADA_GRAD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/AdaGrad.html), [ADA_DELTA](https://tribuo.org/learn/4.1/javadoc/org/tribuo/math/optimisers/AdaDelta.html), [ADAM](https://tribuo.org/learn/4.1/javadoc/org/tribuo/math/optimisers/Adam.html), and [RMS_PROP](https://tribuo.org/learn/4.1/javadoc/org/tribuo/math/optimisers/RMSProp.html).

Check warning on line 64 in _ml-commons-plugin/algorithms.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.AcronymParentheses] 'ADA': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.

Raw Output:
{"message": "[OpenSearch.AcronymParentheses] 'ADA': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.", "location": {"path": "_ml-commons-plugin/algorithms.md", "range": {"start": {"line": 64, "column": 510}}}, "severity": "WARNING"}
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
**Objectives supported:** [ABSOLUTE_LOSS](https://tribuo.org/learn/4.2/javadoc/org/tribuo/regression/sgd/objectives/AbsoluteLoss.html), [HUBER](https://tribuo.org/learn/4.2/javadoc/org/tribuo/regression/sgd/objectives/Huber.html), and [SQUARED_LOSS](https://tribuo.org/learn/4.2/javadoc/org/tribuo/regression/sgd/objectives/SquaredLoss.html).
**momentum_type supported:** [STANDARD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.Momentum.html#STANDARD:~:text=No%20momentum.-,STANDARD,-public%20static%20final), and [NESTEROV](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.Momentum.html#STANDARD:~:text=Standard%20momentum.-,NESTEROV,-public%20static%20final).
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

### Parameters

Parameter | Type | Description | Default value
:--- |:--- | :--- | :---
`learningRate` | Double | The initial step size used in an iterative optimization algorithm. | `0.01`
`momentumFactor` | Double | The extra weight factors that accelerate the rate at which the weight is adjusted. This helps move the minimization routine out of local minima. | `0`
`target` | String | The name of the target variable to predict. Identifies which feature the model will learn to predict during training. | `NA`
`learning_rate` | Double | The initial step size used in an iterative optimization algorithm. | `0.01`
`momentum_factor` | Double | The extra weight factors that accelerate the rate at which the weight is adjusted. This helps move the minimization routine out of local minima. | `0`
`epsilon` | Double | The value for stabilizing gradient inversion. | `1.00E-06`
`beta1` | Double | The exponential decay rates for the moment estimates. | `0.9`
`beta2` | Double | The exponential decay rates for the moment estimates. | `0.99`
`decayRate` | Double | The Root Mean Squared Propagation (RMSProp). | `0.9`
`momentumType` | String | The defined Stochastic Gradient Descent (SGD) momentum type that helps accelerate gradient vectors in the right directions, leading to a fast convergence.| `STANDARD`
`optimizerType` | String | The optimizer used in the model. | `SIMPLE_SGD`
`decay_rate` | Double | The Root Mean Squared Propagation (RMSProp). | `0.9`

Check failure on line 78 in _ml-commons-plugin/algorithms.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: RMSProp. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: RMSProp. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_ml-commons-plugin/algorithms.md", "range": {"start": {"line": 78, "column": 60}}}, "severity": "ERROR"}
`momentum_type` | String | The defined Stochastic Gradient Descent (SGD) momentum type that helps accelerate gradient vectors in the right directions, leading to a fast convergence.| `STANDARD`
`optimiser` | String | The optimizer used in the model. | `SIMPLE_SGD`
`objective` | String | The Objective function used. | `SQUARED_LOSS`
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
`epochs` | Integer | The number of iterations. | `5`|
`batch_size` | Integer | The size of min batches. | `1`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The minimum batch size"?

kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
`logging_interval` | Integer | The frequency of logging during training iterations. Set to -1 to disable logging. | `-1`
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
`seed` | Long | Random seed for reproducible results. Controls the initialization of random number generators. | `12345`
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved



### Supported APIs
Expand Down Expand Up @@ -412,23 +423,27 @@

A classification algorithm, logistic regression models the probability of a discrete outcome given an input variable. In ML Commons, these classifications include both binary and multi-class. The most common is the binary classification, which takes two values, such as "true/false" or "yes/no", and predicts the outcome based on the values specified. Alternatively, a multi-class output can categorize different inputs based on type. This makes logistic regression most useful for situations where you are trying to determine how your inputs fit best into a specified category.

**Optimisers supported:** [SIMPLE_SGD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.html#:~:text=learning%20rate%20SGD.-,getSimpleSGD,-public%20static%C2%A0), [LINEAR_DECAY_SGD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.html#:~:text=linear%20decay%20SGD.-,getLinearDecaySGD,-public%20static%C2%A0), [SQRT_DECAY_SGD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.html#:~:text=sqrt%20decay%20SGD.-,getSqrtDecaySGD,-public%20static%C2%A0), [ADA_GRAD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/AdaGrad.html), [ADA_DELTA](https://tribuo.org/learn/4.1/javadoc/org/tribuo/math/optimisers/AdaDelta.html), [ADAM](https://tribuo.org/learn/4.1/javadoc/org/tribuo/math/optimisers/Adam.html), and [RMS_PROP](https://tribuo.org/learn/4.1/javadoc/org/tribuo/math/optimisers/RMSProp.html).
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
**Objectives supported:** [HINGE](https://tribuo.org/learn/4.2/javadoc/org/tribuo/classification/sgd/objectives/Hinge.html), and [LOGMULTICLASS](https://tribuo.org/learn/4.2/javadoc/org/tribuo/classification/sgd/objectives/LogMulticlass.html).
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
**momentum_type supported:** [STANDARD](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.Momentum.html#STANDARD:~:text=No%20momentum.-,STANDARD,-public%20static%20final), and [NESTEROV](https://tribuo.org/learn/4.2/javadoc/org/tribuo/math/optimisers/SGD.Momentum.html#STANDARD:~:text=Standard%20momentum.-,NESTEROV,-public%20static%20final).
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

### Parameters

| Parameter | Type | Description | Default value |
|---|---|---|---|
| `learningRate` | Double | The initial step size used in an iterative optimization algorithm. | `1` |
| `momentumFactor` | Double | The extra weight factors that accelerate the rate at which the weight is adjusted. This helps move the minimization routine out of local minima. | `0` |
| `learning_rate` | Double | The initial step size used in an iterative optimization algorithm. | `1` |
| `momentum_factor` | Double | The extra weight factors that accelerate the rate at which the weight is adjusted. This helps move the minimization routine out of local minima. | `0` |
| `epsilon` | Double | The value for stabilizing gradient inversion. | `0.1` |
| `beta1` | Double | The exponential decay rates for the moment estimates. | `0.9` |
| `beta2` | Double | The exponential decay rates for the moment estimates. | `0.99` |
| `decayRate` | Double | The Root Mean Squared Propagation (RMSProp). | `0.9` |
| `momentumType` | String | The Stochastic Gradient Descent (SGD) momentum that helps accelerate gradient vectors in the right direction, leading to faster convergence between vectors. | `STANDARD` |
| `optimizerType` | String | The optimizer used in the model. | `AdaGrad` |
| `decay_rate` | Double | The Root Mean Squared Propagation (RMSProp). | `0.9` |

Check failure on line 439 in _ml-commons-plugin/algorithms.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: RMSProp. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: RMSProp. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_ml-commons-plugin/algorithms.md", "range": {"start": {"line": 439, "column": 62}}}, "severity": "ERROR"}
| `momentum_type` | String | The Stochastic Gradient Descent (SGD) momentum that helps accelerate gradient vectors in the correct direction, leading to faster convergence between vectors. | `STANDARD` |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On line 79, "correct direction, leading to faster convergence between vectors" is "right directions, leading to a fast convergence". Please make consistent.

kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
| `optimiser` | String | The optimizer used in the model. | `ADA_GRAD` |
rithin-pullela-aws marked this conversation as resolved.
Show resolved Hide resolved
| `target` | String | The target field. | null |
| `objectiveType` | String | The objective function type. | `LogMulticlass` |
| `objective` | String | The objective function type. | `LOGMULTICLASS` |
| `epochs` | Integer | The number of iterations. | `5` |
| `batchSize` | Integer | The size of min batches. | `1` |
| `loggingInterval` | Integer | The interval of logs lost after many iterations. The interval is `1` if the algorithm contains no logs. | `1000` |
| `batch_size` | Integer | The size of min batches. | `1` |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The minimum batch size"?

kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
| `logging_interval` | Integer | The interval of logs lost after many iterations. The interval is `1` if the algorithm contains no logs. | `1000` |

### Supported APIs

Expand Down
Loading