-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathgaussian-process-regression.html
39 lines (39 loc) · 15.4 KB
/
gaussian-process-regression.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
<!doctype html><html lang=en-uk><head><script data-goatcounter=https://ruivieira-dev.goatcounter.com/count async src=//gc.zgo.at/count.js></script><script src=https://unpkg.com/@alpinejs/[email protected]/dist/cdn.min.js></script><script src=https://unpkg.com/[email protected]/dist/cdn.min.js></script><script type=module src=https://ruivieira.dev/js/deeplinks/deeplinks.js></script><link rel=preload href=https://ruivieira.dev/lib/fonts/fa-brands-400.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=https://ruivieira.dev/lib/fonts/fa-regular-400.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=https://ruivieira.dev/lib/fonts/fa-solid-900.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=https://ruivieira.dev/fonts/firacode/FiraCode-Regular.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=preload href=https://ruivieira.dev/fonts/vollkorn/Vollkorn-Regular.woff2 as=font type=font/woff2 crossorigin=anonymous><link rel=stylesheet href=https://ruivieira.dev/css/kbd.css type=text/css><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge"><title>Gaussian Process Regression · Rui Vieira</title>
<link rel=canonical href=https://ruivieira.dev/gaussian-process-regression.html><meta name=viewport content="width=device-width,initial-scale=1"><meta name=robots content="all,follow"><meta name=googlebot content="index,follow,snippet,archive"><meta property="og:title" content="Gaussian Process Regression"><meta property="og:description" content="Example taken from the official Scikit-learn documentation1.
DataWe will start by generating a synthetic dataset. The true generative process is defined as $f(x)=x\sin(x)$.
import numpy as np np.random.seed(42) X = np.linspace(start=0, stop=10, num=1_000).reshape(-1, 1) y = np.squeeze(X * np.sin(X)) We will use this dataset in the next experiment to illustrate how Gaussian Process regression is working.
Example with noise-free targetIn this first example, we will use the true generative process without adding any noise."><meta property="og:type" content="article"><meta property="og:url" content="https://ruivieira.dev/gaussian-process-regression.html"><meta property="article:section" content="posts"><meta property="article:modified_time" content="2023-09-02T17:28:34+01:00"><meta name=twitter:card content="summary"><meta name=twitter:title content="Gaussian Process Regression"><meta name=twitter:description content="Example taken from the official Scikit-learn documentation1.
DataWe will start by generating a synthetic dataset. The true generative process is defined as $f(x)=x\sin(x)$.
import numpy as np np.random.seed(42) X = np.linspace(start=0, stop=10, num=1_000).reshape(-1, 1) y = np.squeeze(X * np.sin(X)) We will use this dataset in the next experiment to illustrate how Gaussian Process regression is working.
Example with noise-free targetIn this first example, we will use the true generative process without adding any noise."><link rel=stylesheet href=https://ruivieira.dev/css/styles.css><!--[if lt IE 9]><script src=https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js></script><script src=https://oss.maxcdn.com/respond/1.4.2/respond.min.js></script><![endif]--><link rel=icon type=image/png href=https://ruivieira.dev/images/favicon.ico></head><body class="max-width mx-auto px3 ltr" x-data="{currentHeading: undefined}"><div class="content index py4"><div id=header-post><a id=menu-icon href=#><i class="fas fa-eye fa-lg"></i></a>
<a id=menu-icon-tablet href=#><i class="fas fa-eye fa-lg"></i></a>
<a id=top-icon-tablet href=# onclick='$("html, body").animate({scrollTop:0},"fast")' style=display:none aria-label="Top of Page"><i class="fas fa-chevron-up fa-lg"></i></a>
<span id=menu><span id=nav><ul><li><a href=https://ruivieira.dev/>Home</a></li><li><a href=https://ruivieira.dev/blog/>Blog</a></li><li><a href=https://ruivieira.dev/draw/>Drawings</a></li><li><a href=https://ruivieira.dev/map/>All pages</a></li><li><a href=https://ruivieira.dev/search.html>Search</a></li></ul></span><br><div id=share style=display:none></div><div id=toc><h4>Contents</h4><nav id=TableOfContents><ul><li><a href=#data :class="{'toc-h2':true, 'toc-highlight': currentHeading == '#data' }">Data</a></li></ul></nav><h4>Related</h4><nav><ul><li class="header-post toc"><span class=backlink-count>1</span>
<a href=https://ruivieira.dev/machine-learning.html>Machine Learning</a></li></ul></nav></div></span></div><article class=post itemscope itemtype=http://schema.org/BlogPosting><header><h1 class=posttitle itemprop="name headline">Gaussian Process Regression</h1><div class=meta><div class=postdate>Updated <time datetime="2023-09-02 17:28:34 +0100 BST" itemprop=datePublished>2023-09-02</time>
<span class=commit-hash>(<a href=https://ruivieira.dev/log/index.html#d64c4a5>d64c4a5</a>)</span></div></div></header><div class=content itemprop=articleBody><p>Example taken from the official <a href=https://ruivieira.dev/scikit-learn.html>Scikit-learn</a> documentation<sup id=fnref:1><a href=#fn:1 class=footnote-ref role=doc-noteref>1</a></sup>.</p><h2 id=data x-intersect="currentHeading = '#data'">Data</h2><p>We will start by generating a synthetic dataset. The true generative process is defined as $f(x)=x\sin(x)$.</p><div class=highlight><pre tabindex=0 style=background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span><span style=font-weight:700>import</span> <span style=color:#555>numpy</span> <span style=font-weight:700>as</span> <span style=color:#555>np</span>
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>np<span style=font-weight:700>.</span>random<span style=font-weight:700>.</span>seed(<span style=color:#099>42</span>)
</span></span><span style=display:flex><span>X <span style=font-weight:700>=</span> np<span style=font-weight:700>.</span>linspace(start<span style=font-weight:700>=</span><span style=color:#099>0</span>, stop<span style=font-weight:700>=</span><span style=color:#099>10</span>, num<span style=font-weight:700>=</span><span style=color:#099>1_000</span>)<span style=font-weight:700>.</span>reshape(<span style=font-weight:700>-</span><span style=color:#099>1</span>, <span style=color:#099>1</span>)
</span></span><span style=display:flex><span>y <span style=font-weight:700>=</span> np<span style=font-weight:700>.</span>squeeze(X <span style=font-weight:700>*</span> np<span style=font-weight:700>.</span>sin(X))
</span></span></code></pre></div><p><img src=https://ruivieira.dev/Gaussian%20Process%20Regression_files/figure-gfm/cell-3-output-1.png alt loading=lazy></p><p>We will use this dataset in the next experiment to illustrate how Gaussian Process regression is working.</p><h1 id=example-with-noise-free-target x-intersect="currentHeading = '#example-with-noise-free-target'">Example with noise-free target</h1><p>In this first example, we will use the true generative process without adding any noise. For training the Gaussian Process regression, we will only select few samples.</p><div class=highlight><pre tabindex=0 style=background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span>rng <span style=font-weight:700>=</span> np<span style=font-weight:700>.</span>random<span style=font-weight:700>.</span>RandomState(<span style=color:#099>1</span>)
</span></span><span style=display:flex><span>training_indices <span style=font-weight:700>=</span> rng<span style=font-weight:700>.</span>choice(np<span style=font-weight:700>.</span>arange(y<span style=font-weight:700>.</span>size), size<span style=font-weight:700>=</span><span style=color:#099>6</span>, replace<span style=font-weight:700>=</span><span style=font-weight:700>False</span>)
</span></span><span style=display:flex><span>X_train, y_train <span style=font-weight:700>=</span> X[training_indices], y[training_indices]
</span></span></code></pre></div><p>Now, we fit a Gaussian process on these few training data samples. We will use a radial basis function (<a href=https://ruivieira.dev/kernel-functions.html#radial-basis-function-(rbf)>RBF</a>) kernel and a constant parameter to fit the amplitude.</p><div class=highlight><pre tabindex=0 style=background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span><span style=font-weight:700>from</span> <span style=color:#555>sklearn.gaussian_process</span> <span style=font-weight:700>import</span> GaussianProcessRegressor
</span></span><span style=display:flex><span><span style=font-weight:700>from</span> <span style=color:#555>sklearn.gaussian_process.kernels</span> <span style=font-weight:700>import</span> RBF
</span></span><span style=display:flex><span>
</span></span><span style=display:flex><span>kernel <span style=font-weight:700>=</span> <span style=color:#099>1</span> <span style=font-weight:700>*</span> RBF(length_scale<span style=font-weight:700>=</span><span style=color:#099>1.0</span>, length_scale_bounds<span style=font-weight:700>=</span>(<span style=color:#099>1e-2</span>, <span style=color:#099>1e2</span>))
</span></span><span style=display:flex><span>gaussian_process <span style=font-weight:700>=</span> GaussianProcessRegressor(kernel<span style=font-weight:700>=</span>kernel, n_restarts_optimizer<span style=font-weight:700>=</span><span style=color:#099>9</span>)
</span></span><span style=display:flex><span>gaussian_process<span style=font-weight:700>.</span>fit(X_train, y_train)
</span></span><span style=display:flex><span>gaussian_process<span style=font-weight:700>.</span>kernel_
</span></span></code></pre></div><pre><code>5.02**2 * RBF(length_scale=1.43)
</code></pre><p>After fitting our model, we see that the hyperparameters of the kernel have been optimized. Now, we will use our kernel to compute the mean prediction of the full dataset and plot the 95% confidence interval.</p><p><img src=https://ruivieira.dev/Gaussian%20Process%20Regression_files/figure-gfm/cell-6-output-1.png alt loading=lazy></p><p>We see that for a prediction made on a data point close to the one from the training set, the 95% confidence has a small amplitude. Whenever a sample falls far from training data, our model’s prediction is less accurate and the model prediction is less precise (higher uncertainty).</p><h1 id=example-with-noisy-targets x-intersect="currentHeading = '#example-with-noisy-targets'">Example with noisy targets</h1><p>We can repeat a similar experiment adding an additional noise to the target this time. It will allow seeing the effect of the noise on the fitted model.</p><p>We add some random Gaussian noise to the target with an arbitrary standard deviation.</p><div class=highlight><pre tabindex=0 style=background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span>noise_std <span style=font-weight:700>=</span> <span style=color:#099>0.75</span>
</span></span><span style=display:flex><span>y_train_noisy <span style=font-weight:700>=</span> y_train <span style=font-weight:700>+</span> rng<span style=font-weight:700>.</span>normal(loc<span style=font-weight:700>=</span><span style=color:#099>0.0</span>, scale<span style=font-weight:700>=</span>noise_std, size<span style=font-weight:700>=</span>y_train<span style=font-weight:700>.</span>shape)
</span></span></code></pre></div><p>We create a similar Gaussian process model. In addition to the kernel, this time, we specify the parameter alpha which can be interpreted as the variance of a Gaussian noise.</p><div class=highlight><pre tabindex=0 style=background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4><code class=language-python data-lang=python><span style=display:flex><span>gaussian_process <span style=font-weight:700>=</span> GaussianProcessRegressor(
</span></span><span style=display:flex><span> kernel<span style=font-weight:700>=</span>kernel, alpha<span style=font-weight:700>=</span>noise_std<span style=font-weight:700>**</span><span style=color:#099>2</span>, n_restarts_optimizer<span style=font-weight:700>=</span><span style=color:#099>9</span>
</span></span><span style=display:flex><span>)
</span></span><span style=display:flex><span>gaussian_process<span style=font-weight:700>.</span>fit(X_train, y_train_noisy)
</span></span><span style=display:flex><span>mean_prediction, std_prediction <span style=font-weight:700>=</span> gaussian_process<span style=font-weight:700>.</span>predict(X, return_std<span style=font-weight:700>=</span><span style=font-weight:700>True</span>)
</span></span></code></pre></div><p>Let’s plot the mean prediction and the uncertainty region as before.</p><p><img src=https://ruivieira.dev/Gaussian%20Process%20Regression_files/figure-gfm/cell-9-output-1.png alt loading=lazy></p><p>The noise affects the predictions close to the training samples: the predictive uncertainty near to the training samples is larger because we explicitly model a given level target noise independent of the input variable.</p><div class=footnotes role=doc-endnotes><hr><ol><li id=fn:1><p><a href=https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html>https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html</a> <a href=#fnref:1 class=footnote-backref role=doc-backlink>↩︎</a></p></li></ol></div></div></article><div id=footer-post-container><div id=footer-post><div id=nav-footer style=display:none><ul><li><a href=https://ruivieira.dev/>Home</a></li><li><a href=https://ruivieira.dev/blog/>Blog</a></li><li><a href=https://ruivieira.dev/draw/>Drawings</a></li><li><a href=https://ruivieira.dev/map/>All pages</a></li><li><a href=https://ruivieira.dev/search.html>Search</a></li></ul></div><div id=toc-footer style=display:none><nav id=TableOfContents><ul><li><a href=#data>Data</a></li></ul></nav></div><div id=share-footer style=display:none></div><div id=actions-footer><a id=menu-toggle class=icon href=# onclick='return $("#nav-footer").toggle(),!1' aria-label=Menu><i class="fas fa-bars fa-lg" aria-hidden=true></i> Menu</a>
<a id=toc-toggle class=icon href=# onclick='return $("#toc-footer").toggle(),!1' aria-label=TOC><i class="fas fa-list fa-lg" aria-hidden=true></i> TOC</a>
<a id=share-toggle class=icon href=# onclick='return $("#share-footer").toggle(),!1' aria-label=Share><i class="fas fa-share-alt fa-lg" aria-hidden=true></i> share</a>
<a id=top style=display:none class=icon href=# onclick='$("html, body").animate({scrollTop:0},"fast")' aria-label="Top of Page"><i class="fas fa-chevron-up fa-lg" aria-hidden=true></i> Top</a></div></div></div><footer id=footer><div class=footer-left>Copyright © 2024 Rui Vieira</div><div class=footer-right><nav><ul><li><a href=https://ruivieira.dev/>Home</a></li><li><a href=https://ruivieira.dev/blog/>Blog</a></li><li><a href=https://ruivieira.dev/draw/>Drawings</a></li><li><a href=https://ruivieira.dev/map/>All pages</a></li><li><a href=https://ruivieira.dev/search.html>Search</a></li></ul></nav></div></footer></div></body><link rel=stylesheet href=https://ruivieira.dev/css/fa.min.css><script src=https://ruivieira.dev/js/jquery-3.6.0.min.js></script><script src=https://ruivieira.dev/js/mark.min.js></script><script src=https://ruivieira.dev/js/main.js></script><script>MathJax={tex:{inlineMath:[["$","$"],["\\(","\\)"]]},svg:{fontCache:"global"}}</script><script type=text/javascript id=MathJax-script async src=https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js></script></html>