Added single and double pass ComputeMeanAndCovariance functions, plus basic self-test

[mattforestyoung]

So how do these functions look for single and double pass ComputeMeanAndCovariance? I can add a toggle later so the user can de-meaning on or off, but currently both functions de-mean by default.

I've also added a very basic self-test for the functions to confirm they work. In regards to how we actually test the results there are three options, which I've illustrated with the centroid:

// Method 1 (explicit but space inefficient) 
EXPECT_EQ(centroid_sp[0], 0.0); EXPECT_EQ(centroid_sp[1], 0.0);
EXPECT_EQ(centroid_sp[2], 0.0); EXPECT_EQ(centroid_sp[3], 1.0);

// Method 2 (iterative so scales well but doesn't show which element failed)
for (int i = 0; i < 4; i++)
{
  EXPECT_EQ(centroid_expected[i], centroid_sp[i]);
}

// Method 3 (Eigen approx, most space efficient but least informative)
EXPECT_TRUE(centroid_sp.isApprox(centroid_expected));

So do these functions look ok? And which calculated vs expected comparison method do we prefer?

[SergioRAgostinho]

👋 A couple of points to discuss.

1. Try this on a copy of your repo first.

    $ git checkout master 
    $ git pull <upstream i.e. taketwo's remote. check git remote add> master
    $ git checkout add-covariance-functions
    $ git rebase master
    $ git push -f origin add-covariance-functions
   

   You're including a copy of things which were already merged to master on your working branch. We just want the latest updates.

2. And which calculated vs expected comparison method do we prefer?

   Have a read at this section of the google test primer. Notice the first code snippet that appears after it. Copied here for convenience

   ASSERT_EQ(x.size(), y.size()) << "Vectors x and y are of unequal length";
   
   for (int i = 0; i < x.size(); ++i) {
    EXPECT_EQ(x[i], y[i]) << "Vectors x and y differ at index " << i;
   }

   This solves all your problems. Also have a look at the file pcl_tests.h. Submit a PR against PCL modifying those to include the extra custom message to provide the failure index. If possible provide also the ASSERT variants.

[mattforestyoung]

You're including a copy of things which were already merged to master on your working branch. We just want the latest updates.

Yes I saw the original commits piling up and increasing, so thanks for the tip. I've gone though the steps you posted, but I see the commits are still there. Should they have disappeared if that process had worked?

Have a read at this section of the google test primer.

Ah, missed that so thanks for pointing it out. I'll make the necessary changes and looking at submitting another PR to the main PCL repo.

[SergioRAgostinho]

Yes I saw the original commits piling up and increasing, so thanks for the tip. I've gone though the steps you posted, but I see the commits are still there. Should they have disappeared if that process had worked?

I cloned Sergey's and your forks and the rebase is failing. You need to solve the rebase conflicts before pushing again.

$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: Added detail to readme file
Using index info to reconstruct a base tree...
M	README.md
.git/rebase-apply/patch:13: trailing whitespace.
1. **Clone this repository** 
.git/rebase-apply/patch:20: trailing whitespace.
    
.git/rebase-apply/patch:24: trailing whitespace.
   Note: CMake 3.5 or higher is required. Also note that the generator specified after the `-G` can be something other than Unix Makefiles, E.g. Ninja. 
.git/rebase-apply/patch:26: trailing whitespace.
3. **Install `datamash`** 
.git/rebase-apply/patch:34: trailing whitespace.
4. **Build and run benchmarks** 
warning: squelched 3 whitespace errors
warning: 8 lines add whitespace errors.
Falling back to patching base and 3-way merge...
Auto-merging README.md
CONFLICT (content): Merge conflict in README.md
error: Failed to merge in the changes.
Patch failed at 0001 Added detail to readme file
Use 'git am --show-current-patch' to see the failed patch

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

Launch

$ git mergetool

save the final version of what you want.

$ git status
#if README.md appears in git status
$ git add README.md 
$ git rebase --continue
# otherwise
$ git rebase --skip

Then apply the push force.

[taketwo]

Regarding the test functions. Actually I don't like those two that Sergio referenced, they are against the guidelines of writing custom test assertions for Google Test. But below them there is an example of custom assertions done properly. So I'd propose to take these as an example and implement something like:

template <typename VectorT>
      ::testing::AssertionResult VectorNear (const char* expr1,
                                             const char* expr2,
                                             const char* abs_error_expr,
                                             const VectorT& v1,
                                             const VectorT& v2,
                                             double abs_error)
      {
        const VectorT diff = ((p1) - (p2)).cwiseAbs ();
        if ((diff.array () < abs_error).all ())
          return ::testing::AssertionSuccess ();
        return ::testing::AssertionFailure ()
               << "Some of the element-wise differences exceed " << abs_error_expr
               << " (which evaluates to " << abs_error << ")" << std::endl
               << "Difference: " << diff.transpose () << std::endl
               << "  Value of: " << expr2 << std::endl
               << "    Actual: " << v2.transpose () << std::endl
               << "  Expected: " << expr1 << std::endl
               << "  Which is: " << v1.transpose ();
}


/// Expect that element-wise differences between two vectors
/// are each within abs_error.
#define EXPECT_VECTOR_NEAR(expected, actual, abs_error)     \
  EXPECT_PRED_FORMAT3(::pcl::test::internal::VectorNear,    \
                      (expected), (actual), abs_error)

[mattforestyoung]

I cloned Sergey's and your forks and the rebase is failing. You need to solve the rebase conflicts before pushing again.

I think I understand what you're asking for, but I don't seem to have any conflicts. Would you mind taking a look at this output and clarifying the situation? Thanks!

matt@matt:~/pcl-mean-and-covariance$ git checkout master 
Already on 'master'
Your branch is up-to-date with 'origin/master'.

matt@matt:~/pcl-mean-and-covariance$ git pull upstream master
From git://github.com/taketwo/pcl-mean-and-covariance
 * branch            master     -> FETCH_HEAD
Already up-to-date.

matt@matt:~/pcl-mean-and-covariance$ git checkout add-covariance-functions 
Switched to branch 'add-covariance-functions'
Your branch is up-to-date with 'origin/add-covariance-functions'.

matt@matt:~/pcl-mean-and-covariance$ git rebase master
Current branch add-covariance-functions is up to date.

matt@matt:~/pcl-mean-and-covariance$ git push -f origin add-covariance-functions 
Everything up-to-date

matt@matt:~/pcl-mean-and-covariance$ git remote -v
origin	https://github.com/msy22/pcl-mean-and-covariance.git (fetch)
origin	https://github.com/msy22/pcl-mean-and-covariance.git (push)
upstream	git://github.com/taketwo/pcl-mean-and-covariance (fetch)
upstream	git://github.com/taketwo/pcl-mean-and-covariance (push)

[SergioRAgostinho]

You're right in your assessment. It doesn't exhibit any conflict whatsoever. My only explanation is your master branch not being in the exact same state as the one in Sergey's repo. Duplicate the directory somewhere and try this on the duplicate:

$ git checkout add-covariance-functions
$ git branch -D master
$ git fetch upstream
$ git checkout upstream/master
$ git checkout -b master

This should reset your master branch to the exact same state as Sergey's. Give it a try to the rebase procedure then.

[mattforestyoung]

Ok, I copied everything into another folder (pcl-mean-and-covariance2) and then ran the following commands:

$ git checkout add-covariance-functions
$ git branch -D master
$ git fetch upstream
$ git checkout upstream/master
$ git checkout -b master
$ git pull upstream master
$ git checkout add-covariance-functions
$ git rebase master
$ git push -f origin add-covariance-functions

Terminal output of commands (collapsible)

matt@matt:~/pcl-mean-and-covariance2$ git checkout add-covariance-functions 
Already on 'add-covariance-functions'
Your branch is up-to-date with 'origin/add-covariance-functions'.

matt@matt:~/pcl-mean-and-covariance2$ git branch -D master
Deleted branch master (was 9b14b65).

matt@matt:~/pcl-mean-and-covariance2$ git fetch upstream 

matt@matt:~/pcl-mean-and-covariance2$ git checkout upstream/master 
Note: checking out 'upstream/master'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 53d72b9... Adding detail to installation instructions (#1)

matt@matt:~/pcl-mean-and-covariance2$ git checkout -b master
Switched to a new branch 'master'

matt@matt:~/pcl-mean-and-covariance2$ git pull upstream master
From git://github.com/taketwo/pcl-mean-and-covariance
 * branch            master     -> FETCH_HEAD
Already up-to-date.

matt@matt:~/pcl-mean-and-covariance2$ git checkout add-covariance-functions 
Switched to branch 'add-covariance-functions'
Your branch is up-to-date with 'origin/add-covariance-functions'.

matt@matt:~/pcl-mean-and-covariance2$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: Added detail to readme file
Using index info to reconstruct a base tree...
M	README.md
<stdin>:14: trailing whitespace.
1. **Clone this repository** 
<stdin>:21: trailing whitespace.
    
<stdin>:25: trailing whitespace.
   Note: CMake 3.5 or higher is required. Also note that the generator specified after the `-G` can be something other than Unix Makefiles, E.g. Ninja. 
<stdin>:27: trailing whitespace.
3. **Install `datamash`** 
<stdin>:35: trailing whitespace.
4. **Build and run benchmarks** 
warning: squelched 3 whitespace errors
warning: 8 lines add whitespace errors.
Falling back to patching base and 3-way merge...
Auto-merging README.md
CONFLICT (content): Merge conflict in README.md
Failed to merge in the changes.
Patch failed at 0001 Added detail to readme file
The copy of the patch that failed is found in:
   /home/matt/pcl-mean-and-covariance2/.git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

#####################################################################
# At this point I simply copied the changes from README.md.REMOTE to README.md
#####################################################################

matt@matt:~/pcl-mean-and-covariance2$ git add README.md
matt@matt:~/pcl-mean-and-covariance2$ git rebase --continue
Applying: Added detail to readme file
Applying: Updated git clone description to match code snippet
Applying: Corrected instructions to make an out-of-source build
Applying: Got a basic addition function working (I think)
Applying: Added includes for PCL point and cloud types
Applying: Added single pass explanation of code
Applying: Added more comments explaining the single pass function
Applying: Added modified double pass co-variance calculation
Applying: Added de-meaning to double pass calculation
Applying: Completed very basic self-test of single and double-pass calculations
matt@matt:~/pcl-mean-and-covariance2$ git push -f origin add-covariance-functions 
Counting objects: 47, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (36/36), done.
Writing objects: 100% (36/36), 3.95 KiB | 0 bytes/s, done.
Total 36 (delta 20), reused 0 (delta 0)
remote: Resolving deltas: 100% (20/20), completed with 5 local objects.
To https://github.com/msy22/pcl-mean-and-covariance.git
 + f9f86f7...f846e7c add-covariance-functions -> add-covariance-functions (forced update)

It's made the commit history quite messy locally, but I think this has achieved what you're wanting? The commits attached to this PR have been trimmed back to just the ones relevant to the add-covariance-functions branch. If this doesn't resolve the issue, it's not worth wasting more time on so I'll just delete everything locally and start again with a fresh fork and local repo.

[SergioRAgostinho]

It's made the commit history quite messy locally, but I think this has achieved what you're wanting?

That was exactly the process. I have to admit that something looks weird with the master branch. @taketwo did you remove previous @msy22's commits from the master branch? I have the impression the first 4 commits were there before and now they're not.

Anyway @msy22 you got the gist of it. The take home lesson is never touch the master branch of your fork unless to want to pull the latest changes from upstream. If you ever do anything else other than

$ git pull upstream master

on your master branch stop for a second and reassure yourself of what you're doing.

On to the next items on the agenda 👍

[mattforestyoung]

Anyway @msy22 you got the gist of it. The take home lesson is never touch the master branch of your fork unless to want to pull the latest changes from upstream.

Cool, I appreciate the lesson and I'll keep that in mind for future.

On to the next items on the agenda +1

I've pushed more changes, adding your suggested changes to the EXPECT_EQ assertion. It looks quite ugly, and could be split into separate TEST blocks perhaps. But for now it does the job.

I've also added a boolean toggle to the computeMean... functions. If turned off, the functions will still subtract the centroid, but it will be set to zero, which seems cleaner than having if statements littered throughout the functions.

As for making the assertion changes to pcl_tests.h, @taketwo's suggestion is more thorough. But perhaps that needs it's own discussion, and is a little tangential to the normal/covariance accuracy stuff.

[mattforestyoung]

On a side note, I'm also doing some more research into numerical problems and what the underlying mathematical reason for the incorrect normal calculation is (at this stage I think it's "catastrophic cancellation"). I see that the double-pass covariance calculation follows a pretty standard algorithm. But I'm not sure where the single pass algorithm is from. Do either of you know where the original sources for this calculation are?

[SergioRAgostinho]

Disclaimer: I'm claiming this based on a (potentially faulty) memory of having looked into it before. It is likely using the covariance known identity presented here (Second expression). Be sure to have a read at
https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Covariance .

[mattforestyoung]

Ah, I see what's happened, yes I was reading those pages and looking at those equations, but an error at my end meant they didn't match what I have written on paper. Everything matches now so I'll carry on.

[mattforestyoung]

Ok, so now that I understand the co-variance calculations better, I have a question regarding the difference between them. The single-pass method is mathematically the same as the double-pass method, except the equation has been expanded out. According to what I've read this is ostensibly because the single pass method only accesses the inputs once instead of twice. In terms of code, this just means that the point variable is called in one for loop instead of two. Now, to see if this is true I modified this line in bench_mean_and_covariance.cpp

From:

pcl::experimental::Function <pcl::Type, Scalar> (*CloudIn ...

To:

pcl::experimental::computeMeanAndCovarianceMatrixDoublePass <pcl::Type, Scalar> (*CloudIn ...

To force the benchmark to use the double-pass version instead of the single-pass version. I then ran the benchmark with a few different sample rates and iterations, but the results show pretty much the same result. It's only when you get to 60000 iterations (roughly the number of points in a Velodyne HDL-32 scan) that you start seeing a difference, but it's pretty negligible.

Terminal output of `make benchmarks` (20 samples, 4 iterations)

Running benchmarks
Compiler: gcc-4-8-4
Precision: float
Flag: none
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZ             | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
Complete.
Compiler: gcc-4-8-4
Precision: float
Flag: native
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZ             | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
Complete.
Compiler: gcc-4-8-4
Precision: float
Flag: sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZ             | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
Complete.
Compiler: gcc-4-8-4
Precision: float
Flag: no-sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZ             | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: none
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZ             | Proposed        |            Null |              20 |               4 |         1.00268 |        93.50000 |        10695.19 | 
|XYZN            | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: native
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZ             | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZ             | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: no-sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | PCL             |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZ             | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
|XYZN            | Proposed        |            Null |              20 |               4 |         1.00000 |        93.25000 |        10723.86 | 
Complete.
Results saved to: /home/matt/pcl-mean-and-covariance/build/bench/bench_mean_and_covariance_gcc-4-8-4.csv
[100%] Built target benchmarks

Terminal output of `make benchmarks` (30 samples, 50 iterations)

Running benchmarks
Compiler: gcc-4-8-4
Precision: float
Flag: none
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |              50 |         1.00000 |        93.42000 |        10704.35 | 
|XYZN            | PCL             |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
|XYZ             | Proposed        |            Null |              30 |              50 |         0.99979 |        93.40000 |        10706.64 | 
|XYZN            | Proposed        |            Null |              30 |              50 |         1.00021 |        93.42000 |        10704.35 | 
Complete.
Compiler: gcc-4-8-4
Precision: float
Flag: native
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |              50 |         1.00000 |        93.42000 |        10704.35 | 
|XYZN            | PCL             |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
|XYZ             | Proposed        |            Null |              30 |              50 |         0.99957 |        93.38000 |        10708.93 | 
|XYZN            | Proposed        |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
Complete.
Compiler: gcc-4-8-4
Precision: float
Flag: sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
|XYZN            | PCL             |            Null |              30 |              50 |         1.00000 |        93.42000 |        10704.35 | 
|XYZ             | Proposed        |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
|XYZN            | Proposed        |            Null |              30 |              50 |         1.00000 |        93.42000 |        10704.35 | 
Complete.
Compiler: gcc-4-8-4
Precision: float
Flag: no-sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
|XYZN            | PCL             |            Null |              30 |              50 |         1.00000 |        93.38000 |        10708.93 | 
|XYZ             | Proposed        |            Null |              30 |              50 |         1.00021 |        93.42000 |        10704.35 | 
|XYZN            | Proposed        |            Null |              30 |              50 |         1.00043 |        93.42000 |        10704.35 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: none
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
|XYZN            | PCL             |            Null |              30 |              50 |         1.00000 |        93.38000 |        10708.93 | 
|XYZ             | Proposed        |            Null |              30 |              50 |         1.00021 |        93.42000 |        10704.35 | 
|XYZN            | Proposed        |            Null |              30 |              50 |         1.00000 |        93.38000 |        10708.93 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: native
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |              50 |         1.00000 |        93.36000 |        10711.23 | 
|XYZN            | PCL             |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
|XYZ             | Proposed        |            Null |              30 |              50 |         1.00021 |        93.38000 |        10708.93 | 
|XYZN            | Proposed        |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |              50 |         1.00000 |        93.42000 |        10704.35 | 
|XYZN            | PCL             |            Null |              30 |              50 |         1.00000 |        93.36000 |        10711.23 | 
|XYZ             | Proposed        |            Null |              30 |              50 |         1.00000 |        93.42000 |        10704.35 | 
|XYZN            | Proposed        |            Null |              30 |              50 |         1.00064 |        93.42000 |        10704.35 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: no-sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
|XYZN            | PCL             |            Null |              30 |              50 |         1.00000 |        93.42000 |        10704.35 | 
|XYZ             | Proposed        |            Null |              30 |              50 |         1.00000 |        93.40000 |        10706.64 | 
|XYZN            | Proposed        |            Null |              30 |              50 |         1.00000 |        93.42000 |        10704.35 | 
Complete.
Results saved to: /home/matt/pcl-mean-and-covariance/build/bench/bench_mean_and_covariance_gcc-4-8-4.csv
[100%] Built target benchmarks

Terminal output of `make benchmarks` (30 samples, 60000 iterations)

Running benchmarks
Compiler: gcc-4-8-4
Precision: float
Flag: none
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |           60000 |         1.00000 |        94.91180 |        10536.10 | 
|XYZN            | PCL             |            Null |              30 |           60000 |         1.00000 |        94.81872 |        10546.44 | 
|XYZ             | Proposed        |            Null |              30 |           60000 |         0.99935 |        94.85018 |        10542.94 | 
|XYZN            | Proposed        |            Null |              30 |           60000 |         1.00107 |        94.92002 |        10535.19 | 
Complete.
Compiler: gcc-4-8-4
Precision: float
Flag: native
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |           60000 |         1.00000 |        94.94473 |        10532.44 | 
|XYZN            | PCL             |            Null |              30 |           60000 |         1.00000 |        95.21972 |        10502.03 | 
|XYZ             | Proposed        |            Null |              30 |           60000 |         1.00067 |        95.00807 |        10525.42 | 
|XYZN            | Proposed        |            Null |              30 |           60000 |         0.99696 |        94.92980 |        10534.10 | 
Complete.
Compiler: gcc-4-8-4
Precision: float
Flag: sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |           60000 |         1.00000 |        95.10640 |        10514.54 | 
|XYZN            | PCL             |            Null |              30 |           60000 |         1.00000 |        95.22648 |        10501.28 | 
|XYZ             | Proposed        |            Null |              30 |           60000 |         0.99869 |        94.98148 |        10528.37 | 
|XYZN            | Proposed        |            Null |              30 |           60000 |         1.00009 |        95.23458 |        10500.39 | 
Complete.
Compiler: gcc-4-8-4
Precision: float
Flag: no-sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |           60000 |         1.00000 |        95.18362 |        10506.01 | 
|XYZN            | PCL             |            Null |              30 |           60000 |         1.00000 |        95.15280 |        10509.41 | 
|XYZ             | Proposed        |            Null |              30 |           60000 |         0.99961 |        95.14660 |        10510.10 | 
|XYZN            | Proposed        |            Null |              30 |           60000 |         0.99993 |        95.14657 |        10510.10 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: none
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |           60000 |         1.00000 |        95.17930 |        10506.49 | 
|XYZN            | PCL             |            Null |              30 |           60000 |         1.00000 |        95.30877 |        10492.21 | 
|XYZ             | Proposed        |            Null |              30 |           60000 |         0.99927 |        95.11027 |        10514.11 | 
|XYZN            | Proposed        |            Null |              30 |           60000 |         0.99902 |        95.21563 |        10502.48 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: native
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |           60000 |         1.00000 |        94.89728 |        10537.71 | 
|XYZN            | PCL             |            Null |              30 |           60000 |         1.00000 |        95.15618 |        10509.04 | 
|XYZ             | Proposed        |            Null |              30 |           60000 |         1.00109 |        95.00098 |        10526.21 | 
|XYZN            | Proposed        |            Null |              30 |           60000 |         1.00023 |        95.17833 |        10506.59 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |           60000 |         1.00000 |        95.13505 |        10511.37 | 
|XYZN            | PCL             |            Null |              30 |           60000 |         1.00000 |        95.06075 |        10519.59 | 
|XYZ             | Proposed        |            Null |              30 |           60000 |         1.00439 |        95.55300 |        10465.40 | 
|XYZN            | Proposed        |            Null |              30 |           60000 |         1.00189 |        95.24015 |        10499.77 | 
Complete.
Compiler: gcc-4-8-4
Precision: double
Flag: no-sse2
Celero
Timer resolution: 0.001000 us
Writing results to: out
|     Group      |   Experiment    |   Prob. Space   |     Samples     |   Iterations    |    Baseline     |  us/Iteration   | Iterations/sec  | 
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|XYZ             | PCL             |            Null |              30 |           60000 |         1.00000 |        94.93450 |        10533.58 | 
|XYZN            | PCL             |            Null |              30 |           60000 |         1.00000 |        95.12377 |        10512.62 | 
|XYZ             | Proposed        |            Null |              30 |           60000 |         1.00153 |        95.07997 |        10517.46 | 
|XYZN            | Proposed        |            Null |              30 |           60000 |         1.00059 |        95.18020 |        10506.39 | 
Complete.
Results saved to: /home/matt/pcl-mean-and-covariance/build/bench/bench_mean_and_covariance_gcc-4-8-4.csv
[100%] Built target benchmarks

... and if this claim about the superior speed of the single-pass method was true, I would have expected all the "proposed" calculations to show a much larger us/Iteration.

So have I interpreted this output correctly and the methods aren't really much faster than each other? Or is there something about my implementation that is wrong?

On a side note, it appears that the output logged at the end to bench_mean_and_covariance_gcc-4-8-4.csv provides only the baseline results, not the benchmarked comparison.