Skip to content

Commit

Permalink
SF_TPTest: Avoid issue due to rounding differences
Browse files Browse the repository at this point in the history
The recent introduction of decimal multiplier constants changed the last
digits of the result.

But we are actually not interested in an arbitrary precise result, so
let's use the tol parameter.
  • Loading branch information
t-b committed Apr 14, 2022
1 parent 3d50bb2 commit f33460a
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions Packages/Testing-MIES/UTF_SweepFormulaHardware.ipf
Original file line number Diff line number Diff line change
Expand Up @@ -213,19 +213,19 @@ static Function TestSweepFormulaTP(string device)
SetDimLabel LAYERS, 0, DA0, wRef
SetDimLabel LAYERS, 1, DA1, wRef
SetScale d, 0, 0, "MΩ", wRef
CHECK_EQUAL_WAVES(tpResult, wRef)
CHECK_EQUAL_WAVES(tpResult, wRef, tol = 1e-12)

formula = "tp(inst, channels(DA), sweeps())"
WAVE tpResult = SF_FormulaExecutor(DirectToFormulaParser(formula), graph=graph)
CHECK_EQUAL_WAVES(tpResult, wRef)
CHECK_EQUAL_WAVES(tpResult, wRef, tol = 1e-12)

formula = "tp(1, channels(DA), sweeps())"
WAVE tpResult = SF_FormulaExecutor(DirectToFormulaParser(formula), graph=graph)
CHECK_EQUAL_WAVES(tpResult, wRef)
CHECK_EQUAL_WAVES(tpResult, wRef, tol = 1e-12)

formula = "tp(2, channels(DA), sweeps())"
WAVE tpResult = SF_FormulaExecutor(DirectToFormulaParser(formula), graph=graph)
CHECK_EQUAL_WAVES(tpResult, wRef)
CHECK_EQUAL_WAVES(tpResult, wRef, tol = 1e-12)

formula = "tp(base, channels(DA), sweeps())"
WAVE tpResult = SF_FormulaExecutor(DirectToFormulaParser(formula), graph=graph)
Expand Down

0 comments on commit f33460a

Please sign in to comment.