Retry failed profiled very-hot compilations at hot instead of warm

Before this change, a profiled-very-hot compilation that fails
due to excessiveComplexity would not be retried, relying on the
existing compiled body. If the existing compiled body is at
warm (or even cold) opt level, then the performance could be
suboptimal.

With this commit profiled-very-hot compilations that fail will be
retried at the hot optimization level, but without any profiling.
We avoid profiling compilations because they are more complex than
non-profiled ones and the likelihood of a failure for a
profiled-hot compilation is relatively high when we know that a
profiled-very-hot compilation for the same method already failed.

Signed-off-by: Marius Pirvu <[email protected]>

runtime/compiler/control/CompilationThread.cpp

@@ -2347,24 +2347,29 @@ bool TR::CompilationInfo::shouldRetryCompilation(J9VMThread *vmThread, TR_Method
                bodyInfo = 0;
                if (comp->allowRecompilation() && entry->_optimizationPlan && entry->_optimizationPlan->getOptLevel() > minHotness)
                   {
-                  // Compile only if the method is interpreted or if it's compiled with
-                  // profiling information.
+                  // Compile if the method is interpreted.
                   if (entry->_oldStartPC == 0) // interpreter method
                      {
                      tryCompilingAgain = true;
                      }
-                  else // Does the existing body contain profiling info?
+                  else
                      {
+                     // Try again if existing body is compiled with profiling information
+                     // because we don't want to be stuck in profiling mode forever.
                      bodyInfo = TR::Recompilation::getJittedBodyInfoFromPC(entry->_oldStartPC);
                      if (bodyInfo->getIsProfilingBody())
                         tryCompilingAgain = true;
-                     // if existing body is invalidated, retry the compilation
+                     // If existing body is invalidated, retry the compilation.
                      else if (bodyInfo->getIsInvalidated())
                         tryCompilingAgain = true;
-                     // if the existing body uses pre-existence, retry the compilation
-                     // otherwise we will revert to interpreted when failing this compilation
+                     // If the existing body uses pre-existence, retry the compilation,
+                     // otherwise we will revert to interpreted when failing this compilation.
                      else if (bodyInfo->getUsesPreexistence())
                         tryCompilingAgain = true;
+                     // If the compilation that failed is at a higher opt level than hot
+                     // but the existing compiled body is lower than hot, try to recompile at hot.
+                     else if (comp->getOptLevel() > hot && bodyInfo->getHotness() <= warm)
+                        tryCompilingAgain = true;
                      }
                   }
 
@@ -2373,11 +2378,30 @@ bool TR::CompilationInfo::shouldRetryCompilation(J9VMThread *vmThread, TR_Method
                   TR_Hotness hotness = entry->_optimizationPlan->getOptLevel();
                   TR_Hotness newHotness;
                   if (hotness == veryHot)
-                     newHotness = warm; // skip over hot and go down two levels
+                     {
+                     if (comp->isProfilingCompilation())
+                        {
+                        // A hot compilation might succeed where a
+                        // profiled very-hot compilation might fail.
+                        newHotness = hot;
+                        // Prevent internal switching to profiling.
+                        entry->_optimizationPlan->setDoNotSwitchToProfiling(true);
+                        }
+                     else
+                        {
+                        // Very-hot without profiling is as expensive as hot,
+                        // so we should not retry at hot because it may fail again.
+                        newHotness = warm;
+                        }
+                     }
                   else if (hotness <= scorching)
+                     {
                      newHotness = (TR_Hotness)(hotness - 1);
+                     }
                   else // Why would we use hotness greater than scorching
+                     {
                      newHotness = noOpt;
+                     }
                   entry->_optimizationPlan->setOptLevel(newHotness);
                   entry->_optimizationPlan->setInsertInstrumentation(false); // prevent profiling
                   entry->_optimizationPlan->setUseSampling(false); // disable recompilation of this method