Impossible to instrument the body of runConcurrentTest when it is used from java

[dmitrii-artuhov]

Intro

This issue relates to the general purpose model-checker API function: runConcurrentTest (the placeholder naming)
How the function is supposed to work: lincheck model checker algorithm is run on arbitrary code via lambda body

class JunitTest {
  val invocations = 100
  @Test
  fun test() = runConcurrentTest(invocations) {
    var counter = 0
    val t1 = thread { counter++ }
    val t2 = thread { counter++ }
    t1.join(); t2.join()
    check(counter == 2) // will detect an error an throw and exception
  }
}

Problem

When the we make the bytecode transformation in ensureHierarchyTransformed(block) method call, the block is a lambda which contains the code to model-check (basically the body of runConcurrentTest). The problem with instrumenting the contents of this lambda lie in the way how kotlin & java compile them to bytecode.

1. Kotlin compiles lambdas (and method references) to anonymous classes, which can be transformed by dynamic agents (by calling method redefineClass).
2. Java compiles them to hidden classes, which are not modifiable via redefineClasses (https://bugs.openjdk.org/browse/JDK-8230502). From this follows that dynamic agent cannot instrument the body of the java-compiled lambda (the transformation will throw execption, but our code omits such classes even earlier in instrumentation.isModifiable(clazz) call).

What are the outcomes for java:

• If we pass the test class to the runConcurrentTest and write the body as a test class method, then we should be able to run such tests (by calling ensureHierarchyTransformed(testClass) instead of calling it on block):

class JavaClass {
  public void block() { /* code to test */ }
  @Test
  public void test() { runConcurrentTest(iterations, this::class, this::block) }
}

• If we decide to write the code to test inside a lambda body, then its instrumentation is not possible with dynamic agent (no matter if we pass the test class or not):

class JavaClass {
  @Test
  public void test() {
     runConcurrentTest(iterations, this::class, () -> {
        // code to test -> not possible
     })
  }
}

[eupp]

Related issue #451