build is not identical on macos with sandboxing

[johnynek]

Using homebrew bazel 0.4.4 with:
build --spawn_strategy=sandboxed --genrule_strategy=sandboxed

in .bazelrc, I get:

1,2d0 < MD5 (bazel-bin/test/ConsumeSuite.jar) = 20c88ca52467d81b0d25ed8a659cd61b < MD5 (bazel-bin/test/ConsumeSuite_ijar.ja
r) = 79ec42ce14a22ca7dc682d02513acc4e 5,33d2 < MD5 (bazel-bin/test/Exported.jar) = 6483b2788b3c7432be142a46c68635db < MD5 (bazel-bin/test/Exported_deploy.jar) = c6d6f95d3707186abda4ef0de6d60c73 < MD5 (bazel-bin/t
est/Exported_ijar.jar) = 53bcef2585e5895119c161b67164fb1e < MD5 (bazel-bin/test/HelloLib.jar) = 61479b32fa0705d6fae59b316b56b930 < MD5 (bazel-bin/test/HelloLibTest.jar) = 552eb5fce6989e067ec3d2674f114e59 < MD5 (b
azel-bin/test/HelloLibTestSuite_test_suite_HelloLibTest.scala.jar) = 552eb5fce6989e067ec3d2674f114e59 < MD5 (bazel-bin/test/HelloLibTestSuite_test_suite_HelloLibTest2.scala.jar) = 7a85f7c5cf24e7d0a1f160a8ccb0479e
 < MD5 (bazel-bin/test/HelloLib_deploy.jar) = 5b7c1dc54299eac6efce701a2d9e7ca5 < MD5 (bazel-bin/test/HelloLib_ijar.jar) = 894d6691bf9a8adbdebff7908500be99 < MD5 (bazel-bin/test/JavaBinary.jar) = e3e3f10010b8e115e
b6a4773abcd6e80 < MD5 (bazel-bin/test/JavaBinary2.jar) = e3e3f10010b8e115eb6a4773abcd6e80 < MD5 (bazel-bin/test/JavaOnlySources.jar) = a61223a56316d03669ca377d9eed6199 < MD5 (bazel-bin/test/MacroTest.jar) = 92bc8
84c944d87b6eb10b7de626e3290 < MD5 (bazel-bin/test/MixJavaScalaLib.jar) = 4d6cb84301333a33cd7a3f1206cfb4b7 < MD5 (bazel-bin/test/MixJavaScalaLibBinary.jar) = 8d4c20e184dda305a7e5d26ce42e4a72 < MD5 (bazel-bin/test/
MixJavaScalaLib_ijar.jar) = dfdb53113ffde4900b26170b38942f95 < MD5 (bazel-bin/test/MoreScalaLibBinary.jar) = 241caceb6c386ac4e191d184a3e3cff2 < MD5 (bazel-bin/test/MoreScalaLibResources.jar) = c6037d4f8cb87574a2a
ca9a3a9d49430 < MD5 (bazel-bin/test/MoreScalaLibResources_ijar.jar) = 345549ab0e31226904bb119bcab8b683 < MD5 (bazel-bin/test/OtherLib.jar) = 300a9abbab8d6a0eff16892360e7b27d < MD5 (bazel-bin/test/OtherLib_deploy.
jar) = 9fb66f5464ea390cfeef2b12b5ec96fd < MD5 (bazel-bin/test/OtherLib_ijar.jar) = 4d296355b187a0734943c543c4be3c7b < MD5 (bazel-bin/test/Runtime.jar) = f5ea166a4e542bb01b2c3e80d48f88ca < MD5 (bazel-bin/test/Runt
ime_deploy.jar) = 90123710c39f188dd18ae9f0ab79cd93 < MD5 (bazel-bin/test/Runtime_ijar.jar) = 4b7241516ffd5e71c0e50fd89a0cb298 < MD5 (bazel-bin/test/ScalaBinary.jar) = cab090694eb1b27b2433b3aa7ddd2570 < MD5 (bazel
-bin/test/ScalaLibBinary.jar) = 6aa9c9bf4d3dd4d3d6062fb2ee23a994 < MD5 (bazel-bin/test/ScalaLibResources.jar) = 08d2aed03ae7c7b9536c0471fcb889c2 < MD5 (bazel-bin/test/ScalaLibResources_ijar.jar) = 4715ed984ca0a6a
00931486cd4993fa3 36,39d4 < MD5 (bazel-bin/test/SuiteSrcs_lib_DataRoot1.scala.jar) = 77c7765b84db207dbced8f2f7f9b8440 < MD5 (bazel-bin/test/SuiteSrcs_lib_DataRoot1.scala_ijar.jar) = 20efe6708acf2b368c80eb115b4ad4
e9 < MD5 (bazel-bin/test/SuiteSrcs_lib_DataRoot2.scala.jar) = 29dbed3a1b4bed5a2d95f4a92cf80663 < MD5 (bazel-bin/test/SuiteSrcs_lib_DataRoot2.scala_ijar.jar) = 15c7ed396fac028f83d6b9a07db9ef14
 Test test_build_is_identical failed

[ittaiz]

Isn't this #74 and #111? Which we've closed in favor of changing to #76

[johnynek]

@ittaiz I don't see how it is related to that. If I disable sandboxing, the test passes. Only with sandboxing enabled to I get different hashes.

Did I misunderstand your point? The issues you link to above were about md5 vs md5sum and doing the shell test in bazel, no?

[ittaiz]

sorry for the late reply. it seems I've misunderstood you, thanks for the explanation

[chrismcbride]

There is a similar issue with the go rules.

bazel-contrib/rules_go#198

It has to do with the randomly generated sandbox paths being included as debug symbols.

[johnynek]

@chrismcbride Thanks for the pointer!

[oscar-stripe]

I somehow can't reproduce this now. I was trying to fix it and see if some paths were getting in somewhere. If anyone else can repro, let me know. I hope it is not sporadic.

[ittaiz]

I might be mixing stuff up but when I run test_run.sh on HEAD of rules_scala with nothing in my .bazelrc I get the following error:

____Loading complete. Analyzing... ____Found 68 targets... ____Elapsed time: 0.114s, Critical Path: 0.00s ____Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes. ____Loading package: test/data ____Loading package: @bazel_tools//tools/cpp ____Loading package: @local_jdk// ____Loading package: @local_config_cc// ____Loading package: @local_config_xcode// ____Loading complete. Analyzing... ____Loading package: @bazel_tools//third_party/java/jdk/langtools ____Loading package: src/java/com/google/devtools/build/lib ____Found 68 targets... ____Building... ____[0 / 1] BazelWorkspaceStatusAction stable-status.txt ____[1 / 51] Writing file build-info-redacted.properties ____[30 / 271] Writing file src/scala/io/bazel/rules_scala/tut_support/tut_compiler_lib_MANIFEST.MF [for host] ____[30 / 283] Writing file test/src/main/scala/scala/test/twitter_scrooge/mixed_MANIFEST.MF ____[30 / 286] Writing file test/src/main/scala/scala/test/twitter_scrooge/justscrooges_MANIFEST.MF ____[86 / 291] Extracting interface @libthrift//jar:jar ____[86 / 291] Extracting interface @util_logging//jar:jar [for host] ____[97 / 291] Creating runfiles tree bazel-out/local-fastbuild/bin/test/JavaBinary.runfiles ____[225 / 291] Still waiting for 16 jobs to complete: Running (sandbox): scala //scala/support:test_reporter, 10 s scala //src/scala/io/bazel/rules_scala/tut_support:tut_compiler_lib, 10 s scala //test:JavaOnlySources, 10 s scala //test:MoreScalaLibResources, 10 s scala //test:ScalaLibResources, 10 s scala //test:SuiteSrcs_lib_DataRoot1.scala, 10 s scala //test/tut:test_dep [for host], 10 s scala //test/tut:test_dep, 10 s scala //src/scala/io/bazel/rules_scala/tut_support:tut_compiler_lib [for host], 10 s scala //test:SuiteSrcs_lib_DataRoot2.scala, 10 s Scheduling: scala //test:MixJavaScalaLib, 10 s scala //test:MacroTest, 10 s scala //test:Runtime, 10 s scala //src/scala/io/bazel/rules_scala/scrooge_support:focused_zip_importer [for host], 10 s scala //test:Exported, 10 s scala //test/src/main/scala/scala/test/srcjars:source_jar, 10 s ____[234 / 291] Still waiting for 11 jobs to complete: Running (sandbox): scala //scala/support:test_reporter, 37 s scala //test:MixJavaScalaLib, 19 s scala //test:MacroTest, 18 s scala //test:Runtime, 18 s scala //src/scala/io/bazel/rules_scala/scrooge_support:focused_zip_importer [for host], 15 s scala //test:Exported, 15 s scala //test/src/main/scala/scala/test/srcjars:source_jar, 14 s scala //test:ConsumeSuite, 14 s scala //test:ScalaLibBinary, 11 s scala //test:MoreScalaLibBinary, 10 s Scheduling: Executing genrule //test/tut:some_example__tut, 11 s ____From scala //test:MacroTest: warning: there was one deprecation warning; re-run with -deprecation for details one warning found ____From Executing genrule //test/tut:some_example__tut: [tut] compiling: test/tut/SomeExample.md wrote: bazel-out/local-fastbuild/genfiles/test/tut/some_example_tut.md ____From scala //test:HelloLib: test/HelloLib.scala:20: warning: Remove the if expression and use the condition directly. def dumb(x: Int) = if (x == 3) true else false ^ one warning found ____From scala //test:ScalaBinary: Compiler runtime: 7894ms. ____From scala //src/scala/scripts:generator_lib [for host]: warning: there were two deprecation warnings; re-run with -deprecation for details one warning found ____From scala //test/src/main/scala/scala/test/twitter_scrooge:twodeep: warning: there was one feature warning; re-run with -feature for details one warning found ____From scala //test/src/main/scala/scala/test/twitter_scrooge:mixed: warning: there was one feature warning; re-run with -feature for details one warning found ____Elapsed time: 148.072s, Critical Path: 143.13s 42d41 < MD5 (bazel-bin/test/libOtherJavaLib-hjar.jar) = f57034a0379fbc0ca5b42e1677aa45f5

Is this what you're looking for? Because as far as I can see I'm not using sandboxing.
Running this on OS X of course.

[gkk-stripe]

I just ran bash test_run.sh on MacOSX with bazel 0.5.2 and it passed. I came here from README that says test_run.sh doesn't pass due to this issue. Did something get fixed?

[ittaiz]

I don't think so I just ran with HEAD~2 and it failed. I think that it passes on some machines (Oscar- it always passes on yours, right?) but fails on others (I think it never passed on my machine).

…

On Wed, Jul 12, 2017 at 9:38 PM gkk-stripe ***@***.***> wrote: I just ran bash test_run.sh on MacOSX with bazel 0.5.2 and it passed. I came here from README that says test_run.sh doesn't pass due to this issue. Did something get fixed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#136 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF-7ftnfliitwVeHLbpMv0JUnYGIJks5sNRKigaJpZM4L48S_> .

[ittaiz]

I'm closing this as I haven't been able to reproduce this for some time.
We've done a lot of work in related areas and upgraded the bazel version and any of them could have contributed.
Closing for now, please reopen if you see something else

[ittaiz]

Reopening since I've started hitting this again locally :(
Can't dive into it right now but there is probably still something lingering.
I think the first thing to do is change the test to also keep a list of file names to md5 so that if the test fails we have a finer granularity on where were the diffs. Maybe also keep a copy of the jars so we can inspect them.

[johnynek]

@gkk-stripe also saw this on our Linux CI but it seemed like macros were at play. There may be two different issues. Clearly a macro can be non reproducible since it can use the clock or it could use a identity based hashCode.

[ittaiz]

You mean Scala macros right?

…

On Mon, 4 Dec 2017 at 17:21 P. Oscar Boykin ***@***.***> wrote: @gkk-stripe <https://github.com/gkk-stripe> also saw this on our Linux CI but it seemed like macros were at play. There may be two different issues. Clearly a macro can be non reproducible since it can use the clock or it could use a identity based hashCode. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#136 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF6sURXZJTTYtcZa4sVSlDfT2J9xiks5s9A39gaJpZM4L48S_> .

[johnynek]

yes

[ittaiz]

So I'm hitting this with build is identical in this repo. Wdyt about my suggestion?

…

On Mon, 4 Dec 2017 at 19:44 P. Oscar Boykin ***@***.***> wrote: yes — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#136 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFypu88MsiLLRoP7zGmE0yJBGHCrkks5s9C-KgaJpZM4L48S_> .

[johnynek]

I like your suggestion! I'd love to get to the bottom of this.

[gkossakowski]

The most common reason for macros to be non-deterministic is to rely on object identity for hash code. If you put them in a map, the order of iteration is non-deterministic.

[ittaiz]

#396 and #398 should have resolved the current round