Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP, Depends #24] Build spark jar #25

Closed
wants to merge 3 commits into from
Closed

[WIP, Depends #24] Build spark jar #25

wants to merge 3 commits into from

Conversation

dazza-codes
Copy link
Contributor

@dazza-codes dazza-codes commented Sep 13, 2017 •
edited
Loading

DO NOT MERGE THIS

I want to use an alternative solution. But I don't want to close this PR right away until the alternative is working.

This is an example of how to resolve #21 for one project.

Depends on #24

This works, but the uploads are slow. It might be because the system is busy. It might be faster to run the assembly on the remote systems, but this can be changed later, if necessary.

@dazza-codes
Copy link
Contributor Author

When this runs, it looks something like this:

$ bundle exec cap ld4p_dev spark:upload_assembly
00:00 spark:upload_assembly
      01 mkdir -p /opt/ld4p-data-pipeline/current/lib
    ✔ 01 root@ld4p_dev_spark_master 0.757s
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 10.01%
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 20.01%
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 30.01%
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 40.01%
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 50.01%
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 60.01%
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 70.01%
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 80.01%
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 90.0%
      Uploading ./SparkStreamingConvertors/M21toBibFContinousConvApp/target/scala-2.11/M21toBibFContinousConvApp-assembly-1.0.0-SNAPSHOT.jar 100.0%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 10.11%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 20.22%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 30.11%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 40.22%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 50.1%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 60.21%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 70.1%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 80.21%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 90.09%
      Uploading ./SparkStreamingConvertors/target/scala-2.10/SparkStreamingConvertors-assembly-0.1-SNAPSHOT.jar 100.0%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 10.0%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 20.0%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 30.0%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 40.01%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 50.01%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 60.01%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 70.01%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 80.0%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 90.0%
      Uploading ./SparkStreamingConvertors/M21toBibFDumpConvApp/target/scala-2.11/M21toBibFDumpConvApp-assembly-1.0.0-SNAPSHOT.jar 100.0%

# etc for all the other spark nodes

@dazza-codes dazza-codes mentioned this pull request Sep 13, 2017
Open
3 tasks
@dazza-codes dazza-codes changed the title [Depends #24] Build spark jar [WIP, Depends #24] Build spark jar Sep 13, 2017
@dazza-codes
Copy link
Contributor Author

I want to use an alternative solution. But I don't want to close this PR right away until the alternative is working.

@dazza-codes dazza-codes deleted the build-spark-jar branch September 13, 2017 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Capistrano builds for various targets & sub-projects
1 participant
@dazza-codes