Skip to content

Commit

Permalink
Make the mini example that we will include in ch02
Browse files Browse the repository at this point in the history
  • Loading branch information
holdenk committed Sep 10, 2014
1 parent ea3b245 commit f761657
Show file tree
Hide file tree
Showing 5 changed files with 12 additions and 22 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ src_managed/
project/boot/
project/plugins/project/
sbt/*.jar
mini-complete-example/sbt/*.jar

# Scala-IDE specific
.scala_dependencies
Expand Down
7 changes: 6 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,9 @@ script:
- "export IAM=`whoami`"
- sudo pip install urllib3
- sudo mkdir -p /user/hive && sudo chown -R $IAM /user/hive
- "./run-all-examples"
- "./run-all-examples"
# Now do our mini example
- cd mini-complete-example
- ./sbt/sbt clean compile package assembly
- ./sbt/sbt clean
- mvn clean && mvn package
18 changes: 3 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
![buildstatus](https://travis-ci.org/holdenk/learning-spark-examples.svg?branch=master)(https://travis-ci.org/holdenk/learning-spark-examples)
Examples for Learning Spark
===============
Examples for the Learning Spark book.
Examples for the Learning Spark book. These examples require a number of libraries and as such have long build files. We have also added a stand alone example with minimal dependcies and a small build file
in the mini-complete-example directory.

Requirements
==
Expand All @@ -15,19 +16,6 @@ Requirements
* R & the CRAN package Imap are required for the ChapterSixExample
* The Python examples require urllib3

Scala examples
===

You can build and run the Scala examples with sbt, just run
sbt/sbt compile package run

Java examples
===

You can build and run the Java examples wih maven, just run
mvn package
mvn exec:java -Dexec.mainClass="com.oreilly.learningsparkexamples.java.[EXAMPLE]"

Python examples
===

Expand All @@ -39,5 +27,5 @@ Spark Submit
You can also create an assembly jar with all of the dependcies for running either the java or scala
versions of the code and run the job with the spark-submit script

./sbt/sbt assembly
./sbt/sbt assembly OR mvn package
cd $SPARK_HOME; ./bin/spark-submit --class com.oreilly.learningsparkexamples.[lang].[example] ../learning-spark-examples/target/scala-2.10/learning-spark-examples-assembly-0.0.1.jar
6 changes: 1 addition & 5 deletions mini-complete-example/build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,7 @@ scalaVersion := "2.10.4"

// additional libraries
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.0.2",
)

resolvers ++= Seq(

"org.apache.spark" %% "spark-core" % "1.0.2"
)

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<modelVersion>4.0.0</modelVersion>
<name>examples</name>
<packaging>jar</packaging>
<version>1.0</version>
<version>0.0.1</version>
<repositories>
<repository>
<id>Akka repository</id>
Expand Down

0 comments on commit f761657

Please sign in to comment.