syntax.us Let the syntax do the talking
Blog Contact Posts Questions Tags Hire Me

Question:
In H2O how to breakpoint droplet in IntelliJ?

After spending time with the H2O Sparkling Water examples, I gained enough skill to want to build a small application which connects Spark Scala code with H2O.

I started my effort by installing Spark on my Ubuntu-Virtualbox using information at the URL below:

spark_howto_install

After I installed Spark, I installed IntelliJ and found that installation to be easy.

Then I attended a Meetup at H2O and learned that H2O.ai offers a github-repository aimed directly at this use-case.

I git-cloned the repo with these shell commands:
dan@feb ~ $ 
dan@feb ~ $ 
dan@feb ~ $ git clone https://github.com/h2oai/h2o-droplets.git
Cloning into 'h2o-droplets'...
remote: Counting objects: 182, done.        
remote: Total 182 (delta 0), reused 0 (delta 0), pack-reused 182        
Receiving objects: 100% (182/182), 72.23 KiB | 0 bytes/s, done.
Resolving deltas: 100% (82/82), done.
Checking connectivity... done.
dan@feb ~ $ 
dan@feb ~ $ 
dan@feb ~ $ cd h2o-droplets/
dan@feb ~/h2o-droplets $ 
dan@feb ~/h2o-droplets $ ll
total 64
drwxrwxr-x  6 dan dan  4096 Feb 26 06:44 ./
drwx------ 74 dan dan 28672 Feb 26 06:44 ../
drwxrwxr-x  8 dan dan  4096 Feb 26 06:44 .git/
drwxrwxr-x  4 dan dan  4096 Feb 26 06:44 h2o-java-droplet/
drwxrwxr-x  4 dan dan  4096 Feb 26 06:44 h2o-scala-droplet/
-rw-rw-r--  1 dan dan   393 Feb 26 06:44 README.md
drwxrwxr-x  5 dan dan  4096 Feb 26 06:44 sparkling-water-droplet/
dan@feb ~/h2o-droplets $ 
dan@feb ~/h2o-droplets $ 
dan@feb ~/h2o-droplets $ cd sparkling-water-droplet/
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ ll
total 112
drwxrwxr-x 5 dan dan 4096 Feb 26 06:44 ./
drwxrwxr-x 6 dan dan 4096 Feb 26 06:44 ../
-rw-rw-r-- 1 dan dan 5122 Feb 26 06:44 build.gradle
drwxrwxr-x 2 dan dan 4096 Feb 26 06:44 data/
-rw-rw-r-- 1 dan dan  292 Feb 26 06:44 .gitignore
drwxrwxr-x 4 dan dan 4096 Feb 26 06:44 gradle/
-rw-rw-r-- 1 dan dan   23 Feb 26 06:44 gradle.properties
-rwxrwxr-x 1 dan dan 5080 Feb 26 06:44 gradlew*
-rw-rw-r-- 1 dan dan 2404 Feb 26 06:44 gradlew.bat
-rw-rw-r-- 1 dan dan 1849 Feb 26 06:44 README.md
-rw-rw-r-- 1 dan dan   46 Feb 26 06:44 settings.gradle
drwxrwxr-x 3 dan dan 4096 Feb 26 06:44 src/
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ ll
total 120
drwxrwxr-x 7 dan dan 4096 Feb 26 06:46 ./
drwxrwxr-x 6 dan dan 4096 Feb 26 06:44 ../
drwxrwxr-x 5 dan dan 4096 Feb 26 06:46 build/
-rw-rw-r-- 1 dan dan 5122 Feb 26 06:44 build.gradle
drwxrwxr-x 2 dan dan 4096 Feb 26 06:44 data/
-rw-rw-r-- 1 dan dan  292 Feb 26 06:44 .gitignore
drwxrwxr-x 4 dan dan 4096 Feb 26 06:44 gradle/
drwxrwxr-x 3 dan dan 4096 Feb 26 06:45 .gradle/
-rw-rw-r-- 1 dan dan   23 Feb 26 06:44 gradle.properties
-rwxrwxr-x 1 dan dan 5080 Feb 26 06:44 gradlew*
-rw-rw-r-- 1 dan dan 2404 Feb 26 06:44 gradlew.bat
-rw-rw-r-- 1 dan dan 1849 Feb 26 06:44 README.md
-rw-rw-r-- 1 dan dan   46 Feb 26 06:44 settings.gradle
drwxrwxr-x 3 dan dan 4096 Feb 26 06:44 src/
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
So that worked well.

Next I decided to work with this directory because it fit my use-case exactly:

https://github.com/h2oai/h2o-droplets/tree/master/sparkling-water-droplet

Then I ran a shell command recommended by README.md:
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ ./gradlew build
:compileJava UP-TO-DATE
:compileScala
Download https://repo1.maven.org/maven2/ai/h2o/sparkling-water-core_2.10/0.2.10/sparkling-water-core_2.10-0.2.10.pom
Download https://repo1.maven.org/maven2/ai/h2o/sparkling-water-examples_2.10/0.2.10/sparkling-water-examples_2.10-0.2.10.pom
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-scala_2.10/0.1.27.1056/h2o-scala_2.10-0.1.27.1056.pom
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-app/0.1.27.1056/h2o-app-0.1.27.1056.pom
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-core/0.1.27.1056/h2o-core-0.1.27.1056.pom
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-web/0.1.27.1056/h2o-web-0.1.27.1056.pom
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-algos/0.1.27.1056/h2o-algos-0.1.27.1056.pom
Download https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.3.0/jackson-core-2.3.0.pom
Download https://repo1.maven.org/maven2/ai/h2o/sparkling-water-core_2.10/0.2.10/sparkling-water-core_2.10-0.2.10.jar
Download https://repo1.maven.org/maven2/ai/h2o/sparkling-water-examples_2.10/0.2.10/sparkling-water-examples_2.10-0.2.10.jar
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-scala_2.10/0.1.27.1056/h2o-scala_2.10-0.1.27.1056.jar
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-app/0.1.27.1056/h2o-app-0.1.27.1056.jar
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-core/0.1.27.1056/h2o-core-0.1.27.1056.jar
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-web/0.1.27.1056/h2o-web-0.1.27.1056.jar
Download http://h2o-release.s3.amazonaws.com/h2o-dev/master/1056/maven/repo/ai/h2o/h2o-algos/0.1.27.1056/h2o-algos-0.1.27.1056.jar
:processResources UP-TO-DATE
:classes
:jar
:assemble
:compileTestJava UP-TO-DATE
:compileTestScala UP-TO-DATE
:scalatest
[ant:scalatest] Discovery starting.
[ant:scalatest] Discovery completed in 133 milliseconds.
[ant:scalatest] Run starting. Expected test count is: 0
[ant:scalatest] DiscoverySuite:
[ant:scalatest] Run completed in 308 milliseconds.
[ant:scalatest] Total number of tests run: 0
[ant:scalatest] Suites: completed 1, aborted 0
[ant:scalatest] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[ant:scalatest] No tests were executed.
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test UP-TO-DATE
:check UP-TO-DATE
:build

BUILD SUCCESSFUL

Total time: 55.14 secs
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 
dan@feb ~/h2o-droplets/sparkling-water-droplet $ 


Next, I loaded a file named build.gradle into IntelliJ.

After bit of poking at it I was able to get IntelliJ to create a new project.

Next I loaded the Scala plugin into IntelliJ.

The steps I followed for that task came from Google:

http://www.google.com/search?q=how+to+load+Scala+plugin+into+IntelliJ

To effectively write code in any language, I want the ability to step through the code after a breakpoint.

For this use-case I wanted to step through the syntax in this script:

https://github.com/h2oai/h2o-droplets/blob/master/sparkling-water-droplet/src/main/scala/water/droplets/SparklingWaterDroplet.scala

I follow two steps to do this in IntelliJ:
  • click on the LHS where I want the break-point.
  • click the green-run-debug control


That is how it is supposed to work.

Fortunately I knew that was how it was supposed to work.

To get it to actually work, I spent an hour trying a variety of experiments and tinkerings.

If you are new to IntelliJ and you want to use it to step through SparklingWater code, be prepared to get confused by thoroughly buggy behavior.

On my laptop, I can now use IntelliJ to walk through H2O-Spark-Scala scripts using the two step procedure described above.

If you have problems getting it to work, I suggest you tinker with it in a variety of ways like reinstalling it and trying different copies of the H2O repo.

Also IntelliJ often wants to wander into an unusable state where gets really busy reading thousands of files in the file system.

If you see this behavior just recognize that you have 40 minutes to run errands while IntelliJ takes over your laptop.

If you are diligent and patient, it is possible to get IntelliJ into a state where it can step through a 20 line Scala script.

syntax.us Let the syntax do the talking
Blog Contact Posts Questions Tags Hire Me