syntax.us Let the syntax do the talking
Blog Contact Posts Questions Tags Hire Me

Question:
In Spark how to control INFO logging?

When I started working with Spark I noticed the logging level is way too chatty.

I control the logging level by creating a file here:
~/spark/conf/log4j.properties
I create the file from this file:
~/spark/conf/log4j.properties.template
Now the file looks like this:
# Set everything to be logged to the console
log4j.rootCategory=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=ERROR
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=ERROR
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=ERROR
Now I see only interesting output and my Spark scripts run faster.

syntax.us Let the syntax do the talking
Blog Contact Posts Questions Tags Hire Me