# Quick Tips & Tricks I Learned Working With Spark

A small collection of tips & tricks I learned working with Spark so far, I hope it can help you as well. If you have more tricks, please let me know!

### Tip 1 - Importing dependencies on Spark-Shell

Most of the time, I want to use spark-shell with my project dependencies, it can be Cassandra Driver, Case Class or help method. I have learned the best way is just import the whole fatjar (I’m using sbt assembly) containing the driver program and all its dependencies.

### Tip 2 - Connecting to Cassandra from Spark-Shell

This trick is based on a small piece of the awesome post Installing the Cassandra / Spark OSS Stack from @AlTobey. I can connect to my Cassandra cluster from Spark Shell:

Note: it requires you bundle Spark Cassandra Connector inside your Driver Project.

### Tip 3 - Making Spark-Shell quieter

spark-shell is too verbose, I prefer a quieter version, so I’ve changed my log4j level from INFO to WARN and it’s much pleasant now. I installed Spark with homebrew, so my configuration is located here:

### Tip 4 - Avoiding namespace clashes

This is actually a Scala trick, we use io.smx as our internal namespace, and Spark has a namespace org.apache.spark.io so each time I try to use my ns it clashes with Spark’s

### Tip 5 - Running initialization commands on SparkShell - Similar .bashrc

When I’m working on a project I need the same initialization code on spark shell, for example, connect Spark-Cassandra driver. In order to achieve that, I just use the switcher -i:

\$ spark-shell -i ~/shell-init.scala --jars /projects/spark-cassandra-driver.jar


This is my shell-init.scala file:

Do you know something that turns working with Spark even more pleasant? Get in touch and lets improve this list!