Object-Oriented Meets Functional
Scala is relatively new language based on the JVM. The main difference between other “Object Oriented Languages” and Scala is that everything in Scala is an object. The primitive types that are defined in Java, such as int or boolean, are objects in Scala. Functions are treated as objects, too. As objects, they can be passed as arguments, allowing a functional programming approach to writing applications for Apache Spark.
If you have programmed in Java or C#, you should feel right at home with Scala with very little effort.
You can also run or compile Scala programs from commandline or from IDEs such as Eclipse.
Test on HDP 2.3.2 with Spark 1.4.1
Run Sandbox and connect to it from SSH
Let’s open a shell to our Sandbox through SSH:_
|
1 |
ssh -p 2222 root@127.0.0.1 |
Launch Spark-Shell
To learn and experiment with data, I prefer the interactivity of the Scala shell. Let’s launch the Spark shell as it is a fully capable Scala shell.
|
1 |
spark-shell --master yarn-client --driver-memory 512m --executor-memory 512m |
Values
Values can be either mutable or immutable. Mutable values are expressed by the var keyword.
|
1 2 3 |
var a: Int = 5 a = a + 1 println(a) |
Immutable values are expressed by the val keyword.
|
1 2 3 |
val b: Int = 7 b = b + 1 //Error println(b) |
Type Inference
Scala like Java is a strongly typed language. But, it is very smart in type inference, which alleviates the onus of specifying types from the programmers.
|
1 2 3 4 5 6 7 8 |
var c = 9 println(c.getClass) val d = 9.9 println(d.getClass) val e = “Hello“ println(e.getClass) |
Functions
Functions are defined with the def keyword. In Scala, the last expression in the function body is returned without the need of the return keyword.
|
1 2 3 4 5 6 |
def cube(x: Int): Int = { val x2 = x * x * x x2 } cube(3) |
You can write the function more succinctly by leaving out the braces and the return type as return type can be easily inferred.
|
1 2 |
def cube(x: Int) = x*x*x cube(4) |
Anonymous Functions
Anonymous functions can be assigned as a var or val value. It can also be passed to and returned from other functions.
|
1 2 |
val sqr: Int => Int = x => x * x sqr(2) |
Or, anonymous functions can be further shortened to
|
1 2 |
val thrice: Int => Int = _ * 3 thrice(15) |
where _ is a the shorthand for whatever the input was.
Collections
Scala has very convenient set of collections including Lists, Sets, Tuples, Iterators and Options. When you combine these data structures with anonymous fucntions and closures it is very expressive.
|
1 2 3 4 5 6 7 |
val strs = Array("This", "is", "happening") strs.map(_.reverse) strs.reduce(_+""+_) strs.map(_.reverse) strs.reduce(_+" "+_) |
This is far from comprehensive. To learn more visit http://scala-lang.org