Scala notes

From raju

maniplulating dataframes

declare but not initialize a variable

Scala has very weird syntax to declare but not initialize a variable.

In the interpreter, I use

var x: Type = _

For example

var a: DataFrame = _
var b: Int = _
var c: Array[String] = _

In a function, I use

var x: Type = null.asInstanceOf[Type]

multiple variables vs tuples

This creates two variables

scala> var (a,b) = (3,4)
a: Int = 3
b: Int = 4

This creates a tuple

scala> var x = (3,4)
x: (Int, Int) = (3,4)

scala> x._1
res25: Int = 3

scala> x._2
res26: Int = 4

Filter dataframe on multiple columns

df.filter("month = 'JAN' and year > 2000").show()

where df is a DataFrame with month and year columns in it.

count the number of items in each group


where df is a DataFrame with a "month" column. Sample output will look like

|  month|   count|
|    JAN|12345678|
|    FEB| 2345678|

Add two data frames

    val df = df1.unionAll(df2)

Ref:- -> search for unionAll

tags | append two data frames

Read tab separated files

    def tsv2df(tsvFile : String) : DataFrame = {
        val df =
            .option("header", "true") // Use first line of all files as header
            .option("inferSchema", "true") // Automatically infer data types
    def df2parquet(df : DataFrame, parquetFile: String) : Unit = {

Also demonstrates | how to return a value

External Links