Learning Patterns Your Source for Quality Technology Courseware

Introduction to Spark 3 with Scala: Lab Setup Instructions (Linux: Your Environment)

Below are the standard requirements for this course. If you have any questions or issues, please contact us.

Important Note: Student lab files are required on each computer used for the course. The links for these are not in this lab setup, and you should receive them separately.

Other notes:

Hardware and classroom setup.

Each student and the instructor shall have a work environment that fulfills the listed requirements.

Lab Files: Each student and instructor must have lab files installed (links to these files are generally sent separately via e-mail).

Other instructor requirements for the classroom

Install Java Development Kit – JDK 11 (11.0.x)

Spark-Shell Environment Setup

export SPARK_LABS=$HOME/spark-labs-scala

$ spark-shell

... Warnings and logging omitted ...

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://mycomputer2574:4040
Spark context available as 'sc' (master = local[*], app id = local-1655209447467).
Spark session available as 'spark'.

Welcome to

      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.1

Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 11.0.5)

Type in expressions to have them evaluated.

Type :help for more information.