IntelliJ IDEA
 
Get IntelliJ IDEA
You are viewing the documentation for an earlier version of IntelliJ IDEA.

Create and run Spark application on cluster

Last modified: 11 February 2024

This tutorial covers a basic scenario of working with Spark: we'll create a simple application, build it with Gradle, upload it to an AWS EMR cluster, and monitor jobs in Spark and Hadoop YARN.

We'll go through the following steps:

  1. Create a new Spark project from scratch using the Spark project wizard. The wizard lets you select your build tool (SBT, Maven, or Gradle) and JDK and ensures you have all necessary Spark dependencies.

  2. Submit the Spark application to AWS EMR. We'll use a special gutter icon, which creates a ready-to-use run configuration.

  3. Monitor the application in Spark monitoring

  4. Open a Spark job from Hadoop YARN Resource Manager