Spark monitoring

Create a connection to a Spark server

1. In the Big Data Tools window, click and select Spark under the Monitoring section. The Big Data Tools Connection dialog opens.

2. The Big Data Tools Connection dialog opens.

   

   Mandatory parameters:

   • URL: the path to the target server.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Enable tunneling. Creates an SSH tunnel to the remote host. It can be useful if the target server is in a private network but an SSH connection to the host in the network is available.

     Select the checkbox and specify a configuration of an SSH connection (click ... to create a new SSH configuration).

   • Per project: select to enable these connection settings only for the current project. Deselect it if you want this connection to be visible in other projects.

   • Enable connection: deselect if, for some reasons, you want to restrict using this connection. By default, the newly created connections are enabled.

   • Enable HTTP basic authentication: connection with the HTTP authentication using the specified username and password.

   • Enable HTTP proxy: connection with the HTTP proxy using the specified host, port, username, and password.

   • HTTP Proxy: connection with the HTTP or SOCKS Proxy authentication. Select if you want to use IDEA HTTP Proxy settings or use custom settings with the specified host name, port, login, and password.

   • Kerberos authentication settings: opens the Kerberos authentication settings.

     

     Specify the following options:

     • Enable Kerberos auth: select to use the Kerberos authentication protocol.

     • Krb5 config file: a file that contains Kerberos configuration information.

     • JAAS login config file: a file that consists of one or more entries, each specifying which underlying authentication technology should be used for a particular application or applications.

     • Use subject credentials only: allows the mechanism to obtain credentials from some vendor-specific location. Select this checkbox and provide the username and password.

     • To include additional login information into IntelliJ IDEA log, select the Kerberos debug logging and JGSS debug logging.

       Note that the Kerberos settings are effective for all you Spark connections.

3. Once you fill in the settings, click Test connection to ensure that all configuration parameters are correct. Then click OK.

Establish connection using the running job

1. In the notebook that involves any Spark execution, run the corresponding cell.

2. Click the Spark job link in the output area. In the displayed balloon message, select the Create connection link.

   

3. In the Big Data Tools Connections dialog, enter a unique connection name in the Name field. This is to distinguish between different opened connections in the Spark monitoring tool window.

4. You can click the Text connection button to check if the connection has been established successfully, then click OK to finalize configuring.

Adjust layout

• In the list of the application jobs, select a job to preview.

• To focus on a particular stage, switch to the Stages tab.

  

• To manage visibility of the monitoring areas, use the following buttons:

  Item	Description
  	Shows details for the selected stage.
  	Shows the list of the tasks executed during the selected stage.

  

• Click to preview any monitoring data in a browser.

Filter out the monitoring data

• Use the following buttons in the Applications, Jobs, and Stages tabs to show details for the jobs and stages with specific status.

  Item	Description
  	Show running applications, jobs, or stages
  	Show succeeded applications, jobs, or stages
  	Show failed jobs or stages
  	Show jobs or stages with unknown status
  	Show skipped stages

• Filter the list of applications by a start time and end time. Besides, you can specify the limit of the items in the filtered list.

  

• Manage content within a table:

  • Click a column header to change the order of data in the column.

  • Click Show/Hide columns on the toolbar to select the columns to be shown in the table: