DataSpell
 
2023.1
Shortcuts: Windows
Get DataSpell
You are viewing the documentation for an earlier version of DataSpell.

Big Data Tools

Last modified: 21 June 2023

The Big Data Tools plugin is available for DataSpell 2021.3 and later. It provides specific capabilities to monitor and process data with Zeppelin, AWS S3, Apache Spark, Apache Kafka, Apache Hive, Apache Flink, Google Cloud Storage, MinIO, Linode, DigitalOcean Spaces, Microsoft Azure, and Hadoop Distributed File System (HDFS).

You can create new or edit existing local or remote Zeppelin notebooks, execute code paragraphs, preview the resulting tables and graphs, and export the results to various formats.

User interface of the IDE with the Big Data Tools plugin enabled

Getting started with Big Data Tools in DataSpell

The basic workflow for big data processing in DataSpell includes the following steps:

Configure your environment

  1. Install the Big Data Tools plugin.

  2. Configure a connection to the target server.

  3. Work with your notebooks and data files.

Work with notebooks

  1. Create and edit a notebook.

  2. Execute the notebook.

  3. Analyze your data

Get familiar with the user interface

When you install the Big Data Tools plugin for DataSpell, the following user interface elements appear:

Big Data Tools window

The Big Data Tools window appears in the rightmost group of the tool windows. The window displays the list of the configured servers, notebooks, and files structured by folders. Even when no connections are configured, you can see the available types of servers to connect to.

Basic operations on notebooks are available from the context menu.

Big Data Tools window

For the basic operations with the servers, use the window toolbar:

Item

Description

Add connection

Add a new connection to a server.

Delete connection

Delete the selected connection.

Search in notebooks

For Zeppelin: find a note in your Zeppelin server.

For storages: navigate to a file.

Refresh Connection

Refresh connections to all configured servers.

Connection settings

Open the connection settings for the selected server.

Open in Editor

Only for storages. Open the storage in a separate tab of your editor

If you have any questions regarding the Big Data Tools plugin, click the Support link and select one of the available options. You can join the support Slack channel, submit a ticket in the YouTrack system, or copy the support email to send your question.

BDT support options

Notebook editor

Zeppelin notebook editor

In the notebook editor, you can add and execute Python code paragraphs. When editing your code paragraph, you can use all the coding assistance features available for a particular language. Code warnings and errors will be highlighted in the corresponding code constructs in the scrollbar. The results of paragraph execution are shown in the preview area below each paragraph.

Use the notebook editor toolbar for the basic operations with notebooks:

Item

Description

Run all

Executes all paragraphs in the notebook.

Stop execution

Stops execution of the notebook paragraphs.

Clear all outputs

Clears output previews for all paragraphs.

Additional actions

Select Export Note Code to HTML to save the note as an HTML file. Select Alter Code Visibility to hide code sections in paragraphs (by default, both code and result sections are shown). Select Show State Viewer Window to open State Viewer.

Interpreter bindings

Opens the Interpreter Bindings dialog to configure interpreters for the selected notebook.

Open in a browser

Click this button to open the notebook in the browser or copy a link to it.

Navigate

Allows you to jump to a particular paragraph of a notebook.

Minimap

Shows the minimap for quick navigation through the notebook.

A toolbar of a local note contains a list of available Zeppelin servers, so that you can select one to execute the note.

The notebook editor toolbar also shows the status of the last paragraph execution: Finished, Aborted, or Failed and the synchronization status of State Viewer.

Monitoring tool windows

These windows appear when you have connected to a Spark or Hadoop server.

Spark monitoring: jobs