Configure Big Data Tools environment

Install the required plugins

1. Whatever you do in DataSpell, you do it in a project. So, open an existing project (File | Open) or create a new project (File | New | Project).

2. Press Ctrl+Alt+S to open the IDE settings and select Plugins | Marketplace.

3. Install the Big Data Tools plugin.

4. Restart the IDE. After the restart, the Big Data Tools tab appears in the rightmost group of the tool windows. Click it to open the Big Data Tools window.

Configure a server connection

1. In the Big Data Tools window, click and select the server type. The Big Data Tools Connection dialog opens.

2. In the Big Data Tools Connection dialog, specify the following parameters depending on the server type:

   • File Systems: FS | Local, FS | HDFS

   • Storages: AWS S3, Minio, Linode, Digital Open Spaces, GS, Azure, Yandex Object Storage, Alibaba

   • Monitoring: Spark, Hadoop, Kafka

   • Notebooks: Zeppelin

   • Data Processing Platforms: AWS EMR

   

   Mandatory parameters:

   • Root path: a path to the root directory.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   

   Mandatory parameters:

   • Root path: a path to the root directory on the target server.

     When the connection is successfully established the Driver home path field shows the target IP address of connection including a port number. Example: hdfs://127.0.0.1:65224/.

   • Config path a path to the HDFS configuration files directory. See the samples of configuration files.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   • Enable tunneling (Only NameNode operation). Creates an SSH tunnel to the remote host. It can be useful if the target server is in a private network but an SSH connection to the host in the network is available. SSH tunneling currently works only for operators with the following name nodes: list files, get meta info

     Select the checkbox and specify a configuration of an SSH connection (click ... to create a new SSH configuration).

   Note that the Big Data Tools plugin uses the HADOOP_USER_NAME env variable to login to the server. It this variable is not defined then the user.name property is used.

   See more examples of the Hadoop File System configuration files.

   

   Mandatory parameters:

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Bucket filter and Filter type help define a specific set of buckets to preview and work with. You can set filters that contain, match, start with the pattern, or use regular expression.

   • Region: an AWS region of the specified bucket. You can select one from the list or let DataSpell to auto detect it.

   • Authentication type: the authentication method. You can use your account credentials (by default), or opt to entering the access and secret keys. You can also use a named profile that is located in the default AWS config location (~/.aws/credentials on Linux or macOS, or C:\Users\<USERNAME>\.aws\credentials on Windows). If needed you can specify any profile from a custom credential file.

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   • Use custom endpoint: select if you want to specify a custom endpoint and a signing region.

   • HTTP Proxy: select if you want to use IDE proxy settings or if you want to specify custom proxy settings.

   • Enable tunneling. Creates an SSH tunnel to the remote host. It can be useful if the target server is in a private network but an SSH connection to the host in the network is available.

   

   Mandatory parameters:

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Select if you want to specify a custom endpoint or a region. You can select a region from the list or let DataSpell to auto-detect it.

   • Authentication type: the authentication method. You can use your account credentials (by default), or opt to entering the access and secret keys. You can also use a named profile that is located in the default AWS config location (~/.aws/credentials on Linux or macOS, or C:\Users\<USERNAME>\.aws\credentials on Windows). If needed you can specify any profile from a custom credential file.

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   • HTTP Proxy: select if you want to use IDE proxy settings or if you want to specify custom proxy settings.

   • Click the

     Open SSH Key Settings to create an SSH connection authenticated with a private key file. You need to specify the Amazon EC2 key pair private key in the EMR SSH Keystore dialog.

   

   Mandatory parameters:

   • Endpoint: specify an endpoint to connect to.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Bucket filter and Filter type help define a specific set of buckets to preview and work with. You can set filters that contain, match, start with the pattern, or use regular expression.

   • Access credentials: Access Key and Secret Key.

   • HTTP Proxy: select if you want to use IDE proxy settings or if you want to specify custom proxy settings.

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   

   Mandatory parameters:

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Region: a region of the specified bucket. You can select one from the list or let DataSpell to auto detect it.

   • Bucket filter and Filter type help define a specific set of buckets to preview and work with. You can set filters that contain, match, start with the pattern, or use regular expression.

   • Access credentials: Access Key and Secret Key.

   • HTTP Proxy: select if you want to use IDE proxy settings or if you want to specify custom proxy settings.

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   

   Mandatory parameters:

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Region: a Digital Ocean region of the specified bucket. You can select one from the list or let DataSpell to auto detect it.

   • Bucket filter and Filter type help define a specific set of buckets to preview and work with. You can set filters that contain, match, start with the pattern, or use regular expression.

   • Access credentials: Access Key and Secret Key.

   • HTTP Proxy: select if you want to use IDE proxy settings or if you want to specify custom proxy settings.

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   

   Mandatory parameters:

   • Bucket: a name of the basic container to store your data in Google Storage.

   • Google app credentials: a path to the Cloud Storage JSON file.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Base directory (root by default): storage base directory.

   • Proxy: select if you want to use IDE proxy settings or if you want to specify custom proxy settings.

   

   Mandatory parameters:

   • Endpoint: specify an endpoint to connect to.

   • Container: a name of the basic container to store your data in Microsoft Azure.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Root path: a path to the root directory in the specified bucket.

   • Authentication type: the authentication method. You can access the storage by username and key, by a connection string, or using a SAS token.

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   

   Mandatory parameters:

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Bucket filter and Filter type help define a specific set of buckets to preview and work with. You can set filters that contain, match, start with the pattern, or use regular expression.

   • Authentication type: the authentication method. You can use your account credentials (by default), or opt to entering the access and secret keys. You can also use a named profile that is located in the default Yandex Object Storage config location. If needed you can specify any profile from a custom credential file.

   • HTTP Proxy: select if you want to use IDE proxy settings or if you want to specify custom proxy settings.

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   

   Mandatory parameters:

   • Region: an Alibaba OSS. Select one from the list.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   • Buckets: select All buckets in the account or specify Custom roots. If all buckets are selected, you can set up a filter to reduce the number of visible items.

   • Authentication type: the authentication method. You can use your account credentials (by default) or opt to enter the access and secret keys.

     You can also use a named profile that is located in the default OSS config location (~/.oss/credentials on Linux or macOS, or C:\Users\<USERNAME>\.oss\credentials on Windows). If needed you can specify any profile from a custom credential file.

   • HTTP Proxy: select if you want to use IDE proxy settings or if you want to specify custom proxy settings.

   

   Mandatory parameters:

   • URL: the path to the target server.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Enable tunneling. Creates an SSH tunnel to the remote host. It can be useful if the target server is in a private network but an SSH connection to the host in the network is available.

     Select the checkbox and specify a configuration of an SSH connection (click ... to create a new SSH configuration).

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   • Enable HTTP basic authentication: connection with the HTTP authentication using the specified username and password.

   • Enable HTTP proxy: connection with the HTTP proxy using the specified host, port, username, and password.

   • HTTP Proxy: connection with the HTTP or SOCKS Proxy authentication. Select if you want to use IDEA HTTP Proxy settings or use custom settings with the specified host name, port, login, and password.

   • Kerberos authentication settings: opens the Kerberos authentication settings.

     

     Specify the following options:

     • Enable Kerberos auth: select to use the Kerberos authentication protocol.

     • Krb5 config file: a file that contains Kerberos configuration information.

     • JAAS login config file: a file that consists of one or more entries, each specifying which underlying authentication technology should be used for a particular application or applications.

     • Use subject credentials only: allows the mechanism to obtain credentials from some vendor-specific location. Select this checkbox and provide the username and password.

     • To include additional login information into DataSpell log, select the Kerberos debug logging and JGSS debug logging.

       Note that the Kerberos settings are effective for all you Spark connections.

   

   Mandatory parameters:

   • URL: the path to the target server.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Enable tunneling. Creates an SSH tunnel to the remote host. It can be useful if the target server is in a private network but an SSH connection to the host in the network is available.

     Select the checkbox and specify a configuration of an SSH connection (click ... to create a new SSH configuration).

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   • Enable HTTP basic authentication: connection with the HTTP authentication using the specified username and password.

   • Enable HTTP proxy: connection with the HTTP proxy using the specified host, port, username, and password.

   • HTTP Proxy: connection with the HTTP or SOCKS Proxy authentication. Select if you want to use IDEA HTTP Proxy settings or use custom settings with the specified host name, port, login, and password.

   • Kerberos authentication settings: opens the Kerberos authentication settings.

     

     Specify the following options:

     • Enable Kerberos auth: select to use the Kerberos authentication protocol.

     • Krb5 config file: a file that contains Kerberos configuration information.

     • JAAS login config file: a file that consists of one or more entries; each specifies which underlying authentication technology should be used for a particular application or applications.

     • Use subject credentials only: allows the mechanism to obtain credentials from some vendor-specific location. Select this checkbox and provide the username and password.

     • To include additional login information into DataSpell log, select the Kerberos debug logging and JGSS debug logging.

       Note that the Kerberos settings are effective for all you Spark connections.

   • You can also reuse any of the existing Spark connections. Just select it from the Spark Monitoring list.

   

   Mandatory parameters:

   • URL: the path to the target server.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Properties: the list of configurable connection parameters. See Kafka reference documentation for more details. You can specify a file with the properties or the properties will be retrieved from Kafka documentation. You start typing a property name and press Ctrl+Space to get a target property.

   • Enable tunneling. Creates an SSH tunnel to the remote host. It can be useful if the target server is in a private network but an SSH connection to the host in the network is available.

     Select the checkbox and specify a configuration of an SSH connection (click ... to create a new SSH configuration).

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   • Click the question mark next to the Kafka support is limited message to preview the list of the currently supported features.

   

   Mandatory parameters:

   • URL: the path to the target server.

   • Login and Password: your credentials to access the target server.

   • Name: the name of the connection to distinguish it between the other connections.

   Optionally, you can set up:

   • Login as anonymous: select to login without using your credentials.

   • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

   • Library Versions: Scala Version, Spark Version, and Hadoop Version: these values are derived from the plugin bundles. If needed, specify any alternative version values.

   • Enable HTTP basic authentication: connection with the HTTP authentication using the specified username and password.

   • Proxy: connection with the HTTP or SOCKS Proxy authentication. Select if you want to use IDEA HTTP Proxy settings or use custom settings with the specified host name, port, login, and password.

   • Enable tunneling. Creates an SSH tunnel to the remote host. It can be useful if the target server is in a private network but an SSH connection to the host in the network is available.

     Select the checkbox and specify a configuration of an SSH connection (click ... to create a new SSH configuration).

   • Notifications. Select Enable cell execution notification if you want to be notified when execution time exceeds the specified time interval (60 seconds by default).

3. Once you fill in the settings, click Test connection to ensure that all configuration parameters are correct. Then click OK.

You can disable any connection if you temporarily do not need it. Right-click the corresponding item in the BigDataTools window and select Disable Connection from the context menu. The server changes its visual appearance and behavior: you cannot preview its content. To restore the connection, right-click it and select Enable Connection from the context menu.

For your convenience, you can rename the server root and copy a path to it. To quickly access all the required actions, right-click the target server in the BigDataTools window and select the corresponding command from the context menu.

Configure notebook dependencies

1. From the main menu, select File | Project Structure.

2. In the Project Structure dialog, select Modules in the list of the Project Settings. Then select any of the configured connections in the list of the modules and double-click System Dependencies.

3. Inspect the list of the added libraries. Click the list and start typing to search for a particular library.

   

4. If needed, modify the list of the libraries

   • Click to add a new library.

   • Click and specify the URL of the external documentation.

   • Click to select the items that you want DataSpell to ignore (folders, archives and folders within the archives), and click OK.

   • Click to remove the selected ordinary library from the library or restore the selected excluded items. The items themselves will stay in the library.