Use cloud storage data sources

1. Open the notebook.

2. Go to Main menu | Tools | Attached data or click the Attached data icon on the left-hand sidebar.

3. Click New connection and select Cloud storage.

   

4. In the New connection dialog, select Amazon S3.

   

5. In the New Amazon S3 cloud storage connection dialog, fill in the following fields:

   • Display name: to specify the name for this data source in your system

   • AWS access key and AWS secret access key: to access your AWS account (details here)

   • Region: to specify your AWS region

   • Amazon Bucket name: to specify the name of the bucket you want to mount

   • Custom options: to specify additional parameters. See the example below

     tip

     You can use the endpoint_url parameter to connect to other bucket providers from this list.

     note

     When using an IAM role associated with Datalore for authentication, httpPutResponseHopLimit>1 is required.

   • Endpoint URL: to specify the website of the bucket you want to mount

   

6. (Optional) Click Test connection to make sure the provided parameters are correct.

7. Click Create data source to finish the procedure.

1. Open the notebook.

2. Go to Main menu | Tools | Attached data or click the Attached data icon on the left-hand sidebar.

3. Click New connection and select New cloud storage.

4. In the New connection dialog, select Google cloud storage.

5. In the New Google cloud storage connection dialog, fill in the following fields:

   • Display name: to specify the name for this bucket in your system

   • GCS Bucket name: to specify the name of the bucket you want to mount (details here).

   • GCS key file content: to enter the content of the Google service account key file (.json format).

   

6. (Optional) Click Test connection to make sure the provided parameters are correct.

7. Click Create data source to finish the procedure.

1. Open the notebook.

2. Go to Main menu | Tools | Attached data or click the Attached data icon on the left-hand sidebar.

3. Click New connection and select New cloud storage.

4. In the New connection dialog, select SMB/CIFS.

5. In the SMB/CIFS cloud storage connection dialog, fill in the following fields:

   • Display name: to specify the name for this data source in your system

   • Host: to specify the SMB server hostname

   • Port: to specify the SMB port

   • Username and Password: to specify the user credentials

   • Domain (Optional): to specify the SMB server domain

   • Custom options: to specify additional parameters

   

6. (Optional) Click Test connection to make sure the provided parameters are correct.

7. Click Create data source to finish the procedure.

This procedure adds a cloud storage data source to your workspace resources without attaching it to any notebook automatically. Such data sources are later available for all notebooks of the respective workspace.

1. Select Cloud storages from the menu on the left. This will open the Cloud storages list.

   

2. Click New connection in the upper right corner of the list.

3. In the New cloud storage connection, select the cloud storage type.

4. Proceed by following the steps described for the respective cloud storage type.

Cloud storage data sources added to a workspace or attached to a specific notebook are available across the entire workspace and can be attached to any notebook from it.

1. Open the notebook.

2. Select Main menu | Tools | Attached data.

3. In the Attached data tool, click Select data to attach and select the required data source from the list.

   

1. Go to Main menu | Tools | Attached data or click the Attached data icon on the left-hand sidebar. You will see all your data sources including attached buckets.

2. To change access type, click the pencil icon next to the bucket type and select your option (Read-only access or Read-write access).

   

3. To add file to the bucket storage, click Upload files, select your option, and complete the procedure accordingly. You csn find more details about uploading or creating file in Attached files.

4. Click the ellipsis to use more options:

   • Open detail view: to view and manage the stored files

   • Edit cloud storage: to edit the connection parameters for the storage

   • Copy directory path: to copy the path to the storage to the clipboard

   • Connect using boto3: to connect to the cloud storage using boto3 in the notebook code (for Amazon S3 only)

   • Pass credentials to env: to pass the cloud storage credentials to the environment

   • Detach cloud storage: to detach the storage from the notebook

   

1. Select Cloud storage from the menu on the left. This will open the list of your bucket data sources.

2. To edit the parameters of a data source, click the respective list item and make your changes in the Edit [data_source_name] connection dialog.

3. To rename a data source, right-click the list item, select Rename from the menu, and provide a new name.

4. To clone the item to other workspaces:

   1. Right-click the item.

   2. Select Clone to other workspaces.

      

   3. In the Clone [data_source_name] to other workspaces dialog, expand the Workspaces dropdown list.

   4. Select the workspaces where you want to clone the data source and click anywhere outside the dropdown.

      

   5. Click the Clone button. This will close the dialog, followed by a success notification.

5. To delete a data source, right-click the respective list item and select Delete from the menu.

The Custom_options is a field used for optional parameters when creating an Amazon S3 data source. Below are two example of how it can be used.

• To enable SSE-C for S3 data sources, specify the following in the Custom_options: In the Custom_options field, specify the following:

  use_sse=c:/path/to/keys/file

  where:

  /path/to/keys/file is the file that contain keys. Make sure permissions are 600.

• (For Datalore Enterprise only) To provide access based on a role associated with that of an EC2 instance profile, add public_bucket=0,iam_role into the Custom_options field.