Connect to BigQuery
BigQuery is a REST-based web service that allows you to run complex analytical SQL-based queries under large data sets.
To connect to BigQuery, you can use the following approaches:
You can jump to these sections if you already have a configured service account with the necessary permissions. Otherwise, consider reading Configuring the Google Cloud project.
Configuring the Google Cloud project
Step 1. Create or select a project
To create a project, select in the Navigation menu of the Google Cloud Platform. Alternatively, click the Create a project list on the Dashboard page and click New project.
To select an existing project, click a project that you want to use in the Select a project list on the Dashboard page.
Google user account
When you use authorization with the Google user account, you need to receive the authorization code in a web browser.
Step 1. Connect to BigQuery from PyCharm
In the Database tool window ( ) , click the Data Source Properties icon
.On the Data Sources tab in the Data Sources and Drivers dialog, click the Add icon (
) and select BigQuery.Check if there is a Download missing driver files link at the bottom of the data source settings area. As you click this link, PyCharm downloads drivers that are required to interact with a database. The IDE does not include bundled drivers in order to have a smaller size of the installation package and to keep driver versions up-to-date for each IDE version.
You can specify your drivers for the data source if you do not want to download the provided drivers. For more information about creating a database connection with your driver, see Add a user driver to an existing connection.
From the Authentication list, select Google User Account.
In the Project ID field, type the project identifier.
Usually, it is a part of the service account email identifier that goes after the at sign (
@). In our case, it isbigqueryproject-322409.To ensure that the connection to the data source is successful, click the Test Connection link.
From the Authorization Code Required dialog, cut the URL, paste it into the address bar of your web browser and press Enter to follow the URL.
Authorize access to your BigQuery application in your Google account.
Copy the authorization code received from Google, paste it in the Authorization Code Required dialog, and click OK.
To ensure that the connection to the data source is successful, click the Test Connection link.
(Optional) If you are connecting to a data source that contains a lot of databases and schemas, in the Schemas tab, select the schemas that you need to work with.
Click OK to create the data source.
Application Default Credentials
For authorization, the BigQuery driver uses data from a special file. By default, the path to the file is ~/.config/gcloud/application_default_credentials.json.
In PyCharm 2021.3.3 and later, you can set a custom location for the credentials file by using the GOOGLE_APPLICATION_CREDENTIALS environment variable.
Step 1. Create a service account
In the Navigation menu of the Google Cloud Platform, select .
On the Service accounts page, click Create service account.
On the Service account details step in the wizard, type a project name and description.
On the Grant this service account access to the project step in the wizard, select roles for this service account.
For example, for read-only access, select BigQuery Data Viewer, BigQuery Job User, and BigQuery User from the BigQuery menu. Alternatively, select for access to all resources within the project. For more information about roles and permissions, see Predefined roles and permissions at cloud.google.com.
Step 2. Generating service account keys
On the Service accounts page (), click the configuration icon in the Actions column and select Manage keys.
Click the Add key list and select Create new key.
In the Create private key dialog, select JSON and click Create.
The private key download is started automatically.
Step 3. Connect to BigQuery from PyCharm
In the Database tool window ( ) , click the Data Source Properties icon
.On the Data Sources tab in the Data Sources and Drivers dialog, click the Add icon (
) and select BigQuery.Check if there is a Download missing driver files link at the bottom of the data source settings area. As you click this link, PyCharm downloads drivers that are required to interact with a database. The IDE does not include bundled drivers in order to have a smaller size of the installation package and to keep driver versions up-to-date for each IDE version.
You can specify your drivers for the data source if you do not want to download the provided drivers. For more information about creating a database connection with your driver, see Add a user driver to an existing connection.
From the Authentication list, select Application Default Credentials.
Click the Advanced tab.
In the VM environment field, click the Edit environment variables icon.
In the Environment Variables dialog, click the Add button (
) and add the following variable: GOOGLE_APPLICATION_CREDENTIALS.
To ensure that the connection to the data source is successful, click the Test Connection link.
(Optional) If you are connecting to a data source that contains a lot of databases and schemas, in the Schemas tab, select the schemas that you need to work with.
Click OK to create the data source.
Google service account
You need to create a service account that is going to be used by your application. For more information about service accounts, see Service accounts at cloud.google.com.
Step 1. Create a service account
In the Navigation menu of the Google Cloud Platform, select .
On the Service accounts page, click Create service account.
On the Service account details step in the wizard, type a project name and description.
On the Grant this service account access to the project step in the wizard, select roles for this service account.
For example, for read-only access, select BigQuery Data Viewer, BigQuery Job User, and BigQuery User from the BigQuery menu. Alternatively, select for access to all resources within the project. For more information about roles and permissions, see Predefined roles and permissions at cloud.google.com.
Step 2. Generating service account keys
On the Service accounts page (), click the configuration icon in the Actions column and select Manage keys.
Click the Add key list and select Create new key.
In the Create private key dialog, select JSON and click Create.
The private key download is started automatically.
Step 3. Connecting to BigQuery from PyCharm
In the Database tool window ( ) , click the Data Source Properties icon
.On the Data Sources tab in the Data Sources and Drivers dialog, click the Add icon (
) and select BigQuery.Check if there is a Download missing driver files link at the bottom of the data source settings area. As you click this link, PyCharm downloads drivers that are required to interact with a database. The IDE does not include bundled drivers in order to have a smaller size of the installation package and to keep driver versions up-to-date for each IDE version.
You can specify your drivers for the data source if you do not want to download the provided drivers. For more information about creating a database connection with your driver, see Add a user driver to an existing connection.
From the Authentication list, select Google Service Account.
In the Service account email field, type the service account email identifier.
You can get the identifier on the Service accounts page () on the Google Cloud Platform (see Generating service account keys). In our case, it is
intellij-ide@bigqueryproject-322409.iam.gserviceaccount.com.In the Project ID field, type the project identifier.
Usually, it is a part of the service account email identifier that goes after the at sign (
@). In our case, it isbigqueryproject-322409.In the Key file field, type the full path to the JSON key file that you downloaded on Step 3. Generating service account keys.
To ensure that the connection to the data source is successful, click the Test Connection link.
(Optional) If you are connecting to a data source that contains a lot of databases and schemas, in the Schemas tab, select the schemas that you need to work with.
Click OK to create the data source.
Access and refresh tokens (OAuth 2.0)
When you use OAuth 2.0 for authentication, your users are authenticated after they agree to terms that are presented to them on a user consent screen. The OAuth consent screen is a dialog that displays a notification about who requests access to the user's data and a type of this data.
Before you generate a client ID and a client secret, you need to configure the OAuth consent screen.
Step 1. Create OAuth consent screen
In the Navigation menu on the Google cloud platform, click .
Follow the wizard and complete the configuration. For more information about user consent, see the User consent section at support.google.com.
Step 2. Getting a client ID and a client secret
In the Navigation menu on the Google cloud platform, click .
On the Credentials page, click .
From the Application type list, select Web application.
In the Authorised redirect URIs section, click Add URI and paste the following URI:
https://developers.google.com/oauthplaygroundClick Create.
In the OAuth client created window, you will see your client ID and client secret. Alternatively, you can click the created OAuth client ID configuration and get your credentials here.
Having a client ID and a client secret, you can generate authorization code and tokens.
Step 3. Generating access and refresh tokens at OAuth 2.0 Playground
Open the OAuth 2.0 Playground in a separate browser tab.
Click the OAuth 2.0 Configuration icon, select the Use your own OAuth credentials checkbox.
Paste your client ID and client secret in OAuth Client ID and OAuth Client secret fields.
Click Close in the OAuth 2.0 Configuration window.
In the wizard on Step 1. Select & authorize APIs, select .
Click Authorize APIs.
Follow the wizard in a browser and give permissions to the application.
On Step 2 Exchange authorization code for tokens, click Exchange authorization code for tokens.
Copy received refresh and access tokens.
Step 4. Connect to BigQuery from PyCharm
In the Database tool window ( ) , click the Data Source Properties icon
.On the Data Sources tab in the Data Sources and Drivers dialog, click the Add icon (
) and select BigQuery.Check if there is a Download missing driver files link at the bottom of the data source settings area. As you click this link, PyCharm downloads drivers that are required to interact with a database. The IDE does not include bundled drivers in order to have a smaller size of the installation package and to keep driver versions up-to-date for each IDE version.
You can specify your drivers for the data source if you do not want to download the provided drivers. For more information about creating a database connection with your driver, see Add a user driver to an existing connection.
From the Authentication list, select Access and Refresh Tokens.
In the ProjectId field, type your project name.
Usually, it is a part of the service account email identifier that goes after the at sign (
@). In our case, it isbigqueryproject-322409.In the Access token field, paste your access token.
In the Refresh token field, paste your refresh token.
In the Client ID field, paste your client ID.
In the Client secret field, paste your client secret.
To ensure that the connection to the data source is successful, click the Test Connection link.
(Optional) If you are connecting to a data source that contains a lot of databases and schemas, in the Schemas tab, select the schemas that you need to work with.
Click OK to create the data source.
