Shared indexes

One of the possible ways of reducing the indexing time is by using shared indexes. Unlike the regular indexes that are built locally, shared indexes are generated once and are later reused on another computer whenever they are needed.

PyCharm can build and use shared indexes for your project's code. Whenever PyCharm needs to reindex your application, it will use the available shared indexes and will build local indexes for the rest of the project. Normally, this is faster than building local indexes for the entire application from scratch.

Shared project indexes

Shared project indexes are built for project sources and their dependencies. By generating indexes on a single computer and distributing them to other computers, you can avoid duplicating the indexing process on each machine. This way, each computer can benefit from the precomputed indexes without having to perform the expensive indexing operations individually. This is the main advantage of shared indexes over ordinary indexes.

Using shared indexes is reasonable for large projects, where indexing might take a lot of time, creating inconveniences for the teams involved. For smaller projects, we recommend other ways of reducing the indexing time.

Make sure the plugin is installed

To be able to use shared project indexes, the Shared Project Indexes bundled plugin must be enabled in the settings:

Press Ctrl+Alt+S to open settings and then select Plugins.
Switch to the Installed tab, type Shared Project Indexes, and make sure that the checkbox next to it is selected.
Otherwise, select the checkbox to enable the plugin.
Apply the changes and close the dialog. Restart the IDE if prompted.

Install the plugin

To be able to use shared project indexes, install the Shared Project Indexes plugin:

Press Ctrl+Alt+S to open settings and then select Plugins.
Switch to the Marketplace tab and type Shared Project Indexes in the search field.
Once the plugin is found in the list of available plugins, click Install next to it.
Apply the changes and close the dialog. Restart the IDE if prompted.

Before you begin

The computer that builds project shared indexes and any computers that use the project shared indexes:

Should use the same IDE version to ensure index compatibility.
Can have different operating systems.
However, in previous PyCharm versions, shared project indexes were OS-specific. Refer to the documentation that corresponds to your IDE version by using the version switcher in the top-left corner of this page.

Before incorporating shared project indexes into your infrastructure, you can evaluate how much time they can save on indexing by running the command-line tool.

If you are satisfied with the results, you can keep using shared indexes. In more complex scenarios, you can customize sample scripts from the IntelliJ Shared Indexes Tool Example repository.

Evaluate indexing time with shared indexes

Before using shared project indexes in your workflow, you can see how much time they are able to save you. During evaluation, the tool will generate some files in the directory that you have specified.

Download the latest version of the command-line tool and unzip the archive.
In the command line, navigate to the directory of the command line tool and execute the binary file by running:
.\bin\ij-shared-indexes-tool-cli.bat boost
./bin/ij-shared-indexes-tool-cli boost
./bin/ij-shared-indexes-tool-cli boost
with the following options:
- --ij: (mandatory) path to a locally installed IDE.
  The path to the locally installed IDE may differ depending on how it was installed.
- --project: (mandatory) path to the project for which you want to build shared indexes.
- --port: port for running a local server. The default port is 25561.
- --data-directory: directory for storing generated indexes and server data. The default directory is ./ij-shared-indexes-tool-data.
For example:
./bin/ij-shared-indexes-tool-cli boost --ij "/Users/jetbrains/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-1/232.7754.76/PyCharm 2023.2 EAP.app" --project "/Users/jetbrains/PycharmProjects/mySampleProject"
Wait for some time. After that, you will see the indexing time with shared indexes and without them.

By looking at the difference in time, you can decide whether you want to use shared project indexes in your workflow or not.

Generate project indexes

Use these steps to generate index files. After the indexes have been generated, you will need to upload the files to a file server or S3-compatible storage.

Download the latest version of the command-line tool and unzip the archive.
In the command line, navigate to the directory of the command line tool and execute the binary file via the command line by running:
.\bin\ij-shared-indexes-tool-cli.bat indexes
./bin/ij-shared-indexes-tool-cli indexes
./bin/ij-shared-indexes-tool-cli indexes
with the following options:
- --ij: (mandatory) path to a locally installed IDE.
- --project: (mandatory) path to the project for which you want to build shared indexes.
- --base-url: (mandatory) Indexes server URL. The base URL that will be used to define the full URL where shared indexes are hosted.
- --data-directory: directory for storing generated indexes and server data. The default directory is ./ij-shared-indexes-tool-data. You should upload files from this directory to the server specified in the base-url parameter.
Once the project indexes have been generated, you will see a message with the information required to configure shared indexes in your project.

After you have generated indexes, two folders appear in the directory you have specified when executing the indexes command: server and temp. The index files are stored within the server directory.

Upload files to the storage

After you have generated indexes, two folders appear in the directory you have specified when executing the indexes command: server and temp. The index files are stored within the server directory.

Uploading shared indexes to a remote storage

Upload the data and the project folders from the server directory to your file server or S3-compatible storage. The storage location should have the same base URL as the URL that was defined using the base-url parameter when generating the indexes.

Publishing shared indexes when testing locally

When testing shared indexes, you can use the server command to start a server on the generated files.

In the command line, navigate to the directory of the command line tool and execute the binary file by running:
.\bin\ij-shared-indexes-tool-cli.bat server
./bin/ij-shared-indexes-tool-cli server
./bin/ij-shared-indexes-tool-cli server
with the following options:
- --port: port for running a local server.
- --server-directory: directory for server data. The default directory is ./ij-shared-indexes-tool-data/server.
For example:
.\bin\ij-shared-indexes-tool-cli.bat server --port 8080 --server-directory "C:\Users\jetbrains\ij-shared-indexes-tool-data\server"
./bin/ij-shared-indexes-tool-cli server --port 8080 --server-directory "/Users/jetbrains/ij-shared-indexes-tool-data/server"
./bin/ij-shared-indexes-tool-cli server --port 8080 --server-directory "/home/jetbrains/ij-shared-indexes-tool-data/server"

Configure access to shared indexes

Once the project indexes are uploaded to the file storage, they can be downloaded and applied on another computer.

In the project directory, create a new file intellij.yaml and copy the information from the terminal to the new file, for example:
If the project hasn't been indexed yet or you have cleared the system cache, the IDE will download the shared indexes and use them.
In the default configuration, the IDE will download the shared indexes automatically. You may choose to change this and confirm every download manually.

Project indexes will be downloaded to index/shared_indexes in the IDE system directory.

When the indexes are being downloaded, the progress is displayed in the status bar.

When everything is ready, you will see a corresponding notification in the Notifications tool window.

Change the way indexes are downloaded

In the default configuration, the IDE will be downloading shared indexes automatically. You can confirm every download manually.

From the intellij.yaml file, remove the consents section and restart your project.
In the Settings dialog (Ctrl+Alt+S) , select Tools | Shared Indexes and in the Project Shared Indexes area, select Ask before download.
Apply the changes and close the dialog.

Shared indexes CLI command reference

boost

Measures indexing speed up with project shared indexes. Options:

--project: (mandatory) Path to a project.
--ij: (mandatory) Path to a locally installed IDE.
--port: Port for running a local server. The default port is 25561.
--data-directory: Directory to use for generation and server data.

indexes

Generates project shared indexes and makes them ready to be uploaded. Options:

--project: (mandatory) Path to a project.
--ij: (mandatory) Path to a locally installed IDE.
--base-url: (mandatory) Indexes server URL.
--data-directory: Directory to use for generation and server data.

server

Starts a server on local indexes. Options:

--port: port for running a local server. The default port is 25561.
--server-directory: directory for server data. The default directory is ./ij-shared-indexes-tool-data/server.

version

Displays shared indexes version information. Options:

--ij: (mandatory) path to a locally installed IDE..

You can keep using shared indexes from the command line. In more complex scenarios, you can customize sample scripts from the IntelliJ Shared Indexes Tool Example repository.

Last modified: 11 October 2024