Remote development overview

Remote development lets you use the IDE interface on a thin client while having a powerful remote host to check out and load your project, index, analyze, build, run, debug, and test your code.

The remote host is a physical (Linux) or virtual machine hosting the source code and running a headless PyCharm instance. The thin client, for example, a laptop, connects to the host and transparently provides full access to all IDE features.

Architecture and definitions

To better understand bits and pieces of this architecture, you need to get familiar with the following definitions:

Server

This is a physical or virtual machine to host the source code and run the headless PyCharm. It performs most of the IDE features.

Requirements: Linux (macOS and Windows are coming later)

Client

This is a physical machine used by you to connect to the server and provide a thin client for working. For example, this could be your laptop. However, there are no restrictions for clients specifications.

Requirements: macOS, Linux, Windows

IDE Backend

This is a compatible JetBrains IDE running on the server. It is run as a server process, and therefore has no visible user interface - it’s headless. This is the same executable as you might currently be running for your JetBrains IDE.

JetBrains Client

This is a thin client you see after you launch JetBrains Gateway and establish a connection to a remote server. It gets connected to the IDE backend and gives you the ability to do remote development as if it is local.

In contrast to remote-desktop solutions, JetBrains Client has an advanced editor based on the JetBrains Rider’s RD protocol, which makes typing feel instant. Additionally, the advanced code intelligence implemented on the PyCharm server is presented in a way that looks as if it is running locally.

JetBrains Client is based on the IntelliJ platform, therefore it imports any local IDE settings found on the local machine (Client). For example, if there is the PyCharm 2021.3 version installed on Client, the downloaded and just launched JetBrains Client will import its settings, so the remote development experience will be personalized for you.

JetBrains Client's version is always equal to the version of the backen IDE.

JetBrains Gateway

This is an entry point for remote development. It runs on Client, and lets you create new remote environments or connect to the existing ones. After you request an environment in JetBrains Gateway, it launches JetBrains Client that is specific to that environment.

JetBrains Gateway is responsible for the following tasks:

Allowing you to set up a remote connection to any kind of remote backend you have
Exploring existing recent remote projects
Connecting to either the new or recent projects
Downloading JetBrains Client of the appropriate version from the indicated storage
Launching JetBrains Client
Maintaining the connection. It is the JetBrains Gateway process that configures and establishes a connection to Server (SSH or port forwarding or any other custom connection) and launches JetBrains Client against the opened connection or a port. Moreover, JetBrains Gateway is responsible for holding the connection, handling reconnects (with or without UI consent), and indicating the status information about the current connections.

Workflows

There are two types of main workflows available in remote development:

Server-to-client flow: when an organization first sets up a remote headless server, obtains a link (customized or not) which then can be used by you on Client (for example, your laptop) to connect to the available server.
Client-to-server flow: when you set up and install the remote server from the Client UI using the SSH or other connection.

Connect to a remote project at manually launched remote IDE (Server-to-client flow)

If you already have PyCharm installed on your remote server, you can launch it manually and connect to the remote project started in that IDE. It works the same way as from JetBrains Gateway.

Use this approach if your company has a custom orchestration or in case your remote IDE starts automatically on its side.

The process can be described as follows:

Start a backend project in the remote IDE.
Select one of the connection links generated by the backend.
Open the link on your local machine.

The main script to run a remote IDE is remote-dev-server.sh, located in the bin subdirectory of your unpacked IDE.

Connect to remote IDE

Ensure you have downloaded and installed JetBrains Gateway or an IDE with this bundled plugin on your local machine.
Ensure you have an SSH connection to the remote machine and the PyCharm IDE on it.
In the remote server's terminal, run the following command:
remote-dev-server.sh run /path_to_project/ --ssh-link-host host_server_address --ssh-link-user remote_side_user --ssh-link-port ssh_connection_port
See example:
If you don't pass the parameters, the script will use the default ones: 22 for port, system user's username, and host from the hostname command.
~/PY-213.5744.248/bin/remote-dev-server.sh run ~/home/jetbrains/PythonProjects/SampleApp/ --ssh-link-host ec2-13-50-136-85.eu-north-1.compute.amazonaws.com
If the project starts successfully, as an output, you should receive the following 3 links inside terminal:
- Join link: tcp://127.0.0.1:5990...[ ]: contains the local address and port where the remote IDE is listening now.
  Example:
  tcp://127.0.0.1:5991#jt=71e779a1-070e-4fd3-a4e6-83c1f8cb65a5&p=IU&fp=A42F6041C2878D7A22339ECABF4A9C294F643BG05E91E3DFC06035E03DB71BF8&cb=222.4167.29&jb=17.0.4b469.53
  To use it, ensure the remote machine should be accessible by this local address. For example, for inside-Docker IDE with forwarded/open ports.
  Check "ssh -L" for detailed information.
- Http link: https://code-with-me.jetbrains.com/remoteDev...[ ]: contains information about your host-port-user, the IDE and its version.
  Example:
  https://code-with-me.jetbrains.com/remoteDev#idePath=%2Fhome%2Fubuntu%2F.cache%2FJetBrains%2FRemoteDev%2Fdist%2F3e4cd27bfbb95_ideaIU-2022.2.2&projectPath=%2Fhome%2Fubuntu%2Fdevelopment%2Fspring-petclinic&host=ec2-xx-xx-xxx-xx.eu-north-1.compute.amazonaws.com&port=22&user=ubuntu&type=ssh&deploy=false
  When opening in your local browser, it displays a welcome page and tries to call the local Gateway application with pre-filled connection settings values.
  If no JetBrains Gateway application is found on the local machine, you'll be able to download it from the welcome page.
- Gateway link: jetbrains-gateway://connect#idePath... [ ]: also contains information about your host-port-user, IDE and its version.
  Example:
  jetbrains-gateway://connect#idePath=%2Fhome%2Fubuntu%2F.cache%2FJetBrains%2FRemoteDev%2Fdist%2F3e4cd27bfbb95_ideaIU-2022.2.2&projectPath=%2Fhome%2Fubuntu%2Fdevelopment%2Fspring-petclinic&host=ec2-xx-xx-xxx-xx.eu-north-1.compute.amazonaws.com&port=22&user=ubuntu&type=ssh&deploy=false
  When opening in your local browser, it launches the local JetBrains Gateway application directly without a welcome page.
Copy the generated link and paste it into your local browser and allow it to launch Open JetBrains Gateway.
All these links can be also opened in the already launched JetBrains Gateway.
For this, on the JetBrains Gateway welcome screen, paste a link in the Connect to a Running IDE field, click Connect.
PyCharm downloads the required version of JetBrains Client and opens the remote project inside it.

Registering previously installed remote IDE

Since version 221.5481, you can manually register an existing backend IDE on the remote server and make it visible for the Gateway.

To register the installed IDE and make it appear in the list of available builds follow these steps:

Enter remote server by SSH
Locate the folder with unpacked IDE, enter bin folder
In the command line, execute the following command:
remote-dev-server.sh registerBackendLocationForGateway
For example,
sh WebStorm-221.5591.52/bin/remote-dev-server.sh registerBackendLocationForGateway

You can use those links in the different connection scenarios such as connection via SSH, via custom connection, or via browser.

For the detailed information on how to start your Remote Development session, refer to Connect and work with JetBrains Gateway.

Connect using SSH

Start JetBrains Gateway.
Select Connect via SSH.
Invoke the SSH settings and configure options for the already running backend.
JetBrains Gateway connects by SSH, gets the connection link from the existing backend. JetBrains Gateway forwards a port from the connection link to the local machine and prepares a new connection link with that local-to-client port.
It also downloads and launches JetBrains Client with this link. The JetBrains Client version is specified in the link. JetBrains Gateway operates in the background and forwards the port while JetBrains Client is still active.

Configure the custom connection

Start JetBrains Gateway.
Select some custom connection UI provided by the JetBrains Gateway plugin.
A custom plugin may connect to some cloud service, fetch a list of backends from that service, create backends, and so on.
JetBrains Gateway gets the connection link (in most cases it points to the local host and port) from the custom plugin, and with this link, downloads and launches JetBrains Client. The JetBrains Client version is specified in the link. JetBrains Gateway may stay in the background in case it should maintain connection between Client and Server.

Connect through browser

In the browser, click a specifically created link that describes a backend location.
In the case of SSH, it contains a username, a server name, the SSH port, project location on the server, and so on.
This link leads to a landing page that suggests opening JetBrains Gateway or downloading it if JetBrains Gateway is missing on Client. JetBrains Gateway handles link opening from the browser and connects to a specified backend.

Client-to-server workflow

The client-to-server workflow means that you don't have any IDE installed on your remote server. In this case, you first need to download and install JetBrains Gateway. For more detailed information, refer to Launch JetBrains Gateway and connect to a remote server.

Connect via SSH

Download and install JetBrains Gateway.
In the JetBrains Gateway wizard, select Connect via SSH to connect to a remote server.
On the next page of the wizard, select the server to which you want to connect. If there is no IDE on the remote server, JetBrains Gateway will download it.
Depending on the user preference, the IDE may be downloaded from JetBrains servers, uploaded from the Client machine, or downloaded from a custom location.
The RD protocol is specific to each version of the IDE backend. The matching version of JetBrains Client needs to be downloaded on the server.

After this, JetBrains Gateway starts the PyCharm server on the remote machine, establishes the TCP tunnel, and starts JetBrains Client with the appropriate arguments to have it connected.

The TCP connection link format is as follows:

                tcp://127.0.0.1:PORT#jt=ONE_TIME_CONNECTION_TOKEN fp=SERVER_FINGERPRINT cb=THIN_CLIENT_BUILD jb=THIN_CLIENT_RUNTIME_VERSION
            

Check the following example:

                tcp://127.0.0.1:5990#jt=3f2f2471-84f2-4d8b-ac67-c1698f5a1be7 fp=ABC88C383B0A62654C4EF75052C60D4D475885075DCA5B85733BD8D4B9E28CC
                0 cb=213.2667 jb=11_0_11b1620.1
            

Extensibility

The IDE backend

The IDE backend is the full-blown JetBrains IDE, which is just being launched in a special headless unattended mode.

The backend can be extended with all the diversity of PyCharm plugins in the following ways:

By unpacking required plugins to plugins/ folder of the distribution
By running the following code (requires network connection to JetBrains Plugin Marketplace):
./bin/remote-dev-server.sh installPlugins <pathtoproject> <PLUGIN_ID1> <PLUGIN_ID2> ...

If a plugin provides the new set of inspections and features, all of those will be shown on JetBrains Client.

The only plugins that will not have any effect on Remote Development are the ones that aggressively modify the UI . Such plugins should be installed on the JetBrains Client side.

JetBrains Gateway SDK

JetBrains Gateway can be extended like any other IntelliJ platform-based product.

You can use one of the following ways:

Set up a new project with https://github.com/JetBrains/gradle-intellij-plugin/ (gradle-intellij-plugin should be >= 1.1.4)
Use the following settings to build your plugin against JetBrains Gateway:
intellij { version.set("213.2667-CUSTOM-SNAPSHOT") type.set("GW") instrumentCode.set(false) }
You may see available versions at https://www.jetbrains.com/intellij-repository/snapshots (see group com.jetbrains.gateway)

JetBrains Gateway / Thick PyCharm on a Client

Apart from the basic SSH and Code With Me connections, a vendor can customize JetBrains Gateway for its own orchestration service. This can be done within the custom deal between the JetBrains and the vendor.

JetBrains Gateway is based on the IntelliJ platform, it has APIs for connections and interactions with JetBrains Client.

Check the following example:

A big organization wants to write its own orchestration. The basic SSH flow is not enough due to security reasons. The organization writes an internal plugin and delivers it to its developers. Developers can install this plugin in JetBrains Gateway or in PyCharm on their laptops.

This is a very brief introduction of the APIs (they are not yet final and indeed this is not the full scope, but they explain the overall idea).

JetBrains Client

JetBrains Client is not designed to be extensible for the connection part. However, you can develop and install all the variety of PyCharm plugins, which modify the UI, keyboard shortcuts, themes, and other parts that touch the IDE UI interaction, but not its functionality.

Last modified: 08 November 2022