Spark
tip
Prior to PyCharm 2023.2, Spark was a part of the Big Data Tools plugin. In PyCharm 2023.2, the plugin was divided into six plugins. If you used Big Data Tools in 2023.1 or before, then after updating PyCharm to 2023.2, you will have all these new plugins automatically installed, including Spark.
note
If you install the Spark plugin, the following auxiliary plugins are automatically installed:
Metastore Core lets you connect to data processing and monitoring platforms, such as AWS EMR, Google Cloud Dataproc, or Hadoop YARN Resource Manager.
Remote File Systems lets you connect to remote storages, such as AWS S3, Google Cloud Storage, Microsoft Azure.
Big Data File Viewer provides a preview for Parquet, ORC, and Avro files.
Big Data Tools Core facilitates integration with big data tools — for example, it lets you share MFA and OAuth 2.0 data between AWS Glue authentication in Kafka schema registry and other AWS connections (such as AWS S3, AWS EMR, or AWS Glue monitoring).
Thanks for your feedback!