Getting the dependencies right
Motivation
Modules and models are typically interconnected by a network of dependencies of various types. Assuming you have understood the basic principles and categorisations of modules and models, as described at the MPS project structure page, we can now dive deeper as learn all the details.
Getting dependencies right in MPS is a frequent cause of frustration among inexperienced users as well as seasoned veterans. This page aims to solve the problem once and for all. You should be able to find all the relevant information categorised into sections by the various module and dependency types.
All programming languages have a notion of "imports". In Java you get packages and the "import" statement. In Ruby or Python you have "modules" and "require" or "import" statements. In MPS we provide a similar mechanism for packaging code and expressing dependencies in a way that works universally across languages - code is packages into models and these models can express mutual dependencies.
In addtion, since MPS is a multi-language development environment, models can specify the languages (aka syntaxes) enabled in them. This is different from when writing code in Java, Ruby or other languages, where the language to be used is given and fixed by conventions (e.g. the file extension).
Useful keyboard shortcuts
Whenever positioned on a model or a node in the left-hand-side Project Tool Window or when editing in the editor, you can invoke quick actions with the keyboard that will add dependencies or used languages into the current model as well as its containing solution.
Ctrl+L - Add a used language
Ctrl+M - Add a dependency
Ctrl+R - Add a dependency that contains a root concept of a given name
Ctrl+Shift+A - brings up a generic action-selection dialog, in which you can select the desired action applicable in the current context
Solution
Solutions represent programs written in one or more languages. They typically serve four purposes:
Regular solutions - these solutions hold end user code. The IDE typically generates the code on demand and compiles any generated Java code. The compiled code is not loaded into the IDE, but it can be run on demand in a separate Java process.
Sandbox solutions - these solutions are mostly used by language designers during the process of language design to play with the code in order to interactively evolve their languages. The IDE typically generates the code, but does not attempt to compile the generated code.
Runtime solutions - these solutions contain code that other modules (Solutions, Languages or Generators) depend on. The code can consist of MPS models as well as of Java classes, sources or jar files. The IDE will generate and compile the code and reload the classes whenever they get compiled or changed externally.
Plugin solutions - these solutions extend the IDE functionality in some way. They can contribute new menu entries, add side tool panel windows, define custom preference screens from the Project settings dialog, etc. Again, MPS will generate and compile the code and keep reloading the classes, whenever they change. Additionally, the IDE functionality will be updated accordingly.
We'll start with the properties valid for all solutions and then cover the specifics of sandbox, runtime and plugin solutions.
Common
Properties
Name - name of the solution
File path - path the module file
Generator output path - points to the folder, where generated sources should be placed
Module version - an internal version number of the module, should not be changed directly by a user
Left-side panel - contains model roots, each of which may hold one or more models.
Right-side panel - displays the directory structure under the model root currently selected in the left-side panel. Folders and jar files can be selected and marked/unmarked as being models of the current model root.
Model root types
Solutions contain model roots, which in turn contain models. Each model root typically points to a folder and the contained models lie in one or more sub-folders of that folder. Depending on the type of contained models, the model roots are of different kinds:
default - the standard MPS model root type holding MPS models
java_classes - a set of directories or jar files containing Java class files
javasource_stubs - a set of directories or jar files containing Java sources
Dependencies
The dependencies of a solutions are other solutions and languages, the models of which will be visible from within this solution.
The Export flag then specifies whether the dependency should be transitively added as a dependency to all modules that depend on the current solution. For example, of module A depends on B with export on and C depends on A, then C depends on B.
Used Languages
The languages as well as devkits that the solution's models may use are listed among used languages. Used languages are specified on the model level and the Used Languages tab on modules only shows a collection of used languages of its all models.
Java
This is where the different kinds of Solutions differ mostly.
The Java tab contains several options to customize the level of language features as well as the process of compilation and class-loading. Some of these options are only visible when the area with "more advanced" settings is expanded by clicking on the little arrow icon on the left side:
Not all combinations of these settings make sence so the UI will gray out the visual elements that are irrelevant with respect to the other selected options.
Language level - specifies the java language features that should be supported in the solution.
Compilation - specifies which compiler to use to compile the Java code generated from the module.
with MPS - indicates, whether the generated artifacts should be compiled with the Java compiler directly in MPS as part of the generation process
External compiler - use an external compiler. MPS doesn’t take any part in the compilation, it assumes an external compiler will take care of the compilation, but doesn’t mandate any compilation process to be performed. For example, for a java libraries containing compiled classes, there’s no compilation process, and the “external” option here means that MPS expects compilation to happen independently of MPS itself.
Request external IDEA compilation - (for internal use by MPS developers) use IntelliJ IDEA for compilation. This option is only useful for developing MPS itself and requires an additional plugin to work. The -Dmps.internal=true vm option must be enabled for this option to show up.
None - No compilation should be done. MPS does not expect this module to contain any classes and so will not include the module content in the classpath.
Compiler output path - specifies the folder in which to store the compiled code.
Class loader - specifies how the compiled classes should be loaded.
MPS - It tells MPS to manage classpath of a module based on the information available in JavaModuleFacet (namely, the output classes location and the java libraries + the dependencies on other MPS modules). Modules coming from various contributors have the classpath of the contributor as a parent classloader and can access e.g. plugin libraries (in case of a plugin contributor). The difference from the “Module contributor” option below is that MPS creates a module classloader (that respects classes_gen + module dependencies + parent classpath) when classloading by MPS, while relies solely on contributor’s classpath in case of “Module Contributor” (i.e. no MPS dependencies or classes_gen).
Module contributor - indicates that the code that provided the module is also responsible for providing a classloader for the module classes. Most common contributors are app/project libraries and plugins. E.g. a plugin contributor provides a classloader that adds the plugin classes (those in plugin-home/lib/*) in addition to the actual MPS module classes.
N/A - do not load the classes.
Contributes extensions to MPS - indicates whether the compiled classes should be loaded into MPS as its extensions (e.g. actions, menus, etc.). It is mostly used for scenarios when the code needs to integrate with IDEA or another plugin, i.e. when some classes have to be visible to both IDEA and MPS as coming from the same classpath.
Source Paths - Java sources that should be made available to other Java code in the project, i.e. to get included into module compilation along with generated sources.
Libraries - Java classes and jars that are required to be included on classpath at compile-time as well as at run-time.
Facets
Custom Generation - allows attaching a generator plan to the module. More on explicit generator plans is to be found at the Generator plans documentation.
Java - checked, if the solution relies on Java on some way. Keep this checked in most cases.
Tests - checked, if the solution contains test models
Solution models
Solutions contain one or more models. Models can be mutually nested and form hierarchies, just like, for example, Java packages can. The properties dialog hides a few configuration options that can be tweaked:
Dependencies
Models from the current or imported modules can be listed here, so that their elements become accessible in code of this model.
Used languages
The languages used by this model must be listed here.
Advanced
A few extra options are listed on the Advanced tab:
Do not generate - exclude this model from code generation, perhaps because it cannot be meaningfully generated
File path - location of the model file
Languages engaged on generation - lists languages needed for proper generation of the model, if the languages are not directly or indirectly associated with any of the used languages and thus the generator fails finding these languages automatically
Virtual packages
Nodes in models can be logically organised into hierarchies of virtual packages. Use the Set Virtual Package option from the node's context pop-up menu and specify a name, possibly separating nested virtual folder names with the dot symbol.
Adding external Java classes and jars to a project - runtime solutions
Runtime solutions represent libraries of reusable code in MPS. They may contain models holding MPS code as well as stub models that refer to external Java sources, classes or jar files. To properly include external Java code in a project, you need to follow a few steps:
Create a new Solution in your project.
Copy the desired jar files, classes or Java sources into a directory under the solution's root folder, such as "/libs".
In the Solution's properties dialog (Alt+Enter) attach the Java code. This requires a few steps:
First, in the left-hand side panel of the Common tab, which is used to manage model roots (i.e. locations of models that belong to the module), delete the existing model root unless you want to preserve or create code that you write in MPS as part of your runtime library.
Second, also in the Common tab, click on Add Model Root and select the desired type of files:
Java Classes - for classes or jars
Java sources - for Java sources
Kotlin Common - for Kotlin/Common libraries
Kotlin JVM - for compiled Kotlin/JVM libraries
Then navigate MPS to your lib folder (the folder holding the jar files or the folder holding the root of the package hierarchy of your Java classes or sources). This will add a new entry into the list of model roots. This entry will represent the "libs" folder.
Third, when the new model root is selected in the left-hand side panel, select the folder(s) or jar(s) listed in the right-side panel of the properties dialog and click the blue "Sources" button at the top of the right-hand side panel. The selected folders and jars will be shown with a blue icon next to them in the right-hand side panel.
Switch to the Dependencies tab and add the JDK solution as a dependency. Do not forget to turn on the Export flag. This will allow your jar files to refer to the JDK classes.
Then head to the Java tab to add all the jars or the classes root folders to the Libraries part of the window, otherwise the solution using the library classes would not be able to compile. When using java_sourcestubs, add the sources into the Source paths part of the Java tab window, instead. Also add any dependencies of the libraries that you have added to the module.
Additionally, still on the Java> tab, expand the panel hiding behind the Regular MPS module without contributions of MPS extensions label and set Contributes extensions to MPS option to Yes to have MPS reload the classes properly.
A new folder named stubs should appear in your solution
.
Now after you import the solution into another module (solution, language, generator) the classes will become available in that module's models.
The languages that want to use the runtime solution as a runtime dependency of the code that gets generated by the language's generator will need to refer to it in the Runtime Solutions section of the Runtime tab of their module properties.
Include the runtime jars in the build models - If you use the MPS build scripts to build a plugin with your languages you should also package the runtime libraries in the plugin by copying them into the desired location within the target layout definition.
If you forget to do so, you'll be getting errors like "No such path in local layout" when rebuilding the build script's model.
If your runtime solution uses libraries that are located outside of the project folder, your build script should specify the path to the libraries with a folder macro.
Language
Languages represent a language definition and consist of several models, each of which represent a distinct aspect of the language. Languages may contain one or more Generator modules. The properties dialog for languages is in many ways similar to the one of Solutions. Below we will only mention the differences:
Common
A language typically has a single model root that points to a directory, in which all the models for the distinct aspects are located.
Name - name of the language
File path - path the module file
Generator output path - points to the folder, where generated sources should be placed
Language version - a number representing the version of the language incremented with each new migration created in the language, the number should not be changed directly by a user
Module version - an internal version number of the module, should not be changed directly by a user
Left-side panel - contains model roots, each of which may hold one or more models.
Right-side panel - displays the directory structure under the model root currently selected in the left-side panel. Folders and jar files can be selected and marked/unmarked as being models of the current model root.
Dependencies
The dependencies of a language are other solutions and languages, the models of which will be visible from within this solution. The Export flag then specifies whether the dependency should be transitively added as a dependency to all modules that depend on the current language.
A dependency on a language offers thee Scope options:
Default - only makes the models of the other language/solution available for references (aggregation)
The structure aspects manifest languages they incorporate by aggregation (i.e. using a foreign concept in a child role), there's no need to import such languages explicitly into a model that uses the aggregating language or to 'extend' language modules.
Extends - allows the language to define concepts extending concepts from the there language
Generation Target - specifies that the current language is generated into the other language, thus placing a generator ordering constraint that the other language must only be generated after the current one has finished generating
Used Languages
This is the same as for solutions.
Runtime
Runtime Solutions - lists solutions of reusable code that the language requires. For more information about creating such a solution, refer to the "Adding external Java classes and jars to a project - runtime solutions" section.
Accessory models - lists accessory models that the language needs. Nodes contained in these accessory models are implicitly available on the Java classpath and the Dependencies of any model using this language.
Java
Source Paths - Java sources that should be made available to other Java code in the project, i.e. to get included into module compilation along with generated sources.
Libraries - Java classes and jars that are required to be included on classpath at compile-time as well as at run-time.
Facets
This is the same as for solutions.
Language models/aspects
Dependencies / Used Languages / Advanced
These settings are the same and have the same meaning as the settings on any other models, as described in the Solution section.
Generator
The generator module settings are very similar to those of other module types:
Common
Name - name of the generator module
File path - path the module file
Alias - a logical name representing the generator in the UI, since the name typically contains a rather lengthy value
Left-side panel - contains model roots, each of which may hold one or more models.
Right-side panel - displays the directory structure under the model root currently selected in the left-side panel. Folders and jar files can be selected and marked/unmarked as being models of the current model root.
Dependencies
This is the same as for solutions. Additionally generator modules may depend on other generator modules and specify Scope:
Default - only makes the models of the other language/solution available for references
Extends - the current generator will be able to extend the generator elements of the extended generator
Design - the target generator is only needed to be referred from a priority rule of this generator
Used Languages
This is the same as for languages.
Generators priorities
This tab allows to define priority rules for generators, in order to properly order the generators in the generation process. Additionally, three options are configurable through the check-boxes at the bottom of the dialog:
Generate Templates - indicates, whether the generator templates should be generated and compiled into Java, or whether they should be instead interpreted by the generator during generation
Java
Source Paths - Java sources that should be made available to other Java code in the project, i.e. to get included into module compilation along with generated sources.
Libraries - Java classes and jars that are required to be included on classpath at compile-time as well as at run-time.
Facets
This is the same as for languages.
Generator models
This is the same as for solutions.