Custom Persistence Cookbook

This document will use the xmlPersistence sample bundled with MPS to teach you how to define, deploy and use your own persistence formats.

What is custom persistence?

MPS normally saves models in its own XML-based format. However, there are cases when you may want to load or save model files in your own format.

Stub models

These are read-only models that typically represent library code. Suppose, for example, that we want to leverage MPS to describe one of the existing languages. BaseLanguage, for instance, is a good description of Java. In such a case, the libraries written for that original language should be accessible from within MPS. It is the stubs aspect of a language that helps to create stub models for such libraries. Stub models are then used as targets for references that need to reference the code outside of MPS.

Persistence for your language in your own custom format

Sometimes the default MPS persistence format for models is too heavyweight for your needs or you may want to use the persisted models in some other tools external to MPS and so you want to customize the persistence format. Another useful example - if all you need from MPS is to merely edit files of your own DSL using the MPS editor, it would be useful to store the model in text format so that it could be edited in any text editor.

Generic persistence for any MPS models in your own custom format

When you want to implement a generic persistence that would be in some respect superior to the three formats provided by MPS out-of-the-box.

Each of these cases requires slightly different handling.

Extension approaches

Generally, there are two approaches how you can tackle custom persistence:

Provide a custom ModelFactory that will create your own models when given a data source.
Provide a custom ModelRoot that will manage data sources and create models around these data sources on demand.

Terminology

SModel and EditableSModel (org.jetbrains.mps.openapi.model)

Interfaces that can represent a model or an editable model inside MPS. Can be loaded from persistence storage. Editable models can be saved to a persistent storage.

Noteworthy implementations:

SModelBase - a default SModel implementation
EditableSModelBase - a default EditableSModel implementation
RegularModelDescriptor - should be extended from for stub models
EditableModelDescriptor - should be extended from for custom persistence

SModelData (jetbrains.mps.extapi.model)

An interface that represents the model contents in memory. It holds and can manipulate its roots and nodes.

Noteworthy implementations:

jetbrains.mps.smodel.SModel - a default all purpose implementation
jetbrains.mps.smodel.DefaultSModel - an implementation that allows read optimization through the use of “headers”.

DataSource (org.jetbrains.mps.openapi.persistence)

Represents a location of the persisted data that represent a model. Files, internet content or database content are examples of possible data sources. FileDataSource and StreamDataSource are the most frequently utilized implementations in custom persistence.

DataSourceType (org.jetbrains.mps.openapi.persistence.datasource)

Uniquely identifies a Data Source kind. FileExtensionDataSourceType implementation class is used to represent data sources attached to files.

ModelRoot (org.jetbrains.mps.openapi.persistence)

Represents a logically connected group of models that come from a related physical origin, such as a file or a directory. It manages these models - creates, loads and saves them.

Noteworthy implementations:

ModelRootBase
FileBasedModelRoot

Memento

A generic abstraction of a hierarchical configuration information storage that preserves the information between sessions. Mementos, in essence, resemble hierarchically organized hashmaps.

SourceRootKind

Model roots can be of several kinds - typically sources, tests and excluded. This interface is used to communicate the desired/supported kinds of roots. The SourceTootKinds enumeration contains some predefined kinds.

ModelFactory (org.jetbrains.mps.openapi.persistence)

Represents a data source loading/saving/upgrading strategy. Its load(), create() and save() methods work with instances of the SModel and the DataSource interfaces.

ModelRootFactory

A factory for model roots. It must be registered in the plugin.xml file in order to take effect.

ModelRootEntry

A UI element that interacts with the user when a new model root is being specified.

ModelRootEntryFactory

A factory for ModelRootEntries. It must be registered in the plugin.xml file in order to take effect.

ModelLoadResult (jetbrains.mps.smodel)

Represents the result of model loading. It holds the loaded ModelData and a state of the loaded model data (jetbrains.mps.smodel.loading.ModelLoadingState)

Model id

Uniquely identifies a model. Other models use the id to represent the referenced model.

There are two options:

A model can derive its id from some reproducible fact, such as the location of its datasource.
A model can have an arbitrary id and store it in its datasource, typically at the beginning of a file. The model must reconstruct its id each time it is loaded.

The PersistenceFacade.createModelId() can be used to create an id from a description string. The string must have a special format 'factoryDesignator:text' (for example 'path:a/b/c.xml'), where:

factoryDesignator is an 'identification' of an id factory to use for constructing the id. These factories must have been registered through the PersistenceFacade.setModelIdFactory() method.
There are some factories already available:
- 'i' (IntegerSModelId) - creates an IntegerSModelId which gives models an integer value (parsed as hex from 'text'), models should persist the id themselves.
- 'path' (RelativePathSModelId) - creates a RelativePathSModelId which identifies a model with its relative path as specified in 'text'. The id can be reconstructed from the model’s datasource path.
- 'r' (RegularSModelId) - Uses a UUID parsed from 'text', models must take care of storing the ide.
- 'f' (ForeignSModelId) - Uses the string stored in 'text' directly, models must take care of storing the ide.
text - the text to pass to the factory as a seed information to construct the id.

SModelName (org.jetbrains.mps.openapi.model)

Represents a name of a model (for example jetbrains.mps.samples.xmlPersistence@generator), which normally consists of several parts:

Namespace (jetbrains.mps.samples)
Simple name (xmlPersistence)
Stereotype (@generator)

Model references

Represent a model within its containing module. They are part of the internal API and should be eliminated from the public API eventually.

PersistenceFacade

A singleton class that represents a registry of models and model root factories. It also provides helper methods to transform String to/from model.module references and node ids.

ModelSaveException

Thrown when the model does not fit the persistence format.

ModelLoadException

Thrown when the persistence format does not contain all the necessary data to construct the desired model.

ModelReadException

Used to report errors when loading a model.

Problem (org.jetbrains.mps.openapi.model.SModel)

Represents a persistence problem. It is typically reported through an exception and MPS will visualize the problem to the user eventually. It keeps references to the location of the problem as well as the node.

Noteworthy implementation:

jetbrains.mps.extapi.model.PersistenceProblem

Look around the sample xml model factory project

When this project is built and the generated plugin deployed into MPS, MPS will offer a new type of persistence for models, when they are being created. This new persistence will only allow the jetbrains.mps.xml.core language to be used in the model and persist the model into a single XML document.

Using the custom persistence plugin in MPS

After installing the generated plugin into MPS, when you are creating new models you are able to specify the XML file persistence provider for them:

Just as we specify in the XmlPersistenceModelDescriptor.importedLanguageIds() method, the new model will have the jetbrains.mps.core.xml language added as a used language and an empty Root Node will be added to it:

Whatever you type into the root node will be persisted immediately into a corresponding xml file. Changes to the underlying xml file will be reflected in the model upon opening in the editor.

Project structure

If you open the xmlPersistence sample project, you will see three solutions, each of which fulfills a separate role in the puzzle.

The xmlPersistence module defines the actual persistence logic, xmlPersistence.build contains a build script and xmlPersistence.ideaPlugin contains a customized plugin descriptor. Although the build script would normally provide a reasonable plugin descriptor by itself, this time we need to customize the descriptor, thus we include it in the project explicitly.

The plugin descriptor provides the standard plugin information plus it registers our jetbrains.mps.persistence.XmlModelPersistence class as mps.ModelFactoryProvider. This is the additional bit that requires us to provide an explicit plugin descriptor.

The persistence format

The xmlPersistence module implements the persistence logic. The persistence type in MPS is set on the per-model level. In our simplified case, the sample can store a model, which is restricted to a single XMLFile root element of the jetbrains.mps.core.xml language, into plain XML documents. The actual XML parsing logic resides in the XmlConverter class, while the XmlModelPersistence class implements the essential interfaces for hooking into the internal workings of MPS.

XmlPersistenceModelDescriptor

This class extends EditableModelDescriptor and will represent models in MPS.

These are the noteworthy points:

The importedLanguageIds() method ensures that instances of this model have the jetbrains.mps.core.xml language imported.
The save() method assumes a single root in the model, which will be persisted into the model file. It also assumes StreamDataSource. It leverages a helper RegularTextUnit class that converts a model in the jetbrains.mps.core.xml language into xml text.
The createModel() method checks whether the model file exists.
- If not, an new empty in-memory model (SModel instance) is returned.
- If the file exists, it is read, parsed and converted to an SModel instance with a single root XmlFile in it.

XmlModelPersistence

This class extends ModelFactory and handles the creation and persistence lifecycle of XmlPersistenceModelDescriptor instances.

supports()

The supports() method checks the type of the data source. We only accept FileSystemBasedDataSource and StreamDataSource at the same time. The method must be called from all persistence-related methods manually to prevent ClassCastExceptions from within these methods.

getPreferredDataSourceTypes()

This method returns a collection of DataSourceTypes that this ModelFactory handles. This implementation returns its own XML_TYPE, which is a type based on a file extension.

getType()

This method returns an instance of ModelFactoryType that describes the model factory. There are three built-in types in MPS, all defined in the PreinstalledModelFactoryTypes enum.

Our model factory defines its own type:

The getFormatTitle() method is worth a special mention here, since the string it returns will be used to represent the storage format to the future users.

create()

This method will create a new instance of XmlPersistenceModelDescriptor and initialize it with path-based id. Then it adds a new root node with the name identical to the name of the model to represent an empty model.

load()

This method will create a new instance of XmlPersistenceModelDescriptor and initialize it with path-based id. The model itself is not loaded at this point.

save()

This method delegates to the save() method of the model.

Look around the sample stub project

The propertyPersistence sample project demonstrates the use of model roots to implement stub models for property files. Folders on disk become models and property files in them become root nodes of these models.

Using the custom stub persistence

Once the plugin is deployed to MPS, users will be able to set a new type of model root in their solutions - Property files.

If this model root is selected, all folders marked by the user as 'Sources' (in the right-hand side panel) that contain property files (files with the .properties extension) will become models.

These models (named 'client' and 'server' in the image above) are stub models for the property files. They are read-only and other models in the project can reference them. The language used to describe property files in MPS, called jetbrains.mps.samples.PropertyDefinition, is also part of the propertyPersistence plugin.

The plugin project structure

The project contains the jetbrains.mps.samples.PropertyDefinition language and a sandbox solution to test the language on simple property definitions.

The jetbrains.mps.samples.propertyPersistence.build solution contains a build script that packages three of the modules into an MPS plugin.

Custom plugin.xml definition is used.

It registers our custom factories as mps.modelRootFactory and mps.modelRootEntry as extensions to MPS so that they can take effect. Additionally, the location of the packaged language is specified with mps.languageLibrary.

Stub Models implementation

The actual implementation of all the necessary classes is in the jetbrains.mps.samples.propertyPersistenceDef solution.

Each of the classes has a well-defined purpose:

PropertyFilesDataSource - a FolderDataSource extension that only specifies that files with the .properties extension from the current folder should be included in the data source.
PropertyFilesStubModelRootEntryFactory - a factory that creates instances of PropertyFilesStubModelRootEntry when asked by MPS.
PropertyFilesStubModelRootEntry - ensures that the user can specify and configure this kind of model root in the UI of a module. Since our model root uses files and folders, this class only delegates to FileBasedModelRootEntry.
PropertyFilesStubModelDescriptor - the model implementation that extends RegularModelDescriptor. It parses property files, creates root nodes and populates freshly created models with them. Similarly to the xmlPersistence the model descriptor class also ensures that a proper language is imported (jetbrains.mps.samples.PropertyDefinition) into the models when created. Since stub models are read-only, there is no need to save modes.
PropertyFilesStubModelRootFactory - a factory that creates instances of PropertyFilesStubModelRoots when asked by MPS.
PropertyFilesStubModelRoot - the actual implementation of a model root. It extends FileBasedModelRoot

PropertyFilesStubModelRoot

Some more details about this class:

loadModels() - this method gets all the directories marked by the user as SOURCE and for each directory recursively creates PropertyFilesStubModelDescriptor instance.
getType() - this method returns a string identifier of the root type. The identifier is used in the plugin.xml to identify associated factories.
getSupportedFileKinds1() - returns all supported file kinds (SOURCES, TESTS, EXCLUDED, etc.) which the user can choose from to mark files and directories with when setting up the model root in the module settings dialog.
load(Memento) - reads the path of the model root from the memento. It has been saved by the parent FileBasedModelRoot class previously so we rely on it being present in the memento. Using the path it then sets the contentDirectory and source root properties. Note: This method serves as illustration of working with mementos. In fact it is not needed in this example as the parent class already reads the memento and sets the contentDirectory as well as source root.

Build the MPS plugin

Both, the xmlPersistencePlugin and the propertyPersistencePlugin projects contain standard build scripts that zips the essential modules into IDEA/MPS plugins.

After rebuilding a build script you should be able to run the build script and get the plugin generated:

This will create the plugin so we can distribute it to the users.

Debugging the plugin

MPS can also give you a hand when you want to test your fresh persistence implementation right away, directly in MPS. You create a Run Configuration off the Deploy Plugins template:

Then you specify your plugin (after rebuilding the project and re-running the build script) to be deployed by the Run Configuration:

Finally, you run the configuration to get the plugin installed into MPS (MPS will restart).

After this step you will be able to use the custom persistence provider for models and test whether it behaves as expected.

The plugin will be installed into the .MPSxxx sub-directory of your home folder on Windows, or $HOME/Library/Application Support/MPSxxx on Mac. You may uninstall or disable the plugin through the Plugin Manager UI or by deleting the plugin folder manually.

Build the IntelliJ IDEA plugin

Building an IntelliJ IDEA plugin isn't really that much different from building an MPS plugin. You only need to change the dependency from mps to mpsPlugin in the build script and set the artifacts location to wherever your MPS IDEA plugin has been deployed.

After rebuilding the project and running the build script, you get a plugin to deploy into IDEA.

Using the custom XML persistence plugin in IntelliJ IDEA

The IDEA plugin also allows for XML documents to be edited with an MPS-based projectional editor and yet persisted into plain xml files. Since .xml files are associated with the XML editor in IDEA, we get the default IDEA's editor open when we click on an XML file.

However, since the sample.xml file is located under the configured MPS models root, MPS will invoke our custom persistence plugin and have it build a model out of it. When you hit Alt+Insert to open a class by name, you get the option to open the sample model:

Then you can edit it in a projectional editor and have your changes persisted into the original xml file:

Last modified: 08 March 2023