Generator cookbook

How does the generator process the rules?

Generation gradually converts an input model into an output model, which may or may not be then turned into text with TextGen. The generation process itself consists of steps. Each step has three phases:

1. Executing pre-mapping scripts

2. Template-based model transformation

3. Executing post-mapping scripts

The template-based model transformation phase consists of one or more micro-steps. A micro-step is a single-pass model transformation of and input model into a transient (output) model. The generator during a generation step will iterate over the model and search in the generators of all languages involved in this step for rules applicable to the nodes in the input model. When no applicable rules are found, the generation step stops. The next generation step (if any) will receive the output model of previous generation step as its input.

You can check out full description of the generator algorithm in the generator documentation.

Can I have more generators for a single language?

No, MPS only allows one generator per language at the moment. If you need several generators for a language, it is advisable to create empty languages that extend your language and in each of these extending languages to define the desired alternative generators.

Can I have multiple mapping configurations per generator?

Yes, they will all be treated as equal during the generation process. Additionally, each mapping configuration  can have its generation process priority specified separately, which gives you more flexibility to tune the generation.

How to generate multiple targets from single source?

You may want to generate code for several target platforms from the same code-base. There are at least two ways you can achieve that:

• Put all the generation rules for different platforms into a single generator, perhaps logically separated into virtual packages and multiple mapping configurations, and use the  condition  of each rule to activate the rule based on the currently selected target. The developer will indicate the intended target platform by setting a  flag  in her code

• Have no generator in you language directly and provide generators for each target platform in a separate  empty  language, which extends the original language. The developer will select the desired target platform by importing the appropriate extension language.

What macros are available in the templates?

• property macro  - computes a value for a property

• reference macro  - computes the target (node) of a reference

• node macro  - is used to control template filling at generation time

  • $IF$ - conditional generation

  • $CALL$ - insert another template at the current position and process it passing in the current node

  • $LOOP$ - iterate over a collection of nodes and set the current node for each iteration

  • $COPY_SRC$ - copies the specified node and replaces the wrapped node. Reduction rules are applied to the copied node and its children during the process

  • $COPY_SRCL$ - copies the specified collection of nodes and replaces the wrapped code. Reduction rules are applied to the copied nodes and their children during the process

  • $MAP_SRC$ - sets a node to be used as the current node inside the wrapped code

  • $MAP_SRCL$ - sets a collection of nodes, each of which should to be used in turn as the current nodes inside the wrapped code

  • $SWITCH$ - chooses the template to use for generation from multiple choices

  • $LABEL$ - stores the wrapped node in a  mapping label  for easy discovery by other macros

  • $VAR$ - sets a value into a variable, which will then be accessible in the wrapped nodes through  genContext.varName

  • $TRACE$ - stores information required to trace back the original node for the wrapped node and stores the mapping between a source node and the resulting generated text into the trace.info file - when Save Transient Models  is on, this enables the Reveal Origin Node  option in the Debug  pop-up menu item in transient models

  • $WEAVE$ - invokes a specific weaving rule

  • $INSERT$ - inserts a node into the output model at the current position

Where shall I put utility classes?

Create a new model inside the generator. Make sure it does not have the generator  stereotype attached to it. The model should typically depend on  BaseLanguage  so that you can create classes in it. The original generator  model should then import the utility model.

Where shall I put runtime classes?

Classes and libraries that the generated code relies on should be put into separate  runtime solutions, as described in the "Adding external Java classes and jars to a project - runtime solutions" section of Getting the dependencies right.

How do I generate unique names?

Use the genContext  parameter, which gives you access to the generator context object and call:

unique name from <base name> in context <node>

base name  - is an arbitrary string that must be part of the generated name

context node  - If specified, then MPS tries its best to generated names 'contained' in a scope (usually a root node). Then when names are re-calculated (due to changes in input model or in generator model), this won't affect other names outside the scope. The  context node  parameter is optional, though we recommend to specify it to guarantee generation stability.

The uniqueness of generated names is secured throughout the whole generation session.
 Clashing with names that weren't generated using this service is still possible.

How to generate enumeration datatypes?

Having to reduce an enumeration datatype into a Java enum, using the $SWITCH$  macro is usually the best option.

Notice the way the actual enumeration datatype is tested for equality using the is()  method:

How to I change packages of the generated Java classes?

When generating complex Java code, Java classes, interfaces and enums frequently need to be organized into packages. By default, the MPS generator reflects the structure of MPS models in the structure of the packages of the generated Java code. Each BaseLanguage model translates into a Java package with the same name. If you need to manipulate the structure of the generated Java packages, use the  packageName property of the  Classifier  concept. The property is available for root classifiers:

What is the difference between root mapping rules and reduction rules?

During generation models, which have the form of a tree (Abstract Syntax Tree, see Basic notions ), are translated into the generated models, which are also trees. Typically root nodes are generated into root nodes (e.g. a robot Script into a Java class) and non-root nodes into non-root nodes (e.g. a robot Step command into a Java BlockStatement ). Thus templates for root mapping rules represent a starting point of the generated model and cannot specify template context, unlike reduction templates, which may hold context and indicate the actual generation contents with the  template fragment marks.

What happens to untransformed roots?

Roots, for which there are no applicable transformation rules, are carried unchanged over to the next generation step.

How to generate multiple nodes for a single node?

A template fragment in a template indicates, which node will be replacing the current input node. A template fragment can only be attached to a single node inside the template, but a generator template can contain multiple template fragments, provided the are all attached to nodes in the same role under the same parent. This way multiple nodes can be generated for a single input node.

Why is editing the macros so weird?

Indeed, the generator macros and template fragment marks are a bit rough to edit at first. But once you understand the underlying principles of how they work, you'll be able to use them efficiently.

The macros have been implemented as  Node/property/reference attributes (see the  Attributes  section of the  Structure  chapter). As such, they can be used in any language and do not require any up-front support to be available in that language. This is important, because any language can be used as a generation target in MPS. Macros are thus attributes attached to the node that they wrap. Deleting or replacing the node wrapped by a macro will delete the macro as well and you have to re-enter the macro to attach it to a new node. A recommended approach is to enter the code for the template first and then start adding macros. If you need to change the node wrapped by a macro, you can at least preserve the  Inspector window contents on the macro on the old node by copy-pasting it into the macro on the new node.

How can I generate root nodes for non-root nodes?

Simply create a root mapping rule for the concept, perhaps further restrict it with a condition. The rule is called root, because it generates a root node, not necessarily that it takes a root as input.

Alternatively you can either use  conditional root rules, which insert root nodes based on evaluating a boolean predicate, or with  pre-processing scripts, which can inspect the input model and create root nodes as needed.

How to remove no-longer needed roots from the model?

Root nodes are removed automatically once they have been transformed using root mapping rules. For roots that are not directly transformed use the abandon root rule to prevent them from being propagated into the next generation step.

How do I generate nodes for nodes in accessory models?

Nodes in the accessory models are not handled by the generator in any way. In order to generate nodes based on nodes in accessory models, you can leverage either the  conditional root rules, or  pre-procesing scripts.

Reduction rules can then be used to further generate the "inserted nodes" into the desired target language.

How to use mapping labels?

Mapping labels are defined in mapping configurations (e.g.  main ). They must then be populated by wrapping the node that you want to be stored with macros, such as $LOOP$, $LABEL$, $COPY_SRC$ and others.

How to handle references during generation?

Reference macros should be used to set proper target nodes on generated references. You typically use reference macros together with  mapping  labels.

The fundamental task here is that for a reference  R that points to a node  N, we need to ensure that a reference G(R) generated from  R must point to a node  G(N) that was generated from  N.

if R -> N, N generates to G(N), R generates into G(R) then G(R) -> G(N)

Mapping labels serve as dictionaries, which store nodes as keys and whatever they have been generated into as values - N -> G(N) in our example here. The generator has to be instructed explicitly to store the mapping between a node and its generated node into a mapping label. Template fragments as well as node macros allow the user to specify a mapping label into which the generated node should be saved.

When reducing  R into G(R), a reference macro on  G(R) can retrieve  G(N) from the mapping label using  N as a key when calling the generator context in genContext.get output by label and input.

Chapter 7 of the generator tutorial covers mapping labels in more detail.

Can I debug the generation process?

The first option is to enable saving transient models. The intermediate models that MPS creates during generation will be preserved and you can access them in the  Project View  panel on the left-hand side of your screen:

The saved transient models allow you to investigate how the models looked like at various stages of the generation process.

The second option is using the Generator Tracer Tool. For this tool the "Save transient models" option also has to be set to on. The Generator Tracer Tool gives you the ability to investigate in detail the generation process for a selected node. When you select a node in your original model or in any of the transient ones, the context menu gives you the options to either trace the generation process forward or backward. As a result you get a tree-shaped report giving you access to all the stages, through which the particular node went during generation and the rules that influence its generation.

Additionally, the $TRACE$  generator macro gives you the option to mark parts of your templates for tracing, so that you will be able to navigate back from a generated node in a transient model to its original node using the Reveal origin node  option in the Context menu -> Language Debug  menu:

How do I figure out the actual generation steps?

Use the Show Generation Plan  action from the generator context menu. This will give a detailed report showing all the planned generation phases and listing the mapping configurations run in each phase.

Can I specify the order in which generators should be run?

The Generator priorities  tab in the generator's Module properties dialog enables you to specify ordering rules between two distinct mappings configurations.

For more complex scenarios, the new functionality of Generation plans gives you full control over the generator process.

How to check that the model has no errors before starting the generator?

Run the Check model (language/solution)  context-menu action on the model/language/solution to check.

The  Preferences -> Build, Execution, Deployment -> Generator -> Check models for errors before generation configuration flag specifies whether the model checker should be run automatically each time the generator is triggered.

How to extend an existing generator?

There is one rule to follow - make sure your extending generator is run before the extended one - give it higher priority in the generator order priority dialog.

How to build an extensible generator?

By extending the generator, extensions can alter the semantics of the original language. The MPS generator is extensible by design – it resolves all generator rules and mapping configurations from all involved languages and builds a global generation plan. Any language that is attached to the project will have its rules included. The plan specifies the execution order for the generator rules based on their mutual relative priorities expressed in mapping configurations. This enables language extensions to inject their own desired generation rules into the most suitable generation phase. Since priorities are expressed as a collection of relative ordering between mapping configurations, a language extension does not need to know about all other generators involved in generation in order to work. Potential (and rare) clashes are detected and left up to the developer to resolve. Once created, the generation plan is used to iteratively invoke the generators, potentially leveraging parallelism of the underlying hardware for mutually independent rules.

Providing additional reduction rules is one way to extend a language. Using a Generator Switch is another option. If the parent language uses a generator switch to choose the right reduction rules, the language extension may extend that generator switch with its additional logic for picking the reduction rules – typically to include new rules contributed by extension languages.

What shall I use TextGen for?

While the generator performs model-to-model transformations, TextGen does mapping to text. This is typically the last step in generation and is fully managed by the generator. Roots are picked by the generator and converted to files. The TextGen intentionally offers very little flexibility to the language designer. It is the Generator, where the generation process should primarily be configured and handled. If you feel a need to workaround the limitations of TextGen, you're most likely trying things the wrong way. Generator is the place where you have the most flexibility.

How to copy the generated output to a different location?

The CopyOutcome attribute of the jetbrains.mps.lang.makeup language can be used to ensure that the final generated text for a node gets copied into a specified location.

How to generate multiple files per root node?

TextGen only allows one file per root node. Also, you can only have one TextGen component per concept. The flexibility should be encoded into the generator, instead. You get the generator create multiple copies of the node, maybe wrapping each of them in a new node of different concepts. These new nodes then have their TextDef defined to perform the desired model-to-file conversions.