Generator User Guide Demo6

Generator User Guide Demo 6

This demo is probably the most exciting one of all demos, because here we are going to create a 'real' language in the sense that this language will actually define a couple of higher-level concepts.
We will also see how easily this DSL can be integrated with existing languages - 'jetbrains.mps.sampleXML' and 'generator_demo.demoLang5' (created earlier in the Demo 5).

The core idea

We'll allow for a more convenient syntax for entering XML Elements in the test models. Developers will use new syntax for button and label, which will be more compact and intuitive. At the same time, we'll be able to reuse the generator defined in the demoLang5 language that we built in Demo 5. The new demoLang6 language will translate the new concepts into XML Elements, which then can be accepted by the demoLang5 generator.

New Language

• create new language 'generator_demo.demoLang6'

• in language properties dialog add extended languages : 'jetbrains.mps.sampleXML' and 'jetbrains.mps.baseLanguage'

• create a new generator for this language, if it does not exist (see Demo 1 for details)

• in demoLang6 structure model create new concept named 'Button' extending the concept 'ElementPart' (from 'jetbrains.mps.sampleXML')

• in 'Button' concept declare a property named 'text' and set an alias (we need an alias to make auto-completion menu look nice):

• create an editor for the 'Button' concept:

• create a similar concept named 'Label'

• re-generate the language (Ctrl-F9)

New Test Model

Now let's try out our new DSL:

• go to the 'test_models' solution

• clone the 'test5' model into 'test6' (this time DO NOT replace engage on generation language 
  generator_demo.demoLang5 -> generator_demo.demoLang6, since we still need the demoLang5 generator to run with test6)

• add the 'generator_demo.demoLang6' language to the used languages section (new!), so we can use the demoLang6 language for editing our model.

• in the 'test6' model open the 'Panel' document and replace the two 'label' elements with our new button and label concepts:

With the new DSL our 'code' became clearer, shorter and less error prone. For instance, users now no longer can add elements inside labels or buttons.

Generator

We could generate methods and method calls out of button and label mimicking what has been done earlier for XML Elements. But this isn't a good idea, because this way we would introduce indirect dependency between the demoLang6 generator and the demoLang5 generator (the demoLang6 generator would have to know about the implementation details of the demoLang5 generator).

Luckily, we have a far better option. As we are on a higher level of abstraction now (comparing to XML), we can simply reduce our semantically rich concepts into lower-level concepts (XML Elements). We don't care anymore about who and how will transform those lower-level (XML) concepts further into even lower-level concepts.

Reduction Rules

We'll employ reduction rules to convert our new concepts into XML Elements. from the sampleXML language.

• open the (empty) mapping configuration 'main' in the demoLang6 generator

• add a new reduction rule applicable to the Button concept

• choose an in-line template as the rule consequence (Control + Space):

The template is going to be pretty straightforward and it will not require any surrounding 'context'. That's why it is not necessary to create a full-blown external template as we did in other demos.

Since we want to use XML Elements from the 'jetbrains.mps.sampleXML' language in generator templates, we first have to declare that language among Used Languages by the main@generator model:

• back in the main mapping configuration choose Element as the template's content node:

• create the 'button' XML element with a 'text' attribute

• attach a property macro to the attribute value

• enter the code in the macro's value function so that the value of the text property gets propagated into the XML button:

That's it. Our high-level button is being reduced into a 'button' XML element with a 'text' attribute. The XML element will presumably be further transformed down to Java by somebody else.

• add a similar reduction rule applicable to the Label concept

First Test (Failure)

Re-build the generator and generate the 'test6' model.

There will be a bunch of error and warning messages in the MPS messages view.
Click on the 'was input node' error message. MPS will show you the node to, which the generator was trying to apply the rule that failed:

This is a reduction rule in the demoLang5 generator (we are generating the 'test6' model with two generators: demoLang5 and demoLang6, remember?) and it has failed to find a static method declaration through the mapping label method.

The method label is being assigned the generated static methods inside the insert_Button and insert_Label templates, so you may wonder what went wrong. Why couldn't we find the methods created inside the DempApp class and stored in the mapping label?

To answer that question let's take a closer look at the generation process as a whole.

As the first step, the generator creates a transient model, which will serve as an output model for the current generation phase. This model name is 'test6@0'.

Then generator applies our conditional root rule and creates the 'DemoApp' class in output model.

Then, at some point, the insert_Panel template is invoked and its LOOP-macro is executed (we can see the panel factory method containing the panel initialization code that we are familiar with from the insert_Panel template). The LOOP-macro creates a statement 'component.add()' for each child of 'panel' element and reduces the child to generate an actual argument to the 'component.add()' method call.

Our 'panel' element contains three children - a button, a label and a text, which has been turned into a label by a pre-processing generation script. Both the button and the first label are high-level abstractions introduced by demoLang6 and their reduction yields corresponding XML Elements. (We can see it clearly in the first screenshot, where the XML element representing a button is passed as an actual argument to the call to the 'component.add()' method. This is reported as a warning by the generator earlier up in the trace output in yellow color).

This ends the generation of 'test6@1_1' and the generator now creates a new transient model, named 'test6@1_2'. This model becomes a new output model and the former output model becomes a new input model.

The generator starts to generate a new model 'test6@2_2' pretending that there was nothing before. The generator has no memory about previous activities with only one exception - conditional root rules are never applied twice. Thus, this time, the output 'DemoApp' class is created by copying of 'DemoApp' class from the input model, not by applying a conditional root rule.

At this stage, the generator attempts to reduce further the XML Elements that represent the button and the label. The only rule available is the reduce_Element reduction rule from demoLang5. It will attempt to replace the XML Elements with references to static factory method declarations, however, since none have been created, none can be found in the method mapping label. The method definitions were not created simply because the XML Elements were still represented as the Button and Label demoLang6 concepts at the time when the LOOP macro inside DempApp was run.

So this is a timing issue between the demoLang5 and demoLang6 generators. The latter should be made run first.

Now that we understand what is happening, the next question is how to fix it.

We are relying on the demoLang5 generator and we know that the demoLang5 generator works well providing that the input model is a 'valid XML' (we have tested it in Demo 5).
In Demo 6 our input model is not exactly a 'valid XML'. Moreover, we witnessed that the transient model 'test6@1_1' is not a valid model at all - a method call can't accept an XML element as argument (you can even find a warning message "child 'Expression' is expected for role 'actualArgument' but was 'Element'" in the MPS message view).

Now we could probably return to the demoLang5 generator and make some 'improvements' to allow it to handle 'invalid' input models.
Fortunately, there is a better option - we can divide the generation process in two steps so that the original input model gets transformed into a 'valid XML' model first, and then this 'valid XML' model will be transformed to Java by demoLang5 generator in a second step.

Dividing Generation Process into Steps

We will certainly get 'valid XML' model from the 'test6' input model if we reduce button and label (in the 'Panel' document) into XML Elements. This transformation must happen before demoLang5 starts transforming XML Elements into Java. In other words, the reduction rules specified in the demoLang6 generator must be applied before any rules in the demoLang5 generator.

Let's specify this constraint:

• go to the generator in generator_demo.demoLang6 and open the generator properties dialog

• add an extension dependency on the demoLang5 generator
  

  

• in Generators priorities tab add a desired priority rule

This will ensure the demoLang6 generator runs before the demoLang5 generator.

Second Test

Re-build the language and preview generated text for the 'test6' model. You should now run into no issues during generation and the generated code will correctly include the new button and label.

Saving Transient Models

Our demoLang5 and demoLang6 generators are working together well, because we have divided the generation process into two distinct steps.
In the first step the demoLang6 generator reduces our high-level concepts (button and label) into corresponding XML elements and produces a 'valid XML' model as output.
In the second step the demoLang5 generator generates Java from that 'valid XML' model.

Thus there should be at least one transient model between step 1 and step 2.
Let's take a look at what models have actually been created in the course of generation:

• in Project Settings enable the Save transient models on generation option:

• generate the 'test6' model

• in the project tree find and expand the node named 'transient models' (this node is at the bottom)
  There are (surprisingly) five transient models.
  We can browse the transient models and we will find sometimes subtle and sometimes dramatic differences between them. This will give us a clue about what kind of transformation has taken place on each step.

For instance, open the 'Panel' node in model 'test@1_0' (this model has been generated in very first step).

We can see that high-level button and label concepts have been replaced with their XML counterparts, but the text 'Hello everybody!' is still there.

In the next model 'test6@2_0' the text 'Hello everybody!' has been replaced with a 'label' element (as a result of running the pre-processing script 'fix_text' in the demoLang5 generator).

Model 'test6@2_1' contains the generated DemoApp class.

And model 'test6@2_3' contains the same class, but the string "MPS!" is replaced with "JetBrains MPS!" (as a result of the post-processing script 'refine_text' in the demoLang5 generator).

Root Nodes Copying and Reduction Rules

As we can see, the model 'test6@1_0' is identical to the original input model 'test6', except for high-level button and label concepts that have been reduced to XML elements.
This is what we wanted, but it is probably still not clear, how roots in 'test6@1_0' have been created (we don't have any 'root rules' that would create a Document) and why our reduction rules have been applied.
From our previous experience with reduction rules we remember that we had to create a COPY_SRC _macro for reduction to take place. But in the _demoLang6 generator we don't have any _COPY_SRC _macros, do we?

This way the input Documents (Button, Label and Panel) from 'test6' have been copied to model 'test6@1_0' and reduction has been applied to the button and label concepts that were inside the 'Panel' Document.

Returning to the _COPY_SRC _macro - it doesn't 'invoke' a reduction (as it might seem). Instead it merely applies the same copying procedure to the mapped node and, if any reduction rules happen to be applicable, then the transformation occurs.

Using Generation Tracer Tool

We already know how to save and browse transient models. Actually, with the option to Save transient models on generation activated, MPS not only saves transient models, but also collects a great deal of data regarding the process of transformation.
This data can be viewed using the Generation Tracer Tool.

For instance, open the 'Panel' Document in the transient model 'test6@1_0', select 'label' and choose Show Generation Traceback in the popup menu.

The Generation Tracer View will be opened.

In the root of this tree there is the 'label' node, for which we have requested the traceback info (the blue arrow denotes an output node) and the rest of the tree shows the sequence of events, in reversed order, that led to this output.

Looking at this tree and clicking on its nodes (in order to open them in the editor) we can reproduce the process of transformation to a great level of detail.

For instance, we can see that the output 'label' is created by the template in the reduce Label rule.

... and the reduce Label rule has been applied to the input node 'Label' while the 'Panel' Document (the root input node) was being copied.