HowTo -- Integrating into the MPS Make Framework

Build Facets

Overview

Like basically any build or make system, the MPS make executes a sequence of steps, or targets, to build an artifact. A global ordering of the necessary make steps is derived from relative priorities specified for each build targets (target A has to run before B, and B has to run before C, so the global order is A, B, C).

A complete build process may address several concerns, for example generating models into text, compiling these models, deploying them to the server, and/or generating .png files from graphviz source files. In MPS, such different build aspects are implemented with build facets. A facet is a collection of targets that address a common concern.

The targets within a facet can exchange configuration parameters. For example, a target that is declared to run early in the overall make process may collect configuration parameters and pass them to the second facet, which then uses the parameters. The mechanism to achieve this intra-facet parameter exchange is called properties. In addition, targets can use queries to obtain information from the user during the make process.

The overall make process is organized along the pipes and filters pattern. The targets act as filters, working on a stream of data being delivered to them. The data flowing among targets is called resources. There are different kinds of resources, all represented as different Java interfaces and tuples:

MResource contains MPS models created by users, those that are contained in the project's solutions and languages
GResource represents the results of the generation process, which includes the output models, that is the final state of the models after generation has completed. These are transient models, which may be inspected by using the Save Transient Models build option
TResource represents the result of text-gen
CResource represents a collection of Java classes
DResource represents a collection of delta changes to models (IDelta)
TextGenOutcomeResource represents the text files generated by textgen

Build targets specify an interface. According to the pipes and filters pattern, the interface describes the kind of data that flows into and out of a make
target. It is specified in terms of the resouce types mentioned above, as well as in terms of the kind of processing the target applies to these resources. The following four processing policies are defined:

transform is the default. This policy consumes instances of the input resource type and produces instances of the output resource type (e.g. it may
consume MResources and produce TResources.)
consume consumes the declared input, but produces no output. * produce consumes nothing, but produces output
pass through does not access any resources, neither produce nor consume.

Note that the make process is more coarse grained than model generation. In other words, there is one facet that runs all the model generators. If one needs
to "interject" additional targets into the MPS generation process (as opposed to doing something before or after model generation), this requires refactoring
the generate facets. This is beyond the scope of this discussion.

Building an Example Facet

As part of the mbeddr.com project to build a C base language for MPS, the actual C compiler has to be integrated into the MPS build process. More
specifically, programs written in the C base language contain a way to generate a Makefile. This Makefile has to be executed once it and all the
corresponding .c and .h files have been generated, i.e. at the very end of the MPS make process.

To do this, we built a make facet with two targets. The first one inspects input models and collects the absolute paths of the directories that may contain a
Makefile after textgen. The second target then checks if there is actually a file called Makefile in this directory and then runs make there. The two
targets exchange the directories via properties, as discussed in the overview above.

The first target: Collecting Directories

Facets live in the plugins aspect of a language definition. Make sure you include the {{jetbrains.mps.make.facets} language into the plugins model,
so you can create instances of FacetDeclaration. A facet is executed as part of the make process of a model if that model uses the language that
declares the facet.

The facet is called runMake. It depends on TextGen and Generate. The dependencies to those two facets has to be specified so we can then declare our targets' priorities relative to targets in those facets.

facet runMake extends <none>
  Required: TextGen, Generate

The first target is called collectPaths. It is specified as {{transform IMResource -> IMResource} in order to get in touch with the input models. The
facet specifies, as priorities, after configure and before generate. The latter is obvious, since we want to get at the models before they are
generated into text. The former priority essentially says that we want this target to run after the make process has been initialized (in other words: if
you want to do something "at the beginning", use these two priorities.)

target collectPathes overrides <none> {
  resources policy: transform IMResource -> IMResource
  Dependencies:
    after configure
    before generate

We then declare a property pathes which we use to store information about the modules that contain make files, and the paths to the directories in which
the generated code will reside.

Properies:
     list<[string, string]> pathes;

Let's now look at the implementation code of the target. Here is the basic structure. We first initialize the pathes list. We then iterate of the
input (which is a collection of resources) and do something with each input (explained below). We then use the output statement to output the input
data, i.e. we just pass through whatever came into out target. We use the success statement to finish this target successfully (using success
at the end is optional, since this is the default). If something goes wrong, the failure statement can be used to terminate the target unsuccessfully.

(input)->void {
  pathes = new linkedlist<[string, string]>;
  input.forEach({~inpt =>
    (module, models) res = ((module, models)) inpt;
    // do stuff. See below.
  });
  output input;
  success;
}

The actual processing is straight forward Java programming against MPS data structures:

res.models.forEach({~model =>
  string path = res.module.getGeneratorOutputPath() + "/" +
                model.getLongName().replaceAll("\\.", "/");
  string locationInfo = res.module.getModuleFqName() + "/" +
                        model.getLongName();
  pathes.add([path, locationInfo]);
});

We use the getGeneratorOutputPath method to get the path to which the particular module generates its code (this can be configured by the user in the
model properties). We then get the model's dotted name and replace the dots to slashes, since this is where the generated files of a model in that module will
end up (inspect any example MPS project to see this). We then store the module's name and the model's name, separated by a slash, as a way of improving the
logging messages in our second target (via the variable locationInfo}). We add the two strings to the {{pathes collection. This pathes property
is queried by the second target in the facet.

The second Target: Running Make

This one uses the pass through policy since it does not have to deal with resources. All the input it needs it can get from the properties of the collectPaths target discussed above. This second target runs after collectPaths}, {{after textGen and before reconcile. It is obvious that is has to run after collectPaths}, since it uses the property data populated by it. It has to run after {{textGen}, otherwise the make files aren't there yet. And it has to run before {{reconcile}, because basically everything has to run before {{reconcile

target callMake overrides <none>
  resources policy: pass through
  Dependencies:
    after collectPathes
    after textGen
    before reconcile

Let us now look at the implementation code. We start by grabbing all those entries from the collectPathes.pathes property that actually contain a
Makefile. If none is found, we return with success.

sequence<[string, string]> modelDirectoriesWithMakefile =
     collectPathes.pathes.
     where({~it => new File(it[0] + "/Makefile").exists();
  });
  if (modelDirectoriesWithMakefile.isEmpty) { success; }

We then use the progress indicator language to set up the progress bar with as many work units as we have directories with make files in them.

begin work "run make" covering ALL units of total work left,
     expecting modelDirectoriesWithMakefile.size units;

We then iterate over all the entries in the {{modelDirectoriesWithMakefile} collection. In the loop we advance the progress indicator and then use Java
standard APIs to run the make file.

foreach dirInfoTuple in modelDirectoriesWithMakefile {
    try {
      advance 1 units of "run make"
         with comment "running make for " + dirInfoTuple[1];
      Process process = Runtime.getRuntime().
         exec("make", new string[0], new File(dirInfoTuple[0]));
      if (process.waitFor() > 0) {
        error "make failed with exit code " + process.exitValue() + " for " +
              dirInfoTuple[1];
      } else {
        info "make finished successfully for " + dirInfoTuple[1];
      }
    } catch (Exception ex) {
      error ex.getMessage(), ex;
    }
  }

To wrap up the target, we use the finish statement to clean up the progress bar.

finish "run make";

Last modified: 18 June 2020