Archive for the ‘Xtext’ Category

Xtext cross references and scoping – an overview (Part 3)

Sunday, August 7th, 2011


Default Imports


A question regularly asked is how to implement built in types for your DSL. Enums or particular keywords for those types, i.e. hard coding them in the grammar, is possible, but the library approach is considered good (if not best) practice. The basic idea is shipping a model file containing the built in types along with the rest of the language infrastructure, as usually it is possible to model them just like a user defined type.

Of course, it would be nice if those types were visible without the user having to explicitly import that model file or the corresponding namespace.

The Approach

The solution proposed here assumes that the importing project has a “dependency” to the jar containing the default model (for want of a better term). That is, the default model is on the classpath of the importing model. One possibility is that the default model is contained in a “library” plugin project and the importing plugin has a dependency to the former.

URI Imports

If you are using the import uri mechanism, i.e. you are explicitly importing a particular model file, you can hook into the ImportUriGlobalScopeProvider. You simply add the default imports to the set of imports picked up from the model itself

public class MyImportUriGlobalScopeProvider extends
    ImportUriGlobalScopeProvider {

  protected LinkedHashSet<URI> getImportedUris(EObject context) {
    LinkedHashSet<URI> temp = super.getImportedUris(context);
    return temp;

You could make this change known analogous to the way the ImportUriGlobalScopeProvider is bound in the AbstractXRuntimeModule

public void configureIScopeProviderDelegate( binder) {

Namespace Imports

When using namespace imports (so that an element can be referenced via its simple name rather than the fully qualified one) the hook is to be found in the ImportedNamespaceAwareLocalScopeProvider. In Xtext 1.x something like the following snippet does the trick.

public class MyImportedNamespaceAwareLocalScopeProvider extends
    ImportedNamespaceAwareLocalScopeProvider {

  protected Set<ImportNormalizer> getImportNormalizer(EObject context) {
    Set<ImportNormalizer> temp = super.getImportNormalizer(context);
    temp.add(new ImportNormalizer(
      new QualifiedName("builtin.types.namespace.*")
    return temp;

In Xtext 2 there is a dedicated method for implicit imports (also the QualifiedName-API has changed).

public class MyImportedNamespaceAwareLocalScopeProvider extends
    ImportedNamespaceAwareLocalScopeProvider {

  protected List<ImportNormalizer> getImplicitImports(boolean ignoreCase) {
    List<ImportNormalizer> temp=new ArrayList();
    temp.add(new ImportNormalizer(
      true, ignoreCase));
    return temp;

In both cases the change is made know in the runtime module as above, now binding MyImportedNamespaceAwareLocalScopeProvider.

It should be clear that you can also use this approach for dealing with “split packages”, i.e. spreading the same namespace over several files. Usually, you would have to import “your own” namespace in order to refer to elements from another file using their simple names. Note that the getImplicitImports method in the Xtext 2 example is not the right hook now, as there you don’t have access to the context element (for obtaining its namespace).

Xtext cross references and scoping – an overview (Part 2)

Tuesday, August 2nd, 2011


The Out-of-the-box/Usability Trade-off

Scopes are used by a number of components (most notably linking and content assist). This often tempts developers to implement the scope provider such that only objects that are indeed allowed to be linked are put in the scope. After all, then validation and code completion need not be adapted at all (hooray! everything works out of the box), as only valid links can be established and only valid links are proposed. However, scopes are not about validity, they are about visibility.

An analogous problem is grammar design itself. There may be a restriction that entity names have to start with an upper case letter. You can enforce that within the grammar, e.g. by defining a corresponding terminal rule. However, this will make the grammar error prone and the error messages for the user will not be very helpful. There might be a restriction that the value of a certain feature may only be between 5 and 49. Go ahead and put that into the grammar (it’s possible after all), you will have plenty of fun (in particular with other integer value features). Obviously, there are often several ways to achieve something and choices must be made.

The decision what to put into the scope should be made very carefully. There is no simple rule to be followed. Here are a few guide lines, I find sensible:

Single user language

If you develop a language/IDE for yourself only, take the path simplest for you. You know the language, you know the intended meaning, you (should) know why a cross reference could not be resolved. There is nobody who can argue that validity and visibility are not the same.

Multi user language

If you develop a language/IDE for use by somebody else, think thoroughly about usability. Usually, the user should be provided with a message specifically why a certain cross reference is invalid (you cannot extend entity X because it is final, the type of feature Y must implement a method Z and the type you try to use does not do so).
Of course this is a lot of work. You may still have to adapt scoping (reduce the number of visible objects, but not too much), adapt validation (check semantic constraints that must be satisfied), adapt code completion (suggest only valid objects). However, you can provide more meaningful error messages and specific quick fixes (great! still more work). And don’t underestimate how stupid your editor may look.

variable x;
variable y=x+3; /*could not resolve reference to x 
developer's insider knowledge: only initialised 
variables may be used on the right hand side of an assignment)
user reaction: st*&%/$/"§)..., look at the line above*/

“Follow up references”

A use case, where a drastic narrowing of the scope is almost always OK, is a reference that “follows” another one. Take feature calls as an example.

If you already established a link to myType and now only properties of that type are valid, it makes no sense, having all other possible properties on the scope. On seeing a “could not resolve reference” error, the user can navigate to myType and see that there is indeed no such property.

Xtext Cross References and Scoping – an Overview (Part 1)

Monday, August 1st, 2011


The Grammar Aspect

As a side note: There are not many naming conventions the grammar language actually enforces. However, almost all conventions that have an actual impact on the behaviour have to do with cross referencing (I am referring to features with the name “name”, “importURI” and “importedNamespace”).


Xtext distinguishes containment references (the referenced object is contained/declared/defined in the referencing object; in the model file reference and definition basically coincide) from cross references (the referenced object is defined somewhere else, the referencing object has a link to the referenced object). I assume the reader is aware that this series talks only about the second type.

In the grammar of your DSL you specify that there should be a cross reference to an object, you specify the type of the referenced object and you specify the syntax of the name used for establishing the reference. Note that within the grammar you can only restrict the type of the referenced object. Further restrictions on the semantic validity of a cross reference have to be enforced elsewhere. Consider the following well-known grammar snippet

Entity: ... 'extends' extends=[Entity] ...;

The square brackets indicate that there will be a cross reference. The word “Entity” within the brackets indicates that the referenced object is of type Entity. It is very important to understand, that this Entity does not refer to the Grammar rule with name Entity. Further this rule states that the name that is to be establish the reference has the ID-syntax. This is because [Type] is a shorthand notation for [Type|ID] which means “cross reference to an object of type Type via a name that has the syntax of an ID”. The pipe symbol here is not indicating an alternative! It simply separates the referenced type from the string-returning-grammar-rule-describing-the-allowed-syntax. A more verbose version of the above snippet would be

EntityRule returns Entity: ... 'extends' extends=[Entity|ID]...;

Given you inherit the default ID-terminal rule, in your model file you can now write “… extends SuperType …” but not “… extends the.package.of.SuperType…”, i.e. you can use only “simple” names to establish the reference but not qualified ones. For that you would have to change the grammar to something like

Entity: ... 'extends' extends=[Entity|Fqn]...;
Fqn: ID ('.' ID)*;

For a reference you can only define one target type. That is, the following is not valid

Rule: ... ref=[(Type1|Type2) | ID]...

You would have to introduce a common super type of the two

SuperType: Type1|Type2;
Rule: ...ref=[SuperType|ID]...

Note that if you have no influence over the meta model (i.e. you cannot simply introduce a common super type for Type1 and Type2), you usually take the most specific existing common supertype (usually at least EObject is a candidate) and adapt the framework at the suitable hooks (i.e. validation, scoping, content assist).

A common language design problem is the following

Rule1: ... ( refToType1=[Type1] | refToType2=[Type2] )... //or
Rule2: ... (ref=[Type] | simpleValue=ID)...

The problem is that the parser has to decide which alternative is to be used. This is not possible as there is no syntactic indication, in each case an ID is read. Also the developer cannot argue that “in case a Type1 is found link it, in case Type2 is found link that” should be the way to go. The parser is not responsible for linking, it only installs proxies to be resolved later. The usual pattern for resolving these problems would be the introduction of common super types or the introduction of keywords into the language.


As stated above, a name provider is responsible for giving a name to an object so that it may be referenced at all. In order to provide nice out-of-the-box behaviour, the following convention is followed. If an object has a feature with a string-type, whose name is “name”, the default name provider implementation will use the value of that feature for calculating the name of an object. This is why cross referencing works (immediately) for

Entity: 'entity' name=ID ...; //or
Entity: 'entity' name=STRING ...;

but not for

Entity: 'entity' id=ID ...;

If you look at the workflow for generating the language infrastructure, you will notice two naming fragments (SimpleNameFragment, QualifiedNameFragment) that may be used. When using the first one, only the name of the object itself is considered. When using the second, the actual name of an object is composed using all the names found in the containment hierarchy. Given the grammar snippet

Entity: 'entity' name=ID '{' attributes+=Attribute* '}';
Attribute: 'attribute' name=ID;

and the model file

entity X {attribute Att1 attribute Att2}
entity Y {attribute Att1 attribute Att2}

both fragments would yield entities with names X and Y. The simple name fragment would yield two attributes with name Att1 and two with name Att2. Note that here you might run into problems when trying to reference a specific Att1. The qualified name fragment would calculate the following names for the attributes: X.Att1, X.Att2, Y.Att1, Y.Att2


Import constructs are very common in languages. Xtext supports two import types out of the box, importing a particular file via a URI pointing to that file and importing a namespace (analogous to Java imports). Note that these imports only affect the visibility of objects. The import does not make the imported stuff part of the importing model. You choose the import semantic to be used by enabling the corresponding generator fragment (often the uri import goes along with the simple names and the name space import with qualified names). The grammar convention going along with the import semantics can also be found in the documentation.
A URI import of a particular file usually looks as follows:

Import: 'import' importURI=STRING;

Again the name of the feature is important. If any feature with name “importURI” is found the value (in case it is a string-value) is interpreted as the URI of a model file whose objects should be made visible. Usually one uses STRING (rather than ID or some other datatype rule).
A namespace import normally looks as follows:

Import: 'import' importedNamespace=FqnWithWildCard;
FqnWithWildCard: Fqn('.*')?;

Now if you have a model like

import my.package.*
import my.other.package.*
entity X extends Y...

the cross reference resolution would not only look for elements with name Y, but also my.package.Y and my.other.package.Y. So it normalises the name to be used for cross referencing (Y) against all possible imports as well.

Xtext Cross References and Scoping – an Overview (Part 0)

Monday, August 1st, 2011


The ability to reference objects that are defined elsewhere, along with goodies like navigation and code completion, is one of the central features of Xtext. When working on your DSL, it is likely that next to validation this is one of the first places where you want/have to do customisations and the number of questions on that topic in the eclipse forum suggests that it is not trivial to do that. One important reason may be the following. The Xtext philosophy is that as much as possible of the standard behaviour should be provided out of the box. However for cross referencing many components are involved and internally a lot of “magic” happens. Many developers are probably not aware of that fact and think that adapting the scope provider (without really having understood its purpose) is all that ever needs to be done.

It has been said that the documentation on that topic (scoping in particular) is good… once you have understood the topic. And, in my opinion, there is truth in that statement. If you have understood the underlying concepts, the documentation is technically very helpful and contains almost everything you need. The problem is that it lacks an easy to understand overview that allows a beginner to get an idea how the different concepts and framework components actually interact. A series of posts is an attempt to provide such an overview. It will not be published in a “sensible” order. It will not be complete. It will not be (very) specific to a particular Xtext version and will not deal with referencing Java types or Xbase. It will not answer questions on specific implementation details. Its purpose is to give developers a rough overview on the topic, giving some background for understanding the documentation, hints for identifying entry points for language specific customisations and some best practices.

Cross References in a(n enormously simplified!) Nutshell

(This will definitely not be the final version of this paragraph)
In Xtext cross references are established via names. This has a number of consequences, e.g an object that is to be referred to must have a name at all, names are not unique and they do not carry any information on where to look for the object. This means that, if cross referencing is to work out of the box, there must be framework components that deal with all of these aspects. And as Xtext tries to make customisation as simple as possible, there are many components that interact.

The name provider is responsible for attaching a name to an object (note that as soon as an object has a name it is by default globally visible). The scope determines which objects are actually visible from a particular location (as a first approximation think of the scope as a lookup table mapping names to objects). Note that “visible” must not be confused with semantically valid, i.e. an object may well be visible with respect to a reference but not be a valid reference. The scope provider (with the heavy use of many helpers, e.g. the index of exported objects) is responsible for creating the scope for each reference. The linker is responsible for reading the name in the model file and choosing from the scope the object to be cross referenced. If no such object can be found, you will get the “could not resolve cross reference”-Error.
The content assist engine will ask the scope for all potential objects and will suggest the names that would be syntactically valid at the current position.

Posts in this series

Feedback welcome

If you find errors, miss topics, think that the explanations should be improved etc., feel free to leave a comment. If time permits, I will try to incorporate them in this overview.