basic (semantic) highlighting in TMF Xtext

In this post, I want to point to the basic concepts used for semantic highlighting in Xtext. It covers aspects similar to those described in this TMF newsgroup thread. The example is based on the grammar generated with a new Xtext project (0.7.0). Just to be safe, here is the grammar

Model : (imports+=Import)* (elements+=Type)*;
Import : 'import' importURI=STRING;
Type: SimpleType | Entity;
SimpleType: 'type' name=ID;
Entity : 'entity' name=ID ('extends' extends=[Entity])? '{' properties+=Property*'}';
Property: 'property' name=ID ':' type=[Type] (many?='[]')?; 

In a nutshell, two new classes are implemented and are made known in the UIModule. The first one introduces additional style ids and their corresponding default values, the second one does the actual work of calculating which editor regions are to be highlighted. The central call is acceptor.addPosition(startOffset, length, styleId). It informs the highlighter that the editor region from startOffset upto startOffset+length is to be highlighted in the text style connected with the id styleId. Most effort will have to be invested into (efficiently) calculating the first two parameters from the model information. Usually it will be necessary to navigate from the semantic model to the parse tree (NodeUtil is your friend) in order to identify the region connected with some model element.

The first example illustrates the pure call to the acceptor, the second one serves as a starter for actual semantic highlighting. I have tried to make the source code as informative as possible. Please leave a comment if some aspect should be explained in more detail.

Example 1
For configuring style IDs:

public class MySemanticHighlightingConfiguration implements
    ISemanticHighlightingConfiguration {

  // provide an id string for the highlighting calculator
  public static final String DUMMYHL = "dummyHighlighting";

  // configure the acceptor providing the id, the description string
  // that will appear in the preference page and the initial text style
  public void configure(IHighlightingConfigurationAcceptor acceptor) {
    acceptor.acceptDefaultHighlighting(DUMMYHL,
      "AdditionalDummyType", dummytype());
  }

  // method for calculating an actual text styles
  public TextStyle dummytype() {
    TextStyle textStyle = new TextStyle();
    textStyle.setBackgroundColor(new RGB(155, 55, 255));
    textStyle.setColor(new RGB(5, 10, 20));
    textStyle.setStyle(SWT.ITALIC);
    return textStyle;
  }
}

For determining the actual highlighting:

public class MySemanticHighlightingCalculator implements
    ISemanticHighlightingCalculator {

  // Dummy-Highlighting
  // of each block of 10 characters the last 5 are to be highlighted
  // this is not semantic but very briefly illustrates what kind of
  // information is processed
  public void provideHighlightingFor(XtextResource resource,
      IHighlightedPositionAcceptor acceptor) {
    if (resource.getContents().size() > 0) {
      // fetch the model
      Model m = (Model) resource.getContents().get(0);
      // fetch the length of the input file
      int l = NodeUtil.getNodeAdapter(m).getParserNode().getLength();

      for (int i = 10; i < l; i = i + 10) {
        // parameters: offset to begin highlighting, lenght, style id
        // the TextAttributeProvider can handle
        acceptor.addPosition(i - 5, 5,
          MySemanticHighlightingConfiguration.DUMMYHL);
      }
    }
  }
}

Making these implementations known in the (already existing) UI module:

public class DslUiModule extends myProject.AbstractDslUiModule {
  public Class bindSemanticHighlightingCalculator() {
    return MySemanticHighlightingCalculator.class;
  }

  public Class bindSemanticConfig() {
    return MySemanticHighlightingConfiguration.class;
  }
}

Example 2
In the second example, I want the highlighting to be a bit more semantic. In the configuration class, we add additional ids and text styles…

  public void configure(IHighlightingConfigurationAcceptor acceptor) {
    acceptor.acceptDefaultHighlighting(DUMMYHL,
      "AdditionalDummyType", dummytype());
    acceptor.acceptDefaultHighlighting(SLIMENTITY,
      "Entity with few attributes", slimtype());
    acceptor.acceptDefaultHighlighting(FATENTITY,
      "Entity with many attributes", fattype());
  }

  public static final String SLIMENTITY = "entityWithFewAttributes";
  public static final String FATENTITY = "entityWithManyAttributes";

  public TextStyle slimtype() {
    TextStyle textStyle = new TextStyle();
    textStyle.setBackgroundColor(new RGB(200, 200, 200));
    textStyle.setColor(new RGB(50, 100, 0));
    textStyle.setStyle(SWT.NORMAL);
    return textStyle;
  }

  public TextStyle fattype() {
    TextStyle textStyle = slimtype().copy();
    textStyle.setBackgroundColor(new RGB(128, 128, 128));
    textStyle.setStyle(SWT.BOLD);
    return textStyle;
  }

and in the calculation class, we implement a more sophisticated algorithm. We want to distinguish between entities with few properties (less than two) and those with many properties (more than one). Further we want to highlight those simple types which are not used in any entity. We simplify the task by not counting inherited properties (and by ignoring references made in other dsl files). Feel free to make the needed adaptions (… don’t forget to take care of the case were an entity extends itself via some arbitrary chain).

  public void provideHighlightingFor(XtextResource resource,
      IHighlightedPositionAcceptor acceptor) {
    // fetch the model
    Model m = (Model) resource.getContents().get(0);

    // fetch entities
    List modelelements = m.getElements();
    List classes = EcoreUtil2.typeSelect(modelelements,
        Entity.class);

    // for a naive implementation for collecting used simple types
    Collection usedsimpletypes = new HashSet();

    // highlight the entities according to the number of properties defined
    for (Entity entity : classes) {
      int numberOfOwnProperties = entity.getProperties().size();
      String highlightID = MySemanticHighlightingConfiguration.SLIMENTITY;
      if (numberOfOwnProperties > 1) {
        highlightID = MySemanticHighlightingConfiguration.FATENTITY;
      }
      //call a helper method doing the highlighting for a semantic element
      //telling it which feature of the element is to be highlighted how
      highlightFirstFeature(entity, 
          DslPackage.Literals.TYPE__NAME.getName(), 
          highlightID, acceptor);

      // collect the used simple types while going along
      usedsimpletypes.addAll(getUsedSimpleTypes(entity));
    }

    // -------------
    // highlight unused simple types

    for (SimpleType simpleType : 
        EcoreUtil2.typeSelect(modelelements, SimpleType.class)) {
      if (!usedsimpletypes.contains(simpleType)) {
        highlightFirstFeature(simpleType,
            DslPackage.Literals.TYPE__NAME.getName(),
            SemanticHighlightingConfig.DUMMYHL, acceptor);
      }
    }
  }

  private Collection getUsedSimpleTypes(Entity entity) {
    Collection result = new HashSet();
    // iterate over the entity's properties and
    // add all simple types to the result list
    for (Property property : entity.getProperties()) {
      if (property.getType() instanceof SimpleType) {
        result.add((SimpleType) property.getType());
      }
    }
    return result;
  }

  //helper method that takes care of highlighting the first feature element
  //of a semantic object using a given text style ID
  private void highlightFirstFeature(EObject semobject, String featurename,
      String highlightID, IHighlightedPositionAcceptor acceptor) {
    // fetch the parse node for the entity
    LeafNode nodetohighlight = getFirstFeatureNode(semobject, featurename);
    acceptor.addPosition(nodetohighlight.getOffset(),
        nodetohighlight.getLength(), highlightID);
  }

  // adapted from Sebastian Zarnekow's semantic highlighting implementation
  // navigate to the parse node corresponding to the semantic object and
  // fetch the leaf node that corresponds to the first feature with the given
  // name
  public LeafNode getFirstFeatureNode(EObject semantic, String feature) {
    NodeAdapter adapter = NodeUtil.getNodeAdapter(semantic);
    if (adapter != null) {
      CompositeNode node = adapter.getParserNode();
      if (node != null) {
        if (feature == null)
          return null;
        for (AbstractNode child : node.getChildren()) {
          if (child instanceof LeafNode) {
            if (feature.equals(((LeafNode) child).getFeature())) {
              return (LeafNode) child;
            }
          }
        }
      }
    }
    return null;
  }


In the screenshots you can see that the additional id has made its way to the preference page of the DSL where its attributes can be modified. Note that in the second one, the background color for the AdditionalDummyType has been changed.

4 Responses to “basic (semantic) highlighting in TMF Xtext”

  1. Heiko Behrens Says:

    Thank you for this valuable blog post, Leipzig (or should I say Alex?). I will refer to this quite often in the future.

  2. Karsten Thoms Says:

    Hi Alex! Many thanks for your article. This was the missing peace in the Xtext manual.

    With Xtext 1.0.1 the described binding in the UI module does not work as you described for Xtext 0.7.2. I got it working this way:

    public class DslUiModule extends myProject.AbstractDslUiModule {
    @Override
    public void configure(Binder binder) {
    super.configure(binder);
    binder.bind(ISemanticHighlightingCalculator.class).to(MyDslSemanticHighlightingCalculator.class);
    }
    }

    Best wishes,
    ~Karsten

  3. Christian Ernst Says:

    I use the semantic highlighting since Xtext 0.7 and got all relevant information from this post.

    Now I updated to Xtext 2.0. The NodeAdapter part doesnt work anymore. It would be great to update the description for Xtext 2.0.

    Thanks, Christian.

  4. Alexander Nittka Says:

    Hi,

    when I wrote that post, documentation on semantic highlighting was something like non-existent. The current documentation contains code snippets showing the entry point. NodeModelUtils contains useful helpers for navigating between semantic and parse model (which indeed has changed, but you should be able to adapt that). Otherwise, the basic ideas remain the same. Actually, there are more pressing topics than highlighting to blog about… if one had time.

    Alex

Leave a Reply