Understanding Eclipse's Plugin Build

This article provides a concise definition of the term Eclipse Plugin for the purpose at hand:

What Is This Plugin Thing You Are Talking About?

When we say 'Plugin' in this article, we mean an OSGi bundle that is running on an Eclipse runtime that has the Equinox Extension Point Registry installed.

If the above is clear to you, skip to the next paragraph. Otherwise, suffice to say that OSGi is a Java-based framework standard that drives a large number of distributed systems. OSGi assembles systems from parts called bundles. There are multiple implementations. The Eclipse built-in one is called Equinox.

Eclipse historically has a concept of Extension Points and Extensions. Eclipse only started using OSGi as its driver later in its development. OSGi does not know the Extension Point/Extension concepts, so some sort of adapter was needed for compatibility. This adapter is added to the OSGi runtime using a mechanism called "Extension Bundles". The OSGi standard says: "Extension bundles can deliver optional parts of the Framework implementation or provide functionality that must reside on the boot class path."

So, to recap, a Plugin is an ordinary OSGi Bundle with one difference: It contains at least one of the following: 

• An Extension-Point Definition in a file schema/*.exsd
• Usage in a file plugin.xml

If the plugin uses the Extension Points that the Eclipse platform's Plugins expose to help manage for example widgets, editors or help files, the plugin you write will contribute functionality and plug into the platform.

Who Is Building This for Me?

Underlying the build of Eclipse plugins is an automation originally provided by the Eclipse Plugin Development Environment (PDE) known as PDE/Build. PDE/Build is an Ant-driven facility, which uses small descriptor files to assemble the Java archives that make up the plugins. The build for each plugin is driven by a file called build.properties.

The following section describes how the plugin build is executed, based on the information in the build.properties file.

Note that if PDE/Build is quite antiquated. Like many things Eclipse, it mainly has the distinction of being first. It is also an onboard tool that is included in every Eclipse edition that can build plugins.

So to ensure clarity, in this article, we explain PDE/Builds behaviour. In practice, PDE/Build has often been replaced by the Tycho/Maven or bnd build systems. Both are great but are reaching into other areas.

You can liken the relationship of PDE/Build to those other Build systems to how the onboard crank and factory-supplied tools in your car boot compare to a lift ramp and tool set up in an auto repair shop. The crank and tools are limited, and will not do very heavy lifting, but for your small job and for learning they will be enough. They are also always there. So if you just have or just want Eclipse, and nothing else, PDE/Build is the dog food to feed yourself. (No slurs on dogs and their tastes intended here.)

Build Structure

The class diagram explains how the artefacts that are part of the build specification for PDE/Build connect to each other. Let us start reading the diagram from the green marker which represents the Build. Our aim is to assemble a binary plugin, which is a Java archive.

Under all the covers, a plugin is an OSGi bundle. An OSGi bundle requires at a minimum an OSGi manifest (META-INF/MANIFEST.MF) and may contain a number of archives to be added to the bundle's classpath. Apart from that, an OSGi bundle can contain any number of resources. PDE/Build creates a plugin that conforms to these requirements.

In its manifest, an OSGi bundle, and hence an Eclipse plugin, describes what other bundles and packages it would require to function. This is usually the same set that is required to compile them, as most dependencies will be static. The lines (headers) in the manifest that describe the dependencies are:

• Import-Package: A comma-separated list of Java packages this bundle requires. Each package can be qualified with a version range.
• Require-Bundle: A comma-separated list of bundles that this bundle requires. Each bundle can be qualified with a version range.

As the initial step (step 0.) PDE/Build reads the manifest of the bundle to be compiled in order to find the bundles it needs to compile against. It draws those bundles from a collection in the background known as the Target Platform in Eclipse parlance.

PDE/Build then resolves the relevant part of the Target Platform into a Target Import (step 1.) so that the requirements listed in the manifest are satisfied. This list of plugins is the first component of the classpath, known as the Plugin-Classpath.

However, the assumption that the runtime dependencies are identical to the compile-time dependencies is not always completely correct and additional components may need to be added for classes to compile. These extra classpath dependencies are listed in the build.properties file. The details of that will be shown below.

The classpath that a bundle uses to operate is composed of Java resources. These can reside straight in the root of the plugin, or be wrapped up in any number of Java Archives (JARs). The build properties file contains a list of them. PDE/Build compiles each library in turn. For each library, detailed instructions are listed in the build.properties file. The syntactical details will be shown below. 

Compilation compiles all Java source files and copies all non-Java files from all source folders of the library. However, there may be some Java source files or resources that should not be copied and deployed. This means we have a two-level selection. First, all files have to be collected from all source folders, and of these, some have to be excluded. The rest are compiled and copied as expected. The result is archived with a manifest. If one is provided, the provided one will replace the generated one.

The next step is to create the actual bundle. Here a number of resources are packed into the binary archive. A set of include patterns creates the set of files considered for packaging, while a set of exclude patterns removes undesirable files from this set. The result forms the archive. It is important to note that the manifest is not created or maintained automatically. In fact, the idea to automate the generation of the manifest from source is behind the excellent OSGi build tool bnd.

Formalized Build Algorithm

The following instructions describe the build functionality in pseudo-code. The UML sequence diagram below describes the same process in a graphical format.

1. Read the MANIFEST.MF file to identify which dependencies should be provided from the current development target. 
2. Use the OSGi resolver to select the packages to expose. Each package is added to the classpath as a Target Import.
3. Compile each library in order and assembles the resulting resources and class files into an archive. This process uses a classpath based on:
   • The packages in the target that OSGi has resolved.
   • Libraries that it has already compiled.
   • Libraries that are present as resources in the source code.
   • Libraries stored in other plugins in the target platform.
4. Select all source files from each source directory mentioned.
5. Exclude all source files matched by any of the Ant patterns supplied.
6. Assemble the plugin using:
   • The class files of the root library (the one with the special name “.”)
   • The archive files of the other internal libraries.
   • The plugin resources listed.

Sequence Diagram

The UML sequence diagram below shows how PDE/Build interacts with the build.

Build File Syntax and Structure

The descriptor file that controls the build of plugins is build.properties, the Feature and Plug-in Build Configuration Properties file. Below are the names of the properties for the functionalities described above. For consistency, the descriptions are largely taken from the Eclipse help website.

Note that the property keys ending with "includes" or "excludes" have values that are Ant "patterns".  For example, *.jar indicates all jar files in the top-level directory. The simple asterisk patterns do not recurse into subdirectories by default.  For all Java files in all subdirectories for example, the pattern is **/*.java.   The pattern ** matches any number of directory levels.  To describe whole sub-trees, use a directory name followed by a slash, like this: xyz/.

Declaration of the Libraries and Build Order:

• jars.compile.order - defines the order in which jars should be compiled (in case there are multiple libraries).

Definition of Each Library

• source.<library> - source folders that will be compiled (e.g. source.xyz.jar=src/, src-ant/).
• extra.<library>-  classpaths to be used in addition to the bundle-classpath resolved from the dependencies. These can be 
  • File paths like ../someplugin/xyz.jar, or
  • Platform URLs like platform:/plugins/org.apache.ant/ant.jar
• exclude.<library> - the files that should not be copied into the library by the compiler, see below
• manifest.<library> - indicate the file that will be used as a manifest for the library. The file must be located in one of the source folders being used as input of the jar.
• output.<library> - the output folder receiving the result of the compilation;

Old-Style Classpath Additions

PDE also included a less granular form of classpath extension that is deprecated today:

• jars.extra.classpath - same effect as extra.<library> except that the entries are applied to all libraries.

Do not use it. It may stop working.

Definition of the Content of the Plugin or Bundle

• bin.includes - lists files that will be included in the binary version of the plug-in being built;
• bin.excludes - lists files to exclude from the binary build;

An Example File

Below is the build.properties file for the Ant Core plugin. It shows all the features described above by building 

• in order given by jars.compile.order 
• with some extra classpath for all archives, given by jars.extra.classpath
  • the root library . from src/ to bin/, and 
  • the ant support library lib/antsupportlib.jar from src_ant/ to src_ant_bin/ 
• and including a few resource files into the binary plugin using bin.includes

###############################################################################
# Copyright (c) 2000, 2013 IBM Corporation and others.
#
# This program and the accompanying materials
# are made available under the terms of the Eclipse Public License 2.0
# which accompanies this distribution, and is available at
# https://www.eclipse.org/legal/epl-2.0/
#
# SPDX-License-Identifier: EPL-2.0
# 
# Contributors:
#     IBM Corporation - initial API and implementation
###############################################################################

source.. = src/
output.. = bin/
output.lib/antsupportlib.jar = src_ant_bin/
source.lib/antsupportlib.jar = src_ant/
src.includes = about.html,\
               schema/,\
               about_files/

bin.includes = plugin.xml,\
               plugin.properties,\
               .,\
               about.html,\
               META-INF/,\
               about_files/,\
               lib/antsupportlib.jar

jars.compile.order = .,lib/antsupportlib.jar

jars.extra.classpath = platform:/plugin/org.apache.ant/lib/ant.jar,platform:/plugin/org.apache.ant/lib/ant-launcher.jar

javacWarnings..=-unavoidableGenericProblems

Running the Export

This is nice, but how do I actually build this now? Do I have to ramp up a PDE/Build setup? Actually no. There are two facilities that run PDE/Build in the background without actually saying so. These are the Plugin Export, the Installation into Running Instance and the Launch Eclipse Application Buttons. They hide on the right-hand side of the top of the Plugin Manifest Editor.

• Plugin Export and Installation into Running Instance are triggered by the plugin icon on the right. Plugin Export will carry out the build and then export to a directory of your choosing.
• Installation into Running Instance provides an option for the case that you want to add the Plugin you are compiling to the Eclipse IDE you are currently running. This will create a type of installation file, pack your plugin up in it, install it into your Eclipse in the way you normally install new software, and then reboot as you do for new software.
• The play icon in the left stands for "Launch Eclipse Application." This also packages your plugin as Installation into Running Host does, but instead of installing into your Eclipse, it creates a new installation of Eclipse and installs your plugin into that. It then starts the selected Eclipse Application using the Eclipse Application Launcher.
• The bug icon in the middle stands for "Launch Eclipse Application in Debug Mode." This will do what "Launch Eclipse Application" does, but it will connect your running IDE to the new Eclipse copy using the Debugger. This means that if you have breakpoints in your plugin source code, the debugger in your IDE will stop there.

In all, a big lot of potentially brittle functionality lies behind three very small buttons. The number of settings hiding under these three buttons is substantial and too much to cover here. Essentially, all we want for now is the plugin export.

Under the Hood: Build File Generation and Execution

We know that under the hood PDE/Build is driven by the Ant build system. Consequently we would expect that if we click the box to generate an Ant script in the user-interface described above, this would produce the Ant script that is actually being executed by PDE/Build build. Surprisingly, if you look at the outcome we will find something like the snippet below:

<project default="plugin_export" name="build">
<target name="plugin_export">
<pde.exportPlugins destination="/home/jg/Desktop" exportSource="false" exportType="directory" plugins="com.codebots.botlearn.ant" useJARFormat="true"/>
</target>
</project>

That does not seem right. After a brief forensic analysis of the Eclipse source code, we find that this particular ant task is nothing but a call-back to a click in the user interface. Ultimately, after a few hops, it calls the method FeatureBasedExportOperation.run(). An Operationin Eclipse is the "to do" part of a click or other IDE interaction, while the FeatureBasedExport hints at the fact that the plugin is being packaged to be exported. In the end, both the ant task and the user interface click call BuildScriptGenerator.generate()to generate another build script which is ultimately executed. For those interested, below is the full call hierarchy.

pde.exportPlugins (ant task/extension point) → PluginExportTask.new() → BaseExportTask.execute() → PluginExportTask.getExportJob() → PluginExportOperation.new() → FeatureBasedExportOperation.run() → FeatureExportOperation.doExport() → BuildScriptGenerator.generate()

Now we are really curious. Where can we get this mysterious build script? The answer is in the Eclipse Manual in the section Generating Ant Scripts. 

To create scripts, you can simply (sic!) select 'Create Ant Build File' while a suitable manifest file (plugin.xml, fragment.xml or feature.xml) is selected in the Navigator or Package Explorer views. The command will generate the build script.

More forensics show that this user interface interaction is actually carried out when PDE/Build is used to build a whole system. Here, the Operation is wrapped up in the ant task <eclipse.buildScript/>.

So, what code is actually generated when we trigger this function? Below is a call graph of a generated build.xml for a plugin that contributes a single ant library lib/com.codebots.botlearn.ant.jar in addition to the root library.

This graph was generated using the vizant Ant visualisation library. In it you see the dependency and sub-call relationships within an ant file.

What we can see is that all tasks initialize using init and that init determines the  properties that the build uses.

Below is a description of the functionality of the various targets. Targets in italics can be specialized using aspects, as shown in one of the following sections.

• build.update.jar: Builds or updates the jar that contains the plugin. It builds the jars using build.jars, the gathers the results using gather.bin.parts.build.jars:  Invoke each of the compilation targets (see below) that will be built for this bundle.
  • Compilation Targets with generated names:  The build.xml defines a compilation target for each jar being built (eg @dot for the bundle itself, or library.jar). These target names generated based on the build.properties specification.
• gather.bin.parts:  This target copies the build results (i.e. the jars and any other files being included in the bundle) to the final destination directory.  It is also the target where the version qualifiers are replaced in the manifest file (see Version Qualifiers).
• build.sources:  builds the source zips for each of the compilation targets.
• gather.sources: gathers the source zips for the bundle.
• gather.logs:  collects the log files from the build.
• clean:  removes results of previous builds.

In the nascence of Java, zip archives where used and Eclipse reflected this. This functionality is outdated and is not used today. However, targets are still present in the generated build file.

• zip.plugin: does the same thing as build.update.jar, except that it creates a zip which also includes sources. 
• build.zips: does nothing.

The Eclipse IDE uses a cached file system to accelerate work. This means that if the files "underneath" the cache change and shift, as would be expected with a file-oriented tool like Ant, the user interface does not reflect the changes. It looks like nothing happened. To avoid this, the script contains a convenience target.

• refresh: brings the workbench file caching up-to-date with changes on disk.

Custom Builds: Overriding or Advising the Build

 It is nice to see how this automatic build works in general, but what if I want my own tools to fire during this build, say for documentation generation or coverage? While this is by no means easy and Maven/Tycho may be what you will eventually be looking for, PDE/Build can either completely override the standard build or provide advice with an aspect-oriented approach. We will describe this in the following two sections.

Overriding the Build

If the property setting custom=true is found in the build.properties file, this indicates that the build script is hand-crafted as opposed to generated. The generator will not run and produce a new build file. The existing script will be called instead.

Since the script is invoked from the outside as part of PDE/Build's top-level control, the provide script has to expose the same signature as the automatically generated file. It has to have the same targets, expecting the same properties. A tall order.

The easiest way to assure this is to generate the build.xml using the User Interface as shown above and then amend it as desirable.

Overriding the whole build procedure used to be the way to go if one wanted any custom functionality in the build. This made for brittle builds because if a new version of PDE/Build changed the expected signatures, encased builds would break.

Aspect-Oriented Build

To address the problem of brittle builds, Custom Build Steps were introduced. In this approach, much like in aspect-oriented programming, an additional file is added to the build that provides Advice, generally before and after specific execution steps. In our case, these steps are Ant targets.

The templates subdirectory of the PDE/Build plugin directory contains a template called customBuildCallbacks.xml for this file that can be copied into the plugin and edited as required.

When the property customBuildCallbacks is set, the code generator that creates the build.xml file takes the value of the property as the path to an Ant file that contains the advice. 

Advice Targets

Some advice targets names depend on the libraries being built, but most are constant. Below is a table that describes the advice targets, the regular build target they surround, and the properties they receive when they are called.

For example, line 4 in the table describes operation: build for target: jars. This translates to an around advice provided by the couple of targets pre.build.jars and post.build.jars. Both receive property 7, which in the table below is the build.result.folder.  That is the directory where results of the Java compilation are assembled.

<classpath refid="library.jar.classpath"/>
<property name="mypath" refid="library.jar.classpath" />

Configuring the Subant Call

The advice targets are isolated from the rest of the build, because they are executed via Ant's subant task. To given more control to the developer writing the advice, three of subant's control parameters are accessible and can be set by adding the following properties prefixed by customBuildCallbacks. to the build.properties file.

Advanced Features

There are a number of additional features, some of which have already be hinted at previously.

Source Code for the Debugger

Debugging a Plugin-based system is not easy. Deep abstraction combined with slim documentation is a toxic mixture. If there is no easy way to come back to the source code as the definitive source of truth, projects can easily be stymied. Consequently, it is a courtesy to not only ship functionality but also a means to debug; And that means shipping source. Eclipse hence creates source plugins that are associated with the binary plugins whose source code they contain.

Behind the scenes, there are two ways to package source code into plugins in Eclipse. PDE/Build's default approach to building source code bundles still reflects the older method, but it can also generate the newer style Individual Source Bundles. All of this may affect the name of the bundle that is eventually generated, but not building individual plugins, which is what we are discussing here. 

There is, however, one aspect that may be of interest: custom content. The mechanism that Eclipse uses will copy all source code and resources from library folders, but occasionally it may be useful to add some other resources. This is what the confusingly named src. properties are for. They have nothing to do with the source you are intending to compile. Instead, they are used for assembling archives of your source code, so these can be looked up by a debugger later.

• src.includes - selects files to include in the source archive build
• src.excludes - selects files to exclude from the source archive build

As noted earlier, both properties are Ant pattern expressions.

PDE Support to Manage your Dependencies

Automatic Management of Dependencies is a useful tool that helps with finding classes that you depend on. With tons of plugins and bundles everywhere, that can be challenging. You give this tool a list of plugins or bundles that you guess might contain what you want, and the tool finds the ones you really need and adds them to the manifest. The list of candidates is kept in the property below, one plugin or bundle per line.

You list the dependencies after the header additional.bundles. These dependencies are not added to the MANIFEST.MF file immediately; however, you can start coding right away as if they were.

At any time, you can click the add dependencies hyperlink to have PDE analyze your code and generate the correct dependencies in your MANIFEST.MF file via either the Require-Bundle or Import-Package headers.

Distinguishing Build Versions

Version Qualifiers disambiguate small increments of the build that have the same semantic version and the same Eclipse version semantic. Usually, these are created using a local timestamp. PDE/Build performs this replacement based on the property qualifier in build.properties. There are a couple of different settings for the value of this property:

In a simple plugin build, the qualifier will be a timestamp in the form YYYYMMDDHHMM (i.e. 200605121600).

Compilation

Compiler control is quite important. Fortunately, the Eclipse PDE documentation has a good description of the compiler options that can be set in the build.properties. In this section I only want to mention a number of conceptual peculiarities of the build system:

• Compiler Adapters
• Reuse of Project Compiler Settings

Compiler Adapters: Building with Other Languages

The build scripts ultimately use the Java compiler. But what if a bundle uses a different language that can produce class files? PDE/Build solves this issue by asking for an adapter to be installed to provide access to the compiler for the language. Using this, Scala, Groovy, or AspectJ can be compiled. Read the compiler options documentation if you require this feature.

Reuse of Project Compiler Settings

Initially, the build.properties file was used to set all aspects of compilation. But the Java Developer Tooling also keeps a set of settings in a different place: the .settings/org.eclipse.jdt.core.prefs directory. This creates the issue that compilation in the IDE can produce a different result to compilation with a controlled build. As a consequence, PDE/Build was enhanced with a setting in build.properties that prompts it to read JDT compiler settings for configuration. The setting is activated by adding the property javacProjectSettings=true.

Slightly Overloaded: The Plugin Manifest Editor

The Plugin Manifest Editor is an editor with multiple tabs that tries to bring together all the required settings for a Plugin's build and runtime. While beautiful and well thought out, it is also heavily laden with substance and as intricate as an Asian hotel's high-quality buffet. Standing in front of it you are easily overwhelmed. In the following section, I am aiming to show you where the various concepts from the previous sections of this article are reflected.

The most important point to understand about the manifest editor is that is effectively broken into three sections. The first three panels deal only with OSGi-specific information. The second three panels only deal with specific aspects of Eclipse. The last three panels are assisted text editors which simplify editing the underlying Files. These are intended for work with properties that are not exposed through the form-based editors in the first two sections.

1. OSGi-specific Forms: These first three panels, with one exception, only edit the OSGi manifest.
   • Overview: The only important parts of the first panel are the general information and the execution environments forms. They identify the plugin and describe under what circumstances it can operate as a minimum. This data is reflected in the first few header entries in the manifest file. The sections on the right-hand side of the overview panel are intended as advised, but they are so compressed that they are more confusing than helpful. After reading the article to this point you might be able to understand what the intention is —  the article has roughly 4000 words to this point and the information panel probably has less than 100.
   • Dependencies: this panel splits into two distinct units, which confuses because they are lumped together. Required plugins and imported packages describe the dependency of the OSGi bundle. Automated management of dependencies and dependency analysis are Eclipse PDE tools wanting to be helpful with managing these requirements.
     1. OSGi Dependencies: the plugin and can describe what it needs from the environment either by defining the java packages it would like to import (imported packages), or by requiring that certain bundles need to be on the classpath (required plugins), or both.
     2. Eclipse PDE tools: needless to say, it's not easy to define these requirements without any help. This is where automated management of dependencies tries to support: It searches in a number of candidates bundles whenever the IDE finds the class name it does not know. If the facility can find the class, the developer can then add one of the two dependency mechanisms (bundle/package) described above for OSGi. The candidate bundles are stored in the build.properties file and can select from the current build target platform.
        Four helpful tools are hiding in the lower right-hand corner: Two call-graph-like facilities, that allow exploring who this plugin is transitively calling and who is calling the plugin. This functionality can also be used to discover usage cycles. Finally, classes that are not referenced from inside the plugin can probably be removed from the import. The unused dependencies tool supports this analysis.
   • Runtime: This tab lumps together the outside and inside view of the bundle. 
     1. The Exported Package panel and the Package Visibility panel are actually part of the same editor. The package export defines what packages the bundle makes available to its clients. The package visibility adds annotations to analyze the illegal or problematic access. This second panel addresses a very specialized functionality and is consequently very confusing to the regular user.
     2. The Classpath panel describes which libraries make up the classpath of the bundle. Confusingly, this panel is empty most of the time. This is because the route of the bundle is always perceived as being on the classpath. As soon as another library gets added the classpath root also appears in this panel.
2. Eclipse-Specific Forms: these three editors describe the functionality of Eclipse-specific frameworks (plugin.xml) and Eclipse PDE (build.properties). Unfortunately, the three panels are quite uneven in the frequency of use.
   • Extensions: This panel allows to define extensions that the plugin would like to use. It is not immediately obvious that the plugin does not need to depend on the plugin that provides the Extension Point. Another niggle I have with this panel is that it is not clear that you can easily get to the plugin the defines extension Point by using the "find declaring extension point" facility and then exporting the plugin.
   • Extension Points: This panel is intended to allow you to define new extension points. If used in this way, it creates new extension point schema files. However, this is a functionality that is almost never used. Given that it has such prominence it should be equipped with a warning saying that this is a rare activity.
   • Build: This panel formalizes the core of build.properties. The tick box at the very top enables and disables the use of the custom build system. The top two panels allow you to define the libraries and the source code directories that constitute them. The left middle panel describes what is assembled into the binary plugin that is shipped for runtime. The right middle panel describes what is added to the source plugin that is associated.  The folded extra classpath entries at the bottom define the deprecated additional classpath members for all libraries. This panel is almost unusable unless you already know how PDE/Build functions. There also is no help and no definition of the terms used in the panel. Assisted Text Editors: These editors are synchronized with some of the forms. This is quite confusing. Most of them do not have a canonization that formats them in a homogenous fashion.
     1. MANIFEST.MF: The parser in the editor is syntactic, not semantic. Given that most OSGI keys are known, it would be nice to have some more dedicated support in this panel. Even an extensible parser should not be too hard.
     2. plugin.xml: This is a nice editor for the plugin.xml. It is positive that there is no additional panel for the exsd schema definition, as this is hardly ever used.
     3. build.properties: The same issues that exist with the manifest editor also exist here. The editor does not use a context-aware parser, although this would be easily possible.