Tuesday, February 21, 2012

Making P2 working with RPM - milestone I

One of the biggest benefits of RPM is that it allows you to verify your installation and discover any modifications that had been made to your system with (or usually without) your notice.
Unfortunately, such a functionality enforces certain process during package installation - a package should be ready to use just after it is deployed into proper location.
It does not take a lot to understand that P2 will not work with RPM. P2 requires at last the p2 director to be run, but doing so causes instant security alerts for Fedora administrators. Therefore it is a bad idea to launch Eclipse as root. So you cannot actually install anything into your shared installation, unless you use a reconciler approach, ie. you install a small reconciler application, put everything in dropins and hope for the best. The problem is that the reconciler is a legacy solution that works on the "best-can-do" approach, which sometimes is very far from the "best", or even "working".
It is not that P2 or RPM are bad, they are not just designed to cooperate with each other.
This blog post describes my first attempt of getting P2 and RPM working together. The idea is pretty simple:
  • An administrator should not launch Eclipse and modify shared installation directly. Protecting the shared configuration seems to be a rather easy task, so I will deal with it later. As a temporary workaround I have deleted master P2 folder. Ugly, but it works.
  • An administrator deploys plugins and features in the shape, which are ready to be run, together with an .info which is in the bundles.info standard. The administrator is responsible for correctness and completeness of the bundles being installed. This work will be done in Fedora packaging.
  • P2 reads all the .info file, assembles one big master installation on first run, and let the user install anything he wants into his ~/.eclipse folder using P2 UI. Easier said that done - the description is below.

Step I - reading multiple .info files during Eclipse start 
If you have ever wondered how Eclipse knows which plugins should it load, the answer is pretty simple. Eclipse loads just one plugin (with dependencies) - org.eclipse.equinox.simpleconfigurator, which then finds a file named bundles.info with the list of all files. P2 is just an interface between the world and that file [1]. Modifying this file directly or indirectly is not an option for RPM, but making each fedora package delivering its own pregenerated info file is acceptable. So let's make the  simpleconfigurator concatenating multiple .info files.

Step II - spoofing a profile
It is necessary to manipulate the SimpleProfileRegistry to create a fake profile when no profile was found:
public synchronized IProfile getProfile(String id) {
  Profile profile = internalGetProfile(id);
  if (SELF.equals(id)) {
   id = "PlatformProfile";
   self = id;
  if (profile == null) {
   try {
    PlatformAdmin platformAdmin = (PlatformAdmin) ServiceHelper.getService(EngineActivator.getContext(), PlatformAdmin.class.getName());
    IProvisioningAgent agent = (IProvisioningAgent) ServiceHelper.getService(EngineActivator.getContext(), IProvisioningAgent.class.getName());
    IProfileRegistry registry = (IProfileRegistry) agent.getService(IProfileRegistry.class.getName());
    IEngine engine = (IEngine) agent.getService(IEngine.class.getName());
    IMetadataRepositoryManager repoMgr = (IMetadataRepositoryManager) agent.getService(IMetadataRepositoryManager.class.getName());
    IArtifactRepositoryManager artifactRepoMgr = (IArtifactRepositoryManager) agent.getService(IArtifactRepositoryManager.class.getName());

    Collection ius = new Reify().reify(platformAdmin);


    return spoofUpProfile(id, agent, registry, engine, ius);
   } catch (ProvisionException e) {
    return null;

  return profile.snapshot();
The Reify class is a magic class based on the earlier Pascal work - it basically extracts all the P2 data from the running installation. The actual spoofUpProfile method sets properties, creates and executes the plan:

 private synchronized IProfile spoofUpProfile(String id, IProvisioningAgent agent, IProfileRegistry registry, IEngine engine, Collection ius) throws ProvisionException {

  Map prop = new HashMap();

  Location installLocation = (Location) ServiceHelper.getService(EngineActivator.getContext(), Location.class.getName(), Location.INSTALL_FILTER);
  File installFolder = new File(installLocation.getURL().getPath());

  Location configurationLocation = (Location) ServiceHelper.getService(EngineActivator.getContext(), Location.class.getName(), Location.CONFIGURATION_FILTER);
  File configurationFolder = new File(configurationLocation.getURL().getPath());

  // We need to check that the configuration folder is not a file system root. 
  // some of the profiles resources are stored as siblings to the configuration folder.
  // also see bug 230384
  if (configurationFolder.getParentFile() == null)
   throw new IllegalArgumentException("Configuration folder must not be a file system root."); //$NON-NLS-1$

  File launcherConfigFile = new File(configurationFolder, "eclipse.ini.ignored");

  //  prop.put("org.eclipse.update.install.features", "true");
  prop.put(IProfile.PROP_ENVIRONMENTS, "osgi.nl=en_US,osgi.ws=gtk,osgi.arch=x86_64,osgi.os=linux");

  prop.put(IProfile.PROP_INSTALL_FOLDER, installFolder.getAbsolutePath());
  prop.put(IProfile.PROP_SHARED_CACHE, installFolder.getAbsolutePath());
  prop.put(IProfile.PROP_ROAMING, Boolean.FALSE.toString());
  prop.put(IProfile.PROP_CONFIGURATION_FOLDER, configurationFolder.getAbsolutePath());
  prop.put(IProfile.PROP_CACHE, configurationFolder.getParentFile().getAbsolutePath());
  prop.put(IProfile.PROP_LAUNCHER_CONFIGURATION, launcherConfigFile.getAbsolutePath());
  prop.put("org.eclipse.update.install.features", "true");

  IProfile profile = registry.addProfile(id, prop);
  IProvisioningPlan plan = helper.getPlan(profile, ius);

  ((Profile) profile).setChanged(false);
  IPhaseSet phaseSet = PhaseSetFactory.createDefaultPhaseSetExcluding(new String[] {PhaseSetFactory.PHASE_CHECK_TRUST, PhaseSetFactory.PHASE_COLLECT, PhaseSetFactory.PHASE_CONFIGURE, PhaseSetFactory.PHASE_UNCONFIGURE, PhaseSetFactory.PHASE_UNINSTALL});
  IStatus status = engine.perform(plan, phaseSet, null);

  if (!status.isOK())
   return null;

  return profile;

Creating a plan is actually kind of hack. The SimpleProfileRegistry is in the engine, and it cannot reference IPlanner service, because it would cause cyclic dependencies. So I have created a helper field, where the Director sets the proper service:
 public void start(BundleContext ctx) throws Exception {
  context = ctx;
  SimpleProfileRegistry.helper = new IPlannerHelper() {

   public IProvisioningPlan getPlan(IProfile profile, Collection ius) {
    ProfileChangeRequest request = new ProfileChangeRequest(profile);
    IProvisioningAgent provisioningAgent = profile.getProvisioningAgent();
    IPlanner planner = (IPlanner) provisioningAgent.getService(IPlanner.class.getName());
    return planner.getProvisioningPlan(request, new ProvisioningContext(profile.getProvisioningAgent()), null);
One more thing to do is to start both required plugins when dropins are started to ensure that the helper is registered. This is not perfect solution, but once I will find out how to do it better, I will correct it.
  ServiceHelper.getService(getContext(), IProvisioningAgent.SERVICE_NAME);
  ServiceHelper.getService(getContext(), IPlanner.class);

The P2 data generated from the running platform are partially inaccurate. First of all, there is  no P2 repositories (and we do not actually move any jars), and the information about which plugins should be run is lost. I have temporary workarounded that by modyfing SimpleConfiguratorManipulator and hardcoding certain plugin names.

Summing up, there is still a lot of work to be done, but the basic scenario, where an administrator assembles shared installation, and a regular user is able to add custom plugins using P2, and RPM is still happy about it, seems to work.

A complete patch is accessible here. Feel free to suggest updates, comment or criticize, and stay tuned for next patch versions.

Big THANK YOU to @irbull for his help.

BTW. I have no idea if this patch will ever get into P2 repository. It will become a part of Fedora Eclipse build process most likely.

Wednesday, February 8, 2012

Fedora Eclipse Build: far Orbit

Today I will dwell on the issue that hit me when I tried to build a recent Eclipse - something pre-M5. Previous builds of Eclipse using eclipse-build were done for M1, so it is easy to imagine what could go wrong.

It started very innocently - unsatisfied package javax.servlet_2.6 in a number of plugins. This clearly indicated a problem with the build - for some reason the package javax.servlet was missing or not built properly.

And here comes the first surprise - javax.servlet package is not Eclipse package, it is a 3rd party library. A short glimpse at the dependecies.properties (a file that maps non-Eclipse library to Fedora libraries) and here it its mapping:


And there... there is package javax.servlet_3.0 exported!

That's interesting. Quick search for 2.6 version... There is no such version of javax.servlet specification. Something is wrong. How is it possible that half of the Eclipse plugins uses a version of the javax.servlet that does not exists?

The answer was finally found here:

The Servlet 3 specification is a breaking change for implementers, but is
binary compatible for client applications.  Following the version guidelines
the javax.servlet packages should move from version 2.5 -> 2.6
The workaround for that issue was not simple. I could not patch Eclipse to use 3.0 (too many patches), nor downgrade version in tomcat-servlet-3.0-api - because of too many dependencies.

Thankfully, OSGi allows for exporting the same package more than once:

Export-Package: javax.servlet;version="3.0", javax.servlet;version="2.6",

One patch for Tomcat, and one problem in Eclipse Fedora Build solved :-).

The problem that I am trying to highlight is that Eclipse Foundation maintains its own version of libraries in Orbit, and it may surprise users. I do not want to say that renaming 3.0 to 2.6 was unjustified (because it was), but on the other hand, does Eclipse have to maintain its own copies of libraries, which are unusable for the rest of the world?

Maybe it is a good time to start discussion about Orbit removal, and a true, transparent connection to other libraries, where each local modification should be sent to the upstream?