Monday, May 28, 2012

Mylyn and KDE keys conflict

Mylyn allows to you reveal what is inside of a folder by clicking on it while holding the 'Alt' key. For some reason it worked for me in Fedora as 'Fullscreen off'. Now I know why :-). The reason is this default KDE setting:


One rule that I always follow while creating key shortcuts is that 'Windows'/'Meta' key is used for application management purposes and interaction with the operating system, while all other key combinations (without 'Windows'/'Meta' key) serve for application purposes.

Changing the 'Modifier key' to 'Meta' restores Mylyn functionality and reduces the space of possible conflicts with other applications.

BTW. You can imagine my despair when Unity was introduced to Ubuntu and the 'Meta' key always activated it...

Wednesday, May 23, 2012

Improving Eclipse dropins - What P2 and polish politcs had in common

This week started for me with sounds of well known tune:

Hello console, my old friend,
I've come to work with you again,
Because the reconciler softly creeping,
Left its staff while I was dreaming,
And the bundles that were planted in my dropins
Still remain
Within the space of profilins*.

It's all about the situation where certain operations,  described here in length, could break P2 profile in such a way, that P2 thought that physically removed bundle was still installed. The reproduction steps in the bug are really lengthy, but RPM version is short:

sudo yum install eclipse
sudo yum install eclipse-cdt
eclipse
sudo yum remove eclipse-cdt
sudo yum install eclipse-changelog 
eclipse #what? CDT is still in the profile?

Further investigation has revealed that it was changelog bug (in the first place), because it did not required cdt although it should. During the reconciliation phase, cdt was not yet removed from the profile, so P2 believed it was able to prevent its uninstallation, and during debug, explained its plan with friendly messages:

[p2] Mon May 21 13:07:14 CEST 2012 - [Start Level Event Dispatcher]
[reconciler] [plan] Some units will not be uninstalled:
[p2] Mon May 21 13:07:14 CEST 2012 - [Start Level Event Dispatcher]
[reconciler] [plan] org.eclipse.cdt.ui 5.4.0.201203191015
[p2] Mon May 21 13:07:14 CEST 2012 - [Start Level Event Dispatcher]
[reconciler] [plan] org.eclipse.cdt.core 5.4.0.201203191015

So, my first attempt to fix the issue was rather naive - uninstall removed bundles first, then perform all the reconciliation that has left. Result:  proper behaviour in this test case, but all other dropins tests broken. Not good.

But thanks to @dj_houghton and @prapicault, I was able to came up with something much easier, that not only fixes my unit tests case, but also manages to not break any other reconciler test:

@@ -123,6 +126,15 @@ public class ProfileSynchronizer {
   if (moveResult.getSeverity() == IStatus.ERROR || moveResult.getSeverity() == IStatus.CANCEL)
    return moveResult;
 
+  if (!request.getRemovals().isEmpty()) {
+   Collection requirements = new ArrayList();
+   for (IInstallableUnit unit : request.getRemovals()) {
+    RequiredCapability req = new RequiredCapability(IInstallableUnit.NAMESPACE_IU_ID, unit.getId(), new VersionRange(unit.getVersion(), true, unit.getVersion(), true), null, 0, 0, false, null);
+    requirements.add(req);
+   }
+   request.addExtraRequirements(requirements);
+  }
+
   // now create a plan for the rest of the work and execute it
   IStatus addRemoveResult = performAddRemove(request, context, monitor);
   if (addRemoveResult.getSeverity() == IStatus.ERROR || addRemoveResult.getSeverity() == IStatus.CANCEL)

The patch should be read: for each removed bundle add a negative requirement to the request, so P2 will remove those bundles no matter what. That's so simple. I do not know why it took me 3 days :-(.

And now the explanation of the title. The whole situation with P2 refusing to uninstall deleted bundles reminds me of a polish health care. It's a well know fact that medical help should reach people needing it in 12 minutes because even seriously wounded usually do not pass away in that time. So the norm for an ambulance to reach its target was 12 minutes, but the ratio of timely arrivals was to low (< 85%). So the parliament extended the time to 16 minutes :-).

If all goes well, this fix may hit Fedora as yet another 0 day update.

And a small bonus - have you noticed that (at least on Linux) you are finally informed what will be the action invoked? Check the screenshot below:


For a long time it was almost a matter of luck - you never knew if you hit the button or drop down menu.

* I will buy** a beer to a person that finds a good rhyme here.
** Applicable when we meet in person.

EDIT: Added proper twitter link to LeNettoyeur which turned out to be prapicault.

Saturday, May 19, 2012

Eclipse in Fedora - 0 day update

Eclipse bundled by Fedora will contain additional fixes that are not yet a part of Juno release:

1. Predefined update sites


You may say that's all Eclipses have that, but it is not quite true. You have not just used Eclipse with a shared configuration. However, in Linux, this is a default installation, so many many people reported this problem, and for some Linux distributions it was a blocker that prevented them from including Eclipse. But things are getting better - Fedora, and eclipse-build, now include a fix for that issue.

2. Discover renamed jars in dropins.

Again, this issue does not happen often in regular Eclipse installations. But if for some reason you need to rename a jar in dropins, P2 will refuse to load it. In Fedora, each jar should be packaged only once (f.e. junit), so each discovered duplicated library is replaced with a symlink. Sometimes Eclipse refused to resolve dropins, now we know why :-).

3. Reconciler debugging options are now in a default installation.

As of previously, when anyone reported the problem with Eclipse not being loaded fully, we had to guide him through creating the .options file, putting it in right directory and gathering logs. Now it is easier. To get a hint what is going on with reconciler it is enough to invoke following commands:
$ cd /usr/lib64/eclipse
$ ./eclipse -debug -clean > reconciler.log

4. Eclipse cannot be run as root.

This is the most controversial change, but when you try to launch Eclipse as root, you will get:
# eclipse
CompilerOracle: exclude org/eclipse/core/internal/dtree/DataTreeNode.forwardDeltaWith
CompilerOracle: exclude org/eclipse/jdt/internal/compiler/lookup/ParameterizedMethodBinding.
CompilerOracle: exclude org/eclipse/cdt/internal/core/dom/parser/cpp/semantics/CPPTemplates.instantiateTemplate
CompilerOracle: exclude org/eclipse/cdt/internal/core/pdom/dom/cpp/PDOMCPPLinkage.addBinding
CompilerOracle: exclude org/python/pydev/editor/codecompletion/revisited/PythonPathHelper.isValidSourceFile
CompilerOracle: exclude org/python/pydev/ui/filetypes/FileTypesPreferencesPage.getDottedValidSourceFiles
This application should not be run by the owner of the shared configuration.

and exit code will be 14.

The reason behind that is pretty simple - when you invoke the Eclipse as root, everything what you have in dropins will be written into the master configurations. And later, when you change your installation layout using RPM, things will go ugly, because the master configuration will not reflect what is really installed.

If you really, really need to launch Eclipse as root in Fedora, and you know what consequences are to expect, please remove -preventMasterEclipseLaunch from your /usr/lib64/eclipse.ini.

Note 1. All paths used in this blog post are for 64 bit systems. If you happen to run 32 bit Fedora, please replace every lib64 with lib, so the base Eclipse installation folder will be /usr/lib/eclipse.

Note 2. Fedora 17 (a.k.a. Beefy Miracle) is out pretty soon. You can grab beta here.

Wednesday, May 16, 2012

Making P2 working with RPM - last milestone

Maybe you remember my last post about making P2 and RPM working robustly together. Having the Eclipse Platform M7 packaged in Fedora, and Fedora frozen, I have finally found time to look at that work again.

So, first of all, I was surprised how poor programmer was the past me. Then I have fixed a couple of hardcoded paths, got the solution working again, and even managed to prepare and install my first RPM-wrapped feature into the modified version of Eclipse without breaking the ownership of files.

Ownership of files is a pretty important aspect of Fedora security mechanism - a file installed by RPM into /usr should not be changed because otherwise the verification of system will fail - the system will be believed to be compromised.  P2 changes many files in Eclipse installation if it only is allowed to do so.

After the installation - Eclipse successfully started with my new RPM feature installed. But then I have discovered a hole in my logic. And then my logic appeared to be like a cheese:


I totally forgot about the fact that once I generate a profile from an assembled installation, P2 will start working as usual, and I will not be able to uninstall anything. I'm not sure if you know it, but P2 is an API to control the bundles.info file, which is

only.

P2 does not care if your bundles.info got changed. It will overwrite it.

My attempts to solve that problem made the scope of changes growing very fast. I even created wiki page and put there some things that need to be taken into account.

I'm sorry folks, especially for those haunted by issues similar to:

You will not get a revamped version of P2. I do not think it is possible and I'm officially tossing this idea. You have to be patient - there is no instant cure for those issues. Eclipse Shared Installation works poorly out of the box (see bug 358471), and things get even worse if you take into account additional restrictions that Fedora puts on P2.

In the meantime - what do you think about preventing Eclipse from being run by root by accident?  It is terribly easy to do that on Linux, but then all your dropins stuff goes into master profile and you cannot revert that.

Monday, May 7, 2012

Little things that matter

Installing Eclipse in Fedora is deadly easy. It is enough to type
$ sudo yum groupinstall Eclipse
$ eclipse
to enjoy latest and greatest Eclipse installation, including all major projects (Eclipse itself, EMF, GEF, Mylyn, Git, WTP and others). Now it is time for the small thing - error reporting. Up until now, it was possible to do this using Mylyn:
Eclipse packaged by Fedora is now better integrated with the OS, so it is possible to report errors directly to Red Hat bugzilla.
Delivering feedback couldn't be easier!

Thursday, May 3, 2012

Mylyn hidden gem - finding tasks to read.

We all know that feeling, having a dozen or so of incoming changes in Mylyn tasks, and a few hundreds of them in the list. Scrolling the whole list just to find those changed tasks may be a real pain.


I was wondering how it is possible that Mylyn does not offer filtering incoming changes. But quick look into Mylyn user guide revealed, that it is possible, but you need to switch to 'Scheduled' mode:


And then all dates will be visible - and some scheduling categories appear - and 'Incoming' tasks. That's what I was looking for:


I do not think this is a good idea to have such an important functionality so hidden - I have filled bug 378334: Make focusing on incoming tasks easier.

Tuesday, February 21, 2012

Making P2 working with RPM - milestone I

One of the biggest benefits of RPM is that it allows you to verify your installation and discover any modifications that had been made to your system with (or usually without) your notice.
Unfortunately, such a functionality enforces certain process during package installation - a package should be ready to use just after it is deployed into proper location.
It does not take a lot to understand that P2 will not work with RPM. P2 requires at last the p2 director to be run, but doing so causes instant security alerts for Fedora administrators. Therefore it is a bad idea to launch Eclipse as root. So you cannot actually install anything into your shared installation, unless you use a reconciler approach, ie. you install a small reconciler application, put everything in dropins and hope for the best. The problem is that the reconciler is a legacy solution that works on the "best-can-do" approach, which sometimes is very far from the "best", or even "working".
It is not that P2 or RPM are bad, they are not just designed to cooperate with each other.
This blog post describes my first attempt of getting P2 and RPM working together. The idea is pretty simple:
  • An administrator should not launch Eclipse and modify shared installation directly. Protecting the shared configuration seems to be a rather easy task, so I will deal with it later. As a temporary workaround I have deleted master P2 folder. Ugly, but it works.
  • An administrator deploys plugins and features in the shape, which are ready to be run, together with an .info which is in the bundles.info standard. The administrator is responsible for correctness and completeness of the bundles being installed. This work will be done in Fedora packaging.
  • P2 reads all the .info file, assembles one big master installation on first run, and let the user install anything he wants into his ~/.eclipse folder using P2 UI. Easier said that done - the description is below.




Step I - reading multiple .info files during Eclipse start 
If you have ever wondered how Eclipse knows which plugins should it load, the answer is pretty simple. Eclipse loads just one plugin (with dependencies) - org.eclipse.equinox.simpleconfigurator, which then finds a file named bundles.info with the list of all files. P2 is just an interface between the world and that file [1]. Modifying this file directly or indirectly is not an option for RPM, but making each fedora package delivering its own pregenerated info file is acceptable. So let's make the  simpleconfigurator concatenating multiple .info files.



Step II - spoofing a profile
It is necessary to manipulate the SimpleProfileRegistry to create a fake profile when no profile was found:
public synchronized IProfile getProfile(String id) {
  Profile profile = internalGetProfile(id);
  if (SELF.equals(id)) {
   id = "PlatformProfile";
   self = id;
  }
  if (profile == null) {
   try {
    PlatformAdmin platformAdmin = (PlatformAdmin) ServiceHelper.getService(EngineActivator.getContext(), PlatformAdmin.class.getName());
    IProvisioningAgent agent = (IProvisioningAgent) ServiceHelper.getService(EngineActivator.getContext(), IProvisioningAgent.class.getName());
    IProfileRegistry registry = (IProfileRegistry) agent.getService(IProfileRegistry.class.getName());
    IEngine engine = (IEngine) agent.getService(IEngine.class.getName());
    IMetadataRepositoryManager repoMgr = (IMetadataRepositoryManager) agent.getService(IMetadataRepositoryManager.class.getName());
    IArtifactRepositoryManager artifactRepoMgr = (IArtifactRepositoryManager) agent.getService(IArtifactRepositoryManager.class.getName());

    Collection ius = new Reify().reify(platformAdmin);

    ius.add(Reify.createDefaultBundleConfigurationUnit());
    ius.add(Reify.createUpdateConfiguratorConfigurationUnit());
    ius.add(Reify.createDropinsConfigurationUnit());
    ius.add(Reify.createDefaultFeatureConfigurationUnit(""));

    return spoofUpProfile(id, agent, registry, engine, ius);
   } catch (ProvisionException e) {
    e.printStackTrace();
    return null;
   }
  }

  return profile.snapshot();
 }
The Reify class is a magic class based on the earlier Pascal work - it basically extracts all the P2 data from the running installation. The actual spoofUpProfile method sets properties, creates and executes the plan:

 private synchronized IProfile spoofUpProfile(String id, IProvisioningAgent agent, IProfileRegistry registry, IEngine engine, Collection ius) throws ProvisionException {

  Map prop = new HashMap();

  Location installLocation = (Location) ServiceHelper.getService(EngineActivator.getContext(), Location.class.getName(), Location.INSTALL_FILTER);
  File installFolder = new File(installLocation.getURL().getPath());

  Location configurationLocation = (Location) ServiceHelper.getService(EngineActivator.getContext(), Location.class.getName(), Location.CONFIGURATION_FILTER);
  File configurationFolder = new File(configurationLocation.getURL().getPath());

  // We need to check that the configuration folder is not a file system root. 
  // some of the profiles resources are stored as siblings to the configuration folder.
  // also see bug 230384
  if (configurationFolder.getParentFile() == null)
   throw new IllegalArgumentException("Configuration folder must not be a file system root."); //$NON-NLS-1$

  File launcherConfigFile = new File(configurationFolder, "eclipse.ini.ignored");

  //  prop.put("org.eclipse.update.install.features", "true");
  prop.put(IProfile.PROP_ENVIRONMENTS, "osgi.nl=en_US,osgi.ws=gtk,osgi.arch=x86_64,osgi.os=linux");

  prop.put(IProfile.PROP_INSTALL_FOLDER, installFolder.getAbsolutePath());
  prop.put(IProfile.PROP_SHARED_CACHE, installFolder.getAbsolutePath());
  prop.put(IProfile.PROP_ROAMING, Boolean.FALSE.toString());
  prop.put(IProfile.PROP_CONFIGURATION_FOLDER, configurationFolder.getAbsolutePath());
  prop.put(IProfile.PROP_CACHE, configurationFolder.getParentFile().getAbsolutePath());
  prop.put(IProfile.PROP_LAUNCHER_CONFIGURATION, launcherConfigFile.getAbsolutePath());
  prop.put("org.eclipse.update.install.features", "true");

  IProfile profile = registry.addProfile(id, prop);
  IProvisioningPlan plan = helper.getPlan(profile, ius);

  ((Profile) profile).setChanged(false);
  IPhaseSet phaseSet = PhaseSetFactory.createDefaultPhaseSetExcluding(new String[] {PhaseSetFactory.PHASE_CHECK_TRUST, PhaseSetFactory.PHASE_COLLECT, PhaseSetFactory.PHASE_CONFIGURE, PhaseSetFactory.PHASE_UNCONFIGURE, PhaseSetFactory.PHASE_UNINSTALL});
  IStatus status = engine.perform(plan, phaseSet, null);

  if (!status.isOK())
   return null;

  return profile;
 }

Creating a plan is actually kind of hack. The SimpleProfileRegistry is in the engine, and it cannot reference IPlanner service, because it would cause cyclic dependencies. So I have created a helper field, where the Director sets the proper service:
 public void start(BundleContext ctx) throws Exception {
  context = ctx;
  SimpleProfileRegistry.helper = new IPlannerHelper() {

   public IProvisioningPlan getPlan(IProfile profile, Collection ius) {
    ProfileChangeRequest request = new ProfileChangeRequest(profile);
    request.addAll(ius);
    IProvisioningAgent provisioningAgent = profile.getProvisioningAgent();
    IPlanner planner = (IPlanner) provisioningAgent.getService(IPlanner.class.getName());
    return planner.getProvisioningPlan(request, new ProvisioningContext(profile.getProvisioningAgent()), null);
   }
  };
 }
One more thing to do is to start both required plugins when dropins are started to ensure that the helper is registered. This is not perfect solution, but once I will find out how to do it better, I will correct it.
  ServiceHelper.getService(getContext(), IProvisioningAgent.SERVICE_NAME);
  ServiceHelper.getService(getContext(), IPlanner.class);

The P2 data generated from the running platform are partially inaccurate. First of all, there is  no P2 repositories (and we do not actually move any jars), and the information about which plugins should be run is lost. I have temporary workarounded that by modyfing SimpleConfiguratorManipulator and hardcoding certain plugin names.

Summing up, there is still a lot of work to be done, but the basic scenario, where an administrator assembles shared installation, and a regular user is able to add custom plugins using P2, and RPM is still happy about it, seems to work.

A complete patch is accessible here. Feel free to suggest updates, comment or criticize, and stay tuned for next patch versions.

Big THANK YOU to @irbull for his help.

BTW. I have no idea if this patch will ever get into P2 repository. It will become a part of Fedora Eclipse build process most likely.