Friday, December 28, 2012

Fedora Eclipse - update to a snapshot version

I decided to update Eclipse version to the repository snapshot and push this new build into Fedora 18. It's available in updates-testing repository. This is not something that is welcome in Fedora world as unreleased software tends to be unstable.
But this time is a little bit different. If you open the 'Install Software' dialog, you will notice that a new category appeared - Juno SR1 Patches:


This patch fixes, among others,  the most annoying issue with Eclipse 4.2.1 -
Bug 391868 - very slow switching editors; especially XML.
Only this bug made it worth to push a new build into Fedora 18. If you are interested in what else was fixed, look at this bugzilla query

Happy Hacking!


Thursday, December 27, 2012

Installing ADT into Fedora Eclipse

ADT have quite a long history of not working with Fedora Eclipse due to the various ClassCastExceptions and ClassNotFoundExceptions. Majority of those problems is caused by a simple fact - JVM provides a number of java packages that are also provided by Eclipse plugins (javax.xml, org.w3c.dom and others) and were part of Eclipse before they were introduced to JVM.

So, installing any plugin (like org.apache.xerces required by ADT) that required javax.xml caused two instances of those package to live together in different classloaders. Whenever they met, CCE or CNFE was thrown. Typical scenario was:
  • Plugin A is started and loads javax.xml (VM version)
  • Plugin B is started and loads javax.xml (bundle version).
  • Plugin C calls A, gets some javax.xml class instance, and passes it to plugin B. Since each plugin has a separate classloader, CCE is thrown.
Anyway, the solution is now known - basically you need to install eclipse and eclipse-dtp to get the hierarchy stack right - after that you may install ADT and it will work (if only you ensure that you have at least this version of xerces-j2).

Unfortunately there is no way to fix the Eclipse package itself - as it is not broken. It is just the matter of integration - and it will not happen until there is someone who is willing to package ADT for Fedora. I'm going to propose this task as a GSoC project (unless someone steps up earlier).

Friday, November 23, 2012

Do you still use x86 arch?

The title is deliberately misleading, because this (r)evolution has not happened yet, although signs of x86 empire falling down are ubiquitous - I bet you hold one in your pocked, or maybe you even read this post on an arm-powered device. If that's not convincing you, there is even more signs:
  • Engadget says that Microsoft blames OEMs for lower-than-expected Windows 8 sales. What seems to be outstanding here is that not enough of new hardware is being sold. My personal and subjective observations show that personal usage of PC dropped significantly - it's much more convenient to consume internet content using tablet-ish devices on your sofa than be limited by a weight of a laptop or netbook (which is still heavy compared to a tablet).
  • Furthermore, Intel CEO is said to retire early. That may sound like a conspiracy theory, but, honestly, have you ever seen a CEO stepping "unexpectedly" back without a reason? Which might be the Intel absence in the mobile world? This is not the end of the bad news to Intel.
  • Apple is said to ditch x86 in favour of ARM. No more comments needed here.

But why?

The answer is pretty simple - power efficiency. According to this blog, Intel processor requires 4 times more power than ARM one while adding only 30% of performance. In other words, your battery will last three times longer with an ARM. That's significant, isn't it?

Fedora has been working on ARM for quite some time. There are even emulators that you can run on your x86 hardware.

Eclipse was built for ARM some time ago, but it was a PDEBuild style, which is another empire that is falling down (but that's completely separated story). I'm currently in the middle of CBI build. You may not share my excitement, but for me it is a really huge thing - after reall, really long time I have finally nailed the reason for which Tycho was failing with an ugly NPE - and the reason was named JIT, you know, the thing that compiles your java bytecode to the native instruction sets in order to execute them faster. ARM OpenJDK Jit turned out to optimize one loop so much that it was executed only once.

So, keep your fingers crossed, and arm yourself up!

Thursday, October 25, 2012

EclipseCon Europe 2012 - Impressions

This year EclipseCon was extremely interesting. There was a lot of talks, but three main topics will remain in my mind for longer. I could even call them "surprises".

JavaFX
Kai Todter and Tom Shindl prepared two separate implementations and adopted JavaFX for the RCP purposes. I was aware they were working on it, but results were astonishing. The coolest think that I saw was when Tom installed a bundle just by dropping it on the top of his application  (maybe something like this could be implemented for regular Eclipse, too?). Another cool issue was native builds for the MacOS, which included Java (this is pretty important as Apple itself does not ship it). I really think that JavaFX has the potential to be the next SWT, and I'd like to see it in mainstream as soon as possible. And no, Oracle did not bribed me with a Raspberry Pi.

Orion
The Orion project seems to gather momentum. There was even an example of a real-world application called "Scripted" and done by VMware folks. If you add to this a Mozilla keynote, where Mozilla OS was presented, you cannot miss the value of Orion. It looks like the future consumer market will be totally dominated by JavaScript applications.

Maven/Tycho
Tycho is a black horse of this conference for me. Despite suffering from serious integration issues (as explained by Igor Fedorenko), it is widely adapter for build (not necessarily for development). I have not seen anyone talking about PDEBuild (except me), so the only thing I can say now is:
The King is dead. Long live the King.
Face it, whether you like it or not.



Last, but not least, there will be a new EclipseCon family member. In 2013 expect:
  • EclipseCon (March)
  • EclipseCon France (June)
  • EclipseCon Europe (October)
More awesome talks!

Tuesday, September 11, 2012

Ignorantia iuris nocet

I recall Platform switch to 4.2 nearly a year ago, followed by appropriate discussion in the cross project newsgroup.

No objections then, although 4.2 was unusable at that point. Really. Even basic and the most core projects were throwing vile exceptions during 5-minutes-what's-the-impression test.

Even after M6, Juno was still barely usable - being an Eclipse packager means being a very early adopter. And then something amazing happened - with each build there was a clearly visible improvement. Less bugs. Less exceptions. Less surprises. A few months more and Juno would be the state-of-art software.

The platform team indicated that it is understaffed and more community help is needed to make Juno successful in the meantime.

And nothing really happened, probably because of a common belief that Platform is granted*, and expected to always work. That has been proven to not work. Open Source is about freedom, and freedom is about contributions. The blog title explains it all. It's too late. There was a time to protest and demand switch back to 3.x stream nearly a year ago. This could be possible then, but... it would require at least some testing, which was important, but not urgent ;-).

I do not think switching to 3.x is doable anymore. So, maybe instead of complaining about 4.x shape, it is time to fix some bugs?

It's not about charity, it's not about common good. It's all about being as much egoistic as possible. It's all about solving your problems!  Freedom is about contributions.

I have opened a bug, in which I request to lower the entry barrier to the platform. If you have an idea what could help to grow the commiter pool, please comment.

* This statement was formulated by Pascal, although I failed to find a proper link to it.

Thursday, August 23, 2012

Freedom is about contributions

The most annoying thing about Fedora is that it breaks from time to time. Failures vary in their scope. Sometimes they're really subtle (false security alerts) and sometimes I really want to check if my laptop is as shock resistant as manufacturer claims.

That's the cost of being at the bleeding edge. People experiment, do constant changes, improvements, and introduce new bugs. One may even say that the number of newly fixed bugs could be used to measure  software vitality ;-).

On the other hand, some things that I really, really love make up for everything:
  • Freedom - usually it means that you can do what you want with your software. For example, compile KDE with transparency support:
    Can you see the orange button through the KCalc?
    Many folks tends to forget about the most important aspect of freedom - which is fixing bugs that annoy you, for yours and others convenience! Remember, complaining is a really bad habit. Don't complain. Report it, fix it or live with it! That's why there is so many Linux distributions. Each one fixes something :-)
  • People. If you are ready to invest into a community, the community will invest back into you! This is very strongly related to the previous bullet!
 But what if you can't code? There's plenty of things you can do:
  • document problems you have solved. 
  • report bugs (seriously, bug which has not been reported is a bug that does not exist).
It's not that you have to. But hey, don't complain!

I got inspired to write this post and remember those obvious obviousnesses by the creator of my beloved game, which he wrote during his studies, and gave up development due to his PhD and company he runs. But recently - community funded a new game release! That's the power of community (even if it is not open source one)!

Wednesday, August 22, 2012

Eclipse Platform CBI build just hit Fedora!

I think that by now almost everyone had heard of Tycho - a maven-like approach for building OSGi bundles, and the CBI initiative, which is about making Eclipse builds as easy as possible.

I'm proud to announce that the early adopter of everything, Fedora Rawhide, has just received an Eclipse built with Tycho.

The adoption was rather painful, and there's still a lot of work to be done, but hey, that's the first Eclipse Platform CBI adoption*!

If you dare, test it.
sudo yum --enablerepo="rawhide" install eclipse
All bugs should be reported to RedHat bugzilla, because they will cover packaging issues, not issues with Eclipse itself.
*That I am aware of.

Friday, August 17, 2012

Creating noarch packages with arch-specific dependencies.

If you look at Eclipse package, you will notice that despite being written mostly in Java, it is arch-specific, and therefore it is installed either into /usr/lib/eclipse/ or /usr/lib64/eclipse/. This is because Eclipse contains a few platform specific libraries  - and SWT is the most important of them. Don't get me wrong - SWT is not just the platform specific library, it is also a common JAVA API to create application that look like native on Windows, Linux and MacOs despite using the same code base.

This is very problematic for Fedora packaging, because any Eclipse plugin depending on SWT and not containing any native code is a noarch plugin, and it should be located in a noarch directory (/usr/share/eclipse).

There is no problem during runtime, because when you start an arch-specific eclipse, it can always locate the noarch directory and load all plugins that are there.

But compilation is a problem, because in many cases you just need the SWT API on classpath to compile Eclipse plugin and two most important arch-specifc macros, %{_libdir} and %{_isa}, are banned from being used in noarch packages, and they will not point you to the right SWT installation?


There are three possible solutions:
  • Use Tycho. Tycho is a wonderful Maven extension that makes building of Eclipse plugins easy. 
  • Use eclipse-pdebuild script. This script is located in %{_bin} folder and it automagically discovers where is Eclipse (and SWT) installed.
  • At this point you should go back and reconsider the two previous bullets. I do not recommend any other build system (especially pure ant) for building Eclipse plugins. However, if you have to, you can always use a magic command ${build-classpath swt}, which will locate the swt.jar, instead of hardcoding a path to it.
Warning:
You should be very very careful when you are using any arch-specific things in a noarch package, especially if you don't know the build system by heart. The risk is that some arch specific path may find its way into the code as a constant, search path or something similar, while the package will be installed as a noarch one. As a result, it will work only on one architecture, and debugging will be very difficult.

This blogpost is based on a clarification request regarding the use of %{_libdir} in noarch packages.

Tuesday, August 14, 2012

Debugging Fedora builds with mock

The problem that every build engineer faces sooner or later is: what packages introduces dependency to foo to our build?

The commonly used build tool amongst Fedora packages is fedpkg, which allow for building a package in a mocked configuration, without need to install any unwanted packages.

fedpkg mockbuild creates a mock system, install everything what's needed there, and performs a build. Then deletes the mock location afterwards.

What should you do if your build has unwanted dependencies?

Abandon fedpkg in favour of mock command.

The typicall build command may then look like:
mock --no-cleanup-after --rebuild $SRPM --resultdir=$RESULT -r fedora-rawhide-x86_64 

where --no-clean-after option means that mock will not delete anything after the build. This allows us for calling:
mock -r fedora-rawhide-x86_64 --shell 
, which will open a shell that runs inside our mock system. At this point we can use regular rpm to find what packages require "foo".
rpm -q --whatrequires foo

It's that simple!

Wednesday, July 18, 2012

Pre-indexing documentation with tycho

Older Eclipses had this irritating feature of indexing help on first access. I think that many of you remember that :-). The issue has been fixed in Eclipse 3.1. Wait. Actually only the mechanism to do the indexing at build time was delivered, and all content providers should have used it.

The indexing at build time was usually achieved by two steps:
1. Declare an extension saying where your index will be created:
<extension point="org.eclipse.help.toc">
   <index path="index">
   </index>
</extension>

2. Ensure that the index is actually built via customBuildCallbacks.xml
<project name="Build specific targets and properties" default="noDefault">
    <target name="post.build.jars">
        <help.buildHelpIndex manifest="plugin.xml" destination="."/>
    </target>
</project>
The problem with the customBuildCallbacks is that it allows for to much creativity, which is nearly as desired in build systems as in accountancy :). Thus it should not be used. So, the question is, how to make it work with maven? I wish the answer was simple, but I have no other choice then present it in the most brutal form of xml:
<build>
    <plugins>
        <plugin>
            <groupId>org.eclipse.tycho.extras
            <artifactId>tycho-eclipserun-plugin</artifactId>
            <!-- this is actually present in any 0.14+ version -->
            <version>0.16.0-SNAPSHOT</version>
            <configuration>
                <!-- this is to fix some lucene 3.x compatibility issue -->
		<argLine>-Dhelp.lucene.tokenizer=standard</argLine>
                <!-- actuall indexer call -->
		<appArgLine>-application org.eclipse.ant.core.antRunner -buildfile customBuildCallbacks.xml build.index</appArgLine>
                <dependencies>
                    <!-- list of bundles that we need -->
                    <dependency>
                        <artifactId>org.eclipse.ant.core</artifactId>
                        <type>eclipse-plugin</type>
                    </dependency>
		    <dependency>
                        <artifactId>org.apache.ant</artifactId>
                        <type>eclipse-plugin</type>
                    </dependency>
                    <dependency>
                        <artifactId>org.eclipse.help.base</artifactId>
                        <type>eclipse-plugin</type>
                    </dependency>
                </dependencies>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>eclipse-run</goal>
                    </goals>
                    <phase>compile</phase>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
It should be read as: assemble an eclipse with org.apache.core,org.eclipse.ant.core, org.eclipse.help.base and all required dependencies, and then invoke the indexer. I hope this will be useful to someone during migration to tycho :-). Best regards Sources: Eclipse Help and Tycho wiki.

Tuesday, July 17, 2012

Should Maven be enforced by Eclipse Foundation?

I think I can safely assume that you are all accustomed to the idea of Eclipse Long Term Support and it's side product, which is Common Build Initiative. If I'm wrong, then you'd better start reading those wikis ;-).

The basic idea is to let anyone to build any project (including platform) by invoking a very simple command:
mvn clean install
No need to look for any dependencies. No need to check for any version. Just checkout the source and run this command.

Beautiful idea, but easier said than done.


The biggest obstacle is that some Eclipse projects were created a long time before Maven was, and customized their builds to the every possible limit. Some of them rely on the manual builds invocation, and store pre-compiled code in the repo (not only the native one, but also precompiled classes, which, as you already know from this blog, is a big problem for Fedora packagers).

So it is time to choose: should we preserve the pdebuild compatibility, which means that:
  • CBI is monolithic. You build entire platform or nothing.
  • CBI is not usable for building a single project.
  • Either maven-ant-run plugin is used to bridge PDEBuild and Maven, or the build has to be maintained in two places.
Or maybe we should do it right by restructuring all projects to support maven, which means:
  • Some work for projects that use customBuildCallbacks.
  • Forget about the PDEBuild
  • Easy projects builds.
The discussion is already happening @cbi-dev. Be there!

Monday, May 28, 2012

Mylyn and KDE keys conflict

Mylyn allows to you reveal what is inside of a folder by clicking on it while holding the 'Alt' key. For some reason it worked for me in Fedora as 'Fullscreen off'. Now I know why :-). The reason is this default KDE setting:


One rule that I always follow while creating key shortcuts is that 'Windows'/'Meta' key is used for application management purposes and interaction with the operating system, while all other key combinations (without 'Windows'/'Meta' key) serve for application purposes.

Changing the 'Modifier key' to 'Meta' restores Mylyn functionality and reduces the space of possible conflicts with other applications.

BTW. You can imagine my despair when Unity was introduced to Ubuntu and the 'Meta' key always activated it...

Wednesday, May 23, 2012

Improving Eclipse dropins - What P2 and polish politcs had in common

This week started for me with sounds of well known tune:

Hello console, my old friend,
I've come to work with you again,
Because the reconciler softly creeping,
Left its staff while I was dreaming,
And the bundles that were planted in my dropins
Still remain
Within the space of profilins*.

It's all about the situation where certain operations,  described here in length, could break P2 profile in such a way, that P2 thought that physically removed bundle was still installed. The reproduction steps in the bug are really lengthy, but RPM version is short:

sudo yum install eclipse
sudo yum install eclipse-cdt
eclipse
sudo yum remove eclipse-cdt
sudo yum install eclipse-changelog 
eclipse #what? CDT is still in the profile?

Further investigation has revealed that it was changelog bug (in the first place), because it did not required cdt although it should. During the reconciliation phase, cdt was not yet removed from the profile, so P2 believed it was able to prevent its uninstallation, and during debug, explained its plan with friendly messages:

[p2] Mon May 21 13:07:14 CEST 2012 - [Start Level Event Dispatcher]
[reconciler] [plan] Some units will not be uninstalled:
[p2] Mon May 21 13:07:14 CEST 2012 - [Start Level Event Dispatcher]
[reconciler] [plan] org.eclipse.cdt.ui 5.4.0.201203191015
[p2] Mon May 21 13:07:14 CEST 2012 - [Start Level Event Dispatcher]
[reconciler] [plan] org.eclipse.cdt.core 5.4.0.201203191015

So, my first attempt to fix the issue was rather naive - uninstall removed bundles first, then perform all the reconciliation that has left. Result:  proper behaviour in this test case, but all other dropins tests broken. Not good.

But thanks to @dj_houghton and @prapicault, I was able to came up with something much easier, that not only fixes my unit tests case, but also manages to not break any other reconciler test:

@@ -123,6 +126,15 @@ public class ProfileSynchronizer {
   if (moveResult.getSeverity() == IStatus.ERROR || moveResult.getSeverity() == IStatus.CANCEL)
    return moveResult;
 
+  if (!request.getRemovals().isEmpty()) {
+   Collection requirements = new ArrayList();
+   for (IInstallableUnit unit : request.getRemovals()) {
+    RequiredCapability req = new RequiredCapability(IInstallableUnit.NAMESPACE_IU_ID, unit.getId(), new VersionRange(unit.getVersion(), true, unit.getVersion(), true), null, 0, 0, false, null);
+    requirements.add(req);
+   }
+   request.addExtraRequirements(requirements);
+  }
+
   // now create a plan for the rest of the work and execute it
   IStatus addRemoveResult = performAddRemove(request, context, monitor);
   if (addRemoveResult.getSeverity() == IStatus.ERROR || addRemoveResult.getSeverity() == IStatus.CANCEL)

The patch should be read: for each removed bundle add a negative requirement to the request, so P2 will remove those bundles no matter what. That's so simple. I do not know why it took me 3 days :-(.

And now the explanation of the title. The whole situation with P2 refusing to uninstall deleted bundles reminds me of a polish health care. It's a well know fact that medical help should reach people needing it in 12 minutes because even seriously wounded usually do not pass away in that time. So the norm for an ambulance to reach its target was 12 minutes, but the ratio of timely arrivals was to low (< 85%). So the parliament extended the time to 16 minutes :-).

If all goes well, this fix may hit Fedora as yet another 0 day update.

And a small bonus - have you noticed that (at least on Linux) you are finally informed what will be the action invoked? Check the screenshot below:


For a long time it was almost a matter of luck - you never knew if you hit the button or drop down menu.

* I will buy** a beer to a person that finds a good rhyme here.
** Applicable when we meet in person.

EDIT: Added proper twitter link to LeNettoyeur which turned out to be prapicault.

Saturday, May 19, 2012

Eclipse in Fedora - 0 day update

Eclipse bundled by Fedora will contain additional fixes that are not yet a part of Juno release:

1. Predefined update sites


You may say that's all Eclipses have that, but it is not quite true. You have not just used Eclipse with a shared configuration. However, in Linux, this is a default installation, so many many people reported this problem, and for some Linux distributions it was a blocker that prevented them from including Eclipse. But things are getting better - Fedora, and eclipse-build, now include a fix for that issue.

2. Discover renamed jars in dropins.

Again, this issue does not happen often in regular Eclipse installations. But if for some reason you need to rename a jar in dropins, P2 will refuse to load it. In Fedora, each jar should be packaged only once (f.e. junit), so each discovered duplicated library is replaced with a symlink. Sometimes Eclipse refused to resolve dropins, now we know why :-).

3. Reconciler debugging options are now in a default installation.

As of previously, when anyone reported the problem with Eclipse not being loaded fully, we had to guide him through creating the .options file, putting it in right directory and gathering logs. Now it is easier. To get a hint what is going on with reconciler it is enough to invoke following commands:
$ cd /usr/lib64/eclipse
$ ./eclipse -debug -clean > reconciler.log

4. Eclipse cannot be run as root.

This is the most controversial change, but when you try to launch Eclipse as root, you will get:
# eclipse
CompilerOracle: exclude org/eclipse/core/internal/dtree/DataTreeNode.forwardDeltaWith
CompilerOracle: exclude org/eclipse/jdt/internal/compiler/lookup/ParameterizedMethodBinding.
CompilerOracle: exclude org/eclipse/cdt/internal/core/dom/parser/cpp/semantics/CPPTemplates.instantiateTemplate
CompilerOracle: exclude org/eclipse/cdt/internal/core/pdom/dom/cpp/PDOMCPPLinkage.addBinding
CompilerOracle: exclude org/python/pydev/editor/codecompletion/revisited/PythonPathHelper.isValidSourceFile
CompilerOracle: exclude org/python/pydev/ui/filetypes/FileTypesPreferencesPage.getDottedValidSourceFiles
This application should not be run by the owner of the shared configuration.

and exit code will be 14.

The reason behind that is pretty simple - when you invoke the Eclipse as root, everything what you have in dropins will be written into the master configurations. And later, when you change your installation layout using RPM, things will go ugly, because the master configuration will not reflect what is really installed.

If you really, really need to launch Eclipse as root in Fedora, and you know what consequences are to expect, please remove -preventMasterEclipseLaunch from your /usr/lib64/eclipse.ini.

Note 1. All paths used in this blog post are for 64 bit systems. If you happen to run 32 bit Fedora, please replace every lib64 with lib, so the base Eclipse installation folder will be /usr/lib/eclipse.

Note 2. Fedora 17 (a.k.a. Beefy Miracle) is out pretty soon. You can grab beta here.

Wednesday, May 16, 2012

Making P2 working with RPM - last milestone

Maybe you remember my last post about making P2 and RPM working robustly together. Having the Eclipse Platform M7 packaged in Fedora, and Fedora frozen, I have finally found time to look at that work again.

So, first of all, I was surprised how poor programmer was the past me. Then I have fixed a couple of hardcoded paths, got the solution working again, and even managed to prepare and install my first RPM-wrapped feature into the modified version of Eclipse without breaking the ownership of files.

Ownership of files is a pretty important aspect of Fedora security mechanism - a file installed by RPM into /usr should not be changed because otherwise the verification of system will fail - the system will be believed to be compromised.  P2 changes many files in Eclipse installation if it only is allowed to do so.

After the installation - Eclipse successfully started with my new RPM feature installed. But then I have discovered a hole in my logic. And then my logic appeared to be like a cheese:


I totally forgot about the fact that once I generate a profile from an assembled installation, P2 will start working as usual, and I will not be able to uninstall anything. I'm not sure if you know it, but P2 is an API to control the bundles.info file, which is

only.

P2 does not care if your bundles.info got changed. It will overwrite it.

My attempts to solve that problem made the scope of changes growing very fast. I even created wiki page and put there some things that need to be taken into account.

I'm sorry folks, especially for those haunted by issues similar to:

You will not get a revamped version of P2. I do not think it is possible and I'm officially tossing this idea. You have to be patient - there is no instant cure for those issues. Eclipse Shared Installation works poorly out of the box (see bug 358471), and things get even worse if you take into account additional restrictions that Fedora puts on P2.

In the meantime - what do you think about preventing Eclipse from being run by root by accident?  It is terribly easy to do that on Linux, but then all your dropins stuff goes into master profile and you cannot revert that.

Monday, May 7, 2012

Little things that matter

Installing Eclipse in Fedora is deadly easy. It is enough to type
$ sudo yum groupinstall Eclipse
$ eclipse
to enjoy latest and greatest Eclipse installation, including all major projects (Eclipse itself, EMF, GEF, Mylyn, Git, WTP and others). Now it is time for the small thing - error reporting. Up until now, it was possible to do this using Mylyn:
Eclipse packaged by Fedora is now better integrated with the OS, so it is possible to report errors directly to Red Hat bugzilla.
Delivering feedback couldn't be easier!

Thursday, May 3, 2012

Mylyn hidden gem - finding tasks to read.

We all know that feeling, having a dozen or so of incoming changes in Mylyn tasks, and a few hundreds of them in the list. Scrolling the whole list just to find those changed tasks may be a real pain.


I was wondering how it is possible that Mylyn does not offer filtering incoming changes. But quick look into Mylyn user guide revealed, that it is possible, but you need to switch to 'Scheduled' mode:


And then all dates will be visible - and some scheduling categories appear - and 'Incoming' tasks. That's what I was looking for:


I do not think this is a good idea to have such an important functionality so hidden - I have filled bug 378334: Make focusing on incoming tasks easier.

Tuesday, February 21, 2012

Making P2 working with RPM - milestone I

One of the biggest benefits of RPM is that it allows you to verify your installation and discover any modifications that had been made to your system with (or usually without) your notice.
Unfortunately, such a functionality enforces certain process during package installation - a package should be ready to use just after it is deployed into proper location.
It does not take a lot to understand that P2 will not work with RPM. P2 requires at last the p2 director to be run, but doing so causes instant security alerts for Fedora administrators. Therefore it is a bad idea to launch Eclipse as root. So you cannot actually install anything into your shared installation, unless you use a reconciler approach, ie. you install a small reconciler application, put everything in dropins and hope for the best. The problem is that the reconciler is a legacy solution that works on the "best-can-do" approach, which sometimes is very far from the "best", or even "working".
It is not that P2 or RPM are bad, they are not just designed to cooperate with each other.
This blog post describes my first attempt of getting P2 and RPM working together. The idea is pretty simple:
  • An administrator should not launch Eclipse and modify shared installation directly. Protecting the shared configuration seems to be a rather easy task, so I will deal with it later. As a temporary workaround I have deleted master P2 folder. Ugly, but it works.
  • An administrator deploys plugins and features in the shape, which are ready to be run, together with an .info which is in the bundles.info standard. The administrator is responsible for correctness and completeness of the bundles being installed. This work will be done in Fedora packaging.
  • P2 reads all the .info file, assembles one big master installation on first run, and let the user install anything he wants into his ~/.eclipse folder using P2 UI. Easier said that done - the description is below.




Step I - reading multiple .info files during Eclipse start 
If you have ever wondered how Eclipse knows which plugins should it load, the answer is pretty simple. Eclipse loads just one plugin (with dependencies) - org.eclipse.equinox.simpleconfigurator, which then finds a file named bundles.info with the list of all files. P2 is just an interface between the world and that file [1]. Modifying this file directly or indirectly is not an option for RPM, but making each fedora package delivering its own pregenerated info file is acceptable. So let's make the  simpleconfigurator concatenating multiple .info files.



Step II - spoofing a profile
It is necessary to manipulate the SimpleProfileRegistry to create a fake profile when no profile was found:
public synchronized IProfile getProfile(String id) {
  Profile profile = internalGetProfile(id);
  if (SELF.equals(id)) {
   id = "PlatformProfile";
   self = id;
  }
  if (profile == null) {
   try {
    PlatformAdmin platformAdmin = (PlatformAdmin) ServiceHelper.getService(EngineActivator.getContext(), PlatformAdmin.class.getName());
    IProvisioningAgent agent = (IProvisioningAgent) ServiceHelper.getService(EngineActivator.getContext(), IProvisioningAgent.class.getName());
    IProfileRegistry registry = (IProfileRegistry) agent.getService(IProfileRegistry.class.getName());
    IEngine engine = (IEngine) agent.getService(IEngine.class.getName());
    IMetadataRepositoryManager repoMgr = (IMetadataRepositoryManager) agent.getService(IMetadataRepositoryManager.class.getName());
    IArtifactRepositoryManager artifactRepoMgr = (IArtifactRepositoryManager) agent.getService(IArtifactRepositoryManager.class.getName());

    Collection ius = new Reify().reify(platformAdmin);

    ius.add(Reify.createDefaultBundleConfigurationUnit());
    ius.add(Reify.createUpdateConfiguratorConfigurationUnit());
    ius.add(Reify.createDropinsConfigurationUnit());
    ius.add(Reify.createDefaultFeatureConfigurationUnit(""));

    return spoofUpProfile(id, agent, registry, engine, ius);
   } catch (ProvisionException e) {
    e.printStackTrace();
    return null;
   }
  }

  return profile.snapshot();
 }
The Reify class is a magic class based on the earlier Pascal work - it basically extracts all the P2 data from the running installation. The actual spoofUpProfile method sets properties, creates and executes the plan:

 private synchronized IProfile spoofUpProfile(String id, IProvisioningAgent agent, IProfileRegistry registry, IEngine engine, Collection ius) throws ProvisionException {

  Map prop = new HashMap();

  Location installLocation = (Location) ServiceHelper.getService(EngineActivator.getContext(), Location.class.getName(), Location.INSTALL_FILTER);
  File installFolder = new File(installLocation.getURL().getPath());

  Location configurationLocation = (Location) ServiceHelper.getService(EngineActivator.getContext(), Location.class.getName(), Location.CONFIGURATION_FILTER);
  File configurationFolder = new File(configurationLocation.getURL().getPath());

  // We need to check that the configuration folder is not a file system root. 
  // some of the profiles resources are stored as siblings to the configuration folder.
  // also see bug 230384
  if (configurationFolder.getParentFile() == null)
   throw new IllegalArgumentException("Configuration folder must not be a file system root."); //$NON-NLS-1$

  File launcherConfigFile = new File(configurationFolder, "eclipse.ini.ignored");

  //  prop.put("org.eclipse.update.install.features", "true");
  prop.put(IProfile.PROP_ENVIRONMENTS, "osgi.nl=en_US,osgi.ws=gtk,osgi.arch=x86_64,osgi.os=linux");

  prop.put(IProfile.PROP_INSTALL_FOLDER, installFolder.getAbsolutePath());
  prop.put(IProfile.PROP_SHARED_CACHE, installFolder.getAbsolutePath());
  prop.put(IProfile.PROP_ROAMING, Boolean.FALSE.toString());
  prop.put(IProfile.PROP_CONFIGURATION_FOLDER, configurationFolder.getAbsolutePath());
  prop.put(IProfile.PROP_CACHE, configurationFolder.getParentFile().getAbsolutePath());
  prop.put(IProfile.PROP_LAUNCHER_CONFIGURATION, launcherConfigFile.getAbsolutePath());
  prop.put("org.eclipse.update.install.features", "true");

  IProfile profile = registry.addProfile(id, prop);
  IProvisioningPlan plan = helper.getPlan(profile, ius);

  ((Profile) profile).setChanged(false);
  IPhaseSet phaseSet = PhaseSetFactory.createDefaultPhaseSetExcluding(new String[] {PhaseSetFactory.PHASE_CHECK_TRUST, PhaseSetFactory.PHASE_COLLECT, PhaseSetFactory.PHASE_CONFIGURE, PhaseSetFactory.PHASE_UNCONFIGURE, PhaseSetFactory.PHASE_UNINSTALL});
  IStatus status = engine.perform(plan, phaseSet, null);

  if (!status.isOK())
   return null;

  return profile;
 }

Creating a plan is actually kind of hack. The SimpleProfileRegistry is in the engine, and it cannot reference IPlanner service, because it would cause cyclic dependencies. So I have created a helper field, where the Director sets the proper service:
 public void start(BundleContext ctx) throws Exception {
  context = ctx;
  SimpleProfileRegistry.helper = new IPlannerHelper() {

   public IProvisioningPlan getPlan(IProfile profile, Collection ius) {
    ProfileChangeRequest request = new ProfileChangeRequest(profile);
    request.addAll(ius);
    IProvisioningAgent provisioningAgent = profile.getProvisioningAgent();
    IPlanner planner = (IPlanner) provisioningAgent.getService(IPlanner.class.getName());
    return planner.getProvisioningPlan(request, new ProvisioningContext(profile.getProvisioningAgent()), null);
   }
  };
 }
One more thing to do is to start both required plugins when dropins are started to ensure that the helper is registered. This is not perfect solution, but once I will find out how to do it better, I will correct it.
  ServiceHelper.getService(getContext(), IProvisioningAgent.SERVICE_NAME);
  ServiceHelper.getService(getContext(), IPlanner.class);

The P2 data generated from the running platform are partially inaccurate. First of all, there is  no P2 repositories (and we do not actually move any jars), and the information about which plugins should be run is lost. I have temporary workarounded that by modyfing SimpleConfiguratorManipulator and hardcoding certain plugin names.

Summing up, there is still a lot of work to be done, but the basic scenario, where an administrator assembles shared installation, and a regular user is able to add custom plugins using P2, and RPM is still happy about it, seems to work.

A complete patch is accessible here. Feel free to suggest updates, comment or criticize, and stay tuned for next patch versions.

Big THANK YOU to @irbull for his help.

BTW. I have no idea if this patch will ever get into P2 repository. It will become a part of Fedora Eclipse build process most likely.


Wednesday, February 8, 2012

Fedora Eclipse Build: far Orbit

Today I will dwell on the issue that hit me when I tried to build a recent Eclipse - something pre-M5. Previous builds of Eclipse using eclipse-build were done for M1, so it is easy to imagine what could go wrong.

It started very innocently - unsatisfied package javax.servlet_2.6 in a number of plugins. This clearly indicated a problem with the build - for some reason the package javax.servlet was missing or not built properly.

And here comes the first surprise - javax.servlet package is not Eclipse package, it is a 3rd party library. A short glimpse at the dependecies.properties (a file that maps non-Eclipse library to Fedora libraries) and here it its mapping:

javax.servlet_3.0.0.v201112011016.jar=/usr/share/java/tomcat/tomcat-servlet-3.0-api.jar

And there... there is package javax.servlet_3.0 exported!

That's interesting. Quick search for 2.6 version... There is no such version of javax.servlet specification. Something is wrong. How is it possible that half of the Eclipse plugins uses a version of the javax.servlet that does not exists?

The answer was finally found here:

The Servlet 3 specification is a breaking change for implementers, but is
binary compatible for client applications.  Following the version guidelines
the javax.servlet packages should move from version 2.5 -> 2.6
The workaround for that issue was not simple. I could not patch Eclipse to use 3.0 (too many patches), nor downgrade version in tomcat-servlet-3.0-api - because of too many dependencies.

Thankfully, OSGi allows for exporting the same package more than once:

Export-Package: javax.servlet;version="3.0", javax.servlet;version="2.6",

One patch for Tomcat, and one problem in Eclipse Fedora Build solved :-).

The problem that I am trying to highlight is that Eclipse Foundation maintains its own version of libraries in Orbit, and it may surprise users. I do not want to say that renaming 3.0 to 2.6 was unjustified (because it was), but on the other hand, does Eclipse have to maintain its own copies of libraries, which are unusable for the rest of the world?

Maybe it is a good time to start discussion about Orbit removal, and a true, transparent connection to other libraries, where each local modification should be sent to the upstream?





Monday, January 30, 2012

Life-saving Eclipse functionality

As you know, I am working on Eclipse Fedora Build. Eclipse 3.8 ("almost M5") will get soon into Fedora Rawhide repository very soon.

Testing the build involved running new Eclipse and a number of of older packages, which included egit, jgit and others. My  local git repository got quickly corrupted (not sure yet why), so I created a patch, copied it into a safe place overwriting the previous one, deleted the repository, and cloned it again.

When I tried to apply the patch, it turned out that it was empty, and last couple days of my work was gone (because I happily overwrote the old patch).

Fortunately, Eclipse has a feature that deals with that kind of situations:


and then, a nice version selector appears:





My work was wonderfully resurrected!

Thank you, Eclipse!



Thursday, January 26, 2012

Fedora Eclipse Build: Introduction

It has been almost a month, since I stared my work for Red Hat.

I am taking care of Eclipse Build, which is even more complicated than regular Eclipse build (yes, Kim, that *is* possible).

The first and the most important assumption of Linux Eclipse Build is: no binaries allowed.

Some people call that kind of builds liberating. That is the last word that I'd use now ;-).

I will describe nuances that I have met during preparing Eclipse Juno for Fedora 17 in the following posts, but right now I'd like to mention just the first issue I have met:

If you want to build Eclipse, you need a builder, namely, a PDE Build. It looks like a chicken an egg problem, and the only solution is to build the builder using ant and OpenJDK. There is only one problem with that (OK, one serious problem and a lot of smaller ones that I will skip for now): Eclipse Java Compiler is not consistent with OpenJDK javac, or, stating it more clearly, JDT code cannot be compiled with javac.

That's not a big technical problem, as patching the source code before the build and making it javac compatible (and JDT incompatible) is not a very difficult thing, but it is open a rather serious question.

How does it come to the situation, where you cannot build Eclipse from scratch, but you have to use binaries to build Eclipse? Isn't it a bit against open-source philosophy? Today need we just Eclipse Compiler, which cannot be built (unless customized) from source. Tomorrow may we depend on something far less open.

Would not it make sense to put a requirement on Eclipse that some core plugins must compile with OpenJDK? I do not expect here mirroring all OpenJDK bugs in Eclipse compiler, but at least allowing for true open source build of it...

Here is a bug for those of you who are interested in the cause of the javac problem when building JDT.

In the next post: do Orbit users get what they expect?

EDIT: here is the link to the patch that makes JDT compile with OpenJDK.