Discussion:
Building Subclipse with Hudson/Buckminster
Stephen Elsemore
2010-05-27 20:01:57 UTC
Permalink
As Mark Phippard suggested (http://subclipse.tigris.org/issues/show_bug.cgi?id=1144), it would be very nice if Subclipse builds could be automated using Buckminster on Hudson. I will gladly work on such a project but, based on my initial foray into using Buckminster on Hudson, I can see that I will need some help figuring out how to make it all work.

@Thomas Hallgren
Since you are the lead in the Buckminster project, and also active in the Subclipse project, you are probably the ideal source for such help, if you have the time and inclination. Perhaps I can describe to you what I’ve done so far and where my problem areas seem to be, and you can use that as context to point me in the right direction?

I started out knowing nothing about Buckminster and very little about Hudson. My primary resources so far have been:

Buckminster book
http://www.eclipse.org/downloads/download.php?file=/tools/buckminster/doc/BuckyBook.pdf
(mainly chapters 13 “Building a p2 Update Site” and 15 “Hello XML World”)

Ralf Ebert tutorial “Headless Eclipse RCP builds with Buckminster and Hudson”
http://www.ralfebert.de/blog/eclipsercp/rcp_builds

Eclipse forum discussion “Mini-Tutorial: Fetching, archiving and re-using a targt platform with Buckminster/Hudson”
http://www.eclipse.org/forums/index.php?t=tree&th=163429&#page_top

As background, you should also know that everything that I’ve tried so far has been on Mac OSX. Several of the early bumps that I encountered were related to that. For example, at first nothing at all was working with headless Buckminster. I finally figured out that my problem is described here:

http://www.eclipse.org/forums/index.php?t=tree&goto=517230&

When I configured the Hudson Buckminster plugin, I’d selected the option to “Install automatically”, which causes the headless Buckminster to be downloaded and installed the first time it is needed. To fix the problem, I had to delete the version that had been automatically downloaded, then load it manually using the parameters that were suggested in the above link. Then I changed the Hudson Buckminster configuration to use the version that I manually loaded, and that fixed the problem.

I suppose it’s possible that some of my remaining issues could also be OS-related.

Anyway, here’s what I’ve done so far:

First I installed Buckminster plugins into Eclipse and Hudson. Then, rather than diving right in and trying to tackle building Subclipse, I decided to try it out with a simple test plugin that does nothing more than contribute a bare bones view. To do this, I basically followed Ralf’s tutorial. The tutorial builds and packages an RCP application, rather than an Eclipse plugin, so I made the changes that I thought were necessary to make it work with a plugin.

Here’s what I have for Eclipse projects, all checked out from an SVN repository:

org.elsemore.steve.MyApplication (Plug-in Project with a view)
org.elsemore.steve.MyApplication.feature (Feature Project that includes org.elsemore.steve.MyApplication plug-in and has org.eclipse.core.runtime and org.eclipse.ui as dependencies)
org.elsemore.steve.MyApplication.site (Feature Project that includes org.elsemore.steve.MyApplication.feature)

org.elsemore.steve.MyApplication.site is not to be deployed. It just hosts the files for the Buckminster build (.cquery, .rmap, .target).

I created a new free-style job in Hudson, with the following attributes:

Project name: MyApplication

I at first named it “My Application” (with space). Big mistake! I found that importtargetdefinition does not work correctly (but doesn’t give any indication that it did not work correctly) when there are spaces in the path to the target platform file. It took me awhile to figure that one out.

Source Code Management:

Selected “Subversion” and added 3 Repository URL’s:

<my repository root>/trunk/org.elsemore.steve.MyApplication
<my repository root>/trunk/org.elsemore.steve.MyApplication.feature
<my repository root>/trunk/org.elsemore.steve.MyApplication.site

I realized at this point that I had several options:

1. Let Hudson populate the workspace by checking out my projects from Subversion.
2. Use an RMAP that uses an SVN provider in the searchPath and let the Buckminster import materialize the workspace (as is done in the Hello XML World example).

I went with option 1 because it matches the RCP tutorial approach, and because it would seem to minimize the chance of conflicts when I eventually get around to trying to build Subclipse itself.

Run Buckminster Build Step:

Commands:

importtargetdefinition -A '${WORKSPACE}org.elsemore.steve.MyApplication.site/platform.target'
import '${WORKSPACE}org.elsemore.steve.MyApplication.site/site.cquery'
build
perform -D target.os=* -D target.ws=* -D target.arch=* org.elsemore.steve.MyApplication.site#site.p2

Here is how I created platform.target file:

Eclipse->Preferences->Plug-in Development->Target Platform
Add...
Select “Nothing: Start with an empty target definition”, Next
From http://download.eclipse.org/eclipse/updates/3.5 site, select Eclipse SDK
Uncheck “Include required software” and check “Include all environments”, Finish
Select newly created Target Platform, click Move...
Select org.elsemore.steve.MyApplication.site folder and enter “platform.target” for file name. Finish.
Apply.

(I am very unsure about this step.)

Here are the contents of site.cquery:

<?xml version="1.0" encoding="UTF-8"?>
<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="site.rmap">
<cq:rootRequest name="org.elsemore.steve.MyApplication.site" componentType="eclipse.feature"/>
</cq:componentQuery>

And here are the contents of site.rmap:

<?xml version="1.0" encoding="UTF-8"?>
<rmap
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.eclipse.org/buckminster/RMap-1.0"
xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0"
xmlns:mp="http://www.eclipse.org/buckminster/MavenProvider-1.0"
xmlns:pp="http://www.eclipse.org/buckminster/PDEMapProvider-1.0">

<searchPath name="resources">
<provider readerType="local" componentTypes="osgi.bundle,eclipse.feature"
mutable="true" source="true">
<uri format="file:///{0}/{1}/">
<bc:propertyRef key="workspace.root" />
<bc:propertyRef key="buckminster.component" />
</uri>
</provider>
</searchPath>

<locator searchPathRef="resources"/>
<redirect href="http://www.eclipse.org/buckminster/samples/rmaps/dogfood.rmap" pattern=".*"/>
</rmap>

Results of running the build:

1. Resources are checked out into workspace as expected.

2. importtargetdefinition runs for a long time, with lots of “Retrying request” messages, but ultimately completes and seems to have properly imported the target platform. I can see that all the required jar files are now in:

<workspace>/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool/plugins

Ultimately, I would like to avoid refetching the target platform for every build. In an effort to achieve this, I experimented with using the approach described here:

http://www.eclipse.org/forums/index.php?t=tree&th=163429&

So far, I haven’t been able to make that work, which I’ll explain later.

3. build fails with this error:

Error: file /Users/stephenelsemore/.hudson/jobs/MyApplication/workspace/org.elsemore.steve.MyApplication: The project was not built since its build path is incomplete. Cannot find the class file for org.eclipse.swt.widgets.Composite. Fix the build path then try building this project
Error: file /Users/stephenelsemore/.hudson/jobs/MyApplication/workspace/org.elsemore.steve.MyApplication/src/org/elsemore/steve/myapplication/views/SteveView.java, line 1: The type org.eclipse.swt.widgets.Composite cannot be resolved. It is indirectly referenced from required .class files

At first I thought that maybe the build path simply did not include the target platform stuff that was created by step 2. But I don’t believe it’s that simple. If I remove the SteveView class from my project (eliminating the SWT dependency), the build will work fine (even though the Activator class in my project still requires org.eclipse.jface.resource, org.eclipse.ui.plugin, org.osgi.framework). So I’m not sure why the error. I can see that the <workspace>/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool/plugins folder contains SWT jar files for every platform imaginable (which is perhaps overkill), but Buckminster seems to not find or ignore the one that is needed? Or more likely I am missing something in the configuration? I’ve experimented a lot with this, but am so far still stuck.

4. (if I remove SteveView class from my project to avoid build error) perform -D target.os=* -D target.ws=* -D target.arch=* org.elsemore.steve.MyApplication.site#site.p2:

Works as expected. Update site is built here:

<workspace>/buckminster.output/org.elsemore.steve.MyApplication.site_1.0.0-eclipse.feature/site.p2

So, if I could get around the build error in step 3, I’d be making some progress. Any help would be appreciated!

Regarding my failed attempt to archive and re-use a target platform (step 2, above), this is what I tried:

Create a job exactly as described above, but also check “Archive and publish an Eclipse Platform”.

Path to Target Platform: .metadata/.plugins/org.eclipse.pde.core/.bundle_pool
Name: MyTargetPlatform

Run this job.

Create a new job similar to the original job, except that the Run Buckminster build step specifies:

Target Platform: MyTargetPlatform (which now shows up in the dropdown)

In the new job, I removed the importtargetdefinition command, assuming that this is no longer needed now that the archived target platform is being used (I have a feeling I’m starting to go wrong here).

When I run this new job, it fails with this error on the import (which is perhaps not related to the target platform?):

INFO: import '/Users/stephenelsemore/.hudson/jobs/MyApplicationCannedPlatform/workspace/org.elsemore.steve.MyApplication.site/site.cquery'
ERROR [0000] : No suitable provider for component org.eclipse.ui:osgi.bundle was found in resourceMap file:/Users/stephenelsemore/.hudson/jobs/MyApplicationCannedPlatform/workspace/org.elsemore.steve.MyApplication.site/site.rmap
ERROR [0000] : No suitable provider for component org.eclipse.ui:osgi.bundle was found in searchPath resources
ERROR [0000] : Resolution attempt ended with exception: Provider local(file:////Users/stephenelsemore/.hudson/jobs/MyApplicationCannedPlatform/workspace/org.eclipse.ui/): Missing CSpec source required by component type osgi.bundle
ERROR Provider local(file:////Users/stephenelsemore/.hudson/jobs/MyApplicationCannedPlatform/workspace/org.eclipse.ui/): Missing CSpec source required by component type osgi.bundle

If you are still reading, THANKS! I’d be very thankful for any information you can give me to help me resolve my problems. Once I get past these issues, I’m sure things will get even more complicated when it comes to applying such a solution to Subclipse builds (so far there’s very little demand for MyApplication), so any ideas or suggestions that you might have regarding that would be fantastic.

Regards,
Steve

------------------------------------------------------
http://subclipse.tigris.org/ds/viewMessage.do?dsForumId=1043&dsMessageId=2614181

To unsubscribe from this discussion, e-mail: [dev-***@subclipse.tigris.org].
Thomas Hallgren
2010-05-28 06:33:49 UTC
Permalink
Hi Stephen,
Looks like you've done a real deep dive in Buckminster land.

I would recommend a simpler approach where you let Buckminster resolve and populate both your target platform and your workspace. Keeping them apart physically is a good thing of course, but you can do that and still have the benefits of one single resolution.

The approach outlined here assumes that you are using Buckminster 3.6 and a 3.6 RC2 IDE. The IDE is of course optional but it's great for the initial steps and to get things up and running smoothly.

1. Create feature that describes your intended update site. It's just a normal Eclipse feature. Normally, this feature is not included in the resulting site. It just acts as a description.

2. Create a CQUERY/RMAP combination that works. In the RMAP, try using "svn" providers for the source (with source="true") and p2 providers for all bundles that you wish to get as binaries. The CQUERY must point to the feature created in #1.

Do not use our dogfood RMAP. It's not maintained.

3. Use your IDE at first. Create a target platform with one single "directory" container and set that target platform as the active one.

4. Run the CQUERY "Resolve and Materialize". The first time you do this, you are likely to run into problems with components that cannot be resolved (like the one you describe). Once it resolves correctly, you will see that Buckminster populates both your target platform and your workspace.

5. Verify that everything builds in your IDE. Rigth click on the feature created in #1 and select "Buckminster" -> "Perform" -> "site.p2". This should build the update site for you.

Once the above works, the Buckminster part of your build is complete. You should now be able to repeat the same thing headlessly.

1. Import the CQUERY using buckminster import <url>
2. Issue a buckminster build
3. Issue a buckminster perform <feature name>#site.p2

The rest is all about Hudson.

Please note that if Hudson is used when populating your workspace, you will miss two things:
1. The target platform is not resolved and materialized by Buckminster.
2. SVN meta-data is not available since the workspace projects are not "shared" using the team provider. This means that qualifier replacement using revision number or timestamp doesn't work.

If you want a simple example of a complete build of a site, you can take a look at the last one that I did for xtext. It's in the form of two projects. One 'releng' that is a pure release engineering project. It contains the cspec, the rmap, and needed properties:

http://dev.eclipse.org/svnroot/tools/org.eclipse.buckminster/trunk/testbench/org.eclipse.xtext.releng.buckminster

and then there's an example of a feature that describes a site.

http://dev.eclipse.org/svnroot/tools/org.eclipse.buckminster/trunk/testbench/org.eclipse.xtext.site

Don't hesitate to ask if you have more questions. I'm more then happy to help with this. You can also try our mailing lists and newsgroups. You'll get quick answers there, both regarding Buckminster itself and the Buckminster Hudson plugin.

Kind Regards,
Thomas Hallgren

------------------------------------------------------
http://subclipse.tigris.org/ds/viewMessage.do?dsForumId=1043&dsMessageId=2614362

To unsubscribe from this discussion, e-mail: [dev-***@subclipse.tigris.org].
Mark Phippard
2010-05-28 12:04:55 UTC
Permalink
A quick question I have ...

When I build Subclipse for release, I do it using a copy of Eclipse
3.2 that I leave lying around. Is this still possible using
Buckminster? Obviously the build process could run with something
newer, but the classes it is resolving against when it compiles need
to be Eclipse 3.2 for proper backwards compat. at least that is what
I always thought.

Mark
Post by Thomas Hallgren
Hi Stephen,
Looks like you've done a real deep dive in Buckminster land.
I would recommend a simpler approach where you let Buckminster resolve and populate both your target platform and your workspace. Keeping them apart physically is a good thing of course, but you can do that and still have the benefits of one single resolution.
The approach outlined here assumes that you are using Buckminster 3.6 and a 3.6 RC2 IDE. The IDE is of course optional but it's great for the initial steps and to get things up and running smoothly.
1. Create feature that describes your intended update site. It's just a normal Eclipse feature. Normally, this feature is not included in the resulting site. It just acts as a description.
2. Create a CQUERY/RMAP combination that works. In the RMAP, try using "svn" providers for the source (with source="true") and p2 providers for all bundles that you wish to get as binaries. The CQUERY must point to the feature created in #1.
Do not use our dogfood RMAP. It's not maintained.
3. Use your IDE at first. Create a target platform with one single "directory" container and set that target platform as the active one.
4. Run the CQUERY "Resolve and Materialize". The first time you do this, you are likely to run into problems with components that cannot be resolved (like the one you describe). Once it resolves correctly, you will see that Buckminster populates both your target platform and your workspace.
5. Verify that everything builds in your IDE. Rigth click on the feature created in #1 and select "Buckminster" -> "Perform" -> "site.p2". This should build the update site for you.
Once the above works, the Buckminster part of your build is complete. You should now be able to repeat the same thing headlessly.
1. Import the CQUERY using buckminster import <url>
2. Issue a buckminster build
3. Issue a buckminster perform <feature name>#site.p2
The rest is all about Hudson.
1. The target platform is not resolved and materialized by Buckminster.
2. SVN meta-data is not available since the workspace projects are not "shared" using the team provider. This means that qualifier replacement using revision number or timestamp doesn't work.
http://dev.eclipse.org/svnroot/tools/org.eclipse.buckminster/trunk/testbench/org.eclipse.xtext.releng.buckminster
and then there's an example of a feature that describes a site.
http://dev.eclipse.org/svnroot/tools/org.eclipse.buckminster/trunk/testbench/org.eclipse.xtext.site
Don't hesitate to ask if you have more questions. I'm more then happy to help with this. You can also try our mailing lists and newsgroups. You'll get quick answers there, both regarding Buckminster itself and the Buckminster Hudson plugin.
Kind Regards,
Thomas Hallgren
------------------------------------------------------
http://subclipse.tigris.org/ds/viewMessage.do?dsForumId=1043&dsMessageId=2614362
--
Thanks

Mark Phippard
http://markphip.blogspot.com/

------------------------------------------------------
http://subclipse.tigris.org/ds/viewMessage.do?dsForumId=1043&dsMessageId=2614453

To unsubscribe from this discussion, e-mail: [dev-***@subclipse.tigris.org].
Stephen Elsemore
2010-05-28 13:56:17 UTC
Permalink
Thanks, Thomas, this is just what I needed to get going!

Regarding Mark's question, is this something that we would resolve by
pointing the p2 providers in the RMAP to the appropriate Eclipse download
sites?

Steve
Post by Mark Phippard
A quick question I have ...
When I build Subclipse for release, I do it using a copy of Eclipse
3.2 that I leave lying around. Is this still possible using
Buckminster? Obviously the build process could run with something
newer, but the classes it is resolving against when it compiles need
to be Eclipse 3.2 for proper backwards compat. at least that is what
I always thought.
Mark
Post by Thomas Hallgren
Hi Stephen,
Looks like you've done a real deep dive in Buckminster land.
I would recommend a simpler approach where you let Buckminster resolve and
populate both your target platform and your workspace. Keeping them apart
physically is a good thing of course, but you can do that and still have the
benefits of one single resolution.
The approach outlined here assumes that you are using Buckminster 3.6 and a
3.6 RC2 IDE. The IDE is of course optional but it's great for the initial
steps and to get things up and running smoothly.
1. Create feature that describes your intended update site. It's just a
normal Eclipse feature. Normally, this feature is not included in the
resulting site. It just acts as a description.
2. Create a CQUERY/RMAP combination that works. In the RMAP, try using "svn"
providers for the source (with source="true") and p2 providers for all
bundles that you wish to get as binaries. The CQUERY must point to the
feature created in #1.
Do not use our dogfood RMAP. It's not maintained.
3. Use your IDE at first. Create a target platform with one single
"directory" container and set that target platform as the active one.
4. Run the CQUERY "Resolve and Materialize". The first time you do this, you
are likely to run into problems with components that cannot be resolved (like
the one you describe). Once it resolves correctly, you will see that
Buckminster populates both your target platform and your workspace.
5. Verify that everything builds in your IDE. Rigth click on the feature
created in #1 and select "Buckminster" -> "Perform" -> "site.p2". This should
build the update site for you.
Once the above works, the Buckminster part of your build is complete. You
should now be able to repeat the same thing headlessly.
1. Import the CQUERY using buckminster import <url>
2. Issue a buckminster build
3. Issue a buckminster perform <feature name>#site.p2
The rest is all about Hudson.
Please note that if Hudson is used when populating your workspace, you will
1. The target platform is not resolved and materialized by Buckminster.
2. SVN meta-data is not available since the workspace projects are not
"shared" using the team provider. This means that qualifier replacement using
revision number or timestamp doesn't work.
If you want a simple example of a complete build of a site, you can take a
look at the last one that I did for xtext. It's in the form of two projects.
One 'releng' that is a pure release engineering project. It contains the
http://dev.eclipse.org/svnroot/tools/org.eclipse.buckminster/trunk/testbench/
org.eclipse.xtext.releng.buckminster
and then there's an example of a feature that describes a site.
http://dev.eclipse.org/svnroot/tools/org.eclipse.buckminster/trunk/testbench/
org.eclipse.xtext.site
Don't hesitate to ask if you have more questions. I'm more then happy to help
with this. You can also try our mailing lists and newsgroups. You'll get
quick answers there, both regarding Buckminster itself and the Buckminster
Hudson plugin.
Kind Regards,
Thomas Hallgren
------------------------------------------------------
http://subclipse.tigris.org/ds/viewMessage.do?dsForumId=1043&dsMessageId=2614
362
------------------------------------------------------
http://subclipse.tigris.org/ds/viewMessage.do?dsForumId=1043&dsMessageId=2614485

To unsubscribe from this discussion, e-mail: [dev-***@subclipse.tigris.org].
Loading...