Squeak311Proposal


The technical goal of Squeak 3.11 development is to provide a tool chain to enable many contributors to add code to the base without bottlenecks. The validation of this goal being the production of a new 3.11 image built using this tool chain.

 


 

Current Practice

 

For a change to appear in a release image, the following things must happen, in rough order:

  1. Discover: A person acquainted with the release process learns of the existence of the change
  2. Load: Someone adds the change to an early version of the release image
  3. Test: Someone unit tests the resultant image
  4. Feedback: Sufficient time passes to gain confidence in the change
  5. Package: Somebody packages up the change so that beta testers can use and test the new image
  6. Release: Somebody keeps or discards the change from the release, based on feedback

 

Currently, all of these steps involve the time of a release team member, and thus do not scale well. Also, the last one, discarding a change, is very hard in the current process (update streams)

 

Process Improvements

 

We are streamlining the process of incorporating changes into the release. We are making two policy changes, and using four new tools.

 

 

Stable/Unstable Policy

The stable version will include changes that have undergone a period of testing and found stable.

 

The unstable version will include everything in the stable version, as well as some less tested changes.

 

Open Release Policy

We don't want the scarcity of release team members to be a bottleneck in getting changes into the unstable version and exposing them to testing from those who try the unstable version. The release team will still have to bless changes to put them in the stable version.

 

Email Notification

We have added to SqueakSource an emailing capability. When new versions of packages are uploaded, SqueakSource will send out an email notification to interested parties. Using this feature, we created a mailing list that receives commit notifications for all projects that are part of 3.11. The goal is to make code reviewing a bit more involuntary. So far, we have fixed about 5 bugs as a direct result of someone doing an involuntary code review just because it was in their inbox.

 

Sake/Tasks

A place to put image building scripts, as tasks. Tasks define dependents and pre-requisites, and are not always required to be run in a specific order. Sake/Tasks is intended to continue the role of this installer wiki, beyond the bootstrap stage (i.e. once LPF Packages and Tasks are loaded). Scripts are managed in the image, rather than on the web, and the scripts are under version control with Monticello.

 

Sake/Packages

A set of tasks, again defining dependents and pre-requisites, for each loadable package. The entire definition set managed with Monticello. For more information see Packages.

 

Sake/Bob, The Build Server

 

A further set of tasks, under the control of a periodic scheduler.

The build server will have several distinct jobs:

  1. Building the base image
  2. Building downloadable, platform-specific packages of squeak
  3. Building Packages
  4. Testing Image/Package distributions

 

Build Server Task: building the base image

 

People will add tasks to the script that generates the image from the previous version. Mostly, these will be a list of mantis bug fixes to be applied. Currently, this list is on this wiki as one-line installer scripts (such as "Installer fixBug: 6721"); it is being transitioned to Sake/Tasks, where it will be version controlled.

 

When the build control script changes, The build server will run it and build new versions of the following:

 

  1. The resultant image/changes pair, standing alone
  2. (Maybe) the original image, some resources, and a script suitable for running offline. On running, this script will reproduce the image, and duplicate whatever issues the build server found

 

This will be done for both the stable and unstable scripts, and for derivative images thereof.

 

Build server task: Packaging squeak distributions

In addition to building the basic image, the build server will be responsible for packaging up the image, VM, and associated files into platform-specific formats (Windows installer, Mac .dmg, debian .deb package, .rpm, etc.)

 

Build server task: Building packages

The basic image includes a number of packages pre-installed. The builder will update these when it builds a new image. This serves two purposes:

  1. those using other versions of squeak can easily install the 3.11 version of a specific package
  2. those with an older version of 3.11 can get the latest version by updating the base packages

 

Build server task: Testing

The build server will take a previously built image, install some packages (none, the set selected for the "full" image, or a random selection), and run all tests in the image. It will produce for download the image it tested against and a script to debug the tests that failed

 

Conclusion: How does this improve the change harvesting process?

 

Discover

The open release policy will take away a bit of the burden on the release team to find changes, listen to suggestions from the community, and remember them. Under this policy, the process of discovering changes can be parallelized across the entire community.

 

We could also perhaps automate this somewhat. Installer can query mantis, so we may use this feature to write a task to collate potential fixes.

 

Loading

Most changes come from mantis. A user can add a short installer script to any bug report on mantis.

Release tasks can then indicate dependencies upon particular fixes as in the following example:

 

TasksReleaseAfterSqueak310 >> taskCleanDeprecated
 	^ self define: [ :task |
 			
 		task dependsOn: { '6866 Refs Stop CurrentProjectRefactoring Removal' }.
 				
 		task action: [ 
 			Preferences noviceModeSettingChanged. "remove any old morphic key bindings"
 			Installer mc unload: '39Deprecated'.			
 		]
 	].

 

Testing

Unit testing is done automatically by the build server; no need to do it manually anymore

 

Packaging

Packaging is done automatically by the build server; no need to do it manually anymore

 

Feedback

There are two kinds of feedback a release team wants:

  1. People trying the latest code, to make sure it works
  2. People examining the latest code, to make sure it looks ok

 

The way to get the first type of feedback is to make sure it is really easy for people to try out the latest changes. The build server gives beta testers two easy options:

  1. Download and use the image built last night
  2. In your existing image, update all the packages to those build last night

 

To get the second type of feedback, we must make it really easy to see the code added to the image. This is why we have set up a [http://lists.squeakfoundation.org/packages

mailing list] that shows the latest changes made to the packages and code in the latest release. This way, interested people can do semi-involuntary code reviews just by reading some of the emails that go to that list. Easier reviews = more reviews = fewer bugs.

 

Release

Under the previous policy, everything in a release is unstable, up until the moment that someone sends an email to squeak-dev saying "Squeak version X is now released and is available for download from the usual locations", at which point all changes became "stable".

 

Under the new policy, this is much more evolutionary; anyone is welcome to use the latest changes, and can be reasonably confident of their stability. Marking a new version is still a stronger declaration of stability, but we have added another level of granularity.

 

Also, since the image is assembled from a script, it is now possible, for the first time in squeak's history, to take a buggy change out of the unstable version of squeak. This could be done to an extent with the update stream, but it would create a situation where two squeak versions with the same update number had different changes, depending on whether they were generated before or after the deletion. Usually, one had to write another update that undid the previous update. Removing an update was flat out impossible in the 3.10 system; removing a change meant every change since then had to be re-made. This had to be done once, and caused the release team a great amount of pain.

 

This is, however, easy in this system. The night the change is removed, the build server will build a new version, with a new update number, which does not include the change. Users may or may not be able to upgrade an image to the new version, but that is acceptable.

 

Squeak 3.11 Deliverable

 

Since the primary goal of 3.11 is the proving of the process. The base image itself will have no significant new features. Some tidying is planned together with the usual barrage of fixes, each of which will be fully documented in the image. In addition the inclusion of LevelPlayingField (and the ability to keep it updated) will provide the community with some significant benefits. Namely, speed, atomic loading, and some standard package types.

 

The Squeak 3.11 release will be released as '-basic' image. Contents of this image are as follows:

 

  1. LPF as standard - Installer, Launcher, PackageInfo, Monticello 1.5 & 1.6, MonticelloConfigurations
  2. Sake/Packages - Package definitions for everything else
  3. Sake/Tasks - Documentation of changes and bug fixes since 3.10 (optional)
  4. Clean Image - Remove some things that can be unloaded and reloaded easily
  5. Remove Traits Task - For those who prefer/need a flattened image
  6. Fixes from Mantis - Currently 40+ in the list
  7. Underscores in selectors - If possible in the time frame

 

Derivative images

 

  1. -test - basic with tests loaded
  2. -minimal - basic with all removable packages removed, including LPF.
  3. -kernel - Pavel's shrink script applied
  4. -full - (fun squeak? Edgar?)
  5. -dev - Squeak-dev (Damien is moving over to use Sake/Packages to build)
  6. -web - Squeak-web
  7. -dev-beta - Squeak-dev beta
  8. -web-beta - Squeak-web beta
  9. -seaside - The seaside one-click-experience

 

Subsequent Releases

 

Having proved the tool chain subsequent releases will be time boxed. Packaging 4-6 months worth of fixes at a time, halving current release cycles.

 

Squeak 3.12

 

  1. Continue to move modules into loadable packages
  2. Look to Pharo for some refactorings
  3. Thread safe transcript as a default Logging client.
  4. Adopt Logging conventions into the kernel, replacing Transcript show:
  5. Lots of other ideas...