I’ve been participating remotely in the Fedora Activity Day (FAD)  in Raleigh, NC.  The sound quality over Fedora Talk is extremely good calling in with a regular phone (with the exception of other people forgetting to mute), though it is harder to hear some of people not so close to the phone.  The combination of gobby, IRC and the live audio is almost as good as being their in person.  So big thanks all the people that made those things happen!

I think I understand the purpose of this FAD better now and it has clarified for me the problems that we have with rawhide and the test releases:

  • We need to freeze rawhide to put out our test and final releases
  • When rawhide is frozen it makes it harder for developers to do their work.
  • The test releases need to go out as soon as possible because if released too late they are obsoleted by all the new changes added to rawhide.  Because of this we can’t test the test releases too long before making them available because too much else will have changed if we take too long to test it.
  • We can’t publicly test the release candidates because of the short amount of time to create, test, and stage them.  This is due to the size of the content and not wanting to delay rawhide any more than is necessary.

In terms of formal testing for our upcoming releases, we create installable images (test releases) for wider community testing.  Opinions vary on the value of these releases because once installed, testers are encouraged to update to the latest rawhide packages–sometimes upwards of hundreds of packages.  If this is the case is the return on investment for resources invested to create, qualify, and distribute the test releases worth it?  We definitely see an increased interest on the mailing lists and more bugs being filed, yet we do not have any hard data to definitively say. For Fedora 12, the Alpha release was dropped for this reason, combined with a shorter schedule to get back to our standard “Halloween/May Day” six month schedule.

Seth Vidal said it best, “Some people want to check a package into a repo and others want to build a distro.”  I think there are probably people that want to do both too.

Even with freezes, rawhide changes a lot.  Eventually the changes in rawhide do stop, but not until close to the end.  During Fedora 11 I ran repodiff every day comparing the current day of rawhide to the previous day.  Some interesting results I found were:

  • 1,940 package changes between Beta Freeze and Final Freeze
  • 620 packages changes between Final Freeze and the first RC compose
  • 90 packages changes between the first RC Compose and GA

Note: some of these changes could have been the same package being rebuilt multiple times.

I have always been sceptical of the wisdom of considering rawhide our primary testing vehicle.  From a formal QA perspective this is the nightmare of all testing scenarios–the product you are trying to test never stops changing.  To Fedora’s credit our releases have been pretty solid (with occasional exceptions) anyway which continues to amaze me.  I still secretly wonder how solid and polished our releases could be if we took a more traditional approach to the allowed amount of change.  I think the constant changes do contribute to our pattern of slipping test releases and final release dates by a week or two.

Some people argue that a week or two late in the world of software releases isn’t that big of a deal and maybe it isn’t.  Traditional software projects often have a much longer development duration and a week or two late for a project that took a year or more to complete isn’t that big of a deal.  For a project that seeks to release every six months, a week or two late is a bigger percentage slip compared to the total development time.  For a “time based release”– it also shortens the time allotted to the next release.

The proposal (as I understand it) from the FAD would allow rawhide to continue without freezes–removing the resulting holdups to developers.  It would also specify a process to examine the changes added to the release under development after a certain point in the development cycle.  Read all about it at https://fedoraproject.org/wiki/No_Frozen_Rawhide_Proposal.