Fedora Talk Powers NFR's Success

We had what I’m calling a min-Fedora Activity Day today for No Frozen Rawhide (NFR).  As announced earlier this week, a few of us met on Fedora Talk and Gobby (two of my favorite Fedora collaboration tools!) to make sure we had a clear announcement plan for No Frozen Rawhide.

Although not part of the original plan, we found great success writing use cases to explain each of the interaction points different roles would have to the changes in the release composition process.  Along the way we encountered holes in the existing processes and identified ambiguous terminology.  The best part of the meeting was clearly explaining each of the release trees.  Next week I intend to create a stand-alone diagram for it.

I’m a firm believer in choosing the right tools for the job.  I’m also convinced that the full power of this meeting was our ability to talk in real time using Fedora Talk.  All five us where in different geographical locations on opposite sides of the United States.  As it was, it took three solid hours to generate the substantial amount of information that we did.  I’m not even sure a full day of IRC could have generated as much success and solid information.

Thanks to Jesse Keating, Paul Frields, Luke Macken, and James Laska for giving up a big chunk of their day.  These are my favorite kind of Fedora events–where a group of folks collaboratively solve problems and brings greater clarity to our release processes.

We’re hoping that the switch over to No Frozen Rawhide will be a non-event where developers and testers can benefit from the new release process and easily find answers to the questions they have.  See how we are doing by reviewing the page we have so far and leave feedback if you have more ideas.

Fast Spaceless Backups

The title is slightly misleading, but it is true as long as the original source does not change.  I’ve also made a copy of an entire rawhide tree (~13G) in less than five seconds.

I’ve been syncing rawhide trees at home for a while and have a fairly old machine (Mac G4 400 MHz PPC running F10) that I use for file serving, http installs, unison, backups, yum repos, proxying, irssi, etc.  It only has 140G of disc space.  I could buy a larger drive, but I’ve found this is enough space to serve the files that I need and it forces me to keep things clean by throwing away files that are old or not used.  The G4 also makes for a great headless always on machine because according to my Kill A Watt meter it only consumes about 40 watts of electricity.

I was looking for a way to store multiple days of rawhide trees, knowing that usually only part of the tree changes each day and not wanting to duplicate a lot of data.  Inspired by a script that Jesse Keating wrote do to this I started exploring rsync and the fantastic --link-dest= argument.

This technique is also super handy if you want to do a quick backup of a massive amount of data and want to do it really fast–incurring space consumption only if the source changes later.

rsync -av --link-dest=/home/bozo/source /home/bozo/source  /home/bozo/target

The command above creates new hard links from the source to the target.  One tricky thing about the --link-dest= argument is that you must provide the full path.  Also, because hard links are involved the source and target must be on the same file systems.

As packages are removed from the source directory the target remains in tact.  All that has happened is that the link count for the changed file has been reduced. As long as the link count is greater than one the file remains. More on how this works.

This same technique works well for constructing other local Fedora trees with lots of similar files.  If the file is available locally, rsync creates a hard link instead of downloading it.

Update  (2009-01-15): Having given a little more thought to Chris Tyler’s important clarification, maybe I’ve oversold this concept as a good backup mechanism when really it should be more focused on its benefits specifcally for managing trees or directories containing binaries or packages.  In other words, this approach is best for copying or linking content that does not get modified or is completely replaced by a newer version of the file.

Reducing Rawhide Reflux

As a follow-up to a previous post titled What is Rawhide For? we had a great discussion at FUDCon Boston during the Friday afternoon hackfest.  Admittedly we didn’t hack on any code, though Seth Vidal wrote some based on one of the proposed solutions in less than an hour!

Jesse Keating acted as the facilitator and did a fantastic job helping to keep the conversation focused on brainstorming and understanding each other’s ideas versus summarily dismissing new suggestions as not being feasible.  I saw this same approach work extremely well in the post-mortem of Fedora 9 too.  To me this is the essence of great collaboration and arriving at innovative ideas we might not have otherwise.  I really believe being able to meet in person played a big part too.

We met for a couple of hours and the results from our conversation are here https://fedoraproject.org/wiki/JohnPoelstra/ImproveRawhideF10. This is one of the obvious benefits of having live events like this!

What is Rawhide For?

Rawhide serves several purposes in Fedora and yet I don’t think we clearly delineate what they are.  It is often said in Fedoraland that the best way to test the Fedora is to “run rawhide.”  There is a difference between installing and running rawhide to experience the absolute latest and greatest Fedora has to offer and intentionally testing it.  I just think we need to be clearer about our approach and use of rawhide for testing (specific QA) because sometimes it doesn’t make sense.

Take a step back and consider what a strange testing target rawhide is:

  1. An “officially” available version for reproducing bugs or test results only exists for one day–contrast that with our build environment that was created specifically with the intention of always being able to recreate binaries in their original environment.
  2. For most people, the economical way to get rawhide is a mirror–there is no simple definitive way to determine that you have “the originally composed version” of rawhide for that day.  There are a few methods that can give you “reasonable certainty”, but nothing like a single check sum or a way to confirm that the tree you’ve downloaded is exactly the same as what was composed.
  3. There is no “last known good version” when things completely get messed up and you need to reinstall from scratch except the Alpha, Beta, and Preview Release.
  4. You never know when it will install or not.  It isn’t smoke tested–we leave that for *everyone* to do themselves based on their own install attempt.  Even if Fedora hosted an automated smoke test to determine if it installs you still have the problem of mirrors being out of sync and not knowing if what you have is the exact same group of packages that passed the smoke test.
  5. It is the community’s only access point for obtaining the (what is close to) the Release Candidate for testing.  I know several people will disagree with me immediately here–we’ve had the argument several times on IRC.  I still think the underlying assumptions that it is “close enough” and “most likely the same thing” are not good enough.  I think we have been lucky so far and there was a situation with F9 where the RC contained a different package version than what was in rawhide.
  6. We often place a higher value on daily rawhide than the Alpha and Beta releases by proclaiming that they “don’t really matter that much because they are ‘simply snapshots’ of rawhide.”  The community at large seems to focus more on the Alpha, Beta, and Preview releases as evidenced by spikes in traffic on fedora-test-list after these releases.
  7. We consider rawhide our primary testing target yet there is no practical way to create a test matrix around it because it changes every day.  Instead we create test matrices for the Alpha, Beta, and Preview releases which….. see the previous point.  How do we know when we have completed a full test run?  How can you thoroughly test a moving target?

I know I will hear the stock replies of “but it is rawhide” or “this is Fedora, not an enterprise distro like RHEL” and while I agree that “rawhide is rawhide” (a circular, but well understood argument for long term rawhide veterans) and that Fedora is not striving to be an “enterprise distro” that doesn’t mean one of Fedora’s goals has to be emphasizing a testing process that is flawed and could be better if we all put our collective brains together to come up with something better :-)  We innovate in so many other areas… why not innovate here?

I would like to advocate that we reconsider the value we place on rawhide and the emphasis we place around the Alpha, Beta, and Preview Release.  I think a good place to start would be document in our test plan where using rawhide for test results makes sense and where it does not.  I believe rawhide does have its place, but I think we are trying to use it to cover too many bases and could do more effective testing with a more refined approach which in the end will make Fedora better!

What do other people think?  Is there something here worth throwing around on fedora-test-list@redhat.com with a following up discussing at FUDCon Boston this week?

Which Rawhide

Since starting the daily dose of rawhide I am gradually becoming more acquainted with the ins and outs of rawhide. A particular day or release of rawhide only exists for a twenty-four hour period–longer if something goes wrong with the compose process.

Rawhide is composed in Fedora’s infrastructure and made available at http://download.fedora.redhat.com/pub/fedora/linux/development. It is also made available and synchronized by the Fedora mirrors to improve download speed for users and reduce the burden on Red Hat’s infrastructure. To the uninitiated, compose is a special term which refers to creating or building an installable Linux distribution, in this case Fedora or Red Hat Enterprise Linux. See the Fedora Mirroring Page to learn more about becoming a mirror or understanding how the process works.

Because of the time it takes the mirrors to synchronize with Fedora’s infrastructure, combined with the current state of an individual mirror, sometimes it is difficult  to tell which day of rawhide you are getting.  In addition sometimes a tree cannot be installed if files are missing from the images directory.  Snake provides a simple way to find out.

$ su -c 'yum install snake'

$ snake-rawhide-status http://download.fedora.redhat.com/pub/fedora/linux/development/i386/os
checking http://download.fedora.redhat.com/pub/fedora/linux/development/i386/os
Fedora development i386 (20080429): ok

snake-tree info provides additional details

$ /usr/sbin/snake-tree info http://download.fedora.redhat.com/pub/fedora/linux/development/i386/os
No data source used
Name         : Fedora development i386 (20080429)
Arch         : i386
Id           : 1209473593.37
Version      : development
Family       : Fedora
Variant      :
Time         : 2008-04-29 05:53 PDT
URI's        :
Images       : i386 (kernel, family, timestamp, variant, boot.iso, initrd, version, arch),
xen (kernel, family, timestamp, variant, initrd, version, arch)

Each of these commands translates some of the less readable information from the .treeinfo which is created when a tree is composed. For example


family = Fedora
timestamp = 1209473593.37
variant =
totaldiscs = 1
version = development
discnum = 1
packagedir =
arch = i386

kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img
boot.iso = images/boot.iso
kernel = images/xen/vmlinuz
initrd = images/xen/initrd.img

instimage = images/minstg2.img
mainimage = images/stage2.img

Note: the .treeinfo file is a hidden file and thus not normally presented in a directory listing unless specified.

Even if you’ve verified date of a tree and that the images are present, there are still no guarantees that the tree itself is internally consistent–in other words there is no simple way to know whether a particular mirror has completely sync’d all of the latest packages.  And if there is, please post a comment and tell me how :-)