Rawhide serves several purposes in Fedora and yet I don’t think we clearly delineate what they are. It is often said in Fedoraland that the best way to test the Fedora is to “run rawhide.” There is a difference between installing and running rawhide to experience the absolute latest and greatest Fedora has to offer and intentionally testing it. I just think we need to be clearer about our approach and use of rawhide for testing (specific QA) because sometimes it doesn’t make sense.
Take a step back and consider what a strange testing target rawhide is:
- An “officially” available version for reproducing bugs or test results only exists for one day–contrast that with our build environment that was created specifically with the intention of always being able to recreate binaries in their original environment.
- For most people, the economical way to get rawhide is a mirror–there is no simple definitive way to determine that you have “the originally composed version” of rawhide for that day. There are a few methods that can give you “reasonable certainty”, but nothing like a single check sum or a way to confirm that the tree you’ve downloaded is exactly the same as what was composed.
- There is no “last known good version” when things completely get messed up and you need to reinstall from scratch except the Alpha, Beta, and Preview Release.
- You never know when it will install or not. It isn’t smoke tested–we leave that for *everyone* to do themselves based on their own install attempt. Even if Fedora hosted an automated smoke test to determine if it installs you still have the problem of mirrors being out of sync and not knowing if what you have is the exact same group of packages that passed the smoke test.
- It is the community’s only access point for obtaining the (what is close to) the Release Candidate for testing. I know several people will disagree with me immediately here–we’ve had the argument several times on IRC. I still think the underlying assumptions that it is “close enough” and “most likely the same thing” are not good enough. I think we have been lucky so far and there was a situation with F9 where the RC contained a different package version than what was in rawhide.
- We often place a higher value on daily rawhide than the Alpha and Beta releases by proclaiming that they “don’t really matter that much because they are ‘simply snapshots’ of rawhide.” The community at large seems to focus more on the Alpha, Beta, and Preview releases as evidenced by spikes in traffic on fedora-test-list after these releases.
- We consider rawhide our primary testing target yet there is no practical way to create a test matrix around it because it changes every day. Instead we create test matrices for the Alpha, Beta, and Preview releases which….. see the previous point. How do we know when we have completed a full test run? How can you thoroughly test a moving target?
I know I will hear the stock replies of “but it is rawhide” or “this is Fedora, not an enterprise distro like RHEL” and while I agree that “rawhide is rawhide” (a circular, but well understood argument for long term rawhide veterans) and that Fedora is not striving to be an “enterprise distro” that doesn’t mean one of Fedora’s goals has to be emphasizing a testing process that is flawed and could be better if we all put our collective brains together to come up with something better 🙂 We innovate in so many other areas… why not innovate here?
I would like to advocate that we reconsider the value we place on rawhide and the emphasis we place around the Alpha, Beta, and Preview Release. I think a good place to start would be document in our test plan where using rawhide for test results makes sense and where it does not. I believe rawhide does have its place, but I think we are trying to use it to cover too many bases and could do more effective testing with a more refined approach which in the end will make Fedora better!
What do other people think? Is there something here worth throwing around on fedora-test-list@redhat.com with a following up discussing at FUDCon Boston this week?
June 16, 2008 at 3:02 pm
Hi Bucky,
Thanks for your comment.
I’m not arguing that rawhide needs to be stabilized…. I think there are lots of contexts where it makes a lot of sense as it is.
I am attempting to argue that based on the points presented it is not always a suitable target for formal QA and that we need to reconsider where and when we use it. In doing so we can make our testing process stronger and more reliable.
John
June 16, 2008 at 12:22 pm
Hmm. It’s likely I’m missing something, and I figure it’s better to ask and be educated, even at the risk of sounding like a troll, than be silent and never find out.
But.
Isn’t Fedora itself the thing you’re describing? Fedora does not have the same feature freeze that RHEL does, so I think it’s pretty accurate to say that it is a feature-fluid suite of programs guaranteed to install and run at any point in time, and as new as possible. As new as you can get and still have some expectation of utility.
Rawhide is the thing that is exactly one level less stable than that (so new as to lack the guarantee of usability, but not so very new that the packages fail to compile–which I suppose would be the next level up from that). Stabilize rawhide any more than it is, and I’m not certain what the shining distinction would be between Fedora and rawhide anymore.
June 16, 2008 at 7:05 am
Yes, please, I’d love to flesh this out @ FUDCon.