This report is impressive… receiving 7,872 new reports and resolving 7,802 bug reports in a single month while continuing to hold a backlog of 46,000 open bugs! This says two things to me… they have a process for triaging and resolving bugs and they have a tremendous amount of people actively reporting issues.
In the past I’ve been skeptical about the size and effectiveness of Ubuntu’s community because the only evidence usually cited is distrowatch.com, googletrends, and glowing reports (I assumed biased) from Canonical employees at live events. I’ve never heard any hard numbers at the live events, just lots of enthusiasm about how things are “getting bigger and better” and that “collecting good data is hard”.
I think these bug stats are a great first step and show a lot of involvement and hard work. I wonder what the average run rate is per month? Is this regular monthly activity or the results of a special event for the recent release? I found a few more stats, but nothing that showed a corresponding monthly close rate. Without reporting the number of bugs closed each month it is hard to get a full picture.
These are Fedora’s run rates for the past year. For greater perspective look at the graph here.
I tend to think that comparing Fedora to Ubuntu as distros is an invalid comparison. Both projects have very different business and support models combined with different purposes. One thing both projects have in common is community. I am told that Ubuntu’s bug process is primarily community driven and this led me to wonder what Fedora can learn from Ubuntu about bug triage and bug handling.
Because I think there is enormous value in making sure our bug database is current and that we identify and track important release blocker bugs, I’ve been working for the past ten months to help jump start the Fedora bug triage process. We have made some good strides, but I think we are capable of more. Think what we could do with the remaining untriaged NEW rawhide bugs if we had more than three or four people at our weekly meetings? We need to energize our existing community and draw in new people to triage bugs, but the things we have tried to date haven’t gotten us very far.
If you have new ideas or want to help make a difference in Fedora, please join us at our weekly bug triage meeting on IRC, each Tuesday at 15:00 UTC (10 AM EST). We meet on the #fedora-meeting channel of irc.freenode.net.
November 13, 2008 at 7:37 am
Mackenzie: the business about patches vs. debdiffs is something I’ve been railing against for some time, and ultimately seems to be a misconception somebody in the triage community picked up and has been spreading as a rather virulent meme. Developers who are actually applying the things are in fact usually quite happy with patches – it’s just that triagers seem to have got this idea that we need the specific format known as a debdiff (which implies having dealt with the changelog and such).
For people who want to become part of the development community, they need to be able to prepare the whole shebang because that’s what they’d need to upload to the archive, but there’s really no reason why somebody who’s gone to the effort of sending us a normal patch (unless it has zero lines of context or something) should need to be asked to massage it into a subtly different format.
November 3, 2008 at 11:59 am
david:
Bugs are *not* closed automatically after a certain time period. That was tried once, and there was a huge backlash, so now bugs say at the top “this bug has been marked for expiration” but they don’t actually expire. It shows up there when the reporter has given up on the bug and is no longer responding to triagers. We cannot then figure out if they stopped responding because it was fixed or if it still exists. At that point, it’s a judgement call. If there’ve been multiple releases in the timeframe, I’m likely to just close it with a message that if it still exists, please open it and give the information that was requested. If it’s not that old of a bug, I’ll just add a reminder to please provide the information requested so that we can keep working on it.
spot:
Yes, it’s true that we use Malone for tracking “please package ____” requests.
The biggest weakness I see in Ubuntu’s bug process is that we often have a *lot* of triaged-with-patch bugs sitting there for ages. They could’ve been fixed a while ago, but oh no, we need a debdiff, we can’t just use a patch. So then we have to find someone that knows packaging (really, not everyone that can handle a bit of C knows how to package), and get them to make a debdiff. And then we have to wait and wait and hope somebody with upload rights notices.
Launchpad does make it much easier to find duplicates, I think. It’s search algorithm is better, I guess. I think every bug I’ve filed on Mozilla has been marked a duplicate, yet no bugs even remotely similar to the one I was filing showed up in any searches.
November 1, 2008 at 10:48 am
John
I have been trying to figure out how to get bugzilla to give me some stats on how many bugs have been historically open for long term. Say compare the number of bugs that were opened in FC3 and still open in FC6 vs the number of bugs opened in F7 and still open in F9. It seems like it is too generic a request for bugzilla. I assume I am just not familiar enough with it to get the information that I am after. Any information or assistance that you can provide would be appreciated. Feel free to email me.
Thanks
November 1, 2008 at 8:15 am
Are you filtering out Review bugs? FWIW, I don’t think Ubuntu uses their bugtracking system for new package submissions (I could be wrong though)
October 31, 2008 at 10:02 pm
Apport is only turned on in the development distro and one metric Launchpad does not track is where the bug exists, so it is not clear how many of these bug reports are actually from Apport vs hand reported.
October 31, 2008 at 6:51 pm
one reason Ubuntu gets and handles more bugs than most distros is Apport. It makes it easy to file a bug and for Ubuntu to spot duplicates automatically. There alone you are likely to have the majority of that impressive high number.
Another thing is that all bugs default to having a timeout, if nothing happens for a set number of days, the bug is simply closed. This might be a worthwhile technic in keeping the amount of open bugs low, but it certainly also risks losing bugs and disappointing users due to lack of interest in their actual problems (remembering that fora bugreport to be filed, the user actually cares enough to go through the process, he likely also cares enough to be ticked off when the bug just gets closed)
What I have observed interacting with their bug teams, if they can’t resolve the bug within a few comments the volunteers will tend to drop the bug, the paid developers generally do not do this.
E.g. I have an open bug that causes my iwl3945 to not work under 8.04 and 8.10 which they have aside asking for some basic information ignored. This makes Ubuntu pretty useless on my machine since I have no network. Ignoring certain bugs is worst than others – if I can boot, I have network and X comes up any other problem can be fixed with an update. Ignoring bugs that hamper this would tend to scare me, low hanging fruit is nice target for the average contributor but you also have to have the willingness to spend serious time and energy on hard bugs which I unfortunately from personal experience do not see much in Ubuntu.
What I think the rest of us to learn from Ubuntu is to leverage users more for triaging easy bugs in applications they care about (I try to triage and handle bugs for Banshee in Fedora e.g.), we could use tools like Apport. Apport is put bluntly fucking amazing as a user, I don’t have to go back and reproduce crashes, nor wait for the developer to figure out which log files and general bits of information I forgot to include in my report – it makes it dead easy to spot mistakes when they happen so the road between bug discovery and a fix being in the users hand is short. We really need this. However the failing of such tools currently is that we have no standard, each major project seems to do their own and nobody sits down to agree on how this should work. What I like about a fedora approach is the persistence in fixing bugs, when a bug is not outright ignored (sadly happens due to lack of time), it is generally followed to the very end.
So I take these stats with a grain of salt, never the less I think they have some good ideas in mobilizing people and automating work we really should look at finding common ground on.
October 31, 2008 at 5:17 pm
As with many things, metrics can be hard to sort out.
Is their bug tracker easier to use? I’ll avoid bugzilla when I can file a trac ticket on a project’s page (if it has one).
Do their contributers file bugs or use mailing lists first? Are they educated enough to look for dups. Closing a lot of badly opened bugs as “not a bug” can lead to different stats.
Are they sharing details with Debian effectively or ineffectively? Does this point at forking Debian too much? Is there an RFE or feature process that ways heavy on the bugtracker? (Note: I don’t know the answers to any of these).
Basically, we need more information to tell if this means anything in any sort of comparison, though I agree, it’s nice to see numbers — drawing any sort of conclusion from them though, it’s easy to get to the wrong conclusions.
Do we have lots of bugs that get opened and never dealt with? Absolutely, and that leads to a bad user experience when that happens. I think “bug open time graphed over time” is perhaps an interesting metric in that regard, but I’m not sure you can gauge community size by bug counts.
High bug counts are not a good thing to have, nor are high open counts. High closed counts are only meaningful if they indicate a bug was in a “fixed” transition, and they probably mean less if they aren’t also brought back upstream.
Numbers are fun though 🙂