Well, I think the title of this post is a bit strong, but let me explain you my point.

I work for Canonical, a 100% distributed company where the majority of employees work remotely. We meet every once in a while to work together for a week, but, normally, we don’t see each other.

I joined the company a year and a half ago and, even back then, the people who had been there for more than 3 years stated “It is big now!! Do you remember when we knew the name of everybody?”. Well, Canonical is not THAT big, but it is getting harder and harder to remember people’s name and, because it is a distributed environment, people’s faces. Even harder, we rely a lot on IRC communication so you have to match a real name, with a face, and with an IRC nickname. For me this is almost impossible to achieve.

When I wrote an email I relied too much on email addresses auto-completion and this was making the problem bigger. “I think his name started with an M, and it might continue with an E, no, wait, an I. Here he is!”. I don’t do that any more. I write a good amount of emails, but not too many to not be able to spend one minute writing down the addresses it goes to.

Canonical has an internal web tool called “Directory” where you can quickly search a person (by team, name, IRC nickname, town, manager, email address, etc.) and you will soon find the record with a picture, real name, email address and IRC nickname. Then I copy and paste the email address to the “To:” field. It does the trick for me. It helps me visualizing who I am writing to and helps me matching the face with an IRC nickname, so the next time I get a ping from someone I can see the face behind.

Does your company has such a web tool? I really recommend it.

If you think that testing software is an unskilled activity that “even a two-year-old can perform”, keep reading, I’ll try to change your mind. If you do not agree with that sentence, keep reading, you may be interested in joining us.

Software testing is generally seen as the poor cousin of programming. While the bad reputation of testers happens in all software environments, this is more common in free software communities, probably because the “show me the code” motto is too deeply attached to the open source communities. This, unfortunately, is too often translated in unreliable software released with a lot of bugs (some of them critical).

Testing software, as any human activity, is a task that almost everybody can perform to some sort of proficiency. However, that does not mean that it is an unskilled activity. You have to know what to do. You need to have (or to develop), among others, excellent communication skills, technical writing skills, software architecture knowledge, technical research expertise, a critical mindset, etc.

We cannot leave quality to good luck. We cannot rely in having millions of users who will find bugs as they use the applications. Our users want to use the software, not to find bugs and report them. FOSS projects in general and Ubuntu in particular need a new way of rethinking testing as a skilled activity and an opportunity to contribute to the project.

We want to build a Testing Team in Ubuntu to try to minimize the impact of bugs in the released versions. This team would have a mailing list and regular meetings on IRC. Activities will be diverse and will include things like: formal manual testing, exploratory testing, writing new test cases, organize and conduct community testing days, automated testing and developing new tools (yes, if you like to code, there’s also a place for you).

We would love you to join us and make it happen.

We are having a session at UDS Lucid to discuss this topic (scheduled for Wednesday). You can subscribe to the blueprint as well.

This Thursday Karmic reaches the last milestone before the final release. As for every milestone, we need to test all the ISO images we produce, with every possible installation.

All of these test cases will appear, with instructions to follow, in the ISO tracker. If you don’t know how to use the tracker, this blog post will serve as starting guide.

One of the complains of the new comers is that they don’t know which test case needs testing. The coordination is done at #ubuntu-testing at Freenode and not everybody can access IRC. This time, Dave Morley and I, will try something new. As the RC images start appearing and testing begins, we are going to update in Twitter, using #ubuntutesting as tag.

If you want to help us testing RC images, please, follow us in Twitter and make sure to search for #ubuntutesting for updates. And if you’re helping testing, please, tweet about it!

Of course, this is an extra way to get informed. Coordination will happen, as usual, at #ubuntu-testing IRC channel.

Ubuntu 9.10 (Karmic Koala) Beta was released last Thursday. I am so glad to announced that we 98.9% coverage of the test cases in the ISO tracker. I would like to thank the community members that helped testing the ISOs, specially those who joined recently. Thanks! I am discussing with the Community team about the possibility of including this participation in the Ubuntu Hall Of Fame, just as the bug triagers or sponsors are.

I will blog about Release Candidate ISO testing as we approach the milestone week

Also, and because we are getting new contributions, I would like to comment some of the reports we got, so we can improve every milestone.

Not really a failure

We got this comment, in a test case marked as failure:

I have a tablet fujitsu p1630 and the stylus works in the life cd! great, congratulations!
(missing is the calibration tool which should be loaded. The stylus is not properly calibrated and cannot reach the top line (where the application menus sit!).[...]

In the ISO test tracker we mark as failures those experiences that prevented us to do what we want to achieve in that test case. I.e. If we want to install, and the partition manager fails, that’s a failure. If we do install (or can access to the Live environement, as in this case), the test didn’t fail as such. We would mark that as success, but will link the non-critical bugs that we find.

Usability bugs are bugs

The lack of colour in the default options during installation could cause problems for new users.
The default setting of Mute, for sound could cause problems for new users.

These are great examples of usability bugs. Thanks for noticing them! Usability bugs are bugs, so do not only put them as comments in your report, also go and file bugs in Launchpad for them. They will help a lot to new users to understand how Ubuntu works.

Love Ubuntu? Want to help?
Karmic Beta candidate images have started to appear in the ISO tracker.

You don’t know how to use it? It is pretty easy!

As part as the Ubuntu Developer Week, I gave a Mago tutorial on how to add new test cases to available applications in the Mago library.

I have reformatted a bit the logs to make the reading easier and it is now part of the Mago documentation. Enjoy!

Mago tutorial

This post is going to be all about the ISO Testing Tracker. You might know it already, it’s been with us for a while. But, like old friends, sometimes you need a reminder that they are there and give them a hug. But before we continue with the tracker, let’s start with the basics:

What’s ISO Testing?

If you are reading this, you would probably know that Ubuntu has a 6-month release cycle, which means that every 6 months we have a new Ubuntu release, ready to install.

The good thing about software fixed cycles is that you can schedule anything you want. In fact, for Ubuntu, we schedule just at the beginning of the cycle up to 8 development releases. You can check at the Karmic schedule that there was an Alpha 1, Alpha 2, Alpha 3, Alpha 4, Alpha 5 and Alpha 6. And that there’s going to be a Beta and a Release Candidate before the “real” Ubuntu 9.10.

Those milestones need to be tested properly, because we release CDs for people to test on their hardware, and we have to make sure that everything is more or less OK. (with “more or less” I mean that these milestones are still software in development, so you have to be careful, anyway).

How do we track milestone testing?

So, here it is how the tracker appears in the show. Milestone releases happen normally on Thursdays. On Monday, in scheduled milestone weeks, testers start looking at the tracker looking for the candidate images: the ISOs that might be announced as the “Karmic Alpha something”. But these images need to be tested before they get announced.

How is the test tracker structured?

At Ubuntu, we produce builds for many different *buntu flavours. In the ISO Tracker we have a category for each of them. You can filter by category using the “Filter” menu in the bottom left of the page. That would allow you to concentrate in the flavour you want to test.

Filter by category

Once you select a category, you will be able to select an image to download for that category. A category like “Kubuntu” may have different images: i386 Live CD, amd64 alternate, etc., and all of them need to be tested before releasing a milestone.

Kubuntu Karmic Alpha 6 images

If you click now in any of those images, you will have access to a direct link of the ISO to download and a selection of tests that you may run for that image. For every test, there is a link to the test case description in the test case wiki. The test cases are easy to follow when running any of them.

So, to sum up, every category contains a set of ISO images and every image contains at least one test case.

Iso Tracker structure

How do I help?

As I wrote before, the week of the release of a milestone we need a lot of help testing the candidate images. If we want full coverage we need to run more than 150 test cases. If you have a spare machine or a virtual machine installed, you can help making Ubuntu better by running the untested test cases during the week of the release of a milestone.

First thing, you will need an account in the ISO tracker. Unfortunately, your Launchpad Open ID won’t work for this site (indeed something that needs to be improved). Click on “Log in” and fill the requested data. As the page explains, if you already have an account in Ubuntu Brainstorm, you can use it for the ISO Testing Tracker.

Create a QA account

Once you have an account just select the image you want to test, choose a test case for that image, start running the installation in a virtual machine or a spare machine and report any bugs you may find in Launchpad.

Once finished, you have to report back in the ISO Testing Tracker. Click on the test case you have run and report Pass/Fail depending on your success. You can also add the bug numbers of the bugs you found. They will be linked to the Launchpad bug and it will help the release team to track them.

Please, report back!

The time is NOW!

Next week, on Thursday 1st of October, we are releasing Ubuntu Karmic Beta, which eventually will become Ubuntu 9.10. This is a very important milestone, because many people upgrade during the beta, so it is very important to release something very stable at this point. From Monday, 27th of September, we will be testing the candidate images at #ubuntu-testing. I will send reminders to the different mailing lists later this week and at the beginning of next week, but you can start right now creating your account and making yourself familiar with the ISO Testing Tracker.

Help us releasing Ubuntu better and better every 6 months! And remember that we are always at #ubuntu-testing if you need to ask any question!

If you search for software testing automation in your favourite search engine you will find a lot of matches about testing automation of web applications, lots of proprietary software to test proprietary desktop software, lots of FOSS to test web applications and some blogs about testing proprietary desktop software.

What about testing desktop FOSS? How do I find people interested in these topics? Is there anybody out there? Hello?

Fortunately, there is a growing community in the FOSS world interested in testing desktop applications automatically. To try to keep things going, we are having weekly meetings at #ubuntu-testing (irc.freenode.org) every Wednesday at 16:30UTC, running for half an hour.

If you are interested in these topics, please, join us and feel free to add any items you may want to discuss to the agenda.

It is great to see more and more people interested in Mago.

When I started the project about a year ago (by that time it was called Ubuntu Desktop Testing), it was an in-house project I created to solve the problems I was having on the desktop testing I had to do as part of my daily job.

Up to three sessions at the Ubuntu Developer Week, a talk at Gran Canaria Desktop Summit, some UDS and two changes of name later, Mago seems to have found its path and we are seeing a lot of new contributors adding fantastic ideas to it.

To avoid losing any of these ideas, I have created a Roadmap in the Mago site to track the status of the new Mago features. Please, check it out to see what’s happening on the Free Desktop testing automation world.

In about a couple of hours (16UTC) I will be giving a Mago tutorial as part of the Ubuntu Developer Week.

Please, join us on #ubuntu-classroom for a quick hands-on Mago training session!