The testing role

First I would like to expand a bit more regarding my view of the dynamics in the testing profession, before focusing on session-based testing.  In my opinion, the greatest value in the context-driven test school is their redefinition of the tester role.  The boundary being drawn, that is transforming traditional testing, is how and where scripting is utilized.

Most scripted tests are “checks” according to context-driven testing, much like what has been built in order to regression test; just checking to see that nothing has changed recently.  Scripted tests have a disadvantage that they take time to create and maintain, and their usage is easily made obsolete by feature changes.  What ultimately results, as the designated most-essential scripted tasks, are those that target features posing the most risk.

Who scripts those tasks then?  In the case of automation tests, oftentimes coders with high-level skills create these scripts, and the line between developer and tester becomes fuzzy.  With test-driven development, an Agile process, indeed the tester is often doing less scripting in his/her role as a user advocate.  As such, the Agile tester provides stories, questions, and suggests issues.  In this sense, I see context-driven testing and Agile testing as having a common ground.  The former has made an attempt to make the distinction clearer in saying that traditional scripting is not really necessary for the latest incarnation of the tester.

With this in mind, I see the traditional role of testers narrowing, and being focused more on objective risk assessment, “threat” prevention, and generally, user advocacy.  It is a less technical role than the traditional tester has now, in that it must be able to re-create scenarios that are oblivious to intended usage, and explore the deviations in human behavior, rather than follow usage defined in product specifications.  I wouldn’t want to write the certification test for assessing these new skills.

Joking aside, there is much to be said for the efforts of those in the context-driven school who are grooming these testers of the future with “tester coaching” sessions.  User-advocacy and risk-assessment skills are how the new breed of testers will shine.

Then there are the real world applications, and non-applications, of this idealized new tester. Realistically, if this tester was in the field, exclusively practicing these newly evolved skills, he/she would be a specialist.  In practice, these new skills may need to be be mixed with scripting skills, as there are so many small companies/departments looking to add testers who do it all; an automated test developer who also does manual, exploratory testing.  Indeed, I have seen Business Analysts and User Experience engineers performing many “user advocacy” test tasks.  I have also seen developers dictating the test methodology they preferred, as often other product teams consider testing as a service that is customizable.  In such cases, this new testing can be unheard-of, considered unusual, and not considered.

I may have created an expectation that user-advocacy testing is the sort of testing that will be exercised in session-based testing. Lets see what I think after the next post.

I may have also made context-driven testing sound highly reliant on the absence of scripting.  I blame the impact of James Bach’s delivery style for that.

The factory school

Before experimenting with the session test and blogging the experience, here are some views regarding the traditional or “factory school” of testing, and some perceived relationships.

To make the factory school a little more tangible, let’s take an example of a factory school, the ISTQB, per James Bach in the previously linked YouTube video.  Let it be known that they have certified me, and that is their main business, software tester certification.

Regarding an important facet of context-driven testing, session-based testing, we can ask if ISTQB includes it in its curriculum.  It does, and its definition of exploratory testing includes the main elements of the session test (in the syllabus there are references to Bach’s book, as well as IEEE standards).  If the ISTQB allows session test, does context-driven test allow non-exploratory testing?  My notes say that a context-driven tester may, for example, allow automation scripting if the test target doesn’t have human interface, like in performance and security testing.  In these cases, and as the name suggests, in this context, the suggested best practice is a scripted, machine-driven  test.

It cannot be said that the ISTQB holds exploratory testing as a major methodology (and in factory schools outside of ISTQB, exploratory testing is less defined, and often ad-hoc testing).  I believe this is where the differences between these two schools are.  The context-driven school emphasizes testing software human interfaces using session-based testing, a method this school has devised to push exploratory testing potential to its limits.  To achieve this emphasis, the context-driven testing school de-emphasizes the effectiveness of many uses of point-to-point traditional scripting.  This is interesting, because in doing so, the school also re-defines the role of the tester.  More is written regarding this view in the next post.

A few subjective words, first, on my take on the ISTQB, because some testing professionals have concern with what it does and stands for.  I am no staunch supporter, but it cannot be surprising that there are such organizations in light of the fact that there are so few college programs educating testers.  I must admit that I never heard of the ISTQB before coming to Sydney, but now understand how an employer in a similar job market would value such accreditation.  I also understand those who think such certification is an exercise that only benefits the issuer.  Ultimately however, I see so many job openings requiring ISTQB certification, that I believe those who are troubled by their existence don’t actually have issues with ISTQB, or the certificate holders, but rather with the (smaller) employers who seek aversion to product failure risk.  Personally, I can’t fault these employers, these certificates do establish some level of expertise, and a dedication to the profession.  With including accreditation, job seekers also reduce their risk of being denied consideration.

It should be mentioned that the ISTQB does more than just issue employment assessment standards.  It also should be known that after next week’s posts I will join the AST and syndicate the blog there.  I believe in enough respects, the two organizations share similar structures and purpose.

Below the surface

I have listened to several James Bach podcasts, and as with the talk I went to, he has an engaging delivery style, and can even sensationalize topics, such as the ones that make context-driven testing so different.  (He often has reminded me of Jack Black in his speaking style, and imagine it is by no means is imitative.)  Once I got past the first impressions, and dug deeper below the surface, being curious what foundations lie underneath, the following is noted from that further investigation.

Let it first be said that little was found after Internet searches regarding transitioning to context-driven testing, and that the following tenets were garnered from one talk on YouTube “Switching to the Context Driven School of Testing”.

James says that context-driven testing can be any of three things: 1. A worldview, 2. A test approach, 3. A community. That it is one of four schools of testing, with the most popular school being termed as the factory school.  He says that the factory school is overdependent on adherence to written specifications, and is often nothing more than “ceremonial” testing.

As I was interested in how there can be no best practices, it’s noted that he believes that a tester creates a custom set of best practices for what is being tested, and that the tester must take responsibility for engineering applicable test solutions.  He says the tester must become a test methodologist, using techniques that are not always transferable to other projects.

I also was interested what the relationship with Agile was, and had a theory, as one of the Agile principals being “Working software over comprehensive documentation”, that context-driven testing was a by-product of Agile.  Bach says no, it’s not a by-product, Agile is too restricting, and insists on using a capitol A instead of just being agile, when naming its best practices.

All in all, I characterize the observations above as only big broad strokes, and in the talk’s conclusion, it offers just an open-ended solution in how one is to switch to context-driven testing.  The solution suggested is that the community is duty-bound to help with specifics, and that tester coaching sessions will also provide transition assistance.

This is but one talk of many given, and of many more pages written about the subject, and I knew I had further to dig. There is so much written and shared, that I had to dash another theory that this type of testing was a scheme to brand-name something and then hope to market it exclusively.  Sorry, I did not investigate these ideas at the office, but rather on the Internet, remember!

After some time, it became clear how deep the nitty-gritty lay.  It seemed the school’s foundation was built on one nugget; exploratory testing. Exploratory testing is, in many ways, quite a jewel.  It’s flexible, time and cost saving, challenging, empirical, and productive.  Then it was compared to test scripting, by James, in another podcast, as a method that loops rather than use the straight-line technique that scripting employs. This concept helped define the method’s structure, previously assumed to be overly improvisational.  With the mention of a specific practice that includes exploratory testing, called the session-based test, the broadness of the stroke began to narrow, at last.

First impressions

I must have first heard the term context-driven testing at a talk given by James Bach, in Sydney, concerning test estimation.  At least, I believe he must have mentioned it.  To tell the truth, if he did, it didn’t register, and I didn’t make a note to myself to look the term up.  Sorry if this is confusing, but it’s possible that he never actually mentioned the term at all.  I just imagine he must have….

What did profoundly resonate that evening was his description of a tester’s primary responsibility; to make the software usable.  So impressed by this perspective, I added this to my LinkedIn profile:

Testing goals and methods

The expected goals from software testers are often: attention to details, finding bugs, and creating use cases… among others. Above all, what the tester must have in mind foremost is:  How usable is this software?

Implementing usability is how a design moves from a plan to the customer’s desktop. It’s what the developer needs most help with, after getting the design functional. Optimal usability is the aim of a tester’s art and skill.

The art of testing is overlooked when not recognizing the bridge a tester creates between code and human. The skills in test are important in delivering the product per requirements, on time and in budget.

I know how the user sees, thinks and acts with the given interface. I know how to inform the coder to also understand customer inclinations, to perfect the desired experience. I know how to document the steps needed to reproduce targeted function and style of use.

Pretty heady stuff, eh? I have removed the above from my profile since, but have found a place for it here, as this may be a safer place for such. Here it serves as an illustration of how strongly his words had inspired me.

Not until another Meetup did I hear the term.  Imagine that, I didn’t hear about it on the Internet!  Richard Robinson mentioned it, and then stood up and sketched on a whiteboard how he would test something that a developer, who had dropped by, was working on.  He didn’t really describe how it was practiced, he just mentioned it, and it was then I made a note to myself to look the term up.

When I looked it up, it was a bit confusing. Context-driven testing has no best practices? I must admit I was beginning to think of it as something that could be lumped together under the (blog) category of “Voodoo”.

I listened to podcasts, read pages, and watched YouTubes. Scripting tests and context-driven testing were presented as polar opposite techniques.  How could the writer of the tester’s bible propagate such heresy?  No wonder there is some controversy regarding this school of testing.

So my first impression was disbelief, having been “Factory School” testing for quite a while.  But I was also an exploratory tester, and had some faith in those two, Kaner and Bach.

What to expect here

Happy blogging!

That is what WordPress has just wished me in this new venture.

In order to make clear to the reader what to expect, here is the mission, and the deliverables.

Perhaps the tagline of the blog, “When does a software tester change test approaches?” is a more accurate description of what is expected. This will be an investigation, and this blogger’s transition to context-driven testing is by no means guaranteed.

I am an initiate to this school of testing, in the sense that the term is new to me.  I have a suspicion that I have been using its principals in my testing, since it relies on exploratory techniques, so in that sense, I do have experience with its use.  With that in mind, although transition is not guaranteed, it does have a great head start.

For any other initiates, let’s start off with the basics, and how this testing may have relevance.  Context-driven testing is a term that was created by two of Software Testing’s most influential contributors, Cem Kaner and James Bach.  It is a school of testing that was given its name in November 1999.  I remember buying Cem Kaner’s book, Testing Computer Software, years before that.  It is known as the Software Testing “bible”, and advertised as the “best selling software testing book of all time”.  James Bach is the foremost proponent of this school, having written books with and without Cem Kaner, and he has places alongside many a search engine query on context-driven testing.  If you are new with this term, because of these two fellows, perhaps give it some investigation of your own.

I hope to document here how I am to go about analyzing this test approach.  I can tell you now that I have doubts concerning how test passes are documented, as scripts created before testing are quite clear on what software functions are tested, and context-driven tests may not be as traceable.  Many times I have wondered, when exploratory testing, whether a specific function was already previously exercised, and exactly what test permutations were employed at that time.

Yes, this will, to a degree, be testing the testing methodology.  That is the mission.

When does a software tester change test approaches?  Certainly, in such a dynamic domain as software development and its testing, change is inevitable.  That change is the deliverable.