We are now almost a year into development on the game. Each week I meet with the team to discuss where we are at and to strategize the week ahead.
At this last week’s meeting I sent out some thoughts on beta testing. Below is an excerpt from last week’s meeting agenda, only slightly edited (to remove references to what we are working on).
I’m passing it along in case any of you find this “peek backstage” somewhat interesting…
Let’s talk beta testing
Everyone has heard me talk about my desire to be into beta at the end of March. My suspicion is that this has some of you shaking your head, thinking, “Not possible, why try?”
Well, when I use the word “beta” there is room for interpretation. So, I’ll explain what I mean.
This discussion was triggered because Roberta and I had a conversation about my vision of ‘beta testing’ and the need for “approvals” along the way. The word “Approve” is like the phrase “Beta Test.” It needs further definition and interpretation.
To me, there are three approvals needed. Is it good enough for coding to start? Is it good enough to go to beta? And is it ready for final approval?
Something can be “Approved for beta” much sooner than something that is, “Approved for Paying Customers.”
Let’s pick a simple example. Imagine the VR (low poly) version of the first scene. I think we all agree that it is nowhere near good enough to be “Approved for Paying Customers”. To get it to that level will need many hours, and probably several days, of artist time. Arguably, right now, it isn’t even good enough to be “Approved for Beta”.
In other words, I’d like us to think in terms of three Approval levels (viewed from an artist’s perspective):
– Approved for coding — this can happen as soon as we have an ‘approved’ grey boxed scene.
– Approved for beta — this can happen as soon as we have something that looks decent and has enough of the graphics such that the level is playable.
– Approved for Paying Customers – everything is in place and is bug-free.
That middle Approval is the one that is mushy and relates to the definition of the word “Beta.”
There are multiple flavors of beta testing, and there are different groups of potential beta testers.
Limited Release Beta Testing – release to a small carefully selected group a version of the game, along with specific instructions as to what we want tested.
Wide Release Beta Testing –essentially the entire game needs to be in place.
Here’s the way I see it playing out:
In April, May, June and July we will do a series of Limited Release Beta Tests. We will have specific goals for each of these and the beta testers will be hand-picked.
The schedule that follows is entirely made up, but representative of what our beta testing schedule might look like:
April 1 Beta (Computer) – 5 to 10 beta testers – Pick some friends and watch them play through the first couple of regions, just to see if they can figure out the user interface based on the directions provided. My thought is that this testing could be done at a table at some local place with one of us watching — Starbucks or perhaps at one of our houses and invite beta testers who live in the area? Alternately, it could be done by asking testers to screenshare while playing the game while it runs on our computer.
April 7 beta (Quest 2) – 5 to 10 beta testers – Pick some friends and watch them play through the first couple of regions, just to see if they can figure out the user interface based on the directions provided. My thought is that this testing could be done at a booth at some local point with one of us watching — Starbucks or perhaps one of our houses and invite beta testers who live in the area?
April 30th Beta (Computer and VR) – 20 beta testers – Rerun the testing above and see if the beta testers get farther and if the previously identified issues have been solved.
May 15th Beta (Computer and VR) – 20 beta testers – This might be the first time we have the beta testers “play” parts of the game without us watching. That said, the release is probably only a portion of the game and will be accompanied by a walkthrough and specific documentation saying what to — and what not to — try.
June 1st Beta (Computer and VR) – 30 beta testers – The first fully playable game! It still won’t have everything in it, but this would be the first version of the game that could – theoretically, at least using the walkthrough (and a series of saved games that we provide) — be played end to end. This may be the first version that has professional voice acting and even perhaps sound effects and/or music.
July 1st Beta (Computer and VR) – 100 beta testers – The first time we let the game out to a wider audience. We include a walkthrough, and tons of information about what is, and what isn’t, in the beta test. We also include saved games so that as people crash, they can get going again.
July 15th (Computer and VR) – 250 beta testers – This is the “Approved for Paying Customers” beta. The first time we send out an actual release candidate with everything there!
August 1st (Computer and VR) – Submission to the approval process at *****, and possible release to the public of the computer version.
Bottom line: We will start with a small face-to-face group, with a tiny piece of the game, and graduate to a large group with all the game, but do so in a tightly controlled, carefully paced, manner.
Wait!!!! There’s another kind of testing we need to talk about
Roberta is the person charged with the most important kind of testing there is. Customers will forgive a game that crashes. They will forgive confusion, bugs, bad graphics, just about anything. But there is one thing they will never forgive.
I call it: A lack of fun and a well-thought-out game.
We are competing for our player’s time with other games, movies, music, books, going to the park, taking a walk, and many other things. We have to be more fun than any of those other things, or they WILL do those other things. That’s human nature. Pretty graphics, beautiful music, amazing sound effects, challenging puzzles, fast execution times, etc., cannot save us. Ultimately when we mush everything together, it has to be more fun than anything else the player could be doing, or why would they want to play our game?
Roberta’s job is to make sure the game is fun and the design is tight. She has at least one hand tied behind her back, because [***DELETED***].
She will not be able to start serious play testing for herself until the whole game is playable throughout. Even then, play testing is meaningless until all the puzzles and “look/use” messages are in, as well as any significant background objects. (And it should be noted that Roberta was the best QA person and beta-tester that Sierra had – she’s very thorough and somewhat of a perfectionist — and finds ‘everything’!)
Within reason, she CAN play test a scene that does not have final graphics. If a room is supposed to have a green tree, and instead has a pink cube representing the tree, Roberta most likely can test the room. As long as someone tells her, “Ignore the pink cube. That’s really a green tree that we are still working on,” she can test the room and see if the messaging about the tree feels right. In other words, as long as everything is mostly there, even if graphically there is work to be done, she can start testing. At a minimum, to really get a sense of the game, she does need at least placeholders for all graphics, sound effects, speech, and music. The coding for the game should be as near to complete as possible. Once again, it is ok to have a door that doesn’t open, as long as someone tells her, “There is a door here that is still being worked on and is not animated yet.” It’s better if the door is there, or at least some cube that represents the door, but really, for Roberta to provide meaningful feedback on playtesting, all of the game messages and puzzles must be in place.
My goal is that she can start playtesting the game around the same time that we start our beta testing. I’d like to think that by mid-April or the beginning of May there will be something that needs plenty of graphic and sound work but is a fully playable game.
Once Roberta starts playing the game, she will start sending out bug reports, and you will be shocked by how many bug reports she generates. Not all of the bugs will be “bugs” in the classic sense. If she sees something that needs a message, she will add it. If she sees some graphics that she doesn’t like, she will mention it. If she wants graphics added, she will have them added. If something in a room seems distracting to the player, she will nuke it. Her job is to be the eyes and ears of the player. It will take months to work our way through Roberta’s lists. We will be frustrated and tired of seeing bugs/changes, but you need to keep in mind that this is the most important testing that will occur, and that Roberta will see things that you or I might not, or that we might not care about. She will solicit feedback from all of us, and from the beta testers, but will ultimately go with her gut. She will accept some advice and will ignore other advice. Expect it and just recognize that is how she works. It will be worth all the pain when the game is done.
Let’s talk about “BUZZ”
We have to be very careful in beta testing. We live in a world which is dominated by social media. Especially in the early days, the game that beta testers will see will not be representative of what will ultimately be released to the public. We can ask the beta testers to not discuss the game publicly but, realistically, they will. They will want to tell their friends they are beta testing the new game from Roberta Williams, and the first question they will be asked is, “How is it?” If they respond by saying, “I think it will be ok, but it needs a lot of work,” we will be dead in the water. We need super positive things said about us. I would not pay for a game that is mediocre — and if I wouldn’t do it, I sure wouldn’t ask others to do so. We need our beta testers telling their friends how great the game is. We need to generate excitement wherever we can.
Factually speaking, the early releases will not be great, so how do we guarantee that beta testers say only great things? I’m not sure it can be done, but to the extent it can be, it will be by making sure that we under-promise and over-deliver. The beta testers need to be carefully chosen and carefully prepared. We need to tell them what they will be getting, set their expectations (maybe low), and then exceed their expectations. We also have to make sure they understand what we allow them to say publicly, what we want them to say publicly, and what we do not want them to say publicly. I don’t want to ask anyone to lie on our behalf, but the truth is that they’ll be seeing something which is NOT the game we’ll actually be releasing. They’ll be helping us look for things that we have no way of testing without their help. We need the beta testers. We are too close to the product and beta testers will be seeing the game with fresh eyes. They have different speed computers, different graphics cards, different pre-conceived notions on how a game should be played, different brands of mice, different keyboards, different amounts of memory, etc. We want their feedback, and we do want them to share their experience on social media, so we get some buzz going, but we need to ensure they rave about how awesome the game will be when released. I have no doubt they will if we help them understand the beta process correctly.