Weblog Consulting Register Login Subscribe to Free Influx Emails Join our Facebook Group Branding Street Cred Technology Lifestyle Style Entertainment Interviews Art InfluxTV Map Widgets RSS — All Subjects RSS — More Feeds

Search

Advanced Search

Contact Privacy Policy

Arnold creatives, Roger Baldacci and Lawson Clarke made this film for the Hatch Show, it's a rather alarming take down of focus group testing .



I traded a few emails with Roger to learn more about how the film was made and what inspired it.

1. Tell us your background- years in the biz etc?

I’m an Executive Vice President, Creative Director at Arnold. I currently run the ESPN business at Arnold and also work as a senior writer, partnered with art director Paul Renner. I’ve been in the business about 17 years.  Prior to Arnold I worked at Fallon in Minneapolis.
 
2. What inspired this labor of hate, if you are at liberty to tell us?

I call this a labor of hate, because I simply hate concept testing with focus groups.  I have witnessed untold creative concepts die at the hands of people who are only there for the $50 and free M&M’s and Pepsi.
 
There are just so many flaws to the system that I believe it’s not scientifically valid on any level.  These flaws include:

1. Moderators can lead respondents
2. Alpha respondents can lead/sway the rest of the group
3. Spots that are easiest to understand/most familiar often win out
4. Concepts aren’t accurately reflected with cartoon storyboards
5. The mindset of respondents is to criticize no matter how objective you ask them to be. It’s human nature,
6. It's not reflective of the media buy. I’m in this business and I can’t tell you what commercials I saw last night. Brands are built by a cumulative effect on multiple sources.  You can’t expect brand preference to go up after ONE viewing of a bad animatic.
7. Time consuming. Creating these takes time and energy from the creative department.
8.Cost prohibitive. It’s not unusual to spend $300K to find out
9. Preconceived notions. Clients and agencies tend to have ideas of what they want to get from the focus groups so they look for those. One client felt we had a brand linkage problem when the moderator asked the groups to name a commercial in the client’s category. They didn’t name our commercials. I said that if they recalled our commercial, but couldn’t remember it was from us—that’s a brand linkage issue. But they couldn’t even name our spot which means we had a media issue.
10. Kills the creative process. If a spot tests well in groups, clients are unwilling to deviate from it when it comes time to shoot. So trying to get a good director to plus the boards? Forget it. The magic of ad-libs from the talent on shoot day? Forget it.
 
But let me be clear, I’m not against focus groups and learning information about your target audience. I think if you poll planners in all agencies, they would much prefer to do up front market research over concept testing. The issue I have is with testing unfinished work in front of people.
 
Sure, the stakes are high for CMO’s with the average tenure being 22 months on the job before they hope to take their success and move on to another job. But look at the movie industry. The average development costs for a typical Hollywood movie is around 100 million. Do the studios do animatics or board o’ matics of their movies with tests audiences to see which movie they should produce and how best they can “optimize it?” Of course not. They green light a script, film it, edit it, mix it and THEN they have test screenings. Now they’re just asking to see if there was something the audience didn’t get or found confusing. This may result in re-editing it or perhaps reshooting one or two scenes.

But imagine showing an animatic of the film Adaptation or Eternal Sunshine of the Spotless mind to test audiences?
 
3. Why did you pick the Apple spot? Were there any other candidates?

We thought about other high profile spots that people within the industry would know, but in the end, we decided to go with the most iconic spot in the history of advertising. There was something so poetic about taking such a high profile Super Bowl spot for a visionary brand like Apple, (who don’t test, clearly) and subject it to the whims of people who need extra cash to pay their cable bill that month.
 
4. How did you make the film?- do the recruit and my guess is that's not an Arnold planner in the film....

We used a real focus group facility who donated the time and recruited real respondents at cost. The moderator was an Arnold planner who recently took a job on the West coast. We briefed the respondents exactly the way we would a normal focus group—telling them that we “had nothing to do with this commercial” and that we were looking for their “honest feedback, both good and bad.” We told them this was a potential Super Bowl spot. We had two cameramen in the back room behind the mirror, each responsible for close ups on one side of the table. For the majority of the shots, however,  we used the actual focus group camera and audio from the facility.

5. Do you think creatives believe all focus group testing is like this?

The film has been posted on many blogs (in many languages) and you can tell by the responses that this has struck a nerve with many creatives. We have all been there and we’ve all see this. So, yeah, this is pretty much how all (concept testing) focus groups tend to go.
 
6. What does it feel like when you have to sit in the other room in a session like this and listen to people talk about your work?

Any good creative has a thick skin, so hearing your concept get ripped to shreds is not new.  You smirk. You make witty, sarcastic remarks about the respondents, down a fistful of salted peanuts and move on. 
 
Frankly, if focus groups liked your concept then maybe you’re doing something wrong.

Again, focus groups typically reward the familiar. I have run into this situation many times. One time, the agency recommended spot lost out to the back up spot because it featured a husband speaking to his wife in the kitchen. Very easy to understand. The recommended spot featured intricate cg work and was actually more on strategy than the husband/wife spot. We produced the spot that “won out” in testing and the client proceeded to do everything they could in the edit to make it more like the spot that died under the harsh florescent lights of the focus group room.
 
7. How have people in Arnold reacted to the film?

People have loved it. Both Lawson and I have received positive feedback from inside and outside the agency walls. It has been forwarded to many people in the business, including the production side. People dig it because it’s so painfully real.   
 
8. If you were a planner, what would you do to help creative teams develop great work?

The biggest mistake planners make is to pretend they’re in the creative department. If you want in, by all means put your book together and jump in. There’s plenty of room behind the two way mirror at the testing facility.
 
What creatives are looking for from planners is a simple, unique insight into the human condition of our target audience. Granted, that’s not always easy. But telling creatives that teens like music and technology is not helpful. Don’t give us charts and graphs because we know we always want to be in the upper right hand quadrant.
Give us a simple sentence that can get our gears turning. Tell us something we didn’t know about our audience. We all know that milk is good for your bones and does a body good. Just hand us a postcard with a sentence like “You know. It really sucks when you want milk and you don’t have it.”
 
9. What stuff is inspiring you these days?
 
There has been very little that has inspired me within the advertising industry lately. For an industry that prides itself on originality, it’s amazing how often we copy each other.  Remember shaky cam? Remember large type to small type? Remember the “organic” layout with a bunch of stuff photographed on the page with copy wrapped around it? Remember binding in little booklets into magazines? Remember that whacky, offbeat Swedish humour?
 
I tend find inspiration from films, photography and contemporary art. I know that sounds pretentious. So if I had to pick one commercial thing that inspires me I would say the writing of screenwriter Charlie Kaufman and also the writing staff of NBC’s “The Office.”

Thanks to Katie for finding the film. We are looking forward to the sequel on quant pre-testing!
 


Posted by Ed Cotton