More survey nonsense

Nearly half of departing Health P.E.I. employees who took part in exit surveys as part of an effort to improve staff retention cited a “toxic workplace” as one of their main reasons for leaving the provincial health authority.

That information comes from an internal report on retention that consultant Garth Waite prepared for Health P.E.I.

CBC News obtained the report through a freedom of information request.

Health P.E.I. gave Waite the names of 55 employees who left jobs with the health authority between March 2020 and January 2021. Of those people, 31 completed exit surveys that became the basis of his report. 45% of departing Health P.E.I. staff surveyed say ‘toxic workplace’ main reason for leaving

Ignoring the fact that the CBC is resorting to sensationalist misleading headlines to gain ‘clicks’, I find it hard to believe that an organization full of smart educated people like Health PEI would rely upon a simple survey to try and gain insight to the deluge of people leaving the organization (we expect the knuckleheads at Ch’town city hall to use these methods, not health professionals). With peoples lives at stake one would expect Health PEI would try harder. Is it not common practice for organizations of this size to have an even an exit interview? It was a requirement for every organization I worked for for the past 25 years.

I assume the consultant hired to do this work was specifically instructed to use this method, because his background would illustrate he understands how to gain an in-depth understanding of how people come to understand, act and manage their day-to-day situations in particular settings. Something a survey would be hard to provide.

Responsive interviews

Responsive interviewing is yet another type to add to the vast typology (including: naturalistic, feminist, oral history, in-depth, free, organisational, culture, investigative, epistemic, agonistic, platonic, phenomenological, ethnographic, walk along (go along), informal, long, standardised, convergent, socratic, concept, clarification, open, theory elaboration, exit interview, evaluation, problem centred, walking interview, unstructured via U of Amsterdam .)

I like this explanation.

In ordinary conversations, people mostly focus on the immediate outcome—how was the date, who won the game. Qualitative researchers are more likely to look at events as they unfold over time, looking at chains of causes and consequences and searching for patterns—not just what happened at the last city council meeting but how council members make decisions or how citizens become engaged in public issues. In qualitative interviews, researchers seek

In qualitative interviews, researchers seek more depth but on a narrower range of issues than people do in normal conversations. Researchers plan interview questions in advance, organizing them so they are linked to one another to obtain the information needed to complete a whole picture. Instead of chatting on this and that, a researcher has to encourage the interviewee to answer thoughtfully, openly, and in detail on the topic at hand. If a researcher heard that a community group had held a meeting, he or she would want to know who was there, what was said, and what decisions, if any, were made. He or she would want to know the history of the issues, the controversies, and something about the decision makers—who they were, their concerns, their disagreements. The researcher might want to know about the tone of the meeting, whether anyone got angry and stomped out, or whether people laughed and generally seemed to be having a good time. This depth, detail, and richness is what Clifford Geertz (1973) called thick description . To get such depth and detail, responsive interviewers structure an interview around three types of linked questions: main questions, probes, and follow-up questions. Main questions assure that each of the separate parts of a research question are answered. Probes are standard expressions that encourage interviewees to keep talking on the subject, providing examples and details. Follow-up questions ask interviewees to elaborate on key concepts, themes, ideas, or events that they have mentioned to provide the researcher with more depth. Overall, qualitative interviewing requires intense listening, a respect for and curiosity about people’s experiences and perspectives, and the ability to ask about what is not yet understood. Qualitative interviewers listen to hear the meaning of what interviewees tell them. When they cannot figure out that meaning, they ask follow-up questions to gain clarity and precision.
Qualitative Interviewing – The Art of Hearing Data

One cannot observe everything closely, therefore one must discriminate and try to select the significant. When practicing a branch of science, the ‘trained’ observer deliberately looks for specific things which his training has taught him are significant, but in research he often has to rely on his own discrimination, guided only by his general scientific knowledge, judgment and perhaps an hypothesis which he entertains.

Powers of observation can be developed by cultivating the habit of watching things with an active, enquiring mind. It is no exaggeration to say that well developed habits of observation are more important in research than large accumulations of academic learning.

Training in observation follows the same principles as training in any activity. At first one must do things consciously and laboriously, but with practice the activities gradually become automatic and unconscious and a habit is established. Effective scientific observation also requires a good background, for only by being familiar with the usual can we notice something as being unusual or unexplained.
The Art of Scientific Investigation

Books on Interviewing

I sent out this short list of reading material which goes into more detail about what I often talk to people about. My knowledge of interviewing is not so much on the science but how to present yourself (acting) to others so that you can get answers to your research question(s). These books cover just about all you need t know. My favourite is Mental Models by Indi Young – it’s concise and easy to understand. Indi Young and Erika Hall are just about my favourite people working in design right now. Erika Hall is one of the few reasons why I still use twitter.

Interviewing Users: How to Uncover Compelling Insights

Mental Models ch. 4, 5, 6, 7, 8

The User Experience Team of One

Qualitative Research: A Guide to Design and Implementation

Columbo interview technique

Kara Pernice introduces 3 techniques for facilitating user tests: Echo, Boomerang, and Columbo. Echo and Boomerang are fairly common and many would use these consciously or unconsciously, but the Columbo method is something I can’t remember consciously using in the past (silence is usually an effective technique to get people talking, as is talking about unrelated topics at the beginning of a session). It’s been an entertaining deep dive reading more about this method, it’s surprisingly used across a number of disciplines, but most enjoyable of all is watching old episodes of the TV series for which it is named after. You can learn a lot about interviewing people just by watching that show alone.

Changing minds has a simple introduction to the method.

Charlottetown short-term rental survey

But the city’s manager of planning Alex Forbes said the survey, written by city staff, isn’t trying to steer respondents to any particular conclusions.

“We attempted to write the survey in a manner that didn’t necessarily indicate that this was a preferred direction. It was to try to get all sides of the issue,” Forbes said.

As to the lack of an option to suggest the city limit short-term rentals to owner-occupied residences, Forbes said the goal wasn’t to spend “reams and reams of time” developing a “perfect” survey.

Alex Forbes gives the classic excuse when facing criticism over his departments research effort. The effort looks to be very weak and poorly written.

I would be the first to admit that I am not an expert in this particular form of research but I have prepared enough research plans to know when I see a weak effort. Personally, I have found the utility of surveys to be rather limited – you get so much more useful data from qualitative methods – and thus I can count the number of times I have written a survey on one hand.

I do know they have value, like other quantitive methods, in their ability to draw some general conclusions.

With an issue as important as this I would expect that the people who produced this survey, who may very well not have much research expertise, might do what many designers or engineers do when faced with a task for which they are unfamiliar, study and ask for help (hello stack overflow). We don’t all have the luxury of focusing on any one skill for an extended period of time, so the willingness to learn as you go, in order to best serve the goals of your project is essential. And as with any research effort, it’s also helpful to gain feedback from various stakeholders, and do a trial, to test the effectiveness of your research design.

Here are a few links which they could have started with, which took me all of 5 minutes to find on google:

How to create a quantitative research questionnaire correctly

28 Tips for Creating Great Qualitative Surveys

The Difference Between Quantitative vs. Qualitative Research

Good Practices in Survey Design Step-by-Step

Fundamentals of Survey Research Methodology

… go and sit in the lounges of luxury hotels and on the doorsteps of the flophouses. Sit on the Gold Coast settees and on the slum shakedowns. Sit in the Orchestra Hall and in the Star and Garter Burlesque. In short, go and get the seat of your pants dirty in real research. Robert Park

In an ideal world I could split my days between “making” and collecting qualitative research data from observation and interviews.

To learn something new requires interviewing, not just chatting. Poor interviews produce inaccurate information that can take your business in the wrong direction. Interviewing is a skill that at times can be fundamentally different than what you do normally in conversation. Great interviewers leverage their natural style of interacting with people but make deliberate, specific choices about what to say, when to say it, how to say it, and when to say nothing. Doing this well is hard and takes years of practice.
Interviewing Users

Leave Your Own Beliefs Behind

In Dan Saffer’s book, Designing for Interaction: Creating Smart Applications and Clever Devices, he interviews well-known researcher and speaker Brenda Laurel, PhD. He asked her what designers should look for when doing research. Her answer emphasizes the importance of shedding assumptions and precepts before asking research questions.

“The first step is to deliberately identify one’s own biases and beliefs about the subject of study and to ‘hang them at the door’ so as to avoid self-fulfilling prophecies. One must then frame the research question and carefully identify the audiences, contexts, and research methods that are most likely to yield actionable results. Those last two words are the most important: actionable results. Often, the success of a research program hangs upon how the question is framed: ‘why don’t girls play computer games?’ vs. ‘how does play vary by gender?’”
Indi Young

How to ask better questions

Some simple good advice which can be applicable to user interviews as well. From the video:

#1 Can it be answered quickly?
#2 Build up from easy questions
#3 Give examples
#4 Don’t ask questions you could ask Google
#5 Don’t ask broad questions

Exploratory research

I was looking for this explanation earlier this week. I literally forget everything if it isn’t some db somewhere.

Exploratory research is an approach where the researchers are not seeking full answers or conclusive evidence. They try to find information about the research problem and to gain more knowledge about the researched phenomena. An exploratory approach helps the researchers to better understand the field they are studying. This approach gives a better understanding of the problem but it does not give the one and only solution to the research question. It leaves room for further research and multiple solutions. The researchers have to be willing to change the way of their thinking if this approach gives them solutions towards totally new ideas.
(Research Methodology, 2016.)

Women & Mobile: A Global Opportunity

Women & Mobile: A Global Opportunity is a study on the mobile phone gender gap in low and middle-income countries.

Mobile phone ownership in low and middle-income countries has skyrocketed in the past several years. But a woman is still 21% less likely to own a mobile phone than a man. Closing this gender gap would bring the benefits of mobile phones to an additional 300 million women. By extending the benefits of mobile phone ownership to more women, a host of social and economic goals can be advanced.
The Women & Mobile report is the first comprehensive view of women and mobile phones in the developing world. This report, sponsored by the GSMA Development Fund and Cherie Blair Foundation for Women, explores the commercial and social opportunity for closing the mobile gender gap. The report builds off of a survey conducted with women on three continents to show their mobile phone ownership, usage, barriers to adoption and preferences. The report shows how mobile phone ownership can improve access to educational, health, business and employment opportunities and help women lead more secure, connected and productive lives. It also includes ten case studies highlighting the strategies and tactics that both mobile network operators and non-profit organizations across the globe are implementing to increase the usage and impact of mobile phones around the world.

Download study (.pdf). Via Putting People First.

The Smart Phone: A Ubiquitous Input Device

A paper by Rafael Ballagas, Jan Borchers Michael Rohs, and Jennifer G. Sheridan.

Mark Weiser envisioned ubiquitous computing as a world where computation and communication would be conveniently at hand and distributed throughout our everyday environment. [1] As mobile phones are rapidly becoming more powerful, this is beginning to become reality. Your mobile phone is the first truly pervasive computer. It helps you to both keep in touch with others and to manage everyday tasks. Consequently, it’s always with you. Technological trends result in ever more features packed into this small, convenient form factor. Smart phones can already see, hear, and sense their environment. But, as Weiser pointed out: “Prototype tabs, pads and boards are just the beginning of ubiquitous computing. The real power of the concept comes not from any one of these devices; it emerges from the interaction of all of them.” Therefore, we will show how modern mobile phones (Weiser’s tabs) can interact with their environment – especially large situated displays (Weiser’s boards).
The emerging capabilities of smart phones are fueling a rise in the use of mobile phones as input devices to the resources available in the environment such as situated displays, vending machines, and home appliances. The ubiquity of mobile phones gives them great potential to be the default physical interface for ubiquitous computing applications. This would provide the foundation for new interaction paradigms, similar to the way the mouse and keyboard on desktop systems enabled the WIMP (windows, icons, menus, pointers) paradigm of the graphical user interface to emerge. However, before this potential is realized, we must find interaction techniques that are intuitive, efficient, and enjoyable for applications in the ubiquitous computing domain.
In this article, we survey the different interaction techniques that use mobile phones as input devices to ubiquitous computing environments, including two techniques that we have developed ourselves. We use the word “smart phone” to describe an en-hanced mobile phone. In our analysis, we blur the line between smart phones and personal digital assistants (PDAs), such as the PalmPilot, because the feature sets continue to converge.

The Smart Phone: A Ubiquitous Input Device Paper (.pdf)

Research as a Political Tool

This is exactly what lead to my interest in user research and the formation of our user experience group at a former employer. The “value of user research is often to cut through the politics and convince stakeholders to make good design decisions”. (Adrian Chong)

Next time you read an article about a user research success story, ask yourself if the conclusions of that research weren’t just common sense (or at least common sense to good UI designers) to begin with. Ask yourself if a good designer couldn’t have concluded the same conclusion that the user research seemed to reach.
Then ask yourself if you could articulate your “common sense” recommendation to a person who doesn’t understand design at all. To someone who may, in fact, be hostile to your so-called “expert” recommendations?
This is one area where research can help: explaining a user interface design strategy to stakeholders, peers, and bosses who have their own agendas and biases.

The comments to the article are quite informative.
User Research Smoke & Mirrors, Part 3: Research as a Political Tool.

User experience: A research agenda

“Over the last decade, ‘user experience’ (UX) became a buzzword in the field of human – computer interaction (HCI) and interaction design. As technology matured, interactive products became not only more useful and usable, but also fashionable, fascinating things to desire. Driven by the impression that a narrow focus on interactive products as tools does not capture the variety and emerging aspects of technology use, practitioners and researchers alike, seem to readily embrace the notion of UX as a viable alternative to traditional HCI. And, indeed, the term promises change and a fresh look, without being too specific about its definite meaning. The present introduction to the special issue on ‘Empirical studies of the user experience’ attempts to give a provisional answer to the question of what is meant by ‘the user experience’. It provides a cursory sketch of UX and how we think UX research will look like in the future. It is not so much meant as a forecast of the future, but as a proposal – a stimulus for further UX research. ” Read the complete article. Via infodesign.