Testing & Iteration. Find out what is true, then change the product.

Warm-up: what do you already believe?

Before reading, answer three questions from instinct. Do not look anything up. The point is to surface the beliefs you carry into testing, because the most common reason solo creators never test is a quiet assumption that it needs a lab, a budget or a team. Your answers will be revisited at the end of Part two.

The person who built a product cannot un-know how it works, so their sense of "obvious" is broken; the only cure is to watch people who do not share that knowledge, and to trust what they do over what anyone says.

In 1990 a Stanford graduate student, Elizabeth Newton, ran a simple experiment. One person tapped the rhythm of a well-known song like "Happy Birthday" on a table; another person listened and tried to name the song. Before the listeners guessed, the tappers predicted how often they would succeed: about half the time. The real figure was 3 songs out of 120, about 2.5%. The tappers could hear the melody in their heads while tapping. The listeners heard knocking. Chip and Dan Heath popularized the study in Made to Stick (2007) as the clearest picture of the curse of knowledge: once you know something, you cannot imagine not knowing it.

You are the tapper. You hear the melody of your product every time you look at it. You know that the small grey icon top-right opens sync settings because you drew it. You know that "Collections" means tagged groups because you named the database table. You know that swiping left archives because you wrote the gesture handler. A first-time user hears knocking.

Why your intuition fails in a specific way

The curse of knowledge does not make you wrong about everything. It makes you wrong in a predictable direction: you underestimate how hard discovery is, how ambiguous your labels are, and how much people skim. You tend to be right about what is technically possible and wrong about what is noticeable. That is useful, because it tells you where to point your tests: at discovery (can they find it?), comprehension (do they know what it means?), and expectation (does it do what they thought it would?).

There are three other biases stacked on top. Familiarity: you have seen each screen hundreds of times, so problems feel normal. Effort justification: you spent a week on a feature, so you want it to be good. Confirmation: you notice the one user who loves the feature and forget the six who never opened it. None of these go away with experience. Senior designers test because they know they have them too.

The evidence ladder: do > say

Not all evidence is equal. Jakob Nielsen's blunt 2001 article "First Rule of Usability? Don't Listen to Users" made the point: what people say they do, and what they say they want, is a poor predictor of what they actually do. People are not lying. They are reconstructing, guessing and being polite. The ladder below ranks evidence for a solo creator by how much it should move your decisions.

Two refinements make the ladder practical. First, combine levels: numbers (level 6) tell you where people drop off; a usability test (level 5) tells you why. Second, past beats future. "Tell me about the last time you exported a document" (level 4) is far more trustworthy than "Would you use an export feature?" (level 3), because one is a memory and the other is a prediction. Rob Fitzpatrick's The Mom Test (2013) is built entirely on this rule.

In the wild

• Superhumanemail client, 2018 onward
  Rahul Vohra described using Sean Ellis's question, "How would you feel if you could no longer use this product?", and treating the share of "very disappointed" answers (with 40% as the benchmark) as a product–market-fit signal, then interviewing the segments behind the numbers. Even a survey can be anchored to a concrete loss rather than a vague "do you like it".
• Steve Krug's "Don't Make Me Think"book, 2000; 3rd ed. 2014
  Krug built a whole philosophy on watching ordinary people use websites and noticing that they scan, satisfice and muddle through rather than read and choose. The only way to see muddling is to watch; nobody reports it.
• Any app store review sectioniOS and Android
  Read your one-star reviews next to your own feature list. The features you are proudest of rarely appear; the problems that hurt are usually ones you stopped noticing months ago. Strangers report the knocking you cannot hear.

A handful of people doing realistic tasks reveals most of the big, common problems in a design, so many tiny rounds with fixes in between beat one large study; but "five is enough" is a rule about common problems in one flow, not a law.

In 1993 Jakob Nielsen and Thomas Landauer modeled how usability problems are discovered. If each problem has a chance L of being hit by any one participant, then after n participants you have found a share of 1 − (1 − L)ⁿ. Across the projects they studied, the average L was about 31%. Plug that in and five users find roughly 84% of the problems. The curve rises steeply and then flattens: user six mostly repeats what users one to five already showed you.

Nielsen's 2000 article "Why You Only Need to Test with 5 Users" turned this into a recommendation: rather than spend your budget on 15 users at once, run three rounds of five, fixing the problems between rounds. The second round tests your fixes and finds the problems the first set was hiding (you cannot discover that step three is confusing while everyone is stuck on step one).

The limits of five

The formula assumes every problem has the same L, and that your five people are similar and doing the same tasks. Reality breaks all three. Laura Faulkner (2003) ran a study with 60 participants and drew random sets of five from them: some sets found most of the known problems, while unlucky sets found barely over half. Jared Spool and Will Schroeder (2001) tested large e-commerce sites and found that five users came nowhere near most problems, because each person took different paths through a huge site. The honest reading:

• Five finds the common problems in the flow you tested. Those are the ones that matter most, so it is a great deal.
• Rare problems need more people or different methods (analytics, support tickets, long-term use).
• Different audiences need their own five. Writers and editors, new and returning, phone and desktop: each group is a separate pool.
• Five is for finding problems, not measuring them. "3 of 5 failed" means "this is a real problem", not "60% of users fail." For percentages you need far larger samples.

Krug's monthly morning

Steve Krug's Rocket Surgery Made Easy turns this into a routine anyone can run: test with three people, one morning a month, debrief the same day, and fix the few most serious problems before the next round. He argues three is enough because you are looking for the problems that most people will hit, and because a fixed monthly date means testing actually happens. For a solo creator, the calendar is the method. A date on the calendar beats a plan to "test properly once the redesign is done."

1. Week before. Pick the flow to test (usually whatever you just changed or whatever support complains about). Write 3 to 5 tasks. Recruit 3 people.
2. Morning, 9:00 to 12:00. Three sessions of about 45 minutes, with 15-minute gaps to write notes while they're fresh.
3. Lunch, 12:30. Debrief yourself: list everything you saw, pick the top three problems.
4. Afternoon. Decide the smallest change that fixes each. Krug's advice is to tweak rather than redesign: the least you can do that solves it.
5. Next month. Test again, including the fixed flow.

In the wild

• GOV.UKGovernment Digital Service, 2012 onward
  GDS teams publicly documented frequent, small-sample usability testing as a routine part of building services, including with people who have low digital skills or access needs. Cadence and coverage of different audiences beat one big study.
• Krug's demo test"Rocket Surgery Made Easy", 2010
  Krug shows a complete recorded session to illustrate how much one 20-minute test reveals to people who have never watched one. The first session you watch changes how you design; the point is to start.

You need people who can plausibly do the task and who do not know your product's internals; you rarely need perfect matches, so recruit loosely, screen for the few traits that change behavior, and build a standing list so next month is easy.

Krug's advice is to "recruit loosely and grade on a curve": unless your product requires specific domain knowledge, almost anyone who uses similar software will reveal the big problems, and you mentally adjust for people who are less experienced than your real users. What matters most is that participants are not you: not designers or developers who will critique instead of use, and not people who already know how the product works.

The traits worth screening for are the ones that change behavior: Do they do the job your product is for (take notes, track money, read long articles)? Which platform do they use daily? Have they used your product before (and do you want that)? Everything else, such as age or job title, matters far less than founders tend to think.

Channels, ranked for a solo creator

A screener in five questions

A screener is a short form that filters participants. Write it so the "right" answer is not obvious, or people will tell you what you want to hear.

Incentives and courtesy

Pay people, or give them something of real value. A gift card, a year of your product, or a donation in their name all work. Amounts vary by country and audience: a small voucher works for a 20-minute consumer session, while professionals (doctors, lawyers, engineers) expect considerably more. Paying is not only fair; it changes the relationship from a favor to a job, which makes people less polite and more honest. Send the thank-you within 24 hours.

In the wild

• Continuous interviewingTeresa Torres, "Continuous Discovery Habits", 2021
  Torres recommends automating recruitment, for example with an in-product prompt that invites users to book a short call, so that talking to customers every week does not depend on a fresh recruiting effort each time. Automate the pipeline and the habit survives busy weeks.
• Beta programsTestFlight on iOS
  Many indie iOS apps keep a TestFlight beta group of engaged users who opted in. A beta group doubles as a recruiting list for tests of new features, though not for first-run tests.

A good task gives a realistic goal and the motivation behind it, in the user's words, and never names the button, label or path you want them to find; a bad task is an instruction in disguise.

The task is the most powerful lever in a usability test, and the easiest to ruin. If your task says "Use the Collections feature to group your notes," you have told the participant the feature exists, what it is called and roughly what it does. They will scan for the word "Collections," find it, and succeed. You will learn that people can match words. You will not learn whether anyone would ever think to group notes, or where they would look.

A good task is a scenario: a short story with a goal and a reason, like real life. "You're preparing for three exams next week and your notes for each subject are mixed together. Get them organized so you can revise one subject at a time." Now the participant has to translate a real need into your interface, which is exactly what real users do.

Bad and good pairs

More rules for tasks

• Plant the data. If the task is "find the article about sleep," make sure a test account contains it, among realistic decoys.
• Order tasks like real life: sign up before you organize. But put a task you most care about early, in case time runs out.
• Allow multiple right paths. Write down what counts as success before the session, not during.
• Hand tasks over on paper or in chat so the wording is identical for each person and they can re-read it.
• Let participants invent one task: "What would you normally do with an app like this? Show me." Their own task is often the most revealing.
• Pilot the tasks on anyone (even a friend) before the real session, to catch ambiguous wording.

In the wild

• Krug's sample test script"Rocket Surgery Made Easy", 2010
  Krug provides a word-for-word script, including how to hand over tasks on separate slips so each participant hears exactly the same thing. Consistency across sessions is what lets you compare them.
• Nielsen Norman Grouparticles on task writing
  NN/g repeatedly warns against tasks that contain words from the UI, because participants "word-match" instead of thinking. Leaking a label turns a test of understanding into a test of reading.

Your job in the session is to make the participant comfortable thinking out loud, then get out of the way: neutral prompts, no hints, no defending, and every question answered with a question.

Thinking aloud is the core technique: you ask participants to say what they are looking at, what they are trying to do and what they expect, as they go. It was brought into usability work at IBM by Clayton Lewis in the early 1980s, and Nielsen has called it the single most valuable usability engineering method. It works because it externalizes the participant's mental model at the moment of action: "I'd expect this to be under the menu… no… maybe this gear?" is pure gold, and you would never get it from a later interview.

Thinking aloud is unnatural, so you have to teach it gently at the start and prompt it during the session without steering. The hardest discipline for a solo creator is silence. You built this. When they struggle, every instinct says help. Don't. The struggle is the finding.

The session shape

Phrases to use and to avoid

Remote, in person and unmoderated

Recording and consent

Ask before you record, in writing and again on the call. Tell people exactly what you record (screen, voice, face), who will see it (you), how long you keep it, and that they can stop at any time and still get the thank-you. Never test with their real private data unless they offer; prepare a demo account instead. If you want to quote someone publicly, ask separately. Under laws like the EU's GDPR and India's Digital Personal Data Protection Act (2023), recordings of people are personal data, so keep them only as long as you need them and delete them on schedule. When in doubt, take notes instead of recording.

In the wild

• Krug's test script"Rocket Surgery Made Easy", 2010
  The script opens by reassuring the participant that they can't do anything wrong and asking them to think out loud, and it is freely shared so anyone can reuse it. Borrow a proven script rather than improvising your welcome.
• Nielsen Norman Group"Thinking Aloud: The #1 Usability Tool", 2012
  The article argues thinking aloud is cheap, robust and flexible enough for any stage, and that even inexperienced facilitators get useful results. You do not need training to start; you need restraint.

A test is only worth what you write down and act on: capture behavior with timestamps, rate each problem by frequency, impact and persistence, lay the findings out so patterns jump out, and fix the few most serious ones with the smallest change that works.

Sessions produce a fog of impressions. By lunchtime you will remember the funniest moment and the participant who agreed with you, and forget the quiet hesitations that matter most. The cure is structure: one note format for every session, one rating scale for every problem, one sheet where all findings meet.

A note-taking template

Note what you saw and heard, separately from what you think it means. Mixing them is how bias creeps in.

Rating severity

Nielsen's severity scale, published in the mid-1990s alongside his heuristic evaluation work, combines three factors: frequency (how many users hit it), impact (how hard it is to overcome) and persistence (whether it bites once or every time). He also mentioned market impact, which for you means: does it touch your core promise?

Synthesis: the rainbow spreadsheet

Tomer Sharon, then a researcher at Google, described the rainbow spreadsheet in a 2013 Smashing Magazine article: one row per observation, one column per participant, a colored cell where that participant showed that behavior. It was designed for teams watching together, but it is just as useful alone, because it turns a pile of notes into a picture of patterns. Rows with many filled cells are your common problems; single cells are anecdotes worth watching.

Fixing the right things

Not every problem deserves a fix, and not every fix deserves a redesign. Krug's guidance: focus ruthlessly on the most serious problems, and prefer the smallest change that solves them (a label, a default, moving one control) over a rebuild, because tweaks are cheap to test again next month and rebuilds introduce new problems. Plot what you found on severity versus effort.

In the wild

• Rainbow spreadsheetTomer Sharon, Smashing Magazine, 2013
  Built so observers could fill it together during sessions and leave with shared conclusions, avoiding a slow report. Synthesis can be visual and fast; a report nobody reads is not the goal.
• Krug's debrief"Rocket Surgery Made Easy", 2010
  Krug recommends a short debrief over lunch where each observer names the most serious problems they saw, followed by agreement on what to fix before next month. Limit the fix list. Small lists get done.

Match the fidelity of the prototype to the question you're asking: paper for structure and concept, clickable mockups for flow and labels, coded prototypes for feel, speed and real data; higher fidelity costs more and makes people critique polish instead of ideas.

Fidelity is not a ladder you must climb in order. It is a set of tools, each good at answering certain questions. Carolyn Snyder's Paper Prototyping (2003) showed that a hand-drawn interface, with a person playing "computer" by swapping paper screens, finds many of the same problems as a working product, at a fraction of the cost. Bill Buxton's Sketching User Experiences (2007) argued that sketches invite critique because they look unfinished: people feel free to say "this is wrong" to a drawing in a way they never will to a polished screen.

What each level can and cannot tell you

In the wild

• Google Ventures design sprintJake Knapp, "Sprint", 2016
  The sprint builds a realistic-looking "facade" prototype in one day (often in Keynote) and tests it with five customers on the last day. A prototype only needs to be real enough for the questions you're asking.
• The Palm Pilot wooden blockJeff Hawkins, 1990s
  Hawkins famously carried a block of wood the size of the planned device in his pocket, "using" it in meetings to test whether the form and core tasks fit daily life. The lowest fidelity can test the biggest question: does this fit into a life?

Each quick method answers one narrow question with less effort than a full usability test; pick by question: where do they click first, what do they remember, which do they understand, where would they look, how do they group things.

Full sessions are rich but slow. Quick methods trade richness for speed and numbers: many of them run unmoderated, with 20 to 50 participants, in a day. They are best used to settle a specific design question before or between usability rounds.

First-click testing

Show a screen and a task ("Where would you click to read this article offline?"), record only the first click. Research by Bob Bailey and Cari Wolfson found that when people's first click was correct, they went on to complete the task about 87% of the time, against about 46% when the first click was wrong. The first click predicts the rest. That makes first-click tests a cheap check on labels and layout, especially navigation and home screens.

Five-second tests

Show a screen for five seconds, hide it, ask: "What was this page for? What do you remember?" It tests whether the purpose and the main hierarchy come through at a glance, which is exactly what a landing page or an empty home screen must do. If people remember your decorative illustration and not what the product does, your hierarchy is upside down (see Visual Design). It cannot test usability; nobody uses anything in five seconds.

Preference tests (with caveats)

Show two or more designs and ask which people prefer and why. Useful for tone and aesthetic direction (two icon styles, two onboarding illustrations). Dangerous for anything about usability: people prefer what looks familiar and attractive, and their preference does not predict which one they can use. Always ask a comprehension question first ("What do you think this button does?"), then the preference, and treat the "why" answers as more valuable than the vote.

Tree testing

Strip your navigation down to a text hierarchy (no visual design at all) and ask people where they would go to do a task. It isolates your information architecture from your layout. If people can't find "Offline reading" in a bare tree, no amount of icon polish will save it. Tools like Optimal Workshop's Treejack automate it; you can also do it with a nested list on paper. Watch two numbers: success (did they end in the right place?) and directness (did they get there without backtracking?).

Card sorting

Give people cards with your content or features on them and ask them to group the cards and name the groups. In an open sort they invent the groups, which reveals their mental model and vocabulary. In a closed sort you supply the group names and see whether your categories make sense. Donna Spencer's Card Sorting (2009) is the standard guide. Use open sorts early, when you're designing navigation; use tree tests afterwards to check the structure you built. For more on structure itself see Structure & Interaction.

In the wild

• Lyssnaformerly UsabilityHub, renamed 2023
  Offers five-second, first-click, preference and design-survey tests with an optional participant panel. These methods are commodities now; the skill is asking the right question.
• Optimal WorkshopTreejack, OptimalSort
  Popularized online tree testing and card sorting with success and directness metrics and similarity matrices. Test structure separately from screens.

Ten broad rules of thumb, distilled from hundreds of real usability problems, let you inspect any interface and catch many common problems without users; each one is a lens that makes a category of failure visible.

Jakob Nielsen and Rolf Molich introduced heuristic evaluation in 1990. In 1994 Nielsen refined the list by analyzing 249 usability problems and finding the set of principles that explained them best. The ten have barely changed since (NN/g refreshed the wording in 2020), because they describe how people work, not how technology looks. They also echo Norman's principles from The Design of Everyday Things: feedback, mapping, constraints, conceptual models (see Foundations and The Human Mind).

Nielsen also found that a single evaluator catches only a fraction of the problems (around a third, in his studies), and that several evaluators working independently find far more, because different people notice different things. You are one evaluator. That's a reason to use the heuristics as a structured lens, do several passes, and still test with users.

1 · Visibility of system status

The system should always tell people what is going on, through timely feedback. Every action needs an answer: "I heard you," "I'm working," "Done," or "That failed." Silence makes people tap again, wait anxiously, or leave. Rough timing guide: under 0.1 s feels instant; up to about 1 s keeps flow without a spinner; beyond a few seconds needs progress (see Motion & Feel).

2 · Match between the system and the real world

Speak the user's language: words, concepts and order that are familiar to them, not to your database. Internal names (entity, sync conflict, workspace object) leak in because you see them every day in code. Order should follow the real-world task: a recipe app lists ingredients before steps.

3 · User control and freedom

People act by mistake all the time. They need a clearly marked "emergency exit": undo, cancel, back, close, without a long process. Undo is better than confirmation dialogs, because people click through confirmations automatically but can always recover with undo (see Structure & Interaction).

4 · Consistency and standards

People should not wonder whether different words, positions or actions mean the same thing. Two kinds: internal consistency (your app agrees with itself) and external (it follows platform conventions, which Nielsen elsewhere calls Jakob's Law: users spend most of their time in other products). Follow the platform unless you have a strong reason not to (see Systems & Platforms).

5 · Error prevention

Better than a good error message is a design that prevents the error. Norman distinguishes slips (right goal, wrong action, like tapping Delete instead of Save) from mistakes (wrong goal from a wrong mental model). Prevent slips with spacing, constraints and good defaults; prevent mistakes with clear previews and explanations.

6 · Recognition rather than recall

Minimize what people must remember. Show options, recent items and examples instead of expecting them to remember commands, codes or what was on the previous screen. Working memory is small (see The Human Mind); every "remember this" is a tax.

7 · Flexibility and efficiency of use

Shortcuts, hidden from novices, speed up experts. The best designs let the same person grow: menus show keyboard shortcuts next to commands, so people learn them by using the menu. Command palettes (⌘K) serve both: novices search in words, experts type three letters.

8 · Aesthetic and minimalist design

Every extra unit of information competes with the relevant ones and reduces their visibility. This is not "make it pretty" or "make it empty": it is "make what matters win." Rams's "less, but better" and Tufte's data-ink ratio are the same idea (see Visual Design).

9 · Help users recognize, diagnose and recover from errors

Error messages should be in plain language, say precisely what went wrong, and suggest a fix, right where the problem is. Keep what the person typed. For the words themselves, see Words.

10 · Help and documentation

Ideally nothing needs explaining. When it does, help should be easy to search, focused on the person's task, short, and available in context, at the moment of need.

Practice: spot the violations

In the wild

• Gmail Undo SendGoogle, a Labs experiment from 2009, a standard setting since 2015
  A short delay before sending lets people pull back an email, turning an irreversible action into a reversible one. Heuristic 3 can be solved with time instead of dialogs.
• Command palettesSublime Text, VS Code, Linear, Raycast, Figma
  A single search box that runs any command, often listing its shortcut. Heuristics 6 and 7 at once: recognition for novices, speed for experts.
• Stripe's form validationStripe Checkout and Elements
  Card fields format numbers as you type, detect the card brand, and flag problems inline next to the field. Heuristics 1, 5 and 9 designed together reduce errors and make the rest easy to fix.

A heuristic review asks "what rules does this break?"; a cognitive walkthrough asks, step by step, "would a first-time user know what to do here, and would they know it worked?"; together they let you inspect your own product with structure instead of vibes.

The cognitive walkthrough was developed in the early 1990s by Clayton Lewis, Peter Polson, Cathleen Wharton and John Rieman, and published as a practitioner's guide in 1994. It is built on a theory of exploratory learning: new users don't read manuals; they try things that look like they lead toward their goal. So for each step of a task, you ask four questions on behalf of a new user:

A worked example

Task: "Save this article to read offline on the plane." User: a new user, comfortable with phones, never used this app. Correct path: tap ⋯ → "Keep on device".

Two concrete fixes fall straight out: rename to "Download for offline" (Q3), and show "Downloaded · available offline" with an icon on the article itself (Q4). Notice that the heuristics would have flagged these too (2 and 1), but the walkthrough tied them to a specific step and a specific user, which makes fixing them obvious.

Running an expert review on your own product

1. Define the user and tasks. Write who (new, returning, power user) and the 3 to 5 most important tasks. Without a defined user, every judgment becomes your own taste.
2. Walk each task as that user. Use a fresh install, a clean account, the smallest supported device. Ask the four questions at each step; write a "failure story" for every no.
3. Sweep the heuristics. One pass per heuristic across all screens in the tasks. Use the checklist below.
4. Rate severity. Same 0 to 4 scale as for tests.
5. Wait a day, then review your findings. Cut anything you can't defend with a specific user moment.
6. Turn the top issues into test tasks for your next usability round, to confirm they're real.

In the wild

• "The Cognitive Walkthrough Method: A Practitioner's Guide"Wharton, Rieman, Lewis & Polson, 1994
  The original guide treats the walkthrough as a way to evaluate ease of learning by exploration, especially for walk-up-and-use systems. It shines for first-run flows and anything people use rarely.
• "The Streamlined Cognitive Walkthrough"Rick Spencer, Microsoft, CHI 2000
  Spencer reported that the full method was too slow and defensive for a product team, and cut it down to two questions with ground rules (no designing during the walkthrough, no defending). A lighter method you actually run beats a perfect one you don't.

Useful critique starts from an objective ("this screen exists so that…") and asks whether each element serves it; to see your own work freshly, change the conditions you see it under, and to get useful critique from others, ask narrow questions.

Adam Connor and Aaron Irizarry's Discussing Design (2015) separates critique from feedback in general. Critique is analysis against objectives: "Given that the goal is X, this element does or does not help, because Y." Reaction ("I don't like the blue") is not critique; neither is a direct fix ("make the button bigger"), because it skips the why. As a solo creator you must critique yourself, which means writing the objective first, otherwise you will judge by mood.

The objective-first critique

Write it as a sentence with a person, a situation and an outcome. Then, for each element on the screen, ask the three critique questions: What is this for? Does it serve the objective? Is it the best way to serve it? The answers are keep, change, move, or cut. Pair this with the visual hierarchy test: is the most important element for the objective also the most visually prominent?

Fresh-eyes tricks

Your brain stops seeing what it sees every day. Each trick below breaks familiarity in a different way, so you notice what a stranger would. Artists have used some of these for centuries: painters check a canvas in a mirror because flipping it reveals lopsided composition the eye had adapted to.

Red-team your own design

For one hour, become the design's enemy. Write down every way it could fail: the user who skims, the user on a train with one hand, the user who pastes 40,000 words, the user who misunderstands the label, the user who never comes back after day one, the user who wants to leave and take their data. A good habit is a pre-mortem (Gary Klein's technique): imagine it's six months later and the feature failed; write the story of why. Then fix or test the most plausible story.

Getting useful critique online

Posting a screenshot with "Thoughts?" gets you taste reactions from people who don't know your user or goal. Asking a narrow question with context gets you critique.

Good critique questions are answerable and observable: "What do you think this screen is for?", "Where would you tap to…?", "What's the first thing you notice?", "What would you expect to happen if you tapped X?", "Which of these two labels is clearer for this task, and why?" When you receive critique, thank, ask follow-ups, and decide later. Do not defend in the thread.

In the wild

• iA WriterInformation Architects
  Oliver Reichenstein's essays on the app's design repeatedly frame decisions against one objective (help people focus on writing) and explain what was removed for it. Writing your reasoning publicly is a form of self-critique: weak arguments look weak in prose.
• Pixar's BraintrustEd Catmull, "Creativity, Inc.", 2014
  Catmull describes a group that gives candid notes on films in progress, with the rule that the director decides what to do with them. Separate receiving critique from deciding; it keeps critique honest and you in charge.

Three questions cover most of what a small product needs to know: do new people reach the moment of value (activation), do they come back (retention), and how much do they use it (engagement); everything else is either a detail of these or a vanity number.

A design change is a hypothesis about behavior. Metrics are how you see behavior at a scale no usability test can reach. But a dashboard with 40 charts is as useless as none, because you will stare at whatever moved and invent a story. Pick a small set, define them precisely, and look at them on a schedule.

Activation

The share of new users who reach the first moment of real value, defined as a specific action: "created a note and opened it again later," "imported a book and read 5 pages." Not sign-up. See Problems Worth Solving for designing that moment.

Retention

The share of a group of users (a cohort, usually by sign-up week) still active N days or weeks later. The single best sign that the product is valuable.

Engagement

How much retained users use the product: sessions per week, notes created, minutes read. Useful only alongside retention, and only when "more" is actually better for the user.

Conversion

The share of people who take a specific step: visitor to sign-up, trial to paid.

Churn

The share of users or subscribers who leave in a period; the mirror image of retention.

Funnels: where people leave

A funnel lines up the steps toward an outcome and shows how many people make it through each. Its job is to point you at the step with the worst drop, so that your next usability test focuses there.

Cohort retention curves: the truest chart

Group users by when they started, and plot what share is still active over time. Every product's curve falls; the question is whether it flattens. A curve that flattens means a core of people found lasting value: you have something. A curve that keeps sliding toward zero means people try it and leave, and no amount of acquisition will fix that. When newer cohorts' curves sit above older ones, your changes are working.

Vanity metrics

Eric Ries, in The Lean Startup (2011), called numbers that make you feel good but don't inform decisions "vanity metrics," contrasting them with "actionable" ones. Total sign-ups, total downloads and page views only go up; they can't tell you whether anything you did worked. The test: if this number changed, would I do something differently? If not, stop looking at it.

In the wild

• Facebook's early activation ruleoften-cited growth story
  Facebook's growth team is widely reported to have focused on getting new users to a certain number of friends within their first days, because users who did tended to stay. An activation metric names the behavior that predicts retention. Find yours by comparing who stayed with who left.
• "Lean Analytics"Croll & Yoskovitz, 2013
  Introduces the "One Metric That Matters" for each stage of a business. Focus beats breadth: one number at a time is something you can actually move.

Start from the user-experience goal, choose a signal that shows it, and only then a metric (Google's HEART goals–signals–metrics method); summarize the value you create in one North Star number; and measure with the minimum data, named clearly, collected respectfully.

In a 2010 CHI paper, "Measuring the User Experience on a Large Scale," Kerry Rodden, Hilary Hutchinson and Xin Fu at Google described the HEART framework: five categories of user-centered metrics, Happiness, Engagement, Adoption, Retention and Task success, plus a process, Goals → Signals → Metrics, for choosing them. The process matters more than the acronym: it stops you from measuring what's easy instead of what matters.

A North Star for a small app

A North Star metric is a single number that captures the core value users get, and that tends to lead to revenue and retention. It should count value delivered, not activity or revenue. For a reading app, "articles read to the end per week"; for a habit tracker, "users who checked in on 4+ days this week"; for a writing app, "documents edited on 3+ days in a week." A good one rises only when users are actually better off.

Naming events

Your events are a vocabulary you'll live with for years. A common convention is object + past-tense action: note_created, article_finished, export_completed. Keep a short tracking plan: event name, when it fires, properties, why you need it. If you can't write the "why," don't track it.

Privacy-respecting analytics

You can learn what you need without tracking individuals across the web. Measure aggregates, avoid personal data, keep it on infrastructure you trust, and tell people plainly what you collect. Some tools built with this in mind:

• Plausibleweb analytics, open source, EU-based
  Lightweight, cookieless page and goal analytics with a simple dashboard. Enough for landing pages and funnels without a consent banner circus (check your own legal situation).
• TelemetryDeckapp analytics, privacy-focused
  Built for app developers (popular among indie Apple-platform developers), with anonymized user identifiers and signal-based events. Product usage counts for native apps without collecting personal data.
• PostHogopen-source product analytics
  Funnels, retention, feature flags and session replay, available as a cloud service or self-hosted. Self-hosting keeps data on your infrastructure; you're responsible for configuring it to collect only what you need (mask inputs in replays).

Related principles live in Honest & Calm. The short version: collect the minimum, aggregate early, never record what people type, and write a privacy note you'd be happy for users to read.

Quantitative and qualitative together

An A/B test randomly splits people between versions so the only systematic difference is your change; it's the gold standard for causation, but detecting realistic improvements takes thousands of users per variant, so small products should mostly ship bold changes, roll out in stages and compare carefully.

Ron Kohavi, who led experimentation at Microsoft and wrote Trustworthy Online Controlled Experiments (2020, with Diane Tang and Ya Xu), has reported that most ideas tested at large companies fail to improve the metrics they targeted. That is humbling and useful: intuition about which change will help is unreliable even for experts. A/B tests protect against that. The catch is statistical power. Randomness means two identical groups will show different conversion rates by chance; to tell a real 10% improvement from noise, you need enough people that the noise is small.

Sample size intuition

A handy approximation (Lehr's rule, for 80% power at the usual 5% significance) is: users per variant ≈ 16 × p(1 − p) ÷ d², where p is the baseline rate and d the absolute difference you want to detect. Small baselines and small effects explode the number. Play with it.

Rules if you do run one

• Decide the sample size and the metric before you start, and don't stop early when it "looks significant." Evan Miller's essay "How Not To Run an A/B Test" (2010) shows how repeatedly peeking and stopping at the first significant result produces false winners far more often than the nominal 5%.
• Run full weeks, because weekday and weekend users behave differently.
• Check that the split is actually even; a lopsided split means a bug.
• Watch a guardrail metric (retention, complaints, refunds) so a "win" on clicks doesn't hide harm.

Alternatives for small products

Fake doors, honestly

In the wild

• Booking.com and Microsoft Binglarge-scale experimentation cultures
  Both are known for running very large numbers of controlled experiments, with infrastructure to do so safely. That culture depends on enormous traffic. Copy the humility, not the method.
• Evan Miller's sample size calculatorevanmiller.org
  A free, widely used calculator alongside clear essays on the pitfalls of testing. Check the numbers before you design the test, not after.

Support emails, reviews, in-app feedback and churn are a free, continuous research stream; treat them as data to tag and count, not as a to-do list, and dig beneath each request for the problem behind it.

Every message a user sends cost them effort, so it signals something real. But feature requests are proposed solutions from people who don't know your constraints or your other users. "Add folders" might mean "I can't find things," which search could fix better. The job is to collect, tag, count and then ask about the underlying problem.

The channels

Interviewing existing users

Talk to users regularly, not only when something breaks. Teresa Torres recommends weekly touchpoints with customers. The core rule is from Chapter 1: ask about specific past behavior. "Tell me about the last time you saved an article. Where were you? What happened next?" Story-based questions reveal context, workarounds and the real job. For interview technique in depth, see Problems Worth Solving.

In the wild

• Indie apps with a named makercommon pattern among small developers
  Many one-person apps sign their support replies and feedback screens with the developer's own name. Users write differently to a person than to a "team": more context, more patience.
• Superhuman's segmentationRahul Vohra, 2018
  After the "very disappointed" survey, the team focused on what the users who'd be most disappointed loved, and on what held back people who were close to that. Not all feedback is equal; weigh it by who it comes from.

Using your own product daily for real work finds problems no test will (the ones that only appear on day 40, with messy data, when you're tired); but you are a biased power user, so log your use deliberately, and complement it with tests of people who aren't you.

"Eating your own dog food" became a common phrase in software after it circulated inside Microsoft in the late 1980s. Many of the best-loved tools are made by people who use them all day: Basecamp's team runs its company on Basecamp; Linear has said it builds Linear in Linear. Daily use reveals slowness, paper cuts, and what happens after months of accumulated data, which no 45-minute session can.

But dogfooding has a blind spot shaped exactly like the curse of knowledge. You never experience first-run again; you know every shortcut; you forgive what you planned to fix. So turn it into a method: a diary, scheduled "beginner days," and a structured bug bash.

A diary of your own use

Keep a running note while you use your product for real work. Every time something annoys, slows or surprises you, write one line immediately: date, what you were trying to do, what happened, how it felt. Don't fix it on the spot (that breaks your work and biases you toward easy fixes). Review the diary weekly and move recurring items into your issue tracker.

Beginner days

Once a month, use your product as a beginner would: delete the app and reinstall it, sign up with a new email, go through onboarding without skipping, on your oldest device, with no shortcuts allowed. Do the core task from scratch. It won't fully cure the curse, but it re-exposes you to first-run, which you otherwise never see after launch.

A bug bash for one

1. Timebox it. 90 minutes before each release, phone on silent.
2. Change roles every 15 minutes. The beginner, the power user, the impatient one (tapping fast, double-tapping), the offline one (airplane mode), the accessibility user (screen reader, largest text size; see Accessible & Inclusive), the destroyer (paste huge text, emoji, right-to-left text, rotate, background the app mid-save).
3. Log, don't fix. One line per issue with steps to reproduce.
4. Triage with severity (0 to 4) at the end. Ship-blockers are 4s and data-loss risks.

In the wild

• Basecamp37signals
  The company runs its own projects in Basecamp and has written about how this shapes the product. Deep daily use is a quality engine for the things you do yourself.
• Linearissue tracker, 2019 onward
  The team has described building Linear in Linear, which helps explain its focus on speed and keyboard flow. Dogfooding pushes products toward what their makers value; make sure that's what your users value too.

Iteration is a disciplined loop: state what you expect, ship the smallest change that tests it, look at the evidence, and decide (keep going, change direction, or remove); writing each turn of the loop down is what turns activity into learning.

Eric Ries's Build–Measure–Learn loop, from The Lean Startup (2011), describes startups as machines for validated learning: build the minimum needed to test an assumption, measure what happens, learn whether to persevere or pivot. Ries points out that you plan the loop backwards: decide what you need to learn, then what you'd measure, then what to build.

The iteration log

One row per change. Write the first three columns before you ship. This is the solo version of pre-registering an experiment: it protects you from your own hindsight.

Persevere, change, or kill

Ryan Singer's Shape Up (Basecamp, 2019) adds a useful discipline: set an appetite (how much time this problem is worth) before you start, and use a circuit breaker: if a project doesn't ship within its cycle, it doesn't get extended automatically; it has to be re-pitched. For a solo creator, that means "two weekends for this; if it isn't working, I stop and rethink."

Deleting features

Every feature has a cost after it ships: code to maintain, settings to explain, screens that dilute the core, tests to run, support questions. In most products, usage follows a long tail: a few features carry most of the value. Rams's "less, but better" applies to features as much as to form.

1. Measure use and ask who. Low use by your most loyal users is different from low use by nobody.
2. Count the cost. Bugs, support questions, settings, code complexity, design constraints.
3. Announce early and honestly. Say what's going, why, when, and how to keep their data (export).
4. Remove completely. Code, settings, docs, onboarding mentions. Half-removed features haunt you.
5. Log it and watch. Note support volume and retention for a month after.

In the wild

• Instagramfrom Burbn, 2010
  Kevin Systrom and Mike Krieger's first app, Burbn, was a check-in app with many features; they noticed photo sharing was what people actually used, and stripped the product down to it. Usage evidence can tell you which small part of your product is the real product.
• Slackfrom the game Glitch, 2013
  Slack began as the internal chat tool of a team building an online game; when the game was shut down, the tool became the company. A pivot keeps what you learned and changes what you bet on.
• Shape UpRyan Singer, Basecamp, 2019
  Six-week cycles with fixed time and variable scope, appetites set before work starts, and a circuit breaker on overruns. Fixed time with flexible scope forces the trade-offs iteration needs.

Wrap-up

Testing isn't a phase; it's how you stay honest with yourself. The whole guide reduces to a monthly rhythm: watch three people, read your three numbers, count your feedback tags, review your diary, update your log, and make the few changes the evidence supports. An hour a week, sustained, will make your product better than most built by teams.

Practice assignment for the next two weeks

1. Days 1–2. Exercises 1.1 and 2.1: map your curse of knowledge, write predictions, put six test dates in your calendar. Start the dogfooding diary (16.1) today and keep it going throughout.
2. Days 3–4. Exercises 3.1 and 4.1: screener, test list, three booked sessions, five de-leaked tasks, piloted.
3. Day 5. Exercise 10.1: walkthrough and heuristic checklist on your first-run flow. Fix any obvious severity-4 issues before testing.
4. Day 6 (test morning). Exercise 6.1: three sessions, rainbow sheet, severity, three evidence-backed issues. Compare with your predictions.
5. Days 7–9. Exercises 12.1 and 13.1: activation definition, funnel, HEART rows, North Star, tracking plan. Exercise 15.1: voice-of-user log with 30 entries.
6. Days 10–12. Ship the top fixes. Exercise 17.1: start the iteration log with expectations written before shipping. Exercise 14.1 for any change you hoped to A/B test.
7. Days 13–14. Exercise 11.1 on your home screen, then a bug bash (16.1). Write a one-page "What I learned about my users" note, and book next month's three participants.

Reading list

• Rocket Surgery Made Easy
  Steve Krug, 2010 · The do-it-yourself usability testing method this guide adapts, with scripts and a demo session. Short and practical. Read it first.
• Don't Make Me Think, Revisited
  Steve Krug, 3rd ed. 2014 · How people really use screens (scanning, satisficing, muddling through) and why testing reveals it.
• Just Enough Research
  Erika Hall, 2013; 2nd ed. 2019 · Research for people without research departments: what to do, how much, and how to avoid fooling yourself.
• The Mom Test
  Rob Fitzpatrick, 2013 · How to talk to users so they can't lie to you: past behavior, specifics, no pitching.
• Continuous Discovery Habits
  Teresa Torres, 2021 · Turning customer contact into a weekly habit, with opportunity trees to connect findings to decisions.
• "10 Usability Heuristics for User Interface Design" and "Why You Only Need to Test with 5 Users"
  Jakob Nielsen, Nielsen Norman Group (nngroup.com) · The primary sources for Chapters 2 and 9. Free, short, and worth reading in the original.
• Usability Engineering
  Jakob Nielsen, 1993 · The classic textbook on discount usability; dated in examples, timeless in method.
• Paper Prototyping
  Carolyn Snyder, 2003 · Everything about testing before code, including how to play the computer.
• Discussing Design
  Adam Connor & Aaron Irizarry, 2015 · How to give, get and use critique, framed around objectives.
• "Measuring the User Experience on a Large Scale"
  Rodden, Hutchinson & Fu, CHI 2010 · The HEART paper. Short; read the goals–signals–metrics section twice.
• Lean Analytics
  Alistair Croll & Benjamin Yoskovitz, 2013 · Which numbers matter at which stage, and the One Metric That Matters.
• Trustworthy Online Controlled Experiments
  Ron Kohavi, Diane Tang & Ya Xu, 2020 · The definitive A/B testing book. Read the pitfalls chapters even if you never run a test.
• "How Not To Run an A/B Test"
  Evan Miller, 2010 · A short essay on peeking and stopping early. Ten minutes that will save you from false wins.
• The Lean Startup
  Eric Ries, 2011 · Build–Measure–Learn, vanity versus actionable metrics, persevere or pivot.
• Shape Up
  Ryan Singer, Basecamp, 2019 (free online) · Appetites, fixed time and variable scope, circuit breakers: a calm way for a small team (or one person) to iterate.