What is a prototype?

To quote Wikipedia “A prototype is an early sample, model, or release of a product built to test a concept or process.”

For the purposes of this discussion, I will distinguish a prototype from a first-iteration or minimum-viable product (MVP).  The primary purpose of a prototype is to “test a concept” while the other two have greater emphasis on producing reusable code.

A software prototype is often executable code, but it could also consist of an interactive wireframe or other construct. You’ve probably written a few yourself.

What are prototypes good for?

The purpose of a prototype is to test a concept or process. Therefore, it seems obvious that they are best used when you need to learn:

1. Can this algorithm solve the problem?
2. Is this interface concept easy for users to learn?
3. Will database X satisfy our use case?

Use a prototype when your primary goal is learning. When you are fairly confident in the answer already, other approaches such as iterative development are more appropriate.

Prototype like a star

The following 5 steps can help you get the most value out of prototyping, with the least time and effort.

Establish the question

Clearly articulate what you hope to learn or demonstrate through the prototyping process.  Rather than a vague question, specify a specific measurement whenever possible. 

If you are unsure of the question you are asking, it’s premature to start on a prototype.  A whiteboard or shared Google doc should be your next step.

It is ok to have multiple related questions, so long as they are each clearly articulated. 

Examples of measurements:

1. How tightly can algorithm X control process Y?
2. How long do users take to complete task X with the new user interface concept?
3. How quickly can database X retrieve query Y given the proposed schema?

How it Helps
Having a specific goal in mind will let you or your team member determine the fastest way to get an answer. 

Bonus: Establish a binary hypothesis

A testable hypothesis is even better than measurement alone.  Usually a hypothesis can be created by applying a threshold to the measurement.  For example:

1. Can algorithm X control process Y within tolerance Z?
2. Can users complete task X with the new user interface concept at least 20% faster than they do with the current user interface?
3. Can database X retrieve query Y within time Z given the proposed schema?

In all these cases, the measurement implied should still be made and documented. However we have also added a pass/fail criteria related to some outcome we care about.

How it Helps
Testing a specific hypothesis has two advantages over measurement alone:

1. It forces you to consider the pass/fail criteria before you know what the outcome will be. This makes you and your stakeholders less susceptible to the sunk-cost fallacy.  
2. It identifies the region-of-interest for the measurements. If the outcome is a large margin on either side of this decision point, the exact value becomes irrelevant and you won’t need to spend time pinpointing it.

Create a plan

Now that you have a goal, take a moment to plan the best way of achieving it.  It’s best to first consider a few options. 

At the brainstorming phase “plans” may only be a few words on a whiteboard, but they should still have two parts:

1. The Prototype – what will the prototype be? What does it do and how is it built?
2. The Experiment – how will you use the prototype to answer the question or hypothesis?

Assuming you came up with a few options, the two primary factors in the selection should be:

1. Will I be sufficiently confident in the result?
2. How long will this take (time/effort)?

A prototyping effort is wasted if it doesn’t answer the question with enough confidence to make some decision.  Avoid prototypes that have a significant risk of failing to answer the question.  Of the remaining options, the fastest is almost always the correct choice. By definition, any other option involves spending time/effort/money on something other than your primary goal.  

Now that you have a high-level plan, take a moment to elaborate it slightly. Formalize the hypothesis and test conditions.  If your test involves any sort of statistical analysis, make sure you consider your sample sizes and analysis process before you start building and measuring.  This work has to be done anyway, but doing it up-front reduces the risk of an oversight that could invalidate your results later.

Example:
I recently asked for some prototypes demonstrating whether a certain web page could be rendered in under 1 second, using different technologies.  However, we jumped in a little too quickly and forgot to define some of the test conditions. This resulted in inconsistent measurements which had to be repeated. Worse, if we hadn’t noticed the inconsistencies, it could have lead to incorrect decisions.

Now you just need to execute your plan. I do have a couple of suggestions to get there faster and avoid common pitfalls:

Build the prototype quickly

When building the prototype, time is of the essence. In most cases, accuracy is also important since a misleading result is worse than none at all. The one thing you usually should not worry about is reusability.

While I don’t advocate militantly throwing away prototype code, it’s equally important not to code for the future. Given that the purpose of a prototype is to learn, there’s a good chance that the path being evaluated will not be taken and any effort to make it reusable will be wasted.

Run the experiment slowly

The actual experiment is the moment of truth. Often this takes far less time than building the prototype did. So take your time and do it right.

Meticulous notes and attention to detail during the experiment can both verify the integrity of the experiment itself and reveal oversights in the experimental design or the prototype.  Don’t write off unexpected results, record them carefully and with context.  While finding problems at this stage sucks, it’s much better than realizing your mistake after you’ve committed to a bad decision.

Document your results

The outcome of a prototype is learning, so document what you learned!

The level of documentation should be appropriate to the complexity of the prototype and the type of information.  

• Small prototypes that generate a decision you act on right away need less documentation.
• Large prototypes or outcomes likely to apply to future projects should be documented more carefully.

In all cases, you should leave enough documentation and artefacts for someone to repeat your experiment and reach the same conclusion. I recommend a two part strategy for documentation:

1. Clear Summary: Spend time to capture the essence of what you learned very clearly.
2. Data Dump: Store all your code, data, and working notes in whatever format you have handy.  I’ve been known to attach photos from a paper notebook and zip files with code that I’m not proud of. This costs almost nothing and is still likely to be “good enough” if you ever do need the data.

Put it into action

Starting tomorrow, you can

1. Scan your roadmap: are there any significant questions you need to answer?
2. Would a prototype be a good way to answer the question? Start planning it now.

Over the next month, you can

1. See how quickly you can answer the question above? Can the prototype be completed in 2 weeks? 2 days?
2. What worked well for you?
3. What didn’t?

Use the comments section below to let us know!

The forum is moderated and your first post may not appear until it has been manually approved.