Mistakes in A/B Testing: Guide to Failing the Right Way

Failing fast and often will help you learn from your mistakes sooner rather than later. This is an advice you hear often from successful product managers. But what you hear less often is that not every failure is a successful learning experience.

A product’s success is largely dependent on coming up with a hypothesis and designing the right tests. Without those elements, you might draw the wrong conclusions and steer your project in the wrong direction.

In his guest post for the GoPractice blog, Ethan Garr, VP of product at TelTech.co, shares some hard-earned experience in product testing. Through concrete case studies, Ethan shows us how to avoid key pitfalls when designing tests for hypotheses.

P.S. If you want to learn how data can help you build and grow products, take a look at Simulator by GoPractice!

Continue reading “Mistakes in A/B Testing: Guide to Failing the Right Way”

Peeking problem – the fatal mistake in A/B testing and experimentation

You can make many mistakes while designing, running, and analyzing A/B tests, but one of them is outstandingly tricky. Called the “peeking problem,” this mistake is a side effect of checking the results and taking action before the A/B test is over.

An interesting thing about the peeking problem is that even masters of A/B testing (those who have learned to check if the observed difference is statistically significant or not) still make this mistake.

P.S. If you want to learn how data can help you build and grow products, take a look at Simulator by GoPractice!

Continue reading “Peeking problem – the fatal mistake in A/B testing and experimentation”

Growth expert Sean Ellis joins GoPractice team

Recently, Sean Ellis, entrepreneur and the author of Hacking Growth, joined our team on GoPractice. Sean brings years of invaluable experience in product management to our fast-growing community. But what is even more fascinating is what made him interested in Simulator in the first place and convinced him to help make it an even better experience. Here is the journey that led Sean to GoPractice.

P.S. If you want to learn how data can help you build and grow products, take a look at Simulator by GoPractice!

Continue reading “Growth expert Sean Ellis joins GoPractice team”

iOS 14 & IDFA & Attribution: A Global Change in the Mobile Advertising Market

The mobile industry is undergoing one of the most fundamental changes of recent years. Apple has decided that early 2021, app developers will no longer have access to IDFA by default.

IDFA is a unique device identifier used for ad attribution, retargeting, alike audiences, analytics and other tasks. After the change, in order to receive the IDFA, an app developer must explicitly request the user’s permission (which is similar to allowing push notifications in an app). According to various estimates, the share of users who will provide access to their IDFA doesn’t exceed 10%.

Apple has provided privacy-friendly alternatives for attribution, but they fail to cover even a small fraction of the tasks that teams working on developing and promoting mobile apps currently have.

This shift means that mobile marketing (estimated at $80 billion), and by extension the mobile industry, are about to change drastically. In this essay, we will discuss in detail what will change, how it will affect the main players in the mobile advertising market such as developers, ad systems, attribution service providers, and advertisers.

P.S. If you want to learn how data can help you build and grow products, take a look at Simulator by GoPractice!

Continue reading “iOS 14 & IDFA & Attribution: A Global Change in the Mobile Advertising Market”

Key product management quotes from Zero to One

There are a few books that I consider must-read for anyone working on products. Today I’ll be talking about my favourite ideas from Peter Thiel’s iconic book Zero to One: Notes on Startups, or How to Build the Future

Here’s what the scholar Nassim Taleb has to say about Zero to One: “When a risk taker writes a book, read it. In the case of Peter Thiel, read it twice. Or, to be safe, three times. This is a classic.”

Taleb’s remark on Zero to One surprised me. And here is why.

At this point, I was already familiar with a few books written by Nassim Taleb. I had also read (for a few times) Thiel’s class notes that later became the manuscript for Zero to One. I really liked the harmony and sequence of thoughts in each of these books, but at the same time for me the perspectives on life of the authors seemed rather opposite.

One way or another, Zero to One is beyond amazing. I enjoy re-reading it every once in a while, and each time I find something new. And this is exactly what sets aside great books from good ones.

P.S. If you want to learn how data can help you build and grow products, take a look at GoPractice! Simulator.

Continue reading “Key product management quotes from Zero to One”

How to forecast key product metrics through cohort analysis

Forecasting the dynamics of revenue, audience, and other key metrics is an important process for any product that is in its growth phase. Having a good forecast helps to prioritize projects at the planning stage, and then helps to keep track of how quickly you are growing against the forecast, allowing you to spot problems as early as possible.

The very process of creating a forecasting model allows you to synchronize the team in terms of understanding the product’s growth model. It also provides a tool for assessing the impact of working on different areas of the model.

Today we will talk about building audience and revenue forecasts for your product using cohort analysis. We will also find out the pitfalls and difficulties of this process.

P.S. If you want to learn how data can help you build and grow products, take a look at GoPractice! Simulator.

Continue reading “How to forecast key product metrics through cohort analysis”

Why every team member should know the key product metrics

When I worked at Facebook, the Workplace analytics team had a cool tradition: The team’s weekly meetings always started with a small data quiz.

The winner of the previous week’s competition would prepare a question about the product’s key metrics. For example, “what was last month’s MAU?” or “how many new users joined last week?” or “what proportion of the new companies reach 10 users?” or “what was last month’s revenue?” The question had one requirement: Its answer had to be found on the team’s dashboard.

The participants were to write down the answer without getting help from computers, which meant we could only use our memory to do so. The person whose answer was closest to the correct number got +1 point in the chart, and the person who was the farthest lost 1 point. Every six months, a winner was chosen and the game started again.

I participated in five seasons and won three of them. In one of the final rounds, I was tied with another analyst. The team arranged the final round, where we had to answer five questions in a blitz quiz. I managed to score the winning point and won the mug that you see in the photo below.

I told this story not because I wanted to brag about winning the quiz (well, this too, to be honest). In almost every quiz, the respondents’ guesses on metrics were distributed across a wide range, which I found surprising.

Why? Well, first of all, it was the analysts who played the game. They were the people who worked with data most of their time and should have been good at navigating it. Second, these analysts were working at Facebook, a company that has a very advanced and strong data culture. At Facebook, each team has clear goals, dashboards are available to all the company’s employees, and all meetings start with progress updates on key metrics. How could these people be so wrong in answering questions about the product they were working on?

If you decide to play this game with your company’s employees, you will most likely be as surprised as I was. It will turn out that most people have very vague ideas about the key metrics of your product and business. And some people will have no idea at all.

In this essay, we will discuss why it is important for team members to remember at least approximate values ​​of the key product metrics, why this usually doesn’t happen, and how to get there.

P.S. If you want to learn how data can help you build and grow products, take a look at GoPractice! Simulator.

Continue reading “Why every team member should know the key product metrics”

Growing your company through launching a successful product for new audiences

In the previous essays of this series, we discussed how established companies and products can grow by entering new markets through movement into adjacent dependent segments in the value chain and building new products for an established user base.

Today we will talk about growing through expanding your product to new audiences. Here, the most typical growth paths are geographical expansion and movement up and down the market (B2C – SMB – Mid Market – Enterprise).

We will explore several examples where companies have successfully—and at times, not so successfully—used this growth path. We will also try to define a set of questions that will help to make a more informed decision about choosing this vector for a particular company.

P.S. If you want to learn how data can help you build and grow products, take a look at Simulator by GoPractice!

Continue reading “Growing your company through launching a successful product for new audiences”

Long-term retention – the foundation of sustainable product growth

The most common metrics gaming companies focus on are Day-1, Day-7, and Day-30 retention rate. While these metrics are of great help early in the journey, it’s long-term retention which is key to lasting success and a seat in the top-grossing charts. This post makes a case for long term-retention and why your focus should be first and foremost there.

P.S. If you want to learn how data can help you build and grow products, take a look at Simulator by GoPractice!

Continue reading “Long-term retention – the foundation of sustainable product growth”

How the backlog issues prioritization saved the company from closure during the pandemic: Stories of launching three products

I first met Vitaly almost five years ago in Palo Alto when he was participating in an acceleration program at 500 Startups with a product called Concert With Me (an event recommendation service). Over the next two years, the service reached a multimillion dollar annual revenue, but then shut down due to unexpected changes in Facebook’s policies. Then the team created a new go-to-market strategy and released Tendee, a SaaS marketing automation product for the event industry. It met a product-market fit in Europe and the US and brought several large customers and reached dozens of thousands of dollars in MRR. But it had to be put on hold due to the Covid19 pandemic.

In need of a survival plan, the team decided to introduce to the world the prioritization tool they had used within the company for almost two years.

The new product led to striking results, as well as some curious realizations and insights into how sharply the experience of product development differs in conformity with the added value level and market type.

I got in touch with Vit a week ago, and he shared his story with me. It was a gripping conversation, so it occurred to us we could develop it into an article where Vitaly would share his experience in the product development process, as well as reflections on the fundamental differences between the launch of his latest backlog-grooming product (Ducalis.io) and previous event marketing products.

P.S. If you want to learn how data can help you build and grow products, take a look at Simulator by GoPractice!

Continue reading “How the backlog issues prioritization saved the company from closure during the pandemic: Stories of launching three products”