Ertürk Poyraz | Actionable Insight Derived From Data
A/B Test Mobile Game: Cookie Cats
This project is the analysis of the results of an A/B testing on the mobile game "Cookie Cats".
Table of Contents
-
About the Dataset | Kaggle
-
Overview on Excel
-
Analysis and Visualization on R
-
Conclusion
Access Data | Kaggle
Analysis and Visualization on R
Context
This dataset includes A/B test results of Cookie Cats to examine what happens when the first gate in the game was moved from level 30 to level 40. When a player installed the game, he or she was randomly assigned to either gate_30 or gate_40.
​
Content
The data we have is from 90,189 players that installed the game while the AB-test was running. The variables are:
userid: A unique number that identifies each player.
version: Whether the player was put in the control group (gate_30 - a gate at level 30) or the group with the moved gate (gate_40 - a gate at level 40).
sum_gamerounds: the number of game rounds played by the player during the first 14 days after install.
retention_1: Did the player come back and play 1 day after installing?
retention_7: Did the player come back and play 7 days after installing?
When a player installed the game, he or she was randomly assigned to either.
​
Source : DataCamp
​
Overview on Excel
After importing our dataset, we start off by loading the required packages
We have 5 features and 90190 observations.
Our dataset has no dupliactes, NULL values or N/As
gate_30 | 44700
gate_40 | 45489
There are 3994 players who downloaded and never played the game
4.4%
Retention
At this point we have a solid understanding on how our users behave:
​
-
3994 of downloaders plays the game a total of 0 times. Which is 4.4%
-
The number of people who reach level X decreases exponentially. This is true for both variations but our box plot indicates better result from gate 30
Significance
X-squared | 3.1591
df | 1
p-value | 0.0755
X-squared | 9.9591
df | 1
p-value | 0.0016
At this point we know that,
We can not reject the null hypothesis for 1 day retention. Which means we can not reject that gate placement variations have no effect on 1 day retention.
This makes sense since most players don't reach level 30 or 40 within 1 day.
Looks like this is not the case for 7 day retention. Players do reach those levels at 7 days and percentages seem to differ for gate 30 and 40. We need to test if the percentage change is significant
Conclusion
In conclusion,
The placement of a stop-by at gate 30 or gate 40 have changed the 7 day retention significantly. Since these levels are not achieved by players in just 1 day; 1 day retention has not changed significantly. The product manager's choice should be gate 30.