Requirements for django A/B testing tool
May 24, 2009 – 22:38Thought recently about an A/B testing tool to use in our django projects…
My requirements are:
- Survive model changes. That’s other version of the app under test may use a different schema or totally different db. I guess that automatically rules out plain middleware approach..
- May target specific category of visitors/users. Say, only staff or just those, who opted-in for a beta testing or just US visitors, et cetera.. That excludes simple nginx/fastcgi balancers.
- Calculate OEC (Overall Evaluation Criterion) and the level of statistical significance.
My links on the topic:
- General introduction Controlled Experiments on the Web: Survey and Practical Guide
- A thread on django-users
- Another thread on stackoverflow
- Simple middleware-based approach that works by switching templates.
