This is a decent bit of work but necessary if you want trustworthy user numbers. RULES: >12k per variant (arm) - can't use GA4 Three people that I know of have run analyses on Universal Analytics with this issue present, so they are now rerunning the numbers with the "User Metrics" flag turned off in GA (see Georgi's articles in the comments). Iqbal Ali and I also discussed this yesterday and it's important to flag this as a possible decision destroyer on GA4. ![]() Quite a few people asking me about this had not read Georgi's articles. I've been asked about this issue 5 times in the last week, so people are getting around to talking about it, even if interpretations, solutions and guidelines aren't always apparent. Lucia van den Brink ✨️also shared some useful explanations and a video explaining the problem BUT we all still seem to be fielding questions about this. He was the first to flag this issue, but it's not on everyone's radar, which is concerning. If you didn't know about this, you can read up now :-) Firstly, read Georgi Georgiev's articles on this topic (see comments), as they give important background. WARNING: If you have >12k users per variant in your AB test, you can't do reliable analysis using GA4. As Georgi's article says it is much better to use users for other reasons as well. Using sessions instead of users is not an option in GA4 since GA4 uses HLL++ for sessions as well, and at an even lower precision. This only applies to A/B testing platforms where you are using the data from GA in the testing platform.ģ. Again, also remember that 1.5% is the 95% confidence interval, that the mean error even at high cardinalities is more like half of that.Ģ. ![]() The issue is that 1.5% of 100,000 users is a lot of users when plugged into an A/B testing algorithm that is not expecting that potential variance. HLL++ is extremely accurate with cardinalities less than 12k, then after that the error levels jump up but still are relatively low (like 95% ~1.5% or so, you can see a graph here: ). I won't claim to have investigated all the math presented here, but my take here is:ġ. I'd still maintain that HLL++ in GA4 is an annoyance rather than something that really undermines the data - but it is worth highlighting that with A/B testing these estimates can cause issues. In the article I published yesterday ( ) I did link to this very research from Georgi Georgiev. This is a good discussion related to HLL++ and A/B testing. I'm ready to work on the community but only if Google reciprocates with a matched effort of their own. I firmly believe that a strong community around GA can create a feedback loop that turns it into a genre-defining powerhouse just like Universal Analytics once was. I'm hopeful for its future, not because of the tool itself but because it's possible to reinvent the community around GA. If you can't use Explorations or the BQ export (which are not impacted by cardinality), maybe it's time to look at some tool that does not impose these limitations? GA4's self-imposed cardinality limitation is _not_ a reason to sabotage your data collection. End users should stop absurd limitations bully them into submission. Be responsible experts and let your customers and followers learn about the broader world of analytics from you – the one that exists outside Google's marketing stack. Stop with the storyline that GA4 is a version update to Universal Analytics, and that the users of the latter must immediately migrate to the former or something bad will happen. It just makes them look like gullible sycophants. Agencies and partners must stop with the migration threats. It's not because we can't build a community ourselves – it's because we need to see Google taking a stronger role and responsibility in showing that the community's efforts are noticed and appreciated. We desperately need strong advocacy and community leadership from Google. ![]() Far too many obvious bugs emerge (and re-emerge) in the tool, when they should have been caught at the earliest stages of QA. Google should stop delegating quality assurance to its users. In the article, I address some of the stakeholder groups I think have the biggest possibility of enacting change. Google used to be *so good* at this! I don't know what's happened, but it seems like the Google Analytics community today comprises hundreds of frustrated users writing about bugs and workarounds rather than innovating and playing with the capabilities of the tool itself. I personally believe that the way to make GA4 a viable contender among modern digital analytics tools is to start with the community. I was in a ranty mood, so I wrote this article to reflect on my own experiences of working with Google Analytics.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |