7 ms·

From the article "... To make GitLab better faster, we need more data on how users are using GitLab. SaaS telemetry products, which provide analytics on user be

by jfkw 7y ago

From the article "... To make GitLab better faster, we need more data on how users are using GitLab. SaaS telemetry products, which provide analytics on user behavior inside web-based applications, have come a long way in the past few years. They are an important tool for rapidly improving user experiences because you can understand what users are doing (or not doing) in the app. GitLab has a lot of features, and a lot of users, and it is time that we use telemetry to get the data we need for our product managers to improve the experience." This doesn't seem like a sufficiently good justification add telemetry features which some users may find objectionable. Perhaps those resources would be better spent on a feature that let the user who is annoyed by a slow operation to: - Begin recording user interaction telemetry locally - Perform the slow or buggy operation - Stop recording, generating a data bundle file. - Allow the user to review the data bundle in human-readable format - Optionally take the data bundle to a less secured system as needed - Submit the data bundle to GitLab engineering as a well-filed issue Done that way, I think most users would welcome the interaction with Gitlab. For telemetry, not so much.

jkdufair 7y ago
That sounds like recipe for near-zero participation. I think both the intent and execution are spot-on here for GitLab. They are offering a very straightforward option for using GitLab, telemetry-free.
kazlock 7y ago
I'm not arguing for or against tracking, but I disagree with some of these assumptions: 1) 99% of users who have a bad experience are not going to be bothered to figure out how to record, review, and submit their session for the benefit of the service provider. If a page has issues loading, they're just going to give up and move on to the next thing. 2) User tracking isn't really related to performance in the first place. If some server side operation is failing or taking a long time to load, engineers will find out (and probably get paged) with user-agnostic internal performance metrics. 3) User tracking isn't just about finding out what's broken with the site, its about understanding how the site is being used in general so that you can validate your assumptions and make product decisions backed by data.
- codedokode 7y ago
  > If a page has issues loading, they're just going to give up and move on to the next thing. It means that they are fine with these issues and don't need to have them fixed.
  - kazlock 7y ago
    Or it means they're just going to switch to $competitor's product that doesn't have this issue.
rkagerer 7y ago
You don't need telemetry or telepathy to hear the chorus of users screaming at the top of their lungs where to shove your telemetry feature.
emilycook 7y ago
Hello! Thanks for the feedback, we opened an issue here that might help clear some of this up: https://gitlab.com/gitlab-org/growth/product/issues/164 https://gitlab.com/gitlab-org/growth/product/issues/164
TeMPOraL 7y ago
Telemetry is a lazy, paternalistic way of improving user experience. It's a method that lets you ignore the actual feedback you're getting from your users, because that involves dealing with actual human beings - who don't always write politely or coherently. Telemetry lets you avoid collecting feedback in a way that respects the user - through in-house studies, by contracting with outside users to perform on-site studies, and just by asking people and reading what they say (and treating it seriously). GitLab, you're not a cut-throat company run by the Ferengi, desperate to eke out a tiny bit of extra profit by whatever means necessary. You have what it takes to do it right and set an example for others.
- hvidgaard 7y ago
  That is simply not true. Both matter, but telemetry is something you can trust far more than user feedback. Want to know how much a functionality is used, or how long a user spends doing a particular action - telemetry is the answer. Compared to direct user feedback where a user might use some functionality twice a year, but rank it high as something that needs to be improved because it's sort of backwards. Instead of improving the performance and flow of the functionality they use hundreds of times daily, because they're reasonably happy with how it works. It's not that you don't want to fix the former, but it's obviously not business critical to perform often, and it's not considered high priority yet.
  - TeMPOraL 7y ago
    > Both matter, but telemetry is something you can trust far more than user feedback. Disagree. User feedback, however biased or incoherent, is a direct representation of real people with real issues. Telemetry is a stream of context-free data from which you need to infer the hidden user feedback. Telemetry data seems to me to be much more prone to finding in it whatever it is that you want to find, or all the other failure modes that happen when people who analyze it have more skills in statistical tools than in doing science. > Compared to direct user feedback where a user might use some functionality twice a year, but rank it high as something that needs to be improved because it's sort of backwards. Instead of improving the performance and flow of the functionality they use hundreds of times daily, because they're reasonably happy with how it works. And all I want, all my life when using software, is for companies to fucking listen to that. If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. I want you to fix the 5%-time case. There's no direct relationship between how often you use a feature and how critical it is. To give a simplified example, when I'm using 3D Studio Max, 99% of my time is spent modelling, and only 1% (or less) is spent on saving the file. And yet, without that feature, I wouldn't even buy the program. This is of course a silly example, but most software has plenty of very important features executed very rarely (and some that are executed rarely because the experience sucks). Relying only on naive telemetry analysis is going to lead you to misprioritizing work on improving the product.

by jfkw 7y ago

jkdufair 7y ago

That sounds like recipe for near-zero participation. I think both the intent and execution are spot-on here for GitLab. They are offering a very straightforward option for using GitLab, telemetry-free.

kazlock 7y ago

I'm not arguing for or against tracking, but I disagree with some of these assumptions: 1) 99% of users who have a bad experience are not going to be bothered to figure out how to record, review, and submit their session for the benefit of the service provider. If a page has issues loading, they're just going to give up and move on to the next thing. 2) User tracking isn't really related to performance in the first place. If some server side operation is failing or taking a long time to load, engineers will find out (and probably get paged) with user-agnostic internal performance metrics. 3) User tracking isn't just about finding out what's broken with the site, its about understanding how the site is being used in general so that you can validate your assumptions and make product decisions backed by data.

codedokode 7y ago
> If a page has issues loading, they're just going to give up and move on to the next thing. It means that they are fine with these issues and don't need to have them fixed.
- kazlock 7y ago
  Or it means they're just going to switch to $competitor's product that doesn't have this issue.

rkagerer 7y ago

You don't need telemetry or telepathy to hear the chorus of users screaming at the top of their lungs where to shove your telemetry feature.

emilycook 7y ago

Hello! Thanks for the feedback, we opened an issue here that might help clear some of this up: https://gitlab.com/gitlab-org/growth/product/issues/164 https://gitlab.com/gitlab-org/growth/product/issues/164

TeMPOraL 7y ago

Telemetry is a lazy, paternalistic way of improving user experience. It's a method that lets you ignore the actual feedback you're getting from your users, because that involves dealing with actual human beings - who don't always write politely or coherently. Telemetry lets you avoid collecting feedback in a way that respects the user - through in-house studies, by contracting with outside users to perform on-site studies, and just by asking people and reading what they say (and treating it seriously). GitLab, you're not a cut-throat company run by the Ferengi, desperate to eke out a tiny bit of extra profit by whatever means necessary. You have what it takes to do it right and set an example for others.

hvidgaard 7y ago
That is simply not true. Both matter, but telemetry is something you can trust far more than user feedback. Want to know how much a functionality is used, or how long a user spends doing a particular action - telemetry is the answer. Compared to direct user feedback where a user might use some functionality twice a year, but rank it high as something that needs to be improved because it's sort of backwards. Instead of improving the performance and flow of the functionality they use hundreds of times daily, because they're reasonably happy with how it works. It's not that you don't want to fix the former, but it's obviously not business critical to perform often, and it's not considered high priority yet.
- TeMPOraL 7y ago
  > Both matter, but telemetry is something you can trust far more than user feedback. Disagree. User feedback, however biased or incoherent, is a direct representation of real people with real issues. Telemetry is a stream of context-free data from which you need to infer the hidden user feedback. Telemetry data seems to me to be much more prone to finding in it whatever it is that you want to find, or all the other failure modes that happen when people who analyze it have more skills in statistical tools than in doing science. > Compared to direct user feedback where a user might use some functionality twice a year, but rank it high as something that needs to be improved because it's sort of backwards. Instead of improving the performance and flow of the functionality they use hundreds of times daily, because they're reasonably happy with how it works. And all I want, all my life when using software, is for companies to fucking listen to that. If I say I'm reasonably happy with things I spend 95% of my time, then I am reasonably happy. I want you to fix the 5%-time case. There's no direct relationship between how often you use a feature and how critical it is. To give a simplified example, when I'm using 3D Studio Max, 99% of my time is spent modelling, and only 1% (or less) is spent on saving the file. And yet, without that feature, I wouldn't even buy the program. This is of course a silly example, but most software has plenty of very important features executed very rarely (and some that are executed rarely because the experience sucks). Relying only on naive telemetry analysis is going to lead you to misprioritizing work on improving the product.