7 ms·

Google’s painful Gmail OAuth verification process

105 points by kitx 7y ago

PaulHoule 7y ago
To the contrary it will be very good for innovation because it means people will build extensions for other email providers such as Fastmail.
- icebraining 7y ago
  Does Fastmail even provide an API like this?
  - PaulHoule 7y ago
    Sure, IMAP, POP, CALDav, same as all the other mail providers... And the optimized JMAP protocol too. The fact that I am getting voted down proves how absolute the groupthink is... If you work in Google's ecosystem, however, the only possible exit is being bought by Google... Get bought by a competitor and they'll just turn you off.
    - joshuamorton 7y ago
      So the answer is no then, because Gmail still provides imap and pop, which are unaffected by this.
      - PaulHoule 7y ago
        But how does the credential management work?
        joshuamorton 7y ago
        I don't recall since the last time I set it up was a while ago, but I think just your password + an app specific code if you use 2fa normally. But yeah I'm pretty confident this doesn't apply to outlook.
        PaulHoule 7y ago
        So why don't we see more startups use open protocols for access to email rather than make things that are GMail specific? Why do you get voted down for just suggesting that they do so?
        joshuamorton 7y ago
        I'm not sure, my expectation would be because oauth is more secure and feature rich than either imap or pop. Jmap I'm less familiar with.
        kitx 7y ago
        IMAP can be awkward with sync, pagination and has never played great with labels vs folders. JMAP looks promising.
      - JetSpiegel 7y ago
        Both disabled by default (ridiculous for an email provider), and if you enable 2FA get ready to do some yak shaving.
- pmlnr 7y ago
  No. If you want, build an email app with features or with it's own plugins. Not extensions around closed, single point of failure services.
privateSFacct 7y ago
As a gmail user - good to hear this. In the long run trust is going to be a much more important commodity that letting a spam app into your gmail. If you look at the service that want access to all your gmail data - many promise something "free" but then mine that data (in the fine print) to send you offers, alert you to "savings" etc. I automatically turn down apps that say they need access to my entire google drive and all email. Why not just ask for permissions for a single app specific folder? Ie, fax apps -> they should just store inbound faxes into one folder rather than asking for full drive access.
- Alex3917 7y ago
  > Why not just ask for permissions for a single app specific folder? Basically because Gmail A) doesn't have folders B) doesn't have permissions based on tags. Otherwise most Gmail API apps would have this option. The same thing is a problem when you want to delegate access to an email account, where there should be a way to delegate access based on tags, but there just isn't. What you can now do is create Gmail Add-Ons, which only have access to this specific thread that's open when you click to activate the add-on. E.g. this is how we created https://www.prettyfwd.com https://www.prettyfwd.com
  - orion138 7y ago
    Looks interesting but it’s unclear to me, do you copy the threads content? One other comment, there seems to be a spammer on the review page...
    - Alex3917 7y ago
      > do you copy the threads content? Basically we take each message of the thread, normalize the typography so that it can be consistently styled, and then strip off the signatures and quoted reply text while leaving any quoted inline replies. There are some open source libraries that do parts of this, but none of them worked well enough for this particular use case so we ended up just developing some new techniques ourselves. The other thing the tech does is it lets people run much more accurate NLP on threads. This isn't fully productized yet, so right now there are just a handful of companies we're working with to clean and normalize their email data. In terms of the spammer, yeah I see that but I'm not sure what I can really do. As the app creator I don't seem to have any special ability to delete comments or flag things as spam.
nuclearghost 7y ago
This is very helpful to see, along with the gmass blog mentioned within. We've been going through this process for months and it's definitely a moving target with no clear path to resolution. The whole process feels a bit Kafkaesque.
jonbronson 7y ago
As both a gmail user and developer interested in applications to help me manage my personal information, this is incredibly depressing to hear. The idea of a verification process itself is great, and I applaud that effort. But some of these barriers seems put in place solely to kill competition and prevent startups from filling the personal data needs before Google comes up with its own plan. These exorbitant fees of $15,000 and $75,000 are completely unjustifiable.
- Klathmon 7y ago
  It's the direction that everyone seems to be moving. Wall up the gardens, remove your own access to your information, and remove the ability to share and integrate across platforms. The weird thing is how quickly the sentiment turned from my perspective. I felt like one day most technical people applauded the ability to have total real-time access to your data, to be able to write code or use open source code to plug into these systems and augment them for better. To be able to have a startup or small company write software that can work with your google/facebook/apple/whatever account and use the information there in new ways. Then all of a sudden (I feel like it was between 1-2 years ago, but I can also barely remember what I had for lunch yesterday so don't quote me on that!), technical people started slamming companies for "allowing someone to access their data", I saw lots of headlines about how it was unethical for Facebook to allow users to share their information with other companies (don't get me wrong, facebook does plenty wrong, but to call out the ability to share info specifically seemed so wrong). Then APIs started shutting down, access is now only allowed for other big players, and it's getting harder and harder to integrate outside of a single player's walled garden. I get why the companies are doing it (someone told them the only ethical thing was to lock users in!?), but I don't get why HN and other technical circles are applauding it. Maybe i'm on the wrong side of history here, but I just feel like it's never a bad thing to allow me to share my information if I want. I think it should be clearly defined what i'm sharing, I think it should be obvious that i'm sharing it, and I think that some auditing and controls are obviously a good thing, but not this almost absolute shutdown of any ability for me to export or use information from these services on my own. But maybe I'm really in a bubble, and maybe users really shouldn't be given the choice to share their personal information, but it just feels so wrong and so "holier than thou" to make that choice for them.
dktoao 7y ago
Even for non-Gmail apps, this process is incredibly painful. I have an app that has been stuck in the process for weeks. Once you have read through the incredibly confusing and out of date documentation and submit what you think is the correct set of setting to comply with their policy, you then have to deal with the reviewer who will email you once every week if you are lucky. Usually to understand what they are asking you to fix you have to email them back and forth a few times. I love the platform, but they need to fix this aspect of it.
- creeble 7y ago
  They emailed you once a week? You are indeed lucky. I guess I need to bug them more, I haven't heard anything in weeks (busy with implementation). One warning: choose the email address for your Google developer account carefully, there doesn't seem to be a way to change it later. It is forever tied to your permissions and approvals, afaict.
- samcrawford 7y ago
  Worse still, in six months times the requirements will change and your previously approved scopes will no longer be approved. I've also had to deal with broken OAuth verification forms on Google's site (400 errors from their backend, with no UI feedback), and the complete inability to get a response from a human.
- hw 7y ago
  We've been in the Oauth review process for almost a month, and getting maybe 1 response per week as well. Plenty of times it's just the reviewer not reading the instructions we sent them and, well, time to wait another week. Then there's them saying our app doesn't need to be verified (apparently reviewer was looking at a different API permission instead of the ones we requested for OAuth), so there's been a bunch of unnecessary back and forth. We've gone through app review processes at other companies like Facebook, and it's all the same - plenty of time wasted with mostly ineptitude on the reviewer's side. Sometimes it feels like there's just one person working in Google/Facebook's basement doing these app reviews for minimum wage. I understand the need to be thorough on these app reviews especially if the app touches sensitive user data, but when the reviewer doesn't even read the instructions provided to them properly, would you trust them to be thorough when it comes to ensuring the apps aren't malicious?
deleted 7y ago
[deleted]
jacobsenscott 7y ago
This terrible, but building a business based on third party api's is always a tremendous risk. This isn't the first time a bunch of small apps have been killed off by some company making their api's inaccessible. Also, for people who are pushing for more government regulation of service providers - this is the lite version of what you are asking for.
- jonbronson 7y ago
  > this is the lite version of what you are asking for I'd say this is the heavy version. A new startup can put together and run a GDPR compliant web app for over a year for far less than $15,000.
  - tick_tock_tick 7y ago
    As long as no requests that you delete their data or provide them with a copy. As soon as that starts happening $15k doesn't go very far in man hours to comply.
    - icebraining 7y ago
      From personal experience, yes, it really does. Why do you think otherwise?
    - vertex-four 7y ago
      Just because you’re terrible at documenting where you put users’ information doesn’t mean everyone is.
  - jacobsenscott 7y ago
    It seems easy because you don't need to prove it. You can do your best and then launch it. But you can't prove it for under $15k and if you are successful eventually you'll need to prove it. Anyway, I'm not talking about the GDPR. That's just bureaucrats testing the water. The US congress is starting to look for ways to get their cut as well. It will get to the point where you need to prove compliance with multiple conflicting regulations first, before launching. Now you need to spend $100,000 on auditors before your first push to heroku. Most people in software, especially startups, have never had any contact with a regulated industry and don't know what they are in for.
wjossey 7y ago
As someone who tried to develop a tool to help pause their box to improve focus throughout the day, I got bit by this process as well. Basically google just went dark on me altogether. Has been months since their last reply and I kept trying to follow up. The feature I needed elevated permissions on was the ability to add filters, which unfortunately is buried with a bunch of other more dangerous permissions. Looks like I’ll never get to launch the product :( On the plus side, it works fine for just me! So, I just built a tool only I can use.
gonyea 7y ago
This all makes sense to me. If you're not providing enough value to users to cover the >$15k fee, you're just an attack vector for user data. Consistency of the process aside, I'm really not sure what people would expect. (I work at Google, yadda yadda, but have nothing to do with this.)
- Alex3917 7y ago
  At $10 a month, that’s only 125 users. How many apps with less than 125 users have we seen become a target for hackers? Has there ever been a case like that?
- infamia 7y ago
  Startups and open source projects get screwed because of Google's lack of nuance on this issue. Google could have created a tiered fee structure based on number of users, threat vector, etc. but didn't for whatever reason.
  - gonyea 7y ago
    Goal #1: Don't let the users "get screwed." Goal #2: Don't have a solution that requires an army of people to manage. If no one's willing to help fund your idea, you're out of luck. That seems... really understandable, given that every major government is literally investing millions trying to hack into Google's user data.
- nixpulvis 7y ago
  Fuck this, it's EXACTLY the problem in the valley. Small players should be empowered, not stifled.
  - human20190310 7y ago
    I wouldn't want someone who won't bet $15K-$75K on their own product to have access to any of my data.
    - 7y ago
raxxorrax 7y ago
It is not too hard to understand them trying to leverage the potential market of identity providers. Facebooks Libra basically tries to do the same. It would be a waste to use any services attached to it in my opinion. Otherwise oauth is a great technology, but interests may make it not worthwhile.
mariusz331 7y ago
A friend and I have been working on a side project that depends on Google Auth to send emails on a user's behalf. It's one of the app's two critical features. We're not necessarily deterred by this story, but we'll start rethinking our dependence on Google. A $15-75k fee is something that's hard to stomach at our stage. We have about 10 Gmail users excited to try our product and they might not have an issue accepting the "Unverified App" screen because we have earned some of their trust through phone calls and meetings. However, converting people that come across the app organically will be difficult. We aren't sure when the right time will be to pay the fee and become verified. Anyone have ideas on strategy here? It could help us and other developers in the same position. We'd like to avoid raising money but this might be a good reason to - investors may see Google verification as a competitive advantage.
bengotow 7y ago
I'm the author of the Mailspring email client and I've been dealing with this Oauth verification process for the last three months. Mailspring has "pro" features that leverage a small backend API, but it syncs your mail on your computer and your mail data, passwords, tokens, etc. never leave your machine. I care very much about data privacy and I wouldn't use the app myself if it was sending mail data to the cloud. I'm a big fan of Google watching out for their users. I know of at least one very sketchy company that has shut down because of this new policy, which is great. But after three months, they basically told me: "Your desktop app makes a network request to a third party server, you must pay $15,000 for a security audit." Their process has been vague and I wish they'd make an effort to understand whether an audit is really necessary. Their security contractors are going to be laughing all the way to the bank as they review my web service that never sees Gmail data in the first place. Thankfully, Mailspring makes a bit of money and I can afford to do this to keep it alive. But fast-forward a few years and this is going to devastate innovation and development of third party mail clients. (And I think Google prefers it this way.) If the app didn't already have critical mass, or if I was just starting a mail app now, I'd probably throw my hands up and give up rather than emailing them dozens of times and coughing up $15k.
bbcfake 7y ago
Total waste of time
nixpulvis 7y ago
Someone needs to make it trivial to host your own email, and sell it as reliable. I think you could probably sell more than just techies on it, given how your email is a critical system to many people in modern times.
- sn_master 7y ago
  Host it where tho ? Wouldn't you need a 24/7 running server ?
hexo 7y ago
Stop using it and show google you don't care about their "services" as market dictates.

105 points by kitx 7y ago

To the contrary it will be very good for innovation because it means people will build extensions for other email providers such as Fastmail.

icebraining 7y ago
Does Fastmail even provide an API like this?
- PaulHoule 7y ago
  Sure, IMAP, POP, CALDav, same as all the other mail providers... And the optimized JMAP protocol too. The fact that I am getting voted down proves how absolute the groupthink is... If you work in Google's ecosystem, however, the only possible exit is being bought by Google... Get bought by a competitor and they'll just turn you off.
  - joshuamorton 7y ago
    So the answer is no then, because Gmail still provides imap and pop, which are unaffected by this.
    - PaulHoule 7y ago
      But how does the credential management work?
      - joshuamorton 7y ago
        I don't recall since the last time I set it up was a while ago, but I think just your password + an app specific code if you use 2fa normally. But yeah I'm pretty confident this doesn't apply to outlook.
        PaulHoule 7y ago
        So why don't we see more startups use open protocols for access to email rather than make things that are GMail specific? Why do you get voted down for just suggesting that they do so?
        joshuamorton 7y ago
        I'm not sure, my expectation would be because oauth is more secure and feature rich than either imap or pop. Jmap I'm less familiar with.
        kitx 7y ago
        IMAP can be awkward with sync, pagination and has never played great with labels vs folders. JMAP looks promising.
    - JetSpiegel 7y ago
      Both disabled by default (ridiculous for an email provider), and if you enable 2FA get ready to do some yak shaving.
pmlnr 7y ago
No. If you want, build an email app with features or with it's own plugins. Not extensions around closed, single point of failure services.

privateSFacct 7y ago

As a gmail user - good to hear this. In the long run trust is going to be a much more important commodity that letting a spam app into your gmail. If you look at the service that want access to all your gmail data - many promise something "free" but then mine that data (in the fine print) to send you offers, alert you to "savings" etc. I automatically turn down apps that say they need access to my entire google drive and all email. Why not just ask for permissions for a single app specific folder? Ie, fax apps -> they should just store inbound faxes into one folder rather than asking for full drive access.

Alex3917 7y ago
> Why not just ask for permissions for a single app specific folder? Basically because Gmail A) doesn't have folders B) doesn't have permissions based on tags. Otherwise most Gmail API apps would have this option. The same thing is a problem when you want to delegate access to an email account, where there should be a way to delegate access based on tags, but there just isn't. What you can now do is create Gmail Add-Ons, which only have access to this specific thread that's open when you click to activate the add-on. E.g. this is how we created https://www.prettyfwd.com https://www.prettyfwd.com
- orion138 7y ago
  Looks interesting but it’s unclear to me, do you copy the threads content? One other comment, there seems to be a spammer on the review page...
  - Alex3917 7y ago
    > do you copy the threads content? Basically we take each message of the thread, normalize the typography so that it can be consistently styled, and then strip off the signatures and quoted reply text while leaving any quoted inline replies. There are some open source libraries that do parts of this, but none of them worked well enough for this particular use case so we ended up just developing some new techniques ourselves. The other thing the tech does is it lets people run much more accurate NLP on threads. This isn't fully productized yet, so right now there are just a handful of companies we're working with to clean and normalize their email data. In terms of the spammer, yeah I see that but I'm not sure what I can really do. As the app creator I don't seem to have any special ability to delete comments or flag things as spam.

nuclearghost 7y ago

This is very helpful to see, along with the gmass blog mentioned within. We've been going through this process for months and it's definitely a moving target with no clear path to resolution. The whole process feels a bit Kafkaesque.

jonbronson 7y ago

As both a gmail user and developer interested in applications to help me manage my personal information, this is incredibly depressing to hear. The idea of a verification process itself is great, and I applaud that effort. But some of these barriers seems put in place solely to kill competition and prevent startups from filling the personal data needs before Google comes up with its own plan. These exorbitant fees of $15,000 and $75,000 are completely unjustifiable.

Klathmon 7y ago
It's the direction that everyone seems to be moving. Wall up the gardens, remove your own access to your information, and remove the ability to share and integrate across platforms. The weird thing is how quickly the sentiment turned from my perspective. I felt like one day most technical people applauded the ability to have total real-time access to your data, to be able to write code or use open source code to plug into these systems and augment them for better. To be able to have a startup or small company write software that can work with your google/facebook/apple/whatever account and use the information there in new ways. Then all of a sudden (I feel like it was between 1-2 years ago, but I can also barely remember what I had for lunch yesterday so don't quote me on that!), technical people started slamming companies for "allowing someone to access their data", I saw lots of headlines about how it was unethical for Facebook to allow users to share their information with other companies (don't get me wrong, facebook does plenty wrong, but to call out the ability to share info specifically seemed so wrong). Then APIs started shutting down, access is now only allowed for other big players, and it's getting harder and harder to integrate outside of a single player's walled garden. I get why the companies are doing it (someone told them the only ethical thing was to lock users in!?), but I don't get why HN and other technical circles are applauding it. Maybe i'm on the wrong side of history here, but I just feel like it's never a bad thing to allow me to share my information if I want. I think it should be clearly defined what i'm sharing, I think it should be obvious that i'm sharing it, and I think that some auditing and controls are obviously a good thing, but not this almost absolute shutdown of any ability for me to export or use information from these services on my own. But maybe I'm really in a bubble, and maybe users really shouldn't be given the choice to share their personal information, but it just feels so wrong and so "holier than thou" to make that choice for them.

dktoao 7y ago

Even for non-Gmail apps, this process is incredibly painful. I have an app that has been stuck in the process for weeks. Once you have read through the incredibly confusing and out of date documentation and submit what you think is the correct set of setting to comply with their policy, you then have to deal with the reviewer who will email you once every week if you are lucky. Usually to understand what they are asking you to fix you have to email them back and forth a few times. I love the platform, but they need to fix this aspect of it.

creeble 7y ago
They emailed you once a week? You are indeed lucky. I guess I need to bug them more, I haven't heard anything in weeks (busy with implementation). One warning: choose the email address for your Google developer account carefully, there doesn't seem to be a way to change it later. It is forever tied to your permissions and approvals, afaict.
samcrawford 7y ago
Worse still, in six months times the requirements will change and your previously approved scopes will no longer be approved. I've also had to deal with broken OAuth verification forms on Google's site (400 errors from their backend, with no UI feedback), and the complete inability to get a response from a human.
hw 7y ago
We've been in the Oauth review process for almost a month, and getting maybe 1 response per week as well. Plenty of times it's just the reviewer not reading the instructions we sent them and, well, time to wait another week. Then there's them saying our app doesn't need to be verified (apparently reviewer was looking at a different API permission instead of the ones we requested for OAuth), so there's been a bunch of unnecessary back and forth. We've gone through app review processes at other companies like Facebook, and it's all the same - plenty of time wasted with mostly ineptitude on the reviewer's side. Sometimes it feels like there's just one person working in Google/Facebook's basement doing these app reviews for minimum wage. I understand the need to be thorough on these app reviews especially if the app touches sensitive user data, but when the reviewer doesn't even read the instructions provided to them properly, would you trust them to be thorough when it comes to ensuring the apps aren't malicious?

deleted 7y ago

[deleted]

jacobsenscott 7y ago

This terrible, but building a business based on third party api's is always a tremendous risk. This isn't the first time a bunch of small apps have been killed off by some company making their api's inaccessible. Also, for people who are pushing for more government regulation of service providers - this is the lite version of what you are asking for.

jonbronson 7y ago
> this is the lite version of what you are asking for I'd say this is the heavy version. A new startup can put together and run a GDPR compliant web app for over a year for far less than $15,000.
- tick_tock_tick 7y ago
  As long as no requests that you delete their data or provide them with a copy. As soon as that starts happening $15k doesn't go very far in man hours to comply.
  - icebraining 7y ago
    From personal experience, yes, it really does. Why do you think otherwise?
  - vertex-four 7y ago
    Just because you’re terrible at documenting where you put users’ information doesn’t mean everyone is.
- jacobsenscott 7y ago
  It seems easy because you don't need to prove it. You can do your best and then launch it. But you can't prove it for under $15k and if you are successful eventually you'll need to prove it. Anyway, I'm not talking about the GDPR. That's just bureaucrats testing the water. The US congress is starting to look for ways to get their cut as well. It will get to the point where you need to prove compliance with multiple conflicting regulations first, before launching. Now you need to spend $100,000 on auditors before your first push to heroku. Most people in software, especially startups, have never had any contact with a regulated industry and don't know what they are in for.

wjossey 7y ago

As someone who tried to develop a tool to help pause their box to improve focus throughout the day, I got bit by this process as well. Basically google just went dark on me altogether. Has been months since their last reply and I kept trying to follow up. The feature I needed elevated permissions on was the ability to add filters, which unfortunately is buried with a bunch of other more dangerous permissions. Looks like I’ll never get to launch the product :( On the plus side, it works fine for just me! So, I just built a tool only I can use.

gonyea 7y ago

This all makes sense to me. If you're not providing enough value to users to cover the >$15k fee, you're just an attack vector for user data. Consistency of the process aside, I'm really not sure what people would expect. (I work at Google, yadda yadda, but have nothing to do with this.)

Alex3917 7y ago
At $10 a month, that’s only 125 users. How many apps with less than 125 users have we seen become a target for hackers? Has there ever been a case like that?
infamia 7y ago
Startups and open source projects get screwed because of Google's lack of nuance on this issue. Google could have created a tiered fee structure based on number of users, threat vector, etc. but didn't for whatever reason.
- gonyea 7y ago
  Goal #1: Don't let the users "get screwed." Goal #2: Don't have a solution that requires an army of people to manage. If no one's willing to help fund your idea, you're out of luck. That seems... really understandable, given that every major government is literally investing millions trying to hack into Google's user data.
nixpulvis 7y ago
Fuck this, it's EXACTLY the problem in the valley. Small players should be empowered, not stifled.
- human20190310 7y ago
  I wouldn't want someone who won't bet $15K-$75K on their own product to have access to any of my data.
  - 7y ago

raxxorrax 7y ago

It is not too hard to understand them trying to leverage the potential market of identity providers. Facebooks Libra basically tries to do the same. It would be a waste to use any services attached to it in my opinion. Otherwise oauth is a great technology, but interests may make it not worthwhile.

mariusz331 7y ago

A friend and I have been working on a side project that depends on Google Auth to send emails on a user's behalf. It's one of the app's two critical features. We're not necessarily deterred by this story, but we'll start rethinking our dependence on Google. A $15-75k fee is something that's hard to stomach at our stage. We have about 10 Gmail users excited to try our product and they might not have an issue accepting the "Unverified App" screen because we have earned some of their trust through phone calls and meetings. However, converting people that come across the app organically will be difficult. We aren't sure when the right time will be to pay the fee and become verified. Anyone have ideas on strategy here? It could help us and other developers in the same position. We'd like to avoid raising money but this might be a good reason to - investors may see Google verification as a competitive advantage.

bengotow 7y ago

I'm the author of the Mailspring email client and I've been dealing with this Oauth verification process for the last three months. Mailspring has "pro" features that leverage a small backend API, but it syncs your mail on your computer and your mail data, passwords, tokens, etc. never leave your machine. I care very much about data privacy and I wouldn't use the app myself if it was sending mail data to the cloud. I'm a big fan of Google watching out for their users. I know of at least one very sketchy company that has shut down because of this new policy, which is great. But after three months, they basically told me: "Your desktop app makes a network request to a third party server, you must pay $15,000 for a security audit." Their process has been vague and I wish they'd make an effort to understand whether an audit is really necessary. Their security contractors are going to be laughing all the way to the bank as they review my web service that never sees Gmail data in the first place. Thankfully, Mailspring makes a bit of money and I can afford to do this to keep it alive. But fast-forward a few years and this is going to devastate innovation and development of third party mail clients. (And I think Google prefers it this way.) If the app didn't already have critical mass, or if I was just starting a mail app now, I'd probably throw my hands up and give up rather than emailing them dozens of times and coughing up $15k.

bbcfake 7y ago

Total waste of time

nixpulvis 7y ago

Someone needs to make it trivial to host your own email, and sell it as reliable. I think you could probably sell more than just techies on it, given how your email is a critical system to many people in modern times.

sn_master 7y ago
Host it where tho ? Wouldn't you need a 24/7 running server ?

hexo 7y ago

Stop using it and show google you don't care about their "services" as market dictates.