6 ms·

Hey Folks, The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations shardin

Hey Folks, The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a free tier that enables developers to easily use our platform, others see it as an opportunity for exploitation. Even more unfortunate is that it has a negative effect on legitimate users. It’s a fine balance that has been highlighted by several users within this thread. Spam filtering is not a perfect science, and we’re constantly tweaking things -- with our customers in mind. This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues (which was also highlighted by a few within this thread). If this isn’t the case email me: cramsdale@google.com. For those asking, “hey, why am I being penalized for being a new customer?” See my previous comment about spam filtering not being a perfect science. Then email me. We’re here and we want to help. -- Chris (Lead PM for App Engine)

je42 10y ago
Isn't there an API to find out why a message was SPAM filtered ? Or are you afraid that "rogue" applications will use it to produce messages that are SPAM but yet not trigger the SPAM filter ?
- jcrites 10y ago
  The recent post about Candy Japan's attempt to combat credit card fraud gives a good sense of what fighting fraud and abuse is like: https://news.ycombinator.com/item?id=11431881 https://news.ycombinator.com/item?id=11431881 As an email service provider it's like that and more, since (1) once an abuser uses your service, they've gotten the benefit immediately and keep it even if their account is discovered as fraudulent later, e.g. stolen CC number & chargeback. (2) Abusive users can directly harm good users such as by harming the deliverability of the overall platform. It's not just bad debt, it's bad experience too. (3) Unlike Candy Japan where fraudsters mostly just wanted to check CC numbers and not actually buy product, email abusers really want to send emails (4) It can be hard to tell good and bad senders apart because some companies with an internet presence aren't email savvy and might make mistakes or might get hacked. Spam filters are always tough because if you give someone transparency into which actions of theirs that you consider abuse, then they will quickly detect and route around your attempt to block them. (See Candy Japan article) It's pretty easy for a human to guess what might be the sign of their fraud and run a few experiments to see what gets flagged e.g. By comparison a machine learning system might be hard to outsmart, but then it's also challenging to explain and troubleshoot false positives. Hence what's effective is often a combination of machine-learned filters and heuristics along with manual overrides by human judgment. All other things equal, new users are a lot more likely to engage in fraud than existing ones, and so tend to be under more suspicion. Aside from B2B fraud where companies take out lines of credit and then go bankrupt intentionally, it's uncommon for existing established customers to turn fraudulent - they're already vetted. (Consider: who is more likely to be fraudulent. The first time subscriber to Candy Japan, or a subscriber who has been using it for 12 months and is about to buy their 13th month?) It's not a great experience as a new user to be under suspicion, but if it's temporary and easily overridden by a human it can be a decent trade-off - the need to reach out acts a deterrent to spammers but does not deter legitimate users as much (speaking generally).
fpgaminer 10y ago
Just a quick thank you for the free tier that App Engine offers. I use it to provide an educational app for my wife, who's a teacher, that she uses as part of her classes. It's nice to be able to provide that to her and her class, without hitting the bank (the school does that enough...). So, thank you.
brianmcconnell 10y ago
Long time App Engine user here. If you're sending email as part of your app, I'd recommend using Mailgun (or similar services). You can set up DKIM, inbound routes, etc, and generally have more control over how your mail is processed. I never got the impression App Engine really wanted to be in the email delivery business, so if that's important to you, there are tools that are focused on that (and free up to a monthly quota). My $0.02.
- siva7891 10y ago
  I completely agree with your recommendation on Mailgun. Using it for past 4 months, never had a problem with their service.
- wastedhours 10y ago
  +1 as well, after the changes to Mandrill recently, was looking for a new provider and can also recommend Mailgun (at least, for ease and configurability in set up and testing so far, not yet in production).
  - pritambarhate 10y ago
    Yes, Mandrill should have kept the free quota after DKIM and SPF verification. Now, I will be trying out the competitors (Sendgrid and Mailgun) and if I find them at par with Mandrill then for paid projects, I might use them rather than Mandrill.
    - mrcncpt 10y ago
      For our clients, the appeal of Mandrill is the Mailchimp Template Editor. I think this is where their focus is going and so if you don't need that, then definitely check out competitors.
      - pritambarhate 10y ago
        Out of around 18 projects, in which we have used Mandrill, only in 1 we have used the template editor. (I think only about half went for paid accounts when the project went live.) Most of our need was simple and quick transactional messages (Registration, Forgot Password, etc.). The main reason we chose Mandrill was ease of set up over Amazon SES. (Even though we had AWS accounts for all projects.) Anyways, it's up to Mandrill to choose who they want to serve. I really liked the service though. Best of Luck.
jcrites 10y ago
Possible reason that email service providers and other email sending services might see an increase in new signups and free tier use recently: http://blog.mandrill.com/important-changes-to-mandrill.html http://blog.mandrill.com/important-changes-to-mandrill.html Mandrill in essence raised their price, eliminated free tier, and required prepurchased blocks of sending capacity IIRC The numbers Mandrill released about that business suggest that they had a large number of low volume senders, who may now be looking for a new home.
markdown 10y ago
Hi Chris, > This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues. 1. My app is not new. It's been running without issue for 2 or 3 years. 2. My app is not on the free tier. 3. On average, my app sends out under 8 emails a day, and it's done the same for over 2 years. How your algorithm considers my app a spam risk sure beats me. I'm missing 11 days worth of quote requests from customers. Are these recoverable? (is there a hidden outgoing-spam bin?) My app started sending mail again today after I changed the src of an image in it from https://example.appspot.com/images/logo.png https://example.appspot.com/images/logo.png to http://www.example.com/images/logo.png http://www.example.com/images/logo.png How well do you think email clients are going to like emails with images embedded in them without HTTPS?
- tim333 10y ago
  You might look at Sendgrid. It's free for those sort of volumes
- klewandowski 10y ago
  Hi, if you haven't already sent Chris an email, can you send me the details of your account and I'll get answers for you today? lilkim@google.com
  - markdown 10y ago
    Thank you, I appreciate that. I sent him an email 14 hours ago and haven't received a response. I'll wait a day and if I still don't hear from him, will contact you.
JackWebbHeller 10y ago
Chris, you say Gmail tightened up their spam filters... Did this take place in the last ~6 weeks or so? I've noticed the spam filter on Gmail for Google Apps has gotten significantly less accurate recently, resulting in far more false-positives than usual. Any ideas if this is a known issue? I can only presume it's more accurate overall, but it was definitely a noticeable change for our organisation.
- emdd 10y ago
  Yep, this has been an unfortunate thing for us as well.

je42 10y ago

Isn't there an API to find out why a message was SPAM filtered ? Or are you afraid that "rogue" applications will use it to produce messages that are SPAM but yet not trigger the SPAM filter ?

jcrites 10y ago
The recent post about Candy Japan's attempt to combat credit card fraud gives a good sense of what fighting fraud and abuse is like: https://news.ycombinator.com/item?id=11431881 https://news.ycombinator.com/item?id=11431881 As an email service provider it's like that and more, since (1) once an abuser uses your service, they've gotten the benefit immediately and keep it even if their account is discovered as fraudulent later, e.g. stolen CC number & chargeback. (2) Abusive users can directly harm good users such as by harming the deliverability of the overall platform. It's not just bad debt, it's bad experience too. (3) Unlike Candy Japan where fraudsters mostly just wanted to check CC numbers and not actually buy product, email abusers really want to send emails (4) It can be hard to tell good and bad senders apart because some companies with an internet presence aren't email savvy and might make mistakes or might get hacked. Spam filters are always tough because if you give someone transparency into which actions of theirs that you consider abuse, then they will quickly detect and route around your attempt to block them. (See Candy Japan article) It's pretty easy for a human to guess what might be the sign of their fraud and run a few experiments to see what gets flagged e.g. By comparison a machine learning system might be hard to outsmart, but then it's also challenging to explain and troubleshoot false positives. Hence what's effective is often a combination of machine-learned filters and heuristics along with manual overrides by human judgment. All other things equal, new users are a lot more likely to engage in fraud than existing ones, and so tend to be under more suspicion. Aside from B2B fraud where companies take out lines of credit and then go bankrupt intentionally, it's uncommon for existing established customers to turn fraudulent - they're already vetted. (Consider: who is more likely to be fraudulent. The first time subscriber to Candy Japan, or a subscriber who has been using it for 12 months and is about to buy their 13th month?) It's not a great experience as a new user to be under suspicion, but if it's temporary and easily overridden by a human it can be a decent trade-off - the need to reach out acts a deterrent to spammers but does not deter legitimate users as much (speaking generally).

fpgaminer 10y ago

Just a quick thank you for the free tier that App Engine offers. I use it to provide an educational app for my wife, who's a teacher, that she uses as part of her classes. It's nice to be able to provide that to her and her class, without hitting the bank (the school does that enough...). So, thank you.

brianmcconnell 10y ago

Long time App Engine user here. If you're sending email as part of your app, I'd recommend using Mailgun (or similar services). You can set up DKIM, inbound routes, etc, and generally have more control over how your mail is processed. I never got the impression App Engine really wanted to be in the email delivery business, so if that's important to you, there are tools that are focused on that (and free up to a monthly quota). My $0.02.

siva7891 10y ago
I completely agree with your recommendation on Mailgun. Using it for past 4 months, never had a problem with their service.
wastedhours 10y ago
+1 as well, after the changes to Mandrill recently, was looking for a new provider and can also recommend Mailgun (at least, for ease and configurability in set up and testing so far, not yet in production).
- pritambarhate 10y ago
  Yes, Mandrill should have kept the free quota after DKIM and SPF verification. Now, I will be trying out the competitors (Sendgrid and Mailgun) and if I find them at par with Mandrill then for paid projects, I might use them rather than Mandrill.
  - mrcncpt 10y ago
    For our clients, the appeal of Mandrill is the Mailchimp Template Editor. I think this is where their focus is going and so if you don't need that, then definitely check out competitors.
    - pritambarhate 10y ago
      Out of around 18 projects, in which we have used Mandrill, only in 1 we have used the template editor. (I think only about half went for paid accounts when the project went live.) Most of our need was simple and quick transactional messages (Registration, Forgot Password, etc.). The main reason we chose Mandrill was ease of set up over Amazon SES. (Even though we had AWS accounts for all projects.) Anyways, it's up to Mandrill to choose who they want to serve. I really liked the service though. Best of Luck.

jcrites 10y ago

Possible reason that email service providers and other email sending services might see an increase in new signups and free tier use recently: http://blog.mandrill.com/important-changes-to-mandrill.html http://blog.mandrill.com/important-changes-to-mandrill.html Mandrill in essence raised their price, eliminated free tier, and required prepurchased blocks of sending capacity IIRC The numbers Mandrill released about that business suggest that they had a large number of low volume senders, who may now be looking for a new home.

markdown 10y ago

Hi Chris, > This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues. 1. My app is not new. It's been running without issue for 2 or 3 years. 2. My app is not on the free tier. 3. On average, my app sends out under 8 emails a day, and it's done the same for over 2 years. How your algorithm considers my app a spam risk sure beats me. I'm missing 11 days worth of quote requests from customers. Are these recoverable? (is there a hidden outgoing-spam bin?) My app started sending mail again today after I changed the src of an image in it from https://example.appspot.com/images/logo.png https://example.appspot.com/images/logo.png to http://www.example.com/images/logo.png http://www.example.com/images/logo.png How well do you think email clients are going to like emails with images embedded in them without HTTPS?

tim333 10y ago
You might look at Sendgrid. It's free for those sort of volumes
klewandowski 10y ago
Hi, if you haven't already sent Chris an email, can you send me the details of your account and I'll get answers for you today? lilkim@google.com
- markdown 10y ago
  Thank you, I appreciate that. I sent him an email 14 hours ago and haven't received a response. I'll wait a day and if I still don't hear from him, will contact you.

JackWebbHeller 10y ago

Chris, you say Gmail tightened up their spam filters... Did this take place in the last ~6 weeks or so? I've noticed the spam filter on Gmail for Google Apps has gotten significantly less accurate recently, resulting in far more false-positives than usual. Any ideas if this is a known issue? I can only presume it's more accurate overall, but it was definitely a noticeable change for our organisation.

emdd 10y ago
Yep, this has been an unfortunate thing for us as well.