5 ms·

What you can do is add your gmail account as an IMAP account in Outlook or another email client and sort it there. Hard to believe that we must do that because

What you can do is add your gmail account as an IMAP account in Outlook or another email client and sort it there. Hard to believe that we must do that because of their anti sorting stance.

drawkbox 17y ago
It isn't really an anti-sorting stance. If you have worked on appengine and with Google File System you will see that it is fast almost solely because there is no complete dataset. So sorting is really tough on 64K or 64MB chunks spread around many many machines. Google is so fast and scalable exactly because there is no complete dataset easily attainable. GFS works in 1000 item chunks and that is about the best you can get with that setup. Searching, counts/increments and metadata about ALL of your data in the GFS or email in this case, is a tough problem. The same underlying data system is used for gmail, reader, etc.
- icey 17y ago
  I wonder how difficult it would be for them to create a metadata file per account that would only contain things like sent from, sent to, size, subject, date sent (standard email metadata). Then if someone wants to go digging around, send the metadata down as JSON and let the sorting happen client side and let the client return the order the messages should render.
  - drawkbox 17y ago
    It is possible but it also means creating more data. I could easily see this being a pay element to gmail in the future. Basically it could be almost like a bot/spider/worker that gets a fairly recent snapshot of your data as metadata and allows you to clean up or organize based on it. It could be pretty tasking though and may need alot of engineering because right now your gmail data might be spread across thousands of machines. With more data/years this only going to get more tasking and costly. At a certain point with data we have been living in a relative small sandbox in terms of data. As our lives spread to terrabytes of data and across many services, we to will be unable to run atomic operations on the whole of your data.
- TheElder 17y ago
  Thanks for the insight.
- misterbwong 17y ago
  Thanks for that-had no idea how GFS worked. Seems like Google could easily treat it as a searching issue rather than a sorting issue, though. It would be easier than full text searching (which it already does).
  - blasdel 17y ago
    Gmail does not support rich full-text search -- the indexer used is very limited. Among other things no synonyms, word stems, or subwords are indexed.
    - jwecker 17y ago
      This is the biggest problem I have with gmail, and one of the principle reasons why I don't feel I'll be using it "permanently." My search queries very rarely find the emails that I'm looking for.
- blasdel 17y ago
  Every time you mention GFS, you're really talking about Bigtable (which happens to be built on top of GFS)
  - drawkbox 17y ago
    Yes I say GFS because that is the core of why it is this way. From search to reader to gmail etc. All google data operates this way because of the architecture. It is all in 1000 buckets. Ever gone beyond page 100 in google results? You can't. Even though there are millions of results you can only get to page 99 at 10 results per page because it too is built on GFS.
    - TheElder 17y ago
      You are correct: http://www.google.com/search?q=blue&start=1000 http://www.google.com/search?q=blue&start=1000 > Sorry, Google does not serve more than 1000 results for any query. (You asked for results starting from 1000.)
      - prabodh 17y ago
        Results 811 - 817 of about 687,000,000 for test [definition]. (2.19 seconds) after seeing this, I doubt whether 687,000,000 is correct or not..
        thwarted 17y ago
        Term counting is the hello world of map-reduce examples, so while that number is obviously not exact (exactly 687 million?), it's reasonably accurate, considering the actual use of the count, which is more useless the bigger it is. No one created the opposite of Google Whack, finding the terms that yielded the most results; that's largely uninteresting when the counts are in the hundreds of millions range.
    - phsr 17y ago
      Found this out working at a Web Consultant when working with a google appliance. Specifically this page: http://bidmc.org/Search.aspx http://bidmc.org/Search.aspx
- pyre 17y ago
  Why does it have to be a 'sorting' stance? Why can't you just search for something like 'size:>1MB' to get all emails that are larger than 1MB? Even if they aren't 'sorted' one can easily scale the search to narrow in on really large attachments.
brown9-2 17y ago
Just curious, but will adding my gmail account as an IMAP account in another client cause any synchronization issues with accessing my mail on mail.google.com ?
- TheElder 17y ago
  I never had any problems. Just note that, when you read a mail on your IMAP account, it'll mark it as read on your webmail. Same for deleting/moving and whatever else you do.
- pyre 17y ago
  The only issue is that IMAP was never designed to deal with labels, so labels are mapped to folders. This can cause issues though. Deleting something from a 'folder' only removes that label from it. Then you have to go to 'all mail' to delete it. You can work around this issue by using the Advanced IMAP Settings and hiding the 'all mail' folder in IMAP, then setting the 'delete mail once all visible labels are removed' setting. I had issues with using this and offlineimap to sync... Sometimes when I moved a mail from one 'folder' to another offlineimap would delete it from one folder and then create it in another, but Gmail's IMAP server would see that the mail had no labels on it (in the middle of the operation) and then move it to the trash instead. This issue shouldn't come up with a client like Outlook or Thunderbird though because of the way that they operate (I believe that there is a 'move' command in the IMAP protocol that they use).

What you can do is add your gmail account as an IMAP account in Outlook or another email client and sort it there. Hard to believe that we must do that because of their anti sorting stance.

drawkbox 17y ago

It isn't really an anti-sorting stance. If you have worked on appengine and with Google File System you will see that it is fast almost solely because there is no complete dataset. So sorting is really tough on 64K or 64MB chunks spread around many many machines. Google is so fast and scalable exactly because there is no complete dataset easily attainable. GFS works in 1000 item chunks and that is about the best you can get with that setup. Searching, counts/increments and metadata about ALL of your data in the GFS or email in this case, is a tough problem. The same underlying data system is used for gmail, reader, etc.

icey 17y ago
I wonder how difficult it would be for them to create a metadata file per account that would only contain things like sent from, sent to, size, subject, date sent (standard email metadata). Then if someone wants to go digging around, send the metadata down as JSON and let the sorting happen client side and let the client return the order the messages should render.
- drawkbox 17y ago
  It is possible but it also means creating more data. I could easily see this being a pay element to gmail in the future. Basically it could be almost like a bot/spider/worker that gets a fairly recent snapshot of your data as metadata and allows you to clean up or organize based on it. It could be pretty tasking though and may need alot of engineering because right now your gmail data might be spread across thousands of machines. With more data/years this only going to get more tasking and costly. At a certain point with data we have been living in a relative small sandbox in terms of data. As our lives spread to terrabytes of data and across many services, we to will be unable to run atomic operations on the whole of your data.
TheElder 17y ago
Thanks for the insight.
misterbwong 17y ago
Thanks for that-had no idea how GFS worked. Seems like Google could easily treat it as a searching issue rather than a sorting issue, though. It would be easier than full text searching (which it already does).
- blasdel 17y ago
  Gmail does not support rich full-text search -- the indexer used is very limited. Among other things no synonyms, word stems, or subwords are indexed.
  - jwecker 17y ago
    This is the biggest problem I have with gmail, and one of the principle reasons why I don't feel I'll be using it "permanently." My search queries very rarely find the emails that I'm looking for.
blasdel 17y ago
Every time you mention GFS, you're really talking about Bigtable (which happens to be built on top of GFS)
- drawkbox 17y ago
  Yes I say GFS because that is the core of why it is this way. From search to reader to gmail etc. All google data operates this way because of the architecture. It is all in 1000 buckets. Ever gone beyond page 100 in google results? You can't. Even though there are millions of results you can only get to page 99 at 10 results per page because it too is built on GFS.
  - TheElder 17y ago
    You are correct: http://www.google.com/search?q=blue&start=1000 http://www.google.com/search?q=blue&start=1000 > Sorry, Google does not serve more than 1000 results for any query. (You asked for results starting from 1000.)
    - prabodh 17y ago
      Results 811 - 817 of about 687,000,000 for test [definition]. (2.19 seconds) after seeing this, I doubt whether 687,000,000 is correct or not..
      - thwarted 17y ago
        Term counting is the hello world of map-reduce examples, so while that number is obviously not exact (exactly 687 million?), it's reasonably accurate, considering the actual use of the count, which is more useless the bigger it is. No one created the opposite of Google Whack, finding the terms that yielded the most results; that's largely uninteresting when the counts are in the hundreds of millions range.
  - phsr 17y ago
    Found this out working at a Web Consultant when working with a google appliance. Specifically this page: http://bidmc.org/Search.aspx http://bidmc.org/Search.aspx
pyre 17y ago
Why does it have to be a 'sorting' stance? Why can't you just search for something like 'size:>1MB' to get all emails that are larger than 1MB? Even if they aren't 'sorted' one can easily scale the search to narrow in on really large attachments.

brown9-2 17y ago

Just curious, but will adding my gmail account as an IMAP account in another client cause any synchronization issues with accessing my mail on mail.google.com ?

TheElder 17y ago
I never had any problems. Just note that, when you read a mail on your IMAP account, it'll mark it as read on your webmail. Same for deleting/moving and whatever else you do.
pyre 17y ago
The only issue is that IMAP was never designed to deal with labels, so labels are mapped to folders. This can cause issues though. Deleting something from a 'folder' only removes that label from it. Then you have to go to 'all mail' to delete it. You can work around this issue by using the Advanced IMAP Settings and hiding the 'all mail' folder in IMAP, then setting the 'delete mail once all visible labels are removed' setting. I had issues with using this and offlineimap to sync... Sometimes when I moved a mail from one 'folder' to another offlineimap would delete it from one folder and then create it in another, but Gmail's IMAP server would see that the mail had no labels on it (in the middle of the operation) and then move it to the trash instead. This issue shouldn't come up with a client like Outlook or Thunderbird though because of the way that they operate (I believe that there is a 'move' command in the IMAP protocol that they use).