7 ms·

I’m not really sure they articulated why they had to switch very clearly. They didn’t like managing Mongo. They said they couldn’t use Mongo’s hosted solution B

I’m not really sure they articulated why they had to switch very clearly. They didn’t like managing Mongo. They said they couldn’t use Mongo’s hosted solution BUT they switch to hosted Postgres. Why not just overcome the limitations preventing them from switching to hosted MongoDB?

lykr0n 8y ago
It looks like they wanted to run Mongo inside their own Amazon account- under their direct control. AWS could do that with Postgres while MongoDB couldn't.
- flatiron 8y ago
  they talked about NTP. which to me means Mongo on EC2 maybe?
  - lykr0n 8y ago
    That's what I understood they were doing and they didn't want to do that anymore.
    - wftglf 8y ago
      Yep that's what we were doing, and the management software (OpsManager) was also running on EC2 instances. We messed up the VPC configuration so that NTP didn't work on some of the instances - which unsurprisingly broke authentication between OpsManager and the db instances
- scarface74 8y ago
  That makes me sad. I would love for AWS to have a hosted Mongo solution or some type of NoSQL database besides DynamoDB. Do people use ElasticSearch as a primary data store? In my limited experience implementations don’t treat it as the source of truth.
  - snuxoll 8y ago
    ElasticSearch doesn’t have adequate data integrity to treat it as a primary data store for most use cases.
  - jstarfish 8y ago
    Yes, but it only works if absolute-goddamn-integrity is not a concern. We unfortunately inherited a cluster from someone who thought it would be appropriate as a SoT for forensic data, which causes me no end of grief.
  - lykr0n 8y ago
    You could do what they are doing here
  - dhd415 8y ago
    Elastic maintains a page documenting the current status of their "resiliency" efforts on Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html https://www.elastic.co/guide/en/elasticsearch/resiliency/cur...
- h1d 8y ago
  What's wrong with running a EC2 instance with Mongo in it?
  - lykr0n 8y ago
    Nothing, but they outline their problems in the article
emn13 8y ago
The way I read it, mongo was hosted the same way postgres was - on AWS VMs. OpsManager wanted to be more like a SaaS. I'm a little skeptical of the idea that you can do custom software development with a custom database schema and realistically expect to outsource DB management. But sure, you can try. And in any case, you'd hope a largely read-only and document oriented dataset like a newspaper's has a relatively simple schema; without too many crazy schema quirks.
- scarface74 8y ago
  Why wouldn’t you be able to outsource database server management? Are you referring to outsourcing schema management?
  - emn13 8y ago
    By server management - do you mean VM? Sure, you can outsource that. I mean that what and how a home-grown application uses a database tends to mean that database (software/schema/optimization/whatever) management cannot be application agnostic. So if you outsource this, you're either effectively hiring a consultant that still needs to deal with and learn details of specifically your application, or you should assume it's going to cost you some time to do yourself. I'm sure you can get advice or buy know-how; but they're too coupled to think you're not also going to need to spend some time too. (At least: assuming your workload is large enough and complicated enough that naive brute force isn't an attractive option).
    - scarface74 8y ago
      Everything you are saying is true, but when people talk about managed services, for the most part they are referring to someone else managing the VM, the operating system and the server application running on the VM - in this case the database.
      - emn13 8y ago
        Exactly! That's why I was confused - they didn't replace mongodb with colloquially "hosted" postgres; they stayed at the same level of (partial) hosting. I.e., they didn't "switch to hosted postgres", in any normal sense.
koolba 8y ago
They probably mean that they wanted the DB hosted within the private subnet of their VPC. With a traditional hosted offering you’d connect over the public net to a system hosted by the SaaS provider. With RDS they could spin up the DB within their own VPC.
- PerfectElement 8y ago
  Couldn't they do this by using Atlas and choosing AWS as the host?
  - koolba 8y ago
    The article says the work began in July 2017. Was it available at that time?
  - thawkins 8y ago
    Atlas runs tne hosts inside mongodbs AWS account, we have the same restriant, becuase of client privacy isses and gdpr compliance we can let a 3rd party host the data, amazon is ok beacuse we control tbe data on tbe instances.
    - merlynn 8y ago
      Client privacy issues should not preclude the use of MongoDB Atlas... Atlas offers encryption of data at rest as well as the ability to manage your own keys.

lykr0n 8y ago

It looks like they wanted to run Mongo inside their own Amazon account- under their direct control. AWS could do that with Postgres while MongoDB couldn't.

flatiron 8y ago
they talked about NTP. which to me means Mongo on EC2 maybe?
- lykr0n 8y ago
  That's what I understood they were doing and they didn't want to do that anymore.
  - wftglf 8y ago
    Yep that's what we were doing, and the management software (OpsManager) was also running on EC2 instances. We messed up the VPC configuration so that NTP didn't work on some of the instances - which unsurprisingly broke authentication between OpsManager and the db instances
scarface74 8y ago
That makes me sad. I would love for AWS to have a hosted Mongo solution or some type of NoSQL database besides DynamoDB. Do people use ElasticSearch as a primary data store? In my limited experience implementations don’t treat it as the source of truth.
- snuxoll 8y ago
  ElasticSearch doesn’t have adequate data integrity to treat it as a primary data store for most use cases.
- jstarfish 8y ago
  Yes, but it only works if absolute-goddamn-integrity is not a concern. We unfortunately inherited a cluster from someone who thought it would be appropriate as a SoT for forensic data, which causes me no end of grief.
- lykr0n 8y ago
  You could do what they are doing here
- dhd415 8y ago
  Elastic maintains a page documenting the current status of their "resiliency" efforts on Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html https://www.elastic.co/guide/en/elasticsearch/resiliency/cur...
h1d 8y ago
What's wrong with running a EC2 instance with Mongo in it?
- lykr0n 8y ago
  Nothing, but they outline their problems in the article

emn13 8y ago

The way I read it, mongo was hosted the same way postgres was - on AWS VMs. OpsManager wanted to be more like a SaaS. I'm a little skeptical of the idea that you can do custom software development with a custom database schema and realistically expect to outsource DB management. But sure, you can try. And in any case, you'd hope a largely read-only and document oriented dataset like a newspaper's has a relatively simple schema; without too many crazy schema quirks.

scarface74 8y ago
Why wouldn’t you be able to outsource database server management? Are you referring to outsourcing schema management?
- emn13 8y ago
  By server management - do you mean VM? Sure, you can outsource that. I mean that what and how a home-grown application uses a database tends to mean that database (software/schema/optimization/whatever) management cannot be application agnostic. So if you outsource this, you're either effectively hiring a consultant that still needs to deal with and learn details of specifically your application, or you should assume it's going to cost you some time to do yourself. I'm sure you can get advice or buy know-how; but they're too coupled to think you're not also going to need to spend some time too. (At least: assuming your workload is large enough and complicated enough that naive brute force isn't an attractive option).
  - scarface74 8y ago
    Everything you are saying is true, but when people talk about managed services, for the most part they are referring to someone else managing the VM, the operating system and the server application running on the VM - in this case the database.
    - emn13 8y ago
      Exactly! That's why I was confused - they didn't replace mongodb with colloquially "hosted" postgres; they stayed at the same level of (partial) hosting. I.e., they didn't "switch to hosted postgres", in any normal sense.

koolba 8y ago

They probably mean that they wanted the DB hosted within the private subnet of their VPC. With a traditional hosted offering you’d connect over the public net to a system hosted by the SaaS provider. With RDS they could spin up the DB within their own VPC.

PerfectElement 8y ago
Couldn't they do this by using Atlas and choosing AWS as the host?
- koolba 8y ago
  The article says the work began in July 2017. Was it available at that time?
- thawkins 8y ago
  Atlas runs tne hosts inside mongodbs AWS account, we have the same restriant, becuase of client privacy isses and gdpr compliance we can let a 3rd party host the data, amazon is ok beacuse we control tbe data on tbe instances.
  - merlynn 8y ago
    Client privacy issues should not preclude the use of MongoDB Atlas... Atlas offers encryption of data at rest as well as the ability to manage your own keys.