nikhilkuria

5 ms·

1. 1.▲
   by nikhilkuria 8y ago

   Rather than doing a one-time copy of the data, we have applications continually writing to our on-premise MySQL. This data has to reactively reach our datastore on AWS. Also, this is not a 1-1 copy. There are several transformations and cro
2. 2.▲
   by nikhilkuria 8y ago

   There was this one time when we had trouble finding some errors on Cloudwatch logs. This one library was helpful, https://github.com/jorgebastida/awslogs
3. 3.▲
   by nikhilkuria 8y ago

   Thanks! For testing on our local machines, we use SAM Local, https://aws.amazon.com/blogs/aws/new-aws-sam-local-beta-buil... . This has very similar capabilities as local stack. Still, there is always a delay fro
4. 4.▲
   by nikhilkuria 8y ago

   Exactly. The biggest motivation to use Kinesis was the other services we could plug in from 'AWS ecosystem'. Lambdas have Kinesis triggers available natively. We did not have any concern with the latency of Kinesis. Now I'm c
5. 5.▲
   by nikhilkuria 8y ago

   Your assumption is correct. One of our goals was to transform and restructure the data we have on our on-site premise MySQL. This is where the Lambdas come in. The Kinesis streams trigger the lambda, and the Lambdas filter and process the r
6. 6.▲
   by nikhilkuria 8y ago

   Yeah, it all depends on how much load or incoming records you expect. In our case, we use the pipeline to import inventory, say images from from our partners. So, we might have a few million images coming in many times a month. Lambda is su