7 ms·
Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter
- simonw 10y agoFirst impression: this is a brilliant piece of software design. The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow. The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 this looks like my ideal abstraction.
- wellsjohnston 10y agoThank you :) Writing this was a necessity for me, being a 1-person data team coming from a Node.js background.
- rpedela 10y agoYes, I concur! I am definitely trying this out. I have a couple use cases where I think lambda functions would be useful but I don't currently have the time to figure out how to write and execute them.
- koolba 10y ago... Except it's not! The "lambda" here isn't AWS Lambda. It's a locally executed function. Now if this scheduled a bunch of real Lambdas to execute the work for each bucket then yes that'd be awesome.
- glogla 10y agoThat would be glorious.
- kot-behemoth 10y agoIt should be fairly easily doable with Gordon (https://github.com/jorgebastida/gordon https://github.com/jorgebastida/gordon), and scheduling via CloudWatch Events. Or Airflow.
- simonw 10y agoBah. My first impression was totally wrong in that case. Here's hoping someone builds a version of this that executes magically in the lambda cloud.
- illumin8 10y agoWell, you could run it on a large EC2 instance (x1.32xlarge?!:O) and it would be running the lambdas on the cloud, technically... ;-)
- dschnurr 10y agoMight make sense to rename this to avoid confusion with AWS Lambda (I immediately thought it was related). Otherwise, looks like an awesome library!
- deleted 10y ago[deleted]
- wellsjohnston 10y agoAh yeah, just realizing this...what would you recommend?
- andkon 10y agofunc-s3?
- cocktailpeanuts 10y agoI also came here thinking this is some sort of aws lambda triggered within the context of certain s3 file. I would say anyone who's heard of AWS lambda would think that way. Maybe functional-s3?
- stevewilhelm 10y agofp-4-s3?
- wellsjohnston 10y agoOkay this seems like a good alternative. I just renamed the repo. Renaming it on npm...is a bit cumbersome :|
- deleted 10y ago[deleted]
- gbrits 10y agos3-dataflow / s3-pipe
- lancefisher 10y ago
- DenisM 10y agoSo... the client-side code iterates S3 objects matching a certain filter, and then schedules a lambda for each one of those objects. Is that right? Or does the iteration procedure itself is a lambda? Also, when you chain several operators together, where does the chaining happen? I'd like to understand where different parts of the code are being executed.
- wellsjohnston 10y agoFirst, a list of keys is generated based on the set context (and modifier functions). "context" returns a Request object, allowing you to call a lambda function (each, forEach, map, reduce, filter). Each lambda function returns a Promise, allowing you to chain them together. They will operate over the same context, in sequence. Edit: This is not related to aws lambda...sorry for the confusion
- daviding 10y agoOn a quick page-down through the code, I think this is not related to AWS-Lambda, it's more 'local lambda' where the map etc is run locally.
- tjholowaychuk 10y agoI thought this too. You could easily ship/eval the toString()'s of the functions in individual Lambda functions, the name is definitely confusing haha.
- avip 10y agoThis is a nice project. For real-world use cases, we have good alternatives: 1. Migrate s3 ==> gc and use BigQuery which does support udf 2. Register to databricks (I'm not affiliated) 3. (for the brave) poke aws support to implement udf on Athena
- stolendog 10y agowhere actually you can use it ? in which cases? can you provide examples?
- wellsjohnston 10y agoSure. We have application logs that come in to s3 and are stored by date prefix. I have CRON jobs that run node scripts that do various counts/statistics.
- dhpe 10y agoReally nice to have a generic functional interface to S3. Thanks.
- hayd 10y agosee also aws athena https://aws.amazon.com/athena/ https://aws.amazon.com/athena/ ?
- wellsjohnston 10y agoI did not know about this...looks like Amazon's version of BigQuery. Fantastic!
- nstj 10y agoSomehow I'd overlooked this too: nice find
- nstj 10y agoSomehow I'd overlooked this too: nice find
- dajohnson89 10y agoThat seems cool but paying per query (per TB scanned) frightens me. I imagine having to fret about how efficient my queries are...
- illumin8 10y agoIt's not that bad. You can compress the data on S3 in ORC or Parquet format, and you only pay for the compressed data you read, so 1TB can be 130GB after compression. Plus, these formats store summary data, so queries like SELECT COUNT don't have to do a full table scan - they can read just a few KB of summary data for the result.
- dajohnson89 10y agoBut that's a lot of work....Just to have sane costs for reads of your data
- illumin8 10y agoIt's actually just two commands: 1. hive 2. INSERT INTO parquet_table SELECT * FROM csv_table;
- kvz 10y agoGetting aan index of (millions of) files on s3 is very slow for us, like, days. Is there anything you do to work around this? It seems since this is not an AWS Lambda project the client first has to acquire an index from S3 before concurrency benefits set in?
- wellsjohnston 10y agoThis does not have to do with AWS Lambda, I'm thinking about renaming it to "functional-s3", or something similar. To answer your question, there isn't really a workaround for this yet, although indexing should be much quicker than "days". All the keys are listed recursively before running the lambda expression locally. If you have a huge number of files, this can take several minutes, maybe hours depending on the scope. A workaround I've been considering is using a generator function to list the keys; that way, the lambda expression can start immediately, generating keys as it needs them.
- cle 10y agoIs this susceptible to any of S3's eventual consistency constraints?
- wellsjohnston 10y agoI'm not aware of S3's consistency constraints. What are those?
- primax 10y agohttp://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction....
- bpicolo 10y agoAnything you read from S3 is, so yes. The best way to prevent eventual consistency issues in s3 is to use immutable files. Then you have consistency-now. They don't have explicit SLAs on this, unfortunately, but I've heard rumored that internal pagers start firing with consistency behind on the order of hours.
- hoodoof 10y agoIts weird how S3 seems to be the unwanted stepchild of AWS. So many obvious innovations just aren't turning up. For example, strangely, AWS introduced tagging for S3 resources, but you can't search/filter by tag, nor is the tag even returned when you get a list of objects, you can only get the tag with an object request. The word "pointless" springs to mind. In fact it's strange that there is NO useful filtering at all apart from the very useful folder/hierarchy/prefix filtering. But apart from that you can't do wildcard searches or filters or date filters or tag filters. I'm building an application right now that needs to get a list of all the jpg files - the only way to do that is get every single object in the bucket and manually filter out the unwanted ones - feels like its 1988 again. It seems like it would also be valuable for there to be alternate interfaces to S3 such as the ability to send data via ftp or SMTP or sftp or whatever, but there are no such interfaces. Hopefully Google will goad AWS into action on S3 innovation by implementing such features.
- kot-behemoth 10y agoWhile great points, I think it might then go beyond the "Simple" in the S3 name itself. Wasn't the original purpose of the service to have it as a dumb storage, and you'll layer metadata as required? I.e. storing indices separately with whatever functionality is needed (be it date/path filtering).
- hoodoof 10y agoPerhaps true. I'll never do that though because I'd have to use DynamoDB, which is a technology that is high on my list of "technologies that I am least enthused about". Also, I really shouldn't have to go to all the work of creating and maintaining a metadata database and implementing a query API just because I want to do searches more powerful than "list all objects" - that's Amazon's job.
- maxaf 10y agoMoreover, even if you had gone to the trouble of building such an API, S3 still doesn't offer bulk operations, so you'd have to operate on each matching object... one object at a time.
- _Marak_ 10y agoIf anyone is interested in this same kind of architecture for multi-cloud file-system providers ( no cloud lock-in ), please check out this project: https://github.com/bigcompany/hook.io-vfs https://github.com/bigcompany/hook.io-vfs Used in production, but it could use some contributors.
- wcdolphin 10y agoI thinking having the default be destructive for mapping is a strange design decision. That is going to bite someone one day soon.
- wellsjohnston 10y agoGood point...I will make it so that destructive is opt-in