7 ms·
A friend built a system where the ad campaigns with all their targeting rules were Lua scripts and it was also fast and simple in a glorious way!
by nattaylor 2y ago
A friend built a system where the ad campaigns with all their targeting rules were Lua scripts and it was also fast and simple in a glorious way!
- ing33k 2y agoI tried a similar approach with a team I was working with. We were building it on top of redis and after some basic benchmarks gave up the idea and figured out from the docs that that eval script is blocking.
- btilly 2y agoWhen I did that on a project I solved that problem by replicating redis to the same container that the ads were being served from. Replication was very fast, and all that was blocked was the local redis. I worked to make the matching rules in the script efficient, and the blockage was truly not a problem.
- catlifeonmars 2y agoDoesn’t local redis kind of miss the point of using redis in the first place? On the surface, that’s a big chunk of additional complexity for something that could be done internal to an application. Was this meant to be an incremental step in a larger refactor?
- ing33k 2y agowell, redis manages the replication without fuss. Unless the framework / language one uses supports it out of the box, there is no point in re implementing replication. https://www.erlang.org/docs/17/reference_manual/distributed https://www.erlang.org/docs/17/reference_manual/distributed
- catlifeonmars 2y agoOh so this would effectively serve the same purpose as a centralized cluster.
- ing33k 2y agoYeah with the added advantage of reduced latency. The system I worked on had a CRUD application which was in Django , ultimately the rules was stored in a Redis.. The ad serving application was in golang that connected to this local redis instances that were replicas of the central redis.. https://redis.io/docs/latest/commands/replicaof/ https://redis.io/docs/latest/commands/replicaof/
- btilly 2y agoExactly the same for the one that I did, except that the ad serving was in Python. But golang is probably the better choice.
- bastawhiz 2y agoLua in Redis is not useful for serving as a runtime for running Lua application code; it's valuable for allowing you to perform a series of steps atomically. You can read and write data without worrying about something else changing: only one piece of code can write at a time. We use it for rate limiting, which requires reading and writing atomically, for instance.
- knighthack 2y agoWhy wasn't Python used? I imagine for such a use case it wouldn't be that much heavier to use Python vs Lua.
- qwertox 2y agoSpeed, probably. Lua is much faster.
- nattaylor 2y agoYes, speed. 1,000,000+ QPS and 1,000s of ads to evaluate is common. At that scale it's very distributed, so replication is another challenge. In a way, its a search problem where bid≈relevance and targeting_match≈recall, so I've seen Solr used here too.
- catlifeonmars 2y agoIntegration complexity could be a factor. I’m making an assumption about the architecture, but embedding Lua in a program is dead simple and you can do so without introducing external dependencies. Python IIRC requires you to ship and package the standard library or have it already installed, the Lua interpreter can be statically embedded in a program.