6 ms·

> Instead of having 1000 rows per shopSKU, why not just have one row per shopping cartSKU? At what point that row is inserted?

bijowo1676 1mo ago
per my reading of the article, the protection is only needed for a few seconds, while payment is being processed by the payment system. so the row is inserted when Payment is initiated, and row is deleted when Payment succeeds What is oversell protection? Reserve: When payment starts, we mark items as reserved (a short hold, e.g. several minutes). Claim: When payment succeeds, we permanently deduct quantity from the inventory ledger (source of truth). but that system could be easily improved to reserve item when user Adds item to a cart, to prevent scenario when user adds item to a cart, goes through checkout, and after initiating payment gets "soldout error": 1. Let user add item to a cart by default (happy path) 2. Initiate async check in the background for SKU and quantity 2a. The check sums up rows for all SKUs and compares to Inventory table (very cheap check since its done to only active shopping carts) 3. After few seconds the check comes back, and we let user know that item is soldout, before/the moment user goes to Checkout.
- soontimes 1mo ago
  Ok, but before inserting you must ensure that inventory is not depleted, which means you need to know the count and you need to lock the row. So you still have contention on that item. Them having a 1k buffer allows not to take a lock on a single row every time, and only do it when buffer is empty
  - bijowo1676 1mo ago
    there is no need to lock the row, since you a dealing with a shopping cart, not individual item piece. when you run aggregate functions, lock is no needed, it is actually better to run it with SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; for aggregation the check for oversold items is extremely cheap: with current_order as ( select $SKU1, $q2 as quantity union select $SKU2, $q2 as quantity ), with carts as ( select sku, sum(quantity) as reserved from active_carts group by sku ), with warehouse as ( select sku, available_units from inventory group by sku ) select * from current_order inner join carts using (sku) inner join warehouse using (sku) where warehouse.available_units - carts.reserved < current_order.quantity assuming there are indexes on sku field in both, results in efficient index seek and agg over 2 tables
    - edoceo 1mo ago
      Thanks! I don't uSe 'with' enough
    - codedokode 1mo ago
      The item is reserved when the user decides to place an order, but before paying for it. Not when a product is added to the cart because the user can keep it there for a month and end up not buying. You reserve the product by creating an "active_cart" entry. Your solution has a problem, that when you run the check, it might say the product is available, but before you create an "active_cart" to reserve it from thread A, another thread B reserves it and you end up reserving a product that is not available anymore. You end up with SUM(active_cart.quantity) > inventory.available_units. That is exactly why the database has locks - to prevent this situation. With locks, thread A decrements inventory.available_units and that row is locked until the end of transaction. Other threads (if they do SELECT FOR UPDATE instead of SELECT) cannot see the old, invalid value until thread A either commits and the value is updated or rollbacks. However, locks cause performance issues and that is why shopify uses the architecture from the article - instead of 100 users fighting for the lock on the same row with available amount, each user locks only rows with units they plan to buy. Interestingly, MySQL docs has the documentation page with a similar case: https://dev.mysql.com/blog-archive/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/ https://dev.mysql.com/blog-archive/mysql-8-0-1-using-skip-lo...
    - soontimes 1mo ago
      I don’t understand how this should prevent oversold. You have a check that reports empty or oversold inventory. But how does that check prevent 2 concurrent actors fighting for the last item from inserting 2 rows?
      - bijowo1676 1mo ago
        how does current design resolve concurrent actors fighting for the last item ? there is ultimately needs to be some global mechanism resolving this conflict. Currently it is an order in which db engine processes transactions by locking rows for a transaction, whoever got the first lock, wins the last remaining items. my design is the same, except it does not need this dance with moving rows between tables, locking them, and the cludge with replenishment process. in the simplest form, run the sum() over active non-finished orders and compare to inventory. you get the same result: whoever got the first to run sum() and get positive answer will get the last remaining items. but the problem as formulated, imho, is not even correctly defined. Shopify incorrectly formulated the very problem they are trying to solve. Trying to solve it at the payment time is too late, its better to resolve it earlier, before the checkout. the "PAY" button should only do one thing: deduct money from cc and that's it. Resolving inventory availability must be solved way earlier, the moment user clicks Checkout, not when user clicks Pay. So ideally, the error for oversold items should be shown to a user when he clicks Checkout, not when he click PAY

bijowo1676 1mo ago

per my reading of the article, the protection is only needed for a few seconds, while payment is being processed by the payment system. so the row is inserted when Payment is initiated, and row is deleted when Payment succeeds What is oversell protection? Reserve: When payment starts, we mark items as reserved (a short hold, e.g. several minutes). Claim: When payment succeeds, we permanently deduct quantity from the inventory ledger (source of truth). but that system could be easily improved to reserve item when user Adds item to a cart, to prevent scenario when user adds item to a cart, goes through checkout, and after initiating payment gets "soldout error": 1. Let user add item to a cart by default (happy path) 2. Initiate async check in the background for SKU and quantity 2a. The check sums up rows for all SKUs and compares to Inventory table (very cheap check since its done to only active shopping carts) 3. After few seconds the check comes back, and we let user know that item is soldout, before/the moment user goes to Checkout.

soontimes 1mo ago
Ok, but before inserting you must ensure that inventory is not depleted, which means you need to know the count and you need to lock the row. So you still have contention on that item. Them having a 1k buffer allows not to take a lock on a single row every time, and only do it when buffer is empty
- bijowo1676 1mo ago
  there is no need to lock the row, since you a dealing with a shopping cart, not individual item piece. when you run aggregate functions, lock is no needed, it is actually better to run it with SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; for aggregation the check for oversold items is extremely cheap: with current_order as ( select $SKU1, $q2 as quantity union select $SKU2, $q2 as quantity ), with carts as ( select sku, sum(quantity) as reserved from active_carts group by sku ), with warehouse as ( select sku, available_units from inventory group by sku ) select * from current_order inner join carts using (sku) inner join warehouse using (sku) where warehouse.available_units - carts.reserved < current_order.quantity assuming there are indexes on sku field in both, results in efficient index seek and agg over 2 tables
  - edoceo 1mo ago
    Thanks! I don't uSe 'with' enough
  - codedokode 1mo ago
    The item is reserved when the user decides to place an order, but before paying for it. Not when a product is added to the cart because the user can keep it there for a month and end up not buying. You reserve the product by creating an "active_cart" entry. Your solution has a problem, that when you run the check, it might say the product is available, but before you create an "active_cart" to reserve it from thread A, another thread B reserves it and you end up reserving a product that is not available anymore. You end up with SUM(active_cart.quantity) > inventory.available_units. That is exactly why the database has locks - to prevent this situation. With locks, thread A decrements inventory.available_units and that row is locked until the end of transaction. Other threads (if they do SELECT FOR UPDATE instead of SELECT) cannot see the old, invalid value until thread A either commits and the value is updated or rollbacks. However, locks cause performance issues and that is why shopify uses the architecture from the article - instead of 100 users fighting for the lock on the same row with available amount, each user locks only rows with units they plan to buy. Interestingly, MySQL docs has the documentation page with a similar case: https://dev.mysql.com/blog-archive/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/ https://dev.mysql.com/blog-archive/mysql-8-0-1-using-skip-lo...
  - soontimes 1mo ago
    I don’t understand how this should prevent oversold. You have a check that reports empty or oversold inventory. But how does that check prevent 2 concurrent actors fighting for the last item from inserting 2 rows?
    - bijowo1676 1mo ago
      how does current design resolve concurrent actors fighting for the last item ? there is ultimately needs to be some global mechanism resolving this conflict. Currently it is an order in which db engine processes transactions by locking rows for a transaction, whoever got the first lock, wins the last remaining items. my design is the same, except it does not need this dance with moving rows between tables, locking them, and the cludge with replenishment process. in the simplest form, run the sum() over active non-finished orders and compare to inventory. you get the same result: whoever got the first to run sum() and get positive answer will get the last remaining items. but the problem as formulated, imho, is not even correctly defined. Shopify incorrectly formulated the very problem they are trying to solve. Trying to solve it at the payment time is too late, its better to resolve it earlier, before the checkout. the "PAY" button should only do one thing: deduct money from cc and that's it. Resolving inventory availability must be solved way earlier, the moment user clicks Checkout, not when user clicks Pay. So ideally, the error for oversold items should be shown to a user when he clicks Checkout, not when he click PAY