6 ms·

Finally! I've been using the assistants api in building an ai mock interviewer (https://comp.lol https://comp.lol) but the responses were painfully slow when us

by andher 3y ago

Finally! I've been using the assistants api in building an ai mock interviewer (https://comp.lol https://comp.lol) but the responses were painfully slow when using the latest iterations of the gpt-4 model. This will make things so much more responsive

cosmotic 3y ago
I'd still want to see the entire response all at once. Having it stream in while I read it would be very distracting and make it difficult for me to read.
- TowerTall 3y ago
  yes, it like surfing porn in the early internet year using a dialup modem. One line a the time until you finally can see enough of the picture (reply) to realize that is was not the reply you were looking for. LLM streaming must be a cost saving feature to prevent you from overloading the servers by asking to many questions with in a short time frame. Annoying feature IMHO
  - Kiro 3y ago
    How is hiding it behind a loading spinner any better? You still can't spam it with questions since you need to wait for it to finish. With streaming you can at least hit the stop button if it looks incorrect, so you actually spam it more with it enabled.
    - silversmith 3y ago
      For me, the constant visual changes of new parts being streamed in are annoying, and straining on the eyes. Ideally, web frontends would honor `prefers-reduced-motion` and buffer the response when set.
      - Prosammer 3y ago
        Personally, I've fallen in love with that visual effect of streaming text you're talking about. It's a bit pavlovian, but I think in my head it signifies that I'm reading something high signal (even though it isn't always).
  - SoulAuctioneer 3y ago
    It's more about UX, to reduce the perceived delay. LLMs inherently stream their responses, but if you wait until the LLM has finished inference, the user is sitting around twiddling their thumbs.
- deleted 3y ago
  [deleted]
- qwertox 3y ago
  It's a request the front-end developer should be confronted with, not OpenAI. The website could as well buffer the incoming stream until the used clicks an area to request the display of the next block of the response, once he has finished reading the initial sentences.
pieterhg 3y ago
Same it was super slow and unusable when I tried. 10 seconds for a reply or smth. GPT4 API itself was way faster

by andher 3y ago

cosmotic 3y ago

I'd still want to see the entire response all at once. Having it stream in while I read it would be very distracting and make it difficult for me to read.

TowerTall 3y ago
yes, it like surfing porn in the early internet year using a dialup modem. One line a the time until you finally can see enough of the picture (reply) to realize that is was not the reply you were looking for. LLM streaming must be a cost saving feature to prevent you from overloading the servers by asking to many questions with in a short time frame. Annoying feature IMHO
- Kiro 3y ago
  How is hiding it behind a loading spinner any better? You still can't spam it with questions since you need to wait for it to finish. With streaming you can at least hit the stop button if it looks incorrect, so you actually spam it more with it enabled.
  - silversmith 3y ago
    For me, the constant visual changes of new parts being streamed in are annoying, and straining on the eyes. Ideally, web frontends would honor `prefers-reduced-motion` and buffer the response when set.
    - Prosammer 3y ago
      Personally, I've fallen in love with that visual effect of streaming text you're talking about. It's a bit pavlovian, but I think in my head it signifies that I'm reading something high signal (even though it isn't always).
- SoulAuctioneer 3y ago
  It's more about UX, to reduce the perceived delay. LLMs inherently stream their responses, but if you wait until the LLM has finished inference, the user is sitting around twiddling their thumbs.
deleted 3y ago
[deleted]
qwertox 3y ago
It's a request the front-end developer should be confronted with, not OpenAI. The website could as well buffer the incoming stream until the used clicks an area to request the display of the next block of the response, once he has finished reading the initial sentences.

pieterhg 3y ago

Same it was super slow and unusable when I tried. 10 seconds for a reply or smth. GPT4 API itself was way faster