8 ms·
OpenAI: Streaming is now available in the Assistants API
- bytemonitor 2y agoThanks for posting. I got an example working with functions and tool_calls if anyone needs it. I could not find good examples in the docs. https://medium.com/@hawkflow.ai/openai-streaming-assistants-example-77e53ca18fb4 https://medium.com/@hawkflow.ai/openai-streaming-assistants-...
- ProjectArcturis 3y agoHas anyone put out a voice-to-text interface for OpenAI? Or anything in the Ollama-verse?
- the_newest 3y agoThere’s whisper.
- eightysixfour 3y agoOpenAI has a voice to text interface for OpenAI…
- willsmith72 3y agoThe mobile app is pretty good Horrendous in non english languages though, the accents are extremely American
- notRobot 3y agoIs there a way to use the mobile app on PCs? I tried with Windows Subsystem for Android but the app refused to work.
- arthurcolle 3y agoSigh another week lost to the void
- nextworddev 3y agoElaborate?
- castles 3y ago"YET ANOTHER shiny new toy to distract me. Can't help myself even though I think it's mostly a waste of time" Am I just projecting? Relatable, in any case :)
- arthurcolle 3y agoYep, you captured the moment ^_^
- __m 3y agoI immediately implemented streaming into my rocketchat gpt bot, was definitely a distraction but my colleagues liked it. No more waiting until the complete response is sent.
- andher 3y agoFinally! I've been using the assistants api in building an ai mock interviewer (https://comp.lol https://comp.lol) but the responses were painfully slow when using the latest iterations of the gpt-4 model. This will make things so much more responsive
- cosmotic 3y agoI'd still want to see the entire response all at once. Having it stream in while I read it would be very distracting and make it difficult for me to read.
- TowerTall 3y agoyes, it like surfing porn in the early internet year using a dialup modem. One line a the time until you finally can see enough of the picture (reply) to realize that is was not the reply you were looking for. LLM streaming must be a cost saving feature to prevent you from overloading the servers by asking to many questions with in a short time frame. Annoying feature IMHO
- Kiro 3y agoHow is hiding it behind a loading spinner any better? You still can't spam it with questions since you need to wait for it to finish. With streaming you can at least hit the stop button if it looks incorrect, so you actually spam it more with it enabled.
- silversmith 3y agoFor me, the constant visual changes of new parts being streamed in are annoying, and straining on the eyes. Ideally, web frontends would honor `prefers-reduced-motion` and buffer the response when set.
- Prosammer 3y agoPersonally, I've fallen in love with that visual effect of streaming text you're talking about. It's a bit pavlovian, but I think in my head it signifies that I'm reading something high signal (even though it isn't always).
- m-p-3 3y agoI thought this was about making the OpenAI app available as a digital assistant on Android, as a replacement to Google. Oh well..
- XCSme 3y agoAny way to have a consistent system prompt across queries without sending it (and using tokens) for each completion?
- eightysixfour 3y agoThe Assistant API handles that, it has the system prompt as part of the assistant that you interact with.
- XCSme 3y agoAnd can you share the assistant with other users? Also, the system prompt in assistants doesn't consume tokens?
- arthurcolle 3y agoThe assistant has its own "instructions" (replacement for system prompt) and then on each run, you have the option to add more guidance to the run explicitly, without modifying the assistant instructions (system prompt) It's a little bit different but kind of the same
- baobabKoodaa 3y agoNo, adding run instructions will replace existing instructions for that run
- arthurcolle 3y agoThat's what they say ;)
- AgentME 3y agoThis was one of the limitations of the Assistants API that made me entirely ignore it up until now. I am curious if the Assistants API lets you edit/remove/retry messages yet. I don't see anything implying this has changed. It's annoying that the Assistants API doesn't give you enough control to support basic things that the ChatGPT app does.
- xvector 3y agoEdit/remove/retry is just including the whole conversation over again (IIUC this is even how the app works.) It's part of why the API is so expensive
- AgentME 3y agoThe Assistants API doesn't let you recreate the conversation (with edits or not) because you can't (re)create messages with role=assistant.
- arthurcolle 3y agoNot true
- baobabKoodaa 3y agoHow do you create messages as role assistant?
- d4rkp4ttern 3y agoThis was indeed true in the beginning, and I don’t know if this has changed. Inserting messages with Assistant role is crucial for many reasons, such as if you want to implement caching, or otherwise edit/compress a previous assistant response for cost or other reason. At the time I implemented a work-around in Langroid[1]: since you can only insert a “user” role message, prepend the content with ASSISTANT: whenever you want it to be treated as an assistant role. This actually works as expected and I was able to do caching. I explained it in this forum: https://community.openai.com/t/add-custom-roles-to-messages-in-threads/485446/4?u=darthfader https://community.openai.com/t/add-custom-roles-to-messages-... [1] the Langroid code that adds a message with a given role, using this above “assistant spoofing trick”: https://github.com/langroid/langroid/blob/main/langroid/agent/openai_assistant.py#L447 https://github.com/langroid/langroid/blob/main/langroid/agen...
- potsandpans 3y agoOpenai banned my account for suspicious payment activities, and I never was able to talk to a real person. Just several layers of chat bots posing as people. I literally want to give them my money and can't. Every few weeks for shirts and giggles i send an email to them saying, "any update on this?"
- dbish 3y agoOne of the reasons I tend to use any of their options through Azure where available. Azure support has a more straight forward (though still sometimes slow) process for account issues.
- GaggiX 3y agoI guess it's time for Claude 3 (I imagine you were using it for the LLMs).
- thejohnconway 3y agoMy Anthropic account was suspended for suspicious activity, even though I never used it. I had forgotten I had signed up, and tried to sign up using a new email with the same phone number. Locked out forever.
- jasondclinton 3y agoplease contact support: https://support.anthropic.com/en/ https://support.anthropic.com/en/ . we'll get it fixed. sorry!
- slimsag 3y agoWelcome to the future. You might be able to get an enterprise sales contract with human support.
- ukuina 3y agoI suspected as much when one of their support "personnel" used the phrase "I apologize for the earlier confusion..." (there was no confusion, I was simply contradicting what they were saying)
- simonw 3y agoThrowing a feature request in here just in case someone from OpenAI sees it. I'd really like it if the streaming versions of their APIs could return a token usage count at the end. The non-streaming APIs do this right now: curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "A short fun fact about pigeons" } ] }' Returns: { "id": "chatcmpl-92UiIWQaf442wq7Eyp7kF8ge0e3fE", "object": "chat.completion", "created": 1710381746, "model": "gpt-3.5-turbo-0125", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Pigeons are one of the few bird species that can drink water by sucking it up through their beaks, rather than tilting their heads back to swallow." }, "logprobs": null, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 14, "completion_tokens": 33, "total_tokens": 47 }, "system_fingerprint": "fp_4f0b692a78" } Note the "usage" block there telling me how many tokens were used (which tells me how much this cost). But if I add "stream": true I get back an SSE stream that looks like this: ... data: {"id":"chatcmpl-92Uk81oNjrcUJQnPX8fSNqFINLfSI","object":"chat.completion.chunk","created":1710381860,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_4f0b692a78","choices":[{"index":0,"delta":{"content":"."},"logprobs":null,"finish_reason":null}]} data: {"id":"chatcmpl-92Uk81oNjrcUJQnPX8fSNqFINLfSI","object":"chat.completion.chunk","created":1710381860,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_4f0b692a78","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]} data: [DONE] There's no "usage" block, which means I have to try and account for the tokens myself. This is really inconvenient! I noticed the other day that the Claude streaming API returns a "usage" block with the last message. I'd love it if OpenAI's API did the same thing. I need this right now because I'm starting to build features for end users of my own software, and I want to be able to give them X,000 tokens "free" before starting to charge them for extras. Counting those tokens myself (probably using tiktoken) is code I'd rather not have to write - especially since features like tools/functions or images make counting tokens a lot less obvious.
- harrisonjackson 3y agoThis and/or being able to fetch the responses with their token usage by id. What is that ID for without a way to retrieve the completions with it?
- megous 3y agoThis website is now like 30% about this probability based autocomplete nonsense. Feels like all those bitcoin hypes and "running everything on blockchain" fad of few years ago. Now it's running everything through "large autocomplete" model. I really hope this will fade and focus will turn back to highlighting some broader actual human ingenuity in IT, rather than constant stream of "we used autocomplete for this new thing" or "we build this new API for this glorified autocomplete". Boring.
- XCSme 3y agoI just added this "autocomplete" in my app, and customers emailed to say they actually love it: https://docs.uxwizz.com/guides/ask-ai-new https://docs.uxwizz.com/guides/ask-ai-new
- megous 3y agoYes, customers will love anything that helps them. You can get customers to love you by adding any kind of automation for stuff they had to do by hand up to that point. Does this mean there should be 10 articles per day shared about "I added XLSX import to my app, so my customers don't have to do data entry via dialogs"? My point is about repetitiveness of LLM topics. Not about usefullness of LLM itself. And LLMs are glorified autocomplete. Their internals are maybe interesting, but that's often not what's being discussed here or even written about in the shared articles.
- kfajdsl 3y agoI've gotten so used to having an LLM integrated into my editor that when I work on the occasional spreadsheet (or really anything with syntax that I only use occasionally and no integrated AI) it's pretty jarring to have to go to another tab to look up what function to use for a formula (even if that other tab is ChatGPT).
- chaxor 3y ago"old man yells at cloud" Seriously though, it's not going away no matter how much anyone hates it. Emails and blogs will continue to be written with it, letters of recommendation will be/are written with it, Presidential speeches will be written with it, academic articles will be / are written with it (almost all ml and cs research is), news is written with it... It's not going to stop, but it will _probably_/_very likely_ get better. There is no tool, no human, no method to determine if text is generated with one of these models at high F-score (only sometimes high precision, low recall domains for silly examples). We're stuck with it. Like the English teacher and their despised spell check.
- pedrovhb 3y agoFor all the brilliance in the AI and infra departments of OpenAI, their official Python library (which is the flagship one as I understand) feels pretty unidiomatic, designed without much thought for common patterns in the language. 2012 JavaScript called, it wants its callbacks wrapped in objects back. Why do we have a context manager named "stream" for which you call `.until_done()`? This could've been an iterator, or better - an asynchronous iterator, since this is streaming over the network. We could be destructing instances of named tuples with pattern matching, or even just doing `"".join(delta.text for delta in prompt (...)`. But no here subclass this instead, tells me the wrapper around a web API.
- deleted 3y ago[deleted]
- willsmith72 3y agoEverything feels unidiomatic. The API design is bad, the frontends they build are horrific, reliability and availability are shocking. And yet the AI is so good I put up with them everyday If they ever grow into a proper product org they'll be unstoppable.
- athyuttamre 3y agoHi there, I help design the OpenAI APIs. Would you be able to share more? You can reply here or email me at atty@openai.com. (Please don't hold back; we would love to hear the pain points so we can fix them.)
- msp26 3y agoHey, random question. Is there a technical reason why log probs aren't available when using function calling? It's not a problem, I've already found a workaround. I was just curious haha. In general I feel like the function calling/tool use is a bit cumbersome and restrictive so I prefer to write the typescript in the functions namespace myself and just use json_mode.
- 3y ago
- jerrygoyal 3y agoI am interested to use the assistant api for my commercial project but it is not clear from the article what the token count looks like? - is it counted for a single user message or the sum of all previous messages? - if there's a file, will it be counted every time a user interacts or only the first time?
- visarga 3y agoI think - it is correlated to the sum, every new interaction adds the whole history again - yes, but you probably pay for the retrieved fragments, not the whole file
- brandall10 3y agoOn the second point, there was an issue on launch where it would not find a relevant fragment and appear to load the whole file into the context. Unsure if this has changed but it freaked quite a few folks out OpenAI discussion forums w/ escalating costs.
- milar 3y agoHas tool use accuracy improved?
- zerop 3y agothey should do streaming for voice inputs on the chatgpt app. right now it's very slow. Voice interfaces need to be streaming
- mrtksn 3y agoAssistant API is too much of a beta still. I was about to release an app based on the new Assistant API but just a day before the release the response times increased to 8s flat. When I have function calls, that meant up to a minute to get a response. I had to dismantle everything Assistant API and implement it with Chat API. Which turned out to be great because in Assistant API the context management was very bad and after a few back and forth messages the cost ballooned to over 10K tokens per message. When I looked closely at the Assistant API and Chat API, I noticed that Assistant API is just a wrapper over Chat API and acts as a web service that stores the previous messages(so slow response problem was probably due to the web server which keeps track of the context). So I went ahead and implemented my own Assistant API which has more control. For example, I set max token cost per message and if the context balloons over that, I make a request with the context and ask OpenAI to create a summary with all the facts so far, add that summary as a system prompt and my context gets compressed back into reasonable territory.
- infecto 3y agoI still don't even know what the Assistant API is supposed to afford me.
- mrtksn 3y agoIt's useful if you just need to hook up a chat assistant and don't want to bother with the busywork doing it. All you care is loading the messages from the thread(which are conveniently kept for you) and add new messages.
- infecto 3y agoYeah that was kind of my idea, it does not serve much if any purpose and only limits the capability.
- lobsterthief 3y agoIs the training method similar? For example, a company chatbot would need to know it’s a chatbot for Company Y.
- johnfurneaux 3y agoAdore. Congrats team. For us the API is epic. We'd just ask for focus on performance.