6 ms·

Fails "car wash" test miserably. Ask it: > I want to wash my car. The car wash is 50 meters away. Should I walk or drive? Ref: https://opper.ai/blog/car-wash-

by dim13 2mo ago

Fails "car wash" test miserably. Ask it: > I want to wash my car. The car wash is 50 meters away. Should I walk or drive? Ref: https://opper.ai/blog/car-wash-test https://opper.ai/blog/car-wash-test

deleted 2mo ago
[deleted]
SwellJoe 2mo ago
Even very large models failed tests like this up until a year or two ago...this is a very, very, small model.
HelloUsername 2mo ago
[dead]
woadwarrior01 2mo ago
Works with the Bonsai-27B-1bit-mlx model on my mac. Edit: this is running with MLX, and not with WebGPU on the browser, also it is a fail. """ You should *walk*. Here’s why: - *Distance*: 50 meters is very short — about 0.3 miles or 0.5 kilometers. - *Time*: Walking takes roughly 2–3 minutes. Driving would take longer due to parking, starting the engine, and maneuvering. - *Effort*: Walking is light exercise and avoids parking hassle. - *Safety*: Less risk of misjudging parking space or damaging the car. Unless you have a disability that makes walking difficult, walking is the most practical and efficient choice. """
- geon 2mo ago
  How is that not a fail?
  - geon 2mo ago
    [flagged]
  - whstl 2mo ago
    I think GP also failed the car wash test.
Xalutiono 2mo ago
Thats absolutly not the point of these models. They are here to route things, do tool calls, ask experts.
- OutOfHere 2mo ago
  That is completely false. It is a reasonable type of question and a reasonable expectation. You can't route correctly if you can't do reasoning. Imagine a prompt: ``` INSTRUCTION: You're a customer service agent that routes customer service requests. CUSTOMER MESSAGE: I haven't received my order and want a refund. CONTEXT: A signature is required at delivery. No one was available to sign. You can route to one of: * Billing & refunds * Shipping & tracking * Order cancelation * Level II customer service ``` The dumb model, e.g. 1.7B, wrongly routes to Billing. The smart model, e.g. GPT-5.6-Medium, correctly routes to Shipping. It matters. In the real world, the CONTEXT will even be 10-100x larger and noisier.
  - Xalutiono 2mo ago
    IT is not black and white and just because you are able to bring up a random example doesn't mean you are right. These models are build to be small, we have a lot of diverse tasks which do not need fable or whatever because if you are a company and you have to route 100.000 requests per day, costs matter. And these small models, if they are not good enough for my specific use case, i can easily finetune (which i have btw.). You can also easily solve your example by having a second stage. If the customer is unhappy about the first thing or an employee flags it as 'wrong', you can either send it automatically to a more expensive model or the employee fixes it and you got automatic training data.

by dim13 2mo ago

Fails "car wash" test miserably. Ask it: > I want to wash my car. The car wash is 50 meters away. Should I walk or drive? Ref: https://opper.ai/blog/car-wash-test https://opper.ai/blog/car-wash-test

deleted 2mo ago

[deleted]

SwellJoe 2mo ago

Even very large models failed tests like this up until a year or two ago...this is a very, very, small model.

HelloUsername 2mo ago

[dead]

woadwarrior01 2mo ago

Works with the Bonsai-27B-1bit-mlx model on my mac. Edit: this is running with MLX, and not with WebGPU on the browser, also it is a fail. """ You should *walk*. Here’s why: - *Distance*: 50 meters is very short — about 0.3 miles or 0.5 kilometers. - *Time*: Walking takes roughly 2–3 minutes. Driving would take longer due to parking, starting the engine, and maneuvering. - *Effort*: Walking is light exercise and avoids parking hassle. - *Safety*: Less risk of misjudging parking space or damaging the car. Unless you have a disability that makes walking difficult, walking is the most practical and efficient choice. """

geon 2mo ago
How is that not a fail?
- geon 2mo ago
  [flagged]
- whstl 2mo ago
  I think GP also failed the car wash test.

Xalutiono 2mo ago

Thats absolutly not the point of these models. They are here to route things, do tool calls, ask experts.

OutOfHere 2mo ago
That is completely false. It is a reasonable type of question and a reasonable expectation. You can't route correctly if you can't do reasoning. Imagine a prompt: ``` INSTRUCTION: You're a customer service agent that routes customer service requests. CUSTOMER MESSAGE: I haven't received my order and want a refund. CONTEXT: A signature is required at delivery. No one was available to sign. You can route to one of: * Billing & refunds * Shipping & tracking * Order cancelation * Level II customer service ``` The dumb model, e.g. 1.7B, wrongly routes to Billing. The smart model, e.g. GPT-5.6-Medium, correctly routes to Shipping. It matters. In the real world, the CONTEXT will even be 10-100x larger and noisier.
- Xalutiono 2mo ago
  IT is not black and white and just because you are able to bring up a random example doesn't mean you are right. These models are build to be small, we have a lot of diverse tasks which do not need fable or whatever because if you are a company and you have to route 100.000 requests per day, costs matter. And these small models, if they are not good enough for my specific use case, i can easily finetune (which i have btw.). You can also easily solve your example by having a second stage. If the customer is unhappy about the first thing or an employee flags it as 'wrong', you can either send it automatically to a more expensive model or the employee fixes it and you got automatic training data.