8 ms·

Does every token requires a full model computation?

• onedognight 2y ago
  No, you can cache some of the work you did when processing the previous tokens. This is one of the key optimization ideas designed into the architecture.