Gemini 1206 Exp has a 2 million token context window, even if that isn’t the effective context it probably performs much better in that regard than gpt 4o and such. Haven’t tested yet because I don’t want to get ratelimited from ai studio incase they monitor that
Frankly the “shorter” conversations I had at a few tens of thousand of tokens were already noticeably more consistent than before, e. g. it referenced previous responses significantly later
2 days ago and I might already have to adjust the timelines.
Nvidia’s new Digits costs 3K and is the size of a mac mini. Two of them can supposedly run a 400B parameter language model which is crazy. So maybe the hardware issues aren’t as persistent for robotics.
And also Hailuo has a single-image reference mode now that works like a lora. It’s super consistent for faces, even if the rest is a bit quirky.