Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
CobrastanJorji20 hours ago | parent | context | favorite | on: Ultramarine 1 Flash As an aside, I do not like unnecessarily misleading graphs. Own your strengths and weaknesses. In the "Results" section, they choose to compare Ultramarine only to models that it's better than on every metric. They ignore models like Fable 5.1, GPT-6 Astra, Claude Opus 5.5, etc. Then, in the "Score against cost per task" graph, which is focused on price, suddenly all of those models that perform better than Ultramarine come roaring back because they are also more expensive. Just list yourselves as the second best sometimes. Your thing is impressive. Don't let the idiot marketing guy hide that behind stupid tricks. help 
digitcatphd17 hours ago | next[–] Yes, for me when they do this I immediately dismiss the model, don't even bother testing it. If you need to lie with statistics your model isn't good enough. reply lwhi15 hours ago | prev | next[–] Yes, I'm far more likely to purchase if I can understand the full picture. It's not quite as good as frontier models, but it's 60 times cheaper. Honesty and transparency are attractive. reply thatsabadlook15 hours ago | parent | next[–] Maybe you are more likely to buy things when benchmarks are honest, but openai and anthropic have been cheating at those for years. They get purchased a lot. Most purchases are emotional things, even for technical products. That's why they sell llms like it's crack, a weapon, or if you don't you will starve to death in the future. One thing I like about the Chinese and euro labs is they don't market their products in harmful ways, but that's a tangent. reply cadamsdotcom14 hours ago | root | parent | next[–] Sounds like there's a gap in the market for an honest player to fill. reply thatsabadlook14 hours ago | root | parent | next[–] It's more or less all stupid anyways. Benchmarks can give someone and understanding of what a model was trained on but they don't explain tone, consistency, the type of answers, the restrictions, what the money is going to, how good it is on your task, etc. Most use of llms is gambling. So a lot of people get hooked on their one machine or whatever despite all reasons to the contrary. Some of the most deranged LLM users I e encountered are hooked on a variety of models but still use ones from like 2 years ago for specific tasks for emotional reasons. The truth is, for 99.999% of use cases the top 10 models are more or less interchangeable. Either they can mostly do what someone wants, or none of them should probably be used for that application even if s leading lab is advertising it. reply brookst11 hours ago | root | parent | prev | next[–] Not really. Few people will make purchasing decisions based on honesty of marketing materials over actual quality of product. reply adrithmetiqa14 hours ago | parent | prev | next[–] I really would like to understand how it can be 60 times cheaper. Is that a like for like comparison? Is there a more equivalent frontier model that is closer in price? reply arendtio18 hours ago | prev | next[–] I think they just decided to use the formatting different than the others. Normally, the background highlight color indicates the leader for every benchmark, but they chose to highlight their column even though there are better models. APEX Agents => Claude Fable 5 and Gemini. reply embedding-shape13 hours ago | parent | next[–] > I think they just decided to use the formatting different than the others. 500px of the landing page is dedicated to share how they're more "token efficient" than the base model they used. Apparently this -6.5% difference is worth highlighting as the very top feature of this fine-tune they've made. How much is 6.5% percent concretely though? Assuming the six benchmarks have ~30 tasks each (180 in total), and we're doing 120 tok/s lets say, time/task for GLM-5.3 Flash would be ~25 minutes and this "token efficient fine-tune" would arrive at ~23 minutes. Again, a 6.5% difference is highlighted as the most impactful change for this fine-tune, but it's a marginal difference. They're definitely trying to mislead people into thinking this is something more than it is, otherwise this landing page would have been much more realistic and less self-patted. reply torginus16 hours ago | prev | next[–] > Ultramarine 1 Flash is Fable-class on knowledge work spanning finance, law, consulting, and operations. It is also 60x cheaper than Fable. This is not a general purpose model, it makes sense to only bench it on tasks it's supposed to be good at, which is to help out finance-related knowledge work. reply N_Lens14 hours ago | prev | next[–] That looks like extreme cherrypicking and doing this speaks against the Ultramarine team. Virtue is at a premium in these times. Not going to be trying it just based on this observed behaviour. reply jchook19 hours ago | prev | next[–] The benchmarks are not ideal. reply icantevenhold14 hours ago | parent | next[–] When are they ever? I can’t remember a time when I found a benchmark to be genuinely useful even for deterministic software reply falsemyrmidon12 hours ago | root | parent | next[–] I like them for computer games and hardware. But yeah, generally functional capabilities are much more important. reply nurettin16 hours ago | prev | next[–] This is not an aside at all. It is on point. It seems like whatever statistical trick they used also inflated gemini 3.8 to absurd scores as well. reply yieldcrv19 hours ago | prev[–] in my startup, our publicist used to make buzz feed style listicles, consistently listing us as the second or third solution but it was way deeper than that, because it wasn't actually about people looking up some search string, it was actually about predictably pinging the google alerts for the top company and their founders, and then my same publicist messaging them as a rando on linkedin or email suggesting that they join us for a fireside chat at a conference they had their hands in with the fireside chat in front of everyone, and the record of it, boosting us to the collective conscious riding off of the more established brand's presence but yeah our personal google results also looked amazing for several years too everything serves a parallel purpose, but yeah there's almost no utility in self aggrandizing on your own homepage reply
Consider applying for YC's Winter 2027 batch! Applications are open till November 2. Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact Search: