Hacker News
points by charcircuit 4 days ago | 0 comments
Scale is all you need.

That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)

[–] dannyw 4 days ago
Pretty much. See “Scaling Laws for Neural Language Models”, a must read IMO https://arxiv.org/html/2001.08361v1

Compute, data, and parameter count are the three logarithmic values that correspond to model performance.

So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.

If you find this fascinating and want to dive more, I encourage checking out nanochat (solid GPT-2 repro; minimal LoC); and trying to make architectural tweaks with a coding agent.

“Learning through experimentation” has never been easier or more fun in my opinion. If you have a GPU, even an older 8GB card (or a few bucks for Runpod), you can pretrain your own small language model and start experimenting / tweaking it.

[–] addag 4 days ago
> So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.

I don't get how changing the attention head (i.e sliding window attention instead of vanilla transformer) is only seen as a "compute" improvement. Indeed, it improves the complexity from quadratic to linear, but there was a reason why people used transformer and not sliding windows for so long (they were already there in 2021).

Making sliding window attention work with performances on par with vanilla transformers is a non trivial problem and is not solved by only putting more compute. This still requires "smart" ideas in that direction.

[–] janalsncm 4 days ago
In Qwen 3.5, 75% of the layers are linear attention. I think the theory is that linear attention is usually enough, but some regular attention layers help to handle more sophisticated long-range relationships.
[–] hujun236 4 days ago
The 75/25 split is the part I want pinned down. If a few full-attention layers are what actually bind long-range dependencies, then "you can drop most softmax attention" and "you still need some" can both be true. Is the Qwen 3.5 mix from a model card or a config, and does it say whether those full layers are early, late, or interleaved?
[–] api 4 days ago
Makes me wonder if the brain just does it with brute force.

The neocortex would be akin to something like a 100T neural network, and that’s with the maybe overly reductionist assumption that only synapses are doing anything when there’s a lot of biology happening in there. Every cell is filled with its own computation and state. It could be closer to a 10E model if all that is in play.

The neocortex is nowhere near the whole brain.

Architecturally the brain could be described as a slow but ultra-wide machine with colocated memory and thus an effectively enormous memory bandwidth. That lets it evaluate its whole parameter set continuously.

I keep thinking the brain must be algorithmically doing something super clever but maybe not. Maybe old mother nature just found the “bitter lesson.” Maybe it’s algorithmically inefficient and messy but there’s just a whole lot of it.

[–] ngruhn 4 days ago
It's running on 20 watts though. There must be some tricks in there.
[–] api 3 days ago
The simple explanation is that von Neumann architecture sucks for intelligence. It's not a von Neumann machine at all. It's an analog neural network (plus other systems in operation) with colocated memory.

A more far-out explanation is that some form of stochastic quantum computing is going on, which has been proposed at many levels by biologists. There's some work being done on this idea. It would not be the discrete qubit form we are trying to do, which is impossible in a noisy wet environment, but something analog and stochastic.

Both are possible.

Also note that our current chips, in spite of the tiny feature sizes and a ton of optimization work, are not even close to the lower limit for energy cost of computation. They still are not very efficient.

[–] captainbland 4 days ago
I think after you factor it in being on the extreme end of slow but wide (like laptop GPUs) and being a native representation of the neural network rather than a computed silicon approximation, and being more similar to a recurrent/continuous learning model rather than a huge all at once back prop, are there more tricks required?

My thought is you could probably set up something similar with today's technological capabilities but it would probably take decades to train and be expensive to develop which is commercially unacceptable.

[–] K0balt 4 days ago
Analog computation probably. Theoretically, technology like thermal wells and others that harness physics to find the local minimum energy state allow inference with multiple orders of magnitude improvements in size, energy, and speed. Rough back of the napkin gives SD-card size 1t parameter models on about 2.5 watts using thermal wells (cryogenic cooling required, afaik), achieving density approximately similar to NAND flash per parameter.
[–] peterfirefly 3 days ago
We know that the quality of the astrocytes matters a lot. It's not just about the quality of the neurons or their number or the way they are connected.

I'm afraid we don't know much more than that yet.

We have created frankenrodents with human astrocytes and they are a lot smarter than their unfranked counterparts. I bet we could learn a lot if we repeated the experiment with astrocytes from humans with different, known IQs.

[–] famouswaffles 3 days ago
Given we are comparing two different substrates, there's no reason to assume this means anything.
[–] Ohentis 3 days ago
It just seems like we're missing some pretty important stuff. The amount of "training data" humans process in out lifetimes is nowhere near enough to achieve the abilities we have, so clearly the bulk of our brains abilities must be hereditary. The only problem is: by what mechanism? Is such a complex model able to fit on a single cell? Are we somehow influenced by our mother's body in ways that are outside of that thinking? Are our brains just somehow much better architecturely than current ml approaches? The more I think about the more I feel that true AGI is probably still lifetimes of knowledge away.
[–] amelius 4 days ago
Yes if you took a transformer network and added connections until it became a full network, then it would still work (?) but it would be a lot more expensive.

(?) We actually don't know that but it is very likely.