My guess is that in formally verifiable domains you can use RL-esque processes to get lots of good high-quality training data with reward signals (does it verify?), but the same does not apply to "scruffier" work. I would also imagine that the data volume of high-quality scientific work in "scruffy" domains is too low for the large language models to have a good manifold for "high-quality biology research". Would love to be proven wrong if anyone knows of anyone doing great scientific work with large language models in non-verifiable domains though.
[–] serioussecurity 3 days ago
[–] szvsw 3 days ago
In an area I am more familiar with though still not my domain: when the stiffness matrix and computational linear algebra methods collide, it unlocks a world of computational structural design and analysis by commoditizing the underlying mathematics. This is a simple example, but you get the idea.
Obviously with CFD, mechanics, etc there is always massive work being done to try to unlock new computational methods, faster/parallelizable computational methods, differentiable ones, etc etc, but the general feeling is that the core physics are well modeled (NS millennium problem not withstanding) and are not lacking in predictive power for useful applications. I would have assumed that the same would be generally true for computational biology etc, so just curious to hear you elaborate more. It seems plausible to me that turning the brute force of “the models” (as in, swarms of AI compute) to work on/develop/solve/address whatever representational problems there are in the computational approximations/numerical methods you mention, particularly if that is a bottleneck to performing more extensive methodological/experimental exploration in silico to enable better exploitation/use of limited resources in wet lab.
Again, I know nothing about computational biology, just stating my intuition and curious what your take is. Probably the limits of my knowledge can be demonstrated by the fact that the only modeling methods that come to mind as being vaguely relevant would be, like,… molecular dynamics, maybe discrete element method in a weird way but doubtful, and various pop science understandings of protein folding etc.
[–] smj-edison 3 days ago
1. Biology has a very nasty habit of not cleanly abstracting. It's not like we can treat every protein as a "diffuse mass" without losing a ton of important interactions. This is unlike something like CFD where you can treat a mass of water molecules with certain constants. Think of something like cancer: a small mutation to a DNA strand can cascade into a tumor. You have to deal with the mutation happening at an atomic level while scaling to a whole organism (yes I know there's models that can approximate this, but they can't capture all the interactions).
2. This brings us to modelling at the atomic level. To model a system of atoms with the Schrödinger equation, it naively scales O(6^n) per particle. Note that means every nucleus and electron, so if you have say 10 carbon atoms and 22 hydrogen you're already at 10^127 operations. We can cut corners a bit and end up with something like CCSD which scales O(n^7), or cut more corners and scale O(n^3) for DFT. DFT is pretty much the last method where bond forming and breaking is modelled, so after more corner cutting you lose losing reactions, which are important for understanding why something is happening. Most protein folding simulations (moreso prior to alphafold) run a very crude approximation that models atoms as balls on springs (things like AMBER and CHARMM), which obviously can't handle reactions.
3. If this wasn't bad enough, you also can only move through time at about 1 femtosecond (10^-15), so it takes a ton of simulation steps to get even a nanosecond in. Forget millisecond or second lengths where things start to happen.
So, while the math of the physics exists, computationally it's practically intractible. There's some exciting work with something called Machine Learning Interatomic Potentials (MLIP) which approximate DFT using a neural network which are about 1000x faster than DFT, and iirc scale much more linearly (look at things like MACE and orbmol if you're curious). So we're making progress, but we've gone from so ridiculously impossible to pretty much impossible for large systems.
I have a couple ideas about developing custom microchips to make this faster, which is why I'm pursuing this, but there's still a massive way to go!
[–] Daniel_Van_Zant 3 days ago
Systems sciences are ones where each scale has lots of things with unique properties. If you think of “biological things between 2-4m” the amount of unique objects with unique properties and unique interactions is absolutely mind boggling. These sciences generally do not lend themselves to clean highly parsimonious models. The best you can do is coarse-grain a thing as a system (some very roughly defined set of interactions between different things) with certain behavior while accepting that you are losing enormous amounts of predictive power.
I suspect AI will remain worse at systems sciences for a long time because the “possible space” of models for a particular system is much larger than in the reductionist sciences and navigating it takes a human taste and instinct that LLMs don’t seem to have (yet).
[–] smj-edison 3 days ago
I don't think LLMs will just magically be able to connect all the dots, because "garbage in, garbage out" still applies. And there's just too many invisible intermediate states, there is literally just not enough information to fully constrain a model. Hence my interest in scaling computational chemistry to get these intermediates :)
[–] IAmBroom 2 days ago
Your "reductionist sciences" I would instead phrase as "exact sciences", where the input variations are known (even the most generous particle theory only has a 'few' fundamental particle types, such as up-quarks and photons) and their interactions are either well-known (Fermi Principle is apparently inviolable), or reasonably expected to be known someday (exact relationship of gravitons, IF they exist, with other particles).
The "inexact sciences" use either truly infinite, or practically uncountable, inputs - such as the possible shapes of a single protein, which affects the outputs.
You could criticize this grouping by saying the number of particles in the Universe is practically uncountable, but sufficiently accurate statistical models can be built (a galaxy exerts <this much> gravitational force). We can't do the same with the "inexact sciences": we can't predict how a mole of protein will react with a mole of enzymes, without physical testing. OK, maybe for some very low levels of complication this could be reliably predicted, but in general it may be a !P problem.
[–] minimaltom 3 days ago
Is there anything like a getting-started for working with MACE or orbmol, a 'hello world'? Like maybe simulating Allolactose binding to lac1 ?
[–] smj-edison 3 days ago
1. The first thing you need to do is start with some atomic structure. You can often find pre-existing pieces (through websites like pubchem), but you need to cobble them together. I use chemcraft and gaussview for this, but those are paid options. I've heard good things about the FOSS program Avogadro though.
2. Once you've built the structure of interest, you want to calculate properties of it. Note that we only infrequently simulate the atom's movement, since that's quite expensive. There's a lot of techniques that let us get the most bang for buck, like calculating transition states, or states of lowest energy which we call intermediates, or running something called an IRC, which follows the molecule to its lowest energy point (note we pretty much always think about these things in terms of energy). The traditional programs I use for this are Gaussian and ORCA (ORCA is free but you do have to create an account on their forum). Newer methods include MACE and orbmol, which are pretty much only used for trajectories (e.g. running through time) as far as I know. This newer crop of programs are much more FOSS, but they aren't as rigorous so my lab uses them only as a "rough draft". I pulled in one of my current notebooks if you're curious to try it out: https://github.com/smj-edison/qc-tools/blob/main/examples/ml...
3. Interpreting results/debugging problems. This is currently done in my workflow using a python script to extract values from the job runfiles (oh yeah, did I mention the terrible DX? A lot of this stuff is done by hand-writing a job text file and then using regex to parse the logfile). We do a lot of analysis based on relative energies (this is something called transition state theory of reactions).
I definitely glossed over a lot of details, but this is long enough as is. Definitely let me know if you have questions! If you do want a "hello world" though, I'd definitely check out the notebook I linked to. You'll have to create a file called "start.xyz" with some sort of chemistry program (or use AI to pull one for you), and then you can run the system through time. I would use a system I'm researching right now, but I'm not sure what our lab's policy is on broadcasting things to the world.
[–] selimthegrim 3 days ago
[–] gewetensleegte 2 days ago
the NS problem is not physics and not predictive, it happens at scales smaller than molecules, so it doesn't occur in real fluids
also the "NS millennium prize problem" hasn't been solved until OpenAI actually gets it published in a Math journal, and I would say it has been ruined until they do
[–] IAmBroom 2 days ago
1. How is it not physics? It's goal (as laymen me crudely understands it) is to determine if a singularity could be physically realized in a perfect liquid.
2. Why does it happen at scales smaller than molecules? Unless it requires properties that a liquid made of particles sized like real atoms cannot achieve, I don't understand.
TY.
[–] Frost1x 3 days ago
With that said part of the reason some domains may not have robust cheap verifiers is because they’re difficult to produce already, so it may be a bigger lift.
[–] parineum 3 days ago
[–] lioeters 3 days ago
[–] parineum 3 days ago
You've got to procure, use and dispose of a ton of equipment. And it all costs a good amount of money.
[–] jacquesm 3 days ago
[–] lioeters 2 days ago
[–] jacquesm 2 days ago
https://globalmecho.com/lights-out-lab/
Crazy stuff.
[–] lioeters 1 day ago
(Ah, one of their clients is FANUC, a Japanese robotics company I learned about from the Wikipedia article on lights-out manufacturing.)
"Dark factory" is the perfect term, that the labs can operate without lights, heating, and other human needs - but I also like how it has a cyberpunk feel. Another aspect I find attractive is that it treats a laboratory as an automated system, like a program that is implemented in hardware and software (and "wetware", humans participating in the system) - probably including some kind of AI as part of the process.
How about a "lights-out company", or dark business. (Well the latter sounds too suspicious, haha.) Surely there are people working on automated business systems, modelling it as a machine or program with inputs and outputs, self-modifying based on reward functions, adaptive and evolutionary.
Now that the production of software is getting automated, mechanized and industrialized, I've heard people say that the differentiating factor for successful software is the domain-specific knowledge of the businesses where it's being applied. Perhaps that's becoming true of mathematics too, where pure maths - like software for developers - is dominated by automation, the cost of production so low that humans cannot compete with the machines. The term "software factory" is pretty old, but it's taken on a new meaning where the role of humans is mainly managing an automated system, often with language models working day and night without requiring much oversight.
[–] Cu3PO42 2 days ago
[–] kidel001 2 days ago
[–] Creamsicle47 3 days ago
[–] kypro 3 days ago
Obviously it's easy to understand why having strong abilities in maths and coding can help with solving problems in physics. But it's also true of knowledge and mental models more broadly.
For example, say somehow you managed to train a frontier model without any data about elephants. If you did this it would be relatively easy to later train elephants into the weights. You'd basically just need to train it about elephants at a very superficial level. After all, it it already knows what a mammal is. It knows what legs are. It knows what eyes are. It understands how it's digestive system works. All the things that makes the biology of elephants difficult doesn't need to be learnt from scratch.
I assure you people on HN will be surprised at how quickly frontier models will become competent in domains that are harder to RL. They won't need anywhere near as much data as people currently believe.
[–] Kotlopou 3 days ago
[–] nomadpenguin 3 days ago
[–] jacquesm 3 days ago
[–] rcpt 3 days ago
AI has dominion over the digital world, when it can verify things in Lean it's able to crush problems like they're a game of Go.
But when the real world and digital world don't quite connect AI is bottlenecked.
[–] crabmusket 3 days ago
[–] kelseyfrog 3 days ago
[–] rcpt 3 days ago
[–] itchyjunk 3 days ago
[–] Daniel_Van_Zant 3 days ago
[–] pfdietz 2 days ago
https://www.science.org/content/blog-post/chemical-biology-e...
Many of the harnesses used to search for solutions rely on quick/cheap correctness checks to discard incorrect hypotheses. With something like biology, you might need to run a wet lab experiment to reject an idea. This can add days per iteration, slowing progress down by a factor of 100 or more.
The sad reality is that many computational approximations for biological phenomenon are relatively weak and lack the predictive power we need.