Hacker News
points by CMay 2 days ago | 0 comments
Did y'all do any particular work that makes these models great for moderation? One of the issues between cloud models vs local is that often people are trying to push the limit of what their hardware can do and it doesn't leave too much headroom for moderation to make the models a little more constrained for kids or schools to use locally, or just for family in general who don't want accidental unhinged results.

In the past I've noticed that some LFM were way better at moderation than much larger models. Not necessarily prejudging what is acceptable, but allowing you to interrogate content with a series of questions to customize the moderation. Other models I tested were far worse.

Decision models are like the next optimization in that direction, but none of the benchmarks for the popular models ever show how good they are at tasks like content moderation. The smaller and more efficient we can get that workload, the more that it can simply be on by default in a lot of use cases without feeling like you're giving up a lot of performance, creative freedom or accuracy. You can't have a high number of false positives or people would rather take control back.

We need local content moderation to be cool, efficient, accurate and configurable so that people can get the higher quality results which unsafe models tend to produce, while filtering out the unsafe results fast enough that it can even be done in an agentic way and fed back into the model to steer its results.

[–] agairola 1 day ago
on moderation, we run an open guardrail benchmark at raxIT Labs and the hosted d1 models do well. d1.1 is #1 of 30 at checking model output for harmful content (94.4 balanced accuracy), d1-ultra is #1 on harmful user requests (88.2). we haven't run the new 2B open weights yet so no number for those. results and method: https://dm-as-guardrails.raxitlabs.com
[–] CMay 1 day ago
This tracks with my experience with their models. Nice work. It would be interesting to see what they could do if they optimized even more for moderation like this without going so far as to hurt the model performance overall.
[–] samstevens 2 days ago
Can you describe more in terms of what moderation you want? I can say with confidence that we don't train for the typical adult-only content you can see on some "de-safe'd" models, but it's not clear to me exactly what you're asking about. We can also discuss over email if you prefer.
[–] CMay 1 day ago
I'm largely referring to the fundamental components that can be combined to arrive at a decision. Is this a real life image or text, or is it some sort of art or fiction? Is this in a context of historical events for education, etc? Is this sarcasm? Is this a real threat? Is this nudity? Does this contain violence and gore? Is this suggestive? How old is this character? Is this content sufficiently related to the content it's replying to, or is it spam? Does this response deviate too much from the input request? Is this a business logo?

There was this book called The Inquisitive Mood and it very much reminds me of the chain interrogation process you can do with models now.

This isn't exactly a single use case, it's more like there are use cases for image editing software that use generative models, there are use cases for local LLMs to get a sense for abstract user intent, or to check if a model's action is in line with policy and so on.

As I said, the Liquid AI models are already very good at a lot of this, but considering how good the models are at it, I'm curious how much y'all train on tasks that are moderation-adjacent. If y'all don't have any model that's really specifically trained for these kinds of tasks, it would be interesting to see how close y'all could get if you tried.

This is an actually important problem in the AI space especially for local development.

There are other moderation tasks that are harder, like "is this copyrighted?" or "is this a McDonald's logo?" because of how much copyrighted material there is and how many businesses there are, but it might be possible to get a good feel for "is this likely a trademark?", "is this a business logo?", "does this text seem to describe a popular character?"

To some extent you can combine a model like this with a good database to optimize the filters and identification. Then you can provide a few reference image or text examples that are closer to what you're trying to filter out and the model can say, "yeah, this content is pretty close to your examples."

Google and OpenAI have already released some of their open source safety models and they were really quite bad, from my testing. Yet, Liquid AI hasn't really leaned into promoting their models for moderation purposes despite being excellent. I know there is a lot of political intersection in moderation, especially in the past many years, but I think a model that is smart enough and excels at letting people apply their own rules with fairly good accuracy is important right now.