I was hesitant, so I went with a hassle-free return route when I bought mine.

My stack uses a local llamacpp ROCM build, modified by https://github.com/stew675/llama-cpp-rdna-boosts

The build uses my machine's ROCM and I just link it instead of embedded all that in an image, like the published ones do.

Then deploying it via docker compose to my local machine. The models I want to run are stored in a docker volume, so it's easy to switch the model while keeping the same build.

The biggest negative for me is that I don't think it's enough to do parallelization through llamacpp (I'd prefer to work on one thing faster than two things slowly). I haven't gotten vllm working yet. Also, once you have one, you constantly start thinking about a second card...