points by skeeter2020 15 hours ago | 0 comments
Why are you writing tests on the actual provider? IME these types of tests are slow, brittle and don't give you much value. Even if you get a true positive, what can you do about it?
[–] gavinray 14 hours ago
[–] dionian 15 hours ago
I assume integration tests.
[–] viccis 12 hours ago
When it comes to platforms like Databricks, integration tests are honestly the only useful ones. Grug remains vindicated as always.
Our connectors receive a generic query IR from a Datafusion-based engine
Each connector is then responsible for faithfully translating the query IR into the semantically equivalent query for the underlying data source
For a SQL DB this would be a SQL query, for MongoDB a JSON predicate query, for csv/json files it would be an in-memory predicate engine, etc
Running the generated query against the target data source and asserting that this doesn't throw an error and that the records returned are both correct and in the right order is the true test