Meta.json not found and compaction failed

Prometheus server
1

Hi All,
I’ve had some trouble with my Prometheus install lately.
It started with a crash because it ran out of open files. I managed to fix that by adding LimitNOFILE=500000 to the service control file.
But now im getting these messages in the messages log.

Aug  1 08:42:32 SERVER prometheus[2659]: ts=2023-07-31T22:42:32.174Z caller=db.go:1486 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/prometheus/01H6AMNH259613RCPZ5XAAK0HE err="open /data/prometheus/01H6AMNH259613RCPZ5XAAK0HE/meta.json: no such file or directory"
Aug  1 08:42:32 SERVER prometheus[2659]: ts=2023-07-31T22:42:32.174Z caller=db.go:1486 level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/data/prometheus/01H6ATND7M7GMZMYB4AF92GNNE err="open /data/prometheus/01H6ATND7M7GMZMYB4AF92GNNE/meta.json: no such file or directory"
Aug  1 08:43:32 SERVER prometheus[2659]: ts=2023-07-31T22:43:32.178Z caller=db.go:982 level=error component=tsdb msg="compaction failed" err="plan compaction: open /data/prometheus/01H6AMNH259613RCPZ5XAAK0HE/meta.json: no such file or directory"

I was running 2.21 so i upgraded to 2.46 hoping that would help, it hasn’t yet. But it hasn’t been back up for two hours yet, so i seel what happens then.

Does anyone have any suggestions on how to fix this ?

2

It seems to be only complaining about a missing meta.json file in two folders in the datastore.
Can i delete those two folders ?