Prometheus cannot scrap offline data from openmetrics

General Help/Support
1

We have a system where we export prometheus data for offline handling (troubleshooting) in JSON format.
We export the JSON data and convert it to openmetrics, which works good.
After that we use promtool to convert the openmetrics file to create a file which can then be loaded to an offline prometheus data.

Normally we have more than 9k data points, but we tried with lesser but no luck

promtool tsdb create-blocks-from openmetrics openmetrics.mod2 ./output
This generates two files
01F5TQ6KR4CQE0FZXXVXK0B6ZK
01F5TQYDS0S2XF6V913DPW8CFT
Then load the files to prometheus, in ./data . using prometheus-2.26.0.linux
We restart prometheus server but prometheus couldnot scrape any metrics from the new data.

level=info ts=2021-05-16T14:19:23.090Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1620986400047 maxt=1620987233502 ulid=01F5TQ6KR4CQE0FZXXVXK0B6ZK
level=info ts=2021-05-16T14:19:23.090Z caller=repair.go:57 component=tsdb msg=“Found healthy block” mint=1621163132601 maxt=1621166400000 ulid=01F5TQYDS0S2XF6V913DPW8CFT
level=info ts=2021-05-16T14:19:23.124Z caller=head.go:696 component=tsdb msg=“Replaying on-disk memory mappable chunks if any”
level=info ts=2021-05-16T14:19:23.125Z caller=head.go:710 component=tsdb msg=“On-disk memory mappable chunks replay completed” duration=1.777456ms
level=info ts=2021-05-16T14:19:23.125Z caller=head.go:716 component=tsdb msg=“Replaying WAL, this may take a while”
level=info ts=2021-05-16T14:19:23.151Z caller=head.go:742 component=tsdb msg=“WAL checkpoint loaded”
level=info ts=2021-05-16T14:19:23.153Z caller=head.go:768 component=tsdb msg=“WAL segment loaded” segment=2 maxSegment=6
level=info ts=2021-05-16T14:19:23.179Z caller=head.go:768 component=tsdb msg=“WAL segment loaded” segment=3 maxSegment=6
level=info ts=2021-05-16T14:19:23.181Z caller=head.go:768 component=tsdb msg=“WAL segment loaded” segment=4 maxSegment=6
level=info ts=2021-05-16T14:19:23.184Z caller=head.go:768 component=tsdb msg=“WAL segment loaded” segment=5 maxSegment=6
level=info ts=2021-05-16T14:19:23.186Z caller=head.go:768 component=tsdb msg=“WAL segment loaded” segment=6 maxSegment=6
level=info ts=2021-05-16T14:19:23.187Z caller=head.go:773 component=tsdb msg=“WAL replay completed” checkpoint_replay_duration=25.509896ms wal_replay_duration=35.606615ms total_replay_duration=62.922454ms

Regards,
Surojit

2

Can you please explain more the issue? Thanks.

3

The idea was to load promethes data from server A to server B.
Server A scrapes data on from different pods.

Server B is in a remote site which wants to load the same data (offline) and do post processing of data using grafana etc tools.

What is the best way to load data from the live server A to server B
Regards,
Surojit

4

thanks for the awesome information.