Hi all!
I’m very new to Prometheus so i’m sure i’m doing some simple misstake here… =) I want to backfill with some historical using promtool. So i have a text file in the below format.
About 150k lines, data ranging from late 2019 to today.
TYPE volume_space_physical_used gauge
volume_space_physical_used{aggr=“aggr1”,cluster=“mycluster”,instance=“localhost”,datacenter=“mydatacenter”,job=“manual”,node=“mynode”,style=“flexvol”,svm=“mysvm”,volume=“myvolume”} 28271448064 1569710476.0
EOF
I am testing on Windows, so i start it up using:
prometheus.exe --storage.tsdb.retention.time=4y
Next i run the promtool to import the data:
c:\promtool tsdb create-blocks-from openmetrics c:\new2.txt
BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE
01GF0SJFQDB1X5F53NN0XTVKPZ 1569710476000 1569711505001 17m9.001s 22 22 22 5126
01GF0SJG0QSNVBF8T9QKDADW7X 1570313870000 1570317696001 1h3m46.001s 22 22 22 5126
01GF0SJGAKMEYSKZCJ4YMH1PSG 1570920905000 1570921072001 2m47.001s 39 39 39 7957
And so on. Looks good, import seems to be working.
I then restart prometheus to check how it behaves:
ts=2022-10-10T11:23:32.331Z caller=main.go:543 level=info msg=“Starting Prometheus Server” mode=server version=“(version=2.39.1, branch=HEAD, revision=dcd6af9e0d56165c6f5c64ebbc1fae798d24933a)”
ts=2022-10-10T11:23:32.332Z caller=main.go:548 level=info build_context=“(go=go1.19.2, user=root@7c59f010d6a7, date=20221007-16:03:13)”
ts=2022-10-10T11:23:32.333Z caller=main.go:549 level=info host_details=(windows)
ts=2022-10-10T11:23:32.333Z caller=main.go:550 level=info fd_limits=N/A
ts=2022-10-10T11:23:32.333Z caller=main.go:551 level=info vm_limits=N/A
ts=2022-10-10T11:23:32.355Z caller=web.go:559 level=info component=web msg=“Start listening for connections” address=0.0.0.0:9090
ts=2022-10-10T11:23:32.360Z caller=main.go:980 level=info msg=“Starting TSDB …”
ts=2022-10-10T11:23:32.366Z caller=tls_config.go:195 level=info component=web msg=“TLS is disabled.” http2=false
ts=2022-10-10T11:23:32.366Z caller=repair.go:56 level=info component=tsdb msg=“Found healthy block” mint=1569710476000 maxt=1569711505001 ulid=01GF0SP30XVY9SHZPAFGMJJMEA
ts=2022-10-10T11:23:32.366Z caller=repair.go:56 level=info component=tsdb msg=“Found healthy block” mint=1570313870000 maxt=1570317696001 ulid=01GF0SP3ADAGAEZ0XMAHM0C4KX
ts=2022-10-10T11:23:32.366Z caller=repair.go:56 level=info component=tsdb msg=“Found healthy block” mint=1570920905000 maxt=1570921072001 ulid=01GF0SP3S8YZEMH6TJ3DEQ9KK9
and so on for a lot of blocks. Looks good, seems like it finds some stuff to work with.
Problem 1: Looks like Promethues starts deleting stuff, despite the 4y retention:
ts=2022-10-10T11:25:49.627Z caller=compact.go:460 level=info component=tsdb msg=“compact blocks” count=2 mint=1573947514000 maxt=1573949437001 ulid=01GF0T9P2AHFY6H0Z086TSV8NE sources=“[01GF0SP74KAA9H7J6064S29XSZ 01GF0SP7P0S9QEH83V13QE9412]” duration=113.2907ms
ts=2022-10-10T11:25:49.677Z caller=db.go:1526 level=info component=tsdb msg=“Deleting obsolete block” block=01GF0SP74KAA9H7J6064S29XSZ
ts=2022-10-10T11:25:49.688Z caller=db.go:1526 level=info component=tsdb msg=“Deleting obsolete block” block=01GF0SP7P0S9QEH83V13QE9412
ts=2022-10-10T11:25:49.844Z caller=compact.go:460 level=info component=tsdb msg=“compact blocks” count=2 mint=1576367824000 maxt=1576368714001 ulid=01GF0T9P92PT6Q602P3XHSZ1DN sources=“[01GF0SPAHEGB75B197DVYBBWA9 01GF0SPB2EWTS533K3TFZMJ1DJ]” duration=114.3349ms
ts=2022-10-10T11:25:49.894Z caller=db.go:1526 level=info component=tsdb msg=“Deleting obsolete block” block=01GF0SPAHEGB75B197DVYBBWA9
And so on for a bit. It doesn’t delete everything though as i can still see blocks in the ‘data’ folder.
Problem 2: I get no matches when searching, but i can see that the metric is there if i go to the metrics explorer. What i am doing wrong?
Do i need a .yml file with some parameters that match what i am loading or something?
Cheers!