Unknown stored as zero with subsecond precision and N timestamp

[human]

It seems that my (cygwin) 1.2.15 version of RRD stores UNKNOWN values as zero when the rrdupdate is done with timestamp N and not a hard timestamp. Steps to reproduce (some output removed to keep it short):

D:\>rrdtool create seconds.rrd --start 1153730000 --step 10 DS:rh5:GAUGE:30:U:U RRA:AVERAGE:0.5:2:10000 RRA:AVERAGE:0.5:20:10000 RRA:AVERAGE:0.5:1200:10000
 
:: store one minute of data with "hard" times. notice all values are 15 or U
D:\>rrdtool update seconds.rrd 1153730005:15
D:\>rrdtool update seconds.rrd 1153730010:U
D:\>rrdtool update seconds.rrd 1153730015:U
D:\>rrdtool update seconds.rrd 1153730020:15
D:\>rrdtool update seconds.rrd 1153730025:15
D:\>rrdtool update seconds.rrd 1153730030:15
D:\>rrdtool update seconds.rrd 1153730035:15
D:\>rrdtool update seconds.rrd 1153730040:U
D:\>rrdtool update seconds.rrd 1153730045:U
D:\>rrdtool update seconds.rrd 1153730050:15
D:\>rrdtool update seconds.rrd 1153730055:15
D:\>rrdtool update seconds.rrd 1153730060:15
 
:: store one minute of data with N times. notice all values are 15 or U again
D:\>rrdtool update seconds.rrd N:15
:: sleep 5 seconds
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:15
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:U
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:U
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:15
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:15
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:15
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:15
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:U
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:U
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:15
D:\>ping 127.0.0.1 -n 5
D:\>rrdtool update seconds.rrd N:15
 
:: fetch data stored with absolute times
:: no problem here. Unknown values are ignored and average is still 15
D:\>rrdtool fetch seconds.rrd AVERAGE --start 1153730000 --end 1153730070
                            rh5
1153730020: 1.5000000000e+01
1153730040: 1.5000000000e+01
1153730060: 1.5000000000e+01
1153730080: nan
 
:: fetch data stored with N-times
:: notice the averages are too low.
D:\>rrdtool fetch seconds.rrd AVERAGE --start -70
                            rh5
1153741040: nan
1153741060: nan
1153741080: 1.3449607709e+01
1153741100: 1.4747995162e+01
1153741120: nan

I've now changed my setup to first calculate the current data and use that in the rrdtool update. But shouldn't N also work and ignore unknown values?

[oetiker]

Hi Wilfred,

I looked at your report again, the 'problem' you see is that you are skimming a border case ... rrdtool does handle subsecond resolution for updates (and N is a subsecond timestamp), but if this does get mixed with unknown data, then rrdtool can not 'remember' subsecond unknown resolution. This means that only partially known seconds get to be fully known seconds in the final analysis ... this again leads to lower values for border cases ... check out my demo:

R=./rrdtool

$R create unknown-0-1.rrd   \
            --step 10 \
            --start 1000000000 \
            DS:DS1:GAUGE:30:0:100   \
            RRA:AVERAGE:0.5:1:5 \

$R updatev unknown-0-1.rrd 1000000005:15
$R updatev unknown-0-1.rrd 1000000011.1:U
$R updatev unknown-0-1.rrd 1000000016.9:U
$R updatev unknown-0-1.rrd 1000000021.1:15


return_value = 0
return_value = 0
[1000000010]RRA[AVERAGE][1]DS[DS1] = 1.5000000000e+01
return_value = 0
return_value = 0
[1000000020]RRA[AVERAGE][1]DS[DS1] = 1.1625003750e+01

for the first step we know only half the seconds (5) but they are all known so we get 15 as a result

for the second step we know 3.1 seconds (16.9 to 20) so here the average should be 15 as wel. but since unknown seconds get rounded down, we get

  15*3.1/4 = 11.6