Keep Datafiles Sorted

[sruffell]

The datafiles and inclusions in the datafiles are not guaranteed to be kept in sorted order in light of adds / removes. This meant that if a user in the future adds an interval, and then attempts to iterate over the database, the order was not guaranteed to be preserved.

This is a draft PR since it is built on the work of the following PRs:

• Idempotent start #352
• #319 delete interval silently fails #347
• Introduce 'journal.size' configuration item #307

[lauft]

The datafiles and inclusions in the datafiles are not guaranteed to be kept in sorted order in light of adds / removes. This meant that if a user in the future adds an interval, and then attempts to iterate over the database, the order was not guaranteed to be preserved.

Can you specify this a bit/provide an example script?

[sruffell]

The following section shows the test that would fail. In it, Database::addInterval () is called with intervals out of order, but then when trying to iterate over the database, the order isn't sorted, which is implied by the code that does not want to load the entire database.

4cce6b2#diff-de69d8da4e4696c24f8977414a514e29R39-R77

Granted, I cannot currently think of a way for a user of the timew binary to create this condition, but I think a future user of the Database object shouldn't have to commit and reload the database in order to make sure newly added intervals are sorted.

I first was wondering about this when writing b43fdf2#diff-41766cdf08b0c4d6c7a51271c51b9f85R117. I thought why can't I just use lower_bound or otherwise stop looking through the datafile if I've passed the point the new interval should be. I think this also could be an issue if we implement the import command, since I do not believe we should assume the input will be in sorted order relative to entries that may already be in the target database.

[sruffell]

The datafiles and inclusions in the datafiles are not guaranteed to be kept in sorted order in light of adds / removes. This meant that if a user in the future adds an interval, and then attempts to iterate over the database, the order was not guaranteed to be preserved.

Can you specify this a bit/provide an example script?

Actually, I was able to conceive of an example that shows the error against the current dev branch.

The following script will report the wrong running time. I added an interval with the same tags as the current interval in the past. Because the newly added interval was placed at the end of the collection of the database, when interval summarize is walking backwards to show the totals it shows both the current interval (30 min) and the interval that was added a month ago (1 hour).

#!/bin/sh
set -x

export TIMEWARRIORDB=/tmp/timewarriordb
rm -rf ${TIMEWARRIORDB}
mkdir -p ${TIMEWARRIORDB}/data
:> ${TIMEWARRIORDB}/timewarrior.cfg

src/timew start :quiet 1h ago proja
src/timew start :quiet 30min ago projb
src/timew track $(date -d '-2 months' +%Y-%m-%dT%H:%M:%S) - $(date -d '-2 months + 1 hour' +%Y-%m-%dT%H:%M:%S) projb

+ export TIMEWARRIORDB=/tmp/timewarriordb
+ rm -rf /tmp/timewarriordb
+ mkdir -p /tmp/timewarriordb/data
+ :
+ src/timew start :quiet 1h ago proja
+ src/timew start :quiet 30min ago projb
+ date -d -2 months +%Y-%m-%dT%H:%M:%S
+ date -d -2 months + 1 hour +%Y-%m-%dT%H:%M:%S
+ src/timew track 2020-06-14T07:04:02 - 2020-06-14T08:04:02 projb
Recorded projb
  Started 2020-06-14T07:04:02
  Ended              08:04:02
  Total               1:30:00

Although, still there is an issue with intervalSummarize since even when the DB is in sorted order, it only reports the total for the most recent interval.

Here is the output when the same script is run against the wip-sorted-datafiles branch:

+ export TIMEWARRIORDB=/tmp/timewarriordb
+ rm -rf /tmp/timewarriordb
+ mkdir -p /tmp/timewarriordb/data
+ :
+ src/timew start :quiet 1h ago proja
+ src/timew start :quiet 30min ago projb
+ date -d -2 months +%Y-%m-%dT%H:%M:%S
+ date -d -2 months + 1 hour +%Y-%m-%dT%H:%M:%S
+ src/timew track 2020-06-14T07:09:31 - 2020-06-14T08:09:31 projb
Recorded projb
  Started 2020-06-14T07:09:31
  Ended              08:09:31
  Total               0:30:00

I opened #368 to finally eliminate this source of confusion.