in the correct timezone, with smaller milis
According to the spec, this should work because:
> The timestamp of the breadcrumb. Recommended. A timestamp representing when
> the breadcrumb occurred. The format is either a string as defined in [RFC
> 3339](https://tools.ietf.org/html/rfc3339) or a numeric (integer or float)
> value representing the number of seconds that have elapsed since the [Unix
> epoch](https://en.wikipedia.org/wiki/Unix_time). Breadcrumbs are most useful
> when they include a timestamp, as it creates a timeline leading up to an
> event.
I've done a full grep on Issue.objects, Project.objects and get_object_or_404
equivelents, and applying some common sense. The goal: avoid having
confusing/half-broken pages in the UI.
On index-usage: I've decided not to update the indexes. The assumption is:
`is_deleted` items will be a tiny minority of items in general, making the
cost/benefit analysis not turn out favorably (just scanning them out as a final
step is more efficient). Also: sqlite is able to use the correct index without
adding a special one, proof:
```
EXPLAIN QUERY PLAN SELECT [..] WHERE ("issues_issue"."project_id" = 1 AND "issues_issue"."is_muted" = (0) AND "issues_issue"."is_resolved" = (0)) ORDER BY "issues_issue"."last_seen" DESC LIMIT 250;
QUERY PLAN
`--SEARCH issues_issue USING INDEX issue_list_open (project_id=? AND is_resolved=? AND is_muted=?)
EXPLAIN QUERY PLAN SELECT [..] WHERE ("issues_issue"."project_id" = 1 AND "issues_issue"."is_muted" = (0) AND "issues_issue"."is_resolved" = (0) AND "issues_issue"."is_deleted" = 0) ORDER BY "issues_issue"."last_seen" DESC LIMIT 250;
QUERY PLAN
`--SEARCH issues_issue USING INDEX issue_list_open (project_id=? AND is_resolved=? AND is_muted=?)
```
See #139 for the 0/1 notation in the above.
(Project-indexes: not an issue, the scale is "below relevance for indexes")
Implemented using a batch-wise dependency-scanner in delayed
(snappea) style.
* no real point-of-entry in the (regular, non-admin) UI yet.
* no hiding of Projects which are delete-in-progress from the UI
* lack of DRY
* some unnessary work (needed in the Issue-context, but not here)
is still being done.
See #50
Counting incurs looking at all records which is too expensive if you have e.g.
1_000_000 issues.
Note that we take a different approach than the one for Events (where we
count-with-timeout). Reason for switching:
https://sqlite.org/forum/forumpost/fa65709226
For Events we have a known count for the non-query case (denormalized/counted
value), so we preserve what we had there. For Issues the trouble of keeping
counts right for muted/etc. is not (currently) worth it.
This will hopefully help when getting issue-reports for those that
have not set up dogfooding.
See [Dogfooding Bugsink](https://www.bugsink.com/docs/dogfooding/)
no need to calculate event_qs_count (which is potentially expensive) if that's
not used in display.
when counting was moved from the template to the view (in 1eea9268a5) it was
made unconditional; here we restore that behavior.
When searching by tag, there is no need to join with Event; especially when
just counting results or determining first/last digest_order (for navigation).
(For the above "no need" to be actually true, digest_order was denormalized
into EventTag).
The above is implemented in `search_events_optimized`.
Further improvements:
* the bounds of `digest_order` are fetched only once; for first/last this info
is reused.
* explicitly pass `event_qs_count` to the templates
* non-event pages used to calculate a "last event" to generate a tab with a
correct event.id; since we simply have the "last" idiom, better use that.
this also makes clear the "none" idiom was never needed, we remove it again.
Results:
Locally (60K event DB, 30K events on largest issue) my testbatch now
runs in 25% of time (overall).
* The effect on the AND-ing are in fact very large (13% runtime remaining)
* The event details page is not noticably improved.
Done by denormalizing EventTag.issue, and adding that into an index. Targets:
* get-event-within-query (when it's 'last' or 'first')
* .count (of search query results)
* min/max (for the first/prev/next/last buttons)
(The min/max query's performance significantly improved by the addition of
the index, but was also rewritten into a simple SELECT rather than MIN/MAX).
When this code was written, I thought I had spectacularly improved performance.
I now believe this was based on an error in my measurements, but that this
still represents (mostly) an improvement, so I'll let it stand and will take
it from here in subsequent commits.
In b76e474ef1, the event-navigation was changed into the next/prev idiom (I
think completely, i.e. also from the .html files, but did not check) but the
elif structure and error message did not fully reflect that (it still talked
about digest_order/id, but nav is now one of the primary methods)
I briefly considered removing the lookup-by-digest-order-only, but I figure it
may come in handy at some point (if only for users to directly edit the url)
and did not check whether this is actually unused.
"possibly expensive" turned out to be "actually expensive". On 'emu', with 1.5M
events, the counts take 85 and 154 ms for Project and Issue respectively;
bottlenecking our digestion to ~3 events/s.
Note: this is single-issue, single-project (presumably, the cost would be lower
for more spread-out cases)
Note on indexes: Event already has indexes for both Project & Issue (though as
the first item in a multi-column index). Without checking further: that appears
to not "magically solve counting".
This commit also optimizes the .count() on the issue-detail event list (via
Paginator).
This commit also slightly changes the value passed as `stored_event_count` to
be used for `get_random_irrelevance` to be the post-evication value. That won't
matter much in practice, but is slightly more correct IMHO.
Triggered by issue_event_list being more than 5s on "emu" (my 1,500,000 event
test-machine). Reason: sorting those events on non-indexed field. Switching
to a field-with-index solved it.
I then analysed (grepped) for "ordering" and "order_by" and set indexes
accordingly and more or less indiscriminately (i.e. even on tables that are
assumed to have relatively few rows, such as Project & Team).
