Commit Graph

279 Commits

Author SHA1 Message Date
Klaas van Schelven
7b7cd66dfb Project quota: pick up on settings-changes 2026-01-07 14:02:43 +01:00
Klaas van Schelven
7d550708f1 Rename 'count' for explictness (pure refactoring) 2026-01-07 11:41:47 +01:00
Klaas van Schelven
802768f63d Remove 'matter of taste' comment and max()
the amount of extra mental overhead this introduces is not offset by the
gain in correctness
2026-01-07 11:40:54 +01:00
Klaas van Schelven
1d261b4b7c Setting & check for site-wide per-month event ingestion maximum 2026-01-07 11:29:49 +01:00
Klaas van Schelven
fb5424db22 Renames/comment rewrites for understanding (pure refactoring) 2026-01-07 09:23:53 +01:00
Klaas van Schelven
e6c163c674 When minidumps feature is turned off, don't 500 when sent
Fix #293
2026-01-06 19:33:07 +01:00
Klaas van Schelven
5cf185388e Fix exception for unsupported envelope items
See #293
2026-01-06 16:23:45 +01:00
Klaas van Schelven
b7f6331cc6 Fix never_evict for the "conditional ummute" case
The key phrase from our codebase was:

> # .save() will be called by the caller of this function

But this wasn't the case for the conditionally called path.

Adds data-fixing migration too.

Fix #292
2026-01-06 11:07:53 +01:00
Klaas van Schelven
4fe8bd3fad ingest ParseError: don't raise a 500; make this the SDK's problem (400)
500 was just fine when I was still calibrating the envelope parsing, but AFAICT it's
correct now, so I'd rather not get notified about 'remaining problems' (which are SDK problems)
2025-12-09 15:58:48 +01:00
Klaas van Schelven
ec6d480cd1 Cleanup: harmonize import-style for exceptions 2025-12-09 15:56:02 +01:00
Klaas van Schelven
bee889f2cd Raise 413 for the 'content too large' case 2025-11-26 12:45:22 +01:00
Klaas van Schelven
4137565de9 Note about (crashpad/minidump) guid 2025-11-16 19:54:13 +01:00
Klaas van Schelven
8283b80b35 Minidump API Endpoint: custom/extra fields support 2025-11-16 09:29:20 +01:00
Klaas van Schelven
661d83bd93 minidumps: FEATURE flag 2025-11-15 13:33:49 +01:00
Klaas van Schelven
ab065a6329 api_catch_all: header-based
rather than try-and-recover, just look at the headers and show body/POST etc.
this avoids hard-to-reason about situations where either of those won't work
because the other has already been executed; in combination with reasoning
about max size usage the explicit solution is simply easier to reason about.

further:

* makes api_catch_all one of the content_encoding-ready views.
* implement a max length for the ingest api view
2025-11-11 15:25:51 +01:00
Klaas van Schelven
937df4cbb8 minidump endpoint: support content encoding
adds readline() method to GeneratorReader (ChatGPT-generated; eyeballed for
correctness) to match the Django FILES/POST handling expectations.
2025-11-11 13:50:07 +01:00
Klaas van Schelven
72aab81d7d Add ContentEncodingCheckMiddleware 2025-11-11 13:39:44 +01:00
Klaas van Schelven
f5d7b430f2 Merge branch 'main' into minidumps 2025-11-11 10:07:06 +01:00
Klaas van Schelven
d5db5e328b Merge branch brotli-bombs-tests 2025-11-11 10:01:23 +01:00
Klaas van Schelven
54c96eb680 Minidump upload: more explicit errors (and logging) 2025-11-11 09:48:00 +01:00
Klaas van Schelven
690a92a1f9 Merge branch 'main' into minidumps 2025-11-09 21:56:18 +01:00
Klaas van Schelven
0432451e8e Fix inefficient bytes concatenation when KEEP_ENVELOPES != 0 2025-11-09 21:11:35 +01:00
Klaas van Schelven
a6ead89ca8 Remove event.debug_info
basically unused
2025-11-09 20:58:39 +01:00
Klaas van Schelven
60be8fa4a4 Add end-to-end test for brotli/envelope
proving that 2.0.5 solves the problem of (well-formatted) bombs
2025-11-08 23:08:49 +01:00
Klaas van Schelven
1aa8e95892 Assign local variable for easier debugging 2025-11-08 20:37:51 +01:00
Klaas van Schelven
cb8f913cbe Document 2 TODOs 2025-11-05 13:23:31 +01:00
Klaas van Schelven
c4cf038a93 minidump after-digest cleanup
envelope-based happy path only
2025-11-05 12:59:42 +01:00
Klaas van Schelven
48a818bed1 Tests for envelope minidump API 2025-11-05 12:36:17 +01:00
Klaas van Schelven
2e2a8cfeeb envelope endpoint tests: slight cleanup 2025-11-05 12:09:43 +01:00
Klaas van Schelven
d807ea2c50 Minidump: via envelope interface
See #82
2025-11-05 11:10:14 +01:00
Klaas van Schelven
d945b39259 Dead code removal
this was inlined in 7f831f52d4
2025-11-05 09:27:10 +01:00
Klaas van Schelven
31596a9b44 /store/ endpoint: non-immediate digestion
this makes this consistent with the work we did in the previous commit
at the price of being slightly more inefficient.

but it's a deprecated endpoint anyway
2025-11-05 09:20:35 +01:00
Klaas van Schelven
7f831f52d4 Remove DIGEST_IMMEDIATELY option
Although DIGEST_IMMEDIATELY=True is theoretically a nice thing to
have, the upkeep is not worth it now that we're about to introduce
minidump ingestion.

The only thing that you're saving is the round-trip via the filesystem,
but performance of that is negligable, and if you're configuring
DIGEST_IMMEDIATELY you're actually _not_ in the performance-critical path
anyway.

Getting rid of it _also_ harmonizes/reduces the number of paths to test.

It's approximately 1% of our installed base.
2025-11-05 09:03:17 +01:00
Klaas van Schelven
de9a37aab6 Fix tests for UnclosableBytesIO
i.e. fix tests for 391e22bcf0

(the changes in the present commit in `ingest/tests.py` are not strictly necessary
but they are principly right)
2025-11-05 08:39:40 +01:00
Klaas van Schelven
391e22bcf0 parser: event_output_stream closing pushed in
in preparation of the minidump handling through the envelope path,
which probably requires dealing with the whole envelope as a whole.

"theoretically" this might be less efficient, but [a] see the notes
at the top of the parser on how we think about streaming parsing and
[b] the inefficiencies are in the "immediate" path anyway (which has
the assumtion that it's not in high-performance envs).

"blind commit" (tests not run against this commit).
2025-11-04 14:51:21 +01:00
Klaas van Schelven
e7aad45db2 Minidumps: PoC for minidump 'endpoint'
See #82
2025-11-04 10:47:04 +01:00
Klaas van Schelven
20e065c6e8 Comment about FileNotError 2025-10-27 21:53:31 +01:00
Klaas van Schelven
b0b2573d17 Releases API
Fix #191
See #146
2025-09-11 09:55:15 +02:00
Klaas van Schelven
829cea1a80 detection of a new release through an event ⇏ triggering of a TurningPoint
This more exactly expresses semantics by itself, and is also in preparation of
creating releases through the API (which have no triggering event)

See #146
2025-09-11 09:55:09 +02:00
Klaas van Schelven
8b8b61ea3c Use safe_join in vacuum utility
given the listdir right before it: not strictly necessary
but easier to reason about than yet another 'nosec'
2025-09-02 13:30:09 +02:00
Klaas van Schelven
10658d70a5 vacuum_ingest_dir Minor stylistic post-copilot cleanup
Fix #163
2025-09-02 13:24:21 +02:00
copilot-swe-agent[bot]
257b5f1777 Implement vacuum_ingest_dir management command
(Klaas removed copilot's testcases)

See #163

Co-authored-by: vanschelven <223833+vanschelven@users.noreply.github.com>
2025-09-02 13:17:48 +02:00
Klaas van Schelven
4ad3c5efcf Hardening of Temporary-Directory Usage
Defends against certain forms of local privilege escalation, i.e.
understood to be defense in depth rather than a security issue given
the recommended ways of deploying (docker container or in a single-use
single-server)

Fix #174

See https://github.com/python/cpython/pull/23901
2025-08-30 15:10:50 +02:00
Klaas van Schelven
7594227b2b Add some tests for header validation
See #179
2025-08-02 22:25:43 +02:00
Klaas van Schelven
c62b0a47e4 Remove input value from exception
Using patterns here to play well with our excellent dogfood error
tracker:

When the non-validating value is displayed in the exception value, this makes
grouping not work (every event is its own issue); the approach has no upside,
because in a dogfooded setup the local vars are known anyway.
2025-08-02 22:14:13 +02:00
Klaas van Schelven
1ef1119fd9 Bandit: nosec & explain 2025-08-01 10:08:16 +02:00
Klaas van Schelven
5fb48e1e90 sent_at validation: support 00+00
Fix #179
2025-08-01 10:01:41 +02:00
Klaas van Schelven
abb84172bb header and envelope parsers: filter, don't fail on validation failures
See #179
2025-08-01 09:48:40 +02:00
Klaas van Schelven
354af7ea0a Fix issues as reported by bandit or mark as nosec
Nothing worrying, but good to have checked this regardless
and important to have a green pipeline.

Fix #175
2025-07-30 12:16:40 +02:00
Klaas van Schelven
2fede39985 Envelope parsing: validate headers as per the docs
headers means: envelope headers and item headers.

Provides more robustness and a layer of defense-in-depth
Only those headers that we might rely on in a near future (event-based)
are included.

See #173
2025-07-29 23:46:48 +02:00