Commit Graph

2632 Commits

Author SHA1 Message Date
Astrid
e91ecebb39
Merge branch 'master' into perf/export 2019-10-18 13:21:19 +02:00
Astrid
49adcd680a
Merge pull request #132 from nephanim/perf/frontpercent
Improve frontpercent performance
2019-10-18 13:20:43 +02:00
Astrid
bca296533b
Merge pull request #130 from nephanim/feature/import-switches
Export/import displaynames and switches
2019-10-18 13:20:34 +02:00
Ske
49dede7ff8 Remove compiler warning from API 2019-10-18 13:19:17 +02:00
Ske
83c709b69d Sanitize more inputs 2019-10-18 13:14:36 +02:00
Noko
adea7be07c Improve export performance
Refactored ExportSystem to:
- Only fetch message counts once (instead of a query per member)
- Fetch switches using the newly refactored GetTruncatedSwitchList (gets switches and their members in one shot instead of querying for switch members one switch at a time)

- Added a new MessageCountsPerMember method to MemberStore to support the above
- Modified GetTruncatedSwitchList query to retrieve switches with no members (wasn't important for frontpercent, but is if we're reusing that method here)

This doesn't require any index changes beyond those that support GetTruncatedSwitchList, though we can see a small benefit with an index on messages covering the member column (valuable for other reasons outside of these additions).
2019-10-06 02:03:28 -05:00
Noko
ef7b825aa6
Merge pull request #1 from nephanim/feature/import-switches
Bring performance branch in line with export/import changes
2019-10-05 22:51:11 -05:00
Noko
bc0213c9a1 Added indexes to support frontpercent query changes
Note: need to determine conventions to use for index names and settings going forward. There are also some indexes in production that aren't in here and are probably good additions to a standard set.
2019-10-05 15:30:55 -05:00
Noko
845ec90c3e Improve frontpercent performance
Refactored GetTruncatedSwitchList to:
- Only fetch switches in the requested range
- Fetch switch members in bulk rather than one switch at a time

This uses a new GetSwitchMembersList method that requires the following indexes:

CREATE INDEX ix_switches_system
ON public.switches USING btree
(system ASC NULLS LAST)
INCLUDE("timestamp")
TABLESPACE pg_default;

CREATE INDEX ix_switch_members_switch
ON public.switch_members USING btree
(switch ASC NULLS LAST)
INCLUDE(member)
TABLESPACE pg_default;
2019-10-05 15:08:27 -05:00
Ske
1988b29fbc Refactor command system 2019-10-05 07:41:00 +02:00
Noko
7406f18bf5 Import switches
- ImportSystem builds a mapping of data file HID to current system HID
- Switches in a data file are reconciled with system members' actual IDs using this mapping
- SwitchStore provides a RegisterSwitches method to register multiple switches
- RegisterSwitches only imports a switch if one does not exist with the same timestamp
- The number of switches created is logged
2019-09-29 14:40:13 -05:00
Noko
633c181e84 Add displayname to PK export/import
- Add display_name property to member on PK export
- Set display_name if provided when importing from PK export file
2019-09-28 23:29:32 -05:00
xBelladonna
658bcf1faf Display guild user nickname on message card if set 2019-09-22 02:04:20 +09:30
Ske
0ec522ca0a Display member roles in message query card 2019-09-21 15:19:38 +02:00
Ske
86d0b24758 Include event ID in Sentry dump 2019-09-02 20:37:24 +02:00
Ske
7d1accd58d Fix file name of tips and tricks page 2019-08-31 19:13:37 +02:00
Ske
1bfdb43c1f Fix message model table 2019-08-31 19:13:10 +02:00
Ske
cfd8a2cfdb Fix README formatting 2019-08-31 19:12:48 +02:00
Ske
dbade9b6bc Various minor site/documentation changes 2019-08-31 12:32:28 +02:00
Grey Himmel
ad6624ce02
Order members regardless of name case
Add `toLower()` method to the sorting used in system lists, so members are ordered regardless of if the first letter of their name is upper or lowercase
2019-08-25 20:25:37 -04:00
Ske
9b0b8b4353 Fix Clyde, also without an attachment 2019-08-14 07:21:03 +02:00
Ske
74e0508065 Increase webhook name limit to 80 2019-08-14 07:16:48 +02:00
Ske
5ef8a9303d Update .gitignore 2019-08-13 23:28:24 +02:00
Ske
ec49d10d66 Add .editorconfig 2019-08-13 23:28:03 +02:00
Ske
4e69531103 Ignore invalid colors in member card 2019-08-13 21:49:43 +02:00
Ske
4d2c843442 Handle busy indicator without permission properly 2019-08-12 18:07:29 +02:00
Ske
fb7f31d42d Invalidate proxy caches on more editing commands 2019-08-12 17:49:07 +02:00
Ske
d8503001eb Show an error when a bad import URL is given 2019-08-12 17:47:35 +02:00
Ske
8396e17a87 Attempt to fix webhook invocation errors 2019-08-12 16:38:34 +02:00
Ske
5f79aaf960 Fix API service injection 2019-08-12 06:54:28 +02:00
Ske
1604500f2a Optimize Docker build files 2019-08-12 06:49:18 +02:00
Ske
5728f94e00 Fix stats command 2019-08-12 06:24:37 +02:00
Ske
a48940476c Eagerly discard messages before shard init 2019-08-12 06:10:55 +02:00
Ske
7945603cdb Include invite link in error message 2019-08-12 06:03:20 +02:00
Ske
d3f821a266 Send a request ID on error 2019-08-12 05:56:05 +02:00
Ske
145ecb91ad Refactor proxy service 2019-08-12 05:47:55 +02:00
Ske
fda98bc25f Lower message cache size 2019-08-12 05:03:18 +02:00
Ske
17c8b7b765 Properly retry client creation on failure 2019-08-12 04:42:16 +02:00
Ske
5b13c1b100 Cache the entire webhook client rather than just the token 2019-08-12 04:32:01 +02:00
Ske
1b32cd8b6d Log webhook send progress 2019-08-12 04:16:17 +02:00
Ske
c68ce1bdd3 Log gateway latency to console 2019-08-12 04:05:22 +02:00
Ske
aa04d2055e Avoid deleting messages from the store if they're not webhooks 2019-08-12 03:51:54 +02:00
Ske
423d23faf7 Cache account lookup in memory when proxying 2019-08-12 03:48:08 +02:00
Ske
5aa47278cb Revert "Attempt to speed up Docker building"
This reverts commit f5c5291877.
2019-08-12 02:44:48 +02:00
Ske
c94c6c1c41 Increase maximum connection pool size to 1000 2019-08-12 02:34:59 +02:00
Ske
213db91ade Report database query stats as tags 2019-08-12 02:33:56 +02:00
Ske
f5c5291877 Attempt to speed up Docker building 2019-08-12 02:33:28 +02:00
Ske
99ec3cd355 Instrument and log database queries 2019-08-12 02:05:30 +02:00
Ske
ea476d4051 Yield event handlers to task queue before handling 2019-08-12 01:15:55 +02:00
Ske
a689d61788 Collect webhook response time and shard latency 2019-08-12 00:57:23 +02:00