Commit Graph

236 Commits

Author SHA1 Message Date
Ske
9c9e48a799 Upgrade to .NET Core 3.1 2019-12-22 00:40:57 +01:00
Ske
5fc654b758 Add index to message table to speed up count querying 2019-12-21 21:42:34 +01:00
Ske
3b72fa720b Add warning when setting proxy tags conflicting with another member's 2019-12-21 21:42:06 +01:00
Ske
530e3aa697 Fix null normalising utility 2019-12-02 16:56:35 +01:00
Ske
378cba09e5 Add channel blacklists for logging and proxying
Necessary database migrations for this commit:
    alter table servers add column log_blacklist bigint[] not null default array[]::bigint[];
    alter table servers add column blacklist bigint[] not null default array[]::bigint[];
2019-11-03 19:15:50 +01:00
Ske
53124776d4 Fix proxy tag field default type in schema 2019-11-02 23:38:25 +01:00
Ske
7c3b4b9af9 Add command to delete all switches 2019-11-02 22:46:51 +01:00
Ske
f259beae90 Fix importing members with no proxy tags 2019-11-02 20:45:47 +01:00
Ske
1a79d52bfb Add importing of Tupperbox multibrackets 2019-10-30 14:11:24 +01:00
Ske
49dc25ee02 Allow keeping proxy tags when proxying messages.
Required database migration:

Closes #75.
2019-10-30 09:26:50 +01:00
Ske
8604d25ffe Fix legacy prefix/suffix API fields 2019-10-30 09:19:49 +01:00
Ske
5d63181cac Fix exporting 2019-10-30 09:09:09 +01:00
Ske
393ee16c1b Add support for multiple proxy tags
Tangentially closes #103.
2019-10-30 08:18:59 +01:00
Noko
96b03495a4 Fix import from Tupperbox regression defect (#137)
* Fix bulk importer parameter and remove unused method

-Fixes AddSwitchesBulk post-refactor by passing in the switch timestamp instead of a set of PKMembers
-Removes unused RegisterSwitches method which was replaced by AddSwitchesBulk

* Fix import from Tupperbox regression defect

When converting a Tupperbox import file to the PluralKit format on import, this sets the Tupper's name as the member ID which seems like a reasonable default. I'm also initializing the switches collection (even though it's going to be empty in this case).

This fixes a regression defect introduced when switch import was implemented.

* Making converted Tupper DataFileMember Id a GUID instead

Member names aren't necessarily unique (even if they should be), and no reason not to use something we know will be unique...
2019-10-29 08:41:44 +01:00
Ske
545e8df6a7 Reduce the log level of various common events 2019-10-27 23:44:27 +01:00
Ske
46672d264c Refactor log channel storage 2019-10-27 23:01:20 +01:00
Ske
523469e5e6 Remove preemptive warning messages for unproxyable members
The member length limit is now long enough that it's unlikely to
hit the cap by accident. An error will still be displayed if you
attempt to perform a message proxy.
2019-10-27 21:57:31 +01:00
Ske
e6e1ab023a Fix type mismatch on system export 2019-10-27 14:49:46 +01:00
Noko
dd37c8a6d6 Fix bulk importer parameter and remove unused method (#136)
-Fixes AddSwitchesBulk post-refactor by passing in the switch timestamp instead of a set of PKMembers
-Removes unused RegisterSwitches method which was replaced by AddSwitchesBulk
2019-10-27 11:37:16 +01:00
Ske
6a73b3bdd6 Refactor data stores, merging the Store classes 2019-10-26 19:45:44 +02:00
Noko
593e9d9aef Fix edge case of importing exactly enough members to hit the cap
It's OK if import will bring us to the cap (1000), just don't let the count go beyond that.
2019-10-20 18:24:00 -05:00
Noko
406f005b4f Bulk import switches for pk;import
We're now using binary import for switches and switch_members when importing a system profile, rather than importing them one switch at a time.

This adds a pass-through method to the PerformanceTrackingConnection that can be used for other bulk import applications.
2019-10-20 14:38:43 -05:00
Noko
397da2e1fa Added max member count to limits
A given system can now have up to 1000 members. Within 50 members of that limit, a warning will display whenever a new member is created via the bot. Once the limit is reached, a final warning will appear indicating that no additional members can be created unless members are first deleted. Attempting to create a new member at that point by any method will result in an error message indicating that the limit has been reached.

Respecting this in pk;import required some restructuring to tease apart which members already exist and which ones need to be created prior to creating any members as it seems preferable to fail early and give the user the ability to intervene rather than pushing the system to the member cap and requiring manual deletion of "lower priority" members before others can be created. One consequence of the restructure is that existing members are being read in bulk which is a performance improvement of 25-70% depending on how many switches need to be imported (the more members you have, the more noticeable this is).
2019-10-20 02:16:57 -05:00
Ske
4ce4efcc61 Fix importing brackets from Tupperbox. Closes #123 2019-10-18 13:34:54 +02:00
Astrid
e91ecebb39
Merge branch 'master' into perf/export 2019-10-18 13:21:19 +02:00
Astrid
49adcd680a
Merge pull request #132 from nephanim/perf/frontpercent
Improve frontpercent performance
2019-10-18 13:20:43 +02:00
Noko
adea7be07c Improve export performance
Refactored ExportSystem to:
- Only fetch message counts once (instead of a query per member)
- Fetch switches using the newly refactored GetTruncatedSwitchList (gets switches and their members in one shot instead of querying for switch members one switch at a time)

- Added a new MessageCountsPerMember method to MemberStore to support the above
- Modified GetTruncatedSwitchList query to retrieve switches with no members (wasn't important for frontpercent, but is if we're reusing that method here)

This doesn't require any index changes beyond those that support GetTruncatedSwitchList, though we can see a small benefit with an index on messages covering the member column (valuable for other reasons outside of these additions).
2019-10-06 02:03:28 -05:00
Noko
ef7b825aa6
Merge pull request #1 from nephanim/feature/import-switches
Bring performance branch in line with export/import changes
2019-10-05 22:51:11 -05:00
Noko
bc0213c9a1 Added indexes to support frontpercent query changes
Note: need to determine conventions to use for index names and settings going forward. There are also some indexes in production that aren't in here and are probably good additions to a standard set.
2019-10-05 15:30:55 -05:00
Noko
845ec90c3e Improve frontpercent performance
Refactored GetTruncatedSwitchList to:
- Only fetch switches in the requested range
- Fetch switch members in bulk rather than one switch at a time

This uses a new GetSwitchMembersList method that requires the following indexes:

CREATE INDEX ix_switches_system
ON public.switches USING btree
(system ASC NULLS LAST)
INCLUDE("timestamp")
TABLESPACE pg_default;

CREATE INDEX ix_switch_members_switch
ON public.switch_members USING btree
(switch ASC NULLS LAST)
INCLUDE(member)
TABLESPACE pg_default;
2019-10-05 15:08:27 -05:00
Noko
7406f18bf5 Import switches
- ImportSystem builds a mapping of data file HID to current system HID
- Switches in a data file are reconciled with system members' actual IDs using this mapping
- SwitchStore provides a RegisterSwitches method to register multiple switches
- RegisterSwitches only imports a switch if one does not exist with the same timestamp
- The number of switches created is logged
2019-09-29 14:40:13 -05:00
Noko
633c181e84 Add displayname to PK export/import
- Add display_name property to member on PK export
- Set display_name if provided when importing from PK export file
2019-09-28 23:29:32 -05:00
Ske
74e0508065 Increase webhook name limit to 80 2019-08-14 07:16:48 +02:00
Ske
5f79aaf960 Fix API service injection 2019-08-12 06:54:28 +02:00
Ske
213db91ade Report database query stats as tags 2019-08-12 02:33:56 +02:00
Ske
99ec3cd355 Instrument and log database queries 2019-08-12 02:05:30 +02:00
Ske
b593c32950 Add more extensive debug-level logging
- Logging of event IDs
- Logging of command invocations (at DEBUG)
2019-08-12 00:07:29 +02:00
Ske
942022d408 Track database handles over metrics 2019-08-11 22:56:20 +02:00
Ske
200ba6d84c Add member proxy display names 2019-08-09 10:12:38 +02:00
Ske
fe22ba4f32 Truncate member profile in long system list 2019-08-04 13:52:41 +02:00
xBelladonna
29eefd7f5a Handle collision case when generating hid 2019-07-27 12:56:35 +02:00
Ske
b14666aa4f Store the ID of the trigger message in the database 2019-07-24 13:38:54 +02:00
Ske
11eabe2e3d Update D.NET version and add bulk deletion support 2019-07-21 17:16:04 +02:00
Ske
3103df856c Only log deleted messages if they were found previously 2019-07-21 03:44:26 +02:00
Ske
5118a17b84 Undo breaking metric context ID change 2019-07-21 00:11:46 +02:00
Ske
bb6378df63 Add more complex process performance tracking 2019-07-21 00:01:02 +02:00
Ske
0b4335ccaa Add logger to API service setup 2019-07-18 17:26:06 +02:00
Ske
961bfe9094 Add basic logging framework 2019-07-18 17:13:42 +02:00
Ske
a188c64a0e Clean up unused imports 2019-07-17 13:38:23 +02:00
Ske
f53344cd89 Fix front percent returning negative percentages and ranges 2019-07-17 13:37:43 +02:00
Ske
02b41413b3 Add basic InfluxDB reporter 2019-07-16 23:34:22 +02:00
Ske
2d58705e85 Add basic WIP metrics system 2019-07-16 21:59:06 +02:00
Ske
d5da3bf004 Clamp frontpercent range to actual switch range 2019-07-16 21:18:46 +02:00
Ske
b6ba24d171 Show front percent for switches with no fronter
Closes #113.
2019-07-15 21:51:41 +02:00
Ske
18e4d7c9ac Add preliminary Sentry support 2019-07-15 21:02:50 +02:00
Ske
da307c2dc9 Display member count on system card 2019-07-15 17:36:10 +02:00
Ske
7e999f0a1d Ensure switch fronter order is stable 2019-07-14 21:48:10 +02:00
Ske
ebc311ecc3 Remove message query reaction AND open DB connection when obtaining one 2019-07-14 05:23:27 +02:00
Ske
a41e20a0a3 Fix importing with no existing system 2019-07-11 22:46:18 +02:00
Ske
d829630a35 Fix database connection pool contention (maybe)
Instead of acquiring a connection per service per request, we
acquire connections more often but at a more granular level, meaning
they're also disposed of more quickly instead of staying for a long time
in case of long-running commands or leaks.
2019-07-11 21:25:23 +02:00
Ske
1bd5e7e3ca Clarify datetime format in switch move error 2019-07-10 11:09:08 +02:00
Ske
9a5152a74c Add member routes to API 2019-07-10 00:19:18 +02:00
Ske
4874879979 Add basic API, only with system endpoints 2019-07-09 20:39:29 +02:00
Ske
f4b0607572 Add basic documentation site 2019-07-03 10:25:24 +02:00
Biquet
f46fbdf7d4 Fix missing RedQuestion emoji (#110) 2019-07-02 15:55:30 +02:00
Ske
b5c9793578 Add config file loading 2019-07-01 00:55:41 +02:00
Ske
42147fd9cc Add front percent command 2019-06-30 23:41:01 +02:00
Ske
2c3c46002a Add message lookup and log channel setting commands 2019-06-21 13:49:58 +02:00
Ske
06edc9d61e Add API token commands 2019-06-20 21:15:57 +02:00
Ske
7a10a28019 Only show the two most significant delta-time components 2019-06-15 12:49:30 +02:00
Ske
f4a53ce815 Refactor date/time format constants 2019-06-15 12:33:24 +02:00
Ske
1e1ef4495f Add Tupperbox importing support 2019-06-15 11:55:11 +02:00
Ske
9be7514fb9 Fix Newtonsoft.Json NuGet dependency 2019-06-15 11:02:46 +02:00
Ske
652afffb8c Add importing and exporting function 2019-06-14 22:48:19 +02:00
Ske
cd9a3e0abd Expand system time selection logic 2019-06-13 23:42:39 +02:00
Ske
72cb838ad7 Add system time zone command 2019-06-13 20:33:17 +02:00
Ske
d109ca7b57 Add switch deletion command 2019-06-13 17:05:50 +02:00
Ske
7e9b7dcc98 Add switch commands for adding and moving 2019-06-13 16:53:04 +02:00
Ske
4c6790432b Add system linking commands 2019-05-21 23:40:26 +02:00
Ske
8b8ec80944 Move database/mapper setup code to Core 2019-05-19 22:03:45 +02:00
Ske
72a2fadff8 bot: add member deletion command 2019-05-13 23:08:44 +02:00
Ske
62dc2ce78e bot: add birthday command 2019-05-13 22:44:49 +02:00
Ske
b42e052fee bot: add member lookup command 2019-05-12 00:44:02 +02:00
Ske
cf2598baa5 bot: add color change command 2019-05-11 23:56:56 +02:00
Ske
9b49f22048 bot: enable .NET configuration management 2019-05-08 20:08:56 +02:00
Ske
c5d2b7c251 refactor project structure 2019-05-08 00:06:27 +02:00