Necessary database migrations for this commit:
alter table servers add column log_blacklist bigint[] not null default array[]::bigint[];
alter table servers add column blacklist bigint[] not null default array[]::bigint[];
* Fix bulk importer parameter and remove unused method
-Fixes AddSwitchesBulk post-refactor by passing in the switch timestamp instead of a set of PKMembers
-Removes unused RegisterSwitches method which was replaced by AddSwitchesBulk
* Fix import from Tupperbox regression defect
When converting a Tupperbox import file to the PluralKit format on import, this sets the Tupper's name as the member ID which seems like a reasonable default. I'm also initializing the switches collection (even though it's going to be empty in this case).
This fixes a regression defect introduced when switch import was implemented.
* Making converted Tupper DataFileMember Id a GUID instead
Member names aren't necessarily unique (even if they should be), and no reason not to use something we know will be unique...
The member length limit is now long enough that it's unlikely to
hit the cap by accident. An error will still be displayed if you
attempt to perform a message proxy.
-Fixes AddSwitchesBulk post-refactor by passing in the switch timestamp instead of a set of PKMembers
-Removes unused RegisterSwitches method which was replaced by AddSwitchesBulk
We're now using binary import for switches and switch_members when importing a system profile, rather than importing them one switch at a time.
This adds a pass-through method to the PerformanceTrackingConnection that can be used for other bulk import applications.
A given system can now have up to 1000 members. Within 50 members of that limit, a warning will display whenever a new member is created via the bot. Once the limit is reached, a final warning will appear indicating that no additional members can be created unless members are first deleted. Attempting to create a new member at that point by any method will result in an error message indicating that the limit has been reached.
Respecting this in pk;import required some restructuring to tease apart which members already exist and which ones need to be created prior to creating any members as it seems preferable to fail early and give the user the ability to intervene rather than pushing the system to the member cap and requiring manual deletion of "lower priority" members before others can be created. One consequence of the restructure is that existing members are being read in bulk which is a performance improvement of 25-70% depending on how many switches need to be imported (the more members you have, the more noticeable this is).
Refactored ExportSystem to:
- Only fetch message counts once (instead of a query per member)
- Fetch switches using the newly refactored GetTruncatedSwitchList (gets switches and their members in one shot instead of querying for switch members one switch at a time)
- Added a new MessageCountsPerMember method to MemberStore to support the above
- Modified GetTruncatedSwitchList query to retrieve switches with no members (wasn't important for frontpercent, but is if we're reusing that method here)
This doesn't require any index changes beyond those that support GetTruncatedSwitchList, though we can see a small benefit with an index on messages covering the member column (valuable for other reasons outside of these additions).
Note: need to determine conventions to use for index names and settings going forward. There are also some indexes in production that aren't in here and are probably good additions to a standard set.
Refactored GetTruncatedSwitchList to:
- Only fetch switches in the requested range
- Fetch switch members in bulk rather than one switch at a time
This uses a new GetSwitchMembersList method that requires the following indexes:
CREATE INDEX ix_switches_system
ON public.switches USING btree
(system ASC NULLS LAST)
INCLUDE("timestamp")
TABLESPACE pg_default;
CREATE INDEX ix_switch_members_switch
ON public.switch_members USING btree
(switch ASC NULLS LAST)
INCLUDE(member)
TABLESPACE pg_default;
- ImportSystem builds a mapping of data file HID to current system HID
- Switches in a data file are reconciled with system members' actual IDs using this mapping
- SwitchStore provides a RegisterSwitches method to register multiple switches
- RegisterSwitches only imports a switch if one does not exist with the same timestamp
- The number of switches created is logged
Instead of acquiring a connection per service per request, we
acquire connections more often but at a more granular level, meaning
they're also disposed of more quickly instead of staying for a long time
in case of long-running commands or leaks.
