We're now using binary import for switches and switch_members when importing a system profile, rather than importing them one switch at a time.
This adds a pass-through method to the PerformanceTrackingConnection that can be used for other bulk import applications.
Some of the command rewrite changes resulted in the response messages for importing a system being swapped. When importing without an existing system (ctx.System == null), we want to display the "new system" message. Otherwise, show the count added/modified.
A given system can now have up to 1000 members. Within 50 members of that limit, a warning will display whenever a new member is created via the bot. Once the limit is reached, a final warning will appear indicating that no additional members can be created unless members are first deleted. Attempting to create a new member at that point by any method will result in an error message indicating that the limit has been reached.
Respecting this in pk;import required some restructuring to tease apart which members already exist and which ones need to be created prior to creating any members as it seems preferable to fail early and give the user the ability to intervene rather than pushing the system to the member cap and requiring manual deletion of "lower priority" members before others can be created. One consequence of the restructure is that existing members are being read in bulk which is a performance improvement of 25-70% depending on how many switches need to be imported (the more members you have, the more noticeable this is).
Refactored ExportSystem to:
- Only fetch message counts once (instead of a query per member)
- Fetch switches using the newly refactored GetTruncatedSwitchList (gets switches and their members in one shot instead of querying for switch members one switch at a time)
- Added a new MessageCountsPerMember method to MemberStore to support the above
- Modified GetTruncatedSwitchList query to retrieve switches with no members (wasn't important for frontpercent, but is if we're reusing that method here)
This doesn't require any index changes beyond those that support GetTruncatedSwitchList, though we can see a small benefit with an index on messages covering the member column (valuable for other reasons outside of these additions).
Note: need to determine conventions to use for index names and settings going forward. There are also some indexes in production that aren't in here and are probably good additions to a standard set.
Refactored GetTruncatedSwitchList to:
- Only fetch switches in the requested range
- Fetch switch members in bulk rather than one switch at a time
This uses a new GetSwitchMembersList method that requires the following indexes:
CREATE INDEX ix_switches_system
ON public.switches USING btree
(system ASC NULLS LAST)
INCLUDE("timestamp")
TABLESPACE pg_default;
CREATE INDEX ix_switch_members_switch
ON public.switch_members USING btree
(switch ASC NULLS LAST)
INCLUDE(member)
TABLESPACE pg_default;
- ImportSystem builds a mapping of data file HID to current system HID
- Switches in a data file are reconciled with system members' actual IDs using this mapping
- SwitchStore provides a RegisterSwitches method to register multiple switches
- RegisterSwitches only imports a switch if one does not exist with the same timestamp
- The number of switches created is logged
Add `toLower()` method to the sorting used in system lists, so members are ordered regardless of if the first letter of their name is upper or lowercase
