Commit Graph

406 Commits

Author SHA1 Message Date
Astrid
77d63c2838
Merge branch 'main' into feat/ap 2020-12-08 12:25:01 +01:00
spiral
115017980e
Add pk;member group 2020-11-22 19:57:01 -05:00
spiral
8e5fb6520b Add configurable autoproxy latch timeout 2020-11-22 09:42:40 -05:00
spiral
37294b68da Add disabling autoproxy per-account 2020-11-22 09:42:40 -05:00
spiral
41247c68a6 Add disabling front/latch autoproxy per-member 2020-11-22 09:42:40 -05:00
Ske
8c12b3f037 Update library versions (still on .NET 3.1 though) 2020-11-15 14:15:30 +01:00
Ske
634173e205 Update to D#+ 4.0.0-rc1 2020-11-15 13:53:31 +01:00
spiral
71461b3163
Add looking up group by displayname; document this 2020-11-14 10:42:51 -05:00
Ske
81cd5496d5 Refactor command message deletion 2020-10-23 12:18:28 +02:00
Astrid
9976789467
Merge branch 'main' into feature/public-reminder 2020-10-23 11:13:50 +02:00
Astrid
025fbfe120
Merge pull request #228 from dev-kittens/fix/memberlimits
Fix per-system member limits not being considered in certain conditionals
2020-10-23 11:11:11 +02:00
spiral
2eade8cb8a
Fix error message when importing >1k members 2020-10-22 15:50:56 -05:00
spiral
9da023e97a
change command message timeout to 2 hours 2020-10-19 03:57:43 -05:00
dev-kittens
a5f7bacb3e
Add member is public reminder message on member creation (if system has private members) 2020-10-19 03:50:51 -05:00
spiral
ece7a523ba
whoops 2020-10-18 01:06:20 -05:00
dev-kittens
56bb5e975f
Add handling command reactions
and, uhh, rename invoker_id to author_id
2020-10-18 00:59:36 -05:00
dev-kittens
05cc30279a
- Add storing bot response messages in postgres
- Add scheduled task to clean up said store
2020-10-18 00:18:52 -05:00
Ske
9282d5e9fb Add some more resiliency to DB errors 2020-10-14 23:35:10 +02:00
dev-kittens
019800f396 Fix per-system member limit being ignored on import 2020-10-09 14:24:15 +00:00
dev-kittens
d80d3c5f04 Fix limit message on creating new members 2020-10-09 14:23:31 +00:00
Ske
5b262fdc7b Fix typo in DB migration >.> 2020-10-09 12:21:07 +02:00
Ske
1fda6ec919 Add per-system member/group limit override 2020-10-09 12:18:29 +02:00
Ske
d875472c9f Upgrade Npgsql 2020-09-09 22:23:10 +02:00
Ske
6a64a13d10 Disable App.Metrics logging under INF 2020-09-09 22:22:43 +02:00
Ske
18ac9f882d Add in-app connection string settings 2020-09-09 22:21:46 +02:00
Ske
c7612df37e Major database refactor (again) 2020-08-29 13:46:27 +02:00
Ske
a2128da2a1 Fix some ordering 2020-08-28 11:41:41 +02:00
Ske
935c9660e8 Support destructuring of proxy tag object 2020-08-28 11:37:53 +02:00
Ske
f89ca4c867 Use Serilog in API 2020-08-27 23:36:02 +02:00
Ske
9955dc29f9 Add Elastic-bound logging of (almost) all database update events 2020-08-27 21:28:36 +02:00
Ske
8d27148bdf More updates to event infrastructure 2020-08-27 18:20:20 +02:00
Ske
2d66c44848 Change group limit to 250 2020-08-26 23:22:33 +02:00
Ske
2e6b1826fc Add experimental Elasticsearch logging output 2020-08-26 22:29:24 +02:00
Ske
001aff6298 Restrict member count in group list to public members 2020-08-26 01:50:24 +02:00
Ske
0d04725a19 Include full debug symbols when building in Release mode 2020-08-25 20:36:38 +02:00
Ske
87619a728e Update group migration date 2020-08-25 19:17:16 +02:00
Ske
10d6968734 Add metrics for group count 2020-08-25 18:26:52 +02:00
Ske
8688f4d0bf Merge branch 'main' into feature/groups 2020-08-21 18:31:49 +02:00
Ske
10c01da39b Couple more slight tweaks :) 2020-08-21 17:08:49 +02:00
Ske
1bb5d203df Various additional tweaks/additions to groups 2020-08-20 21:43:17 +02:00
Ske
9e251352c7 Various fixes and improvements 2020-08-16 12:10:54 +02:00
Ske
cedb050b8a Add group delete command 2020-08-08 14:56:34 +02:00
Ske
a0fc9d3826 Move some extension methods into their respective class files 2020-08-05 20:24:51 +02:00
Ske
5e28e0aba1 Add group privacy command/structures 2020-07-28 19:17:01 +02:00
Ske
47d5ad0004 Enforce group count limit on creation 2020-07-28 19:17:01 +02:00
Ske
f47d366c8d Show member groups on member card 2020-07-28 19:17:01 +02:00
Ske
ec9ee5c794 Show member count on group card 2020-07-28 19:17:01 +02:00
Ske
9d5be07f0c Add group icon and privacy to the database schema 2020-07-28 19:17:01 +02:00
Ske
8a28d836c7 Add group member list command 2020-07-28 19:17:00 +02:00
Ske
0f4c40b344 Add group add/remove commands 2020-07-28 19:17:00 +02:00
Ske
6c5cb8cea7 Add group name/description/list commands 2020-07-28 19:17:00 +02:00
Ske
253ae43c7f Add super basic group model/command 2020-07-28 19:17:00 +02:00
Ske
94076dc9be Split HandlerQueue logic into Predicate/Handler 2020-07-22 01:19:23 +02:00
Ske
577629c28a Switch to ConcurrentDictionary for HandlerQueue 2020-07-22 01:14:45 +02:00
Ske
8b8cf932a3 Add null check in HandlerQueue 2020-07-22 01:06:26 +02:00
Ske
9f523b3c5f Refactor system/member privacy commands 2020-07-08 00:47:16 +02:00
Ske
467b95b1b1 Refactor and simplify member list code 2020-07-07 20:57:22 +02:00
Ske
9424e50187 Re-add private setters for member class 2020-07-03 11:25:05 +02:00
Ske
0598c53f62 Migrate guild objects to the patch system 2020-06-29 15:20:28 +02:00
Ske
467ce78522 Retire more IDataStore methods 2020-06-29 14:54:11 +02:00
Ske
9c1efc7886 Move system updates to the same patch system as members 2020-06-29 14:39:19 +02:00
Ske
c5697b33e2 Finally retire the PKMember setters! 2020-06-29 14:15:30 +02:00
Ske
281b669391 Refactor member updates to use a patch object 2020-06-29 13:57:48 +02:00
Ske
178e256583 Fix system guild settings model mismatch 2020-06-24 16:52:49 +02:00
Ske
3b47785aa0 Fix front percent lookup
Dapper can't map lists of ID types in an input position so we select it to a plain ulong list :)
2020-06-24 14:02:47 +02:00
Ske
4ac6465648 Fix system guild helper method (again?) 2020-06-24 14:02:47 +02:00
Fiona
2de6d399ce Create GetMemberByDisplayName() 2020-06-22 21:06:14 +10:00
Ske
6fa76e85c4 Make duration parsing case-insensitive 2020-06-21 16:07:14 +02:00
Ske
d3cf382e3b Migrate to date/time format extension methods 2020-06-21 16:05:04 +02:00
Ske
dcffbef805 Migrate to more privacy helper extensions 2020-06-21 15:51:08 +02:00
Ske
a31782864d Fix message count and created not being mapped properly
This weirdly only happens when going through ListedMember, perhaps the inheritance is making it so Dapper doesn't detect the backing field or soemthing... Adding a setter to the autoproperty makes things work, though.
2020-06-21 14:52:39 +02:00
Ske
ae4e8f97d0 Add member avatar privacy 2020-06-20 16:00:50 +02:00
Ske
dd9cc3ef0c Respect description privacy when searching members 2020-06-18 17:33:37 +02:00
Ske
56eae82b0a Move most references to PKMember.Name to go through helper extepsions for privacy 2020-06-18 17:08:36 +02:00
Ske
761270f0c3 Clean up member privacy command 2020-06-17 23:06:49 +02:00
Ske
dd2690c3f4 Get rid of compiler warning 2020-06-17 22:39:54 +02:00
Ske
e92700f339 Comment out color privacy stuff for now (revisit later?) 2020-06-17 21:51:40 +02:00
BeeFox-sys
721a4502bb
Feature/granular member privacy (#174)
* Some reasons this needs to exist for it to run on my machine? I don't think it would hurt to have it in other machines so

* Add options to member model

* Add Privacy to member embed

* Added member privacy display list

* Update database settings

* apparetnly this is nolonger needed?

* Fix sql call

* Fix more sql errors

* Added in settings control

* Add all subject to system privacy

* Basic API Privacy

* Name privacy in logs

* update todo

* remove CheckReadMemberPermission

* Added name privacy to log embed

* update todo

* Update todo

* Update api to handle privacy

* update todo

* Update systemlist full to respect privacy (as well as system list)

* include colour as option for member privacy subject

* move todo file (why was it there?)

* Update TODO.md

* Update TODO.md

* Update TODO.md

* Deleted to create pr

* Update command usage and add to the command tree

* Make api respect created privacy

* Add editing privacy through the api

* Fix pronoun privacy field in api

* Fix info leak of display name in api

* deprecate privacy field in api

* Deprecate privacy diffrently

* Update API

* Update documentation

* Update documentation

* Remove comment in yml

* Update userguide

* Update migration (fix typo in 5.sql too)

* Sanatize names

* some full stops

* Fix after merge

* update migration

* update schema version

* update edit command

* update privacy filter

* fix a dumb mistake

* clarify on what name privacy does

* make it easier on someone else

* Update docs

* Comment out unused code

* Add aliases for `member privacy all public` and `member privacy all private`
2020-06-17 21:31:39 +02:00
Ske
627f544ee8 Migrate API to ASP.NET Core Auth services + refactor 2020-06-16 01:15:59 +02:00
Ske
21f8caf006 Update type signature of MessageContext rows 2020-06-15 12:54:22 +02:00
Ske
5aa55359b3 Always return a row from message_context even with both system and guild is undefined (closes #183) 2020-06-15 12:54:00 +02:00
Ske
088bc15e5e Upgrade Autofac dependency 2020-06-15 00:52:20 +02:00
Ske
b9cbd241de Migrate to type-safe model ID structs 2020-06-14 21:37:04 +02:00
Ske
0f041c2e3c Add unit test project and some early tests 2020-06-14 18:56:53 +02:00
Ske
55b1c6499d Remove duplicated ProxyName function 2020-06-14 15:37:07 +02:00
Ske
c780781184 Remove unused imports 2020-06-13 23:12:44 +02:00
Ske
d135042a58 Change default console log level back 2020-06-13 22:39:19 +02:00
Ske
2702c45b4f General bits of cleanup 2020-06-13 22:20:24 +02:00
Ske
6d06474d26 Refactor sort/filter code once again
Now we handle sorting on the bot side, but still filter in the database
2020-06-13 21:49:31 +02:00
Ske
f1b28b7fb6 Move schema migration stuff to Database 2020-06-13 19:54:16 +02:00
Ske
c927967c3b Migrate GetSystemById 2020-06-13 19:42:04 +02:00
Ske
511cf0ab55 Get rid of some unused ctors 2020-06-13 19:41:51 +02:00
Ske
90ac186183 Extract Database to interface 2020-06-13 19:36:43 +02:00
Ske
c875c8af9f Various small fixes, setting guild config now works 2020-06-13 19:15:50 +02:00
Ske
2038f023a0 Move count stat collecting to raw SQL 2020-06-13 19:15:29 +02:00
Ske
767a37e637 Use async transactions for AddSwitch 2020-06-13 19:14:42 +02:00
Ske
37b99f9521 Wrap DbTransaction too 2020-06-13 18:49:05 +02:00
Ske
e176ccbab5 Several more database-y refactors
- DbConnectionFactory renamed to "Database", will now be the primary entry point for DB stuff
- Created IPKConnection interface mostly containing async extensions to IDbConnection, use this going forward
- Reworked the Connection/Command wrappers (that have performance/logging extensions)
- Probably more stuff that I forgot???
2020-06-13 18:31:20 +02:00
Ske
a915ddb41c Extract system/member guild settings and refactor DB access
(also refactor MemberAvatar now that I'm here)
2020-06-13 16:03:57 +02:00
Ske
ed511a6236 Refactor server config models/commands 2020-06-13 13:58:27 +02:00
Ske
23c595f675 Extract ProxyTag to separate file 2020-06-13 13:18:21 +02:00
Ske
cb8f29cc47 Remove setters where possible 2020-06-13 13:11:08 +02:00
Ske
e8b69dfe29 Remove JSON attributes from models 2020-06-13 13:06:41 +02:00
Ske
292676b694 Optimize last message search in member list view 2020-06-13 02:45:06 +02:00
Ske
d246c9bf33 Change HID function types to char(5) as in DB 2020-06-13 01:54:29 +02:00
Ske
920c8cff01 Optimize message count migration 2020-06-13 01:25:12 +02:00
Ske
8ac2f1e4b8 Move hid generation to the database. Closes #157. 2020-06-13 00:43:48 +02:00
Ske
c39c51426f Remove setters for function models 2020-06-13 00:42:25 +02:00
Ske
761a82740f Remove unused functions and classes 2020-06-12 23:55:25 +02:00
Ske
729930a562 Move system tag/icon from ProxyMember to MessageContent 2020-06-12 23:30:10 +02:00
Ske
2a39489c4d Move proxy name/avatar logic to the bot itself 2020-06-12 23:24:36 +02:00
Ske
3d62a0d33c Split message/proxy data up in MessageContext and ProxyMember 2020-06-12 23:13:21 +02:00
Ske
ba441a15cc Too many refactors in one:
- Allowed adding ephemeral(ish) views and functions
- Moved message_count to a concrete database field
- Moved most proxy logic to a stored procedure
- Moved database files around and refactored schema manager
2020-06-12 20:29:50 +02:00
Ske
24f1363bb0 Return AsyncDisposable from DbConnectionFactory 2020-06-12 00:54:50 +02:00
Ske
49acc4d9e2 Refactor proxy handling code
- Move reaction handlers to the ReactionAdded event instead of
  ProxyService
- Split tag matching off into ProxyTagParser
- Split autoproxy matching off into Autoproxier
- General cleanup and simplification
2020-06-11 23:22:53 +02:00
Ske
ccb6ba5d30 Minor renames and rewordings 2020-06-11 21:32:04 +02:00
Lillian Berry
50c9675963 Fix formatting 2020-06-11 21:19:40 +02:00
Lillian Berry
59af6f02e8 Add disabling reaction pings 2020-06-11 21:19:40 +02:00
Ske
39f92309b7 Refactor import/export database code 2020-06-11 21:11:50 +02:00
Ske
62207a9255 Reduce flush to disk interval further 2020-05-31 03:22:41 +02:00
Ske
c3d439dc5f Properly handle duplicate message insertions to the database 2020-05-06 23:49:06 +02:00
Ske
f803b9a392 Add a character limit to avatar URLs (256 for now) 2020-05-05 16:04:17 +02:00
Ske
91c2eb3992 Fix plain-text logging output file 2020-05-01 20:13:15 +02:00
Ske
483a9d6ed9 Save log file as both text and JSON 2020-04-29 00:25:31 +02:00
Ske
4fe40b220e Optimize latch-mode autoproxy query 2020-04-16 18:23:56 +02:00
Ske
93e53be7db Bump member count limit for the time being 2020-04-09 17:25:01 +02:00
Ske
a1d91784a0 Fix importing older export files 2020-03-07 17:30:22 +01:00
Ske
c5a91c4575 Fix Tupperbox importing birthdays 2020-03-04 19:49:02 +01:00
Ske
45c5e5ed42 Normalize colors to lowercase hex 2020-02-25 16:37:52 +01:00
BeeFox-sys
e455bd6d12
Fix import validation issue (#156) 2020-02-25 16:33:49 +01:00
Ske
9d4993b121 Fix Tupperbox import errors 2020-02-23 12:45:26 +01:00
Ske
26e7ef3c23 Add further bounds/sanity checking to import files 2020-02-22 20:11:37 +01:00
Ske
de141d629b Fix newline rendering in embeds on iOS 2020-02-20 22:57:37 +01:00
Ske
e9cc8ed424 Add logger bot cleanup support 2020-02-15 14:17:25 +01:00
Ske
48342a2890 Fix spurious empty-string-description errors 2020-02-12 23:18:31 +01:00
Ske
2148e29f54 Log updates to guild settings 2020-02-12 19:07:40 +01:00
Ske
d0d3579b17 Add server-specific member avatars 2020-02-12 17:42:12 +01:00
Ske
6d5004bf54 Large refactor and project restructuring 2020-02-12 15:16:19 +01:00
Ske
30ed293dc6 Change no-year sentinel value to 0004
This allows setting the date "Feb 29" with no year, since the year 0004 is a leap year in the Gregorian calendar, while the year 0001 isn't.
2020-02-09 22:36:02 +01:00
Ske
951b089c97 Handle imports from other systems/instances with different HIDs 2020-02-03 15:16:57 +01:00
Ske
7bdc3020b0 Handle malformed proxy tag importing correctly 2020-02-03 15:05:05 +01:00
Ske
026932049a Properly invalidate account caches when unlinking 2020-02-03 15:04:53 +01:00
Ske
de427d8bfe Properly invalidate account caches when deleting systems 2020-02-03 14:47:29 +01:00
Ske
82dfe43d5a Rework caching
This does a *lot* of things. Essentially, it replaces the existing
individual proxy- and autoproxy caches on the bot end with a
global cache (in Core) that handles all the caching at once, and
automatically invalidates the cache once something changes in the
datastore.

This allows us to do proxying and autoproxying with *zero database
queries* (best-case).
2020-02-01 14:40:57 +01:00
Ske
a60be64551 Ensure metrics instance is a singleton 2020-02-01 12:33:43 +01:00
Ske
3b2167e16f Shuffle dependencies around 2020-01-26 01:30:39 +01:00
Ske
1ea0526ef8 Migrate DI container to Autofac 2020-01-26 01:27:45 +01:00
Ske
b371c2f34d Fix Npgsql deprecation error 2020-01-25 16:46:34 +01:00
Ske
83cfb3eb46 Add autoproxy functionality 2020-01-24 20:28:48 +01:00
Ske
57bc576de6 Add autoproxy management commands 2020-01-23 21:20:22 +01:00
Ske
ca37c7e6ca Put the autoproxy settings in the correct class 👀 2020-01-23 20:29:22 +01:00
Ske
bf70a6e3e1 Add schema changes for autoproxy 2020-01-23 20:19:50 +01:00
Ske
b347d2d557 Add front history pagination; upgrade more store methods 2020-01-18 00:58:35 +01:00
Ske
8a689ac0f2 Upgrade various store methods to IAsyncEnumerable 2020-01-18 00:02:17 +01:00
Ske
0dd8988c08 Fix member count including private members in system card 2020-01-17 17:54:07 +01:00
Ske
991c00a4e6 Increase precision in exported timestamps 2020-01-17 17:43:06 +01:00
Ske
98613c4287 Add system and member privacy support 2020-01-11 16:51:13 +01:00
Ske
66eb012e4a Fix database errors on first startup 2020-01-07 16:31:46 +01:00
Ske
64b633e6b4 Fix member avatar URL not showing up in API responses 2020-01-07 15:52:32 +01:00
Ske
54aa9fb7d7 Change handling of Tupperbox per-member tags 2019-12-28 15:53:11 +01:00
Ske
0f22285824 Upgrade API serialisation code to enable potential context-based serialisation 2019-12-28 15:52:59 +01:00
Ske
73b2631280 Optimize database queries for proxy filtering 2019-12-28 12:00:52 +01:00
Ske
4a30e56298 Add basic database schema migration system 2019-12-26 21:43:05 +01:00
Ske
4d07886ec8 Add server-specific display names 2019-12-26 20:39:47 +01:00
Ske
6a4c131d6d Upgrade every dependency to latest compatible version 2019-12-23 15:53:20 +01:00
Ske
f2ce09d467 Upgrade to Npgsql 4.1.2 2019-12-23 13:55:56 +01:00
Ske
b4ad1e5041 Use asynchronous logging 2019-12-23 00:29:30 +01:00
Ske
d2205ea7e6 Update documentation with new features 2019-12-22 14:26:53 +01:00
Ske
639c813ce7 Add per-server per-system proxy enable toggle 2019-12-22 14:15:56 +01:00
Ske
d4c1a6b1fc Add beginnings of new export/data system 2019-12-22 00:41:53 +01:00
Ske
9c9e48a799 Upgrade to .NET Core 3.1 2019-12-22 00:40:57 +01:00
Ske
5fc654b758 Add index to message table to speed up count querying 2019-12-21 21:42:34 +01:00
Ske
3b72fa720b Add warning when setting proxy tags conflicting with another member's 2019-12-21 21:42:06 +01:00
Ske
530e3aa697 Fix null normalising utility 2019-12-02 16:56:35 +01:00
Ske
378cba09e5 Add channel blacklists for logging and proxying
Necessary database migrations for this commit:
    alter table servers add column log_blacklist bigint[] not null default array[]::bigint[];
    alter table servers add column blacklist bigint[] not null default array[]::bigint[];
2019-11-03 19:15:50 +01:00
Ske
53124776d4 Fix proxy tag field default type in schema 2019-11-02 23:38:25 +01:00
Ske
7c3b4b9af9 Add command to delete all switches 2019-11-02 22:46:51 +01:00
Ske
f259beae90 Fix importing members with no proxy tags 2019-11-02 20:45:47 +01:00
Ske
1a79d52bfb Add importing of Tupperbox multibrackets 2019-10-30 14:11:24 +01:00
Ske
49dc25ee02 Allow keeping proxy tags when proxying messages.
Required database migration:

Closes #75.
2019-10-30 09:26:50 +01:00
Ske
8604d25ffe Fix legacy prefix/suffix API fields 2019-10-30 09:19:49 +01:00
Ske
5d63181cac Fix exporting 2019-10-30 09:09:09 +01:00
Ske
393ee16c1b Add support for multiple proxy tags
Tangentially closes #103.
2019-10-30 08:18:59 +01:00
Noko
96b03495a4 Fix import from Tupperbox regression defect (#137)
* Fix bulk importer parameter and remove unused method

-Fixes AddSwitchesBulk post-refactor by passing in the switch timestamp instead of a set of PKMembers
-Removes unused RegisterSwitches method which was replaced by AddSwitchesBulk

* Fix import from Tupperbox regression defect

When converting a Tupperbox import file to the PluralKit format on import, this sets the Tupper's name as the member ID which seems like a reasonable default. I'm also initializing the switches collection (even though it's going to be empty in this case).

This fixes a regression defect introduced when switch import was implemented.

* Making converted Tupper DataFileMember Id a GUID instead

Member names aren't necessarily unique (even if they should be), and no reason not to use something we know will be unique...
2019-10-29 08:41:44 +01:00
Ske
545e8df6a7 Reduce the log level of various common events 2019-10-27 23:44:27 +01:00
Ske
46672d264c Refactor log channel storage 2019-10-27 23:01:20 +01:00
Ske
523469e5e6 Remove preemptive warning messages for unproxyable members
The member length limit is now long enough that it's unlikely to
hit the cap by accident. An error will still be displayed if you
attempt to perform a message proxy.
2019-10-27 21:57:31 +01:00
Ske
e6e1ab023a Fix type mismatch on system export 2019-10-27 14:49:46 +01:00
Noko
dd37c8a6d6 Fix bulk importer parameter and remove unused method (#136)
-Fixes AddSwitchesBulk post-refactor by passing in the switch timestamp instead of a set of PKMembers
-Removes unused RegisterSwitches method which was replaced by AddSwitchesBulk
2019-10-27 11:37:16 +01:00
Ske
6a73b3bdd6 Refactor data stores, merging the Store classes 2019-10-26 19:45:44 +02:00
Noko
593e9d9aef Fix edge case of importing exactly enough members to hit the cap
It's OK if import will bring us to the cap (1000), just don't let the count go beyond that.
2019-10-20 18:24:00 -05:00
Noko
406f005b4f Bulk import switches for pk;import
We're now using binary import for switches and switch_members when importing a system profile, rather than importing them one switch at a time.

This adds a pass-through method to the PerformanceTrackingConnection that can be used for other bulk import applications.
2019-10-20 14:38:43 -05:00
Noko
397da2e1fa Added max member count to limits
A given system can now have up to 1000 members. Within 50 members of that limit, a warning will display whenever a new member is created via the bot. Once the limit is reached, a final warning will appear indicating that no additional members can be created unless members are first deleted. Attempting to create a new member at that point by any method will result in an error message indicating that the limit has been reached.

Respecting this in pk;import required some restructuring to tease apart which members already exist and which ones need to be created prior to creating any members as it seems preferable to fail early and give the user the ability to intervene rather than pushing the system to the member cap and requiring manual deletion of "lower priority" members before others can be created. One consequence of the restructure is that existing members are being read in bulk which is a performance improvement of 25-70% depending on how many switches need to be imported (the more members you have, the more noticeable this is).
2019-10-20 02:16:57 -05:00
Ske
4ce4efcc61 Fix importing brackets from Tupperbox. Closes #123 2019-10-18 13:34:54 +02:00
Astrid
e91ecebb39
Merge branch 'master' into perf/export 2019-10-18 13:21:19 +02:00
Astrid
49adcd680a
Merge pull request #132 from nephanim/perf/frontpercent
Improve frontpercent performance
2019-10-18 13:20:43 +02:00
Noko
adea7be07c Improve export performance
Refactored ExportSystem to:
- Only fetch message counts once (instead of a query per member)
- Fetch switches using the newly refactored GetTruncatedSwitchList (gets switches and their members in one shot instead of querying for switch members one switch at a time)

- Added a new MessageCountsPerMember method to MemberStore to support the above
- Modified GetTruncatedSwitchList query to retrieve switches with no members (wasn't important for frontpercent, but is if we're reusing that method here)

This doesn't require any index changes beyond those that support GetTruncatedSwitchList, though we can see a small benefit with an index on messages covering the member column (valuable for other reasons outside of these additions).
2019-10-06 02:03:28 -05:00
Noko
ef7b825aa6
Merge pull request #1 from nephanim/feature/import-switches
Bring performance branch in line with export/import changes
2019-10-05 22:51:11 -05:00
Noko
bc0213c9a1 Added indexes to support frontpercent query changes
Note: need to determine conventions to use for index names and settings going forward. There are also some indexes in production that aren't in here and are probably good additions to a standard set.
2019-10-05 15:30:55 -05:00
Noko
845ec90c3e Improve frontpercent performance
Refactored GetTruncatedSwitchList to:
- Only fetch switches in the requested range
- Fetch switch members in bulk rather than one switch at a time

This uses a new GetSwitchMembersList method that requires the following indexes:

CREATE INDEX ix_switches_system
ON public.switches USING btree
(system ASC NULLS LAST)
INCLUDE("timestamp")
TABLESPACE pg_default;

CREATE INDEX ix_switch_members_switch
ON public.switch_members USING btree
(switch ASC NULLS LAST)
INCLUDE(member)
TABLESPACE pg_default;
2019-10-05 15:08:27 -05:00