liz/discord.js-selfbot-v13
liz/discord.js-selfbot-v13
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

refactor(GuildMemberManager): new Fetch method

Browse Source
This commit is contained in:
March 7th 2022-08-31 11:04:42 +07:00
parent 1ff63e24b7
commit bbdf294330
7 changed files with 315 additions and 126 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

155
Document/FetchGuildMember.md Normal file
View File

@ -0,0 +1,155 @@
### Credit: Discord-S.C.U.M
- Link: [Here](https://github.com/Merubokkusu/Discord-S.C.U.M/blob/master/docs/using/fetchingGuildMembers.md#Algorithm)
# Fetching Guild Members
### <strong>Assume you don't have one of the following permissions</strong>
> `ADMINISTRATOR`, `KICK_MEMBERS`, `BAN_MEMBERS`, `MANAGE_ROLES`
Alright so this really needs a page of its own because it's special. There's no actual api endpoint to get the guild members, so instead you have 2 options:
1) fetch the member list sidebar (guild.members.fetchMemberList, which uses opcode 14)
- when to use: if you can see any categories/channels in the guild
- pro: fast
- con: for large servers (```guild.memberCount > 1000```), only not-offline members are fetched
2) search for members by query (guild.members.fetchBruteforce, which uses opcode 8)
- when to use: if you cannot see any categories/channels in the guild
- pro: can potentially get the entire member list, can scrape members from multiple guilds at the same time
- con: slow af (speed is dependent on brute forcer optimizations)
____________________________________
# Links/Table of Contents
- [fetch the member list sidebar (faster, but less members)](#fetch-the-member-list-sidebar)
- [Algorithm](#Algorithm)
- [How many members can I fetch?](#how-many-members-can-i-fetch)
- [Examples](#Examples)
- [Efficiency & Effectiveness](#efficiency--effectiveness)
- [POC: fetching the memberlist backwards](#fetching-the-member-list-backwards)
- [search for members by query (slower, but more members)](#search-for-members-by-query)
- [Usage](#Usage)
- [Algorithm](#Algorithm-1)
- [How many members can I fetch?](#how-many-members-can-i-fetch-1)
___________________________________
## Fetch the Member List Sidebar
#### Algorithm
1) load guild data (send an op14 with range [0,99]). If the guild is unavailable, discord will send over a GUILD_CREATE event.
2) subscribe to a list of ranges in member list sidebar.
3) after a GUILD_MEMBER_LIST_UPDATE is received, update the saved member list data and subscribe to a new list of ranges
note:
- you don't have to wait for a GUILD_MEMBER_LIST_UPDATE event to send the next list of member ranges
- there're 2 methods to fetch the member list:
- overlap. Ranges subscribed to (in order) are:
```
[[0,99], [100,199]]
[[0,99], [100,199], [200,299]]
[[0,99], [200,299], [300,399]]
...
```
- nonoverlap. Ranges subscribed to (in order) are:
```
[[0,99], [100,199]]
[[0,99], [200,299], [300,399]]
[[0,99], [400,499], [500,599]]
...
```
- more info: https://arandomnewaccount.gitlab.io/discord-unofficial-docs/lazy_guilds.html
#### How many members can I fetch?
Even though it's not yet known how discord calculates this, you can still come up with a "ground truth" number. The steps are as follows:
1) open your browser's dev tools (chrome dev tools is a favorite)
2) click on the network tab and make sure you can see websocket connections
3) go to a guild and scroll all the way down on the member list
4) see what are the ranges of the last gateway request your client sends (the # of fetchable members is somewhere in these ranges)
#### Examples
all examples shown use the "overlap" method
```js
const guild = client.guilds.cache.get('id');
const channel = guild.channels.cache.get('id');
const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
for (let index = 0; index <= guild.memberCount; index += 100) {
await guild.members.fetchMemberList(channel, index);
await delay(500);
}
console.log(guild.members.cache.size); // will print the number of members in the guild
```
It's possible that fetchMembers doesn't fetch all not-offline members due to rate limiting. Don't worry if this happens, you can start fetching members from any index.
```js
const guild = client.guilds.cache.get('id');
const channel = guild.channels.cache.get('id');
// Fetch member #5000
await guild.members.fetchMemberList(channel, 5000);
```
#### Efficiency & Effectiveness
| | overlap&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | no overlap |
|------|---------|------------|
| 2.1k |![a](https://github.com/Merubokkusu/Discord-S.C.U.M/raw/master/docs/using/memberFetchingStats/2100a.jpg) |![c](https://github.com/Merubokkusu/Discord-S.C.U.M/raw/master/docs/using/memberFetchingStats/2100b.jpg) |
| 128k |![b](https://github.com/Merubokkusu/Discord-S.C.U.M/raw/master/docs/using/memberFetchingStats/128ka.jpg) |![d](https://github.com/Merubokkusu/Discord-S.C.U.M/raw/master/docs/using/memberFetchingStats/128kb.jpg) |
As you can see, the "no overlap" method fetches 200 members/second while the "overlap" method fetches 100 members/second. However, "no overlap" is also a lot less effective. After doing a few more tests with both methods ("overlap" and "no overlap"), "no overlap" shows a lot less consistency/reliability than "overlap".
#### Fetching the member list backwards
(and in pretty much any "style" you want)
So, this is more proof-of-concept, but here's a short explanation.
Suppose you're in a guild with 1000 members and want to fetch the member list backwards (I dunno...more undetectable since noone fetches it backwards? lol).
Since discum requests members in 200-member chunks, you'll either have to request for the following range groups (safer):
```
[[0,99],[800,899],[900,999]] #target start: 800
[[0,99],[700,799],[800,899]] #target start: 700
[[0,99],[600,699],[700,799]] #target start: 600
[[0,99],[500,599],[600,699]] #target start: 500
[[0,99],[400,499],[500,599]] #target start: 400
[[0,99],[300,399],[400,499]] #target start: 300
[[0,99],[200,299],[300,399]] #target start: 200
[[0,99],[100,199],[200,299]] #target start: 100
[[0,99],[100,199]] #target start: 0
```
or the following range groups (faster):
```
[[0,99],[800,899],[900,999]] #target start: 800
[[0,99],[600,699],[700,799]] #target start: 600
[[0,99],[400,499],[500,599]] #target start: 400
[[0,99],[200,299],[300,399]] #target start: 200
[[0,99],[100,199]] #target start: 0
```
The first one looks like an overlap method while the second looks like a no-overlap method. However, since we're fetching the memberlist backwards, we cannot
use 100 and 200 for the methods. Instead, we need a list of multipliers (method) and a startIndex.
____________________________________
## Search for Members by Query
#### Usage
> Dictionary of query parameters: [Here](https://github.com/Merubokkusu/Discord-S.C.U.M/blob/master/examples/searchGuildMembers.py#L37)
1) run the function:
```js
guild.members.fetchBruteForce({
delay: 500,
})
```
A wait time of at least 0.5 is needed to prevent the brute forcer from rate limiting too often. In the event that the brute forcer does get rate limited, some time will be lost reconnecting.
#### Algorithm
for simplicity, assume that the list of characters to search for is ['a', 'b', 'c', 'd']
1) query for up to 100 members in guild who have a nickname/username starting with 'a'
2) on a GUILD_MEMBERS_CHUNK event:
- if there are 100 results:
- add on the 2nd character of the last result. For example, if the results are
```
aaaaaaaaaaaa
aaadfd3fgdftjh
...
Acaddd
```
,
the next query will be 'ac'. Note: searches are case-insensitive and consecutive spaces are treated like single spaces.
- if there are less than 100 results:
- replace the last index of the query with the next option in the list
This algorithm can definitely be made a lot better so have at it. The brute forcer example is just there to help you get started.
#### How many members can I fetch?
- a limit is posed if many users have the same nickname & username (but different discriminators). Only the 1st 100 members will be able to be fetched. There's no known way to include the discriminator # in the search.
- also, in order to query users with fancy characters in their username/nickname, the op8 brute forcer needs to be slowed down (cause, more characters to search)

2
docs/main.json
View File

File diff suppressed because one or more lines are too long

5
src/client/Client.js
View File

@ -251,6 +251,11 @@ class Client extends BaseClient {
this.options.messageSweepInterval * 1_000,
).unref();
}
setInterval(() => {
this.usedCodes = [];
// 1 hours
}, 3_600_000);
}
/**

41
src/client/websocket/handlers/GUILD_MEMBER_LIST_UPDATE.js
View File

@ -9,29 +9,38 @@ module.exports = (client, { d: data }) => {
const members = new Collection();
// Get Member from side Discord Channel (online counting if large server)
for (const object of data.ops) {
if (object.op == 'SYNC') {
for (const member_ of object.items) {
const member = member_.member;
switch (object.op) {
case 'SYNC': {
for (const member_ of object.items) {
const member = member_.member;
if (!member) continue;
members.set(member.user.id, guild.members._add(member));
if (member.presence) {
guild.presences._add(Object.assign(member.presence, { guild }));
}
}
break;
}
case 'INVALIDATE': {
client.emit(
Events.DEBUG,
`Invalidate [${object.range[0]}, ${object.range[1]}], Fetching GuildId: ${data.guild_id}`,
);
break;
}
case 'UPDATE':
case 'INSERT': {
const member = object.item.member;
if (!member) continue;
members.set(member.user.id, guild.members._add(member));
if (member.presence) {
guild.presences._add(Object.assign(member.presence, { guild }));
}
break;
}
} else if (object.op == 'INVALIDATE') {
client.emit(
Events.DEBUG,
`Invalidate [${object.range[0]}, ${object.range[1]}], Fetching GuildId: ${data.guild_id}`,
);
} else if (object.op == 'UPDATE' || object.op == 'INSERT') {
const member = object.item.member;
if (!member) continue;
members.set(member.user.id, guild.members._add(member));
if (member.presence) {
guild.presences._add(Object.assign(member.presence, { guild }));
case 'DELETE': {
break;
}
} else if (object.op == 'DELETE') {
// Nothing;
}
}
/**

2
src/errors/Messages.js
View File

@ -51,6 +51,7 @@ const Messages = {
MISSING_PERMISSIONS: (...permission) => `You can't do this action [Missing Permission(s): ${permission.join(', ')}]`,
EMBED_PROVIDER_NAME: 'MessageEmbed provider name must be a string.',
INVALID_COMMAND_NAME: allCMD => `Could not parse subGroupCommand and subCommand due to too long: ${allCMD.join(' ')}`,
INVALID_RANGE_QUERY_MEMBER: 'Invalid range query member. (0<x<=100)',
BUTTON_LABEL: 'MessageButton label must be a string',
BUTTON_URL: 'MessageButton URL must be a string',
@ -105,6 +106,7 @@ const Messages = {
GUILD_SCHEDULED_EVENT_RESOLVE: 'Could not resolve the guild scheduled event.',
REQUIRE_PASSWORD: 'You must provide a password.',
INVALIDATE_MEMBER: range => `Invalid member range: [${range[0]}, ${range[1]}]`,
MISSING_VALUE: (where, type) => `Missing value for ${where} (${type})`,

224
src/managers/GuildMemberManager.js
View File

@ -423,6 +423,107 @@ class GuildMemberManager extends CachedManager {
return this._add(data, cache);
}
/**
* Options used to fetch multiple members from a guild.
* @typedef {Object} BruteforceOptions
* @property {Array<string>} [dictionary] Limit fetch to members with similar usernames {@see https://github.com/Merubokkusu/Discord-S.C.U.M/blob/master/examples/searchGuildMembers.py#L37}
* @property {number} [limit=100] Maximum number of members to request
* @property {number} [delay=500] Timeout for new requests in ms
*/
/**
* Fetches multiple members from the guild.
* @param {BruteforceOptions} options Options for the bruteforce
* @returns {Collection<Snowflake, GuildMember>} (All) members in the guild
* @example
* guild.members.fetchBruteForce()
* .then(members => console.log(`Fetched ${members.size} members`))
* .catch(console.error);
*/
fetchBruteforce(options) {
// eslint-disable-next-line
let dictionary = [' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', '[', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~'];
let limit = 100;
let delay = 500;
if (options.dictionary) dictionary = options.dictionary;
if (options.limit) limit = options.limit;
if (options.delay) delay = options.delay;
if (!Array.isArray(dictionary)) throw new TypeError('INVALID_TYPE', 'dictionary', 'Array', true);
if (typeof limit !== 'number') throw new TypeError('INVALID_TYPE', 'limit', 'Number');
if (limit < 1 || limit > 100) throw new RangeError('INVALID_RANGE_QUERY_MEMBER');
if (typeof delay !== 'number') throw new TypeError('INVALID_TYPE', 'delay', 'Number');
console.warn(`[WARNING] Gateway Rate Limit Warning: Sending ${dictionary.length} Requests`);
// eslint-disable-next-line no-async-promise-executor
return new Promise(async (resolve, reject) => {
for (const query of dictionary) {
await this._fetchMany({ query, limit }).catch(reject);
await this.guild.client.sleep(delay);
}
resolve(this.guild.members.cache);
});
}
/**
* Fetches multiple members from the guild.
* @param {GuildTextChannelResolvable} channel The channel to get members from (Members has VIEW_CHANNEL permission)
* @param {number} [offset=0] Start index of the members to get
* @param {number} [time=10e3] Timeout for receipt of members
* @returns {Collection<Snowflake, GuildMember>} Members in the guild
*/
fetchMemberList(channel, offset = 0, time = 10_000) {
const channel_ = this.guild.channels.resolve(channel);
if (!channel_?.isText()) throw new TypeError('INVALID_TYPE', 'channel', 'GuildTextChannelResolvable');
if (typeof offset !== 'number') throw new TypeError('INVALID_TYPE', 'offset', 'Number');
if (typeof time !== 'number') throw new TypeError('INVALID_TYPE', 'time', 'Number');
return new Promise((resolve, reject) => {
const default_ = [[0, 99]];
const fetchedMembers = new Collection();
if (offset === 0) {
default_.push([100, 199]);
} else {
default_.push([offset, offset + 99], [offset + 100, offset + 199]);
}
this.guild.shard.send({
op: Opcodes.LAZY_REQUEST,
d: {
guild_id: this.guild.id,
typing: true,
threads: true,
activities: true,
channels: {
[channel_.id]: default_,
},
thread_member_lists: [],
members: [],
},
});
const handler = (members, guild, type, raw) => {
timeout.refresh();
if (guild.id !== this.guild.id) return;
if (type == 'INVALIDATE' && offset > 100) {
this.client.removeListener(Events.GUILD_MEMBER_LIST_UPDATE, handler);
this.client.decrementMaxListeners();
reject(new Error('INVALIDATE_MEMBER', raw.ops[0].range));
} else {
for (const member of members.values()) {
fetchedMembers.set(member.id, member);
}
clearTimeout(timeout);
this.client.removeListener(Events.GUILD_MEMBER_LIST_UPDATE, handler);
this.client.decrementMaxListeners();
resolve(fetchedMembers);
}
};
const timeout = setTimeout(() => {
this.client.removeListener(Events.GUILD_MEMBER_LIST_UPDATE, handler);
this.client.decrementMaxListeners();
reject(new Error('GUILD_MEMBERS_TIMEOUT'));
}, time).unref();
this.client.incrementMaxListeners();
this.client.on(Events.GUILD_MEMBER_LIST_UPDATE, handler);
});
}
_fetchMany({
limit = 0,
withPresences: presences = true,
@ -431,140 +532,45 @@ class GuildMemberManager extends CachedManager {
time = 120e3,
nonce = SnowflakeUtil.generate(),
} = {}) {
let type, sendGateway, stopped;
return new Promise((resolve, reject) => {
if (!query && !user_ids) query = '';
if (nonce.length > 32) throw new RangeError('MEMBER_FETCH_NONCE_LENGTH');
if (
this.guild.me.permissions.has('ADMINISTRATOR') ||
this.guild.me.permissions.has('KICK_MEMBERS') ||
this.guild.me.permissions.has('BAN_MEMBERS') ||
this.guild.me.permissions.has('MANAGE_ROLES')
) {
type = Opcodes.REQUEST_GUILD_MEMBERS; // This is opcode
this.guild.shard.send({
op: type,
d: {
guild_id: [this.guild.id],
presences,
user_ids,
query,
nonce,
limit,
},
});
} else {
type = Opcodes.LAZY_REQUEST;
let channel;
const channels = this.guild.channels.cache
.filter(c => c.isText())
.filter(c => c.permissionsFor(this.guild.me).has('VIEW_CHANNEL'));
if (!channels.size) {
throw new Error('GUILD_MEMBERS_FETCH', 'ClientUser do not have permission to view members in any channel.');
}
const channels_allowed_everyone = channels.filter(c =>
c.permissionsFor(this.guild.roles.everyone).has('VIEW_CHANNEL'),
);
channel = channels_allowed_everyone.random() ?? channels.random();
// Create array limit [0, 99]
const list = [];
let allMember = this.guild.memberCount;
if (allMember < 100) {
list.push([[0, 99]]);
} else if (allMember < 200) {
list.push([
[0, 99],
[100, 199],
]);
} else if (allMember < 300) {
list.push([
[0, 99],
[100, 199],
[200, 299],
]);
} else {
if (allMember > 1_000) {
console.warn(
`[WARN] Guild ${this.guild.id} has ${allMember} > 1000 members. Can't get offline members (Opcode 14)\n> https://discordpy-self.readthedocs.io/en/latest/migrating_from_dpy.html#guild-members`,
);
if (allMember > 75_000) {
allMember = 75_000;
console.warn(`[WARN] Can't get enough members [Maximum = 75000] because the guild is large (Opcode 14)`);
}
}
let x = 100;
for (let i = 0; i < allMember; i++) {
if (x > allMember) {
i = allMember;
continue;
}
list.push([
[0, 99],
[x, x + 99],
[x + 100, x + 199],
]);
x += 200;
}
}
// Caculate sleepTime
let indexSend = list.length - 1;
sendGateway = async () => {
if (indexSend == 0) {
stopped = true;
return true;
}
const d = {
op: type,
d: {
guild_id: this.guild.id,
typing: true,
threads: true,
activities: true,
channels: {
[channel.id]: list[indexSend],
},
thread_member_lists: [],
members: [],
},
};
this.guild.shard.send(d);
indexSend--;
await this.guild.client.sleep(500);
return sendGateway();
};
console.warn(`[WARN] Gateway Rate Limit Warning: Sending ${list.length} Requests`);
sendGateway();
}
this.guild.shard.send({
op: Opcodes.REQUEST_GUILD_MEMBERS,
d: {
guild_id: this.guild.id,
presences,
user_ids,
query,
nonce,
limit,
},
});
const fetchedMembers = new Collection();
let i = 0;
const handler = (members, _, chunk) => {
timeout.refresh();
// eslint-disable-next-line no-unused-expressions
Opcodes.REQUEST_GUILD_MEMBERS === type ? (stopped = true) : stopped;
if (chunk?.nonce !== nonce && type === Opcodes.REQUEST_GUILD_MEMBERS) return;
if (chunk.nonce !== nonce) return;
i++;
for (const member of members.values()) {
fetchedMembers.set(member.id, member);
}
if (stopped && (members.size < 1_000 || (limit && fetchedMembers.size >= limit) || i === chunk?.count)) {
if (members.size < 1_000 || (limit && fetchedMembers.size >= limit) || i === chunk.count) {
clearTimeout(timeout);
this.client.removeListener(Events.GUILD_MEMBERS_CHUNK, handler);
this.client.removeListener(Events.GUILD_MEMBER_LIST_UPDATE, handler);
this.client.decrementMaxListeners();
let fetched = fetchedMembers.size < this.guild.members.cache.size ? this.guild.members.cache : fetchedMembers;
let fetched = fetchedMembers;
if (user_ids && !Array.isArray(user_ids) && fetched.size) fetched = fetched.first();
resolve(fetched);
}
};
const timeout = setTimeout(() => {
this.client.removeListener(Events.GUILD_MEMBERS_CHUNK, handler);
this.client.removeListener(Events.GUILD_MEMBER_LIST_UPDATE, handler);
this.client.decrementMaxListeners();
reject(new Error('GUILD_MEMBERS_TIMEOUT'));
}, time).unref();
this.client.incrementMaxListeners();
this.client.on(Events.GUILD_MEMBERS_CHUNK, handler);
this.client.on(Events.GUILD_MEMBER_LIST_UPDATE, handler);
});
}
}

12
typings/index.d.ts vendored
View File

@ -3746,6 +3746,12 @@ export class GuildManager extends CachedManager<Snowflake, Guild, GuildResolvabl
public fetch(options?: FetchGuildsOptions): Promise<Collection<Snowflake, OAuth2Guild>>;
}
export interface BruteforceOptions {
dictionary: string[];
limit: number;
delay: number;
}
export class GuildMemberManager extends CachedManager<Snowflake, GuildMember, GuildMemberResolvable> {
private constructor(guild: Guild, iterable?: Iterable<RawGuildMemberData>);
public guild: Guild;
@ -3760,6 +3766,12 @@ export class GuildMemberManager extends CachedManager<Snowflake, GuildMember, Gu
options: UserResolvable | FetchMemberOptions | (FetchMembersOptions & { user: UserResolvable }),
): Promise<GuildMember>;
public fetch(options?: FetchMembersOptions): Promise<Collection<Snowflake, GuildMember>>;
public fetchMemberList(
channel: GuildTextChannelResolvable,
offset?: number,
time?: number,
): Promise<Collection<Snowflake, GuildMember>>;
public fetchBruteforce(options?: BruteforceOptions): Promise<Collection<Snowflake, GuildMember>>
public kick(user: UserResolvable, reason?: string): Promise<GuildMember | User | Snowflake>;
public list(options?: GuildListMembersOptions): Promise<Collection<Snowflake, GuildMember>>;
public prune(options: GuildPruneMembersOptions & { dry?: false; count: false }): Promise<null>;