FoundKeyGang/FoundKey
FoundKeyGang/FoundKey
7
19
Fork 22
Code Issues 37 Pull requests 4 Releases Activity

backend: add automatic dead instance detection #204

Merged
toast merged 3 commits from dead-instance into main 2022-10-16 13:45:29 +00:00
Conversation 1 Commits 3 Files changed 1 +23 -7
1 changed files with 23 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files
Showing only changes of commit 91a4f38871 - Show all commits

28
packages/backend/src/queue/processors/deliver.ts
View file

@ -12,36 +12,52 @@ import { Cache } from '@/misc/cache.js';
import { Instance } from '@/models/entities/instance.js';
import { StatusError } from '@/misc/fetch.js';
import { DeliverJobData } from '@/queue/types.js';
import { LessThan } from 'typeorm';
const logger = new Logger('deliver');
let latest: string | null = null;
const suspendedHostsCache = new Cache<Instance[]>(1000 * 60 * 60);
// dead host list is a linear scan, so cache it longer
const deadHostsCache = new Cache<Instance[]>(1000 * 60 * 60 * 24);
const deadThreshold = 1000 * 60 * 60 * 24 * 30; // 1 month
toast marked this conversation as resolved Outdated
Johann150 commented 2022-10-16 12:39:00 +00:00
Outdated
Review
Copy link

You should probably use the time constants here:

import { DAY } from '@/const.ts';

//...
const deadHostsCache = new Cache<Instance[]>(DAY);
const deadThreshold = 30 * DAY;
You should probably use the time constants here: ```typescript import { DAY } from '@/const.ts'; //... const deadHostsCache = new Cache<Instance[]>(DAY); const deadThreshold = 30 * DAY; ```
toast commented 2022-10-16 13:03:50 +00:00
Outdated
Review
Copy link

Should probably also do that for prelude/time.ts huh
TODO ig

Should probably also do that for prelude/time.ts huh TODO ig
export default async (job: Bull.Job<DeliverJobData>) => {
const { host } = new URL(job.data.to);
const puny = toPuny(host);
// ブロックしてたら中断
const meta = await fetchMeta();
if (meta.blockedHosts.includes(toPuny(host))) {
if (meta.blockedHosts.includes(puny)) {
return 'skip (blocked)';
}
// isSuspendedなら中断
// isSuspended
let suspendedHosts = suspendedHostsCache.get(null);
if (suspendedHosts == null) {
suspendedHosts = await Instances.find({
where: {
suspendedHosts = await Instances.findBy({
isSuspended: true,
},
});
suspendedHostsCache.set(null, suspendedHosts);
}
if (suspendedHosts.map(x => x.host).includes(toPuny(host))) {
if (suspendedHosts.some(x => x.host == puny)) {
return 'skip (suspended)';
}
// dead
let deadHosts = deadHostsCache.get(null);
if (deadHosts == null) {
const deadTime = new Date(Date.now() - deadThreshold);
deadHosts = await Instances.findBy({
lastCommunicatedAt: LessThan(deadTime),
});
deadHostsCache.set(null, deadHosts);
}
toast marked this conversation as resolved Outdated
Johann150 commented 2022-10-16 12:51:08 +00:00
Outdated
Review
Copy link

This whole bit with the caching I don't really understand. Couldn't you just as well just request that one instance? Perhaps something like this:

const deadTime = new Date(Date.now() - deadThreshold);
const isDead = await Instances.countBy({
	host,
	lastCommunicatedAt: LessThan(deadTime),
});

if (isDead) return 'skip (dead instance)';

This would also have the advantage of immediately resuming contact with an instance instead of having to wait for the cache to time out.

This whole bit with the caching I don't really understand. Couldn't you just as well just request that one instance? Perhaps something like this: ```typescript const deadTime = new Date(Date.now() - deadThreshold); const isDead = await Instances.countBy({ host, lastCommunicatedAt: LessThan(deadTime), }); if (isDead) return 'skip (dead instance)'; ``` This would also have the advantage of immediately resuming contact with an instance instead of having to wait for the cache to time out.
toast commented 2022-10-16 13:03:00 +00:00
Outdated
Review
Copy link

Would it make sense to do that for suspended instances too? (same logic)

Would it make sense to do that for suspended instances too? (same logic)
Johann150 commented 2022-10-16 13:04:57 +00:00
Outdated
Review
Copy link

I think yes? I don't know why it was done that way and how the different performances are but it might well have just been waves hands classic Misskey.

I think yes? I don't know why it was done that way and how the different performances are ~~but it might well have just been *waves hands* classic Misskey~~.
👍 1
Johann150 commented 2022-10-16 13:25:53 +00:00
Outdated
Review
Copy link

I think you removed the definition of deadTime.

I think you removed the definition of `deadTime`.
toast commented 2022-10-16 13:30:44 +00:00
Outdated
Review
Copy link

how to tell I'm not finished caffeinating 👍

how to tell I'm not finished caffeinating 👍
if (deadHosts.some(x => x.host == puny)) {
return 'skip (dead instance)';
}
try {
if (latest !== (latest = JSON.stringify(job.data.content, null, 2))) {
logger.debug(`delivering ${latest}`);