Api Endpoint POST /api/drive/files returns 500

puniko commented

2022-09-11 07:56:55 +00:00

Contributor

I mostly can't open my Misskey Drive. Folders Endpoint is loading fine, but the Files endpoint returns 500 more often than not. Mostly happens on root directory in the drive so far and its probably related to it having a big chunk of files in it.

{
	"error": {
		"message": "Cannot read properties of null (reading 'message')"
	}
}

Logs in backend don't show anything. just mentiones that api request is taking a long time.

Checked the code a bit, and couldn't really find anything related to messages in the files api and db api area. I suspect the culprit might be hiding somewhere in the pack function in packages/backend/src/models/repositories/user.ts maybe related to unread messages.

Its weird tho that it sometimes does work

I mostly can't open my Misskey Drive. Folders Endpoint is loading fine, but the Files endpoint returns 500 more often than not. Mostly happens on root directory in the drive so far and its probably related to it having a big chunk of files in it. ``` { "error": { "message": "Cannot read properties of null (reading 'message')" } } ``` Logs in backend don't show anything. just mentiones that api request is taking a long time. Checked the code a bit, and couldn't really find anything related to messages in the files api and db api area. I suspect the culprit might be hiding somewhere in the [`pack`](https://akkoma.dev/FoundKeyGang/FoundKey/src/commit/d9f443c6c14b2f6ad5121a6958f4448e6cef5b29/packages/backend/src/models/repositories/user.ts#L241) function in [`packages/backend/src/models/repositories/user.ts`](https://akkoma.dev/FoundKeyGang/FoundKey/src/branch/main/packages/backend/src/models/repositories/user.ts) maybe related to unread messages. Its weird tho that it sometimes does work

Johann150 commented

2022-09-11 11:41:08 +00:00

Owner

I suspect if it only sometimes fails, and there are no errors in the log, and there are a lot of files in your drive, and the logs say it is taking a long time that it may be related to a database timeout.

I also think that the specific error of reading 'message' could more likely come from the database timeout resulting in something unexpected being thrown from the database drive and the normal error handler then tries to get the message property of that.

I suspect if it only sometimes fails, and there are no errors in the log, and there are a lot of files in your drive, and the logs say it is taking a long time that it may be related to a database timeout. I also think that the specific error of `reading 'message'` could more likely come from the database timeout resulting in something unexpected being thrown from the database drive and the normal error handler then tries to get the `message` property of that.

puniko commented

2022-09-12 12:14:09 +00:00

Author

Contributor

I suspect if it only sometimes fails

Well, the thing is, i have to reopen my drive like 20 - 30 times until a request actually comes through without failing.

I also think that the specific error of reading 'message' could more likely come from the database timeout resulting in something unexpected being thrown from the database drive and the normal error handler then tries to get the message property of that.

could very well be too, but in that case, it would at least log the timeout in the misskey logs (it usually does). haven't seen one there.

the thing with drive is, that it takes an awefull long time to load. not sure if it can be improved by setting some indices or something.

> I suspect if it only sometimes fails Well, the thing is, i have to reopen my drive like 20 - 30 times until a request actually comes through without failing. > I also think that the specific error of reading 'message' could more likely come from the database timeout resulting in something unexpected being thrown from the database drive and the normal error handler then tries to get the message property of that. could very well be too, but in that case, it would at least log the timeout in the misskey logs (it usually does). haven't seen one there. the thing with drive is, that it takes an awefull long time to load. not sure if it can be improved by setting some indices or something.

toast commented

2022-09-12 13:32:20 +00:00

Owner

Puni, how big is your db?
If it's under 10gb, I'm wondering how feasible it might be for me to get a copy to experiment with.
It contains user data, but I think it might be possible to yeet all of that out?
Really we should have a process for stuff like this, ideally.

Puni, how big is your db? If it's under 10gb, I'm wondering how feasible it might be for me to get a copy to experiment with. It contains user data, but I think it might be possible to yeet all of that out? Really we should have a process for stuff like this, ideally.

puniko commented

2022-09-12 14:04:23 +00:00

Author

Contributor

Puni, how big is your db?

Around 30GB

> Puni, how big is your db? Around 30GB

puniko commented

2022-09-14 07:59:28 +00:00

Author

Contributor

I think this is be an issue with long running query. When i try to query all files in root directory (on psql directly) it takes an aweful long time to complete (like 1min or more). consequtive queries to perform faster tho and after running the query, drive loads up without issue.

not sure how to fix this tho. i mean, i could make a cron, that runs that query every now and then, but that isn't really a fix

I think this is be an issue with long running query. When i try to query all files in root directory (on psql directly) it takes an aweful long time to complete (like 1min or more). consequtive queries to perform faster tho and after running the query, drive loads up without issue. not sure how to fix this tho. i mean, i could make a cron, that runs that query every now and then, but that isn't really a fix

toast commented

2022-10-21 10:59:37 +00:00

Owner

With #203 merged, is this any better @puniko ? (it's not in preview2, so you'd have to merge it in or update to HEAD to get it)
I suspect it might be helpful here.

With #203 merged, is this any better @puniko ? (it's not in preview2, so you'd have to merge it in or update to HEAD to get it) I suspect it might be helpful here.

puniko commented

2022-10-21 11:08:57 +00:00

Author

Contributor

thanks for the hint. might merge it when i next update it or so. probably also takes some time to be sure of it working.

(i currently do the cron thingy i mentioned before, which is preventing this to happen)

thanks for the hint. might merge it when i next update it or so. probably also takes some time to be sure of it working. (i currently do the cron thingy i mentioned before, which is preventing this to happen)

puniko commented

2022-11-08 14:56:59 +00:00

Author

Contributor

ok, i have ran it for a bit without having the cron on. it is indeed an improvement, but only because it doesn't discard loaded folders when files fail. on the files, i still get a 500 after a while.