[Discussion] IPFS support?

nullobsi commented

2022-11-01 01:56:05 +00:00

Contributor

Hi! I want to implement some IPFS support into Misskey/Foundkey, and I wanted to get some feedback about how an implementation should work and throw around some ideas

First off, I'm not sure if this would fit into the main FoundKey repository or the goals, but I still wanted to ask around about how some of y'all think an implementation should work :p

Also, I know deletion is an issue with IPFS, so any implementation would be opt-in and add addition warnings on the front-end about IPFS's permenance.

I had some ideas:

Simply store IPFS gateway URLs

This wouldn't require any changes to the DB and not that much changes on the frontend either. The backend would purely handle connecting to a local IPFS node to upload the files, which would then be served via the gateway URL.
Pros:

Simple!
Would work for most cases.

Cons:

Would probably need regexes to determine how to handle IPFS files when you want to delete them
Tons of if statements to handle these IPFS links
Probably not elegant or maintainable
Not permanent, survives a rollback

Store IPFS CIDs in the DriveFile database

It would need schema changes but I feel would be the most elegant solution and is what I'm working on right now. Add an ipfsCID column to DriveFile, leave all other fields null; the client could then simply use ipfs:// URLs if the IPFS extension is installed or most likely use a new exposed meta property ipfsPublicGateway to get file links.
Pros:

Would be a bit more code to add, but I feel would be a bit more maintainable and patchable compared to the previous option.
More explicit as to what files are on IPFS and what files are not.

Cons:

Requires a breaking DB change, so you can't just change forks, and must apply the patch.

...

For both options, there would need to be backend changes to support both a local IPFS gateway to upload and pin files, and to serve the IPFS links from a chosen public gateway.

Also, I'm not sure how IPFS would interact with webpublic or thumbnails. My idea was to forgoe thumbnails and webpublic entirely when IPFS is enabled, but I have another idea that involves storing the webpublic, thumbnail, and original file under an IPFS folder. To me though, this feels like it would be another Misskey-ism and not fit very well with IPFS. Maybe the thumbails should be stored exclusively local, and don't use webpublic?

Another idea to improve file availability: Expose IPFS Peer IDs through Nodeinfo so that other instances can add Fediverse peers as IPFS peers, which would help connect nodes and make files load faster through the DHT.

Hi! I want to implement some IPFS support into Misskey/Foundkey, and I wanted to get some feedback about how an implementation should work and throw around some ideas First off, I'm not sure if this would fit into the main FoundKey repository or the goals, but I still wanted to ask around about how some of y'all think an implementation should work :p Also, I know deletion is an issue with IPFS, so any implementation would be opt-in and add addition warnings on the front-end about IPFS's permenance. I had some ideas: ## Simply store IPFS gateway URLs This wouldn't require any changes to the DB and not that much changes on the frontend either. The backend would purely handle connecting to a local IPFS node to upload the files, which would then be served via the gateway URL. Pros: - Simple! - Would work for most cases. Cons: - Would probably need regexes to determine how to handle IPFS files when you want to delete them - Tons of if statements to handle these IPFS links - Probably not elegant or maintainable - Not permanent, survives a rollback ## Store IPFS CIDs in the DriveFile database It would need schema changes but I feel would be the most elegant solution and is what I'm working on right now. Add an ipfsCID column to DriveFile, leave all other fields null; the client could then simply use ipfs:// URLs if the IPFS extension is installed or most likely use a new exposed `meta` property `ipfsPublicGateway` to get file links. Pros: - Would be a bit more code to add, but I feel would be a bit more maintainable and patchable compared to the previous option. - More explicit as to what files are on IPFS and what files are not. Cons: - Requires a breaking DB change, so you can't just change forks, and must apply the patch. ## ... For both options, there would need to be backend changes to support both a local IPFS gateway to upload and pin files, and to serve the IPFS links from a chosen public gateway. Also, I'm not sure how IPFS would interact with webpublic or thumbnails. My idea was to forgoe thumbnails and webpublic entirely when IPFS is enabled, but I have another idea that involves storing the webpublic, thumbnail, and original file under an IPFS folder. To me though, this feels like it would be another Misskey-ism and not fit very well with IPFS. Maybe the thumbails should be stored exclusively local, and don't use webpublic? Another idea to improve file availability: Expose IPFS Peer IDs through Nodeinfo so that other instances can add Fediverse peers as IPFS peers, which would help connect nodes and make files load faster through the DHT.

Johann150 commented

2022-11-02 18:24:41 +00:00

Owner

For clarification because you didn't explicitly say it: What do you want to store on IPFS? From my understanding of what you wrote it seems you only want to store drive files there, and not activities?

From the two alternatives I would prefer the first one that doesn't add a column. Maybe you could do something similar to S3 in regards to files (at least once they are uploaded)?

I'm also not sure how many people would want to use IPFS. At least I'm not really interested, because:

Regarding knowledge about IPFS itself, I have to say that I don't really know anything about it, so I can't really comment on how good of an idea it is to implement in the first place. Maybe others?

For clarification because you didn't explicitly say it: What do you want to store on IPFS? From my understanding of what you wrote it seems you only want to store drive files there, and not activities? From the two alternatives I would prefer the first one that doesn't add a column. Maybe you could do something similar to S3 in regards to files (at least once they are uploaded)? I'm also not sure how many people would want to use IPFS. At least I'm not really interested, because: Regarding knowledge about IPFS itself, I have to say that I don't really know anything about it, so I can't really comment on how good of an idea it is to implement in the first place. Maybe others?

nullobsi commented

2022-11-02 18:34:58 +00:00

Author

Contributor

Only drive files, not activities.

I think it improve the availability and speed of media download, without having to have copies of it sitting on everyone's servers. (I mean, there would be copies, but they would be addressable from the same CID hash and those copies would contribute to the overall availability).

Only drive files, not activities. I think it improve the availability and speed of media download, without having to have copies of it sitting on everyone's servers. (I mean, there would be copies, but they would be addressable from the same CID hash and those copies would contribute to the overall availability).

toast commented

2022-12-03 10:37:57 +00:00

Owner

There's not much interest in this.
I've worked with IPFS previously in the context of distrodev and it really does not perform very well at all.
The gateways perform even worse.
Furthermore, the goal is to "keep the files from sitting on everyone's drives", but IPFS has that as the goal, while for us it's mostly incidental (and usually just thumbnails, and may not remain for that long).

The "proper" way of doing this would be to turn every instance into an IPFS node that can cache all of its interested stuff and supplant the drive functionality entirely...
But yeah there's clearly not much interest or point to doing that at all.

There's not much interest in this. I've worked with IPFS previously in the context of distrodev and it really does not perform very well *at all*. The gateways perform even worse. Furthermore, the goal is to "keep the files from sitting on everyone's drives", but IPFS has that as the goal, while for us it's mostly incidental (and usually just thumbnails, and may not remain for that long). The "proper" way of doing this would be to turn every instance into an IPFS node that can cache all of its interested stuff and supplant the drive functionality entirely... But yeah there's clearly not much interest or point to doing that at all.

toast closed this issue

2022-12-03 10:38:00 +00:00

[Discussion] IPFS support? #222

Simply store IPFS gateway URLs

Store IPFS CIDs in the DriveFile database

...