Commit graph

1318 commits

Author SHA1 Message Date
Claire
76761d5fc0
Add ability for admins to delete canonical email blocks (#16644)
* Add admin option to remove canonical email blocks from a deleted account

* Add tootctl canonical_email_blocks to inspect and remove canonical email blocks
2021-12-17 23:02:14 +01:00
Claire
7f803c41e2
Add ability to purge undeliverable domains from admin interface (#16686)
* Add ability to purge undeliverable domains from admin interface

* Add tests
2021-12-17 23:01:21 +01:00
Eugen Rochko
0fb9536d38
Add batch suspend for accounts in admin UI (#17009) 2021-12-05 21:48:39 +01:00
Jeong Arm
1c826471e7
Fix admin statuses order(#16937) (#16969)
* Fix #16937

* Add test for statuses order
2021-11-26 22:12:27 +01:00
Eugen Rochko
7de0ee7aba
Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
Claire
013bee6afb
Fix filtering DMs from non-followed users (#17042) 2021-11-25 23:46:30 +01:00
Eugen Rochko
6e50134a42
Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire
02a87431cf
Fix error when suspending user with an already-existing canonical email block (#17036)
* Fix error when suspending user with an already-existing canonical email block

Fixes #17033

While attempting to create a `CanonicalEmailBlock` with an existing hash would
raise an `ActiveRecord::RecordNotUnique` error, this being done within a
transaction would cancel the whole transaction. For this reason, checking for
uniqueness in Rails would query the database within the transaction and avoid
invalidating the whole transaction for this reason.

A race condition is still possible, where multiple accounts sharing a canonical
email would be blocked in concurrent transactions, in which only one would
succeed, but that is way less likely to happen that the current issue, and can
always be retried after the first failure, unlike the current situation.

* Add tests
2021-11-24 17:41:03 +01:00
Claire
87085a5152
Fix AccountNote not having a maximum length (#16942) 2021-11-06 00:12:25 +01:00
Eugen Rochko
39cdf61ab7
Add support for structured data and more OpenGraph tags to link cards (#16938)
Save preview cards under their canonical URL

Increase max redirects to follow from 2 to 3
2021-11-05 23:23:05 +01:00
Claire
989c67d29d
Fix handling announcements with links (#16941)
Broken since #15827
2021-11-05 21:14:35 +01:00
Claire
ec059317fa
Fix some link previews being incorrectly generated from other prior links (#16885)
* Add tests

* Fix some link previews being incorrectly generated from different prior links

PR #12403 added a cache to avoid redundant queries when the OEmbed endpoint can
be guessed from the URL. This caching mechanism is not perfectly correct as
there is no guarantee that all pages from a given domain share the same
OEmbed provider endpoint.

This PR prevents the FetchOEmbedService from caching OEmbed endpoint that
cannot be generalized by replacing a fully-qualified URL from the endpoint's
parameters, greatly reducing the number of incorrect cached generalizations.
2021-10-21 20:39:35 +02:00
Claire
3f9b28ce26
Add support for fetching Create and Announce activities by URI (#16383)
* Add support for fetching Create and Announce activities by URI

This should improve compatibility with ZAP and offer a way to fetch boosts,
which is currently not possible.

* Add tests
2021-10-21 01:14:04 +02:00
Takeshi Umeda
17f4e457b3
Add remove from followers api (#16864)
* Add followed_by? to account_interactions

* Add RemoveFromFollowersService

* Fix AccountBatch to use RemoveFromFollowersService

* Add remove from followers API
2021-10-18 12:02:35 +02:00
Jeong Arm
f4081d1564
Allow keeping only 1 boosts/favs on auto deleting posts (#16653)
* Allow keeping 1 boosts/favs on auto deleting posts

* Fix tests
2021-10-14 21:11:14 +02:00
Claire
216570ad98
Fix scheduled statuses decreasing statuses counts (#16791)
* Add tests

* Fix scheduled statuses decreasing statuses counts

Fixes #16774
2021-10-14 19:59:21 +02:00
Claire
5159ba26e4
Fix error when rendering public pages with media attachments (#16763)
* Add tests

* Fix error when rendering public pages with media attachments

* Add tests

* Fix tests

* Please CodeClimate
2021-10-13 15:27:19 +02:00
Claire
84ceebe1c4
Fix media attachment size validation not correctly accounting for file type (#16819)
* Fix media attachment size validation not correctly accounting for file type

Fixes a regression introduced in #16724 caused by the fact that kt-paperclip
now correctly runs validations before processing, meaning that file size
verification could not rely on our before_post_processing hook.

Moved the `before_post_processing` hooks to `before_validate` to make sure
the media attachment type is set correctly before the file gets validated.

* Add tests
2021-10-06 14:49:32 +01:00
Claire
24f9ea7818
Fix webauthn secure key authentication (#16792)
* Add tests

* Fix webauthn secure key authentication

Fixes #16769
2021-09-30 05:26:29 +02:00
Claire
fc3ae1343d
Switch from unmaintained paperclip to kt-paperclip (#16724)
* Switch from unmaintained paperclip to kt-paperclip

* Drop some compatibility monkey-patches not required by kt-paperclip

* Drop media spoof check monkey-patching

It's broken with kt-paperclip and hopefully it won't be needed anymore

* Fix regression introduced by paperclip 6.1.0

* Do not rely on pathname to call FastImage

* Add test for ogg vorbis file with cover art

* Add audio/vorbis to the accepted content-types

This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…

* Restore missing for_as_default method

* Refactor Attachmentable concern and delay Paperclip's content-type spoof check

Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.

* Please CodeClimate

* Add audio/vorbis to the unreliable set

It doesn't correspond to a file format and thus has no extension associated.
2021-09-29 23:52:36 +02:00
Eugen Rochko
52e5c07948
Change routing paths to use usernames in web UI (#16171) 2021-09-26 05:46:13 +02:00
Claire
e0af97164a
Fix followers synchronization mechanism not working when URI has empty path (#16744)
Follow-up to #16510, forgot the controller exposing the actual followers…
2021-09-15 18:51:16 +02:00
Claire
db57bff11d
Stop setting a shortcode to newly-created media attachments (#16730)
* Stop setting a shortcode to newly-created media attachments

The WebUI has stopped using the “short media URL” in ages. This isn't used
anywhere except for mail notifications.

Deprecating it would allow us to eventually get rid of at least a database
column and corruption-prone index, as well as a controller.

* Fix tests
2021-09-13 18:59:37 +02:00
Claire
12cd097e7c
Fix addressing of remote groups' followers (#16700)
Fixes #16699
2021-09-08 23:33:23 +02:00
Claire
2b18f7a943
Fix processing mentions to domains with non-ascii TLDs (#16689)
Fixes #16602
2021-09-01 22:06:40 +02:00
Claire
94bcf45321
Fix authentication failures after going halfway through a sign-in attempt (#16607)
* Add tests

* Add security-related tests

My first (unpublished) attempt at fixing the issues introduced (extremely
hard-to-exploit) security vulnerabilities, addressing them in a test.

* Fix authentication failures after going halfway through a sign-in attempt

* Refactor `authenticate_with_sign_in_token` and `authenticate_with_two_factor` to make the two authentication steps more obvious
2021-08-25 22:52:41 +02:00
Claire
6702148472
Add tests for SuspendAccountService and UnsuspendAccountService (#16627)
* Add tests for SuspendAccountService

* Add tests for UnsuspendAccountService
2021-08-20 10:53:33 +01:00
Claire
5efb1ff337
Fix followers synchronization mechanism not working when URI has empty path (#16510)
* Fix followers synchronization mechanism not working when URI has empty path

To my knowledge, there is no current implementation on the fediverse
that can use bare domains (e.g., actor is at https://example.org instead of
something like https://example.org/actor) that also plans to support the
followers synchronization mechanism. However, Mastodon's current implementation
would exclude such accounts from followers list.

Also adds tests and rename them to reflect the proper method names.

* Move url prefix regexp to its own constant
2021-08-11 17:48:42 +02:00
Claire
13b08610a0
Fix crash when encountering invalid account fields (#16598)
* Add test

* Fix crash when encountering invalid account fields
2021-08-11 16:40:55 +02:00
Claire
4ac78e2a06
Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Claire
763ab0c7eb
Fix owned account notes not being deleted when an account is deleted (#16579)
* Add account_notes relationship

* Add tests

* Fix owned account notes not being deleted when an account is deleted

* Add post-migration to clean up orphaned account notes
2021-08-08 15:29:57 +02:00
Claire
30ce6e395c
Fix user email address being banned on self-deletion (#16503)
* Add tests

* Fix user email address being banned on self-deletion

Fixes #16498
2021-07-14 05:35:49 +02:00
Claire
5a1e072517
Change references to tootsuite/mastodon to mastodon/mastodon (#16491)
* Change references to tootsuite/mastodon to mastodon/mastodon

* Remove obsolete test fixture

* Replace occurrences of tootsuite/mastodon with mastodon/mastodon in CHANGELOG

And a few other places
2021-07-13 15:46:20 +02:00
Eugen Rochko
771c9d4ba8
Add ability to skip sign-in token authentication for specific users (#16427)
Remove "active within last two weeks" exception for sign in token requirement

Change admin reset password to lock access until the password is reset
2021-07-08 05:31:28 +02:00
Claire
225c6582d1
Add tests for BootstrapTimelineService (#16476) 2021-07-07 21:12:43 +02:00
Ikko Ashimine
67226acf7e
Fix typo in tag_feed_spec.rb (#16466)
existant -> existent
2021-07-05 19:16:21 +02:00
Claire
49219508bc
Fix anonymous access to outbox not being cached by the reverse proxy (#16458)
* Fix anonymous access to outbox not being cached by the reverse proxy

Up until now, anonymous access to outbox was marked as public, but with a
0 duration for caching, which means remote proxies would only serve from cache
when the server was completely overwhelmed.

Changed that cache duration to one minute, so that repeated anonymous access
to one account's outbox can be appropriately cached.

Also added `Signature` to the `Vary` header in case a page is requested, so
that authenticated fetches are never served from cache (which only contains
public toots).

* Remove Vary: Accept header from webfinger controller

Indeed, we have stopped returning xrd, and only ever return jrd, so the
Accept request header does not matter anymore.

* Cache negative webfinger hits for 3 minutes
2021-07-03 21:13:47 +02:00
Claire
63b807cffc
Fix serialization of followers/following counts when user hides their network (#16418)
* Add tests

* Fix serialization of followers/following counts when user hides their network

Fixes #16382

Signed-off-by: Claire <claire.github-309c@sitedethib.com>
2021-06-21 20:14:47 +02:00
Eugen Rochko
d174d12c83
Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
Claire
afb7882189
Fix blocking someone not clearing up list feeds (#16205) 2021-05-10 17:31:55 +02:00
Eugen Rochko
1294f9ee4f
Remove PubSubHubbub-related columns from accounts table (#16170) 2021-05-07 19:32:58 +02:00
Eugen Rochko
74081433d0
Change trending hashtags to be affected be reblogs (#16164)
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Eugen Rochko
2c77d97e0d
Add joined date to profiles in web UI (#16169) 2021-05-07 14:33:19 +02:00
Claire
566fc90913
Add Ruby 3.0 support (#16046)
* Fix issues with POSIX::Spawn, Terrapin and Ruby 3.0

Also improve the Terrapin monkey-patch for the stderr/stdout issue.

* Fix keyword argument handling throughout the codebase

* Monkey-patch Paperclip to fix keyword arguments handling in validators

* Change validation_extensions to please CodeClimate

* Bump microformats from 4.2.1 to 4.3.1

* Allow Ruby 3.0

* Add Ruby 3.0 test target to CircleCI

* Add test for admin dashboard warnings

* Fix admin dashboard warnings on Ruby 3.0
2021-05-06 14:22:54 +02:00
Claire
8c44b723bb
Change confirmations controller to redirect to / for approved users (#16151)
Clicking the confirmation link multiple times currently leads to entering
account settings, which can be confusing. This commit changes that so that
it redirects to the root path, so it behaves the same way as clicking only
once in most cases.
2021-05-03 15:45:19 +02:00
abcang
7f0c49c58a
Improve tag search query (#16104) 2021-04-25 06:33:28 +02:00
Eugen Rochko
daccc07dc1
Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
Claire
a6564d56d6
Fix edge case where accepted follow cannot be processed because of follow limit (#16098) 2021-04-23 22:51:21 +02:00
Eugen Rochko
9cc283f0b4
Change the nouns "toot" and "status" to "post" (#16080) 2021-04-21 18:31:24 +02:00
Claire
0b36e3419d
Fix processing of remote Delete activities (#16084)
* Add tests

* Ensure deleted statuses are marked as such

* Save some redis memory by not storing URIs in delete_upon_arrival values

* Avoid possible race condition when processing incoming Deletes

* Avoid potential duplicate Delete forwards

* Lower lock durations to reduce issues in case of hard crash of the Rails process

* Check for `lock.aquired?` and improve comment

* Refactor RedisLock usage in app/lib/activitypub

* Fix using incorrect or non-existent sender for relaying Deletes
2021-04-21 04:46:09 +02:00
Eugen Rochko
b3ceb3dcc4
Add canonical e-mail blocks for suspended accounts (#16049)
Prevent new accounts from being created using the same underlying
e-mail as a suspended account using extensions and period
permutations. Stores e-mails as a SHA256 hash
2021-04-17 03:14:25 +02:00
Eugen Rochko
3b8d085436
Fix app name, website and redirect URIs not having a maximum length (#16042)
Fix app scopes not being validated
2021-04-15 16:28:43 +02:00
Eugen Rochko
ce2148c571
Add policy param to POST /api/v1/push/subscriptions (#16040)
With possible values `all`, `followed`, `follower`, and `none`,
control from whom notifications will generate a Web Push alert
2021-04-15 05:00:25 +02:00
Eugen Rochko
120965eb0b
Change Web Push API deliveries to use request pooling (#16014) 2021-04-12 14:25:34 +02:00
Eugen Rochko
f7117646af
Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Eugen Rochko
619fad6cf8
Remove spam check and dependency on nilsimsa gem (#16011) 2021-04-11 11:22:50 +02:00
Eugen Rochko
7183d9a113
Change multiple mentions with same username to render with domain (#15718)
Fix #15506
2021-04-10 11:51:02 +02:00
Claire
a650a1157d
Fix /admin/tags/:id crashing since Rails 6.1 update (#15953)
Raw SQL passed to `pluck` now has to be explicitly marked as SQL via
Arel.sql, see https://github.com/rails/rails/pull/27947
2021-03-26 18:36:16 +01:00
Claire
cbd0ee1d07
Update Mastodon to Rails 6.1 (#15910)
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Claire
1c4dee4554
Fix Mastodon not understanding as:Public and Public (#15948)
Fixes #5551
2021-03-24 10:19:40 +01:00
Claire
051efed5ed
Bypass MX validation for explicitly allowed domains (#15930)
* Bypass MX validation for explicitly allowed domains

This spares some lookups and prevent issues in some edge cases with
local domains.

* Add tests

* Fix test
2021-03-19 23:48:47 +01:00
Claire
741d0952b1
Improve account counters handling (#15913)
* Improve account counters handling

* Use ActiveRecord::Base::sanitize_sql to pass values instead of interpolating them

Keep using string interpolation for `key` as it is safe and using
“ActiveRecord::Base::sanitize_sql_hash_for_assignment” would require stitching
bits of SQL in a way that is not more easily checked for safety.

* Add migration hook to catch PostgreSQL versions earlier than 9.5
2021-03-19 13:14:57 +01:00
Claire
b358229834
Further preparation for Rails 6 (#15916)
* Use ActiveRecord::Result#to_ary instead of deprecated to_hash

They do the same thing, and to_hash has been removed from Rails 6.1

* Explicitly name polymorphic indexes to workaround a bug in Rails 6.1

cf. https://github.com/rails/rails/issues/41693

* Fix incorrect usage of “foreign_key” in migration script

* Use `ActiveModel::Errors#delete` instead of deprecated clear method

* Fix link headers tests on Rails 6.1

Rails 6.1 adds values to the Link header by default, thus it is not a
LinkHeader object anymore. Fix the test to parse the Link header instead
of assuming it is a LinkHeader.
2021-03-19 02:45:34 +01:00
Claire
a4dcaef53b
Prepare Mastodon for zeitwerk autoloader (#15917)
* Prepare Mastodon for zeitwerk autoloader (Rails 6)

Add inflections and rename/move a few classes.

In particular, app/lib/exceptions.rb and app/lib/sanitize_config.rb
were manually loaded while still in autoload paths.

* Add inflection for Url → URL
2021-03-19 02:42:43 +01:00
Claire
5027abecd1
Fix cache_collection crashing when given an empty collection (#15921)
* Fix cache_collection crashing when given an empty collection

* Add tests
2021-03-18 00:41:32 +01:00
Claire
43eff898a0
Prepare Mastodon for Rails 6 (#15911)
* Fix misuse of foreign_type

* Fix use of removed "add_template_helper"

* Use response.media_type instead of response.content_type in tests

* Fix CSV export controller test on Rails 6

Rails 6 sets a "filename*" field in the Content-Disposition header to
explicitly encode the filename as UTF-8.

This changes checks the first part of the Content-Disposition header so
it matches in both Rails 5 and Rails 6.

* Fix emoji formatting with Rails 6

* Make emoji output more idiomatic and robust

* Switch from redis-rails gem to built-in Rails redis cache storage
2021-03-17 10:09:55 +01:00
Claire
5cc45d22d3
Remove subscription_expires_at leftover from OStatus (#15857) 2021-03-12 05:25:24 +01:00
Claire
5614e6724e
Fix URL scanning in note length validator and preview card fetching (#15827)
* Add tests

* Fix URL scanning in note length validator and preview card fetching
2021-03-04 00:12:26 +01:00
Claire
65db262550
Update twitter-text from 1.14 to 3.1.0 and fix toot character counting (#15382)
* Update twitter-text from 1.14 to 3.1.0

* Disable emoji parsing

* Properly depend on twitter-text for url detection

* Fix some URLs being wrongly detected client-side

* Add test for server-side validation of non-autolinkable URLs

* Fix server-side status length counting
2021-03-02 12:02:56 +01:00
Eugen Rochko
9aa37b32c3
Add details to error response for POST /api/v1/accounts in REST API (#15803) 2021-03-01 04:59:13 +01:00
Claire
5f4c0b79c2
Change ResolveAccountService's handling of skip_webfinger (#15750)
* Change ResolveAccountService's handling of skip_webfinger

Change it so it never makes any webfinger query, as the name would imply.

* Add tests

* Change FollowService to not take an URI for target_account

* Restore domain-block check in FollowService

* Fix tests
2021-02-24 06:32:13 +01:00
Eugen Rochko
8331fdf7e0
Add server rules (#15769) 2021-02-21 19:50:12 +01:00
Claire
be3b9f8151
Fix URI of repeat follow requests not being recorded (#15662)
* Fix URI of repeat follow requests not being recorded

In case we receive a “repeat” or “duplicate” follow request, we automatically
fast-forward the accept with the latest received Activity `id`, but we don't
record it.

In general, a “repeat” or “duplicate” follow request may happen if for some
reason (e.g. inconsistent handling of Block or Undo Accept activities, an
instance being brought back up from the dead, etc.) the local instance thought
the remote actor were following them while the remote actor thought otherwise.

In those cases, the remote instance does not know about the older Follow
activity `id`, so keeping that record serves no purpose, but knowing the most
recent one is useful if the remote implementation at some point refers to it
by `id` without inlining it.

* Add tests
2021-02-11 01:53:44 +01:00
ThibG
a044ddac5b
Fix race conditions on account migration creation (#15597)
* Atomically check for processing lock in Move handler

* Prevent race condition when creating account migrations

Fixes #15595

* Add tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-02-02 14:49:57 +01:00
Shubhendra Singh Chauhan
c8d11b8bdb
Fixed code quality issues (#15541)
* Added .deepsource.toml

* Removed bad use of `alias`

* Fixed operand order in the binary expression

* Prefixed unused method arguments with an underscore

* Replaced the old OpenSSL algorithmic constants with the newer strings initializers.

* Removed unnecessary UTF-8 encoding comment
2021-01-31 21:26:09 +01:00
abcang
7ab53f221a
Improved performance of notification preloading (#15640)
* Improved performance of notification preloading

* Remove Cacheable from Notification

* Fix test
2021-01-31 21:24:57 +01:00
ThibG
54d4e5252b
Use Rails' index_by where it makes sense (#15542)
* Use Rails' index_by where it makes sense

* Fix tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-12 09:27:38 +01:00
11d603101a
Fix muting users with duration via the REST api (#15516) 2021-01-10 12:47:21 +01:00
ThibG
f1f96ebf02
Fix being able to import more than allowed number of follows (#15384)
* Fix being able to import more than allowed number of follows

Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.

This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).

* Catch validation errors to not re-queue failed follows

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-26 23:52:46 +01:00
ThibG
3249d35bdc
Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
ThibG
1cf2c3a810
Fix external user creation failing when invite request text is required (#15405)
* Fix external user creation failing when invite request text is required

Also fixes tootctl-based user creation.

* Add test about invites when invite request text is otherwise required

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 17:14:32 +01:00
Eugen Rochko
9915d11c0d
Fix unnecessary queries when batch-removing statuses, 100x faster (#15387) 2020-12-22 17:13:55 +01:00
ThibG
43961035a9
Fix some notifications not being deleted on poll/status deletion (#15402)
* Fix deleting polls not deleting notifications

* Fix fav notification deletion when deleting a toot

* Refactor DeleteAccountService spec

* Add DeleteAccountService tests for other associations and notifications

* Add favourite handling spec in status removal

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-21 18:22:17 +01:00
ThibG
a60d9335d8
Fix resolving accounts sometimes creating duplicate records for a given AP id (#15364)
* Fix ResolveAccountService accepting mismatching acct: URI

* Set attributes that should be updated regardless of suspension

* Fix key fetching

* Automatically merge remote accounts with duplicate `uri`

* Add tests

* Add "tootctl accounts fix-duplicates"

Finds duplicate accounts sharing a same ActivityPub `id`, re-fetch them and
merge them under the canonical `acct:` URI.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-18 23:26:26 +01:00
Eugen Rochko
eb35be0431
Fix follow limit preventing re-following of a moved account (#14207) 2020-12-18 09:18:31 +01:00
ThibG
b1feb47055
Improve searching for private toots from URL (#14856)
* Improve searching for private toots from URL

Most of the time, when sharing toots, people use the toot URL rather than
the toot URI, which makes sense since it is the user-facing URL.

In Mastodon's case, the URL and URI are different, and Mastodon does not
have an index on URL, which means searching a private toot by URL is done
with a slow query that will only succeed for very recent toots.

This change gets rid of the slow query, and attempts to guess the URI from
URL instead, as Mastodon's are predictable.

* Add tests

* Only return status with guessed uri if url matches

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-17 06:51:49 +01:00
ThibG
8357969559
Fix admins being able to suspend their instance actor (#14567)
* Fix admin being able to suspend their own instance account

* Add text about the instance's own actor in admin view

* Change instance actor notice from flash message to template

* Do not list local instance actor in account moderation list
2020-12-15 17:23:58 +01:00
Eugen Rochko
1f564051b6
Change RTL detection to rely on unicode-bidi paragraph by paragraph (#14573) 2020-12-15 12:56:43 +01:00
Eugen Rochko
216b85b053
Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
ThibG
49eb4d4ddf
Add honeypot fields and minimum fill-out time for sign-up form (#15276)
* Add honeypot fields to limit non-specialized spam

Add two honeypot fields: a fake website input and a fake password confirmation
one. The label/placeholder/aria-label tells not to fill them, and they are
hidden in CSS, so legitimate users should not fall into these.

This should cut down on some non-Mastodon-specific spambots.

* Require a 3 seconds delay before submitting the registration form

* Fix tests

* Move registration form time check to model validation

* Give people a chance to clear the honeypot fields

* Refactor honeypot translation strings

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-10 06:27:26 +01:00
127c543a6e
rename replies_policy enumerables (#15304) 2020-12-09 04:34:17 +01:00
ThibG
c43f4cd3bb
Fix not being able to unfavorite toots one has lost access to (#15192)
Fixes #15191
2020-11-21 06:18:09 +01:00
ThibG
8b8004a962
Fix webfinger redirect handling in ResolveAccountService (#15187)
* Fix webfinger redirect handling in ResolveAccountService

ResolveAccountService#process_webfinger! handled a one-step webfinger
redirection, but only accepting the result if it matched the exact URI passed
as input, defeating the point of a redirection check.

Instead, use the same logic as in `ActivityPub::FetchRemoteAccountService`,
updating the resulting `acct:` URI with the result of the first webfinger
query.

* Add tests
2020-11-19 19:52:06 +01:00
ThibG
96c1e71329
Add import/export feature for bookmarks (#14956)
* Add ability to export bookmarks

* Add support for importing bookmarks

* Add bookmark import tests

* Add bookmarks export test
2020-11-19 17:48:13 +01:00
Eugen Rochko
8532429af7
Fix 2FA/sign-in token sessions being valid after password change (#14802)
If someone tries logging in to an account and is prompted for a 2FA
code or sign-in token, even if the account's password or e-mail is
updated in the meantime, the session will show the prompt and allow
the login process to complete with a valid 2FA code or sign-in token
2020-11-12 23:05:01 +01:00
Eugen Rochko
337dc6e0ad
Fix updating account counters when account_stat is not yet created (#15108) 2020-11-09 16:00:23 +01:00
Eugen Rochko
3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
Takeshi Umeda
d6fe0c94ca
Add account sensitized (#14361)
* Add account sensitized

* Fix i18n normalize

* Fix description and spec

* Fix spec

* Fix wording
2020-11-04 20:45:01 +01:00
ThibG
ca56527140
Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
Eugen Rochko
5e1364c448
Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00