Merge remote-tracking branch 'origin/develop' into sixohsix/pleroma-post_expiration

.dockerignore

@@ -0,0 +1,12 @@
+.*
+*.md
+AGPL-3
+CC-BY-SA-4.0
+COPYING
+*file
+elixir_buildpack.config
+docs/
+test/
+
+# Required to get version
+!.git

CHANGELOG.md

@@ -4,27 +4,33 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
-### Added
+### Security
-- Expiring/ephemeral activites. All activities can have expires_on value set, which controls when they should be deleted automatically.
+- OStatus: eliminate the possibility of a protocol downgrade attack.
-- Mastodon API: in post_status, the expires_in parameter lets you set the number of minutes until an activity expires. It must be at least one hour.
+- OStatus: prevent following locked accounts, bypassing the approval process.
-- Mastodon API: all status JSON responses contain a `pleroma.expires_on` item which states when an activity will expire. The value is only shown to the user who created the activity. To everyone else it's empty.
-- Configuration: `ActivityExpiration.enabled` controls whether expired activites will get deleted at the appropriate time. Enabled by default.
 
 ### Changed
 - **Breaking:** Configuration: A setting to explicitly disable the mailer was added, defaulting to true, if you are using a mailer add `config :pleroma, Pleroma.Emails.Mailer, enabled: true` to your config
+- **Breaking:** Configuration: `/media/` is now removed when `base_url` is configured, append `/media/` to your `base_url` config to keep the old behaviour if desired
 - Configuration: OpenGraph and TwitterCard providers enabled by default
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
 - Federation: Return 403 errors when trying to request pages from a user's follower/following collections if they have `hide_followers`/`hide_follows` set
 - NodeInfo: Return `skipThreadContainment` in `metadata` for the `skip_thread_containment` option
+- NodeInfo: Return `mailerEnabled` in `metadata`
 - Mastodon API: Unsubscribe followers when they unfollow a user
 - AdminAPI: Add "godmode" while fetching user statuses (i.e. admin can see private statuses)
+- Improve digest email template
 
 ### Fixed
 - Not being able to pin unlisted posts
+- Objects being re-embedded to activities after being updated (e.g faved/reposted). Running 'mix pleroma.database prune_objects' again is advised.
+- Favorites timeline doing database-intensive queries
 - Metadata rendering errors resulting in the entire page being inaccessible
+- `federation_incoming_replies_max_depth` option being ignored in certain cases
 - Federation/MediaProxy not working with instances that have wrong certificate order
 - Mastodon API: Handling of search timeouts (`/api/v1/search` and `/api/v2/search`)
 - Mastodon API: Embedded relationships not being properly rendered in the Account entity of Status entity
+- Mastodon API: follower/following counters not being nullified, when `hide_follows`/`hide_followers` is set
+- Mastodon API: `muted` in the Status entity, using author's account to determine if the tread was muted
 - Mastodon API: Add `account_id`, `type`, `offset`, and `limit` to search API (`/api/v1/search` and `/api/v2/search`)
 - Mastodon API, streaming: Fix filtering of notifications based on blocks/mutes/thread mutes
 - ActivityPub C2S: follower/following collection pages being inaccessible even when authentifucated if `hide_followers`/ `hide_follows` was set
@@ -32,11 +38,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Rich Media: Parser failing when no TTL can be found by image TTL setters
 - Rich Media: The crawled URL is now spliced into the rich media data.
 - ActivityPub S2S: sharedInbox usage has been mostly aligned with the rules in the AP specification.
+- ActivityPub S2S: remote user deletions now work the same as local user deletions.
+- ActivityPub S2S: POST requests are now signed with `(request-target)` pseudo-header.
+- Not being able to access the Mastodon FE login page on private instances
+- Invalid SemVer version generation, when the current branch does not have commits ahead of tag/checked out on a tag
+- Pleroma.Upload base_url was not automatically whitelisted by MediaProxy. Now your custom CDN or file hosting will be accessed directly as expected.
+- Report email not being sent to admins when the reporter is a remote user
+- MRF: ensure that subdomain_match calls are case-insensitive
+- Reverse Proxy limiting `max_body_length` was incorrectly defined and only checked `Content-Length` headers which may not be sufficient in some circumstances
+- MRF: fix use of unserializable keyword lists in describe() implementations
+- ActivityPub: Deactivated user deletion
 
 ### Added
+- Expiring/ephemeral activites. All activities can have expires_on value set, which controls when they should be deleted automatically.
+- Mastodon API: in post_status, the expires_in parameter lets you set the number of minutes until an activity expires. It must be at least one hour.
+- Mastodon API: all status JSON responses contain a `pleroma.expires_on` item which states when an activity will expire. The value is only shown to the user who created the activity. To everyone else it's empty.
+- Configuration: `ActivityExpiration.enabled` controls whether expired activites will get deleted at the appropriate time. Enabled by default.
+- Conversations: Add Pleroma-specific conversation endpoints and status posting extensions. Run the `bump_all_conversations` task again to create the necessary data.
+- **Breaking:** MRF describe API, which adds support for exposing configuration information about MRF policies to NodeInfo.
+  Custom modules will need to be updated by adding, at the very least, `def describe, do: {:ok, %{}}` to the MRF policy modules.
 - MRF: Support for priming the mediaproxy cache (`Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`)
 - MRF: Support for excluding specific domains from Transparency.
 - MRF: Support for filtering posts based on who they mention (`Pleroma.Web.ActivityPub.MRF.MentionPolicy`)
+- MRF: Support for filtering posts based on ActivityStreams vocabulary (`Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`)
 - MRF (Simple Policy): Support for wildcard domains.
 - Support for wildcard domains in user domain blocks setting.
 - Configuration: `quarantined_instances` support wildcard domains.
@@ -47,21 +71,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: Add support for categories for custom emojis by reusing the group feature. <https://github.com/tootsuite/mastodon/pull/11196>
 - Mastodon API: Add support for muting/unmuting notifications
 - Mastodon API: Add support for the `blocked_by` attribute in the relationship API (`GET /api/v1/accounts/relationships`). <https://github.com/tootsuite/mastodon/pull/10373>
+- Mastodon API: Add support for the `domain_blocking` attribute in the relationship API (`GET /api/v1/accounts/relationships`).
 - Mastodon API: Add `pleroma.deactivated` to the Account entity
 - Mastodon API: added `/auth/password` endpoint for password reset with rate limit.
 - Mastodon API: /api/v1/accounts/:id/statuses now supports nicknames or user id
+- Mastodon API: Improve support for the user profile custom fields
 - Admin API: Return users' tags when querying reports
 - Admin API: Return avatar and display name when querying users
 - Admin API: Allow querying user by ID
 - Admin API: Added support for `tuples`.
+- Admin API: Added endpoints to run mix tasks pleroma.config migrate_to_db & pleroma.config migrate_from_db
 - Added synchronization of following/followers counters for external users
 - Configuration: `enabled` option for `Pleroma.Emails.Mailer`, defaulting to `false`.
 - Configuration: Pleroma.Plugs.RateLimiter `bucket_name`, `params` options.
+- Configuration: `user_bio_length` and `user_name_length` options.
 - Addressable lists
 - Twitter API: added rate limit for `/api/account/password_reset` endpoint.
 - ActivityPub: Add an internal service actor for fetching ActivityPub objects.
 - ActivityPub: Optional signing of ActivityPub object fetches.
 - Admin API: Endpoint for fetching latest user's statuses
+- Pleroma API: Add `/api/v1/pleroma/accounts/confirmation_resend?email=<email>` for resending account confirmation.
+- Relays: Added a task to list relay subscriptions.
+- Mix Tasks: `mix pleroma.database fix_likes_collections`
+- Federation: Remove `likes` from objects.
 
 ### Changed
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
@@ -69,6 +101,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - RichMedia: parsers and their order are configured in `rich_media` config.
 - RichMedia: add the rich media ttl based on image expiration time.
 
+### Removed
+- Emoji: Remove longfox emojis.
+- Remove `Reply-To` header from report emails for admins.
+- ActivityPub: The `accept_blocks` configuration setting.
+
 ## [1.0.1] - 2019-07-14
 ### Security
 - OStatus: fix an object spoofing vulnerability.
@@ -79,6 +116,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Rich media: Do not crawl private IP ranges
 
 ### Added
+- Digest email for inactive users
 - Add a generic settings store for frontends / clients to use.
 - Explicit addressing option for posting.
 - Optional SSH access mode. (Needs `erlang-ssh` package on some distributions).
@@ -105,6 +143,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Configuration: `notify_email` option
 - Configuration: Media proxy `whitelist` option
 - Configuration: `report_uri` option
+- Configuration: `email_notifications` option
 - Configuration: `limit_to_local_content` option
 - Pleroma API: User subscriptions
 - Pleroma API: Healthcheck endpoint

Dockerfile

@@ -0,0 +1,39 @@
+FROM rinpatch/elixir:1.9.0-rc.0-alpine as build
+
+COPY . .
+
+ENV MIX_ENV=prod
+
+RUN apk add git gcc g++ musl-dev make &&\
+	echo "import Mix.Config" > config/prod.secret.exs &&\
+	mix local.hex --force &&\
+	mix local.rebar --force &&\
+	mix deps.get --only prod &&\
+	mkdir release &&\
+	mix release --path release
+
+FROM alpine:latest
+
+ARG HOME=/opt/pleroma
+ARG DATA=/var/lib/pleroma
+
+RUN echo "http://nl.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories &&\
+	apk update &&\
+	apk add ncurses postgresql-client &&\
+	adduser --system --shell /bin/false --home ${HOME} pleroma &&\
+	mkdir -p ${DATA}/uploads &&\
+	mkdir -p ${DATA}/static &&\
+	chown -R pleroma ${DATA} &&\
+	mkdir -p /etc/pleroma &&\
+	chown -R pleroma /etc/pleroma
+
+USER pleroma
+
+COPY --from=build --chown=pleroma:0 /release ${HOME}
+
+COPY ./config/docker.exs /etc/pleroma/config.exs
+COPY ./docker-entrypoint.sh ${HOME}
+
+EXPOSE 4000
+
+ENTRYPOINT ["/opt/pleroma/docker-entrypoint.sh"]

README.md

@@ -21,7 +21,7 @@ If you want to run your own server, feel free to contact us at @lain@pleroma.soy
 Currently Pleroma is not packaged by any OS/Distros, but feel free to reach out to us at [#pleroma-dev on freenode](https://webchat.freenode.net/?channels=%23pleroma-dev) or via matrix at <https://matrix.heldscal.la/#/room/#freenode_#pleroma-dev:matrix.org> for assistance. If you want to change default options in your Pleroma package, please **discuss it with us first**.
 
 ### Docker
-While we don’t provide docker files, other people have written very good ones. Take a look at <https://github.com/angristan/docker-pleroma> or <https://github.com/sn0w/pleroma-docker>.
+While we don’t provide docker files, other people have written very good ones. Take a look at <https://github.com/angristan/docker-pleroma> or <https://glitch.sh/sn0w/pleroma-docker>.
 
 ### Dependencies
 

config/config.exs

@@ -253,6 +253,12 @@
   skip_thread_containment: true,
   limit_to_local_content: :unauthenticated,
   dynamic_configuration: false,
+  user_bio_length: 5000,
+  user_name_length: 100,
+  max_account_fields: 10,
+  max_remote_account_fields: 20,
+  account_field_name_length: 512,
+  account_field_value_length: 512,
   external_user_synchronization: true
 
 config :pleroma, :markup,
@@ -302,7 +308,6 @@
   default_mascot: :pleroma_fox_tan
 
 config :pleroma, :activitypub,
-  accept_blocks: true,
   unfollow_blocked: true,
   outgoing_blocks: true,
   follow_handshake_timeout: 500,
@@ -337,6 +342,10 @@
 
 config :pleroma, :mrf_subchain, match_actor: %{}
 
+config :pleroma, :mrf_vocabulary,
+  accept: [],
+  reject: []
+
 config :pleroma, :rich_media,
   enabled: true,
   ignore_hosts: [],
@@ -508,6 +517,17 @@
 
 config :pleroma, Pleroma.Emails.Mailer, adapter: Swoosh.Adapters.Sendmail, enabled: false
 
+config :pleroma, Pleroma.Emails.UserEmail,
+  logo: nil,
+  styling: %{
+    link_color: "#d8a070",
+    background_color: "#2C3645",
+    content_background_color: "#1B2635",
+    header_color: "#d8a070",
+    text_color: "#b9b9ba",
+    text_muted_color: "#b9b9ba"
+  }
+
 config :prometheus, Pleroma.Web.Endpoint.MetricsExporter, path: "/api/pleroma/app_metrics"
 
 config :pleroma, Pleroma.ScheduledActivity,
@@ -515,6 +535,14 @@
   total_user_limit: 300,
   enabled: true
 
+config :pleroma, :email_notifications,
+  digest: %{
+    active: false,
+    schedule: "0 0 * * 0",
+    interval: 7,
+    inactivity_threshold: 7
+  }
+
 config :pleroma, :oauth2,
   token_expires_in: 600,
   issue_new_refresh_token: true,
@@ -535,7 +563,9 @@
   relation_id_action: {60_000, 2},
   statuses_actions: {10_000, 15},
   status_id_action: {60_000, 3},
-  password_reset: {1_800_000, 5}
+  password_reset: {1_800_000, 5},
+  account_confirmation_resend: {8_640_000, 5},
+  ap_routes: {60_000, 15}
 
 config :pleroma, Pleroma.ActivityExpiration, enabled: true
 

config/docker.exs

@@ -0,0 +1,68 @@
+import Config
+
+config :pleroma, Pleroma.Web.Endpoint,
+  url: [host: System.get_env("DOMAIN", "localhost"), scheme: "https", port: 443],
+  http: [ip: {0, 0, 0, 0}, port: 4000]
+
+config :pleroma, :instance,
+  name: System.get_env("INSTANCE_NAME", "Pleroma"),
+  email: System.get_env("ADMIN_EMAIL"),
+  notify_email: System.get_env("NOTIFY_EMAIL"),
+  limit: 5000,
+  registrations_open: false,
+  dynamic_configuration: true
+
+config :pleroma, Pleroma.Repo,
+  adapter: Ecto.Adapters.Postgres,
+  username: System.get_env("DB_USER", "pleroma"),
+  password: System.fetch_env!("DB_PASS"),
+  database: System.get_env("DB_NAME", "pleroma"),
+  hostname: System.get_env("DB_HOST", "db"),
+  pool_size: 10
+
+# Configure web push notifications
+config :web_push_encryption, :vapid_details, subject: "mailto:#{System.get_env("NOTIFY_EMAIL")}"
+
+config :pleroma, :database, rum_enabled: false
+config :pleroma, :instance, static_dir: "/var/lib/pleroma/static"
+config :pleroma, Pleroma.Uploaders.Local, uploads: "/var/lib/pleroma/uploads"
+
+# We can't store the secrets in this file, since this is baked into the docker image
+if not File.exists?("/var/lib/pleroma/secret.exs") do
+  secret = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
+  signing_salt = :crypto.strong_rand_bytes(8) |> Base.encode64() |> binary_part(0, 8)
+  {web_push_public_key, web_push_private_key} = :crypto.generate_key(:ecdh, :prime256v1)
+
+  secret_file =
+    EEx.eval_string(
+      """
+      import Config
+
+      config :pleroma, Pleroma.Web.Endpoint,
+        secret_key_base: "<%= secret %>",
+        signing_salt: "<%= signing_salt %>"
+
+      config :web_push_encryption, :vapid_details,
+        public_key: "<%= web_push_public_key %>",
+        private_key: "<%= web_push_private_key %>"
+      """,
+      secret: secret,
+      signing_salt: signing_salt,
+      web_push_public_key: Base.url_encode64(web_push_public_key, padding: false),
+      web_push_private_key: Base.url_encode64(web_push_private_key, padding: false)
+    )
+
+  File.write("/var/lib/pleroma/secret.exs", secret_file)
+end
+
+import_config("/var/lib/pleroma/secret.exs")
+
+# For additional user config
+if File.exists?("/var/lib/pleroma/config.exs"),
+  do: import_config("/var/lib/pleroma/config.exs"),
+  else:
+    File.write("/var/lib/pleroma/config.exs", """
+    import Config
+
+    # For additional configuration outside of environmental variables
+    """)

config/emoji.txt

@@ -1,30 +1,2 @@
 firefox, /emoji/Firefox.gif, Gif,Fun
 blank, /emoji/blank.png, Fun
-f_00b, /emoji/f_00b.png
-f_00b11b, /emoji/f_00b11b.png
-f_00b33b, /emoji/f_00b33b.png
-f_00h, /emoji/f_00h.png
-f_00t, /emoji/f_00t.png
-f_01b, /emoji/f_01b.png
-f_03b, /emoji/f_03b.png
-f_10b, /emoji/f_10b.png
-f_11b, /emoji/f_11b.png
-f_11b00b, /emoji/f_11b00b.png
-f_11b22b, /emoji/f_11b22b.png
-f_11h, /emoji/f_11h.png
-f_11t, /emoji/f_11t.png
-f_12b, /emoji/f_12b.png
-f_21b, /emoji/f_21b.png
-f_22b, /emoji/f_22b.png
-f_22b11b, /emoji/f_22b11b.png
-f_22b33b, /emoji/f_22b33b.png
-f_22h, /emoji/f_22h.png
-f_22t, /emoji/f_22t.png
-f_23b, /emoji/f_23b.png
-f_30b, /emoji/f_30b.png
-f_32b, /emoji/f_32b.png
-f_33b, /emoji/f_33b.png
-f_33b00b, /emoji/f_33b00b.png
-f_33b22b, /emoji/f_33b22b.png
-f_33h, /emoji/f_33h.png
-f_33t, /emoji/f_33t.png

config/test.exs

@@ -29,7 +29,8 @@
   email: "admin@example.com",
   notify_email: "noreply@example.com",
   skip_thread_containment: false,
-  federating: false
+  federating: false,
+  external_user_synchronization: false
 
 config :pleroma, :activitypub, sign_object_fetches: false
 
@@ -70,7 +71,8 @@
 config :pleroma, :rate_limit,
   search: [{1000, 30}, {1000, 30}],
   app_account_creation: {10_000, 5},
-  password_reset: {1000, 30}
+  password_reset: {1000, 30},
+  ap_routes: nil
 
 config :pleroma, :http_security, report_uri: "https://endpoint.com"
 
@@ -80,6 +82,8 @@
 config :pleroma, :database, rum_enabled: rum_enabled
 IO.puts("RUM enabled: #{rum_enabled}")
 
+config :joken, default_signer: "yU8uHKq+yyAkZ11Hx//jcdacWc8yQ1bxAAGrplzB0Zwwjkp35v0RK9SO8WTPr6QZ"
+
 config :pleroma, Pleroma.ReverseProxy.Client, Pleroma.ReverseProxy.ClientMock
 
 if File.exists?("./config/test.secret.exs") do

docker-entrypoint.sh

@@ -0,0 +1,14 @@
+#!/bin/ash
+
+set -e
+
+echo "-- Waiting for database..."
+while ! pg_isready -U ${DB_USER:-pleroma} -d postgres://${DB_HOST:-db}:5432/${DB_NAME:-pleroma} -t 1; do
+    sleep 1s
+done
+
+echo "-- Running migrations..."
+$HOME/bin/pleroma_ctl migrate
+
+echo "-- Starting!"
+exec $HOME/bin/pleroma start

docs/api/admin_api.md

@@ -575,6 +575,29 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
     - 404 Not Found `"Not found"`
   - On success: 200 OK `{}`
 
+
+## `/api/pleroma/admin/config/migrate_to_db`
+### Run mix task pleroma.config migrate_to_db
+Copy settings on key `:pleroma` to DB.
+- Method `GET`
+- Params: none
+- Response:
+
+```json
+{}
+```
+
+## `/api/pleroma/admin/config/migrate_from_db`
+### Run mix task pleroma.config migrate_from_db
+Copy all settings from DB to `config/prod.exported_from_db.secret.exs` with deletion from DB.
+- Method `GET`
+- Params: none
+- Response:
+
+```json
+{}
+```
+
 ## `/api/pleroma/admin/config`
 ### List config settings
 List config settings only works with `:pleroma => :instance => :dynamic_configuration` setting to `true`.
@@ -604,6 +627,9 @@ Tuples can be passed as `{"tuple": ["first_val", Pleroma.Module, []]}`.
 Keywords can be passed as lists with 2 child tuples, e.g.
 `[{"tuple": ["first_val", Pleroma.Module]}, {"tuple": ["second_val", true]}]`.
 
+If value contains list of settings `[subkey: val1, subkey2: val2, subkey3: val3]`, it's possible to remove only subkeys instead of all settings passing `subkeys` parameter. E.g.:
+{"group": "pleroma", "key": "some_key", "delete": "true", "subkeys": [":subkey", ":subkey3"]}.
+
 Compile time settings (need instance reboot):
 - all settings by this keys:
   - `:hackney_pools`
@@ -622,6 +648,7 @@ Compile time settings (need instance reboot):
     - `key` (string or string with leading `:` for atoms)
     - `value` (string, [], {} or {"tuple": []})
     - `delete` = true (optional, if parameter must be deleted)
+    - `subkeys` [(string with leading `:` for atoms)] (optional, works only if `delete=true` parameter is passed, otherwise will be ignored)
   ]
 
 - Request (example):

docs/api/differences_in_mastoapi_responses.md

@@ -60,12 +60,19 @@ Has these additional fields under the `pleroma` object:
 - `show_role`: boolean, nullable, true when the user wants his role (e.g admin, moderator) to be shown
 - `no_rich_text` - boolean, nullable, true when html tags are stripped from all statuses requested from the API
 
+## Conversations
+
+Has an additional field under the `pleroma` object:
+
+- `recipients`: The list of the recipients of this Conversation. These will be addressed when replying to this conversation.
+
 ## Account Search
 
 Behavior has changed:
 
 - `/api/v1/accounts/search`: Does not require authentication
 
+
 ## Notifications
 
 Has these additional fields under the `pleroma` object:
@@ -81,6 +88,7 @@ Additional parameters can be added to the JSON body/Form data:
 - `to`: A list of nicknames (like `lain@soykaf.club` or `lain` on the local server) that will be used to determine who is going to be addressed by this post. Using this will disable the implicit addressing by mentioned names in the `status` body, only the people in the `to` list will be addressed. The normal rules for for post visibility are not affected by this and will still apply.
 - `visibility`: string, besides standard MastoAPI values (`direct`, `private`, `unlisted` or `public`) it can be used to address a List by setting it to `list:LIST_ID`.
 - `expires_on`: datetime (iso8601), sets when the posted activity should expire. When a posted activity expires it will be deleted from the server, and a delete request for it will be federated.
+- `in_reply_to_conversation_id`: Will reply to a given conversation, addressing only the people who are part of the recipient set of that conversation. Sets the visibility to `direct`.
 
 ## PATCH `/api/v1/update_credentials`
 

docs/api/pleroma_api.md

@@ -245,6 +245,14 @@ See [Admin-API](Admin-API.md)
 - PATCH `/api/v1/pleroma/accounts/update_banner`: Set/clear user banner image
 - PATCH `/api/v1/pleroma/accounts/update_background`: Set/clear user background image
 
+## `/api/v1/pleroma/accounts/confirmation_resend`
+### Resend confirmation email
+* Method `POST`
+* Params:
+    * `email`: email of that needs to be verified
+* Authentication: not required
+* Response: 204 No Content 
+
 ## `/api/v1/pleroma/mascot`
 ### Gets user mascot image
 * Method `GET`
@@ -311,3 +319,38 @@ See [Admin-API](Admin-API.md)
   "healthy": true # Instance state
 }
 ```
+
+# Pleroma Conversations
+
+Pleroma Conversations have the same general structure that Mastodon Conversations have. The behavior differs in the following ways when using these endpoints:
+
+1. Pleroma Conversations never add or remove recipients, unless explicitly changed by the user. 
+2. Pleroma Conversations statuses can be requested by Conversation id.
+3. Pleroma Conversations can be replied to.
+
+Conversations have the additional field "recipients" under the "pleroma" key. This holds a list of all the accounts that will receive a message in this conversation.
+
+The status posting endpoint takes an additional parameter, `in_reply_to_conversation_id`, which, when set, will set the visiblity to direct and address only the people who are the recipients of that Conversation.
+
+
+## `GET /api/v1/pleroma/conversations/:id/statuses`
+### Timeline for a given conversation
+* Method `GET`
+* Authentication: required
+* Params: Like other timelines
+* Response: JSON, statuses (200 - healthy, 503 unhealthy).
+
+## `GET /api/v1/pleroma/conversations/:id`
+### The conversation with the given ID.
+* Method `GET`
+* Authentication: required
+* Params: None
+* Response: JSON, statuses (200 - healthy, 503 unhealthy).
+
+## `PATCH /api/v1/pleroma/conversations/:id`
+### Update a conversation. Used to change the set of recipients.
+* Method `PATCH`
+* Authentication: required
+* Params:
+    * `recipients`: A list of ids of users that should receive posts to this conversation. This will replace the current list of recipients, so submit the full list. The owner of owner of the conversation will always be part of the set of recipients, though.
+* Response: JSON, statuses (200 - healthy, 503 unhealthy)

docs/config.md

@@ -18,6 +18,7 @@ Note: `strip_exif` has been replaced by `Pleroma.Upload.Filter.Mogrify`.
 
 ## Pleroma.Uploaders.S3
 * `bucket`: S3 bucket name
+* `bucket_namespace`: S3 bucket namespace
 * `public_endpoint`: S3 endpoint that the user finally accesses(ex. "https://s3.dualstack.ap-northeast-1.amazonaws.com")
 * `truncated_namespace`: If you use S3 compatible service such as Digital Ocean Spaces or CDN, set folder name or "" etc.
 For example, when using CDN to S3 virtual host format, set "".
@@ -25,7 +26,7 @@ At this time, write CNAME to CDN in public_endpoint.
 
 ## Pleroma.Upload.Filter.Mogrify
 
-* `args`: List of actions for the `mogrify` command like `"strip"` or `["strip", "auto-orient", {"impode", "1"}]`.
+* `args`: List of actions for the `mogrify` command like `"strip"` or `["strip", "auto-orient", {"implode", "1"}]`.
 
 ## Pleroma.Upload.Filter.Dedupe
 
@@ -102,6 +103,7 @@ config :pleroma, Pleroma.Emails.Mailer,
   * `Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy`: Rejects posts from likely spambots by rejecting posts from new users that contain links.
   * `Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`: Crawls attachments using their MediaProxy URLs so that the MediaProxy cache is primed.
   * `Pleroma.Web.ActivityPub.MRF.MentionPolicy`: Drops posts mentioning configurable users. (see `:mrf_mention` section)
+  * `Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`: Restricts activities to a configured set of vocabulary. (see `:mrf_vocabulary` section)
 * `public`: Makes the client API in authentificated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
 * `quarantined_instances`: List of ActivityPub instances where private(DMs, followers-only) activities will not be send.
 * `managed_config`: Whenether the config for pleroma-fe is configured in this config or in ``static/config.json``
@@ -125,9 +127,15 @@ config :pleroma, Pleroma.Emails.Mailer,
 * `safe_dm_mentions`: If set to true, only mentions at the beginning of a post will be used to address people in direct messages. This is to prevent accidental mentioning of people when talking about them (e.g. "@friend hey i really don't like @enemy"). Default: `false`.
 * `healthcheck`: If set to true, system data will be shown on ``/api/pleroma/healthcheck``.
 * `remote_post_retention_days`: The default amount of days to retain remote posts when pruning the database.
+* `user_bio_length`: A user bio maximum length (default: `5000`)
+* `user_name_length`: A user name maximum length (default: `100`)
 * `skip_thread_containment`: Skip filter out broken threads. The default is `false`.
 * `limit_to_local_content`: Limit unauthenticated users to search for local statutes and users only. Possible values: `:unauthenticated`, `:all` and `false`. The default is `:unauthenticated`.
 * `dynamic_configuration`: Allow transferring configuration to DB with the subsequent customization from Admin api.
+* `max_account_fields`: The maximum number of custom fields in the user profile (default: `10`)
+* `max_remote_account_fields`: The maximum number of custom fields in the remote user profile (default: `20`)
+* `account_field_name_length`: An account field name maximum length (default: `512`)
+* `account_field_value_length`: An account field value maximum length (default: `512`)
 * `external_user_synchronization`: Enabling following/followers counters synchronization for external users.
 
 
@@ -275,6 +283,10 @@ config :pleroma, :mrf_subchain,
 ## :mrf_mention
 * `actors`: A list of actors, for which to drop any posts mentioning.
 
+## :mrf_vocabulary
+* `accept`: A list of ActivityStreams terms to accept.  If empty, all supported messages are accepted.
+* `reject`: A list of ActivityStreams terms to reject.  If empty, no messages are rejected.
+
 ## :media_proxy
 * `enabled`: Enables proxying of remote media to the instance’s proxy
 * `base_url`: The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host/CDN fronts.
@@ -328,7 +340,6 @@ config :pleroma, Pleroma.Web.Endpoint,
 This will make Pleroma listen on `127.0.0.1` port `8080` and generate urls starting with `https://example.com:2020`
 
 ## :activitypub
-* ``accept_blocks``: Whether to accept incoming block activities from other instances
 * ``unfollow_blocked``: Whether blocks result in people getting unfollowed
 * ``outgoing_blocks``: Whether to federate blocks to other instances
 * ``deny_follow_blocked``: Whether to disallow following an account that has blocked the user in question
@@ -540,6 +551,23 @@ Authentication / authorization settings.
 * `oauth_consumer_template`: OAuth consumer mode authentication form template. By default it's `consumer.html` which corresponds to `lib/pleroma/web/templates/o_auth/o_auth/consumer.html.eex`.
 * `oauth_consumer_strategies`: the list of enabled OAuth consumer strategies; by default it's set by `OAUTH_CONSUMER_STRATEGIES` environment variable. Each entry in this space-delimited string should be of format `<strategy>` or `<strategy>:<dependency>` (e.g. `twitter` or `keycloak:ueberauth_keycloak_strategy` in case dependency is named differently than `ueberauth_<strategy>`).
 
+## :email_notifications
+
+Email notifications settings.
+
+  - digest - emails of "what you've missed" for users who have been
+    inactive for a while.
+    - active: globally enable or disable digest emails
+    - schedule: When to send digest email, in [crontab format](https://en.wikipedia.org/wiki/Cron).
+      "0 0 * * 0" is the default, meaning "once a week at midnight on Sunday morning"
+    - interval: Minimum interval between digest emails to one user
+    - inactivity_threshold: Minimum user inactivity threshold
+
+## Pleroma.Emails.UserEmail
+
+- `:logo` - a path to a custom logo. Set it to `nil` to use the default Pleroma logo.
+- `:styling` - a map with color settings for email templates.
+
 ## OAuth consumer mode
 
 OAuth consumer mode allows sign in / sign up via external OAuth providers (e.g. Twitter, Facebook, Google, Microsoft, etc.).

docs/config/howto_mediaproxy.md

@@ -1,8 +1,8 @@
 # How to activate mediaproxy
 ## Explanation
 
-Without the `mediaproxy` function, Pleroma don't store any remote content like pictures, video etc. locally. So every time you open Pleroma, the content is loaded from the source server, from where the post is coming. This can result in slowly loading content or/and increased bandwidth usage on the source server.
+Without the `mediaproxy` function, Pleroma doesn't store any remote content like pictures, video etc. locally. So every time you open Pleroma, the content is loaded from the source server, from where the post is coming. This can result in slowly loading content or/and increased bandwidth usage on the source server.
-With the `mediaproxy` function you can use the cache ability of nginx, to cache these content, so user can access it faster, cause it's loaded from your server.
+With the `mediaproxy` function you can use nginx to cache this content, so users can access it faster, because it's loaded from your server.
 
 ## Activate it
 

lib/mix/tasks/pleroma/benchmark.ex

@@ -26,4 +26,48 @@ def run(["tag"]) do
       end
     })
   end
+
+  def run(["render_timeline", nickname]) do
+    start_pleroma()
+    user = Pleroma.User.get_by_nickname(nickname)
+
+    activities =
+      %{}
+      |> Map.put("type", ["Create", "Announce"])
+      |> Map.put("blocking_user", user)
+      |> Map.put("muting_user", user)
+      |> Map.put("user", user)
+      |> Map.put("limit", 80)
+      |> Pleroma.Web.ActivityPub.ActivityPub.fetch_public_activities()
+      |> Enum.reverse()
+
+    inputs = %{
+      "One activity" => Enum.take_random(activities, 1),
+      "Ten activities" => Enum.take_random(activities, 10),
+      "Twenty activities" => Enum.take_random(activities, 20),
+      "Forty activities" => Enum.take_random(activities, 40),
+      "Eighty activities" => Enum.take_random(activities, 80)
+    }
+
+    Benchee.run(
+      %{
+        "Parallel rendering" => fn activities ->
+          Pleroma.Web.MastodonAPI.StatusView.render("index.json", %{
+            activities: activities,
+            for: user,
+            as: :activity
+          })
+        end,
+        "Standart rendering" => fn activities ->
+          Pleroma.Web.MastodonAPI.StatusView.render("index.json", %{
+            activities: activities,
+            for: user,
+            as: :activity,
+            parallel: false
+          })
+        end
+      },
+      inputs: inputs
+    )
+  end
 end

lib/mix/tasks/pleroma/config.ex

@@ -15,7 +15,7 @@ defmodule Mix.Tasks.Pleroma.Config do
 
       mix pleroma.config migrate_to_db
 
-  ## Transfers config from DB to file.
+  ## Transfers config from DB to file `config/env.exported_from_db.secret.exs`
 
       mix pleroma.config migrate_from_db ENV
   """

lib/mix/tasks/pleroma/database.ex

@@ -8,6 +8,7 @@ defmodule Mix.Tasks.Pleroma.Database do
   alias Pleroma.Repo
   alias Pleroma.User
   require Logger
+  require Pleroma.Constants
   import Mix.Pleroma
   use Mix.Task
 
@@ -35,6 +36,10 @@ defmodule Mix.Tasks.Pleroma.Database do
   ## Remove duplicated items from following and update followers count for all users
 
       mix pleroma.database update_users_following_followers_counts
+
+  ## Fix the pre-existing "likes" collections for all objects
+
+      mix pleroma.database fix_likes_collections
   """
   def run(["remove_embedded_objects" | args]) do
     {options, [], []} =
@@ -99,10 +104,15 @@ def run(["prune_objects" | args]) do
       NaiveDateTime.utc_now()
       |> NaiveDateTime.add(-(deadline * 86_400))
 
-    public = "https://www.w3.org/ns/activitystreams#Public"
-
     from(o in Object,
-      where: fragment("?->'to' \\? ? OR ?->'cc' \\? ?", o.data, ^public, o.data, ^public),
+      where:
+        fragment(
+          "?->'to' \\? ? OR ?->'cc' \\? ?",
+          o.data,
+          ^Pleroma.Constants.as_public(),
+          o.data,
+          ^Pleroma.Constants.as_public()
+        ),
       where: o.inserted_at < ^time_deadline,
       where:
         fragment("split_part(?->>'actor', '/', 3) != ?", o.data, ^Pleroma.Web.Endpoint.host())
@@ -119,4 +129,36 @@ def run(["prune_objects" | args]) do
       )
     end
   end
+
+  def run(["fix_likes_collections"]) do
+    import Ecto.Query
+
+    start_pleroma()
+
+    from(object in Object,
+      where: fragment("(?)->>'likes' is not null", object.data),
+      select: %{id: object.id, likes: fragment("(?)->>'likes'", object.data)}
+    )
+    |> Pleroma.RepoStreamer.chunk_stream(100)
+    |> Stream.each(fn objects ->
+      ids =
+        objects
+        |> Enum.filter(fn object -> object.likes |> Jason.decode!() |> is_map() end)
+        |> Enum.map(& &1.id)
+
+      Object
+      |> where([object], object.id in ^ids)
+      |> update([object],
+        set: [
+          data:
+            fragment(
+              "jsonb_set(?, '{likes}', '[]'::jsonb, true)",
+              object.data
+            )
+        ]
+      )
+      |> Repo.update_all([], timeout: :infinity)
+    end)
+    |> Stream.run()
+  end
 end

lib/mix/tasks/pleroma/digest.ex

@@ -0,0 +1,41 @@
+defmodule Mix.Tasks.Pleroma.Digest do
+  use Mix.Task
+
+  @shortdoc "Manages digest emails"
+  @moduledoc """
+  Manages digest emails
+
+  ## Send digest email since given date (user registration date by default)
+  ignoring user activity status.
+
+  ``mix pleroma.digest test <nickname> <since_date>``
+
+  Example: ``mix pleroma.digest test donaldtheduck 2019-05-20``
+  """
+  def run(["test", nickname | opts]) do
+    Mix.Pleroma.start_pleroma()
+
+    user = Pleroma.User.get_by_nickname(nickname)
+
+    last_digest_emailed_at =
+      with [date] <- opts,
+           {:ok, datetime} <- Timex.parse(date, "{YYYY}-{0M}-{0D}") do
+        datetime
+      else
+        _ -> user.inserted_at
+      end
+
+    patched_user = %{user | last_digest_emailed_at: last_digest_emailed_at}
+
+    with %Swoosh.Email{} = email <- Pleroma.Emails.UserEmail.digest_email(patched_user) do
+      {:ok, _} = Pleroma.Emails.Mailer.deliver(email)
+
+      Mix.shell().info("Digest email have been sent to #{nickname} (#{user.email})")
+    else
+      _ ->
+        Mix.shell().info(
+          "Cound't find any mentions for #{nickname} since #{last_digest_emailed_at}"
+        )
+    end
+  end
+end

lib/mix/tasks/pleroma/instance.ex

@@ -183,6 +183,7 @@ def run(["gen" | rest]) do
         )
 
       secret = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
+      jwt_secret = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
       signing_salt = :crypto.strong_rand_bytes(8) |> Base.encode64() |> binary_part(0, 8)
       {web_push_public_key, web_push_private_key} = :crypto.generate_key(:ecdh, :prime256v1)
       template_dir = Application.app_dir(:pleroma, "priv") <> "/templates"
@@ -200,6 +201,7 @@ def run(["gen" | rest]) do
           dbuser: dbuser,
           dbpass: dbpass,
           secret: secret,
+          jwt_secret: jwt_secret,
           signing_salt: signing_salt,
           web_push_public_key: Base.url_encode64(web_push_public_key, padding: false),
           web_push_private_key: Base.url_encode64(web_push_private_key, padding: false),

lib/mix/tasks/pleroma/relay.ex

@@ -5,6 +5,7 @@
 defmodule Mix.Tasks.Pleroma.Relay do
   use Mix.Task
   import Mix.Pleroma
+  alias Pleroma.User
   alias Pleroma.Web.ActivityPub.Relay
 
   @shortdoc "Manages remote relays"
@@ -22,6 +23,10 @@ defmodule Mix.Tasks.Pleroma.Relay do
   ``mix pleroma.relay unfollow <relay_url>``
 
   Example: ``mix pleroma.relay unfollow https://example.org/relay``
+
+  ## List relay subscriptions
+
+  ``mix pleroma.relay list``
   """
   def run(["follow", target]) do
     start_pleroma()
@@ -44,4 +49,17 @@ def run(["unfollow", target]) do
       {:error, e} -> shell_error("Error while following #{target}: #{inspect(e)}")
     end
   end
+
+  def run(["list"]) do
+    start_pleroma()
+
+    with %User{following: following} = _user <- Relay.get_actor() do
+      following
+      |> Enum.map(fn entry -> URI.parse(entry).host end)
+      |> Enum.uniq()
+      |> Enum.each(&shell_info(&1))
+    else
+      e -> shell_error("Error while fetching relay subscription list: #{inspect(e)}")
+    end
+  end
 end

lib/mix/tasks/pleroma/user.ex

@@ -31,8 +31,8 @@ defmodule Mix.Tasks.Pleroma.User do
       mix pleroma.user invite [OPTION...]
 
     Options:
-    - `--expires_at DATE` - last day on which token is active (e.g. "2019-04-05")
+    - `--expires-at DATE` - last day on which token is active (e.g. "2019-04-05")
-    - `--max_use NUMBER` - maximum numbers of token uses
+    - `--max-use NUMBER` - maximum numbers of token uses
 
   ## List generated invites
 

lib/pleroma/activity.ex

@@ -99,6 +99,7 @@ def with_set_thread_muted_field(query, %User{} = user) do
     from([a] in query,
       left_join: tm in ThreadMute,
       on: tm.user_id == ^user.id and tm.context == fragment("?->>'context'", a.data),
+      as: :thread_mute,
       select: %Activity{a | thread_muted?: not is_nil(tm.id)}
     )
   end
@@ -227,6 +228,29 @@ def get_create_by_object_ap_id(ap_id) when is_binary(ap_id) do
 
   def get_create_by_object_ap_id(_), do: nil
 
+  def create_by_object_ap_id_with_object(ap_ids) when is_list(ap_ids) do
+    from(
+      activity in Activity,
+      where:
+        fragment(
+          "coalesce((?)->'object'->>'id', (?)->>'object') = ANY(?)",
+          activity.data,
+          activity.data,
+          ^ap_ids
+        ),
+      where: fragment("(?)->>'type' = 'Create'", activity.data),
+      inner_join: o in Object,
+      on:
+        fragment(
+          "(?->>'id') = COALESCE(?->'object'->>'id', ?->>'object')",
+          o.data,
+          activity.data,
+          activity.data
+        ),
+      preload: [object: o]
+    )
+  end
+
   def create_by_object_ap_id_with_object(ap_id) when is_binary(ap_id) do
     from(
       activity in Activity,
@@ -266,8 +290,8 @@ defp get_in_reply_to_activity_from_object(%Object{data: %{"inReplyTo" => ap_id}}
 
   defp get_in_reply_to_activity_from_object(_), do: nil
 
-  def get_in_reply_to_activity(%Activity{data: %{"object" => object}}) do
+  def get_in_reply_to_activity(%Activity{} = activity) do
-    get_in_reply_to_activity_from_object(Object.normalize(object))
+    get_in_reply_to_activity_from_object(Object.normalize(activity))
   end
 
   def normalize(obj) when is_map(obj), do: get_by_ap_id_with_object(obj["id"])

lib/pleroma/activity/search.ex

@@ -9,6 +9,8 @@ defmodule Pleroma.Activity.Search do
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.Visibility
 
+  require Pleroma.Constants
+
   import Ecto.Query
 
   def search(user, search_query, options \\ []) do
@@ -39,7 +41,7 @@ def maybe_restrict_author(query, _), do: query
   defp restrict_public(q) do
     from([a, o] in q,
       where: fragment("?->>'type' = 'Create'", a.data),
-      where: "https://www.w3.org/ns/activitystreams#Public" in a.recipients
+      where: ^Pleroma.Constants.as_public() in a.recipients
     )
   end
 

lib/pleroma/application.ex

@@ -3,11 +3,14 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Application do
+  import Cachex.Spec
   use Application
 
   @name Mix.Project.config()[:name]
   @version Mix.Project.config()[:version]
   @repository Mix.Project.config()[:source_url]
+  @env Mix.env()
+
   def name, do: @name
   def version, do: @version
   def named_version, do: @name <> " " <> @version
@@ -21,120 +24,25 @@ def user_agent do
   # See http://elixir-lang.org/docs/stable/elixir/Application.html
   # for more information on OTP Applications
   def start(_type, _args) do
-    import Cachex.Spec
-
     Pleroma.Config.DeprecationWarnings.warn()
     setup_instrumenters()
 
     # Define workers and child supervisors to be supervised
     children =
       [
-        # Start the Ecto repository
+        Pleroma.Repo,
-        %{id: Pleroma.Repo, start: {Pleroma.Repo, :start_link, []}, type: :supervisor},
+        Pleroma.Config.TransferTask,
-        %{id: Pleroma.Config.TransferTask, start: {Pleroma.Config.TransferTask, :start_link, []}},
+        Pleroma.Emoji,
-        %{id: Pleroma.Emoji, start: {Pleroma.Emoji, :start_link, []}},
+        Pleroma.Captcha,
-        %{id: Pleroma.Captcha, start: {Pleroma.Captcha, :start_link, []}},
+        Pleroma.FlakeId,
-        %{
+        Pleroma.ScheduledActivityWorker,
-          id: :cachex_used_captcha_cache,
+        Pleroma.ActiviyExpirationWorker
-          start:
-            {Cachex, :start_link,
-             [
-               :used_captcha_cache,
-               [
-                 ttl_interval:
-                   :timer.seconds(Pleroma.Config.get!([Pleroma.Captcha, :seconds_valid]))
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_user,
-          start:
-            {Cachex, :start_link,
-             [
-               :user_cache,
-               [
-                 default_ttl: 25_000,
-                 ttl_interval: 1000,
-                 limit: 2500
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_object,
-          start:
-            {Cachex, :start_link,
-             [
-               :object_cache,
-               [
-                 default_ttl: 25_000,
-                 ttl_interval: 1000,
-                 limit: 2500
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_rich_media,
-          start:
-            {Cachex, :start_link,
-             [
-               :rich_media_cache,
-               [
-                 default_ttl: :timer.minutes(120),
-                 limit: 5000
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_scrubber,
-          start:
-            {Cachex, :start_link,
-             [
-               :scrubber_cache,
-               [
-                 limit: 2500
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_idem,
-          start:
-            {Cachex, :start_link,
-             [
-               :idempotency_cache,
-               [
-                 expiration:
-                   expiration(
-                     default: :timer.seconds(6 * 60 * 60),
-                     interval: :timer.seconds(60)
-                   ),
-                 limit: 2500
-               ]
-             ]}
-        },
-        %{id: Pleroma.FlakeId, start: {Pleroma.FlakeId, :start_link, []}},
-        %{
-          id: Pleroma.ScheduledActivityWorker,
-          start: {Pleroma.ScheduledActivityWorker, :start_link, []}
-        },
-        %{
-          id: Pleroma.ActivityExpirationWorker,
-          start: {Pleroma.ActivityExpirationWorker, :start_link, []}
-        }
       ] ++
+        cachex_children() ++
         hackney_pool_children() ++
         [
-          %{
+          Pleroma.Web.Federator.RetryQueue,
-            id: Pleroma.Web.Federator.RetryQueue,
+          Pleroma.Stats,
-            start: {Pleroma.Web.Federator.RetryQueue, :start_link, []}
-          },
-          %{
-            id: Pleroma.Web.OAuth.Token.CleanWorker,
-            start: {Pleroma.Web.OAuth.Token.CleanWorker, :start_link, []}
-          },
-          %{
-            id: Pleroma.Stats,
-            start: {Pleroma.Stats, :start_link, []}
-          },
           %{
             id: :web_push_init,
             start: {Task, :start_link, [&Pleroma.Web.Push.init/0]},
@@ -151,22 +59,20 @@ def start(_type, _args) do
             restart: :temporary
           }
         ] ++
-        streamer_child() ++
+        oauth_cleanup_child(oauth_cleanup_enabled?()) ++
-        chat_child() ++
+        streamer_child(@env) ++
+        chat_child(@env, chat_enabled?()) ++
         [
-          # Start the endpoint when the application starts
+          Pleroma.Web.Endpoint,
-          %{
+          Pleroma.Gopher.Server
-            id: Pleroma.Web.Endpoint,
-            start: {Pleroma.Web.Endpoint, :start_link, []},
-            type: :supervisor
-          },
-          %{id: Pleroma.Gopher.Server, start: {Pleroma.Gopher.Server, :start_link, []}}
         ]
 
     # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html
     # for other strategies and supported options
     opts = [strategy: :one_for_one, name: Pleroma.Supervisor]
-    Supervisor.start_link(children, opts)
+    result = Supervisor.start_link(children, opts)
+    :ok = after_supervisor_start()
+    result
   end
 
   defp setup_instrumenters do
@@ -203,32 +109,71 @@ def enabled_hackney_pools do
       end
   end
 
-  if Pleroma.Config.get(:env) == :test do
+  defp cachex_children do
-    defp streamer_child, do: []
+    [
-    defp chat_child, do: []
+      build_cachex("used_captcha", ttl_interval: seconds_valid_interval()),
-  else
+      build_cachex("user", default_ttl: 25_000, ttl_interval: 1000, limit: 2500),
-    defp streamer_child do
+      build_cachex("object", default_ttl: 25_000, ttl_interval: 1000, limit: 2500),
-      [%{id: Pleroma.Web.Streamer, start: {Pleroma.Web.Streamer, :start_link, []}}]
+      build_cachex("rich_media", default_ttl: :timer.minutes(120), limit: 5000),
-    end
+      build_cachex("scrubber", limit: 2500),
-
+      build_cachex("idempotency", expiration: idempotency_expiration(), limit: 2500)
-    defp chat_child do
+    ]
-      if Pleroma.Config.get([:chat, :enabled]) do
-        [
-          %{
-            id: Pleroma.Web.ChatChannel.ChatChannelState,
-            start: {Pleroma.Web.ChatChannel.ChatChannelState, :start_link, []}
-          }
-        ]
-      else
-        []
-      end
-    end
   end
 
+  defp idempotency_expiration,
+    do: expiration(default: :timer.seconds(6 * 60 * 60), interval: :timer.seconds(60))
+
+  defp seconds_valid_interval,
+    do: :timer.seconds(Pleroma.Config.get!([Pleroma.Captcha, :seconds_valid]))
+
+  defp build_cachex(type, opts),
+    do: %{
+      id: String.to_atom("cachex_" <> type),
+      start: {Cachex, :start_link, [String.to_atom(type <> "_cache"), opts]},
+      type: :worker
+    }
+
+  defp chat_enabled?, do: Pleroma.Config.get([:chat, :enabled])
+
+  defp oauth_cleanup_enabled?,
+    do: Pleroma.Config.get([:oauth2, :clean_expired_tokens], false)
+
+  defp streamer_child(:test), do: []
+
+  defp streamer_child(_) do
+    [Pleroma.Web.Streamer]
+  end
+
+  defp oauth_cleanup_child(true),
+    do: [Pleroma.Web.OAuth.Token.CleanWorker]
+
+  defp oauth_cleanup_child(_), do: []
+
+  defp chat_child(:test, _), do: []
+
+  defp chat_child(_env, true) do
+    [Pleroma.Web.ChatChannel.ChatChannelState]
+  end
+
+  defp chat_child(_, _), do: []
+
   defp hackney_pool_children do
     for pool <- enabled_hackney_pools() do
       options = Pleroma.Config.get([:hackney_pools, pool])
       :hackney_pool.child_spec(pool, options)
     end
   end
+
+  defp after_supervisor_start do
+    with digest_config <- Application.get_env(:pleroma, :email_notifications)[:digest],
+         true <- digest_config[:active] do
+      PleromaJobQueue.schedule(
+        digest_config[:schedule],
+        :digest_emails,
+        Pleroma.DigestEmailWorker
+      )
+    end
+
+    :ok
+  end
 end

lib/pleroma/captcha/captcha.ex

@@ -12,7 +12,7 @@ defmodule Pleroma.Captcha do
   use GenServer
 
   @doc false
-  def start_link do
+  def start_link(_) do
     GenServer.start_link(__MODULE__, [], name: __MODULE__)
   end
 

lib/pleroma/config/transfer_task.ex

@@ -6,7 +6,7 @@ defmodule Pleroma.Config.TransferTask do
   use Task
   alias Pleroma.Web.AdminAPI.Config
 
-  def start_link do
+  def start_link(_) do
     load_and_update_env()
     if Pleroma.Config.get(:env) == :test, do: Ecto.Adapters.SQL.Sandbox.checkin(Pleroma.Repo)
     :ignore

lib/pleroma/constants.ex

@@ -0,0 +1,9 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Constants do
+  use Const
+
+  const(as_public, do: "https://www.w3.org/ns/activitystreams#Public")
+end

lib/pleroma/conversation.ex

@@ -4,6 +4,7 @@
 
 defmodule Pleroma.Conversation do
   alias Pleroma.Conversation.Participation
+  alias Pleroma.Conversation.Participation.RecipientShip
   alias Pleroma.Repo
   alias Pleroma.User
   use Ecto.Schema
@@ -39,6 +40,15 @@ def get_for_ap_id(ap_id) do
     Repo.get_by(__MODULE__, ap_id: ap_id)
   end
 
+  def maybe_create_recipientships(participation, activity) do
+    participation = Repo.preload(participation, :recipients)
+
+    if participation.recipients |> Enum.empty?() do
+      recipients = User.get_all_by_ap_id(activity.recipients)
+      RecipientShip.create(recipients, participation)
+    end
+  end
+
   @doc """
   This will
   1. Create a conversation if there isn't one already
@@ -60,6 +70,7 @@ def create_or_bump_for(activity, opts \\ []) do
           {:ok, participation} =
             Participation.create_for_user_and_conversation(user, conversation, opts)
 
+          maybe_create_recipientships(participation, activity)
           participation
         end)
 

lib/pleroma/conversation/participation.ex

@@ -5,6 +5,7 @@
 defmodule Pleroma.Conversation.Participation do
   use Ecto.Schema
   alias Pleroma.Conversation
+  alias Pleroma.Conversation.Participation.RecipientShip
   alias Pleroma.Repo
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ActivityPub
@@ -17,6 +18,9 @@ defmodule Pleroma.Conversation.Participation do
     field(:read, :boolean, default: false)
     field(:last_activity_id, Pleroma.FlakeId, virtual: true)
 
+    has_many(:recipient_ships, RecipientShip)
+    has_many(:recipients, through: [:recipient_ships, :user])
+
     timestamps()
   end
 
@@ -65,6 +69,14 @@ def for_user(user, params \\ %{}) do
     |> Pleroma.Pagination.fetch_paginated(params)
   end
 
+  def for_user_and_conversation(user, conversation) do
+    from(p in __MODULE__,
+      where: p.user_id == ^user.id,
+      where: p.conversation_id == ^conversation.id
+    )
+    |> Repo.one()
+  end
+
   def for_user_with_last_activity_id(user, params \\ %{}) do
     for_user(user, params)
     |> Enum.map(fn participation ->
@@ -81,4 +93,46 @@ def for_user_with_last_activity_id(user, params \\ %{}) do
     end)
     |> Enum.filter(& &1.last_activity_id)
   end
+
+  def get(_, _ \\ [])
+  def get(nil, _), do: nil
+
+  def get(id, params) do
+    query =
+      if preload = params[:preload] do
+        from(p in __MODULE__,
+          preload: ^preload
+        )
+      else
+        __MODULE__
+      end
+
+    Repo.get(query, id)
+  end
+
+  def set_recipients(participation, user_ids) do
+    user_ids =
+      [participation.user_id | user_ids]
+      |> Enum.uniq()
+
+    Repo.transaction(fn ->
+      query =
+        from(r in RecipientShip,
+          where: r.participation_id == ^participation.id
+        )
+
+      Repo.delete_all(query)
+
+      users =
+        from(u in User,
+          where: u.id in ^user_ids
+        )
+        |> Repo.all()
+
+      RecipientShip.create(users, participation)
+      :ok
+    end)
+
+    {:ok, Repo.preload(participation, :recipients, force: true)}
+  end
 end

lib/pleroma/conversation/participation_recipient_ship.ex

@@ -0,0 +1,34 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Conversation.Participation.RecipientShip do
+  use Ecto.Schema
+
+  alias Pleroma.Conversation.Participation
+  alias Pleroma.Repo
+  alias Pleroma.User
+
+  import Ecto.Changeset
+
+  schema "conversation_participation_recipient_ships" do
+    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:participation, Participation)
+  end
+
+  def creation_cng(struct, params) do
+    struct
+    |> cast(params, [:user_id, :participation_id])
+    |> validate_required([:user_id, :participation_id])
+  end
+
+  def create(%User{} = user, participation), do: create([user], participation)
+
+  def create(users, participation) do
+    Enum.each(users, fn user ->
+      %__MODULE__{}
+      |> creation_cng(%{user_id: user.id, participation_id: participation.id})
+      |> Repo.insert!()
+    end)
+  end
+end

lib/pleroma/digest_email_worker.ex

@@ -0,0 +1,39 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.DigestEmailWorker do
+  import Ecto.Query
+
+  @queue_name :digest_emails
+
+  def perform do
+    config = Pleroma.Config.get([:email_notifications, :digest])
+    negative_interval = -Map.fetch!(config, :interval)
+    inactivity_threshold = Map.fetch!(config, :inactivity_threshold)
+    inactive_users_query = Pleroma.User.list_inactive_users_query(inactivity_threshold)
+
+    now = NaiveDateTime.truncate(NaiveDateTime.utc_now(), :second)
+
+    from(u in inactive_users_query,
+      where: fragment(~s(? #> '{"email_notifications","digest"}' @> 'true'), u.info),
+      where: u.last_digest_emailed_at < datetime_add(^now, ^negative_interval, "day"),
+      select: u
+    )
+    |> Pleroma.Repo.all()
+    |> Enum.each(&PleromaJobQueue.enqueue(@queue_name, __MODULE__, [&1]))
+  end
+
+  @doc """
+  Send digest email to the given user.
+  Updates `last_digest_emailed_at` field for the user and returns the updated user.
+  """
+  @spec perform(Pleroma.User.t()) :: Pleroma.User.t()
+  def perform(user) do
+    with %Swoosh.Email{} = email <- Pleroma.Emails.UserEmail.digest_email(user) do
+      Pleroma.Emails.Mailer.deliver_async(email)
+    end
+
+    Pleroma.User.touch_last_digest_emailed_at(user)
+  end
+end

lib/pleroma/emails/admin_email.ex

@@ -63,7 +63,6 @@ def report(to, reporter, account, statuses, comment) do
     new()
     |> to({to.name, to.email})
     |> from({instance_name(), instance_notify_email()})
-    |> reply_to({reporter.name, reporter.email})
     |> subject("#{instance_name()} Report")
     |> html_body(html_body)
   end

lib/pleroma/emails/user_email.ex

@@ -5,23 +5,23 @@
 defmodule Pleroma.Emails.UserEmail do
   @moduledoc "User emails"
 
-  import Swoosh.Email
+  use Phoenix.Swoosh, view: Pleroma.Web.EmailView, layout: {Pleroma.Web.LayoutView, :email}
 
+  alias Pleroma.Config
+  alias Pleroma.User
   alias Pleroma.Web.Endpoint
   alias Pleroma.Web.Router
 
-  defp instance_config, do: Pleroma.Config.get(:instance)
+  defp instance_name, do: Config.get([:instance, :name])
-
-  defp instance_name, do: instance_config()[:name]
 
   defp sender do
-    email = Keyword.get(instance_config(), :notify_email, instance_config()[:email])
+    email = Config.get([:instance, :notify_email]) || Config.get([:instance, :email])
     {instance_name(), email}
   end
 
   defp recipient(email, nil), do: email
   defp recipient(email, name), do: {name, email}
-  defp recipient(%Pleroma.User{} = user), do: recipient(user.email, user.name)
+  defp recipient(%User{} = user), do: recipient(user.email, user.name)
 
   def password_reset_email(user, token) when is_binary(token) do
     password_reset_url = Router.Helpers.reset_password_url(Endpoint, :reset, token)
@@ -87,4 +87,92 @@ def account_confirmation_email(user) do
     |> subject("#{instance_name()} account confirmation")
     |> html_body(html_body)
   end
+
+  @doc """
+  Email used in digest email notifications
+  Includes Mentions and New Followers data
+  If there are no mentions (even when new followers exist), the function will return nil
+  """
+  @spec digest_email(User.t()) :: Swoosh.Email.t() | nil
+  def digest_email(user) do
+    notifications = Pleroma.Notification.for_user_since(user, user.last_digest_emailed_at)
+
+    mentions =
+      notifications
+      |> Enum.filter(&(&1.activity.data["type"] == "Create"))
+      |> Enum.map(fn notification ->
+        object = Pleroma.Object.normalize(notification.activity)
+        object = update_in(object.data["content"], &format_links/1)
+
+        %{
+          data: notification,
+          object: object,
+          from: User.get_by_ap_id(notification.activity.actor)
+        }
+      end)
+
+    followers =
+      notifications
+      |> Enum.filter(&(&1.activity.data["type"] == "Follow"))
+      |> Enum.map(fn notification ->
+        %{
+          data: notification,
+          object: Pleroma.Object.normalize(notification.activity),
+          from: User.get_by_ap_id(notification.activity.actor)
+        }
+      end)
+
+    unless Enum.empty?(mentions) do
+      styling = Config.get([__MODULE__, :styling])
+      logo = Config.get([__MODULE__, :logo])
+
+      html_data = %{
+        instance: instance_name(),
+        user: user,
+        mentions: mentions,
+        followers: followers,
+        unsubscribe_link: unsubscribe_url(user, "digest"),
+        styling: styling
+      }
+
+      logo_path =
+        if is_nil(logo) do
+          Path.join(:code.priv_dir(:pleroma), "static/static/logo.png")
+        else
+          Path.join(Config.get([:instance, :static_dir]), logo)
+        end
+
+      new()
+      |> to(recipient(user))
+      |> from(sender())
+      |> subject("Your digest from #{instance_name()}")
+      |> put_layout(false)
+      |> render_body("digest.html", html_data)
+      |> attachment(Swoosh.Attachment.new(logo_path, filename: "logo.png", type: :inline))
+    end
+  end
+
+  defp format_links(str) do
+    re = ~r/<a.+href=['"].*>/iU
+    %{link_color: color} = Config.get([__MODULE__, :styling])
+
+    Regex.replace(re, str, fn link ->
+      String.replace(link, "<a", "<a style=\"color: #{color};text-decoration: none;\"")
+    end)
+  end
+
+  @doc """
+  Generate unsubscribe link for given user and notifications type.
+  The link contains JWT token with the data, and subscription can be modified without
+  authorization.
+  """
+  @spec unsubscribe_url(User.t(), String.t()) :: String.t()
+  def unsubscribe_url(user, notifications_type) do
+    token =
+      %{"sub" => user.id, "act" => %{"unsubscribe" => notifications_type}, "exp" => false}
+      |> Pleroma.JWT.generate_and_sign!()
+      |> Base.encode64()
+
+    Router.Helpers.subscription_url(Endpoint, :unsubscribe, token)
+  end
 end

lib/pleroma/emoji.ex

@@ -24,7 +24,7 @@ defmodule Pleroma.Emoji do
   @ets_options [:ordered_set, :protected, :named_table, {:read_concurrency, true}]
 
   @doc false
-  def start_link do
+  def start_link(_) do
     GenServer.start_link(__MODULE__, [], name: __MODULE__)
   end
 

lib/pleroma/flake_id.ex

@@ -66,6 +66,16 @@ def from_integer(integer) do
   @spec get :: binary
   def get, do: to_string(:gen_server.call(:flake, :get))
 
+  # checks that ID is is valid FlakeID
+  #
+  @spec is_flake_id?(String.t()) :: boolean
+  def is_flake_id?(id), do: is_flake_id?(String.to_charlist(id), true)
+  defp is_flake_id?([c | cs], true) when c >= ?0 and c <= ?9, do: is_flake_id?(cs, true)
+  defp is_flake_id?([c | cs], true) when c >= ?A and c <= ?Z, do: is_flake_id?(cs, true)
+  defp is_flake_id?([c | cs], true) when c >= ?a and c <= ?z, do: is_flake_id?(cs, true)
+  defp is_flake_id?([], true), do: true
+  defp is_flake_id?(_, _), do: false
+
   # -- Ecto.Type API
   @impl Ecto.Type
   def type, do: :uuid
@@ -88,7 +98,7 @@ def dump(value) do
   def autogenerate, do: get()
 
   # -- GenServer API
-  def start_link do
+  def start_link(_) do
     :gen_server.start_link({:local, :flake}, __MODULE__, [], [])
   end
 

lib/pleroma/gopher/server.ex

@@ -6,7 +6,7 @@ defmodule Pleroma.Gopher.Server do
   use GenServer
   require Logger
 
-  def start_link do
+  def start_link(_) do
     config = Pleroma.Config.get(:gopher, [])
     ip = Keyword.get(config, :ip, {0, 0, 0, 0})
     port = Keyword.get(config, :port, 1234)

lib/pleroma/html.ex

@@ -203,6 +203,8 @@ defmodule Pleroma.HTML.Scrubber.Default do
   Meta.allow_tag_with_these_attributes("p", [])
   Meta.allow_tag_with_these_attributes("pre", [])
   Meta.allow_tag_with_these_attributes("strong", [])
+  Meta.allow_tag_with_these_attributes("sub", [])
+  Meta.allow_tag_with_these_attributes("sup", [])
   Meta.allow_tag_with_these_attributes("u", [])
   Meta.allow_tag_with_these_attributes("ul", [])
 
@@ -280,3 +282,31 @@ def scrub({tag, attributes, children}), do: {tag, attributes, children}
   def scrub({_tag, children}), do: children
   def scrub(text), do: text
 end
+
+defmodule Pleroma.HTML.Scrubber.LinksOnly do
+  @moduledoc """
+  An HTML scrubbing policy which limits to links only.
+  """
+
+  @valid_schemes Pleroma.Config.get([:uri_schemes, :valid_schemes], [])
+
+  require HtmlSanitizeEx.Scrubber.Meta
+  alias HtmlSanitizeEx.Scrubber.Meta
+
+  Meta.remove_cdata_sections_before_scrub()
+  Meta.strip_comments()
+
+  # links
+  Meta.allow_tag_with_uri_attributes("a", ["href"], @valid_schemes)
+
+  Meta.allow_tag_with_this_attribute_values("a", "rel", [
+    "tag",
+    "nofollow",
+    "noopener",
+    "noreferrer",
+    "me"
+  ])
+
+  Meta.allow_tag_with_these_attributes("a", ["name", "title"])
+  Meta.strip_everything_not_covered()
+end

lib/pleroma/http/connection.ex

@@ -11,6 +11,7 @@ defmodule Pleroma.HTTP.Connection do
     connect_timeout: 10_000,
     recv_timeout: 20_000,
     follow_redirect: true,
+    force_redirect: true,
     pool: :federation
   ]
   @adapter Application.get_env(:tesla, :adapter)

lib/pleroma/jwt.ex

@@ -0,0 +1,9 @@
+defmodule Pleroma.JWT do
+  use Joken.Config
+
+  @impl true
+  def token_config do
+    default_claims(skip: [:aud])
+    |> add_claim("aud", &Pleroma.Web.Endpoint.url/0, &(&1 == Pleroma.Web.Endpoint.url()))
+  end
+end

lib/pleroma/notification.ex

@@ -18,6 +18,8 @@ defmodule Pleroma.Notification do
   import Ecto.Query
   import Ecto.Changeset
 
+  @type t :: %__MODULE__{}
+
   schema "notifications" do
     field(:seen, :boolean, default: false)
     belongs_to(:user, User, type: Pleroma.FlakeId)
@@ -31,7 +33,7 @@ def changeset(%Notification{} = notification, attrs) do
     |> cast(attrs, [:seen])
   end
 
-  def for_user_query(user, opts) do
+  def for_user_query(user, opts \\ []) do
     query =
       Notification
       |> where(user_id: ^user.id)
@@ -75,6 +77,25 @@ def for_user(user, opts \\ %{}) do
     |> Pagination.fetch_paginated(opts)
   end
 
+  @doc """
+  Returns notifications for user received since given date.
+
+  ## Examples
+
+      iex> Pleroma.Notification.for_user_since(%Pleroma.User{}, ~N[2019-04-13 11:22:33])
+      [%Pleroma.Notification{}, %Pleroma.Notification{}]
+
+      iex> Pleroma.Notification.for_user_since(%Pleroma.User{}, ~N[2019-04-15 11:22:33])
+      []
+  """
+  @spec for_user_since(Pleroma.User.t(), NaiveDateTime.t()) :: [t()]
+  def for_user_since(user, date) do
+    from(n in for_user_query(user),
+      where: n.updated_at > ^date
+    )
+    |> Repo.all()
+  end
+
   def set_read_up_to(%{id: user_id} = _user, id) do
     query =
       from(
@@ -82,7 +103,10 @@ def set_read_up_to(%{id: user_id} = _user, id) do
         where: n.user_id == ^user_id,
         where: n.id <= ^id,
         update: [
-          set: [seen: true]
+          set: [
+            seen: true,
+            updated_at: ^NaiveDateTime.utc_now()
+          ]
         ]
       )
 

lib/pleroma/object/fetcher.ex

@@ -114,7 +114,7 @@ defp maybe_date_fetch(headers, date) do
     end
   end
 
-  def fetch_and_contain_remote_object_from_id(id) do
+  def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
     Logger.info("Fetching object #{id} via AP")
 
     date =
@@ -141,4 +141,9 @@ def fetch_and_contain_remote_object_from_id(id) do
         {:error, e}
     end
   end
+
+  def fetch_and_contain_remote_object_from_id(%{"id" => id}),
+    do: fetch_and_contain_remote_object_from_id(id)
+
+  def fetch_and_contain_remote_object_from_id(_id), do: {:error, "id must be a string"}
 end

lib/pleroma/plugs/set_format_plug.ex

@@ -0,0 +1,24 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Plugs.SetFormatPlug do
+  import Plug.Conn, only: [assign: 3, fetch_query_params: 1]
+
+  def init(_), do: nil
+
+  def call(conn, _) do
+    case get_format(conn) do
+      nil -> conn
+      format -> assign(conn, :format, format)
+    end
+  end
+
+  defp get_format(conn) do
+    conn.private[:phoenix_format] ||
+      case fetch_query_params(conn) do
+        %{query_params: %{"_format" => format}} -> format
+        _ -> nil
+      end
+  end
+end

lib/pleroma/reverse_proxy/reverse_proxy.ex

@@ -109,7 +109,11 @@ def call(conn = %{method: method}, url, opts) when method in @methods do
       end
 
     with {:ok, code, headers, client} <- request(method, url, req_headers, hackney_opts),
-         :ok <- header_length_constraint(headers, Keyword.get(opts, :max_body_length)) do
+         :ok <-
+           header_length_constraint(
+             headers,
+             Keyword.get(opts, :max_body_length, @max_body_length)
+           ) do
       response(conn, client, url, code, headers, opts)
     else
       {:ok, code, headers} ->
@@ -200,7 +204,11 @@ defp chunk_reply(conn, client, opts, sent_so_far, duration) do
          {:ok, data} <- client().stream_body(client),
          {:ok, duration} <- increase_read_duration(duration),
          sent_so_far = sent_so_far + byte_size(data),
-         :ok <- body_size_constraint(sent_so_far, Keyword.get(opts, :max_body_size)),
+         :ok <-
+           body_size_constraint(
+             sent_so_far,
+             Keyword.get(opts, :max_body_length, @max_body_length)
+           ),
          {:ok, conn} <- chunk(conn, data) do
       chunk_reply(conn, client, opts, sent_so_far, duration)
     else

lib/pleroma/scheduled_activity_worker.ex

@@ -16,7 +16,7 @@ defmodule Pleroma.ScheduledActivityWorker do
 
   @schedule_interval :timer.minutes(1)
 
-  def start_link do
+  def start_link(_) do
     GenServer.start_link(__MODULE__, nil)
   end
 

lib/pleroma/signature.ex

@@ -15,7 +15,7 @@ def key_id_to_actor_id(key_id) do
       |> Map.put(:fragment, nil)
 
     uri =
-      if String.ends_with?(uri.path, "/publickey") do
+      if not is_nil(uri.path) and String.ends_with?(uri.path, "/publickey") do
         Map.put(uri, :path, String.replace(uri.path, "/publickey", ""))
       else
         uri

lib/pleroma/stats.ex

@@ -7,31 +7,56 @@ defmodule Pleroma.Stats do
   alias Pleroma.Repo
   alias Pleroma.User
 
-  def start_link do
+  use GenServer
-    agent = Agent.start_link(fn -> {[], %{}} end, name: __MODULE__)
+
-    spawn(fn -> schedule_update() end)
+  @interval 1000 * 60 * 60
-    agent
+
+  def start_link(_) do
+    GenServer.start_link(__MODULE__, initial_data(), name: __MODULE__)
+  end
+
+  def force_update do
+    GenServer.call(__MODULE__, :force_update)
   end
 
   def get_stats do
-    Agent.get(__MODULE__, fn {_, stats} -> stats end)
+    %{stats: stats} = GenServer.call(__MODULE__, :get_state)
+
+    stats
   end
 
   def get_peers do
-    Agent.get(__MODULE__, fn {peers, _} -> peers end)
+    %{peers: peers} = GenServer.call(__MODULE__, :get_state)
+
+    peers
   end
 
-  def schedule_update do
+  def init(args) do
-    spawn(fn ->
+    Process.send(self(), :run_update, [])
-      # 1 hour
+    {:ok, args}
-      Process.sleep(1000 * 60 * 60)
-      schedule_update()
-    end)
-
-    update_stats()
   end
 
-  def update_stats do
+  def handle_call(:force_update, _from, _state) do
+    new_stats = get_stat_data()
+    {:reply, new_stats, new_stats}
+  end
+
+  def handle_call(:get_state, _from, state) do
+    {:reply, state, state}
+  end
+
+  def handle_info(:run_update, _state) do
+    new_stats = get_stat_data()
+
+    Process.send_after(self(), :run_update, @interval)
+    {:noreply, new_stats}
+  end
+
+  defp initial_data do
+    %{peers: [], stats: %{}}
+  end
+
+  defp get_stat_data do
     peers =
       from(
         u in User,
@@ -52,8 +77,9 @@ def update_stats do
 
     user_count = Repo.aggregate(User.Query.build(%{local: true, active: true}), :count, :id)
 
-    Agent.update(__MODULE__, fn _ ->
+    %{
-      {peers, %{domain_count: domain_count, status_count: status_count, user_count: user_count}}
+      peers: peers,
-    end)
+      stats: %{domain_count: domain_count, status_count: status_count, user_count: user_count}
+    }
   end
 end

lib/pleroma/upload.ex

@@ -228,7 +228,14 @@ defp url_from_spec(%__MODULE__{name: name}, base_url, {:file, path}) do
           ""
         end
 
-    [base_url, "media", path]
+    prefix =
+      if is_nil(Pleroma.Config.get([__MODULE__, :base_url])) do
+        "media"
+      else
+        ""
+      end
+
+    [base_url, prefix, path]
     |> Path.join()
   end
 

lib/pleroma/uploaders/local.ex

@@ -11,7 +11,7 @@ def get_file(_) do
 
   def put_file(upload) do
     {local_path, file} =
-      case Enum.reverse(String.split(upload.path, "/", trim: true)) do
+      case Enum.reverse(Path.split(upload.path)) do
         [file] ->
           {upload_path(), file}
 
@@ -23,7 +23,7 @@ def put_file(upload) do
 
     result_file = Path.join(local_path, file)
 
-    unless File.exists?(result_file) do
+    if not File.exists?(result_file) do
       File.cp!(upload.tempfile, result_file)
     end
 

lib/pleroma/uploaders/mdii.ex

@@ -3,6 +3,8 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Uploaders.MDII do
+  @moduledoc "Represents uploader for https://github.com/hakaba-hitoyo/minimal-digital-image-infrastructure"
+
   alias Pleroma.Config
   alias Pleroma.HTTP
 

lib/pleroma/uploaders/s3.ex

@@ -6,10 +6,12 @@ defmodule Pleroma.Uploaders.S3 do
   @behaviour Pleroma.Uploaders.Uploader
   require Logger
 
+  alias Pleroma.Config
+
   # The file name is re-encoded with S3's constraints here to comply with previous
   # links with less strict filenames
   def get_file(file) do
-    config = Pleroma.Config.get([__MODULE__])
+    config = Config.get([__MODULE__])
     bucket = Keyword.fetch!(config, :bucket)
 
     bucket_with_namespace =
@@ -34,15 +36,15 @@ def get_file(file) do
   end
 
   def put_file(%Pleroma.Upload{} = upload) do
-    config = Pleroma.Config.get([__MODULE__])
+    config = Config.get([__MODULE__])
     bucket = Keyword.get(config, :bucket)
 
-    {:ok, file_data} = File.read(upload.tempfile)
-
     s3_name = strict_encode(upload.path)
 
     op =
-      ExAws.S3.put_object(bucket, s3_name, file_data, [
+      upload.tempfile
+      |> ExAws.S3.Upload.stream_file()
+      |> ExAws.S3.upload(bucket, s3_name, [
         {:acl, :public_read},
         {:content_type, upload.content_type}
       ])

lib/pleroma/user.ex

@@ -21,6 +21,7 @@ defmodule Pleroma.User do
   alias Pleroma.Web
   alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.ActivityPub.Utils
+  alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.CommonAPI.Utils, as: CommonUtils
   alias Pleroma.Web.OAuth
   alias Pleroma.Web.OStatus
@@ -57,6 +58,7 @@ defmodule Pleroma.User do
     field(:search_type, :integer, virtual: true)
     field(:tags, {:array, :string}, default: [])
     field(:last_refreshed_at, :naive_datetime_usec)
+    field(:last_digest_emailed_at, :naive_datetime)
     has_many(:notifications, Notification)
     has_many(:registrations, Registration)
     embeds_one(:info, User.Info)
@@ -114,7 +116,9 @@ def ap_following(%User{} = user), do: "#{ap_id(user)}/following"
 
   def user_info(%User{} = user, args \\ %{}) do
     following_count =
-      if args[:following_count], do: args[:following_count], else: following_count(user)
+      if args[:following_count],
+        do: args[:following_count],
+        else: user.info.following_count || following_count(user)
 
     follower_count =
       if args[:follower_count], do: args[:follower_count], else: user.info.follower_count
@@ -129,6 +133,28 @@ def user_info(%User{} = user, args \\ %{}) do
     |> Map.put(:follower_count, follower_count)
   end
 
+  def follow_state(%User{} = user, %User{} = target) do
+    follow_activity = Utils.fetch_latest_follow(user, target)
+
+    if follow_activity,
+      do: follow_activity.data["state"],
+      # Ideally this would be nil, but then Cachex does not commit the value
+      else: false
+  end
+
+  def get_cached_follow_state(user, target) do
+    key = "follow_state:#{user.ap_id}|#{target.ap_id}"
+    Cachex.fetch!(:user_cache, key, fn _ -> {:commit, follow_state(user, target)} end)
+  end
+
+  def set_follow_state_cache(user_ap_id, target_ap_id, state) do
+    Cachex.put(
+      :user_cache,
+      "follow_state:#{user_ap_id}|#{target_ap_id}",
+      state
+    )
+  end
+
   def set_info_cache(user, args) do
     Cachex.put(:user_cache, "user_info:#{user.id}", user_info(user, args))
   end
@@ -149,10 +175,10 @@ def following_count(%User{} = user) do
   end
 
   def remote_user_creation(params) do
-    params =
+    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
-      params
+    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
-      |> Map.put(:info, params[:info] || %{})
 
+    params = Map.put(params, :info, params[:info] || %{})
     info_cng = User.Info.remote_user_creation(%User.Info{}, params[:info])
 
     changes =
@@ -161,8 +187,8 @@ def remote_user_creation(params) do
       |> validate_required([:name, :ap_id])
       |> unique_constraint(:nickname)
       |> validate_format(:nickname, @email_regex)
-      |> validate_length(:bio, max: 5000)
+      |> validate_length(:bio, max: bio_limit)
-      |> validate_length(:name, max: 100)
+      |> validate_length(:name, max: name_limit)
       |> put_change(:local, false)
       |> put_embed(:info, info_cng)
 
@@ -185,22 +211,23 @@ def remote_user_creation(params) do
   end
 
   def update_changeset(struct, params \\ %{}) do
+    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
+    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
+
     struct
     |> cast(params, [:bio, :name, :avatar, :following])
     |> unique_constraint(:nickname)
     |> validate_format(:nickname, local_nickname_regex())
-    |> validate_length(:bio, max: 5000)
+    |> validate_length(:bio, max: bio_limit)
-    |> validate_length(:name, min: 1, max: 100)
+    |> validate_length(:name, min: 1, max: name_limit)
   end
 
-  def upgrade_changeset(struct, params \\ %{}) do
+  def upgrade_changeset(struct, params \\ %{}, remote? \\ false) do
-    params =
+    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
-      params
+    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
-      |> Map.put(:last_refreshed_at, NaiveDateTime.utc_now())
 
-    info_cng =
+    params = Map.put(params, :last_refreshed_at, NaiveDateTime.utc_now())
-      struct.info
+    info_cng = User.Info.user_upgrade(struct.info, params[:info], remote?)
-      |> User.Info.user_upgrade(params[:info])
 
     struct
     |> cast(params, [
@@ -213,8 +240,8 @@ def upgrade_changeset(struct, params \\ %{}) do
     ])
     |> unique_constraint(:nickname)
     |> validate_format(:nickname, local_nickname_regex())
-    |> validate_length(:bio, max: 5000)
+    |> validate_length(:bio, max: bio_limit)
-    |> validate_length(:name, max: 100)
+    |> validate_length(:name, max: name_limit)
     |> put_embed(:info, info_cng)
   end
 
@@ -226,6 +253,7 @@ def password_update_changeset(struct, params) do
     |> put_password_hash
   end
 
+  @spec reset_password(User.t(), map) :: {:ok, User.t()} | {:error, Ecto.Changeset.t()}
   def reset_password(%User{id: user_id} = user, data) do
     multi =
       Multi.new()
@@ -240,6 +268,9 @@ def reset_password(%User{id: user_id} = user, data) do
   end
 
   def register_changeset(struct, params \\ %{}, opts \\ []) do
+    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
+    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
+
     need_confirmation? =
       if is_nil(opts[:need_confirmation]) do
         Pleroma.Config.get([:instance, :account_activation_required])
@@ -260,8 +291,8 @@ def register_changeset(struct, params \\ %{}, opts \\ []) do
       |> validate_exclusion(:nickname, Pleroma.Config.get([User, :restricted_nicknames]))
       |> validate_format(:nickname, local_nickname_regex())
       |> validate_format(:email, @email_regex)
-      |> validate_length(:bio, max: 1000)
+      |> validate_length(:bio, max: bio_limit)
-      |> validate_length(:name, min: 1, max: 100)
+      |> validate_length(:name, min: 1, max: name_limit)
       |> put_change(:info, info_change)
 
     changeset =
@@ -330,6 +361,7 @@ def needs_update?(%User{local: false} = user) do
 
   def needs_update?(_), do: true
 
+  @spec maybe_direct_follow(User.t(), User.t()) :: {:ok, User.t()} | {:error, String.t()}
   def maybe_direct_follow(%User{} = follower, %User{local: true, info: %{locked: true}}) do
     {:ok, follower}
   end
@@ -404,6 +436,8 @@ def follow(%User{} = follower, %User{info: info} = followed) do
 
         {1, [follower]} = Repo.update_all(q, [])
 
+        follower = maybe_update_following_count(follower)
+
         {:ok, _} = update_follower_count(followed)
 
         set_cache(follower)
@@ -423,6 +457,8 @@ def unfollow(%User{} = follower, %User{} = followed) do
 
       {1, [follower]} = Repo.update_all(q, [])
 
+      follower = maybe_update_following_count(follower)
+
       {:ok, followed} = update_follower_count(followed)
 
       set_cache(follower)
@@ -450,6 +486,13 @@ def get_by_ap_id(ap_id) do
     Repo.get_by(User, ap_id: ap_id)
   end
 
+  def get_all_by_ap_id(ap_ids) do
+    from(u in __MODULE__,
+      where: u.ap_id in ^ap_ids
+    )
+    |> Repo.all()
+  end
+
   # This is mostly an SPC migration fix. This guesses the user nickname by taking the last part
   # of the ap_id and the domain and tries to get that user
   def get_by_guessed_nickname(ap_id) do
@@ -471,7 +514,7 @@ def set_cache(%User{} = user) do
   end
 
   def update_and_set_cache(changeset) do
-    with {:ok, user} <- Repo.update(changeset) do
+    with {:ok, user} <- Repo.update(changeset, stale_error_field: :id) do
       set_cache(user)
     else
       e -> e
@@ -707,32 +750,75 @@ def update_note_count(%User{} = user) do
     |> update_and_set_cache()
   end
 
-  def update_follower_count(%User{} = user) do
+  @spec maybe_fetch_follow_information(User.t()) :: User.t()
-    follower_count_query =
+  def maybe_fetch_follow_information(user) do
-      User.Query.build(%{followers: user, deactivated: false})
+    with {:ok, user} <- fetch_follow_information(user) do
-      |> select([u], %{count: count(u.id)})
+      user
+    else
+      e ->
+        Logger.error("Follower/Following counter update for #{user.ap_id} failed.\n#{inspect(e)}")
 
-    User
+        user
-    |> where(id: ^user.id)
-    |> join(:inner, [u], s in subquery(follower_count_query))
-    |> update([u, s],
-      set: [
-        info:
-          fragment(
-            "jsonb_set(?, '{follower_count}', ?::varchar::jsonb, true)",
-            u.info,
-            s.count
-          )
-      ]
-    )
-    |> select([u], u)
-    |> Repo.update_all([])
-    |> case do
-      {1, [user]} -> set_cache(user)
-      _ -> {:error, user}
     end
   end
 
+  def fetch_follow_information(user) do
+    with {:ok, info} <- ActivityPub.fetch_follow_information_for_user(user) do
+      info_cng = User.Info.follow_information_update(user.info, info)
+
+      changeset =
+        user
+        |> change()
+        |> put_embed(:info, info_cng)
+
+      update_and_set_cache(changeset)
+    else
+      {:error, _} = e -> e
+      e -> {:error, e}
+    end
+  end
+
+  def update_follower_count(%User{} = user) do
+    if user.local or !Pleroma.Config.get([:instance, :external_user_synchronization]) do
+      follower_count_query =
+        User.Query.build(%{followers: user, deactivated: false})
+        |> select([u], %{count: count(u.id)})
+
+      User
+      |> where(id: ^user.id)
+      |> join(:inner, [u], s in subquery(follower_count_query))
+      |> update([u, s],
+        set: [
+          info:
+            fragment(
+              "jsonb_set(?, '{follower_count}', ?::varchar::jsonb, true)",
+              u.info,
+              s.count
+            )
+        ]
+      )
+      |> select([u], u)
+      |> Repo.update_all([])
+      |> case do
+        {1, [user]} -> set_cache(user)
+        _ -> {:error, user}
+      end
+    else
+      {:ok, maybe_fetch_follow_information(user)}
+    end
+  end
+
+  @spec maybe_update_following_count(User.t()) :: User.t()
+  def maybe_update_following_count(%User{local: false} = user) do
+    if Pleroma.Config.get([:instance, :external_user_synchronization]) do
+      maybe_fetch_follow_information(user)
+    else
+      user
+    end
+  end
+
+  def maybe_update_following_count(user), do: user
+
   def remove_duplicated_following(%User{following: following} = user) do
     uniq_following = Enum.uniq(following)
 
@@ -831,6 +917,13 @@ def block(blocker, %User{ap_id: ap_id} = blocked) do
         blocker
       end
 
+    # clear any requested follows as well
+    blocked =
+      case CommonAPI.reject_follow_request(blocked, blocker) do
+        {:ok, %User{} = updated_blocked} -> updated_blocked
+        nil -> blocked
+      end
+
     blocker =
       if subscribed_to?(blocked, blocker) do
         {:ok, blocker} = unsubscribe(blocked, blocker)
@@ -882,19 +975,26 @@ def muted_notifications?(nil, _), do: false
   def muted_notifications?(user, %{ap_id: ap_id}),
     do: Enum.member?(user.info.muted_notifications, ap_id)
 
-  def blocks?(%User{info: info} = _user, %{ap_id: ap_id}) do
+  def blocks?(%User{} = user, %User{} = target) do
-    blocks = info.blocks
+    blocks_ap_id?(user, target) || blocks_domain?(user, target)
-
-    domain_blocks = Pleroma.Web.ActivityPub.MRF.subdomains_regex(info.domain_blocks)
-
-    %{host: host} = URI.parse(ap_id)
-
-    Enum.member?(blocks, ap_id) ||
-      Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, host)
   end
 
   def blocks?(nil, _), do: false
 
+  def blocks_ap_id?(%User{} = user, %User{} = target) do
+    Enum.member?(user.info.blocks, target.ap_id)
+  end
+
+  def blocks_ap_id?(_, _), do: false
+
+  def blocks_domain?(%User{} = user, %User{} = target) do
+    domain_blocks = Pleroma.Web.ActivityPub.MRF.subdomains_regex(user.info.domain_blocks)
+    %{host: host} = URI.parse(target.ap_id)
+    Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, host)
+  end
+
+  def blocks_domain?(_, _), do: false
+
   def subscribed_to?(user, %{ap_id: ap_id}) do
     with %User{} = target <- get_cached_by_ap_id(ap_id) do
       Enum.member?(target.info.subscribers, user.ap_id)
@@ -1363,6 +1463,80 @@ def showing_reblogs?(%User{} = user, %User{} = target) do
     target.ap_id not in user.info.muted_reblogs
   end
 
+  @doc """
+  The function returns a query to get users with no activity for given interval of days.
+  Inactive users are those who didn't read any notification, or had any activity where
+  the user is the activity's actor, during `inactivity_threshold` days.
+  Deactivated users will not appear in this list.
+
+  ## Examples
+
+      iex> Pleroma.User.list_inactive_users()
+      %Ecto.Query{}
+  """
+  @spec list_inactive_users_query(integer()) :: Ecto.Query.t()
+  def list_inactive_users_query(inactivity_threshold \\ 7) do
+    negative_inactivity_threshold = -inactivity_threshold
+    now = NaiveDateTime.truncate(NaiveDateTime.utc_now(), :second)
+    # Subqueries are not supported in `where` clauses, join gets too complicated.
+    has_read_notifications =
+      from(n in Pleroma.Notification,
+        where: n.seen == true,
+        group_by: n.id,
+        having: max(n.updated_at) > datetime_add(^now, ^negative_inactivity_threshold, "day"),
+        select: n.user_id
+      )
+      |> Pleroma.Repo.all()
+
+    from(u in Pleroma.User,
+      left_join: a in Pleroma.Activity,
+      on: u.ap_id == a.actor,
+      where: not is_nil(u.nickname),
+      where: fragment("not (?->'deactivated' @> 'true')", u.info),
+      where: u.id not in ^has_read_notifications,
+      group_by: u.id,
+      having:
+        max(a.inserted_at) < datetime_add(^now, ^negative_inactivity_threshold, "day") or
+          is_nil(max(a.inserted_at))
+    )
+  end
+
+  @doc """
+  Enable or disable email notifications for user
+
+  ## Examples
+
+      iex> Pleroma.User.switch_email_notifications(Pleroma.User{info: %{email_notifications: %{"digest" => false}}}, "digest", true)
+      Pleroma.User{info: %{email_notifications: %{"digest" => true}}}
+
+      iex> Pleroma.User.switch_email_notifications(Pleroma.User{info: %{email_notifications: %{"digest" => true}}}, "digest", false)
+      Pleroma.User{info: %{email_notifications: %{"digest" => false}}}
+  """
+  @spec switch_email_notifications(t(), String.t(), boolean()) ::
+          {:ok, t()} | {:error, Ecto.Changeset.t()}
+  def switch_email_notifications(user, type, status) do
+    info = Pleroma.User.Info.update_email_notifications(user.info, %{type => status})
+
+    change(user)
+    |> put_embed(:info, info)
+    |> update_and_set_cache()
+  end
+
+  @doc """
+  Set `last_digest_emailed_at` value for the user to current time
+  """
+  @spec touch_last_digest_emailed_at(t()) :: t()
+  def touch_last_digest_emailed_at(user) do
+    now = NaiveDateTime.truncate(NaiveDateTime.utc_now(), :second)
+
+    {:ok, updated_user} =
+      user
+      |> change(%{last_digest_emailed_at: now})
+      |> update_and_set_cache()
+
+    updated_user
+  end
+
   @spec toggle_confirmation(User.t()) :: {:ok, User.t()} | {:error, Changeset.t()}
   def toggle_confirmation(%User{} = user) do
     need_confirmation? = !user.info.confirmation_pending

lib/pleroma/user/info.ex

@@ -16,6 +16,8 @@ defmodule Pleroma.User.Info do
     field(:source_data, :map, default: %{})
     field(:note_count, :integer, default: 0)
     field(:follower_count, :integer, default: 0)
+    # Should be filled in only for remote users
+    field(:following_count, :integer, default: nil)
     field(:locked, :boolean, default: false)
     field(:confirmation_pending, :boolean, default: false)
     field(:confirmation_token, :string, default: nil)
@@ -43,9 +45,12 @@ defmodule Pleroma.User.Info do
     field(:hide_follows, :boolean, default: false)
     field(:hide_favorites, :boolean, default: true)
     field(:pinned_activities, {:array, :string}, default: [])
+    field(:email_notifications, :map, default: %{"digest" => false})
     field(:mascot, :map, default: nil)
     field(:emoji, {:array, :map}, default: [])
     field(:pleroma_settings_store, :map, default: %{})
+    field(:fields, {:array, :map}, default: [])
+    field(:raw_fields, {:array, :map}, default: [])
 
     field(:notification_settings, :map,
       default: %{
@@ -93,6 +98,30 @@ def update_notification_settings(info, settings) do
     |> validate_required([:notification_settings])
   end
 
+  @doc """
+  Update email notifications in the given User.Info struct.
+
+  Examples:
+
+      iex> update_email_notifications(%Pleroma.User.Info{email_notifications: %{"digest" => false}}, %{"digest" => true})
+      %Pleroma.User.Info{email_notifications: %{"digest" => true}}
+
+  """
+  @spec update_email_notifications(t(), map()) :: Ecto.Changeset.t()
+  def update_email_notifications(info, settings) do
+    email_notifications =
+      info.email_notifications
+      |> Map.merge(settings)
+      |> Map.take(["digest"])
+
+    params = %{email_notifications: email_notifications}
+    fields = [:email_notifications]
+
+    info
+    |> cast(params, fields)
+    |> validate_required(fields)
+  end
+
   def add_to_note_count(info, number) do
     set_note_count(info, info.note_count + number)
   end
@@ -223,19 +252,31 @@ def remote_user_creation(info, params) do
       :uri,
       :hub,
       :topic,
-      :salmon
+      :salmon,
+      :hide_followers,
+      :hide_follows,
+      :follower_count,
+      :fields,
+      :following_count
     ])
+    |> validate_fields(true)
   end
 
-  def user_upgrade(info, params) do
+  def user_upgrade(info, params, remote? \\ false) do
     info
     |> cast(params, [
       :ap_enabled,
       :source_data,
       :banner,
       :locked,
-      :magic_key
+      :magic_key,
+      :follower_count,
+      :following_count,
+      :hide_follows,
+      :fields,
+      :hide_followers
     ])
+    |> validate_fields(remote?)
   end
 
   def profile_update(info, params) do
@@ -251,10 +292,40 @@ def profile_update(info, params) do
       :background,
       :show_role,
       :skip_thread_containment,
+      :fields,
+      :raw_fields,
       :pleroma_settings_store
     ])
+    |> validate_fields()
   end
 
+  def validate_fields(changeset, remote? \\ false) do
+    limit_name = if remote?, do: :max_remote_account_fields, else: :max_account_fields
+    limit = Pleroma.Config.get([:instance, limit_name], 0)
+
+    changeset
+    |> validate_length(:fields, max: limit)
+    |> validate_change(:fields, fn :fields, fields ->
+      if Enum.all?(fields, &valid_field?/1) do
+        []
+      else
+        [fields: "invalid"]
+      end
+    end)
+  end
+
+  defp valid_field?(%{"name" => name, "value" => value}) do
+    name_limit = Pleroma.Config.get([:instance, :account_field_name_length], 255)
+    value_limit = Pleroma.Config.get([:instance, :account_field_value_length], 255)
+
+    is_binary(name) &&
+      is_binary(value) &&
+      String.length(name) <= name_limit &&
+      String.length(value) <= value_limit
+  end
+
+  defp valid_field?(_), do: false
+
   @spec confirmation_changeset(Info.t(), keyword()) :: Changeset.t()
   def confirmation_changeset(info, opts) do
     need_confirmation? = Keyword.get(opts, :need_confirmation)
@@ -348,4 +419,27 @@ def remove_reblog_mute(info, ap_id) do
 
     cast(info, params, [:muted_reblogs])
   end
+
+  # ``fields`` is an array of mastodon profile field, containing ``{"name": "…", "value": "…"}``.
+  # For example: [{"name": "Pronoun", "value": "she/her"}, …]
+  def fields(%{fields: [], source_data: %{"attachment" => attachment}}) do
+    limit = Pleroma.Config.get([:instance, :max_remote_account_fields], 0)
+
+    attachment
+    |> Enum.filter(fn %{"type" => t} -> t == "PropertyValue" end)
+    |> Enum.map(fn fields -> Map.take(fields, ["name", "value"]) end)
+    |> Enum.take(limit)
+  end
+
+  def fields(%{fields: fields}), do: fields
+
+  def follow_information_update(info, params) do
+    info
+    |> cast(params, [
+      :hide_followers,
+      :hide_follows,
+      :follower_count,
+      :following_count
+    ])
+  end
 end

lib/pleroma/user/search.ex

@@ -44,7 +44,7 @@ defp format_query(query_string) do
     query_string = String.trim_leading(query_string, "@")
 
     with [name, domain] <- String.split(query_string, "@"),
-         formatted_domain <- String.replace(domain, ~r/[!-\-|@|[-`|{-~|\/|:]+/, "") do
+         formatted_domain <- String.replace(domain, ~r/[!-\-|@|[-`|{-~|\/|:|\s]+/, "") do
       name <> "@" <> to_string(:idna.encode(formatted_domain))
     else
       _ -> query_string

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -23,6 +23,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   import Pleroma.Web.ActivityPub.Visibility
 
   require Logger
+  require Pleroma.Constants
 
   # For Announce activities, we filter the recipients based on following status for any actors
   # that match actual users.  See issue #164 for more information about why this is necessary.
@@ -64,12 +65,12 @@ defp check_actor_is_active(actor) do
     if not is_nil(actor) do
       with user <- User.get_cached_by_ap_id(actor),
            false <- user.info.deactivated do
-        :ok
+        true
       else
-        _e -> :reject
+        _e -> false
       end
     else
-      :ok
+      true
     end
   end
 
@@ -118,10 +119,10 @@ def increase_poll_votes_if_vote(%{
 
   def increase_poll_votes_if_vote(_create_data), do: :noop
 
-  def insert(map, local \\ true, fake \\ false) when is_map(map) do
+  def insert(map, local \\ true, fake \\ false, bypass_actor_check \\ false) when is_map(map) do
     with nil <- Activity.normalize(map),
          map <- lazy_put_activity_defaults(map, fake),
-         :ok <- check_actor_is_active(map["actor"]),
+         true <- bypass_actor_check || check_actor_is_active(map["actor"]),
          {_, true} <- {:remote_limit_error, check_remote_limit(map)},
          {:ok, map} <- MRF.filter(map),
          {recipients, _, _} = get_recipients(map),
@@ -207,8 +208,6 @@ def stream_out_participations(%Object{data: %{"context" => context}}, user) do
   def stream_out_participations(_, _), do: :noop
 
   def stream_out(activity) do
-    public = "https://www.w3.org/ns/activitystreams#Public"
-
     if activity.data["type"] in ["Create", "Announce", "Delete"] do
       object = Object.normalize(activity)
       # Do not stream out poll replies
@@ -216,7 +215,7 @@ def stream_out(activity) do
         Pleroma.Web.Streamer.stream("user", activity)
         Pleroma.Web.Streamer.stream("list", activity)
 
-        if Enum.member?(activity.data["to"], public) do
+        if get_visibility(activity) == "public" do
           Pleroma.Web.Streamer.stream("public", activity)
 
           if activity.local do
@@ -238,13 +237,8 @@ def stream_out(activity) do
             end
           end
         else
-          # TODO: Write test, replace with visibility test
+          if get_visibility(activity) == "direct",
-          if !Enum.member?(activity.data["cc"] || [], public) &&
+            do: Pleroma.Web.Streamer.stream("direct", activity)
-               !Enum.member?(
-                 activity.data["to"],
-                 User.get_cached_by_ap_id(activity.data["actor"]).follower_address
-               ),
-             do: Pleroma.Web.Streamer.stream("direct", activity)
         end
       end
     end
@@ -273,6 +267,9 @@ def create(%{to: to, actor: actor, context: context, object: object} = params, f
     else
       {:fake, true, activity} ->
         {:ok, activity}
+
+      {:error, message} ->
+        {:error, message}
     end
   end
 
@@ -391,7 +388,8 @@ def unannounce(
   def follow(follower, followed, activity_id \\ nil, local \\ true) do
     with data <- make_follow_data(follower, followed, activity_id),
          {:ok, activity} <- insert(data, local),
-         :ok <- maybe_federate(activity) do
+         :ok <- maybe_federate(activity),
+         _ <- User.set_follow_state_cache(follower.ap_id, followed.ap_id, activity.data["state"]) do
       {:ok, activity}
     end
   end
@@ -413,7 +411,7 @@ def delete(%User{ap_id: ap_id, follower_address: follower_address} = user) do
            "actor" => ap_id,
            "object" => %{"type" => "Person", "id" => ap_id}
          },
-         {:ok, activity} <- insert(data, true, true),
+         {:ok, activity} <- insert(data, true, true, true),
          :ok <- maybe_federate(activity) do
       {:ok, user}
     end
@@ -514,13 +512,15 @@ def flag(
   end
 
   defp fetch_activities_for_context_query(context, opts) do
-    public = ["https://www.w3.org/ns/activitystreams#Public"]
+    public = [Pleroma.Constants.as_public()]
 
     recipients =
       if opts["user"], do: [opts["user"].ap_id | opts["user"].following] ++ public, else: public
 
     from(activity in Activity)
     |> maybe_preload_objects(opts)
+    |> maybe_preload_bookmarks(opts)
+    |> maybe_set_thread_muted_field(opts)
     |> restrict_blocked(opts)
     |> restrict_recipients(recipients, opts["user"])
     |> where(
@@ -534,6 +534,7 @@ defp fetch_activities_for_context_query(context, opts) do
       )
     )
     |> exclude_poll_votes(opts)
+    |> exclude_id(opts)
     |> order_by([activity], desc: activity.id)
   end
 
@@ -555,7 +556,7 @@ def fetch_latest_activity_id_for_context(context, opts \\ %{}) do
   end
 
   def fetch_public_activities(opts \\ %{}) do
-    q = fetch_activities_query(["https://www.w3.org/ns/activitystreams#Public"], opts)
+    q = fetch_activities_query([Pleroma.Constants.as_public()], opts)
 
     q
     |> restrict_unlisted()
@@ -626,6 +627,7 @@ def fetch_user_activities(user, reading_user, params \\ %{}) do
     params =
       params
       |> Map.put("type", ["Create", "Announce"])
+      |> Map.put("user", reading_user)
       |> Map.put("actor_id", user.ap_id)
       |> Map.put("whole_db", true)
       |> Map.put("pinned_activity_ids", user.info.pinned_activities)
@@ -646,10 +648,9 @@ defp user_activities_recipients(%{"godmode" => true}) do
 
   defp user_activities_recipients(%{"reading_user" => reading_user}) do
     if reading_user do
-      ["https://www.w3.org/ns/activitystreams#Public"] ++
+      [Pleroma.Constants.as_public()] ++ [reading_user.ap_id | reading_user.following]
-        [reading_user.ap_id | reading_user.following]
     else
-      ["https://www.w3.org/ns/activitystreams#Public"]
+      [Pleroma.Constants.as_public()]
     end
   end
 
@@ -753,8 +754,8 @@ defp restrict_state(query, _), do: query
 
   defp restrict_favorited_by(query, %{"favorited_by" => ap_id}) do
     from(
-      activity in query,
+      [_activity, object] in query,
-      where: fragment(~s(? <@ (? #> '{"object","likes"}'\)), ^ap_id, activity.data)
+      where: fragment("(?)->'likes' \\? (?)", object.data, ^ap_id)
     )
   end
 
@@ -790,14 +791,20 @@ defp restrict_reblogs(query, _), do: query
 
   defp restrict_muted(query, %{"with_muted" => val}) when val in [true, "true", "1"], do: query
 
-  defp restrict_muted(query, %{"muting_user" => %User{info: info}}) do
+  defp restrict_muted(query, %{"muting_user" => %User{info: info}} = opts) do
     mutes = info.mutes
 
-    from(
+    query =
-      activity in query,
+      from([activity] in query,
-      where: fragment("not (? = ANY(?))", activity.actor, ^mutes),
+        where: fragment("not (? = ANY(?))", activity.actor, ^mutes),
-      where: fragment("not (?->'to' \\?| ?)", activity.data, ^mutes)
+        where: fragment("not (?->'to' \\?| ?)", activity.data, ^mutes)
-    )
+      )
+
+    unless opts["skip_preload"] do
+      from([thread_mute: tm] in query, where: is_nil(tm))
+    else
+      query
+    end
   end
 
   defp restrict_muted(query, _), do: query
@@ -834,7 +841,7 @@ defp restrict_unlisted(query) do
         fragment(
           "not (coalesce(?->'cc', '{}'::jsonb) \\?| ?)",
           activity.data,
-          ^["https://www.w3.org/ns/activitystreams#Public"]
+          ^[Pleroma.Constants.as_public()]
         )
     )
   end
@@ -874,6 +881,12 @@ defp exclude_poll_votes(query, _) do
     end
   end
 
+  defp exclude_id(query, %{"exclude_id" => id}) when is_binary(id) do
+    from(activity in query, where: activity.id != ^id)
+  end
+
+  defp exclude_id(query, _), do: query
+
   defp maybe_preload_objects(query, %{"skip_preload" => true}), do: query
 
   defp maybe_preload_objects(query, _) do
@@ -892,7 +905,7 @@ defp maybe_set_thread_muted_field(query, %{"skip_preload" => true}), do: query
 
   defp maybe_set_thread_muted_field(query, opts) do
     query
-    |> Activity.with_set_thread_muted_field(opts["user"])
+    |> Activity.with_set_thread_muted_field(opts["muting_user"] || opts["user"])
   end
 
   defp maybe_order(query, %{order: :desc}) do
@@ -971,7 +984,7 @@ def fetch_activities_bounded_query(query, recipients, recipients_with_public) do
       where:
         fragment("? && ?", activity.recipients, ^recipients) or
           (fragment("? && ?", activity.recipients, ^recipients_with_public) and
-             "https://www.w3.org/ns/activitystreams#Public" in activity.recipients)
+             ^Pleroma.Constants.as_public() in activity.recipients)
     )
   end
 
@@ -1010,16 +1023,23 @@ defp object_to_user_data(data) do
           "url" => [%{"href" => data["image"]["url"]}]
         }
 
+    fields =
+      data
+      |> Map.get("attachment", [])
+      |> Enum.filter(fn %{"type" => t} -> t == "PropertyValue" end)
+      |> Enum.map(fn fields -> Map.take(fields, ["name", "value"]) end)
+
     locked = data["manuallyApprovesFollowers"] || false
     data = Transmogrifier.maybe_fix_user_object(data)
 
     user_data = %{
       ap_id: data["id"],
       info: %{
-        "ap_enabled" => true,
+        ap_enabled: true,
-        "source_data" => data,
+        source_data: data,
-        "banner" => banner,
+        banner: banner,
-        "locked" => locked
+        fields: fields,
+        locked: locked
       },
       avatar: avatar,
       name: data["name"],
@@ -1043,6 +1063,71 @@ defp object_to_user_data(data) do
     {:ok, user_data}
   end
 
+  def fetch_follow_information_for_user(user) do
+    with {:ok, following_data} <-
+           Fetcher.fetch_and_contain_remote_object_from_id(user.following_address),
+         following_count when is_integer(following_count) <- following_data["totalItems"],
+         {:ok, hide_follows} <- collection_private(following_data),
+         {:ok, followers_data} <-
+           Fetcher.fetch_and_contain_remote_object_from_id(user.follower_address),
+         followers_count when is_integer(followers_count) <- followers_data["totalItems"],
+         {:ok, hide_followers} <- collection_private(followers_data) do
+      {:ok,
+       %{
+         hide_follows: hide_follows,
+         follower_count: followers_count,
+         following_count: following_count,
+         hide_followers: hide_followers
+       }}
+    else
+      {:error, _} = e ->
+        e
+
+      e ->
+        {:error, e}
+    end
+  end
+
+  defp maybe_update_follow_information(data) do
+    with {:enabled, true} <-
+           {:enabled, Pleroma.Config.get([:instance, :external_user_synchronization])},
+         {:ok, info} <- fetch_follow_information_for_user(data) do
+      info = Map.merge(data.info, info)
+      Map.put(data, :info, info)
+    else
+      {:enabled, false} ->
+        data
+
+      e ->
+        Logger.error(
+          "Follower/Following counter update for #{data.ap_id} failed.\n" <> inspect(e)
+        )
+
+        data
+    end
+  end
+
+  defp collection_private(data) do
+    if is_map(data["first"]) and
+         data["first"]["type"] in ["CollectionPage", "OrderedCollectionPage"] do
+      {:ok, false}
+    else
+      with {:ok, %{"type" => type}} when type in ["CollectionPage", "OrderedCollectionPage"] <-
+             Fetcher.fetch_and_contain_remote_object_from_id(data["first"]) do
+        {:ok, false}
+      else
+        {:error, {:ok, %{status: code}}} when code in [401, 403] ->
+          {:ok, true}
+
+        {:error, _} = e ->
+          e
+
+        e ->
+          {:error, e}
+      end
+    end
+  end
+
   def user_data_from_user_object(data) do
     with {:ok, data} <- MRF.filter(data),
          {:ok, data} <- object_to_user_data(data) do
@@ -1054,7 +1139,8 @@ def user_data_from_user_object(data) do
 
   def fetch_and_prepare_user_from_ap_id(ap_id) do
     with {:ok, data} <- Fetcher.fetch_and_contain_remote_object_from_id(ap_id),
-         {:ok, data} <- user_data_from_user_object(data) do
+         {:ok, data} <- user_data_from_user_object(data),
+         data <- maybe_update_follow_information(data) do
       {:ok, data}
     else
       e -> Logger.error("Could not decode user at fetch #{ap_id}, #{inspect(e)}")

lib/pleroma/web/activity_pub/mrf.ex

@@ -28,11 +28,43 @@ defp get_policies(_), do: []
 
   @spec subdomains_regex([String.t()]) :: [Regex.t()]
   def subdomains_regex(domains) when is_list(domains) do
-    for domain <- domains, do: ~r(^#{String.replace(domain, "*.", "(.*\\.)*")}$)
+    for domain <- domains, do: ~r(^#{String.replace(domain, "*.", "(.*\\.)*")}$)i
   end
 
   @spec subdomain_match?([Regex.t()], String.t()) :: boolean()
   def subdomain_match?(domains, host) do
     Enum.any?(domains, fn domain -> Regex.match?(domain, host) end)
   end
+
+  @callback describe() :: {:ok | :error, Map.t()}
+
+  def describe(policies) do
+    {:ok, policy_configs} =
+      policies
+      |> Enum.reduce({:ok, %{}}, fn
+        policy, {:ok, data} ->
+          {:ok, policy_data} = policy.describe()
+          {:ok, Map.merge(data, policy_data)}
+
+        _, error ->
+          error
+      end)
+
+    mrf_policies =
+      get_policies()
+      |> Enum.map(fn policy -> to_string(policy) |> String.split(".") |> List.last() end)
+
+    exclusions = Pleroma.Config.get([:instance, :mrf_transparency_exclusions])
+
+    base =
+      %{
+        mrf_policies: mrf_policies,
+        exclusions: length(exclusions) > 0
+      }
+      |> Map.merge(policy_configs)
+
+    {:ok, base}
+  end
+
+  def describe, do: get_policies() |> describe()
 end

lib/pleroma/web/activity_pub/mrf/anti_followbot_policy.ex

@@ -62,4 +62,7 @@ def filter(%{"type" => "Follow", "actor" => actor_id} = message) do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/anti_link_spam_policy.ex

@@ -5,6 +5,8 @@
 defmodule Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy do
   alias Pleroma.User
 
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
   require Logger
 
   # has the user successfully posted before?
@@ -22,6 +24,7 @@ defp contains_links?(%{"content" => content} = _object) do
 
   defp contains_links?(_), do: false
 
+  @impl true
   def filter(%{"type" => "Create", "actor" => actor, "object" => object} = message) do
     with {:ok, %User{} = u} <- User.get_or_fetch_by_ap_id(actor),
          {:contains_links, true} <- {:contains_links, contains_links?(object)},
@@ -45,4 +48,7 @@ def filter(%{"type" => "Create", "actor" => actor, "object" => object} = message
 
   # in all other cases, pass through
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/drop_policy.ex

@@ -12,4 +12,7 @@ def filter(object) do
     Logger.info("REJECTING #{inspect(object)}")
     {:reject, object}
   end
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/ensure_re_prepended.ex

@@ -39,4 +39,6 @@ def filter(%{"type" => "Create", "object" => child_object} = object) do
   end
 
   def filter(object), do: {:ok, object}
+
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/hellthread_policy.ex

@@ -4,6 +4,9 @@
 
 defmodule Pleroma.Web.ActivityPub.MRF.HellthreadPolicy do
   alias Pleroma.User
+
+  require Pleroma.Constants
+
   @moduledoc "Block messages with too much mentions (configurable)"
 
   @behaviour Pleroma.Web.ActivityPub.MRF
@@ -19,12 +22,12 @@ defp delist_message(message, threshold) when threshold > 0 do
         when follower_collection? and recipients > threshold ->
           message
           |> Map.put("to", [follower_collection])
-          |> Map.put("cc", ["https://www.w3.org/ns/activitystreams#Public"])
+          |> Map.put("cc", [Pleroma.Constants.as_public()])
 
         {:public, recipients} when recipients > threshold ->
           message
           |> Map.put("to", [])
-          |> Map.put("cc", ["https://www.w3.org/ns/activitystreams#Public"])
+          |> Map.put("cc", [Pleroma.Constants.as_public()])
 
         _ ->
           message
@@ -51,10 +54,10 @@ defp get_recipient_count(message) do
     recipients = (message["to"] || []) ++ (message["cc"] || [])
     follower_collection = User.get_cached_by_ap_id(message["actor"]).follower_address
 
-    if Enum.member?(recipients, "https://www.w3.org/ns/activitystreams#Public") do
+    if Enum.member?(recipients, Pleroma.Constants.as_public()) do
       recipients =
         recipients
-        |> List.delete("https://www.w3.org/ns/activitystreams#Public")
+        |> List.delete(Pleroma.Constants.as_public())
         |> List.delete(follower_collection)
 
       {:public, length(recipients)}
@@ -87,4 +90,8 @@ def filter(%{"type" => "Create"} = message) do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe,
+    do: {:ok, %{mrf_hellthread: Pleroma.Config.get(:mrf_hellthread) |> Enum.into(%{})}}
 end

lib/pleroma/web/activity_pub/mrf/keyword_policy.ex

@@ -3,6 +3,8 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.ActivityPub.MRF.KeywordPolicy do
+  require Pleroma.Constants
+
   @moduledoc "Reject or Word-Replace messages with a keyword or regex"
 
   @behaviour Pleroma.Web.ActivityPub.MRF
@@ -31,12 +33,12 @@ defp check_reject(%{"object" => %{"content" => content, "summary" => summary}} =
   defp check_ftl_removal(
          %{"to" => to, "object" => %{"content" => content, "summary" => summary}} = message
        ) do
-    if "https://www.w3.org/ns/activitystreams#Public" in to and
+    if Pleroma.Constants.as_public() in to and
          Enum.any?(Pleroma.Config.get([:mrf_keyword, :federated_timeline_removal]), fn pattern ->
            string_matches?(content, pattern) or string_matches?(summary, pattern)
          end) do
-      to = List.delete(to, "https://www.w3.org/ns/activitystreams#Public")
+      to = List.delete(to, Pleroma.Constants.as_public())
-      cc = ["https://www.w3.org/ns/activitystreams#Public" | message["cc"] || []]
+      cc = [Pleroma.Constants.as_public() | message["cc"] || []]
 
       message =
         message
@@ -94,4 +96,36 @@ def filter(%{"type" => "Create", "object" => %{"content" => _content}} = message
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe do
+    # This horror is needed to convert regex sigils to strings
+    mrf_keyword =
+      Pleroma.Config.get(:mrf_keyword, [])
+      |> Enum.map(fn {key, value} ->
+        {key,
+         Enum.map(value, fn
+           {pattern, replacement} ->
+             %{
+               "pattern" =>
+                 if not is_binary(pattern) do
+                   inspect(pattern)
+                 else
+                   pattern
+                 end,
+               "replacement" => replacement
+             }
+
+           pattern ->
+             if not is_binary(pattern) do
+               inspect(pattern)
+             else
+               pattern
+             end
+         end)}
+      end)
+      |> Enum.into(%{})
+
+    {:ok, %{mrf_keyword: mrf_keyword}}
+  end
 end

lib/pleroma/web/activity_pub/mrf/mediaproxy_warming_policy.ex

@@ -53,4 +53,7 @@ def filter(
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/mention_policy.ex

@@ -21,4 +21,7 @@ def filter(%{"type" => "Create"} = message) do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/no_placeholder_text_policy.ex

@@ -19,4 +19,7 @@ def filter(
 
   @impl true
   def filter(object), do: {:ok, object}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/noop_policy.ex

@@ -10,4 +10,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.NoOpPolicy do
   def filter(object) do
     {:ok, object}
   end
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/normalize_markup.ex

@@ -21,4 +21,6 @@ def filter(%{"type" => "Create", "object" => child_object} = object) do
   end
 
   def filter(object), do: {:ok, object}
+
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/reject_non_public.ex

@@ -10,7 +10,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.RejectNonPublic do
 
   @behaviour Pleroma.Web.ActivityPub.MRF
 
-  @public "https://www.w3.org/ns/activitystreams#Public"
+  require Pleroma.Constants
 
   @impl true
   def filter(%{"type" => "Create"} = object) do
@@ -19,8 +19,8 @@ def filter(%{"type" => "Create"} = object) do
     # Determine visibility
     visibility =
       cond do
-        @public in object["to"] -> "public"
+        Pleroma.Constants.as_public() in object["to"] -> "public"
-        @public in object["cc"] -> "unlisted"
+        Pleroma.Constants.as_public() in object["cc"] -> "unlisted"
         user.follower_address in object["to"] -> "followers"
         true -> "direct"
       end
@@ -44,4 +44,8 @@ def filter(%{"type" => "Create"} = object) do
 
   @impl true
   def filter(object), do: {:ok, object}
+
+  @impl true
+  def describe,
+    do: {:ok, %{mrf_rejectnonpublic: Pleroma.Config.get(:mrf_rejectnonpublic) |> Enum.into(%{})}}
 end

lib/pleroma/web/activity_pub/mrf/simple_policy.ex

@@ -8,6 +8,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
   @moduledoc "Filter activities depending on their origin instance"
   @behaviour MRF
 
+  require Pleroma.Constants
+
   defp check_accept(%{host: actor_host} = _actor_info, object) do
     accepts =
       Pleroma.Config.get([:mrf_simple, :accept])
@@ -89,14 +91,10 @@ defp check_ftl_removal(%{host: actor_host} = _actor_info, object) do
     object =
       with true <- MRF.subdomain_match?(timeline_removal, actor_host),
            user <- User.get_cached_by_ap_id(object["actor"]),
-           true <- "https://www.w3.org/ns/activitystreams#Public" in object["to"] do
+           true <- Pleroma.Constants.as_public() in object["to"] do
-        to =
+        to = List.delete(object["to"], Pleroma.Constants.as_public()) ++ [user.follower_address]
-          List.delete(object["to"], "https://www.w3.org/ns/activitystreams#Public") ++
-            [user.follower_address]
 
-        cc =
+        cc = List.delete(object["cc"], user.follower_address) ++ [Pleroma.Constants.as_public()]
-          List.delete(object["cc"], user.follower_address) ++
-            ["https://www.w3.org/ns/activitystreams#Public"]
 
         object
         |> Map.put("to", to)
@@ -179,4 +177,16 @@ def filter(%{"id" => actor, "type" => obj_type} = object)
   end
 
   def filter(object), do: {:ok, object}
+
+  @impl true
+  def describe do
+    exclusions = Pleroma.Config.get([:instance, :mrf_transparency_exclusions])
+
+    mrf_simple =
+      Pleroma.Config.get(:mrf_simple)
+      |> Enum.map(fn {k, v} -> {k, Enum.reject(v, fn v -> v in exclusions end)} end)
+      |> Enum.into(%{})
+
+    {:ok, %{mrf_simple: mrf_simple}}
+  end
 end

lib/pleroma/web/activity_pub/mrf/subchain_policy.ex

@@ -37,4 +37,7 @@ def filter(%{"actor" => actor} = message) do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/tag_policy.ex

@@ -19,7 +19,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.TagPolicy do
      - `mrf_tag:disable-any-subscription`: Reject any follow requests
   """
 
-  @public "https://www.w3.org/ns/activitystreams#Public"
+  require Pleroma.Constants
 
   defp get_tags(%User{tags: tags}) when is_list(tags), do: tags
   defp get_tags(_), do: []
@@ -70,9 +70,9 @@ defp process_tag(
        ) do
     user = User.get_cached_by_ap_id(actor)
 
-    if Enum.member?(to, @public) do
+    if Enum.member?(to, Pleroma.Constants.as_public()) do
-      to = List.delete(to, @public) ++ [user.follower_address]
+      to = List.delete(to, Pleroma.Constants.as_public()) ++ [user.follower_address]
-      cc = List.delete(cc, user.follower_address) ++ [@public]
+      cc = List.delete(cc, user.follower_address) ++ [Pleroma.Constants.as_public()]
 
       object =
         object
@@ -103,9 +103,10 @@ defp process_tag(
        ) do
     user = User.get_cached_by_ap_id(actor)
 
-    if Enum.member?(to, @public) or Enum.member?(cc, @public) do
+    if Enum.member?(to, Pleroma.Constants.as_public()) or
-      to = List.delete(to, @public) ++ [user.follower_address]
+         Enum.member?(cc, Pleroma.Constants.as_public()) do
-      cc = List.delete(cc, @public)
+      to = List.delete(to, Pleroma.Constants.as_public()) ++ [user.follower_address]
+      cc = List.delete(cc, Pleroma.Constants.as_public())
 
       object =
         object
@@ -164,4 +165,7 @@ def filter(%{"actor" => actor, "type" => "Create"} = message),
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/user_allowlist_policy.ex

@@ -32,4 +32,13 @@ def filter(%{"actor" => actor} = object) do
   end
 
   def filter(object), do: {:ok, object}
+
+  @impl true
+  def describe do
+    mrf_user_allowlist =
+      Config.get([:mrf_user_allowlist], [])
+      |> Enum.into(%{}, fn {k, v} -> {k, length(v)} end)
+
+    {:ok, %{mrf_user_allowlist: mrf_user_allowlist}}
+  end
 end

lib/pleroma/web/activity_pub/mrf/vocabulary_policy.ex

@@ -0,0 +1,37 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.MRF.VocabularyPolicy do
+  @moduledoc "Filter messages which belong to certain activity vocabularies"
+
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
+  def filter(%{"type" => "Undo", "object" => child_message} = message) do
+    with {:ok, _} <- filter(child_message) do
+      {:ok, message}
+    else
+      {:reject, nil} ->
+        {:reject, nil}
+    end
+  end
+
+  def filter(%{"type" => message_type} = message) do
+    with accepted_vocabulary <- Pleroma.Config.get([:mrf_vocabulary, :accept]),
+         rejected_vocabulary <- Pleroma.Config.get([:mrf_vocabulary, :reject]),
+         true <-
+           length(accepted_vocabulary) == 0 || Enum.member?(accepted_vocabulary, message_type),
+         false <-
+           length(rejected_vocabulary) > 0 && Enum.member?(rejected_vocabulary, message_type),
+         {:ok, _} <- filter(message["object"]) do
+      {:ok, message}
+    else
+      _ -> {:reject, nil}
+    end
+  end
+
+  def filter(message), do: {:ok, message}
+
+  def describe,
+    do: {:ok, %{mrf_vocabulary: Pleroma.Config.get(:mrf_vocabulary) |> Enum.into(%{})}}
+end

lib/pleroma/web/activity_pub/publisher.ex

@@ -11,6 +11,8 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
   alias Pleroma.Web.ActivityPub.Relay
   alias Pleroma.Web.ActivityPub.Transmogrifier
 
+  require Pleroma.Constants
+
   import Pleroma.Web.ActivityPub.Visibility
 
   @behaviour Pleroma.Web.Federator.Publisher
@@ -44,7 +46,7 @@ def is_representable?(%Activity{} = activity) do
   """
   def publish_one(%{inbox: inbox, json: json, actor: %User{} = actor, id: id} = params) do
     Logger.info("Federating #{id} to #{inbox}")
-    host = URI.parse(inbox).host
+    %{host: host, path: path} = URI.parse(inbox)
 
     digest = "SHA-256=" <> (:crypto.hash(:sha256, json) |> Base.encode64())
 
@@ -54,6 +56,7 @@ def publish_one(%{inbox: inbox, json: json, actor: %User{} = actor, id: id} = pa
 
     signature =
       Pleroma.Signature.sign(actor, %{
+        "(request-target)": "post #{path}",
         host: host,
         "content-length": byte_size(json),
         digest: digest,
@@ -117,8 +120,6 @@ defp get_cc_ap_ids(ap_id, recipients) do
     |> Enum.map(& &1.ap_id)
   end
 
-  @as_public "https://www.w3.org/ns/activitystreams#Public"
-
   defp maybe_use_sharedinbox(%User{info: %{source_data: data}}),
     do: (is_map(data["endpoints"]) && Map.get(data["endpoints"], "sharedInbox")) || data["inbox"]
 
@@ -145,7 +146,7 @@ def determine_inbox(
       type == "Delete" ->
         maybe_use_sharedinbox(user)
 
-      @as_public in to || @as_public in cc ->
+      Pleroma.Constants.as_public() in to || Pleroma.Constants.as_public() in cc ->
         maybe_use_sharedinbox(user)
 
       length(to) + length(cc) > 1 ->

lib/pleroma/web/activity_pub/relay.ex

@@ -14,6 +14,7 @@ def get_actor do
     |> User.get_or_create_service_actor_by_ap_id()
   end
 
+  @spec follow(String.t()) :: {:ok, Activity.t()} | {:error, any()}
   def follow(target_instance) do
     with %User{} = local_user <- get_actor(),
          {:ok, %User{} = target_user} <- User.get_or_fetch_by_ap_id(target_instance),
@@ -21,12 +22,17 @@ def follow(target_instance) do
       Logger.info("relay: followed instance: #{target_instance}; id=#{activity.data["id"]}")
       {:ok, activity}
     else
+      {:error, _} = error ->
+        Logger.error("error: #{inspect(error)}")
+        error
+
       e ->
         Logger.error("error: #{inspect(e)}")
         {:error, e}
     end
   end
 
+  @spec unfollow(String.t()) :: {:ok, Activity.t()} | {:error, any()}
   def unfollow(target_instance) do
     with %User{} = local_user <- get_actor(),
          {:ok, %User{} = target_user} <- User.get_or_fetch_by_ap_id(target_instance),
@@ -34,20 +40,27 @@ def unfollow(target_instance) do
       Logger.info("relay: unfollowed instance: #{target_instance}: id=#{activity.data["id"]}")
       {:ok, activity}
     else
+      {:error, _} = error ->
+        Logger.error("error: #{inspect(error)}")
+        error
+
       e ->
         Logger.error("error: #{inspect(e)}")
         {:error, e}
     end
   end
 
+  @spec publish(any()) :: {:ok, Activity.t(), Object.t()} | {:error, any()}
   def publish(%Activity{data: %{"type" => "Create"}} = activity) do
     with %User{} = user <- get_actor(),
          %Object{} = object <- Object.normalize(activity) do
       ActivityPub.announce(user, object, nil, true, false)
     else
-      e -> Logger.error("error: #{inspect(e)}")
+      e ->
+        Logger.error("error: #{inspect(e)}")
+        {:error, inspect(e)}
     end
   end
 
-  def publish(_), do: nil
+  def publish(_), do: {:error, "Not implemented"}
 end

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -19,12 +19,14 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   import Ecto.Query
 
   require Logger
+  require Pleroma.Constants
 
   @doc """
   Modifies an incoming AP object (mastodon format) to our internal format.
   """
   def fix_object(object, options \\ []) do
     object
+    |> strip_internal_fields
     |> fix_actor
     |> fix_url
     |> fix_attachments
@@ -33,7 +35,6 @@ def fix_object(object, options \\ []) do
     |> fix_emoji
     |> fix_tag
     |> fix_content_map
-    |> fix_likes
     |> fix_addressing
     |> fix_summary
     |> fix_type(options)
@@ -102,8 +103,7 @@ def fix_explicit_addressing(object) do
 
     follower_collection = User.get_cached_by_ap_id(Containment.get_actor(object)).follower_address
 
-    explicit_mentions =
+    explicit_mentions = explicit_mentions ++ [Pleroma.Constants.as_public(), follower_collection]
-      explicit_mentions ++ ["https://www.w3.org/ns/activitystreams#Public", follower_collection]
 
     fix_explicit_addressing(object, explicit_mentions, follower_collection)
   end
@@ -115,11 +115,11 @@ def fix_implicit_addressing(%{"to" => to, "cc" => cc} = object, followers_collec
 
     if followers_collection not in recipients do
       cond do
-        "https://www.w3.org/ns/activitystreams#Public" in cc ->
+        Pleroma.Constants.as_public() in cc ->
           to = to ++ [followers_collection]
           Map.put(object, "to", to)
 
-        "https://www.w3.org/ns/activitystreams#Public" in to ->
+        Pleroma.Constants.as_public() in to ->
           cc = cc ++ [followers_collection]
           Map.put(object, "cc", cc)
 
@@ -151,20 +151,6 @@ def fix_actor(%{"attributedTo" => actor} = object) do
     |> Map.put("actor", Containment.get_actor(%{"actor" => actor}))
   end
 
-  # Check for standardisation
-  # This is what Peertube does
-  # curl -H 'Accept: application/activity+json' $likes | jq .totalItems
-  # Prismo returns only an integer (count) as "likes"
-  def fix_likes(%{"likes" => likes} = object) when not is_map(likes) do
-    object
-    |> Map.put("likes", [])
-    |> Map.put("like_count", 0)
-  end
-
-  def fix_likes(object) do
-    object
-  end
-
   def fix_in_reply_to(object, options \\ [])
 
   def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
@@ -347,13 +333,15 @@ def fix_content_map(object), do: object
 
   def fix_type(object, options \\ [])
 
-  def fix_type(%{"inReplyTo" => reply_id} = object, options) when is_binary(reply_id) do
+  def fix_type(%{"inReplyTo" => reply_id, "name" => _} = object, options)
+      when is_binary(reply_id) do
     reply =
-      if Federator.allowed_incoming_reply_depth?(options[:depth]) do
+      with true <- Federator.allowed_incoming_reply_depth?(options[:depth]),
-        Object.normalize(reply_id, true)
+           {:ok, object} <- get_obj_helper(reply_id, options) do
+        object
       end
 
-    if reply && (reply.data["type"] == "Question" and object["name"]) do
+    if reply && reply.data["type"] == "Question" do
       Map.put(object, "type", "Answer")
     else
       object
@@ -480,8 +468,7 @@ def handle_incoming(
          {:ok, %User{} = follower} <- User.get_or_fetch_by_ap_id(follower),
          {:ok, activity} <- ActivityPub.follow(follower, followed, id, false) do
       with deny_follow_blocked <- Pleroma.Config.get([:user, :deny_follow_blocked]),
-           {_, false} <-
+           {_, false} <- {:user_blocked, User.blocks?(followed, follower) && deny_follow_blocked},
-             {:user_blocked, User.blocks?(followed, follower) && deny_follow_blocked},
            {_, false} <- {:user_locked, User.locked?(followed)},
            {_, {:ok, follower}} <- {:follow, User.follow(follower, followed)},
            {_, {:ok, _}} <-
@@ -609,16 +596,22 @@ def handle_incoming(
     with %User{ap_id: ^actor_id} = actor <- User.get_cached_by_ap_id(object["id"]) do
       {:ok, new_user_data} = ActivityPub.user_data_from_user_object(object)
 
-      banner = new_user_data[:info]["banner"]
+      banner = new_user_data[:info][:banner]
-      locked = new_user_data[:info]["locked"] || false
+      locked = new_user_data[:info][:locked] || false
+      attachment = get_in(new_user_data, [:info, :source_data, "attachment"]) || []
+
+      fields =
+        attachment
+        |> Enum.filter(fn %{"type" => t} -> t == "PropertyValue" end)
+        |> Enum.map(fn fields -> Map.take(fields, ["name", "value"]) end)
 
       update_data =
         new_user_data
         |> Map.take([:name, :bio, :avatar])
-        |> Map.put(:info, %{"banner" => banner, "locked" => locked})
+        |> Map.put(:info, %{banner: banner, locked: locked, fields: fields})
 
       actor
-      |> User.upgrade_changeset(update_data)
+      |> User.upgrade_changeset(update_data, true)
       |> User.update_and_set_cache()
 
       ActivityPub.update(%{
@@ -656,20 +649,7 @@ def handle_incoming(
       nil ->
         case User.get_cached_by_ap_id(object_id) do
           %User{ap_id: ^actor} = user ->
-            {:ok, followers} = User.get_followers(user)
+            User.delete(user)
-
-            Enum.each(followers, fn follower ->
-              User.unfollow(follower, user)
-            end)
-
-            {:ok, friends} = User.get_friends(user)
-
-            Enum.each(friends, fn followed ->
-              User.unfollow(user, followed)
-            end)
-
-            User.invalidate_cache(user)
-            Repo.delete(user)
 
           nil ->
             :error
@@ -727,8 +707,7 @@ def handle_incoming(
         } = _data,
         _options
       ) do
-    with true <- Pleroma.Config.get([:activitypub, :accept_blocks]),
+    with %User{local: true} = blocked <- User.get_cached_by_ap_id(blocked),
-         %User{local: true} = blocked <- User.get_cached_by_ap_id(blocked),
          {:ok, %User{} = blocker} <- User.get_or_fetch_by_ap_id(blocker),
          {:ok, activity} <- ActivityPub.unblock(blocker, blocked, id, false) do
       User.unblock(blocker, blocked)
@@ -742,8 +721,7 @@ def handle_incoming(
         %{"type" => "Block", "object" => blocked, "actor" => blocker, "id" => id} = _data,
         _options
       ) do
-    with true <- Pleroma.Config.get([:activitypub, :accept_blocks]),
+    with %User{local: true} = blocked = User.get_cached_by_ap_id(blocked),
-         %User{local: true} = blocked = User.get_cached_by_ap_id(blocked),
          {:ok, %User{} = blocker} = User.get_or_fetch_by_ap_id(blocker),
          {:ok, activity} <- ActivityPub.block(blocker, blocked, id, false) do
       User.unfollow(blocker, blocked)
@@ -798,7 +776,6 @@ def prepare_object(object) do
     |> add_mention_tags
     |> add_emoji_tags
     |> add_attributed_to
-    |> add_likes
     |> prepare_attachments
     |> set_conversation
     |> set_reply_to_uri
@@ -985,22 +962,6 @@ def add_attributed_to(object) do
     |> Map.put("attributedTo", attributed_to)
   end
 
-  def add_likes(%{"id" => id, "like_count" => likes} = object) do
-    likes = %{
-      "id" => "#{id}/likes",
-      "first" => "#{id}/likes?page=1",
-      "type" => "OrderedCollection",
-      "totalItems" => likes
-    }
-
-    object
-    |> Map.put("likes", likes)
-  end
-
-  def add_likes(object) do
-    object
-  end
-
   def prepare_attachments(object) do
     attachments =
       (object["attachment"] || [])
@@ -1016,6 +977,7 @@ def prepare_attachments(object) do
   defp strip_internal_fields(object) do
     object
     |> Map.drop([
+      "likes",
       "like_count",
       "announcements",
       "announcement_count",
@@ -1090,10 +1052,6 @@ def upgrade_user_from_ap_id(ap_id) do
         PleromaJobQueue.enqueue(:transmogrifier, __MODULE__, [:user_upgrade, user])
       end
 
-      if Pleroma.Config.get([:instance, :external_user_synchronization]) do
-        update_following_followers_counters(user)
-      end
-
       {:ok, user}
     else
       %User{} = user -> {:ok, user}
@@ -1126,27 +1084,4 @@ def maybe_fix_user_object(data) do
     data
     |> maybe_fix_user_url
   end
-
-  def update_following_followers_counters(user) do
-    info = %{}
-
-    following = fetch_counter(user.following_address)
-    info = if following, do: Map.put(info, :following_count, following), else: info
-
-    followers = fetch_counter(user.follower_address)
-    info = if followers, do: Map.put(info, :follower_count, followers), else: info
-
-    User.set_info_cache(user, info)
-  end
-
-  defp fetch_counter(url) do
-    with {:ok, %{body: body, status: code}} when code in 200..299 <-
-           Pleroma.HTTP.get(
-             url,
-             [{:Accept, "application/activity+json"}]
-           ),
-         {:ok, data} <- Jason.decode(body) do
-      data["totalItems"]
-    end
-  end
 end

lib/pleroma/web/activity_pub/utils.ex

@@ -18,6 +18,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
   import Ecto.Query
 
   require Logger
+  require Pleroma.Constants
 
   @supported_object_types ["Article", "Note", "Video", "Page", "Question", "Answer"]
   @supported_report_states ~w(open closed resolved)
@@ -250,20 +251,6 @@ def insert_full_object(%{"object" => %{"type" => type} = object_data} = map)
 
   def insert_full_object(map), do: {:ok, map, nil}
 
-  def update_object_in_activities(%{data: %{"id" => id}} = object) do
-    # TODO
-    # Update activities that already had this. Could be done in a seperate process.
-    # Alternatively, just don't do this and fetch the current object each time. Most
-    # could probably be taken from cache.
-    relevant_activities = Activity.get_all_create_by_object_ap_id(id)
-
-    Enum.map(relevant_activities, fn activity ->
-      new_activity_data = activity.data |> Map.put("object", object.data)
-      changeset = Changeset.change(activity, data: new_activity_data)
-      Repo.update(changeset)
-    end)
-  end
-
   #### Like-related helpers
 
   @doc """
@@ -346,8 +333,7 @@ def update_element_in_object(property, element, object) do
            |> Map.put("#{property}_count", length(element))
            |> Map.put("#{property}s", element),
          changeset <- Changeset.change(object, data: new_data),
-         {:ok, object} <- Object.update_and_set_cache(changeset),
+         {:ok, object} <- Object.update_and_set_cache(changeset) do
-         _ <- update_object_in_activities(object) do
       {:ok, object}
     end
   end
@@ -388,6 +374,7 @@ def update_follow_state_for_all(
         [state, actor, object]
       )
 
+      User.set_follow_state_cache(actor, object, state)
       activity = Activity.get_by_id(activity.id)
       {:ok, activity}
     rescue
@@ -396,12 +383,16 @@ def update_follow_state_for_all(
     end
   end
 
-  def update_follow_state(%Activity{} = activity, state) do
+  def update_follow_state(
+        %Activity{data: %{"actor" => actor, "object" => object}} = activity,
+        state
+      ) do
     with new_data <-
            activity.data
            |> Map.put("state", state),
          changeset <- Changeset.change(activity, data: new_data),
-         {:ok, activity} <- Repo.update(changeset) do
+         {:ok, activity} <- Repo.update(changeset),
+         _ <- User.set_follow_state_cache(actor, object, state) do
       {:ok, activity}
     end
   end
@@ -418,7 +409,7 @@ def make_follow_data(
       "type" => "Follow",
       "actor" => follower_id,
       "to" => [followed_id],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "object" => followed_id,
       "state" => "pending"
     }
@@ -510,7 +501,7 @@ def make_announce_data(
       "actor" => ap_id,
       "object" => id,
       "to" => [user.follower_address, object.data["actor"]],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "context" => object.data["context"]
     }
 
@@ -530,7 +521,7 @@ def make_unannounce_data(
       "actor" => ap_id,
       "object" => activity.data,
       "to" => [user.follower_address, activity.data["actor"]],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "context" => context
     }
 
@@ -547,7 +538,7 @@ def make_unlike_data(
       "actor" => ap_id,
       "object" => activity.data,
       "to" => [user.follower_address, activity.data["actor"]],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "context" => context
     }
 
@@ -556,7 +547,7 @@ def make_unlike_data(
 
   def add_announce_to_object(
         %Activity{
-          data: %{"actor" => actor, "cc" => ["https://www.w3.org/ns/activitystreams#Public"]}
+          data: %{"actor" => actor, "cc" => [Pleroma.Constants.as_public()]}
         },
         object
       ) do
@@ -765,7 +756,7 @@ defp get_updated_targets(
        ) do
     cc = Map.get(data, "cc", [])
     follower_address = User.get_cached_by_ap_id(data["actor"]).follower_address
-    public = "https://www.w3.org/ns/activitystreams#Public"
+    public = Pleroma.Constants.as_public()
 
     case visibility do
       "public" ->

lib/pleroma/web/activity_pub/views/object_view.ex

@@ -66,8 +66,10 @@ def collection(collection, iri, page) do
       "orderedItems" => items
     }
 
-    if offset < total do
+    if offset + length(items) < total do
       Map.put(map, "next", "#{iri}?page=#{page + 1}")
+    else
+      map
     end
   end
 end

lib/pleroma/web/activity_pub/views/user_view.ex

@@ -65,7 +65,7 @@ def render("user.json", %{user: %User{nickname: nil} = user}),
     do: render("service.json", %{user: user})
 
   def render("user.json", %{user: %User{nickname: "internal." <> _} = user}),
-    do: render("service.json", %{user: user})
+    do: render("service.json", %{user: user}) |> Map.put("preferredUsername", user.nickname)
 
   def render("user.json", %{user: user}) do
     {:ok, user} = User.ensure_keys_present(user)
@@ -80,6 +80,17 @@ def render("user.json", %{user: user}) do
       |> Transmogrifier.add_emoji_tags()
       |> Map.get("tag", [])
 
+    fields =
+      user.info
+      |> User.Info.fields()
+      |> Enum.map(fn %{"name" => name, "value" => value} ->
+        %{
+          "name" => Pleroma.HTML.strip_tags(name),
+          "value" => Pleroma.HTML.filter_tags(value, Pleroma.HTML.Scrubber.LinksOnly)
+        }
+      end)
+      |> Enum.map(&Map.put(&1, "type", "PropertyValue"))
+
     %{
       "id" => user.ap_id,
       "type" => "Person",
@@ -98,6 +109,7 @@ def render("user.json", %{user: user}) do
         "publicKeyPem" => public_key
       },
       "endpoints" => endpoints,
+      "attachment" => fields,
       "tag" => (user.info.source_data["tag"] || []) ++ user_tags
     }
     |> Map.merge(maybe_make_image(&User.avatar_url/2, "icon", user))

lib/pleroma/web/activity_pub/visibility.ex

@@ -8,14 +8,14 @@ defmodule Pleroma.Web.ActivityPub.Visibility do
   alias Pleroma.Repo
   alias Pleroma.User
 
-  @public "https://www.w3.org/ns/activitystreams#Public"
+  require Pleroma.Constants
 
   @spec is_public?(Object.t() | Activity.t() | map()) :: boolean()
   def is_public?(%Object{data: %{"type" => "Tombstone"}}), do: false
   def is_public?(%Object{data: data}), do: is_public?(data)
   def is_public?(%Activity{data: data}), do: is_public?(data)
   def is_public?(%{"directMessage" => true}), do: false
-  def is_public?(data), do: @public in (data["to"] ++ (data["cc"] || []))
+  def is_public?(data), do: Pleroma.Constants.as_public() in (data["to"] ++ (data["cc"] || []))
 
   def is_private?(activity) do
     with false <- is_public?(activity),
@@ -73,10 +73,10 @@ def get_visibility(object) do
     cc = object.data["cc"] || []
 
     cond do
-      @public in to ->
+      Pleroma.Constants.as_public() in to ->
         "public"
 
-      @public in cc ->
+      Pleroma.Constants.as_public() in cc ->
         "unlisted"
 
       # this should use the sql for the object's activity

lib/pleroma/web/admin_api/admin_api_controller.ex

@@ -379,6 +379,16 @@ def status_delete(%{assigns: %{user: user}} = conn, %{"id" => id}) do
     end
   end
 
+  def migrate_to_db(conn, _params) do
+    Mix.Tasks.Pleroma.Config.run(["migrate_to_db"])
+    json(conn, %{})
+  end
+
+  def migrate_from_db(conn, _params) do
+    Mix.Tasks.Pleroma.Config.run(["migrate_from_db", Pleroma.Config.get(:env), "true"])
+    json(conn, %{})
+  end
+
   def config_show(conn, _params) do
     configs = Pleroma.Repo.all(Config)
 
@@ -392,9 +402,9 @@ def config_update(conn, %{"configs" => configs}) do
       if Pleroma.Config.get([:instance, :dynamic_configuration]) do
         updated =
           Enum.map(configs, fn
-            %{"group" => group, "key" => key, "delete" => "true"} ->
+            %{"group" => group, "key" => key, "delete" => "true"} = params ->
-              {:ok, _} = Config.delete(%{group: group, key: key})
+              {:ok, config} = Config.delete(%{group: group, key: key, subkeys: params["subkeys"]})
-              nil
+              config
 
             %{"group" => group, "key" => key, "value" => value} ->
               {:ok, config} = Config.update_or_create(%{group: group, key: key, value: value})

lib/pleroma/web/admin_api/config.ex

@@ -55,8 +55,19 @@ def update_or_create(params) do
 
   @spec delete(map()) :: {:ok, Config.t()} | {:error, Changeset.t()}
   def delete(params) do
-    with %Config{} = config <- Config.get_by_params(params) do
+    with %Config{} = config <- Config.get_by_params(Map.delete(params, :subkeys)) do
-      Repo.delete(config)
+      if params[:subkeys] do
+        updated_value =
+          Keyword.drop(
+            :erlang.binary_to_term(config.value),
+            Enum.map(params[:subkeys], &do_transform_string(&1))
+          )
+
+        Config.update(config, %{value: updated_value})
+      else
+        Repo.delete(config)
+        {:ok, nil}
+      end
     else
       nil ->
         err =

lib/pleroma/web/auth/authenticator.ex

@@ -21,8 +21,7 @@ def get_user(plug), do: implementation().get_user(plug)
   def create_from_registration(plug, registration),
     do: implementation().create_from_registration(plug, registration)
 
-  @callback get_registration(Plug.Conn.t()) ::
+  @callback get_registration(Plug.Conn.t()) :: {:ok, Registration.t()} | {:error, any()}
-              {:ok, Registration.t()} | {:error, any()}
   def get_registration(plug), do: implementation().get_registration(plug)
 
   @callback handle_error(Plug.Conn.t(), any()) :: any()

lib/pleroma/web/chat_channel.ex

@@ -33,9 +33,11 @@ def handle_in("new_msg", %{"text" => text}, %{assigns: %{user_name: user_name}}
 end
 
 defmodule Pleroma.Web.ChatChannel.ChatChannelState do
+  use Agent
+
   @max_messages 20
 
-  def start_link do
+  def start_link(_) do
     Agent.start_link(fn -> %{max_id: 1, messages: []} end, name: __MODULE__)
   end
 

lib/pleroma/web/common_api/common_api.ex

@@ -5,6 +5,7 @@
 defmodule Pleroma.Web.CommonAPI do
   alias Pleroma.Activity
   alias Pleroma.ActivityExpiration
+  alias Pleroma.Conversation.Participation
   alias Pleroma.Formatter
   alias Pleroma.Object
   alias Pleroma.ThreadMute
@@ -172,21 +173,25 @@ defp normalize_and_validate_choice_indices(choices, count) do
     end)
   end
 
-  def get_visibility(%{"visibility" => visibility}, in_reply_to)
+  def get_visibility(_, _, %Participation{}) do
+    {"direct", "direct"}
+  end
+
+  def get_visibility(%{"visibility" => visibility}, in_reply_to, _)
       when visibility in ~w{public unlisted private direct},
       do: {visibility, get_replied_to_visibility(in_reply_to)}
 
-  def get_visibility(%{"visibility" => "list:" <> list_id}, in_reply_to) do
+  def get_visibility(%{"visibility" => "list:" <> list_id}, in_reply_to, _) do
     visibility = {:list, String.to_integer(list_id)}
     {visibility, get_replied_to_visibility(in_reply_to)}
   end
 
-  def get_visibility(_, in_reply_to) when not is_nil(in_reply_to) do
+  def get_visibility(_, in_reply_to, _) when not is_nil(in_reply_to) do
     visibility = get_replied_to_visibility(in_reply_to)
     {visibility, visibility}
   end
 
-  def get_visibility(_, in_reply_to), do: {"public", get_replied_to_visibility(in_reply_to)}
+  def get_visibility(_, in_reply_to, _), do: {"public", get_replied_to_visibility(in_reply_to)}
 
   def get_replied_to_visibility(nil), do: nil
 
@@ -212,7 +217,9 @@ def post(user, %{"status" => status} = data) do
     with status <- String.trim(status),
          attachments <- attachments_from_ids(data),
          in_reply_to <- get_replied_to_activity(data["in_reply_to_status_id"]),
-         {visibility, in_reply_to_visibility} <- get_visibility(data, in_reply_to),
+         in_reply_to_conversation <- Participation.get(data["in_reply_to_conversation_id"]),
+         {visibility, in_reply_to_visibility} <-
+           get_visibility(data, in_reply_to, in_reply_to_conversation),
          {_, false} <-
            {:private_to_public, in_reply_to_visibility == "direct" && visibility != "direct"},
          {content_html, mentions, tags} <-
@@ -225,8 +232,9 @@ def post(user, %{"status" => status} = data) do
          mentioned_users <- for({_, mentioned_user} <- mentions, do: mentioned_user.ap_id),
          addressed_users <- get_addressed_users(mentioned_users, data["to"]),
          {poll, poll_emoji} <- make_poll_data(data),
-         {to, cc} <- get_to_and_cc(user, addressed_users, in_reply_to, visibility),
+         {to, cc} <-
-         context <- make_context(in_reply_to),
+           get_to_and_cc(user, addressed_users, in_reply_to, visibility, in_reply_to_conversation),
+         context <- make_context(in_reply_to, in_reply_to_conversation),
          cw <- data["spoiler_text"] || "",
          sensitive <- data["sensitive"] || Enum.member?(tags, {"#nsfw", "nsfw"}),
          {:ok, expires_at} <- check_expiry_date(data["expires_at"]),
@@ -321,8 +329,7 @@ def pin(id_or_ap_id, %{ap_id: user_ap_id} = user) do
            }
          } = activity <- get_by_id_or_ap_id(id_or_ap_id),
          true <- Visibility.is_public?(activity),
-         %{valid?: true} = info_changeset <-
+         %{valid?: true} = info_changeset <- User.Info.add_pinnned_activity(user.info, activity),
-           User.Info.add_pinnned_activity(user.info, activity),
          changeset <-
            Ecto.Changeset.change(user) |> Ecto.Changeset.put_embed(:info, info_changeset),
          {:ok, _user} <- User.update_and_set_cache(changeset) do

lib/pleroma/web/common_api/utils.ex

@@ -8,6 +8,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   alias Calendar.Strftime
   alias Pleroma.Activity
   alias Pleroma.Config
+  alias Pleroma.Conversation.Participation
   alias Pleroma.Formatter
   alias Pleroma.Object
   alias Pleroma.Plugs.AuthenticationPlug
@@ -19,11 +20,17 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   alias Pleroma.Web.MediaProxy
 
   require Logger
+  require Pleroma.Constants
 
   # This is a hack for twidere.
   def get_by_id_or_ap_id(id) do
     activity =
-      Activity.get_by_id_with_object(id) || Activity.get_create_by_object_ap_id_with_object(id)
+      with true <- Pleroma.FlakeId.is_flake_id?(id),
+           %Activity{} = activity <- Activity.get_by_id_with_object(id) do
+        activity
+      else
+        _ -> Activity.get_create_by_object_ap_id_with_object(id)
+      end
 
     activity &&
       if activity.data["type"] == "Create" do
@@ -41,32 +48,61 @@ def get_replied_to_activity(id) when not is_nil(id) do
 
   def get_replied_to_activity(_), do: nil
 
-  def attachments_from_ids(data) do
+  def attachments_from_ids(%{"media_ids" => ids, "descriptions" => desc} = _) do
-    if Map.has_key?(data, "descriptions") do
+    attachments_from_ids_descs(ids, desc)
-      attachments_from_ids_descs(data["media_ids"], data["descriptions"])
-    else
-      attachments_from_ids_no_descs(data["media_ids"])
-    end
   end
 
-  def attachments_from_ids_no_descs(ids) do
+  def attachments_from_ids(%{"media_ids" => ids} = _) do
-    Enum.map(ids || [], fn media_id ->
+    attachments_from_ids_no_descs(ids)
-      Repo.get(Object, media_id).data
-    end)
   end
 
+  def attachments_from_ids(_), do: []
+
+  def attachments_from_ids_no_descs([]), do: []
+
+  def attachments_from_ids_no_descs(ids) do
+    Enum.map(ids, fn media_id ->
+      case Repo.get(Object, media_id) do
+        %Object{data: data} = _ -> data
+        _ -> nil
+      end
+    end)
+    |> Enum.filter(& &1)
+  end
+
+  def attachments_from_ids_descs([], _), do: []
+
   def attachments_from_ids_descs(ids, descs_str) do
     {_, descs} = Jason.decode(descs_str)
 
-    Enum.map(ids || [], fn media_id ->
+    Enum.map(ids, fn media_id ->
-      Map.put(Repo.get(Object, media_id).data, "name", descs[media_id])
+      case Repo.get(Object, media_id) do
+        %Object{data: data} = _ ->
+          Map.put(data, "name", descs[media_id])
+
+        _ ->
+          nil
+      end
     end)
+    |> Enum.filter(& &1)
   end
 
-  @spec get_to_and_cc(User.t(), list(String.t()), Activity.t() | nil, String.t()) ::
+  @spec get_to_and_cc(
+          User.t(),
+          list(String.t()),
+          Activity.t() | nil,
+          String.t(),
+          Participation.t() | nil
+        ) ::
           {list(String.t()), list(String.t())}
-  def get_to_and_cc(user, mentioned_users, inReplyTo, "public") do
+
-    to = ["https://www.w3.org/ns/activitystreams#Public" | mentioned_users]
+  def get_to_and_cc(_, _, _, _, %Participation{} = participation) do
+    participation = Repo.preload(participation, :recipients)
+    {Enum.map(participation.recipients, & &1.ap_id), []}
+  end
+
+  def get_to_and_cc(user, mentioned_users, inReplyTo, "public", _) do
+    to = [Pleroma.Constants.as_public() | mentioned_users]
     cc = [user.follower_address]
 
     if inReplyTo do
@@ -76,9 +112,9 @@ def get_to_and_cc(user, mentioned_users, inReplyTo, "public") do
     end
   end
 
-  def get_to_and_cc(user, mentioned_users, inReplyTo, "unlisted") do
+  def get_to_and_cc(user, mentioned_users, inReplyTo, "unlisted", _) do
     to = [user.follower_address | mentioned_users]
-    cc = ["https://www.w3.org/ns/activitystreams#Public"]
+    cc = [Pleroma.Constants.as_public()]
 
     if inReplyTo do
       {Enum.uniq([inReplyTo.data["actor"] | to]), cc}
@@ -87,12 +123,12 @@ def get_to_and_cc(user, mentioned_users, inReplyTo, "unlisted") do
     end
   end
 
-  def get_to_and_cc(user, mentioned_users, inReplyTo, "private") do
+  def get_to_and_cc(user, mentioned_users, inReplyTo, "private", _) do
-    {to, cc} = get_to_and_cc(user, mentioned_users, inReplyTo, "direct")
+    {to, cc} = get_to_and_cc(user, mentioned_users, inReplyTo, "direct", nil)
     {[user.follower_address | to], cc}
   end
 
-  def get_to_and_cc(_user, mentioned_users, inReplyTo, "direct") do
+  def get_to_and_cc(_user, mentioned_users, inReplyTo, "direct", _) do
     if inReplyTo do
       {Enum.uniq([inReplyTo.data["actor"] | mentioned_users]), []}
     else
@@ -100,7 +136,7 @@ def get_to_and_cc(_user, mentioned_users, inReplyTo, "direct") do
     end
   end
 
-  def get_to_and_cc(_user, mentions, _inReplyTo, {:list, _}), do: {mentions, []}
+  def get_to_and_cc(_user, mentions, _inReplyTo, {:list, _}, _), do: {mentions, []}
 
   def get_addressed_users(_, to) when is_list(to) do
     User.get_ap_ids_by_nicknames(to)
@@ -230,8 +266,12 @@ defp maybe_add_nsfw_tag({text, mentions, tags}, %{"sensitive" => sensitive})
 
   defp maybe_add_nsfw_tag(data, _), do: data
 
-  def make_context(%Activity{data: %{"context" => context}}), do: context
+  def make_context(_, %Participation{} = participation) do
-  def make_context(_), do: Utils.generate_context_id()
+    Repo.preload(participation, :conversation).conversation.ap_id
+  end
+
+  def make_context(%Activity{data: %{"context" => context}}, _), do: context
+  def make_context(_, _), do: Utils.generate_context_id()
 
   def maybe_add_attachments(parsed, _attachments, true = _no_links), do: parsed
 
@@ -241,20 +281,18 @@ def maybe_add_attachments({text, mentions, tags}, attachments, _no_links) do
   end
 
   def add_attachments(text, attachments) do
-    attachment_text =
+    attachment_text = Enum.map(attachments, &build_attachment_link/1)
-      Enum.map(attachments, fn
-        %{"url" => [%{"href" => href} | _]} = attachment ->
-          name = attachment["name"] || URI.decode(Path.basename(href))
-          href = MediaProxy.url(href)
-          "<a href=\"#{href}\" class='attachment'>#{shortname(name)}</a>"
-
-        _ ->
-          ""
-      end)
-
     Enum.join([text | attachment_text], "<br>")
   end
 
+  defp build_attachment_link(%{"url" => [%{"href" => href} | _]} = attachment) do
+    name = attachment["name"] || URI.decode(Path.basename(href))
+    href = MediaProxy.url(href)
+    "<a href=\"#{href}\" class='attachment'>#{shortname(name)}</a>"
+  end
+
+  defp build_attachment_link(_), do: ""
+
   def format_input(text, format, options \\ [])
 
   @doc """
@@ -314,7 +352,7 @@ def make_note_data(
         sensitive \\ false,
         merge \\ %{}
       ) do
-    object = %{
+    %{
       "type" => "Note",
       "to" => to,
       "cc" => cc,
@@ -324,18 +362,20 @@ def make_note_data(
       "context" => context,
       "attachment" => attachments,
       "actor" => actor,
-      "tag" => tags |> Enum.map(fn {_, tag} -> tag end) |> Enum.uniq()
+      "tag" => Keyword.values(tags) |> Enum.uniq()
     }
+    |> add_in_reply_to(in_reply_to)
+    |> Map.merge(merge)
+  end
 
-    object =
+  defp add_in_reply_to(object, nil), do: object
-      with false <- is_nil(in_reply_to),
-           %Object{} = in_reply_to_object <- Object.normalize(in_reply_to) do
-        Map.put(object, "inReplyTo", in_reply_to_object.data["id"])
-      else
-        _ -> object
-      end
 
-    Map.merge(object, merge)
+  defp add_in_reply_to(object, in_reply_to) do
+    with %Object{} = in_reply_to_object <- Object.normalize(in_reply_to) do
+      Map.put(object, "inReplyTo", in_reply_to_object.data["id"])
+    else
+      _ -> object
+    end
   end
 
   def format_naive_asctime(date) do
@@ -367,17 +407,16 @@ def to_masto_date(%NaiveDateTime{} = date) do
     |> String.replace(~r/(\.\d+)?$/, ".000Z", global: false)
   end
 
-  def to_masto_date(date) do
+  def to_masto_date(date) when is_binary(date) do
-    try do
+    with {:ok, date} <- NaiveDateTime.from_iso8601(date) do
-      date
+      to_masto_date(date)
-      |> NaiveDateTime.from_iso8601!()
+    else
-      |> NaiveDateTime.to_iso8601()
+      _ -> ""
-      |> String.replace(~r/(\.\d+)?$/, ".000Z", global: false)
-    rescue
-      _e -> ""
     end
   end
 
+  def to_masto_date(_), do: ""
+
   defp shortname(name) do
     if String.length(name) < 30 do
       name
@@ -422,7 +461,7 @@ def maybe_notify_mentioned_recipients(
 
     object_data =
       cond do
-        !is_nil(object) ->
+        not is_nil(object) ->
           object.data
 
         is_map(data["object"]) ->
@@ -466,9 +505,9 @@ def maybe_notify_subscribers(recipients, _), do: recipients
 
   def maybe_extract_mentions(%{"tag" => tag}) do
     tag
-    |> Enum.filter(fn x -> is_map(x) end)
+    |> Enum.filter(fn x -> is_map(x) && x["type"] == "Mention" end)
-    |> Enum.filter(fn x -> x["type"] == "Mention" end)
     |> Enum.map(fn x -> x["href"] end)
+    |> Enum.uniq()
   end
 
   def maybe_extract_mentions(_), do: []

lib/pleroma/web/controller_helper.ex

@@ -33,4 +33,80 @@ defp param_to_integer(val, default) when is_binary(val) do
   end
 
   defp param_to_integer(_, default), do: default
+
+  def add_link_headers(
+        conn,
+        method,
+        activities,
+        param \\ nil,
+        params \\ %{},
+        func3 \\ nil,
+        func4 \\ nil
+      ) do
+    params =
+      conn.params
+      |> Map.drop(["since_id", "max_id", "min_id"])
+      |> Map.merge(params)
+
+    last = List.last(activities)
+
+    func3 = func3 || (&mastodon_api_url/3)
+    func4 = func4 || (&mastodon_api_url/4)
+
+    if last do
+      max_id = last.id
+
+      limit =
+        params
+        |> Map.get("limit", "20")
+        |> String.to_integer()
+
+      min_id =
+        if length(activities) <= limit do
+          activities
+          |> List.first()
+          |> Map.get(:id)
+        else
+          activities
+          |> Enum.at(limit * -1)
+          |> Map.get(:id)
+        end
+
+      {next_url, prev_url} =
+        if param do
+          {
+            func4.(
+              Pleroma.Web.Endpoint,
+              method,
+              param,
+              Map.merge(params, %{max_id: max_id})
+            ),
+            func4.(
+              Pleroma.Web.Endpoint,
+              method,
+              param,
+              Map.merge(params, %{min_id: min_id})
+            )
+          }
+        else
+          {
+            func3.(
+              Pleroma.Web.Endpoint,
+              method,
+              Map.merge(params, %{max_id: max_id})
+            ),
+            func3.(
+              Pleroma.Web.Endpoint,
+              method,
+              Map.merge(params, %{min_id: min_id})
+            )
+          }
+        end
+
+      conn
+      |> put_resp_header("link", "<#{next_url}>; rel=\"next\", <#{prev_url}>; rel=\"prev\"")
+    else
+      conn
+    end
+  end
 end

lib/pleroma/web/fallback_redirect_controller.ex

@@ -0,0 +1,77 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Fallback.RedirectController do
+  use Pleroma.Web, :controller
+  require Logger
+  alias Pleroma.User
+  alias Pleroma.Web.Metadata
+
+  def api_not_implemented(conn, _params) do
+    conn
+    |> put_status(404)
+    |> json(%{error: "Not implemented"})
+  end
+
+  def redirector(conn, _params, code \\ 200)
+
+  # redirect to admin section
+  # /pleroma/admin -> /pleroma/admin/
+  #
+  def redirector(conn, %{"path" => ["pleroma", "admin"]} = _, _code) do
+    redirect(conn, to: "/pleroma/admin/")
+  end
+
+  def redirector(conn, _params, code) do
+    conn
+    |> put_resp_content_type("text/html")
+    |> send_file(code, index_file_path())
+  end
+
+  def redirector_with_meta(conn, %{"maybe_nickname_or_id" => maybe_nickname_or_id} = params) do
+    with %User{} = user <- User.get_cached_by_nickname_or_id(maybe_nickname_or_id) do
+      redirector_with_meta(conn, %{user: user})
+    else
+      nil ->
+        redirector(conn, params)
+    end
+  end
+
+  def redirector_with_meta(conn, params) do
+    {:ok, index_content} = File.read(index_file_path())
+
+    tags =
+      try do
+        Metadata.build_tags(params)
+      rescue
+        e ->
+          Logger.error(
+            "Metadata rendering for #{conn.request_path} failed.\n" <>
+              Exception.format(:error, e, __STACKTRACE__)
+          )
+
+          ""
+      end
+
+    response = String.replace(index_content, "<!--server-generated-meta-->", tags)
+
+    conn
+    |> put_resp_content_type("text/html")
+    |> send_resp(200, response)
+  end
+
+  def index_file_path do
+    Pleroma.Plugs.InstanceStatic.file_path("index.html")
+  end
+
+  def registration_page(conn, params) do
+    redirector(conn, params)
+  end
+
+  def empty(conn, _params) do
+    conn
+    |> put_status(204)
+    |> text("")
+  end
+end

lib/pleroma/web/federator/retry_queue.ex

@@ -13,7 +13,7 @@ def init(args) do
     {:ok, %{args | queue_table: queue_table, running_jobs: :sets.new()}}
   end
 
-  def start_link do
+  def start_link(_) do
     enabled =
       if Pleroma.Config.get(:env) == :test,
         do: true,

lib/pleroma/web/mailer/subscription_controller.ex

@@ -0,0 +1,20 @@
+defmodule Pleroma.Web.Mailer.SubscriptionController do
+  use Pleroma.Web, :controller
+
+  alias Pleroma.JWT
+  alias Pleroma.Repo
+  alias Pleroma.User
+
+  def unsubscribe(conn, %{"token" => encoded_token}) do
+    with {:ok, token} <- Base.decode64(encoded_token),
+         {:ok, claims} <- JWT.verify_and_validate(token),
+         %{"act" => %{"unsubscribe" => type}, "sub" => uid} <- claims,
+         %User{} = user <- Repo.get(User, uid),
+         {:ok, _user} <- User.switch_email_notifications(user, type, false) do
+      render(conn, "unsubscribe_success.html", email: user.email)
+    else
+      _err ->
+        render(conn, "unsubscribe_failure.html")
+    end
+  end
+end

lib/pleroma/web/mastodon_api/mastodon_api.ex

@@ -13,10 +13,8 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPI do
   alias Pleroma.User
   alias Pleroma.Web.CommonAPI
 
+  @spec follow(User.t(), User.t(), map) :: {:ok, User.t()} | {:error, String.t()}
   def follow(follower, followed, params \\ %{}) do
-    options = cast_params(params)
-    reblogs = options[:reblogs]
-
     result =
       if not User.following?(follower, followed) do
         CommonAPI.follow(follower, followed)
@@ -24,19 +22,25 @@ def follow(follower, followed, params \\ %{}) do
         {:ok, follower, followed, nil}
       end
 
-    with {:ok, follower, followed, _} <- result do
+    with {:ok, follower, _followed, _} <- result do
-      reblogs
+      options = cast_params(params)
-      |> case do
+
-        false -> CommonAPI.hide_reblogs(follower, followed)
+      case reblogs_visibility(options[:reblogs], result) do
-        _ -> CommonAPI.show_reblogs(follower, followed)
-      end
-      |> case do
         {:ok, follower} -> {:ok, follower}
         _ -> {:ok, follower}
       end
     end
   end
 
+  defp reblogs_visibility(false, {:ok, follower, followed, _}) do
+    CommonAPI.hide_reblogs(follower, followed)
+  end
+
+  defp reblogs_visibility(_, {:ok, follower, followed, _}) do
+    CommonAPI.show_reblogs(follower, followed)
+  end
+
+  @spec get_followers(User.t(), map()) :: list(User.t())
   def get_followers(user, params \\ %{}) do
     user
     |> User.get_followers_query()

lib/pleroma/web/mastodon_api/mastodon_api_controller.ex

@@ -4,6 +4,10 @@
 
 defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   use Pleroma.Web, :controller
+
+  import Pleroma.Web.ControllerHelper,
+    only: [json_response: 3, add_link_headers: 5, add_link_headers: 4, add_link_headers: 3]
+
   alias Ecto.Changeset
   alias Pleroma.Activity
   alias Pleroma.Bookmark
@@ -46,6 +50,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   import Ecto.Query
 
   require Logger
+  require Pleroma.Constants
 
   @rate_limited_relations_actions ~w(follow unfollow)a
 
@@ -74,6 +79,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   plug(RateLimiter, :app_account_creation when action == :account_register)
   plug(RateLimiter, :search when action in [:search, :search2, :account_search])
   plug(RateLimiter, :password_reset when action == :password_reset)
+  plug(RateLimiter, :account_confirmation_resend when action == :account_confirmation_resend)
 
   @local_mastodon_name "Mastodon-Local"
 
@@ -132,7 +138,9 @@ def update_credentials(%{assigns: %{user: user}} = conn, params) do
     emojis_text = (user_params["display_name"] || "") <> (user_params["note"] || "")
 
     user_info_emojis =
-      ((user.info.emoji || []) ++ Formatter.get_emoji_map(emojis_text))
+      user.info
+      |> Map.get(:emoji, [])
+      |> Enum.concat(Formatter.get_emoji_map(emojis_text))
       |> Enum.dedup()
 
     info_params =
@@ -151,6 +159,12 @@ def update_credentials(%{assigns: %{user: user}} = conn, params) do
         end)
       end)
       |> add_if_present(params, "default_scope", :default_scope)
+      |> add_if_present(params, "fields", :fields, fn fields ->
+        fields = Enum.map(fields, fn f -> Map.update!(f, "value", &AutoLinker.link(&1)) end)
+
+        {:ok, fields}
+      end)
+      |> add_if_present(params, "fields", :raw_fields)
       |> add_if_present(params, "pleroma_settings_store", :pleroma_settings_store, fn value ->
         {:ok, Map.merge(user.info.pleroma_settings_store, value)}
       end)
@@ -337,71 +351,6 @@ def custom_emojis(conn, _params) do
     json(conn, mastodon_emoji)
   end
 
-  defp add_link_headers(conn, method, activities, param \\ nil, params \\ %{}) do
-    params =
-      conn.params
-      |> Map.drop(["since_id", "max_id", "min_id"])
-      |> Map.merge(params)
-
-    last = List.last(activities)
-
-    if last do
-      max_id = last.id
-
-      limit =
-        params
-        |> Map.get("limit", "20")
-        |> String.to_integer()
-
-      min_id =
-        if length(activities) <= limit do
-          activities
-          |> List.first()
-          |> Map.get(:id)
-        else
-          activities
-          |> Enum.at(limit * -1)
-          |> Map.get(:id)
-        end
-
-      {next_url, prev_url} =
-        if param do
-          {
-            mastodon_api_url(
-              Pleroma.Web.Endpoint,
-              method,
-              param,
-              Map.merge(params, %{max_id: max_id})
-            ),
-            mastodon_api_url(
-              Pleroma.Web.Endpoint,
-              method,
-              param,
-              Map.merge(params, %{min_id: min_id})
-            )
-          }
-        else
-          {
-            mastodon_api_url(
-              Pleroma.Web.Endpoint,
-              method,
-              Map.merge(params, %{max_id: max_id})
-            ),
-            mastodon_api_url(
-              Pleroma.Web.Endpoint,
-              method,
-              Map.merge(params, %{min_id: min_id})
-            )
-          }
-        end
-
-      conn
-      |> put_resp_header("link", "<#{next_url}>; rel=\"next\", <#{prev_url}>; rel=\"prev\"")
-    else
-      conn
-    end
-  end
-
   def home_timeline(%{assigns: %{user: user}} = conn, params) do
     params =
       params
@@ -430,6 +379,7 @@ def public_timeline(%{assigns: %{user: user}} = conn, params) do
       |> Map.put("local_only", local_only)
       |> Map.put("blocking_user", user)
       |> Map.put("muting_user", user)
+      |> Map.put("user", user)
       |> ActivityPub.fetch_public_activities()
       |> Enum.reverse()
 
@@ -491,12 +441,9 @@ def get_context(%{assigns: %{user: user}} = conn, %{"id" => id}) do
          activities <-
            ActivityPub.fetch_activities_for_context(activity.data["context"], %{
              "blocking_user" => user,
-             "user" => user
+             "user" => user,
+             "exclude_id" => activity.id
            }),
-         activities <-
-           activities |> Enum.filter(fn %{id: aid} -> to_string(aid) != to_string(id) end),
-         activities <-
-           activities |> Enum.filter(fn %{data: %{"type" => type}} -> type == "Create" end),
          grouped_activities <- Enum.group_by(activities, fn %{id: id} -> id < activity.id end) do
       result = %{
         ancestors:
@@ -531,8 +478,8 @@ def get_poll(%{assigns: %{user: user}} = conn, %{"id" => id}) do
       |> put_view(StatusView)
       |> try_render("poll.json", %{object: object, for: user})
     else
-      nil -> render_error(conn, :not_found, "Record not found")
+      error when is_nil(error) or error == false ->
-      false -> render_error(conn, :not_found, "Record not found")
+        render_error(conn, :not_found, "Record not found")
     end
   end
 
@@ -880,8 +827,8 @@ def get_mascot(%{assigns: %{user: user}} = conn, _params) do
   end
 
   def favourited_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
-    with %Activity{data: %{"object" => object}} <- Activity.get_by_id(id),
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
-         %Object{data: %{"likes" => likes}} <- Object.normalize(object) do
+         %Object{data: %{"likes" => likes}} <- Object.normalize(activity) do
       q = from(u in User, where: u.ap_id in ^likes)
 
       users =
@@ -897,8 +844,8 @@ def favourited_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
   end
 
   def reblogged_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
-    with %Activity{data: %{"object" => object}} <- Activity.get_by_id(id),
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
-         %Object{data: %{"announcements" => announces}} <- Object.normalize(object) do
+         %Object{data: %{"announcements" => announces}} <- Object.normalize(activity) do
       q = from(u in User, where: u.ap_id in ^announces)
 
       users =
@@ -939,6 +886,7 @@ def hashtag_timeline(%{assigns: %{user: user}} = conn, params) do
       |> Map.put("local_only", local_only)
       |> Map.put("blocking_user", user)
       |> Map.put("muting_user", user)
+      |> Map.put("user", user)
       |> Map.put("tag", tags)
       |> Map.put("tag_all", tag_all)
       |> Map.put("tag_reject", tag_reject)
@@ -1220,10 +1168,9 @@ def user_favourites(%{assigns: %{user: for_user}} = conn, %{"id" => id} = params
 
       recipients =
         if for_user do
-          ["https://www.w3.org/ns/activitystreams#Public"] ++
+          [Pleroma.Constants.as_public()] ++ [for_user.ap_id | for_user.following]
-            [for_user.ap_id | for_user.following]
         else
-          ["https://www.w3.org/ns/activitystreams#Public"]
+          [Pleroma.Constants.as_public()]
         end
 
       activities =
@@ -1346,6 +1293,7 @@ def list_timeline(%{assigns: %{user: user}} = conn, %{"list_id" => id} = params)
         params
         |> Map.put("type", "Create")
         |> Map.put("blocking_user", user)
+        |> Map.put("user", user)
         |> Map.put("muting_user", user)
 
       # we must filter the following list for the user to avoid leaking statuses the user
@@ -1686,45 +1634,35 @@ def suggestions(%{assigns: %{user: user}} = conn, _) do
         |> String.replace("{{user}}", user)
 
       with {:ok, %{status: 200, body: body}} <-
-             HTTP.get(
+             HTTP.get(url, [], adapter: [recv_timeout: timeout, pool: :default]),
-               url,
-               [],
-               adapter: [
-                 recv_timeout: timeout,
-                 pool: :default
-               ]
-             ),
            {:ok, data} <- Jason.decode(body) do
         data =
           data
           |> Enum.slice(0, limit)
           |> Enum.map(fn x ->
-            Map.put(
+            x
-              x,
+            |> Map.put("id", fetch_suggestion_id(x))
-              "id",
+            |> Map.put("avatar", MediaProxy.url(x["avatar"]))
-              case User.get_or_fetch(x["acct"]) do
+            |> Map.put("avatar_static", MediaProxy.url(x["avatar_static"]))
-                {:ok, %User{id: id}} -> id
-                _ -> 0
-              end
-            )
-          end)
-          |> Enum.map(fn x ->
-            Map.put(x, "avatar", MediaProxy.url(x["avatar"]))
-          end)
-          |> Enum.map(fn x ->
-            Map.put(x, "avatar_static", MediaProxy.url(x["avatar_static"]))
           end)
 
-        conn
+        json(conn, data)
-        |> json(data)
       else
-        e -> Logger.error("Could not retrieve suggestions at fetch #{url}, #{inspect(e)}")
+        e ->
+          Logger.error("Could not retrieve suggestions at fetch #{url}, #{inspect(e)}")
       end
     else
       json(conn, [])
     end
   end
 
+  defp fetch_suggestion_id(attrs) do
+    case User.get_or_fetch(attrs["acct"]) do
+      {:ok, %User{id: id}} -> id
+      _ -> 0
+    end
+  end
+
   def status_card(%{assigns: %{user: user}} = conn, %{"id" => status_id}) do
     with %Activity{} = activity <- Activity.get_by_id(status_id),
          true <- Visibility.visible_for_user?(activity, user) do
@@ -1803,7 +1741,7 @@ def conversations(%{assigns: %{user: user}} = conn, params) do
 
     conversations =
       Enum.map(participations, fn participation ->
-        ConversationView.render("participation.json", %{participation: participation, user: user})
+        ConversationView.render("participation.json", %{participation: participation, for: user})
       end)
 
     conn
@@ -1816,7 +1754,7 @@ def conversation_read(%{assigns: %{user: user}} = conn, %{"id" => participation_
            Repo.get_by(Participation, id: participation_id, user_id: user.id),
          {:ok, participation} <- Participation.mark_as_read(participation) do
       participation_view =
-        ConversationView.render("participation.json", %{participation: participation, user: user})
+        ConversationView.render("participation.json", %{participation: participation, for: user})
 
       conn
       |> json(participation_view)
@@ -1839,6 +1777,16 @@ def password_reset(conn, params) do
     end
   end
 
+  def account_confirmation_resend(conn, params) do
+    nickname_or_email = params["email"] || params["nickname"]
+
+    with %User{} = user <- User.get_by_nickname_or_email(nickname_or_email),
+         {:ok, _} <- User.try_send_confirmation_email(user) do
+      conn
+      |> json_response(:no_content, "")
+    end
+  end
+
   def try_render(conn, target, params)
       when is_binary(target) do
     case render(conn, target, params) do

lib/pleroma/web/mastodon_api/views/account_view.ex

@@ -28,7 +28,7 @@ def render("mention.json", %{user: user}) do
       id: to_string(user.id),
       acct: user.nickname,
       username: username_from_nickname(user.nickname),
-      url: user.ap_id
+      url: User.profile_url(user)
     }
   end
 
@@ -37,11 +37,11 @@ def render("relationship.json", %{user: nil, target: _target}) do
   end
 
   def render("relationship.json", %{user: %User{} = user, target: %User{} = target}) do
-    follow_activity = Pleroma.Web.ActivityPub.Utils.fetch_latest_follow(user, target)
+    follow_state = User.get_cached_follow_state(user, target)
 
     requested =
-      if follow_activity && !User.following?(target, user) do
+      if follow_state && !User.following?(user, target) do
-        follow_activity.data["state"] == "pending"
+        follow_state == "pending"
       else
         false
       end
@@ -50,13 +50,13 @@ def render("relationship.json", %{user: %User{} = user, target: %User{} = target
       id: to_string(target.id),
       following: User.following?(user, target),
       followed_by: User.following?(target, user),
-      blocking: User.blocks?(user, target),
+      blocking: User.blocks_ap_id?(user, target),
-      blocked_by: User.blocks?(target, user),
+      blocked_by: User.blocks_ap_id?(target, user),
       muting: User.mutes?(user, target),
       muting_notifications: User.muted_notifications?(user, target),
       subscribing: User.subscribed_to?(user, target),
       requested: requested,
-      domain_blocking: false,
+      domain_blocking: User.blocks_domain?(user, target),
       showing_reblogs: User.showing_reblogs?(user, target),
       endorsed: false
     }
@@ -72,6 +72,13 @@ defp do_render("account.json", %{user: user} = opts) do
     image = User.avatar_url(user) |> MediaProxy.url()
     header = User.banner_url(user) |> MediaProxy.url()
     user_info = User.get_cached_user_info(user)
+
+    following_count =
+      ((!user.info.hide_follows or opts[:for] == user) && user_info.following_count) || 0
+
+    followers_count =
+      ((!user.info.hide_followers or opts[:for] == user) && user_info.follower_count) || 0
+
     bot = (user.info.source_data["type"] || "Person") in ["Application", "Service"]
 
     emojis =
@@ -87,12 +94,18 @@ defp do_render("account.json", %{user: user} = opts) do
       end)
 
     fields =
-      (user.info.source_data["attachment"] || [])
+      user.info
-      |> Enum.filter(fn %{"type" => t} -> t == "PropertyValue" end)
+      |> User.Info.fields()
-      |> Enum.map(fn fields -> Map.take(fields, ["name", "value"]) end)
+      |> Enum.map(fn %{"name" => name, "value" => value} ->
+        %{
+          "name" => Pleroma.HTML.strip_tags(name),
+          "value" => Pleroma.HTML.filter_tags(value, Pleroma.HTML.Scrubber.LinksOnly)
+        }
+      end)
+
+    raw_fields = Map.get(user.info, :raw_fields, [])
 
     bio = HTML.filter_tags(user.bio, User.html_filter_policy(opts[:for]))
-
     relationship = render("relationship.json", %{user: opts[:for], target: user})
 
     %{
@@ -102,11 +115,11 @@ defp do_render("account.json", %{user: user} = opts) do
       display_name: display_name,
       locked: user_info.locked,
       created_at: Utils.to_masto_date(user.inserted_at),
-      followers_count: user_info.follower_count,
+      followers_count: followers_count,
-      following_count: user_info.following_count,
+      following_count: following_count,
       statuses_count: user_info.note_count,
       note: bio || "",
-      url: user.ap_id,
+      url: User.profile_url(user),
       avatar: image,
       avatar_static: image,
       header: header,
@@ -117,6 +130,7 @@ defp do_render("account.json", %{user: user} = opts) do
       source: %{
         note: HTML.strip_tags((user.bio || "") |> String.replace("<br>", "\n")),
         sensitive: false,
+        fields: raw_fields,
         pleroma: %{}
       },
 

lib/pleroma/web/mastodon_api/views/conversation_view.ex

@@ -11,8 +11,8 @@ defmodule Pleroma.Web.MastodonAPI.ConversationView do
   alias Pleroma.Web.MastodonAPI.AccountView
   alias Pleroma.Web.MastodonAPI.StatusView
 
-  def render("participation.json", %{participation: participation, user: user}) do
+  def render("participation.json", %{participation: participation, for: user}) do
-    participation = Repo.preload(participation, conversation: :users)
+    participation = Repo.preload(participation, conversation: [], recipients: [])
 
     last_activity_id =
       with nil <- participation.last_activity_id do
@@ -28,7 +28,7 @@ def render("participation.json", %{participation: participation, user: user}) do
 
     # Conversations return all users except the current user.
     users =
-      participation.conversation.users
+      participation.recipients
       |> Enum.reject(&(&1.id == user.id))
 
     accounts =

lib/pleroma/web/mastodon_api/views/status_view.ex

@@ -5,8 +5,12 @@
 defmodule Pleroma.Web.MastodonAPI.StatusView do
   use Pleroma.Web, :view
 
+  require Pleroma.Constants
+
   alias Pleroma.Activity
   alias Pleroma.ActivityExpiration
+  alias Pleroma.Conversation
+  alias Pleroma.Conversation.Participation
   alias Pleroma.HTML
   alias Pleroma.Object
   alias Pleroma.Repo
@@ -25,19 +29,19 @@ defp get_replied_to_activities([]), do: %{}
   defp get_replied_to_activities(activities) do
     activities
     |> Enum.map(fn
-      %{data: %{"type" => "Create", "object" => object}} ->
+      %{data: %{"type" => "Create"}} = activity ->
-        object = Object.normalize(object)
+        object = Object.normalize(activity)
-        object.data["inReplyTo"] != "" && object.data["inReplyTo"]
+        object && object.data["inReplyTo"] != "" && object.data["inReplyTo"]
 
       _ ->
         nil
     end)
     |> Enum.filter(& &1)
-    |> Activity.create_by_object_ap_id()
+    |> Activity.create_by_object_ap_id_with_object()
     |> Repo.all()
     |> Enum.reduce(%{}, fn activity, acc ->
       object = Object.normalize(activity)
-      Map.put(acc, object.data["id"], activity)
+      if object, do: Map.put(acc, object.data["id"], activity), else: acc
     end)
   end
 
@@ -69,12 +73,14 @@ defp reblogged?(activity, user) do
 
   def render("index.json", opts) do
     replied_to_activities = get_replied_to_activities(opts.activities)
+    parallel = unless is_nil(opts[:parallel]), do: opts[:parallel], else: true
 
     opts.activities
     |> safe_render_many(
       StatusView,
       "status.json",
-      Map.put(opts, :replied_to_activities, replied_to_activities)
+      Map.put(opts, :replied_to_activities, replied_to_activities),
+      parallel
     )
   end
 
@@ -89,6 +95,7 @@ def render(
     reblogged_activity =
       Activity.create_by_object_ap_id(activity_object.data["id"])
       |> Activity.with_preloaded_bookmark(opts[:for])
+      |> Activity.with_set_thread_muted_field(opts[:for])
       |> Repo.one()
 
     reblogged = render("status.json", Map.put(opts, :activity, reblogged_activity))
@@ -143,6 +150,7 @@ def render("status.json", %{activity: %{data: %{"object" => _object}} = activity
     object = Object.normalize(activity)
 
     user = get_user(activity.data["actor"])
+    user_follower_address = user.follower_address
 
     like_count = object.data["like_count"] || 0
     announcement_count = object.data["announcement_count"] || 0
@@ -158,7 +166,11 @@ def render("status.json", %{activity: %{data: %{"object" => _object}} = activity
     mentions =
       (object.data["to"] ++ tag_mentions)
       |> Enum.uniq()
-      |> Enum.map(fn ap_id -> User.get_cached_by_ap_id(ap_id) end)
+      |> Enum.map(fn
+        Pleroma.Constants.as_public() -> nil
+        ^user_follower_address -> nil
+        ap_id -> User.get_cached_by_ap_id(ap_id)
+      end)
       |> Enum.filter(& &1)
       |> Enum.map(fn user -> AccountView.render("mention.json", %{user: user}) end)
 
@@ -178,7 +190,7 @@ def render("status.json", %{activity: %{data: %{"object" => _object}} = activity
     thread_muted? =
       case activity.thread_muted? do
         thread_muted? when is_boolean(thread_muted?) -> thread_muted?
-        nil -> CommonAPI.thread_muted?(user, activity)
+        nil -> (opts[:for] && CommonAPI.thread_muted?(opts[:for], activity)) || false
       end
 
     attachment_data = object.data["attachment"] || []
@@ -232,7 +244,20 @@ def render("status.json", %{activity: %{data: %{"object" => _object}} = activity
       if user.local do
         Pleroma.Web.Router.Helpers.o_status_url(Pleroma.Web.Endpoint, :notice, activity)
       else
-        object.data["external_url"] || object.data["id"]
+        object.data["url"] || object.data["external_url"] || object.data["id"]
+      end
+
+    direct_conversation_id =
+      with {_, true} <- {:include_id, opts[:with_direct_conversation_id]},
+           {_, %User{} = for_user} <- {:for_user, opts[:for]},
+           %{data: %{"context" => context}} when is_binary(context) <- activity,
+           %Conversation{} = conversation <- Conversation.get_for_ap_id(context),
+           %Participation{id: participation_id} <-
+             Participation.for_user_and_conversation(for_user, conversation) do
+        participation_id
+      else
+        _e ->
+          nil
       end
 
     %{
@@ -273,7 +298,8 @@ def render("status.json", %{activity: %{data: %{"object" => _object}} = activity
         in_reply_to_account_acct: reply_to_user && reply_to_user.nickname,
         content: %{"text/plain" => content_plaintext},
         spoiler_text: %{"text/plain" => summary_plaintext},
-        expires_at: expires_at
+        expires_at: expires_at,
+        direct_conversation_id: direct_conversation_id
       }
     }
   end

lib/pleroma/web/media_proxy/media_proxy.ex

@@ -4,6 +4,7 @@
 
 defmodule Pleroma.Web.MediaProxy do
   alias Pleroma.Config
+  alias Pleroma.Upload
   alias Pleroma.Web
 
   @base64_opts [padding: false]
@@ -26,7 +27,18 @@ defp local?(url), do: String.starts_with?(url, Pleroma.Web.base_url())
   defp whitelisted?(url) do
     %{host: domain} = URI.parse(url)
 
-    Enum.any?(Config.get([:media_proxy, :whitelist]), fn pattern ->
+    mediaproxy_whitelist = Config.get([:media_proxy, :whitelist])
+
+    upload_base_url_domain =
+      if !is_nil(Config.get([Upload, :base_url])) do
+        [URI.parse(Config.get([Upload, :base_url])).host]
+      else
+        []
+      end
+
+    whitelist = mediaproxy_whitelist ++ upload_base_url_domain
+
+    Enum.any?(whitelist, fn pattern ->
       String.equivalent?(domain, pattern)
     end)
   end

lib/pleroma/web/nodeinfo/nodeinfo_controller.ex

@@ -34,64 +34,18 @@ def schemas(conn, _params) do
   def raw_nodeinfo do
     stats = Stats.get_stats()
 
-    exclusions = Config.get([:instance, :mrf_transparency_exclusions])
-
-    mrf_simple =
-      Config.get(:mrf_simple)
-      |> Enum.map(fn {k, v} -> {k, Enum.reject(v, fn v -> v in exclusions end)} end)
-      |> Enum.into(%{})
-
-    # This horror is needed to convert regex sigils to strings
-    mrf_keyword =
-      Config.get(:mrf_keyword, [])
-      |> Enum.map(fn {key, value} ->
-        {key,
-         Enum.map(value, fn
-           {pattern, replacement} ->
-             %{
-               "pattern" =>
-                 if not is_binary(pattern) do
-                   inspect(pattern)
-                 else
-                   pattern
-                 end,
-               "replacement" => replacement
-             }
-
-           pattern ->
-             if not is_binary(pattern) do
-               inspect(pattern)
-             else
-               pattern
-             end
-         end)}
-      end)
-      |> Enum.into(%{})
-
-    mrf_policies =
-      MRF.get_policies()
-      |> Enum.map(fn policy -> to_string(policy) |> String.split(".") |> List.last() end)
-
     quarantined = Config.get([:instance, :quarantined_instances], [])
 
     staff_accounts =
       User.all_superusers()
       |> Enum.map(fn u -> u.ap_id end)
 
-    mrf_user_allowlist =
-      Config.get([:mrf_user_allowlist], [])
-      |> Enum.into(%{}, fn {k, v} -> {k, length(v)} end)
-
     federation_response =
       if Config.get([:instance, :mrf_transparency]) do
-        %{
+        {:ok, data} = MRF.describe()
-          mrf_policies: mrf_policies,
+
-          mrf_simple: mrf_simple,
+        data
-          mrf_keyword: mrf_keyword,
+        |> Map.merge(%{quarantined_instances: quarantined})
-          mrf_user_allowlist: mrf_user_allowlist,
-          quarantined_instances: quarantined,
-          exclusions: length(exclusions) > 0
-        }
       else
         %{}
       end
@@ -165,6 +119,7 @@ def raw_nodeinfo do
         },
         accountActivationRequired: Config.get([:instance, :account_activation_required], false),
         invitesEnabled: Config.get([:instance, :invites_enabled], false),
+        mailerEnabled: Config.get([Pleroma.Emails.Mailer, :enabled], false),
         features: features,
         restrictedNicknames: Config.get([Pleroma.User, :restricted_nicknames]),
         skipThreadContainment: Config.get([:instance, :skip_thread_containment], false)

lib/pleroma/web/oauth/oauth_controller.ex

@@ -365,8 +365,7 @@ def registration_details(%Plug.Conn{} = conn, %{"authorization" => auth_attrs})
   def register(%Plug.Conn{} = conn, %{"authorization" => _, "op" => "connect"} = params) do
     with registration_id when not is_nil(registration_id) <- get_session_registration_id(conn),
          %Registration{} = registration <- Repo.get(Registration, registration_id),
-         {_, {:ok, auth}} <-
+         {_, {:ok, auth}} <- {:create_authorization, do_create_authorization(conn, params)},
-           {:create_authorization, do_create_authorization(conn, params)},
          %User{} = user <- Repo.preload(auth, :user).user,
          {:ok, _updated_registration} <- Registration.bind_to_user(registration, user) do
       conn

lib/pleroma/web/oauth/token.ex

@@ -44,8 +44,7 @@ def get_by_refresh_token(%App{id: app_id} = _app, token) do
     |> Repo.find_resource()
   end
 
-  @spec exchange_token(App.t(), Authorization.t()) ::
+  @spec exchange_token(App.t(), Authorization.t()) :: {:ok, Token.t()} | {:error, Changeset.t()}
-          {:ok, Token.t()} | {:error, Changeset.t()}
   def exchange_token(app, auth) do
     with {:ok, auth} <- Authorization.use_token(auth),
          true <- auth.app_id == app.id do

lib/pleroma/web/oauth/token/clean_worker.ex

@@ -6,36 +6,30 @@ defmodule Pleroma.Web.OAuth.Token.CleanWorker do
   @moduledoc """
   The module represents functions to clean an expired oauth tokens.
   """
+  use GenServer
+
+  @ten_seconds 10_000
+  @one_day 86_400_000
 
-  # 10 seconds
-  @start_interval 10_000
   @interval Pleroma.Config.get(
-              # 24 hours
               [:oauth2, :clean_expired_tokens_interval],
-              86_400_000
+              @one_day
             )
-  @queue :background
 
   alias Pleroma.Web.OAuth.Token
 
-  def start_link, do: GenServer.start_link(__MODULE__, nil)
+  def start_link(_), do: GenServer.start_link(__MODULE__, %{})
 
   def init(_) do
-    if Pleroma.Config.get([:oauth2, :clean_expired_tokens], false) do
+    Process.send_after(self(), :perform, @ten_seconds)
-      Process.send_after(self(), :perform, @start_interval)
+    {:ok, nil}
-      {:ok, nil}
-    else
-      :ignore
-    end
   end
 
   @doc false
   def handle_info(:perform, state) do
+    Token.delete_expired_tokens()
+
     Process.send_after(self(), :perform, @interval)
-    PleromaJobQueue.enqueue(@queue, __MODULE__, [:clean])
     {:noreply, state}
   end
-
-  # Job Worker Callbacks
-  def perform(:clean), do: Token.delete_expired_tokens()
 end

lib/pleroma/web/ostatus/activity_representer.ex

@@ -9,6 +9,7 @@ defmodule Pleroma.Web.OStatus.ActivityRepresenter do
   alias Pleroma.Web.OStatus.UserRepresenter
 
   require Logger
+  require Pleroma.Constants
 
   defp get_href(id) do
     with %Object{data: %{"external_url" => external_url}} <- Object.get_cached_by_ap_id(id) do
@@ -34,7 +35,7 @@ defp get_mentions(to) do
     Enum.map(to, fn id ->
       cond do
         # Special handling for the AP/Ostatus public collections
-        "https://www.w3.org/ns/activitystreams#Public" == id ->
+        Pleroma.Constants.as_public() == id ->
           {:link,
            [
              rel: "mentioned",
@@ -182,6 +183,7 @@ def to_simple_form(%{data: %{"type" => "Announce"}} = activity, user, with_autho
     author = if with_author, do: [{:author, UserRepresenter.to_simple_form(user)}], else: []
 
     retweeted_activity = Activity.get_create_by_object_ap_id(activity.data["object"])
+    retweeted_object = Object.normalize(retweeted_activity)
     retweeted_user = User.get_cached_by_ap_id(retweeted_activity.data["actor"])
 
     retweeted_xml = to_simple_form(retweeted_activity, retweeted_user, true)
@@ -196,7 +198,7 @@ def to_simple_form(%{data: %{"type" => "Announce"}} = activity, user, with_autho
       {:"activity:verb", ['http://activitystrea.ms/schema/1.0/share']},
       {:id, h.(activity.data["id"])},
       {:title, ['#{user.nickname} repeated a notice']},
-      {:content, [type: 'html'], ['RT #{retweeted_activity.data["object"]["content"]}']},
+      {:content, [type: 'html'], ['RT #{retweeted_object.data["content"]}']},
       {:published, h.(inserted_at)},
       {:updated, h.(updated_at)},
       {:"ostatus:conversation", [ref: h.(activity.data["context"])],