Merge branch 'develop' into feature/bulk-confirmation

2020-09-23 11:56:22 -05:00 · 2020-09-23 11:56:22 -05:00 · f3a1f9c3bb
commit f3a1f9c3bb
parent 34d7e864db e02101e15c
274 changed files with 8654 additions and 9573 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -13,10 +13,50 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Renamed `:await_up_timeout` in `:connections_pool` namespace to `:connect_timeout`, old name is deprecated.
 - Renamed `:timeout` in `pools` namespace to `:recv_timeout`, old name is deprecated.
 - The `discoverable` field in the `User` struct will now add a NOINDEX metatag to profile pages when false.
 - Users with the `discoverable` field set to false will not show up in searches.
 - Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option).
 ### Added
 - Media preview proxy (requires media proxy be enabled; see `:media_preview_proxy` config for more details).
 - Pleroma API: Importing the mutes users from CSV files.
 - Experimental websocket-based federation between Pleroma instances.
 - Admin API: Importing emoji from a zip file
 ### Removed
- **Breaking:** Removed `Pleroma.Workers.Cron.StatsWorker` setting from Oban `:crontab`.
+- **Breaking:** `Pleroma.Workers.Cron.StatsWorker` setting from Oban `:crontab` (moved to a simpler implementation).
 - **Breaking:** `Pleroma.Workers.Cron.ClearOauthTokenWorker` setting from Oban `:crontab` (moved to scheduled jobs).
 - **Breaking:** `Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker` setting from Oban `:crontab` (moved to scheduled jobs).
 - Removed `:managed_config` option. In practice, it was accidentally removed with 2.0.0 release when frontends were
 switched to a new configuration mechanism, however it was not officially removed until now.
 ### Fixed
 - Add documented-but-missing chat pagination.
 - Allow sending out emails again.
 ## [2.1.2] - 2020-09-17
 ### Security
 - Fix most MRF rules either crashing or not being applied to objects passed into the Common Pipeline (ChatMessage, Question, Answer, Audio, Event).
 ### Fixed
 - Welcome Chat messages preventing user registration with MRF Simple Policy applied to the local instance.
 - Mastodon API: the public timeline returning an error when the `reply_visibility` parameter is set to `self` for an unauthenticated user.
 - Mastodon Streaming API: Handler crashes on authentication failures, resulting in error logs.
 - Mastodon Streaming API: Error logs on client pings.
 - Rich media: Log spam on failures. Now the error is only logged once per attempt.
 ### Changed
 - Rich Media: A HEAD request is now done to the url, to ensure it has the appropriate content type and size before proceeding with a GET.
 ### Upgrade notes
 1. Restart Pleroma
 ## [2.1.1] - 2020-09-08
@ -33,6 +73,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Added
 - Rich media failure tracking (along with `:failure_backoff` option).
 <details>
  <summary>Admin API Changes</summary>
 - Add `PATCH /api/pleroma/admin/instance_document/:document_name` to modify the Terms of Service and Instance Panel HTML pages via Admin API
 </details>
 ### Fixed
 - Default HTTP adapter not respecting pool setting, leading to possible OOM.
 - Fixed uploading webp images when the Exiftool Upload Filter is enabled by skipping them
--- a/README.md
+++ b/README.md
@ -18,15 +18,16 @@ If you are running Linux (glibc or musl) on x86/arm, the recommended way to inst
 ### From Source
 If your platform is not supported, or you just want to be able to edit the source code easily, you may install Pleroma from source.
 - [Debian-based](https://docs-develop.pleroma.social/backend/installation/debian_based_en/)
 - [Debian-based (jp)](https://docs-develop.pleroma.social/backend/installation/debian_based_jp/)
 - [Alpine Linux](https://docs-develop.pleroma.social/backend/installation/alpine_linux_en/)
 - [Arch Linux](https://docs-develop.pleroma.social/backend/installation/arch_linux_en/)
 - [CentOS 7](https://docs-develop.pleroma.social/backend/installation/centos7_en/)
 - [Debian-based](https://docs-develop.pleroma.social/backend/installation/debian_based_en/)
 - [Debian-based (jp)](https://docs-develop.pleroma.social/backend/installation/debian_based_jp/)
 - [FreeBSD](https://docs-develop.pleroma.social/backend/installation/freebsd_en/)
 - [Gentoo Linux](https://docs-develop.pleroma.social/backend/installation/gentoo_en/)
 - [NetBSD](https://docs-develop.pleroma.social/backend/installation/netbsd_en/)
 - [OpenBSD](https://docs-develop.pleroma.social/backend/installation/openbsd_en/)
 - [OpenBSD (fi)](https://docs-develop.pleroma.social/backend/installation/openbsd_fi/)
 - [CentOS 7](https://docs-develop.pleroma.social/backend/installation/centos7_en/)
 ### OS/Distro packages
 Currently Pleroma is not packaged by any OS/Distros, but if you want to package it for one, we can guide you through the process on our [community channels](#community-channels). If you want to change default options in your Pleroma package, please **discuss it with us first**.
--- a/config/config.exs
+++ b/config/config.exs
@ -130,6 +130,7 @@
    dispatch: [
      {:_,
       [
         {"/api/fedsocket/v1", Pleroma.Web.FedSockets.IncomingHandler, []},
         {"/api/v1/streaming", Pleroma.Web.MastodonAPI.WebsocketHandler, []},
         {"/websocket", Phoenix.Endpoint.CowboyWebSocket,
          {Phoenix.Transports.WebSocket,
@ -148,6 +149,16 @@
    "SameSite=Lax"
  ]
 config :pleroma, :fed_sockets,
  enabled: false,
  connection_duration: :timer.hours(8),
  rejection_duration: :timer.minutes(15),
  fed_socket_fetches: [
    default: 12_000,
    interval: 3_000,
    lazy: false
  ]
 # Configures Elixir's Logger
 config :logger, :console,
  level: :debug,
@ -216,7 +227,6 @@
  allow_relay: true,
  public: true,
  quarantined_instances: [],
  managed_config: true,
  static_dir: "instance/static/",
  allowed_post_formats: [
    "text/plain",
@ -424,6 +434,8 @@
  proxy_opts: [
    redirect_on_failure: false,
    max_body_length: 25 * 1_048_576,
    # Note: max_read_duration defaults to Pleroma.ReverseProxy.max_read_duration_default/1
    max_read_duration: 30_000,
    http: [
      follow_redirect: true,
      pool: :media
@ -438,6 +450,14 @@
 config :pleroma, Pleroma.Web.MediaProxy.Invalidation.Script, script_path: nil
 # Note: media preview proxy depends on media proxy to be enabled
 config :pleroma, :media_preview_proxy,
  enabled: false,
  thumbnail_max_width: 600,
  thumbnail_max_height: 600,
  image_quality: 85,
  min_content_length: 100 * 1024
 config :pleroma, :chat, enabled: true
 config :phoenix, :format_encoders, json: Jason
@ -530,8 +550,10 @@
  log: false,
  queues: [
    activity_expiration: 10,
    token_expiration: 5,
    federator_incoming: 50,
    federator_outgoing: 50,
    ingestion_queue: 50,
    web_push: 50,
    mailer: 10,
    transmogrifier: 20,
@ -543,8 +565,6 @@
  ],
  plugins: [Oban.Plugins.Pruner],
  crontab: [
    {"0 0 * * *", Pleroma.Workers.Cron.ClearOauthTokenWorker},
    {"* * * * *", Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker},
    {"0 0 * * 0", Pleroma.Workers.Cron.DigestEmailsWorker},
    {"0 0 * * *", Pleroma.Workers.Cron.NewUsersDigestWorker}
  ]
@ -655,7 +675,7 @@
  account_confirmation_resend: {8_640_000, 5},
  ap_routes: {60_000, 15}
-config :pleroma, Pleroma.ActivityExpiration, enabled: true
+config :pleroma, Pleroma.Workers.PurgeExpiredActivity, enabled: true, min_lifetime: 600
 config :pleroma, Pleroma.Plugs.RemoteIp, enabled: true
@ -743,8 +763,8 @@
  ],
  media: [
    size: 50,
-    max_waiting: 10,
+    max_waiting: 20,
-    recv_timeout: 10_000
+    recv_timeout: 15_000
  ],
  upload: [
    size: 25,
@ -789,6 +809,8 @@
 config :ex_aws, http_client: Pleroma.HTTP.ExAws
 config :web_push_encryption, http_client: Pleroma.HTTP
 config :pleroma, :instances_favicons, enabled: false
 config :floki, :html_parser, Floki.HTMLParser.FastHtml
--- a/config/description.exs
+++ b/config/description.exs
@ -270,6 +270,19 @@
      }
    ]
  },
  %{
    group: :pleroma,
    key: :fed_sockets,
    type: :group,
    description: "Websocket based federation",
    children: [
      %{
        key: :enabled,
        type: :boolean,
        description: "Enable FedSockets"
      }
    ]
  },
  %{
    group: :pleroma,
    key: Pleroma.Emails.Mailer,
@ -764,12 +777,6 @@
          "*.quarantined.com"
        ]
      },
      %{
        key: :managed_config,
        type: :boolean,
        description:
          "Whenether the config for pleroma-fe is configured in this config or in static/config.json"
      },
      %{
        key: :static_dir,
        type: :string,
@ -1880,6 +1887,7 @@
        suggestions: [
          redirect_on_failure: false,
          max_body_length: 25 * 1_048_576,
          max_read_duration: 30_000,
          http: [
            follow_redirect: true,
            pool: :media
@ -1900,6 +1908,11 @@
              "Limits the content length to be approximately the " <>
                "specified length. It is validated with the `content-length` header and also verified when proxying."
          },
          %{
            key: :max_read_duration,
            type: :integer,
            description: "Timeout (in milliseconds) of GET request to remote URI."
          },
          %{
            key: :http,
            label: "HTTP",
@ -1946,6 +1959,43 @@
      }
    ]
  },
  %{
    group: :pleroma,
    key: :media_preview_proxy,
    type: :group,
    description: "Media preview proxy",
    children: [
      %{
        key: :enabled,
        type: :boolean,
        description:
          "Enables proxying of remote media preview to the instance's proxy. Requires enabled media proxy."
      },
      %{
        key: :thumbnail_max_width,
        type: :integer,
        description:
          "Max width of preview thumbnail for images (video preview always has original dimensions)."
      },
      %{
        key: :thumbnail_max_height,
        type: :integer,
        description:
          "Max height of preview thumbnail for images (video preview always has original dimensions)."
      },
      %{
        key: :image_quality,
        type: :integer,
        description: "Quality of the output. Ranges from 0 (min quality) to 100 (max quality)."
      },
      %{
        key: :min_content_length,
        type: :integer,
        description:
          "Min content length to perform preview, in bytes. If greater than 0, media smaller in size will be served as is, without thumbnailing."
      }
    ]
  },
  %{
    group: :pleroma,
    key: Pleroma.Web.MediaProxy.Invalidation.Http,
@ -2290,8 +2340,6 @@
        type: {:list, :tuple},
        description: "Settings for cron background jobs",
        suggestions: [
          {"0 0 * * *", Pleroma.Workers.Cron.ClearOauthTokenWorker},
          {"* * * * *", Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker},
          {"0 0 * * 0", Pleroma.Workers.Cron.DigestEmailsWorker},
          {"0 0 * * *", Pleroma.Workers.Cron.NewUsersDigestWorker}
        ]
@ -2397,7 +2445,7 @@
  %{
    group: :pleroma,
    key: Pleroma.Formatter,
-    label: "Auto Linker",
+    label: "Linkify",
    type: :group,
    description:
      "Configuration for Pleroma's link formatter which parses mentions, hashtags, and URLs.",
@ -2474,14 +2522,20 @@
  },
  %{
    group: :pleroma,
-    key: Pleroma.ActivityExpiration,
+    key: Pleroma.Workers.PurgeExpiredActivity,
    type: :group,
-    description: "Expired activity settings",
+    description: "Expired activities settings",
    children: [
      %{
        key: :enabled,
        type: :boolean,
-        description: "Whether expired activities will be sent to the job queue to be deleted"
+        description: "Enables expired activities addition & deletion"
      },
      %{
        key: :min_lifetime,
        type: :integer,
        description: "Minimum lifetime for ephemeral activity (in seconds)",
        suggestions: [600]
      }
    ]
  },
--- a/config/test.exs
+++ b/config/test.exs
@ -19,6 +19,11 @@
  level: :warn,
  format: "\n[$level] $message\n"
 config :pleroma, :fed_sockets,
  enabled: false,
  connection_duration: 5,
  rejection_duration: 5
 config :pleroma, :auth, oauth_consumer_strategies: []
 config :pleroma, Pleroma.Upload,
--- a/docs/API/admin_api.md
+++ b/docs/API/admin_api.md
@ -1334,3 +1334,166 @@ Loads json generated from `config/descriptions.exs`.
 { }
 ```
 ## GET /api/pleroma/admin/users/:nickname/chats
 ### List a user's chats
 - Params: None
 - Response:
 ```json
 [
   {
      "sender": {
        "id": "someflakeid",
        "username": "somenick",
        ...
      },
      "receiver": {
        "id": "someflakeid",
        "username": "somenick",
        ...
      },
      "id" : "1",
      "unread" : 2,
      "last_message" : {...}, // The last message in that chat
      "updated_at": "2020-04-21T15:11:46.000Z"
   }
 ]
 ```
 ## GET /api/pleroma/admin/chats/:chat_id
 ### View a single chat
 - Params: None
 - Response:
 ```json
 {
  "sender": {
    "id": "someflakeid",
    "username": "somenick",
    ...
  },
  "receiver": {
    "id": "someflakeid",
    "username": "somenick",
    ...
  },
  "id" : "1",
  "unread" : 2,
  "last_message" : {...}, // The last message in that chat
  "updated_at": "2020-04-21T15:11:46.000Z"
 }
 ```
 ## GET /api/pleroma/admin/chats/:chat_id/messages
 ### List the messages in a chat
 - Params: `max_id`, `min_id`
 - Response:
 ```json
 [
  {
    "account_id": "someflakeid",
    "chat_id": "1",
    "content": "Check this out :firefox:",
    "created_at": "2020-04-21T15:11:46.000Z",
    "emojis": [
      {
        "shortcode": "firefox",
        "static_url": "https://dontbulling.me/emoji/Firefox.gif",
        "url": "https://dontbulling.me/emoji/Firefox.gif",
        "visible_in_picker": false
      }
    ],
    "id": "13",
    "unread": true
  },
  {
    "account_id": "someflakeid",
    "chat_id": "1",
    "content": "Whats' up?",
    "created_at": "2020-04-21T15:06:45.000Z",
    "emojis": [],
    "id": "12",
    "unread": false
  }
 ]
 ```
 ## DELETE /api/pleroma/admin/chats/:chat_id/messages/:message_id
 ### Delete a single message
 - Params: None
 - Response:
 ```json
 {
  "account_id": "someflakeid",
  "chat_id": "1",
  "content": "Check this out :firefox:",
  "created_at": "2020-04-21T15:11:46.000Z",
  "emojis": [
    {
      "shortcode": "firefox",
      "static_url": "https://dontbulling.me/emoji/Firefox.gif",
      "url": "https://dontbulling.me/emoji/Firefox.gif",
      "visible_in_picker": false
    }
  ],
  "id": "13",
  "unread": false
 }
 ```
 ## `GET /api/pleroma/admin/instance_document/:document_name`
 ### Get an instance document
 - Authentication: required
 - Response:
 Returns the content of the document
 ```html
 <h1>Instance panel</h1>
 ```
 ## `PATCH /api/pleroma/admin/instance_document/:document_name`
 - Params:
  - `file` (the file to be uploaded, using multipart form data.)
 ### Update an instance document
 - Authentication: required
 - Response:
 ``` json
 {
  "url": "https://example.com/instance/panel.html"
 }
 ```
 ## `DELETE /api/pleroma/admin/instance_document/:document_name`
 ### Delete an instance document
 - Response:
 ``` json
 {
  "url": "https://example.com/instance/panel.html"
 }
 ```
--- a/docs/API/pleroma_api.md
+++ b/docs/API/pleroma_api.md
@ -44,6 +44,22 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
 * Response: HTTP 200 on success, 500 on error
 * Note: Users that can't be followed are silently skipped.
 ## `/api/pleroma/blocks_import`
 ### Imports your blocks.
 * Method: `POST`
 * Authentication: required
 * Params:
    * `list`: STRING or FILE containing a whitespace-separated list of accounts to block
 * Response: HTTP 200 on success, 500 on error
 ## `/api/pleroma/mutes_import`
 ### Imports your mutes.
 * Method: `POST`
 * Authentication: required
 * Params:
    * `list`: STRING or FILE containing a whitespace-separated list of accounts to mute
 * Response: HTTP 200 on success, 500 on error
 ## `/api/pleroma/captcha`
 ### Get a new captcha
 * Method: `GET`
--- a/docs/configuration/cheatsheet.md
+++ b/docs/configuration/cheatsheet.md
@ -18,7 +18,7 @@ To add configuration to your config file, you can copy it from the base config.
 * `notify_email`: Email used for notifications.
 * `description`: The instance’s description, can be seen in nodeinfo and ``/api/v1/instance``.
 * `limit`: Posts character limit (CW/Subject included in the counter).
-* `discription_limit`: The character limit for image descriptions.
+* `description_limit`: The character limit for image descriptions.
 * `chat_limit`: Character limit of the instance chat messages.
 * `remote_limit`: Hard character limit beyond which remote posts will be dropped.
 * `upload_limit`: File size limit of uploads (except for avatar, background, banner).
@ -40,7 +40,6 @@ To add configuration to your config file, you can copy it from the base config.
 * `allow_relay`: Enable Pleroma’s Relay, which makes it possible to follow a whole instance.
 * `public`: Makes the client API in authenticated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network. Note that there is a dependent setting restricting or allowing unauthenticated access to specific resources, see `restrict_unauthenticated` for more details.
 * `quarantined_instances`: List of ActivityPub instances where private (DMs, followers-only) activities will not be send.
 * `managed_config`: Whenether the config for pleroma-fe is configured in [:frontend_configurations](#frontend_configurations) or in ``static/config.json``.
 * `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML).
 * `extended_nickname_format`: Set to `true` to use extended local nicknames format (allows underscores/dashes). This will break federation with
    older software for theses nicknames.
@ -226,6 +225,16 @@ Enables the worker which processes posts scheduled for deletion. Pinned posts ar
 * `enabled`: whether expired activities will be sent to the job queue to be deleted
 ## FedSockets
 FedSockets is an experimental feature allowing for Pleroma backends to federate using a persistant websocket connection as opposed to making each federation a seperate http connection. This feature is currently off by default. It is configurable throught he following options.
 ### :fedsockets
 * `enabled`: Enables FedSockets for this instance. `false` by default.
 * `connection_duration`: Time an idle websocket is kept open.
 * `rejection_duration`: Failures to connect via FedSockets will not be retried for this period of time.
 * `fed_socket_fetches` and `fed_socket_rejections`: Settings passed to `cachex` for the fetch registry, and rejection stacks. See `Pleroma.Web.FedSockets` for more details.
 ## Frontends
 ### :frontend_configurations
@ -315,6 +324,14 @@ This section describe PWA manifest instance-specific values. Currently this opti
  * `enabled`: Enables purge cache
  * `provider`: Which one of  the [purge cache strategy](#purge-cache-strategy) to use.
 ## :media_preview_proxy
 * `enabled`: Enables proxying of remote media preview to the instance’s proxy. Requires enabled media proxy (`media_proxy/enabled`).
 * `thumbnail_max_width`: Max width of preview thumbnail for images (video preview always has original dimensions).
 * `thumbnail_max_height`: Max height of preview thumbnail for images (video preview always has original dimensions).
 * `image_quality`: Quality of the output. Ranges from 0 (min quality) to 100 (max quality).
 * `min_content_length`: Min content length to perform preview, in bytes. If greater than 0, media smaller in size will be served as is, without thumbnailing.
 ### Purge cache strategy
 #### Pleroma.Web.MediaProxy.Invalidation.Script
@ -691,9 +708,8 @@ Pleroma has the following queues:
 Pleroma has these periodic job workers:
-`Pleroma.Workers.Cron.ClearOauthTokenWorker` - a job worker to cleanup expired oauth tokens.
+* `Pleroma.Workers.Cron.DigestEmailsWorker` - digest emails for users with new mentions and follows
-
+* `Pleroma.Workers.Cron.NewUsersDigestWorker` - digest emails for admins with new registrations
 Example:
 ```elixir
 config :pleroma, Oban,
@ -705,7 +721,8 @@ config :pleroma, Oban,
    federator_outgoing: 50
  ],
  crontab: [
-    {"0 0 * * *", Pleroma.Workers.Cron.ClearOauthTokenWorker}
+    {"0 0 * * 0", Pleroma.Workers.Cron.DigestEmailsWorker},
    {"0 0 * * *", Pleroma.Workers.Cron.NewUsersDigestWorker}
  ]
 ```
@ -972,7 +989,7 @@ Configure OAuth 2 provider capabilities:
 * `token_expires_in` - The lifetime in seconds of the access token.
 * `issue_new_refresh_token` - Keeps old refresh token or generate new refresh token when to obtain an access token.
-* `clean_expired_tokens` - Enable a background job to clean expired oauth tokens. Defaults to `false`. Interval settings sets in configuration periodic jobs [`Oban.Cron`](#obancron)
+* `clean_expired_tokens` - Enable a background job to clean expired oauth tokens. Defaults to `false`.
 ## Link parsing
@ -1091,3 +1108,10 @@ config :pleroma, :frontends,
 ```
 This would serve the frontend from the the folder at `$instance_static/frontends/pleroma/stable`. You have to copy the frontend into this folder yourself. You can choose the name and ref any way you like, but they will be used by mix tasks to automate installation in the future, the name referring to the project and the ref referring to a commit.
 ## Ephemeral activities (Pleroma.Workers.PurgeExpiredActivity)
 Settings to enable and configure expiration for ephemeral activities
 * `:enabled` - enables ephemeral activities creation
 * `:min_lifetime` - minimum lifetime for ephemeral activities (in seconds). Default: 10 minutes.
--- a/installation/pleroma.nginx
+++ b/installation/pleroma.nginx
@ -9,6 +9,12 @@
 proxy_cache_path /tmp/pleroma-media-cache levels=1:2 keys_zone=pleroma_media_cache:10m max_size=10g
                 inactive=720m use_temp_path=off;
 # this is explicitly IPv4 since Pleroma.Web.Endpoint binds on IPv4 only
 # and `localhost.` resolves to [::0] on some systems: see issue #930
 upstream phoenix {
    server 127.0.0.1:4000 max_fails=5 fail_timeout=60s;
 }
 server {
    server_name    example.tld;
@ -63,19 +69,16 @@ server {
    # the nginx default is 1m, not enough for large media uploads
    client_max_body_size 16m;
    ignore_invalid_headers off;
    location / {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        # this is explicitly IPv4 since Pleroma.Web.Endpoint binds on IPv4 only
+    location / {
-        # and `localhost.` resolves to [::0] on some systems: see issue #930
+        proxy_pass http://phoenix;
        proxy_pass http://127.0.0.1:4000;
        client_max_body_size 16m;
    }
    location ~ ^/(media|proxy) {
@ -83,12 +86,16 @@ server {
        slice              1m;
        proxy_cache_key    $host$uri$is_args$args$slice_range;
        proxy_set_header   Range $slice_range;
        proxy_http_version 1.1;
        proxy_cache_valid  200 206 301 304 1h;
        proxy_cache_lock   on;
        proxy_ignore_client_abort on;
        proxy_buffering    on;
        chunked_transfer_encoding on;
-        proxy_pass         http://127.0.0.1:4000;
+        proxy_pass         http://phoenix;
    }
    location /api/fedsocket/v1 {
        proxy_request_buffering off;
        proxy_pass http://phoenix/api/fedsocket/v1;
    }
 }
--- a/lib/mix/tasks/pleroma/config.ex
+++ b/lib/mix/tasks/pleroma/config.ex
@ -32,7 +32,8 @@ def run(["migrate_from_db" | options]) do
  @spec migrate_to_db(Path.t() | nil) :: any()
  def migrate_to_db(file_path \\ nil) do
-    if Pleroma.Config.get([:configurable_from_database]) do
+    with true <- Pleroma.Config.get([:configurable_from_database]),
         :ok <- Pleroma.Config.DeprecationWarnings.warn() do
      config_file =
        if file_path do
          file_path
@ -46,7 +47,8 @@ def migrate_to_db(file_path \\ nil) do
      do_migrate_to_db(config_file)
    else
-      migration_error()
+      :error -> deprecation_error()
      _ -> migration_error()
    end
  end
@ -120,6 +122,10 @@ defp migration_error do
    )
  end
  defp deprecation_error do
    shell_error("Migration is not allowed until all deprecation warnings have been resolved.")
  end
  if Code.ensure_loaded?(Config.Reader) do
    defp config_header, do: "import Config\r\n\r\n"
    defp read_file(config_file), do: Config.Reader.read_imports!(config_file)
--- a/lib/mix/tasks/pleroma/database.ex
+++ b/lib/mix/tasks/pleroma/database.ex
@ -99,7 +99,7 @@ def run(["fix_likes_collections"]) do
      where: fragment("(?)->>'likes' is not null", object.data),
      select: %{id: object.id, likes: fragment("(?)->>'likes'", object.data)}
    )
-    |> Pleroma.RepoStreamer.chunk_stream(100)
+    |> Pleroma.Repo.chunk_stream(100, :batches)
    |> Stream.each(fn objects ->
      ids =
        objects
@ -133,8 +133,7 @@ def run(["ensure_expiration"]) do
    days = Pleroma.Config.get([:mrf_activity_expiration, :days], 365)
    Pleroma.Activity
-    |> join(:left, [a], u in assoc(a, :expiration))
+    |> join(:inner, [a], o in Object,
    |> join(:inner, [a, _u], o in Object,
      on:
        fragment(
          "(?->>'id') = COALESCE((?)->'object'->> 'id', (?)->>'object')",
@ -144,14 +143,20 @@ def run(["ensure_expiration"]) do
        )
    )
    |> where(local: true)
    |> where([a, u], is_nil(u))
    |> where([a], fragment("(? ->> 'type'::text) = 'Create'", a.data))
-    |> where([_a, _u, o], fragment("?->>'type' = 'Note'", o.data))
+    |> where([_a, o], fragment("?->>'type' = 'Note'", o.data))
-    |> Pleroma.RepoStreamer.chunk_stream(100)
+    |> Pleroma.Repo.chunk_stream(100, :batches)
    |> Stream.each(fn activities ->
      Enum.each(activities, fn activity ->
-        expires_at = Timex.shift(activity.inserted_at, days: days)
+        expires_at =
-        Pleroma.ActivityExpiration.create(activity, expires_at, false)
+          activity.inserted_at
          |> DateTime.from_naive!("Etc/UTC")
          |> Timex.shift(days: days)
        Pleroma.Workers.PurgeExpiredActivity.enqueue(%{
          activity_id: activity.id,
          expires_at: expires_at
        })
      end)
    end)
    |> Stream.run()
--- a/lib/mix/tasks/pleroma/emoji.ex
+++ b/lib/mix/tasks/pleroma/emoji.ex
@ -183,7 +183,7 @@ def run(["gen-pack" | args]) do
    IO.puts("Downloading the pack and generating SHA256")
-    binary_archive = Tesla.get!(client(), src).body
+    {:ok, %{body: binary_archive}} = Pleroma.HTTP.get(src)
    archive_sha = :crypto.hash(:sha256, binary_archive) |> Base.encode16()
    IO.puts("SHA256 is #{archive_sha}")
@ -252,7 +252,7 @@ defp fetch_and_decode!(from) do
  end
  defp fetch("http" <> _ = from) do
-    with {:ok, %{body: body}} <- Tesla.get(client(), from) do
+    with {:ok, %{body: body}} <- Pleroma.HTTP.get(from) do
      {:ok, body}
    end
  end
@ -271,13 +271,5 @@ defp parse_global_opts(args) do
    )
  end
  defp client do
    middleware = [
      {Tesla.Middleware.FollowRedirects, [max_redirects: 3]}
    ]
    Tesla.client(middleware)
  end
  defp default_manifest, do: Pleroma.Config.get!([:emoji, :default_manifest])
 end
--- a/lib/mix/tasks/pleroma/user.ex
+++ b/lib/mix/tasks/pleroma/user.ex
@ -179,7 +179,7 @@ def run(["deactivate_all_from_instance", instance]) do
    start_pleroma()
    Pleroma.User.Query.build(%{nickname: "@#{instance}"})
-    |> Pleroma.RepoStreamer.chunk_stream(500)
+    |> Pleroma.Repo.chunk_stream(500, :batches)
    |> Stream.each(fn users ->
      users
      |> Enum.each(fn user ->
@ -413,7 +413,7 @@ def run(["list"]) do
    start_pleroma()
    Pleroma.User.Query.build(%{local: true})
-    |> Pleroma.RepoStreamer.chunk_stream(500)
+    |> Pleroma.Repo.chunk_stream(500, :batches)
    |> Stream.each(fn users ->
      users
      |> Enum.each(fn user ->
--- a/lib/pleroma/activity.ex
+++ b/lib/pleroma/activity.ex
@ -7,7 +7,6 @@ defmodule Pleroma.Activity do
  alias Pleroma.Activity
  alias Pleroma.Activity.Queries
  alias Pleroma.ActivityExpiration
  alias Pleroma.Bookmark
  alias Pleroma.Notification
  alias Pleroma.Object
@ -60,8 +59,6 @@ defmodule Pleroma.Activity do
    # typical case.
    has_one(:object, Object, on_delete: :nothing, foreign_key: :id)
    has_one(:expiration, ActivityExpiration, on_delete: :delete_all)
    timestamps()
  end
@ -304,14 +301,14 @@ def all_by_actor_and_id(actor, status_ids) do
    |> Repo.all()
  end
-  def follow_requests_for_actor(%Pleroma.User{ap_id: ap_id}) do
+  def follow_requests_for_actor(%User{ap_id: ap_id}) do
    ap_id
    |> Queries.by_object_id()
    |> Queries.by_type("Follow")
    |> where([a], fragment("? ->> 'state' = 'pending'", a.data))
  end
-  def following_requests_for_actor(%Pleroma.User{ap_id: ap_id}) do
+  def following_requests_for_actor(%User{ap_id: ap_id}) do
    Queries.by_type("Follow")
    |> where([a], fragment("?->>'state' = 'pending'", a.data))
    |> where([a], a.actor == ^ap_id)
--- a/lib/pleroma/activity_expiration.ex
+++ b/lib/pleroma/activity_expiration.ex
@ -1,74 +0,0 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.ActivityExpiration do
  use Ecto.Schema
  alias Pleroma.Activity
  alias Pleroma.ActivityExpiration
  alias Pleroma.Repo
  import Ecto.Changeset
  import Ecto.Query
  @type t :: %__MODULE__{}
  @min_activity_lifetime :timer.hours(1)
  schema "activity_expirations" do
    belongs_to(:activity, Activity, type: FlakeId.Ecto.CompatType)
    field(:scheduled_at, :naive_datetime)
  end
  def changeset(%ActivityExpiration{} = expiration, attrs, validate_scheduled_at) do
    expiration
    |> cast(attrs, [:scheduled_at])
    |> validate_required([:scheduled_at])
    |> validate_scheduled_at(validate_scheduled_at)
  end
  def get_by_activity_id(activity_id) do
    ActivityExpiration
    |> where([exp], exp.activity_id == ^activity_id)
    |> Repo.one()
  end
  def create(%Activity{} = activity, scheduled_at, validate_scheduled_at \\ true) do
    %ActivityExpiration{activity_id: activity.id}
    |> changeset(%{scheduled_at: scheduled_at}, validate_scheduled_at)
    |> Repo.insert()
  end
  def due_expirations(offset \\ 0) do
    naive_datetime =
      NaiveDateTime.utc_now()
      |> NaiveDateTime.add(offset, :millisecond)
    ActivityExpiration
    |> where([exp], exp.scheduled_at < ^naive_datetime)
    |> limit(50)
    |> preload(:activity)
    |> Repo.all()
    |> Enum.reject(fn %{activity: activity} ->
      Activity.pinned_by_actor?(activity)
    end)
  end
  def validate_scheduled_at(changeset, false), do: changeset
  def validate_scheduled_at(changeset, true) do
    validate_change(changeset, :scheduled_at, fn _, scheduled_at ->
      if not expires_late_enough?(scheduled_at) do
        [scheduled_at: "an ephemeral activity must live for at least one hour"]
      else
        []
      end
    end)
  end
  def expires_late_enough?(scheduled_at) do
    now = NaiveDateTime.utc_now()
    diff = NaiveDateTime.diff(scheduled_at, now, :millisecond)
    diff > @min_activity_lifetime
  end
 end
--- a/lib/pleroma/application.ex
+++ b/lib/pleroma/application.ex
@ -99,7 +99,7 @@ def start(_type, _args) do
          {Oban, Config.get(Oban)}
        ] ++
        task_children(@env) ++
-        streamer_child(@env) ++
+        dont_run_in_test(@env) ++
        chat_child(@env, chat_enabled?()) ++
        [
          Pleroma.Web.Endpoint,
@ -188,16 +188,17 @@ def build_cachex(type, opts),
  defp chat_enabled?, do: Config.get([:chat, :enabled])
-  defp streamer_child(env) when env in [:test, :benchmark], do: []
+  defp dont_run_in_test(env) when env in [:test, :benchmark], do: []
-  defp streamer_child(_) do
+  defp dont_run_in_test(_) do
    [
      {Registry,
       [
         name: Pleroma.Web.Streamer.registry(),
         keys: :duplicate,
         partitions: System.schedulers_online()
-       ]}
+       ]},
      Pleroma.Web.FedSockets.Supervisor
    ]
  end
--- a/lib/pleroma/chat.ex
+++ b/lib/pleroma/chat.ex
@ -6,7 +6,9 @@ defmodule Pleroma.Chat do
  use Ecto.Schema
  import Ecto.Changeset
  import Ecto.Query
  alias Pleroma.Chat
  alias Pleroma.Repo
  alias Pleroma.User
@ -16,6 +18,7 @@ defmodule Pleroma.Chat do
  It is a helper only, to make it easy to display a list of chats with other people, ordered by last bump. The actual messages are retrieved by querying the recipients of the ChatMessages.
  """
  @type t :: %__MODULE__{}
  @primary_key {:id, FlakeId.Ecto.CompatType, autogenerate: true}
  schema "chats" do
@ -39,16 +42,28 @@ def changeset(struct, params) do
    |> unique_constraint(:user_id, name: :chats_user_id_recipient_index)
  end
  @spec get_by_user_and_id(User.t(), FlakeId.Ecto.CompatType.t()) ::
          {:ok, t()} | {:error, :not_found}
  def get_by_user_and_id(%User{id: user_id}, id) do
    from(c in __MODULE__,
      where: c.id == ^id,
      where: c.user_id == ^user_id
    )
    |> Repo.find_resource()
  end
  @spec get_by_id(FlakeId.Ecto.CompatType.t()) :: t() | nil
  def get_by_id(id) do
-    __MODULE__
+    Repo.get(__MODULE__, id)
    |> Repo.get(id)
  end
  @spec get(FlakeId.Ecto.CompatType.t(), String.t()) :: t() | nil
  def get(user_id, recipient) do
-    __MODULE__
+    Repo.get_by(__MODULE__, user_id: user_id, recipient: recipient)
    |> Repo.get_by(user_id: user_id, recipient: recipient)
  end
  @spec get_or_create(FlakeId.Ecto.CompatType.t(), String.t()) ::
          {:ok, t()} | {:error, Ecto.Changeset.t()}
  def get_or_create(user_id, recipient) do
    %__MODULE__{}
    |> changeset(%{user_id: user_id, recipient: recipient})
@ -60,6 +75,8 @@ def get_or_create(user_id, recipient) do
    )
  end
  @spec bump_or_create(FlakeId.Ecto.CompatType.t(), String.t()) ::
          {:ok, t()} | {:error, Ecto.Changeset.t()}
  def bump_or_create(user_id, recipient) do
    %__MODULE__{}
    |> changeset(%{user_id: user_id, recipient: recipient})
@ -69,4 +86,12 @@ def bump_or_create(user_id, recipient) do
      conflict_target: [:user_id, :recipient]
    )
  end
  @spec for_user_query(FlakeId.Ecto.CompatType.t()) :: Ecto.Query.t()
  def for_user_query(user_id) do
    from(c in Chat,
      where: c.user_id == ^user_id,
      order_by: [desc: c.updated_at]
    )
  end
 end
--- a/lib/pleroma/config/deprecation_warnings.ex
+++ b/lib/pleroma/config/deprecation_warnings.ex
@ -8,7 +8,7 @@ defmodule Pleroma.Config.DeprecationWarnings do
  require Logger
  alias Pleroma.Config
-  @type config_namespace() :: [atom()]
+  @type config_namespace() :: atom() | [atom()]
  @type config_map() :: {config_namespace(), config_namespace(), String.t()}
  @mrf_config_map [
@ -26,6 +26,10 @@ def check_hellthread_threshold do
      !!!DEPRECATION WARNING!!!
      You are using the old configuration mechanism for the hellthread filter. Please check config.md.
      """)
      :error
    else
      :ok
    end
  end
@ -47,16 +51,26 @@ def mrf_user_allowlist do
      config :pleroma, :mrf_user_allowlist, #{inspect(rewritten, pretty: true)}
      """)
      :error
    else
      :ok
    end
  end
  def warn do
-    check_hellthread_threshold()
+    with :ok <- check_hellthread_threshold(),
-    mrf_user_allowlist()
+         :ok <- mrf_user_allowlist(),
-    check_old_mrf_config()
+         :ok <- check_old_mrf_config(),
-    check_media_proxy_whitelist_config()
+         :ok <- check_media_proxy_whitelist_config(),
-    check_welcome_message_config()
+         :ok <- check_welcome_message_config(),
-    check_gun_pool_options()
+         :ok <- check_gun_pool_options(),
         :ok <- check_activity_expiration_config() do
      :ok
    else
      _ ->
        :error
    end
  end
  def check_welcome_message_config do
@ -73,6 +87,10 @@ def check_welcome_message_config do
      \n* `config :pleroma, :instance, welcome_user_nickname` is now `config :pleroma, :welcome, :direct_message, :sender_nickname`
      \n* `config :pleroma, :instance, welcome_message` is now `config :pleroma, :welcome, :direct_message, :message`
      """)
      :error
    else
      :ok
    end
  end
@ -100,8 +118,11 @@ def move_namespace_and_warn(config_map, warning_preface) do
          end
      end)
-    if warning != "" do
+    if warning == "" do
      :ok
    else
      Logger.warn(warning_preface <> warning)
      :error
    end
  end
@ -114,6 +135,10 @@ def check_media_proxy_whitelist_config do
      !!!DEPRECATION WARNING!!!
      Your config is using old format (only domain) for MediaProxy whitelist option. Setting should work for now, but you are advised to change format to scheme with port to prevent possible issues later.
      """)
      :error
    else
      :ok
    end
  end
@ -156,6 +181,25 @@ def check_gun_pool_options do
      Logger.warn(Enum.join([warning_preface | pool_warnings]))
      Config.put(:pools, updated_config)
      :error
    else
      :ok
    end
  end
  @spec check_activity_expiration_config() :: :ok | nil
  def check_activity_expiration_config do
    warning_preface = """
    !!!DEPRECATION WARNING!!!
      Your config is using old namespace for activity expiration configuration. Setting should work for now, but you are advised to change to new namespace to prevent possible issues later:
    """
    move_namespace_and_warn(
      [
        {Pleroma.ActivityExpiration, Pleroma.Workers.PurgeExpiredActivity,
         "\n* `config :pleroma, Pleroma.ActivityExpiration` is now `config :pleroma, Pleroma.Workers.PurgeExpiredActivity`"}
      ],
      warning_preface
    )
  end
 end
--- a/lib/pleroma/config/oban.ex
+++ b/lib/pleroma/config/oban.ex
@ -5,7 +5,11 @@ def warn do
    oban_config = Pleroma.Config.get(Oban)
    crontab =
-      [Pleroma.Workers.Cron.StatsWorker]
+      [
        Pleroma.Workers.Cron.StatsWorker,
        Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker,
        Pleroma.Workers.Cron.ClearOauthTokenWorker
      ]
      |> Enum.reduce(oban_config[:crontab], fn removed_worker, acc ->
        with acc when is_list(acc) <- acc,
             setting when is_tuple(setting) <-
--- a/lib/pleroma/emails/mailer.ex
+++ b/lib/pleroma/emails/mailer.ex
@ -35,6 +35,11 @@ def perform(:deliver_async, email, config), do: deliver(email, config)
  def deliver(email, config \\ [])
  def deliver(email, config) do
    # temporary hackney fix until hackney max_connections bug is fixed
    # https://git.pleroma.social/pleroma/pleroma/-/issues/2101
    email =
      Swoosh.Email.put_private(email, :hackney_options, ssl_options: [versions: [:"tlsv1.2"]])
    case enabled?() do
      true -> Swoosh.Mailer.deliver(email, parse_config(config))
      false -> {:error, :deliveries_disabled}
--- a/lib/pleroma/emoji.ex
+++ b/lib/pleroma/emoji.ex
@ -56,6 +56,9 @@ def get(name) do
    end
  end
  @spec exist?(String.t()) :: boolean()
  def exist?(name), do: not is_nil(get(name))
  @doc "Returns all the emojos!!"
  @spec get_all() :: list({String.t(), String.t(), String.t()})
  def get_all do
--- a/lib/pleroma/emoji/pack.ex
+++ b/lib/pleroma/emoji/pack.ex
@ -17,6 +17,7 @@ defmodule Pleroma.Emoji.Pack do
        }
  alias Pleroma.Emoji
  alias Pleroma.Emoji.Pack
  @spec create(String.t()) :: {:ok, t()} | {:error, File.posix()} | {:error, :empty_values}
  def create(name) do
@ -64,24 +65,93 @@ def delete(name) do
    end
  end
-  @spec add_file(String.t(), String.t(), Path.t(), Plug.Upload.t() | String.t()) ::
+  @spec unpack_zip_emojies(list(tuple())) :: list(map())
-          {:ok, t()} | {:error, File.posix() | atom()}
+  defp unpack_zip_emojies(zip_files) do
-  def add_file(name, shortcode, filename, file) do
+    Enum.reduce(zip_files, [], fn
-    with :ok <- validate_not_empty([name, shortcode, filename]),
+      {_, path, s, _, _, _}, acc when elem(s, 2) == :regular ->
        with(
          filename <- Path.basename(path),
          shortcode <- Path.basename(filename, Path.extname(filename)),
          false <- Emoji.exist?(shortcode)
        ) do
          [%{path: path, filename: path, shortcode: shortcode} | acc]
        else
          _ -> acc
        end
      _, acc ->
        acc
    end)
  end
  @spec add_file(t(), String.t(), Path.t(), Plug.Upload.t()) ::
          {:ok, t()}
          | {:error, File.posix() | atom()}
  def add_file(%Pack{} = pack, _, _, %Plug.Upload{content_type: "application/zip"} = file) do
    with {:ok, zip_files} <- :zip.table(to_charlist(file.path)),
         [_ | _] = emojies <- unpack_zip_emojies(zip_files),
         {:ok, tmp_dir} <- Pleroma.Utils.tmp_dir("emoji") do
      try do
        {:ok, _emoji_files} =
          :zip.unzip(
            to_charlist(file.path),
            [{:file_list, Enum.map(emojies, & &1[:path])}, {:cwd, tmp_dir}]
          )
        {_, updated_pack} =
          Enum.map_reduce(emojies, pack, fn item, emoji_pack ->
            emoji_file = %Plug.Upload{
              filename: item[:filename],
              path: Path.join(tmp_dir, item[:path])
            }
            {:ok, updated_pack} =
              do_add_file(
                emoji_pack,
                item[:shortcode],
                to_string(item[:filename]),
                emoji_file
              )
            {item, updated_pack}
          end)
        Emoji.reload()
        {:ok, updated_pack}
      after
        File.rm_rf(tmp_dir)
      end
    else
      {:error, _} = error ->
        error
      _ ->
        {:ok, pack}
    end
  end
  def add_file(%Pack{} = pack, shortcode, filename, %Plug.Upload{} = file) do
    with :ok <- validate_not_empty([shortcode, filename]),
         :ok <- validate_emoji_not_exists(shortcode),
-         {:ok, pack} <- load_pack(name),
+         {:ok, updated_pack} <- do_add_file(pack, shortcode, filename, file) do
         :ok <- save_file(file, pack, filename),
         {:ok, updated_pack} <- pack |> put_emoji(shortcode, filename) |> save_pack() do
      Emoji.reload()
      {:ok, updated_pack}
    end
  end
-  @spec delete_file(String.t(), String.t()) ::
+  defp do_add_file(pack, shortcode, filename, file) do
    with :ok <- save_file(file, pack, filename) do
      pack
      |> put_emoji(shortcode, filename)
      |> save_pack()
    end
  end
  @spec delete_file(t(), String.t()) ::
          {:ok, t()} | {:error, File.posix() | atom()}
-  def delete_file(name, shortcode) do
+  def delete_file(%Pack{} = pack, shortcode) do
-    with :ok <- validate_not_empty([name, shortcode]),
+    with :ok <- validate_not_empty([shortcode]),
         {:ok, pack} <- load_pack(name),
         :ok <- remove_file(pack, shortcode),
         {:ok, updated_pack} <- pack |> delete_emoji(shortcode) |> save_pack() do
      Emoji.reload()
@ -89,11 +159,10 @@ def delete_file(name, shortcode) do
    end
  end
-  @spec update_file(String.t(), String.t(), String.t(), String.t(), boolean()) ::
+  @spec update_file(t(), String.t(), String.t(), String.t(), boolean()) ::
          {:ok, t()} | {:error, File.posix() | atom()}
-  def update_file(name, shortcode, new_shortcode, new_filename, force) do
+  def update_file(%Pack{} = pack, shortcode, new_shortcode, new_filename, force) do
-    with :ok <- validate_not_empty([name, shortcode, new_shortcode, new_filename]),
+    with :ok <- validate_not_empty([shortcode, new_shortcode, new_filename]),
         {:ok, pack} <- load_pack(name),
         {:ok, filename} <- get_filename(pack, shortcode),
         :ok <- validate_emoji_not_exists(new_shortcode, force),
         :ok <- rename_file(pack, filename, new_filename),
@ -243,9 +312,10 @@ defp validate_emoji_not_exists(shortcode, force \\ false)
  defp validate_emoji_not_exists(_shortcode, true), do: :ok
  defp validate_emoji_not_exists(shortcode, _) do
-    case Emoji.get(shortcode) do
+    if Emoji.exist?(shortcode) do
-      nil -> :ok
+      {:error, :already_exists}
-      _ -> {:error, :already_exists}
+    else
      :ok
    end
  end
@ -386,25 +456,18 @@ defp validate_not_empty(list) do
    end
  end
-  defp save_file(file, pack, filename) do
+  defp save_file(%Plug.Upload{path: upload_path}, pack, filename) do
    file_path = Path.join(pack.path, filename)
    create_subdirs(file_path)
-    case file do
+    with {:ok, _} <- File.copy(upload_path, file_path) do
-      %Plug.Upload{path: upload_path} ->
+      :ok
        # Copy the uploaded file from the temporary directory
        with {:ok, _} <- File.copy(upload_path, file_path), do: :ok
      url when is_binary(url) ->
        # Download and write the file
        file_contents = Tesla.get!(url).body
        File.write(file_path, file_contents)
    end
  end
  defp put_emoji(pack, shortcode, filename) do
    files = Map.put(pack.files, shortcode, filename)
-    %{pack | files: files}
+    %{pack | files: files, files_count: length(Map.keys(files))}
  end
  defp delete_emoji(pack, shortcode) do
--- a/lib/pleroma/gun/conn.ex
+++ b/lib/pleroma/gun/conn.ex
@ -50,10 +50,10 @@ defp do_open(uri, %{proxy: {proxy_host, proxy_port}} = opts) do
    with open_opts <- Map.delete(opts, :tls_opts),
         {:ok, conn} <- Gun.open(proxy_host, proxy_port, open_opts),
-         {:ok, _} <- Gun.await_up(conn, opts[:connect_timeout]),
+         {:ok, protocol} <- Gun.await_up(conn, opts[:connect_timeout]),
         stream <- Gun.connect(conn, connect_opts),
         {:response, :fin, 200, _} <- Gun.await(conn, stream) do
-      {:ok, conn}
+      {:ok, conn, protocol}
    else
      error ->
        Logger.warn(
@ -88,8 +88,8 @@ defp do_open(uri, %{proxy: {proxy_type, proxy_host, proxy_port}} = opts) do
      |> Map.put(:socks_opts, socks_opts)
    with {:ok, conn} <- Gun.open(proxy_host, proxy_port, opts),
-         {:ok, _} <- Gun.await_up(conn, opts[:connect_timeout]) do
+         {:ok, protocol} <- Gun.await_up(conn, opts[:connect_timeout]) do
-      {:ok, conn}
+      {:ok, conn, protocol}
    else
      error ->
        Logger.warn(
@ -106,8 +106,8 @@ defp do_open(%URI{host: host, port: port} = uri, opts) do
    host = Pleroma.HTTP.AdapterHelper.parse_host(host)
    with {:ok, conn} <- Gun.open(host, port, opts),
-         {:ok, _} <- Gun.await_up(conn, opts[:connect_timeout]) do
+         {:ok, protocol} <- Gun.await_up(conn, opts[:connect_timeout]) do
-      {:ok, conn}
+      {:ok, conn, protocol}
    else
      error ->
        Logger.warn(
--- a/lib/pleroma/gun/connection_pool/worker.ex
+++ b/lib/pleroma/gun/connection_pool/worker.ex
@ -15,7 +15,7 @@ def init([_key, _uri, _opts, _client_pid] = opts) do
  @impl true
  def handle_continue({:connect, [key, uri, opts, client_pid]}, _) do
-    with {:ok, conn_pid} <- Gun.Conn.open(uri, opts),
+    with {:ok, conn_pid, protocol} <- Gun.Conn.open(uri, opts),
         Process.link(conn_pid) do
      time = :erlang.monotonic_time(:millisecond)
@ -27,8 +27,12 @@ def handle_continue({:connect, [key, uri, opts, client_pid]}, _) do
      send(client_pid, {:conn_pid, conn_pid})
      {:noreply,
-       %{key: key, timer: nil, client_monitors: %{client_pid => Process.monitor(client_pid)}},
+       %{
-       :hibernate}
+         key: key,
         timer: nil,
         client_monitors: %{client_pid => Process.monitor(client_pid)},
         protocol: protocol
       }, :hibernate}
    else
      err ->
        {:stop, {:shutdown, err}, nil}
@ -53,14 +57,20 @@ def handle_cast({:remove_client, client_pid}, state) do
  end
  @impl true
-  def handle_call(:add_client, {client_pid, _}, %{key: key} = state) do
+  def handle_call(:add_client, {client_pid, _}, %{key: key, protocol: protocol} = state) do
    time = :erlang.monotonic_time(:millisecond)
-    {{conn_pid, _, _, _}, _} =
+    {{conn_pid, used_by, _, _}, _} =
      Registry.update_value(@registry, key, fn {conn_pid, used_by, crf, last_reference} ->
        {conn_pid, [client_pid | used_by], crf(time - last_reference, crf), time}
      end)
    :telemetry.execute(
      [:pleroma, :connection_pool, :client, :add],
      %{client_pid: client_pid, clients: used_by},
      %{key: state.key, protocol: protocol}
    )
    state =
      if state.timer != nil do
        Process.cancel_timer(state[:timer])
@ -83,12 +93,8 @@ def handle_call(:remove_client, {client_pid, _}, %{key: key} = state) do
      end)
    {ref, state} = pop_in(state.client_monitors[client_pid])
-    # DOWN message can receive right after `remove_client` call and cause worker to terminate
+
-    state =
+    Process.demonitor(ref, [:flush])
      if is_nil(ref) do
        state
      else
        Process.demonitor(ref)
    timer =
      if used_by == [] do
@ -98,10 +104,7 @@ def handle_call(:remove_client, {client_pid, _}, %{key: key} = state) do
        nil
      end
-        %{state | timer: timer}
+    {:reply, :ok, %{state | timer: timer}, :hibernate}
      end
    {:reply, :ok, state, :hibernate}
  end
  @impl true
@ -131,7 +134,7 @@ def handle_info({:gun_down, _pid, _protocol, _reason, _killed_streams}, state) d
  @impl true
  def handle_info({:DOWN, _ref, :process, pid, reason}, state) do
    :telemetry.execute(
-      [:pleroma, :connection_pool, :client_death],
+      [:pleroma, :connection_pool, :client, :dead],
      %{client_pid: pid, reason: reason},
      %{key: state.key}
    )
--- a/lib/pleroma/helpers/media_helper.ex
+++ b/lib/pleroma/helpers/media_helper.ex
@ -0,0 +1,150 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Helpers.MediaHelper do
  @moduledoc """
  Handles common media-related operations.
  """
  alias Pleroma.HTTP
  def image_resize(url, options) do
    with executable when is_binary(executable) <- System.find_executable("convert"),
         {:ok, args} <- prepare_image_resize_args(options),
         {:ok, env} <- HTTP.get(url, [], pool: :media),
         {:ok, fifo_path} <- mkfifo() do
      args = List.flatten([fifo_path, args])
      run_fifo(fifo_path, env, executable, args)
    else
      nil -> {:error, {:convert, :command_not_found}}
      {:error, _} = error -> error
    end
  end
  defp prepare_image_resize_args(
         %{max_width: max_width, max_height: max_height, format: "png"} = options
       ) do
    quality = options[:quality] || 85
    resize = Enum.join([max_width, "x", max_height, ">"])
    args = [
      "-resize",
      resize,
      "-quality",
      to_string(quality),
      "png:-"
    ]
    {:ok, args}
  end
  defp prepare_image_resize_args(%{max_width: max_width, max_height: max_height} = options) do
    quality = options[:quality] || 85
    resize = Enum.join([max_width, "x", max_height, ">"])
    args = [
      "-interlace",
      "Plane",
      "-resize",
      resize,
      "-quality",
      to_string(quality),
      "jpg:-"
    ]
    {:ok, args}
  end
  defp prepare_image_resize_args(_), do: {:error, :missing_options}
  # Note: video thumbnail is intentionally not resized (always has original dimensions)
  def video_framegrab(url) do
    with executable when is_binary(executable) <- System.find_executable("ffmpeg"),
         {:ok, env} <- HTTP.get(url, [], pool: :media),
         {:ok, fifo_path} <- mkfifo(),
         args = [
           "-y",
           "-i",
           fifo_path,
           "-vframes",
           "1",
           "-f",
           "mjpeg",
           "-loglevel",
           "error",
           "-"
         ] do
      run_fifo(fifo_path, env, executable, args)
    else
      nil -> {:error, {:ffmpeg, :command_not_found}}
      {:error, _} = error -> error
    end
  end
  defp run_fifo(fifo_path, env, executable, args) do
    pid =
      Port.open({:spawn_executable, executable}, [
        :use_stdio,
        :stream,
        :exit_status,
        :binary,
        args: args
      ])
    fifo = Port.open(to_charlist(fifo_path), [:eof, :binary, :stream, :out])
    fix = Pleroma.Helpers.QtFastStart.fix(env.body)
    true = Port.command(fifo, fix)
    :erlang.port_close(fifo)
    loop_recv(pid)
  after
    File.rm(fifo_path)
  end
  defp mkfifo do
    path = Path.join(System.tmp_dir!(), "pleroma-media-preview-pipe-#{Ecto.UUID.generate()}")
    case System.cmd("mkfifo", [path]) do
      {_, 0} ->
        spawn(fifo_guard(path))
        {:ok, path}
      {_, err} ->
        {:error, {:fifo_failed, err}}
    end
  end
  defp fifo_guard(path) do
    pid = self()
    fn ->
      ref = Process.monitor(pid)
      receive do
        {:DOWN, ^ref, :process, ^pid, _} ->
          File.rm(path)
      end
    end
  end
  defp loop_recv(pid) do
    loop_recv(pid, <<>>)
  end
  defp loop_recv(pid, acc) do
    receive do
      {^pid, {:data, data}} ->
        loop_recv(pid, acc <> data)
      {^pid, {:exit_status, 0}} ->
        {:ok, acc}
      {^pid, {:exit_status, status}} ->
        {:error, status}
    after
      5000 ->
        :erlang.port_close(pid)
        {:error, :timeout}
    end
  end
 end
--- a/lib/pleroma/helpers/qt_fast_start.ex
+++ b/lib/pleroma/helpers/qt_fast_start.ex
@ -0,0 +1,131 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Helpers.QtFastStart do
  @moduledoc """
  (WIP) Converts a "slow start" (data before metadatas) mov/mp4 file to a "fast start" one (metadatas before data).
  """
  # TODO: Cleanup and optimizations
  # Inspirations: https://www.ffmpeg.org/doxygen/3.4/qt-faststart_8c_source.html
  #               https://github.com/danielgtaylor/qtfaststart/blob/master/qtfaststart/processor.py
  #               ISO/IEC 14496-12:2015, ISO/IEC 15444-12:2015
  #               Paracetamol
  def fix(<<0x00, 0x00, 0x00, _, 0x66, 0x74, 0x79, 0x70, _::bits>> = binary) do
    index = fix(binary, 0, nil, nil, [])
    case index do
      :abort -> binary
      [{"ftyp", _, _, _, _}, {"mdat", _, _, _, _} | _] -> faststart(index)
      [{"ftyp", _, _, _, _}, {"free", _, _, _, _}, {"mdat", _, _, _, _} | _] -> faststart(index)
      _ -> binary
    end
  end
  def fix(binary) do
    binary
  end
  # MOOV have been seen before MDAT- abort
  defp fix(<<_::bits>>, _, true, false, _) do
    :abort
  end
  defp fix(
         <<size::integer-big-size(32), fourcc::bits-size(32), rest::bits>>,
         pos,
         got_moov,
         got_mdat,
         acc
       ) do
    full_size = (size - 8) * 8
    <<data::bits-size(full_size), rest::bits>> = rest
    acc = [
      {fourcc, pos, pos + size, size,
       <<size::integer-big-size(32), fourcc::bits-size(32), data::bits>>}
      | acc
    ]
    fix(rest, pos + size, got_moov || fourcc == "moov", got_mdat || fourcc == "mdat", acc)
  end
  defp fix(<<>>, _pos, _, _, acc) do
    :lists.reverse(acc)
  end
  defp faststart(index) do
    {{_ftyp, _, _, _, ftyp}, index} = List.keytake(index, "ftyp", 0)
    # Skip re-writing the free fourcc as it's kind of useless.
    # Why stream useless bytes when you can do without?
    {free_size, index} =
      case List.keytake(index, "free", 0) do
        {{_, _, _, size, _}, index} -> {size, index}
        _ -> {0, index}
      end
    {{_moov, _, _, moov_size, moov}, index} = List.keytake(index, "moov", 0)
    offset = -free_size + moov_size
    rest = for {_, _, _, _, data} <- index, do: data, into: []
    <<moov_head::bits-size(64), moov_data::bits>> = moov
    [ftyp, moov_head, fix_moov(moov_data, offset, []), rest]
  end
  defp fix_moov(
         <<size::integer-big-size(32), fourcc::bits-size(32), rest::bits>>,
         offset,
         acc
       ) do
    full_size = (size - 8) * 8
    <<data::bits-size(full_size), rest::bits>> = rest
    data =
      cond do
        fourcc in ["trak", "mdia", "minf", "stbl"] ->
          # Theses contains sto or co64 part
          [<<size::integer-big-size(32), fourcc::bits-size(32)>>, fix_moov(data, offset, [])]
        fourcc in ["stco", "co64"] ->
          # fix the damn thing
          <<version::integer-big-size(32), count::integer-big-size(32), rest::bits>> = data
          entry_size =
            case fourcc do
              "stco" -> 32
              "co64" -> 64
            end
          [
            <<size::integer-big-size(32), fourcc::bits-size(32), version::integer-big-size(32),
              count::integer-big-size(32)>>,
            rewrite_entries(entry_size, offset, rest, [])
          ]
        true ->
          [<<size::integer-big-size(32), fourcc::bits-size(32)>>, data]
      end
    acc = [acc | data]
    fix_moov(rest, offset, acc)
  end
  defp fix_moov(<<>>, _, acc), do: acc
  for size <- [32, 64] do
    defp rewrite_entries(
           unquote(size),
           offset,
           <<pos::integer-big-size(unquote(size)), rest::bits>>,
           acc
         ) do
      rewrite_entries(unquote(size), offset, rest, [
        acc | <<pos + offset::integer-big-size(unquote(size))>>
      ])
    end
  end
  defp rewrite_entries(_, _, <<>>, acc), do: acc
 end
--- a/lib/pleroma/helpers/uri_helper.ex
+++ b/lib/pleroma/helpers/uri_helper.ex
@ -3,18 +3,22 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Helpers.UriHelper do
-  def append_uri_params(uri, appended_params) do
+  def modify_uri_params(uri, overridden_params, deleted_params \\ []) do
    uri = URI.parse(uri)
-    appended_params = for {k, v} <- appended_params, into: %{}, do: {to_string(k), v}
+
-    existing_params = URI.query_decoder(uri.query || "") |> Enum.into(%{})
+    existing_params = URI.query_decoder(uri.query || "") |> Map.new()
-    updated_params_keys = Enum.uniq(Map.keys(existing_params) ++ Map.keys(appended_params))
+    overridden_params = Map.new(overridden_params, fn {k, v} -> {to_string(k), v} end)
    deleted_params = Enum.map(deleted_params, &to_string/1)
    updated_params =
-      for k <- updated_params_keys, do: {k, appended_params[k] || existing_params[k]}
+      existing_params
      |> Map.merge(overridden_params)
      |> Map.drop(deleted_params)
    uri
    |> Map.put(:query, URI.encode_query(updated_params))
    |> URI.to_string()
    |> String.replace_suffix("?", "")
  end
  def maybe_add_base("/" <> uri, base), do: Path.join([base, uri])
--- a/lib/pleroma/instances/instance.ex
+++ b/lib/pleroma/instances/instance.ex
@ -156,16 +156,12 @@ def get_or_update_favicon(%URI{host: host} = instance_uri) do
  defp scrape_favicon(%URI{} = instance_uri) do
    try do
      with {:ok, %Tesla.Env{body: html}} <-
-             Pleroma.HTTP.get(to_string(instance_uri), [{"accept", "text/html"}],
+             Pleroma.HTTP.get(to_string(instance_uri), [{"accept", "text/html"}], pool: :media),
-               adapter: [pool: :media]
+           {_, [favicon_rel | _]} when is_binary(favicon_rel) <-
-             ),
+             {:parse,
-           favicon_rel <-
+              html |> Floki.parse_document!() |> Floki.attribute("link[rel=icon]", "href")},
-             html
+           {_, favicon} when is_binary(favicon) <-
-             |> Floki.parse_document!()
+             {:merge, URI.merge(instance_uri, favicon_rel) |> to_string()} do
             |> Floki.attribute("link[rel=icon]", "href")
             |> List.first(),
           favicon <- URI.merge(instance_uri, favicon_rel) |> to_string(),
           true <- is_binary(favicon) do
        favicon
      else
        _ -> nil
--- a/lib/pleroma/mfa/token.ex
+++ b/lib/pleroma/mfa/token.ex
@ -10,10 +10,11 @@ defmodule Pleroma.MFA.Token do
  alias Pleroma.Repo
  alias Pleroma.User
  alias Pleroma.Web.OAuth.Authorization
  alias Pleroma.Web.OAuth.Token, as: OAuthToken
  @expires 300
  @type t() :: %__MODULE__{}
  schema "mfa_tokens" do
    field(:token, :string)
    field(:valid_until, :naive_datetime_usec)
@ -24,6 +25,7 @@ defmodule Pleroma.MFA.Token do
    timestamps()
  end
  @spec get_by_token(String.t()) :: {:ok, t()} | {:error, :not_found}
  def get_by_token(token) do
    from(
      t in __MODULE__,
@ -33,33 +35,40 @@ def get_by_token(token) do
    |> Repo.find_resource()
  end
-  def validate(token) do
+  @spec validate(String.t()) :: {:ok, t()} | {:error, :not_found} | {:error, :expired_token}
-    with {:fetch_token, {:ok, token}} <- {:fetch_token, get_by_token(token)},
+  def validate(token_str) do
-         {:expired, false} <- {:expired, is_expired?(token)} do
+    with {:ok, token} <- get_by_token(token_str),
         false <- expired?(token) do
      {:ok, token}
    else
      {:expired, _} -> {:error, :expired_token}
      {:fetch_token, _} -> {:error, :not_found}
      error -> {:error, error}
    end
  end
-  def create_token(%User{} = user) do
+  defp expired?(%__MODULE__{valid_until: valid_until}) do
-    %__MODULE__{}
+    with true <- NaiveDateTime.diff(NaiveDateTime.utc_now(), valid_until) > 0 do
-    |> change
+      {:error, :expired_token}
-    |> assign_user(user)
+    end
    |> put_token
    |> put_valid_until
    |> Repo.insert()
  end
-  def create_token(user, authorization) do
+  @spec create(User.t(), Authorization.t() | nil) :: {:ok, t()} | {:error, Ecto.Changeset.t()}
  def create(user, authorization \\ nil) do
    with {:ok, token} <- do_create(user, authorization) do
      Pleroma.Workers.PurgeExpiredToken.enqueue(%{
        token_id: token.id,
        valid_until: DateTime.from_naive!(token.valid_until, "Etc/UTC"),
        mod: __MODULE__
      })
      {:ok, token}
    end
  end
  defp do_create(user, authorization) do
    %__MODULE__{}
-    |> change
+    |> change()
    |> assign_user(user)
-    |> assign_authorization(authorization)
+    |> maybe_assign_authorization(authorization)
-    |> put_token
+    |> put_token()
-    |> put_valid_until
+    |> put_valid_until()
    |> Repo.insert()
  end
@ -69,15 +78,19 @@ defp assign_user(changeset, user) do
    |> validate_required([:user])
  end
-  defp assign_authorization(changeset, authorization) do
+  defp maybe_assign_authorization(changeset, %Authorization{} = authorization) do
    changeset
    |> put_assoc(:authorization, authorization)
    |> validate_required([:authorization])
  end
  defp maybe_assign_authorization(changeset, _), do: changeset
  defp put_token(changeset) do
    token = Pleroma.Web.OAuth.Token.Utils.generate_token()
    changeset
-    |> change(%{token: OAuthToken.Utils.generate_token()})
+    |> change(%{token: token})
    |> validate_required([:token])
    |> unique_constraint(:token)
  end
@ -89,18 +102,4 @@ defp put_valid_until(changeset) do
    |> change(%{valid_until: expires_in})
    |> validate_required([:valid_until])
  end
  def is_expired?(%__MODULE__{valid_until: valid_until}) do
    NaiveDateTime.diff(NaiveDateTime.utc_now(), valid_until) > 0
  end
  def is_expired?(_), do: false
  def delete_expired_tokens do
    from(
      q in __MODULE__,
      where: fragment("?", q.valid_until) < ^Timex.now()
    )
    |> Repo.delete_all()
  end
 end
--- a/lib/pleroma/migration_helper/notification_backfill.ex
+++ b/lib/pleroma/migration_helper/notification_backfill.ex
@ -19,13 +19,13 @@ def fill_in_notification_types do
    query
    |> Repo.chunk_stream(100)
    |> Enum.each(fn notification ->
-      type =
+      if notification.activity do
-        notification.activity
+        type = type_from_activity(notification.activity)
        |> type_from_activity()
        notification
        |> Ecto.Changeset.change(%{type: type})
        |> Repo.update()
      end
    end)
  end
@ -72,8 +72,7 @@ defp type_from_activity(%{data: %{"type" => type}} = activity) do
        "pleroma:emoji_reaction"
      "Create" ->
-        activity
+        type_from_activity_object(activity)
        |> type_from_activity_object()
      t ->
        raise "No notification type for activity type #{t}"
--- a/lib/pleroma/moderation_log.ex
+++ b/lib/pleroma/moderation_log.ex
@ -320,6 +320,19 @@ def insert_log(%{
    |> insert_log_entry_with_message()
  end
  @spec insert_log(%{actor: User, action: String.t(), subject_id: String.t()}) ::
          {:ok, ModerationLog} | {:error, any}
  def insert_log(%{actor: %User{} = actor, action: "chat_message_delete", subject_id: subject_id}) do
    %ModerationLog{
      data: %{
        "actor" => %{"nickname" => actor.nickname},
        "action" => "chat_message_delete",
        "subject_id" => subject_id
      }
    }
    |> insert_log_entry_with_message()
  end
  @spec insert_log_entry_with_message(ModerationLog) :: {:ok, ModerationLog} | {:error, any}
  defp insert_log_entry_with_message(entry) do
    entry.data["message"]
@ -627,6 +640,17 @@ def get_log_entry_message(%ModerationLog{
    "@#{actor_nickname} updated users: #{users_to_nicknames_string(subjects)}"
  end
  @spec get_log_entry_message(ModerationLog) :: String.t()
  def get_log_entry_message(%ModerationLog{
        data: %{
          "actor" => %{"nickname" => actor_nickname},
          "action" => "chat_message_delete",
          "subject_id" => subject_id
        }
      }) do
    "@#{actor_nickname} deleted chat message ##{subject_id}"
  end
  defp nicknames_to_string(nicknames) do
    nicknames
    |> Enum.map(&"@#{&1}")
--- a/lib/pleroma/object/containment.ex
+++ b/lib/pleroma/object/containment.ex
@ -44,13 +44,6 @@ def get_object(_) do
    nil
  end
  # TODO: We explicitly allow 'tag' URIs through, due to references to legacy OStatus
  # objects being present in the test suite environment.  Once these objects are
  # removed, please also remove this.
  if Mix.env() == :test do
    defp compare_uris(_, %URI{scheme: "tag"}), do: :ok
  end
  defp compare_uris(%URI{host: host} = _id_uri, %URI{host: host} = _other_uri), do: :ok
  defp compare_uris(_id_uri, _other_uri), do: :error
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@ -12,6 +12,7 @@ defmodule Pleroma.Object.Fetcher do
  alias Pleroma.Web.ActivityPub.ObjectValidator
  alias Pleroma.Web.ActivityPub.Transmogrifier
  alias Pleroma.Web.Federator
  alias Pleroma.Web.FedSockets
  require Logger
  require Pleroma.Constants
@ -98,8 +99,8 @@ def fetch_object_from_id(id, options \\ []) do
      {:containment, _} ->
        {:error, "Object containment failed."}
-      {:transmogrifier, {:error, {:reject, nil}}} ->
+      {:transmogrifier, {:error, {:reject, e}}} ->
-        {:reject, nil}
+        {:reject, e}
      {:transmogrifier, _} = e ->
        {:error, e}
@ -182,27 +183,20 @@ defp maybe_date_fetch(headers, date) do
    end
  end
-  def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
+  def fetch_and_contain_remote_object_from_id(prm, opts \\ [])
  def fetch_and_contain_remote_object_from_id(%{"id" => id}, opts),
    do: fetch_and_contain_remote_object_from_id(id, opts)
  def fetch_and_contain_remote_object_from_id(id, opts) when is_binary(id) do
    Logger.debug("Fetching object #{id} via AP")
    date = Pleroma.Signature.signed_date()
    headers =
      [{"accept", "application/activity+json"}]
      |> maybe_date_fetch(date)
      |> sign_fetch(id, date)
    Logger.debug("Fetch headers: #{inspect(headers)}")
    with {:scheme, true} <- {:scheme, String.starts_with?(id, "http")},
-         {:ok, %{body: body, status: code}} when code in 200..299 <- HTTP.get(id, headers),
+         {:ok, body} <- get_object(id, opts),
-         {:ok, data} <- Jason.decode(body),
+         {:ok, data} <- safe_json_decode(body),
         :ok <- Containment.contain_origin_from_id(id, data) do
      {:ok, data}
    else
      {:ok, %{status: code}} when code in [404, 410] ->
        {:error, "Object has been deleted"}
      {:scheme, _} ->
        {:error, "Unsupported URI scheme"}
@ -214,8 +208,44 @@ def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
    end
  end
-  def fetch_and_contain_remote_object_from_id(%{"id" => id}),
+  def fetch_and_contain_remote_object_from_id(_id, _opts),
-    do: fetch_and_contain_remote_object_from_id(id)
+    do: {:error, "id must be a string"}
-  def fetch_and_contain_remote_object_from_id(_id), do: {:error, "id must be a string"}
+  defp get_object(id, opts) do
    with false <- Keyword.get(opts, :force_http, false),
         {:ok, fedsocket} <- FedSockets.get_or_create_fed_socket(id) do
      Logger.debug("fetching via fedsocket - #{inspect(id)}")
      FedSockets.fetch(fedsocket, id)
    else
      _other ->
        Logger.debug("fetching via http - #{inspect(id)}")
        get_object_http(id)
    end
  end
  defp get_object_http(id) do
    date = Pleroma.Signature.signed_date()
    headers =
      [{"accept", "application/activity+json"}]
      |> maybe_date_fetch(date)
      |> sign_fetch(id, date)
    case HTTP.get(id, headers) do
      {:ok, %{body: body, status: code}} when code in 200..299 ->
        {:ok, body}
      {:ok, %{status: code}} when code in [404, 410] ->
        {:error, "Object has been deleted"}
      {:error, e} ->
        {:error, e}
      e ->
        {:error, e}
    end
  end
  defp safe_json_decode(nil), do: {:ok, nil}
  defp safe_json_decode(json), do: Jason.decode(json)
 end
--- a/lib/pleroma/plugs/oauth_scopes_plug.ex
+++ b/lib/pleroma/plugs/oauth_scopes_plug.ex
@ -53,7 +53,7 @@ def drop_auth_info(conn) do
    |> assign(:token, nil)
  end
-  @doc "Filters descendants of supported scopes"
+  @doc "Keeps those of `scopes` which are descendants of `supported_scopes`"
  def filter_descendants(scopes, supported_scopes) do
    Enum.filter(
      scopes,
--- a/lib/pleroma/repo.ex
+++ b/lib/pleroma/repo.ex
@ -49,7 +49,21 @@ def get_assoc(resource, association) do
    end
  end
-  def chunk_stream(query, chunk_size) do
+  @doc """
  Returns a lazy enumerable that emits all entries from the data store matching the given query.
  `returns_as` use to group records. use the `batches` option to fetch records in bulk.
  ## Examples
  # fetch records one-by-one
  iex> Pleroma.Repo.chunk_stream(Pleroma.Activity.Queries.by_actor(ap_id), 500)
  # fetch records in bulk
  iex> Pleroma.Repo.chunk_stream(Pleroma.Activity.Queries.by_actor(ap_id), 500, :batches)
  """
  @spec chunk_stream(Ecto.Query.t(), integer(), atom()) :: Enumerable.t()
  def chunk_stream(query, chunk_size, returns_as \\ :one) do
    # We don't actually need start and end funcitons of resource streaming,
    # but it seems to be the only way to not fetch records one-by-one and
    # have individual records be the elements of the stream, instead of
@ -69,7 +83,12 @@ def chunk_stream(query, chunk_size) do
            records ->
              last_id = List.last(records).id
              if returns_as == :one do
                {records, last_id}
              else
                {[records], last_id}
              end
          end
      end,
      fn _ -> :ok end
--- a/lib/pleroma/repo_streamer.ex
+++ b/lib/pleroma/repo_streamer.ex
@ -1,34 +0,0 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.RepoStreamer do
  alias Pleroma.Repo
  import Ecto.Query
  def chunk_stream(query, chunk_size) do
    Stream.unfold(0, fn
      :halt ->
        {[], :halt}
      last_id ->
        query
        |> order_by(asc: :id)
        |> where([r], r.id > ^last_id)
        |> limit(^chunk_size)
        |> Repo.all()
        |> case do
          [] ->
            {[], :halt}
          records ->
            last_id = List.last(records).id
            {records, last_id}
        end
    end)
    |> Stream.take_while(fn
      [] -> false
      _ -> true
    end)
  end
 end
--- a/lib/pleroma/reverse_proxy/client/tesla.ex
+++ b/lib/pleroma/reverse_proxy/client/tesla.ex
@ -28,7 +28,7 @@ def request(method, url, headers, body, opts \\ []) do
             url,
             body,
             headers,
-             Keyword.put(opts, :adapter, opts)
+             opts
           ) do
      if is_map(response.body) and method != :head do
        {:ok, response.status, response.headers, response.body}
--- a/lib/pleroma/reverse_proxy/reverse_proxy.ex
+++ b/lib/pleroma/reverse_proxy/reverse_proxy.ex
@ -17,6 +17,9 @@ defmodule Pleroma.ReverseProxy do
  @failed_request_ttl :timer.seconds(60)
  @methods ~w(GET HEAD)
  def max_read_duration_default, do: @max_read_duration
  def default_cache_control_header, do: @default_cache_control_header
  @moduledoc """
  A reverse proxy.
@ -391,6 +394,8 @@ defp body_size_constraint(size, limit) when is_integer(limit) and limit > 0 and
  defp body_size_constraint(_, _), do: :ok
  defp check_read_duration(nil = _duration, max), do: check_read_duration(@max_read_duration, max)
  defp check_read_duration(duration, max)
       when is_integer(duration) and is_integer(max) and max > 0 do
    if duration > max do
--- a/lib/pleroma/signature.ex
+++ b/lib/pleroma/signature.ex
@ -39,7 +39,7 @@ def key_id_to_actor_id(key_id) do
  def fetch_public_key(conn) do
    with %{"keyId" => kid} <- HTTPSignatures.signature_for_conn(conn),
         {:ok, actor_id} <- key_id_to_actor_id(kid),
-         {:ok, public_key} <- User.get_public_key_for_ap_id(actor_id) do
+         {:ok, public_key} <- User.get_public_key_for_ap_id(actor_id, force_http: true) do
      {:ok, public_key}
    else
      e ->
@ -50,8 +50,8 @@ def fetch_public_key(conn) do
  def refetch_public_key(conn) do
    with %{"keyId" => kid} <- HTTPSignatures.signature_for_conn(conn),
         {:ok, actor_id} <- key_id_to_actor_id(kid),
-         {:ok, _user} <- ActivityPub.make_user_from_ap_id(actor_id),
+         {:ok, _user} <- ActivityPub.make_user_from_ap_id(actor_id, force_http: true),
-         {:ok, public_key} <- User.get_public_key_for_ap_id(actor_id) do
+         {:ok, public_key} <- User.get_public_key_for_ap_id(actor_id, force_http: true) do
      {:ok, public_key}
    else
      e ->
--- a/lib/pleroma/telemetry/logger.ex
+++ b/lib/pleroma/telemetry/logger.ex
@ -7,7 +7,8 @@ defmodule Pleroma.Telemetry.Logger do
    [:pleroma, :connection_pool, :reclaim, :start],
    [:pleroma, :connection_pool, :reclaim, :stop],
    [:pleroma, :connection_pool, :provision_failure],
-    [:pleroma, :connection_pool, :client_death]
+    [:pleroma, :connection_pool, :client, :dead],
    [:pleroma, :connection_pool, :client, :add]
  ]
  def attach do
    :telemetry.attach_many("pleroma-logger", @events, &handle_event/4, [])
@ -62,7 +63,7 @@ def handle_event(
  end
  def handle_event(
-        [:pleroma, :connection_pool, :client_death],
+        [:pleroma, :connection_pool, :client, :dead],
        %{client_pid: client_pid, reason: reason},
        %{key: key},
        _
@ -73,4 +74,17 @@ def handle_event(
      }"
    end)
  end
  def handle_event(
        [:pleroma, :connection_pool, :client, :add],
        %{clients: [_, _ | _] = clients},
        %{key: key, protocol: :http},
        _
      ) do
    Logger.info(fn ->
      "Pool worker for #{key}: #{length(clients)} clients are using an HTTP1 connection at the same time, head-of-line blocking might occur."
    end)
  end
  def handle_event([:pleroma, :connection_pool, :client, :add], _, _, _), do: :ok
 end
--- a/lib/pleroma/user.ex
+++ b/lib/pleroma/user.ex
@ -25,7 +25,6 @@ defmodule Pleroma.User do
  alias Pleroma.Object
  alias Pleroma.Registration
  alias Pleroma.Repo
  alias Pleroma.RepoStreamer
  alias Pleroma.User
  alias Pleroma.UserRelationship
  alias Pleroma.Web
@ -276,9 +275,9 @@ def binary_id(%User{} = user), do: binary_id(user.id)
  @spec account_status(User.t()) :: account_status()
  def account_status(%User{deactivated: true}), do: :deactivated
  def account_status(%User{password_reset_pending: true}), do: :password_reset_pending
-  def account_status(%User{approval_pending: true}), do: :approval_pending
+  def account_status(%User{local: true, approval_pending: true}), do: :approval_pending
-  def account_status(%User{confirmation_pending: true}) do
+  def account_status(%User{local: true, confirmation_pending: true}) do
    if Config.get([:instance, :account_activation_required]) do
      :confirmation_pending
    else
@ -1687,42 +1686,6 @@ def perform(:delete, %User{} = user) do
  def perform(:deactivate_async, user, status), do: deactivate(user, status)
  @spec perform(atom(), User.t(), list()) :: list() | {:error, any()}
  def perform(:blocks_import, %User{} = blocker, blocked_identifiers)
      when is_list(blocked_identifiers) do
    Enum.map(
      blocked_identifiers,
      fn blocked_identifier ->
        with {:ok, %User{} = blocked} <- get_or_fetch(blocked_identifier),
             {:ok, _block} <- CommonAPI.block(blocker, blocked) do
          blocked
        else
          err ->
            Logger.debug("blocks_import failed for #{blocked_identifier} with: #{inspect(err)}")
            err
        end
      end
    )
  end
  def perform(:follow_import, %User{} = follower, followed_identifiers)
      when is_list(followed_identifiers) do
    Enum.map(
      followed_identifiers,
      fn followed_identifier ->
        with {:ok, %User{} = followed} <- get_or_fetch(followed_identifier),
             {:ok, follower} <- maybe_direct_follow(follower, followed),
             {:ok, _, _, _} <- CommonAPI.follow(follower, followed) do
          followed
        else
          err ->
            Logger.debug("follow_import failed for #{followed_identifier} with: #{inspect(err)}")
            err
        end
      end
    )
  end
  @spec external_users_query() :: Ecto.Query.t()
  def external_users_query do
    User.Query.build(%{
@ -1751,21 +1714,6 @@ def external_users(opts \\ []) do
    Repo.all(query)
  end
  def blocks_import(%User{} = blocker, blocked_identifiers) when is_list(blocked_identifiers) do
    BackgroundWorker.enqueue("blocks_import", %{
      "blocker_id" => blocker.id,
      "blocked_identifiers" => blocked_identifiers
    })
  end
  def follow_import(%User{} = follower, followed_identifiers)
      when is_list(followed_identifiers) do
    BackgroundWorker.enqueue("follow_import", %{
      "follower_id" => follower.id,
      "followed_identifiers" => followed_identifiers
    })
  end
  def delete_notifications_from_user_activities(%User{ap_id: ap_id}) do
    Notification
    |> join(:inner, [n], activity in assoc(n, :activity))
@ -1776,7 +1724,7 @@ def delete_notifications_from_user_activities(%User{ap_id: ap_id}) do
  def delete_user_activities(%User{ap_id: ap_id} = user) do
    ap_id
    |> Activity.Queries.by_actor()
-    |> RepoStreamer.chunk_stream(50)
+    |> Repo.chunk_stream(50, :batches)
    |> Stream.each(fn activities ->
      Enum.each(activities, fn activity -> delete_activity(activity, user) end)
    end)
@ -1822,12 +1770,12 @@ def html_filter_policy(%User{no_rich_text: true}) do
  def html_filter_policy(_), do: Config.get([:markup, :scrub_policy])
-  def fetch_by_ap_id(ap_id), do: ActivityPub.make_user_from_ap_id(ap_id)
+  def fetch_by_ap_id(ap_id, opts \\ []), do: ActivityPub.make_user_from_ap_id(ap_id, opts)
-  def get_or_fetch_by_ap_id(ap_id) do
+  def get_or_fetch_by_ap_id(ap_id, opts \\ []) do
    cached_user = get_cached_by_ap_id(ap_id)
-    maybe_fetched_user = needs_update?(cached_user) && fetch_by_ap_id(ap_id)
+    maybe_fetched_user = needs_update?(cached_user) && fetch_by_ap_id(ap_id, opts)
    case {cached_user, maybe_fetched_user} do
      {_, {:ok, %User{} = user}} ->
@ -1900,8 +1848,8 @@ def public_key(%{public_key: public_key_pem}) when is_binary(public_key_pem) do
  def public_key(_), do: {:error, "key not found"}
-  def get_public_key_for_ap_id(ap_id) do
+  def get_public_key_for_ap_id(ap_id, opts \\ []) do
-    with {:ok, %User{} = user} <- get_or_fetch_by_ap_id(ap_id),
+    with {:ok, %User{} = user} <- get_or_fetch_by_ap_id(ap_id, opts),
         {:ok, public_key} <- public_key(user) do
      {:ok, public_key}
    else
@ -2323,6 +2271,11 @@ def add_pinnned_activity(user, %Pleroma.Activity{id: id}) do
      max_pinned_statuses = Config.get([:instance, :max_pinned_statuses], 0)
      params = %{pinned_activities: user.pinned_activities ++ [id]}
      # if pinned activity was scheduled for deletion, we remove job
      if expiration = Pleroma.Workers.PurgeExpiredActivity.get_expiration(id) do
        Oban.cancel_job(expiration.id)
      end
      user
      |> cast(params, [:pinned_activities])
      |> validate_length(:pinned_activities,
@ -2335,9 +2288,19 @@ def add_pinnned_activity(user, %Pleroma.Activity{id: id}) do
    |> update_and_set_cache()
  end
-  def remove_pinnned_activity(user, %Pleroma.Activity{id: id}) do
+  def remove_pinnned_activity(user, %Pleroma.Activity{id: id, data: data}) do
    params = %{pinned_activities: List.delete(user.pinned_activities, id)}
    # if pinned activity was scheduled for deletion, we reschedule it for deletion
    if data["expires_at"] do
      {:ok, expires_at, _} = DateTime.from_iso8601(data["expires_at"])
      Pleroma.Workers.PurgeExpiredActivity.enqueue(%{
        activity_id: id,
        expires_at: expires_at
      })
    end
    user
    |> cast(params, [:pinned_activities])
    |> update_and_set_cache()
--- a/lib/pleroma/user/import.ex
+++ b/lib/pleroma/user/import.ex
@ -0,0 +1,85 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.User.Import do
  use Ecto.Schema
  alias Pleroma.User
  alias Pleroma.Web.CommonAPI
  alias Pleroma.Workers.BackgroundWorker
  require Logger
  @spec perform(atom(), User.t(), list()) :: :ok | list() | {:error, any()}
  def perform(:mutes_import, %User{} = user, [_ | _] = identifiers) do
    Enum.map(
      identifiers,
      fn identifier ->
        with {:ok, %User{} = muted_user} <- User.get_or_fetch(identifier),
             {:ok, _} <- User.mute(user, muted_user) do
          muted_user
        else
          error -> handle_error(:mutes_import, identifier, error)
        end
      end
    )
  end
  def perform(:blocks_import, %User{} = blocker, [_ | _] = identifiers) do
    Enum.map(
      identifiers,
      fn identifier ->
        with {:ok, %User{} = blocked} <- User.get_or_fetch(identifier),
             {:ok, _block} <- CommonAPI.block(blocker, blocked) do
          blocked
        else
          error -> handle_error(:blocks_import, identifier, error)
        end
      end
    )
  end
  def perform(:follow_import, %User{} = follower, [_ | _] = identifiers) do
    Enum.map(
      identifiers,
      fn identifier ->
        with {:ok, %User{} = followed} <- User.get_or_fetch(identifier),
             {:ok, follower} <- User.maybe_direct_follow(follower, followed),
             {:ok, _, _, _} <- CommonAPI.follow(follower, followed) do
          followed
        else
          error -> handle_error(:follow_import, identifier, error)
        end
      end
    )
  end
  def perform(_, _, _), do: :ok
  defp handle_error(op, user_id, error) do
    Logger.debug("#{op} failed for #{user_id} with: #{inspect(error)}")
    error
  end
  def blocks_import(%User{} = blocker, [_ | _] = identifiers) do
    BackgroundWorker.enqueue(
      "blocks_import",
      %{"user_id" => blocker.id, "identifiers" => identifiers}
    )
  end
  def follow_import(%User{} = follower, [_ | _] = identifiers) do
    BackgroundWorker.enqueue(
      "follow_import",
      %{"user_id" => follower.id, "identifiers" => identifiers}
    )
  end
  def mutes_import(%User{} = user, [_ | _] = identifiers) do
    BackgroundWorker.enqueue(
      "mutes_import",
      %{"user_id" => user.id, "identifiers" => identifiers}
    )
  end
 end
--- a/lib/pleroma/user/query.ex
+++ b/lib/pleroma/user/query.ex
@ -47,6 +47,7 @@ defmodule Pleroma.User.Query do
            is_moderator: boolean(),
            super_users: boolean(),
            invisible: boolean(),
            internal: boolean(),
            followers: User.t(),
            friends: User.t(),
            recipients_from_activity: [String.t()],
@ -80,7 +81,9 @@ defp base_query do
  end
  defp prepare_query(query, criteria) do
-    Enum.reduce(criteria, query, &compose_query/2)
+    criteria
    |> Map.put_new(:internal, false)
    |> Enum.reduce(query, &compose_query/2)
  end
  defp compose_query({key, value}, query)
@ -129,14 +132,12 @@ defp compose_query({:external, _}, query), do: location_query(query, false)
  defp compose_query({:active, _}, query) do
    User.restrict_deactivated(query)
    |> where([u], not is_nil(u.nickname))
    |> where([u], u.approval_pending == false)
  end
  defp compose_query({:legacy_active, _}, query) do
    query
    |> where([u], fragment("not (?->'deactivated' @> 'true')", u.info))
    |> where([u], not is_nil(u.nickname))
  end
  defp compose_query({:deactivated, false}, query) do
@ -145,7 +146,6 @@ defp compose_query({:deactivated, false}, query) do
  defp compose_query({:deactivated, true}, query) do
    where(query, [u], u.deactivated == ^true)
    |> where([u], not is_nil(u.nickname))
  end
  defp compose_query({:confirmation_pending, bool}, query) do
@ -203,10 +203,15 @@ defp compose_query({:limit, limit}, query) do
    limit(query, ^limit)
  end
  defp compose_query({:internal, false}, query) do
    query
    |> where([u], not is_nil(u.nickname))
    |> where([u], not like(u.nickname, "internal.%"))
  end
  defp compose_query(_unsupported_param, query), do: query
  defp location_query(query, local) do
    where(query, [u], u.local == ^local)
    |> where([u], not is_nil(u.nickname))
  end
 end
--- a/lib/pleroma/user/search.ex
+++ b/lib/pleroma/user/search.ex
@ -52,6 +52,7 @@ defp search_query(query_string, for_user, following) do
    |> base_query(following)
    |> filter_blocked_user(for_user)
    |> filter_invisible_users()
    |> filter_discoverable_users()
    |> filter_internal_users()
    |> filter_blocked_domains(for_user)
    |> fts_search(query_string)
@ -122,6 +123,10 @@ defp filter_invisible_users(query) do
    from(q in query, where: q.invisible == false)
  end
  defp filter_discoverable_users(query) do
    from(q in query, where: q.discoverable == true)
  end
  defp filter_internal_users(query) do
    from(q in query, where: q.actor_type != "Application")
  end
--- a/lib/pleroma/utils.ex
+++ b/lib/pleroma/utils.ex
@ -24,4 +24,24 @@ def compile_dir(dir) when is_binary(dir) do
  def command_available?(command) do
    match?({_output, 0}, System.cmd("sh", ["-c", "command -v #{command}"]))
  end
  @doc "creates the uniq temporary directory"
  @spec tmp_dir(String.t()) :: {:ok, String.t()} | {:error, :file.posix()}
  def tmp_dir(prefix \\ "") do
    sub_dir =
      [
        prefix,
        Timex.to_unix(Timex.now()),
        :os.getpid(),
        String.downcase(Integer.to_string(:rand.uniform(0x100000000), 36))
      ]
      |> Enum.join("-")
    tmp_dir = Path.join(System.tmp_dir!(), sub_dir)
    case File.mkdir(tmp_dir) do
      :ok -> {:ok, tmp_dir}
      error -> error
    end
  end
 end
--- a/lib/pleroma/web/activity_pub/activity_pub.ex
+++ b/lib/pleroma/web/activity_pub/activity_pub.ex
@ -5,7 +5,6 @@
 defmodule Pleroma.Web.ActivityPub.ActivityPub do
  alias Pleroma.Activity
  alias Pleroma.Activity.Ir.Topics
  alias Pleroma.ActivityExpiration
  alias Pleroma.Config
  alias Pleroma.Constants
  alias Pleroma.Conversation
@ -85,7 +84,7 @@ defp increase_replies_count_if_reply(%{
  defp increase_replies_count_if_reply(_create_data), do: :noop
-  @object_types ~w[ChatMessage Question Answer Audio Event]
+  @object_types ~w[ChatMessage Question Answer Audio Video Event Article]
  @spec persist(map(), keyword()) :: {:ok, Activity.t() | Object.t()}
  def persist(%{"type" => type} = object, meta) when type in @object_types do
    with {:ok, object} <- Object.create(object) do
@ -102,7 +101,9 @@ def persist(object, meta) do
             local: local,
             recipients: recipients,
             actor: object["actor"]
-           }) do
+           }),
         # TODO: add tests for expired activities, when Note type will be supported in new pipeline
         {:ok, _} <- maybe_create_activity_expiration(activity) do
      {:ok, activity, meta}
    end
  end
@ -111,23 +112,14 @@ def persist(object, meta) do
  def insert(map, local \\ true, fake \\ false, bypass_actor_check \\ false) when is_map(map) do
    with nil <- Activity.normalize(map),
         map <- lazy_put_activity_defaults(map, fake),
-         true <- bypass_actor_check || check_actor_is_active(map["actor"]),
+         {_, true} <- {:actor_check, bypass_actor_check || check_actor_is_active(map["actor"])},
-         {_, true} <- {:remote_limit_error, check_remote_limit(map)},
+         {_, true} <- {:remote_limit_pass, check_remote_limit(map)},
         {:ok, map} <- MRF.filter(map),
         {recipients, _, _} = get_recipients(map),
         {:fake, false, map, recipients} <- {:fake, fake, map, recipients},
         {:containment, :ok} <- {:containment, Containment.contain_child(map)},
-         {:ok, map, object} <- insert_full_object(map) do
+         {:ok, map, object} <- insert_full_object(map),
-      {:ok, activity} =
+         {:ok, activity} <- insert_activity_with_expiration(map, local, recipients) do
        %Activity{
          data: map,
          local: local,
          actor: map["actor"],
          recipients: recipients
        }
        |> Repo.insert()
        |> maybe_create_activity_expiration()
      # Splice in the child object if we have one.
      activity = Maps.put_if_present(activity, :object, object)
@ -138,6 +130,15 @@ def insert(map, local \\ true, fake \\ false, bypass_actor_check \\ false) when
      %Activity{} = activity ->
        {:ok, activity}
      {:actor_check, _} ->
        {:error, false}
      {:containment, _} = error ->
        error
      {:error, _} = error ->
        error
      {:fake, true, map, recipients} ->
        activity = %Activity{
          data: map,
@ -150,8 +151,24 @@ def insert(map, local \\ true, fake \\ false, bypass_actor_check \\ false) when
        Pleroma.Web.RichMedia.Helpers.fetch_data_for_activity(activity)
        {:ok, activity}
-      error ->
+      {:remote_limit_pass, _} ->
-        {:error, error}
+        {:error, :remote_limit}
      {:reject, _} = e ->
        {:error, e}
    end
  end
  defp insert_activity_with_expiration(data, local, recipients) do
    struct = %Activity{
      data: data,
      local: local,
      actor: data["actor"],
      recipients: recipients
    }
    with {:ok, activity} <- Repo.insert(struct) do
      maybe_create_activity_expiration(activity)
    end
  end
@ -164,13 +181,19 @@ def notify_and_stream(activity) do
    stream_out_participations(participations)
  end
-  defp maybe_create_activity_expiration({:ok, %{data: %{"expires_at" => expires_at}} = activity}) do
+  defp maybe_create_activity_expiration(
-    with {:ok, _} <- ActivityExpiration.create(activity, expires_at) do
+         %{data: %{"expires_at" => %DateTime{} = expires_at}} = activity
       ) do
    with {:ok, _job} <-
           Pleroma.Workers.PurgeExpiredActivity.enqueue(%{
             activity_id: activity.id,
             expires_at: expires_at
           }) do
      {:ok, activity}
    end
  end
-  defp maybe_create_activity_expiration(result), do: result
+  defp maybe_create_activity_expiration(activity), do: {:ok, activity}
  defp create_or_bump_conversation(activity, actor) do
    with {:ok, conversation} <- Conversation.create_or_bump_for(activity),
@ -744,7 +767,7 @@ defp restrict_replies(query, %{exclude_replies: true}) do
  end
  defp restrict_replies(query, %{
-         reply_filtering_user: user,
+         reply_filtering_user: %User{} = user,
         reply_visibility: "self"
       }) do
    from(
@ -760,7 +783,7 @@ defp restrict_replies(query, %{
  end
  defp restrict_replies(query, %{
-         reply_filtering_user: user,
+         reply_filtering_user: %User{} = user,
         reply_visibility: "following"
       }) do
    from(
@ -818,7 +841,14 @@ defp restrict_blocked(query, %{blocking_user: %User{} = user} = opts) do
    from(
      [activity, object: o] in query,
      where: fragment("not (? = ANY(?))", activity.actor, ^blocked_ap_ids),
-      where: fragment("not (? && ?)", activity.recipients, ^blocked_ap_ids),
+      where:
        fragment(
          "((not (? && ?)) or ? = ?)",
          activity.recipients,
          ^blocked_ap_ids,
          activity.actor,
          ^user.ap_id
        ),
      where:
        fragment(
          "recipients_contain_blocked_domains(?, ?) = false",
@ -1247,10 +1277,12 @@ defp object_to_user_data(data) do
  def fetch_follow_information_for_user(user) do
    with {:ok, following_data} <-
-           Fetcher.fetch_and_contain_remote_object_from_id(user.following_address),
+           Fetcher.fetch_and_contain_remote_object_from_id(user.following_address,
             force_http: true
           ),
         {:ok, hide_follows} <- collection_private(following_data),
         {:ok, followers_data} <-
-           Fetcher.fetch_and_contain_remote_object_from_id(user.follower_address),
+           Fetcher.fetch_and_contain_remote_object_from_id(user.follower_address, force_http: true),
         {:ok, hide_followers} <- collection_private(followers_data) do
      {:ok,
       %{
@ -1324,8 +1356,8 @@ def user_data_from_user_object(data) do
    end
  end
-  def fetch_and_prepare_user_from_ap_id(ap_id) do
+  def fetch_and_prepare_user_from_ap_id(ap_id, opts \\ []) do
-    with {:ok, data} <- Fetcher.fetch_and_contain_remote_object_from_id(ap_id),
+    with {:ok, data} <- Fetcher.fetch_and_contain_remote_object_from_id(ap_id, opts),
         {:ok, data} <- user_data_from_user_object(data) do
      {:ok, maybe_update_follow_information(data)}
    else
@ -1367,13 +1399,13 @@ def maybe_handle_clashing_nickname(data) do
    end
  end
-  def make_user_from_ap_id(ap_id) do
+  def make_user_from_ap_id(ap_id, opts \\ []) do
    user = User.get_cached_by_ap_id(ap_id)
    if user && !User.ap_enabled?(user) do
      Transmogrifier.upgrade_user_from_ap_id(ap_id)
    else
-      with {:ok, data} <- fetch_and_prepare_user_from_ap_id(ap_id) do
+      with {:ok, data} <- fetch_and_prepare_user_from_ap_id(ap_id, opts) do
        if user do
          user
          |> User.remote_user_changeset(data)
--- a/lib/pleroma/web/activity_pub/mrf.ex
+++ b/lib/pleroma/web/activity_pub/mrf.ex
@ -5,16 +5,34 @@
 defmodule Pleroma.Web.ActivityPub.MRF do
  @callback filter(Map.t()) :: {:ok | :reject, Map.t()}
-  def filter(policies, %{} = object) do
+  def filter(policies, %{} = message) do
    policies
-    |> Enum.reduce({:ok, object}, fn
+    |> Enum.reduce({:ok, message}, fn
-      policy, {:ok, object} -> policy.filter(object)
+      policy, {:ok, message} -> policy.filter(message)
      _, error -> error
    end)
  end
  def filter(%{} = object), do: get_policies() |> filter(object)
  def pipeline_filter(%{} = message, meta) do
    object = meta[:object_data]
    ap_id = message["object"]
    if object && ap_id do
      with {:ok, message} <- filter(Map.put(message, "object", object)) do
        meta = Keyword.put(meta, :object_data, message["object"])
        {:ok, Map.put(message, "object", ap_id), meta}
      else
        {err, message} -> {err, message, meta}
      end
    else
      {err, message} = filter(message)
      {err, message, meta}
    end
  end
  def get_policies do
    Pleroma.Config.get([:mrf, :policies], []) |> get_policies()
  end
--- a/lib/pleroma/web/activity_pub/mrf/activity_expiration_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/activity_expiration_policy.ex
@ -31,10 +31,10 @@ defp note?(activity) do
  defp maybe_add_expiration(activity) do
    days = Pleroma.Config.get([:mrf_activity_expiration, :days], 365)
-    expires_at = NaiveDateTime.utc_now() |> Timex.shift(days: days)
+    expires_at = DateTime.utc_now() |> Timex.shift(days: days)
    with %{"expires_at" => existing_expires_at} <- activity,
-         :lt <- NaiveDateTime.compare(existing_expires_at, expires_at) do
+         :lt <- DateTime.compare(existing_expires_at, expires_at) do
      activity
    else
      _ -> Map.put(activity, "expires_at", expires_at)
--- a/lib/pleroma/web/activity_pub/mrf/keyword_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/keyword_policy.ex
@ -20,9 +20,17 @@ defp string_matches?(string, pattern) do
    String.match?(string, pattern)
  end
-  defp check_reject(%{"object" => %{"content" => content, "summary" => summary}} = message) do
+  defp object_payload(%{} = object) do
    [object["content"], object["summary"], object["name"]]
    |> Enum.filter(& &1)
    |> Enum.join("\n")
  end
  defp check_reject(%{"object" => %{} = object} = message) do
    payload = object_payload(object)
    if Enum.any?(Pleroma.Config.get([:mrf_keyword, :reject]), fn pattern ->
-         string_matches?(content, pattern) or string_matches?(summary, pattern)
+         string_matches?(payload, pattern)
       end) do
      {:reject, "[KeywordPolicy] Matches with rejected keyword"}
    else
@ -30,12 +38,12 @@ defp check_reject(%{"object" => %{"content" => content, "summary" => summary}} =
    end
  end
-  defp check_ftl_removal(
+  defp check_ftl_removal(%{"to" => to, "object" => %{} = object} = message) do
-         %{"to" => to, "object" => %{"content" => content, "summary" => summary}} = message
+    payload = object_payload(object)
-       ) do
+
    if Pleroma.Constants.as_public() in to and
         Enum.any?(Pleroma.Config.get([:mrf_keyword, :federated_timeline_removal]), fn pattern ->
-           string_matches?(content, pattern) or string_matches?(summary, pattern)
+           string_matches?(payload, pattern)
         end) do
      to = List.delete(to, Pleroma.Constants.as_public())
      cc = [Pleroma.Constants.as_public() | message["cc"] || []]
@ -51,35 +59,24 @@ defp check_ftl_removal(
    end
  end
-  defp check_replace(%{"object" => %{"content" => content, "summary" => summary}} = message) do
+  defp check_replace(%{"object" => %{} = object} = message) do
-    content =
+    object =
-      if is_binary(content) do
+      ["content", "name", "summary"]
-        content
+      |> Enum.filter(fn field -> Map.has_key?(object, field) && object[field] end)
-      else
+      |> Enum.reduce(object, fn field, object ->
-        ""
+        data =
      end
    summary =
      if is_binary(summary) do
        summary
      else
        ""
      end
    {content, summary} =
          Enum.reduce(
            Pleroma.Config.get([:mrf_keyword, :replace]),
-        {content, summary},
+            object[field],
-        fn {pattern, replacement}, {content_acc, summary_acc} ->
+            fn {pat, repl}, acc -> String.replace(acc, pat, repl) end
          {String.replace(content_acc, pattern, replacement),
           String.replace(summary_acc, pattern, replacement)}
        end
          )
-    {:ok,
+        Map.put(object, field, data)
-     message
+      end)
-     |> put_in(["object", "content"], content)
+
-     |> put_in(["object", "summary"], summary)}
+    message = Map.put(message, "object", object)
    {:ok, message}
  end
  @impl true
--- a/lib/pleroma/web/activity_pub/mrf/media_proxy_warming_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/media_proxy_warming_policy.ex
@ -12,17 +12,21 @@ defmodule Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy do
  require Logger
-  @options [
+  @adapter_options [
    pool: :media,
    recv_timeout: 10_000
  ]
  def perform(:prefetch, url) do
-    Logger.debug("Prefetching #{inspect(url)}")
+    # Fetching only proxiable resources
    if MediaProxy.enabled?() and MediaProxy.url_proxiable?(url) do
      # If preview proxy is enabled, it'll also hit media proxy (so we're caching both requests)
      prefetch_url = MediaProxy.preview_url(url)
-    url
+      Logger.debug("Prefetching #{inspect(url)} as #{inspect(prefetch_url)}")
-    |> MediaProxy.url()
+
-    |> HTTP.get([], @options)
+      HTTP.get(prefetch_url, [], @adapter_options)
    end
  end
  def perform(:preload, %{"object" => %{"attachment" => attachments}} = _message) do
--- a/lib/pleroma/web/activity_pub/mrf/simple_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/simple_policy.ex
@ -66,7 +66,8 @@ defp check_media_nsfw(
           "type" => "Create",
           "object" => child_object
         } = object
-       ) do
+       )
       when is_map(child_object) do
    media_nsfw =
      Config.get([:mrf_simple, :media_nsfw])
      |> MRF.subdomains_regex()
--- a/lib/pleroma/web/activity_pub/mrf/subchain_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/subchain_policy.ex
@ -28,8 +28,7 @@ def filter(%{"actor" => actor} = message) do
        }"
      )
-      subchain
+      MRF.filter(subchain, message)
      |> MRF.filter(message)
    else
      _e -> {:ok, message}
    end
--- a/lib/pleroma/web/activity_pub/object_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validator.ex
@ -12,11 +12,13 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
  alias Pleroma.Activity
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Object
  alias Pleroma.Object.Containment
  alias Pleroma.User
  alias Pleroma.Web.ActivityPub.ObjectValidators.AcceptRejectValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.AnnounceValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.AnswerValidator
-  alias Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator
+  alias Pleroma.Web.ActivityPub.ObjectValidators.ArticleNoteValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.AudioVideoValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.BlockValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.ChatMessageValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.CreateChatMessageValidator
@ -149,10 +151,20 @@ def validate(%{"type" => "Question"} = object, meta) do
    end
  end
-  def validate(%{"type" => "Audio"} = object, meta) do
+  def validate(%{"type" => type} = object, meta) when type in ~w[Audio Video] do
    with {:ok, object} <-
           object
-           |> AudioValidator.cast_and_validate()
+           |> AudioVideoValidator.cast_and_validate()
           |> Ecto.Changeset.apply_action(:insert) do
      object = stringify_keys(object)
      {:ok, object, meta}
    end
  end
  def validate(%{"type" => "Article"} = object, meta) do
    with {:ok, object} <-
           object
           |> ArticleNoteValidator.cast_and_validate()
           |> Ecto.Changeset.apply_action(:insert) do
      object = stringify_keys(object)
      {:ok, object, meta}
@ -198,7 +210,7 @@ def validate(
        %{"type" => "Create", "object" => %{"type" => objtype} = object} = create_activity,
        meta
      )
-      when objtype in ~w[Question Answer Audio Event] do
+      when objtype in ~w[Question Answer Audio Video Event Article] do
    with {:ok, object_data} <- cast_and_apply(object),
         meta = Keyword.put(meta, :object_data, object_data |> stringify_keys),
         {:ok, create_activity} <-
@ -232,14 +244,18 @@ def cast_and_apply(%{"type" => "Answer"} = object) do
    AnswerValidator.cast_and_apply(object)
  end
-  def cast_and_apply(%{"type" => "Audio"} = object) do
+  def cast_and_apply(%{"type" => type} = object) when type in ~w[Audio Video] do
-    AudioValidator.cast_and_apply(object)
+    AudioVideoValidator.cast_and_apply(object)
  end
  def cast_and_apply(%{"type" => "Event"} = object) do
    EventValidator.cast_and_apply(object)
  end
  def cast_and_apply(%{"type" => "Article"} = object) do
    ArticleNoteValidator.cast_and_apply(object)
  end
  def cast_and_apply(o), do: {:error, {:validator_not_set, o}}
  # is_struct/1 isn't present in Elixir 1.8.x
@ -262,7 +278,8 @@ def stringify_keys(object) when is_list(object) do
  def stringify_keys(object), do: object
  def fetch_actor(object) do
-    with {:ok, actor} <- ObjectValidators.ObjectID.cast(object["actor"]) do
+    with actor <- Containment.get_actor(object),
         {:ok, actor} <- ObjectValidators.ObjectID.cast(actor) do
      User.get_or_fetch_by_ap_id(actor)
    end
  end
--- a/lib/pleroma/web/activity_pub/object_validators/article_note_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/article_note_validator.ex
@ -2,7 +2,7 @@
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
-defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
+defmodule Pleroma.Web.ActivityPub.ObjectValidators.ArticleNoteValidator do
  use Ecto.Schema
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
@ -25,14 +25,19 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
    # TODO: Write type
    field(:tag, {:array, :map}, default: [])
    field(:type, :string)
    field(:name, :string)
    field(:summary, :string)
    field(:content, :string)
    field(:context, :string)
    # short identifier for PleromaFE to group statuses by context
    field(:context_id, :integer)
    # TODO: Remove actor on objects
    field(:actor, ObjectValidators.ObjectID)
    field(:attributedTo, ObjectValidators.ObjectID)
    field(:summary, :string)
    field(:published, ObjectValidators.DateTime)
    field(:emoji, ObjectValidators.Emoji, default: %{})
    field(:sensitive, :boolean, default: false)
@ -40,13 +45,11 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioValidator do
    field(:replies_count, :integer, default: 0)
    field(:like_count, :integer, default: 0)
    field(:announcement_count, :integer, default: 0)
-    field(:inReplyTo, :string)
+    field(:inReplyTo, ObjectValidators.ObjectID)
    field(:url, ObjectValidators.Uri)
    # short identifier for PleromaFE to group statuses by context
    field(:context_id, :integer)
-    field(:likes, {:array, :string}, default: [])
+    field(:likes, {:array, ObjectValidators.ObjectID}, default: [])
-    field(:announcements, {:array, :string}, default: [])
+    field(:announcements, {:array, ObjectValidators.ObjectID}, default: [])
  end
  def cast_and_apply(data) do
@ -62,19 +65,14 @@ def cast_and_validate(data) do
  end
  def cast_data(data) do
    data = fix(data)
    %__MODULE__{}
    |> changeset(data)
  end
-  defp fix_url(%{"url" => url} = data) when is_list(url) do
+  defp fix_url(%{"url" => url} = data) when is_map(url) do
-    attachment =
+    Map.put(data, "url", url["href"])
      Enum.find(url, fn x -> is_map(x) and String.starts_with?(x["mimeType"], "audio/") end)
    link_element = Enum.find(url, fn x -> is_map(x) and x["mimeType"] == "text/html" end)
    data
    |> Map.put("attachment", [attachment])
    |> Map.put("url", link_element["href"])
  end
  defp fix_url(data), do: data
@ -83,8 +81,9 @@ defp fix(data) do
    data
    |> CommonFixes.fix_defaults()
    |> CommonFixes.fix_attribution()
-    |> Transmogrifier.fix_emoji()
+    |> CommonFixes.fix_actor()
    |> fix_url()
    |> Transmogrifier.fix_emoji()
  end
  def changeset(struct, data) do
@ -97,8 +96,8 @@ def changeset(struct, data) do
  def validate_data(data_cng) do
    data_cng
-    |> validate_inclusion(:type, ["Audio"])
+    |> validate_inclusion(:type, ["Article", "Note"])
-    |> validate_required([:id, :actor, :attributedTo, :type, :context, :attachment])
+    |> validate_required([:id, :actor, :attributedTo, :type, :context, :context_id])
    |> CommonValidations.validate_any_presence([:cc, :to])
    |> CommonValidations.validate_fields_match([:actor, :attributedTo])
    |> CommonValidations.validate_actor_presence()
--- a/lib/pleroma/web/activity_pub/object_validators/attachment_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/attachment_validator.ex
@ -5,6 +5,7 @@
 defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do
  use Ecto.Schema
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Web.ActivityPub.ObjectValidators.UrlObjectValidator
  import Ecto.Changeset
@ -15,7 +16,11 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator do
    field(:mediaType, :string, default: "application/octet-stream")
    field(:name, :string)
-    embeds_many(:url, UrlObjectValidator)
+    embeds_many :url, UrlObjectValidator, primary_key: false do
      field(:type, :string)
      field(:href, ObjectValidators.Uri)
      field(:mediaType, :string, default: "application/octet-stream")
    end
  end
  def cast_and_validate(data) do
@ -37,7 +42,18 @@ def changeset(struct, data) do
    struct
    |> cast(data, [:type, :mediaType, :name])
-    |> cast_embed(:url, required: true)
+    |> cast_embed(:url, with: &url_changeset/2)
    |> validate_inclusion(:type, ~w[Link Document Audio Image Video])
    |> validate_required([:type, :mediaType, :url])
  end
  def url_changeset(struct, data) do
    data = fix_media_type(data)
    struct
    |> cast(data, [:type, :href, :mediaType])
    |> validate_inclusion(:type, ["Link"])
    |> validate_required([:type, :href, :mediaType])
  end
  def fix_media_type(data) do
@ -75,6 +91,7 @@ defp fix_url(data) do
  def validate_data(cng) do
    cng
    |> validate_inclusion(:type, ~w[Document Audio Image Video])
    |> validate_required([:mediaType, :url, :type])
  end
 end
--- a/lib/pleroma/web/activity_pub/object_validators/audio_video_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/audio_video_validator.ex
@ -0,0 +1,134 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.ActivityPub.ObjectValidators.AudioVideoValidator do
  use Ecto.Schema
  alias Pleroma.EarmarkRenderer
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidator
  alias Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes
  alias Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
  alias Pleroma.Web.ActivityPub.Transmogrifier
  import Ecto.Changeset
  @primary_key false
  @derive Jason.Encoder
  embedded_schema do
    field(:id, ObjectValidators.ObjectID, primary_key: true)
    field(:to, ObjectValidators.Recipients, default: [])
    field(:cc, ObjectValidators.Recipients, default: [])
    field(:bto, ObjectValidators.Recipients, default: [])
    field(:bcc, ObjectValidators.Recipients, default: [])
    # TODO: Write type
    field(:tag, {:array, :map}, default: [])
    field(:type, :string)
    field(:name, :string)
    field(:summary, :string)
    field(:content, :string)
    field(:context, :string)
    # short identifier for PleromaFE to group statuses by context
    field(:context_id, :integer)
    # TODO: Remove actor on objects
    field(:actor, ObjectValidators.ObjectID)
    field(:attributedTo, ObjectValidators.ObjectID)
    field(:published, ObjectValidators.DateTime)
    field(:emoji, ObjectValidators.Emoji, default: %{})
    field(:sensitive, :boolean, default: false)
    embeds_many(:attachment, AttachmentValidator)
    field(:replies_count, :integer, default: 0)
    field(:like_count, :integer, default: 0)
    field(:announcement_count, :integer, default: 0)
    field(:inReplyTo, ObjectValidators.ObjectID)
    field(:url, ObjectValidators.Uri)
    field(:likes, {:array, ObjectValidators.ObjectID}, default: [])
    field(:announcements, {:array, ObjectValidators.ObjectID}, default: [])
  end
  def cast_and_apply(data) do
    data
    |> cast_data
    |> apply_action(:insert)
  end
  def cast_and_validate(data) do
    data
    |> cast_data()
    |> validate_data()
  end
  def cast_data(data) do
    %__MODULE__{}
    |> changeset(data)
  end
  defp fix_url(%{"url" => url} = data) when is_list(url) do
    attachment =
      Enum.find(url, fn x ->
        mime_type = x["mimeType"] || x["mediaType"] || ""
        is_map(x) and String.starts_with?(mime_type, ["video/", "audio/"])
      end)
    link_element =
      Enum.find(url, fn x ->
        mime_type = x["mimeType"] || x["mediaType"] || ""
        is_map(x) and mime_type == "text/html"
      end)
    data
    |> Map.put("attachment", [attachment])
    |> Map.put("url", link_element["href"])
  end
  defp fix_url(data), do: data
  defp fix_content(%{"mediaType" => "text/markdown", "content" => content} = data)
       when is_binary(content) do
    content =
      content
      |> Earmark.as_html!(%Earmark.Options{renderer: EarmarkRenderer})
      |> Pleroma.HTML.filter_tags()
    Map.put(data, "content", content)
  end
  defp fix_content(data), do: data
  defp fix(data) do
    data
    |> CommonFixes.fix_defaults()
    |> CommonFixes.fix_attribution()
    |> CommonFixes.fix_actor()
    |> Transmogrifier.fix_emoji()
    |> fix_url()
    |> fix_content()
  end
  def changeset(struct, data) do
    data = fix(data)
    struct
    |> cast(data, __schema__(:fields) -- [:attachment])
    |> cast_embed(:attachment)
  end
  def validate_data(data_cng) do
    data_cng
    |> validate_inclusion(:type, ["Audio", "Video"])
    |> validate_required([:id, :actor, :attributedTo, :type, :context, :attachment])
    |> CommonValidations.validate_any_presence([:cc, :to])
    |> CommonValidations.validate_fields_match([:actor, :attributedTo])
    |> CommonValidations.validate_actor_presence()
    |> CommonValidations.validate_host_match()
  end
 end
--- a/lib/pleroma/web/activity_pub/object_validators/common_fixes.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/common_fixes.ex
@ -3,6 +3,7 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes do
  alias Pleroma.Object.Containment
  alias Pleroma.Web.ActivityPub.Utils
  # based on Pleroma.Web.ActivityPub.Utils.lazy_put_objects_defaults
@ -19,4 +20,12 @@ def fix_attribution(data) do
    data
    |> Map.put_new("actor", data["attributedTo"])
  end
  def fix_actor(data) do
    actor = Containment.get_actor(data)
    data
    |> Map.put("actor", actor)
    |> Map.put("attributedTo", actor)
  end
 end
--- a/lib/pleroma/web/activity_pub/object_validators/create_generic_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/create_generic_validator.ex
@ -10,9 +10,10 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.CreateGenericValidator do
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Object
  alias Pleroma.Web.ActivityPub.ObjectValidators.CommonFixes
  alias Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
  import Ecto.Changeset
  import Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
  @primary_key false
@ -75,14 +76,15 @@ defp fix(data, meta) do
    data
    |> fix_context(meta)
    |> fix_addressing(meta)
    |> CommonFixes.fix_actor()
  end
  def validate_data(cng, meta \\ []) do
    cng
    |> validate_required([:actor, :type, :object])
    |> validate_inclusion(:type, ["Create"])
-    |> validate_actor_presence()
+    |> CommonValidations.validate_actor_presence()
-    |> validate_any_presence([:to, :cc])
+    |> CommonValidations.validate_any_presence([:to, :cc])
    |> validate_actors_match(meta)
    |> validate_context_match(meta)
    |> validate_object_nonexistence()
--- a/lib/pleroma/web/activity_pub/object_validators/note_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/note_validator.ex
@ -1,73 +0,0 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.ActivityPub.ObjectValidators.NoteValidator do
  use Ecto.Schema
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Web.ActivityPub.Transmogrifier
  import Ecto.Changeset
  @primary_key false
  embedded_schema do
    field(:id, ObjectValidators.ObjectID, primary_key: true)
    field(:to, ObjectValidators.Recipients, default: [])
    field(:cc, ObjectValidators.Recipients, default: [])
    field(:bto, ObjectValidators.Recipients, default: [])
    field(:bcc, ObjectValidators.Recipients, default: [])
    # TODO: Write type
    field(:tag, {:array, :map}, default: [])
    field(:type, :string)
    field(:name, :string)
    field(:summary, :string)
    field(:content, :string)
    field(:context, :string)
    # short identifier for PleromaFE to group statuses by context
    field(:context_id, :integer)
    field(:actor, ObjectValidators.ObjectID)
    field(:attributedTo, ObjectValidators.ObjectID)
    field(:published, ObjectValidators.DateTime)
    field(:emoji, ObjectValidators.Emoji, default: %{})
    field(:sensitive, :boolean, default: false)
    # TODO: Write type
    field(:attachment, {:array, :map}, default: [])
    field(:replies_count, :integer, default: 0)
    field(:like_count, :integer, default: 0)
    field(:announcement_count, :integer, default: 0)
    field(:inReplyTo, ObjectValidators.ObjectID)
    field(:url, ObjectValidators.Uri)
    field(:likes, {:array, :string}, default: [])
    field(:announcements, {:array, :string}, default: [])
  end
  def cast_and_validate(data) do
    data
    |> cast_data()
    |> validate_data()
  end
  defp fix(data) do
    data
    |> Transmogrifier.fix_emoji()
  end
  def cast_data(data) do
    data = fix(data)
    %__MODULE__{}
    |> cast(data, __schema__(:fields))
  end
  def validate_data(data_cng) do
    data_cng
    |> validate_inclusion(:type, ["Note"])
    |> validate_required([:id, :actor, :to, :cc, :type, :content, :context])
  end
 end
--- a/lib/pleroma/web/activity_pub/object_validators/question_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/question_validator.ex
@ -47,8 +47,8 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.QuestionValidator do
    # short identifier for PleromaFE to group statuses by context
    field(:context_id, :integer)
-    field(:likes, {:array, :string}, default: [])
+    field(:likes, {:array, ObjectValidators.ObjectID}, default: [])
-    field(:announcements, {:array, :string}, default: [])
+    field(:announcements, {:array, ObjectValidators.ObjectID}, default: [])
    field(:closed, ObjectValidators.DateTime)
    field(:voters, {:array, ObjectValidators.ObjectID}, default: [])
--- a/lib/pleroma/web/activity_pub/object_validators/url_object_validator.ex
+++ b/lib/pleroma/web/activity_pub/object_validators/url_object_validator.ex
@ -1,24 +0,0 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.ActivityPub.ObjectValidators.UrlObjectValidator do
  use Ecto.Schema
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  import Ecto.Changeset
  @primary_key false
  embedded_schema do
    field(:type, :string)
    field(:href, ObjectValidators.Uri)
    field(:mediaType, :string, default: "application/octet-stream")
  end
  def changeset(struct, data) do
    struct
    |> cast(data, __schema__(:fields))
    |> validate_required([:type, :href, :mediaType])
  end
 end
--- a/lib/pleroma/web/activity_pub/pipeline.ex
+++ b/lib/pleroma/web/activity_pub/pipeline.ex
@ -26,13 +26,17 @@ def common_pipeline(object, meta) do
      {:error, e} ->
        {:error, e}
      {:reject, e} ->
        {:reject, e}
    end
  end
  def do_common_pipeline(object, meta) do
    with {_, {:ok, validated_object, meta}} <-
           {:validate_object, ObjectValidator.validate(object, meta)},
-         {_, {:ok, mrfd_object}} <- {:mrf_object, MRF.filter(validated_object)},
+         {_, {:ok, mrfd_object, meta}} <-
           {:mrf_object, MRF.pipeline_filter(validated_object, meta)},
         {_, {:ok, activity, meta}} <-
           {:persist_object, ActivityPub.persist(mrfd_object, meta)},
         {_, {:ok, activity, meta}} <-
@ -40,7 +44,7 @@ def do_common_pipeline(object, meta) do
         {_, {:ok, _}} <- {:federation, maybe_federate(activity, meta)} do
      {:ok, activity, meta}
    else
-      {:mrf_object, {:reject, _}} -> {:ok, nil, meta}
+      {:mrf_object, {:reject, message, _}} -> {:reject, message}
      e -> {:error, e}
    end
  end
--- a/lib/pleroma/web/activity_pub/publisher.ex
+++ b/lib/pleroma/web/activity_pub/publisher.ex
@ -13,6 +13,7 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
  alias Pleroma.User
  alias Pleroma.Web.ActivityPub.Relay
  alias Pleroma.Web.ActivityPub.Transmogrifier
  alias Pleroma.Web.FedSockets
  require Pleroma.Constants
@ -50,15 +51,35 @@ def is_representable?(%Activity{} = activity) do
  def publish_one(%{inbox: inbox, json: json, actor: %User{} = actor, id: id} = params) do
    Logger.debug("Federating #{id} to #{inbox}")
-    uri = URI.parse(inbox)
+    case FedSockets.get_or_create_fed_socket(inbox) do
      {:ok, fedsocket} ->
        Logger.debug("publishing via fedsockets - #{inspect(inbox)}")
        FedSockets.publish(fedsocket, json)
      _ ->
        Logger.debug("publishing via http - #{inspect(inbox)}")
        http_publish(inbox, actor, json, params)
    end
  end
  def publish_one(%{actor_id: actor_id} = params) do
    actor = User.get_cached_by_id(actor_id)
    params
    |> Map.delete(:actor_id)
    |> Map.put(:actor, actor)
    |> publish_one()
  end
  defp http_publish(inbox, actor, json, params) do
    uri = %{path: path} = URI.parse(inbox)
    digest = "SHA-256=" <> (:crypto.hash(:sha256, json) |> Base.encode64())
    date = Pleroma.Signature.signed_date()
    signature =
      Pleroma.Signature.sign(actor, %{
-        "(request-target)": "post #{uri.path}",
+        "(request-target)": "post #{path}",
        host: signature_host(uri),
        "content-length": byte_size(json),
        digest: digest,
@ -89,15 +110,6 @@ def publish_one(%{inbox: inbox, json: json, actor: %User{} = actor, id: id} = pa
    end
  end
  def publish_one(%{actor_id: actor_id} = params) do
    actor = User.get_cached_by_id(actor_id)
    params
    |> Map.delete(:actor_id)
    |> Map.put(:actor, actor)
    |> publish_one()
  end
  defp signature_host(%URI{port: port, scheme: scheme, host: host}) do
    if port == URI.default_port(scheme) do
      host
--- a/lib/pleroma/web/activity_pub/side_effects.ex
+++ b/lib/pleroma/web/activity_pub/side_effects.ex
@ -7,7 +7,6 @@ defmodule Pleroma.Web.ActivityPub.SideEffects do
  """
  alias Pleroma.Activity
  alias Pleroma.Activity.Ir.Topics
  alias Pleroma.ActivityExpiration
  alias Pleroma.Chat
  alias Pleroma.Chat.MessageReference
  alias Pleroma.FollowingRelationship
@ -188,10 +187,6 @@ def handle(%{data: %{"type" => "Create"}} = activity, meta) do
        Object.increase_replies_count(in_reply_to)
      end
      if expires_at = activity.data["expires_at"] do
        ActivityExpiration.create(activity, expires_at)
      end
      BackgroundWorker.enqueue("fetch_data_for_activity", %{"activity_id" => activity.id})
      meta =
@ -341,7 +336,7 @@ def handle_object_creation(%{"type" => "Answer"} = object_map, meta) do
  end
  def handle_object_creation(%{"type" => objtype} = object, meta)
-      when objtype in ~w[Audio Question Event] do
+      when objtype in ~w[Audio Video Question Event Article] do
    with {:ok, object, meta} <- Pipeline.common_pipeline(object, meta) do
      {:ok, object, meta}
    end
--- a/lib/pleroma/web/activity_pub/transmogrifier.ex
+++ b/lib/pleroma/web/activity_pub/transmogrifier.ex
@ -7,7 +7,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
  A module to handle coding from internal to wire ActivityPub and back.
  """
  alias Pleroma.Activity
  alias Pleroma.EarmarkRenderer
  alias Pleroma.EctoType.ActivityPub.ObjectValidators
  alias Pleroma.Maps
  alias Pleroma.Object
@ -45,7 +44,6 @@ def fix_object(object, options \\ []) do
    |> fix_addressing
    |> fix_summary
    |> fix_type(options)
    |> fix_content
  end
  def fix_summary(%{"summary" => nil} = object) do
@ -168,7 +166,6 @@ def fix_in_reply_to(object, options \\ [])
  def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
      when not is_nil(in_reply_to) do
    in_reply_to_id = prepare_in_reply_to(in_reply_to)
    object = Map.put(object, "inReplyToAtomUri", in_reply_to_id)
    depth = (options[:depth] || 0) + 1
    if Federator.allowed_thread_distance?(depth) do
@ -176,9 +173,8 @@ def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
           %Activity{} <- Activity.get_create_by_object_ap_id(replied_object.data["id"]) do
        object
        |> Map.put("inReplyTo", replied_object.data["id"])
        |> Map.put("inReplyToAtomUri", object["inReplyToAtomUri"] || in_reply_to_id)
        |> Map.put("context", replied_object.data["context"] || object["conversation"])
-        |> Map.drop(["conversation"])
+        |> Map.drop(["conversation", "inReplyToAtomUri"])
      else
        e ->
          Logger.warn("Couldn't fetch #{inspect(in_reply_to_id)}, error: #{inspect(e)}")
@ -276,24 +272,7 @@ def fix_url(%{"url" => url} = object) when is_map(url) do
    Map.put(object, "url", url["href"])
  end
-  def fix_url(%{"type" => "Video", "url" => url} = object) when is_list(url) do
+  def fix_url(%{"url" => url} = object) when is_list(url) do
    attachment =
      Enum.find(url, fn x ->
        media_type = x["mediaType"] || x["mimeType"] || ""
        is_map(x) and String.starts_with?(media_type, "video/")
      end)
    link_element =
      Enum.find(url, fn x -> is_map(x) and (x["mediaType"] || x["mimeType"]) == "text/html" end)
    object
    |> Map.put("attachment", [attachment])
    |> Map.put("url", link_element["href"])
  end
  def fix_url(%{"type" => object_type, "url" => url} = object)
      when object_type != "Video" and is_list(url) do
    first_element = Enum.at(url, 0)
    url_string =
@ -311,7 +290,7 @@ def fix_url(object), do: object
  def fix_emoji(%{"tag" => tags} = object) when is_list(tags) do
    emoji =
      tags
-      |> Enum.filter(fn data -> data["type"] == "Emoji" and data["icon"] end)
+      |> Enum.filter(fn data -> is_map(data) and data["type"] == "Emoji" and data["icon"] end)
      |> Enum.reduce(%{}, fn data, mapping ->
        name = String.trim(data["name"], ":")
@ -373,18 +352,6 @@ def fix_type(%{"inReplyTo" => reply_id, "name" => _} = object, options)
  def fix_type(object, _), do: object
  defp fix_content(%{"mediaType" => "text/markdown", "content" => content} = object)
       when is_binary(content) do
    html_content =
      content
      |> Earmark.as_html!(%Earmark.Options{renderer: EarmarkRenderer})
      |> Pleroma.HTML.filter_tags()
    Map.merge(object, %{"content" => html_content, "mediaType" => "text/html"})
  end
  defp fix_content(object), do: object
  # Reduce the object list to find the reported user.
  defp get_reported(objects) do
    Enum.reduce_while(objects, nil, fn ap_id, _ ->
@ -457,7 +424,7 @@ def handle_incoming(
        %{"type" => "Create", "object" => %{"type" => objtype} = object} = data,
        options
      )
-      when objtype in ~w{Article Note Video Page} do
+      when objtype in ~w{Note Page} do
    actor = Containment.get_actor(data)
    with nil <- Activity.get_create_by_object_ap_id(object["id"]),
@ -551,7 +518,9 @@ def handle_incoming(
        %{"type" => "Create", "object" => %{"type" => objtype}} = data,
        _options
      )
-      when objtype in ~w{Question Answer ChatMessage Audio Event} do
+      when objtype in ~w{Question Answer ChatMessage Audio Video Event Article} do
    data = Map.put(data, "object", strip_internal_fields(data["object"]))
    with {:ok, %User{}} <- ObjectValidator.fetch_actor(data),
         {:ok, activity, _} <- Pipeline.common_pipeline(data, local: false) do
      {:ok, activity}
@ -1031,7 +1000,7 @@ def perform(:user_upgrade, user) do
  def upgrade_user_from_ap_id(ap_id) do
    with %User{local: false} = user <- User.get_cached_by_ap_id(ap_id),
-         {:ok, data} <- ActivityPub.fetch_and_prepare_user_from_ap_id(ap_id),
+         {:ok, data} <- ActivityPub.fetch_and_prepare_user_from_ap_id(ap_id, force_http: true),
         {:ok, user} <- update_user(user, data) do
      TransmogrifierWorker.enqueue("user_upgrade", %{"user_id" => user.id})
      {:ok, user}
--- a/lib/pleroma/web/admin_api/controllers/admin_api_controller.ex
+++ b/lib/pleroma/web/admin_api/controllers/admin_api_controller.ex
@ -23,8 +23,6 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
  alias Pleroma.Web.Endpoint
  alias Pleroma.Web.Router
  require Logger
  @users_page_size 50
  plug(
@ -68,6 +66,12 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
    when action in [:list_user_statuses, :list_instance_statuses]
  )
  plug(
    OAuthScopesPlug,
    %{scopes: ["read:chats"], admin: true}
    when action in [:list_user_chats]
  )
  plug(
    OAuthScopesPlug,
    %{scopes: ["read"], admin: true}
@ -256,6 +260,20 @@ def list_user_statuses(%{assigns: %{user: admin}} = conn, %{"nickname" => nickna
    end
  end
  def list_user_chats(%{assigns: %{user: admin}} = conn, %{"nickname" => nickname} = _params) do
    with %User{id: user_id} <- User.get_cached_by_nickname_or_id(nickname, for: admin) do
      chats =
        Pleroma.Chat.for_user_query(user_id)
        |> Pleroma.Repo.all()
      conn
      |> put_view(AdminAPI.ChatView)
      |> render("index.json", chats: chats)
    else
      _ -> {:error, :not_found}
    end
  end
  def user_toggle_activation(%{assigns: %{user: admin}} = conn, %{"nickname" => nickname}) do
    user = User.get_cached_by_nickname(nickname)
--- a/lib/pleroma/web/admin_api/controllers/chat_controller.ex
+++ b/lib/pleroma/web/admin_api/controllers/chat_controller.ex
@ -0,0 +1,85 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.AdminAPI.ChatController do
  use Pleroma.Web, :controller
  alias Pleroma.Activity
  alias Pleroma.Chat
  alias Pleroma.Chat.MessageReference
  alias Pleroma.ModerationLog
  alias Pleroma.Pagination
  alias Pleroma.Plugs.OAuthScopesPlug
  alias Pleroma.Web.AdminAPI
  alias Pleroma.Web.CommonAPI
  alias Pleroma.Web.PleromaAPI.Chat.MessageReferenceView
  require Logger
  plug(Pleroma.Web.ApiSpec.CastAndValidate)
  plug(
    OAuthScopesPlug,
    %{scopes: ["read:chats"], admin: true} when action in [:show, :messages]
  )
  plug(
    OAuthScopesPlug,
    %{scopes: ["write:chats"], admin: true} when action in [:delete_message]
  )
  action_fallback(Pleroma.Web.AdminAPI.FallbackController)
  defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.Admin.ChatOperation
  def delete_message(%{assigns: %{user: user}} = conn, %{
        message_id: message_id,
        id: chat_id
      }) do
    with %MessageReference{object: %{data: %{"id" => object_ap_id}}} = cm_ref <-
           MessageReference.get_by_id(message_id),
         ^chat_id <- to_string(cm_ref.chat_id),
         %Activity{id: activity_id} <- Activity.get_create_by_object_ap_id(object_ap_id),
         {:ok, _} <- CommonAPI.delete(activity_id, user) do
      ModerationLog.insert_log(%{
        action: "chat_message_delete",
        actor: user,
        subject_id: message_id
      })
      conn
      |> put_view(MessageReferenceView)
      |> render("show.json", chat_message_reference: cm_ref)
    else
      _e ->
        {:error, :could_not_delete}
    end
  end
  def messages(conn, %{id: id} = params) do
    with %Chat{} = chat <- Chat.get_by_id(id) do
      cm_refs =
        chat
        |> MessageReference.for_chat_query()
        |> Pagination.fetch_paginated(params)
      conn
      |> put_view(MessageReferenceView)
      |> render("index.json", chat_message_references: cm_refs)
    else
      _ ->
        conn
        |> put_status(:not_found)
        |> json(%{error: "not found"})
    end
  end
  def show(conn, %{id: id}) do
    with %Chat{} = chat <- Chat.get_by_id(id) do
      conn
      |> put_view(AdminAPI.ChatView)
      |> render("show.json", chat: chat)
    end
  end
 end
--- a/lib/pleroma/web/admin_api/controllers/instance_document_controller.ex
+++ b/lib/pleroma/web/admin_api/controllers/instance_document_controller.ex
@ -0,0 +1,41 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.AdminAPI.InstanceDocumentController do
  use Pleroma.Web, :controller
  alias Pleroma.Plugs.InstanceStatic
  alias Pleroma.Plugs.OAuthScopesPlug
  alias Pleroma.Web.InstanceDocument
  plug(Pleroma.Web.ApiSpec.CastAndValidate)
  action_fallback(Pleroma.Web.AdminAPI.FallbackController)
  defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.Admin.InstanceDocumentOperation
  plug(OAuthScopesPlug, %{scopes: ["read"], admin: true} when action == :show)
  plug(OAuthScopesPlug, %{scopes: ["write"], admin: true} when action in [:update, :delete])
  def show(conn, %{name: document_name}) do
    with {:ok, url} <- InstanceDocument.get(document_name),
         {:ok, content} <- File.read(InstanceStatic.file_path(url)) do
      conn
      |> put_resp_content_type("text/html")
      |> send_resp(200, content)
    end
  end
  def update(%{body_params: %{file: file}} = conn, %{name: document_name}) do
    with {:ok, url} <- InstanceDocument.put(document_name, file.path) do
      json(conn, %{"url" => url})
    end
  end
  def delete(conn, %{name: document_name}) do
    with :ok <- InstanceDocument.delete(document_name) do
      json(conn, %{})
    end
  end
 end
--- a/lib/pleroma/web/admin_api/views/chat_view.ex
+++ b/lib/pleroma/web/admin_api/views/chat_view.ex
@ -0,0 +1,30 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.AdminAPI.ChatView do
  use Pleroma.Web, :view
  alias Pleroma.Chat
  alias Pleroma.User
  alias Pleroma.Web.MastodonAPI
  alias Pleroma.Web.PleromaAPI
  def render("index.json", %{chats: chats} = opts) do
    render_many(chats, __MODULE__, "show.json", Map.delete(opts, :chats))
  end
  def render("show.json", %{chat: %Chat{user_id: user_id}} = opts) do
    user = User.get_by_id(user_id)
    sender = MastodonAPI.AccountView.render("show.json", user: user, skip_visibility_check: true)
    serialized_chat = PleromaAPI.ChatView.render("show.json", opts)
    serialized_chat
    |> Map.put(:sender, sender)
    |> Map.put(:receiver, serialized_chat[:account])
    |> Map.delete(:account)
  end
  def render(view, opts), do: PleromaAPI.ChatView.render(view, opts)
 end
--- a/lib/pleroma/web/admin_api/views/status_view.ex
+++ b/lib/pleroma/web/admin_api/views/status_view.ex
@ -8,6 +8,7 @@ defmodule Pleroma.Web.AdminAPI.StatusView do
  require Pleroma.Constants
  alias Pleroma.Web.AdminAPI
  alias Pleroma.Web.CommonAPI
  alias Pleroma.Web.MastodonAPI
  defdelegate merge_account_views(user), to: AdminAPI.AccountView
@ -17,7 +18,7 @@ def render("index.json", opts) do
  end
  def render("show.json", %{activity: %{data: %{"object" => _object}} = activity} = opts) do
-    user = MastodonAPI.StatusView.get_user(activity.data["actor"])
+    user = CommonAPI.get_user(activity.data["actor"])
    MastodonAPI.StatusView.render("show.json", opts)
    |> Map.merge(%{account: merge_account_views(user)})
--- a/lib/pleroma/web/api_spec.ex
+++ b/lib/pleroma/web/api_spec.ex
@ -13,10 +13,15 @@ defmodule Pleroma.Web.ApiSpec do
  @impl OpenApi
  def spec do
    %OpenApi{
-      servers: [
+      servers:
        if Phoenix.Endpoint.server?(:pleroma, Endpoint) do
          [
            # Populate the Server info from a phoenix endpoint
            OpenApiSpex.Server.from_endpoint(Endpoint)
-      ],
+          ]
        else
          []
        end,
      info: %OpenApiSpex.Info{
        title: "Pleroma",
        description: Application.spec(:pleroma, :description) |> to_string(),
--- a/lib/pleroma/web/api_spec/helpers.ex
+++ b/lib/pleroma/web/api_spec/helpers.ex
@ -72,7 +72,11 @@ def empty_object_response do
  end
  def empty_array_response do
-    Operation.response("Empty array", "application/json", %Schema{type: :array, example: []})
+    Operation.response("Empty array", "application/json", %Schema{
      type: :array,
      items: %Schema{type: :object, example: %{}},
      example: []
    })
  end
  def no_content_response do
--- a/lib/pleroma/web/api_spec/operations/account_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/account_operation.ex
@ -372,6 +372,10 @@ def identity_proofs_operation do
      tags: ["accounts"],
      summary: "Identity proofs",
      operationId: "AccountController.identity_proofs",
      # Validators complains about unused path params otherwise
      parameters: [
        %Reference{"$ref": "#/components/parameters/accountIdOrNickname"}
      ],
      description: "Not implemented",
      responses: %{
        200 => empty_array_response()
@ -469,7 +473,6 @@ defp create_response do
        identifier: %Schema{type: :string},
        message: %Schema{type: :string}
      },
      required: [],
      # Note: example of successful registration with failed login response:
      # example: %{
      #   "identifier" => "missing_confirmed_email",
@ -530,7 +533,7 @@ defp update_credentials_request do
          nullable: true,
          oneOf: [
            %Schema{type: :array, items: attribute_field()},
-            %Schema{type: :object, additionalProperties: %Schema{type: attribute_field()}}
+            %Schema{type: :object, additionalProperties: attribute_field()}
          ]
        },
        # NOTE: `source` field is not supported
--- a/lib/pleroma/web/api_spec/operations/admin/chat_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/admin/chat_operation.ex
@ -0,0 +1,96 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.ApiSpec.Admin.ChatOperation do
  alias OpenApiSpex.Operation
  alias Pleroma.Web.ApiSpec.Schemas.Chat
  alias Pleroma.Web.ApiSpec.Schemas.ChatMessage
  import Pleroma.Web.ApiSpec.Helpers
  def open_api_operation(action) do
    operation = String.to_existing_atom("#{action}_operation")
    apply(__MODULE__, operation, [])
  end
  def delete_message_operation do
    %Operation{
      tags: ["admin", "chat"],
      summary: "Delete an individual chat message",
      operationId: "AdminAPI.ChatController.delete_message",
      parameters: [
        Operation.parameter(:id, :path, :string, "The ID of the Chat"),
        Operation.parameter(:message_id, :path, :string, "The ID of the message")
      ],
      responses: %{
        200 =>
          Operation.response(
            "The deleted ChatMessage",
            "application/json",
            ChatMessage
          )
      },
      security: [
        %{
          "oAuth" => ["write:chats"]
        }
      ]
    }
  end
  def messages_operation do
    %Operation{
      tags: ["admin", "chat"],
      summary: "Get the most recent messages of the chat",
      operationId: "AdminAPI.ChatController.messages",
      parameters:
        [Operation.parameter(:id, :path, :string, "The ID of the Chat")] ++
          pagination_params(),
      responses: %{
        200 =>
          Operation.response(
            "The messages in the chat",
            "application/json",
            Pleroma.Web.ApiSpec.ChatOperation.chat_messages_response()
          )
      },
      security: [
        %{
          "oAuth" => ["read:chats"]
        }
      ]
    }
  end
  def show_operation do
    %Operation{
      tags: ["chat"],
      summary: "Create a chat",
      operationId: "AdminAPI.ChatController.show",
      parameters: [
        Operation.parameter(
          :id,
          :path,
          :string,
          "The id of the chat",
          required: true,
          example: "1234"
        )
      ],
      responses: %{
        200 =>
          Operation.response(
            "The existing chat",
            "application/json",
            Chat
          )
      },
      security: [
        %{
          "oAuth" => ["read"]
        }
      ]
    }
  end
 end
--- a/lib/pleroma/web/api_spec/operations/admin/instance_document_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/admin/instance_document_operation.ex
@ -0,0 +1,115 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.ApiSpec.Admin.InstanceDocumentOperation do
  alias OpenApiSpex.Operation
  alias OpenApiSpex.Schema
  alias Pleroma.Web.ApiSpec.Helpers
  alias Pleroma.Web.ApiSpec.Schemas.ApiError
  def open_api_operation(action) do
    operation = String.to_existing_atom("#{action}_operation")
    apply(__MODULE__, operation, [])
  end
  def show_operation do
    %Operation{
      tags: ["Admin", "InstanceDocument"],
      summary: "Get the instance document",
      operationId: "AdminAPI.InstanceDocumentController.show",
      security: [%{"oAuth" => ["read"]}],
      parameters: [
        Operation.parameter(:name, :path, %Schema{type: :string}, "The document name",
          required: true
        )
        | Helpers.admin_api_params()
      ],
      responses: %{
        200 => document_content(),
        400 => Operation.response("Bad Request", "application/json", ApiError),
        403 => Operation.response("Forbidden", "application/json", ApiError),
        404 => Operation.response("Not Found", "application/json", ApiError)
      }
    }
  end
  def update_operation do
    %Operation{
      tags: ["Admin", "InstanceDocument"],
      summary: "Update the instance document",
      operationId: "AdminAPI.InstanceDocumentController.update",
      security: [%{"oAuth" => ["write"]}],
      requestBody: Helpers.request_body("Parameters", update_request()),
      parameters: [
        Operation.parameter(:name, :path, %Schema{type: :string}, "The document name",
          required: true
        )
        | Helpers.admin_api_params()
      ],
      responses: %{
        200 => Operation.response("InstanceDocument", "application/json", instance_document()),
        400 => Operation.response("Bad Request", "application/json", ApiError),
        403 => Operation.response("Forbidden", "application/json", ApiError),
        404 => Operation.response("Not Found", "application/json", ApiError)
      }
    }
  end
  defp update_request do
    %Schema{
      title: "UpdateRequest",
      description: "POST body for uploading the file",
      type: :object,
      required: [:file],
      properties: %{
        file: %Schema{
          type: :string,
          format: :binary,
          description: "The file to be uploaded, using multipart form data."
        }
      }
    }
  end
  def delete_operation do
    %Operation{
      tags: ["Admin", "InstanceDocument"],
      summary: "Get the instance document",
      operationId: "AdminAPI.InstanceDocumentController.delete",
      security: [%{"oAuth" => ["write"]}],
      parameters: [
        Operation.parameter(:name, :path, %Schema{type: :string}, "The document name",
          required: true
        )
        | Helpers.admin_api_params()
      ],
      responses: %{
        200 => Operation.response("InstanceDocument", "application/json", instance_document()),
        400 => Operation.response("Bad Request", "application/json", ApiError),
        403 => Operation.response("Forbidden", "application/json", ApiError),
        404 => Operation.response("Not Found", "application/json", ApiError)
      }
    }
  end
  defp instance_document do
    %Schema{
      title: "InstanceDocument",
      type: :object,
      properties: %{
        url: %Schema{type: :string}
      },
      example: %{
        "url" => "https://example.com/static/terms-of-service.html"
      }
    }
  end
  defp document_content do
    Operation.response("InstanceDocumentContent", "text/html", %Schema{
      type: :string,
      example: "<h1>Instance panel</h1>"
    })
  end
 end
--- a/lib/pleroma/web/api_spec/operations/chat_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/chat_operation.ex
@ -158,7 +158,8 @@ def messages_operation do
            "The messages in the chat",
            "application/json",
            chat_messages_response()
-          )
+          ),
        404 => Operation.response("Not Found", "application/json", ApiError)
      },
      security: [
        %{
@ -184,7 +185,8 @@ def post_chat_message_operation do
            "application/json",
            ChatMessage
          ),
-        400 => Operation.response("Bad Request", "application/json", ApiError)
+        400 => Operation.response("Bad Request", "application/json", ApiError),
        422 => Operation.response("MRF Rejection", "application/json", ApiError)
      },
      security: [
        %{
--- a/lib/pleroma/web/api_spec/operations/custom_emoji_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/custom_emoji_operation.ex
@ -69,7 +69,7 @@ defp custom_emoji do
          type: :object,
          properties: %{
            category: %Schema{type: :string},
-            tags: %Schema{type: :array}
+            tags: %Schema{type: :array, items: %Schema{type: :string}}
          }
        }
      ],
--- a/lib/pleroma/web/api_spec/operations/emoji_reaction_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/emoji_reaction_operation.ex
@ -23,7 +23,7 @@ def index_operation do
      parameters: [
        Operation.parameter(:id, :path, FlakeID, "Status ID", required: true),
        Operation.parameter(:emoji, :path, :string, "Filter by a single unicode emoji",
-          required: false
+          required: nil
        )
      ],
      security: [%{"oAuth" => ["read:statuses"]}],
--- a/lib/pleroma/web/api_spec/operations/list_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/list_operation.ex
@ -187,8 +187,7 @@ defp add_remove_accounts_request(required) when is_boolean(required) do
        type: :object,
        properties: %{
          account_ids: %Schema{type: :array, description: "Array of account IDs", items: FlakeID}
-        },
+        }
        required: required && [:account_ids]
      },
      required: required
    )
--- a/lib/pleroma/web/api_spec/operations/pleroma_emoji_file_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/pleroma_emoji_file_operation.ex
@ -0,0 +1,139 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.ApiSpec.PleromaEmojiFileOperation do
  alias OpenApiSpex.Operation
  alias OpenApiSpex.Schema
  alias Pleroma.Web.ApiSpec.Schemas.ApiError
  import Pleroma.Web.ApiSpec.Helpers
  def open_api_operation(action) do
    operation = String.to_existing_atom("#{action}_operation")
    apply(__MODULE__, operation, [])
  end
  def create_operation do
    %Operation{
      tags: ["Emoji Packs"],
      summary: "Add new file to the pack",
      operationId: "PleromaAPI.EmojiPackController.add_file",
      security: [%{"oAuth" => ["write"]}],
      requestBody: request_body("Parameters", create_request(), required: true),
      parameters: [name_param()],
      responses: %{
        200 => Operation.response("Files Object", "application/json", files_object()),
        422 => Operation.response("Unprocessable Entity", "application/json", ApiError),
        404 => Operation.response("Not Found", "application/json", ApiError),
        400 => Operation.response("Bad Request", "application/json", ApiError),
        409 => Operation.response("Conflict", "application/json", ApiError)
      }
    }
  end
  defp create_request do
    %Schema{
      type: :object,
      required: [:file],
      properties: %{
        file: %Schema{
          description:
            "File needs to be uploaded with the multipart request or link to remote file",
          anyOf: [
            %Schema{type: :string, format: :binary},
            %Schema{type: :string, format: :uri}
          ]
        },
        shortcode: %Schema{
          type: :string,
          description:
            "Shortcode for new emoji, must be unique for all emoji. If not sended, shortcode will be taken from original filename."
        },
        filename: %Schema{
          type: :string,
          description:
            "New emoji file name. If not specified will be taken from original filename."
        }
      }
    }
  end
  def update_operation do
    %Operation{
      tags: ["Emoji Packs"],
      summary: "Add new file to the pack",
      operationId: "PleromaAPI.EmojiPackController.update_file",
      security: [%{"oAuth" => ["write"]}],
      requestBody: request_body("Parameters", update_request(), required: true),
      parameters: [name_param()],
      responses: %{
        200 => Operation.response("Files Object", "application/json", files_object()),
        404 => Operation.response("Not Found", "application/json", ApiError),
        400 => Operation.response("Bad Request", "application/json", ApiError),
        409 => Operation.response("Conflict", "application/json", ApiError),
        422 => Operation.response("Unprocessable Entity", "application/json", ApiError)
      }
    }
  end
  defp update_request do
    %Schema{
      type: :object,
      required: [:shortcode, :new_shortcode, :new_filename],
      properties: %{
        shortcode: %Schema{
          type: :string,
          description: "Emoji file shortcode"
        },
        new_shortcode: %Schema{
          type: :string,
          description: "New emoji file shortcode"
        },
        new_filename: %Schema{
          type: :string,
          description: "New filename for emoji file"
        },
        force: %Schema{
          type: :boolean,
          description: "With true value to overwrite existing emoji with new shortcode",
          default: false
        }
      }
    }
  end
  def delete_operation do
    %Operation{
      tags: ["Emoji Packs"],
      summary: "Delete emoji file from pack",
      operationId: "PleromaAPI.EmojiPackController.delete_file",
      security: [%{"oAuth" => ["write"]}],
      parameters: [
        name_param(),
        Operation.parameter(:shortcode, :query, :string, "File shortcode",
          example: "cofe",
          required: true
        )
      ],
      responses: %{
        200 => Operation.response("Files Object", "application/json", files_object()),
        400 => Operation.response("Bad Request", "application/json", ApiError),
        404 => Operation.response("Not Found", "application/json", ApiError),
        422 => Operation.response("Unprocessable Entity", "application/json", ApiError)
      }
    }
  end
  defp name_param do
    Operation.parameter(:name, :path, :string, "Pack Name", example: "cofe", required: true)
  end
  defp files_object do
    %Schema{
      type: :object,
      additionalProperties: %Schema{type: :string},
      description: "Object with emoji names as keys and filenames as values"
    }
  end
 end
--- a/lib/pleroma/web/api_spec/operations/pleroma_emoji_pack_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/pleroma_emoji_pack_operation.ex
@ -175,111 +175,6 @@ def update_operation do
    }
  end
  def add_file_operation do
    %Operation{
      tags: ["Emoji Packs"],
      summary: "Add new file to the pack",
      operationId: "PleromaAPI.EmojiPackController.add_file",
      security: [%{"oAuth" => ["write"]}],
      requestBody: request_body("Parameters", add_file_request(), required: true),
      parameters: [name_param()],
      responses: %{
        200 => Operation.response("Files Object", "application/json", files_object()),
        400 => Operation.response("Bad Request", "application/json", ApiError),
        409 => Operation.response("Conflict", "application/json", ApiError)
      }
    }
  end
  defp add_file_request do
    %Schema{
      type: :object,
      required: [:file],
      properties: %{
        file: %Schema{
          description:
            "File needs to be uploaded with the multipart request or link to remote file",
          anyOf: [
            %Schema{type: :string, format: :binary},
            %Schema{type: :string, format: :uri}
          ]
        },
        shortcode: %Schema{
          type: :string,
          description:
            "Shortcode for new emoji, must be unique for all emoji. If not sended, shortcode will be taken from original filename."
        },
        filename: %Schema{
          type: :string,
          description:
            "New emoji file name. If not specified will be taken from original filename."
        }
      }
    }
  end
  def update_file_operation do
    %Operation{
      tags: ["Emoji Packs"],
      summary: "Add new file to the pack",
      operationId: "PleromaAPI.EmojiPackController.update_file",
      security: [%{"oAuth" => ["write"]}],
      requestBody: request_body("Parameters", update_file_request(), required: true),
      parameters: [name_param()],
      responses: %{
        200 => Operation.response("Files Object", "application/json", files_object()),
        400 => Operation.response("Bad Request", "application/json", ApiError),
        409 => Operation.response("Conflict", "application/json", ApiError)
      }
    }
  end
  defp update_file_request do
    %Schema{
      type: :object,
      required: [:shortcode, :new_shortcode, :new_filename],
      properties: %{
        shortcode: %Schema{
          type: :string,
          description: "Emoji file shortcode"
        },
        new_shortcode: %Schema{
          type: :string,
          description: "New emoji file shortcode"
        },
        new_filename: %Schema{
          type: :string,
          description: "New filename for emoji file"
        },
        force: %Schema{
          type: :boolean,
          description: "With true value to overwrite existing emoji with new shortcode",
          default: false
        }
      }
    }
  end
  def delete_file_operation do
    %Operation{
      tags: ["Emoji Packs"],
      summary: "Delete emoji file from pack",
      operationId: "PleromaAPI.EmojiPackController.delete_file",
      security: [%{"oAuth" => ["write"]}],
      parameters: [
        name_param(),
        Operation.parameter(:shortcode, :query, :string, "File shortcode",
          example: "cofe",
          required: true
        )
      ],
      responses: %{
        200 => Operation.response("Files Object", "application/json", files_object()),
        400 => Operation.response("Bad Request", "application/json", ApiError)
      }
    }
  end
  def import_from_filesystem_operation do
    %Operation{
      tags: ["Emoji Packs"],
--- a/lib/pleroma/web/api_spec/operations/status_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/status_operation.ex
@ -55,7 +55,7 @@ def create_operation do
            "application/json",
            %Schema{oneOf: [Status, ScheduledStatus]}
          ),
-        422 => Operation.response("Bad Request", "application/json", ApiError)
+        422 => Operation.response("Bad Request / MRF Rejection", "application/json", ApiError)
      }
    }
  end
--- a/lib/pleroma/web/api_spec/operations/user_import_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/user_import_operation.ex
@ -0,0 +1,80 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.ApiSpec.UserImportOperation do
  alias OpenApiSpex.Operation
  alias OpenApiSpex.Schema
  alias Pleroma.Web.ApiSpec.Schemas.ApiError
  import Pleroma.Web.ApiSpec.Helpers
  @spec open_api_operation(atom) :: Operation.t()
  def open_api_operation(action) do
    operation = String.to_existing_atom("#{action}_operation")
    apply(__MODULE__, operation, [])
  end
  def follow_operation do
    %Operation{
      tags: ["follow_import"],
      summary: "Imports your follows.",
      operationId: "UserImportController.follow",
      requestBody: request_body("Parameters", import_request(), required: true),
      responses: %{
        200 => ok_response(),
        500 => Operation.response("Error", "application/json", ApiError)
      },
      security: [%{"oAuth" => ["write:follow"]}]
    }
  end
  def blocks_operation do
    %Operation{
      tags: ["blocks_import"],
      summary: "Imports your blocks.",
      operationId: "UserImportController.blocks",
      requestBody: request_body("Parameters", import_request(), required: true),
      responses: %{
        200 => ok_response(),
        500 => Operation.response("Error", "application/json", ApiError)
      },
      security: [%{"oAuth" => ["write:blocks"]}]
    }
  end
  def mutes_operation do
    %Operation{
      tags: ["mutes_import"],
      summary: "Imports your mutes.",
      operationId: "UserImportController.mutes",
      requestBody: request_body("Parameters", import_request(), required: true),
      responses: %{
        200 => ok_response(),
        500 => Operation.response("Error", "application/json", ApiError)
      },
      security: [%{"oAuth" => ["write:mutes"]}]
    }
  end
  defp import_request do
    %Schema{
      type: :object,
      required: [:list],
      properties: %{
        list: %Schema{
          description:
            "STRING or FILE containing a whitespace-separated list of accounts to import.",
          anyOf: [
            %Schema{type: :string, format: :binary},
            %Schema{type: :string}
          ]
        }
      }
    }
  end
  defp ok_response do
    Operation.response("Ok", "application/json", %Schema{type: :string, example: "ok"})
  end
 end
--- a/lib/pleroma/web/api_spec/schemas/chat_message.ex
+++ b/lib/pleroma/web/api_spec/schemas/chat_message.ex
@ -4,6 +4,7 @@
 defmodule Pleroma.Web.ApiSpec.Schemas.ChatMessage do
  alias OpenApiSpex.Schema
  alias Pleroma.Web.ApiSpec.Schemas.Emoji
  require OpenApiSpex
@ -18,7 +19,7 @@ defmodule Pleroma.Web.ApiSpec.Schemas.ChatMessage do
      chat_id: %Schema{type: :string},
      content: %Schema{type: :string, nullable: true},
      created_at: %Schema{type: :string, format: :"date-time"},
-      emojis: %Schema{type: :array},
+      emojis: %Schema{type: :array, items: Emoji},
      attachment: %Schema{type: :object, nullable: true},
      card: %Schema{
        type: :object,
--- a/lib/pleroma/web/api_spec/schemas/scheduled_status.ex
+++ b/lib/pleroma/web/api_spec/schemas/scheduled_status.ex
@ -27,9 +27,9 @@ defmodule Pleroma.Web.ApiSpec.Schemas.ScheduledStatus do
          media_ids: %Schema{type: :array, nullable: true, items: %Schema{type: :string}},
          sensitive: %Schema{type: :boolean, nullable: true},
          spoiler_text: %Schema{type: :string, nullable: true},
-          visibility: %Schema{type: VisibilityScope, nullable: true},
+          visibility: %Schema{allOf: [VisibilityScope], nullable: true},
          scheduled_at: %Schema{type: :string, format: :"date-time", nullable: true},
-          poll: %Schema{type: Poll, nullable: true},
+          poll: %Schema{allOf: [Poll], nullable: true},
          in_reply_to_id: %Schema{type: :string, nullable: true}
        }
      }
--- a/lib/pleroma/web/common_api/activity_draft.ex
+++ b/lib/pleroma/web/common_api/activity_draft.ex
@ -202,7 +202,7 @@ defp changes(draft) do
    additional =
      case draft.expires_at do
-        %NaiveDateTime{} = expires_at -> Map.put(additional, "expires_at", expires_at)
+        %DateTime{} = expires_at -> Map.put(additional, "expires_at", expires_at)
        _ -> additional
      end
--- a/lib/pleroma/web/common_api/common_api.ex
+++ b/lib/pleroma/web/common_api/common_api.ex
@ -4,7 +4,6 @@
 defmodule Pleroma.Web.CommonAPI do
  alias Pleroma.Activity
  alias Pleroma.ActivityExpiration
  alias Pleroma.Conversation.Participation
  alias Pleroma.Formatter
  alias Pleroma.Object
@ -49,6 +48,9 @@ def post_chat_message(%User{} = user, %User{} = recipient, content, opts \\ [])
              local: true
            )} do
      {:ok, activity}
    else
      {:common_pipeline, {:reject, _} = e} -> e
      e -> e
    end
  end
@ -381,9 +383,9 @@ def get_replied_to_visibility(activity) do
  def check_expiry_date({:ok, nil} = res), do: res
  def check_expiry_date({:ok, in_seconds}) do
-    expiry = NaiveDateTime.utc_now() |> NaiveDateTime.add(in_seconds)
+    expiry = DateTime.add(DateTime.utc_now(), in_seconds)
-    if ActivityExpiration.expires_late_enough?(expiry) do
+    if Pleroma.Workers.PurgeExpiredActivity.expires_late_enough?(expiry) do
      {:ok, expiry}
    else
      {:error, "Expiry date is too soon"}
@ -551,4 +553,21 @@ def hide_reblogs(%User{} = user, %User{} = target) do
  def show_reblogs(%User{} = user, %User{} = target) do
    UserRelationship.delete_reblog_mute(user, target)
  end
  def get_user(ap_id, fake_record_fallback \\ true) do
    cond do
      user = User.get_cached_by_ap_id(ap_id) ->
        user
      user = User.get_by_guessed_nickname(ap_id) ->
        user
      fake_record_fallback ->
        # TODO: refactor (fake records is never a good idea)
        User.error_user(ap_id)
      true ->
        nil
    end
  end
 end
--- a/lib/pleroma/web/controller_helper.ex
+++ b/lib/pleroma/web/controller_helper.ex
@ -48,13 +48,13 @@ defp param_to_integer(val, default) when is_binary(val) do
  defp param_to_integer(_, default), do: default
-  def add_link_headers(conn, activities, extra_params \\ %{})
+  def add_link_headers(conn, entries, extra_params \\ %{})
-  def add_link_headers(%{assigns: %{skip_link_headers: true}} = conn, _activities, _extra_params),
+  def add_link_headers(%{assigns: %{skip_link_headers: true}} = conn, _entries, _extra_params),
    do: conn
-  def add_link_headers(conn, activities, extra_params) do
+  def add_link_headers(conn, entries, extra_params) do
-    case get_pagination_fields(conn, activities, extra_params) do
+    case get_pagination_fields(conn, entries, extra_params) do
      %{"next" => next_url, "prev" => prev_url} ->
        put_resp_header(conn, "link", "<#{next_url}>; rel=\"next\", <#{prev_url}>; rel=\"prev\"")
@ -78,19 +78,15 @@ defp build_pagination_fields(conn, min_id, max_id, extra_params) do
    }
  end
-  def get_pagination_fields(conn, activities, extra_params \\ %{}) do
+  def get_pagination_fields(conn, entries, extra_params \\ %{}) do
-    case List.last(activities) do
+    case List.last(entries) do
      %{pagination_id: max_id} when not is_nil(max_id) ->
-        %{pagination_id: min_id} =
+        %{pagination_id: min_id} = List.first(entries)
          activities
          |> List.first()
        build_pagination_fields(conn, min_id, max_id, extra_params)
      %{id: max_id} ->
-        %{id: min_id} =
+        %{id: min_id} = List.first(entries)
          activities
          |> List.first()
        build_pagination_fields(conn, min_id, max_id, extra_params)
--- a/lib/pleroma/web/fed_sockets/fed_registry.ex
+++ b/lib/pleroma/web/fed_sockets/fed_registry.ex
@ -0,0 +1,185 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.FedSockets.FedRegistry do
  @moduledoc """
  The FedRegistry stores the active FedSockets for quick retrieval.
  The storage and retrieval portion of the FedRegistry is done in process through
  elixir's `Registry` module for speed and its ability to monitor for terminated processes.
  Dropped connections will be caught by `Registry` and deleted. Since the next
  message will initiate a new connection there is no reason to try and reconnect at that point.
  Normally outside modules should have no need to call or use the FedRegistry themselves.
  """
  alias Pleroma.Web.FedSockets.FedSocket
  alias Pleroma.Web.FedSockets.SocketInfo
  require Logger
  @default_rejection_duration 15 * 60 * 1000
  @rejections :fed_socket_rejections
  @doc """
  Retrieves a FedSocket from the Registry given it's origin.
  The origin is expected to be a string identifying the endpoint "example.com" or "example2.com:8080"
  Will return:
    * {:ok, fed_socket} for working FedSockets
    * {:error, :rejected} for origins that have been tried and refused within the rejection duration interval
    * {:error, some_reason} usually :missing for unknown origins
  """
  def get_fed_socket(origin) do
    case get_registry_data(origin) do
      {:error, reason} ->
        {:error, reason}
      {:ok, %{state: :connected} = socket_info} ->
        {:ok, socket_info}
    end
  end
  @doc """
  Adds a connected FedSocket to the Registry.
  Always returns {:ok, fed_socket}
  """
  def add_fed_socket(origin, pid \\ nil) do
    origin
    |> SocketInfo.build(pid)
    |> SocketInfo.connect()
    |> add_socket_info
  end
  defp add_socket_info(%{origin: origin, state: :connected} = socket_info) do
    case Registry.register(FedSockets.Registry, origin, socket_info) do
      {:ok, _owner} ->
        clear_prior_rejection(origin)
        Logger.debug("fedsocket added: #{inspect(origin)}")
        {:ok, socket_info}
      {:error, {:already_registered, _pid}} ->
        FedSocket.close(socket_info)
        existing_socket_info = Registry.lookup(FedSockets.Registry, origin)
        {:ok, existing_socket_info}
      _ ->
        {:error, :error_adding_socket}
    end
  end
  @doc """
  Mark this origin as having rejected a connection attempt.
  This will keep it from getting additional connection attempts
  for a period of time specified in the config.
  Always returns {:ok, new_reg_data}
  """
  def set_host_rejected(uri) do
    new_reg_data =
      uri
      |> SocketInfo.origin()
      |> get_or_create_registry_data()
      |> set_to_rejected()
      |> save_registry_data()
    {:ok, new_reg_data}
  end
  @doc """
  Retrieves the FedRegistryData from the Registry given it's origin.
  The origin is expected to be a string identifying the endpoint "example.com" or "example2.com:8080"
  Will return:
    * {:ok, fed_registry_data} for known origins
    * {:error, :missing} for uniknown origins
    * {:error, :cache_error} indicating some low level runtime issues
  """
  def get_registry_data(origin) do
    case Registry.lookup(FedSockets.Registry, origin) do
      [] ->
        if is_rejected?(origin) do
          Logger.debug("previously rejected fedsocket requested")
          {:error, :rejected}
        else
          {:error, :missing}
        end
      [{_pid, %{state: :connected} = socket_info}] ->
        {:ok, socket_info}
      _ ->
        {:error, :cache_error}
    end
  end
  @doc """
  Retrieves a map of all sockets from the Registry. The keys are the origins and the values are the corresponding SocketInfo
  """
  def list_all do
    (list_all_connected() ++ list_all_rejected())
    |> Enum.into(%{})
  end
  defp list_all_connected do
    FedSockets.Registry
    |> Registry.select([{{:"$1", :_, :"$3"}, [], [{{:"$1", :"$3"}}]}])
  end
  defp list_all_rejected do
    {:ok, keys} = Cachex.keys(@rejections)
    {:ok, registry_data} =
      Cachex.execute(@rejections, fn worker ->
        Enum.map(keys, fn k -> {k, Cachex.get!(worker, k)} end)
      end)
    registry_data
  end
  defp clear_prior_rejection(origin),
    do: Cachex.del(@rejections, origin)
  defp is_rejected?(origin) do
    case Cachex.get(@rejections, origin) do
      {:ok, nil} ->
        false
      {:ok, _} ->
        true
    end
  end
  defp get_or_create_registry_data(origin) do
    case get_registry_data(origin) do
      {:error, :missing} ->
        %SocketInfo{origin: origin}
      {:ok, socket_info} ->
        socket_info
    end
  end
  defp save_registry_data(%SocketInfo{origin: origin, state: :connected} = socket_info) do
    {:ok, true} = Registry.update_value(FedSockets.Registry, origin, fn _ -> socket_info end)
    socket_info
  end
  defp save_registry_data(%SocketInfo{origin: origin, state: :rejected} = socket_info) do
    rejection_expiration =
      Pleroma.Config.get([:fed_sockets, :rejection_duration], @default_rejection_duration)
    {:ok, true} = Cachex.put(@rejections, origin, socket_info, ttl: rejection_expiration)
    socket_info
  end
  defp set_to_rejected(%SocketInfo{} = socket_info),
    do: %SocketInfo{socket_info | state: :rejected}
 end
--- a/lib/pleroma/web/fed_sockets/fed_socket.ex
+++ b/lib/pleroma/web/fed_sockets/fed_socket.ex
@ -0,0 +1,137 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.FedSockets.FedSocket do
  @moduledoc """
  The FedSocket module abstracts the actions to be taken taken on connections regardless of
  whether the connection started as inbound or outbound.
  Normally outside modules will have no need to call the FedSocket module directly.
  """
  alias Pleroma.Object
  alias Pleroma.Object.Containment
  alias Pleroma.User
  alias Pleroma.Web.ActivityPub.ObjectView
  alias Pleroma.Web.ActivityPub.UserView
  alias Pleroma.Web.ActivityPub.Visibility
  alias Pleroma.Web.FedSockets.FetchRegistry
  alias Pleroma.Web.FedSockets.IngesterWorker
  alias Pleroma.Web.FedSockets.OutgoingHandler
  alias Pleroma.Web.FedSockets.SocketInfo
  require Logger
  @shake "61dd18f7-f1e6-49a4-939a-a749fcdc1103"
  def connect_to_host(uri) do
    case OutgoingHandler.start_link(uri) do
      {:ok, pid} ->
        {:ok, pid}
      error ->
        {:error, error}
    end
  end
  def close(%SocketInfo{pid: socket_pid}),
    do: Process.send(socket_pid, :close, [])
  def publish(%SocketInfo{pid: socket_pid}, json) do
    %{action: :publish, data: json}
    |> Jason.encode!()
    |> send_packet(socket_pid)
  end
  def fetch(%SocketInfo{pid: socket_pid}, id) do
    fetch_uuid = FetchRegistry.register_fetch(id)
    %{action: :fetch, data: id, uuid: fetch_uuid}
    |> Jason.encode!()
    |> send_packet(socket_pid)
    wait_for_fetch_to_return(fetch_uuid, 0)
  end
  def receive_package(%SocketInfo{} = fed_socket, json) do
    json
    |> Jason.decode!()
    |> process_package(fed_socket)
  end
  defp wait_for_fetch_to_return(uuid, cntr) do
    case FetchRegistry.check_fetch(uuid) do
      {:error, :waiting} ->
        Process.sleep(:math.pow(cntr, 3) |> Kernel.trunc())
        wait_for_fetch_to_return(uuid, cntr + 1)
      {:error, :missing} ->
        Logger.error("FedSocket fetch timed out - #{inspect(uuid)}")
        {:error, :timeout}
      {:ok, _fr} ->
        FetchRegistry.pop_fetch(uuid)
    end
  end
  defp process_package(%{"action" => "publish", "data" => data}, %{origin: origin} = _fed_socket) do
    if Containment.contain_origin(origin, data) do
      IngesterWorker.enqueue("ingest", %{"object" => data})
    end
    {:reply, %{"action" => "publish_reply", "status" => "processed"}}
  end
  defp process_package(%{"action" => "fetch_reply", "uuid" => uuid, "data" => data}, _fed_socket) do
    FetchRegistry.register_fetch_received(uuid, data)
    {:noreply, nil}
  end
  defp process_package(%{"action" => "fetch", "uuid" => uuid, "data" => ap_id}, _fed_socket) do
    {:ok, data} = render_fetched_data(ap_id, uuid)
    {:reply, data}
  end
  defp process_package(%{"action" => "publish_reply"}, _fed_socket) do
    {:noreply, nil}
  end
  defp process_package(other, _fed_socket) do
    Logger.warn("unknown json packages received #{inspect(other)}")
    {:noreply, nil}
  end
  defp render_fetched_data(ap_id, uuid) do
    {:ok,
     %{
       "action" => "fetch_reply",
       "status" => "processed",
       "uuid" => uuid,
       "data" => represent_item(ap_id)
     }}
  end
  defp represent_item(ap_id) do
    case User.get_by_ap_id(ap_id) do
      nil ->
        object = Object.get_cached_by_ap_id(ap_id)
        if Visibility.is_public?(object) do
          Phoenix.View.render_to_string(ObjectView, "object.json", object: object)
        else
          nil
        end
      user ->
        Phoenix.View.render_to_string(UserView, "user.json", user: user)
    end
  end
  defp send_packet(data, socket_pid) do
    Process.send(socket_pid, {:send, data}, [])
  end
  def shake, do: @shake
 end
--- a/lib/pleroma/web/fed_sockets/fed_sockets.ex
+++ b/lib/pleroma/web/fed_sockets/fed_sockets.ex
@ -0,0 +1,185 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.FedSockets do
  @moduledoc """
  This documents the FedSockets framework. A framework for federating
  ActivityPub objects between servers via persistant WebSocket connections.
  FedSockets allow servers to authenticate on first contact and maintain that
  connection, eliminating the need to authenticate every time data needs to be shared.
  ## Protocol
  FedSockets currently support 2 types of data transfer:
    * `publish` method which doesn't require a response
    * `fetch` method requires a response be sent
    ### Publish
    The publish operation sends a json encoded map of the shape:
      %{action: :publish, data: json}
    and accepts (but does not require) a reply of form:
      %{"action" => "publish_reply"}
    The outgoing params represent
      * data: ActivityPub object encoded into json
    ### Fetch
    The fetch operation sends a json encoded map of the shape:
      %{action: :fetch, data: id, uuid: fetch_uuid}
    and requires a reply of form:
      %{"action" => "fetch_reply", "uuid" => uuid, "data" => data}
    The outgoing params represent
      * id: an ActivityPub object URI
      * uuid: a unique uuid generated by the sender
    The reply params represent
      * data: an ActivityPub object encoded into json
      * uuid: the uuid sent along with the fetch request
  ## Examples
  Clients of FedSocket transfers shouldn't need to use any of the functions outside of this module.
  A typical publish operation can be performed through the following code, and a fetch operation in a similar manner.
    case FedSockets.get_or_create_fed_socket(inbox) do
      {:ok, fedsocket} ->
        FedSockets.publish(fedsocket, json)
      _ ->
        alternative_publish(inbox, actor, json, params)
    end
  ## Configuration
  FedSockets have the following config settings
  config :pleroma, :fed_sockets,
  enabled: true,
  ping_interval: :timer.seconds(15),
  connection_duration: :timer.hours(1),
  rejection_duration: :timer.hours(1),
  fed_socket_fetches: [
    default: 12_000,
    interval: 3_000,
    lazy: false
  ]
    * enabled - turn FedSockets on or off with this flag. Can be toggled at runtime.
    * connection_duration - How long a FedSocket can sit idle before it's culled.
    * rejection_duration - After failing to make a FedSocket connection a host will be excluded
    from further connections for this amount of time
    * fed_socket_fetches - Use these parameters to pass options to the Cachex queue backing the FetchRegistry
    * fed_socket_rejections - Use these parameters to pass options to the Cachex queue backing the FedRegistry
    Cachex options are
      * default: the minimum amount of time a fetch can wait before it times out.
      * interval: the interval between checks for timed out entries. This plus the default represent the maximum time allowed
      * lazy: leave at false for consistant and fast lookups, set to true for stricter timeout enforcement
  """
  require Logger
  alias Pleroma.Web.FedSockets.FedRegistry
  alias Pleroma.Web.FedSockets.FedSocket
  alias Pleroma.Web.FedSockets.SocketInfo
  @doc """
  returns a FedSocket for the given origin. Will reuse an existing one or create a new one.
  address is expected to be a fully formed URL such as:
  "http://www.example.com" or "http://www.example.com:8080"
  It can and usually does include additional path parameters,
  but these are ignored as the FedSockets are organized by host and port info alone.
  """
  def get_or_create_fed_socket(address) do
    with {:cache, {:error, :missing}} <- {:cache, get_fed_socket(address)},
         {:connect, {:ok, _pid}} <- {:connect, FedSocket.connect_to_host(address)},
         {:cache, {:ok, fed_socket}} <- {:cache, get_fed_socket(address)} do
      Logger.debug("fedsocket created for - #{inspect(address)}")
      {:ok, fed_socket}
    else
      {:cache, {:ok, socket}} ->
        Logger.debug("fedsocket found in cache - #{inspect(address)}")
        {:ok, socket}
      {:cache, {:error, :rejected} = e} ->
        e
      {:connect, {:error, _host}} ->
        Logger.debug("set host rejected for - #{inspect(address)}")
        FedRegistry.set_host_rejected(address)
        {:error, :rejected}
      {_, {:error, :disabled}} ->
        {:error, :disabled}
      {_, {:error, reason}} ->
        Logger.warn("get_or_create_fed_socket error - #{inspect(reason)}")
        {:error, reason}
    end
  end
  @doc """
  returns a FedSocket for the given origin. Will not create a new FedSocket if one does not exist.
  address is expected to be a fully formed URL such as:
    "http://www.example.com" or "http://www.example.com:8080"
  """
  def get_fed_socket(address) do
    origin = SocketInfo.origin(address)
    with {:config, true} <- {:config, Pleroma.Config.get([:fed_sockets, :enabled], false)},
         {:ok, socket} <- FedRegistry.get_fed_socket(origin) do
      {:ok, socket}
    else
      {:config, _} ->
        {:error, :disabled}
      {:error, :rejected} ->
        Logger.debug("FedSocket previously rejected - #{inspect(origin)}")
        {:error, :rejected}
      {:error, reason} ->
        {:error, reason}
    end
  end
  @doc """
  Sends the supplied data via the publish protocol.
  It will not block waiting for a reply.
  Returns :ok but this is not an indication of a successful transfer.
  the data is expected to be JSON encoded binary data.
  """
  def publish(%SocketInfo{} = fed_socket, json) do
    FedSocket.publish(fed_socket, json)
  end
  @doc """
  Sends the supplied data via the fetch protocol.
  It will block waiting for a reply or timeout.
  Returns {:ok, object} where object is the requested object (or nil)
          {:error, :timeout} in the event the message was not responded to
  the id is expected to be the URI of an ActivityPub object.
  """
  def fetch(%SocketInfo{} = fed_socket, id) do
    FedSocket.fetch(fed_socket, id)
  end
  @doc """
  Disconnect all and restart FedSockets.
  This is mainly used in development and testing but could be useful in production.
  """
  def reset do
    FedRegistry
    |> Process.whereis()
    |> Process.exit(:testing)
  end
  def uri_for_origin(origin),
    do: "ws://#{origin}/api/fedsocket/v1"
 end
--- a/lib/pleroma/web/fed_sockets/fetch_registry.ex
+++ b/lib/pleroma/web/fed_sockets/fetch_registry.ex
@ -0,0 +1,151 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.FedSockets.FetchRegistry do
  @moduledoc """
  The FetchRegistry acts as a broker for fetch requests and return values.
  This allows calling processes to block while waiting for a reply.
  It doesn't impose it's own process instead using `Cachex` to handle fetches in process, allowing
  multi threaded processes to avoid bottlenecking.
  Normally outside modules will have no need to call or use the FetchRegistry themselves.
  The `Cachex` parameters can be controlled from the config. Since exact timeout intervals
  aren't necessary the following settings are used by default:
  config :pleroma, :fed_sockets,
    fed_socket_fetches: [
      default: 12_000,
      interval: 3_000,
      lazy: false
    ]
  """
  defmodule FetchRegistryData do
    defstruct uuid: nil,
              sent_json: nil,
              received_json: nil,
              sent_at: nil,
              received_at: nil
  end
  alias Ecto.UUID
  require Logger
  @fetches :fed_socket_fetches
  @doc """
  Registers a json request wth the FetchRegistry and returns the identifying UUID.
  """
  def register_fetch(json) do
    %FetchRegistryData{uuid: uuid} =
      json
      |> new_registry_data
      |> save_registry_data
    uuid
  end
  @doc """
  Reports on the status of a Fetch given the identifying UUID.
  Will return
    * {:ok, fetched_object} if a fetch has completed
    * {:error, :waiting} if a fetch is still pending
    * {:error, other_error} usually :missing to indicate a fetch that has timed out
  """
  def check_fetch(uuid) do
    case get_registry_data(uuid) do
      {:ok, %FetchRegistryData{received_at: nil}} ->
        {:error, :waiting}
      {:ok, %FetchRegistryData{} = reg_data} ->
        {:ok, reg_data}
      e ->
        e
    end
  end
  @doc """
  Retrieves the response to a fetch given the identifying UUID.
  The completed fetch will be deleted from the FetchRegistry
  Will return
    * {:ok, fetched_object} if a fetch has completed
    * {:error, :waiting} if a fetch is still pending
    * {:error, other_error} usually :missing to indicate a fetch that has timed out
  """
  def pop_fetch(uuid) do
    case check_fetch(uuid) do
      {:ok, %FetchRegistryData{received_json: received_json}} ->
        delete_registry_data(uuid)
        {:ok, received_json}
      e ->
        e
    end
  end
  @doc """
  This is called to register a fetch has returned.
  It expects the result data along with the UUID that was sent in the request
  Will return the fetched object or :error
  """
  def register_fetch_received(uuid, data) do
    case get_registry_data(uuid) do
      {:ok, %FetchRegistryData{received_at: nil} = reg_data} ->
        reg_data
        |> set_fetch_received(data)
        |> save_registry_data()
      {:ok, %FetchRegistryData{} = reg_data} ->
        Logger.warn("tried to add fetched data twice - #{uuid}")
        reg_data
      {:error, _} ->
        Logger.warn("Error adding fetch to registry - #{uuid}")
        :error
    end
  end
  defp new_registry_data(json) do
    %FetchRegistryData{
      uuid: UUID.generate(),
      sent_json: json,
      sent_at: :erlang.monotonic_time(:millisecond)
    }
  end
  defp get_registry_data(origin) do
    case Cachex.get(@fetches, origin) do
      {:ok, nil} ->
        {:error, :missing}
      {:ok, reg_data} ->
        {:ok, reg_data}
      _ ->
        {:error, :cache_error}
    end
  end
  defp set_fetch_received(%FetchRegistryData{} = reg_data, data),
    do: %FetchRegistryData{
      reg_data
      | received_at: :erlang.monotonic_time(:millisecond),
        received_json: data
    }
  defp save_registry_data(%FetchRegistryData{uuid: uuid} = reg_data) do
    {:ok, true} = Cachex.put(@fetches, uuid, reg_data)
    reg_data
  end
  defp delete_registry_data(origin),
    do: {:ok, true} = Cachex.del(@fetches, origin)
 end
--- a/lib/pleroma/web/fed_sockets/incoming_handler.ex
+++ b/lib/pleroma/web/fed_sockets/incoming_handler.ex
@ -0,0 +1,88 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.FedSockets.IncomingHandler do
  require Logger
  alias Pleroma.Web.FedSockets.FedRegistry
  alias Pleroma.Web.FedSockets.FedSocket
  alias Pleroma.Web.FedSockets.SocketInfo
  import HTTPSignatures, only: [validate_conn: 1, split_signature: 1]
  @behaviour :cowboy_websocket
  def init(req, state) do
    shake = FedSocket.shake()
    with true <- Pleroma.Config.get([:fed_sockets, :enabled]),
         sec_protocol <- :cowboy_req.header("sec-websocket-protocol", req, nil),
         headers = %{"(request-target)" => ^shake} <- :cowboy_req.headers(req),
         true <- validate_conn(%{req_headers: headers}),
         %{"keyId" => origin} <- split_signature(headers["signature"]) do
      req =
        if is_nil(sec_protocol) do
          req
        else
          :cowboy_req.set_resp_header("sec-websocket-protocol", sec_protocol, req)
        end
      {:cowboy_websocket, req, %{origin: origin}, %{}}
    else
      _ ->
        {:ok, req, state}
    end
  end
  def websocket_init(%{origin: origin}) do
    case FedRegistry.add_fed_socket(origin) do
      {:ok, socket_info} ->
        {:ok, socket_info}
      e ->
        Logger.error("FedSocket websocket_init failed - #{inspect(e)}")
        {:error, inspect(e)}
    end
  end
  # Use the ping to  check if the connection should be expired
  def websocket_handle(:ping, socket_info) do
    if SocketInfo.expired?(socket_info) do
      {:stop, socket_info}
    else
      {:ok, socket_info, :hibernate}
    end
  end
  def websocket_handle({:text, data}, socket_info) do
    socket_info = SocketInfo.touch(socket_info)
    case FedSocket.receive_package(socket_info, data) do
      {:noreply, _} ->
        {:ok, socket_info}
      {:reply, reply} ->
        {:reply, {:text, Jason.encode!(reply)}, socket_info}
      {:error, reason} ->
        Logger.error("incoming error - receive_package: #{inspect(reason)}")
        {:ok, socket_info}
    end
  end
  def websocket_info({:send, message}, socket_info) do
    socket_info = SocketInfo.touch(socket_info)
    {:reply, {:text, message}, socket_info}
  end
  def websocket_info(:close, state) do
    {:stop, state}
  end
  def websocket_info(message, state) do
    Logger.debug("#{__MODULE__} unknown message #{inspect(message)}")
    {:ok, state}
  end
 end
--- a/lib/pleroma/web/fed_sockets/ingester_worker.ex
+++ b/lib/pleroma/web/fed_sockets/ingester_worker.ex
@ -0,0 +1,33 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.FedSockets.IngesterWorker do
  use Pleroma.Workers.WorkerHelper, queue: "ingestion_queue"
  require Logger
  alias Pleroma.Web.Federator
  @impl Oban.Worker
  def perform(%Job{args: %{"op" => "ingest", "object" => ingestee}}) do
    try do
      ingestee
      |> Jason.decode!()
      |> do_ingestion()
    rescue
      e ->
        Logger.error("IngesterWorker error - #{inspect(e)}")
        e
    end
  end
  defp do_ingestion(params) do
    case Federator.incoming_ap_doc(params) do
      {:error, reason} ->
        {:error, reason}
      {:ok, object} ->
        {:ok, object}
    end
  end
 end
--- a/lib/pleroma/web/fed_sockets/outgoing_handler.ex
+++ b/lib/pleroma/web/fed_sockets/outgoing_handler.ex
@ -0,0 +1,151 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.FedSockets.OutgoingHandler do
  use GenServer
  require Logger
  alias Pleroma.Application
  alias Pleroma.Web.ActivityPub.InternalFetchActor
  alias Pleroma.Web.FedSockets
  alias Pleroma.Web.FedSockets.FedRegistry
  alias Pleroma.Web.FedSockets.FedSocket
  alias Pleroma.Web.FedSockets.SocketInfo
  def start_link(uri) do
    GenServer.start_link(__MODULE__, %{uri: uri})
  end
  def init(%{uri: uri}) do
    case initiate_connection(uri) do
      {:ok, ws_origin, conn_pid} ->
        FedRegistry.add_fed_socket(ws_origin, conn_pid)
      {:error, reason} ->
        Logger.debug("Outgoing connection failed - #{inspect(reason)}")
        :ignore
    end
  end
  def handle_info({:gun_ws, conn_pid, _ref, {:text, data}}, socket_info) do
    socket_info = SocketInfo.touch(socket_info)
    case FedSocket.receive_package(socket_info, data) do
      {:noreply, _} ->
        {:noreply, socket_info}
      {:reply, reply} ->
        :gun.ws_send(conn_pid, {:text, Jason.encode!(reply)})
        {:noreply, socket_info}
      {:error, reason} ->
        Logger.error("incoming error - receive_package: #{inspect(reason)}")
        {:noreply, socket_info}
    end
  end
  def handle_info(:close, state) do
    Logger.debug("Sending close frame !!!!!!!")
    {:close, state}
  end
  def handle_info({:gun_down, _pid, _prot, :closed, _}, state) do
    {:stop, :normal, state}
  end
  def handle_info({:send, data}, %{conn_pid: conn_pid} = socket_info) do
    socket_info = SocketInfo.touch(socket_info)
    :gun.ws_send(conn_pid, {:text, data})
    {:noreply, socket_info}
  end
  def handle_info({:gun_ws, _, _, :pong}, state) do
    {:noreply, state, :hibernate}
  end
  def handle_info(msg, state) do
    Logger.debug("#{__MODULE__} unhandled event #{inspect(msg)}")
    {:noreply, state}
  end
  def terminate(reason, state) do
    Logger.debug(
      "#{__MODULE__} terminating outgoing connection for #{inspect(state)} for #{inspect(reason)}"
    )
    {:ok, state}
  end
  def initiate_connection(uri) do
    ws_uri =
      uri
      |> SocketInfo.origin()
      |> FedSockets.uri_for_origin()
    %{host: host, port: port, path: path} = URI.parse(ws_uri)
    with {:ok, conn_pid} <- :gun.open(to_charlist(host), port, %{protocols: [:http]}),
         {:ok, _} <- :gun.await_up(conn_pid),
         reference <-
           :gun.get(conn_pid, to_charlist(path), [
             {'user-agent', to_charlist(Application.user_agent())}
           ]),
         {:response, :fin, 204, _} <- :gun.await(conn_pid, reference),
         headers <- build_headers(uri),
         ref <- :gun.ws_upgrade(conn_pid, to_charlist(path), headers, %{silence_pings: false}) do
      receive do
        {:gun_upgrade, ^conn_pid, ^ref, [<<"websocket">>], _} ->
          {:ok, ws_uri, conn_pid}
      after
        15_000 ->
          Logger.debug("Fedsocket timeout connecting to #{inspect(uri)}")
          {:error, :timeout}
      end
    else
      {:response, :nofin, 404, _} ->
        {:error, :fedsockets_not_supported}
      e ->
        Logger.debug("Fedsocket error connecting to #{inspect(uri)}")
        {:error, e}
    end
  end
  defp build_headers(uri) do
    host_for_sig = uri |> URI.parse() |> host_signature()
    shake = FedSocket.shake()
    digest = "SHA-256=" <> (:crypto.hash(:sha256, shake) |> Base.encode64())
    date = Pleroma.Signature.signed_date()
    shake_size = byte_size(shake)
    signature_opts = %{
      "(request-target)": shake,
      "content-length": to_charlist("#{shake_size}"),
      date: date,
      digest: digest,
      host: host_for_sig
    }
    signature = Pleroma.Signature.sign(InternalFetchActor.get_actor(), signature_opts)
    [
      {'signature', to_charlist(signature)},
      {'date', date},
      {'digest', to_charlist(digest)},
      {'content-length', to_charlist("#{shake_size}")},
      {to_charlist("(request-target)"), to_charlist(shake)},
      {'user-agent', to_charlist(Application.user_agent())}
    ]
  end
  defp host_signature(%{host: host, scheme: scheme, port: port}) do
    if port == URI.default_port(scheme) do
      host
    else
      "#{host}:#{port}"
    end
  end
 end
--- a/lib/pleroma/web/fed_sockets/socket_info.ex
+++ b/lib/pleroma/web/fed_sockets/socket_info.ex
@ -0,0 +1,52 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.FedSockets.SocketInfo do
  defstruct origin: nil,
            pid: nil,
            conn_pid: nil,
            state: :default,
            connected_until: nil
  alias Pleroma.Web.FedSockets.SocketInfo
  @default_connection_duration 15 * 60 * 1000
  def build(uri, conn_pid \\ nil) do
    uri
    |> build_origin()
    |> build_pids(conn_pid)
    |> touch()
  end
  def touch(%SocketInfo{} = socket_info),
    do: %{socket_info | connected_until: new_ttl()}
  def connect(%SocketInfo{} = socket_info),
    do: %{socket_info | state: :connected}
  def expired?(%{connected_until: connected_until}),
    do: connected_until < :erlang.monotonic_time(:millisecond)
  def origin(uri),
    do: build_origin(uri).origin
  defp build_pids(socket_info, conn_pid),
    do: struct(socket_info, pid: self(), conn_pid: conn_pid)
  defp build_origin(uri) when is_binary(uri),
    do: uri |> URI.parse() |> build_origin
  defp build_origin(%{host: host, port: nil, scheme: scheme}),
    do: build_origin(%{host: host, port: URI.default_port(scheme)})
  defp build_origin(%{host: host, port: port}),
    do: %SocketInfo{origin: "#{host}:#{port}"}
  defp new_ttl do
    connection_duration =
      Pleroma.Config.get([:fed_sockets, :connection_duration], @default_connection_duration)
    :erlang.monotonic_time(:millisecond) + connection_duration
  end
 end
--- a/lib/pleroma/web/fed_sockets/supervisor.ex
+++ b/lib/pleroma/web/fed_sockets/supervisor.ex
@ -0,0 +1,59 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.FedSockets.Supervisor do
  use Supervisor
  import Cachex.Spec
  def start_link(opts) do
    Supervisor.start_link(__MODULE__, opts, name: __MODULE__)
  end
  def init(args) do
    children = [
      build_cache(:fed_socket_fetches, args),
      build_cache(:fed_socket_rejections, args),
      {Registry, keys: :unique, name: FedSockets.Registry, meta: [rejected: %{}]}
    ]
    opts = [strategy: :one_for_all, name: Pleroma.Web.Streamer.Supervisor]
    Supervisor.init(children, opts)
  end
  defp build_cache(name, args) do
    opts = get_opts(name, args)
    %{
      id: String.to_atom("#{name}_cache"),
      start: {Cachex, :start_link, [name, opts]},
      type: :worker
    }
  end
  defp get_opts(cache_name, args)
       when cache_name in [:fed_socket_fetches, :fed_socket_rejections] do
    default = get_opts_or_config(args, cache_name, :default, 15_000)
    interval = get_opts_or_config(args, cache_name, :interval, 3_000)
    lazy = get_opts_or_config(args, cache_name, :lazy, false)
    [expiration: expiration(default: default, interval: interval, lazy: lazy)]
  end
  defp get_opts(name, args) do
    Keyword.get(args, name, [])
  end
  defp get_opts_or_config(args, name, key, default) do
    args
    |> Keyword.get(name, [])
    |> Keyword.get(key)
    |> case do
      nil ->
        Pleroma.Config.get([:fed_sockets, name, key], default)
      value ->
        value
    end
  end
 end
--- a/lib/pleroma/web/federator/federator.ex
+++ b/lib/pleroma/web/federator/federator.ex
@ -66,14 +66,17 @@ def perform(:publish, activity) do
  def perform(:incoming_ap_doc, params) do
    Logger.debug("Handling incoming AP activity")
-    params = Utils.normalize_params(params)
+    actor =
      params
      |> Map.get("actor")
      |> Utils.get_ap_id()
    # NOTE: we use the actor ID to do the containment, this is fine because an
    # actor shouldn't be acting on objects outside their own AP server.
-    with {:ok, _user} <- ap_enabled_actor(params["actor"]),
+    with {_, {:ok, _user}} <- {:actor, ap_enabled_actor(actor)},
         nil <- Activity.normalize(params["id"]),
         {_, :ok} <-
-           {:correct_origin?, Containment.contain_origin_from_id(params["actor"], params)},
+           {:correct_origin?, Containment.contain_origin_from_id(actor, params)},
         {:ok, activity} <- Transmogrifier.handle_incoming(params) do
      {:ok, activity}
    else
@ -85,10 +88,13 @@ def perform(:incoming_ap_doc, params) do
        Logger.debug("Already had #{params["id"]}")
        {:error, :already_present}
      {:actor, e} ->
        Logger.debug("Unhandled actor #{actor}, #{inspect(e)}")
        {:error, e}
      e ->
        # Just drop those for now
-        Logger.debug("Unhandled activity")
+        Logger.debug(fn -> "Unhandled activity\n" <> Jason.encode!(params, pretty: true) end)
        Logger.debug(Jason.encode!(params, pretty: true))
        {:error, e}
    end
  end
--- a/lib/pleroma/web/instance_document.ex
+++ b/lib/pleroma/web/instance_document.ex
@ -0,0 +1,62 @@
 # Pleroma: A lightweight social networking server
 # Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 defmodule Pleroma.Web.InstanceDocument do
  alias Pleroma.Config
  alias Pleroma.Web.Endpoint
  @instance_documents %{
    "terms-of-service" => "/static/terms-of-service.html",
    "instance-panel" => "/instance/panel.html"
  }
  @spec get(String.t()) :: {:ok, String.t()} | {:error, atom()}
  def get(document_name) do
    case Map.fetch(@instance_documents, document_name) do
      {:ok, path} -> {:ok, path}
      _ -> {:error, :not_found}
    end
  end
  @spec put(String.t(), String.t()) :: {:ok, String.t()} | {:error, atom()}
  def put(document_name, origin_path) do
    with {_, {:ok, destination_path}} <-
           {:instance_document, Map.fetch(@instance_documents, document_name)},
         :ok <- put_file(origin_path, destination_path) do
      {:ok, Path.join(Endpoint.url(), destination_path)}
    else
      {:instance_document, :error} -> {:error, :not_found}
      error -> error
    end
  end
  @spec delete(String.t()) :: :ok | {:error, atom()}
  def delete(document_name) do
    with {_, {:ok, path}} <- {:instance_document, Map.fetch(@instance_documents, document_name)},
         instance_static_dir_path <- instance_static_dir(path),
         :ok <- File.rm(instance_static_dir_path) do
      :ok
    else
      {:instance_document, :error} -> {:error, :not_found}
      {:error, :enoent} -> {:error, :not_found}
      error -> error
    end
  end
  defp put_file(origin_path, destination_path) do
    with destination <- instance_static_dir(destination_path),
         {_, :ok} <- {:mkdir_p, File.mkdir_p(Path.dirname(destination))},
         {_, {:ok, _}} <- {:copy, File.copy(origin_path, destination)} do
      :ok
    else
      {error, _} -> {:error, error}
    end
  end
  defp instance_static_dir(filename) do
    [:instance, :static_dir]
    |> Config.get!()
    |> Path.join(filename)
  end
 end
--- a/lib/pleroma/web/mastodon_api/controllers/auth_controller.ex
+++ b/lib/pleroma/web/mastodon_api/controllers/auth_controller.ex
@ -5,6 +5,8 @@
 defmodule Pleroma.Web.MastodonAPI.AuthController do
  use Pleroma.Web, :controller
  import Pleroma.Web.ControllerHelper, only: [json_response: 3]
  alias Pleroma.User
  alias Pleroma.Web.OAuth.App
  alias Pleroma.Web.OAuth.Authorization
@ -61,9 +63,7 @@ def password_reset(conn, params) do
    TwitterAPI.password_reset(nickname_or_email)
-    conn
+    json_response(conn, :no_content, "")
    |> put_status(:no_content)
    |> json("")
  end
  defp local_mastodon_root_path(conn) do
--- a/Show more
+++ b/Show more