Merge branch 'develop' of https://git.pleroma.social/pleroma/pleroma into develop

.gitlab/issue_templates/Bug.md

@@ -14,7 +14,7 @@
 * Pleroma version (could be found in the "Version" tab of settings in Pleroma-FE): 
 * Elixir version (`elixir -v` for from source installations, N/A for OTP):
 * Operating system:
-* PostgreSQL version (`postgres -V`):
+* PostgreSQL version (`psql -V`):
 
 
 ### Bug description

CHANGELOG.md

@@ -8,17 +8,37 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Changed
 - **Breaking:** Elixir >=1.9 is now required (was >= 1.8)
 - In Conversations, return only direct messages as `last_status`
+- Using the `only_media` filter on timelines will now exclude reblog media
 - MFR policy to set global expiration for all local Create activities
 - OGP rich media parser merged with TwitterCard
+- Configuration: `:instance, rewrite_policy` moved to `:mrf, policies`, `:instance, :mrf_transparency` moved to `:mrf, :transparency`, `:instance, :mrf_transparency_exclusions` moved to `:mrf, :transparency_exclusions`. Old config namespace is deprecated.
+
 <details>
   <summary>API Changes</summary>
+
+- **Breaking:** Pleroma API: The routes to update avatar, banner and background have been removed.
+- **Breaking:** Image description length is limited now.
 - **Breaking:** Emoji API: changed methods and renamed routes.
+- MastodonAPI: Allow removal of avatar, banner and background.
+- Streaming: Repeats of a user's posts will no longer be pushed to the user's stream.
+- Mastodon API: Added `pleroma.metadata.fields_limits` to /api/v1/instance
+- Mastodon API: On deletion, returns the original post text.
+- Mastodon API: Add `pleroma.unread_count` to the Marker entity.
+</details>
+
+<details>
+  <summary>Admin API Changes</summary>
+
+- Status visibility stats: now can return stats per instance.
+
+- Mix task to refresh counter cache (`mix pleroma.refresh_counter_cache`)
 </details>
 
 ### Removed
 - **Breaking:** removed `with_move` parameter from notifications timeline.
 
 ### Added
+
 - Chats: Added support for federated chats. For details, see the docs.
 - ActivityPub: Added support for existing AP ids for instances migrated from Mastodon.
 - Instance: Add `background_image` to configuration and `/api/v1/instance`
@@ -34,12 +54,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Notifications: Added `follow_request` notification type.
 - Added `:reject_deletes` group to SimplePolicy
 - MRF (`EmojiStealPolicy`): New MRF Policy which allows to automatically download emojis from remote instances
+- Support pagination in emoji packs API (for packs and for files in pack)
+- Support for viewing instances favicons next to posts and accounts
+
 <details>
   <summary>API Changes</summary>
+- Mastodon API: Add pleroma.parents_visible field to statuses.
 - Mastodon API: Extended `/api/v1/instance`.
 - Mastodon API: Support for `include_types` in `/api/v1/notifications`.
 - Mastodon API: Added `/api/v1/notifications/:id/dismiss` endpoint.
-- Mastodon API: Add support for filtering replies in public and home timelines
+- Mastodon API: Add support for filtering replies in public and home timelines.
+- Mastodon API: Support for `bot` field in `/api/v1/accounts/update_credentials`.
+- Mastodon API: Support irreversible property for filters.
+- Mastodon API: Add pleroma.favicon field to accounts.
 - Admin API: endpoints for create/update/delete OAuth Apps.
 - Admin API: endpoint for status view.
 - OTP: Add command to reload emoji packs
@@ -53,6 +80,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Resolving Peertube accounts with Webfinger
 - `blob:` urls not being allowed by connect-src CSP
 - Mastodon API: fix `GET /api/v1/notifications` not returning the full result set
+- Rich Media Previews for Twitter links
+- Fix CSP policy generation to include remote Captcha services
 
 ## [Unreleased (patch)]
 
@@ -91,6 +120,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 2. Run database migrations (inside Pleroma directory):
   - OTP: `./bin/pleroma_ctl migrate`
   - From Source: `mix ecto.migrate`
+3. Reset status visibility counters (inside Pleroma directory):
+  - OTP: `./bin/pleroma_ctl refresh_counter_cache`
+  - From Source: `mix pleroma.refresh_counter_cache`
 
 
 ## [2.0.2] - 2020-04-08
@@ -191,7 +223,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: `pleroma.thread_muted` to the Status entity
 - Mastodon API: Mark the direct conversation as read for the author when they send a new direct message
 - Mastodon API, streaming: Add `pleroma.direct_conversation_id` to the `conversation` stream event payload.
-- Mastodon API: Add `pleroma.unread_count` to the Marker entity
 - Admin API: Render whole status in grouped reports
 - Mastodon API: User timelines will now respect blocks, unless you are getting the user timeline of somebody you blocked (which would be empty otherwise).
 - Mastodon API: Favoriting / Repeating a post multiple times will now return the identical response every time. Before, executing that action twice would return an error ("already favorited") on the second try.

README.md

@@ -34,6 +34,16 @@ Currently Pleroma is not packaged by any OS/Distros, but if you want to package
 ### Docker
 While we don’t provide docker files, other people have written very good ones. Take a look at <https://github.com/angristan/docker-pleroma> or <https://glitch.sh/sn0w/pleroma-docker>.
 
+### Compilation Troubleshooting
+If you ever encounter compilation issues during the updating of Pleroma, you can try these commands and see if they fix things:
+
+- `mix deps.clean --all`
+- `mix local.rebar`
+- `mix local.hex`
+- `rm -r _build`
+
+If you are not developing Pleroma, it is better to use the OTP release, which comes with everything precompiled.
+
 ## Documentation
 - Latest Released revision: <https://docs.pleroma.social>
 - Latest Git revision: <https://docs-develop.pleroma.social>

benchmarks/load_testing/activities.ex

@@ -24,6 +24,7 @@ defmodule Pleroma.LoadTesting.Activities do
   @visibility ~w(public private direct unlisted)
   @types [
     :simple,
+    :simple_filtered,
     :emoji,
     :mentions,
     :hell_thread,
@@ -242,6 +243,15 @@ defp insert_activity(:simple, visibility, group, users, _opts) do
     insert_local_activity(visibility, group, users, "Simple status")
   end
 
+  defp insert_activity(:simple_filtered, visibility, group, users, _opts)
+       when group in @remote_groups do
+    insert_remote_activity(visibility, group, users, "Remote status which must be filtered")
+  end
+
+  defp insert_activity(:simple_filtered, visibility, group, users, _opts) do
+    insert_local_activity(visibility, group, users, "Simple status which must be filtered")
+  end
+
   defp insert_activity(:emoji, visibility, group, users, _opts)
        when group in @remote_groups do
     insert_remote_activity(visibility, group, users, "Remote status with emoji :firefox:")

benchmarks/load_testing/fetcher.ex

@@ -32,10 +32,22 @@ defp fetch_user(user) do
     )
   end
 
+  defp create_filter(user) do
+    Pleroma.Filter.create(%Pleroma.Filter{
+      user_id: user.id,
+      phrase: "must be filtered",
+      hide: true
+    })
+  end
+
+  defp delete_filter(filter), do: Repo.delete(filter)
+
   defp fetch_timelines(user) do
     fetch_home_timeline(user)
+    fetch_home_timeline_with_filter(user)
     fetch_direct_timeline(user)
     fetch_public_timeline(user)
+    fetch_public_timeline_with_filter(user)
     fetch_public_timeline(user, :with_blocks)
     fetch_public_timeline(user, :local)
     fetch_public_timeline(user, :tag)
@@ -61,7 +73,7 @@ defp opts_for_home_timeline(user) do
     }
   end
 
-  defp fetch_home_timeline(user) do
+  defp fetch_home_timeline(user, title_end \\ "") do
     opts = opts_for_home_timeline(user)
 
     recipients = [user.ap_id | User.following(user)]
@@ -84,9 +96,11 @@ defp fetch_home_timeline(user) do
       |> Enum.reverse()
       |> List.last()
 
+    title = "home timeline " <> title_end
+
     Benchee.run(
       %{
-        "home timeline" => fn opts -> ActivityPub.fetch_activities(recipients, opts) end
+        title => fn opts -> ActivityPub.fetch_activities(recipients, opts) end
       },
       inputs: %{
         "1 page" => opts,
@@ -108,6 +122,14 @@ defp fetch_home_timeline(user) do
     )
   end
 
+  defp fetch_home_timeline_with_filter(user) do
+    {:ok, filter} = create_filter(user)
+
+    fetch_home_timeline(user, "with filters")
+
+    delete_filter(filter)
+  end
+
   defp opts_for_direct_timeline(user) do
     %{
       visibility: "direct",
@@ -210,6 +232,14 @@ defp fetch_public_timeline(user) do
     fetch_public_timeline(opts, "public timeline")
   end
 
+  defp fetch_public_timeline_with_filter(user) do
+    {:ok, filter} = create_filter(user)
+    opts = opts_for_public_timeline(user)
+
+    fetch_public_timeline(opts, "public timeline with filters")
+    delete_filter(filter)
+  end
+
   defp fetch_public_timeline(user, :local) do
     opts = opts_for_public_timeline(user, :local)
 

config/config.exs

@@ -97,6 +97,7 @@
     "dat",
     "dweb",
     "gopher",
+    "hyper",
     "ipfs",
     "ipns",
     "irc",
@@ -186,7 +187,9 @@
   notify_email: "noreply@example.com",
   description: "Pleroma: An efficient and flexible fediverse server",
   background_image: "/images/city.jpg",
+  instance_thumbnail: "/instance/thumbnail.jpeg",
   limit: 5_000,
+  description_limit: 5_000,
   chat_limit: 5_000,
   remote_limit: 100_000,
   upload_limit: 16_000_000,
@@ -209,7 +212,6 @@
     Pleroma.Web.ActivityPub.Publisher
   ],
   allow_relay: true,
-  rewrite_policy: Pleroma.Web.ActivityPub.MRF.NoOpPolicy,
   public: true,
   quarantined_instances: [],
   managed_config: true,
@@ -408,6 +410,13 @@
   ],
   whitelist: []
 
+config :pleroma, Pleroma.Web.MediaProxy.Invalidation.Http,
+  method: :purge,
+  headers: [],
+  options: []
+
+config :pleroma, Pleroma.Web.MediaProxy.Invalidation.Script, script_path: nil
+
 config :pleroma, :chat, enabled: true
 
 config :phoenix, :format_encoders, json: Jason
@@ -430,6 +439,11 @@
   ],
   unfurl_nsfw: false
 
+config :pleroma, Pleroma.Web.Preload,
+  providers: [
+    Pleroma.Web.Preload.Providers.Instance
+  ]
+
 config :pleroma, :http_security,
   enabled: true,
   sts: false,
@@ -686,6 +700,17 @@
 
 config :pleroma, Pleroma.Web.ApiSpec.CastAndValidate, strict: false
 
+config :pleroma, :mrf,
+  policies: Pleroma.Web.ActivityPub.MRF.NoOpPolicy,
+  transparency: true,
+  transparency_exclusions: []
+
+config :tzdata, :http_client, Pleroma.HTTP.Tzdata
+
+config :ex_aws, http_client: Pleroma.HTTP.ExAws
+
+config :pleroma, :instances_favicons, enabled: false
+
 # Import environment specific config. This must remain at the bottom
 # of this file so it overrides the configuration defined above.
 import_config "#{Mix.env()}.exs"

config/description.exs

@@ -40,12 +40,13 @@
         key: :link_name,
         type: :boolean,
         description:
-          "If enabled, a name parameter will be added to the url of the upload. For example `https://instance.tld/media/imagehash.png?name=realname.png`."
+          "If enabled, a name parameter will be added to the URL of the upload. For example `https://instance.tld/media/imagehash.png?name=realname.png`."
       },
       %{
         key: :base_url,
+        label: "Base URL",
         type: :string,
-        description: "Base url for the uploads, needed if you use CDN",
+        description: "Base URL for the uploads, needed if you use CDN",
         suggestions: [
           "https://cdn-host.com"
         ]
@@ -58,6 +59,7 @@
       },
       %{
         key: :proxy_opts,
+        label: "Proxy Options",
         type: :keyword,
         description: "Options for Pleroma.ReverseProxy",
         suggestions: [
@@ -85,6 +87,7 @@
           },
           %{
             key: :http,
+            label: "HTTP",
             type: :keyword,
             description: "HTTP options",
             children: [
@@ -193,7 +196,9 @@
       %{
         key: :args,
         type: [:string, {:list, :string}, {:list, :tuple}],
-        description: "List of actions for the mogrify command",
+        description:
+          "List of actions for the mogrify command. It's possible to add self-written settings as string. " <>
+            "For example `[\"auto-orient\", \"strip\", {\"resize\", \"3840x1080>\"}]` string will be parsed into list of the settings.",
         suggestions: [
           "strip",
           "auto-orient",
@@ -479,6 +484,7 @@
   %{
     group: :pleroma,
     key: :uri_schemes,
+    label: "URI Schemes",
     type: :group,
     description: "URI schemes related settings",
     children: [
@@ -492,6 +498,7 @@
           "dat",
           "dweb",
           "gopher",
+          "hyper",
           "ipfs",
           "ipns",
           "irc",
@@ -651,17 +658,17 @@
         key: :invites_enabled,
         type: :boolean,
         description:
-          "Enable user invitations for admins (depends on `registrations_open` being disabled)."
+          "Enable user invitations for admins (depends on `registrations_open` being disabled)"
       },
       %{
         key: :account_activation_required,
         type: :boolean,
-        description: "Require users to confirm their emails before signing in."
+        description: "Require users to confirm their emails before signing in"
       },
       %{
         key: :federating,
         type: :boolean,
-        description: "Enable federation with other instances."
+        description: "Enable federation with other instances"
       },
       %{
         key: :federation_incoming_replies_max_depth,
@@ -679,7 +686,7 @@
         label: "Fed. reachability timeout days",
         type: :integer,
         description:
-          "Timeout (in days) of each external federation target being unreachable prior to pausing federating to it.",
+          "Timeout (in days) of each external federation target being unreachable prior to pausing federating to it",
         suggestions: [
           7
         ]
@@ -689,23 +696,13 @@
         type: :boolean,
         description: "Enable Pleroma's Relay, which makes it possible to follow a whole instance"
       },
-      %{
-        key: :rewrite_policy,
-        type: [:module, {:list, :module}],
-        description:
-          "A list of enabled MRF policies. Module names are shortened (removed leading `Pleroma.Web.ActivityPub.MRF.` part), but on adding custom module you need to use full name.",
-        suggestions:
-          Generator.list_modules_in_dir(
-            "lib/pleroma/web/activity_pub/mrf",
-            "Elixir.Pleroma.Web.ActivityPub.MRF."
-          )
-      },
       %{
         key: :public,
         type: :boolean,
         description:
-          "Makes the client API in authentificated mode-only except for user-profiles." <>
+          "Makes the client API in authenticated mode-only except for user-profiles." <>
-            " Useful for disabling the Local Timeline and The Whole Known Network."
+            " Useful for disabling the Local Timeline and The Whole Known Network. " <>
+            " Note: when setting to `false`, please also check `:restrict_unauthenticated` setting."
       },
       %{
         key: :quarantined_instances,
@@ -742,23 +739,6 @@
           "text/bbcode"
         ]
       },
-      %{
-        key: :mrf_transparency,
-        label: "MRF transparency",
-        type: :boolean,
-        description:
-          "Make the content of your Message Rewrite Facility settings public (via nodeinfo)"
-      },
-      %{
-        key: :mrf_transparency_exclusions,
-        label: "MRF transparency exclusions",
-        type: {:list, :string},
-        description:
-          "Exclude specific instance names from MRF transparency. The use of the exclusions feature will be disclosed in nodeinfo as a boolean value.",
-        suggestions: [
-          "exclusion.com"
-        ]
-      },
       %{
         key: :extended_nickname_format,
         type: :boolean,
@@ -829,6 +809,7 @@
       },
       %{
         key: :safe_dm_mentions,
+        label: "Safe DM mentions",
         type: :boolean,
         description:
           "If enabled, only mentions at the beginning of a post will be used to address people in direct messages." <>
@@ -868,7 +849,7 @@
       %{
         key: :skip_thread_containment,
         type: :boolean,
-        description: "Skip filtering out broken threads. Default: enabled"
+        description: "Skip filtering out broken threads. Default: enabled."
       },
       %{
         key: :limit_to_local_content,
@@ -932,6 +913,7 @@
         children: [
           %{
             key: :totp,
+            label: "TOTP settings",
             type: :keyword,
             description: "TOTP settings",
             suggestions: [digits: 6, period: 30],
@@ -948,7 +930,7 @@
                 type: :integer,
                 suggestions: [30],
                 description:
-                  "a period for which the TOTP code will be valid, in seconds. Defaults to 30 seconds."
+                  "A period for which the TOTP code will be valid, in seconds. Defaults to 30 seconds."
               }
             ]
           },
@@ -962,7 +944,7 @@
                 key: :number,
                 type: :integer,
                 suggestions: [5],
-                description: "number of backup codes to generate."
+                description: "Number of backup codes to generate."
               },
               %{
                 key: :length,
@@ -979,7 +961,7 @@
         key: :instance_thumbnail,
         type: :string,
         description:
-          "The instance thumbnail image. It will appear in [Pleroma Instances](http://distsn.org/pleroma-instances.html)",
+          "The instance thumbnail can be any image that represents your instance and is used by some apps or services when they display information about your instance.",
         suggestions: ["/instance/thumbnail.jpeg"]
       }
     ]
@@ -1002,6 +984,7 @@
     group: :logger,
     type: :group,
     key: :ex_syslogger,
+    label: "ExSyslogger",
     description: "ExSyslogger-related settings",
     children: [
       %{
@@ -1020,7 +1003,7 @@
       %{
         key: :format,
         type: :string,
-        description: "Default: \"$date $time [$level] $levelpad$node $metadata $message\".",
+        description: "Default: \"$date $time [$level] $levelpad$node $metadata $message\"",
         suggestions: ["$metadata[$level] $message"]
       },
       %{
@@ -1034,6 +1017,7 @@
     group: :logger,
     type: :group,
     key: :console,
+    label: "Console Logger",
     description: "Console logger settings",
     children: [
       %{
@@ -1045,7 +1029,7 @@
       %{
         key: :format,
         type: :string,
-        description: "Default: \"$date $time [$level] $levelpad$node $metadata $message\".",
+        description: "Default: \"$date $time [$level] $levelpad$node $metadata $message\"",
         suggestions: ["$metadata[$level] $message"]
       },
       %{
@@ -1058,6 +1042,7 @@
   %{
     group: :quack,
     type: :group,
+    label: "Quack Logger",
     description: "Quack-related settings",
     children: [
       %{
@@ -1168,19 +1153,19 @@
             key: :greentext,
             label: "Greentext",
             type: :boolean,
-            description: "Enables green text on lines prefixed with the > character."
+            description: "Enables green text on lines prefixed with the > character"
           },
           %{
             key: :hideFilteredStatuses,
             label: "Hide Filtered Statuses",
             type: :boolean,
-            description: "Hides filtered statuses from timelines."
+            description: "Hides filtered statuses from timelines"
           },
           %{
             key: :hideMutedPosts,
             label: "Hide Muted Posts",
             type: :boolean,
-            description: "Hides muted statuses from timelines."
+            description: "Hides muted statuses from timelines"
           },
           %{
             key: :hidePostStats,
@@ -1192,7 +1177,7 @@
             key: :hideSitename,
             label: "Hide Sitename",
             type: :boolean,
-            description: "Hides instance name from PleromaFE banner."
+            description: "Hides instance name from PleromaFE banner"
           },
           %{
             key: :hideUserStats,
@@ -1237,14 +1222,14 @@
             label: "NSFW Censor Image",
             type: :string,
             description:
-              "URL of the image to use for hiding NSFW media attachments in the timeline.",
+              "URL of the image to use for hiding NSFW media attachments in the timeline",
             suggestions: ["/static/img/nsfw.74818f9.png"]
           },
           %{
             key: :postContentType,
             label: "Post Content Type",
             type: {:dropdown, :atom},
-            description: "Default post formatting option.",
+            description: "Default post formatting option",
             suggestions: ["text/plain", "text/html", "text/markdown", "text/bbcode"]
           },
           %{
@@ -1273,14 +1258,14 @@
             key: :sidebarRight,
             label: "Sidebar on Right",
             type: :boolean,
-            description: "Change alignment of sidebar and panels to the right."
+            description: "Change alignment of sidebar and panels to the right"
           },
           %{
             key: :showFeaturesPanel,
             label: "Show instance features panel",
             type: :boolean,
             description:
-              "Enables panel displaying functionality of the instance on the About page."
+              "Enables panel displaying functionality of the instance on the About page"
           },
           %{
             key: :showInstanceSpecificPanel,
@@ -1338,7 +1323,7 @@
         key: :mascots,
         type: {:keyword, :map},
         description:
-          "Keyword of mascots, each element must contain both an url and a mime_type key",
+          "Keyword of mascots, each element must contain both an URL and a mime_type key",
         suggestions: [
           pleroma_fox_tan: %{
             url: "/images/pleroma-fox-tan-smol.png",
@@ -1362,7 +1347,7 @@
       %{
         key: :default_user_avatar,
         type: :string,
-        description: "URL of the default user avatar.",
+        description: "URL of the default user avatar",
         suggestions: ["/images/avi.png"]
       }
     ]
@@ -1372,7 +1357,7 @@
     key: :manifest,
     type: :group,
     description:
-      "This section describe PWA manifest instance-specific values. Currently this option relate only for MastoFE",
+      "This section describe PWA manifest instance-specific values. Currently this option relate only for MastoFE.",
     children: [
       %{
         key: :icons,
@@ -1408,10 +1393,49 @@
   },
   %{
     group: :pleroma,
-    key: :mrf_simple,
+    key: :mrf,
-    label: "MRF simple",
+    tab: :mrf,
+    label: "MRF",
     type: :group,
-    description: "Message Rewrite Facility",
+    description: "General MRF settings",
+    children: [
+      %{
+        key: :policies,
+        type: [:module, {:list, :module}],
+        description:
+          "A list of MRF policies enabled. Module names are shortened (removed leading `Pleroma.Web.ActivityPub.MRF.` part), but on adding custom module you need to use full name.",
+        suggestions:
+          Generator.list_modules_in_dir(
+            "lib/pleroma/web/activity_pub/mrf",
+            "Elixir.Pleroma.Web.ActivityPub.MRF."
+          )
+      },
+      %{
+        key: :transparency,
+        label: "MRF transparency",
+        type: :boolean,
+        description:
+          "Make the content of your Message Rewrite Facility settings public (via nodeinfo)"
+      },
+      %{
+        key: :transparency_exclusions,
+        label: "MRF transparency exclusions",
+        type: {:list, :string},
+        description:
+          "Exclude specific instance names from MRF transparency. The use of the exclusions feature will be disclosed in nodeinfo as a boolean value.",
+        suggestions: [
+          "exclusion.com"
+        ]
+      }
+    ]
+  },
+  %{
+    group: :pleroma,
+    key: :mrf_simple,
+    tab: :mrf,
+    label: "MRF Simple",
+    type: :group,
+    description: "Simple ingress policies",
     children: [
       %{
         key: :media_removal,
@@ -1430,7 +1454,7 @@
         key: :federated_timeline_removal,
         type: {:list, :string},
         description:
-          "List of instances to remove from Federated (aka The Whole Known Network) Timeline",
+          "List of instances to remove from the Federated (aka The Whole Known Network) Timeline",
         suggestions: ["example.com", "*.example.com"]
       },
       %{
@@ -1474,14 +1498,15 @@
   %{
     group: :pleroma,
     key: :mrf_activity_expiration,
+    tab: :mrf,
     label: "MRF Activity Expiration Policy",
     type: :group,
-    description: "Adds expiration to all local Create Note activities",
+    description: "Adds automatic expiration to all local activities",
     children: [
       %{
         key: :days,
         type: :integer,
-        description: "Default global expiration time for all local Create activities (in days)",
+        description: "Default global expiration time for all local activities (in days)",
         suggestions: [90, 365]
       }
     ]
@@ -1489,7 +1514,8 @@
   %{
     group: :pleroma,
     key: :mrf_subchain,
-    label: "MRF subchain",
+    tab: :mrf,
+    label: "MRF Subchain",
     type: :group,
     description:
       "This policy processes messages through an alternate pipeline when a given message matches certain criteria." <>
@@ -1510,9 +1536,9 @@
   %{
     group: :pleroma,
     key: :mrf_rejectnonpublic,
-    description:
+    tab: :mrf,
-      "MRF RejectNonPublic settings. RejectNonPublic drops posts with non-public visibility settings.",
+    description: "RejectNonPublic drops posts with non-public visibility settings.",
-    label: "MRF reject non public",
+    label: "MRF Reject Non Public",
     type: :group,
     children: [
       %{
@@ -1531,16 +1557,17 @@
   %{
     group: :pleroma,
     key: :mrf_hellthread,
-    label: "MRF hellthread",
+    tab: :mrf,
+    label: "MRF Hellthread",
     type: :group,
-    description: "Block messages with too much mentions",
+    description: "Block messages with excessive user mentions",
     children: [
       %{
         key: :delist_threshold,
         type: :integer,
         description:
-          "Number of mentioned users after which the message gets delisted (the message can still be seen, " <>
+          "Number of mentioned users after which the message gets removed from timelines and" <>
-            " but it will not show up in public timelines and mentioned users won't get notifications about it). Set to 0 to disable.",
+            "disables notifications. Set to 0 to disable.",
         suggestions: [10]
       },
       %{
@@ -1555,7 +1582,8 @@
   %{
     group: :pleroma,
     key: :mrf_keyword,
-    label: "MRF keyword",
+    tab: :mrf,
+    label: "MRF Keyword",
     type: :group,
     description: "Reject or Word-Replace messages with a keyword or regex",
     children: [
@@ -1585,14 +1613,15 @@
   %{
     group: :pleroma,
     key: :mrf_mention,
-    label: "MRF mention",
+    tab: :mrf,
+    label: "MRF Mention",
     type: :group,
-    description: "Block messages which mention a user",
+    description: "Block messages which mention a specific user",
     children: [
       %{
         key: :actors,
         type: {:list, :string},
-        description: "A list of actors for which any post mentioning them will be dropped.",
+        description: "A list of actors for which any post mentioning them will be dropped",
         suggestions: ["actor1", "actor2"]
       }
     ]
@@ -1600,7 +1629,8 @@
   %{
     group: :pleroma,
     key: :mrf_vocabulary,
-    label: "MRF vocabulary",
+    tab: :mrf,
+    label: "MRF Vocabulary",
     type: :group,
     description: "Filter messages which belong to certain activity vocabularies",
     children: [
@@ -1608,14 +1638,14 @@
         key: :accept,
         type: {:list, :string},
         description:
-          "A list of ActivityStreams terms to accept. If empty, all supported messages are accepted",
+          "A list of ActivityStreams terms to accept. If empty, all supported messages are accepted.",
         suggestions: ["Create", "Follow", "Mention", "Announce", "Like"]
       },
       %{
         key: :reject,
         type: {:list, :string},
         description:
-          "A list of ActivityStreams terms to reject. If empty, no messages are rejected",
+          "A list of ActivityStreams terms to reject. If empty, no messages are rejected.",
         suggestions: ["Create", "Follow", "Mention", "Announce", "Like"]
       }
     ]
@@ -1645,13 +1675,40 @@
       },
       %{
         key: :base_url,
+        label: "Base URL",
         type: :string,
         description:
           "The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host/CDN fronts.",
         suggestions: ["https://example.com"]
       },
+      %{
+        key: :invalidation,
+        type: :keyword,
+        descpiption: "",
+        suggestions: [
+          enabled: true,
+          provider: Pleroma.Web.MediaProxy.Invalidation.Script
+        ],
+        children: [
+          %{
+            key: :enabled,
+            type: :boolean,
+            description: "Enables invalidate media cache"
+          },
+          %{
+            key: :provider,
+            type: :module,
+            description: "Module which will be used to cache purge.",
+            suggestions: [
+              Pleroma.Web.MediaProxy.Invalidation.Script,
+              Pleroma.Web.MediaProxy.Invalidation.Http
+            ]
+          }
+        ]
+      },
       %{
         key: :proxy_opts,
+        label: "Proxy Options",
         type: :keyword,
         description: "Options for Pleroma.ReverseProxy",
         suggestions: [
@@ -1679,6 +1736,7 @@
           },
           %{
             key: :http,
+            label: "HTTP",
             type: :keyword,
             description: "HTTP options",
             children: [
@@ -1722,6 +1780,45 @@
       }
     ]
   },
+  %{
+    group: :pleroma,
+    key: Pleroma.Web.MediaProxy.Invalidation.Http,
+    type: :group,
+    description: "HTTP invalidate settings",
+    children: [
+      %{
+        key: :method,
+        type: :atom,
+        description: "HTTP method of request. Default: :purge"
+      },
+      %{
+        key: :headers,
+        type: {:list, :tuple},
+        description: "HTTP headers of request.",
+        suggestions: [{"x-refresh", 1}]
+      },
+      %{
+        key: :options,
+        type: :keyword,
+        description: "Request options.",
+        suggestions: [params: %{ts: "xxx"}]
+      }
+    ]
+  },
+  %{
+    group: :pleroma,
+    key: Pleroma.Web.MediaProxy.Invalidation.Script,
+    type: :group,
+    description: "Script invalidate settings",
+    children: [
+      %{
+        key: :script_path,
+        type: :string,
+        description: "Path to shell script. Which will run purge cache.",
+        suggestions: ["./installation/nginx-cache-purge.sh.example"]
+      }
+    ]
+  },
   %{
     group: :pleroma,
     key: :gopher,
@@ -1735,6 +1832,7 @@
       },
       %{
         key: :ip,
+        label: "IP",
         type: :tuple,
         description: "IP address to bind to",
         suggestions: [{0, 0, 0, 0}]
@@ -1748,7 +1846,7 @@
       %{
         key: :dstport,
         type: :integer,
-        description: "Port advertised in urls (optional, defaults to port)",
+        description: "Port advertised in URLs (optional, defaults to port)",
         suggestions: [9999]
       }
     ]
@@ -1756,6 +1854,7 @@
   %{
     group: :pleroma,
     key: :activitypub,
+    label: "ActivityPub",
     type: :group,
     description: "ActivityPub-related settings",
     children: [
@@ -1778,7 +1877,7 @@
         key: :note_replies_output_limit,
         type: :integer,
         description:
-          "The number of Note replies' URIs to be included with outgoing federation (`5` to match Mastodon hardcoded value, `0` to disable the output)."
+          "The number of Note replies' URIs to be included with outgoing federation (`5` to match Mastodon hardcoded value, `0` to disable the output)"
       },
       %{
         key: :follow_handshake_timeout,
@@ -1791,6 +1890,7 @@
   %{
     group: :pleroma,
     key: :http_security,
+    label: "HTTP security",
     type: :group,
     description: "HTTP security settings",
     children: [
@@ -1829,7 +1929,7 @@
         key: :report_uri,
         label: "Report URI",
         type: :string,
-        description: "Adds the specified url to report-uri and report-to group in CSP header",
+        description: "Adds the specified URL to report-uri and report-to group in CSP header",
         suggestions: ["https://example.com/report-uri"]
       }
     ]
@@ -1837,9 +1937,10 @@
   %{
     group: :web_push_encryption,
     key: :vapid_details,
+    label: "Vapid Details",
     type: :group,
     description:
-      "Web Push Notifications configuration. You can use the mix task mix web_push.gen.keypair to generate it",
+      "Web Push Notifications configuration. You can use the mix task mix web_push.gen.keypair to generate it.",
     children: [
       %{
         key: :subject,
@@ -1906,6 +2007,7 @@
   },
   %{
     group: :pleroma,
+    label: "Pleroma Admin Token",
     type: :group,
     description:
       "Allows to set a token that can be used to authenticate with the admin api without using an actual user by giving it as the `admin_token` parameter",
@@ -1913,7 +2015,7 @@
       %{
         key: :admin_token,
         type: :string,
-        description: "Token",
+        description: "Admin token",
         suggestions: ["We recommend a secure random string or UUID"]
       }
     ]
@@ -2078,24 +2180,24 @@
     key: :rich_media,
     type: :group,
     description:
-      "If enabled the instance will parse metadata from attached links to generate link previews.",
+      "If enabled the instance will parse metadata from attached links to generate link previews",
     children: [
       %{
         key: :enabled,
         type: :boolean,
-        description: "Enables RichMedia parsing of URLs."
+        description: "Enables RichMedia parsing of URLs"
       },
       %{
         key: :ignore_hosts,
         type: {:list, :string},
-        description: "List of hosts which will be ignored by the metadata parser.",
+        description: "List of hosts which will be ignored by the metadata parser",
         suggestions: ["accounts.google.com", "xss.website"]
       },
       %{
         key: :ignore_tld,
         label: "Ignore TLD",
         type: {:list, :string},
-        description: "List TLDs (top-level domains) which will ignore for parse metadata.",
+        description: "List TLDs (top-level domains) which will ignore for parse metadata",
         suggestions: ["local", "localdomain", "lan"]
       },
       %{
@@ -2123,31 +2225,32 @@
   %{
     group: :auto_linker,
     key: :opts,
+    label: "Auto Linker",
     type: :group,
     description: "Configuration for the auto_linker library",
     children: [
       %{
         key: :class,
         type: [:string, false],
-        description: "Specify the class to be added to the generated link. Disable to clear",
+        description: "Specify the class to be added to the generated link. Disable to clear.",
         suggestions: ["auto-linker", false]
       },
       %{
         key: :rel,
         type: [:string, false],
-        description: "Override the rel attribute. Disable to clear",
+        description: "Override the rel attribute. Disable to clear.",
         suggestions: ["ugc", "noopener noreferrer", false]
       },
       %{
         key: :new_window,
         type: :boolean,
-        description: "Link urls will open in new window/tab"
+        description: "Link URLs will open in new window/tab"
       },
       %{
         key: :truncate,
         type: [:integer, false],
         description:
-          "Set to a number to truncate urls longer then the number. Truncated urls will end in `..`",
+          "Set to a number to truncate URLs longer then the number. Truncated URLs will end in `..`",
         suggestions: [15, false]
       },
       %{
@@ -2158,7 +2261,7 @@
       %{
         key: :extra,
         type: :boolean,
-        description: "Link urls with rarely used schemes (magnet, ipfs, irc, etc.)"
+        description: "Link URLs with rarely used schemes (magnet, ipfs, irc, etc.)"
       }
     ]
   },
@@ -2204,6 +2307,7 @@
   },
   %{
     group: :pleroma,
+    label: "Pleroma Authenticator",
     type: :group,
     description: "Authenticator",
     children: [
@@ -2217,6 +2321,7 @@
   %{
     group: :pleroma,
     key: :ldap,
+    label: "LDAP",
     type: :group,
     description:
       "Use LDAP for user authentication. When a user logs in to the Pleroma instance, the name and password" <>
@@ -2303,6 +2408,7 @@
       },
       %{
         key: :uid,
+        label: "UID",
         type: :string,
         description:
           "LDAP attribute name to authenticate the user, e.g. when \"cn\", the filter will be \"cn=username,base\"",
@@ -2318,11 +2424,12 @@
     children: [
       %{
         key: :enforce_oauth_admin_scope_usage,
+        label: "Enforce OAuth admin scope usage",
         type: :boolean,
         description:
           "OAuth admin scope requirement toggle. " <>
             "If enabled, admin actions explicitly demand admin OAuth scope(s) presence in OAuth token " <>
-            "(client app must support admin scopes). If disabled and token doesn't have admin scope(s)," <>
+            "(client app must support admin scopes). If disabled and token doesn't have admin scope(s), " <>
             "`is_admin` user flag grants access to admin-specific actions."
       },
       %{
@@ -2334,6 +2441,7 @@
       },
       %{
         key: :oauth_consumer_template,
+        label: "OAuth consumer template",
         type: :string,
         description:
           "OAuth consumer mode authentication form template. By default it's `consumer.html` which corresponds to" <>
@@ -2342,6 +2450,7 @@
       },
       %{
         key: :oauth_consumer_strategies,
+        label: "OAuth consumer strategies",
         type: {:list, :string},
         description:
           "The list of enabled OAuth consumer strategies. By default it's set by OAUTH_CONSUMER_STRATEGIES environment variable." <>
@@ -2470,7 +2579,7 @@
       %{
         key: :enabled,
         type: :boolean,
-        description: "enables new users admin digest email when `true`",
+        description: "Enables new users admin digest email when `true`",
         suggestions: [false]
       }
     ]
@@ -2478,6 +2587,7 @@
   %{
     group: :pleroma,
     key: :oauth2,
+    label: "OAuth2",
     type: :group,
     description: "Configure OAuth 2 provider capabilities",
     children: [
@@ -2496,7 +2606,7 @@
       %{
         key: :clean_expired_tokens,
         type: :boolean,
-        description: "Enable a background job to clean expired oauth tokens. Default: disabled."
+        description: "Enable a background job to clean expired OAuth tokens. Default: disabled."
       }
     ]
   },
@@ -2580,6 +2690,7 @@
       },
       %{
         key: :relation_id_action,
+        label: "Relation ID action",
         type: [:tuple, {:list, :tuple}],
         description: "For actions on relation with a specific user (follow, unfollow)",
         suggestions: [{1000, 10}, [{10_000, 10}, {10_000, 50}]]
@@ -2593,6 +2704,7 @@
       },
       %{
         key: :status_id_action,
+        label: "Status ID action",
         type: [:tuple, {:list, :tuple}],
         description:
           "For fav / unfav or reblog / unreblog actions on the same status by the same user",
@@ -2608,6 +2720,7 @@
   },
   %{
     group: :esshd,
+    label: "ESSHD",
     type: :group,
     description:
       "Before enabling this you must add :esshd to mix.exs as one of the extra_applications " <>
@@ -2646,8 +2759,9 @@
   },
   %{
     group: :mime,
+    label: "Mime Types",
     type: :group,
-    description: "Mime types",
+    description: "Mime Types settings",
     children: [
       %{
         key: :types,
@@ -2706,6 +2820,7 @@
   %{
     group: :pleroma,
     key: :http,
+    label: "HTTP",
     type: :group,
     description: "HTTP settings",
     children: [
@@ -2754,6 +2869,7 @@
   %{
     group: :pleroma,
     key: :markup,
+    label: "Markup Settings",
     type: :group,
     children: [
       %{
@@ -2794,8 +2910,9 @@
   },
   %{
     group: :pleroma,
+    tab: :mrf,
     key: :mrf_normalize_markup,
-    label: "MRF normalize markup",
+    label: "MRF Normalize Markup",
     description: "MRF NormalizeMarkup settings. Scrub configured hypertext markup.",
     type: :group,
     children: [
@@ -2851,6 +2968,7 @@
   },
   %{
     group: :cors_plug,
+    label: "CORS plug config",
     type: :group,
     children: [
       %{
@@ -2923,6 +3041,7 @@
   %{
     group: :pleroma,
     key: :web_cache_ttl,
+    label: "Web cache TTL",
     type: :group,
     description:
       "The expiration time for the web responses cache. Values should be in milliseconds or `nil` to disable expiration.",
@@ -2945,9 +3064,10 @@
   %{
     group: :pleroma,
     key: :static_fe,
+    label: "Static FE",
     type: :group,
     description:
-      "Render profiles and posts using server-generated HTML that is viewable without using JavaScript.",
+      "Render profiles and posts using server-generated HTML that is viewable without using JavaScript",
     children: [
       %{
         key: :enabled,
@@ -2965,18 +3085,18 @@
       %{
         key: :post_title,
         type: :map,
-        description: "Configure title rendering.",
+        description: "Configure title rendering",
         children: [
           %{
             key: :max_length,
             type: :integer,
-            description: "Maximum number of characters before truncating title.",
+            description: "Maximum number of characters before truncating title",
             suggestions: [100]
           },
           %{
             key: :omission,
             type: :string,
-            description: "Replacement which will be used after truncating string.",
+            description: "Replacement which will be used after truncating string",
             suggestions: ["..."]
           }
         ]
@@ -2986,8 +3106,11 @@
   %{
     group: :pleroma,
     key: :mrf_object_age,
+    label: "MRF Object Age",
+    tab: :mrf,
     type: :group,
-    description: "Rejects or delists posts based on their age when received.",
+    description:
+      "Rejects or delists posts based on their timestamp deviance from your server's clock.",
     children: [
       %{
         key: :threshold,
@@ -3000,7 +3123,7 @@
         type: {:list, :atom},
         description:
           "A list of actions to apply to the post. `:delist` removes the post from public timelines; " <>
-            "`:strip_followers` removes followers from the ActivityPub recipient list, ensuring they won't be delivered to home timelines; " <>
+            "`:strip_followers` removes followers from the ActivityPub recipient list ensuring they won't be delivered to home timelines; " <>
             "`:reject` rejects the message entirely",
         suggestions: [:delist, :strip_followers, :reject]
       }
@@ -3028,13 +3151,13 @@
       %{
         key: :workers,
         type: :integer,
-        description: "Number of workers to send notifications.",
+        description: "Number of workers to send notifications",
         suggestions: [3]
       },
       %{
         key: :overflow_workers,
         type: :integer,
-        description: "Maximum number of workers created if pool is empty.",
+        description: "Maximum number of workers created if pool is empty",
         suggestions: [2]
       }
     ]
@@ -3325,5 +3448,18 @@
         suggestions: [false]
       }
     ]
+  },
+  %{
+    group: :pleroma,
+    key: :instances_favicons,
+    type: :group,
+    description: "Control favicons for instances",
+    children: [
+      %{
+        key: :enabled,
+        type: :boolean,
+        description: "Allow/disallow displaying and getting instances favicons"
+      }
+    ]
   }
 ]

config/test.exs

@@ -111,6 +111,8 @@
 
 config :pleroma, Pleroma.Web.ApiSpec.CastAndValidate, strict: true
 
+config :pleroma, :instances_favicons, enabled: true
+
 if File.exists?("./config/test.secret.exs") do
   import_config "test.secret.exs"
 else

docs/API/admin_api.md

@@ -488,35 +488,39 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 
 ### Change the user's email, password, display and settings-related fields
 
-- Params:
+* Params:
-  - `email`
+  * `email`
-  - `password`
+  * `password`
-  - `name`
+  * `name`
-  - `bio`
+  * `bio`
-  - `avatar`
+  * `avatar`
-  - `locked`
+  * `locked`
-  - `no_rich_text`
+  * `no_rich_text`
-  - `default_scope`
+  * `default_scope`
-  - `banner`
+  * `banner`
-  - `hide_follows`
+  * `hide_follows`
-  - `hide_followers`
+  * `hide_followers`
-  - `hide_followers_count`
+  * `hide_followers_count`
-  - `hide_follows_count`
+  * `hide_follows_count`
-  - `hide_favorites`
+  * `hide_favorites`
-  - `allow_following_move`
+  * `allow_following_move`
-  - `background`
+  * `background`
-  - `show_role`
+  * `show_role`
-  - `skip_thread_containment`
+  * `skip_thread_containment`
-  - `fields`
+  * `fields`
-  - `discoverable`
+  * `discoverable`
-  - `actor_type`
+  * `actor_type`
 
-- Response:
+* Responses:
+
+Status: 200
 
 ```json
 {"status": "success"}
 ```
 
+Status: 400
+
 ```json
 {"errors":
   {"actor_type": "is invalid"},
@@ -525,8 +529,10 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
  }
 ```
 
+Status: 404
+
 ```json
-{"error": "Unable to update user."}
+{"error": "Not found"}
 ```
 
 ## `GET /api/pleroma/admin/reports`
@@ -1112,6 +1118,10 @@ Loads json generated from `config/descriptions.exs`.
 
 ### Stats
 
+- Query Params:
+  - *optional* `instance`: **string** instance hostname (without protocol) to get stats for
+- Example: `https://mypleroma.org/api/pleroma/admin/stats?instance=lain.com`
+
 - Response:
 
 ```json
@@ -1224,4 +1234,66 @@ Loads json generated from `config/descriptions.exs`.
 - Response:
   - On success: `204`, empty response
   - On failure:
-    - 400 Bad Request `"Invalid parameters"` when `status` is missing
+    - 400 Bad Request `"Invalid parameters"` when `status` is missing
+
+## `GET /api/pleroma/admin/media_proxy_caches`
+
+### Get a list of all banned MediaProxy URLs in Cachex
+
+- Authentication: required
+- Params:
+- *optional* `page`: **integer** page number
+- *optional* `page_size`: **integer** number of log entries per page (default is `50`)
+
+- Response:
+
+``` json
+{
+  "urls": [
+    "http://example.com/media/a688346.jpg",
+    "http://example.com/media/fb1f4d.jpg"
+  ]
+}
+
+```
+
+## `POST /api/pleroma/admin/media_proxy_caches/delete`
+
+### Remove a banned MediaProxy URL from Cachex
+
+- Authentication: required
+- Params:
+  - `urls` (array)
+
+- Response:
+
+``` json
+{
+  "urls": [
+    "http://example.com/media/a688346.jpg",
+    "http://example.com/media/fb1f4d.jpg"
+  ]
+}
+
+```
+
+## `POST /api/pleroma/admin/media_proxy_caches/purge`
+
+### Purge a MediaProxy URL
+
+- Authentication: required
+- Params:
+  - `urls` (array)
+  - `ban` (boolean)
+
+- Response:
+
+``` json
+{
+  "urls": [
+    "http://example.com/media/a688346.jpg",
+    "http://example.com/media/fb1f4d.jpg"
+  ]
+}
+
+```

docs/API/differences_in_mastoapi_responses.md

@@ -27,6 +27,7 @@ Has these additional fields under the `pleroma` object:
 - `expires_at`: a datetime (iso8601) that states when the post will expire (be deleted automatically), or empty if the post won't expire
 - `thread_muted`: true if the thread the post belongs to is muted
 - `emoji_reactions`: A list with emoji / reaction maps. The format is `{name: "☕", count: 1, me: true}`. Contains no information about the reacting users, for that use the `/statuses/:id/reactions` endpoint.
+- `parent_visible`: If the parent of this post is visible to the user or not.
 
 ## Media Attachments
 
@@ -51,11 +52,14 @@ The `id` parameter can also be the `nickname` of the user. This only works in th
 
 Has these additional fields under the `pleroma` object:
 
+- `ap_id`: nullable URL string, ActivityPub id of the user
+- `background_image`: nullable URL string, background image of the user
 - `tags`: Lists an array of tags for the user
-- `relationship{}`: Includes fields as documented for Mastodon API https://docs.joinmastodon.org/entities/relationship/
+- `relationship` (object): Includes fields as documented for Mastodon API https://docs.joinmastodon.org/entities/relationship/
 - `is_moderator`: boolean, nullable,  true if user is a moderator
 - `is_admin`: boolean, nullable, true if user is an admin
 - `confirmation_pending`: boolean, true if a new user account is waiting on email confirmation to be activated
+- `hide_favorites`: boolean, true when the user has hiding favorites enabled
 - `hide_followers`: boolean, true when the user has follower hiding enabled
 - `hide_follows`: boolean, true when the user has follow hiding enabled
 - `hide_followers_count`: boolean, true when the user has follower stat hiding enabled
@@ -66,6 +70,8 @@ Has these additional fields under the `pleroma` object:
 - `allow_following_move`: boolean, true when the user allows automatically follow moved following accounts
 - `unread_conversation_count`: The count of unread conversations. Only returned to the account owner.
 - `unread_notifications_count`: The count of unread notifications. Only returned to the account owner.
+- `notification_settings`: object, can be absent. See `/api/pleroma/notification_settings` for the parameters/keys returned.
+- `favicon`: nullable URL string, Favicon image of the user's instance
 
 ### Source
 
@@ -177,10 +183,12 @@ Additional parameters can be added to the JSON body/Form data:
 - `pleroma_settings_store` - Opaque user settings to be saved on the backend.
 - `skip_thread_containment` - if true, skip filtering out broken threads
 - `allow_following_move` - if true, allows automatically follow moved following accounts
-- `pleroma_background_image` - sets the background image of the user.
+- `pleroma_background_image` - sets the background image of the user. Can be set to "" (an empty string) to reset.
 - `discoverable` - if true, discovery of this account in search results and other services is allowed.
 - `actor_type` - the type of this account.
 
+All images (avatar, banner and background) can be reset to the default by sending an empty string ("") instead of a file.
+
 ### Pleroma Settings Store
 
 Pleroma has mechanism that allows frontends to save blobs of json for each user on the backend. This can be used to save frontend-specific settings for a user that the backend does not need to know about.
@@ -215,6 +223,8 @@ Has theses additional parameters (which are the same as in Pleroma-API):
 `GET /api/v1/instance` has additional fields
 
 - `max_toot_chars`: The maximum characters per post
+- `chat_limit`: The maximum characters per chat message
+- `description_limit`: The maximum characters per image description
 - `poll_limits`: The limits of polls
 - `upload_limit`: The maximum upload file size
 - `avatar_upload_limit`: The same for avatars
@@ -223,6 +233,7 @@ Has theses additional parameters (which are the same as in Pleroma-API):
 - `background_image`: A background image that frontends can use
 - `pleroma.metadata.features`: A list of supported features
 - `pleroma.metadata.federation`: The federation restrictions of this instance
+- `pleroma.metadata.fields_limits`: A list of values detailing the length and count limitation for various instance-configurable fields.
 - `vapid_public_key`: The public key needed for push messages
 
 ## Markers
@@ -234,3 +245,43 @@ Has these additional fields under the `pleroma` object:
 ## Streaming
 
 There is an additional `user:pleroma_chat` stream. Incoming chat messages will make the current chat be sent to this `user` stream. The `event` of an incoming chat message is `pleroma:chat_update`. The payload is the updated chat with the incoming chat message in the `last_message` field.
+
+## Not implemented
+
+Pleroma is generally compatible with the Mastodon 2.7.2 API, but some newer features and non-essential features are omitted. These features usually return an HTTP 200 status code, but with an empty response. While they may be added in the future, they are considered low priority.
+
+### Suggestions
+
+*Added in Mastodon 2.4.3*
+
+- `GET /api/v1/suggestions`: Returns an empty array, `[]`
+
+### Trends
+
+*Added in Mastodon 3.0.0*
+
+- `GET /api/v1/trends`: Returns an empty array, `[]`
+
+### Identity proofs
+
+*Added in Mastodon 2.8.0*
+
+- `GET /api/v1/identity_proofs`: Returns an empty array, `[]`
+
+### Endorsements
+
+*Added in Mastodon 2.5.0*
+
+- `GET /api/v1/endorsements`: Returns an empty array, `[]`
+
+### Profile directory
+
+*Added in Mastodon 3.0.0*
+
+- `GET /api/v1/directory`: Returns HTTP 404
+
+### Featured tags
+
+*Added in Mastodon 3.0.0*
+
+- `GET /api/v1/featured_tags`: Returns HTTP 404

docs/API/pleroma_api.md

@@ -450,18 +450,44 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 * Response: JSON, list with updated files for updated pack (hashmap -> shortcode => filename) with status 200, either error status with error message.
 
 ## `GET /api/pleroma/emoji/packs`
+
 ### Lists local custom emoji packs
+
 * Method `GET`
 * Authentication: not required
-* Params: None
+* Params:
-* Response: JSON, "ok" and 200 status and the JSON hashmap of pack name to pack contents
+  * `page`: page number for packs (default 1)
+  * `page_size`: page size for packs (default 50)
+* Response: `packs` key with JSON hashmap of pack name to pack contents and `count` key for count of packs.
+
+```json
+{
+  "packs": {
+    "pack_name": {...}, // pack contents
+    ...
+  },
+  "count": 0 // packs count
+}
+```
 
 ## `GET /api/pleroma/emoji/packs/:name`
+
 ### Get pack.json for the pack
+
 * Method `GET`
 * Authentication: not required
-* Params: None
+* Params:
-* Response: JSON, pack json with `files` and `pack` keys with 200 status or 404 if the pack does not exist
+  * `page`: page number for files (default 1)
+  * `page_size`: page size for files (default 30)
+* Response: JSON, pack json with `files`, `files_count` and `pack` keys with 200 status or 404 if the pack does not exist.
+
+```json
+{
+  "files": {...},
+  "files_count": 0, // emoji count in pack
+  "pack": {...}
+}
+```
 
 ## `GET /api/pleroma/emoji/packs/:name/archive`
 ### Requests a local pack archive from the instance

docs/administration/updating.md

@@ -1,6 +1,6 @@
 # Updating your instance
 
-You should **always check the release notes/changelog** in case there are config deprecations, special update special update steps, etc.
+You should **always check the [release notes/changelog](https://git.pleroma.social/pleroma/pleroma/-/releases)** in case there are config deprecations, special update steps, etc.
 
 Besides that, doing the following is generally enough:
 

docs/configuration/cheatsheet.md

@@ -18,6 +18,7 @@ To add configuration to your config file, you can copy it from the base config.
 * `notify_email`: Email used for notifications.
 * `description`: The instance’s description, can be seen in nodeinfo and ``/api/v1/instance``.
 * `limit`: Posts character limit (CW/Subject included in the counter).
+* `discription_limit`: The character limit for image descriptions.
 * `chat_limit`: Character limit of the instance chat messages.
 * `remote_limit`: Hard character limit beyond which remote posts will be dropped.
 * `upload_limit`: File size limit of uploads (except for avatar, background, banner).
@@ -36,31 +37,15 @@ To add configuration to your config file, you can copy it from the base config.
 * `federation_incoming_replies_max_depth`: Max. depth of reply-to activities fetching on incoming federation, to prevent out-of-memory situations while fetching very long threads. If set to `nil`, threads of any depth will be fetched. Lower this value if you experience out-of-memory crashes.
 * `federation_reachability_timeout_days`: Timeout (in days) of each external federation target being unreachable prior to pausing federating to it.
 * `allow_relay`: Enable Pleroma’s Relay, which makes it possible to follow a whole instance.
-* `rewrite_policy`: Message Rewrite Policy, either one or a list. Here are the ones available by default:
+* `public`: Makes the client API in authenticated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network. See also: `restrict_unauthenticated`.
-    * `Pleroma.Web.ActivityPub.MRF.NoOpPolicy`: Doesn’t modify activities (default).
-    * `Pleroma.Web.ActivityPub.MRF.DropPolicy`: Drops all activities. It generally doesn’t makes sense to use in production.
-    * `Pleroma.Web.ActivityPub.MRF.SimplePolicy`: Restrict the visibility of activities from certain instances (See [`:mrf_simple`](#mrf_simple)).
-    * `Pleroma.Web.ActivityPub.MRF.TagPolicy`: Applies policies to individual users based on tags, which can be set using pleroma-fe/admin-fe/any other app that supports Pleroma Admin API. For example it allows marking posts from individual users nsfw (sensitive).
-    * `Pleroma.Web.ActivityPub.MRF.SubchainPolicy`: Selectively runs other MRF policies when messages match (See [`:mrf_subchain`](#mrf_subchain)).
-    * `Pleroma.Web.ActivityPub.MRF.RejectNonPublic`: Drops posts with non-public visibility settings (See [`:mrf_rejectnonpublic`](#mrf_rejectnonpublic)).
-    * `Pleroma.Web.ActivityPub.MRF.EnsureRePrepended`: Rewrites posts to ensure that replies to posts with subjects do not have an identical subject and instead begin with re:.
-    * `Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy`: Rejects posts from likely spambots by rejecting posts from new users that contain links.
-    * `Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`: Crawls attachments using their MediaProxy URLs so that the MediaProxy cache is primed.
-    * `Pleroma.Web.ActivityPub.MRF.MentionPolicy`: Drops posts mentioning configurable users. (See [`:mrf_mention`](#mrf_mention)).
-    * `Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`: Restricts activities to a configured set of vocabulary. (See [`:mrf_vocabulary`](#mrf_vocabulary)).
-    * `Pleroma.Web.ActivityPub.MRF.ObjectAgePolicy`: Rejects or delists posts based on their age when received. (See [`:mrf_object_age`](#mrf_object_age)).
-    * `Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy`: Adds expiration to all local Create activities (see [`:mrf_activity_expiration`](#mrf_activity_expiration)).
-* `public`: Makes the client API in authenticated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
 * `quarantined_instances`: List of ActivityPub instances where private(DMs, followers-only) activities will not be send.
 * `managed_config`: Whenether the config for pleroma-fe is configured in [:frontend_configurations](#frontend_configurations) or in ``static/config.json``.
 * `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML).
-* `mrf_transparency`: Make the content of your Message Rewrite Facility settings public (via nodeinfo).
-* `mrf_transparency_exclusions`: Exclude specific instance names from MRF transparency.  The use of the exclusions feature will be disclosed in nodeinfo as a boolean value.
 * `extended_nickname_format`: Set to `true` to use extended local nicknames format (allows underscores/dashes). This will break federation with
     older software for theses nicknames.
 * `max_pinned_statuses`: The maximum number of pinned statuses. `0` will disable the feature.
 * `autofollowed_nicknames`: Set to nicknames of (local) users that every new user should automatically follow.
-* `no_attachment_links`: Set to true to disable automatically adding attachment link text to statuses.
+* `attachment_links`: Set to true to enable automatically adding attachment link text to statuses.
 * `welcome_message`: A message that will be send to a newly registered users as a direct message.
 * `welcome_user_nickname`: The nickname of the local user that sends the welcome message.
 * `max_report_comment_size`: The maximum size of the report comment (Default: `1000`).
@@ -78,11 +63,30 @@ To add configuration to your config file, you can copy it from the base config.
 * `external_user_synchronization`: Enabling following/followers counters synchronization for external users.
 * `cleanup_attachments`: Remove attachments along with statuses. Does not affect duplicate files and attachments without status. Enabling this will increase load to database when deleting statuses on larger instances.
 
+## Message rewrite facility
+
+### :mrf
+* `policies`: Message Rewrite Policy, either one or a list. Here are the ones available by default:
+    * `Pleroma.Web.ActivityPub.MRF.NoOpPolicy`: Doesn’t modify activities (default).
+    * `Pleroma.Web.ActivityPub.MRF.DropPolicy`: Drops all activities. It generally doesn’t makes sense to use in production.
+    * `Pleroma.Web.ActivityPub.MRF.SimplePolicy`: Restrict the visibility of activities from certains instances (See [`:mrf_simple`](#mrf_simple)).
+    * `Pleroma.Web.ActivityPub.MRF.TagPolicy`: Applies policies to individual users based on tags, which can be set using pleroma-fe/admin-fe/any other app that supports Pleroma Admin API. For example it allows marking posts from individual users nsfw (sensitive).
+    * `Pleroma.Web.ActivityPub.MRF.SubchainPolicy`: Selectively runs other MRF policies when messages match (See [`:mrf_subchain`](#mrf_subchain)).
+    * `Pleroma.Web.ActivityPub.MRF.RejectNonPublic`: Drops posts with non-public visibility settings (See [`:mrf_rejectnonpublic`](#mrf_rejectnonpublic)).
+    * `Pleroma.Web.ActivityPub.MRF.EnsureRePrepended`: Rewrites posts to ensure that replies to posts with subjects do not have an identical subject and instead begin with re:.
+    * `Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy`: Rejects posts from likely spambots by rejecting posts from new users that contain links.
+    * `Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`: Crawls attachments using their MediaProxy URLs so that the MediaProxy cache is primed.
+    * `Pleroma.Web.ActivityPub.MRF.MentionPolicy`: Drops posts mentioning configurable users. (See [`:mrf_mention`](#mrf_mention)).
+    * `Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`: Restricts activities to a configured set of vocabulary. (See [`:mrf_vocabulary`](#mrf_vocabulary)).
+    * `Pleroma.Web.ActivityPub.MRF.ObjectAgePolicy`: Rejects or delists posts based on their age when received. (See [`:mrf_object_age`](#mrf_object_age)).
+* `transparency`: Make the content of your Message Rewrite Facility settings public (via nodeinfo).
+* `transparency_exclusions`: Exclude specific instance names from MRF transparency.  The use of the exclusions feature will be disclosed in nodeinfo as a boolean value.
+
 ## Federation
 ### MRF policies
 
 !!! note
-    Configuring MRF policies is not enough for them to take effect. You have to enable them by specifying their module in `rewrite_policy` under [:instance](#instance) section.
+    Configuring MRF policies is not enough for them to take effect. You have to enable them by specifying their module in `policies` under [:mrf](#mrf) section.
 
 #### :mrf_simple
 * `media_removal`: List of instances to remove media from.
@@ -151,7 +155,7 @@ config :pleroma, :mrf_user_allowlist, %{
   * `:strip_followers` removes followers from the ActivityPub recipient list, ensuring they won't be delivered to home timelines
   * `:reject` rejects the message entirely
 
-#### mrf_steal_emoji
+#### :mrf_steal_emoji
 * `hosts`: List of hosts to steal emojis from
 * `rejected_shortcodes`: Regex-list of shortcodes to reject
 * `size_limit`: File size limit (in bytes), checked before an emoji is saved to the disk
@@ -268,7 +272,7 @@ This section describe PWA manifest instance-specific values. Currently this opti
 
 #### Pleroma.Web.MediaProxy.Invalidation.Script
 
-This strategy allow perform external bash script to purge cache.
+This strategy allow perform external shell script to purge cache.
 Urls of attachments pass to script as arguments.
 
 * `script_path`: path to external script.
@@ -284,8 +288,8 @@ config :pleroma, Pleroma.Web.MediaProxy.Invalidation.Script,
 This strategy allow perform custom http request to purge cache.
 
 * `method`: http method. default is `purge`
-* `headers`: http headers. default is empty
+* `headers`: http headers.
-* `options`: request options. default is empty
+* `options`: request options.
 
 Example:
 ```elixir
@@ -967,19 +971,26 @@ config :pleroma, :database_config_whitelist, [
 
 ### :restrict_unauthenticated
 
-Restrict access for unauthenticated users to timelines (public and federate), user profiles and statuses.
+Restrict access for unauthenticated users to timelines (public and federated), user profiles and statuses.
 
-* `timelines` - public and federated timelines
+* `timelines`: public and federated timelines
-  * `local` - public timeline
+  * `local`: public timeline
-  * `federated`
+  * `federated`: federated timeline (includes public timeline)
-* `profiles` - user profiles
+* `profiles`: user profiles
   * `local`
   * `remote`
-* `activities` - statuses
+* `activities`: statuses
   * `local`
   * `remote`
 
+Note: setting `restrict_unauthenticated/timelines/local` to `true` has no practical sense if `restrict_unauthenticated/timelines/federated` is set to `false` (since local public activities will still be delivered to unauthenticated users as part of federated timeline). 
 
 ## Pleroma.Web.ApiSpec.CastAndValidate
 
 * `:strict` a boolean, enables strict input validation (useful in development, not recommended in production). Defaults to `false`.
+
+## :instances_favicons
+
+Control favicons for instances.
+
+* `enabled`: Allow/disallow displaying and getting instances favicons

docs/configuration/howto_theming_your_instance.md

@@ -60,7 +60,7 @@ Example of `my-awesome-theme.json` where we add the name "My Awesome Theme"
 
 ### Set as default theme
 
-Now we can set the new theme as default in the [Pleroma FE configuration](General-tips-for-customizing-Pleroma-FE.md).
+Now we can set the new theme as default in the [Pleroma FE configuration](../../../frontend/CONFIGURATION).
 
 Example of adding the new theme in the back-end config files
 ```elixir

docs/configuration/mrf.md

@@ -34,9 +34,9 @@ config :pleroma, :instance,
 To use `SimplePolicy`, you must enable it. Do so by adding the following to your `:instance` config object, so that it looks like this:
 
 ```elixir
-config :pleroma, :instance,
+config :pleroma, :mrf,
   [...]
-  rewrite_policy: Pleroma.Web.ActivityPub.MRF.SimplePolicy
+  policies: Pleroma.Web.ActivityPub.MRF.SimplePolicy
 ```
 
 Once `SimplePolicy` is enabled, you can configure various groups in the `:mrf_simple` config object. These groups are:
@@ -58,8 +58,8 @@ Servers should be configured as lists.
 This example will enable `SimplePolicy`, block media from `illegalporn.biz`, mark media as NSFW from `porn.biz` and `porn.business`, reject messages from `spam.com`, remove messages from `spam.university` from the federated timeline and block reports (flags) from `whiny.whiner`:
 
 ```elixir
-config :pleroma, :instance,
+config :pleroma, :mrf,
-  rewrite_policy: [Pleroma.Web.ActivityPub.MRF.SimplePolicy]
+  policies: [Pleroma.Web.ActivityPub.MRF.SimplePolicy]
 
 config :pleroma, :mrf_simple,
   media_removal: ["illegalporn.biz"],
@@ -75,7 +75,7 @@ The effects of MRF policies can be very drastic. It is important to use this fun
 
 ## Writing your own MRF Policy
 
-As discussed above, the MRF system is a modular system that supports pluggable policies. This means that an admin may write a custom MRF policy in Elixir or any other language that runs on the Erlang VM, by specifying the module name in the `rewrite_policy` config setting.
+As discussed above, the MRF system is a modular system that supports pluggable policies. This means that an admin may write a custom MRF policy in Elixir or any other language that runs on the Erlang VM, by specifying the module name in the `policies` config setting.
 
 For example, here is a sample policy module which rewrites all messages to "new message content":
 
@@ -125,8 +125,8 @@ end
 If you save this file as `lib/pleroma/web/activity_pub/mrf/rewrite_policy.ex`, it will be included when you next rebuild Pleroma.  You can enable it in the configuration like so:
 
 ```elixir
-config :pleroma, :instance,
+config :pleroma, :mrf,
-  rewrite_policy: [
+  policies: [
     Pleroma.Web.ActivityPub.MRF.SimplePolicy,
     Pleroma.Web.ActivityPub.MRF.RewritePolicy
   ]

docs/configuration/storing_remote_media.md

@@ -33,6 +33,6 @@ as soon as the post is received by your instance.
 Add to your `prod.secret.exs`:
 
 ```
-config :pleroma, :instance,
+config :pleroma, :mrf,
-  rewrite_policy: [Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy]
+  policies: [Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy]
 ```

docs/dev.md

@@ -20,4 +20,4 @@ This document contains notes and guidelines for Pleroma developers.
 
 ## Auth-related configuration, OAuth consumer mode etc.
 
-See `Authentication` section of [`docs/configuration/cheatsheet.md`](docs/configuration/cheatsheet.md#authentication).
+See `Authentication` section of [the configuration cheatsheet](configuration/cheatsheet.md#authentication).

docs/index.md

@@ -0,0 +1,26 @@
+# Introduction to Pleroma
+## What is Pleroma?
+Pleroma is a federated social networking platform, compatible with Mastodon and other ActivityPub implementations. It is free software licensed under the AGPLv3.
+It actually consists of two components: a backend, named simply Pleroma, and a user-facing frontend, named Pleroma-FE. It also includes the Mastodon frontend, if that's your thing.
+It's part of what we call the fediverse, a federated network of instances which speak common protocols and can communicate with each other.
+One account on an instance is enough to talk to the entire fediverse!
+
+## How can I use it?
+
+Pleroma instances are already widely deployed, a list can be found at <https://the-federation.info/pleroma> and <https://fediverse.network/pleroma>.
+
+If you don't feel like joining an existing instance, but instead prefer to deploy your own instance, that's easy too!
+Installation instructions can be found in the installation section of these docs.
+
+## I got an account, now what?
+Great! Now you can explore the fediverse! Open the login page for your Pleroma instance (e.g. <https://pleroma.soykaf.com>) and login with your username and password. (If you don't have an account yet, click on Register)
+
+### Pleroma-FE
+The default front-end used by Pleroma is Pleroma-FE. You can find more information on what it is and how to use it in the [Introduction to Pleroma-FE](../frontend).
+
+### Mastodon interface
+If the Pleroma interface isn't your thing, or you're just trying something new but you want to keep using the familiar Mastodon interface, we got that too!
+Just add a "/web" after your instance url (e.g. <https://pleroma.soycaf.com/web>) and you'll end on the Mastodon web interface, but with a Pleroma backend! MAGIC!
+The Mastodon interface is from the Glitch-soc fork. For more information on the Mastodon interface you can check the [Mastodon](https://docs.joinmastodon.org/) and [Glitch-soc](https://glitch-soc.github.io/docs/) documentation.
+
+Remember, what you see is only the frontend part of Mastodon, the backend is still Pleroma.

docs/installation/alpine_linux_en.md

@@ -225,10 +225,7 @@ sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress
 
 #### Further reading
 
-* [Backup your instance](../administration/backup.md)
+{! backend/installation/further_reading.include !}
-* [Hardening your instance](../configuration/hardening.md)
-* [How to activate mediaproxy](../configuration/howto_mediaproxy.md)
-* [Updating your instance](../administration/updating.md)
 
 ## Questions
 

docs/installation/arch_linux_en.md

@@ -200,10 +200,7 @@ sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress
 
 #### Further reading
 
-* [Backup your instance](../administration/backup.md)
+{! backend/installation/further_reading.include !}
-* [Hardening your instance](../configuration/hardening.md)
-* [How to activate mediaproxy](../configuration/howto_mediaproxy.md)
-* [Updating your instance](../administration/updating.md)
 
 ## Questions
 

docs/installation/debian_based_en.md

@@ -186,10 +186,7 @@ sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress
 
 #### Further reading
 
-* [Backup your instance](../administration/backup.md)
+{! backend/installation/further_reading.include !}
-* [Hardening your instance](../configuration/hardening.md)
-* [How to activate mediaproxy](../configuration/howto_mediaproxy.md)
-* [Updating your instance](../administration/updating.md)
 
 ## Questions
 

docs/installation/debian_based_jp.md

@@ -175,10 +175,7 @@ sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress
 
 #### その他の設定とカスタマイズ
 
-* [Backup your instance](../administration/backup.md)
+{! backend/installation/further_reading.include !}
-* [Hardening your instance](../configuration/hardening.md)
-* [How to activate mediaproxy](../configuration/howto_mediaproxy.md)
-* [Updating your instance](../administration/updating.md)
 
 ## 質問ある？
 

docs/installation/further_reading.include

@@ -0,0 +1,5 @@
+* [How Federation Works/Why is my Federated Timeline empty?](https://blog.soykaf.com/post/how-federation-works/)
+* [Backup your instance](../administration/backup.md)
+* [Updating your instance](../administration/updating.md)
+* [Hardening your instance](../configuration/hardening.md)
+* [How to activate mediaproxy](../configuration/howto_mediaproxy.md)

docs/installation/gentoo_en.md

@@ -283,10 +283,7 @@ If you opted to allow sudo for the `pleroma` user but would like to remove the a
 
 #### Further reading
 
-* [Backup your instance](../administration/backup.md)
+{! backend/installation/further_reading.include !}
-* [Hardening your instance](../configuration/hardening.md)
-* [How to activate mediaproxy](../configuration/howto_mediaproxy.md)
-* [Updating your instance](../administration/updating.md)
 
 ## Questions
 

docs/installation/netbsd_en.md

@@ -196,3 +196,11 @@ incorrect timestamps. You should have ntpd running.
 ## Instances running NetBSD
 
 * <https://catgirl.science>
+
+#### Further reading
+
+{! backend/installation/further_reading.include !}
+
+## Questions
+
+Questions about the installation or didn’t it work as it should be, ask in [#pleroma:matrix.org](https://matrix.heldscal.la/#/room/#freenode_#pleroma:matrix.org) or IRC Channel **#pleroma** on **Freenode**.

docs/installation/openbsd_en.md

@@ -242,3 +242,11 @@ If your instance is up and running, you can create your first user with administ
 ```
 LC_ALL=en_US.UTF-8 MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress> --admin
 ```
+
+#### Further reading
+
+{! backend/installation/further_reading.include !}
+
+## Questions
+
+Questions about the installation or didn’t it work as it should be, ask in [#pleroma:matrix.org](https://matrix.heldscal.la/#/room/#freenode_#pleroma:matrix.org) or IRC Channel **#pleroma** on **Freenode**.

docs/installation/otp_en.md

@@ -270,10 +270,7 @@ This will create an account withe the username of 'joeuser' with the email addre
 
 ## Further reading
 
-* [Backup your instance](../administration/backup.md)
+{! backend/installation/further_reading.include !}
-* [Hardening your instance](../configuration/hardening.md)
-* [How to activate mediaproxy](../configuration/howto_mediaproxy.md)
-* [Updating your instance](../administration/updating.md)
 
 ## Questions
 

docs/introduction.md

@@ -1,65 +0,0 @@
-# Introduction to Pleroma
-## What is Pleroma?
-Pleroma is a federated social networking platform, compatible with GNU social, Mastodon and other OStatus and ActivityPub implementations. It is free software licensed under the AGPLv3.
-It actually consists of two components: a backend, named simply Pleroma, and a user-facing frontend, named Pleroma-FE. It also includes the Mastodon frontend, if that's your thing.
-It's part of what we call the fediverse, a federated network of instances which speak common protocols and can communicate with each other.
-One account on an instance is enough to talk to the entire fediverse!
-
-## How can I use it?
-
-Pleroma instances are already widely deployed, a list can be found at <http://distsn.org/pleroma-instances.html>. Information on all existing fediverse instances can be found at <https://fediverse.network/>.
-
-If you don't feel like joining an existing instance, but instead prefer to deploy your own instance, that's easy too!
-Installation instructions can be found in the installation section of these docs.
-
-## I got an account, now what?
-Great! Now you can explore the fediverse! Open the login page for your Pleroma instance (e.g. <https://pleroma.soykaf.com>) and login with your username and password. (If you don't have an account yet, click on Register)
-
-At this point you will have two columns in front of you.
-
-### Left column
-
-- first block: here you can see your avatar, your nickname and statistics (Statuses, Following, Followers). Clicking your profile pic will open your profile.
-Under that you have a text form which allows you to post new statuses. The number on the bottom of the text form is a character counter, every instance can have a different character limit (the default is 5000).
-If you want to mention someone, type @ + name of the person. A drop-down menu will help you in finding the right person.
-Under the text form there are also several visibility options and there is the option to use rich text.
-Under that the icon on the left is for uploading media files and attach them to your post. There is also an emoji-picker and an option to post a poll.
-To post your status, simply press Submit.
-On the top right you will also see a wrench icon. This opens your personal settings.
-
-- second block: Here you can switch between the different timelines:
-   - Timeline: all the people that you follow
-   - Interactions: here you can switch between different timelines where there was interaction with your account. There is Mentions, Repeats and Favorites, and New follows
-   - Direct Messages: these are the Direct Messages sent to you
-   - Public Timeline: all the statutes from the local instance
-   - The Whole Known Network: all public posts the instance knows about, both local and remote!
-   - About: This isn't a Timeline but shows relevant info about the instance. You can find a list of the moderators and admins, Terms of Service, MRF policies and enabled features.
-- Optional third block: This is the Instance panel that can be activated, but is deactivated by default. It's fully customisable and by default has links to the pleroma-fe and Mastodon-fe.
-- fourth block: This is the Notifications block, here you will get notified whenever somebody mentions you, follows you, repeats or favorites one of your statuses.
-
-### Right column
-This is where the interesting stuff happens!
-Depending on the timeline you will see different statuses, but each status has a standard structure:
-
-- Profile pic, name and link to profile. An optional left-arrow if it's a reply to another status (hovering will reveal the reply-to status). Clicking on the profile pic will uncollapse the user's profile.
-- A `+` button on the right allows you to Expand/Collapse an entire discussion thread. It also updates in realtime!
-- An arrow icon allows you to open the status on the instance where it's originating from.
-- The text of the status, including mentions and attachements. If you click on a mention, it will automatically open the profile page of that person.
-- Three buttons (left to right): Reply, Repeat, Favorite. There is also a forth button, this is a dropdown menu for simple moderation like muting the conversation or, if you have moderation rights, delete the status from the server.
-
-### Top right
-
-- The magnifier icon opens the search screen where you can search for statuses, people and hashtags. It's also possible to import statusses from remote servers by pasting the url to the post in the search field.
-- The gear icon gives you general settings
-- If you have admin rights, you'll see an icon that opens the admin interface
-- The last icon is to log out
-
-### Bottom right
-On the bottom right you have a chatbox. Here you can communicate with people on the same instance in realtime. It is local-only, for now, but there are plans to make it extendable to the entire fediverse!
-
-### Mastodon interface
-If the Pleroma interface isn't your thing, or you're just trying something new but you want to keep using the familiar Mastodon interface, we got that too!
-Just add a "/web" after your instance url (e.g. <https://pleroma.soycaf.com/web>) and you'll end on the Mastodon web interface, but with a Pleroma backend! MAGIC!
-The Mastodon interface is from the Glitch-soc fork. For more information on the Mastodon interface you can check the [Mastodon](https://docs.joinmastodon.org/) and [Glitch-soc](https://glitch-soc.github.io/docs/) documentation.
-
-Remember, what you see is only the frontend part of Mastodon, the backend is still Pleroma.

installation/nginx-cache-purge.sh.example

@@ -13,7 +13,7 @@ CACHE_DIRECTORY="/tmp/pleroma-media-cache"
 ## $3 - (optional) the number of parallel processes to run for grep.
 get_cache_files() {
     local max_parallel=${3-16}
-    find $2 -maxdepth 2 -type d | xargs -P $max_parallel -n 1 grep -E Rl "^KEY:.*$1" | sort -u
+    find $2 -maxdepth 2 -type d | xargs -P $max_parallel -n 1 grep -E -Rl "^KEY:.*$1" | sort -u
 }
 
 ## Removes an item from the given cache zone.
@@ -37,4 +37,4 @@ purge() {
 
 }
 
-purge $1
+purge $@

lib/mix/tasks/pleroma/config.ex

@@ -52,6 +52,7 @@ def migrate_to_db(file_path \\ nil) do
 
   defp do_migrate_to_db(config_file) do
     if File.exists?(config_file) do
+      shell_info("Migrating settings from file: #{Path.expand(config_file)}")
       Ecto.Adapters.SQL.query!(Repo, "TRUNCATE config;")
       Ecto.Adapters.SQL.query!(Repo, "ALTER SEQUENCE config_id_seq RESTART;")
 

lib/mix/tasks/pleroma/instance.ex

@@ -145,7 +145,7 @@ def run(["gen" | rest]) do
           options,
           :uploads_dir,
           "What directory should media uploads go in (when using the local uploader)?",
-          Pleroma.Config.get([Pleroma.Uploaders.Local, :uploads])
+          Config.get([Pleroma.Uploaders.Local, :uploads])
         )
         |> Path.expand()
 
@@ -154,7 +154,7 @@ def run(["gen" | rest]) do
           options,
           :static_dir,
           "What directory should custom public files be read from (custom emojis, frontend bundle overrides, robots.txt, etc.)?",
-          Pleroma.Config.get([:instance, :static_dir])
+          Config.get([:instance, :static_dir])
         )
         |> Path.expand()
 

lib/mix/tasks/pleroma/refresh_counter_cache.ex

@@ -17,30 +17,53 @@ defmodule Mix.Tasks.Pleroma.RefreshCounterCache do
   def run([]) do
     Mix.Pleroma.start_pleroma()
 
-    ["public", "unlisted", "private", "direct"]
+    instances =
-    |> Enum.each(fn visibility ->
+      Activity
-      count = status_visibility_count_query(visibility)
+      |> distinct([a], true)
-      name = "status_visibility_#{visibility}"
+      |> select([a], fragment("split_part(?, '/', 3)", a.actor))
-      CounterCache.set(name, count)
+      |> Repo.all()
-      Mix.Pleroma.shell_info("Set #{name} to #{count}")
+
+    instances
+    |> Enum.with_index(1)
+    |> Enum.each(fn {instance, i} ->
+      counters = instance_counters(instance)
+      CounterCache.set(instance, counters)
+
+      Mix.Pleroma.shell_info(
+        "[#{i}/#{length(instances)}] Setting #{instance} counters: #{inspect(counters)}"
+      )
     end)
 
     Mix.Pleroma.shell_info("Done")
   end
 
-  defp status_visibility_count_query(visibility) do
+  defp instance_counters(instance) do
+    counters = %{"public" => 0, "unlisted" => 0, "private" => 0, "direct" => 0}
+
     Activity
-    |> where(
+    |> where([a], fragment("(? ->> 'type'::text) = 'Create'", a.data))
+    |> where([a], fragment("split_part(?, '/', 3) = ?", a.actor, ^instance))
+    |> select(
+      [a],
+      {fragment(
+         "activity_visibility(?, ?, ?)",
+         a.actor,
+         a.recipients,
+         a.data
+       ), count(a.id)}
+    )
+    |> group_by(
       [a],
       fragment(
-        "activity_visibility(?, ?, ?) = ?",
+        "activity_visibility(?, ?, ?)",
         a.actor,
         a.recipients,
-        a.data,
+        a.data
-        ^visibility
       )
     )
-    |> where([a], fragment("(? ->> 'type'::text) = 'Create'", a.data))
+    |> Repo.all(timeout: :timer.minutes(30))
-    |> Repo.aggregate(:count, :id, timeout: :timer.minutes(30))
+    |> Enum.reduce(counters, fn {visibility, count}, acc ->
+      Map.put(acc, visibility, count)
+    end)
   end
 end

lib/pleroma/application.ex

@@ -35,11 +35,11 @@ def user_agent do
   # See http://elixir-lang.org/docs/stable/elixir/Application.html
   # for more information on OTP Applications
   def start(_type, _args) do
-    Pleroma.Config.Holder.save_default()
+    Config.Holder.save_default()
     Pleroma.HTML.compile_scrubbers()
     Config.DeprecationWarnings.warn()
     Pleroma.Plugs.HTTPSecurityPlug.warn_if_disabled()
-    Pleroma.Repo.check_migrations_applied!()
+    Pleroma.ApplicationRequirements.verify!()
     setup_instrumenters()
     load_custom_modules()
 
@@ -148,7 +148,8 @@ defp cachex_children do
       build_cachex("idempotency", expiration: idempotency_expiration(), limit: 2500),
       build_cachex("web_resp", limit: 2500),
       build_cachex("emoji_packs", expiration: emoji_packs_expiration(), limit: 10),
-      build_cachex("failed_proxy_url", limit: 2500)
+      build_cachex("failed_proxy_url", limit: 2500),
+      build_cachex("banned_urls", default_ttl: :timer.hours(24 * 30), limit: 5_000)
     ]
   end
 

lib/pleroma/application_requirements.ex

@@ -0,0 +1,107 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.ApplicationRequirements do
+  @moduledoc """
+  The module represents the collection of validations to runs before start server.
+  """
+
+  defmodule VerifyError, do: defexception([:message])
+
+  import Ecto.Query
+
+  require Logger
+
+  @spec verify!() :: :ok | VerifyError.t()
+  def verify! do
+    :ok
+    |> check_migrations_applied!()
+    |> check_rum!()
+    |> handle_result()
+  end
+
+  defp handle_result(:ok), do: :ok
+  defp handle_result({:error, message}), do: raise(VerifyError, message: message)
+
+  # Checks for pending migrations.
+  #
+  def check_migrations_applied!(:ok) do
+    unless Pleroma.Config.get(
+             [:i_am_aware_this_may_cause_data_loss, :disable_migration_check],
+             false
+           ) do
+      {_, res, _} =
+        Ecto.Migrator.with_repo(Pleroma.Repo, fn repo ->
+          down_migrations =
+            Ecto.Migrator.migrations(repo)
+            |> Enum.reject(fn
+              {:up, _, _} -> true
+              {:down, _, _} -> false
+            end)
+
+          if length(down_migrations) > 0 do
+            down_migrations_text =
+              Enum.map(down_migrations, fn {:down, id, name} -> "- #{name} (#{id})\n" end)
+
+            Logger.error(
+              "The following migrations were not applied:\n#{down_migrations_text}If you want to start Pleroma anyway, set\nconfig :pleroma, :i_am_aware_this_may_cause_data_loss, disable_migration_check: true"
+            )
+
+            {:error, "Unapplied Migrations detected"}
+          else
+            :ok
+          end
+        end)
+
+      res
+    else
+      :ok
+    end
+  end
+
+  def check_migrations_applied!(result), do: result
+
+  # Checks for settings of RUM indexes.
+  #
+  defp check_rum!(:ok) do
+    {_, res, _} =
+      Ecto.Migrator.with_repo(Pleroma.Repo, fn repo ->
+        migrate =
+          from(o in "columns",
+            where: o.table_name == "objects",
+            where: o.column_name == "fts_content"
+          )
+          |> repo.exists?(prefix: "information_schema")
+
+        setting = Pleroma.Config.get([:database, :rum_enabled], false)
+
+        do_check_rum!(setting, migrate)
+      end)
+
+    res
+  end
+
+  defp check_rum!(result), do: result
+
+  defp do_check_rum!(setting, migrate) do
+    case {setting, migrate} do
+      {true, false} ->
+        Logger.error(
+          "Use `RUM` index is enabled, but were not applied migrations for it.\nIf you want to start Pleroma anyway, set\nconfig :pleroma, :database, rum_enabled: false\nOtherwise apply the following migrations:\n`mix ecto.migrate --migrations-path priv/repo/optional_migrations/rum_indexing/`"
+        )
+
+        {:error, "Unapplied RUM Migrations detected"}
+
+      {false, true} ->
+        Logger.error(
+          "Detected applied migrations to use `RUM` index, but `RUM` isn't enable in settings.\nIf you want to use `RUM`, set\nconfig :pleroma, :database, rum_enabled: true\nOtherwise roll `RUM` migrations back.\n`mix ecto.rollback --migrations-path priv/repo/optional_migrations/rum_indexing/`"
+        )
+
+        {:error, "RUM Migrations detected"}
+
+      _ ->
+        :ok
+    end
+  end
+end

lib/pleroma/config/config_db.ex

@@ -167,7 +167,9 @@ defp only_full_update?(%ConfigDB{group: group, key: key}) do
     end)
   end
 
-  @spec delete(map()) :: {:ok, ConfigDB.t()} | {:error, Changeset.t()}
+  @spec delete(ConfigDB.t() | map()) :: {:ok, ConfigDB.t()} | {:error, Changeset.t()}
+  def delete(%ConfigDB{} = config), do: Repo.delete(config)
+
   def delete(params) do
     search_opts = Map.delete(params, :subkeys)
 

lib/pleroma/config/deprecation_warnings.ex

@@ -3,9 +3,23 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Config.DeprecationWarnings do
+  alias Pleroma.Config
+
   require Logger
   alias Pleroma.Config
 
+  @type config_namespace() :: [atom()]
+  @type config_map() :: {config_namespace(), config_namespace(), String.t()}
+
+  @mrf_config_map [
+    {[:instance, :rewrite_policy], [:mrf, :policies],
+     "\n* `config :pleroma, :instance, rewrite_policy` is now `config :pleroma, :mrf, policies`"},
+    {[:instance, :mrf_transparency], [:mrf, :transparency],
+     "\n* `config :pleroma, :instance, mrf_transparency` is now `config :pleroma, :mrf, transparency`"},
+    {[:instance, :mrf_transparency_exclusions], [:mrf, :transparency_exclusions],
+     "\n* `config :pleroma, :instance, mrf_transparency_exclusions` is now `config :pleroma, :mrf, transparency_exclusions`"}
+  ]
+
   def check_hellthread_threshold do
     if Config.get([:mrf_hellthread, :threshold]) do
       Logger.warn("""
@@ -39,5 +53,35 @@ def mrf_user_allowlist do
   def warn do
     check_hellthread_threshold()
     mrf_user_allowlist()
+    check_old_mrf_config()
+  end
+
+  def check_old_mrf_config do
+    warning_preface = """
+    !!!DEPRECATION WARNING!!!
+    Your config is using old namespaces for MRF configuration. They should work for now, but you are advised to change to new namespaces to prevent possible issues later:
+    """
+
+    move_namespace_and_warn(@mrf_config_map, warning_preface)
+  end
+
+  @spec move_namespace_and_warn([config_map()], String.t()) :: :ok
+  def move_namespace_and_warn(config_map, warning_preface) do
+    warning =
+      Enum.reduce(config_map, "", fn
+        {old, new, err_msg}, acc ->
+          old_config = Config.get(old)
+
+          if old_config do
+            Config.put(new, old_config)
+            acc <> err_msg
+          else
+            acc
+          end
+      end)
+
+    if warning != "" do
+      Logger.warn(warning_preface <> warning)
+    end
   end
 end

lib/pleroma/config/loader.ex

@@ -12,6 +12,11 @@ defmodule Pleroma.Config.Loader do
     :swarm
   ]
 
+  @reject_groups [
+    :postgrex,
+    :tesla
+  ]
+
   if Code.ensure_loaded?(Config.Reader) do
     @reader Config.Reader
 
@@ -47,7 +52,8 @@ defp filter(configs) do
   @spec filter_group(atom(), keyword()) :: keyword()
   def filter_group(group, configs) do
     Enum.reject(configs[group], fn {key, _v} ->
-      key in @reject_keys or (group == :phoenix and key == :serve_endpoints) or group == :postgrex
+      key in @reject_keys or group in @reject_groups or
+        (group == :phoenix and key == :serve_endpoints)
     end)
   end
 end

lib/pleroma/counter_cache.ex

@@ -10,32 +10,70 @@ defmodule Pleroma.CounterCache do
   import Ecto.Query
 
   schema "counter_cache" do
-    field(:name, :string)
+    field(:instance, :string)
-    field(:count, :integer)
+    field(:public, :integer)
+    field(:unlisted, :integer)
+    field(:private, :integer)
+    field(:direct, :integer)
   end
 
   def changeset(struct, params) do
     struct
-    |> cast(params, [:name, :count])
+    |> cast(params, [:instance, :public, :unlisted, :private, :direct])
-    |> validate_required([:name])
+    |> validate_required([:instance])
-    |> unique_constraint(:name)
+    |> unique_constraint(:instance)
   end
 
-  def get_as_map(names) when is_list(names) do
+  def get_by_instance(instance) do
     CounterCache
-    |> where([cc], cc.name in ^names)
+    |> select([c], %{
-    |> Repo.all()
+      "public" => c.public,
-    |> Enum.group_by(& &1.name, & &1.count)
+      "unlisted" => c.unlisted,
-    |> Map.new(fn {k, v} -> {k, hd(v)} end)
+      "private" => c.private,
+      "direct" => c.direct
+    })
+    |> where([c], c.instance == ^instance)
+    |> Repo.one()
+    |> case do
+      nil -> %{"public" => 0, "unlisted" => 0, "private" => 0, "direct" => 0}
+      val -> val
+    end
   end
 
-  def set(name, count) do
+  def get_sum do
+    CounterCache
+    |> select([c], %{
+      "public" => type(sum(c.public), :integer),
+      "unlisted" => type(sum(c.unlisted), :integer),
+      "private" => type(sum(c.private), :integer),
+      "direct" => type(sum(c.direct), :integer)
+    })
+    |> Repo.one()
+  end
+
+  def set(instance, values) do
+    params =
+      Enum.reduce(
+        ["public", "private", "unlisted", "direct"],
+        %{"instance" => instance},
+        fn param, acc ->
+          Map.put_new(acc, param, Map.get(values, param, 0))
+        end
+      )
+
     %CounterCache{}
-    |> changeset(%{"name" => name, "count" => count})
+    |> changeset(params)
     |> Repo.insert(
-      on_conflict: [set: [count: count]],
+      on_conflict: [
+        set: [
+          public: params["public"],
+          private: params["private"],
+          unlisted: params["unlisted"],
+          direct: params["direct"]
+        ]
+      ],
       returning: true,
-      conflict_target: :name
+      conflict_target: :instance
     )
   end
 end

lib/pleroma/emails/admin_email.ex

@@ -10,7 +10,7 @@ defmodule Pleroma.Emails.AdminEmail do
   alias Pleroma.Config
   alias Pleroma.Web.Router.Helpers
 
-  defp instance_config, do: Pleroma.Config.get(:instance)
+  defp instance_config, do: Config.get(:instance)
   defp instance_name, do: instance_config()[:name]
 
   defp instance_notify_email do
@@ -72,6 +72,8 @@ def report(to, reporter, account, statuses, comment) do
     <p>Reported Account: <a href="#{user_url(account)}">#{account.nickname}</a></p>
     #{comment_html}
     #{statuses_html}
+    <p>
+    <a href="#{Pleroma.Web.base_url()}/pleroma/admin/#/reports/index">View Reports in AdminFE</a>
     """
 
     new()

lib/pleroma/emoji/loader.ex

@@ -108,7 +108,7 @@ defp load_pack(pack_dir, emoji_groups) do
       if File.exists?(emoji_txt) do
         load_from_file(emoji_txt, emoji_groups)
       else
-        extensions = Pleroma.Config.get([:emoji, :pack_extensions])
+        extensions = Config.get([:emoji, :pack_extensions])
 
         Logger.info(
           "No emoji.txt found for pack \"#{pack_name}\", assuming all #{

lib/pleroma/emoji/pack.ex

@@ -1,6 +1,7 @@
 defmodule Pleroma.Emoji.Pack do
-  @derive {Jason.Encoder, only: [:files, :pack]}
+  @derive {Jason.Encoder, only: [:files, :pack, :files_count]}
   defstruct files: %{},
+            files_count: 0,
             pack_file: nil,
             path: nil,
             pack: %{},
@@ -8,6 +9,7 @@ defmodule Pleroma.Emoji.Pack do
 
   @type t() :: %__MODULE__{
           files: %{String.t() => Path.t()},
+          files_count: non_neg_integer(),
           pack_file: Path.t(),
           path: Path.t(),
           pack: map(),
@@ -16,7 +18,7 @@ defmodule Pleroma.Emoji.Pack do
 
   alias Pleroma.Emoji
 
-  @spec create(String.t()) :: :ok | {:error, File.posix()} | {:error, :empty_values}
+  @spec create(String.t()) :: {:ok, t()} | {:error, File.posix()} | {:error, :empty_values}
   def create(name) do
     with :ok <- validate_not_empty([name]),
          dir <- Path.join(emoji_path(), name),
@@ -26,10 +28,28 @@ def create(name) do
     end
   end
 
-  @spec show(String.t()) :: {:ok, t()} | {:error, atom()}
+  defp paginate(entities, 1, page_size), do: Enum.take(entities, page_size)
-  def show(name) do
+
+  defp paginate(entities, page, page_size) do
+    entities
+    |> Enum.chunk_every(page_size)
+    |> Enum.at(page - 1)
+  end
+
+  @spec show(keyword()) :: {:ok, t()} | {:error, atom()}
+  def show(opts) do
+    name = opts[:name]
+
     with :ok <- validate_not_empty([name]),
          {:ok, pack} <- load_pack(name) do
+      shortcodes =
+        pack.files
+        |> Map.keys()
+        |> Enum.sort()
+        |> paginate(opts[:page], opts[:page_size])
+
+      pack = Map.put(pack, :files, Map.take(pack.files, shortcodes))
+
       {:ok, validate_pack(pack)}
     end
   end
@@ -120,10 +140,10 @@ def list_remote(url) do
     end
   end
 
-  @spec list_local() :: {:ok, map()}
+  @spec list_local(keyword()) :: {:ok, map(), non_neg_integer()}
-  def list_local do
+  def list_local(opts) do
     with {:ok, results} <- list_packs_dir() do
-      packs =
+      all_packs =
         results
         |> Enum.map(fn name ->
           case load_pack(name) do
@@ -132,9 +152,13 @@ def list_local do
           end
         end)
         |> Enum.reject(&is_nil/1)
+
+      packs =
+        all_packs
+        |> paginate(opts[:page], opts[:page_size])
         |> Map.new(fn pack -> {pack.name, validate_pack(pack)} end)
 
-      {:ok, packs}
+      {:ok, packs, length(all_packs)}
     end
   end
 
@@ -146,7 +170,7 @@ def get_archive(name) do
     end
   end
 
-  @spec download(String.t(), String.t(), String.t()) :: :ok | {:error, atom()}
+  @spec download(String.t(), String.t(), String.t()) :: {:ok, t()} | {:error, atom()}
   def download(name, url, as) do
     uri = url |> String.trim() |> URI.parse()
 
@@ -197,7 +221,12 @@ def load_pack(name) do
         |> Map.put(:path, Path.dirname(pack_file))
         |> Map.put(:name, name)
 
-      {:ok, pack}
+      files_count =
+        pack.files
+        |> Map.keys()
+        |> length()
+
+      {:ok, Map.put(pack, :files_count, files_count)}
     else
       {:error, :not_found}
     end
@@ -296,7 +325,9 @@ defp downloadable?(pack) do
     # Otherwise, they'd have to download it from external-src
     pack.pack["share-files"] &&
       Enum.all?(pack.files, fn {_, file} ->
-        File.exists?(Path.join(pack.path, file))
+        pack.path
+        |> Path.join(file)
+        |> File.exists?()
       end)
   end
 
@@ -440,7 +471,7 @@ defp list_packs_dir do
     # with the API so it should be sufficient
     with {:create_dir, :ok} <- {:create_dir, File.mkdir_p(emoji_path)},
          {:ls, {:ok, results}} <- {:ls, File.ls(emoji_path)} do
-      {:ok, results}
+      {:ok, Enum.sort(results)}
     else
       {:create_dir, {:error, e}} -> {:error, :create_dir, e}
       {:ls, {:error, e}} -> {:error, :ls, e}

lib/pleroma/filter.ex

@@ -34,10 +34,18 @@ def get(id, %{id: user_id} = _user) do
     Repo.one(query)
   end
 
-  def get_filters(%User{id: user_id} = _user) do
+  def get_active(query) do
+    from(f in query, where: is_nil(f.expires_at) or f.expires_at > ^NaiveDateTime.utc_now())
+  end
+
+  def get_irreversible(query) do
+    from(f in query, where: f.hide)
+  end
+
+  def get_filters(query \\ __MODULE__, %User{id: user_id}) do
     query =
       from(
-        f in Pleroma.Filter,
+        f in query,
         where: f.user_id == ^user_id,
         order_by: [desc: :id]
       )
@@ -95,4 +103,34 @@ def update(%Pleroma.Filter{} = filter, params) do
     |> validate_required([:phrase, :context])
     |> Repo.update()
   end
+
+  def compose_regex(user_or_filters, format \\ :postgres)
+
+  def compose_regex(%User{} = user, format) do
+    __MODULE__
+    |> get_active()
+    |> get_irreversible()
+    |> get_filters(user)
+    |> compose_regex(format)
+  end
+
+  def compose_regex([_ | _] = filters, format) do
+    phrases =
+      filters
+      |> Enum.map(& &1.phrase)
+      |> Enum.join("|")
+
+    case format do
+      :postgres ->
+        "\\y(#{phrases})\\y"
+
+      :re ->
+        ~r/\b#{phrases}\b/i
+
+      _ ->
+        nil
+    end
+  end
+
+  def compose_regex(_, _), do: nil
 end

lib/pleroma/following_relationship.ex

@@ -124,6 +124,7 @@ def get_follow_requests(%User{id: id}) do
     |> join(:inner, [r], f in assoc(r, :follower))
     |> where([r], r.state == ^:follow_pending)
     |> where([r], r.following_id == ^id)
+    |> where([r, f], f.deactivated != true)
     |> select([r, f], f)
     |> Repo.all()
   end

lib/pleroma/html.ex

@@ -109,7 +109,7 @@ def extract_first_external_url(object, content) do
       result =
         content
         |> Floki.parse_fragment!()
-        |> Floki.filter_out("a.mention,a.hashtag,a[rel~=\"tag\"]")
+        |> Floki.filter_out("a.mention,a.hashtag,a.attachment,a[rel~=\"tag\"]")
         |> Floki.attribute("a", "href")
         |> Enum.at(0)
 

lib/pleroma/http/ex_aws.ex

@@ -0,0 +1,22 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.HTTP.ExAws do
+  @moduledoc false
+
+  @behaviour ExAws.Request.HttpClient
+
+  alias Pleroma.HTTP
+
+  @impl true
+  def request(method, url, body \\ "", headers \\ [], http_opts \\ []) do
+    case HTTP.request(method, url, body, headers, http_opts) do
+      {:ok, env} ->
+        {:ok, %{status_code: env.status, headers: env.headers, body: env.body}}
+
+      {:error, reason} ->
+        {:error, %{reason: reason}}
+    end
+  end
+end

lib/pleroma/http/http.ex

@@ -16,6 +16,7 @@ defmodule Pleroma.HTTP do
   require Logger
 
   @type t :: __MODULE__
+  @type method() :: :get | :post | :put | :delete | :head
 
   @doc """
   Performs GET request.
@@ -28,6 +29,9 @@ def get(url, headers \\ [], options \\ [])
   def get(nil, _, _), do: nil
   def get(url, headers, options), do: request(:get, url, "", headers, options)
 
+  @spec head(Request.url(), Request.headers(), keyword()) :: {:ok, Env.t()} | {:error, any()}
+  def head(url, headers \\ [], options \\ []), do: request(:head, url, "", headers, options)
+
   @doc """
   Performs POST request.
 
@@ -42,7 +46,7 @@ def post(url, body, headers \\ [], options \\ []),
   Builds and performs http request.
 
   # Arguments:
-  `method` - :get, :post, :put, :delete
+  `method` - :get, :post, :put, :delete, :head
   `url` - full url
   `body` - request body
   `headers` - a keyworld list of headers, e.g. `[{"content-type", "text/plain"}]`
@@ -52,7 +56,7 @@ def post(url, body, headers \\ [], options \\ []),
   `{:ok, %Tesla.Env{}}` or `{:error, error}`
 
   """
-  @spec request(atom(), Request.url(), String.t(), Request.headers(), keyword()) ::
+  @spec request(method(), Request.url(), String.t(), Request.headers(), keyword()) ::
           {:ok, Env.t()} | {:error, any()}
   def request(method, url, body, headers, options) when is_binary(url) do
     uri = URI.parse(url)

lib/pleroma/http/tzdata.ex

@@ -0,0 +1,25 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.HTTP.Tzdata do
+  @moduledoc false
+
+  @behaviour Tzdata.HTTPClient
+
+  alias Pleroma.HTTP
+
+  @impl true
+  def get(url, headers, options) do
+    with {:ok, %Tesla.Env{} = env} <- HTTP.get(url, headers, options) do
+      {:ok, {env.status, env.headers, env.body}}
+    end
+  end
+
+  @impl true
+  def head(url, headers, options) do
+    with {:ok, %Tesla.Env{} = env} <- HTTP.head(url, headers, options) do
+      {:ok, {env.status, env.headers}}
+    end
+  end
+end

lib/pleroma/instances/instance.ex

@@ -17,6 +17,8 @@ defmodule Pleroma.Instances.Instance do
   schema "instances" do
     field(:host, :string)
     field(:unreachable_since, :naive_datetime_usec)
+    field(:favicon, :string)
+    field(:favicon_updated_at, :naive_datetime)
 
     timestamps()
   end
@@ -25,7 +27,7 @@ defmodule Pleroma.Instances.Instance do
 
   def changeset(struct, params \\ %{}) do
     struct
-    |> cast(params, [:host, :unreachable_since])
+    |> cast(params, [:host, :unreachable_since, :favicon, :favicon_updated_at])
     |> validate_required([:host])
     |> unique_constraint(:host)
   end
@@ -120,4 +122,48 @@ defp parse_datetime(datetime) when is_binary(datetime) do
   end
 
   defp parse_datetime(datetime), do: datetime
+
+  def get_or_update_favicon(%URI{host: host} = instance_uri) do
+    existing_record = Repo.get_by(Instance, %{host: host})
+    now = NaiveDateTime.utc_now()
+
+    if existing_record && existing_record.favicon_updated_at &&
+         NaiveDateTime.diff(now, existing_record.favicon_updated_at) < 86_400 do
+      existing_record.favicon
+    else
+      favicon = scrape_favicon(instance_uri)
+
+      if existing_record do
+        existing_record
+        |> changeset(%{favicon: favicon, favicon_updated_at: now})
+        |> Repo.update()
+      else
+        %Instance{}
+        |> changeset(%{host: host, favicon: favicon, favicon_updated_at: now})
+        |> Repo.insert()
+      end
+
+      favicon
+    end
+  end
+
+  defp scrape_favicon(%URI{} = instance_uri) do
+    try do
+      with {:ok, %Tesla.Env{body: html}} <-
+             Pleroma.HTTP.get(to_string(instance_uri), [{:Accept, "text/html"}]),
+           favicon_rel <-
+             html
+             |> Floki.parse_document!()
+             |> Floki.attribute("link[rel=icon]", "href")
+             |> List.first(),
+           favicon <- URI.merge(instance_uri, favicon_rel) |> to_string(),
+           true <- is_binary(favicon) do
+        favicon
+      else
+        _ -> nil
+      end
+    rescue
+      _ -> nil
+    end
+  end
 end

lib/pleroma/migration_helper/notification_backfill.ex

@@ -3,7 +3,6 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.MigrationHelper.NotificationBackfill do
-  alias Pleroma.Notification
   alias Pleroma.Object
   alias Pleroma.Repo
   alias Pleroma.User
@@ -25,18 +24,27 @@ def fill_in_notification_types do
         |> type_from_activity()
 
       notification
-      |> Notification.changeset(%{type: type})
+      |> Ecto.Changeset.change(%{type: type})
       |> Repo.update()
     end)
   end
 
+  defp get_by_ap_id(ap_id) do
+    q =
+      from(u in User,
+        select: u.id
+      )
+
+    Repo.get_by(q, ap_id: ap_id)
+  end
+
   # This is copied over from Notifications to keep this stable.
   defp type_from_activity(%{data: %{"type" => type}} = activity) do
     case type do
       "Follow" ->
         accepted_function = fn activity ->
-          with %User{} = follower <- User.get_by_ap_id(activity.data["actor"]),
+          with %User{} = follower <- get_by_ap_id(activity.data["actor"]),
-               %User{} = followed <- User.get_by_ap_id(activity.data["object"]) do
+               %User{} = followed <- get_by_ap_id(activity.data["object"]) do
             Pleroma.FollowingRelationship.following?(follower, followed)
           end
         end

lib/pleroma/notification.ex

@@ -130,6 +130,7 @@ def for_user_query(user, opts \\ %{}) do
     |> preload([n, a, o], activity: {a, object: o})
     |> exclude_notification_muted(user, exclude_notification_muted_opts)
     |> exclude_blocked(user, exclude_blocked_opts)
+    |> exclude_filtered(user)
     |> exclude_visibility(opts)
   end
 
@@ -158,6 +159,20 @@ defp exclude_notification_muted(query, user, opts) do
     |> where([n, a, o, tm], is_nil(tm.user_id))
   end
 
+  defp exclude_filtered(query, user) do
+    case Pleroma.Filter.compose_regex(user) do
+      nil ->
+        query
+
+      regex ->
+        from([_n, a, o] in query,
+          where:
+            fragment("not(?->>'content' ~* ?)", o.data, ^regex) or
+              fragment("?->>'actor' = ?", o.data, ^user.ap_id)
+        )
+    end
+  end
+
   @valid_visibilities ~w[direct unlisted public private]
 
   defp exclude_visibility(query, %{exclude_visibilities: visibility})
@@ -337,6 +352,7 @@ def dismiss(%{id: user_id} = _user, id) do
     end
   end
 
+  @spec create_notifications(Activity.t(), keyword()) :: {:ok, [Notification.t()] | []}
   def create_notifications(activity, options \\ [])
 
   def create_notifications(%Activity{data: %{"to" => _, "type" => "Create"}} = activity, options) do
@@ -367,6 +383,7 @@ defp do_create_notifications(%Activity{} = activity, options) do
         do_send = do_send && user in enabled_receivers
         create_notification(activity, user, do_send)
       end)
+      |> Enum.reject(&is_nil/1)
 
     {:ok, notifications}
   end
@@ -480,6 +497,10 @@ def get_potential_receiver_ap_ids(%{data: %{"type" => type, "object" => object_i
     end
   end
 
+  def get_potential_receiver_ap_ids(%{data: %{"type" => "Follow", "object" => object_id}}) do
+    [object_id]
+  end
+
   def get_potential_receiver_ap_ids(activity) do
     []
     |> Utils.maybe_notify_to_recipients(activity)
@@ -554,7 +575,8 @@ def skip?(%Activity{} = activity, %User{} = user) do
       :follows,
       :non_followers,
       :non_follows,
-      :recently_followed
+      :recently_followed,
+      :filtered
     ]
     |> Enum.find(&skip?(&1, activity, user))
   end
@@ -623,6 +645,26 @@ def skip?(:recently_followed, %Activity{data: %{"type" => "Follow"}} = activity,
     end)
   end
 
+  def skip?(:filtered, %{data: %{"type" => type}}, _) when type in ["Follow", "Move"], do: false
+
+  def skip?(:filtered, activity, user) do
+    object = Object.normalize(activity)
+
+    cond do
+      is_nil(object) ->
+        false
+
+      object.data["actor"] == user.ap_id ->
+        false
+
+      not is_nil(regex = Pleroma.Filter.compose_regex(user, :re)) ->
+        Regex.match?(regex, object.data["content"])
+
+      true ->
+        false
+    end
+  end
+
   def skip?(_, _, _), do: false
 
   def for_user_and_activity(user, activity) do

lib/pleroma/object/fetcher.ex

@@ -83,8 +83,8 @@ def fetch_object_from_id(id, options \\ []) do
       {:transmogrifier, {:error, {:reject, nil}}} ->
         {:reject, nil}
 
-      {:transmogrifier, _} ->
+      {:transmogrifier, _} = e ->
-        {:error, "Transmogrifier failure."}
+        {:error, e}
 
       {:object, data, nil} ->
         reinject_object(%Object{}, data)

lib/pleroma/pagination.ex

@@ -64,6 +64,12 @@ def fetch_paginated(query, params, :offset, table_binding) do
   @spec paginate(Ecto.Query.t(), map(), type(), atom() | nil) :: [Ecto.Schema.t()]
   def paginate(query, options, method \\ :keyset, table_binding \\ nil)
 
+  def paginate(list, options, _method, _table_binding) when is_list(list) do
+    offset = options[:offset] || 0
+    limit = options[:limit] || 0
+    Enum.slice(list, offset, limit)
+  end
+
   def paginate(query, options, :keyset, table_binding) do
     query
     |> restrict(:min_id, options, table_binding)

lib/pleroma/plugs/http_security_plug.ex

@@ -69,10 +69,11 @@ defp csp_string do
     img_src = "img-src 'self' data: blob:"
     media_src = "media-src 'self'"
 
+    # Strict multimedia CSP enforcement only when MediaProxy is enabled
     {img_src, media_src} =
       if Config.get([:media_proxy, :enabled]) &&
            !Config.get([:media_proxy, :proxy_opts, :redirect_on_failure]) do
-        sources = get_proxy_and_attachment_sources()
+        sources = build_csp_multimedia_source_list()
         {[img_src, sources], [media_src, sources]}
       else
         {[img_src, " https:"], [media_src, " https:"]}
@@ -81,14 +82,14 @@ defp csp_string do
     connect_src = ["connect-src 'self' blob: ", static_url, ?\s, websocket_url]
 
     connect_src =
-      if Pleroma.Config.get(:env) == :dev do
+      if Config.get(:env) == :dev do
         [connect_src, " http://localhost:3035/"]
       else
         connect_src
       end
 
     script_src =
-      if Pleroma.Config.get(:env) == :dev do
+      if Config.get(:env) == :dev do
         "script-src 'self' 'unsafe-eval'"
       else
         "script-src 'self'"
@@ -107,29 +108,28 @@ defp csp_string do
     |> :erlang.iolist_to_binary()
   end
 
-  defp get_proxy_and_attachment_sources do
+  defp build_csp_multimedia_source_list do
     media_proxy_whitelist =
       Enum.reduce(Config.get([:media_proxy, :whitelist]), [], fn host, acc ->
         add_source(acc, host)
       end)
 
-    media_proxy_base_url =
+    media_proxy_base_url = build_csp_param(Config.get([:media_proxy, :base_url]))
-      if Config.get([:media_proxy, :base_url]),
-        do: URI.parse(Config.get([:media_proxy, :base_url])).host
 
-    upload_base_url =
+    upload_base_url = build_csp_param(Config.get([Pleroma.Upload, :base_url]))
-      if Config.get([Pleroma.Upload, :base_url]),
-        do: URI.parse(Config.get([Pleroma.Upload, :base_url])).host
 
-    s3_endpoint =
+    s3_endpoint = build_csp_param(Config.get([Pleroma.Uploaders.S3, :public_endpoint]))
-      if Config.get([Pleroma.Upload, :uploader]) == Pleroma.Uploaders.S3,
+
-        do: URI.parse(Config.get([Pleroma.Uploaders.S3, :public_endpoint])).host
+    captcha_method = Config.get([Pleroma.Captcha, :method])
+
+    captcha_endpoint = build_csp_param(Config.get([captcha_method, :endpoint]))
 
     []
     |> add_source(media_proxy_base_url)
     |> add_source(upload_base_url)
     |> add_source(s3_endpoint)
     |> add_source(media_proxy_whitelist)
+    |> add_source(captcha_endpoint)
   end
 
   defp add_source(iodata, nil), do: iodata
@@ -139,6 +139,16 @@ defp add_csp_param(csp_iodata, nil), do: csp_iodata
 
   defp add_csp_param(csp_iodata, param), do: [[param, ?;] | csp_iodata]
 
+  defp build_csp_param(nil), do: nil
+
+  defp build_csp_param(url) when is_binary(url) do
+    %{host: host, scheme: scheme} = URI.parse(url)
+
+    if scheme do
+      [scheme, "://", host]
+    end
+  end
+
   def warn_if_disabled do
     unless Config.get([:http_security, :enabled]) do
       Logger.warn("

lib/pleroma/plugs/static_fe_plug.ex

@@ -9,7 +9,7 @@ defmodule Pleroma.Plugs.StaticFEPlug do
   def init(options), do: options
 
   def call(conn, _) do
-    if enabled?() and accepts_html?(conn) do
+    if enabled?() and requires_html?(conn) do
       conn
       |> StaticFEController.call(:show)
       |> halt()
@@ -20,10 +20,7 @@ def call(conn, _) do
 
   defp enabled?, do: Pleroma.Config.get([:static_fe, :enabled], false)
 
-  defp accepts_html?(conn) do
+  defp requires_html?(conn) do
-    case get_req_header(conn, "accept") do
+    Phoenix.Controller.get_format(conn) == "html"
-      [accept | _] -> String.contains?(accept, "text/html")
-      _ -> false
-    end
   end
 end

lib/pleroma/plugs/uploaded_media.ex

@@ -10,6 +10,8 @@ defmodule Pleroma.Plugs.UploadedMedia do
   import Pleroma.Web.Gettext
   require Logger
 
+  alias Pleroma.Web.MediaProxy
+
   @behaviour Plug
   # no slashes
   @path "media"
@@ -35,8 +37,7 @@ def call(%{request_path: <<"/", @path, "/", file::binary>>} = conn, opts) do
         %{query_params: %{"name" => name}} = conn ->
           name = String.replace(name, "\"", "\\\"")
 
-          conn
+          put_resp_header(conn, "content-disposition", "filename=\"#{name}\"")
-          |> put_resp_header("content-disposition", "filename=\"#{name}\"")
 
         conn ->
           conn
@@ -47,7 +48,8 @@ def call(%{request_path: <<"/", @path, "/", file::binary>>} = conn, opts) do
 
     with uploader <- Keyword.fetch!(config, :uploader),
          proxy_remote = Keyword.get(config, :proxy_remote, false),
-         {:ok, get_method} <- uploader.get_file(file) do
+         {:ok, get_method} <- uploader.get_file(file),
+         false <- media_is_banned(conn, get_method) do
       get_media(conn, get_method, proxy_remote, opts)
     else
       _ ->
@@ -59,6 +61,14 @@ def call(%{request_path: <<"/", @path, "/", file::binary>>} = conn, opts) do
 
   def call(conn, _opts), do: conn
 
+  defp media_is_banned(%{request_path: path} = _conn, {:static_dir, _}) do
+    MediaProxy.in_banned_urls(Pleroma.Web.base_url() <> path)
+  end
+
+  defp media_is_banned(_, {:url, url}), do: MediaProxy.in_banned_urls(url)
+
+  defp media_is_banned(_, _), do: false
+
   defp get_media(conn, {:static_dir, directory}, _, opts) do
     static_opts =
       Map.get(opts, :static_plug_opts)

lib/pleroma/repo.ex

@@ -11,9 +11,7 @@ defmodule Pleroma.Repo do
   import Ecto.Query
   require Logger
 
-  defmodule Instrumenter do
+  defmodule Instrumenter, do: use(Prometheus.EctoInstrumenter)
-    use Prometheus.EctoInstrumenter
-  end
 
   @doc """
   Dynamically loads the repository url from the
@@ -51,35 +49,6 @@ def get_assoc(resource, association) do
     end
   end
 
-  def check_migrations_applied!() do
-    unless Pleroma.Config.get(
-             [:i_am_aware_this_may_cause_data_loss, :disable_migration_check],
-             false
-           ) do
-      Ecto.Migrator.with_repo(__MODULE__, fn repo ->
-        down_migrations =
-          Ecto.Migrator.migrations(repo)
-          |> Enum.reject(fn
-            {:up, _, _} -> true
-            {:down, _, _} -> false
-          end)
-
-        if length(down_migrations) > 0 do
-          down_migrations_text =
-            Enum.map(down_migrations, fn {:down, id, name} -> "- #{name} (#{id})\n" end)
-
-          Logger.error(
-            "The following migrations were not applied:\n#{down_migrations_text}If you want to start Pleroma anyway, set\nconfig :pleroma, :i_am_aware_this_may_cause_data_loss, disable_migration_check: true"
-          )
-
-          raise Pleroma.Repo.UnappliedMigrationsError
-        end
-      end)
-    else
-      :ok
-    end
-  end
-
   def chunk_stream(query, chunk_size) do
     # We don't actually need start and end funcitons of resource streaming,
     # but it seems to be the only way to not fetch records one-by-one and
@@ -107,7 +76,3 @@ def chunk_stream(query, chunk_size) do
     )
   end
 end
-
-defmodule Pleroma.Repo.UnappliedMigrationsError do
-  defexception message: "Unapplied Migrations detected"
-end

lib/pleroma/stats.ex

@@ -97,20 +97,11 @@ def calculate_stat_data do
     }
   end
 
-  def get_status_visibility_count do
+  def get_status_visibility_count(instance \\ nil) do
-    counter_cache =
+    if is_nil(instance) do
-      CounterCache.get_as_map([
+      CounterCache.get_sum()
-        "status_visibility_public",
+    else
-        "status_visibility_private",
+      CounterCache.get_by_instance(instance)
-        "status_visibility_unlisted",
+    end
-        "status_visibility_direct"
-      ])
-
-    %{
-      public: counter_cache["status_visibility_public"] || 0,
-      unlisted: counter_cache["status_visibility_unlisted"] || 0,
-      private: counter_cache["status_visibility_private"] || 0,
-      direct: counter_cache["status_visibility_direct"] || 0
-    }
   end
 end

lib/pleroma/upload.ex

@@ -63,6 +63,10 @@ def store(upload, opts \\ []) do
     with {:ok, upload} <- prepare_upload(upload, opts),
          upload = %__MODULE__{upload | path: upload.path || "#{upload.id}/#{upload.name}"},
          {:ok, upload} <- Pleroma.Upload.Filter.filter(opts.filters, upload),
+         description = Map.get(opts, :description) || upload.name,
+         {_, true} <-
+           {:description_limit,
+            String.length(description) <= Pleroma.Config.get([:instance, :description_limit])},
          {:ok, url_spec} <- Pleroma.Uploaders.Uploader.put_file(opts.uploader, upload) do
       {:ok,
        %{
@@ -75,9 +79,12 @@ def store(upload, opts \\ []) do
              "href" => url_from_spec(upload, opts.base_url, url_spec)
            }
          ],
-         "name" => Map.get(opts, :description) || upload.name
+         "name" => description
        }}
     else
+      {:description_limit, _} ->
+        {:error, :description_too_long}
+
       {:error, error} ->
         Logger.error(
           "#{__MODULE__} store (using #{inspect(opts.uploader)}) failed: #{inspect(error)}"

lib/pleroma/user.ex

@@ -89,7 +89,7 @@ defmodule Pleroma.User do
     field(:keys, :string)
     field(:public_key, :string)
     field(:ap_id, :string)
-    field(:avatar, :map)
+    field(:avatar, :map, default: %{})
     field(:local, :boolean, default: true)
     field(:follower_address, :string)
     field(:following_address, :string)
@@ -115,7 +115,7 @@ defmodule Pleroma.User do
     field(:is_moderator, :boolean, default: false)
     field(:is_admin, :boolean, default: false)
     field(:show_role, :boolean, default: true)
-    field(:settings, :map, default: nil)
+    field(:mastofe_settings, :map, default: nil)
     field(:uri, ObjectValidators.Uri, default: nil)
     field(:hide_followers_count, :boolean, default: false)
     field(:hide_follows_count, :boolean, default: false)
@@ -263,37 +263,60 @@ def account_status(%User{deactivated: true}), do: :deactivated
   def account_status(%User{password_reset_pending: true}), do: :password_reset_pending
 
   def account_status(%User{confirmation_pending: true}) do
-    case Config.get([:instance, :account_activation_required]) do
+    if Config.get([:instance, :account_activation_required]) do
-      true -> :confirmation_pending
+      :confirmation_pending
-      _ -> :active
+    else
+      :active
     end
   end
 
   def account_status(%User{}), do: :active
 
-  @spec visible_for?(User.t(), User.t() | nil) :: boolean()
+  @spec visible_for(User.t(), User.t() | nil) ::
-  def visible_for?(user, for_user \\ nil)
+          :visible
+          | :invisible
+          | :restricted_unauthenticated
+          | :deactivated
+          | :confirmation_pending
+  def visible_for(user, for_user \\ nil)
 
-  def visible_for?(%User{invisible: true}, _), do: false
+  def visible_for(%User{invisible: true}, _), do: :invisible
 
-  def visible_for?(%User{id: user_id}, %User{id: user_id}), do: true
+  def visible_for(%User{id: user_id}, %User{id: user_id}), do: :visible
 
-  def visible_for?(%User{local: local} = user, nil) do
+  def visible_for(%User{} = user, nil) do
-    cfg_key =
+    if restrict_unauthenticated?(user) do
-      if local,
+      :restrict_unauthenticated
-        do: :local,
+    else
-        else: :remote
+      visible_account_status(user)
-
+    end
-    if Config.get([:restrict_unauthenticated, :profiles, cfg_key]),
-      do: false,
-      else: account_status(user) == :active
   end
 
-  def visible_for?(%User{} = user, for_user) do
+  def visible_for(%User{} = user, for_user) do
-    account_status(user) == :active || superuser?(for_user)
+    if superuser?(for_user) do
+      :visible
+    else
+      visible_account_status(user)
+    end
   end
 
-  def visible_for?(_, _), do: false
+  def visible_for(_, _), do: :invisible
+
+  defp restrict_unauthenticated?(%User{local: local}) do
+    config_key = if local, do: :local, else: :remote
+
+    Config.get([:restrict_unauthenticated, :profiles, config_key], false)
+  end
+
+  defp visible_account_status(user) do
+    status = account_status(user)
+
+    if status in [:active, :password_reset_pending] do
+      :visible
+    else
+      status
+    end
+  end
 
   @spec superuser?(User.t()) :: boolean()
   def superuser?(%User{local: true, is_admin: true}), do: true
@@ -365,8 +388,8 @@ defp fix_follower_address(%{nickname: nickname} = params),
   defp fix_follower_address(params), do: params
 
   def remote_user_changeset(struct \\ %User{local: false}, params) do
-    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
+    bio_limit = Config.get([:instance, :user_bio_length], 5000)
-    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
+    name_limit = Config.get([:instance, :user_name_length], 100)
 
     name =
       case params[:name] do
@@ -425,8 +448,8 @@ def remote_user_changeset(struct \\ %User{local: false}, params) do
   end
 
   def update_changeset(struct, params \\ %{}) do
-    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
+    bio_limit = Config.get([:instance, :user_bio_length], 5000)
-    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
+    name_limit = Config.get([:instance, :user_name_length], 100)
 
     struct
     |> cast(
@@ -465,6 +488,7 @@ def update_changeset(struct, params \\ %{}) do
     |> validate_format(:nickname, local_nickname_regex())
     |> validate_length(:bio, max: bio_limit)
     |> validate_length(:name, min: 1, max: name_limit)
+    |> validate_inclusion(:actor_type, ["Person", "Service"])
     |> put_fields()
     |> put_emoji()
     |> put_change_if_present(:bio, &{:ok, parse_bio(&1, struct)})
@@ -515,14 +539,11 @@ defp put_emoji(changeset) do
   end
 
   defp put_change_if_present(changeset, map_field, value_function) do
-    if value = get_change(changeset, map_field) do
+    with {:ok, value} <- fetch_change(changeset, map_field),
-      with {:ok, new_value} <- value_function.(value) do
+         {:ok, new_value} <- value_function.(value) do
-        put_change(changeset, map_field, new_value)
+      put_change(changeset, map_field, new_value)
-      else
-        _ -> changeset
-      end
     else
-      changeset
+      _ -> changeset
     end
   end
 
@@ -597,12 +618,12 @@ def force_password_reset_async(user) do
   def force_password_reset(user), do: update_password_reset_pending(user, true)
 
   def register_changeset(struct, params \\ %{}, opts \\ []) do
-    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
+    bio_limit = Config.get([:instance, :user_bio_length], 5000)
-    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
+    name_limit = Config.get([:instance, :user_name_length], 100)
 
     need_confirmation? =
       if is_nil(opts[:need_confirmation]) do
-        Pleroma.Config.get([:instance, :account_activation_required])
+        Config.get([:instance, :account_activation_required])
       else
         opts[:need_confirmation]
       end
@@ -623,7 +644,7 @@ def register_changeset(struct, params \\ %{}, opts \\ []) do
     |> validate_confirmation(:password)
     |> unique_constraint(:email)
     |> unique_constraint(:nickname)
-    |> validate_exclusion(:nickname, Pleroma.Config.get([User, :restricted_nicknames]))
+    |> validate_exclusion(:nickname, Config.get([User, :restricted_nicknames]))
     |> validate_format(:nickname, local_nickname_regex())
     |> validate_format(:email, @email_regex)
     |> validate_length(:bio, max: bio_limit)
@@ -638,7 +659,7 @@ def register_changeset(struct, params \\ %{}, opts \\ []) do
   def maybe_validate_required_email(changeset, true), do: changeset
 
   def maybe_validate_required_email(changeset, _) do
-    if Pleroma.Config.get([:instance, :account_activation_required]) do
+    if Config.get([:instance, :account_activation_required]) do
       validate_required(changeset, [:email])
     else
       changeset
@@ -658,7 +679,7 @@ defp put_following_and_follower_address(changeset) do
   end
 
   defp autofollow_users(user) do
-    candidates = Pleroma.Config.get([:instance, :autofollowed_nicknames])
+    candidates = Config.get([:instance, :autofollowed_nicknames])
 
     autofollowed_users =
       User.Query.build(%{nickname: candidates, local: true, deactivated: false})
@@ -685,7 +706,7 @@ def post_register_action(%User{} = user) do
 
   def try_send_confirmation_email(%User{} = user) do
     if user.confirmation_pending &&
-         Pleroma.Config.get([:instance, :account_activation_required]) do
+         Config.get([:instance, :account_activation_required]) do
       user
       |> Pleroma.Emails.UserEmail.account_confirmation_email()
       |> Pleroma.Emails.Mailer.deliver_async()
@@ -742,7 +763,7 @@ def follow_all(follower, followeds) do
   defdelegate following(user), to: FollowingRelationship
 
   def follow(%User{} = follower, %User{} = followed, state \\ :follow_accept) do
-    deny_follow_blocked = Pleroma.Config.get([:user, :deny_follow_blocked])
+    deny_follow_blocked = Config.get([:user, :deny_follow_blocked])
 
     cond do
       followed.deactivated ->
@@ -758,7 +779,6 @@ def follow(%User{} = follower, %User{} = followed, state \\ :follow_accept) do
 
         follower
         |> update_following_count()
-        |> set_cache()
     end
   end
 
@@ -787,7 +807,6 @@ defp do_unfollow(%User{} = follower, %User{} = followed) do
         {:ok, follower} =
           follower
           |> update_following_count()
-          |> set_cache()
 
         {:ok, follower, followed}
 
@@ -945,7 +964,7 @@ def get_cached_by_nickname(nickname) do
   end
 
   def get_cached_by_nickname_or_id(nickname_or_id, opts \\ []) do
-    restrict_to_local = Pleroma.Config.get([:instance, :limit_to_local_content])
+    restrict_to_local = Config.get([:instance, :limit_to_local_content])
 
     cond do
       is_integer(nickname_or_id) or FlakeId.flake_id?(nickname_or_id) ->
@@ -1139,35 +1158,25 @@ defp follow_information_changeset(user, params) do
     ])
   end
 
+  @spec update_follower_count(User.t()) :: {:ok, User.t()}
   def update_follower_count(%User{} = user) do
-    if user.local or !Pleroma.Config.get([:instance, :external_user_synchronization]) do
+    if user.local or !Config.get([:instance, :external_user_synchronization]) do
-      follower_count_query =
+      follower_count = FollowingRelationship.follower_count(user)
-        User.Query.build(%{followers: user, deactivated: false})
-        |> select([u], %{count: count(u.id)})
 
-      User
+      user
-      |> where(id: ^user.id)
+      |> follow_information_changeset(%{follower_count: follower_count})
-      |> join(:inner, [u], s in subquery(follower_count_query))
+      |> update_and_set_cache
-      |> update([u, s],
-        set: [follower_count: s.count]
-      )
-      |> select([u], u)
-      |> Repo.update_all([])
-      |> case do
-        {1, [user]} -> set_cache(user)
-        _ -> {:error, user}
-      end
     else
       {:ok, maybe_fetch_follow_information(user)}
     end
   end
 
-  @spec update_following_count(User.t()) :: User.t()
+  @spec update_following_count(User.t()) :: {:ok, User.t()}
   def update_following_count(%User{local: false} = user) do
-    if Pleroma.Config.get([:instance, :external_user_synchronization]) do
+    if Config.get([:instance, :external_user_synchronization]) do
-      maybe_fetch_follow_information(user)
+      {:ok, maybe_fetch_follow_information(user)}
     else
-      user
+      {:ok, user}
     end
   end
 
@@ -1176,7 +1185,7 @@ def update_following_count(%User{local: true} = user) do
 
     user
     |> follow_information_changeset(%{following_count: following_count})
-    |> Repo.update!()
+    |> update_and_set_cache()
   end
 
   def set_unread_conversation_count(%User{local: true} = user) do
@@ -1251,7 +1260,7 @@ def unmute(%User{} = muter, %User{} = mutee) do
   end
 
   def subscribe(%User{} = subscriber, %User{} = target) do
-    deny_follow_blocked = Pleroma.Config.get([:user, :deny_follow_blocked])
+    deny_follow_blocked = Config.get([:user, :deny_follow_blocked])
 
     if blocks?(target, subscriber) and deny_follow_blocked do
       {:error, "Could not subscribe: #{target.nickname} is blocking you"}
@@ -1297,7 +1306,8 @@ def block(%User{} = blocker, %User{} = blocked) do
 
     unsubscribe(blocked, blocker)
 
-    if following?(blocked, blocker), do: unfollow(blocked, blocker)
+    unfollowing_blocked = Config.get([:activitypub, :unfollow_blocked], true)
+    if unfollowing_blocked && following?(blocked, blocker), do: unfollow(blocked, blocker)
 
     {:ok, blocker} = update_follower_count(blocker)
     {:ok, blocker, _} = Participation.mark_all_as_read(blocker, blocked)
@@ -1515,8 +1525,7 @@ def perform(:blocks_import, %User{} = blocker, blocked_identifiers)
       blocked_identifiers,
       fn blocked_identifier ->
         with {:ok, %User{} = blocked} <- get_or_fetch(blocked_identifier),
-             {:ok, _user_block} <- block(blocker, blocked),
+             {:ok, _block} <- CommonAPI.block(blocker, blocked) do
-             {:ok, _} <- ActivityPub.block(blocker, blocked) do
           blocked
         else
           err ->
@@ -1534,7 +1543,7 @@ def perform(:follow_import, %User{} = follower, followed_identifiers)
       fn followed_identifier ->
         with {:ok, %User{} = followed} <- get_or_fetch(followed_identifier),
              {:ok, follower} <- maybe_direct_follow(follower, followed),
-             {:ok, _} <- ActivityPub.follow(follower, followed) do
+             {:ok, _, _, _} <- CommonAPI.follow(follower, followed) do
           followed
         else
           err ->
@@ -1642,7 +1651,7 @@ def html_filter_policy(%User{no_rich_text: true}) do
     Pleroma.HTML.Scrubber.TwitterText
   end
 
-  def html_filter_policy(_), do: Pleroma.Config.get([:markup, :scrub_policy])
+  def html_filter_policy(_), do: Config.get([:markup, :scrub_policy])
 
   def fetch_by_ap_id(ap_id), do: ActivityPub.make_user_from_ap_id(ap_id)
 
@@ -1824,7 +1833,7 @@ defp normalize_tags(tags) do
   end
 
   defp local_nickname_regex do
-    if Pleroma.Config.get([:instance, :extended_nickname_format]) do
+    if Config.get([:instance, :extended_nickname_format]) do
       @extended_local_nickname_regex
     else
       @strict_local_nickname_regex
@@ -1952,8 +1961,8 @@ def get_mascot(%{mascot: %{} = mascot}) when not is_nil(mascot) do
 
   def get_mascot(%{mascot: mascot}) when is_nil(mascot) do
     # use instance-default
-    config = Pleroma.Config.get([:assets, :mascots])
+    config = Config.get([:assets, :mascots])
-    default_mascot = Pleroma.Config.get([:assets, :default_mascot])
+    default_mascot = Config.get([:assets, :default_mascot])
     mascot = Keyword.get(config, default_mascot)
 
     %{
@@ -2048,7 +2057,7 @@ def roles(%{is_moderator: is_moderator, is_admin: is_admin}) do
 
   def validate_fields(changeset, remote? \\ false) do
     limit_name = if remote?, do: :max_remote_account_fields, else: :max_account_fields
-    limit = Pleroma.Config.get([:instance, limit_name], 0)
+    limit = Config.get([:instance, limit_name], 0)
 
     changeset
     |> validate_length(:fields, max: limit)
@@ -2062,8 +2071,8 @@ def validate_fields(changeset, remote? \\ false) do
   end
 
   defp valid_field?(%{"name" => name, "value" => value}) do
-    name_limit = Pleroma.Config.get([:instance, :account_field_name_length], 255)
+    name_limit = Config.get([:instance, :account_field_name_length], 255)
-    value_limit = Pleroma.Config.get([:instance, :account_field_value_length], 255)
+    value_limit = Config.get([:instance, :account_field_value_length], 255)
 
     is_binary(name) && is_binary(value) && String.length(name) <= name_limit &&
       String.length(value) <= value_limit
@@ -2073,10 +2082,10 @@ defp valid_field?(_), do: false
 
   defp truncate_field(%{"name" => name, "value" => value}) do
     {name, _chopped} =
-      String.split_at(name, Pleroma.Config.get([:instance, :account_field_name_length], 255))
+      String.split_at(name, Config.get([:instance, :account_field_name_length], 255))
 
     {value, _chopped} =
-      String.split_at(value, Pleroma.Config.get([:instance, :account_field_value_length], 255))
+      String.split_at(value, Config.get([:instance, :account_field_value_length], 255))
 
     %{"name" => name, "value" => value}
   end
@@ -2106,8 +2115,8 @@ def mascot_update(user, url) do
 
   def mastodon_settings_update(user, settings) do
     user
-    |> cast(%{settings: settings}, [:settings])
+    |> cast(%{mastofe_settings: settings}, [:mastofe_settings])
-    |> validate_required([:settings])
+    |> validate_required([:mastofe_settings])
     |> update_and_set_cache()
   end
 
@@ -2131,7 +2140,7 @@ def confirmation_changeset(user, need_confirmation: need_confirmation?) do
 
   def add_pinnned_activity(user, %Pleroma.Activity{id: id}) do
     if id not in user.pinned_activities do
-      max_pinned_statuses = Pleroma.Config.get([:instance, :max_pinned_statuses], 0)
+      max_pinned_statuses = Config.get([:instance, :max_pinned_statuses], 0)
       params = %{pinned_activities: user.pinned_activities ++ [id]}
 
       user

lib/pleroma/user/search.ex

@@ -52,6 +52,7 @@ defp search_query(query_string, for_user, following) do
     |> base_query(following)
     |> filter_blocked_user(for_user)
     |> filter_invisible_users()
+    |> filter_internal_users()
     |> filter_blocked_domains(for_user)
     |> fts_search(query_string)
     |> trigram_rank(query_string)
@@ -109,6 +110,10 @@ defp filter_invisible_users(query) do
     from(q in query, where: q.invisible == false)
   end
 
+  defp filter_internal_users(query) do
+    from(q in query, where: q.actor_type != "Application")
+  end
+
   defp filter_blocked_user(query, %User{} = blocker) do
     query
     |> join(:left, [u], b in Pleroma.UserRelationship,

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -10,6 +10,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   alias Pleroma.Constants
   alias Pleroma.Conversation
   alias Pleroma.Conversation.Participation
+  alias Pleroma.Filter
   alias Pleroma.Maps
   alias Pleroma.Notification
   alias Pleroma.Object
@@ -321,50 +322,6 @@ defp accept_or_reject(type, %{to: to, actor: actor, object: object} = params) do
     end
   end
 
-  @spec update(map()) :: {:ok, Activity.t()} | {:error, any()}
-  def update(%{to: to, cc: cc, actor: actor, object: object} = params) do
-    local = !(params[:local] == false)
-    activity_id = params[:activity_id]
-
-    data =
-      %{
-        "to" => to,
-        "cc" => cc,
-        "type" => "Update",
-        "actor" => actor,
-        "object" => object
-      }
-      |> Maps.put_if_present("id", activity_id)
-
-    with {:ok, activity} <- insert(data, local),
-         _ <- notify_and_stream(activity),
-         :ok <- maybe_federate(activity) do
-      {:ok, activity}
-    end
-  end
-
-  @spec follow(User.t(), User.t(), String.t() | nil, boolean(), keyword()) ::
-          {:ok, Activity.t()} | {:error, any()}
-  def follow(follower, followed, activity_id \\ nil, local \\ true, opts \\ []) do
-    with {:ok, result} <-
-           Repo.transaction(fn -> do_follow(follower, followed, activity_id, local, opts) end) do
-      result
-    end
-  end
-
-  defp do_follow(follower, followed, activity_id, local, opts) do
-    skip_notify_and_stream = Keyword.get(opts, :skip_notify_and_stream, false)
-    data = make_follow_data(follower, followed, activity_id)
-
-    with {:ok, activity} <- insert(data, local),
-         _ <- skip_notify_and_stream || notify_and_stream(activity),
-         :ok <- maybe_federate(activity) do
-      {:ok, activity}
-    else
-      {:error, error} -> Repo.rollback(error)
-    end
-  end
-
   @spec unfollow(User.t(), User.t(), String.t() | nil, boolean()) ::
           {:ok, Activity.t()} | nil | {:error, any()}
   def unfollow(follower, followed, activity_id \\ nil, local \\ true) do
@@ -388,33 +345,6 @@ defp do_unfollow(follower, followed, activity_id, local) do
     end
   end
 
-  @spec block(User.t(), User.t(), String.t() | nil, boolean()) ::
-          {:ok, Activity.t()} | {:error, any()}
-  def block(blocker, blocked, activity_id \\ nil, local \\ true) do
-    with {:ok, result} <-
-           Repo.transaction(fn -> do_block(blocker, blocked, activity_id, local) end) do
-      result
-    end
-  end
-
-  defp do_block(blocker, blocked, activity_id, local) do
-    unfollow_blocked = Config.get([:activitypub, :unfollow_blocked])
-
-    if unfollow_blocked and fetch_latest_follow(blocker, blocked) do
-      unfollow(blocker, blocked, nil, local)
-    end
-
-    block_data = make_block_data(blocker, blocked, activity_id)
-
-    with {:ok, activity} <- insert(block_data, local),
-         _ <- notify_and_stream(activity),
-         :ok <- maybe_federate(activity) do
-      {:ok, activity}
-    else
-      {:error, error} -> Repo.rollback(error)
-    end
-  end
-
   @spec flag(map()) :: {:ok, Activity.t()} | {:error, any()}
   def flag(
         %{
@@ -495,6 +425,7 @@ def fetch_activities_for_context_query(context, opts) do
     |> maybe_set_thread_muted_field(opts)
     |> restrict_blocked(opts)
     |> restrict_recipients(recipients, opts[:user])
+    |> restrict_filtered(opts)
     |> where(
       [activity],
       fragment(
@@ -834,7 +765,8 @@ defp restrict_media(_query, %{only_media: _val, skip_preload: true}) do
 
   defp restrict_media(query, %{only_media: true}) do
     from(
-      [_activity, object] in query,
+      [activity, object] in query,
+      where: fragment("(?)->>'type' = ?", activity.data, "Create"),
       where: fragment("not (?)->'attachment' = (?)", object.data, ^[])
     )
   end
@@ -1009,6 +941,26 @@ defp restrict_instance(query, %{instance: instance}) do
 
   defp restrict_instance(query, _), do: query
 
+  defp restrict_filtered(query, %{user: %User{} = user}) do
+    case Filter.compose_regex(user) do
+      nil ->
+        query
+
+      regex ->
+        from([activity, object] in query,
+          where:
+            fragment("not(?->>'content' ~* ?)", object.data, ^regex) or
+              activity.actor == ^user.ap_id
+        )
+    end
+  end
+
+  defp restrict_filtered(query, %{blocking_user: %User{} = user}) do
+    restrict_filtered(query, %{user: user})
+  end
+
+  defp restrict_filtered(query, _), do: query
+
   defp exclude_poll_votes(query, %{include_poll_votes: true}), do: query
 
   defp exclude_poll_votes(query, _) do
@@ -1139,6 +1091,7 @@ def fetch_activities_query(recipients, opts \\ %{}) do
     |> restrict_favorited_by(opts)
     |> restrict_blocked(restrict_blocked_opts)
     |> restrict_muted(restrict_muted_opts)
+    |> restrict_filtered(opts)
     |> restrict_media(opts)
     |> restrict_visibility(opts)
     |> restrict_thread_visibility(opts, config)
@@ -1147,6 +1100,7 @@ def fetch_activities_query(recipients, opts \\ %{}) do
     |> restrict_muted_reblogs(restrict_muted_reblogs_opts)
     |> restrict_instance(opts)
     |> restrict_announce_object_actor(opts)
+    |> restrict_filtered(opts)
     |> Activity.restrict_deactivated_users()
     |> exclude_poll_votes(opts)
     |> exclude_chat_messages(opts)
@@ -1419,6 +1373,16 @@ def fetch_and_prepare_user_from_ap_id(ap_id) do
     end
   end
 
+  def maybe_handle_clashing_nickname(nickname) do
+    with %User{} = old_user <- User.get_by_nickname(nickname) do
+      Logger.info("Found an old user for #{nickname}, ap id is #{old_user.ap_id}, renaming.")
+
+      old_user
+      |> User.remote_user_changeset(%{nickname: "#{old_user.id}.#{old_user.nickname}"})
+      |> User.update_and_set_cache()
+    end
+  end
+
   def make_user_from_ap_id(ap_id) do
     user = User.get_cached_by_ap_id(ap_id)
 
@@ -1431,6 +1395,8 @@ def make_user_from_ap_id(ap_id) do
           |> User.remote_user_changeset(data)
           |> User.update_and_set_cache()
         else
+          maybe_handle_clashing_nickname(data[:nickname])
+
           data
           |> User.remote_user_changeset()
           |> Repo.insert()

lib/pleroma/web/activity_pub/activity_pub_controller.ex

@@ -514,7 +514,6 @@ defp ensure_user_keys_present_and_maybe_refresh_for_user(user, for_user) do
     {new_user, for_user}
   end
 
-  # TODO: Add support for "object" field
   @doc """
   Endpoint based on <https://www.w3.org/wiki/SocialCG/ActivityPub/MediaUpload>
 
@@ -525,6 +524,8 @@ defp ensure_user_keys_present_and_maybe_refresh_for_user(user, for_user) do
   Response:
   - HTTP Code: 201 Created
   - HTTP Body: ActivityPub object to be inserted into another's `attachment` field
+
+  Note: Will not point to a URL with a `Location` header because no standalone Activity has been created.
   """
   def upload_media(%{assigns: %{user: %User{} = user}} = conn, %{"file" => file} = data) do
     with {:ok, object} <-

lib/pleroma/web/activity_pub/builder.ex

@@ -14,6 +14,19 @@ defmodule Pleroma.Web.ActivityPub.Builder do
 
   require Pleroma.Constants
 
+  @spec follow(User.t(), User.t()) :: {:ok, map(), keyword()}
+  def follow(follower, followed) do
+    data = %{
+      "id" => Utils.generate_activity_id(),
+      "actor" => follower.ap_id,
+      "type" => "Follow",
+      "object" => followed.ap_id,
+      "to" => [followed.ap_id]
+    }
+
+    {:ok, data, []}
+  end
+
   @spec emoji_react(User.t(), Object.t(), String.t()) :: {:ok, map(), keyword()}
   def emoji_react(actor, object, emoji) do
     with {:ok, data, meta} <- object_action(actor, object) do
@@ -123,6 +136,33 @@ def like(actor, object) do
     end
   end
 
+  # Retricted to user updates for now, always public
+  @spec update(User.t(), Object.t()) :: {:ok, map(), keyword()}
+  def update(actor, object) do
+    to = [Pleroma.Constants.as_public(), actor.follower_address]
+
+    {:ok,
+     %{
+       "id" => Utils.generate_activity_id(),
+       "type" => "Update",
+       "actor" => actor.ap_id,
+       "object" => object,
+       "to" => to
+     }, []}
+  end
+
+  @spec block(User.t(), User.t()) :: {:ok, map(), keyword()}
+  def block(blocker, blocked) do
+    {:ok,
+     %{
+       "id" => Utils.generate_activity_id(),
+       "type" => "Block",
+       "actor" => blocker.ap_id,
+       "object" => blocked.ap_id,
+       "to" => [blocked.ap_id]
+     }, []}
+  end
+
   @spec announce(User.t(), Object.t(), keyword()) :: {:ok, map(), keyword()}
   def announce(actor, object, options \\ []) do
     public? = Keyword.get(options, :public, false)

lib/pleroma/web/activity_pub/mrf.ex

@@ -16,7 +16,7 @@ def filter(policies, %{} = object) do
   def filter(%{} = object), do: get_policies() |> filter(object)
 
   def get_policies do
-    Pleroma.Config.get([:instance, :rewrite_policy], []) |> get_policies()
+    Pleroma.Config.get([:mrf, :policies], []) |> get_policies()
   end
 
   defp get_policies(policy) when is_atom(policy), do: [policy]
@@ -51,7 +51,7 @@ def describe(policies) do
       get_policies()
       |> Enum.map(fn policy -> to_string(policy) |> String.split(".") |> List.last() end)
 
-    exclusions = Pleroma.Config.get([:instance, :mrf_transparency_exclusions])
+    exclusions = Pleroma.Config.get([:mrf, :transparency_exclusions])
 
     base =
       %{

lib/pleroma/web/activity_pub/mrf/anti_link_spam_policy.ex

@@ -27,11 +27,14 @@ defp contains_links?(_), do: false
 
   @impl true
   def filter(%{"type" => "Create", "actor" => actor, "object" => object} = message) do
-    with {:ok, %User{} = u} <- User.get_or_fetch_by_ap_id(actor),
+    with {:ok, %User{local: false} = u} <- User.get_or_fetch_by_ap_id(actor),
          {:contains_links, true} <- {:contains_links, contains_links?(object)},
          {:old_user, true} <- {:old_user, old_user?(u)} do
       {:ok, message}
     else
+      {:ok, %User{local: true}} ->
+        {:ok, message}
+
       {:contains_links, false} ->
         {:ok, message}
 

lib/pleroma/web/activity_pub/mrf/hellthread_policy.ex

@@ -13,8 +13,10 @@ defmodule Pleroma.Web.ActivityPub.MRF.HellthreadPolicy do
 
   defp delist_message(message, threshold) when threshold > 0 do
     follower_collection = User.get_cached_by_ap_id(message["actor"]).follower_address
+    to = message["to"] || []
+    cc = message["cc"] || []
 
-    follower_collection? = Enum.member?(message["to"] ++ message["cc"], follower_collection)
+    follower_collection? = Enum.member?(to ++ cc, follower_collection)
 
     message =
       case get_recipient_count(message) do
@@ -71,7 +73,8 @@ defp get_recipient_count(message) do
   end
 
   @impl true
-  def filter(%{"type" => "Create"} = message) do
+  def filter(%{"type" => "Create", "object" => %{"type" => object_type}} = message)
+      when object_type in ~w{Note Article} do
     reject_threshold =
       Pleroma.Config.get(
         [:mrf_hellthread, :reject_threshold],

lib/pleroma/web/activity_pub/mrf/object_age_policy.ex

@@ -98,7 +98,7 @@ def filter(message), do: {:ok, message}
   @impl true
   def describe do
     mrf_object_age =
-      Pleroma.Config.get(:mrf_object_age)
+      Config.get(:mrf_object_age)
       |> Enum.into(%{})
 
     {:ok, %{mrf_object_age: mrf_object_age}}

lib/pleroma/web/activity_pub/mrf/reject_non_public.ex

@@ -47,5 +47,5 @@ def filter(object), do: {:ok, object}
 
   @impl true
   def describe,
-    do: {:ok, %{mrf_rejectnonpublic: Pleroma.Config.get(:mrf_rejectnonpublic) |> Enum.into(%{})}}
+    do: {:ok, %{mrf_rejectnonpublic: Config.get(:mrf_rejectnonpublic) |> Enum.into(%{})}}
 end

lib/pleroma/web/activity_pub/mrf/simple_policy.ex

@@ -3,21 +3,23 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
-  alias Pleroma.User
-  alias Pleroma.Web.ActivityPub.MRF
   @moduledoc "Filter activities depending on their origin instance"
   @behaviour Pleroma.Web.ActivityPub.MRF
 
+  alias Pleroma.Config
+  alias Pleroma.User
+  alias Pleroma.Web.ActivityPub.MRF
+
   require Pleroma.Constants
 
   defp check_accept(%{host: actor_host} = _actor_info, object) do
     accepts =
-      Pleroma.Config.get([:mrf_simple, :accept])
+      Config.get([:mrf_simple, :accept])
       |> MRF.subdomains_regex()
 
     cond do
       accepts == [] -> {:ok, object}
-      actor_host == Pleroma.Config.get([Pleroma.Web.Endpoint, :url, :host]) -> {:ok, object}
+      actor_host == Config.get([Pleroma.Web.Endpoint, :url, :host]) -> {:ok, object}
       MRF.subdomain_match?(accepts, actor_host) -> {:ok, object}
       true -> {:reject, nil}
     end
@@ -25,7 +27,7 @@ defp check_accept(%{host: actor_host} = _actor_info, object) do
 
   defp check_reject(%{host: actor_host} = _actor_info, object) do
     rejects =
-      Pleroma.Config.get([:mrf_simple, :reject])
+      Config.get([:mrf_simple, :reject])
       |> MRF.subdomains_regex()
 
     if MRF.subdomain_match?(rejects, actor_host) do
@@ -41,7 +43,7 @@ defp check_media_removal(
        )
        when length(child_attachment) > 0 do
     media_removal =
-      Pleroma.Config.get([:mrf_simple, :media_removal])
+      Config.get([:mrf_simple, :media_removal])
       |> MRF.subdomains_regex()
 
     object =
@@ -65,7 +67,7 @@ defp check_media_nsfw(
          } = object
        ) do
     media_nsfw =
-      Pleroma.Config.get([:mrf_simple, :media_nsfw])
+      Config.get([:mrf_simple, :media_nsfw])
       |> MRF.subdomains_regex()
 
     object =
@@ -85,7 +87,7 @@ defp check_media_nsfw(_actor_info, object), do: {:ok, object}
 
   defp check_ftl_removal(%{host: actor_host} = _actor_info, object) do
     timeline_removal =
-      Pleroma.Config.get([:mrf_simple, :federated_timeline_removal])
+      Config.get([:mrf_simple, :federated_timeline_removal])
       |> MRF.subdomains_regex()
 
     object =
@@ -108,7 +110,7 @@ defp check_ftl_removal(%{host: actor_host} = _actor_info, object) do
 
   defp check_report_removal(%{host: actor_host} = _actor_info, %{"type" => "Flag"} = object) do
     report_removal =
-      Pleroma.Config.get([:mrf_simple, :report_removal])
+      Config.get([:mrf_simple, :report_removal])
       |> MRF.subdomains_regex()
 
     if MRF.subdomain_match?(report_removal, actor_host) do
@@ -122,7 +124,7 @@ defp check_report_removal(_actor_info, object), do: {:ok, object}
 
   defp check_avatar_removal(%{host: actor_host} = _actor_info, %{"icon" => _icon} = object) do
     avatar_removal =
-      Pleroma.Config.get([:mrf_simple, :avatar_removal])
+      Config.get([:mrf_simple, :avatar_removal])
       |> MRF.subdomains_regex()
 
     if MRF.subdomain_match?(avatar_removal, actor_host) do
@@ -136,7 +138,7 @@ defp check_avatar_removal(_actor_info, object), do: {:ok, object}
 
   defp check_banner_removal(%{host: actor_host} = _actor_info, %{"image" => _image} = object) do
     banner_removal =
-      Pleroma.Config.get([:mrf_simple, :banner_removal])
+      Config.get([:mrf_simple, :banner_removal])
       |> MRF.subdomains_regex()
 
     if MRF.subdomain_match?(banner_removal, actor_host) do
@@ -153,7 +155,7 @@ def filter(%{"type" => "Delete", "actor" => actor} = object) do
     %{host: actor_host} = URI.parse(actor)
 
     reject_deletes =
-      Pleroma.Config.get([:mrf_simple, :reject_deletes])
+      Config.get([:mrf_simple, :reject_deletes])
       |> MRF.subdomains_regex()
 
     if MRF.subdomain_match?(reject_deletes, actor_host) do
@@ -197,10 +199,10 @@ def filter(object), do: {:ok, object}
 
   @impl true
   def describe do
-    exclusions = Pleroma.Config.get([:instance, :mrf_transparency_exclusions])
+    exclusions = Config.get([:mrf, :transparency_exclusions])
 
     mrf_simple =
-      Pleroma.Config.get(:mrf_simple)
+      Config.get(:mrf_simple)
       |> Enum.map(fn {k, v} -> {k, Enum.reject(v, fn v -> v in exclusions end)} end)
       |> Enum.into(%{})
 

lib/pleroma/web/activity_pub/object_validator.ex

@@ -13,16 +13,58 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidator do
   alias Pleroma.Object
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ObjectValidators.AnnounceValidator
+  alias Pleroma.Web.ActivityPub.ObjectValidators.BlockValidator
   alias Pleroma.Web.ActivityPub.ObjectValidators.ChatMessageValidator
   alias Pleroma.Web.ActivityPub.ObjectValidators.CreateChatMessageValidator
   alias Pleroma.Web.ActivityPub.ObjectValidators.DeleteValidator
   alias Pleroma.Web.ActivityPub.ObjectValidators.EmojiReactValidator
+  alias Pleroma.Web.ActivityPub.ObjectValidators.FollowValidator
   alias Pleroma.Web.ActivityPub.ObjectValidators.LikeValidator
   alias Pleroma.Web.ActivityPub.ObjectValidators.UndoValidator
+  alias Pleroma.Web.ActivityPub.ObjectValidators.UpdateValidator
 
   @spec validate(map(), keyword()) :: {:ok, map(), keyword()} | {:error, any()}
   def validate(object, meta)
 
+  def validate(%{"type" => "Follow"} = object, meta) do
+    with {:ok, object} <-
+           object
+           |> FollowValidator.cast_and_validate()
+           |> Ecto.Changeset.apply_action(:insert) do
+      object = stringify_keys(object)
+      {:ok, object, meta}
+    end
+  end
+
+  def validate(%{"type" => "Block"} = block_activity, meta) do
+    with {:ok, block_activity} <-
+           block_activity
+           |> BlockValidator.cast_and_validate()
+           |> Ecto.Changeset.apply_action(:insert) do
+      block_activity = stringify_keys(block_activity)
+      outgoing_blocks = Pleroma.Config.get([:activitypub, :outgoing_blocks])
+
+      meta =
+        if !outgoing_blocks do
+          Keyword.put(meta, :do_not_federate, true)
+        else
+          meta
+        end
+
+      {:ok, block_activity, meta}
+    end
+  end
+
+  def validate(%{"type" => "Update"} = update_activity, meta) do
+    with {:ok, update_activity} <-
+           update_activity
+           |> UpdateValidator.cast_and_validate()
+           |> Ecto.Changeset.apply_action(:insert) do
+      update_activity = stringify_keys(update_activity)
+      {:ok, update_activity, meta}
+    end
+  end
+
   def validate(%{"type" => "Undo"} = object, meta) do
     with {:ok, object} <-
            object

lib/pleroma/web/activity_pub/object_validators/block_validator.ex

@@ -0,0 +1,42 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.ObjectValidators.BlockValidator do
+  use Ecto.Schema
+
+  alias Pleroma.EctoType.ActivityPub.ObjectValidators
+
+  import Ecto.Changeset
+  import Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
+
+  @primary_key false
+
+  embedded_schema do
+    field(:id, ObjectValidators.ObjectID, primary_key: true)
+    field(:type, :string)
+    field(:actor, ObjectValidators.ObjectID)
+    field(:to, ObjectValidators.Recipients, default: [])
+    field(:cc, ObjectValidators.Recipients, default: [])
+    field(:object, ObjectValidators.ObjectID)
+  end
+
+  def cast_data(data) do
+    %__MODULE__{}
+    |> cast(data, __schema__(:fields))
+  end
+
+  def validate_data(cng) do
+    cng
+    |> validate_required([:id, :type, :actor, :to, :cc, :object])
+    |> validate_inclusion(:type, ["Block"])
+    |> validate_actor_presence()
+    |> validate_actor_presence(field_name: :object)
+  end
+
+  def cast_and_validate(data) do
+    data
+    |> cast_data
+    |> validate_data
+  end
+end

lib/pleroma/web/activity_pub/object_validators/follow_validator.ex

@@ -0,0 +1,44 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.ObjectValidators.FollowValidator do
+  use Ecto.Schema
+
+  alias Pleroma.EctoType.ActivityPub.ObjectValidators
+
+  import Ecto.Changeset
+  import Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
+
+  @primary_key false
+
+  embedded_schema do
+    field(:id, ObjectValidators.ObjectID, primary_key: true)
+    field(:type, :string)
+    field(:actor, ObjectValidators.ObjectID)
+    field(:to, ObjectValidators.Recipients, default: [])
+    field(:cc, ObjectValidators.Recipients, default: [])
+    field(:object, ObjectValidators.ObjectID)
+    field(:state, :string, default: "pending")
+  end
+
+  def cast_data(data) do
+    %__MODULE__{}
+    |> cast(data, __schema__(:fields))
+  end
+
+  def validate_data(cng) do
+    cng
+    |> validate_required([:id, :type, :actor, :to, :cc, :object])
+    |> validate_inclusion(:type, ["Follow"])
+    |> validate_inclusion(:state, ~w{pending reject accept})
+    |> validate_actor_presence()
+    |> validate_actor_presence(field_name: :object)
+  end
+
+  def cast_and_validate(data) do
+    data
+    |> cast_data
+    |> validate_data
+  end
+end

lib/pleroma/web/activity_pub/object_validators/note_validator.ex

@@ -41,7 +41,6 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.NoteValidator do
     field(:announcements, {:array, :string}, default: [])
 
     # see if needed
-    field(:conversation, :string)
     field(:context_id, :string)
   end
 

lib/pleroma/web/activity_pub/object_validators/update_validator.ex

@@ -0,0 +1,59 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.ObjectValidators.UpdateValidator do
+  use Ecto.Schema
+
+  alias Pleroma.EctoType.ActivityPub.ObjectValidators
+
+  import Ecto.Changeset
+  import Pleroma.Web.ActivityPub.ObjectValidators.CommonValidations
+
+  @primary_key false
+
+  embedded_schema do
+    field(:id, ObjectValidators.ObjectID, primary_key: true)
+    field(:type, :string)
+    field(:actor, ObjectValidators.ObjectID)
+    field(:to, ObjectValidators.Recipients, default: [])
+    field(:cc, ObjectValidators.Recipients, default: [])
+    # In this case, we save the full object in this activity instead of just a
+    # reference, so we can always see what was actually changed by this.
+    field(:object, :map)
+  end
+
+  def cast_data(data) do
+    %__MODULE__{}
+    |> cast(data, __schema__(:fields))
+  end
+
+  def validate_data(cng) do
+    cng
+    |> validate_required([:id, :type, :actor, :to, :cc, :object])
+    |> validate_inclusion(:type, ["Update"])
+    |> validate_actor_presence()
+    |> validate_updating_rights()
+  end
+
+  def cast_and_validate(data) do
+    data
+    |> cast_data
+    |> validate_data
+  end
+
+  # For now we only support updating users, and here the rule is easy:
+  # object id == actor id
+  def validate_updating_rights(cng) do
+    with actor = get_field(cng, :actor),
+         object = get_field(cng, :object),
+         {:ok, object_id} <- ObjectValidators.ObjectID.cast(object),
+         true <- actor == object_id do
+      cng
+    else
+      _e ->
+        cng
+        |> add_error(:object, "Can't be updated by this actor")
+    end
+  end
+end

lib/pleroma/web/activity_pub/relay.ex

@@ -28,7 +28,7 @@ def relay_ap_id do
   def follow(target_instance) do
     with %User{} = local_user <- get_actor(),
          {:ok, %User{} = target_user} <- User.get_or_fetch_by_ap_id(target_instance),
-         {:ok, activity} <- ActivityPub.follow(local_user, target_user) do
+         {:ok, _, _, activity} <- CommonAPI.follow(local_user, target_user) do
       Logger.info("relay: followed instance: #{target_instance}; id=#{activity.data["id"]}")
       {:ok, activity}
     else

lib/pleroma/web/activity_pub/side_effects.ex

@@ -6,8 +6,10 @@ defmodule Pleroma.Web.ActivityPub.SideEffects do
   collection, and so on.
   """
   alias Pleroma.Activity
+  alias Pleroma.Activity.Ir.Topics
   alias Pleroma.Chat
   alias Pleroma.Chat.MessageReference
+  alias Pleroma.FollowingRelationship
   alias Pleroma.Notification
   alias Pleroma.Object
   alias Pleroma.Repo
@@ -20,6 +22,104 @@ defmodule Pleroma.Web.ActivityPub.SideEffects do
 
   def handle(object, meta \\ [])
 
+  # Tasks this handle
+  # - Follows if possible
+  # - Sends a notification
+  # - Generates accept or reject if appropriate
+  def handle(
+        %{
+          data: %{
+            "id" => follow_id,
+            "type" => "Follow",
+            "object" => followed_user,
+            "actor" => following_user
+          }
+        } = object,
+        meta
+      ) do
+    with %User{} = follower <- User.get_cached_by_ap_id(following_user),
+         %User{} = followed <- User.get_cached_by_ap_id(followed_user),
+         {_, {:ok, _}, _, _} <-
+           {:following, User.follow(follower, followed, :follow_pending), follower, followed} do
+      if followed.local && !followed.locked do
+        Utils.update_follow_state_for_all(object, "accept")
+        FollowingRelationship.update(follower, followed, :follow_accept)
+        User.update_follower_count(followed)
+        User.update_following_count(follower)
+
+        %{
+          to: [following_user],
+          actor: followed,
+          object: follow_id,
+          local: true
+        }
+        |> ActivityPub.accept()
+      end
+    else
+      {:following, {:error, _}, follower, followed} ->
+        Utils.update_follow_state_for_all(object, "reject")
+        FollowingRelationship.update(follower, followed, :follow_reject)
+
+        if followed.local do
+          %{
+            to: [follower.ap_id],
+            actor: followed,
+            object: follow_id,
+            local: true
+          }
+          |> ActivityPub.reject()
+        end
+
+      _ ->
+        nil
+    end
+
+    {:ok, notifications} = Notification.create_notifications(object, do_send: false)
+
+    meta =
+      meta
+      |> add_notifications(notifications)
+
+    updated_object = Activity.get_by_ap_id(follow_id)
+
+    {:ok, updated_object, meta}
+  end
+
+  # Tasks this handles:
+  # - Unfollow and block
+  def handle(
+        %{data: %{"type" => "Block", "object" => blocked_user, "actor" => blocking_user}} =
+          object,
+        meta
+      ) do
+    with %User{} = blocker <- User.get_cached_by_ap_id(blocking_user),
+         %User{} = blocked <- User.get_cached_by_ap_id(blocked_user) do
+      User.block(blocker, blocked)
+    end
+
+    {:ok, object, meta}
+  end
+
+  # Tasks this handles:
+  # - Update the user
+  #
+  # For a local user, we also get a changeset with the full information, so we
+  # can update non-federating, non-activitypub settings as well.
+  def handle(%{data: %{"type" => "Update", "object" => updated_object}} = object, meta) do
+    if changeset = Keyword.get(meta, :user_update_changeset) do
+      changeset
+      |> User.update_and_set_cache()
+    else
+      {:ok, new_user_data} = ActivityPub.user_data_from_user_object(updated_object)
+
+      User.get_by_ap_id(updated_object["id"])
+      |> User.remote_user_changeset(new_user_data)
+      |> User.update_and_set_cache()
+    end
+
+    {:ok, object, meta}
+  end
+
   # Tasks this handles:
   # - Add like to object
   # - Set up notification
@@ -62,7 +162,10 @@ def handle(%{data: %{"type" => "Announce"}} = object, meta) do
 
     if !User.is_internal_user?(user) do
       Notification.create_notifications(object)
-      ActivityPub.stream_out(object)
+
+      object
+      |> Topics.get_activity_topics()
+      |> Streamer.stream(object)
     end
 
     {:ok, object, meta}
@@ -170,14 +273,20 @@ def handle_object_creation(object) do
     {:ok, object}
   end
 
-  def handle_undoing(%{data: %{"type" => "Like"}} = object) do
+  defp undo_like(nil, object), do: delete_object(object)
-    with %Object{} = liked_object <- Object.get_by_ap_id(object.data["object"]),
+
-         {:ok, _} <- Utils.remove_like_from_object(object, liked_object),
+  defp undo_like(%Object{} = liked_object, object) do
-         {:ok, _} <- Repo.delete(object) do
+    with {:ok, _} <- Utils.remove_like_from_object(object, liked_object) do
-      :ok
+      delete_object(object)
     end
   end
 
+  def handle_undoing(%{data: %{"type" => "Like"}} = object) do
+    object.data["object"]
+    |> Object.get_by_ap_id()
+    |> undo_like(object)
+  end
+
   def handle_undoing(%{data: %{"type" => "EmojiReact"}} = object) do
     with %Object{} = reacted_object <- Object.get_by_ap_id(object.data["object"]),
          {:ok, _} <- Utils.remove_emoji_reaction_from_object(object, reacted_object),
@@ -207,6 +316,11 @@ def handle_undoing(
 
   def handle_undoing(object), do: {:error, ["don't know how to handle", object]}
 
+  @spec delete_object(Object.t()) :: :ok | {:error, Ecto.Changeset.t()}
+  defp delete_object(object) do
+    with {:ok, _} <- Repo.delete(object), do: :ok
+  end
+
   defp send_notifications(meta) do
     Keyword.get(meta, :notifications, [])
     |> Enum.each(fn notification ->

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -172,8 +172,8 @@ def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
         object
         |> Map.put("inReplyTo", replied_object.data["id"])
         |> Map.put("inReplyToAtomUri", object["inReplyToAtomUri"] || in_reply_to_id)
-        |> Map.put("conversation", replied_object.data["context"] || object["conversation"])
         |> Map.put("context", replied_object.data["context"] || object["conversation"])
+        |> Map.drop(["conversation"])
       else
         e ->
           Logger.error("Couldn't fetch #{inspect(in_reply_to_id)}, error: #{inspect(e)}")
@@ -207,7 +207,7 @@ def fix_context(object) do
 
     object
     |> Map.put("context", context)
-    |> Map.put("conversation", context)
+    |> Map.drop(["conversation"])
   end
 
   def fix_attachments(%{"attachment" => attachment} = object) when is_list(attachment) do
@@ -233,18 +233,24 @@ def fix_attachments(%{"attachment" => attachment} = object) when is_list(attachm
             is_map(url) && is_binary(url["href"]) -> url["href"]
             is_binary(data["url"]) -> data["url"]
             is_binary(data["href"]) -> data["href"]
+            true -> nil
           end
 
-        attachment_url =
+        if href do
-          %{"href" => href}
+          attachment_url =
-          |> Maps.put_if_present("mediaType", media_type)
+            %{"href" => href}
-          |> Maps.put_if_present("type", Map.get(url || %{}, "type"))
+            |> Maps.put_if_present("mediaType", media_type)
+            |> Maps.put_if_present("type", Map.get(url || %{}, "type"))
 
-        %{"url" => [attachment_url]}
+          %{"url" => [attachment_url]}
-        |> Maps.put_if_present("mediaType", media_type)
+          |> Maps.put_if_present("mediaType", media_type)
-        |> Maps.put_if_present("type", data["type"])
+          |> Maps.put_if_present("type", data["type"])
-        |> Maps.put_if_present("name", data["name"])
+          |> Maps.put_if_present("name", data["name"])
+        else
+          nil
+        end
       end)
+      |> Enum.filter(& &1)
 
     Map.put(object, "attachment", attachments)
   end
@@ -263,12 +269,18 @@ def fix_url(%{"url" => url} = object) when is_map(url) do
 
   def fix_url(%{"type" => object_type, "url" => url} = object)
       when object_type in ["Video", "Audio"] and is_list(url) do
-    first_element = Enum.at(url, 0)
+    attachment =
+      Enum.find(url, fn x ->
+        media_type = x["mediaType"] || x["mimeType"] || ""
 
-    link_element = Enum.find(url, fn x -> is_map(x) and x["mimeType"] == "text/html" end)
+        is_map(x) and String.starts_with?(media_type, ["audio/", "video/"])
+      end)
+
+    link_element =
+      Enum.find(url, fn x -> is_map(x) and (x["mediaType"] || x["mimeType"]) == "text/html" end)
 
     object
-    |> Map.put("attachment", [first_element])
+    |> Map.put("attachment", [attachment])
     |> Map.put("url", link_element["href"])
   end
 
@@ -446,19 +458,16 @@ def handle_incoming(
       when objtype in ["Article", "Event", "Note", "Video", "Page", "Question", "Answer", "Audio"] do
     actor = Containment.get_actor(data)
 
-    data =
-      Map.put(data, "actor", actor)
-      |> fix_addressing
-
     with nil <- Activity.get_create_by_object_ap_id(object["id"]),
-         {:ok, %User{} = user} <- User.get_or_fetch_by_ap_id(data["actor"]) do
+         {:ok, %User{} = user} <- User.get_or_fetch_by_ap_id(actor),
+         data <- Map.put(data, "actor", actor) |> fix_addressing() do
       object = fix_object(object, options)
 
       params = %{
         to: data["to"],
         object: object,
         actor: user,
-        context: object["conversation"],
+        context: object["context"],
         local: false,
         published: data["published"],
         additional:
@@ -520,66 +529,6 @@ def handle_incoming(
     end
   end
 
-  def handle_incoming(
-        %{"type" => "Follow", "object" => followed, "actor" => follower, "id" => id} = data,
-        _options
-      ) do
-    with %User{local: true} = followed <-
-           User.get_cached_by_ap_id(Containment.get_actor(%{"actor" => followed})),
-         {:ok, %User{} = follower} <-
-           User.get_or_fetch_by_ap_id(Containment.get_actor(%{"actor" => follower})),
-         {:ok, activity} <-
-           ActivityPub.follow(follower, followed, id, false, skip_notify_and_stream: true) do
-      with deny_follow_blocked <- Pleroma.Config.get([:user, :deny_follow_blocked]),
-           {_, false} <- {:user_blocked, User.blocks?(followed, follower) && deny_follow_blocked},
-           {_, false} <- {:user_locked, User.locked?(followed)},
-           {_, {:ok, follower}} <- {:follow, User.follow(follower, followed)},
-           {_, {:ok, _}} <-
-             {:follow_state_update, Utils.update_follow_state_for_all(activity, "accept")},
-           {:ok, _relationship} <-
-             FollowingRelationship.update(follower, followed, :follow_accept) do
-        ActivityPub.accept(%{
-          to: [follower.ap_id],
-          actor: followed,
-          object: data,
-          local: true
-        })
-      else
-        {:user_blocked, true} ->
-          {:ok, _} = Utils.update_follow_state_for_all(activity, "reject")
-          {:ok, _relationship} = FollowingRelationship.update(follower, followed, :follow_reject)
-
-          ActivityPub.reject(%{
-            to: [follower.ap_id],
-            actor: followed,
-            object: data,
-            local: true
-          })
-
-        {:follow, {:error, _}} ->
-          {:ok, _} = Utils.update_follow_state_for_all(activity, "reject")
-          {:ok, _relationship} = FollowingRelationship.update(follower, followed, :follow_reject)
-
-          ActivityPub.reject(%{
-            to: [follower.ap_id],
-            actor: followed,
-            object: data,
-            local: true
-          })
-
-        {:user_locked, true} ->
-          {:ok, _relationship} = FollowingRelationship.update(follower, followed, :follow_pending)
-          :noop
-      end
-
-      ActivityPub.notify_and_stream(activity)
-      {:ok, activity}
-    else
-      _e ->
-        :error
-    end
-  end
-
   def handle_incoming(
         %{"type" => "Accept", "object" => follow_object, "actor" => _actor, "id" => id} = data,
         _options
@@ -673,7 +622,7 @@ def handle_incoming(
   end
 
   def handle_incoming(%{"type" => type} = data, _options)
-      when type in ["Like", "EmojiReact", "Announce"] do
+      when type in ~w{Like EmojiReact Announce} do
     with :ok <- ObjectValidator.fetch_actor_and_object(data),
          {:ok, activity, _meta} <-
            Pipeline.common_pipeline(data, local: false) do
@@ -684,35 +633,13 @@ def handle_incoming(%{"type" => type} = data, _options)
   end
 
   def handle_incoming(
-        %{"type" => "Update", "object" => %{"type" => object_type} = object, "actor" => actor_id} =
+        %{"type" => type} = data,
-          data,
         _options
       )
-      when object_type in [
+      when type in ~w{Update Block Follow} do
-             "Person",
+    with {:ok, %User{}} <- ObjectValidator.fetch_actor(data),
-             "Application",
+         {:ok, activity, _} <- Pipeline.common_pipeline(data, local: false) do
-             "Service",
+      {:ok, activity}
-             "Organization"
-           ] do
-    with %User{ap_id: ^actor_id} = actor <- User.get_cached_by_ap_id(object["id"]) do
-      {:ok, new_user_data} = ActivityPub.user_data_from_user_object(object)
-
-      actor
-      |> User.remote_user_changeset(new_user_data)
-      |> User.update_and_set_cache()
-
-      ActivityPub.update(%{
-        local: false,
-        to: data["to"] || [],
-        cc: data["cc"] || [],
-        object: object,
-        actor: actor_id,
-        activity_id: data["id"]
-      })
-    else
-      e ->
-        Logger.error(e)
-        :error
     end
   end
 
@@ -788,21 +715,6 @@ def handle_incoming(
     end
   end
 
-  def handle_incoming(
-        %{"type" => "Block", "object" => blocked, "actor" => blocker, "id" => id} = _data,
-        _options
-      ) do
-    with %User{local: true} = blocked = User.get_cached_by_ap_id(blocked),
-         {:ok, %User{} = blocker} = User.get_or_fetch_by_ap_id(blocker),
-         {:ok, activity} <- ActivityPub.block(blocker, blocked, id, false) do
-      User.unfollow(blocker, blocked)
-      User.block(blocker, blocked)
-      {:ok, activity}
-    else
-      _e -> :error
-    end
-  end
-
   def handle_incoming(
         %{
           "type" => "Move",

lib/pleroma/web/activity_pub/visibility.ex

@@ -47,6 +47,10 @@ def is_list?(_), do: false
   @spec visible_for_user?(Activity.t(), User.t() | nil) :: boolean()
   def visible_for_user?(%{actor: ap_id}, %User{ap_id: ap_id}), do: true
 
+  def visible_for_user?(nil, _), do: false
+
+  def visible_for_user?(%{data: %{"listMessage" => _}}, nil), do: false
+
   def visible_for_user?(%{data: %{"listMessage" => list_ap_id}} = activity, %User{} = user) do
     user.ap_id in activity.data["to"] ||
       list_ap_id
@@ -54,8 +58,6 @@ def visible_for_user?(%{data: %{"listMessage" => list_ap_id}} = activity, %User{
       |> Pleroma.List.member?(user)
   end
 
-  def visible_for_user?(%{data: %{"listMessage" => _}}, nil), do: false
-
   def visible_for_user?(%{local: local} = activity, nil) do
     cfg_key =
       if local,

lib/pleroma/web/admin_api/controllers/admin_api_controller.ex

@@ -111,8 +111,7 @@ def user_delete(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames})
       action: "delete"
     })
 
-    conn
+    json(conn, nicknames)
-    |> json(nicknames)
   end
 
   def user_follow(%{assigns: %{user: admin}} = conn, %{
@@ -131,8 +130,7 @@ def user_follow(%{assigns: %{user: admin}} = conn, %{
       })
     end
 
-    conn
+    json(conn, "ok")
-    |> json("ok")
   end
 
   def user_unfollow(%{assigns: %{user: admin}} = conn, %{
@@ -151,8 +149,7 @@ def user_unfollow(%{assigns: %{user: admin}} = conn, %{
       })
     end
 
-    conn
+    json(conn, "ok")
-    |> json("ok")
   end
 
   def users_create(%{assigns: %{user: admin}} = conn, %{"users" => users}) do
@@ -191,8 +188,7 @@ def users_create(%{assigns: %{user: admin}} = conn, %{"users" => users}) do
           action: "create"
         })
 
-        conn
+        json(conn, res)
-        |> json(res)
 
       {:error, id, changeset, _} ->
         res =
@@ -363,8 +359,8 @@ defp maybe_parse_filters(filters) do
     filters
     |> String.split(",")
     |> Enum.filter(&Enum.member?(@filters, &1))
-    |> Enum.map(&String.to_atom(&1))
+    |> Enum.map(&String.to_atom/1)
-    |> Enum.into(%{}, &{&1, true})
+    |> Map.new(&{&1, true})
   end
 
   def right_add_multiple(%{assigns: %{user: admin}} = conn, %{
@@ -568,10 +564,10 @@ def update_user_credentials(
       {:error, changeset} ->
         errors = Map.new(changeset.errors, fn {key, {error, _}} -> {key, error} end)
 
-        json(conn, %{errors: errors})
+        {:errors, errors}
 
       _ ->
-        json(conn, %{error: "Unable to update user."})
+        {:error, :not_found}
     end
   end
 
@@ -616,7 +612,7 @@ defp configurable_from_database do
   def reload_emoji(conn, _params) do
     Pleroma.Emoji.reload()
 
-    conn |> json("ok")
+    json(conn, "ok")
   end
 
   def confirm_email(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames}) do
@@ -630,7 +626,7 @@ def confirm_email(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames}
       action: "confirm_email"
     })
 
-    conn |> json("")
+    json(conn, "")
   end
 
   def resend_confirmation_email(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames}) do
@@ -644,14 +640,13 @@ def resend_confirmation_email(%{assigns: %{user: admin}} = conn, %{"nicknames" =
       action: "resend_confirmation_email"
     })
 
-    conn |> json("")
+    json(conn, "")
   end
 
-  def stats(conn, _) do
+  def stats(conn, params) do
-    count = Stats.get_status_visibility_count()
+    counters = Stats.get_status_visibility_count(params["instance"])
 
-    conn
+    json(conn, %{"status_visibility" => counters})
-    |> json(%{"status_visibility" => count})
   end
 
   defp page_params(params) do

lib/pleroma/web/admin_api/controllers/fallback_controller.ex

@@ -17,6 +17,12 @@ def call(conn, {:error, reason}) do
     |> json(%{error: reason})
   end
 
+  def call(conn, {:errors, errors}) do
+    conn
+    |> put_status(:bad_request)
+    |> json(%{errors: errors})
+  end
+
   def call(conn, {:param_cast, _}) do
     conn
     |> put_status(:bad_request)

lib/pleroma/web/admin_api/controllers/media_proxy_cache_controller.ex

@@ -0,0 +1,63 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.AdminAPI.MediaProxyCacheController do
+  use Pleroma.Web, :controller
+
+  alias Pleroma.Plugs.OAuthScopesPlug
+  alias Pleroma.Web.ApiSpec.Admin, as: Spec
+  alias Pleroma.Web.MediaProxy
+
+  plug(Pleroma.Web.ApiSpec.CastAndValidate)
+
+  plug(
+    OAuthScopesPlug,
+    %{scopes: ["read:media_proxy_caches"], admin: true} when action in [:index]
+  )
+
+  plug(
+    OAuthScopesPlug,
+    %{scopes: ["write:media_proxy_caches"], admin: true} when action in [:purge, :delete]
+  )
+
+  action_fallback(Pleroma.Web.AdminAPI.FallbackController)
+
+  defdelegate open_api_operation(action), to: Spec.MediaProxyCacheOperation
+
+  def index(%{assigns: %{user: _}} = conn, params) do
+    cursor =
+      :banned_urls_cache
+      |> :ets.table([{:traverse, {:select, Cachex.Query.create(true, :key)}}])
+      |> :qlc.cursor()
+
+    urls =
+      case params.page do
+        1 ->
+          :qlc.next_answers(cursor, params.page_size)
+
+        _ ->
+          :qlc.next_answers(cursor, (params.page - 1) * params.page_size)
+          :qlc.next_answers(cursor, params.page_size)
+      end
+
+    :qlc.delete_cursor(cursor)
+
+    render(conn, "index.json", urls: urls)
+  end
+
+  def delete(%{assigns: %{user: _}, body_params: %{urls: urls}} = conn, _) do
+    MediaProxy.remove_from_banned_urls(urls)
+    render(conn, "index.json", urls: urls)
+  end
+
+  def purge(%{assigns: %{user: _}, body_params: %{urls: urls, ban: ban}} = conn, _) do
+    MediaProxy.Invalidation.purge(urls)
+
+    if ban do
+      MediaProxy.put_in_banned_urls(urls)
+    end
+
+    render(conn, "index.json", urls: urls)
+  end
+end

lib/pleroma/web/admin_api/views/media_proxy_cache_view.ex

@@ -0,0 +1,11 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.AdminAPI.MediaProxyCacheView do
+  use Pleroma.Web, :view
+
+  def render("index.json", %{urls: urls}) do
+    %{urls: urls}
+  end
+end

lib/pleroma/web/api_spec/cast_and_validate.ex

@@ -40,7 +40,7 @@ def call(%{private: %{open_api_spex: private_data}} = conn, %{
           |> List.first()
 
         _ ->
-          nil
+          "application/json"
       end
 
     private_data = Map.put(private_data, :operation_id, operation_id)

lib/pleroma/web/api_spec/helpers.ex

@@ -39,6 +39,12 @@ def pagination_params do
         :string,
         "Return the newest items newer than this ID"
       ),
+      Operation.parameter(
+        :offset,
+        :query,
+        %Schema{type: :integer, default: 0},
+        "Return items past this number of items"
+      ),
       Operation.parameter(
         :limit,
         :query,

lib/pleroma/web/api_spec/operations/account_operation.ex

@@ -102,6 +102,7 @@ def show_operation do
       parameters: [%Reference{"$ref": "#/components/parameters/accountIdOrNickname"}],
       responses: %{
         200 => Operation.response("Account", "application/json", Account),
+        401 => Operation.response("Error", "application/json", ApiError),
         404 => Operation.response("Error", "application/json", ApiError)
       }
     }
@@ -142,6 +143,7 @@ def statuses_operation do
         ] ++ pagination_params(),
       responses: %{
         200 => Operation.response("Statuses", "application/json", array_of_statuses()),
+        401 => Operation.response("Error", "application/json", ApiError),
         404 => Operation.response("Error", "application/json", ApiError)
       }
     }
@@ -201,14 +203,23 @@ def follow_operation do
       security: [%{"oAuth" => ["follow", "write:follows"]}],
       description: "Follow the given account",
       parameters: [
-        %Reference{"$ref": "#/components/parameters/accountIdOrNickname"},
+        %Reference{"$ref": "#/components/parameters/accountIdOrNickname"}
-        Operation.parameter(
-          :reblogs,
-          :query,
-          BooleanLike,
-          "Receive this account's reblogs in home timeline? Defaults to true."
-        )
       ],
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            type: :object,
+            properties: %{
+              reblogs: %Schema{
+                type: :boolean,
+                description: "Receive this account's reblogs in home timeline? Defaults to true.",
+                default: true
+              }
+            }
+          },
+          required: false
+        ),
       responses: %{
         200 => Operation.response("Relationship", "application/json", AccountRelationship),
         400 => Operation.response("Error", "application/json", ApiError),
@@ -436,6 +447,7 @@ defp create_request do
     }
   end
 
+  # TODO: This is actually a token respone, but there's no oauth operation file yet.
   defp create_response do
     %Schema{
       title: "AccountCreateResponse",
@@ -444,14 +456,20 @@ defp create_response do
       properties: %{
         token_type: %Schema{type: :string},
         access_token: %Schema{type: :string},
-        scope: %Schema{type: :array, items: %Schema{type: :string}},
+        refresh_token: %Schema{type: :string},
-        created_at: %Schema{type: :integer, format: :"date-time"}
+        scope: %Schema{type: :string},
+        created_at: %Schema{type: :integer, format: :"date-time"},
+        me: %Schema{type: :string},
+        expires_in: %Schema{type: :integer}
       },
       example: %{
+        "token_type" => "Bearer",
         "access_token" => "i9hAVVzGld86Pl5JtLtizKoXVvtTlSCJvwaugCxvZzk",
+        "refresh_token" => "i9hAVVzGld86Pl5JtLtizKoXVvtTlSCJvwaugCxvZzz",
         "created_at" => 1_585_918_714,
-        "scope" => ["read", "write", "follow", "push"],
+        "expires_in" => 600,
-        "token_type" => "Bearer"
+        "scope" => "read write follow push",
+        "me" => "https://gensokyo.2hu/users/raymoo"
       }
     }
   end

lib/pleroma/web/api_spec/operations/admin/media_proxy_cache_operation.ex

@@ -0,0 +1,109 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ApiSpec.Admin.MediaProxyCacheOperation do
+  alias OpenApiSpex.Operation
+  alias OpenApiSpex.Schema
+  alias Pleroma.Web.ApiSpec.Schemas.ApiError
+
+  import Pleroma.Web.ApiSpec.Helpers
+
+  def open_api_operation(action) do
+    operation = String.to_existing_atom("#{action}_operation")
+    apply(__MODULE__, operation, [])
+  end
+
+  def index_operation do
+    %Operation{
+      tags: ["Admin", "MediaProxyCache"],
+      summary: "Fetch a paginated list of all banned MediaProxy URLs in Cachex",
+      operationId: "AdminAPI.MediaProxyCacheController.index",
+      security: [%{"oAuth" => ["read:media_proxy_caches"]}],
+      parameters: [
+        Operation.parameter(
+          :page,
+          :query,
+          %Schema{type: :integer, default: 1},
+          "Page"
+        ),
+        Operation.parameter(
+          :page_size,
+          :query,
+          %Schema{type: :integer, default: 50},
+          "Number of statuses to return"
+        )
+      ],
+      responses: %{
+        200 => success_response()
+      }
+    }
+  end
+
+  def delete_operation do
+    %Operation{
+      tags: ["Admin", "MediaProxyCache"],
+      summary: "Remove a banned MediaProxy URL from Cachex",
+      operationId: "AdminAPI.MediaProxyCacheController.delete",
+      security: [%{"oAuth" => ["write:media_proxy_caches"]}],
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            type: :object,
+            required: [:urls],
+            properties: %{
+              urls: %Schema{type: :array, items: %Schema{type: :string, format: :uri}}
+            }
+          },
+          required: true
+        ),
+      responses: %{
+        200 => success_response(),
+        400 => Operation.response("Error", "application/json", ApiError)
+      }
+    }
+  end
+
+  def purge_operation do
+    %Operation{
+      tags: ["Admin", "MediaProxyCache"],
+      summary: "Purge and optionally ban a MediaProxy URL",
+      operationId: "AdminAPI.MediaProxyCacheController.purge",
+      security: [%{"oAuth" => ["write:media_proxy_caches"]}],
+      requestBody:
+        request_body(
+          "Parameters",
+          %Schema{
+            type: :object,
+            required: [:urls],
+            properties: %{
+              urls: %Schema{type: :array, items: %Schema{type: :string, format: :uri}},
+              ban: %Schema{type: :boolean, default: true}
+            }
+          },
+          required: true
+        ),
+      responses: %{
+        200 => success_response(),
+        400 => Operation.response("Error", "application/json", ApiError)
+      }
+    }
+  end
+
+  defp success_response do
+    Operation.response("Array of banned MediaProxy URLs in Cachex", "application/json", %Schema{
+      type: :object,
+      properties: %{
+        urls: %Schema{
+          type: :array,
+          items: %Schema{
+            type: :string,
+            format: :uri,
+            description: "MediaProxy URLs"
+          }
+        }
+      }
+    })
+  end
+end

lib/pleroma/web/api_spec/operations/notification_operation.ex

@@ -163,6 +163,13 @@ def notification do
           description:
             "Status that was the object of the notification, e.g. in mentions, reblogs, favourites, or polls.",
           nullable: true
+        },
+        pleroma: %Schema{
+          type: :object,
+          properties: %{
+            is_seen: %Schema{type: :boolean},
+            is_muted: %Schema{type: :boolean}
+          }
         }
       },
       example: %{
@@ -170,7 +177,8 @@ def notification do
         "type" => "mention",
         "created_at" => "2019-11-23T07:49:02.064Z",
         "account" => Account.schema().example,
-        "status" => Status.schema().example
+        "status" => Status.schema().example,
+        "pleroma" => %{"is_seen" => false, "is_muted" => false}
       }
     }
   end

lib/pleroma/web/api_spec/operations/pleroma_account_operation.ex

@@ -4,7 +4,6 @@
 
 defmodule Pleroma.Web.ApiSpec.PleromaAccountOperation do
   alias OpenApiSpex.Operation
-  alias OpenApiSpex.Schema
   alias Pleroma.Web.ApiSpec.Schemas.AccountRelationship
   alias Pleroma.Web.ApiSpec.Schemas.ApiError
   alias Pleroma.Web.ApiSpec.Schemas.FlakeID
@@ -40,48 +39,6 @@ def confirmation_resend_operation do
     }
   end
 
-  def update_avatar_operation do
-    %Operation{
-      tags: ["Accounts"],
-      summary: "Set/clear user avatar image",
-      operationId: "PleromaAPI.AccountController.update_avatar",
-      requestBody:
-        request_body("Parameters", update_avatar_or_background_request(), required: true),
-      security: [%{"oAuth" => ["write:accounts"]}],
-      responses: %{
-        200 => update_response(),
-        403 => Operation.response("Forbidden", "application/json", ApiError)
-      }
-    }
-  end
-
-  def update_banner_operation do
-    %Operation{
-      tags: ["Accounts"],
-      summary: "Set/clear user banner image",
-      operationId: "PleromaAPI.AccountController.update_banner",
-      requestBody: request_body("Parameters", update_banner_request(), required: true),
-      security: [%{"oAuth" => ["write:accounts"]}],
-      responses: %{
-        200 => update_response()
-      }
-    }
-  end
-
-  def update_background_operation do
-    %Operation{
-      tags: ["Accounts"],
-      summary: "Set/clear user background image",
-      operationId: "PleromaAPI.AccountController.update_background",
-      security: [%{"oAuth" => ["write:accounts"]}],
-      requestBody:
-        request_body("Parameters", update_avatar_or_background_request(), required: true),
-      responses: %{
-        200 => update_response()
-      }
-    }
-  end
-
   def favourites_operation do
     %Operation{
       tags: ["Accounts"],
@@ -136,52 +93,4 @@ defp id_param do
       required: true
     )
   end
-
-  defp update_avatar_or_background_request do
-    %Schema{
-      title: "PleromaAccountUpdateAvatarOrBackgroundRequest",
-      type: :object,
-      properties: %{
-        img: %Schema{
-          nullable: true,
-          type: :string,
-          format: :binary,
-          description: "Image encoded using `multipart/form-data` or an empty string to clear"
-        }
-      }
-    }
-  end
-
-  defp update_banner_request do
-    %Schema{
-      title: "PleromaAccountUpdateBannerRequest",
-      type: :object,
-      properties: %{
-        banner: %Schema{
-          type: :string,
-          nullable: true,
-          format: :binary,
-          description: "Image encoded using `multipart/form-data` or an empty string to clear"
-        }
-      }
-    }
-  end
-
-  defp update_response do
-    Operation.response("PleromaAccountUpdateResponse", "application/json", %Schema{
-      type: :object,
-      properties: %{
-        url: %Schema{
-          type: :string,
-          format: :uri,
-          nullable: true,
-          description: "Image URL"
-        }
-      },
-      example: %{
-        "url" =>
-          "https://cofe.party/media/9d0add56-bcb6-4c0f-8225-cbbd0b6dd773/13eadb6972c9ccd3f4ffa3b8196f0e0d38b4d2f27594457c52e52946c054cd9a.gif"
-      }
-    })
-  end
 end

lib/pleroma/web/api_spec/operations/pleroma_emoji_pack_operation.ex

@@ -33,6 +33,20 @@ def index_operation do
       tags: ["Emoji Packs"],
       summary: "Lists local custom emoji packs",
       operationId: "PleromaAPI.EmojiPackController.index",
+      parameters: [
+        Operation.parameter(
+          :page,
+          :query,
+          %Schema{type: :integer, default: 1},
+          "Page"
+        ),
+        Operation.parameter(
+          :page_size,
+          :query,
+          %Schema{type: :integer, default: 50},
+          "Number of emoji packs to return"
+        )
+      ],
       responses: %{
         200 => emoji_packs_response()
       }
@@ -44,7 +58,21 @@ def show_operation do
       tags: ["Emoji Packs"],
       summary: "Show emoji pack",
       operationId: "PleromaAPI.EmojiPackController.show",
-      parameters: [name_param()],
+      parameters: [
+        name_param(),
+        Operation.parameter(
+          :page,
+          :query,
+          %Schema{type: :integer, default: 1},
+          "Page"
+        ),
+        Operation.parameter(
+          :page_size,
+          :query,
+          %Schema{type: :integer, default: 30},
+          "Number of emoji to return"
+        )
+      ],
       responses: %{
         200 => Operation.response("Emoji Pack", "application/json", emoji_pack()),
         400 => Operation.response("Bad Request", "application/json", ApiError),

lib/pleroma/web/api_spec/operations/status_operation.ex

@@ -84,7 +84,7 @@ def delete_operation do
       operationId: "StatusController.delete",
       parameters: [id_param()],
       responses: %{
-        200 => empty_object_response(),
+        200 => status_response(),
         403 => Operation.response("Forbidden", "application/json", ApiError),
         404 => Operation.response("Not Found", "application/json", ApiError)
       }

lib/pleroma/web/api_spec/schemas/account.ex

@@ -40,20 +40,53 @@ defmodule Pleroma.Web.ApiSpec.Schemas.Account do
       pleroma: %Schema{
         type: :object,
         properties: %{
-          allow_following_move: %Schema{type: :boolean},
+          allow_following_move: %Schema{
-          background_image: %Schema{type: :string, nullable: true},
+            type: :boolean,
+            description: "whether the user allows automatically follow moved following accounts"
+          },
+          background_image: %Schema{type: :string, nullable: true, format: :uri},
           chat_token: %Schema{type: :string},
-          confirmation_pending: %Schema{type: :boolean},
+          confirmation_pending: %Schema{
+            type: :boolean,
+            description:
+              "whether the user account is waiting on email confirmation to be activated"
+          },
           hide_favorites: %Schema{type: :boolean},
-          hide_followers_count: %Schema{type: :boolean},
+          hide_followers_count: %Schema{
-          hide_followers: %Schema{type: :boolean},
+            type: :boolean,
-          hide_follows_count: %Schema{type: :boolean},
+            description: "whether the user has follower stat hiding enabled"
-          hide_follows: %Schema{type: :boolean},
+          },
-          is_admin: %Schema{type: :boolean},
+          hide_followers: %Schema{
-          is_moderator: %Schema{type: :boolean},
+            type: :boolean,
+            description: "whether the user has follower hiding enabled"
+          },
+          hide_follows_count: %Schema{
+            type: :boolean,
+            description: "whether the user has follow stat hiding enabled"
+          },
+          hide_follows: %Schema{
+            type: :boolean,
+            description: "whether the user has follow hiding enabled"
+          },
+          is_admin: %Schema{
+            type: :boolean,
+            description: "whether the user is an admin of the local instance"
+          },
+          is_moderator: %Schema{
+            type: :boolean,
+            description: "whether the user is a moderator of the local instance"
+          },
           skip_thread_containment: %Schema{type: :boolean},
-          tags: %Schema{type: :array, items: %Schema{type: :string}},
+          tags: %Schema{
-          unread_conversation_count: %Schema{type: :integer},
+            type: :array,
+            items: %Schema{type: :string},
+            description:
+              "List of tags being used for things like extra roles or moderation(ie. marking all media as nsfw all)."
+          },
+          unread_conversation_count: %Schema{
+            type: :integer,
+            description: "The count of unread conversations. Only returned to the account owner."
+          },
           notification_settings: %Schema{
             type: :object,
             properties: %{
@@ -66,7 +99,15 @@ defmodule Pleroma.Web.ApiSpec.Schemas.Account do
           },
           relationship: AccountRelationship,
           settings_store: %Schema{
-            type: :object
+            type: :object,
+            description:
+              "A generic map of settings for frontends. Opaque to the backend. Only returned in `verify_credentials` and `update_credentials`"
+          },
+          favicon: %Schema{
+            type: :string,
+            format: :uri,
+            nullable: true,
+            description: "Favicon image of the user's instance"
           }
         }
       },
@@ -74,16 +115,32 @@ defmodule Pleroma.Web.ApiSpec.Schemas.Account do
         type: :object,
         properties: %{
           fields: %Schema{type: :array, items: AccountField},
-          note: %Schema{type: :string},
+          note: %Schema{
+            type: :string,
+            description:
+              "Plaintext version of the bio without formatting applied by the backend, used for editing the bio."
+          },
           privacy: VisibilityScope,
           sensitive: %Schema{type: :boolean},
           pleroma: %Schema{
             type: :object,
             properties: %{
               actor_type: ActorType,
-              discoverable: %Schema{type: :boolean},
+              discoverable: %Schema{
-              no_rich_text: %Schema{type: :boolean},
+                type: :boolean,
-              show_role: %Schema{type: :boolean}
+                description:
+                  "whether the user allows discovery of the account in search results and other services."
+              },
+              no_rich_text: %Schema{
+                type: :boolean,
+                description:
+                  "whether the HTML tags for rich-text formatting are stripped from all statuses requested from the API."
+              },
+              show_role: %Schema{
+                type: :boolean,
+                description:
+                  "whether the user wants their role (e.g admin, moderator) to be shown"
+              }
             }
           }
         }

lib/pleroma/web/api_spec/schemas/status.ex

@@ -62,6 +62,11 @@ defmodule Pleroma.Web.ApiSpec.Schemas.Status do
         }
       },
       content: %Schema{type: :string, format: :html, description: "HTML-encoded status content"},
+      text: %Schema{
+        type: :string,
+        description: "Original unformatted content in plain text",
+        nullable: true
+      },
       created_at: %Schema{
         type: :string,
         format: "date-time",
@@ -184,6 +189,10 @@ defmodule Pleroma.Web.ApiSpec.Schemas.Status do
           thread_muted: %Schema{
             type: :boolean,
             description: "`true` if the thread the post belongs to is muted"
+          },
+          parent_visible: %Schema{
+            type: :boolean,
+            description: "`true` if the parent post is visible to the user"
           }
         }
       },

lib/pleroma/web/common_api/activity_draft.ex

@@ -186,6 +186,7 @@ defp object(draft) do
         draft.poll
       )
       |> Map.put("emoji", emoji)
+      |> Map.put("source", draft.status)
 
     %__MODULE__{draft | object: object}
   end

lib/pleroma/web/common_api/common_api.ex

@@ -25,6 +25,13 @@ defmodule Pleroma.Web.CommonAPI do
   require Pleroma.Constants
   require Logger
 
+  def block(blocker, blocked) do
+    with {:ok, block_data, _} <- Builder.block(blocker, blocked),
+         {:ok, block, _} <- Pipeline.common_pipeline(block_data, local: true) do
+      {:ok, block}
+    end
+  end
+
   def post_chat_message(%User{} = user, %User{} = recipient, content, opts \\ []) do
     with maybe_attachment <- opts[:media_id] && Object.get_by_id(opts[:media_id]),
          :ok <- validate_chat_content_length(content, !!maybe_attachment),
@@ -94,10 +101,14 @@ def unblock(blocker, blocked) do
   def follow(follower, followed) do
     timeout = Pleroma.Config.get([:activitypub, :follow_handshake_timeout])
 
-    with {:ok, follower} <- User.maybe_direct_follow(follower, followed),
+    with {:ok, follow_data, _} <- Builder.follow(follower, followed),
-         {:ok, activity} <- ActivityPub.follow(follower, followed),
+         {:ok, activity, _} <- Pipeline.common_pipeline(follow_data, local: true),
          {:ok, follower, followed} <- User.wait_and_refresh(timeout, follower, followed) do
-      {:ok, follower, followed, activity}
+      if activity.data["state"] == "reject" do
+        {:error, :rejected}
+      else
+        {:ok, follower, followed, activity}
+      end
     end
   end
 

lib/pleroma/web/common_api/utils.ex

@@ -143,7 +143,7 @@ def make_poll_data(%{"poll" => %{"expires_in" => expires_in}} = data)
 
   def make_poll_data(%{poll: %{options: options, expires_in: expires_in}} = data)
       when is_list(options) do
-    limits = Pleroma.Config.get([:instance, :poll_limits])
+    limits = Config.get([:instance, :poll_limits])
 
     with :ok <- validate_poll_expiration(expires_in, limits),
          :ok <- validate_poll_options_amount(options, limits),
@@ -502,7 +502,7 @@ def maybe_extract_mentions(_), do: []
   def make_report_content_html(nil), do: {:ok, {nil, [], []}}
 
   def make_report_content_html(comment) do
-    max_size = Pleroma.Config.get([:instance, :max_report_comment_size], 1000)
+    max_size = Config.get([:instance, :max_report_comment_size], 1000)
 
     if String.length(comment) <= max_size do
       {:ok, format_input(comment, "text/plain")}
@@ -564,7 +564,7 @@ def validate_character_limit("" = _full_payload, [] = _attachments) do
   end
 
   def validate_character_limit(full_payload, _attachments) do
-    limit = Pleroma.Config.get([:instance, :limit])
+    limit = Config.get([:instance, :limit])
     length = String.length(full_payload)
 
     if length <= limit do

lib/pleroma/web/fallback_redirect_controller.ex

@@ -9,6 +9,7 @@ defmodule Fallback.RedirectController do
 
   alias Pleroma.User
   alias Pleroma.Web.Metadata
+  alias Pleroma.Web.Preload
 
   def api_not_implemented(conn, _params) do
     conn
@@ -16,16 +17,7 @@ def api_not_implemented(conn, _params) do
     |> json(%{error: "Not implemented"})
   end
 
-  def redirector(conn, _params, code \\ 200)
+  def redirector(conn, _params, code \\ 200) do
-
-  # redirect to admin section
-  # /pleroma/admin -> /pleroma/admin/
-  #
-  def redirector(conn, %{"path" => ["pleroma", "admin"]} = _, _code) do
-    redirect(conn, to: "/pleroma/admin/")
-  end
-
-  def redirector(conn, _params, code) do
     conn
     |> put_resp_content_type("text/html")
     |> send_file(code, index_file_path())
@@ -43,28 +35,33 @@ def redirector_with_meta(conn, %{"maybe_nickname_or_id" => maybe_nickname_or_id}
   def redirector_with_meta(conn, params) do
     {:ok, index_content} = File.read(index_file_path())
 
-    tags =
+    tags = build_tags(conn, params)
-      try do
+    preloads = preload_data(conn, params)
-        Metadata.build_tags(params)
-      rescue
-        e ->
-          Logger.error(
-            "Metadata rendering for #{conn.request_path} failed.\n" <>
-              Exception.format(:error, e, __STACKTRACE__)
-          )
 
-          ""
+    response =
-      end
+      index_content
-
+      |> String.replace("<!--server-generated-meta-->", tags <> preloads)
-    response = String.replace(index_content, "<!--server-generated-meta-->", tags)
 
     conn
     |> put_resp_content_type("text/html")
     |> send_resp(200, response)
   end
 
-  def index_file_path do
+  def redirector_with_preload(conn, %{"path" => ["pleroma", "admin"]}) do
-    Pleroma.Plugs.InstanceStatic.file_path("index.html")
+    redirect(conn, to: "/pleroma/admin/")
+  end
+
+  def redirector_with_preload(conn, params) do
+    {:ok, index_content} = File.read(index_file_path())
+    preloads = preload_data(conn, params)
+
+    response =
+      index_content
+      |> String.replace("<!--server-generated-meta-->", preloads)
+
+    conn
+    |> put_resp_content_type("text/html")
+    |> send_resp(200, response)
   end
 
   def registration_page(conn, params) do
@@ -76,4 +73,36 @@ def empty(conn, _params) do
     |> put_status(204)
     |> text("")
   end
+
+  defp index_file_path do
+    Pleroma.Plugs.InstanceStatic.file_path("index.html")
+  end
+
+  defp build_tags(conn, params) do
+    try do
+      Metadata.build_tags(params)
+    rescue
+      e ->
+        Logger.error(
+          "Metadata rendering for #{conn.request_path} failed.\n" <>
+            Exception.format(:error, e, __STACKTRACE__)
+        )
+
+        ""
+    end
+  end
+
+  defp preload_data(conn, params) do
+    try do
+      Preload.build_tags(conn, params)
+    rescue
+      e ->
+        Logger.error(
+          "Preloading for #{conn.request_path} failed.\n" <>
+            Exception.format(:error, e, __STACKTRACE__)
+        )
+
+        ""
+    end
+  end
 end

lib/pleroma/web/masto_fe_controller.ex

@@ -49,7 +49,7 @@ def manifest(conn, _params) do
     |> render("manifest.json")
   end
 
-  @doc "PUT /api/web/settings"
+  @doc "PUT /api/web/settings: Backend-obscure settings blob for MastoFE, don't parse/reuse elsewhere"
   def put_settings(%{assigns: %{user: user}} = conn, %{"data" => settings} = _params) do
     with {:ok, _} <- User.mastodon_settings_update(user, settings) do
       json(conn, %{})

lib/pleroma/web/mastodon_api/controllers/account_controller.ex

@@ -20,11 +20,14 @@ defmodule Pleroma.Web.MastodonAPI.AccountController do
   alias Pleroma.Plugs.RateLimiter
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ActivityPub
+  alias Pleroma.Web.ActivityPub.Builder
+  alias Pleroma.Web.ActivityPub.Pipeline
   alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.MastodonAPI.ListView
   alias Pleroma.Web.MastodonAPI.MastodonAPI
   alias Pleroma.Web.MastodonAPI.MastodonAPIController
   alias Pleroma.Web.MastodonAPI.StatusView
+  alias Pleroma.Web.OAuth.OAuthView
   alias Pleroma.Web.OAuth.Token
   alias Pleroma.Web.TwitterAPI.TwitterAPI
 
@@ -99,12 +102,7 @@ def create(%{assigns: %{app: app}, body_params: params} = conn, _params) do
          :ok <- TwitterAPI.validate_captcha(app, params),
          {:ok, user} <- TwitterAPI.register_user(params, need_confirmation: true),
          {:ok, token} <- Token.create_token(app, user, %{scopes: app.scopes}) do
-      json(conn, %{
+      json(conn, OAuthView.render("token.json", %{user: user, token: token}))
-        token_type: "Bearer",
-        access_token: token.token,
-        scope: app.scopes,
-        created_at: Token.Utils.format_created_at(token)
-      })
     else
       {:error, error} -> json_response(conn, :bad_request, %{error: error})
     end
@@ -146,6 +144,13 @@ def update_credentials(%{assigns: %{user: user}, body_params: params} = conn, _p
       |> Enum.filter(fn {_, value} -> not is_nil(value) end)
       |> Enum.into(%{})
 
+    # We use an empty string as a special value to reset
+    # avatars, banners, backgrounds
+    user_image_value = fn
+      "" -> {:ok, nil}
+      value -> {:ok, value}
+    end
+
     user_params =
       [
         :no_rich_text,
@@ -166,9 +171,9 @@ def update_credentials(%{assigns: %{user: user}, body_params: params} = conn, _p
       |> Maps.put_if_present(:name, params[:display_name])
       |> Maps.put_if_present(:bio, params[:note])
       |> Maps.put_if_present(:raw_bio, params[:note])
-      |> Maps.put_if_present(:avatar, params[:avatar])
+      |> Maps.put_if_present(:avatar, params[:avatar], user_image_value)
-      |> Maps.put_if_present(:banner, params[:header])
+      |> Maps.put_if_present(:banner, params[:header], user_image_value)
-      |> Maps.put_if_present(:background, params[:pleroma_background_image])
+      |> Maps.put_if_present(:background, params[:pleroma_background_image], user_image_value)
       |> Maps.put_if_present(
         :raw_fields,
         params[:fields_attributes],
@@ -177,36 +182,44 @@ def update_credentials(%{assigns: %{user: user}, body_params: params} = conn, _p
       |> Maps.put_if_present(:pleroma_settings_store, params[:pleroma_settings_store])
       |> Maps.put_if_present(:default_scope, params[:default_scope])
       |> Maps.put_if_present(:default_scope, params["source"]["privacy"])
+      |> Maps.put_if_present(:actor_type, params[:bot], fn bot ->
+        if bot, do: {:ok, "Service"}, else: {:ok, "Person"}
+      end)
       |> Maps.put_if_present(:actor_type, params[:actor_type])
 
-    changeset = User.update_changeset(user, user_params)
+    # What happens here:
-
+    #
-    with {:ok, user} <- User.update_and_set_cache(changeset) do
+    # We want to update the user through the pipeline, but the ActivityPub
-      user
+    # update information is not quite enough for this, because this also
-      |> build_update_activity_params()
+    # contains local settings that don't federate and don't even appear
-      |> ActivityPub.update()
+    # in the Update activity.
-
+    #
-      render(conn, "show.json", user: user, for: user, with_pleroma_settings: true)
+    # So we first build the normal local changeset, then apply it to the
+    # user data, but don't persist it. With this, we generate the object
+    # data for our update activity. We feed this and the changeset as meta
+    # inforation into the pipeline, where they will be properly updated and
+    # federated.
+    with changeset <- User.update_changeset(user, user_params),
+         {:ok, unpersisted_user} <- Ecto.Changeset.apply_action(changeset, :update),
+         updated_object <-
+           Pleroma.Web.ActivityPub.UserView.render("user.json", user: user)
+           |> Map.delete("@context"),
+         {:ok, update_data, []} <- Builder.update(user, updated_object),
+         {:ok, _update, _} <-
+           Pipeline.common_pipeline(update_data,
+             local: true,
+             user_update_changeset: changeset
+           ) do
+      render(conn, "show.json",
+        user: unpersisted_user,
+        for: unpersisted_user,
+        with_pleroma_settings: true
+      )
     else
       _e -> render_error(conn, :forbidden, "Invalid request")
     end
   end
 
-  # Hotfix, handling will be redone with the pipeline
-  defp build_update_activity_params(user) do
-    object =
-      Pleroma.Web.ActivityPub.UserView.render("user.json", user: user)
-      |> Map.delete("@context")
-
-    %{
-      local: true,
-      to: [user.follower_address],
-      cc: [],
-      object: object,
-      actor: user.ap_id
-    }
-  end
-
   defp normalize_fields_attributes(fields) do
     if Enum.all?(fields, &is_tuple/1) do
       Enum.map(fields, fn {_, v} -> v end)
@@ -231,17 +244,17 @@ def relationships(%{assigns: %{user: _user}} = conn, _), do: json(conn, [])
   @doc "GET /api/v1/accounts/:id"
   def show(%{assigns: %{user: for_user}} = conn, %{id: nickname_or_id}) do
     with %User{} = user <- User.get_cached_by_nickname_or_id(nickname_or_id, for: for_user),
-         true <- User.visible_for?(user, for_user) do
+         :visible <- User.visible_for(user, for_user) do
       render(conn, "show.json", user: user, for: for_user)
     else
-      _e -> render_error(conn, :not_found, "Can't find user")
+      error -> user_visibility_error(conn, error)
     end
   end
 
   @doc "GET /api/v1/accounts/:id/statuses"
   def statuses(%{assigns: %{user: reading_user}} = conn, params) do
     with %User{} = user <- User.get_cached_by_nickname_or_id(params.id, for: reading_user),
-         true <- User.visible_for?(user, reading_user) do
+         :visible <- User.visible_for(user, reading_user) do
       params =
         params
         |> Map.delete(:tagged)
@@ -258,7 +271,17 @@ def statuses(%{assigns: %{user: reading_user}} = conn, params) do
         as: :activity
       )
     else
-      _e -> render_error(conn, :not_found, "Can't find user")
+      error -> user_visibility_error(conn, error)
+    end
+  end
+
+  defp user_visibility_error(conn, error) do
+    case error do
+      :restrict_unauthenticated ->
+        render_error(conn, :unauthorized, "This API requires an authenticated user")
+
+      _ ->
+        render_error(conn, :not_found, "Can't find user")
     end
   end
 
@@ -326,7 +349,7 @@ def follow(%{assigns: %{user: %{id: id}, account: %{id: id}}}, _params) do
     {:error, "Can not follow yourself"}
   end
 
-  def follow(%{assigns: %{user: follower, account: followed}} = conn, params) do
+  def follow(%{body_params: params, assigns: %{user: follower, account: followed}} = conn, _) do
     with {:ok, follower} <- MastodonAPI.follow(follower, followed, params) do
       render(conn, "relationship.json", user: follower, target: followed)
     else
@@ -365,8 +388,7 @@ def unmute(%{assigns: %{user: muter, account: muted}} = conn, _params) do
 
   @doc "POST /api/v1/accounts/:id/block"
   def block(%{assigns: %{user: blocker, account: blocked}} = conn, _params) do
-    with {:ok, _user_block} <- User.block(blocker, blocked),
+    with {:ok, _activity} <- CommonAPI.block(blocker, blocked) do
-         {:ok, _activity} <- ActivityPub.block(blocker, blocked) do
       render(conn, "relationship.json", user: blocker, target: blocked)
     else
       {:error, message} -> json_response(conn, :forbidden, %{error: message})

lib/pleroma/web/mastodon_api/controllers/search_controller.ex

@@ -44,6 +44,7 @@ def search2(conn, params), do: do_search(:v2, conn, params)
   def search(conn, params), do: do_search(:v1, conn, params)
 
   defp do_search(version, %{assigns: %{user: user}} = conn, %{q: query} = params) do
+    query = String.trim(query)
     options = search_options(params, user)
     timeout = Keyword.get(Repo.config(), :timeout, 15_000)
     default_values = %{"statuses" => [], "accounts" => [], "hashtags" => []}
@@ -107,21 +108,21 @@ defp resource_search(_, "statuses", query, options) do
     )
   end
 
-  defp resource_search(:v2, "hashtags", query, _options) do
+  defp resource_search(:v2, "hashtags", query, options) do
     tags_path = Web.base_url() <> "/tag/"
 
     query
-    |> prepare_tags()
+    |> prepare_tags(options)
     |> Enum.map(fn tag ->
       %{name: tag, url: tags_path <> tag}
     end)
   end
 
-  defp resource_search(:v1, "hashtags", query, _options) do
+  defp resource_search(:v1, "hashtags", query, options) do
-    prepare_tags(query)
+    prepare_tags(query, options)
   end
 
-  defp prepare_tags(query, add_joined_tag \\ true) do
+  defp prepare_tags(query, options) do
     tags =
       query
       |> preprocess_uri_query()
@@ -139,13 +140,20 @@ defp prepare_tags(query, add_joined_tag \\ true) do
 
     tags = Enum.map(tags, fn tag -> String.trim_leading(tag, "#") end)
 
-    if Enum.empty?(explicit_tags) && add_joined_tag do
+    tags =
-      tags
+      if Enum.empty?(explicit_tags) && !options[:skip_joined_tag] do
-      |> Kernel.++([joined_tag(tags)])
+        add_joined_tag(tags)
-      |> Enum.uniq_by(&String.downcase/1)
+      else
-    else
+        tags
-      tags
+      end
-    end
+
+    Pleroma.Pagination.paginate(tags, options)
+  end
+
+  defp add_joined_tag(tags) do
+    tags
+    |> Kernel.++([joined_tag(tags)])
+    |> Enum.uniq_by(&String.downcase/1)
   end
 
   # If `query` is a URI, returns last component of its path, otherwise returns `query`

lib/pleroma/web/mastodon_api/controllers/status_controller.ex

@@ -200,11 +200,16 @@ def show(%{assigns: %{user: user}} = conn, %{id: id}) do
 
   @doc "DELETE /api/v1/statuses/:id"
   def delete(%{assigns: %{user: user}} = conn, %{id: id}) do
-    with {:ok, %Activity{}} <- CommonAPI.delete(id, user) do
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
-      json(conn, %{})
+         {:ok, %Activity{}} <- CommonAPI.delete(id, user) do
+      try_render(conn, "show.json",
+        activity: activity,
+        for: user,
+        with_direct_conversation_id: true,
+        with_source: true
+      )
     else
-      {:error, :not_found} = e -> e
+      _e -> {:error, :not_found}
-      _e -> render_error(conn, :forbidden, "Can't delete this post")
     end
   end