Merge branch 'develop' of https://git.pleroma.social/pleroma/pleroma into develop

CHANGELOG.md

@@ -5,6 +5,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [unreleased]
 ### Added
+- Add a generic settings store for frontends / clients to use.
 - Optional SSH access mode. (Needs `erlang-ssh` package on some distributions).
 - [MongooseIM](https://github.com/esl/MongooseIM) http authentication support.
 - LDAP authentication
@@ -48,6 +49,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - OAuth: added job to clean expired access tokens
 - MRF: Support for rejecting reports from specific instances (`mrf_simple`)
 - MRF: Support for stripping avatars and banner images from specific instances (`mrf_simple`)
+- MRF: Support for running subchains.
+- Configuration: `skip_thread_containment` option
 
 ### Changed
 - **Breaking:** Configuration: move from Pleroma.Mailer to Pleroma.Emails.Mailer

config/config.exs

@@ -243,7 +243,8 @@
   max_report_comment_size: 1000,
   safe_dm_mentions: false,
   healthcheck: false,
-  remote_post_retention_days: 90
+  remote_post_retention_days: 90,
+  skip_thread_containment: false
 
 config :pleroma, :app_account_creation, enabled: true, max_requests: 25, interval: 1800
 
@@ -326,6 +327,8 @@
   federated_timeline_removal: [],
   replace: []
 
+config :pleroma, :mrf_subchain, match_actor: %{}
+
 config :pleroma, :rich_media, enabled: true
 
 config :pleroma, :media_proxy,
@@ -459,7 +462,11 @@
 config :esshd,
   enabled: false
 
-oauth_consumer_strategies = String.split(System.get_env("OAUTH_CONSUMER_STRATEGIES") || "")
+oauth_consumer_strategies =
+  System.get_env("OAUTH_CONSUMER_STRATEGIES")
+  |> to_string()
+  |> String.split()
+  |> Enum.map(&hd(String.split(&1, ":")))
 
 ueberauth_providers =
   for strategy <- oauth_consumer_strategies do

docs/api/differences_in_mastoapi_responses.md

@@ -43,6 +43,7 @@ Has these additional fields under the `pleroma` object:
 - `confirmation_pending`: boolean, true if a new user account is waiting on email confirmation to be activated
 - `hide_followers`: boolean, true when the user has follower hiding enabled
 - `hide_follows`: boolean, true when the user has follow hiding enabled
+- `settings_store`: A generic map of settings for frontends. Opaque to the backend. Only returned in `verify_credentials` and `update_credentials`
 
 ### Source
 
@@ -80,6 +81,15 @@ Additional parameters can be added to the JSON body/Form data:
 - `hide_favorites` - if true, user's favorites timeline will be hidden
 - `show_role` - if true, user's role (e.g admin, moderator) will be exposed to anyone in the API
 - `default_scope` - the scope returned under `privacy` key in Source subentity
+- `pleroma_settings_store` - Opaque user settings to be saved on the backend.
+- `skip_thread_containment` - if true, skip filtering out broken threads
+
+### Pleroma Settings Store
+Pleroma has mechanism that allows frontends to save blobs of json for each user on the backend. This can be used to save frontend-specific settings for a user that the backend does not need to know about.
+
+The parameter should have a form of `{frontend_name: {...}}`, with `frontend_name` identifying your type of client, e.g. `pleroma_fe`. It will overwrite everything under this property, but will not overwrite other frontend's settings.
+
+This information is returned in the `verify_credentials` endpoint.
 
 ## Authentication
 

docs/config.md

@@ -86,6 +86,7 @@ config :pleroma, Pleroma.Emails.Mailer,
   * `Pleroma.Web.ActivityPub.MRF.NoOpPolicy`: Doesn’t modify activities (default)
   * `Pleroma.Web.ActivityPub.MRF.DropPolicy`: Drops all activities. It generally doesn’t makes sense to use in production
   * `Pleroma.Web.ActivityPub.MRF.SimplePolicy`: Restrict the visibility of activities from certains instances (See ``:mrf_simple`` section)
+  * `Pleroma.Web.ActivityPub.MRF.SubchainPolicy`: Selectively runs other MRF policies when messages match (see ``:mrf_subchain`` section)
   * `Pleroma.Web.ActivityPub.MRF.RejectNonPublic`: Drops posts with non-public visibility settings (See ``:mrf_rejectnonpublic`` section)
   * `Pleroma.Web.ActivityPub.MRF.EnsureRePrepended`: Rewrites posts to ensure that replies to posts with subjects do not have an identical subject and instead begin with re:.
 * `public`: Makes the client API in authentificated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
@@ -110,6 +111,7 @@ config :pleroma, Pleroma.Emails.Mailer,
 * `safe_dm_mentions`: If set to true, only mentions at the beginning of a post will be used to address people in direct messages. This is to prevent accidental mentioning of people when talking about them (e.g. "@friend hey i really don't like @enemy"). (Default: `false`)
 * `healthcheck`: if set to true, system data will be shown on ``/api/pleroma/healthcheck``.
 * `remote_post_retention_days`: the default amount of days to retain remote posts when pruning the database
+* `skip_thread_containment`: Skip filter out broken threads. the default is `false`.
 
 ## :app_account_creation
 REST API for creating an account settings
@@ -229,6 +231,21 @@ relates to mascots on the mastodon frontend
 * `avatar_removal`: List of instances to strip avatars from
 * `banner_removal`: List of instances to strip banners from
 
+## :mrf_subchain
+This policy processes messages through an alternate pipeline when a given message matches certain criteria.
+All criteria are configured as a map of regular expressions to lists of policy modules.
+
+* `match_actor`: Matches a series of regular expressions against the actor field.
+
+Example:
+
+```
+config :pleroma, :mrf_subchain,
+  match_actor: %{
+    ~r/https:\/\/example.com/s => [Pleroma.Web.ActivityPub.MRF.DropPolicy]
+  }
+```
+
 ## :mrf_rejectnonpublic
 * `allow_followersonly`: whether to allow followers-only posts
 * `allow_direct`: whether to allow direct messages
@@ -497,7 +514,7 @@ Authentication / authorization settings.
 
 * `auth_template`: authentication form template. By default it's `show.html` which corresponds to `lib/pleroma/web/templates/o_auth/o_auth/show.html.eex`.
 * `oauth_consumer_template`: OAuth consumer mode authentication form template. By default it's `consumer.html` which corresponds to `lib/pleroma/web/templates/o_auth/o_auth/consumer.html.eex`.
-* `oauth_consumer_strategies`: the list of enabled OAuth consumer strategies; by default it's set by OAUTH_CONSUMER_STRATEGIES environment variable.
+* `oauth_consumer_strategies`: the list of enabled OAuth consumer strategies; by default it's set by OAUTH_CONSUMER_STRATEGIES environment variable. Each entry in this space-delimited string should be of format `<strategy>` or `<strategy>:<dependency>` (e.g. `twitter` or `keycloak:ueberauth_keycloak_strategy` in case dependency is named differently than `ueberauth_<strategy>`).
 
 ## OAuth consumer mode
 

lib/pleroma/conversation/participation.ex

@@ -79,5 +79,6 @@ def for_user_with_last_activity_id(user, params \\ %{}) do
         | last_activity_id: activity_id
       }
     end)
+    |> Enum.filter(& &1.last_activity_id)
   end
 end

lib/pleroma/emoji.ex

@@ -97,10 +97,22 @@ defp load do
         # There was some other error
         Logger.error("Could not access the custom emoji directory #{emoji_dir_path}: #{e}")
 
-      {:ok, packs} ->
+      {:ok, results} ->
+        grouped = Enum.group_by(results, &File.dir?/1)
+        packs = grouped[true] || []
+        files = grouped[false] || []
+
         # Print the packs we've found
         Logger.info("Found emoji packs: #{Enum.join(packs, ", ")}")
 
+        if not Enum.empty?(files) do
+          Logger.warn(
+            "Found files in the emoji folder. These will be ignored, please move them to a subdirectory\nFound files: #{
+              Enum.join(files, ", ")
+            }"
+          )
+        end
+
         emojis =
           Enum.flat_map(
             packs,

lib/pleroma/user/info.ex

@@ -44,6 +44,7 @@ defmodule Pleroma.User.Info do
     field(:pinned_activities, {:array, :string}, default: [])
     field(:mascot, :map, default: nil)
     field(:emoji, {:array, :map}, default: [])
+    field(:pleroma_settings_store, :map, default: %{})
 
     field(:notification_settings, :map,
       default: %{
@@ -54,6 +55,8 @@ defmodule Pleroma.User.Info do
       }
     )
 
+    field(:skip_thread_containment, :boolean, default: false)
+
     # Found in the wild
     # ap_id -> Where is this used?
     # bio -> Where is this used?
@@ -218,7 +221,9 @@ def profile_update(info, params) do
       :hide_followers,
       :hide_favorites,
       :background,
-      :show_role
+      :show_role,
+      :skip_thread_containment,
+      :pleroma_settings_store
     ])
   end
 

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -4,6 +4,7 @@
 
 defmodule Pleroma.Web.ActivityPub.ActivityPub do
   alias Pleroma.Activity
+  alias Pleroma.Config
   alias Pleroma.Conversation
   alias Pleroma.Notification
   alias Pleroma.Object
@@ -73,7 +74,7 @@ defp check_actor_is_active(actor) do
   end
 
   defp check_remote_limit(%{"object" => %{"content" => content}}) when not is_nil(content) do
-    limit = Pleroma.Config.get([:instance, :remote_limit])
+    limit = Config.get([:instance, :remote_limit])
     String.length(content) <= limit
   end
 
@@ -411,8 +412,8 @@ def delete(%Object{data: %{"id" => id, "actor" => actor}} = object, local \\ tru
   end
 
   def block(blocker, blocked, activity_id \\ nil, local \\ true) do
-    outgoing_blocks = Pleroma.Config.get([:activitypub, :outgoing_blocks])
+    outgoing_blocks = Config.get([:activitypub, :outgoing_blocks])
-    unfollow_blocked = Pleroma.Config.get([:activitypub, :unfollow_blocked])
+    unfollow_blocked = Config.get([:activitypub, :unfollow_blocked])
 
     if unfollow_blocked do
       follow_activity = fetch_latest_follow(blocker, blocked)
@@ -557,14 +558,11 @@ defp restrict_visibility(query, %{visibility: visibility})
 
   defp restrict_visibility(query, %{visibility: visibility})
        when visibility in @valid_visibilities do
-    query =
     from(
       a in query,
       where:
         fragment("activity_visibility(?, ?, ?) = ?", a.actor, a.recipients, a.data, ^visibility)
     )
-
-    query
   end
 
   defp restrict_visibility(_query, %{visibility: visibility})
@@ -574,17 +572,24 @@ defp restrict_visibility(_query, %{visibility: visibility})
 
   defp restrict_visibility(query, _visibility), do: query
 
-  defp restrict_thread_visibility(query, %{"user" => %User{ap_id: ap_id}}) do
+  defp restrict_thread_visibility(query, _, %{skip_thread_containment: true} = _),
-    query =
+    do: query
+
+  defp restrict_thread_visibility(
+         query,
+         %{"user" => %User{info: %{skip_thread_containment: true}}},
+         _
+       ),
+       do: query
+
+  defp restrict_thread_visibility(query, %{"user" => %User{ap_id: ap_id}}, _) do
     from(
       a in query,
       where: fragment("thread_visibility(?, (?)->>'id') = true", ^ap_id, a.data)
     )
-
-    query
   end
 
-  defp restrict_thread_visibility(query, _), do: query
+  defp restrict_thread_visibility(query, _, _), do: query
 
   def fetch_user_activities(user, reading_user, params \\ %{}) do
     params =
@@ -863,6 +868,10 @@ defp maybe_order(query, _), do: query
   def fetch_activities_query(recipients, opts \\ %{}) do
     base_query = from(activity in Activity)
 
+    config = %{
+      skip_thread_containment: Config.get([:instance, :skip_thread_containment])
+    }
+
     base_query
     |> maybe_preload_objects(opts)
     |> maybe_preload_bookmarks(opts)
@@ -882,7 +891,7 @@ def fetch_activities_query(recipients, opts \\ %{}) do
     |> restrict_muted(opts)
     |> restrict_media(opts)
     |> restrict_visibility(opts)
-    |> restrict_thread_visibility(opts)
+    |> restrict_thread_visibility(opts, config)
     |> restrict_replies(opts)
     |> restrict_reblogs(opts)
     |> restrict_pinned(opts)

lib/pleroma/web/activity_pub/mrf.ex

@@ -5,8 +5,8 @@
 defmodule Pleroma.Web.ActivityPub.MRF do
   @callback filter(Map.t()) :: {:ok | :reject, Map.t()}
 
-  def filter(object) do
+  def filter(policies, %{} = object) do
-    get_policies()
+    policies
     |> Enum.reduce({:ok, object}, fn
       policy, {:ok, object} ->
         policy.filter(object)
@@ -16,6 +16,8 @@ def filter(object) do
     end)
   end
 
+  def filter(%{} = object), do: get_policies() |> filter(object)
+
   def get_policies do
     Pleroma.Config.get([:instance, :rewrite_policy], []) |> get_policies()
   end

lib/pleroma/web/activity_pub/mrf/subchain_policy.ex

@@ -0,0 +1,40 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.MRF.SubchainPolicy do
+  alias Pleroma.Config
+  alias Pleroma.Web.ActivityPub.MRF
+
+  require Logger
+
+  @behaviour MRF
+
+  defp lookup_subchain(actor) do
+    with matches <- Config.get([:mrf_subchain, :match_actor]),
+         {match, subchain} <- Enum.find(matches, fn {k, _v} -> String.match?(actor, k) end) do
+      {:ok, match, subchain}
+    else
+      _e -> {:error, :notfound}
+    end
+  end
+
+  @impl true
+  def filter(%{"actor" => actor} = message) do
+    with {:ok, match, subchain} <- lookup_subchain(actor) do
+      Logger.debug(
+        "[SubchainPolicy] Matched #{actor} against #{inspect(match)} with subchain #{
+          inspect(subchain)
+        }"
+      )
+
+      subchain
+      |> MRF.filter(message)
+    else
+      _e -> {:ok, message}
+    end
+  end
+
+  @impl true
+  def filter(message), do: {:ok, message}
+end

lib/pleroma/web/activity_pub/utils.ex

@@ -794,10 +794,11 @@ def get_existing_votes(actor, %{data: %{"id" => id}}) do
     query =
       from(
         [activity, object: object] in Activity.with_preloaded_object(Activity),
+        where: fragment("(?)->>'type' = 'Create'", activity.data),
         where: fragment("(?)->>'actor' = ?", activity.data, ^actor),
         where:
           fragment(
-            "(?)->'inReplyTo' = ?",
+            "(?)->>'inReplyTo' = ?",
             object.data,
             ^to_string(id)
           ),

lib/pleroma/web/common_api/common_api.ex

@@ -132,6 +132,7 @@ def vote(user, object, choices) do
         Enum.map(choices, fn index ->
           answer_data = make_answer_data(user, object, Enum.at(options, index)["name"])
 
+          {:ok, activity} =
             ActivityPub.create(%{
               to: answer_data["to"],
               actor: user,
@@ -139,6 +140,8 @@ def vote(user, object, choices) do
               object: answer_data,
               additional: %{"cc" => answer_data["cc"]}
             })
+
+          activity
         end)
 
       object = Object.get_cached_by_ap_id(object.data["id"])

lib/pleroma/web/mastodon_api/mastodon_api_controller.ex

@@ -117,13 +117,24 @@ def update_credentials(%{assigns: %{user: user}} = conn, params) do
       |> Enum.dedup()
 
     info_params =
-      [:no_rich_text, :locked, :hide_followers, :hide_follows, :hide_favorites, :show_role]
+      [
+        :no_rich_text,
+        :locked,
+        :hide_followers,
+        :hide_follows,
+        :hide_favorites,
+        :show_role,
+        :skip_thread_containment
+      ]
       |> Enum.reduce(%{}, fn key, acc ->
         add_if_present(acc, params, to_string(key), key, fn value ->
           {:ok, ControllerHelper.truthy_param?(value)}
         end)
       end)
       |> add_if_present(params, "default_scope", :default_scope)
+      |> add_if_present(params, "pleroma_settings_store", :pleroma_settings_store, fn value ->
+        {:ok, Map.merge(user.info.pleroma_settings_store, value)}
+      end)
       |> add_if_present(params, "header", :banner, fn value ->
         with %Plug.Upload{} <- value,
              {:ok, object} <- ActivityPub.upload(value, type: :banner) do
@@ -143,7 +154,10 @@ def update_credentials(%{assigns: %{user: user}} = conn, params) do
         CommonAPI.update(user)
       end
 
-      json(conn, AccountView.render("account.json", %{user: user, for: user}))
+      json(
+        conn,
+        AccountView.render("account.json", %{user: user, for: user, with_pleroma_settings: true})
+      )
     else
       _e ->
         conn
@@ -153,7 +167,9 @@ def update_credentials(%{assigns: %{user: user}} = conn, params) do
   end
 
   def verify_credentials(%{assigns: %{user: user}} = conn, _) do
-    account = AccountView.render("account.json", %{user: user, for: user})
+    account =
+      AccountView.render("account.json", %{user: user, for: user, with_pleroma_settings: true})
+
     json(conn, account)
   end
 

lib/pleroma/web/mastodon_api/views/account_view.ex

@@ -124,12 +124,14 @@ defp do_render("account.json", %{user: user} = opts) do
         hide_followers: user.info.hide_followers,
         hide_follows: user.info.hide_follows,
         hide_favorites: user.info.hide_favorites,
-        relationship: relationship
+        relationship: relationship,
+        skip_thread_containment: user.info.skip_thread_containment
       }
     }
     |> maybe_put_role(user, opts[:for])
     |> maybe_put_settings(user, opts[:for], user_info)
     |> maybe_put_notification_settings(user, opts[:for])
+    |> maybe_put_settings_store(user, opts[:for], opts)
   end
 
   defp username_from_nickname(string) when is_binary(string) do
@@ -152,6 +154,15 @@ defp maybe_put_settings(
 
   defp maybe_put_settings(data, _, _, _), do: data
 
+  defp maybe_put_settings_store(data, %User{info: info, id: id}, %User{id: id}, %{
+         with_pleroma_settings: true
+       }) do
+    data
+    |> Kernel.put_in([:pleroma, :settings_store], info.pleroma_settings_store)
+  end
+
+  defp maybe_put_settings_store(data, _, _, _), do: data
+
   defp maybe_put_role(data, %User{info: %{show_role: true}} = user, _) do
     data
     |> Kernel.put_in([:pleroma, :is_admin], user.info.is_admin)

lib/pleroma/web/nodeinfo/nodeinfo_controller.ex

@@ -97,6 +97,7 @@ def raw_nodeinfo do
         "pleroma_api",
         "mastodon_api",
         "mastodon_api_streaming",
+        "polls",
         if Config.get([:media_proxy, :enabled]) do
           "media_proxy"
         end,
@@ -149,6 +150,7 @@ def raw_nodeinfo do
         },
         staffAccounts: staff_accounts,
         federation: federation_response,
+        pollLimits: Config.get([:instance, :poll_limits]),
         postFormats: Config.get([:instance, :allowed_post_formats]),
         uploadLimits: %{
           general: Config.get([:instance, :upload_limit]),

lib/pleroma/web/streamer.ex

@@ -6,6 +6,7 @@ defmodule Pleroma.Web.Streamer do
   use GenServer
   require Logger
   alias Pleroma.Activity
+  alias Pleroma.Config
   alias Pleroma.Conversation.Participation
   alias Pleroma.Notification
   alias Pleroma.Object
@@ -224,11 +225,10 @@ def push_to_socket(topics, topic, %Activity{data: %{"type" => "Announce"}} = ite
         mutes = user.info.mutes || []
         reblog_mutes = user.info.muted_reblogs || []
 
-        parent = Object.normalize(item)
+        with parent when not is_nil(parent) <- Object.normalize(item),
-
+             true <- Enum.all?([blocks, mutes, reblog_mutes], &(item.actor not in &1)),
-        unless is_nil(parent) or item.actor in blocks or item.actor in mutes or
+             true <- Enum.all?([blocks, mutes], &(parent.data["actor"] not in &1)),
-                 item.actor in reblog_mutes or not ActivityPub.contain_activity(item, user) or
+             true <- thread_containment(item, user) do
-                 parent.data["actor"] in blocks or parent.data["actor"] in mutes do
           send(socket.transport_pid, {:text, represent_update(item, user)})
         end
       else
@@ -264,8 +264,8 @@ def push_to_socket(topics, topic, item) do
         blocks = user.info.blocks || []
         mutes = user.info.mutes || []
 
-        unless item.actor in blocks or item.actor in mutes or
+        with true <- Enum.all?([blocks, mutes], &(item.actor not in &1)),
-                 not ActivityPub.contain_activity(item, user) do
+             true <- thread_containment(item, user) do
           send(socket.transport_pid, {:text, represent_update(item, user)})
         end
       else
@@ -279,4 +279,15 @@ defp internal_topic(topic, socket) when topic in ~w[user direct] do
   end
 
   defp internal_topic(topic, _), do: topic
+
+  @spec thread_containment(Activity.t(), User.t()) :: boolean()
+  defp thread_containment(_activity, %User{info: %{skip_thread_containment: true}}), do: true
+
+  defp thread_containment(activity, user) do
+    if Config.get([:instance, :skip_thread_containment]) do
+      true
+    else
+      ActivityPub.contain_activity(activity, user)
+    end
+  end
 end

lib/pleroma/web/templates/layout/app.html.eex

@@ -63,13 +63,14 @@
 
       .scopes-input {
         display: flex;
+        flex-direction: column;
         margin-top: 1em;
         text-align: left;
         color: #89898a;
       }
 
       .scopes-input label:first-child {
-        flex-basis: 40%;
+        height: 2em;
       }
 
       .scopes {
@@ -80,13 +81,22 @@
       }
 
       .scope {
-        flex-basis: 100%;
         display: flex;
+        flex-basis: 100%;
         height: 2em;
         align-items: center;
       }
 
+      .scope:before {
+        color: #b9b9ba;
+        content: "✔\fe0e";
+        margin-left: 1em;
+        margin-right: 1em;
+      }
+
       [type="checkbox"] + label {
+        display: none;
+        cursor: pointer;
         margin: 0.5em;
       }
 
@@ -95,10 +105,12 @@
       }
 
       [type="checkbox"] + label:before {
+        cursor: pointer;
         display: inline-block;
         color: white;
         background-color: #121a24;
         border: 4px solid #121a24;
+        box-shadow: 0px 0px 1px 0 #d8a070;
         box-sizing: border-box;
         width: 1.2em;
         height: 1.2em;
@@ -128,7 +140,8 @@
         border-radius: 4px;
         border: none;
         padding: 10px;
-        margin-top: 30px;
+        margin-top: 20px;
+        margin-bottom: 20px;
         text-transform: uppercase;
         font-size: 16px;
         box-shadow: 0px 0px 2px 0px black,
@@ -147,8 +160,8 @@
         box-sizing: border-box;
         width: 100%;
         background-color: #931014;
+        border: 1px solid #a06060;
         border-radius: 4px;
-        border: none;
         padding: 10px;
         margin-top: 20px;
         font-weight: 500;
@@ -171,12 +184,27 @@
           margin-top: 0
         }
 
-        .scopes-input {
+        .scope {
-          flex-direction: column;
+          flex-basis: 0%;
         }
 
-        .scope {
+        .scope:before {
-          flex-basis: 50%;
+          content: "";
+          margin-left: 0em;
+          margin-right: 1em;
+        }
+
+        .scope:first-child:before {
+          margin-left: 1em;
+          content: "✔\fe0e";
+        }
+
+        .scope:after {
+          content: ",";
+        }
+
+        .scope:last-child:after {
+          content: "";
         }
       }
       .form-row {

lib/pleroma/web/templates/o_auth/o_auth/_scopes.html.eex

@@ -1,13 +1,19 @@
 <div class="scopes-input">
-  <%= label @form, :scope, "Permissions" %>
+  <%= label @form, :scope, "The following permissions will be granted" %>
-
   <div class="scopes">
     <%= for scope <- @available_scopes do %>
       <%# Note: using hidden input with `unchecked_value` in order to distinguish user's empty selection from `scope` param being omitted %>
+      <%= if scope in @scopes do %>
         <div class="scope">
           <%= checkbox @form, :"scope_#{scope}", value: scope in @scopes && scope, checked_value: scope, unchecked_value: "", name: "authorization[scope][]" %>
           <%= label @form, :"scope_#{scope}", String.capitalize(scope) %>
+          <%= if scope in @scopes && scope do %>
+            <%= String.capitalize(scope) %>
+          <% end %>
         </div>
+      <% else %>
+        <%= checkbox @form, :"scope_#{scope}", value: scope in @scopes && scope, checked_value: scope, unchecked_value: "", name: "authorization[scope][]" %>
+      <% end %>
     <% end %>
   </div>
 </div>

lib/pleroma/web/templates/o_auth/o_auth/consumer.html.eex

@@ -1,7 +1,9 @@
 <h2>Sign in with external provider</h2>
 
 <%= form_for @conn, o_auth_path(@conn, :prepare_request), [as: "authorization", method: "get"], fn f -> %>
-  <%= render @view_module, "_scopes.html", Map.put(assigns, :form, f) %>
+  <div style="display: none">
+    <%= render @view_module, "_scopes.html", Map.merge(assigns, %{form: f}) %>
+  </div>
 
   <%= hidden_input f, :client_id, value: @client_id %>
   <%= hidden_input f, :redirect_uri, value: @redirect_uri %>

lib/pleroma/web/templates/o_auth/o_auth/show.html.eex

@@ -6,26 +6,38 @@
 <% end %>
 
 <h2>OAuth Authorization</h2>
-
 <%= form_for @conn, o_auth_path(@conn, :authorize), [as: "authorization"], fn f -> %>
+
+<%= if @params["registration"] in ["true", true] do %>
+  <h3>This is the first time you visit! Please enter your Pleroma handle.</h3>
+  <p>Choose carefully! You won't be able to change this later. You will be able to change your display name, though.</p>
   <div class="input">
-  <%= label f, :name, "Name or email" %>
+    <%= label f, :nickname, "Pleroma Handle" %>
+    <%= text_input f, :nickname, placeholder: "lain" %>
+  </div>
+  <%= hidden_input f, :name, value: @params["name"] %>
+  <%= hidden_input f, :password, value: @params["password"] %>
+  <br>
+<% else %>
+  <div class="input">
+    <%= label f, :name, "Username" %>
     <%= text_input f, :name %>
   </div>
   <div class="input">
     <%= label f, :password, "Password" %>
     <%= password_input f, :password %>
   </div>
-
+  <%= submit "Log In" %>
   <%= render @view_module, "_scopes.html", Map.merge(assigns, %{form: f}) %>
+<% end %>
 
 <%= hidden_input f, :client_id, value: @client_id %>
 <%= hidden_input f, :response_type, value: @response_type %>
 <%= hidden_input f, :redirect_uri, value: @redirect_uri %>
 <%= hidden_input f, :state, value: @state %>
-<%= submit "Authorize" %>
 <% end %>
 
 <%= if Pleroma.Config.oauth_consumer_enabled?() do %>
   <%= render @view_module, Pleroma.Web.Auth.Authenticator.oauth_consumer_template(), assigns %>
 <% end %>
+

lib/pleroma/web/twitter_api/twitter_api_controller.ex

@@ -632,7 +632,15 @@ def raw_empty_array(conn, _params) do
 
   defp build_info_cng(user, params) do
     info_params =
-      ["no_rich_text", "locked", "hide_followers", "hide_follows", "hide_favorites", "show_role"]
+      [
+        "no_rich_text",
+        "locked",
+        "hide_followers",
+        "hide_follows",
+        "hide_favorites",
+        "show_role",
+        "skip_thread_containment"
+      ]
       |> Enum.reduce(%{}, fn key, res ->
         if value = params[key] do
           Map.put(res, key, value == "true")

lib/pleroma/web/twitter_api/views/user_view.ex

@@ -118,7 +118,8 @@ defp do_render("user.json", %{user: user = %User{}} = assigns) do
         "pleroma" =>
           %{
             "confirmation_pending" => user_info.confirmation_pending,
-            "tags" => user.tags
+            "tags" => user.tags,
+            "skip_thread_containment" => user.info.skip_thread_containment
           }
           |> maybe_with_activation_status(user, for_user)
           |> with_notification_settings(user, for_user)

mix.exs

@@ -51,16 +51,27 @@ def application do
   defp elixirc_paths(:test), do: ["lib", "test/support"]
   defp elixirc_paths(_), do: ["lib"]
 
+  # Specifies OAuth dependencies.
+  defp oauth_deps do
+    oauth_strategy_packages =
+      System.get_env("OAUTH_CONSUMER_STRATEGIES")
+      |> to_string()
+      |> String.split()
+      |> Enum.map(fn strategy_entry ->
+        with [_strategy, dependency] <- String.split(strategy_entry, ":") do
+          dependency
+        else
+          [strategy] -> "ueberauth_#{strategy}"
+        end
+      end)
+
+    for s <- oauth_strategy_packages, do: {String.to_atom(s), ">= 0.0.0"}
+  end
+
   # Specifies your project dependencies.
   #
   # Type `mix help deps` for examples and options.
   defp deps do
-    oauth_strategies = String.split(System.get_env("OAUTH_CONSUMER_STRATEGIES") || "")
-
-    oauth_deps =
-      for s <- oauth_strategies,
-          do: {String.to_atom("ueberauth_#{s}"), ">= 0.0.0"}
-
     [
       {:phoenix, "~> 1.4.1"},
       {:plug_cowboy, "~> 2.0"},
@@ -121,7 +132,7 @@ defp deps do
       {:ex_rated, "~> 1.2"},
       {:plug_static_index_html, "~> 1.0.0"},
       {:excoveralls, "~> 0.11.1", only: :test}
-    ] ++ oauth_deps
+    ] ++ oauth_deps()
   end
 
   # Aliases are shortcuts or tasks specific to the current project.
@@ -146,7 +157,8 @@ defp aliases do
   #   * the mix environment if different than prod
   defp version(version) do
     {git_tag, git_pre_release} =
-      with {tag, 0} <- System.cmd("git", ["describe", "--tags", "--abbrev=0"]),
+      with {tag, 0} <-
+             System.cmd("git", ["describe", "--tags", "--abbrev=0"], stderr_to_stdout: true),
            tag = String.trim(tag),
            {describe, 0} <- System.cmd("git", ["describe", "--tags", "--abbrev=8"]),
            describe = String.trim(describe),

priv/repo/migrations/20190603162018_add_object_in_reply_to_index.exs

@@ -0,0 +1,7 @@
+defmodule Pleroma.Repo.Migrations.AddObjectInReplyToIndex do
+  use Ecto.Migration
+
+  def change do
+    create index(:objects, ["(data->>'inReplyTo')"], name: :objects_in_reply_to_index)
+  end
+end

priv/repo/migrations/20190603173419_add_tag_index_to_objects.exs

@@ -0,0 +1,8 @@
+defmodule Pleroma.Repo.Migrations.AddTagIndexToObjects do
+  use Ecto.Migration
+
+  def change do
+    drop_if_exists index(:activities, ["(data #> '{\"object\",\"tag\"}')"], using: :gin, name: :activities_tags)
+    create index(:objects, ["(data->'tag')"], using: :gin, name: :objects_tags)
+  end
+end

test/conversation/participation_test.exs

@@ -86,4 +86,17 @@ test "gets all the participations for a user, ordered by updated at descending"
 
     assert participation_one.last_activity_id == activity_three.id
   end
+
+  test "Doesn't die when the conversation gets empty" do
+    user = insert(:user)
+
+    {:ok, activity} = CommonAPI.post(user, %{"status" => ".", "visibility" => "direct"})
+    [participation] = Participation.for_user_with_last_activity_id(user)
+
+    assert participation.last_activity_id == activity.id
+
+    {:ok, _} = CommonAPI.delete(activity.id, user)
+
+    [] = Participation.for_user_with_last_activity_id(user)
+  end
 end

test/object/containment_test.exs

@@ -6,6 +6,11 @@ defmodule Pleroma.Object.ContainmentTest do
 
   import Pleroma.Factory
 
+  setup_all do
+    Tesla.Mock.mock_global(fn env -> apply(HttpRequestMock, :request, [env]) end)
+    :ok
+  end
+
   describe "general origin containment" do
     test "contain_origin_from_id() catches obvious spoofing attempts" do
       data = %{

test/support/http_request_mock.ex

@@ -243,6 +243,14 @@ def get("https://niu.moe/users/rye", _, _, Accept: "application/activity+json")
      }}
   end
 
+  def get("https://n1u.moe/users/rye", _, _, Accept: "application/activity+json") do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/httpoison_mock/rye.json")
+     }}
+  end
+
   def get("http://mastodon.example.org/users/admin/statuses/100787282858396771", _, _, _) do
     {:ok,
      %Tesla.Env{
@@ -302,6 +310,10 @@ def get("http://mastodon.example.org/users/admin", _, _, Accept: "application/ac
      }}
   end
 
+  def get("http://mastodon.example.org/users/gargron", _, _, Accept: "application/activity+json") do
+    {:error, :nxdomain}
+  end
+
   def get(
         "http://mastodon.example.org/@admin/99541947525187367",
         _,
@@ -546,6 +558,15 @@ def get(
      }}
   end
 
+  def get(
+        "http://gs.example.org:4040/index.php/user/1",
+        _,
+        _,
+        Accept: "application/activity+json"
+      ) do
+    {:ok, %Tesla.Env{status: 406, body: ""}}
+  end
+
   def get("http://gs.example.org/index.php/api/statuses/user_timeline/1.atom", _, _, _) do
     {:ok,
      %Tesla.Env{

test/web/activity_pub/mrf/subchain_policy_test.exs

@@ -0,0 +1,32 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.MRF.SubchainPolicyTest do
+  use Pleroma.DataCase
+
+  alias Pleroma.Web.ActivityPub.MRF.DropPolicy
+  alias Pleroma.Web.ActivityPub.MRF.SubchainPolicy
+
+  @message %{
+    "actor" => "https://banned.com",
+    "type" => "Create",
+    "object" => %{"content" => "hi"}
+  }
+
+  test "it matches and processes subchains when the actor matches a configured target" do
+    Pleroma.Config.put([:mrf_subchain, :match_actor], %{
+      ~r/^https:\/\/banned.com/s => [DropPolicy]
+    })
+
+    {:reject, _} = SubchainPolicy.filter(@message)
+  end
+
+  test "it doesn't match and process subchains when the actor doesn't match a configured target" do
+    Pleroma.Config.put([:mrf_subchain, :match_actor], %{
+      ~r/^https:\/\/borked.com/s => [DropPolicy]
+    })
+
+    {:ok, _message} = SubchainPolicy.filter(@message)
+  end
+end

test/web/activity_pub/utils_test.exs

@@ -1,6 +1,7 @@
 defmodule Pleroma.Web.ActivityPub.UtilsTest do
   use Pleroma.DataCase
   alias Pleroma.Activity
+  alias Pleroma.Object
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.ActivityPub.Utils
@@ -204,4 +205,46 @@ test "make_json_ld_header/0" do
              ]
            }
   end
+
+  describe "get_existing_votes" do
+    test "fetches existing votes" do
+      user = insert(:user)
+      other_user = insert(:user)
+
+      {:ok, activity} =
+        CommonAPI.post(user, %{
+          "status" => "How do I pronounce LaTeX?",
+          "poll" => %{
+            "options" => ["laytekh", "lahtekh", "latex"],
+            "expires_in" => 20,
+            "multiple" => true
+          }
+        })
+
+      object = Object.normalize(activity)
+      {:ok, votes, object} = CommonAPI.vote(other_user, object, [0, 1])
+      assert Enum.sort(Utils.get_existing_votes(other_user.ap_id, object)) == Enum.sort(votes)
+    end
+
+    test "fetches only Create activities" do
+      user = insert(:user)
+      other_user = insert(:user)
+
+      {:ok, activity} =
+        CommonAPI.post(user, %{
+          "status" => "Are we living in a society?",
+          "poll" => %{
+            "options" => ["yes", "no"],
+            "expires_in" => 20
+          }
+        })
+
+      object = Object.normalize(activity)
+      {:ok, [vote], object} = CommonAPI.vote(other_user, object, [0])
+      vote_object = Object.normalize(vote)
+      {:ok, _activity, _object} = ActivityPub.like(user, vote_object)
+      [fetched_vote] = Utils.get_existing_votes(other_user.ap_id, object)
+      assert fetched_vote.id == vote.id
+    end
+  end
 end

test/web/activity_pub/visibilty_test.exs

@@ -1,6 +1,7 @@
 defmodule Pleroma.Web.ActivityPub.VisibilityTest do
   use Pleroma.DataCase
 
+  alias Pleroma.Activity
   alias Pleroma.Web.ActivityPub.Visibility
   alias Pleroma.Web.CommonAPI
   import Pleroma.Factory
@@ -121,4 +122,46 @@ test "get_visibility", %{
   test "get_visibility with directMessage flag" do
     assert Visibility.get_visibility(%{data: %{"directMessage" => true}}) == "direct"
   end
+
+  describe "entire_thread_visible_for_user?/2" do
+    test "returns false if not found activity", %{user: user} do
+      refute Visibility.entire_thread_visible_for_user?(%Activity{}, user)
+    end
+
+    test "returns true if activity hasn't 'Create' type", %{user: user} do
+      activity = insert(:like_activity)
+      assert Visibility.entire_thread_visible_for_user?(activity, user)
+    end
+
+    test "returns false when invalid recipients", %{user: user} do
+      author = insert(:user)
+
+      activity =
+        insert(:note_activity,
+          note:
+            insert(:note,
+              user: author,
+              data: %{"to" => ["test-user"]}
+            )
+        )
+
+      refute Visibility.entire_thread_visible_for_user?(activity, user)
+    end
+
+    test "returns true if user following to author" do
+      author = insert(:user)
+      user = insert(:user, following: [author.ap_id])
+
+      activity =
+        insert(:note_activity,
+          note:
+            insert(:note,
+              user: author,
+              data: %{"to" => [user.ap_id]}
+            )
+        )
+
+      assert Visibility.entire_thread_visible_for_user?(activity, user)
+    end
+  end
 end

test/web/mastodon_api/account_view_test.exs

@@ -67,7 +67,8 @@ test "Represent a user account" do
         hide_favorites: true,
         hide_followers: false,
         hide_follows: false,
-        relationship: %{}
+        relationship: %{},
+        skip_thread_containment: false
       }
     }
 
@@ -132,7 +133,8 @@ test "Represent a Service(bot) account" do
         hide_favorites: true,
         hide_followers: false,
         hide_follows: false,
-        relationship: %{}
+        relationship: %{},
+        skip_thread_containment: false
       }
     }
 
@@ -233,10 +235,26 @@ test "represent an embedded relationship" do
           domain_blocking: false,
           showing_reblogs: true,
           endorsed: false
-        }
+        },
+        skip_thread_containment: false
       }
     }
 
     assert expected == AccountView.render("account.json", %{user: user, for: other_user})
   end
+
+  test "returns the settings store if the requesting user is the represented user and it's requested specifically" do
+    user = insert(:user, %{info: %User.Info{pleroma_settings_store: %{fe: "test"}}})
+
+    result =
+      AccountView.render("account.json", %{user: user, for: user, with_pleroma_settings: true})
+
+    assert result.pleroma.settings_store == %{:fe => "test"}
+
+    result = AccountView.render("account.json", %{user: user, with_pleroma_settings: true})
+    assert result.pleroma[:settings_store] == nil
+
+    result = AccountView.render("account.json", %{user: user, for: user})
+    assert result.pleroma[:settings_store] == nil
+  end
 end

test/web/mastodon_api/mastodon_api_controller_test.exs

@@ -2423,6 +2423,66 @@ test "hides favorites for new users by default", %{conn: conn, current_user: cur
   end
 
   describe "updating credentials" do
+    test "sets user settings in a generic way", %{conn: conn} do
+      user = insert(:user)
+
+      res_conn =
+        conn
+        |> assign(:user, user)
+        |> patch("/api/v1/accounts/update_credentials", %{
+          "pleroma_settings_store" => %{
+            pleroma_fe: %{
+              theme: "bla"
+            }
+          }
+        })
+
+      assert user = json_response(res_conn, 200)
+      assert user["pleroma"]["settings_store"] == %{"pleroma_fe" => %{"theme" => "bla"}}
+
+      user = Repo.get(User, user["id"])
+
+      res_conn =
+        conn
+        |> assign(:user, user)
+        |> patch("/api/v1/accounts/update_credentials", %{
+          "pleroma_settings_store" => %{
+            masto_fe: %{
+              theme: "bla"
+            }
+          }
+        })
+
+      assert user = json_response(res_conn, 200)
+
+      assert user["pleroma"]["settings_store"] ==
+               %{
+                 "pleroma_fe" => %{"theme" => "bla"},
+                 "masto_fe" => %{"theme" => "bla"}
+               }
+
+      user = Repo.get(User, user["id"])
+
+      res_conn =
+        conn
+        |> assign(:user, user)
+        |> patch("/api/v1/accounts/update_credentials", %{
+          "pleroma_settings_store" => %{
+            masto_fe: %{
+              theme: "blub"
+            }
+          }
+        })
+
+      assert user = json_response(res_conn, 200)
+
+      assert user["pleroma"]["settings_store"] ==
+               %{
+                 "pleroma_fe" => %{"theme" => "bla"},
+                 "masto_fe" => %{"theme" => "blub"}
+               }
+    end
+
     test "updates the user's bio", %{conn: conn} do
       user = insert(:user)
       user2 = insert(:user)
@@ -2479,6 +2539,19 @@ test "updates the user's hide_followers status", %{conn: conn} do
       assert user["pleroma"]["hide_followers"] == true
     end
 
+    test "updates the user's skip_thread_containment option", %{conn: conn} do
+      user = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> patch("/api/v1/accounts/update_credentials", %{skip_thread_containment: "true"})
+        |> json_response(200)
+
+      assert response["pleroma"]["skip_thread_containment"] == true
+      assert refresh_record(user).info.skip_thread_containment
+    end
+
     test "updates the user's hide_follows status", %{conn: conn} do
       user = insert(:user)
 

test/web/streamer_test.exs

@@ -11,6 +11,16 @@ defmodule Pleroma.Web.StreamerTest do
   alias Pleroma.Web.Streamer
   import Pleroma.Factory
 
+  setup do
+    skip_thread_containment = Pleroma.Config.get([:instance, :skip_thread_containment])
+
+    on_exit(fn ->
+      Pleroma.Config.put([:instance, :skip_thread_containment], skip_thread_containment)
+    end)
+
+    :ok
+  end
+
   test "it sends to public" do
     user = insert(:user)
     other_user = insert(:user)
@@ -68,6 +78,74 @@ test "it sends to public" do
     Task.await(task)
   end
 
+  describe "thread_containment" do
+    test "it doesn't send to user if recipients invalid and thread containment is enabled" do
+      Pleroma.Config.put([:instance, :skip_thread_containment], false)
+      author = insert(:user)
+      user = insert(:user, following: [author.ap_id])
+
+      activity =
+        insert(:note_activity,
+          note:
+            insert(:note,
+              user: author,
+              data: %{"to" => ["TEST-FFF"]}
+            )
+        )
+
+      task = Task.async(fn -> refute_receive {:text, _}, 1_000 end)
+      fake_socket = %{transport_pid: task.pid, assigns: %{user: user}}
+      topics = %{"public" => [fake_socket]}
+      Streamer.push_to_socket(topics, "public", activity)
+
+      Task.await(task)
+    end
+
+    test "it sends message if recipients invalid and thread containment is disabled" do
+      Pleroma.Config.put([:instance, :skip_thread_containment], true)
+      author = insert(:user)
+      user = insert(:user, following: [author.ap_id])
+
+      activity =
+        insert(:note_activity,
+          note:
+            insert(:note,
+              user: author,
+              data: %{"to" => ["TEST-FFF"]}
+            )
+        )
+
+      task = Task.async(fn -> assert_receive {:text, _}, 1_000 end)
+      fake_socket = %{transport_pid: task.pid, assigns: %{user: user}}
+      topics = %{"public" => [fake_socket]}
+      Streamer.push_to_socket(topics, "public", activity)
+
+      Task.await(task)
+    end
+
+    test "it sends message if recipients invalid and thread containment is enabled but user's thread containment is disabled" do
+      Pleroma.Config.put([:instance, :skip_thread_containment], false)
+      author = insert(:user)
+      user = insert(:user, following: [author.ap_id], info: %{skip_thread_containment: true})
+
+      activity =
+        insert(:note_activity,
+          note:
+            insert(:note,
+              user: author,
+              data: %{"to" => ["TEST-FFF"]}
+            )
+        )
+
+      task = Task.async(fn -> assert_receive {:text, _}, 1_000 end)
+      fake_socket = %{transport_pid: task.pid, assigns: %{user: user}}
+      topics = %{"public" => [fake_socket]}
+      Streamer.push_to_socket(topics, "public", activity)
+
+      Task.await(task)
+    end
+  end
+
   test "it doesn't send to blocked users" do
     user = insert(:user)
     blocked_user = insert(:user)

test/web/twitter_api/twitter_api_controller_test.exs

@@ -1495,7 +1495,7 @@ test "it sets and un-sets hide_follows", %{conn: conn} do
           "hide_follows" => "false"
         })
 
-      user = Repo.get!(User, user.id)
+      user = refresh_record(user)
       assert user.info.hide_follows == false
       assert json_response(conn, 200) == UserView.render("user.json", %{user: user, for: user})
     end
@@ -1548,6 +1548,29 @@ test "it sets and un-sets show_role", %{conn: conn} do
       assert json_response(conn, 200) == UserView.render("user.json", %{user: user, for: user})
     end
 
+    test "it sets and un-sets skip_thread_containment", %{conn: conn} do
+      user = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> post("/api/account/update_profile.json", %{"skip_thread_containment" => "true"})
+        |> json_response(200)
+
+      assert response["pleroma"]["skip_thread_containment"] == true
+      user = refresh_record(user)
+      assert user.info.skip_thread_containment
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> post("/api/account/update_profile.json", %{"skip_thread_containment" => "false"})
+        |> json_response(200)
+
+      assert response["pleroma"]["skip_thread_containment"] == false
+      refute refresh_record(user).info.skip_thread_containment
+    end
+
     test "it locks an account", %{conn: conn} do
       user = insert(:user)
 

test/web/twitter_api/views/user_view_test.exs

@@ -99,7 +99,8 @@ test "A user" do
       "fields" => [],
       "pleroma" => %{
         "confirmation_pending" => false,
-        "tags" => []
+        "tags" => [],
+        "skip_thread_containment" => false
       },
       "rights" => %{"admin" => false, "delete_others_notice" => false},
       "role" => "member"
@@ -154,7 +155,8 @@ test "A user for a given other follower", %{user: user} do
       "fields" => [],
       "pleroma" => %{
         "confirmation_pending" => false,
-        "tags" => []
+        "tags" => [],
+        "skip_thread_containment" => false
       },
       "rights" => %{"admin" => false, "delete_others_notice" => false},
       "role" => "member"
@@ -199,7 +201,8 @@ test "A user that follows you", %{user: user} do
       "fields" => [],
       "pleroma" => %{
         "confirmation_pending" => false,
-        "tags" => []
+        "tags" => [],
+        "skip_thread_containment" => false
       },
       "rights" => %{"admin" => false, "delete_others_notice" => false},
       "role" => "member"
@@ -281,7 +284,8 @@ test "A blocked user for the blocker" do
       "fields" => [],
       "pleroma" => %{
         "confirmation_pending" => false,
-        "tags" => []
+        "tags" => [],
+        "skip_thread_containment" => false
       },
       "rights" => %{"admin" => false, "delete_others_notice" => false},
       "role" => "member"