From 5d467af6c5299fd249a4c7d285be6f0839c635b3 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Mon, 4 Mar 2024 17:50:19 +0100
Subject: [PATCH 01/43] Update notes on security exploit handling

---
 SECURITY.md | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/SECURITY.md b/SECURITY.md
index c009d21d9..d37a8c9ca 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -1,16 +1,21 @@
-# Pleroma backend security policy
-
-## Supported versions
-
-Currently, Pleroma offers bugfixes and security patches only for the latest minor release.
-
-| Version | Support 
-|---------| --------
-| 2.2     | Bugfixes and security patches
+# Akkoma backend security handling
 
 ## Reporting a vulnerability
 
-Please use confidential issues (tick the "This issue is confidential and should only be visible to team members with at least Reporter access." box when submitting) at our [bugtracker](https://git.pleroma.social/pleroma/pleroma/-/issues/new) for reporting vulnerabilities.
+Please send an email (preferably encrypted) or
+a DM via our IRC to one of the following people:
+
+| Forgejo nick | IRC nick      | Email         | GPG                                     |
+| ------------ | ------------- | ------------- | --------------------------------------- |
+| floatinghost | FloatingGhost | *see GPG key* | https://coffee-and-dreams.uk/pubkey.asc |
+
 ## Announcements
 
-New releases are announced at [pleroma.social](https://pleroma.social/announcements/). All security releases are tagged with ["Security"](https://pleroma.social/announcements/tags/security/). You can be notified of them by subscribing to an Atom feed at <https://pleroma.social/announcements/tags/security/feed.xml>. 
+New releases and security issues are announced at
+[meta.akkoma.dev](https://meta.akkoma.dev/c/releases) and
+[@akkoma@ihatebeinga.live](https://ihatebeinga.live/akkoma).
+
+Both also offer RSS feeds
+([meta](https://meta.akkoma.dev/c/releases/7.rss),
+[fedi](https://ihatebeinga.live/users/akkoma.rss))
+so you can keep an eye on it without any accounts.

From dbb6091d0199678dd4ba887c3623d5df14c08c95 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Mon, 4 Mar 2024 17:50:20 +0100
Subject: [PATCH 02/43] Import copy of Plug.Static from Plug 1.15.3

The following commit will apply the needed patch
---
 .../web/plugs/static_no_content_type.ex       | 451 ++++++++++++++++++
 1 file changed, 451 insertions(+)
 create mode 100644 lib/pleroma/web/plugs/static_no_content_type.ex

diff --git a/lib/pleroma/web/plugs/static_no_content_type.ex b/lib/pleroma/web/plugs/static_no_content_type.ex
new file mode 100644
index 000000000..285e4ff30
--- /dev/null
+++ b/lib/pleroma/web/plugs/static_no_content_type.ex
@@ -0,0 +1,451 @@
+# This is almost identical to Plug.Static from Plug 1.15.3 (2024-01-16)
+# It being copied is a temporary measure to fix an urgent bug without
+# needing to wait for merge of a suitable patch upstream
+# The differences are:
+#  - this leading comment
+#  - renaming of the module from 'Plug.Static' to 'Pleroma.Web.Plugs.StaticNoCT'
+
+defmodule Pleroma.Web.Plugs.StaticNoCT do
+  @moduledoc """
+  A plug for serving static assets.
+
+  It requires two options:
+
+    * `:at` - the request path to reach for static assets.
+      It must be a string.
+
+    * `:from` - the file system path to read static assets from.
+      It can be either: a string containing a file system path, an
+      atom representing the application name (where assets will
+      be served from `priv/static`), a tuple containing the
+      application name and the directory to serve assets from (besides
+      `priv/static`), or an MFA tuple.
+
+  The preferred form is to use `:from` with an atom or tuple, since
+  it will make your application independent from the starting directory.
+  For example, if you pass:
+
+      plug Plug.Static, from: "priv/app/path"
+
+  Plug.Static will be unable to serve assets if you build releases
+  or if you change the current directory. Instead do:
+
+      plug Plug.Static, from: {:app_name, "priv/app/path"}
+
+  If a static asset cannot be found, `Plug.Static` simply forwards
+  the connection to the rest of the pipeline.
+
+  ## Cache mechanisms
+
+  `Plug.Static` uses etags for HTTP caching. This means browsers/clients
+  should cache assets on the first request and validate the cache on
+  following requests, not downloading the static asset once again if it
+  has not changed. The cache-control for etags is specified by the
+  `cache_control_for_etags` option and defaults to `"public"`.
+
+  However, `Plug.Static` also supports direct cache control by using
+  versioned query strings. If the request query string starts with
+  "?vsn=", `Plug.Static` assumes the application is versioning assets
+  and does not set the `ETag` header, meaning the cache behaviour will
+  be specified solely by the `cache_control_for_vsn_requests` config,
+  which defaults to `"public, max-age=31536000"`.
+
+  ## Options
+
+    * `:encodings` - list of 2-ary tuples where first value is value of
+      the `Accept-Encoding` header and second is extension of the file to
+      be served if given encoding is accepted by client. Entries will be tested
+      in order in list, so entries higher in list will be preferred. Defaults
+      to: `[]`.
+
+      In addition to setting this value directly it supports 2 additional
+      options for compatibility reasons:
+
+        + `:brotli` - will append `{"br", ".br"}` to the encodings list.
+        + `:gzip` - will append `{"gzip", ".gz"}` to the encodings list.
+
+      Additional options will be added in the above order (Brotli takes
+      preference over Gzip) to reflect older behaviour which was set due
+      to fact that Brotli in general provides better compression ratio than
+      Gzip.
+
+    * `:cache_control_for_etags` - sets the cache header for requests
+      that use etags. Defaults to `"public"`.
+
+    * `:etag_generation` - specify a `{module, function, args}` to be used
+      to generate   an etag. The `path` of the resource will be passed to
+      the function, as well as the `args`. If this option is not supplied,
+      etags will be generated based off of file size and modification time.
+      Note it is [recommended for the etag value to be quoted](https://tools.ietf.org/html/rfc7232#section-2.3),
+      which Plug won't do automatically.
+
+    * `:cache_control_for_vsn_requests` - sets the cache header for
+      requests starting with "?vsn=" in the query string. Defaults to
+      `"public, max-age=31536000"`.
+
+    * `:only` - filters which requests to serve. This is useful to avoid
+      file system access on every request when this plug is mounted
+      at `"/"`. For example, if `only: ["images", "favicon.ico"]` is
+      specified, only files in the "images" directory and the
+      "favicon.ico" file will be served by `Plug.Static`.
+      Note that `Plug.Static` matches these filters against request
+      uri and not against the filesystem. When requesting
+      a file with name containing non-ascii or special characters,
+      you should use urlencoded form. For example, you should write
+      `only: ["file%20name"]` instead of `only: ["file name"]`.
+      Defaults to `nil` (no filtering).
+
+    * `:only_matching` - a relaxed version of `:only` that will
+      serve any request as long as one of the given values matches the
+      given path. For example, `only_matching: ["images", "favicon"]`
+      will match any request that starts at "images" or "favicon",
+      be it "/images/foo.png", "/images-high/foo.png", "/favicon.ico"
+      or "/favicon-high.ico". Such matches are useful when serving
+      digested files at the root. Defaults to `nil` (no filtering).
+
+    * `:headers` - other headers to be set when serving static assets. Specify either
+      an enum of key-value pairs or a `{module, function, args}` to return an enum. The
+      `conn` will be passed to the function, as well as the `args`.
+
+    * `:content_types` - custom MIME type mapping. As a map with filename as key
+      and content type as value. For example:
+      `content_types: %{"apple-app-site-association" => "application/json"}`.
+
+  ## Examples
+
+  This plug can be mounted in a `Plug.Builder` pipeline as follows:
+
+      defmodule MyPlug do
+        use Plug.Builder
+
+        plug Plug.Static,
+          at: "/public",
+          from: :my_app,
+          only: ~w(images robots.txt)
+        plug :not_found
+
+        def not_found(conn, _) do
+          send_resp(conn, 404, "not found")
+        end
+      end
+
+  """
+
+  @behaviour Plug
+  @allowed_methods ~w(GET HEAD)
+
+  import Plug.Conn
+  alias Plug.Conn
+
+  # In this module, the `:prim_file` Erlang module along with the `:file_info`
+  # record are used instead of the more common and Elixir-y `File` module and
+  # `File.Stat` struct, respectively. The reason behind this is performance: all
+  # the `File` operations pass through a single process in order to support node
+  # operations that we simply don't need when serving assets.
+
+  require Record
+  Record.defrecordp(:file_info, Record.extract(:file_info, from_lib: "kernel/include/file.hrl"))
+
+  defmodule InvalidPathError do
+    defexception message: "invalid path for static asset", plug_status: 400
+  end
+
+  @impl true
+  def init(opts) do
+    from =
+      case Keyword.fetch!(opts, :from) do
+        {_, _} = from -> from
+        {_, _, _} = from -> from
+        from when is_atom(from) -> {from, "priv/static"}
+        from when is_binary(from) -> from
+        _ -> raise ArgumentError, ":from must be an atom, a binary or a tuple"
+      end
+
+    encodings =
+      opts
+      |> Keyword.get(:encodings, [])
+      |> maybe_add("br", ".br", Keyword.get(opts, :brotli, false))
+      |> maybe_add("gzip", ".gz", Keyword.get(opts, :gzip, false))
+
+    %{
+      encodings: encodings,
+      only_rules: {Keyword.get(opts, :only, []), Keyword.get(opts, :only_matching, [])},
+      qs_cache: Keyword.get(opts, :cache_control_for_vsn_requests, "public, max-age=31536000"),
+      et_cache: Keyword.get(opts, :cache_control_for_etags, "public"),
+      et_generation: Keyword.get(opts, :etag_generation, nil),
+      headers: Keyword.get(opts, :headers, %{}),
+      content_types: Keyword.get(opts, :content_types, %{}),
+      from: from,
+      at: opts |> Keyword.fetch!(:at) |> Plug.Router.Utils.split()
+    }
+  end
+
+  @impl true
+  def call(
+        conn = %Conn{method: meth},
+        %{at: at, only_rules: only_rules, from: from, encodings: encodings} = options
+      )
+      when meth in @allowed_methods do
+    segments = subset(at, conn.path_info)
+
+    if allowed?(only_rules, segments) do
+      segments = Enum.map(segments, &uri_decode/1)
+
+      if invalid_path?(segments) do
+        raise InvalidPathError, "invalid path for static asset: #{conn.request_path}"
+      end
+
+      path = path(from, segments)
+      range = get_req_header(conn, "range")
+      encoding = file_encoding(conn, path, range, encodings)
+      serve_static(encoding, conn, segments, range, options)
+    else
+      conn
+    end
+  end
+
+  def call(conn, _options) do
+    conn
+  end
+
+  defp uri_decode(path) do
+    # TODO: Remove rescue as this can't fail from Elixir v1.13
+    try do
+      URI.decode(path)
+    rescue
+      ArgumentError ->
+        raise InvalidPathError
+    end
+  end
+
+  defp allowed?(_only_rules, []), do: false
+  defp allowed?({[], []}, _list), do: true
+
+  defp allowed?({full, prefix}, [h | _]) do
+    h in full or (prefix != [] and match?({0, _}, :binary.match(h, prefix)))
+  end
+
+  defp serve_static({content_encoding, file_info, path}, conn, segments, range, options) do
+    %{
+      qs_cache: qs_cache,
+      et_cache: et_cache,
+      et_generation: et_generation,
+      headers: headers,
+      content_types: types
+    } = options
+
+    case put_cache_header(conn, qs_cache, et_cache, et_generation, file_info, path) do
+      {:stale, conn} ->
+        filename = List.last(segments)
+        content_type = Map.get(types, filename) || MIME.from_path(filename)
+
+        conn
+        |> put_resp_header("content-type", content_type)
+        |> put_resp_header("accept-ranges", "bytes")
+        |> maybe_add_encoding(content_encoding)
+        |> merge_headers(headers)
+        |> serve_range(file_info, path, range, options)
+
+      {:fresh, conn} ->
+        conn
+        |> maybe_add_vary(options)
+        |> send_resp(304, "")
+        |> halt()
+    end
+  end
+
+  defp serve_static(:error, conn, _segments, _range, _options) do
+    conn
+  end
+
+  defp serve_range(conn, file_info, path, [range], options) do
+    file_info(size: file_size) = file_info
+
+    with %{"bytes" => bytes} <- Plug.Conn.Utils.params(range),
+         {range_start, range_end} <- start_and_end(bytes, file_size) do
+      send_range(conn, path, range_start, range_end, file_size, options)
+    else
+      _ -> send_entire_file(conn, path, options)
+    end
+  end
+
+  defp serve_range(conn, _file_info, path, _range, options) do
+    send_entire_file(conn, path, options)
+  end
+
+  defp start_and_end("-" <> rest, file_size) do
+    case Integer.parse(rest) do
+      {last, ""} when last > 0 and last <= file_size -> {file_size - last, file_size - 1}
+      _ -> :error
+    end
+  end
+
+  defp start_and_end(range, file_size) do
+    case Integer.parse(range) do
+      {first, "-"} when first >= 0 ->
+        {first, file_size - 1}
+
+      {first, "-" <> rest} when first >= 0 ->
+        case Integer.parse(rest) do
+          {last, ""} when last >= first -> {first, min(last, file_size - 1)}
+          _ -> :error
+        end
+
+      _ ->
+        :error
+    end
+  end
+
+  defp send_range(conn, path, 0, range_end, file_size, options) when range_end == file_size - 1 do
+    send_entire_file(conn, path, options)
+  end
+
+  defp send_range(conn, path, range_start, range_end, file_size, _options) do
+    length = range_end - range_start + 1
+
+    conn
+    |> put_resp_header("content-range", "bytes #{range_start}-#{range_end}/#{file_size}")
+    |> send_file(206, path, range_start, length)
+    |> halt()
+  end
+
+  defp send_entire_file(conn, path, options) do
+    conn
+    |> maybe_add_vary(options)
+    |> send_file(200, path)
+    |> halt()
+  end
+
+  defp maybe_add_encoding(conn, nil), do: conn
+  defp maybe_add_encoding(conn, ce), do: put_resp_header(conn, "content-encoding", ce)
+
+  defp maybe_add_vary(conn, %{encodings: encodings}) do
+    # If we serve gzip or brotli at any moment, we need to set the proper vary
+    # header regardless of whether we are serving gzip content right now.
+    # See: http://www.fastly.com/blog/best-practices-for-using-the-vary-header/
+    if encodings != [] do
+      update_in(conn.resp_headers, &[{"vary", "Accept-Encoding"} | &1])
+    else
+      conn
+    end
+  end
+
+  defp put_cache_header(
+         %Conn{query_string: "vsn=" <> _} = conn,
+         qs_cache,
+         _et_cache,
+         _et_generation,
+         _file_info,
+         _path
+       )
+       when is_binary(qs_cache) do
+    {:stale, put_resp_header(conn, "cache-control", qs_cache)}
+  end
+
+  defp put_cache_header(conn, _qs_cache, et_cache, et_generation, file_info, path)
+       when is_binary(et_cache) do
+    etag = etag_for_path(file_info, et_generation, path)
+
+    conn =
+      conn
+      |> put_resp_header("cache-control", et_cache)
+      |> put_resp_header("etag", etag)
+
+    if etag in get_req_header(conn, "if-none-match") do
+      {:fresh, conn}
+    else
+      {:stale, conn}
+    end
+  end
+
+  defp put_cache_header(conn, _, _, _, _, _) do
+    {:stale, conn}
+  end
+
+  defp etag_for_path(file_info, et_generation, path) do
+    case et_generation do
+      {module, function, args} ->
+        apply(module, function, [path | args])
+
+      nil ->
+        file_info(size: size, mtime: mtime) = file_info
+        <<?", {size, mtime} |> :erlang.phash2() |> Integer.to_string(16)::binary, ?">>
+    end
+  end
+
+  defp file_encoding(conn, path, [_range], _encodings) do
+    # We do not support compression for range queries.
+    file_encoding(conn, path, nil, [])
+  end
+
+  defp file_encoding(conn, path, _range, encodings) do
+    encoded =
+      Enum.find_value(encodings, fn {encoding, ext} ->
+        if file_info = accept_encoding?(conn, encoding) && regular_file_info(path <> ext) do
+          {encoding, file_info, path <> ext}
+        end
+      end)
+
+    cond do
+      not is_nil(encoded) ->
+        encoded
+
+      file_info = regular_file_info(path) ->
+        {nil, file_info, path}
+
+      true ->
+        :error
+    end
+  end
+
+  defp regular_file_info(path) do
+    case :prim_file.read_file_info(path) do
+      {:ok, file_info(type: :regular) = file_info} ->
+        file_info
+
+      _ ->
+        nil
+    end
+  end
+
+  defp accept_encoding?(conn, encoding) do
+    encoding? = &String.contains?(&1, [encoding, "*"])
+
+    Enum.any?(get_req_header(conn, "accept-encoding"), fn accept ->
+      accept |> Plug.Conn.Utils.list() |> Enum.any?(encoding?)
+    end)
+  end
+
+  defp maybe_add(list, key, value, true), do: list ++ [{key, value}]
+  defp maybe_add(list, _key, _value, false), do: list
+
+  defp path({module, function, arguments}, segments)
+       when is_atom(module) and is_atom(function) and is_list(arguments),
+       do: Enum.join([apply(module, function, arguments) | segments], "/")
+
+  defp path({app, from}, segments) when is_atom(app) and is_binary(from),
+    do: Enum.join([Application.app_dir(app), from | segments], "/")
+
+  defp path(from, segments),
+    do: Enum.join([from | segments], "/")
+
+  defp subset([h | expected], [h | actual]), do: subset(expected, actual)
+  defp subset([], actual), do: actual
+  defp subset(_, _), do: []
+
+  defp invalid_path?(list) do
+    invalid_path?(list, :binary.compile_pattern(["/", "\\", ":", "\0"]))
+  end
+
+  defp invalid_path?([h | _], _match) when h in [".", "..", ""], do: true
+  defp invalid_path?([h | t], match), do: String.contains?(h, match) or invalid_path?(t)
+  defp invalid_path?([], _match), do: false
+
+  defp merge_headers(conn, {module, function, args}) do
+    merge_headers(conn, apply(module, function, [conn | args]))
+  end
+
+  defp merge_headers(conn, headers) do
+    merge_resp_headers(conn, headers)
+  end
+end

From 7ef93c0b6db47fa1bb76b22b66387e5a0ac891cf Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Mon, 4 Mar 2024 17:50:20 +0100
Subject: [PATCH 03/43] Add set_content_type to Plug.StaticNoCT

---
 .../web/plugs/static_no_content_type.ex       | 24 ++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/lib/pleroma/web/plugs/static_no_content_type.ex b/lib/pleroma/web/plugs/static_no_content_type.ex
index 285e4ff30..ea00a2d5d 100644
--- a/lib/pleroma/web/plugs/static_no_content_type.ex
+++ b/lib/pleroma/web/plugs/static_no_content_type.ex
@@ -4,6 +4,7 @@
 # The differences are:
 #  - this leading comment
 #  - renaming of the module from 'Plug.Static' to 'Pleroma.Web.Plugs.StaticNoCT'
+#  - additon of set_content_type option
 
 defmodule Pleroma.Web.Plugs.StaticNoCT do
   @moduledoc """
@@ -111,6 +112,13 @@ defmodule Pleroma.Web.Plugs.StaticNoCT do
       and content type as value. For example:
       `content_types: %{"apple-app-site-association" => "application/json"}`.
 
+    * `:set_content_type` - by default Plug.Static (re)sets the content type header
+      using auto-detection and the `:content_types` map. But when set to `false`
+      no content-type header will be inserted instead retaining the original
+      value or lack thereof. This can be useful when custom logic for appropiate
+      content types is needed which cannot be reasonably expressed as a static
+      filename map.
+
   ## Examples
 
   This plug can be mounted in a `Plug.Builder` pipeline as follows:
@@ -175,6 +183,7 @@ defmodule Pleroma.Web.Plugs.StaticNoCT do
       et_generation: Keyword.get(opts, :etag_generation, nil),
       headers: Keyword.get(opts, :headers, %{}),
       content_types: Keyword.get(opts, :content_types, %{}),
+      set_content_type: Keyword.get(opts, :set_content_type, true),
       from: from,
       at: opts |> Keyword.fetch!(:at) |> Plug.Router.Utils.split()
     }
@@ -225,22 +234,31 @@ defmodule Pleroma.Web.Plugs.StaticNoCT do
     h in full or (prefix != [] and match?({0, _}, :binary.match(h, prefix)))
   end
 
+  defp maybe_put_content_type(conn, false, _, _), do: conn
+
+  defp maybe_put_content_type(conn, _, types, filename) do
+    content_type = Map.get(types, filename) || MIME.from_path(filename)
+
+    conn
+    |> put_resp_header("content-type", content_type)
+  end
+
   defp serve_static({content_encoding, file_info, path}, conn, segments, range, options) do
     %{
       qs_cache: qs_cache,
       et_cache: et_cache,
       et_generation: et_generation,
       headers: headers,
-      content_types: types
+      content_types: types,
+      set_content_type: set_content_type
     } = options
 
     case put_cache_header(conn, qs_cache, et_cache, et_generation, file_info, path) do
       {:stale, conn} ->
         filename = List.last(segments)
-        content_type = Map.get(types, filename) || MIME.from_path(filename)
 
         conn
-        |> put_resp_header("content-type", content_type)
+        |> maybe_put_content_type(set_content_type, types, filename)
         |> put_resp_header("accept-ranges", "bytes")
         |> maybe_add_encoding(content_encoding)
         |> merge_headers(headers)

From f7c9793542711bfd7bc8bdf2d8be736a4eea9d15 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Mon, 4 Mar 2024 17:50:21 +0100
Subject: [PATCH 04/43] Sanitise Content-Type of uploads
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The lack thereof enables spoofing ActivityPub objects.

A malicious user could upload fake activities as attachments
and (if having access to remote search) trick local and remote
fedi instances into fetching and processing it as a valid object.

If uploads are hosted on the same domain as the instance itself,
it is possible for anyone with upload access to impersonate(!)
other users of the same instance.
If uploads are exclusively hosted on a different domain, even the most
basic check of domain of the object id and fetch url matching should
prevent impersonation. However, it may still be possible to trick
servers into accepting bogus users on the upload (sub)domain and bogus
notes attributed to such users.
Instances which later migrated to a different domain and have a
permissive redirect rule in place can still be vulnerable.
If — like Akkoma — the fetching server is overly permissive with
redirects, impersonation still works.

This was possible because Plug.Static also uses our custom
MIME type mappings used for actually authentic AP objects.

Provided external storage providers don’t somehow return ActivityStream
Content-Types on their own, instances using those are also safe against
their users being spoofed via uploads.

Akkoma instances using the OnlyMedia upload filter
cannot be exploited as a vector in this way — IF the
fetching server validates the Content-Type of
fetched objects (Akkoma itself does this already).

However, restricting uploads to only multimedia files may be a bit too
heavy-handed. Instead this commit will restrict the returned
Content-Type headers for user uploaded files to a safe subset, falling
back to generic 'application/octet-stream' for anything else.
This will also protect against non-AP payloads as e.g. used in
past frontend code injection attacks.

It’s a slight regression in user comfort, if say PDFs are uploaded,
but this trade-off seems fairly acceptable.

(Note, just excluding our own custom types would offer no protection
 against non-AP payloads and bear a (perhaps small) risk of a silent
 regression should MIME ever decide to add a canonical extension for
 ActivityPub objects)

Now, one might expect there to be other defence mechanisms
besides Content-Type preventing counterfeits from being accepted,
like e.g. validation of the queried URL and AP ID matching.
Inserting a self-reference into our uploads is hard, but unfortunately
*oma does not verify the id in such a way and happily accepts _anything_
from the same domain (without even considering redirects).
E.g. Sharkey (and possibly other *keys) seem to attempt to guard
against this by immediately refetching the object from its ID, but
this is easily circumvented by just uploading two payloads with the
ID of one linking to the other.

Unfortunately *oma is thus _both_ a vector for spoofing and
vulnerable to those spoof payloads, resulting in an easy way
to impersonate our users.

Similar flaws exists for emoji and media proxy.

Subsequent commits will fix this by rigorously sanitising
content types in more areas, hardening our checks, improving
the default config and discouraging insecure config options.
---
 CHANGELOG.md                            | 11 +++++++++++
 config/config.exs                       |  3 ++-
 config/description.exs                  | 13 +++++++++++++
 lib/pleroma/web/plugs/uploaded_media.ex | 21 +++++++++++++++++++--
 4 files changed, 45 insertions(+), 3 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9033050a1..1108c60e8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,17 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## Unreleased
+
+## Added
+
+## Changed
+
+## Fixed
+- Critical security issue allowing Akkoma to be used as a vector for
+  (depending on configuration) impersonation of other users or creation
+  of bogus users and posts on the upload domain
+
 ## 2024.02
 
 ## Added
diff --git a/config/config.exs b/config/config.exs
index 1c531344c..723d173ec 100644
--- a/config/config.exs
+++ b/config/config.exs
@@ -65,7 +65,8 @@ config :pleroma, Pleroma.Upload,
   link_name: false,
   proxy_remote: false,
   filename_display_max_length: 30,
-  base_url: nil
+  base_url: nil,
+  allowed_mime_types: ["image", "audio", "video"]
 
 config :pleroma, Pleroma.Uploaders.Local, uploads: "uploads"
 
diff --git a/config/description.exs b/config/description.exs
index e108aaae8..3ddd874f8 100644
--- a/config/description.exs
+++ b/config/description.exs
@@ -105,6 +105,19 @@ config :pleroma, :config_description, [
           "https://cdn-host.com"
         ]
       },
+      %{
+        key: :allowed_mime_types,
+        label: "Allowed MIME types",
+        type: {:list, :string},
+        description:
+          "List of MIME (main) types uploads are allowed to identify themselves with. Other types may still be uploaded, but will identify as a generic binary to clients. WARNING: Loosening this over the defaults can lead to security issues. Removing types is safe, but only add to the list if you are sure you know what you are doing.",
+        suggestions: [
+          "image",
+          "audio",
+          "video",
+          "font"
+        ]
+      },
       %{
         key: :proxy_remote,
         type: :boolean,
diff --git a/lib/pleroma/web/plugs/uploaded_media.ex b/lib/pleroma/web/plugs/uploaded_media.ex
index 300c33068..c0982b4af 100644
--- a/lib/pleroma/web/plugs/uploaded_media.ex
+++ b/lib/pleroma/web/plugs/uploaded_media.ex
@@ -28,7 +28,9 @@ defmodule Pleroma.Web.Plugs.UploadedMedia do
       |> Keyword.put(:at, "/__unconfigured_media_plug")
       |> Plug.Static.init()
 
-    %{static_plug_opts: static_plug_opts}
+    allowed_mime_types = Pleroma.Config.get([Pleroma.Upload, :allowed_mime_types])
+
+    %{static_plug_opts: static_plug_opts, allowed_mime_types: allowed_mime_types}
   end
 
   def call(%{request_path: <<"/", @path, "/", file::binary>>} = conn, opts) do
@@ -68,13 +70,28 @@ defmodule Pleroma.Web.Plugs.UploadedMedia do
 
   defp media_is_banned(_, _), do: false
 
+  defp get_safe_mime_type(%{allowed_mime_types: allowed_mime_types} = _opts, mime) do
+    [maintype | _] = String.split(mime, "/", parts: 2)
+    if maintype in allowed_mime_types, do: mime, else: "application/octet-stream"
+  end
+
+  defp set_content_type(conn, opts, filepath) do
+    real_mime = MIME.from_path(filepath)
+    clean_mime = get_safe_mime_type(opts, real_mime)
+    put_resp_header(conn, "content-type", clean_mime)
+  end
+
   defp get_media(conn, {:static_dir, directory}, opts) do
     static_opts =
       Map.get(opts, :static_plug_opts)
       |> Map.put(:at, [@path])
       |> Map.put(:from, directory)
+      |> Map.put(:set_content_type, false)
 
-    conn = Plug.Static.call(conn, static_opts)
+    conn =
+      conn
+      |> set_content_type(opts, conn.request_path)
+      |> Pleroma.Web.Plugs.StaticNoCT.call(static_opts)
 
     if conn.halted do
       conn

From bdefbb8fd99227f084220937da954ffe7fce15ee Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Tue, 5 Mar 2024 02:20:16 +0100
Subject: [PATCH 05/43] plug/upload_media: query config only once on init

---
 lib/pleroma/web/plugs/uploaded_media.ex | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/lib/pleroma/web/plugs/uploaded_media.ex b/lib/pleroma/web/plugs/uploaded_media.ex
index c0982b4af..9f13d919b 100644
--- a/lib/pleroma/web/plugs/uploaded_media.ex
+++ b/lib/pleroma/web/plugs/uploaded_media.ex
@@ -28,12 +28,21 @@ defmodule Pleroma.Web.Plugs.UploadedMedia do
       |> Keyword.put(:at, "/__unconfigured_media_plug")
       |> Plug.Static.init()
 
-    allowed_mime_types = Pleroma.Config.get([Pleroma.Upload, :allowed_mime_types])
+    config = Pleroma.Config.get(Pleroma.Upload)
+    allowed_mime_types = Keyword.fetch!(config, :allowed_mime_types)
+    uploader = Keyword.fetch!(config, :uploader)
 
-    %{static_plug_opts: static_plug_opts, allowed_mime_types: allowed_mime_types}
+    %{
+      static_plug_opts: static_plug_opts,
+      allowed_mime_types: allowed_mime_types,
+      uploader: uploader
+    }
   end
 
-  def call(%{request_path: <<"/", @path, "/", file::binary>>} = conn, opts) do
+  def call(
+        %{request_path: <<"/", @path, "/", file::binary>>} = conn,
+        %{uploader: uploader} = opts
+      ) do
     conn =
       case fetch_query_params(conn) do
         %{query_params: %{"name" => name}} = conn ->
@@ -46,10 +55,7 @@ defmodule Pleroma.Web.Plugs.UploadedMedia do
       end
       |> merge_resp_headers([{"content-security-policy", "sandbox"}])
 
-    config = Pleroma.Config.get(Pleroma.Upload)
-
-    with uploader <- Keyword.fetch!(config, :uploader),
-         {:ok, get_method} <- uploader.get_file(file),
+    with {:ok, get_method} <- uploader.get_file(file),
          false <- media_is_banned(conn, get_method) do
       get_media(conn, get_method, opts)
     else

From fef773ca3524ed102e97014b590728b30edf9ab5 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Mon, 4 Mar 2024 17:50:22 +0100
Subject: [PATCH 06/43] Drop media base_url default and recommend different
 domain

Same-domain setups enabled now at least two exploits,
so they ought to be discouraged and definitely not be the default.
---
 CHANGELOG.md                             |  2 ++
 docs/docs/configuration/cheatsheet.md    |  3 ++-
 docs/docs/configuration/hardening.md     | 10 ++++++++++
 lib/mix/tasks/pleroma/instance.ex        | 10 ++++++++++
 lib/pleroma/upload.ex                    | 13 +++++++++++--
 priv/templates/sample_config.eex         |  4 +++-
 test/mix/tasks/pleroma/instance_test.exs |  3 +++
 7 files changed, 41 insertions(+), 4 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1108c60e8..1efe1ecfc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ## Added
 
 ## Changed
+- `Pleroma.Upload, :base_url` now MUST be configured explicitly;
+  use of the same domain as the instance is **strongly** discouraged
 
 ## Fixed
 - Critical security issue allowing Akkoma to be used as a vector for
diff --git a/docs/docs/configuration/cheatsheet.md b/docs/docs/configuration/cheatsheet.md
index 3c9113f88..a04160a1d 100644
--- a/docs/docs/configuration/cheatsheet.md
+++ b/docs/docs/configuration/cheatsheet.md
@@ -598,7 +598,8 @@ the source code is here: [kocaptcha](https://github.com/koto-bank/kocaptcha). Th
 * `uploader`: Which one of the [uploaders](#uploaders) to use.
 * `filters`: List of [upload filters](#upload-filters) to use.
 * `link_name`: When enabled Akkoma will add a `name` parameter to the url of the upload, for example `https://instance.tld/media/corndog.png?name=corndog.png`. This is needed to provide the correct filename in Content-Disposition headers when using filters like `Pleroma.Upload.Filter.Dedupe`
-* `base_url`: The base URL to access a user-uploaded file. Useful when you want to host the media files via another domain or are using a 3rd party S3 provider.
+* `base_url`: The base URL to access a user-uploaded file; MUST be configured explicitly.
+  Using a (sub)domain distinct from the instance endpoint is **strongly** recommended.
 * `proxy_remote`: If you're using a remote uploader, Akkoma will proxy media requests instead of redirecting to it.
 * `proxy_opts`: Proxy options, see `Pleroma.ReverseProxy` documentation.
 * `filename_display_max_length`: Set max length of a filename to display. 0 = no limit. Default: 30.
diff --git a/docs/docs/configuration/hardening.md b/docs/docs/configuration/hardening.md
index 521183f7d..f8ba048dd 100644
--- a/docs/docs/configuration/hardening.md
+++ b/docs/docs/configuration/hardening.md
@@ -17,6 +17,16 @@ This sets the Akkoma application server to only listen to the localhost interfac
 
 This sets the `secure` flag on Akkoma’s session cookie. This makes sure, that the cookie is only accepted over encrypted HTTPs connections. This implicitly renames the cookie from `pleroma_key` to `__Host-pleroma-key` which enforces some restrictions. (see [cookie prefixes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie#Cookie_prefixes))
 
+### `Pleroma.Upload, :uploader, :base_url`
+
+> Recommended value: *anything on a different domain than the instance endpoint; e.g. https://media.myinstance.net/*
+
+Uploads are user controlled and (unless you’re running a true single-user
+instance) should therefore not be considered trusted. But the domain is used
+as a pivilege boundary e.g. by HTTP content security policy and ActivityPub.
+Having uploads on the same domain enabled several past vulnerabilities
+able to be exploited by malicious users.
+
 ### `:http_security`
 
 > Recommended value: `true`
diff --git a/lib/mix/tasks/pleroma/instance.ex b/lib/mix/tasks/pleroma/instance.ex
index 6a7d4f0d3..b442fdb5b 100644
--- a/lib/mix/tasks/pleroma/instance.ex
+++ b/lib/mix/tasks/pleroma/instance.ex
@@ -20,6 +20,7 @@ defmodule Mix.Tasks.Pleroma.Instance do
           output: :string,
           output_psql: :string,
           domain: :string,
+          media_url: :string,
           instance_name: :string,
           admin_email: :string,
           notify_email: :string,
@@ -64,6 +65,14 @@ defmodule Mix.Tasks.Pleroma.Instance do
           ":"
         ) ++ [443]
 
+      media_url =
+        get_option(
+          options,
+          :media_url,
+          "What base url will uploads use? (e.g https://media.example.com/media)\n" <>
+            "  Generally this should NOT use the same domain as the instance       "
+        )
+
       name =
         get_option(
           options,
@@ -207,6 +216,7 @@ defmodule Mix.Tasks.Pleroma.Instance do
         EEx.eval_file(
           template_dir <> "/sample_config.eex",
           domain: domain,
+          media_url: media_url,
           port: port,
           email: email,
           notify_email: notify_email,
diff --git a/lib/pleroma/upload.ex b/lib/pleroma/upload.ex
index 99b6b5215..974d12533 100644
--- a/lib/pleroma/upload.ex
+++ b/lib/pleroma/upload.ex
@@ -39,6 +39,8 @@ defmodule Pleroma.Upload do
   alias Pleroma.Web.ActivityPub.Utils
   require Logger
 
+  @mix_env Mix.env()
+
   @type source ::
           Plug.Upload.t()
           | (data_uri_string :: String.t())
@@ -228,6 +230,13 @@ defmodule Pleroma.Upload do
 
   defp url_from_spec(_upload, _base_url, {:url, url}), do: url
 
+  if @mix_env == :test do
+    defp choose_base_url(prim, sec \\ nil),
+      do: prim || sec || Pleroma.Web.Endpoint.url() <> "/media/"
+  else
+    defp choose_base_url(prim, sec \\ nil), do: prim || sec
+  end
+
   def base_url do
     uploader = Config.get([Pleroma.Upload, :uploader])
     upload_base_url = Config.get([Pleroma.Upload, :base_url])
@@ -235,7 +244,7 @@ defmodule Pleroma.Upload do
 
     case uploader do
       Pleroma.Uploaders.Local ->
-        upload_base_url || Pleroma.Web.Endpoint.url() <> "/media/"
+        choose_base_url(upload_base_url)
 
       Pleroma.Uploaders.S3 ->
         bucket = Config.get([Pleroma.Uploaders.S3, :bucket])
@@ -261,7 +270,7 @@ defmodule Pleroma.Upload do
         end
 
       _ ->
-        public_endpoint || upload_base_url || Pleroma.Web.Endpoint.url() <> "/media/"
+        choose_base_url(public_endpoint, upload_base_url)
     end
   end
 end
diff --git a/priv/templates/sample_config.eex b/priv/templates/sample_config.eex
index 0068969ac..724c5a9d5 100644
--- a/priv/templates/sample_config.eex
+++ b/priv/templates/sample_config.eex
@@ -78,6 +78,8 @@ config :joken, default_signer: "<%= jwt_secret %>"
 
 config :pleroma, configurable_from_database: <%= db_configurable? %>
 
+config :pleroma, Pleroma.Upload,
 <%= if Kernel.length(upload_filters) > 0 do
-"config :pleroma, Pleroma.Upload, filters: #{inspect(upload_filters)}"
+"  filters: #{inspect(upload_filters)},"
 end %>
+  base_url: "<%= media_url %>"
diff --git a/test/mix/tasks/pleroma/instance_test.exs b/test/mix/tasks/pleroma/instance_test.exs
index 5a5a68053..522319371 100644
--- a/test/mix/tasks/pleroma/instance_test.exs
+++ b/test/mix/tasks/pleroma/instance_test.exs
@@ -39,6 +39,8 @@ defmodule Mix.Tasks.Pleroma.InstanceTest do
         tmp_path() <> "setup.psql",
         "--domain",
         "test.pleroma.social",
+        "--media-url",
+        "https://media.pleroma.social/media",
         "--instance-name",
         "Pleroma",
         "--admin-email",
@@ -92,6 +94,7 @@ defmodule Mix.Tasks.Pleroma.InstanceTest do
     assert generated_config =~ "configurable_from_database: true"
     assert generated_config =~ "http: [ip: {127, 0, 0, 1}, port: 4000]"
     assert generated_config =~ "filters: [Pleroma.Upload.Filter.Exiftool]"
+    assert generated_config =~ "base_url: \"https://media.pleroma.social/media\""
     assert File.read!(tmp_path() <> "setup.psql") == generated_setup_psql()
     assert File.exists?(Path.expand("./test/instance/static/robots.txt"))
   end

From 0ec62acb9dc1c6500033086b46c37adefb700c62 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Mon, 4 Mar 2024 18:39:08 +0100
Subject: [PATCH 07/43] Always insert Dedupe upload filter

This actually was already intended before to eradict all future
path-traversal-style exploits and to fix issues with some
characters like akkoma#610 in 0b2ec0ccee. However, Dedupe and
AnonymizeFilename got mixed up. The latter only anonymises the name
in Content-Disposition headers GET parameters (with link_name),
_not_ the upload path.

Even without Dedupe, the upload path is prefixed by an UUID,
so it _should_ already be hard to guess for attackers. But now
we actually can be sure no path shenanigangs occur, uploads
reliably work and save some disk space.

While this makes the final path predictable, this prediction is
not exploitable. Insertion of a back-reference to the upload
itself requires pulling off a successfull preimage attack against
SHA-256, which is deemed infeasible for the foreseeable futures.

Dedupe was already included in the default list in config.exs
since 28cfb2c37a, but this will get overridde by whatever the
config generated by the "pleroma.instance gen" task chose.

Upload+delete tests running in parallel using Dedupe might be flaky, but
this was already true before and needs its own commit to fix eventually.
---
 CHANGELOG.md                                  |  2 +
 config/config.exs                             |  2 +-
 docs/docs/configuration/cheatsheet.md         | 17 +++---
 lib/mix/tasks/pleroma/instance.ex             | 21 +------
 lib/pleroma/upload.ex                         |  2 +-
 test/mix/tasks/pleroma/instance_test.exs      |  2 -
 test/mix/tasks/pleroma/uploads_test.exs       |  1 -
 test/pleroma/object_test.exs                  | 55 ++++++++++---------
 test/pleroma/upload_test.exs                  | 21 +------
 .../web/activity_pub/activity_pub_test.exs    |  8 ---
 .../controllers/mascot_controller_test.exs    |  6 +-
 test/test_helper.exs                          |  4 ++
 12 files changed, 55 insertions(+), 86 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1efe1ecfc..9130a81ae 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ## Changed
 - `Pleroma.Upload, :base_url` now MUST be configured explicitly;
   use of the same domain as the instance is **strongly** discouraged
+- The `Dedupe` upload filter is now always active;
+  `AnonymizeFilenames` is again opt-in
 
 ## Fixed
 - Critical security issue allowing Akkoma to be used as a vector for
diff --git a/config/config.exs b/config/config.exs
index 723d173ec..a366961c0 100644
--- a/config/config.exs
+++ b/config/config.exs
@@ -61,7 +61,7 @@ config :pleroma, Pleroma.Captcha.Kocaptcha, endpoint: "https://captcha.kotobank.
 # Upload configuration
 config :pleroma, Pleroma.Upload,
   uploader: Pleroma.Uploaders.Local,
-  filters: [Pleroma.Upload.Filter.Dedupe],
+  filters: [],
   link_name: false,
   proxy_remote: false,
   filename_display_max_length: 30,
diff --git a/docs/docs/configuration/cheatsheet.md b/docs/docs/configuration/cheatsheet.md
index a04160a1d..9c5bb9901 100644
--- a/docs/docs/configuration/cheatsheet.md
+++ b/docs/docs/configuration/cheatsheet.md
@@ -597,7 +597,7 @@ the source code is here: [kocaptcha](https://github.com/koto-bank/kocaptcha). Th
 
 * `uploader`: Which one of the [uploaders](#uploaders) to use.
 * `filters`: List of [upload filters](#upload-filters) to use.
-* `link_name`: When enabled Akkoma will add a `name` parameter to the url of the upload, for example `https://instance.tld/media/corndog.png?name=corndog.png`. This is needed to provide the correct filename in Content-Disposition headers when using filters like `Pleroma.Upload.Filter.Dedupe`
+* `link_name`: When enabled Akkoma will add a `name` parameter to the url of the upload, for example `https://instance.tld/media/corndog.png?name=corndog.png`. This is needed to provide the correct filename in Content-Disposition headers
 * `base_url`: The base URL to access a user-uploaded file; MUST be configured explicitly.
   Using a (sub)domain distinct from the instance endpoint is **strongly** recommended.
 * `proxy_remote`: If you're using a remote uploader, Akkoma will proxy media requests instead of redirecting to it.
@@ -639,17 +639,18 @@ config :ex_aws, :s3,
 
 ### Upload filters
 
-#### Pleroma.Upload.Filter.AnonymizeFilename
-
-This filter replaces the filename (not the path) of an upload. For complete obfuscation, add
-`Pleroma.Upload.Filter.Dedupe` before AnonymizeFilename.
-
-* `text`: Text to replace filenames in links. If empty, `{random}.extension` will be used. You can get the original filename extension by using `{extension}`, for example `custom-file-name.{extension}`.
-
 #### Pleroma.Upload.Filter.Dedupe
 
+**Always** active; cannot be turned off.
+Renames files to their hash and prevents duplicate files filling up the disk.
 No specific configuration.
 
+#### Pleroma.Upload.Filter.AnonymizeFilename
+
+This filter replaces the declared filename (not the path) of an upload.
+
+* `text`: Text to replace filenames in links. If empty, `{random}.extension` will be used. You can get the original filename extension by using `{extension}`, for example `custom-file-name.{extension}`.
+
 #### Pleroma.Upload.Filter.Exiftool
 
 This filter only strips the GPS and location metadata with Exiftool leaving color profiles and attributes intact.
diff --git a/lib/mix/tasks/pleroma/instance.ex b/lib/mix/tasks/pleroma/instance.ex
index b442fdb5b..44f6b6e70 100644
--- a/lib/mix/tasks/pleroma/instance.ex
+++ b/lib/mix/tasks/pleroma/instance.ex
@@ -36,8 +36,7 @@ defmodule Mix.Tasks.Pleroma.Instance do
           listen_ip: :string,
           listen_port: :string,
           strip_uploads: :string,
-          anonymize_uploads: :string,
-          dedupe_uploads: :string
+          anonymize_uploads: :string
         ],
         aliases: [
           o: :output,
@@ -195,14 +194,6 @@ defmodule Mix.Tasks.Pleroma.Instance do
           "n"
         ) === "y"
 
-      dedupe_uploads =
-        get_option(
-          options,
-          :dedupe_uploads,
-          "Do you want to deduplicate uploaded files? (y/n)",
-          "n"
-        ) === "y"
-
       Config.put([:instance, :static_dir], static_dir)
 
       secret = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
@@ -240,8 +231,7 @@ defmodule Mix.Tasks.Pleroma.Instance do
           upload_filters:
             upload_filters(%{
               strip: strip_uploads,
-              anonymize: anonymize_uploads,
-              dedupe: dedupe_uploads
+              anonymize: anonymize_uploads
             })
         )
 
@@ -329,13 +319,6 @@ defmodule Mix.Tasks.Pleroma.Instance do
         enabled_filters
       end
 
-    enabled_filters =
-      if filters.dedupe do
-        enabled_filters ++ [Pleroma.Upload.Filter.Dedupe]
-      else
-        enabled_filters
-      end
-
     enabled_filters
   end
 end
diff --git a/lib/pleroma/upload.ex b/lib/pleroma/upload.ex
index 974d12533..1158e9449 100644
--- a/lib/pleroma/upload.ex
+++ b/lib/pleroma/upload.ex
@@ -66,7 +66,7 @@ defmodule Pleroma.Upload do
           path: String.t()
         }
 
-  @always_enabled_filters [Pleroma.Upload.Filter.AnonymizeFilename]
+  @always_enabled_filters [Pleroma.Upload.Filter.Dedupe]
 
   defstruct [:id, :name, :tempfile, :content_type, :width, :height, :blurhash, :path]
 
diff --git a/test/mix/tasks/pleroma/instance_test.exs b/test/mix/tasks/pleroma/instance_test.exs
index 522319371..17b2e3267 100644
--- a/test/mix/tasks/pleroma/instance_test.exs
+++ b/test/mix/tasks/pleroma/instance_test.exs
@@ -71,8 +71,6 @@ defmodule Mix.Tasks.Pleroma.InstanceTest do
         "./test/../test/instance/static/",
         "--strip-uploads",
         "y",
-        "--dedupe-uploads",
-        "n",
         "--anonymize-uploads",
         "n"
       ])
diff --git a/test/mix/tasks/pleroma/uploads_test.exs b/test/mix/tasks/pleroma/uploads_test.exs
index 67fb642c1..d00e25a37 100644
--- a/test/mix/tasks/pleroma/uploads_test.exs
+++ b/test/mix/tasks/pleroma/uploads_test.exs
@@ -16,7 +16,6 @@ defmodule Mix.Tasks.Pleroma.UploadsTest do
       Mix.shell(Mix.Shell.IO)
     end)
 
-    File.mkdir_p!("test/uploads")
     :ok
   end
 
diff --git a/test/pleroma/object_test.exs b/test/pleroma/object_test.exs
index 8320660a5..4b0fec1bd 100644
--- a/test/pleroma/object_test.exs
+++ b/test/pleroma/object_test.exs
@@ -22,6 +22,13 @@ defmodule Pleroma.ObjectTest do
     :ok
   end
 
+  # Only works for a single attachment but that's all we need here
+  defp get_attachment_filepath(note, uploads_dir) do
+    %{data: %{"attachment" => [%{"url" => [%{"href" => href}]}]}} = note
+    filename = href |> Path.basename()
+    "#{uploads_dir}/#{filename}"
+  end
+
   test "returns an object by it's AP id" do
     object = insert(:note)
     found_object = Object.get_by_ap_id(object.data["id"])
@@ -95,14 +102,13 @@ defmodule Pleroma.ObjectTest do
       {:ok, %Object{} = attachment} =
         Pleroma.Web.ActivityPub.ActivityPub.upload(file, actor: user.ap_id)
 
-      %{data: %{"attachment" => [%{"url" => [%{"href" => href}]}]}} =
-        note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
+      note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
 
       uploads_dir = Pleroma.Config.get!([Pleroma.Uploaders.Local, :uploads])
 
-      path = href |> Path.dirname() |> Path.basename()
+      path = get_attachment_filepath(note, uploads_dir)
 
-      assert {:ok, ["an_image.jpg"]} == File.ls("#{uploads_dir}/#{path}")
+      assert File.exists?("#{path}")
 
       Object.delete(note)
 
@@ -111,7 +117,7 @@ defmodule Pleroma.ObjectTest do
       assert Object.get_by_id(note.id).data["deleted"]
       refute Object.get_by_id(attachment.id) == nil
 
-      assert {:ok, ["an_image.jpg"]} == File.ls("#{uploads_dir}/#{path}")
+      assert File.exists?("#{path}")
     end
 
     test "in subdirectories" do
@@ -129,14 +135,13 @@ defmodule Pleroma.ObjectTest do
       {:ok, %Object{} = attachment} =
         Pleroma.Web.ActivityPub.ActivityPub.upload(file, actor: user.ap_id)
 
-      %{data: %{"attachment" => [%{"url" => [%{"href" => href}]}]}} =
-        note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
+      note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
 
       uploads_dir = Pleroma.Config.get!([Pleroma.Uploaders.Local, :uploads])
 
-      path = href |> Path.dirname() |> Path.basename()
+      path = get_attachment_filepath(note, uploads_dir)
 
-      assert {:ok, ["an_image.jpg"]} == File.ls("#{uploads_dir}/#{path}")
+      assert File.exists?("#{path}")
 
       Object.delete(note)
 
@@ -145,7 +150,7 @@ defmodule Pleroma.ObjectTest do
       assert Object.get_by_id(note.id).data["deleted"]
       assert Object.get_by_id(attachment.id) == nil
 
-      assert {:ok, []} == File.ls("#{uploads_dir}/#{path}")
+      refute File.exists?("#{path}")
     end
 
     test "with dedupe enabled" do
@@ -168,13 +173,11 @@ defmodule Pleroma.ObjectTest do
       {:ok, %Object{} = attachment} =
         Pleroma.Web.ActivityPub.ActivityPub.upload(file, actor: user.ap_id)
 
-      %{data: %{"attachment" => [%{"url" => [%{"href" => href}]}]}} =
-        note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
+      note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
 
-      filename = Path.basename(href)
+      path = get_attachment_filepath(note, uploads_dir)
 
-      assert {:ok, files} = File.ls(uploads_dir)
-      assert filename in files
+      assert File.exists?("#{path}")
 
       Object.delete(note)
 
@@ -182,8 +185,8 @@ defmodule Pleroma.ObjectTest do
 
       assert Object.get_by_id(note.id).data["deleted"]
       assert Object.get_by_id(attachment.id) == nil
-      assert {:ok, files} = File.ls(uploads_dir)
-      refute filename in files
+      # what if another test runs concurrently using the same image file?
+      refute File.exists?("#{path}")
     end
 
     test "with objects that have legacy data.url attribute" do
@@ -203,14 +206,13 @@ defmodule Pleroma.ObjectTest do
 
       {:ok, %Object{}} = Object.create(%{url: "https://google.com", actor: user.ap_id})
 
-      %{data: %{"attachment" => [%{"url" => [%{"href" => href}]}]}} =
-        note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
+      note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
 
       uploads_dir = Pleroma.Config.get!([Pleroma.Uploaders.Local, :uploads])
 
-      path = href |> Path.dirname() |> Path.basename()
+      path = get_attachment_filepath(note, uploads_dir)
 
-      assert {:ok, ["an_image.jpg"]} == File.ls("#{uploads_dir}/#{path}")
+      assert File.exists?("#{path}")
 
       Object.delete(note)
 
@@ -219,7 +221,7 @@ defmodule Pleroma.ObjectTest do
       assert Object.get_by_id(note.id).data["deleted"]
       assert Object.get_by_id(attachment.id) == nil
 
-      assert {:ok, []} == File.ls("#{uploads_dir}/#{path}")
+      refute File.exists?("#{path}")
     end
 
     test "With custom base_url" do
@@ -238,14 +240,13 @@ defmodule Pleroma.ObjectTest do
       {:ok, %Object{} = attachment} =
         Pleroma.Web.ActivityPub.ActivityPub.upload(file, actor: user.ap_id)
 
-      %{data: %{"attachment" => [%{"url" => [%{"href" => href}]}]}} =
-        note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
+      note = insert(:note, %{user: user, data: %{"attachment" => [attachment.data]}})
 
       uploads_dir = Pleroma.Config.get!([Pleroma.Uploaders.Local, :uploads])
 
-      path = href |> Path.dirname() |> Path.basename()
+      path = get_attachment_filepath(note, uploads_dir)
 
-      assert {:ok, ["an_image.jpg"]} == File.ls("#{uploads_dir}/#{path}")
+      assert File.exists?("#{path}")
 
       Object.delete(note)
 
@@ -254,7 +255,7 @@ defmodule Pleroma.ObjectTest do
       assert Object.get_by_id(note.id).data["deleted"]
       assert Object.get_by_id(attachment.id) == nil
 
-      assert {:ok, []} == File.ls("#{uploads_dir}/#{path}")
+      refute File.exists?("#{path}")
     end
   end
 
diff --git a/test/pleroma/upload_test.exs b/test/pleroma/upload_test.exs
index ad6065b43..27a2d1b97 100644
--- a/test/pleroma/upload_test.exs
+++ b/test/pleroma/upload_test.exs
@@ -188,7 +188,7 @@ defmodule Pleroma.UploadTest do
       refute data["name"] == "an [image.jpg"
     end
 
-    test "escapes invalid characters in url" do
+    test "mangles the filename" do
       File.cp!("test/fixtures/image.jpg", "test/fixtures/image_tmp.jpg")
 
       file = %Plug.Upload{
@@ -200,23 +200,8 @@ defmodule Pleroma.UploadTest do
       {:ok, data} = Upload.store(file)
       [attachment_url | _] = data["url"]
 
-      assert Path.basename(attachment_url["href"]) == "an%E2%80%A6%20image.jpg"
-    end
-
-    test "escapes reserved uri characters" do
-      File.cp!("test/fixtures/image.jpg", "test/fixtures/image_tmp.jpg")
-
-      file = %Plug.Upload{
-        content_type: "image/jpeg",
-        path: Path.absname("test/fixtures/image_tmp.jpg"),
-        filename: ":?#[]@!$&\\'()*+,;=.jpg"
-      }
-
-      {:ok, data} = Upload.store(file)
-      [attachment_url | _] = data["url"]
-
-      assert Path.basename(attachment_url["href"]) ==
-               "%3A%3F%23%5B%5D%40%21%24%26%5C%27%28%29%2A%2B%2C%3B%3D.jpg"
+      refute Path.basename(attachment_url["href"]) == "an%E2%80%A6%20image.jpg"
+      refute Path.basename(attachment_url["href"]) == "an… image.jpg"
     end
   end
 
diff --git a/test/pleroma/web/activity_pub/activity_pub_test.exs b/test/pleroma/web/activity_pub/activity_pub_test.exs
index 5d5388cf5..69b4ac257 100644
--- a/test/pleroma/web/activity_pub/activity_pub_test.exs
+++ b/test/pleroma/web/activity_pub/activity_pub_test.exs
@@ -1304,14 +1304,6 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
       %{test_file: test_file}
     end
 
-    test "strips / from filename", %{test_file: file} do
-      file = %Plug.Upload{file | filename: "../../../../../nested/bad.jpg"}
-      {:ok, %Object{} = object} = ActivityPub.upload(file)
-      [%{"href" => href}] = object.data["url"]
-      assert Regex.match?(~r"/bad.jpg$", href)
-      refute Regex.match?(~r"/nested/", href)
-    end
-
     test "sets a description if given", %{test_file: file} do
       {:ok, %Object{} = object} = ActivityPub.upload(file, description: "a cool file")
       assert object.data["name"] == "a cool file"
diff --git a/test/pleroma/web/pleroma_api/controllers/mascot_controller_test.exs b/test/pleroma/web/pleroma_api/controllers/mascot_controller_test.exs
index 7f02bff8f..8829597eb 100644
--- a/test/pleroma/web/pleroma_api/controllers/mascot_controller_test.exs
+++ b/test/pleroma/web/pleroma_api/controllers/mascot_controller_test.exs
@@ -64,6 +64,10 @@ defmodule Pleroma.Web.PleromaAPI.MascotControllerTest do
 
     assert json_response_and_validate_schema(ret_conn, 200)
 
+    %{"url" => uploaded_url} = Jason.decode!(ret_conn.resp_body)
+
+    assert uploaded_url != nil and is_binary(uploaded_url)
+
     user = User.get_cached_by_id(user.id)
 
     conn =
@@ -72,6 +76,6 @@ defmodule Pleroma.Web.PleromaAPI.MascotControllerTest do
       |> get("/api/v1/pleroma/mascot")
 
     assert %{"url" => url, "type" => "image"} = json_response_and_validate_schema(conn, 200)
-    assert url =~ "an_image"
+    assert url == uploaded_url
   end
 end
diff --git a/test/test_helper.exs b/test/test_helper.exs
index 0fc7a86b9..22a0f33ee 100644
--- a/test/test_helper.exs
+++ b/test/test_helper.exs
@@ -9,6 +9,10 @@ Ecto.Adapters.SQL.Sandbox.mode(Pleroma.Repo, :manual)
 
 {:ok, _} = Application.ensure_all_started(:ex_machina)
 
+# Prepare and later automatically cleanup upload dir
+uploads_dir = Pleroma.Config.get([Pleroma.Uploaders.Local, :uploads], "test/uploads")
+File.mkdir_p!(uploads_dir)
+
 ExUnit.after_suite(fn _results ->
   uploads = Pleroma.Config.get([Pleroma.Uploaders.Local, :uploads], "test/uploads")
   File.rm_rf!(uploads)

From ba558c0c242fed48f0fec1cf7fe86642d07e410b Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Thu, 7 Mar 2024 00:00:25 +0100
Subject: [PATCH 08/43] Limit instance emoji to image types

Else malicious emoji packs or our EmojiStealer MRF can
put payloads into the same domain as the instance itself.
Sanitising the content type should prevent proper clients
from acting on any potential payload.

Note, this does not affect the default emoji shipped with Akkoma
as they are handled by another plug. However, those are fully trusted
and thus not in needed of sanitisation.
---
 lib/pleroma/web/plugs/instance_static.ex | 22 ++++++++++++++++++++--
 lib/pleroma/web/plugs/uploaded_media.ex  |  8 ++------
 lib/pleroma/web/plugs/utils.ex           | 14 ++++++++++++++
 3 files changed, 36 insertions(+), 8 deletions(-)
 create mode 100644 lib/pleroma/web/plugs/utils.ex

diff --git a/lib/pleroma/web/plugs/instance_static.ex b/lib/pleroma/web/plugs/instance_static.ex
index 5f9a6ee83..b72b604a1 100644
--- a/lib/pleroma/web/plugs/instance_static.ex
+++ b/lib/pleroma/web/plugs/instance_static.ex
@@ -3,8 +3,12 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.Plugs.InstanceStatic do
+  import Plug.Conn
+
   require Pleroma.Constants
 
+  alias Pleroma.Web.Plugs.Utils
+
   @moduledoc """
   This is a shim to call `Plug.Static` but with runtime `from` configuration.
 
@@ -43,11 +47,25 @@ defmodule Pleroma.Web.Plugs.InstanceStatic do
     conn
   end
 
-  defp call_static(conn, opts, from) do
+  defp set_static_content_type(conn, "/emoji/" <> _ = request_path) do
+    real_mime = MIME.from_path(request_path)
+    safe_mime = Utils.get_safe_mime_type(%{allowed_mime_types: ["image"]}, real_mime)
+
+    put_resp_header(conn, "content-type", safe_mime)
+  end
+
+  defp set_static_content_type(conn, request_path) do
+    put_resp_header(conn, "content-type", MIME.from_path(request_path))
+  end
+
+  defp call_static(%{request_path: request_path} = conn, opts, from) do
     opts =
       opts
       |> Map.put(:from, from)
+      |> Map.put(:set_content_type, false)
 
-    Plug.Static.call(conn, opts)
+    conn
+    |> set_static_content_type(request_path)
+    |> Pleroma.Web.Plugs.StaticNoCT.call(opts)
   end
 end
diff --git a/lib/pleroma/web/plugs/uploaded_media.ex b/lib/pleroma/web/plugs/uploaded_media.ex
index 9f13d919b..746203087 100644
--- a/lib/pleroma/web/plugs/uploaded_media.ex
+++ b/lib/pleroma/web/plugs/uploaded_media.ex
@@ -11,6 +11,7 @@ defmodule Pleroma.Web.Plugs.UploadedMedia do
   require Logger
 
   alias Pleroma.Web.MediaProxy
+  alias Pleroma.Web.Plugs.Utils
 
   @behaviour Plug
   # no slashes
@@ -76,14 +77,9 @@ defmodule Pleroma.Web.Plugs.UploadedMedia do
 
   defp media_is_banned(_, _), do: false
 
-  defp get_safe_mime_type(%{allowed_mime_types: allowed_mime_types} = _opts, mime) do
-    [maintype | _] = String.split(mime, "/", parts: 2)
-    if maintype in allowed_mime_types, do: mime, else: "application/octet-stream"
-  end
-
   defp set_content_type(conn, opts, filepath) do
     real_mime = MIME.from_path(filepath)
-    clean_mime = get_safe_mime_type(opts, real_mime)
+    clean_mime = Utils.get_safe_mime_type(opts, real_mime)
     put_resp_header(conn, "content-type", clean_mime)
   end
 
diff --git a/lib/pleroma/web/plugs/utils.ex b/lib/pleroma/web/plugs/utils.ex
new file mode 100644
index 000000000..770a3eeb2
--- /dev/null
+++ b/lib/pleroma/web/plugs/utils.ex
@@ -0,0 +1,14 @@
+# Akkoma: Magically expressive social media
+# Copyright © 2024 Akkoma Authors <https://akkoma.dev>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.Plugs.Utils do
+  @moduledoc """
+  Some helper functions shared across several plugs
+  """
+
+  def get_safe_mime_type(%{allowed_mime_types: allowed_mime_types} = _opts, mime) do
+    [maintype | _] = String.split(mime, "/", parts: 2)
+    if maintype in allowed_mime_types, do: mime, else: "application/octet-stream"
+  end
+end

From e88d0a28536c10c24abd9f66ad8bc552b82fe99e Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Thu, 7 Mar 2024 00:18:00 +0100
Subject: [PATCH 09/43] Fix Content-Type of our schema

Strict servers fail to process anything from us otherwise.

Fixes: akkoma#716
---
 lib/pleroma/web/endpoint.ex | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/pleroma/web/endpoint.ex b/lib/pleroma/web/endpoint.ex
index 64593767d..6628fcaf3 100644
--- a/lib/pleroma/web/endpoint.ex
+++ b/lib/pleroma/web/endpoint.ex
@@ -98,6 +98,10 @@ defmodule Pleroma.Web.Endpoint do
     at: "/",
     from: :pleroma,
     only: Pleroma.Web.static_paths(),
+    # JSON-LD is accepted by some servers for AP objects and activities,
+    # thus only enable it here instead of a global extension mapping
+    # (it's our only *.jsonld file anyway)
+    content_types: %{"litepub-0.1.jsonld" => "application/ld+json"},
     # credo:disable-for-previous-line Credo.Check.Readability.MaxLineLength
     gzip: true,
     cache_control_for_etags: @static_cache_control,

From bcc528b2e2c5ac9a9854dd408c4d6e643863541e Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Thu, 7 Mar 2024 01:02:32 -0100
Subject: [PATCH 10/43] Never automatically assign privileged content types

By mapping all extensions related to our custom privileged types
back to innocuous text/plain, our custom types will never automatically
be inserted which was one of the factors making impersonation possible.

Note, this does not invalidate the upload and emoji Content-Type
restrictions from previous commits. Apart from counterfeit AP objects
there are other payloads with standard types this protects against,
e.g. *.js Javascript payloads as used in prior frontend injections.
---
 config/config.exs | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/config/config.exs b/config/config.exs
index a366961c0..85a84208c 100644
--- a/config/config.exs
+++ b/config/config.exs
@@ -149,6 +149,19 @@ config :logger, :ex_syslogger,
   format: "$metadata[$level] $message",
   metadata: [:request_id]
 
+# ———————————————————————————————————————————————————————————————
+#                       W  A  R  N  I  N  G
+# ———————————————————————————————————————————————————————————————
+#
+#  Whenever adding a privileged new custom type for e.g.
+#  ActivityPub objects, ALWAYS map their extension back
+#  to "application/octet-stream".
+#  Else files served by us can automatically end up with
+#  those privileged types causing severe security hazards.
+#  (We need those mappings so Phoenix can assoiate its format
+#   (the "extension") to incoming requests of those MIME types)
+#
+# ———————————————————————————————————————————————————————————————
 config :mime, :types, %{
   "application/xml" => ["xml"],
   "application/xrd+xml" => ["xrd+xml"],
@@ -158,9 +171,13 @@ config :mime, :types, %{
 }
 
 config :mime, :extensions, %{
-  "activity+json" => "application/activity+json"
+  "xrd+xml" => "text/plain",
+  "jrd+json" => "text/plain",
+  "activity+json" => "text/plain"
 }
 
+# ———————————————————————————————————————————————————————————————
+
 config :tesla, :adapter, {Tesla.Adapter.Finch, name: MyFinch}
 
 # Configures http settings, upstream proxy etc.

From 11ae8344eb62fea51b9fc26a063406f2afe253ac Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 10 Mar 2024 18:57:19 +0000
Subject: [PATCH 11/43] Sanitise Content-Type of media proxy URLs

Just as with uploads and emoji before, this can otherwise be used
to place counterfeit AP objects or other malicious payloads.
In this case, even if we never assign a priviliged type to content,
the remote server can and until now we just mimcked whatever it told us.

Preview URLs already handle only specific, safe content types
and redirect to the external host for all else; thus no additional
sanitisiation is needed for them.

Non-previews are all delegated to the modified ReverseProxy module.
It already has consolidated logic for building response headers
making it easy to slip in sanitisation.

Although proxy urls are prefixed by a MAC built from a server secret,
attackers can still achieve a perfect id match when they are able to
change the contents of the pointed to URL. After sending an posts
containing an attachment at a controlled destination, the proxy URL can
be read back and inserted into the payload. After injection of
counterfeits in the target server the content can again be changed
to something innocuous lessening chance of detection.
---
 lib/pleroma/reverse_proxy.ex        | 18 +++++++++++++
 test/pleroma/reverse_proxy_test.exs | 41 +++++++++++++++++++++++++++--
 2 files changed, 57 insertions(+), 2 deletions(-)

diff --git a/lib/pleroma/reverse_proxy.ex b/lib/pleroma/reverse_proxy.ex
index bb4f4def3..f017bf51b 100644
--- a/lib/pleroma/reverse_proxy.ex
+++ b/lib/pleroma/reverse_proxy.ex
@@ -17,6 +17,8 @@ defmodule Pleroma.ReverseProxy do
   @failed_request_ttl :timer.seconds(60)
   @methods ~w(GET HEAD)
 
+  @allowed_mime_types Pleroma.Config.get([Pleroma.Upload, :allowed_mime_types], [])
+
   @cachex Pleroma.Config.get([:cachex, :provider], Cachex)
 
   def max_read_duration_default, do: @max_read_duration
@@ -253,6 +255,7 @@ defmodule Pleroma.ReverseProxy do
     headers
     |> Enum.filter(fn {k, _} -> k in @keep_resp_headers end)
     |> build_resp_cache_headers(opts)
+    |> sanitise_content_type()
     |> build_resp_content_disposition_header(opts)
     |> build_csp_headers()
     |> Keyword.merge(Keyword.get(opts, :resp_headers, []))
@@ -282,6 +285,21 @@ defmodule Pleroma.ReverseProxy do
     end
   end
 
+  defp sanitise_content_type(headers) do
+    original_ct = get_content_type(headers)
+
+    safe_ct =
+      Pleroma.Web.Plugs.Utils.get_safe_mime_type(
+        %{allowed_mime_types: @allowed_mime_types},
+        original_ct
+      )
+
+    [
+      {"content-type", safe_ct}
+      | Enum.filter(headers, fn {k, _v} -> k != "content-type" end)
+    ]
+  end
+
   defp build_resp_content_disposition_header(headers, opts) do
     opt = Keyword.get(opts, :inline_content_types, @inline_content_types)
 
diff --git a/test/pleroma/reverse_proxy_test.exs b/test/pleroma/reverse_proxy_test.exs
index e3e2a1571..fc6ae42bc 100644
--- a/test/pleroma/reverse_proxy_test.exs
+++ b/test/pleroma/reverse_proxy_test.exs
@@ -75,13 +75,16 @@ defmodule Pleroma.ReverseProxyTest do
       Tesla.Mock.mock(fn %{method: :head, url: "/head"} ->
         %Tesla.Env{
           status: 200,
-          headers: [{"content-type", "text/html; charset=utf-8"}],
+          headers: [{"content-type", "image/png"}],
           body: ""
         }
       end)
 
       conn = ReverseProxy.call(Map.put(conn, :method, "HEAD"), "/head")
-      assert html_response(conn, 200) == ""
+
+      assert conn.status == 200
+      assert Conn.get_resp_header(conn, "content-type") == ["image/png"]
+      assert conn.resp_body == ""
     end
   end
 
@@ -252,4 +255,38 @@ defmodule Pleroma.ReverseProxyTest do
       assert {"content-disposition", "attachment; filename=\"filename.jpg\""} in conn.resp_headers
     end
   end
+
+  describe "content-type sanitisation" do
+    test "preserves video type", %{conn: conn} do
+      Tesla.Mock.mock(fn %{method: :get, url: "/content"} ->
+        %Tesla.Env{
+          status: 200,
+          headers: [{"content-type", "video/mp4"}],
+          body: "test"
+        }
+      end)
+
+      conn = ReverseProxy.call(Map.put(conn, :method, "GET"), "/content")
+
+      assert conn.status == 200
+      assert Conn.get_resp_header(conn, "content-type") == ["video/mp4"]
+      assert conn.resp_body == "test"
+    end
+
+    test "replaces application type", %{conn: conn} do
+      Tesla.Mock.mock(fn %{method: :get, url: "/content"} ->
+        %Tesla.Env{
+          status: 200,
+          headers: [{"content-type", "application/activity+json"}],
+          body: "test"
+        }
+      end)
+
+      conn = ReverseProxy.call(Map.put(conn, :method, "GET"), "/content")
+
+      assert conn.status == 200
+      assert Conn.get_resp_header(conn, "content-type") == ["application/octet-stream"]
+      assert conn.resp_body == "test"
+    end
+  end
 end

From fc36b04016303cec5746ec3824e5651b6a2655b1 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 10 Mar 2024 18:57:40 +0000
Subject: [PATCH 12/43] Drop media proxy same-domain default for base_url

Even more than with user uploads, a same-domain proxy setup bears
significant security risks due to serving untrusted content under
the main domain space.

A risky setup like that should never be the default.
---
 docs/docs/configuration/howto_mediaproxy.md | 17 +++++++++++++----
 lib/pleroma/web/media_proxy.ex              | 12 ++++++++++--
 2 files changed, 23 insertions(+), 6 deletions(-)

diff --git a/docs/docs/configuration/howto_mediaproxy.md b/docs/docs/configuration/howto_mediaproxy.md
index 8ad81bdfb..223ad7eed 100644
--- a/docs/docs/configuration/howto_mediaproxy.md
+++ b/docs/docs/configuration/howto_mediaproxy.md
@@ -6,7 +6,16 @@ With the `mediaproxy` function you can use nginx to cache this content, so users
 
 ## Activate it
 
-* Edit your nginx config and add the following location: 
+* Edit your nginx config and add the following location to your main server block:
+```
+location /proxy {
+        return 404;
+}
+```
+
+* Set up a subdomain for the proxy with its nginx config on the same machine
+  *(the latter is not strictly required, but for simplicity we’ll assume so)*
+* In this subdomain’s server block add
 ```
 location /proxy {
         proxy_cache akkoma_media_cache;
@@ -26,9 +35,9 @@ config :pleroma, :media_proxy,
       enabled: true,
       proxy_opts: [
             redirect_on_failure: true
-      ]
-      #base_url: "https://cache.akkoma.social"
+      ],
+      base_url: "https://cache.akkoma.social"
 ```
-If you want to use a subdomain to serve the files, uncomment `base_url`, change the url and add a comma after `true` in the previous line.
+You **really** should use a subdomain to serve proxied files; while we will fix bugs resulting from this, serving arbitrary remote content on your main domain namespace is a significant attack surface.
 
 * Restart nginx and Akkoma
diff --git a/lib/pleroma/web/media_proxy.ex b/lib/pleroma/web/media_proxy.ex
index c5087c42c..19411d58e 100644
--- a/lib/pleroma/web/media_proxy.ex
+++ b/lib/pleroma/web/media_proxy.ex
@@ -14,6 +14,8 @@ defmodule Pleroma.Web.MediaProxy do
 
   @cachex Pleroma.Config.get([:cachex, :provider], Cachex)
 
+  @mix_env Mix.env()
+
   def cache_table, do: @cache_table
 
   @spec in_banned_urls(String.t()) :: boolean()
@@ -144,8 +146,14 @@ defmodule Pleroma.Web.MediaProxy do
     if path = URI.parse(url_or_path).path, do: Path.basename(path)
   end
 
-  def base_url do
-    Config.get([:media_proxy, :base_url], Endpoint.url())
+  if @mix_env == :test do
+    def base_url do
+      Config.get([:media_proxy, :base_url], Endpoint.url())
+    end
+  else
+    def base_url do
+      Config.get!([:media_proxy, :base_url])
+    end
   end
 
   defp proxy_url(path, sig_base64, url_base64, filename) do

From fb54c47f0b7380f233f643699c2d14db9bb6c549 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 10 Mar 2024 19:01:17 +0000
Subject: [PATCH 13/43] Update example nginx config

To account for our subdomain recommendations
---
 docs/docs/configuration/cheatsheet.md |  3 +-
 installation/nginx/akkoma.nginx       | 43 +++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/docs/docs/configuration/cheatsheet.md b/docs/docs/configuration/cheatsheet.md
index 9c5bb9901..40d1319c7 100644
--- a/docs/docs/configuration/cheatsheet.md
+++ b/docs/docs/configuration/cheatsheet.md
@@ -396,7 +396,8 @@ This section describe PWA manifest instance-specific values. Currently this opti
 ## :media_proxy
 
 * `enabled`: Enables proxying of remote media to the instance’s proxy
-* `base_url`: The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host/CDN fronts.
+* `base_url`: The base URL to access a user-uploaded file.
+  Using a (sub)domain distinct from the instance endpoint is **strongly** recommended.
 * `proxy_opts`: All options defined in `Pleroma.ReverseProxy` documentation, defaults to `[max_body_length: (25*1_048_576)]`.
 * `whitelist`: List of hosts with scheme to bypass the mediaproxy (e.g. `https://example.com`)
 * `invalidation`: options for remove media from cache after delete object:
diff --git a/installation/nginx/akkoma.nginx b/installation/nginx/akkoma.nginx
index 18d92f30f..1d91ce22f 100644
--- a/installation/nginx/akkoma.nginx
+++ b/installation/nginx/akkoma.nginx
@@ -75,9 +75,48 @@ server {
     proxy_set_header Host $http_host;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
+    location ~ ^/(media|proxy) {
+        return 404;
+    }
+
     location / {
         proxy_pass http://phoenix;
     }
+}
+
+# Upload and MediaProxy Subdomain
+# (see main domain setup for more details)
+server {
+    server_name    media.example.tld;
+
+    listen         80;
+    listen         [::]:80;
+
+    location / {
+      return         301 https://$server_name$request_uri;
+    }
+}
+
+server {
+    server_name media.example.tld;
+
+    listen 443 ssl http2;
+    listen [::]:443 ssl http2;
+
+    ssl_trusted_certificate   /etc/letsencrypt/live/media.example.tld/chain.pem;
+    ssl_certificate           /etc/letsencrypt/live/media.example.tld/fullchain.pem;
+    ssl_certificate_key       /etc/letsencrypt/live/media.example.tld/privkey.pem;
+    # .. copy all other the ssl_* and gzip_* stuff from main domain
+
+    # the nginx default is 1m, not enough for large media uploads
+    client_max_body_size 16m;
+    ignore_invalid_headers off;
+
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+    proxy_set_header Host $http_host;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
     location ~ ^/(media|proxy) {
         proxy_cache        akkoma_media_cache;
@@ -91,4 +130,8 @@ server {
         chunked_transfer_encoding on;
         proxy_pass         http://phoenix;
     }
+
+    location / {
+        return 404;
+    }
 }

From af041db6dc443194b37217bc107f7c8f72a9b8e5 Mon Sep 17 00:00:00 2001
From: Norm <normandy@biribiri.dev>
Date: Tue, 20 Feb 2024 15:11:26 -0500
Subject: [PATCH 14/43] Limit emoji stealer to alphanum, dash, or underscore
 characters
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

As suggested in b387f4a1c1ff02573f16de0b25403cf501afc3b4, only steal
emoji with alphanumerc, dash, or underscore characters.

Also consolidate all validation logic into a single function.

===

Taken from akkoma#703 with cosmetic tweaks

This matches our existing validation logic from Pleroma.Emoji,
and apart from excluding the dot also POSIX’s Portable Filename
Character Set making it always safe for use in filenames.

Mastodon is even stricter also disallowing U+002D HYPEN-MINUS
and requiring at least two characters.

Given both we and Mastodon reject shortcodes excluded
by this anyway, this doesn’t seem like a loss.
---
 .../activity_pub/mrf/steal_emoji_policy.ex    | 26 ++++++++++---------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
index 02a107c27..456fe88c5 100644
--- a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
@@ -20,6 +20,19 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
     String.match?(shortcode, pattern)
   end
 
+  defp reject_emoji?({shortcode, _url}, installed_emoji) do
+    valid_shortcode? = String.match?(shortcode, ~r/^[a-zA-Z0-9_-]+$/)
+
+    rejected_shortcode? =
+      [:mrf_steal_emoji, :rejected_shortcodes]
+      |> Config.get([])
+      |> Enum.any?(fn pattern -> shortcode_matches?(shortcode, pattern) end)
+
+    emoji_installed? = Enum.member?(installed_emoji, shortcode)
+
+    !valid_shortcode? or rejected_shortcode? or emoji_installed?
+  end
+
   defp steal_emoji({shortcode, url}, emoji_dir_path) do
     url = Pleroma.Web.MediaProxy.url(url)
 
@@ -76,18 +89,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
 
       new_emojis =
         foreign_emojis
-        |> Enum.reject(fn {shortcode, _url} -> shortcode in installed_emoji end)
-        |> Enum.reject(fn {shortcode, _url} ->
-          String.contains?(shortcode, ["/", "\\", ".", ":"])
-        end)
-        |> Enum.filter(fn {shortcode, _url} ->
-          reject_emoji? =
-            [:mrf_steal_emoji, :rejected_shortcodes]
-            |> Config.get([])
-            |> Enum.find(false, fn pattern -> shortcode_matches?(shortcode, pattern) end)
-
-          !reject_emoji?
-        end)
+        |> Enum.reject(&reject_emoji?(&1, installed_emoji))
         |> Enum.map(&steal_emoji(&1, emoji_dir_path))
         |> Enum.filter(& &1)
 

From 111cdb0d86f51d0a3f10b77d3f785f68fc9681d4 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Thu, 7 Mar 2024 13:07:02 +0100
Subject: [PATCH 15/43] Split steal_emoji function for better readability

---
 .../activity_pub/mrf/steal_emoji_policy.ex    | 44 ++++++++++---------
 1 file changed, 24 insertions(+), 20 deletions(-)

diff --git a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
index 456fe88c5..c743ac3a2 100644
--- a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
@@ -33,31 +33,35 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
     !valid_shortcode? or rejected_shortcode? or emoji_installed?
   end
 
-  defp steal_emoji({shortcode, url}, emoji_dir_path) do
+  defp steal_emoji(%{} = response, {shortcode, url}, emoji_dir_path) do
+    extension =
+      url
+      |> URI.parse()
+      |> Map.get(:path)
+      |> Path.basename()
+      |> Path.extname()
+
+    shortcode = Path.basename(shortcode)
+    file_path = Path.join(emoji_dir_path, shortcode <> (extension || ".png"))
+
+    case File.write(file_path, response.body) do
+      :ok ->
+        shortcode
+
+      e ->
+        Logger.warning("MRF.StealEmojiPolicy: Failed to write to #{file_path}: #{inspect(e)}")
+        nil
+    end
+  end
+
+  defp maybe_steal_emoji({shortcode, url}, emoji_dir_path) do
     url = Pleroma.Web.MediaProxy.url(url)
 
     with {:ok, %{status: status} = response} when status in 200..299 <- Pleroma.HTTP.get(url) do
       size_limit = Config.get([:mrf_steal_emoji, :size_limit], 50_000)
 
       if byte_size(response.body) <= size_limit do
-        extension =
-          url
-          |> URI.parse()
-          |> Map.get(:path)
-          |> Path.basename()
-          |> Path.extname()
-
-        shortcode = Path.basename(shortcode)
-        file_path = Path.join(emoji_dir_path, shortcode <> (extension || ".png"))
-
-        case File.write(file_path, response.body) do
-          :ok ->
-            shortcode
-
-          e ->
-            Logger.warning("MRF.StealEmojiPolicy: Failed to write to #{file_path}: #{inspect(e)}")
-            nil
-        end
+        steal_emoji(response, {shortcode, url}, emoji_dir_path)
       else
         Logger.debug(
           "MRF.StealEmojiPolicy: :#{shortcode}: at #{url} (#{byte_size(response.body)} B) over size limit (#{size_limit} B)"
@@ -90,7 +94,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
       new_emojis =
         foreign_emojis
         |> Enum.reject(&reject_emoji?(&1, installed_emoji))
-        |> Enum.map(&steal_emoji(&1, emoji_dir_path))
+        |> Enum.map(&maybe_steal_emoji(&1, emoji_dir_path))
         |> Enum.filter(& &1)
 
       if !Enum.empty?(new_emojis) do

From a8c6c780b4e0d0db9a393ff6bb6a3b904ad94269 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Thu, 7 Mar 2024 23:35:05 +0100
Subject: [PATCH 16/43] StealEmoji: use Content-Type and reject non-images
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

E.g. *key’s emoji URLs typically don’t have file extensions, but
until now we just slapped ".png" at its end hoping for the best.

Furthermore, this gives us a chance to actually reject non-images,
which before was not feasible exatly due to those extension-less URLs
---
 .../activity_pub/mrf/steal_emoji_policy.ex    | 26 ++++++----
 .../mrf/steal_emoji_policy_test.exs           | 51 +++++++++++++++++--
 2 files changed, 62 insertions(+), 15 deletions(-)

diff --git a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
index c743ac3a2..9b2fa3020 100644
--- a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
@@ -33,16 +33,9 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
     !valid_shortcode? or rejected_shortcode? or emoji_installed?
   end
 
-  defp steal_emoji(%{} = response, {shortcode, url}, emoji_dir_path) do
-    extension =
-      url
-      |> URI.parse()
-      |> Map.get(:path)
-      |> Path.basename()
-      |> Path.extname()
-
+  defp steal_emoji(%{} = response, {shortcode, extension}, emoji_dir_path) do
     shortcode = Path.basename(shortcode)
-    file_path = Path.join(emoji_dir_path, shortcode <> (extension || ".png"))
+    file_path = Path.join(emoji_dir_path, shortcode <> "." <> extension)
 
     case File.write(file_path, response.body) do
       :ok ->
@@ -54,14 +47,25 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
     end
   end
 
+  defp get_extension_if_safe(response) do
+    content_type =
+      :proplists.get_value("content-type", response.headers, MIME.from_path(response.url))
+
+    case content_type do
+      "image/" <> _ -> List.first(MIME.extensions(content_type))
+      _ -> nil
+    end
+  end
+
   defp maybe_steal_emoji({shortcode, url}, emoji_dir_path) do
     url = Pleroma.Web.MediaProxy.url(url)
 
     with {:ok, %{status: status} = response} when status in 200..299 <- Pleroma.HTTP.get(url) do
       size_limit = Config.get([:mrf_steal_emoji, :size_limit], 50_000)
+      extension = get_extension_if_safe(response)
 
-      if byte_size(response.body) <= size_limit do
-        steal_emoji(response, {shortcode, url}, emoji_dir_path)
+      if byte_size(response.body) <= size_limit and extension do
+        steal_emoji(response, {shortcode, extension}, emoji_dir_path)
       else
         Logger.debug(
           "MRF.StealEmojiPolicy: :#{shortcode}: at #{url} (#{byte_size(response.body)} B) over size limit (#{size_limit} B)"
diff --git a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
index 59baa3a43..b0f6d351c 100644
--- a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
+++ b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
@@ -45,7 +45,11 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     refute File.exists?(path)
 
     Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox.png"} ->
-      %Tesla.Env{status: 200, body: File.read!("test/fixtures/image.jpg")}
+      %Tesla.Env{
+        status: 200,
+        body: File.read!("test/fixtures/image.jpg"),
+        url: "https://example.org/emoji/firedfox.png"
+      }
     end)
 
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
@@ -72,7 +76,11 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     fullpath = Path.join(path, "fired/fox.png")
 
     Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox"} ->
-      %Tesla.Env{status: 200, body: File.read!("test/fixtures/image.jpg")}
+      %Tesla.Env{
+        status: 200,
+        body: File.read!("test/fixtures/image.jpg"),
+        url: "https://example.org/emoji/firedfox.png"
+      }
     end)
 
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
@@ -86,6 +94,36 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     refute File.exists?(fullpath)
   end
 
+  test "prefers content-type header for extension", %{path: path} do
+    message = %{
+      "type" => "Create",
+      "object" => %{
+        "emoji" => [{"firedfox", "https://example.org/emoji/firedfox.fud"}],
+        "actor" => "https://example.org/users/admin"
+      }
+    }
+
+    Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox.fud"} ->
+      %Tesla.Env{
+        status: 200,
+        body: File.read!("test/fixtures/image.jpg"),
+        url: "https://example.org/emoji/firedfox.wevp",
+        headers: [{"content-type", "image/gif"}]
+      }
+    end)
+
+    clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
+
+    assert {:ok, _message} = StealEmojiPolicy.filter(message)
+
+    assert "firedfox" in installed()
+    assert File.exists?(path)
+
+    assert path
+           |> Path.join("firedfox.gif")
+           |> File.exists?()
+  end
+
   test "reject regex shortcode", %{message: message} do
     refute "firedfox" in installed()
 
@@ -118,7 +156,11 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     refute "firedfox" in installed()
 
     Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox.png"} ->
-      %Tesla.Env{status: 200, body: File.read!("test/fixtures/image.jpg")}
+      %Tesla.Env{
+        status: 200,
+        body: File.read!("test/fixtures/image.jpg"),
+        url: "https://example.org/emoji/firedfox.png"
+      }
     end)
 
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 50_000)
@@ -132,7 +174,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     refute "firedfox" in installed()
 
     Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox.png"} ->
-      {:ok, %Tesla.Env{status: 404, body: "Not found"}}
+      {:ok,
+       %Tesla.Env{status: 404, body: "Not found", url: "https://example.org/emoji/firedfox.png"}}
     end)
 
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)

From 5b126567bb6cd25b8299ee953042710ef645555f Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Thu, 7 Mar 2024 23:39:00 +0100
Subject: [PATCH 17/43] StealEmoji: drop superfluous basename

Since 3 commits ago we restrict shortcodes to a subset of
the POSIX Portable Filename Character Set, therefore
this can never have a directory component.
---
 lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex | 1 -
 1 file changed, 1 deletion(-)

diff --git a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
index 9b2fa3020..da6b8275d 100644
--- a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
@@ -34,7 +34,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
   end
 
   defp steal_emoji(%{} = response, {shortcode, extension}, emoji_dir_path) do
-    shortcode = Path.basename(shortcode)
     file_path = Path.join(emoji_dir_path, shortcode <> "." <> extension)
 
     case File.write(file_path, response.body) do

From fa98b44acf9f04a7c50f591aeaa4a666b3967984 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sat, 9 Mar 2024 22:18:00 +0100
Subject: [PATCH 18/43] Fill out path for newly created packs

Before this was only filled on loading the pack again,
preventing the created pack from being used directly.
---
 lib/pleroma/emoji/pack.ex | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/pleroma/emoji/pack.ex b/lib/pleroma/emoji/pack.ex
index 9049d9097..30ed5a8bd 100644
--- a/lib/pleroma/emoji/pack.ex
+++ b/lib/pleroma/emoji/pack.ex
@@ -31,7 +31,10 @@ defmodule Pleroma.Emoji.Pack do
     with :ok <- validate_not_empty([name]),
          dir <- Path.join(emoji_path(), name),
          :ok <- File.mkdir(dir) do
-      save_pack(%__MODULE__{pack_file: Path.join(dir, "pack.json")})
+      save_pack(%__MODULE__{
+        path: dir,
+        pack_file: Path.join(dir, "pack.json")
+      })
     end
   end
 

From d1c4d0740467455b4fd158c9c3d1b79e500a79f9 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Fri, 8 Mar 2024 03:06:40 +0100
Subject: [PATCH 19/43] Convert StealEmoji to pack.json

This will decouple filenames from shortcodes and
allow more image formats to work instead of only
those included in the auto-load glob. (Albeit we
still saved other formats to disk, wasting space)

Furthermore, this will allow us to make
final URL paths infeasible to predict.
---
 lib/pleroma/emoji/pack.ex                     | 16 ++++-
 .../activity_pub/mrf/steal_emoji_policy.ex    | 58 +++++++++++++------
 2 files changed, 56 insertions(+), 18 deletions(-)

diff --git a/lib/pleroma/emoji/pack.ex b/lib/pleroma/emoji/pack.ex
index 30ed5a8bd..f007cde65 100644
--- a/lib/pleroma/emoji/pack.ex
+++ b/lib/pleroma/emoji/pack.ex
@@ -92,7 +92,7 @@ defmodule Pleroma.Emoji.Pack do
     end)
   end
 
-  @spec add_file(t(), String.t(), Path.t(), Plug.Upload.t()) ::
+  @spec add_file(t(), String.t(), Path.t(), Plug.Upload.t() | binary()) ::
           {:ok, t()}
           | {:error, File.posix() | atom()}
   def add_file(%Pack{} = pack, _, _, %Plug.Upload{content_type: "application/zip"} = file) do
@@ -140,6 +140,14 @@ defmodule Pleroma.Emoji.Pack do
   end
 
   def add_file(%Pack{} = pack, shortcode, filename, %Plug.Upload{} = file) do
+    try_add_file(pack, shortcode, filename, file)
+  end
+
+  def add_file(%Pack{} = pack, shortcode, filename, filedata) when is_binary(filedata) do
+    try_add_file(pack, shortcode, filename, filedata)
+  end
+
+  defp try_add_file(%Pack{} = pack, shortcode, filename, file) do
     with :ok <- validate_not_empty([shortcode, filename]),
          :ok <- validate_emoji_not_exists(shortcode),
          {:ok, updated_pack} <- do_add_file(pack, shortcode, filename, file) do
@@ -485,6 +493,12 @@ defmodule Pleroma.Emoji.Pack do
     end
   end
 
+  defp save_file(file_data, pack, filename) when is_binary(file_data) do
+    file_path = Path.join(pack.path, filename)
+    create_subdirs(file_path)
+    File.write(file_path, file_data, [:binary])
+  end
+
   defp put_emoji(pack, shortcode, filename) do
     files = Map.put(pack.files, shortcode, filename)
     %{pack | files: files, files_count: length(Map.keys(files))}
diff --git a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
index da6b8275d..ed421d93e 100644
--- a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
@@ -6,10 +6,41 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
   require Logger
 
   alias Pleroma.Config
+  alias Pleroma.Emoji.Pack
 
   @moduledoc "Detect new emojis by their shortcode and steals them"
   @behaviour Pleroma.Web.ActivityPub.MRF.Policy
 
+  @pack_name "stolen"
+
+  defp create_pack() do
+    with {:ok, pack} = Pack.create(@pack_name) do
+      Pack.save_metadata(
+        %{
+          "description" => "Collection of emoji auto-stolen from other instances",
+          "homepage" => Pleroma.Web.Endpoint.url(),
+          "can-download" => false,
+          "share-files" => false
+        },
+        pack
+      )
+    end
+  end
+
+  defp load_or_create_pack() do
+    case Pack.load_pack(@pack_name) do
+      {:ok, pack} -> {:ok, pack}
+      {:error, :enoent} -> create_pack()
+      e -> e
+    end
+  end
+
+  defp add_emoji(shortcode, extension, filedata) do
+    {:ok, pack} = load_or_create_pack()
+    filename = shortcode <> "." <> extension
+    Pack.add_file(pack, shortcode, filename, filedata)
+  end
+
   defp accept_host?(host), do: host in Config.get([:mrf_steal_emoji, :hosts], [])
 
   defp shortcode_matches?(shortcode, pattern) when is_binary(pattern) do
@@ -33,15 +64,16 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
     !valid_shortcode? or rejected_shortcode? or emoji_installed?
   end
 
-  defp steal_emoji(%{} = response, {shortcode, extension}, emoji_dir_path) do
-    file_path = Path.join(emoji_dir_path, shortcode <> "." <> extension)
-
-    case File.write(file_path, response.body) do
-      :ok ->
+  defp steal_emoji(%{} = response, {shortcode, extension}) do
+    case add_emoji(shortcode, extension, response.body) do
+      {:ok, _} ->
         shortcode
 
       e ->
-        Logger.warning("MRF.StealEmojiPolicy: Failed to write to #{file_path}: #{inspect(e)}")
+        Logger.warning(
+          "MRF.StealEmojiPolicy: Failed to add #{shortcode}.#{extension}: #{inspect(e)}"
+        )
+
         nil
     end
   end
@@ -56,7 +88,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
     end
   end
 
-  defp maybe_steal_emoji({shortcode, url}, emoji_dir_path) do
+  defp maybe_steal_emoji({shortcode, url}) do
     url = Pleroma.Web.MediaProxy.url(url)
 
     with {:ok, %{status: status} = response} when status in 200..299 <- Pleroma.HTTP.get(url) do
@@ -64,7 +96,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
       extension = get_extension_if_safe(response)
 
       if byte_size(response.body) <= size_limit and extension do
-        steal_emoji(response, {shortcode, extension}, emoji_dir_path)
+        steal_emoji(response, {shortcode, extension})
       else
         Logger.debug(
           "MRF.StealEmojiPolicy: :#{shortcode}: at #{url} (#{byte_size(response.body)} B) over size limit (#{size_limit} B)"
@@ -86,18 +118,10 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
     if host != Pleroma.Web.Endpoint.host() and accept_host?(host) do
       installed_emoji = Pleroma.Emoji.get_all() |> Enum.map(fn {k, _} -> k end)
 
-      emoji_dir_path =
-        Config.get(
-          [:mrf_steal_emoji, :path],
-          Path.join(Config.get([:instance, :static_dir]), "emoji/stolen")
-        )
-
-      File.mkdir_p(emoji_dir_path)
-
       new_emojis =
         foreign_emojis
         |> Enum.reject(&reject_emoji?(&1, installed_emoji))
-        |> Enum.map(&maybe_steal_emoji(&1, emoji_dir_path))
+        |> Enum.map(&maybe_steal_emoji(&1))
         |> Enum.filter(& &1)
 
       if !Enum.empty?(new_emojis) do

From ee5ce87825d598409e4d409a8256677f2f52fbc0 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sat, 9 Mar 2024 21:39:25 +0000
Subject: [PATCH 20/43] test: use pack functions to check for emoji

The hardocded path and filenames assumptions
will be broken with the next commit.
---
 .../mrf/steal_emoji_policy_test.exs           | 44 +++++++++++--------
 1 file changed, 25 insertions(+), 19 deletions(-)

diff --git a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
index b0f6d351c..43de902c7 100644
--- a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
+++ b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
@@ -7,8 +7,23 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
 
   alias Pleroma.Config
   alias Pleroma.Emoji
+  alias Pleroma.Emoji.Pack
   alias Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy
 
+  defp has_pack?() do
+    case Pack.load_pack("stolen") do
+      {:ok, _pack} -> true
+      {:error, :enoent} -> false
+    end
+  end
+
+  defp has_emoji?(shortcode) do
+    case Pack.load_pack("stolen") do
+      {:ok, pack} -> Map.has_key?(pack.files, shortcode)
+      {:error, :enoent} -> false
+    end
+  end
+
   setup do
     emoji_path = [:instance, :static_dir] |> Config.get() |> Path.join("emoji/stolen")
 
@@ -26,7 +41,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
       File.rm_rf!(emoji_path)
     end)
 
-    [message: message, path: emoji_path]
+    [message: message]
   end
 
   test "does nothing by default", %{message: message} do
@@ -38,11 +53,10 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
   end
 
   test "Steals emoji on unknown shortcode from allowed remote host", %{
-    message: message,
-    path: path
+    message: message
   } do
     refute "firedfox" in installed()
-    refute File.exists?(path)
+    refute has_pack?()
 
     Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox.png"} ->
       %Tesla.Env{
@@ -57,14 +71,12 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     assert {:ok, _message} = StealEmojiPolicy.filter(message)
 
     assert "firedfox" in installed()
-    assert File.exists?(path)
+    assert has_pack?()
 
-    assert path
-           |> Path.join("firedfox.png")
-           |> File.exists?()
+    assert has_emoji?("firedfox")
   end
 
-  test "rejects invalid shortcodes", %{path: path} do
+  test "rejects invalid shortcodes" do
     message = %{
       "type" => "Create",
       "object" => %{
@@ -73,8 +85,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
       }
     }
 
-    fullpath = Path.join(path, "fired/fox.png")
-
     Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox"} ->
       %Tesla.Env{
         status: 200,
@@ -86,15 +96,15 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
 
     refute "firedfox" in installed()
-    refute File.exists?(path)
+    refute has_pack?()
 
     assert {:ok, _message} = StealEmojiPolicy.filter(message)
 
     refute "fired/fox" in installed()
-    refute File.exists?(fullpath)
+    refute has_emoji?("fired/fox")
   end
 
-  test "prefers content-type header for extension", %{path: path} do
+  test "prefers content-type header for extension" do
     message = %{
       "type" => "Create",
       "object" => %{
@@ -117,11 +127,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     assert {:ok, _message} = StealEmojiPolicy.filter(message)
 
     assert "firedfox" in installed()
-    assert File.exists?(path)
-
-    assert path
-           |> Path.join("firedfox.gif")
-           |> File.exists?()
+    assert has_emoji?("firedfox")
   end
 
   test "reject regex shortcode", %{message: message} do

From a4fa2ec9af769ac160f3e1a60c3273a6f991b9fc Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sat, 9 Mar 2024 22:41:26 +0100
Subject: [PATCH 21/43] StealEmoji: make final paths infeasible to predict
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Certain attacks rely on predictable paths for their payloads.
If we weren’t so overly lax in our (id, URL) check, the current
counterfeit activity exploit would be one of those.
It seems plausible for future attacks to hinge on
or being made easier by predictable paths too.

In general, letting remote actors place arbitrary data at
a path within our domain of their choosing (sans prefix)
just doesn’t seem like a good idea.

Using fully random filenames would have worked as well, but this
is less friendly for admins checking emoji dirs.
The generated suffix should still be more than enough;
an attacker needs on average 140 trillion attempts to
correctly guess the final path.
---
 .../web/activity_pub/mrf/steal_emoji_policy.ex      | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
index ed421d93e..3a6eae3f2 100644
--- a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
@@ -37,7 +37,16 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
 
   defp add_emoji(shortcode, extension, filedata) do
     {:ok, pack} = load_or_create_pack()
-    filename = shortcode <> "." <> extension
+    # Make final path infeasible to predict to thwart certain kinds of attacks
+    # (48 bits is slighty more than 8 base62 chars, thus 9 chars)
+    salt =
+      :crypto.strong_rand_bytes(6)
+      |> :crypto.bytes_to_integer()
+      |> Base62.encode()
+      |> String.pad_leading(9, "0")
+
+    filename = shortcode <> "-" <> salt <> "." <> extension
+
     Pack.add_file(pack, shortcode, filename, filedata)
   end
 
@@ -71,7 +80,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
 
       e ->
         Logger.warning(
-          "MRF.StealEmojiPolicy: Failed to add #{shortcode}.#{extension}: #{inspect(e)}"
+          "MRF.StealEmojiPolicy: Failed to add #{shortcode} as #{extension}: #{inspect(e)}"
         )
 
         nil

From d1ce5fd911542ccd4894150d75fe288146b4e16f Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 10 Mar 2024 00:44:12 +0100
Subject: [PATCH 22/43] test/steal_emoji: reduce code duplication with mock
 macro

---
 .../mrf/steal_emoji_policy_test.exs           | 57 ++++++++-----------
 1 file changed, 24 insertions(+), 33 deletions(-)

diff --git a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
index 43de902c7..b34269aa5 100644
--- a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
+++ b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
@@ -24,6 +24,25 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     end
   end
 
+  defmacro mock_tesla(
+             url \\ "https://example.org/emoji/firedfox.png",
+             status \\ 200,
+             headers \\ [],
+             get_body \\ File.read!("test/fixtures/image.jpg")
+           ) do
+    quote do
+      Tesla.Mock.mock(fn
+        %{method: :get, url: unquote(url)} ->
+          %Tesla.Env{
+            status: unquote(status),
+            body: unquote(get_body),
+            url: unquote(url),
+            headers: unquote(headers)
+          }
+      end)
+    end
+  end
+
   setup do
     emoji_path = [:instance, :static_dir] |> Config.get() |> Path.join("emoji/stolen")
 
@@ -58,13 +77,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     refute "firedfox" in installed()
     refute has_pack?()
 
-    Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox.png"} ->
-      %Tesla.Env{
-        status: 200,
-        body: File.read!("test/fixtures/image.jpg"),
-        url: "https://example.org/emoji/firedfox.png"
-      }
-    end)
+    mock_tesla()
 
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
 
@@ -85,13 +98,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
       }
     }
 
-    Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox"} ->
-      %Tesla.Env{
-        status: 200,
-        body: File.read!("test/fixtures/image.jpg"),
-        url: "https://example.org/emoji/firedfox.png"
-      }
-    end)
+    mock_tesla()
 
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
 
@@ -113,14 +120,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
       }
     }
 
-    Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox.fud"} ->
-      %Tesla.Env{
-        status: 200,
-        body: File.read!("test/fixtures/image.jpg"),
-        url: "https://example.org/emoji/firedfox.wevp",
-        headers: [{"content-type", "image/gif"}]
-      }
-    end)
+    mock_tesla("https://example.org/emoji/firedfox.fud", 200, [{"content-type", "image/gif"}])
 
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
 
@@ -161,13 +161,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
   test "reject if size is above the limit", %{message: message} do
     refute "firedfox" in installed()
 
-    Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox.png"} ->
-      %Tesla.Env{
-        status: 200,
-        body: File.read!("test/fixtures/image.jpg"),
-        url: "https://example.org/emoji/firedfox.png"
-      }
-    end)
+    mock_tesla()
 
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 50_000)
 
@@ -179,10 +173,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
   test "reject if host returns error", %{message: message} do
     refute "firedfox" in installed()
 
-    Tesla.Mock.mock(fn %{method: :get, url: "https://example.org/emoji/firedfox.png"} ->
-      {:ok,
-       %Tesla.Env{status: 404, body: "Not found", url: "https://example.org/emoji/firedfox.png"}}
-    end)
+    mock_tesla("https://example.org/emoji/firedfox.png", 404, [], "Not found")
 
     clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
 

From 6d003e1acdf4d92cbf02ccba7b627d1fb8f3db6f Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 10 Mar 2024 01:14:51 +0100
Subject: [PATCH 23/43] test/steal_emoji: consolidate configuration setup

---
 .../mrf/steal_emoji_policy_test.exs           | 28 ++++++-------------
 1 file changed, 9 insertions(+), 19 deletions(-)

diff --git a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
index b34269aa5..2103e8539 100644
--- a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
+++ b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
@@ -44,6 +44,11 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
   end
 
   setup do
+    clear_config(:mrf_steal_emoji,
+      hosts: ["example.org"],
+      size_limit: 284_468
+    )
+
     emoji_path = [:instance, :static_dir] |> Config.get() |> Path.join("emoji/stolen")
 
     Emoji.reload()
@@ -66,6 +71,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
   test "does nothing by default", %{message: message} do
     refute "firedfox" in installed()
 
+    clear_config(:mrf_steal_emoji, [])
     assert {:ok, _message} = StealEmojiPolicy.filter(message)
 
     refute "firedfox" in installed()
@@ -79,8 +85,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
 
     mock_tesla()
 
-    clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
-
     assert {:ok, _message} = StealEmojiPolicy.filter(message)
 
     assert "firedfox" in installed()
@@ -100,8 +104,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
 
     mock_tesla()
 
-    clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
-
     refute "firedfox" in installed()
     refute has_pack?()
 
@@ -122,8 +124,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
 
     mock_tesla("https://example.org/emoji/firedfox.fud", 200, [{"content-type", "image/gif"}])
 
-    clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
-
     assert {:ok, _message} = StealEmojiPolicy.filter(message)
 
     assert "firedfox" in installed()
@@ -133,11 +133,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
   test "reject regex shortcode", %{message: message} do
     refute "firedfox" in installed()
 
-    clear_config(:mrf_steal_emoji,
-      hosts: ["example.org"],
-      size_limit: 284_468,
-      rejected_shortcodes: [~r/firedfox/]
-    )
+    clear_config([:mrf_steal_emoji, :rejected_shortcodes], [~r/firedfox/])
 
     assert {:ok, _message} = StealEmojiPolicy.filter(message)
 
@@ -147,11 +143,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
   test "reject string shortcode", %{message: message} do
     refute "firedfox" in installed()
 
-    clear_config(:mrf_steal_emoji,
-      hosts: ["example.org"],
-      size_limit: 284_468,
-      rejected_shortcodes: ["firedfox"]
-    )
+    clear_config([:mrf_steal_emoji, :rejected_shortcodes], ["firedfox"])
 
     assert {:ok, _message} = StealEmojiPolicy.filter(message)
 
@@ -163,7 +155,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
 
     mock_tesla()
 
-    clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 50_000)
+    clear_config([:mrf_steal_emoji, :size_limit], 50_000)
 
     assert {:ok, _message} = StealEmojiPolicy.filter(message)
 
@@ -175,8 +167,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
 
     mock_tesla("https://example.org/emoji/firedfox.png", 404, [], "Not found")
 
-    clear_config(:mrf_steal_emoji, hosts: ["example.org"], size_limit: 284_468)
-
     ExUnit.CaptureLog.capture_log(fn ->
       assert {:ok, _message} = StealEmojiPolicy.filter(message)
     end) =~ "MRF.StealEmojiPolicy: Failed to fetch https://example.org/emoji/firedfox.png"

From d6d838cbe83e8caf3e1fc67a81c3943e880ab290 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 10 Mar 2024 01:35:35 +0100
Subject: [PATCH 24/43] StealEmoji: check remote size before downloading

To save on bandwith and avoid OOMs with large files.
Ofc, this relies on the remote server
 (a) sending a content-length header and
 (b) being honest about the size.

Common fedi servers seem to provide the header and (b) at least raises
the required privilege of an malicious actor to a server infrastructure
admin of an explicitly allowed host.

A more complete defense which still works when faced with
a malicious server requires changes in upstream Finch;
see https://github.com/sneako/finch/issues/224
---
 docs/docs/configuration/cheatsheet.md         |  4 +-
 .../activity_pub/mrf/steal_emoji_policy.ex    | 25 +++++++++-
 .../mrf/steal_emoji_policy_test.exs           | 50 ++++++++++++++++++-
 3 files changed, 75 insertions(+), 4 deletions(-)

diff --git a/docs/docs/configuration/cheatsheet.md b/docs/docs/configuration/cheatsheet.md
index 40d1319c7..c4259c6cf 100644
--- a/docs/docs/configuration/cheatsheet.md
+++ b/docs/docs/configuration/cheatsheet.md
@@ -236,7 +236,9 @@ config :pleroma, :mrf_user_allowlist, %{
 #### :mrf_steal_emoji
 * `hosts`: List of hosts to steal emojis from
 * `rejected_shortcodes`: Regex-list of shortcodes to reject
-* `size_limit`: File size limit (in bytes), checked before an emoji is saved to the disk
+* `size_limit`: File size limit (in bytes), checked before download if possible (and remote server honest),
+   otherwise or again checked before saving emoji to the disk
+* `download_unknown_size`: whether to download an emoji when the remote server doesn’t report its size in advance
 
 #### :mrf_activity_expiration
 
diff --git a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
index 3a6eae3f2..26d3dc592 100644
--- a/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
+++ b/lib/pleroma/web/activity_pub/mrf/steal_emoji_policy.ex
@@ -13,6 +13,10 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
 
   @pack_name "stolen"
 
+  # Config defaults
+  @size_limit 50_000
+  @download_unknown_size false
+
   defp create_pack() do
     with {:ok, pack} = Pack.create(@pack_name) do
       Pack.save_metadata(
@@ -97,11 +101,28 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicy do
     end
   end
 
+  defp is_remote_size_within_limit?(url) do
+    with {:ok, %{status: status, headers: headers} = _response} when status in 200..299 <-
+           Pleroma.HTTP.request(:head, url, nil, [], []) do
+      content_length = :proplists.get_value("content-length", headers, nil)
+      size_limit = Config.get([:mrf_steal_emoji, :size_limit], @size_limit)
+
+      accept_unknown =
+        Config.get([:mrf_steal_emoji, :download_unknown_size], @download_unknown_size)
+
+      content_length <= size_limit or
+        (content_length == nil and accept_unknown)
+    else
+      _ -> false
+    end
+  end
+
   defp maybe_steal_emoji({shortcode, url}) do
     url = Pleroma.Web.MediaProxy.url(url)
 
-    with {:ok, %{status: status} = response} when status in 200..299 <- Pleroma.HTTP.get(url) do
-      size_limit = Config.get([:mrf_steal_emoji, :size_limit], 50_000)
+    with {:remote_size, true} <- {:remote_size, is_remote_size_within_limit?(url)},
+         {:ok, %{status: status} = response} when status in 200..299 <- Pleroma.HTTP.get(url) do
+      size_limit = Config.get([:mrf_steal_emoji, :size_limit], @size_limit)
       extension = get_extension_if_safe(response)
 
       if byte_size(response.body) <= size_limit and extension do
diff --git a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
index 2103e8539..ba5087f1b 100644
--- a/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
+++ b/test/pleroma/web/activity_pub/mrf/steal_emoji_policy_test.exs
@@ -32,6 +32,14 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
            ) do
     quote do
       Tesla.Mock.mock(fn
+        %{method: :head, url: unquote(url)} ->
+          %Tesla.Env{
+            status: unquote(status),
+            body: nil,
+            url: unquote(url),
+            headers: unquote(headers)
+          }
+
         %{method: :get, url: unquote(url)} ->
           %Tesla.Env{
             status: unquote(status),
@@ -46,7 +54,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
   setup do
     clear_config(:mrf_steal_emoji,
       hosts: ["example.org"],
-      size_limit: 284_468
+      size_limit: 284_468,
+      download_unknown_size: true
     )
 
     emoji_path = [:instance, :static_dir] |> Config.get() |> Path.join("emoji/stolen")
@@ -174,5 +183,44 @@ defmodule Pleroma.Web.ActivityPub.MRF.StealEmojiPolicyTest do
     refute "firedfox" in installed()
   end
 
+  test "reject unknown size", %{message: message} do
+    clear_config([:mrf_steal_emoji, :download_unknown_size], false)
+    mock_tesla()
+
+    refute "firedfox" in installed()
+
+    ExUnit.CaptureLog.capture_log(fn ->
+      assert {:ok, _message} = StealEmojiPolicy.filter(message)
+    end) =~
+      "MRF.StealEmojiPolicy: Failed to fetch https://example.org/emoji/firedfox.png: {:remote_size, false}"
+
+    refute "firedfox" in installed()
+  end
+
+  test "reject too large content-size before download", %{message: message} do
+    clear_config([:mrf_steal_emoji, :download_unknown_size], false)
+    mock_tesla("https://example.org/emoji/firedfox.png", 200, [{"content-length", 2 ** 30}])
+
+    refute "firedfox" in installed()
+
+    ExUnit.CaptureLog.capture_log(fn ->
+      assert {:ok, _message} = StealEmojiPolicy.filter(message)
+    end) =~
+      "MRF.StealEmojiPolicy: Failed to fetch https://example.org/emoji/firedfox.png: {:remote_size, false}"
+
+    refute "firedfox" in installed()
+  end
+
+  test "accepts content-size below limit", %{message: message} do
+    clear_config([:mrf_steal_emoji, :download_unknown_size], false)
+    mock_tesla("https://example.org/emoji/firedfox.png", 200, [{"content-length", 2}])
+
+    refute "firedfox" in installed()
+
+    assert {:ok, _message} = StealEmojiPolicy.filter(message)
+
+    assert "firedfox" in installed()
+  end
+
   defp installed, do: Emoji.get_all() |> Enum.map(fn {k, _} -> k end)
 end

From ddd79ff22d7e8ba1931b6aa40996637e3b73a5ac Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 10 Mar 2024 07:15:26 +0100
Subject: [PATCH 25/43] Proactively harden emoji pack against path traversal

No new path traversal attacks are known. But given the many entrypoints
and code flow complexity inside pack.ex, it unfortunately seems
possible a future refactor or addition might reintroduce one.
Furthermore, some old packs might still contain traversing path entries
which could trigger undesireable actions on rename or delete.

To ensure this can never happen, assert safety during path construction.

Path.safe_relative was introduced in Elixir 1.14, but
fortunately, we already require at least 1.14 anyway.
---
 lib/pleroma/emoji/pack.ex        | 55 ++++++++++++++++++++++++--------
 test/pleroma/emoji/pack_test.exs |  6 ++--
 2 files changed, 46 insertions(+), 15 deletions(-)

diff --git a/lib/pleroma/emoji/pack.ex b/lib/pleroma/emoji/pack.ex
index f007cde65..142208854 100644
--- a/lib/pleroma/emoji/pack.ex
+++ b/lib/pleroma/emoji/pack.ex
@@ -26,10 +26,32 @@ defmodule Pleroma.Emoji.Pack do
   alias Pleroma.Emoji.Pack
   alias Pleroma.Utils
 
+  # Invalid/Malicious names are supposed to be filtered out before path joining,
+  # but there are many entrypoints to affected functions so as the code changes
+  # we might accidentally let an unsanitised name slip through.
+  # To make sure, use the below which crash the process otherwise.
+
+  # ALWAYS use this when constructing paths from external name!
+  # (name meaning it must be only a single path component)
+  defp path_join_name_safe(dir, name) do
+    if to_string(name) != Path.basename(name) or name in ["..", ".", ""] do
+      raise "Invalid or malicious pack name: #{name}"
+    else
+      Path.join(dir, name)
+    end
+  end
+
+  # ALWAYS use this to join external paths
+  # (which are allowed to have several components)
+  defp path_join_safe(dir, path) do
+    {:ok, safe_path} = Path.safe_relative(path)
+    Path.join(dir, safe_path)
+  end
+
   @spec create(String.t()) :: {:ok, t()} | {:error, File.posix()} | {:error, :empty_values}
   def create(name) do
     with :ok <- validate_not_empty([name]),
-         dir <- Path.join(emoji_path(), name),
+         dir <- path_join_name_safe(emoji_path(), name),
          :ok <- File.mkdir(dir) do
       save_pack(%__MODULE__{
         path: dir,
@@ -68,7 +90,7 @@ defmodule Pleroma.Emoji.Pack do
           {:ok, [binary()]} | {:error, File.posix(), binary()} | {:error, :empty_values}
   def delete(name) do
     with :ok <- validate_not_empty([name]),
-         pack_path <- Path.join(emoji_path(), name) do
+         pack_path <- path_join_name_safe(emoji_path(), name) do
       File.rm_rf(pack_path)
     end
   end
@@ -110,7 +132,7 @@ defmodule Pleroma.Emoji.Pack do
           Enum.map_reduce(emojies, pack, fn item, emoji_pack ->
             emoji_file = %Plug.Upload{
               filename: item[:filename],
-              path: Path.join(tmp_dir, item[:path])
+              path: path_join_safe(tmp_dir, item[:path])
             }
 
             {:ok, updated_pack} =
@@ -200,6 +222,7 @@ defmodule Pleroma.Emoji.Pack do
          {:ok, results} <- File.ls(emoji_path) do
       names =
         results
+        # items come from File.ls, thus safe
         |> Enum.map(&Path.join(emoji_path, &1))
         |> Enum.reject(fn path ->
           File.dir?(path) and File.exists?(Path.join(path, "pack.json"))
@@ -298,8 +321,8 @@ defmodule Pleroma.Emoji.Pack do
 
   @spec load_pack(String.t()) :: {:ok, t()} | {:error, :file.posix()}
   def load_pack(name) do
-    name = Path.basename(name)
-    pack_file = Path.join([emoji_path(), name, "pack.json"])
+    pack_dir = path_join_name_safe(emoji_path(), name)
+    pack_file = Path.join(pack_dir, "pack.json")
 
     with {:ok, _} <- File.stat(pack_file),
          {:ok, pack_data} <- File.read(pack_file) do
@@ -423,7 +446,13 @@ defmodule Pleroma.Emoji.Pack do
   end
 
   defp create_archive_and_cache(pack, hash) do
-    files = [~c"pack.json" | Enum.map(pack.files, fn {_, file} -> to_charlist(file) end)]
+    files = [
+      ~c"pack.json"
+      | Enum.map(pack.files, fn {_, file} ->
+          {:ok, file} = Path.safe_relative(file)
+          to_charlist(file)
+        end)
+    ]
 
     {:ok, {_, result}} =
       :zip.zip(~c"#{pack.name}.zip", files, [:memory, cwd: to_charlist(pack.path)])
@@ -485,7 +514,7 @@ defmodule Pleroma.Emoji.Pack do
   end
 
   defp save_file(%Plug.Upload{path: upload_path}, pack, filename) do
-    file_path = Path.join(pack.path, filename)
+    file_path = path_join_safe(pack.path, filename)
     create_subdirs(file_path)
 
     with {:ok, _} <- File.copy(upload_path, file_path) do
@@ -494,7 +523,7 @@ defmodule Pleroma.Emoji.Pack do
   end
 
   defp save_file(file_data, pack, filename) when is_binary(file_data) do
-    file_path = Path.join(pack.path, filename)
+    file_path = path_join_safe(pack.path, filename)
     create_subdirs(file_path)
     File.write(file_path, file_data, [:binary])
   end
@@ -510,8 +539,8 @@ defmodule Pleroma.Emoji.Pack do
   end
 
   defp rename_file(pack, filename, new_filename) do
-    old_path = Path.join(pack.path, filename)
-    new_path = Path.join(pack.path, new_filename)
+    old_path = path_join_safe(pack.path, filename)
+    new_path = path_join_safe(pack.path, new_filename)
     create_subdirs(new_path)
 
     with :ok <- File.rename(old_path, new_path) do
@@ -529,7 +558,7 @@ defmodule Pleroma.Emoji.Pack do
 
   defp remove_file(pack, shortcode) do
     with {:ok, filename} <- get_filename(pack, shortcode),
-         emoji <- Path.join(pack.path, filename),
+         emoji <- path_join_safe(pack.path, filename),
          :ok <- File.rm(emoji) do
       remove_dir_if_empty(emoji, filename)
     end
@@ -547,7 +576,7 @@ defmodule Pleroma.Emoji.Pack do
 
   defp get_filename(pack, shortcode) do
     with %{^shortcode => filename} when is_binary(filename) <- pack.files,
-         file_path <- Path.join(pack.path, filename),
+         file_path <- path_join_safe(pack.path, filename),
          {:ok, _} <- File.stat(file_path) do
       {:ok, filename}
     else
@@ -585,7 +614,7 @@ defmodule Pleroma.Emoji.Pack do
   end
 
   defp copy_as(remote_pack, local_name) do
-    path = Path.join(emoji_path(), local_name)
+    path = path_join_name_safe(emoji_path(), local_name)
 
     %__MODULE__{
       name: local_name,
diff --git a/test/pleroma/emoji/pack_test.exs b/test/pleroma/emoji/pack_test.exs
index 4d769789d..f5d2e2eef 100644
--- a/test/pleroma/emoji/pack_test.exs
+++ b/test/pleroma/emoji/pack_test.exs
@@ -93,7 +93,9 @@ defmodule Pleroma.Emoji.PackTest do
     assert updated_pack.files_count == 1
   end
 
-  test "load_pack/1 ignores path traversal in a forged pack name", %{pack: pack} do
-    assert {:ok, ^pack} = Pack.load_pack("../../../../../dump_pack")
+  test "load_pack/1 panics on path traversal in a forged pack name" do
+    assert_raise(RuntimeError, "Invalid or malicious pack name: ../../../../../dump_pack", fn ->
+      Pack.load_pack("../../../../../dump_pack")
+    end)
   end
 end

From c806adbfdbea3c30d6d9ecb92cceaefa240438c4 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Mon, 11 Mar 2024 22:52:46 +0100
Subject: [PATCH 26/43] Refactor Fetcher.get_object for readability

Apart from slightly different error reasons wrt content-type,
this does not change functionality in any way.
---
 lib/pleroma/object/fetcher.ex | 46 ++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 22 deletions(-)

diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex
index 9e62ca69f..47f5f9169 100644
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@@ -275,37 +275,39 @@ defmodule Pleroma.Object.Fetcher do
       |> maybe_date_fetch(date)
       |> sign_fetch(id, date)
 
-    case HTTP.get(id, headers) do
-      {:ok, %{body: body, status: code, headers: headers}} when code in 200..299 ->
-        case List.keyfind(headers, "content-type", 0) do
-          {_, content_type} ->
-            case Plug.Conn.Utils.media_type(content_type) do
-              {:ok, "application", "activity+json", _} ->
-                {:ok, body}
+    with {:ok, %{body: body, status: code, headers: headers}} when code in 200..299 <-
+           HTTP.get(id, headers),
+         {:has_content_type, {_, content_type}} <-
+           {:has_content_type, List.keyfind(headers, "content-type", 0)},
+         {:parse_content_type, {:ok, "application", subtype, type_params}} <-
+           {:parse_content_type, Plug.Conn.Utils.media_type(content_type)} do
+      case {subtype, type_params} do
+        {"activity+json", _} ->
+          {:ok, body}
 
-              {:ok, "application", "ld+json",
-               %{"profile" => "https://www.w3.org/ns/activitystreams"}} ->
-                {:ok, body}
+        {"ld+json", %{"profile" => "https://www.w3.org/ns/activitystreams"}} ->
+          {:ok, body}
 
-              # pixelfed sometimes (and only sometimes) responds with http instead of https
-              {:ok, "application", "ld+json",
-               %{"profile" => "http://www.w3.org/ns/activitystreams"}} ->
-                {:ok, body}
-
-              _ ->
-                {:error, {:content_type, content_type}}
-            end
-
-          _ ->
-            {:error, {:content_type, nil}}
-        end
+        # pixelfed sometimes (and only sometimes) responds with http instead of https
+        {"ld+json", %{"profile" => "http://www.w3.org/ns/activitystreams"}} ->
+          {:ok, body}
 
+        _ ->
+          {:error, {:content_type, content_type}}
+      end
+    else
       {:ok, %{status: code}} when code in [404, 410] ->
         {:error, {"Object has been deleted", id, code}}
 
       {:error, e} ->
         {:error, e}
 
+      {:has_content_type, _} ->
+        {:error, {:content_type, nil}}
+
+      {:parse_content_type, e} ->
+        {:error, {:content_type, e}}
+
       e ->
         {:error, e}
     end

From 93ab6a018e96fe300e677ae35b83e8989a11a09a Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Wed, 13 Mar 2024 19:41:14 -0100
Subject: [PATCH 27/43] mix: fix docs task

---
 mix.exs | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/mix.exs b/mix.exs
index 4e8b3160d..98d65484a 100644
--- a/mix.exs
+++ b/mix.exs
@@ -21,13 +21,13 @@ defmodule Pleroma.Mixfile do
       source_url: "https://akkoma.dev/AkkomaGang/akkoma",
       docs: [
         source_url_pattern: "https://akkoma.dev/AkkomaGang/akkoma/blob/develop/%{path}#L%{line}",
-        logo: "priv/static/images/logo.png",
-        extras: ["README.md", "CHANGELOG.md"] ++ Path.wildcard("docs/**/*.md"),
+        logo: "priv/static/logo-512.png",
+        extras: ["README.md", "CHANGELOG.md"] ++ Path.wildcard("docs/docs/**/*.md"),
         groups_for_extras: [
-          "Installation manuals": Path.wildcard("docs/installation/*.md"),
-          Configuration: Path.wildcard("docs/config/*.md"),
-          Administration: Path.wildcard("docs/admin/*.md"),
-          "Pleroma's APIs and Mastodon API extensions": Path.wildcard("docs/api/*.md")
+          "Installation manuals": Path.wildcard("docs/docs/installation/*.md"),
+          Configuration: Path.wildcard("docs/docs/config/*.md"),
+          Administration: Path.wildcard("docs/docs/admin/*.md"),
+          "Pleroma's APIs and Mastodon API extensions": Path.wildcard("docs/docs/api/*.md")
         ],
         main: "readme",
         output: "priv/static/doc"

From 2bcf633dc2863cea68f6528320817f80b8e0f14c Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Wed, 13 Mar 2024 19:42:51 -0100
Subject: [PATCH 28/43] Document Pleroma.Object.Fetcher

---
 lib/pleroma/object/containment.ex |  3 +++
 lib/pleroma/object/fetcher.ex     | 19 ++++++++++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/lib/pleroma/object/containment.ex b/lib/pleroma/object/containment.ex
index 040537acf..219bc3892 100644
--- a/lib/pleroma/object/containment.ex
+++ b/lib/pleroma/object/containment.ex
@@ -64,6 +64,9 @@ defmodule Pleroma.Object.Containment do
 
   def contain_origin(_id, _data), do: :error
 
+  @doc """
+  Check whether the object id is from the same host as another id
+  """
   def contain_origin_from_id(id, %{"id" => other_id} = _params) when is_binary(other_id) do
     id_uri = URI.parse(id)
     other_uri = URI.parse(other_id)
diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex
index 47f5f9169..c3aaf7a03 100644
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@@ -18,6 +18,14 @@ defmodule Pleroma.Object.Fetcher do
   require Logger
   require Pleroma.Constants
 
+  @moduledoc """
+  This module deals with correctly fetching Acitivity Pub objects in a safe way.
+
+  The core function is `fetch_and_contain_remote_object_from_id/1` which performs
+  the actual fetch and common safety and authenticity checks. Other `fetch_*`
+  function use the former and perform some additional tasks
+  """
+
   defp touch_changeset(changeset) do
     updated_at =
       NaiveDateTime.utc_now()
@@ -103,6 +111,7 @@ defmodule Pleroma.Object.Fetcher do
     end
   end
 
+  @doc "Assumes object already is in our database and refetches from remote to update (e.g. for polls)"
   def refetch_object(%Object{data: %{"id" => id}} = object) do
     with {:local, false} <- {:local, Object.local?(object)},
          {:ok, new_data} <- fetch_and_contain_remote_object_from_id(id),
@@ -114,7 +123,12 @@ defmodule Pleroma.Object.Fetcher do
     end
   end
 
-  # Note: will create a Create activity, which we need internally at the moment.
+  @doc """
+    Fetches a new object and puts it through the processing pipeline for inbound objects
+
+    Note: will also insert a fake Create activity, since atm we internally
+    need everything to be traced back to a Create activity.
+  """
   def fetch_object_from_id(id, options \\ []) do
     with %URI{} = uri <- URI.parse(id),
          # let's check the URI is even vaguely valid first
@@ -185,6 +199,7 @@ defmodule Pleroma.Object.Fetcher do
     |> Maps.put_if_present("bcc", data["bcc"])
   end
 
+  @doc "Identical to `fetch_object_from_id/2` but just directly returns the object or on error `nil`"
   def fetch_object_from_id!(id, options \\ []) do
     with {:ok, object} <- fetch_object_from_id(id, options) do
       object
@@ -235,6 +250,7 @@ defmodule Pleroma.Object.Fetcher do
     end
   end
 
+  @doc "Fetches arbitrary remote object and performs basic safety and authenticity checks"
   def fetch_and_contain_remote_object_from_id(id)
 
   def fetch_and_contain_remote_object_from_id(%{"id" => id}),
@@ -267,6 +283,7 @@ defmodule Pleroma.Object.Fetcher do
   def fetch_and_contain_remote_object_from_id(_id),
     do: {:error, "id must be a string"}
 
+  @doc "Do NOT use; only public for use in tests"
   def get_object(id) do
     date = Pleroma.Signature.signed_date()
 

From baaeffdebcf4efeeacd69ff8311fa6276c6b979d Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Wed, 13 Mar 2024 20:04:31 -0100
Subject: [PATCH 29/43] Update spoofed activity test

Turns out we already had a test for activities spoofed via upload due
to an exploit several years. Back then *oma did not verify content-type
at all and doing so was the only adopted countermeasure.
Even the added test sample though suffered from a mismatching id, yet
nobody seems to have thought it a good idea to tighten id checks, huh

Since we will add stricter id checks later, make id and URL match
and also add a testcase for no content type at all. The new section
will be expanded in subsequent commits.
---
 test/pleroma/object/fetcher_test.exs | 53 ++++++++++++++++++++++------
 1 file changed, 43 insertions(+), 10 deletions(-)

diff --git a/test/pleroma/object/fetcher_test.exs b/test/pleroma/object/fetcher_test.exs
index 8cf0bce48..b2da0a757 100644
--- a/test/pleroma/object/fetcher_test.exs
+++ b/test/pleroma/object/fetcher_test.exs
@@ -14,6 +14,17 @@ defmodule Pleroma.Object.FetcherTest do
   import Mock
   import Tesla.Mock
 
+  defp spoofed_object_with_ids(
+         id \\ "https://patch.cx/objects/spoof",
+         actor_id \\ "https://patch.cx/users/rin"
+       ) do
+    File.read!("test/fixtures/spoofed-object.json")
+    |> Jason.decode!()
+    |> Map.put("id", id)
+    |> Map.put("actor", actor_id)
+    |> Jason.encode!()
+  end
+
   setup do
     mock(fn
       %{method: :get, url: "https://mastodon.example.org/users/userisgone"} ->
@@ -22,15 +33,28 @@ defmodule Pleroma.Object.FetcherTest do
       %{method: :get, url: "https://mastodon.example.org/users/userisgone404"} ->
         %Tesla.Env{status: 404}
 
+      # Spoof: wrong Content-Type
       %{
         method: :get,
-        url:
-          "https://patch.cx/media/03ca3c8b4ac3ddd08bf0f84be7885f2f88de0f709112131a22d83650819e36c2.json"
+        url: "https://patch.cx/objects/spoof_content_type.json"
       } ->
         %Tesla.Env{
           status: 200,
+          url: "https://patch.cx/objects/spoof_content_type.json",
           headers: [{"content-type", "application/json"}],
-          body: File.read!("test/fixtures/spoofed-object.json")
+          body: spoofed_object_with_ids("https://patch.cx/objects/spoof_content_type.json")
+        }
+
+      # Spoof: no Content-Type
+      %{
+        method: :get,
+        url: "https://patch.cx/objects/spoof_content_type"
+      } ->
+        %Tesla.Env{
+          status: 200,
+          url: "https://patch.cx/objects/spoof_content_type",
+          headers: [],
+          body: spoofed_object_with_ids("https://patch.cx/objects/spoof_content_type")
         }
 
       env ->
@@ -129,6 +153,22 @@ defmodule Pleroma.Object.FetcherTest do
     end
   end
 
+  describe "fetcher security and auth checks" do
+    test "it does not fetch a spoofed object without content type" do
+      assert {:error, {:content_type, nil}} =
+               Fetcher.fetch_and_contain_remote_object_from_id(
+                 "https://patch.cx/objects/spoof_content_type"
+               )
+    end
+
+    test "it does not fetch a spoofed object with wrong content type" do
+      assert {:error, {:content_type, _}} =
+               Fetcher.fetch_and_contain_remote_object_from_id(
+                 "https://patch.cx/objects/spoof_content_type.json"
+               )
+    end
+  end
+
   describe "fetching an object" do
     test "it fetches an object" do
       {:ok, object} =
@@ -155,13 +195,6 @@ defmodule Pleroma.Object.FetcherTest do
                )
     end
 
-    test "it does not fetch a spoofed object uploaded on an instance as an attachment" do
-      assert {:error, _} =
-               Fetcher.fetch_object_from_id(
-                 "https://patch.cx/media/03ca3c8b4ac3ddd08bf0f84be7885f2f88de0f709112131a22d83650819e36c2.json"
-               )
-    end
-
     test "does not fetch anything from a rejected instance" do
       clear_config([:mrf_simple, :reject], [{"evil.example.org", "i said so"}])
 

From c4cf4d7f0bb1d1082f74aea8fb54ed50160ab29e Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Wed, 13 Mar 2024 20:12:17 -0100
Subject: [PATCH 30/43] Reject cross-domain redirects when fetching AP objects
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Such redirects on AP queries seem most likely to be a spoofing attempt.
If the object is legit, the id should match the final domain anyway and
users can directly use the canonical URL.

The lack of such a check (and use of the initially queried domain’s
authority instead of the final domain) was enabling the current exploit
to even affect instances which already migrated away from a same-domain
upload/proxy setup in the past, but retained a redirect to not break old
attachments.

(In theory this redirect could, with some effort, have been limited to
 only old files, but common guides employed a catch-all redirect, which
 allows even future uploads to be reachable via an initial query to the
 main domain)

Same-domain redirects are valid and also used by ourselves,
e.g. for redirecting /notice/XXX to /objects/YYY.
---
 lib/pleroma/object/fetcher.ex        | 25 +++++++++++++-
 test/pleroma/object/fetcher_test.exs | 50 ++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+), 1 deletion(-)

diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex
index c3aaf7a03..9c0bf7124 100644
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@@ -26,6 +26,8 @@ defmodule Pleroma.Object.Fetcher do
   function use the former and perform some additional tasks
   """
 
+  @mix_env Mix.env()
+
   defp touch_changeset(changeset) do
     updated_at =
       NaiveDateTime.utc_now()
@@ -283,6 +285,22 @@ defmodule Pleroma.Object.Fetcher do
   def fetch_and_contain_remote_object_from_id(_id),
     do: {:error, "id must be a string"}
 
+  defp check_crossdomain_redirect(final_host, original_url)
+
+  # HOPEFULLY TEMPORARY
+  # Basically none of our Tesla mocks in tests set the (supposed to
+  # exist for Tesla proper) url parameter for their responses
+  # causing almost every fetch in test to fail otherwise
+  if @mix_env == :test do
+    defp check_crossdomain_redirect(nil, _) do
+      {:cross_domain_redirect, false}
+    end
+  end
+
+  defp check_crossdomain_redirect(final_host, original_url) do
+    {:cross_domain_redirect, final_host != URI.parse(original_url).host}
+  end
+
   @doc "Do NOT use; only public for use in tests"
   def get_object(id) do
     date = Pleroma.Signature.signed_date()
@@ -292,8 +310,13 @@ defmodule Pleroma.Object.Fetcher do
       |> maybe_date_fetch(date)
       |> sign_fetch(id, date)
 
-    with {:ok, %{body: body, status: code, headers: headers}} when code in 200..299 <-
+    with {:ok, %{body: body, status: code, headers: headers, url: final_url}}
+         when code in 200..299 <-
            HTTP.get(id, headers),
+         remote_host <-
+           URI.parse(final_url).host,
+         {:cross_domain_redirect, false} <-
+           check_crossdomain_redirect(remote_host, id),
          {:has_content_type, {_, content_type}} <-
            {:has_content_type, List.keyfind(headers, "content-type", 0)},
          {:parse_content_type, {:ok, "application", subtype, type_params}} <-
diff --git a/test/pleroma/object/fetcher_test.exs b/test/pleroma/object/fetcher_test.exs
index b2da0a757..7f6f9c031 100644
--- a/test/pleroma/object/fetcher_test.exs
+++ b/test/pleroma/object/fetcher_test.exs
@@ -57,6 +57,33 @@ defmodule Pleroma.Object.FetcherTest do
           body: spoofed_object_with_ids("https://patch.cx/objects/spoof_content_type")
         }
 
+      # Spoof: cross-domain redirect with original domain id
+      %{method: :get, url: "https://patch.cx/objects/spoof_media_redirect1"} ->
+        %Tesla.Env{
+          status: 200,
+          url: "https://media.patch.cx/objects/spoof",
+          headers: [{"content-type", "application/activity+json"}],
+          body: spoofed_object_with_ids("https://patch.cx/objects/spoof_media_redirect1")
+        }
+
+      # Spoof: cross-domain redirect with final domain id
+      %{method: :get, url: "https://patch.cx/objects/spoof_media_redirect2"} ->
+        %Tesla.Env{
+          status: 200,
+          url: "https://media.patch.cx/objects/spoof_media_redirect2",
+          headers: [{"content-type", "application/activity+json"}],
+          body: spoofed_object_with_ids("https://media.patch.cx/objects/spoof_media_redirect2")
+        }
+
+      # No-Spoof: same domain redirect
+      %{method: :get, url: "https://patch.cx/objects/spoof_redirect"} ->
+        %Tesla.Env{
+          status: 200,
+          url: "https://patch.cx/objects/spoof",
+          headers: [{"content-type", "application/activity+json"}],
+          body: spoofed_object_with_ids("https://patch.cx/objects/spoof_redirect")
+        }
+
       env ->
         apply(HttpRequestMock, :request, [env])
     end)
@@ -167,6 +194,29 @@ defmodule Pleroma.Object.FetcherTest do
                  "https://patch.cx/objects/spoof_content_type.json"
                )
     end
+
+    test "it does not fetch an object via cross-domain redirects (initial id)" do
+      assert {:error, {:cross_domain_redirect, true}} =
+               Fetcher.fetch_and_contain_remote_object_from_id(
+                 "https://patch.cx/objects/spoof_media_redirect1"
+               )
+    end
+
+    test "it does not fetch an object via cross-domain redirects (final id)" do
+      assert {:error, {:cross_domain_redirect, true}} =
+               Fetcher.fetch_and_contain_remote_object_from_id(
+                 "https://patch.cx/objects/spoof_media_redirect2"
+               )
+    end
+
+    test "it accepts same-domain redirects" do
+      assert {:ok, %{"id" => id} = _object} =
+               Fetcher.fetch_and_contain_remote_object_from_id(
+                 "https://patch.cx/objects/spoof_redirect"
+               )
+
+      assert id == "https://patch.cx/objects/spoof_redirect"
+    end
   end
 
   describe "fetching an object" do

From fee57eb376a446ee394812762df148d7b4d19b39 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Wed, 13 Mar 2024 20:21:19 -0100
Subject: [PATCH 31/43] Move actor check into
 fetch_and_contain_remote_object_from_id
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This brings it in line with its name and closes an,
in practice harmless, verification hole.

This was/is the only user of contain_origin making it
safe to change the behaviour on actor-less objects.

Until now refetched objects did not ensure the new actor matches the
domain of the object. We refetch polls occasionally to retrieve
up-to-date vote counts. A malicious AP server could have switched out
the poll after initial posting with a completely different post
attribute to an actor from another server.
While we indeed fell for this spoof before the commit,
it fortunately seems to have had no ill effect in practice,
since the asociated Create activity is not changed. When exposing the
actor via our REST API, we read this info from the activity not the
object.

This at first thought still keeps one avenue for exploit open though:
the updated actor can be from our own domain and a third server be
instructed to fetch the object from us. However this is foiled by an
id mismatch. By necessity of being fetchable and our longstanding
same-domain check, the id must still be from the attacker’s server.
Even the most barebone authenticity check is able to sus this out.
---
 lib/pleroma/object/containment.ex        |  2 +-
 lib/pleroma/object/fetcher.ex            | 10 ++---
 test/pleroma/object/containment_test.exs | 50 ++++++++++++++++++++++--
 test/pleroma/object/fetcher_test.exs     | 20 ++++++++++
 4 files changed, 72 insertions(+), 10 deletions(-)

diff --git a/lib/pleroma/object/containment.ex b/lib/pleroma/object/containment.ex
index 219bc3892..85b777179 100644
--- a/lib/pleroma/object/containment.ex
+++ b/lib/pleroma/object/containment.ex
@@ -62,7 +62,7 @@ defmodule Pleroma.Object.Containment do
   def contain_origin(id, %{"attributedTo" => actor} = params),
     do: contain_origin(id, Map.put(params, "actor", actor))
 
-  def contain_origin(_id, _data), do: :error
+  def contain_origin(_id, _data), do: :ok
 
   @doc """
   Check whether the object id is from the same host as another id
diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex
index 9c0bf7124..16d8194e7 100644
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@@ -143,7 +143,6 @@ defmodule Pleroma.Object.Fetcher do
          {_, {:ok, data}} <- {:fetch, fetch_and_contain_remote_object_from_id(id)},
          {_, nil} <- {:normalize, Object.normalize(data, fetch: false)},
          params <- prepare_activity_params(data),
-         {_, :ok} <- {:containment, Containment.contain_origin(id, params)},
          {_, {:ok, activity}} <-
            {:transmogrifier, Transmogrifier.handle_incoming(params, options)},
          {_, _data, %Object{} = object} <-
@@ -156,9 +155,6 @@ defmodule Pleroma.Object.Fetcher do
       {:scheme, false} ->
         {:error, "URI Scheme Invalid"}
 
-      {:containment, _} ->
-        {:error, "Object containment failed."}
-
       {:transmogrifier, {:error, {:reject, e}}} ->
         {:reject, e}
 
@@ -264,7 +260,8 @@ defmodule Pleroma.Object.Fetcher do
     with {:scheme, true} <- {:scheme, String.starts_with?(id, "http")},
          {:ok, body} <- get_object(id),
          {:ok, data} <- safe_json_decode(body),
-         :ok <- Containment.contain_origin_from_id(id, data) do
+         {_, :ok} <- {:containment, Containment.contain_origin_from_id(id, data)},
+         {_, :ok} <- {:containment, Containment.contain_origin(id, data)} do
       unless Instances.reachable?(id) do
         Instances.set_reachable(id)
       end
@@ -274,6 +271,9 @@ defmodule Pleroma.Object.Fetcher do
       {:scheme, _} ->
         {:error, "Unsupported URI scheme"}
 
+      {:containment, _} ->
+        {:error, "Object containment failed."}
+
       {:error, e} ->
         {:error, e}
 
diff --git a/test/pleroma/object/containment_test.exs b/test/pleroma/object/containment_test.exs
index fb2fb7d49..ad1660069 100644
--- a/test/pleroma/object/containment_test.exs
+++ b/test/pleroma/object/containment_test.exs
@@ -17,16 +17,58 @@ defmodule Pleroma.Object.ContainmentTest do
   end
 
   describe "general origin containment" do
-    test "works for completely actorless posts" do
-      assert :error ==
-               Containment.contain_origin("https://glaceon.social/users/monorail", %{
+    test "handles completly actorless objects gracefully" do
+      assert :ok ==
+               Containment.contain_origin("https://glaceon.social/statuses/123", %{
                  "deleted" => "2019-10-30T05:48:50.249606Z",
                  "formerType" => "Note",
-                 "id" => "https://glaceon.social/users/monorail/statuses/103049757364029187",
+                 "id" => "https://glaceon.social/statuses/123",
                  "type" => "Tombstone"
                })
     end
 
+    test "errors for spoofed actors" do
+      assert :error ==
+               Containment.contain_origin("https://glaceon.social/statuses/123", %{
+                 "actor" => "https://otp.akkoma.dev/users/you",
+                 "id" => "https://glaceon.social/statuses/123",
+                 "type" => "Note"
+               })
+    end
+
+    test "errors for spoofed attributedTo" do
+      assert :error ==
+               Containment.contain_origin("https://glaceon.social/statuses/123", %{
+                 "attributedTo" => "https://otp.akkoma.dev/users/you",
+                 "id" => "https://glaceon.social/statuses/123",
+                 "type" => "Note"
+               })
+    end
+
+    test "accepts valid actors" do
+      assert :ok ==
+               Containment.contain_origin("https://glaceon.social/statuses/123", %{
+                 "actor" => "https://glaceon.social/users/monorail",
+                 "attributedTo" => "https://glaceon.social/users/monorail",
+                 "id" => "https://glaceon.social/statuses/123",
+                 "type" => "Note"
+               })
+
+      assert :ok ==
+               Containment.contain_origin("https://glaceon.social/statuses/123", %{
+                 "actor" => "https://glaceon.social/users/monorail",
+                 "id" => "https://glaceon.social/statuses/123",
+                 "type" => "Note"
+               })
+
+      assert :ok ==
+               Containment.contain_origin("https://glaceon.social/statuses/123", %{
+                 "attributedTo" => "https://glaceon.social/users/monorail",
+                 "id" => "https://glaceon.social/statuses/123",
+                 "type" => "Note"
+               })
+    end
+
     test "contain_origin_from_id() catches obvious spoofing attempts" do
       data = %{
         "id" => "http://example.com/~alyssa/activities/1234.json"
diff --git a/test/pleroma/object/fetcher_test.exs b/test/pleroma/object/fetcher_test.exs
index 7f6f9c031..b289e869d 100644
--- a/test/pleroma/object/fetcher_test.exs
+++ b/test/pleroma/object/fetcher_test.exs
@@ -84,6 +84,19 @@ defmodule Pleroma.Object.FetcherTest do
           body: spoofed_object_with_ids("https://patch.cx/objects/spoof_redirect")
         }
 
+      # Spoof: Actor from another domain
+      %{method: :get, url: "https://patch.cx/objects/spoof_foreign_actor"} ->
+        %Tesla.Env{
+          status: 200,
+          url: "https://patch.cx/objects/spoof_foreign_actor",
+          headers: [{"content-type", "application/activity+json"}],
+          body:
+            spoofed_object_with_ids(
+              "https://patch.cx/objects/spoof_foreign_actor",
+              "https://not.patch.cx/users/rin"
+            )
+        }
+
       env ->
         apply(HttpRequestMock, :request, [env])
     end)
@@ -217,6 +230,13 @@ defmodule Pleroma.Object.FetcherTest do
 
       assert id == "https://patch.cx/objects/spoof_redirect"
     end
+
+    test "it does not fetch a spoofed object with a foreign actor" do
+      assert {:error, "Object containment failed."} =
+               Fetcher.fetch_and_contain_remote_object_from_id(
+                 "https://patch.cx/objects/spoof_foreign_actor"
+               )
+    end
   end
 
   describe "fetching an object" do

From 59a142e0b0e6760e116443567802ef2ab7483178 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Wed, 13 Mar 2024 21:00:23 -0100
Subject: [PATCH 32/43] Never fetch resource from ourselves
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

If it’s not already in the database,
it must be counterfeit (or just not exists at all)

Changed test URLs were only ever used from "local: false" users anyway.
---
 lib/pleroma/object/containment.ex             | 13 +++++++
 lib/pleroma/object/fetcher.ex                 |  4 +++
 .../users_mock/friendica_followers.json       |  2 +-
 .../users_mock/friendica_following.json       |  2 +-
 .../users_mock/masto_closed_followers.json    |  4 +--
 .../masto_closed_followers_page.json          |  2 +-
 .../users_mock/masto_closed_following.json    |  4 +--
 .../masto_closed_following_page.json          |  2 +-
 .../users_mock/pleroma_followers.json         |  8 ++---
 .../users_mock/pleroma_following.json         |  8 ++---
 test/pleroma/object/fetcher_test.exs          |  7 ++++
 test/pleroma/user_test.exs                    | 22 ++++++------
 .../web/activity_pub/activity_pub_test.exs    | 36 +++++++++----------
 test/support/http_request_mock.ex             | 16 ++++-----
 14 files changed, 77 insertions(+), 53 deletions(-)

diff --git a/lib/pleroma/object/containment.ex b/lib/pleroma/object/containment.ex
index 85b777179..48e62a917 100644
--- a/lib/pleroma/object/containment.ex
+++ b/lib/pleroma/object/containment.ex
@@ -47,6 +47,19 @@ defmodule Pleroma.Object.Containment do
   defp compare_uris(%URI{host: host} = _id_uri, %URI{host: host} = _other_uri), do: :ok
   defp compare_uris(_id_uri, _other_uri), do: :error
 
+  @doc """
+  Checks whether an URL to fetch from is from the local server.
+
+  We never want to fetch from ourselves; if it’s not in the database
+  it can’t be authentic and must be a counterfeit.
+  """
+  def contain_local_fetch(id) do
+    case compare_uris(URI.parse(id), Pleroma.Web.Endpoint.struct_url()) do
+      :ok -> :error
+      _ -> :ok
+    end
+  end
+
   @doc """
   Checks that an imported AP object's actor matches the host it came from.
   """
diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex
index 16d8194e7..f256bbf77 100644
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@@ -258,6 +258,7 @@ defmodule Pleroma.Object.Fetcher do
     Logger.debug("Fetching object #{id} via AP")
 
     with {:scheme, true} <- {:scheme, String.starts_with?(id, "http")},
+         {_, :ok} <- {:local_fetch, Containment.contain_local_fetch(id)},
          {:ok, body} <- get_object(id),
          {:ok, data} <- safe_json_decode(body),
          {_, :ok} <- {:containment, Containment.contain_origin_from_id(id, data)},
@@ -271,6 +272,9 @@ defmodule Pleroma.Object.Fetcher do
       {:scheme, _} ->
         {:error, "Unsupported URI scheme"}
 
+      {:local_fetch, _} ->
+        {:error, "Trying to fetch local resource"}
+
       {:containment, _} ->
         {:error, "Object containment failed."}
 
diff --git a/test/fixtures/users_mock/friendica_followers.json b/test/fixtures/users_mock/friendica_followers.json
index 7b86b5fe2..02b287e23 100644
--- a/test/fixtures/users_mock/friendica_followers.json
+++ b/test/fixtures/users_mock/friendica_followers.json
@@ -13,7 +13,7 @@
       "directMessage": "litepub:directMessage"
     }
   ],
-  "id": "http://localhost:8080/followers/fuser3",
+  "id": "http://remote.org/followers/fuser3",
   "type": "OrderedCollection",
   "totalItems": 296
 }
diff --git a/test/fixtures/users_mock/friendica_following.json b/test/fixtures/users_mock/friendica_following.json
index 7c526befc..0908e78f0 100644
--- a/test/fixtures/users_mock/friendica_following.json
+++ b/test/fixtures/users_mock/friendica_following.json
@@ -13,7 +13,7 @@
       "directMessage": "litepub:directMessage"
     }
   ],
-  "id": "http://localhost:8080/following/fuser3",
+  "id": "http://remote.org/following/fuser3",
   "type": "OrderedCollection",
   "totalItems": 32
 }
diff --git a/test/fixtures/users_mock/masto_closed_followers.json b/test/fixtures/users_mock/masto_closed_followers.json
index da296892d..ccc32d15e 100644
--- a/test/fixtures/users_mock/masto_closed_followers.json
+++ b/test/fixtures/users_mock/masto_closed_followers.json
@@ -1,7 +1,7 @@
 {
   "@context": "https://www.w3.org/ns/activitystreams",
-  "id": "http://localhost:4001/users/masto_closed/followers",
+  "id": "http://remote.org/users/masto_closed/followers",
   "type": "OrderedCollection",
   "totalItems": 437,
-  "first": "http://localhost:4001/users/masto_closed/followers?page=1"
+  "first": "http://remote.org/users/masto_closed/followers?page=1"
 }
diff --git a/test/fixtures/users_mock/masto_closed_followers_page.json b/test/fixtures/users_mock/masto_closed_followers_page.json
index 04ab0c4d3..e4f1b3ac0 100644
--- a/test/fixtures/users_mock/masto_closed_followers_page.json
+++ b/test/fixtures/users_mock/masto_closed_followers_page.json
@@ -1 +1 @@
-{"@context":"https://www.w3.org/ns/activitystreams","id":"http://localhost:4001/users/masto_closed/followers?page=1","type":"OrderedCollectionPage","totalItems":437,"next":"http://localhost:4001/users/masto_closed/followers?page=2","partOf":"http://localhost:4001/users/masto_closed/followers","orderedItems":["https://testing.uguu.ltd/users/rin","https://patch.cx/users/rin","https://letsalllovela.in/users/xoxo","https://pleroma.site/users/crushv","https://aria.company/users/boris","https://kawen.space/users/crushv","https://freespeech.host/users/cvcvcv","https://pleroma.site/users/picpub","https://pixelfed.social/users/nosleep","https://boopsnoot.gq/users/5c1896d162f7d337f90492a3","https://pikachu.rocks/users/waifu","https://royal.crablettesare.life/users/crablettes"]}
+{"@context":"https://www.w3.org/ns/activitystreams","id":"http://remote.org/users/masto_closed/followers?page=1","type":"OrderedCollectionPage","totalItems":437,"next":"http://remote.org/users/masto_closed/followers?page=2","partOf":"http://remote.org/users/masto_closed/followers","orderedItems":["https://testing.uguu.ltd/users/rin","https://patch.cx/users/rin","https://letsalllovela.in/users/xoxo","https://pleroma.site/users/crushv","https://aria.company/users/boris","https://kawen.space/users/crushv","https://freespeech.host/users/cvcvcv","https://pleroma.site/users/picpub","https://pixelfed.social/users/nosleep","https://boopsnoot.gq/users/5c1896d162f7d337f90492a3","https://pikachu.rocks/users/waifu","https://royal.crablettesare.life/users/crablettes"]}
diff --git a/test/fixtures/users_mock/masto_closed_following.json b/test/fixtures/users_mock/masto_closed_following.json
index 146d49f9c..34e9e9fe8 100644
--- a/test/fixtures/users_mock/masto_closed_following.json
+++ b/test/fixtures/users_mock/masto_closed_following.json
@@ -1,7 +1,7 @@
 {
   "@context": "https://www.w3.org/ns/activitystreams",
-  "id": "http://localhost:4001/users/masto_closed/following",
+  "id": "http://remote.org/users/masto_closed/following",
   "type": "OrderedCollection",
   "totalItems": 152,
-  "first": "http://localhost:4001/users/masto_closed/following?page=1"
+  "first": "http://remote.org/users/masto_closed/following?page=1"
 }
diff --git a/test/fixtures/users_mock/masto_closed_following_page.json b/test/fixtures/users_mock/masto_closed_following_page.json
index 8d8324699..d398ae3cf 100644
--- a/test/fixtures/users_mock/masto_closed_following_page.json
+++ b/test/fixtures/users_mock/masto_closed_following_page.json
@@ -1 +1 @@
-{"@context":"https://www.w3.org/ns/activitystreams","id":"http://localhost:4001/users/masto_closed/following?page=1","type":"OrderedCollectionPage","totalItems":152,"next":"http://localhost:4001/users/masto_closed/following?page=2","partOf":"http://localhost:4001/users/masto_closed/following","orderedItems":["https://testing.uguu.ltd/users/rin","https://patch.cx/users/rin","https://letsalllovela.in/users/xoxo","https://pleroma.site/users/crushv","https://aria.company/users/boris","https://kawen.space/users/crushv","https://freespeech.host/users/cvcvcv","https://pleroma.site/users/picpub","https://pixelfed.social/users/nosleep","https://boopsnoot.gq/users/5c1896d162f7d337f90492a3","https://pikachu.rocks/users/waifu","https://royal.crablettesare.life/users/crablettes"]}
+{"@context":"https://www.w3.org/ns/activitystreams","id":"http://remote.org/users/masto_closed/following?page=1","type":"OrderedCollectionPage","totalItems":152,"next":"http://remote.org/users/masto_closed/following?page=2","partOf":"http://remote.org/users/masto_closed/following","orderedItems":["https://testing.uguu.ltd/users/rin","https://patch.cx/users/rin","https://letsalllovela.in/users/xoxo","https://pleroma.site/users/crushv","https://aria.company/users/boris","https://kawen.space/users/crushv","https://freespeech.host/users/cvcvcv","https://pleroma.site/users/picpub","https://pixelfed.social/users/nosleep","https://boopsnoot.gq/users/5c1896d162f7d337f90492a3","https://pikachu.rocks/users/waifu","https://royal.crablettesare.life/users/crablettes"]}
diff --git a/test/fixtures/users_mock/pleroma_followers.json b/test/fixtures/users_mock/pleroma_followers.json
index db71d084b..9611990ee 100644
--- a/test/fixtures/users_mock/pleroma_followers.json
+++ b/test/fixtures/users_mock/pleroma_followers.json
@@ -1,14 +1,14 @@
 {
   "type": "OrderedCollection",
   "totalItems": 527,
-  "id": "http://localhost:4001/users/fuser2/followers",
+  "id": "http://remote.org/users/fuser2/followers",
   "first": {
     "type": "OrderedCollectionPage",
     "totalItems": 527,
-    "partOf": "http://localhost:4001/users/fuser2/followers",
+    "partOf": "http://remote.org/users/fuser2/followers",
     "orderedItems": [],
-    "next": "http://localhost:4001/users/fuser2/followers?page=2",
-    "id": "http://localhost:4001/users/fuser2/followers?page=1"
+    "next": "http://remote.org/users/fuser2/followers?page=2",
+    "id": "http://remote.org/users/fuser2/followers?page=1"
   },
   "@context": [
     "https://www.w3.org/ns/activitystreams",
diff --git a/test/fixtures/users_mock/pleroma_following.json b/test/fixtures/users_mock/pleroma_following.json
index 33d087703..27fadbc94 100644
--- a/test/fixtures/users_mock/pleroma_following.json
+++ b/test/fixtures/users_mock/pleroma_following.json
@@ -1,14 +1,14 @@
 {
   "type": "OrderedCollection",
   "totalItems": 267,
-  "id": "http://localhost:4001/users/fuser2/following",
+  "id": "http://remote.org/users/fuser2/following",
   "first": {
     "type": "OrderedCollectionPage",
     "totalItems": 267,
-    "partOf": "http://localhost:4001/users/fuser2/following",
+    "partOf": "http://remote.org/users/fuser2/following",
     "orderedItems": [],
-    "next": "http://localhost:4001/users/fuser2/following?page=2",
-    "id": "http://localhost:4001/users/fuser2/following?page=1"
+    "next": "http://remote.org/users/fuser2/following?page=2",
+    "id": "http://remote.org/users/fuser2/following?page=1"
   },
   "@context": [
     "https://www.w3.org/ns/activitystreams",
diff --git a/test/pleroma/object/fetcher_test.exs b/test/pleroma/object/fetcher_test.exs
index b289e869d..d1f3c070e 100644
--- a/test/pleroma/object/fetcher_test.exs
+++ b/test/pleroma/object/fetcher_test.exs
@@ -237,6 +237,13 @@ defmodule Pleroma.Object.FetcherTest do
                  "https://patch.cx/objects/spoof_foreign_actor"
                )
     end
+
+    test "it does not fetch from localhost" do
+      assert {:error, "Trying to fetch local resource"} =
+               Fetcher.fetch_and_contain_remote_object_from_id(
+                 Pleroma.Web.Endpoint.url() <> "/spoof_local"
+               )
+    end
   end
 
   describe "fetching an object" do
diff --git a/test/pleroma/user_test.exs b/test/pleroma/user_test.exs
index 5a0d77cab..96ca8d0fd 100644
--- a/test/pleroma/user_test.exs
+++ b/test/pleroma/user_test.exs
@@ -326,9 +326,9 @@ defmodule Pleroma.UserTest do
         insert(:user, %{
           local: false,
           nickname: "fuser2",
-          ap_id: "http://localhost:4001/users/fuser2",
-          follower_address: "http://localhost:4001/users/fuser2/followers",
-          following_address: "http://localhost:4001/users/fuser2/following"
+          ap_id: "http://remote.org/users/fuser2",
+          follower_address: "http://remote.org/users/fuser2/followers",
+          following_address: "http://remote.org/users/fuser2/following"
         })
 
       {:ok, user, followed} = User.follow(user, followed, :follow_accept)
@@ -2177,8 +2177,8 @@ defmodule Pleroma.UserTest do
 
   describe "sync followers count" do
     setup do
-      user1 = insert(:user, local: false, ap_id: "http://localhost:4001/users/masto_closed")
-      user2 = insert(:user, local: false, ap_id: "http://localhost:4001/users/fuser2")
+      user1 = insert(:user, local: false, ap_id: "http://remote.org/users/masto_closed")
+      user2 = insert(:user, local: false, ap_id: "http://remote.org/users/fuser2")
       insert(:user, local: true)
       insert(:user, local: false, is_active: false)
       {:ok, user1: user1, user2: user2}
@@ -2272,8 +2272,8 @@ defmodule Pleroma.UserTest do
       other_user =
         insert(:user,
           local: false,
-          follower_address: "http://localhost:4001/users/masto_closed/followers",
-          following_address: "http://localhost:4001/users/masto_closed/following",
+          follower_address: "http://remote.org/users/masto_closed/followers",
+          following_address: "http://remote.org/users/masto_closed/following",
           ap_enabled: true
         )
 
@@ -2294,8 +2294,8 @@ defmodule Pleroma.UserTest do
       other_user =
         insert(:user,
           local: false,
-          follower_address: "http://localhost:4001/users/masto_closed/followers",
-          following_address: "http://localhost:4001/users/masto_closed/following",
+          follower_address: "http://remote.org/users/masto_closed/followers",
+          following_address: "http://remote.org/users/masto_closed/following",
           ap_enabled: true
         )
 
@@ -2316,8 +2316,8 @@ defmodule Pleroma.UserTest do
       other_user =
         insert(:user,
           local: false,
-          follower_address: "http://localhost:4001/users/masto_closed/followers",
-          following_address: "http://localhost:4001/users/masto_closed/following",
+          follower_address: "http://remote.org/users/masto_closed/followers",
+          following_address: "http://remote.org/users/masto_closed/following",
           ap_enabled: true
         )
 
diff --git a/test/pleroma/web/activity_pub/activity_pub_test.exs b/test/pleroma/web/activity_pub/activity_pub_test.exs
index 69b4ac257..c6543ec83 100644
--- a/test/pleroma/web/activity_pub/activity_pub_test.exs
+++ b/test/pleroma/web/activity_pub/activity_pub_test.exs
@@ -1643,8 +1643,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
       user =
         insert(:user,
           local: false,
-          follower_address: "http://localhost:4001/users/fuser2/followers",
-          following_address: "http://localhost:4001/users/fuser2/following"
+          follower_address: "http://remote.org/users/fuser2/followers",
+          following_address: "http://remote.org/users/fuser2/following"
         )
 
       {:ok, info} = ActivityPub.fetch_follow_information_for_user(user)
@@ -1655,7 +1655,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
     test "detects hidden followers" do
       mock(fn env ->
         case env.url do
-          "http://localhost:4001/users/masto_closed/followers?page=1" ->
+          "http://remote.org/users/masto_closed/followers?page=1" ->
             %Tesla.Env{status: 403, body: ""}
 
           _ ->
@@ -1666,8 +1666,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
       user =
         insert(:user,
           local: false,
-          follower_address: "http://localhost:4001/users/masto_closed/followers",
-          following_address: "http://localhost:4001/users/masto_closed/following"
+          follower_address: "http://remote.org/users/masto_closed/followers",
+          following_address: "http://remote.org/users/masto_closed/following"
         )
 
       {:ok, follow_info} = ActivityPub.fetch_follow_information_for_user(user)
@@ -1678,7 +1678,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
     test "detects hidden follows" do
       mock(fn env ->
         case env.url do
-          "http://localhost:4001/users/masto_closed/following?page=1" ->
+          "http://remote.org/users/masto_closed/following?page=1" ->
             %Tesla.Env{status: 403, body: ""}
 
           _ ->
@@ -1689,8 +1689,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
       user =
         insert(:user,
           local: false,
-          follower_address: "http://localhost:4001/users/masto_closed/followers",
-          following_address: "http://localhost:4001/users/masto_closed/following"
+          follower_address: "http://remote.org/users/masto_closed/followers",
+          following_address: "http://remote.org/users/masto_closed/following"
         )
 
       {:ok, follow_info} = ActivityPub.fetch_follow_information_for_user(user)
@@ -1702,8 +1702,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
       user =
         insert(:user,
           local: false,
-          follower_address: "http://localhost:8080/followers/fuser3",
-          following_address: "http://localhost:8080/following/fuser3"
+          follower_address: "http://remote.org/followers/fuser3",
+          following_address: "http://remote.org/following/fuser3"
         )
 
       {:ok, follow_info} = ActivityPub.fetch_follow_information_for_user(user)
@@ -1716,28 +1716,28 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
     test "doesn't crash when follower and following counters are hidden" do
       mock(fn env ->
         case env.url do
-          "http://localhost:4001/users/masto_hidden_counters/following" ->
+          "http://remote.org/users/masto_hidden_counters/following" ->
             json(
               %{
                 "@context" => "https://www.w3.org/ns/activitystreams",
-                "id" => "http://localhost:4001/users/masto_hidden_counters/followers"
+                "id" => "http://remote.org/users/masto_hidden_counters/followers"
               },
               headers: HttpRequestMock.activitypub_object_headers()
             )
 
-          "http://localhost:4001/users/masto_hidden_counters/following?page=1" ->
+          "http://remote.org/users/masto_hidden_counters/following?page=1" ->
             %Tesla.Env{status: 403, body: ""}
 
-          "http://localhost:4001/users/masto_hidden_counters/followers" ->
+          "http://remote.org/users/masto_hidden_counters/followers" ->
             json(
               %{
                 "@context" => "https://www.w3.org/ns/activitystreams",
-                "id" => "http://localhost:4001/users/masto_hidden_counters/following"
+                "id" => "http://remote.org/users/masto_hidden_counters/following"
               },
               headers: HttpRequestMock.activitypub_object_headers()
             )
 
-          "http://localhost:4001/users/masto_hidden_counters/followers?page=1" ->
+          "http://remote.org/users/masto_hidden_counters/followers?page=1" ->
             %Tesla.Env{status: 403, body: ""}
         end
       end)
@@ -1745,8 +1745,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
       user =
         insert(:user,
           local: false,
-          follower_address: "http://localhost:4001/users/masto_hidden_counters/followers",
-          following_address: "http://localhost:4001/users/masto_hidden_counters/following"
+          follower_address: "http://remote.org/users/masto_hidden_counters/followers",
+          following_address: "http://remote.org/users/masto_hidden_counters/following"
         )
 
       {:ok, follow_info} = ActivityPub.fetch_follow_information_for_user(user)
diff --git a/test/support/http_request_mock.ex b/test/support/http_request_mock.ex
index 6772a7421..e831f43f7 100644
--- a/test/support/http_request_mock.ex
+++ b/test/support/http_request_mock.ex
@@ -964,7 +964,7 @@ defmodule HttpRequestMock do
     {:ok, %Tesla.Env{status: 200, body: File.read!("test/fixtures/rich_media/ogp.html")}}
   end
 
-  def get("http://localhost:4001/users/masto_closed/followers", _, _, _) do
+  def get("http://remote.org/users/masto_closed/followers", _, _, _) do
     {:ok,
      %Tesla.Env{
        status: 200,
@@ -973,7 +973,7 @@ defmodule HttpRequestMock do
      }}
   end
 
-  def get("http://localhost:4001/users/masto_closed/followers?page=1", _, _, _) do
+  def get("http://remote.org/users/masto_closed/followers?page=1", _, _, _) do
     {:ok,
      %Tesla.Env{
        status: 200,
@@ -982,7 +982,7 @@ defmodule HttpRequestMock do
      }}
   end
 
-  def get("http://localhost:4001/users/masto_closed/following", _, _, _) do
+  def get("http://remote.org/users/masto_closed/following", _, _, _) do
     {:ok,
      %Tesla.Env{
        status: 200,
@@ -991,7 +991,7 @@ defmodule HttpRequestMock do
      }}
   end
 
-  def get("http://localhost:4001/users/masto_closed/following?page=1", _, _, _) do
+  def get("http://remote.org/users/masto_closed/following?page=1", _, _, _) do
     {:ok,
      %Tesla.Env{
        status: 200,
@@ -1000,7 +1000,7 @@ defmodule HttpRequestMock do
      }}
   end
 
-  def get("http://localhost:8080/followers/fuser3", _, _, _) do
+  def get("http://remote.org/followers/fuser3", _, _, _) do
     {:ok,
      %Tesla.Env{
        status: 200,
@@ -1009,7 +1009,7 @@ defmodule HttpRequestMock do
      }}
   end
 
-  def get("http://localhost:8080/following/fuser3", _, _, _) do
+  def get("http://remote.org/following/fuser3", _, _, _) do
     {:ok,
      %Tesla.Env{
        status: 200,
@@ -1018,7 +1018,7 @@ defmodule HttpRequestMock do
      }}
   end
 
-  def get("http://localhost:4001/users/fuser2/followers", _, _, _) do
+  def get("http://remote.org/users/fuser2/followers", _, _, _) do
     {:ok,
      %Tesla.Env{
        status: 200,
@@ -1027,7 +1027,7 @@ defmodule HttpRequestMock do
      }}
   end
 
-  def get("http://localhost:4001/users/fuser2/following", _, _, _) do
+  def get("http://remote.org/users/fuser2/following", _, _, _) do
     {:ok,
      %Tesla.Env{
        status: 200,

From f07eb4cb55ebcc68bdf7ed502fcbffbdee7dc819 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Fri, 15 Mar 2024 20:31:45 -0100
Subject: [PATCH 33/43] Sanity check fetched user data

In order to properly process incoming notes we need
to be able to map the key id back to an actor.
Also, check collections actually belong to the same server.

Key ids of Hubzilla and Bridgy samples were updated to what
modern versions of those output. If anything still uses the
old format, we would not be able to verify their posts anyway.
---
 lib/pleroma/object/containment.ex             |  8 ++
 lib/pleroma/web/activity_pub/activity_pub.ex  |  6 ++
 .../object_validators/user_validator.ex       | 92 +++++++++++++++++++
 test/fixtures/bridgy/actor.json               |  2 +-
 ...ps___osada.macgirvin.com_channel_mike.json |  2 +-
 .../kaniini@hubzilla.example.org.json         |  2 +-
 6 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100644 lib/pleroma/web/activity_pub/object_validators/user_validator.ex

diff --git a/lib/pleroma/object/containment.ex b/lib/pleroma/object/containment.ex
index 48e62a917..a312f69e8 100644
--- a/lib/pleroma/object/containment.ex
+++ b/lib/pleroma/object/containment.ex
@@ -101,4 +101,12 @@ defmodule Pleroma.Object.Containment do
     do: contain_origin(id, object)
 
   def contain_child(_), do: :ok
+
+  @doc "Checks whether two URIs belong to the same domain"
+  def same_origin(id1, id2) do
+    uri1 = URI.parse(id1)
+    uri2 = URI.parse(id2)
+
+    compare_uris(uri1, uri2)
+  end
 end
diff --git a/lib/pleroma/web/activity_pub/activity_pub.ex b/lib/pleroma/web/activity_pub/activity_pub.ex
index 4a8ce2d3d..deca69613 100644
--- a/lib/pleroma/web/activity_pub/activity_pub.ex
+++ b/lib/pleroma/web/activity_pub/activity_pub.ex
@@ -22,6 +22,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   alias Pleroma.Upload
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.MRF
+  alias Pleroma.Web.ActivityPub.ObjectValidators.UserValidator
   alias Pleroma.Web.ActivityPub.Transmogrifier
   alias Pleroma.Web.Streamer
   alias Pleroma.Web.WebFinger
@@ -1722,6 +1723,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   def fetch_and_prepare_user_from_ap_id(ap_id, additional \\ []) do
     with {:ok, data} <- Fetcher.fetch_and_contain_remote_object_from_id(ap_id),
+         {:valid, {:ok, _, _}} <- {:valid, UserValidator.validate(data, [])},
          {:ok, data} <- user_data_from_user_object(data, additional) do
       {:ok, maybe_update_follow_information(data)}
     else
@@ -1734,6 +1736,10 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
         Logger.debug("Rejected user #{ap_id}: #{inspect(reason)}")
         {:error, e}
 
+      {:valid, reason} ->
+        Logger.debug("Data is not a valid user #{ap_id}: #{inspect(reason)}")
+        {:error, "Not a user"}
+
       {:error, e} ->
         Logger.error("Could not decode user at fetch #{ap_id}, #{inspect(e)}")
         {:error, e}
diff --git a/lib/pleroma/web/activity_pub/object_validators/user_validator.ex b/lib/pleroma/web/activity_pub/object_validators/user_validator.ex
new file mode 100644
index 000000000..90b5404f3
--- /dev/null
+++ b/lib/pleroma/web/activity_pub/object_validators/user_validator.ex
@@ -0,0 +1,92 @@
+# Akkoma: Magically expressive social media
+# Copyright © 2024 Akkoma Authors <https://akkoma.dev/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.ObjectValidators.UserValidator do
+  @moduledoc """
+  Checks whether ActivityPub data represents a valid user
+
+  Users don't go through the same ingest pipeline like activities or other objects.
+  To ensure this can only match a user and no users match in the other pipeline,
+  this is a separate from the generic ObjectValidator.
+  """
+
+  @behaviour Pleroma.Web.ActivityPub.ObjectValidator.Validating
+
+  alias Pleroma.Object.Containment
+  alias Pleroma.Signature
+
+  @impl true
+  def validate(object, meta)
+
+  def validate(%{"type" => type, "id" => _id} = data, meta)
+      when type in ["Person", "Organization", "Group", "Application"] do
+    with :ok <- validate_pubkey(data),
+         :ok <- validate_inbox(data),
+         :ok <- contain_collection_origin(data) do
+      {:ok, data, meta}
+    else
+      {:error, e} -> {:error, e}
+      e -> {:error, e}
+    end
+  end
+
+  def validate(_, _), do: {:error, "Not a user object"}
+
+  defp mabye_validate_owner(nil, _actor), do: :ok
+  defp mabye_validate_owner(actor, actor), do: :ok
+  defp mabye_validate_owner(_owner, _actor), do: :error
+
+  defp validate_pubkey(
+         %{"id" => id, "publicKey" => %{"id" => pk_id, "publicKeyPem" => _key}} = data
+       )
+       when id != nil do
+    with {_, {:ok, kactor}} <- {:key, Signature.key_id_to_actor_id(pk_id)},
+         true <- id == kactor,
+         :ok <- mabye_validate_owner(Map.get(data, "owner"), id) do
+      :ok
+    else
+      {:key, _} ->
+        {:error, "Unable to determine actor id from key id"}
+
+      false ->
+        {:error, "Key id does not relate to user id"}
+
+      _ ->
+        {:error, "Actor does not own its public key"}
+    end
+  end
+
+  # pubkey is optional atm
+  defp validate_pubkey(_data), do: :ok
+
+  defp validate_inbox(%{"id" => id, "inbox" => inbox}) do
+    case Containment.same_origin(id, inbox) do
+      :ok -> :ok
+      :error -> {:error, "Inbox on different doamin"}
+    end
+  end
+
+  defp validate_inbox(_), do: {:error, "No inbox"}
+
+  defp check_field_value(%{"id" => id} = _data, value) do
+    Containment.same_origin(id, value)
+  end
+
+  defp maybe_check_field(data, field) do
+    with val when val != nil <- data[field],
+         :ok <- check_field_value(data, val) do
+      :ok
+    else
+      nil -> :ok
+      _ -> {:error, "#{field} on different domain"}
+    end
+  end
+
+  defp contain_collection_origin(data) do
+    Enum.reduce(["followers", "following", "featured"], :ok, fn
+      field, :ok -> maybe_check_field(data, field)
+      _, error -> error
+    end)
+  end
+end
diff --git a/test/fixtures/bridgy/actor.json b/test/fixtures/bridgy/actor.json
index 5b2d8982b..b4e859a82 100644
--- a/test/fixtures/bridgy/actor.json
+++ b/test/fixtures/bridgy/actor.json
@@ -70,7 +70,7 @@
   "preferredUsername": "jk.nipponalba.scot",
   "summary": "",
   "publicKey": {
-    "id": "jk.nipponalba.scot",
+    "id": "https://fed.brid.gy/jk.nipponalba.scot#key",
     "publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDdarxwzxnNbJ2hneWOYHkYJowk\npyigQtxlUd0VjgSQHwxU9kWqfbrHBVADyTtcqi/4dAzQd3UnCI1TPNnn4LPZY9PW\noiWd3Zl1/EfLFxO7LU9GS7fcSLQkyj5JNhSlN3I8QPudZbybrgRDVZYooDe1D+52\n5KLGqC2ajrIVOiDRTQIDAQAB\n-----END PUBLIC KEY-----"
   },
   "inbox": "https://fed.brid.gy/jk.nipponalba.scot/inbox",
diff --git a/test/fixtures/tesla_mock/https___osada.macgirvin.com_channel_mike.json b/test/fixtures/tesla_mock/https___osada.macgirvin.com_channel_mike.json
index ca76d6e17..70d750e5d 100644
--- a/test/fixtures/tesla_mock/https___osada.macgirvin.com_channel_mike.json
+++ b/test/fixtures/tesla_mock/https___osada.macgirvin.com_channel_mike.json
@@ -37,7 +37,7 @@
         "sharedInbox": "https://osada.macgirvin.com/inbox"
     },
     "publicKey": {
-        "id": "https://osada.macgirvin.com/channel/mike/public_key_pem",
+        "id": "https://osada.macgirvin.com/channel/mike",
         "owner": "https://osada.macgirvin.com/channel/mike",
         "publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAskSyK2VwBNKbzZl9XNJk\nvxU5AAilmRArMmmKSzphdHaVBHakeafUfixvqNrQ/oX2srJvJKcghNmEMrJ6MJ7r\npeEndVOo7pcP4PwVjtnC06p3J711q5tBehqM25BfCLCrB2YqWF6c8zk3CPN3Na21\n8k5s4cO95N/rGN+Po0XFAX/HjKjlpgNpKRDrpxmXxTU8NZfAqeQGJ5oiMBZI9vVB\n+eU7t1L6F5/XWuUCeP4OMrG8oZX822AREba8rknS6DpkWGES0Rx2eNOyYTf6ue75\nI6Ek6rlO+da5wMWr+3BvYMq4JMIwTHzAO+ZqqJPFpzKSiVuAWb2DOX/MDFecVWJE\ntF/R60lONxe4e/00MPCoDdqkLKdwROsk1yGL7z4Zk6jOWFEhIcWy/d2Ya5CpPvS3\nu4wNN4jkYAjra+8TiloRELhV4gpcEk8nkyNwLXOhYm7zQ5sIc5rfXoIrFzALB86W\nG05Nnqg+77zZIaTZpD9qekYlaEt+0OVtt9TTIeTiudQ983l6mfKwZYymrzymH1dL\nVgxBRYo+Z53QOSLiSKELfTBZxEoP1pBw6RiOHXydmJ/39hGgc2YAY/5ADwW2F2yb\nJ7+gxG6bPJ3ikDLYcD4CB5iJQdnTcDsFt3jyHAT6wOCzFAYPbHUqtzHfUM30dZBn\nnJhQF8udPLcXLaj6GW75JacCAwEAAQ==\n-----END PUBLIC KEY-----\n"
     },
diff --git a/test/fixtures/tesla_mock/kaniini@hubzilla.example.org.json b/test/fixtures/tesla_mock/kaniini@hubzilla.example.org.json
index 11c79e11e..c354747cc 100644
--- a/test/fixtures/tesla_mock/kaniini@hubzilla.example.org.json
+++ b/test/fixtures/tesla_mock/kaniini@hubzilla.example.org.json
@@ -1 +1 @@
-{"@context":["https://www.w3.org/ns/activitystreams","https://w3id.org/security/v1","https://hubzilla.example.org/apschema/v1.2"],"type":"Person","id":"https://hubzilla.example.org/channel/kaniini","preferredUsername":"kaniini","name":"kaniini","icon":{"type":"Image","mediaType":"image/jpeg","url":"https://hubzilla.example.org/photo/profile/l/281","height":300,"width":300},"url":{"type":"Link","mediaType":"text/html","href":"https://hubzilla.example.org/channel/kaniini"},"inbox":"https://hubzilla.example.org/inbox/kaniini","outbox":"https://hubzilla.example.org/outbox/kaniini","followers":"https://hubzilla.example.org/followers/kaniini","following":"https://hubzilla.example.org/following/kaniini","endpoints":{"sharedInbox":"https://hubzilla.example.org/inbox"},"publicKey":{"id":"https://hubzilla.example.org/channel/kaniini/public_key_pem","owner":"https://hubzilla.example.org/channel/kaniini","publicKeyPem":"-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvXCDkQPw+1N8B2CUd5s2\nbYvjHt+t7soMNfUiRy0qGbgW46S45k5lCq1KpbFIX3sgGZ4OWjnXVbvjCJi4kl5M\nfm5DBXzpuu05AmjVl8hqk4GejajiE/1Nq0uWHPiOSFWispUjCzzCu65V+IsiE5JU\nvcL6WEf/pYNRq7gYqyT693F7+cO5/rVv9OScx5UOxbIuU1VXYhdHCqAMDJWadC89\nhePrcD3HOQKl06W2tDxHcWk6QjrdsUQGbNOgK/QIN9gSxA+rCFEvH5O0HAhI0aXq\ncOB+vysJUFLeQOAqmAKvKS5V6RqE1GqqT0pDWHack4EmQi0gkgVzo+45xoP6wfDl\nWwG88w21LNxGvGHuN4I8mg6cEoApqKQBSOj086UtfDfSlPC1B+PRD2phE5etucHd\nF/RIWN3SxVzU9BKIiaDm2gwOpvI8QuorQb6HDtZFO5NsSN3PnMnSywPe7kXl/469\nuQRYXrseqyOVIi6WjhvXkyWVKVE5CBz+S8wXHfKph+9YOyUcJeAVMijp9wrjBlMc\noSzOGu79oM7tpMSq/Xo6ePJ/glNOwZR+OKrg92Qp9BGTKDNwGrxuxP/9KwWtGLNf\nOMTtIkxtC3ubhxL3lBxOd7l+Bmum0UJV2f8ogkCgvTpIz05jMoyU8qWl6kkWNQlY\nDropXWaOfy7Lac+G4qlfSgsCAwEAAQ==\n-----END PUBLIC KEY-----\n"},"nomadicLocations":[{"id":"https://hubzilla.example.org/locs/kaniini","type":"nomadicLocation","locationAddress":"acct:kaniini@hubzilla.example.org","locationPrimary":true,"locationDeleted":false}],"signature":{"@context":["https://www.w3.org/ns/activitystreams","https://w3id.org/security/v1"],"type":"RsaSignature2017","nonce":"6b981a2f3bdcffc20252e3b131d4a4569fd2dea9fac543e5196136302f492694","creator":"https://hubzilla.example.org/channel/kaniini/public_key_pem","created":"2018-05-19T08:19:13Z","signatureValue":"ezpT4iCIUzJSeJa/Jsf4EkgbX9enWZG/0eliLXZcvkeCX9mZabaX9LMQRViP2GSlAJBHJu+UqK5LWaoWw9pYkQQHUL+43w2DeBxQicEcPqpT46j6pHuWptfwB8YHTC2/Pb56Y/jseU37j+FW8xVmcGZk4cPqJRLQNojwJlQiFOpBEd4Cel6081W12Pep578+6xBL+h92RJsWznA1gE/NV9dkCqoAoNdiORJg68sVTm0yYxPit2D/DLwXUFeBhC47EZtY3DtAOf7rADGwbquXKug/wtEI47R4p9dJvMWERSVW9O2FmDk8deUjRR3qO1iYGce8O+uMnnBHmuTcToRUHH7mxfMdqjfbcZ9DGBjKtLPSOyVPT9rENeyX8fsksmX0XhfHsNSWkmeDaU5/Au3IY75gDewiGzmzLOpRc6GUnHHro7lMpyMuo3lLZKjNVsFZbx+sXCYwORz5GAMuwIt/iCUdrsQsF5aycqfUAZrFBPguH6DVjbMUqyLvS78sDKiWqgWVhq9VDKse+WuQaJLGBDJNF9APoA6NDMjjIBZfmkGf2mV7ubIYihoOncUjahFqxU5306cNxAcdj2uNcwkgX4BCnBe/L2YsvMHhZrupzDewWWy4fxhktyoZ7VhLSl1I7fMPytjOpb9EIvng4DHGX2t+hKfon2rCGfECPavwiTM="}}
+{"@context":["https://www.w3.org/ns/activitystreams","https://w3id.org/security/v1","https://hubzilla.example.org/apschema/v1.2"],"type":"Person","id":"https://hubzilla.example.org/channel/kaniini","preferredUsername":"kaniini","name":"kaniini","icon":{"type":"Image","mediaType":"image/jpeg","url":"https://hubzilla.example.org/photo/profile/l/281","height":300,"width":300},"url":{"type":"Link","mediaType":"text/html","href":"https://hubzilla.example.org/channel/kaniini"},"inbox":"https://hubzilla.example.org/inbox/kaniini","outbox":"https://hubzilla.example.org/outbox/kaniini","followers":"https://hubzilla.example.org/followers/kaniini","following":"https://hubzilla.example.org/following/kaniini","endpoints":{"sharedInbox":"https://hubzilla.example.org/inbox"},"publicKey":{"id":"https://hubzilla.example.org/channel/kaniini","owner":"https://hubzilla.example.org/channel/kaniini","publicKeyPem":"-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAvXCDkQPw+1N8B2CUd5s2\nbYvjHt+t7soMNfUiRy0qGbgW46S45k5lCq1KpbFIX3sgGZ4OWjnXVbvjCJi4kl5M\nfm5DBXzpuu05AmjVl8hqk4GejajiE/1Nq0uWHPiOSFWispUjCzzCu65V+IsiE5JU\nvcL6WEf/pYNRq7gYqyT693F7+cO5/rVv9OScx5UOxbIuU1VXYhdHCqAMDJWadC89\nhePrcD3HOQKl06W2tDxHcWk6QjrdsUQGbNOgK/QIN9gSxA+rCFEvH5O0HAhI0aXq\ncOB+vysJUFLeQOAqmAKvKS5V6RqE1GqqT0pDWHack4EmQi0gkgVzo+45xoP6wfDl\nWwG88w21LNxGvGHuN4I8mg6cEoApqKQBSOj086UtfDfSlPC1B+PRD2phE5etucHd\nF/RIWN3SxVzU9BKIiaDm2gwOpvI8QuorQb6HDtZFO5NsSN3PnMnSywPe7kXl/469\nuQRYXrseqyOVIi6WjhvXkyWVKVE5CBz+S8wXHfKph+9YOyUcJeAVMijp9wrjBlMc\noSzOGu79oM7tpMSq/Xo6ePJ/glNOwZR+OKrg92Qp9BGTKDNwGrxuxP/9KwWtGLNf\nOMTtIkxtC3ubhxL3lBxOd7l+Bmum0UJV2f8ogkCgvTpIz05jMoyU8qWl6kkWNQlY\nDropXWaOfy7Lac+G4qlfSgsCAwEAAQ==\n-----END PUBLIC KEY-----\n"},"nomadicLocations":[{"id":"https://hubzilla.example.org/locs/kaniini","type":"nomadicLocation","locationAddress":"acct:kaniini@hubzilla.example.org","locationPrimary":true,"locationDeleted":false}],"signature":{"@context":["https://www.w3.org/ns/activitystreams","https://w3id.org/security/v1"],"type":"RsaSignature2017","nonce":"6b981a2f3bdcffc20252e3b131d4a4569fd2dea9fac543e5196136302f492694","creator":"https://hubzilla.example.org/channel","created":"2018-05-19T08:19:13Z","signatureValue":"ezpT4iCIUzJSeJa/Jsf4EkgbX9enWZG/0eliLXZcvkeCX9mZabaX9LMQRViP2GSlAJBHJu+UqK5LWaoWw9pYkQQHUL+43w2DeBxQicEcPqpT46j6pHuWptfwB8YHTC2/Pb56Y/jseU37j+FW8xVmcGZk4cPqJRLQNojwJlQiFOpBEd4Cel6081W12Pep578+6xBL+h92RJsWznA1gE/NV9dkCqoAoNdiORJg68sVTm0yYxPit2D/DLwXUFeBhC47EZtY3DtAOf7rADGwbquXKug/wtEI47R4p9dJvMWERSVW9O2FmDk8deUjRR3qO1iYGce8O+uMnnBHmuTcToRUHH7mxfMdqjfbcZ9DGBjKtLPSOyVPT9rENeyX8fsksmX0XhfHsNSWkmeDaU5/Au3IY75gDewiGzmzLOpRc6GUnHHro7lMpyMuo3lLZKjNVsFZbx+sXCYwORz5GAMuwIt/iCUdrsQsF5aycqfUAZrFBPguH6DVjbMUqyLvS78sDKiWqgWVhq9VDKse+WuQaJLGBDJNF9APoA6NDMjjIBZfmkGf2mV7ubIYihoOncUjahFqxU5306cNxAcdj2uNcwkgX4BCnBe/L2YsvMHhZrupzDewWWy4fxhktyoZ7VhLSl1I7fMPytjOpb9EIvng4DHGX2t+hKfon2rCGfECPavwiTM="}}

From 3e134b07fa4e382f1f4cfdbe90e74f8e73336a4e Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Fri, 15 Mar 2024 18:57:09 -0100
Subject: [PATCH 34/43] fetcher: return final URL after redirects from
 get_object
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Since we reject cross-domain redirects, this doesn’t yet
make a difference, but it’s requried for stricter checking
subsequent commits will introduce.

To make sure (and in case we ever decide to reallow
cross-domain redirects) also use the final location
for containment and reachability checks.
---
 lib/pleroma/object/fetcher.ex        | 27 +++++++++++++++++++--------
 test/pleroma/object/fetcher_test.exs | 27 ++++++++++++++++++++++++---
 2 files changed, 43 insertions(+), 11 deletions(-)

diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex
index f256bbf77..263f9a4fa 100644
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@@ -259,12 +259,12 @@ defmodule Pleroma.Object.Fetcher do
 
     with {:scheme, true} <- {:scheme, String.starts_with?(id, "http")},
          {_, :ok} <- {:local_fetch, Containment.contain_local_fetch(id)},
-         {:ok, body} <- get_object(id),
+         {:ok, final_id, body} <- get_object(id),
          {:ok, data} <- safe_json_decode(body),
-         {_, :ok} <- {:containment, Containment.contain_origin_from_id(id, data)},
-         {_, :ok} <- {:containment, Containment.contain_origin(id, data)} do
-      unless Instances.reachable?(id) do
-        Instances.set_reachable(id)
+         {_, :ok} <- {:containment, Containment.contain_origin_from_id(final_id, data)},
+         {_, :ok} <- {:containment, Containment.contain_origin(final_id, data)} do
+      unless Instances.reachable?(final_id) do
+        Instances.set_reachable(final_id)
       end
 
       {:ok, data}
@@ -305,6 +305,15 @@ defmodule Pleroma.Object.Fetcher do
     {:cross_domain_redirect, final_host != URI.parse(original_url).host}
   end
 
+  if @mix_env == :test do
+    defp get_final_id(nil, initial_url), do: initial_url
+    defp get_final_id("", initial_url), do: initial_url
+  end
+
+  defp get_final_id(final_url, _intial_url) do
+    final_url
+  end
+
   @doc "Do NOT use; only public for use in tests"
   def get_object(id) do
     date = Pleroma.Signature.signed_date()
@@ -325,16 +334,18 @@ defmodule Pleroma.Object.Fetcher do
            {:has_content_type, List.keyfind(headers, "content-type", 0)},
          {:parse_content_type, {:ok, "application", subtype, type_params}} <-
            {:parse_content_type, Plug.Conn.Utils.media_type(content_type)} do
+      final_id = get_final_id(final_url, id)
+
       case {subtype, type_params} do
         {"activity+json", _} ->
-          {:ok, body}
+          {:ok, final_id, body}
 
         {"ld+json", %{"profile" => "https://www.w3.org/ns/activitystreams"}} ->
-          {:ok, body}
+          {:ok, final_id, body}
 
         # pixelfed sometimes (and only sometimes) responds with http instead of https
         {"ld+json", %{"profile" => "http://www.w3.org/ns/activitystreams"}} ->
-          {:ok, body}
+          {:ok, final_id, body}
 
         _ ->
           {:error, {:content_type, content_type}}
diff --git a/test/pleroma/object/fetcher_test.exs b/test/pleroma/object/fetcher_test.exs
index d1f3c070e..a761578f9 100644
--- a/test/pleroma/object/fetcher_test.exs
+++ b/test/pleroma/object/fetcher_test.exs
@@ -693,12 +693,13 @@ defmodule Pleroma.Object.FetcherTest do
         } ->
           %Tesla.Env{
             status: 200,
+            url: "https://mastodon.social/2",
             headers: [{"content-type", "application/activity+json"}],
             body: "{}"
           }
       end)
 
-      assert {:ok, "{}"} = Fetcher.get_object("https://mastodon.social/2")
+      assert {:ok, _, "{}"} = Fetcher.get_object("https://mastodon.social/2")
     end
 
     test "should return ok if the content type is application/ld+json with a profile" do
@@ -709,6 +710,7 @@ defmodule Pleroma.Object.FetcherTest do
         } ->
           %Tesla.Env{
             status: 200,
+            url: "https://mastodon.social/2",
             headers: [
               {"content-type",
                "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\""}
@@ -717,7 +719,7 @@ defmodule Pleroma.Object.FetcherTest do
           }
       end)
 
-      assert {:ok, "{}"} = Fetcher.get_object("https://mastodon.social/2")
+      assert {:ok, _, "{}"} = Fetcher.get_object("https://mastodon.social/2")
 
       Tesla.Mock.mock(fn
         %{
@@ -734,7 +736,7 @@ defmodule Pleroma.Object.FetcherTest do
           }
       end)
 
-      assert {:ok, "{}"} = Fetcher.get_object("https://mastodon.social/2")
+      assert {:ok, _, "{}"} = Fetcher.get_object("https://mastodon.social/2")
     end
 
     test "should not return ok with other content types" do
@@ -745,6 +747,7 @@ defmodule Pleroma.Object.FetcherTest do
         } ->
           %Tesla.Env{
             status: 200,
+            url: "https://mastodon.social/2",
             headers: [{"content-type", "application/json"}],
             body: "{}"
           }
@@ -753,5 +756,23 @@ defmodule Pleroma.Object.FetcherTest do
       assert {:error, {:content_type, "application/json"}} =
                Fetcher.get_object("https://mastodon.social/2")
     end
+
+    test "returns the url after redirects" do
+      Tesla.Mock.mock(fn
+        %{
+          method: :get,
+          url: "https://mastodon.social/5"
+        } ->
+          %Tesla.Env{
+            status: 200,
+            url: "https://mastodon.social/7",
+            headers: [{"content-type", "application/activity+json"}],
+            body: "{}"
+          }
+      end)
+
+      assert {:ok, "https://mastodon.social/7", "{}"} =
+               Fetcher.get_object("https://mastodon.social/5")
+    end
   end
 end

From 9061d148bedb2685c460b098bebae13b05587f84 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 24 Mar 2024 17:32:28 -0100
Subject: [PATCH 35/43] =?UTF-8?q?Ensure=20object=20id=20doesn=E2=80=99t=20?=
 =?UTF-8?q?change=20on=20refetch?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

---
 lib/pleroma/object/fetcher.ex | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex
index 263f9a4fa..e51262de4 100644
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@@ -117,10 +117,12 @@ defmodule Pleroma.Object.Fetcher do
   def refetch_object(%Object{data: %{"id" => id}} = object) do
     with {:local, false} <- {:local, Object.local?(object)},
          {:ok, new_data} <- fetch_and_contain_remote_object_from_id(id),
+         {:id, true} <- {:id, new_data["id"] == id},
          {:ok, object} <- reinject_object(object, new_data) do
       {:ok, object}
     else
       {:local, true} -> {:ok, object}
+      {:id, false} -> {:error, "Object id changed on refetch"}
       e -> {:error, e}
     end
   end

From 48b3a357930b75bf2dcbe74b63a103172c7508e1 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Wed, 20 Mar 2024 19:03:39 -0100
Subject: [PATCH 36/43] Update user reference after fetch

Since we always followed redirects (and until recently allowed fuzzy id
matches), the ap_id of the received object might differ from the iniital
fetch url. This lead to us mistakenly trying to insert a new user with
the same nickname, ap_id, etc as an existing user (which will fail due
to uniqueness constraints) instead of updating the existing one.
---
 lib/pleroma/web/activity_pub/activity_pub.ex | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/pleroma/web/activity_pub/activity_pub.ex b/lib/pleroma/web/activity_pub/activity_pub.ex
index deca69613..1e06bc809 100644
--- a/lib/pleroma/web/activity_pub/activity_pub.ex
+++ b/lib/pleroma/web/activity_pub/activity_pub.ex
@@ -1840,6 +1840,13 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
       with {:ok, data} <- fetch_and_prepare_user_from_ap_id(ap_id, additional) do
         {:ok, _pid} = Task.start(fn -> pinned_fetch_task(data) end)
 
+        user =
+          if data.ap_id != ap_id do
+            User.get_cached_by_ap_id(data.ap_id)
+          else
+            user
+          end
+
         if user do
           user
           |> User.remote_user_changeset(data)

From 8684964c5d03f6c70f73730b3f1ad26784ffb004 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Fri, 15 Mar 2024 23:00:19 -0100
Subject: [PATCH 37/43] Only allow exact id matches
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This protects us from falling for obvious spoofs as from the current
upload exploit (unfortunately we can’t reasonably do anything about
spoofs with exact matches as was possible via emoji and proxy).

Such objects being invalid is supported by the spec, sepcifically
sections 3.1 and 3.2: https://www.w3.org/TR/activitypub/#obj-id

Anonymous objects are not relevant here (they can only exists within
parent objects iiuc) and neither is client-to-server or transient objects
(as those cannot be fetched in the first place).
This leaves us with the requirement for `id` to (a) exist and
(b) be a publicly dereferencable URI from the originating server.
This alone does not yet demand strict equivalence, but the spec then
further explains objects ought to be fetchable _via their ID_.
Meaning an object not retrievable via its ID, is invalid.

This reading is supported by the fact, e.g. GoToSocial (recently) and
Mastodon (for 6+ years) do already implement such strict ID checks,
additionally proving this doesn’t cause federation issues in practice.

However, apart from canonical IDs there can also be additional display
URLs. *omas first redirect those to their canonical location, but *keys
and Mastodon directly serve the AP representation without redirects.

Mastodon and GTS deal with this in two different ways,
but both constitute an effective countermeasure:
 - Mastodon:
   Unless it already is a known AP id, two fetches occur.
   The first fetch just reads the `id` property and then refetches from
   the id. The last fetch requires the returned id to exactly match the
   URL the content was fetched from. (This can be optimised by skipping
   the second fetch if it already matches)
   https://github.com/mastodon/mastodon/blob/05eda8d19330a9c27c0cf07de19a87edff269057/app/helpers/jsonld_helper.rb#L168
   https://github.com/mastodon/mastodon/commit/63f097979990bf5ba9db848b8a253056bad781af

 - GTS:
   Only does a single fetch and then checks if _either_ the id
   _or_ url property (which can be an object) match the original fetch
   URL. This relies on implementations always including their display URL
   as "url" if differing from the id. For actors this is true for all
   investigated implementations, for posts only Mastodon includes an
   "url", but it is also the only one with a differing display URL.
   https://github.com/superseriousbusiness/gotosocial/commit/2bafd7daf542d985ee76d9079a30a602cb7be827#diff-943bbb02c8ac74ac5dc5d20807e561dcdfaebdc3b62b10730f643a20ac23c24fR222

Albeit Mastodon’s refetch offers higher compatibility with theoretical
implmentations using either multiple different display URL or not
denoting any of them as "url" at all, for now we chose to adopt a
GTS-like refetch-free approach to avoid additional implementation
concerns wrt to whether redirects should be allowed when fetching a
canonical AP id and potential for accidentally loosening some checks
(e.g. cross-domain refetches) for one of the fetches.
This may be reconsidered in the future.
---
 lib/pleroma/object/containment.ex             | 35 ++++++++++++
 lib/pleroma/object/fetcher.ex                 |  5 +-
 .../https__info.pleroma.site_activity3.json   |  2 +-
 .../tesla_mock/relay@mastdon.example.org.json |  6 +-
 .../collections/collections_fetcher_test.exs  |  7 ++-
 test/pleroma/object/containment_test.exs      | 50 +++++++++++++++++
 test/pleroma/object/fetcher_test.exs          | 55 ++++++++++++++++++-
 .../web/activity_pub/activity_pub_test.exs    |  8 +--
 test/support/http_request_mock.ex             | 18 ++++++
 9 files changed, 174 insertions(+), 12 deletions(-)

diff --git a/lib/pleroma/object/containment.ex b/lib/pleroma/object/containment.ex
index a312f69e8..37bc20e4d 100644
--- a/lib/pleroma/object/containment.ex
+++ b/lib/pleroma/object/containment.ex
@@ -11,6 +11,9 @@ defmodule Pleroma.Object.Containment do
   Object containment is an important step in validating remote objects to prevent
   spoofing, therefore removal of object containment functions is NOT recommended.
   """
+
+  alias Pleroma.Web.ActivityPub.Transmogrifier
+
   def get_actor(%{"actor" => actor}) when is_binary(actor) do
     actor
   end
@@ -47,6 +50,18 @@ defmodule Pleroma.Object.Containment do
   defp compare_uris(%URI{host: host} = _id_uri, %URI{host: host} = _other_uri), do: :ok
   defp compare_uris(_id_uri, _other_uri), do: :error
 
+  defp compare_uris_exact(uri, uri), do: :ok
+
+  defp compare_uris_exact(%URI{} = id, %URI{} = other),
+    do: compare_uris_exact(URI.to_string(id), URI.to_string(other))
+
+  defp compare_uris_exact(id_uri, other_uri)
+       when is_binary(id_uri) and is_binary(other_uri) do
+    norm_id = String.replace_suffix(id_uri, "/", "")
+    norm_other = String.replace_suffix(other_uri, "/", "")
+    if norm_id == norm_other, do: :ok, else: :error
+  end
+
   @doc """
   Checks whether an URL to fetch from is from the local server.
 
@@ -77,6 +92,26 @@ defmodule Pleroma.Object.Containment do
 
   def contain_origin(_id, _data), do: :ok
 
+  @doc """
+  Check whether the fetch URL (after redirects) exactly (sans tralining slash) matches either
+  the canonical ActivityPub id or the objects url field (for display URLs from *key and Mastodon)
+
+  Since this is meant to be used for fetches, anonymous or transient objects are not accepted here.
+  """
+  def contain_id_to_fetch(url, %{"id" => id} = data) when is_binary(id) do
+    with {:id, :error} <- {:id, compare_uris_exact(id, url)},
+         # "url" can be a "Link" object and this is checked before full normalisation
+         display_url <- Transmogrifier.fix_url(data)["url"],
+         true <- display_url != nil do
+      compare_uris_exact(display_url, url)
+    else
+      {:id, :ok} -> :ok
+      _ -> :error
+    end
+  end
+
+  def contain_id_to_fetch(_url, _data), do: :error
+
   @doc """
   Check whether the object id is from the same host as another id
   """
diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex
index e51262de4..618fb278e 100644
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@@ -263,7 +263,7 @@ defmodule Pleroma.Object.Fetcher do
          {_, :ok} <- {:local_fetch, Containment.contain_local_fetch(id)},
          {:ok, final_id, body} <- get_object(id),
          {:ok, data} <- safe_json_decode(body),
-         {_, :ok} <- {:containment, Containment.contain_origin_from_id(final_id, data)},
+         {_, :ok} <- {:strict_id, Containment.contain_id_to_fetch(final_id, data)},
          {_, :ok} <- {:containment, Containment.contain_origin(final_id, data)} do
       unless Instances.reachable?(final_id) do
         Instances.set_reachable(final_id)
@@ -271,6 +271,9 @@ defmodule Pleroma.Object.Fetcher do
 
       {:ok, data}
     else
+      {:strict_id, _} ->
+        {:error, "Object's ActivityPub id/url does not match final fetch URL"}
+
       {:scheme, _} ->
         {:error, "Unsupported URI scheme"}
 
diff --git a/test/fixtures/tesla_mock/https__info.pleroma.site_activity3.json b/test/fixtures/tesla_mock/https__info.pleroma.site_activity3.json
index 1df73f2c5..dbf74dfe1 100644
--- a/test/fixtures/tesla_mock/https__info.pleroma.site_activity3.json
+++ b/test/fixtures/tesla_mock/https__info.pleroma.site_activity3.json
@@ -3,7 +3,7 @@
         "attributedTo": "http://mastodon.example.org/users/admin",
         "attachment": [],
         "content": "<p>this post was not actually written by Haelwenn</p>",
-        "id": "https://info.pleroma.site/activity2.json",
+        "id": "https://info.pleroma.site/activity3.json",
         "published": "2018-09-01T22:15:00Z",
         "tag": [],
         "to": [
diff --git a/test/fixtures/tesla_mock/relay@mastdon.example.org.json b/test/fixtures/tesla_mock/relay@mastdon.example.org.json
index c1fab7d3b..21dd405c8 100644
--- a/test/fixtures/tesla_mock/relay@mastdon.example.org.json
+++ b/test/fixtures/tesla_mock/relay@mastdon.example.org.json
@@ -11,7 +11,7 @@
     "toot": "http://joinmastodon.org/ns#",
     "Emoji": "toot:Emoji"
   }],
-  "id": "http://mastodon.example.org/users/admin",
+  "id": "http://mastodon.example.org/users/relay",
   "type": "Application",
   "invisible": true,
   "following": "http://mastodon.example.org/users/admin/following",
@@ -24,8 +24,8 @@
   "url": "http://mastodon.example.org/@admin",
   "manuallyApprovesFollowers": false,
   "publicKey": {
-    "id": "http://mastodon.example.org/users/admin#main-key",
-    "owner": "http://mastodon.example.org/users/admin",
+    "id": "http://mastodon.example.org/users/relay#main-key",
+    "owner": "http://mastodon.example.org/users/relay",
     "publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtc4Tir+3ADhSNF6VKrtW\nOU32T01w7V0yshmQei38YyiVwVvFu8XOP6ACchkdxbJ+C9mZud8qWaRJKVbFTMUG\nNX4+6Q+FobyuKrwN7CEwhDALZtaN2IPbaPd6uG1B7QhWorrY+yFa8f2TBM3BxnUy\nI4T+bMIZIEYG7KtljCBoQXuTQmGtuffO0UwJksidg2ffCF5Q+K//JfQagJ3UzrR+\nZXbKMJdAw4bCVJYs4Z5EhHYBwQWiXCyMGTd7BGlmMkY6Av7ZqHKC/owp3/0EWDNz\nNqF09Wcpr3y3e8nA10X40MJqp/wR+1xtxp+YGbq/Cj5hZGBG7etFOmIpVBrDOhry\nBwIDAQAB\n-----END PUBLIC KEY-----\n"
   },
   "attachment": [{
diff --git a/test/pleroma/collections/collections_fetcher_test.exs b/test/pleroma/collections/collections_fetcher_test.exs
index 7a582a3d7..ff1aa84db 100644
--- a/test/pleroma/collections/collections_fetcher_test.exs
+++ b/test/pleroma/collections/collections_fetcher_test.exs
@@ -12,11 +12,14 @@ defmodule Akkoma.Collections.FetcherTest do
   end
 
   test "it should extract items from an embedded array in a Collection" do
+    ap_id = "https://example.com/collection/ordered_array"
+
     unordered_collection =
       "test/fixtures/collections/unordered_array.json"
       |> File.read!()
-
-    ap_id = "https://example.com/collection/ordered_array"
+      |> Jason.decode!()
+      |> Map.put("id", ap_id)
+      |> Jason.encode!(pretty: true)
 
     Tesla.Mock.mock(fn
       %{
diff --git a/test/pleroma/object/containment_test.exs b/test/pleroma/object/containment_test.exs
index ad1660069..f8f40a3ac 100644
--- a/test/pleroma/object/containment_test.exs
+++ b/test/pleroma/object/containment_test.exs
@@ -105,6 +105,56 @@ defmodule Pleroma.Object.ContainmentTest do
         )
     end
 
+    test "contain_id_to_fetch() refuses alternate IDs within the same origin domain" do
+      data = %{
+        "id" => "http://example.com/~alyssa/activities/1234.json",
+        "url" => "http://example.com/@alyssa/status/1234"
+      }
+
+      :error =
+        Containment.contain_id_to_fetch(
+          "http://example.com/~alyssa/activities/1234",
+          data
+        )
+    end
+
+    test "contain_id_to_fetch() allows matching IDs" do
+      data = %{
+        "id" => "http://example.com/~alyssa/activities/1234.json/"
+      }
+
+      :ok =
+        Containment.contain_id_to_fetch(
+          "http://example.com/~alyssa/activities/1234.json/",
+          data
+        )
+
+      :ok =
+        Containment.contain_id_to_fetch(
+          "http://example.com/~alyssa/activities/1234.json",
+          data
+        )
+    end
+
+    test "contain_id_to_fetch() allows display URLs" do
+      data = %{
+        "id" => "http://example.com/~alyssa/activities/1234.json",
+        "url" => "http://example.com/@alyssa/status/1234"
+      }
+
+      :ok =
+        Containment.contain_id_to_fetch(
+          "http://example.com/@alyssa/status/1234",
+          data
+        )
+
+      :ok =
+        Containment.contain_id_to_fetch(
+          "http://example.com/@alyssa/status/1234/",
+          data
+        )
+    end
+
     test "users cannot be collided through fake direction spoofing attempts" do
       _user =
         insert(:user, %{
diff --git a/test/pleroma/object/fetcher_test.exs b/test/pleroma/object/fetcher_test.exs
index a761578f9..c59d77f0f 100644
--- a/test/pleroma/object/fetcher_test.exs
+++ b/test/pleroma/object/fetcher_test.exs
@@ -57,6 +57,46 @@ defmodule Pleroma.Object.FetcherTest do
           body: spoofed_object_with_ids("https://patch.cx/objects/spoof_content_type")
         }
 
+      # Spoof: mismatching ids
+      # Variant 1: Non-exisitng fake id
+      %{
+        method: :get,
+        url:
+          "https://patch.cx/media/03ca3c8b4ac3ddd08bf0f84be7885f2f88de0f709112131a22d83650819e36c2.json"
+      } ->
+        %Tesla.Env{
+          status: 200,
+          url:
+            "https://patch.cx/media/03ca3c8b4ac3ddd08bf0f84be7885f2f88de0f709112131a22d83650819e36c2.json",
+          headers: [{"content-type", "application/activity+json"}],
+          body: spoofed_object_with_ids()
+        }
+
+      %{method: :get, url: "https://patch.cx/objects/spoof"} ->
+        %Tesla.Env{
+          status: 404,
+          url: "https://patch.cx/objects/spoof",
+          headers: [],
+          body: "Not found"
+        }
+
+      # Varaint 2: two-stage payload
+      %{method: :get, url: "https://patch.cx/media/spoof_stage1.json"} ->
+        %Tesla.Env{
+          status: 200,
+          url: "https://patch.cx/media/spoof_stage1.json",
+          headers: [{"content-type", "application/activity+json"}],
+          body: spoofed_object_with_ids("https://patch.cx/media/spoof_stage2.json")
+        }
+
+      %{method: :get, url: "https://patch.cx/media/spoof_stage2.json"} ->
+        %Tesla.Env{
+          status: 200,
+          url: "https://patch.cx/media/spoof_stage2.json",
+          headers: [{"content-type", "application/activity+json"}],
+          body: spoofed_object_with_ids("https://patch.cx/media/unpredictable.json")
+        }
+
       # Spoof: cross-domain redirect with original domain id
       %{method: :get, url: "https://patch.cx/objects/spoof_media_redirect1"} ->
         %Tesla.Env{
@@ -79,7 +119,7 @@ defmodule Pleroma.Object.FetcherTest do
       %{method: :get, url: "https://patch.cx/objects/spoof_redirect"} ->
         %Tesla.Env{
           status: 200,
-          url: "https://patch.cx/objects/spoof",
+          url: "https://patch.cx/objects/spoof_redirect",
           headers: [{"content-type", "application/activity+json"}],
           body: spoofed_object_with_ids("https://patch.cx/objects/spoof_redirect")
         }
@@ -110,6 +150,7 @@ defmodule Pleroma.Object.FetcherTest do
         %{method: :get, url: "https://social.sakamoto.gq/notice/9wTkLEnuq47B25EehM"} ->
           %Tesla.Env{
             status: 200,
+            url: "https://social.sakamoto.gq/objects/f20f2497-66d9-4a52-a2e1-1be2a39c32c1",
             body: File.read!("test/fixtures/fetch_mocks/9wTkLEnuq47B25EehM.json"),
             headers: HttpRequestMock.activitypub_object_headers()
           }
@@ -208,6 +249,18 @@ defmodule Pleroma.Object.FetcherTest do
                )
     end
 
+    test "it does not fetch a spoofed object with id different from URL" do
+      assert {:error, "Object's ActivityPub id/url does not match final fetch URL"} =
+               Fetcher.fetch_and_contain_remote_object_from_id(
+                 "https://patch.cx/media/03ca3c8b4ac3ddd08bf0f84be7885f2f88de0f709112131a22d83650819e36c2.json"
+               )
+
+      assert {:error, "Object's ActivityPub id/url does not match final fetch URL"} =
+               Fetcher.fetch_and_contain_remote_object_from_id(
+                 "https://patch.cx/media/spoof_stage1.json"
+               )
+    end
+
     test "it does not fetch an object via cross-domain redirects (initial id)" do
       assert {:error, {:cross_domain_redirect, true}} =
                Fetcher.fetch_and_contain_remote_object_from_id(
diff --git a/test/pleroma/web/activity_pub/activity_pub_test.exs b/test/pleroma/web/activity_pub/activity_pub_test.exs
index c6543ec83..5ad6d4716 100644
--- a/test/pleroma/web/activity_pub/activity_pub_test.exs
+++ b/test/pleroma/web/activity_pub/activity_pub_test.exs
@@ -312,7 +312,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
   end
 
   test "fetches user featured collection using the first property" do
-    featured_url = "https://friendica.example.com/raha/collections/featured"
+    featured_url = "https://friendica.example.com/featured/raha"
     first_url = "https://friendica.example.com/featured/raha?page=1"
 
     featured_data =
@@ -350,7 +350,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
   end
 
   test "fetches user featured when it has string IDs" do
-    featured_url = "https://example.com/alisaie/collections/featured"
+    featured_url = "https://example.com/users/alisaie/collections/featured"
     dead_url = "https://example.com/users/alisaie/statuses/108311386746229284"
 
     featured_data =
@@ -1720,7 +1720,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
             json(
               %{
                 "@context" => "https://www.w3.org/ns/activitystreams",
-                "id" => "http://remote.org/users/masto_hidden_counters/followers"
+                "id" => "http://remote.org/users/masto_hidden_counters/following"
               },
               headers: HttpRequestMock.activitypub_object_headers()
             )
@@ -1732,7 +1732,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do
             json(
               %{
                 "@context" => "https://www.w3.org/ns/activitystreams",
-                "id" => "http://remote.org/users/masto_hidden_counters/following"
+                "id" => "http://remote.org/users/masto_hidden_counters/followers"
               },
               headers: HttpRequestMock.activitypub_object_headers()
             )
diff --git a/test/support/http_request_mock.ex b/test/support/http_request_mock.ex
index e831f43f7..cc0e22af1 100644
--- a/test/support/http_request_mock.ex
+++ b/test/support/http_request_mock.ex
@@ -572,6 +572,7 @@ defmodule HttpRequestMock do
      }}
   end
 
+  # Mastodon status via display URL
   def get(
         "http://mastodon.example.org/@admin/99541947525187367",
         _,
@@ -581,6 +582,23 @@ defmodule HttpRequestMock do
     {:ok,
      %Tesla.Env{
        status: 200,
+       url: "http://mastodon.example.org/@admin/99541947525187367",
+       body: File.read!("test/fixtures/mastodon-note-object.json"),
+       headers: activitypub_object_headers()
+     }}
+  end
+
+  # same status via its canonical ActivityPub id
+  def get(
+        "http://mastodon.example.org/users/admin/statuses/99541947525187367",
+        _,
+        _,
+        _
+      ) do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       url: "http://mastodon.example.org/users/admin/statuses/99541947525187367",
        body: File.read!("test/fixtures/mastodon-note-object.json"),
        headers: activitypub_object_headers()
      }}

From 61ec592d668956aeb075585391414284016ecba9 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Tue, 26 Mar 2024 15:11:06 -0100
Subject: [PATCH 38/43] Drop obsolete pixelfed workaround

This pixelfed issue was fixed in 2022-12 in
https://github.com/pixelfed/pixelfed/pull/3932

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
---
 lib/pleroma/object/fetcher.ex        |  4 ----
 test/pleroma/object/fetcher_test.exs | 17 -----------------
 2 files changed, 21 deletions(-)

diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex
index 618fb278e..6609b8c1a 100644
--- a/lib/pleroma/object/fetcher.ex
+++ b/lib/pleroma/object/fetcher.ex
@@ -348,10 +348,6 @@ defmodule Pleroma.Object.Fetcher do
         {"ld+json", %{"profile" => "https://www.w3.org/ns/activitystreams"}} ->
           {:ok, final_id, body}
 
-        # pixelfed sometimes (and only sometimes) responds with http instead of https
-        {"ld+json", %{"profile" => "http://www.w3.org/ns/activitystreams"}} ->
-          {:ok, final_id, body}
-
         _ ->
           {:error, {:content_type, content_type}}
       end
diff --git a/test/pleroma/object/fetcher_test.exs b/test/pleroma/object/fetcher_test.exs
index c59d77f0f..4c4831af3 100644
--- a/test/pleroma/object/fetcher_test.exs
+++ b/test/pleroma/object/fetcher_test.exs
@@ -773,23 +773,6 @@ defmodule Pleroma.Object.FetcherTest do
       end)
 
       assert {:ok, _, "{}"} = Fetcher.get_object("https://mastodon.social/2")
-
-      Tesla.Mock.mock(fn
-        %{
-          method: :get,
-          url: "https://mastodon.social/2"
-        } ->
-          %Tesla.Env{
-            status: 200,
-            headers: [
-              {"content-type",
-               "application/ld+json; profile=\"http://www.w3.org/ns/activitystreams\""}
-            ],
-            body: "{}"
-          }
-      end)
-
-      assert {:ok, _, "{}"} = Fetcher.get_object("https://mastodon.social/2")
     end
 
     test "should not return ok with other content types" do

From 31f90bbb52db472f45d38aa90fc1bae43b65d7ff Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Tue, 26 Mar 2024 15:44:44 -0100
Subject: [PATCH 39/43] Register APNG MIME type
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The newest git HEAD of MIME already knows about APNG, but this
hasn’t been released yet. Without this, APNG attachments from
remote posts won’t display as images in frontends.

Fixes: akkoma#657
---
 config/config.exs                               |  5 ++++-
 .../attachment_validator_test.exs               | 17 +++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/config/config.exs b/config/config.exs
index 85a84208c..e0a5eccb1 100644
--- a/config/config.exs
+++ b/config/config.exs
@@ -167,7 +167,10 @@ config :mime, :types, %{
   "application/xrd+xml" => ["xrd+xml"],
   "application/jrd+json" => ["jrd+json"],
   "application/activity+json" => ["activity+json"],
-  "application/ld+json" => ["activity+json"]
+  "application/ld+json" => ["activity+json"],
+  # Can be removed when bumping MIME past 2.0.5
+  # see https://akkoma.dev/AkkomaGang/akkoma/issues/657
+  "image/apng" => ["apng"]
 }
 
 config :mime, :extensions, %{
diff --git a/test/pleroma/web/activity_pub/object_validators/attachment_validator_test.exs b/test/pleroma/web/activity_pub/object_validators/attachment_validator_test.exs
index 9150b8d41..f8dec09d3 100644
--- a/test/pleroma/web/activity_pub/object_validators/attachment_validator_test.exs
+++ b/test/pleroma/web/activity_pub/object_validators/attachment_validator_test.exs
@@ -11,6 +11,23 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.AttachmentValidatorTest do
   import Pleroma.Factory
 
   describe "attachments" do
+    test "works with apng" do
+      attachment =
+        %{
+          "mediaType" => "image/apng",
+          "name" => "",
+          "type" => "Document",
+          "url" =>
+            "https://media.misskeyusercontent.com/io/2859c26e-cd43-4550-848b-b6243bc3fe28.apng"
+        }
+
+      assert {:ok, attachment} =
+               AttachmentValidator.cast_and_validate(attachment)
+               |> Ecto.Changeset.apply_action(:insert)
+
+      assert attachment.mediaType == "image/apng"
+    end
+
     test "works with honkerific attachments" do
       attachment = %{
         "mediaType" => "",

From d4411012007333821e254a7487d75bc295d78ed0 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 17 Mar 2024 15:29:23 -0100
Subject: [PATCH 40/43] Add mix task to detect uploaded spoof payloads

---
 .../docs/administration/CLI_tasks/security.md |  32 +++
 lib/mix/tasks/pleroma/security.ex             | 209 ++++++++++++++++++
 2 files changed, 241 insertions(+)
 create mode 100644 docs/docs/administration/CLI_tasks/security.md
 create mode 100644 lib/mix/tasks/pleroma/security.ex

diff --git a/docs/docs/administration/CLI_tasks/security.md b/docs/docs/administration/CLI_tasks/security.md
new file mode 100644
index 000000000..99b84264c
--- /dev/null
+++ b/docs/docs/administration/CLI_tasks/security.md
@@ -0,0 +1,32 @@
+# Security-related tasks
+
+{! administration/CLI_tasks/general_cli_task_info.include !}
+
+!!! danger
+    Many of these tasks were written in response to a patched exploit.
+    It is recommended to run those very soon after installing its respective security update.
+    Over time with db migrations they might become less accurate or be removed altogether.
+    If you never ran an affected version, there’s no point in running them.
+
+## Spoofed AcitivityPub objects exploit (2024-03, fixed in 3.11.1)
+
+### Search for uploaded spoofing payloads
+
+Scans local uploads for spoofing payloads.
+If the instance is not using the local uploader it was not affected.
+Attachments wil be scanned anyway in case local uploader was used in the past.
+
+!!! note
+    This cannot reliably detect payloads attached to deleted posts.
+
+=== "OTP"
+
+    ```sh
+    ./bin/pleroma_ctl security spoof-uploaded
+    ```
+
+=== "From Source"
+
+    ```sh
+    mix pleroma.security spoof-uploaded
+    ```
diff --git a/lib/mix/tasks/pleroma/security.ex b/lib/mix/tasks/pleroma/security.ex
new file mode 100644
index 000000000..354f227bd
--- /dev/null
+++ b/lib/mix/tasks/pleroma/security.ex
@@ -0,0 +1,209 @@
+# Akkoma: Magically expressive social media
+# Copyright © 2024 Akkoma Authors <https://akkoma.dev/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Mix.Tasks.Pleroma.Security do
+  use Mix.Task
+  import Mix.Pleroma
+
+  alias Pleroma.Config
+
+  require Logger
+
+  @shortdoc """
+  Security-related tasks, like e.g. checking for signs past exploits were abused.
+  """
+
+  # Constants etc
+  defp local_id_prefix(), do: Pleroma.Web.Endpoint.url() <> "/"
+
+  defp local_id_pattern(), do: local_id_prefix() <> "%"
+
+  @activity_exts ["activity+json", "activity%2Bjson"]
+
+  defp activity_ext_url_patterns() do
+    for e <- @activity_exts do
+      for suf <- ["", "?%"] do
+        # Escape literal % for use in SQL patterns
+        ee = String.replace(e, "%", "\\%")
+        "%.#{ee}#{suf}"
+      end
+    end
+    |> List.flatten()
+  end
+
+  # Search for malicious uploads exploiting the lack of Content-Type sanitisation from before 2024-03
+  def run(["spoof-uploaded"]) do
+    Logger.put_process_level(self(), :notice)
+    start_pleroma()
+
+    IO.puts("""
+    +------------------------+
+    |  SPOOF SEARCH UPLOADS  |
+    +------------------------+
+    Checking if any uploads are using privileged types.
+    NOTE if attachment deletion is enabled, payloads used
+         in the past may no longer exist.
+    """)
+
+    do_spoof_uploaded()
+  end
+
+  # +-----------------------------+
+  # | S P O O F - U P L O A D E D |
+  # +-----------------------------+
+  defp do_spoof_uploaded() do
+    files =
+      case Config.get!([Pleroma.Upload, :uploader]) do
+        Pleroma.Uploaders.Local ->
+          uploads_search_spoofs_local_dir(Config.get!([Pleroma.Uploaders.Local, :uploads]))
+
+        _ ->
+          IO.puts("""
+          NOTE:
+            Not using local uploader; thus not affected by this exploit.
+            It's impossible to check for files, but in case local uploader was used before
+            or to check if anyone futilely attempted a spoof, notes will still be scanned.
+          """)
+
+          []
+      end
+
+    emoji = uploads_search_spoofs_local_dir(Config.get!([:instance, :static_dir]))
+
+    post_attachs = uploads_search_spoofs_notes()
+
+    not_orphaned_urls =
+      post_attachs
+      |> Enum.map(fn {_u, _a, url} -> url end)
+      |> MapSet.new()
+
+    orphaned_attachs = upload_search_orphaned_attachments(not_orphaned_urls)
+
+    IO.puts("\nSearch concluded; here are the results:")
+    pretty_print_list_with_title(emoji, "Emoji")
+    pretty_print_list_with_title(files, "Uploaded Files")
+    pretty_print_list_with_title(post_attachs, "(Not Deleted) Post Attachments")
+    pretty_print_list_with_title(orphaned_attachs, "Orphaned Uploads")
+
+    IO.puts("""
+    In total found
+      #{length(emoji)} emoji
+      #{length(files)} uploads
+      #{length(post_attachs)} not deleted posts
+      #{length(orphaned_attachs)} orphaned attachments
+    """)
+  end
+
+  defp uploads_search_spoofs_local_dir(dir) do
+    local_dir = String.replace_suffix(dir, "/", "")
+
+    IO.puts("Searching for suspicious files in #{local_dir}...")
+
+    glob_ext = "{" <> Enum.join(@activity_exts, ",") <> "}"
+
+    Path.wildcard(local_dir <> "/**/*." <> glob_ext, match_dot: true)
+    |> Enum.map(fn path ->
+      String.replace_prefix(path, local_dir <> "/", "")
+    end)
+    |> Enum.sort()
+  end
+
+  defp uploads_search_spoofs_notes() do
+    IO.puts("Now querying DB for posts with spoofing attachments. This might take a while...")
+
+    patterns = [local_id_pattern() | activity_ext_url_patterns()]
+
+    # if jsonb_array_elemsts in FROM can be used with normal Ecto functions, idk how
+    """
+    SELECT DISTINCT a.data->>'actor', a.id, url->>'href'
+    FROM public.objects AS o JOIN public.activities AS a
+         ON o.data->>'id' = a.data->>'object',
+       jsonb_array_elements(o.data->'attachment') AS attachs,
+       jsonb_array_elements(attachs->'url') AS url
+    WHERE o.data->>'type' = 'Note' AND
+          o.data->>'id' LIKE $1::text AND (
+            url->>'href' LIKE $2::text OR
+            url->>'href' LIKE $3::text OR
+            url->>'href' LIKE $4::text OR
+            url->>'href' LIKE $5::text
+          )
+    ORDER BY a.data->>'actor', a.id, url->>'href';
+    """
+    |> Pleroma.Repo.query!(patterns, timeout: :infinity)
+    |> map_raw_id_apid_tuple()
+  end
+
+  defp upload_search_orphaned_attachments(not_orphaned_urls) do
+    IO.puts("""
+    Now querying DB for orphaned spoofing attachment (i.e. their post was deleted,
+    but if :cleanup_attachments was not enabled traces remain in the database)
+    This might take a bit...
+    """)
+
+    patterns = activity_ext_url_patterns()
+
+    """
+    SELECT DISTINCT attach.id, url->>'href'
+    FROM public.objects AS attach,
+         jsonb_array_elements(attach.data->'url') AS url
+    WHERE (attach.data->>'type' = 'Image' OR
+           attach.data->>'type' = 'Document')
+          AND (
+            url->>'href' LIKE $1::text OR
+            url->>'href' LIKE $2::text OR
+            url->>'href' LIKE $3::text OR
+            url->>'href' LIKE $4::text
+          )
+    ORDER BY attach.id, url->>'href';
+    """
+    |> Pleroma.Repo.query!(patterns, timeout: :infinity)
+    |> then(fn res -> Enum.map(res.rows, fn [id, url] -> {id, url} end) end)
+    |> Enum.filter(fn {_, url} -> !(url in not_orphaned_urls) end)
+  end
+
+  # +-----------------------------------+
+  # | module-specific utility functions |
+  # +-----------------------------------+
+  defp pretty_print_list_with_title(list, title) do
+    title_len = String.length(title)
+    title_underline = String.duplicate("=", title_len)
+    IO.puts(title)
+    IO.puts(title_underline)
+    pretty_print_list(list)
+  end
+
+  defp pretty_print_list([]), do: IO.puts("")
+
+  defp pretty_print_list([{a, o} | rest])
+       when (is_binary(a) or is_number(a)) and is_binary(o) do
+    IO.puts("  {#{a}, #{o}}")
+    pretty_print_list(rest)
+  end
+
+  defp pretty_print_list([{u, a, o} | rest])
+       when is_binary(a) and is_binary(u) and is_binary(o) do
+    IO.puts("  {#{u}, #{a}, #{o}}")
+    pretty_print_list(rest)
+  end
+
+  defp pretty_print_list([e | rest]) when is_binary(e) do
+    IO.puts("  #{e}")
+    pretty_print_list(rest)
+  end
+
+  defp pretty_print_list([e | rest]), do: pretty_print_list([inspect(e) | rest])
+
+  defp map_raw_id_apid_tuple(res) do
+    user_prefix = local_id_prefix() <> "users/"
+
+    Enum.map(res.rows, fn
+      [uid, aid, oid] ->
+        {
+          String.replace_prefix(uid, user_prefix, ""),
+          FlakeId.to_string(aid),
+          oid
+        }
+    end)
+  end
+end

From 0648d9ebaa5714647a420881361668a7e3a1c7c8 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Sun, 17 Mar 2024 20:07:16 -0100
Subject: [PATCH 41/43] Add mix tasks to detect spoofed posts and users

At least as far as we can
---
 .../docs/administration/CLI_tasks/security.md |  24 ++++
 lib/mix/tasks/pleroma/security.ex             | 121 ++++++++++++++++++
 2 files changed, 145 insertions(+)

diff --git a/docs/docs/administration/CLI_tasks/security.md b/docs/docs/administration/CLI_tasks/security.md
index 99b84264c..a0208c4e5 100644
--- a/docs/docs/administration/CLI_tasks/security.md
+++ b/docs/docs/administration/CLI_tasks/security.md
@@ -30,3 +30,27 @@ Attachments wil be scanned anyway in case local uploader was used in the past.
     ```sh
     mix pleroma.security spoof-uploaded
     ```
+
+### Search for counterfeit posts in database
+
+Scans all notes in the database for signs of being spoofed.
+
+!!! note
+    Spoofs targeting local accounts can be detected rather reliably
+    (with some restrictions documented in the task’s logs).
+    Counterfeit posts from remote users cannot. A best-effort attempt is made, but
+    a thorough attacker can avoid this and it may yield a small amount of false positives.
+
+    Should you find counterfeit posts of local users, let other admins know so they can delete the too.
+
+=== "OTP"
+
+    ```sh
+    ./bin/pleroma_ctl security spoof-inserted
+    ```
+
+=== "From Source"
+
+    ```sh
+    mix pleroma.security spoof-inserted
+    ```
diff --git a/lib/mix/tasks/pleroma/security.ex b/lib/mix/tasks/pleroma/security.ex
index 354f227bd..dc1f667d7 100644
--- a/lib/mix/tasks/pleroma/security.ex
+++ b/lib/mix/tasks/pleroma/security.ex
@@ -4,6 +4,7 @@
 
 defmodule Mix.Tasks.Pleroma.Security do
   use Mix.Task
+  import Ecto.Query
   import Mix.Pleroma
 
   alias Pleroma.Config
@@ -49,6 +50,23 @@ defmodule Mix.Tasks.Pleroma.Security do
     do_spoof_uploaded()
   end
 
+  # Fuzzy search for potentially counterfeit activities in the database resulting from the same exploit
+  def run(["spoof-inserted"]) do
+    Logger.put_process_level(self(), :notice)
+    start_pleroma()
+
+    IO.puts("""
+    +----------------------+
+    |  SPOOF SEARCH NOTES  |
+    +----------------------+
+    Starting fuzzy search for counterfeit activities.
+    NOTE this can not guarantee detecting all counterfeits
+         and may yield a small percentage of false positives.
+    """)
+
+    do_spoof_inserted()
+  end
+
   # +-----------------------------+
   # | S P O O F - U P L O A D E D |
   # +-----------------------------+
@@ -162,6 +180,109 @@ defmodule Mix.Tasks.Pleroma.Security do
     |> Enum.filter(fn {_, url} -> !(url in not_orphaned_urls) end)
   end
 
+  # +-----------------------------+
+  # | S P O O F - I N S E R T E D |
+  # +-----------------------------+
+  defp do_spoof_inserted() do
+    IO.puts("""
+    Searching for local posts whose Create activity has no ActivityPub id...
+      This is a pretty good indicator, but only for spoofs of local actors
+      and only if the spoofing happened after around late 2021.
+    """)
+
+    idless_create =
+      search_local_notes_without_create_id()
+      |> Enum.sort()
+
+    IO.puts("Done.\n")
+
+    IO.puts("""
+    Now trying to weed out other poorly hidden spoofs.
+    This can't detect all and may have some false positives.
+    """)
+
+    likely_spoofed_posts_set = MapSet.new(idless_create)
+
+    sus_pattern_posts =
+      search_sus_notes_by_id_patterns()
+      |> Enum.filter(fn r -> !(r in likely_spoofed_posts_set) end)
+
+    IO.puts("Done.\n")
+
+    IO.puts("""
+    Finally, searching for spoofed, local user accounts.
+    (It's impossible to detect spoofed remote users)
+    """)
+
+    spoofed_users = search_bogus_local_users()
+
+    pretty_print_list_with_title(sus_pattern_posts, "Maybe Spoofed Posts")
+    pretty_print_list_with_title(idless_create, "Likely Spoofed Posts")
+    pretty_print_list_with_title(spoofed_users, "Spoofed local user accounts")
+
+    IO.puts("""
+    In total found:
+      #{length(spoofed_users)} bogus users
+      #{length(idless_create)} likely spoofed posts
+      #{length(sus_pattern_posts)} maybe spoofed posts
+    """)
+  end
+
+  defp search_local_notes_without_create_id() do
+    Pleroma.Object
+    |> where([o], fragment("?->>'id' LIKE ?", o.data, ^local_id_pattern()))
+    |> join(:inner, [o], a in Pleroma.Activity,
+      on: fragment("?->>'object' = ?->>'id'", a.data, o.data)
+    )
+    |> where([o, a], fragment("NOT (? \\? 'id') OR ?->>'id' IS NULL", a.data, a.data))
+    |> select([o, a], {a.id, fragment("?->>'id'", o.data)})
+    |> order_by([o, a], a.id)
+    |> Pleroma.Repo.all()
+  end
+
+  defp search_sus_notes_by_id_patterns() do
+    [ep1, ep2, ep3, ep4] = activity_ext_url_patterns()
+
+    Pleroma.Object
+    |> where(
+      [o],
+      # for local objects we know exactly how a genuine id looks like
+      # (though a thorough attacker can emulate this)
+      # for remote posts, use some best-effort patterns
+      fragment(
+        """
+         (?->>'id' LIKE ? AND ?->>'id' NOT SIMILAR TO
+          ? || 'objects/[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}')
+        """,
+        o.data,
+        ^local_id_pattern(),
+        o.data,
+        ^local_id_prefix()
+      ) or
+        fragment("?->>'id' LIKE ?", o.data, "%/emoji/%") or
+        fragment("?->>'id' LIKE ?", o.data, "%/media/%") or
+        fragment("?->>'id' LIKE ?", o.data, "%/proxy/%") or
+        fragment("?->>'id' LIKE ?", o.data, ^ep1) or
+        fragment("?->>'id' LIKE ?", o.data, ^ep2) or
+        fragment("?->>'id' LIKE ?", o.data, ^ep3) or
+        fragment("?->>'id' LIKE ?", o.data, ^ep4)
+    )
+    |> join(:inner, [o], a in Pleroma.Activity,
+      on: fragment("?->>'object' = ?->>'id'", a.data, o.data)
+    )
+    |> select([o, a], {a.id, fragment("?->>'id'", o.data)})
+    |> order_by([o, a], a.id)
+    |> Pleroma.Repo.all()
+  end
+
+  defp search_bogus_local_users() do
+    Pleroma.User.Query.build(%{})
+    |> where([u], u.local == false and like(u.ap_id, ^local_id_pattern()))
+    |> order_by([u], u.ap_id)
+    |> select([u], u.ap_id)
+    |> Pleroma.Repo.all()
+  end
+
   # +-----------------------------------+
   # | module-specific utility functions |
   # +-----------------------------------+

From ee7d98b093651b8e1f2050cfe38533a9b6bd7b00 Mon Sep 17 00:00:00 2001
From: Oneric <oneric@oneric.stub>
Date: Thu, 28 Mar 2024 20:24:02 -0100
Subject: [PATCH 42/43] Update Changelog

---
 CHANGELOG.md | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9130a81ae..c46e84fa8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,17 +7,37 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ## Unreleased
 
 ## Added
+- CLI tasks best-effort checking for past abuse of the recent spoofing exploit
+- new `:mrf_steal_emoji, :download_unknown_size` option; defaults to `false`
 
 ## Changed
-- `Pleroma.Upload, :base_url` now MUST be configured explicitly;
+- `Pleroma.Upload, :base_url` now MUST be configured explicitly if used;
   use of the same domain as the instance is **strongly** discouraged
+- `:media_proxy, :base_url` now MUST be configured explicitly if used;
+  use of the same domain as the instance is **strongly** discouraged
+- StealEmoji:
+  - now uses the pack.json format;
+    existing users must migrate with an out-of-band script (check release notes)
+  - only steals shortcodes recognised as valid
+  - URLs of stolen emoji is no longer predictable
 - The `Dedupe` upload filter is now always active;
   `AnonymizeFilenames` is again opt-in
+- received AP data is sanity checked before we attempt to parse it as a user
+- Uploads, emoji and media proxy now restrict Content-Type headers to a safe subset
+- Akkoma will no longer fetch and parse objects hosted on the same domain
 
 ## Fixed
 - Critical security issue allowing Akkoma to be used as a vector for
   (depending on configuration) impersonation of other users or creation
   of bogus users and posts on the upload domain
+- Critical security issue letting Akkoma fall for the above impersonation
+  payloads due to lack of strict id checking
+- Critical security issue allowing domains redirect to to pose as the initial domain
+  (e.g. with media proxy's fallback redirects)
+- refetched objects can no longer attribute themselves to third-party actors
+  (this had no externally visible effect since actor info is read from the Create activity)
+- our litepub JSON-LD schema is now served with the correct content type
+- remote APNG attachments are now recognised as images
 
 ## 2024.02
 

From 3650bb03709bf68be2fd7ebcb16b5fc7189c36da Mon Sep 17 00:00:00 2001
From: FloatingGhost <hannah@coffee-and-dreams.uk>
Date: Sat, 30 Mar 2024 11:44:34 +0000
Subject: [PATCH 43/43] Changelog entry

---
 CHANGELOG.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index c46e84fa8..3d15caf51 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
-## Unreleased
+## 2024.03
 
 ## Added
 - CLI tasks best-effort checking for past abuse of the recent spoofing exploit