Merge branch 'develop' into fedi-absturztau-be

2023-02-12 09:13:56 +01:00 · 2023-02-12 09:13:56 +01:00 · 7bcf64438e
commit 7bcf64438e
parent a9eb94e8d3 f92484fd01
17 changed files with 199 additions and 47 deletions
--- a/.woodpecker.yml
+++ b/.woodpecker.yml
@ -95,7 +95,7 @@ pipeline:

  # Canonical amd64
  ubuntu22:
-    image: hexpm/elixir:1.14.2-erlang-25.1.2-ubuntu-jammy-20220428
+    image: hexpm/elixir:1.14.3-erlang-25.2.2-ubuntu-jammy-20221130
    <<: *on-release
    environment:
      MIX_ENV: prod
@ -122,7 +122,7 @@ pipeline:
      - /bin/sh /entrypoint.sh

  debian-bullseye:
-    image: hexpm/elixir:1.14.2-erlang-25.1.2-debian-bullseye-20221004
+    image: hexpm/elixir:1.14.3-erlang-25.2.2-debian-bullseye-20230109
    <<: *on-release
    environment:
      MIX_ENV: prod
@ -151,7 +151,7 @@ pipeline:

  # Canonical amd64-musl
  musl:
-    image: hexpm/elixir:1.14.2-erlang-25.1.2-alpine-3.16.2
+    image: hexpm/elixir:1.14.3-erlang-25.2.2-alpine-3.17.0
    <<: *on-stable
    environment:
      MIX_ENV: prod
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file.

 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

-## Unreleased
+## 2023.02

 ### Added
 - Prometheus metrics exporting from `/api/v1/akkoma/metrics`
@ -37,6 +37,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Simplified HTTP signature processing
 - Rich media will now hard-exit after 5 seconds, to prevent timeline hangs
 - HTTP Content Security Policy is now far more strict to prevent any potential XSS/CSS leakages
+- Follow requests are now paginated, matches mastodon API spec, so use the Link header to paginate.
+- `internal.fetch` and `relay` actors are now represented with the actor type `Application`

 ### Fixed 
 - /api/v1/accounts/lookup will now respect restrict\_unauthenticated
--- a/config/config.exs
+++ b/config/config.exs
@ -354,7 +354,7 @@

 config :pleroma, :activitypub,
  unfollow_blocked: true,
-  outgoing_blocks: true,
+  outgoing_blocks: false,
  blockers_visible: true,
  follow_handshake_timeout: 500,
  note_replies_output_limit: 5,
--- a/docs/docs/configuration/postgresql.md
+++ b/docs/docs/configuration/postgresql.md
@ -6,6 +6,31 @@ Akkoma performance is largely dependent on performance of the underlying databas

 [PgTune](https://pgtune.leopard.in.ua) can be used to get recommended settings. Be sure to set "Number of Connections" to 20, otherwise it might produce settings hurtful to database performance. It is also recommended to not use "Network Storage" option.

+If your server runs other services, you may want to take that into account. E.g. if you have 4G ram, but 1G of it is already used for other services, it may be better to tell PGTune you only have 3G. In the end, PGTune only provides recomended settings, you can always try to finetune further.
+
+### Example configurations
+
+Here are some configuration suggestions for PostgreSQL 10+.
+
+#### 1GB RAM, 1 CPU
+```
+shared_buffers = 256MB
+effective_cache_size = 768MB
+maintenance_work_mem = 64MB
+work_mem = 13107kB
+```
+
+#### 2GB RAM, 2 CPU
+```
+shared_buffers = 512MB
+effective_cache_size = 1536MB
+maintenance_work_mem = 128MB
+work_mem = 26214kB
+max_worker_processes = 2
+max_parallel_workers_per_gather = 1
+max_parallel_workers = 2
+```
+
 ## Disable generic query plans

 When PostgreSQL receives a query, it decides on a strategy for searching the requested data, this is called a query plan. The query planner has two modes: generic and custom. Generic makes a plan for all queries of the same shape, ignoring the parameters, which is then cached and reused. Custom, on the contrary, generates a unique query plan based on query parameters.
@ -23,26 +48,3 @@ config :pleroma, Pleroma.Repo,
 ```

 A more detailed explaination of the issue can be found at <https://blog.soykaf.com/post/postgresql-elixir-troubles/>.
-
-## Example configurations
-
-Here are some configuration suggestions for PostgreSQL 10+.
-
-### 1GB RAM, 1 CPU
-```
-shared_buffers = 256MB
-effective_cache_size = 768MB
-maintenance_work_mem = 64MB
-work_mem = 13107kB
-```
-
-### 2GB RAM, 2 CPU
-```
-shared_buffers = 512MB
-effective_cache_size = 1536MB
-maintenance_work_mem = 128MB
-work_mem = 26214kB
-max_worker_processes = 2
-max_parallel_workers_per_gather = 1
-max_parallel_workers = 2
-```
--- a/docs/docs/development/index.md
+++ b/docs/docs/development/index.md
@ -1 +1,48 @@
-This section contains notes and guidelines for developers.
+# Contributing to Akkoma
+
+You wish to add a new feature in Akkoma, but don't know how to proceed? This guide takes you through the various steps of the development and contribution process.
+
+If you're looking for stuff to implement or fix, check the [bug-tracker](https://akkoma.dev/AkkomaGang/akkoma/issues) or [forum](https://meta.akkoma.dev/c/requests/5).
+
+Come say hi to us in the [#akkoma-dev chat room](./../#irc)!
+
+## Akkoma Clients
+
+Akkoma is the back-end. Clients have their own repositories and often separate projects. You can check what clients work with Akkoma [on the clients page](../clients/). If you maintain a working client not listed yet, feel free to make a PR [to these docs](./#docs)!
+
+For resources on APIs and such, check the sidebar of this page.
+
+## Docs
+
+The docs are written in Markdown, including certain extensions, and can be found [in the docs folder of the Akkoma repo](https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/docs/). The content itself is stored in the `docs` subdirectory.
+
+## Technology
+
+Akkoma is written in [Elixir](https://elixir-lang.org/) and uses [Postgresql](https://www.postgresql.org/) for database. We use [Git](https://git-scm.com/) for collaboration and tracking code changes. Furthermore it can typically run on [Unix and Unix-like OS'es](https://en.wikipedia.org/wiki/Unix-like). For development, you should use an OS which [can run Akkoma](../installation/debian_based_en/).
+
+It's good to have at least some basic understanding of at least Git and Elixir. If this is completely new for you, there's some [videos explaining Git](https://git-scm.com/doc) and Codeberg has a nice article explaining the typical [pull requests Git flow](https://docs.codeberg.org/collaborating/pull-requests-and-git-flow/). For Elixir, you can follow Elixir's own [Getting Started guide](https://elixir-lang.org/getting-started/introduction.html).
+
+## Setting up a development environment
+
+The best way to start is getting the software to run from source so you can start poking on it. Check out the [guides for setting up an Akkoma instance for development](setting_up_akkoma_dev/#setting-up-a-akkoma-development-environment).
+
+## General overview
+### Modules
+
+Akkoma has several modules. There are modules for [uploading](https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/lib/pleroma/uploaders), [upload filters](https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/lib/pleroma/upload/filter), [translators](https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/lib/pleroma/akkoma/translators)... The most famous ones are without a doubt the [MRF policies](https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/lib/pleroma/web/activity_pub/mrf). Modules are often self contained and a good way to start with development because you don't have to think about much more than just the module itself. We even have an example on [writing your own MRF policy](/configuration/mrf/#writing-your-own-mrf-policy)!
+
+Another easy entry point is the [mix tasks](https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/lib/mix/tasks/pleroma). They too are often self contained and don't need you to go through much of the code.
+
+### Activity Streams/Activity Pub
+
+Akkoma uses Activity Streams for both federation, as well as internal representation. It may be interesting to at least go over the specifications of [Activity Pub](https://www.w3.org/TR/activitypub/), [Activity Streams 2.0](https://www.w3.org/TR/activitystreams-core/), and [Activity Streams Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary/). Note that these are not enough to have a full grasp of how everything works, but should at least give you the basics to understand how messages are passed between and inside Akkoma instances.
+
+## Don't forget
+
+When you make changes, you're expected to create [a Pull Request](https://akkoma.dev/AkkomaGang/akkoma/pulls). You don't have to wait until you finish to create the PR, but please do prefix the title of the PR with "WIP: " for as long as you're still working on it. The sooner you create your PR, the sooner people know what you are working on and the sooner you can get feedback and, if needed, help. You can then simply keep working on it until you are finished.
+
+When doing changes, don't forget to add it to the relevant parts of the [CHANGELOG.md](https://akkoma.dev/AkkomaGang/akkoma/src/branch/develop/CHANGELOG.md).
+
+You're expected to write [tests](https://elixirschool.com/en/lessons/testing/basics). While code is generally stored in the `lib` directory, tests are stored in the `test` directory using a similar folder structure. Feel free to peak at other tests to see how they are done. Obviously tests are expected to pass and properly test the functionality you added. If you feel really confident, you could even try to [write a test first and then write the code needed to make it pass](https://en.wikipedia.org/wiki/Test-driven_development)!
+
+Code is formatted using the default formatter that comes with Elixir. You can format a file with e.g. `mix format /path/to/file.ex`. To check if everything is properly formatted, you can run `mix format --check-formatted`.
--- a/docs/docs/development/setting_up_akkoma_dev.md
+++ b/docs/docs/development/setting_up_akkoma_dev.md
@ -5,17 +5,32 @@ Akkoma requires some adjustments from the defaults for running the instance loca
 ## Installing

 1. Install Akkoma as explained in [the docs](../installation/debian_based_en.md), with some exceptions:
-    * You can use your own fork of the repository and add akkoma as a remote `git remote add akkoma 'https://akkoma.dev/AkkomaGang/akkoma.git'`
-    * You can skip systemd and nginx and all that stuff
    * No need to create a dedicated akkoma user, it's easier to just use your own user
-    * For the DB you can still choose a dedicated user, the mix tasks set it up for you so it's no extra work for you
+    * You can use your own fork of the repository and add akkoma as a remote `git remote add akkoma 'https://akkoma.dev/AkkomaGang/akkoma.git'`
    * For domain you can use `localhost`
+    * For the DB you can still choose a dedicated user. The mix tasks sets it up, so it's no extra work for you
    * instead of creating a `prod.secret.exs`, create `dev.secret.exs`
    * No need to prefix with `MIX_ENV=prod`. We're using dev and that's the default MIX_ENV
+    * You can skip nginx and systemd
+    * For front-end, you'll probably want to install and use the develop branch instead of the stable branch. There's no guarantee that the stable branch of the FE will always work on the develop branch of the BE.
 2. Change the dev.secret.exs
+    * Change the FE settings to use the installed branch (see also [Frontend Management](/configuration/frontend_management/))
    * Change the scheme in `config :pleroma, Pleroma.Web.Endpoint` to http (see examples below)
    * If you want to change other settings, you can do that too
-3. You can now start the server `mix phx.server`. Once it's build and started, you can access the instance on `http://<host>:<port>` (e.g.http://localhost:4000 ) and should be able to do everything locally you normaly can.
+3. You can now start the server with `mix phx.server`. Once it's build and started, you can access the instance on `http://<host>:<port>` (e.g.http://localhost:4000 ) and should be able to do everything locally you normally can.
+
+Example on how to install pleroma-fe and admin-fe using it's develop branch
+```sh
+mix pleroma.frontend install pleroma-fe --ref develop
+mix pleroma.frontend install admin-fe --ref develop
+```
+
+Example config to use the pleroma-fe and admin-fe installed from the develop branch
+```elixir
+config :pleroma, :frontends,
+  primary: %{"name" => "pleroma-fe", "ref" => "develop"},
+  admin: %{"name" => "admin-fe", "ref" => "develop"}
+```

 Example config to change the scheme to http. Change the port if you want to run on another port.
 ```elixir
@ -94,4 +109,4 @@ Update Akkoma as explained in [the docs](../administration/updating.md). Just ma

 ## Working on multiple branches

-If you develop on a separate branch, it's possible you did migrations that aren't merged into another branch you're working on. If you have multiple things you're working on, it's probably best to set up multiple Akkoma instances each with their own database. If you finished with a branch and want to switch back to develop to start a new branch from there, you can drop the database and recreate the database (e.g. by using `config/setup_db.psql`). The commands to drop and recreate the database can be found in [the docs](../administration/backup.md).
+If you develop on a separate branch, it's possible you did migrations that aren't merged into another branch you're working on. In that case, it's probably best to set up multiple Akkoma instances each with their own database. If you finished with a branch and want to switch back to develop to start a new branch from there, you can drop the database and recreate the database (e.g. by using `config/setup_db.psql`). The commands to drop and recreate the database can be found in [the docs](../administration/backup.md).
--- a/lib/pleroma/following_relationship.ex
+++ b/lib/pleroma/following_relationship.ex
@ -155,14 +155,13 @@ def following_count(%User{} = user) do
    |> Repo.aggregate(:count, :id)
  end

-  def get_follow_requests(%User{id: id}) do
+  def get_follow_requests_query(%User{id: id}) do
    __MODULE__
-    |> join(:inner, [r], f in assoc(r, :follower))
+    |> join(:inner, [r], f in assoc(r, :follower), as: :follower)
    |> where([r], r.state == ^:follow_pending)
    |> where([r], r.following_id == ^id)
-    |> where([r, f], f.is_active == true)
-    |> select([r, f], f)
-    |> Repo.all()
+    |> where([r, follower: f], f.is_active == true)
+    |> select([r, follower: f], f)
  end

  def following?(%User{id: follower_id}, %User{id: followed_id}) do
--- a/lib/pleroma/user.ex
+++ b/lib/pleroma/user.ex
@ -273,7 +273,13 @@ def cached_muted_users_ap_ids(user) do
  defdelegate following(user), to: FollowingRelationship
  defdelegate following?(follower, followed), to: FollowingRelationship
  defdelegate following_ap_ids(user), to: FollowingRelationship
-  defdelegate get_follow_requests(user), to: FollowingRelationship
+  defdelegate get_follow_requests_query(user), to: FollowingRelationship
+
+  def get_follow_requests(user) do
+    get_follow_requests_query(user)
+    |> Repo.all()
+  end
+
  defdelegate search(query, opts \\ []), to: User.Search

  @doc """
@ -1994,6 +2000,7 @@ defp create_service_actor(uri, nickname) do
    %User{
      invisible: true,
      local: true,
+      actor_type: "Application",
      ap_id: uri,
      nickname: nickname,
      follower_address: uri <> "/followers"
--- a/lib/pleroma/web/api_spec/operations/follow_request_operation.ex
+++ b/lib/pleroma/web/api_spec/operations/follow_request_operation.ex
@ -19,6 +19,7 @@ def index_operation do
      summary: "Retrieve follow requests",
      security: [%{"oAuth" => ["read:follows", "follow"]}],
      operationId: "FollowRequestController.index",
+      parameters: pagination_params(),
      responses: %{
        200 =>
          Operation.response("Array of Account", "application/json", %Schema{
@ -62,4 +63,22 @@ defp id_param do
      required: true
    )
  end
+
+  defp pagination_params do
+    [
+      Operation.parameter(:max_id, :query, :string, "Return items older than this ID"),
+      Operation.parameter(
+        :since_id,
+        :query,
+        :string,
+        "Return the oldest items newer than this ID"
+      ),
+      Operation.parameter(
+        :limit,
+        :query,
+        %Schema{type: :integer, default: 20},
+        "Maximum number of items to return. Will be ignored if it's more than 40"
+      )
+    ]
+  end
 end
--- a/lib/pleroma/web/mastodon_api/controllers/follow_request_controller.ex
+++ b/lib/pleroma/web/mastodon_api/controllers/follow_request_controller.ex
@ -5,9 +5,13 @@
 defmodule Pleroma.Web.MastodonAPI.FollowRequestController do
  use Pleroma.Web, :controller

+  import Pleroma.Web.ControllerHelper,
+    only: [add_link_headers: 2]
+
  alias Pleroma.User
  alias Pleroma.Web.CommonAPI
  alias Pleroma.Web.Plugs.OAuthScopesPlug
+  alias Pleroma.Pagination

  plug(Pleroma.Web.ApiSpec.CastAndValidate)
  plug(:assign_follower when action != :index)
@ -24,10 +28,15 @@ defmodule Pleroma.Web.MastodonAPI.FollowRequestController do
  defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.FollowRequestOperation

  @doc "GET /api/v1/follow_requests"
-  def index(%{assigns: %{user: followed}} = conn, _params) do
-    follow_requests = User.get_follow_requests(followed)
+  def index(%{assigns: %{user: followed}} = conn, params) do
+    follow_requests =
+      followed
+      |> User.get_follow_requests_query()
+      |> Pagination.fetch_paginated(params, :keyset, :follower)

-    render(conn, "index.json", for: followed, users: follow_requests, as: :user)
+    conn
+    |> add_link_headers(follow_requests)
+    |> render("index.json", for: followed, users: follow_requests, as: :user)
  end

  @doc "POST /api/v1/follow_requests/:id/authorize"
--- a/lib/pleroma/web/mastodon_api/views/account_view.ex
+++ b/lib/pleroma/web/mastodon_api/views/account_view.ex
@ -334,7 +334,8 @@ defp maybe_put_follow_requests_count(
         %User{id: user_id}
       ) do
    count =
-      User.get_follow_requests(user)
+      user
+      |> User.get_follow_requests()
      |> length()

    data
--- a/mix.exs
+++ b/mix.exs
@ -4,7 +4,7 @@ defmodule Pleroma.Mixfile do
  def project do
    [
      app: :pleroma,
-      version: version("3.5.0"),
+      version: version("3.6.0"),
      elixir: "~> 1.12",
      elixirc_paths: elixirc_paths(Mix.env()),
      compilers: [:phoenix] ++ Mix.compilers(),
--- a/priv/repo/migrations/20230202154409_instance_actors_to_actor_type_application.exs
+++ b/priv/repo/migrations/20230202154409_instance_actors_to_actor_type_application.exs
@ -0,0 +1,21 @@
+defmodule Pleroma.Repo.Migrations.InstanceActorsToActorTypeApplication do
+  use Ecto.Migration
+
+  def up do
+    execute("""
+    update users
+    set actor_type = 'Application'
+    where local
+    and (ap_id like '%/relay' or ap_id like '%/internal/fetch')
+    """)
+  end
+
+  def down do
+    execute("""
+    update users
+    set actor_type = 'Person'
+    where local
+    and (ap_id like '%/relay' or ap_id like '%/internal/fetch')
+    """)
+  end
+end
--- a/test/pleroma/web/activity_pub/object_validators/block_validation_test.exs
+++ b/test/pleroma/web/activity_pub/object_validators/block_validation_test.exs
@ -12,6 +12,7 @@ defmodule Pleroma.Web.ActivityPub.ObjectValidators.BlockValidationTest do

  describe "blocks" do
    setup do
+      clear_config([:activitypub, :outgoing_blocks], true)
      user = insert(:user, local: false)
      blocked = insert(:user)

--- a/test/pleroma/web/activity_pub/relay_test.exs
+++ b/test/pleroma/web/activity_pub/relay_test.exs
@ -19,6 +19,12 @@ test "gets an actor for the relay" do
    assert user.ap_id == "#{Pleroma.Web.Endpoint.url()}/relay"
  end

+  test "relay actor is an application" do
+    # See <https://www.w3.org/TR/activitystreams-vocabulary/#dfn-application>
+    user = Relay.get_actor()
+    assert user.actor_type == "Application"
+  end
+
  test "relay actor is invisible" do
    user = Relay.get_actor()
    assert User.invisible?(user)
--- a/test/pleroma/web/common_api_test.exs
+++ b/test/pleroma/web/common_api_test.exs
@ -71,6 +71,7 @@ test "it posts a poll" do

    test "it blocks and federates", %{blocker: blocker, blocked: blocked} do
      clear_config([:instance, :federating], true)
+      clear_config([:activitypub, :outgoing_blocks], true)

      with_mock Pleroma.Web.Federator,
        publish: fn _ -> nil end do
--- a/test/pleroma/web/mastodon_api/controllers/follow_request_controller_test.exs
+++ b/test/pleroma/web/mastodon_api/controllers/follow_request_controller_test.exs
@ -10,6 +10,11 @@ defmodule Pleroma.Web.MastodonAPI.FollowRequestControllerTest do

  import Pleroma.Factory

+  defp extract_next_link_header(header) do
+    [_, next_link] = Regex.run(~r{<(?<next_link>.*)>; rel="next"}, header)
+    next_link
+  end
+
  describe "locked accounts" do
    setup do
      user = insert(:user, is_locked: true)
@ -31,6 +36,23 @@ test "/api/v1/follow_requests works", %{user: user, conn: conn} do
      assert to_string(other_user.id) == relationship["id"]
    end

+    test "/api/v1/follow_requests paginates", %{user: user, conn: conn} do
+      for _ <- 1..21 do
+        other_user = insert(:user)
+        {:ok, _, _, _activity} = CommonAPI.follow(other_user, user)
+        {:ok, _, _} = User.follow(other_user, user, :follow_pending)
+      end
+
+      conn = get(conn, "/api/v1/follow_requests")
+      assert length(json_response_and_validate_schema(conn, 200)) == 20
+      assert [link_header] = get_resp_header(conn, "link")
+      assert link_header =~ "rel=\"next\""
+      next_link = extract_next_link_header(link_header)
+      assert next_link =~ "/api/v1/follow_requests"
+      conn = get(conn, next_link)
+      assert length(json_response_and_validate_schema(conn, 200)) == 1
+    end
+
    test "/api/v1/follow_requests/:id/authorize works", %{user: user, conn: conn} do
      other_user = insert(:user)