Merge branch 'develop' of https://git.pleroma.social/pleroma/pleroma into develop

CHANGELOG.md

@@ -4,8 +4,39 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
+### Removed
+- **Breaking**: Removed 1.0+ deprecated configurations `Pleroma.Upload, :strip_exif` and `:instance, :dedupe_media`
+- **Breaking**: OStatus protocol support
+
+### Changed
+- **Breaking:** Elixir >=1.8 is now required (was >= 1.7)
+- Replaced [pleroma_job_queue](https://git.pleroma.social/pleroma/pleroma_job_queue) and `Pleroma.Web.Federator.RetryQueue` with [Oban](https://github.com/sorentwo/oban) (see [`docs/config.md`](docs/config.md) on migrating customized worker / retry settings)
+- Introduced [quantum](https://github.com/quantum-elixir/quantum-core) job scheduler
+- Enabled `:instance, extended_nickname_format` in the default config
+- Add `rel="ugc"` to all links in statuses, to prevent SEO spam
+- Extract RSS functionality from OStatus
+- MRF (Simple Policy): Also use `:accept`/`:reject` on the actors rather than only their activities
+<details>
+  <summary>API Changes</summary>
+
+- **Breaking:** Admin API: Return link alongside with token on password reset
+- **Breaking:** `/api/pleroma/admin/users/invite_token` now uses `POST`, changed accepted params and returns full invite in json instead of only token string.
+- Admin API: Return `total` when querying for reports
+- Mastodon API: Return `pleroma.direct_conversation_id` when creating a direct message (`POST /api/v1/statuses`)
+- Admin API: Return link alongside with token on password reset
+- Mastodon API: Add `pleroma.direct_conversation_id` to the status endpoint (`GET /api/v1/statuses/:id`)
+- Mastodon API: `pleroma.thread_muted` to the Status entity
+- Mastodon API: Mark the direct conversation as read for the author when they send a new direct message
+</details>
+
 ### Added
 - Refreshing poll results for remote polls
+- Authentication: Added rate limit for password-authorized actions / login existence checks
+- Mix task to re-count statuses for all users (`mix pleroma.count_statuses`)
+- Support for `X-Forwarded-For` and similar HTTP headers which used by reverse proxies to pass a real user IP address to the backend. Must not be enabled unless your instance is behind at least one reverse proxy (such as Nginx, Apache HTTPD or Varnish Cache).
+<details>
+  <summary>API Changes</summary>
+
 - Job queue stats to the healthcheck page
 - Admin API: Add ability to require password reset
 - Mastodon API: Account entities now include `follow_requests_count` (planned Mastodon 3.x addition)
@@ -14,15 +45,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: Add `upload_limit`, `avatar_upload_limit`, `background_upload_limit`, and `banner_upload_limit` to `/api/v1/instance`
 - Mastodon API: Add `pleroma.unread_conversation_count` to the Account entity
 - OAuth: support for hierarchical permissions / [Mastodon 2.4.3 OAuth permissions](https://docs.joinmastodon.org/api/permissions/)
-- Authentication: Added rate limit for password-authorized actions / login existence checks
 - Metadata Link: Atom syndication Feed
-- Mix task to re-count statuses for all users (`mix pleroma.count_statuses`)
 - Mastodon API: Add `exclude_visibilities` parameter to the timeline and notification endpoints
 - Admin API: `/users/:nickname/toggle_activation` endpoint is now deprecated in favor of: `/users/activate`, `/users/deactivate`, both accept `nicknames` array
 - Admin API: `POST/DELETE /api/pleroma/admin/users/:nickname/permission_group/:permission_group` are deprecated in favor of: `POST/DELETE /api/pleroma/admin/users/permission_group/:permission_group` (both accept `nicknames` array), `DELETE /api/pleroma/admin/users` (`nickname` query param or `nickname` sent in JSON body) is deprecated in favor of: `DELETE /api/pleroma/admin/users` (`nicknames` query array param or `nicknames` sent in JSON body).
-
+- Admin API: Add `GET /api/pleroma/admin/relay` endpoint - lists all followed relays
-### Removed
+- Pleroma API: `POST /api/v1/pleroma/conversations/read` to mark all conversations as read
-- **Breaking**: Removed 1.0+ deprecated configurations `Pleroma.Upload, :strip_exif` and `:instance, :dedupe_media`
+- Mastodon API: Add `/api/v1/markers` for managing timeline read markers
 
 ### Changed
 - **Breaking:** Elixir >=1.8 is now required (was >= 1.7)
@@ -36,15 +65,36 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - OStatus: Extract RSS functionality
 - Mastodon API: Add `pleroma.direct_conversation_id` to the status endpoint (`GET /api/v1/statuses/:id`)
 - Mastodon API: Mark the direct conversation as read for the author when they send a new direct message
+</details>
 
 ### Fixed
+- Report emails now include functional links to profiles of remote user accounts
+<details>
+  <summary>API Changes</summary>
+
 - Mastodon API: Fix private and direct statuses not being filtered out from the public timeline for an authenticated user (`GET /api/v1/timelines/public`)
 - Mastodon API: Inability to get some local users by nickname in `/api/v1/accounts/:id_or_nickname`
-- Added `:instance, extended_nickname_format` setting to the default config
+</details>
-- Report emails now include functional links to profiles of remote user accounts
 
-## [1.1.0] - 2019-??-??
+## [1.1.2] - 2019-10-18
-**Breaking:** The stable branch has been changed from `master` to `stable`, `master` now points to `release/1.0`
+### Fixed
+- `pleroma_ctl` trying to connect to a running instance when generating the config, which of course doesn't exist.
+
+## [1.1.1] - 2019-10-18
+### Fixed
+- One of the migrations between 1.0.0 and 1.1.0 wiping user info of the relay user because of unexpected behavior of postgresql's `jsonb_set`, resulting in inability to post in the default configuration. If you were affected, please run the following query in postgres console, the relay user will be recreated automatically:
+```
+delete from users where ap_id = 'https://your.instance.hostname/relay';
+```
+- Bad user search matches
+
+## [1.1.0] - 2019-10-14
+**Breaking:** The stable branch has been changed from `master` to `stable`. If you want to keep using 1.0, the `release/1.0` branch will receive security updates for 6 months after 1.1 release.
+
+**OTP Note:** `pleroma_ctl` in 1.0 defaults to `master` and doesn't support specifying arbitrary branches, making `./pleroma_ctl update` fail. To fix this, fetch a version of `pleroma_ctl` from 1.1 using the command below and proceed with the update normally:
+```
+curl -Lo ./bin/pleroma_ctl 'https://git.pleroma.social/pleroma/pleroma/raw/develop/rel/files/bin/pleroma_ctl'
+```
 ### Security
 - Mastodon API: respect post privacy in `/api/v1/statuses/:id/{favourited,reblogged}_by`
 
@@ -52,16 +102,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - **Breaking:** GNU Social API with Qvitter extensions support
 - Emoji: Remove longfox emojis.
 - Remove `Reply-To` header from report emails for admins.
+- ActivityPub: The `/objects/:uuid/likes` endpoint.
 
 ### Changed
 - **Breaking:** Configuration: A setting to explicitly disable the mailer was added, defaulting to true, if you are using a mailer add `config :pleroma, Pleroma.Emails.Mailer, enabled: true` to your config
 - **Breaking:** Configuration: `/media/` is now removed when `base_url` is configured, append `/media/` to your `base_url` config to keep the old behaviour if desired
 - **Breaking:** `/api/pleroma/notifications/read` is moved to `/api/v1/pleroma/notifications/read` and now supports `max_id` and responds with Mastodon API entities.
-- **Breaking:** `/api/pleroma/admin/users/invite_token` now uses `POST`, changed accepted params and returns full invite in json instead of only token string.
 - Configuration: added `config/description.exs`, from which `docs/config.md` is generated
 - Configuration: OpenGraph and TwitterCard providers enabled by default
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
-- Mastodon API: `pleroma.thread_muted` key in the Status entity
 - Federation: Return 403 errors when trying to request pages from a user's follower/following collections if they have `hide_followers`/`hide_follows` set
 - NodeInfo: Return `skipThreadContainment` in `metadata` for the `skip_thread_containment` option
 - NodeInfo: Return `mailerEnabled` in `metadata`
@@ -70,7 +119,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - AdminAPI: Add "godmode" while fetching user statuses (i.e. admin can see private statuses)
 - Improve digest email template
 – Pagination: (optional) return `total` alongside with `items` when paginating
-- Add `rel="ugc"` to all links in statuses, to prevent SEO spam
+- The `Pleroma.FlakeId` module has been replaced with the `flake_id` library.
 
 ### Fixed
 - Following from Osada
@@ -81,21 +130,28 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: Misskey's endless polls being unable to render
 - Mastodon API: Embedded relationships not being properly rendered in the Account entity of Status entity
 - Mastodon API: Notifications endpoint crashing if one notification failed to render
+- Mastodon API: `exclude_replies` is correctly handled again.
 - Mastodon API: Add `account_id`, `type`, `offset`, and `limit` to search API (`/api/v1/search` and `/api/v2/search`)
 - Mastodon API, streaming: Fix filtering of notifications based on blocks/mutes/thread mutes
-- ActivityPub C2S: follower/following collection pages being inaccessible even when authentifucated if `hide_followers`/ `hide_follows` was set
+- Mastodon API: Fix private and direct statuses not being filtered out from the public timeline for an authenticated user (`GET /api/v1/timelines/public`)
-- Existing user id not being preserved on insert conflict
+- Mastodon API: Ensure the `account` field is not empty when rendering Notification entities.
+- Mastodon API: Inability to get some local users by nickname in `/api/v1/accounts/:id_or_nickname`
+- Mastodon API: Blocks are now treated consistently between the Streaming API and the Timeline APIs
 - Rich Media: Parser failing when no TTL can be found by image TTL setters
 - Rich Media: The crawled URL is now spliced into the rich media data.
 - ActivityPub S2S: sharedInbox usage has been mostly aligned with the rules in the AP specification.
-- Pleroma.Upload base_url was not automatically whitelisted by MediaProxy. Now your custom CDN or file hosting will be accessed directly as expected.
+- ActivityPub C2S: follower/following collection pages being inaccessible even when authentifucated if `hide_followers`/ `hide_follows` was set
-- Report email not being sent to admins when the reporter is a remote user
-- Reverse Proxy limiting `max_body_length` was incorrectly defined and only checked `Content-Length` headers which may not be sufficient in some circumstances
 - ActivityPub: Deactivated user deletion
 - ActivityPub: Fix `/users/:nickname/inbox` crashing without an authenticated user
 - MRF: fix ability to follow a relay when AntiFollowbotPolicy was enabled
-- Mastodon API: Blocks are now treated consistently between the Streaming API and the Timeline APIs
+- ActivityPub: Correct addressing of Undo.
-- Mastodon API: `exclude_replies` is correctly handled again.
+- ActivityPub: Correct addressing of profile update activities.
+- ActivityPub: Polls are now refreshed when necessary.
+- Report emails now include functional links to profiles of remote user accounts
+- Existing user id not being preserved on insert conflict
+- Pleroma.Upload base_url was not automatically whitelisted by MediaProxy. Now your custom CDN or file hosting will be accessed directly as expected.
+- Report email not being sent to admins when the reporter is a remote user
+- Reverse Proxy limiting `max_body_length` was incorrectly defined and only checked `Content-Length` headers which may not be sufficient in some circumstances
 
 ### Added
 - Expiring/ephemeral activites. All activities can have expires_at value set, which controls when they should be deleted automatically.
@@ -109,6 +165,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: Support for the [`tagged` filter](https://github.com/tootsuite/mastodon/pull/9755) in [`GET /api/v1/accounts/:id/statuses`](https://docs.joinmastodon.org/api/rest/accounts/#get-api-v1-accounts-id-statuses)
 - Mastodon API, streaming: Add support for passing the token in the `Sec-WebSocket-Protocol` header
 - Mastodon API, extension: Ability to reset avatar, profile banner, and background
+- Mastodon API: Add support for `fields_attributes` API parameter (setting custom fields)
 - Mastodon API: Add support for categories for custom emojis by reusing the group feature. <https://github.com/tootsuite/mastodon/pull/11196>
 - Mastodon API: Add support for muting/unmuting notifications
 - Mastodon API: Add support for the `blocked_by` attribute in the relationship API (`GET /api/v1/accounts/relationships`). <https://github.com/tootsuite/mastodon/pull/10373>
@@ -117,7 +174,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: added `/auth/password` endpoint for password reset with rate limit.
 - Mastodon API: /api/v1/accounts/:id/statuses now supports nicknames or user id
 - Mastodon API: Improve support for the user profile custom fields
-- Mastodon API: follower/following counters are nullified when `hide_follows`/`hide_followers` and `hide_follows_count`/`hide_followers_count` are set
+- Mastodon API: Add support for `fields_attributes` API parameter (setting custom fields)
+- Mastodon API: Added an endpoint to get multiple statuses by IDs (`GET /api/v1/statuses/?ids[]=1&ids[]=2`)
 - Admin API: Return users' tags when querying reports
 - Admin API: Return avatar and display name when querying users
 - Admin API: Allow querying user by ID
@@ -135,11 +193,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Pleroma API: Add `/api/v1/pleroma/accounts/confirmation_resend?email=<email>` for resending account confirmation.
 - Pleroma API: Email change endpoint.
 - Admin API: Added moderation log
-- Support for `X-Forwarded-For` and similar HTTP headers which used by reverse proxies to pass a real user IP address to the backend. Must not be enabled unless your instance is behind at least one reverse proxy (such as Nginx, Apache HTTPD or Varnish Cache).
 - Web response cache (currently, enabled for ActivityPub)
-- Mastodon API: Added an endpoint to get multiple statuses by IDs (`GET /api/v1/statuses/?ids[]=1&ids[]=2`)
-- ActivityPub: Add ActivityPub actor's `discoverable` parameter.
-- Admin API: Added moderation log filters (user/start date/end date/search/pagination)
 - Reverse Proxy: Do not retry failed requests to limit pressure on the peer
 
 ### Changed

config/config.exs

@@ -59,10 +59,6 @@
     _ -> []
   end
 
-scheduled_jobs =
-  scheduled_jobs ++
-    [{"0 */6 * * * *", {Pleroma.Web.Websub, :refresh_subscriptions, []}}]
-
 config :pleroma, Pleroma.Scheduler,
   global: true,
   overlap: true,
@@ -243,9 +239,7 @@
   federation_incoming_replies_max_depth: 100,
   federation_reachability_timeout_days: 7,
   federation_publisher_modules: [
-    Pleroma.Web.ActivityPub.Publisher,
+    Pleroma.Web.ActivityPub.Publisher
-    Pleroma.Web.Websub,
-    Pleroma.Web.Salmon
   ],
   allow_relay: true,
   rewrite_policy: Pleroma.Web.ActivityPub.MRF.NoOpPolicy,
@@ -328,6 +322,16 @@
   ],
   default_mascot: :pleroma_fox_tan
 
+config :pleroma, :manifest,
+  icons: [
+    %{
+      src: "/static/logo.png",
+      type: "image/png"
+    }
+  ],
+  theme_color: "#282c37",
+  background_color: "#191b22"
+
 config :pleroma, :activitypub,
   unfollow_blocked: true,
   outgoing_blocks: true,

config/description.exs

@@ -581,9 +581,7 @@
         type: [:list, :module],
         description: "List of modules for federation publishing",
         suggestions: [
-          Pleroma.Web.ActivityPub.Publisher,
+          Pleroma.Web.ActivityPub.Publisher
-          Pleroma.Web.Websub,
-          Pleroma.Web.Salmo
         ]
       },
       %{
@@ -1100,6 +1098,45 @@
       }
     ]
   },
+  %{
+    group: :pleroma,
+    key: :manifest,
+    type: :group,
+    description:
+      "This section describe PWA manifest instance-specific values. Currently this option relate only for MastoFE",
+    children: [
+      %{
+        key: :icons,
+        type: {:list, :map},
+        description: "Describe the icons of the app",
+        suggestion: [
+          %{
+            src: "/static/logo.png"
+          },
+          %{
+            src: "/static/icon.png",
+            type: "image/png"
+          },
+          %{
+            src: "/static/icon.ico",
+            sizes: "72x72 96x96 128x128 256x256"
+          }
+        ]
+      },
+      %{
+        key: :theme_color,
+        type: :string,
+        description: "Describe the theme color of the app",
+        suggestions: ["#282c37", "mediumpurple"]
+      },
+      %{
+        key: :background_color,
+        type: :string,
+        description: "Describe the background color of the app",
+        suggestions: ["#191b22", "aliceblue"]
+      }
+    ]
+  },
   %{
     group: :pleroma,
     key: :mrf_simple,

config/releases.exs

@@ -1,6 +1,6 @@
 import Config
 
-config :pleroma, :instance, static_dir: "/var/lib/pleroma/static"
+config :pleroma, :instance, static: "/var/lib/pleroma/static"
 config :pleroma, Pleroma.Uploaders.Local, uploads: "/var/lib/pleroma/uploads"
 
 config_path = System.get_env("PLEROMA_CONFIG_PATH") || "/etc/pleroma/config.exs"

docs/API/admin_api.md

@@ -289,6 +289,14 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 - Response:
   - On success: URL of the unfollowed relay
 
+## `GET /api/pleroma/admin/relay`
+
+### List Relays
+
+- Params: none
+- Response:
+  - On success: JSON array of relays
+
 ## `/api/pleroma/admin/users/invite_token`
 
 ### Create an account registration invite token

docs/API/pleroma_api.md

@@ -367,6 +367,13 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
     * `recipients`: A list of ids of users that should receive posts to this conversation. This will replace the current list of recipients, so submit the full list. The owner of owner of the conversation will always be part of the set of recipients, though.
 * Response: JSON, statuses (200 - healthy, 503 unhealthy)
 
+## `GET /api/v1/pleroma/conversations/read`
+### Marks all user's conversations as read.
+* Method `POST`
+* Authentication: required
+* Params: None
+* Response: JSON, returns a list of Mastodon Conversation entities that were marked as read (200 - healthy, 503 unhealthy).
+
 ## `GET /api/pleroma/emoji/packs`
 ### Lists the custom emoji packs on the server
 * Method `GET`

docs/configuration/cheatsheet.md

@@ -247,6 +247,35 @@ relates to mascots on the mastodon frontend
 * `default_mascot`: An element from `mascots` - This will be used as the default mascot
   on MastoFE (default: `:pleroma_fox_tan`)
 
+## :manifest
+
+This section describe PWA manifest instance-specific values. Currently this option relate only for MastoFE.
+
+* `icons`: Describe the icons of the app, this a list of maps describing icons in the same way as the
+  [spec](https://www.w3.org/TR/appmanifest/#imageresource-and-its-members) describes it.
+
+  Example:
+
+  ```elixir
+  config :pleroma, :manifest,
+    icons: [
+      %{
+        src: "/static/logo.png"
+      },
+      %{
+        src: "/static/icon.png",
+        type: "image/png"
+      },
+      %{
+        src: "/static/icon.ico",
+        sizes: "72x72 96x96 128x128 256x256"
+      }
+    ]
+  ```
+
+* `theme_color`: Describe the theme color of the app. (Example: `"#282c37"`, `"rebeccapurple"`)
+* `background_color`: Describe the background color of the app. (Example: `"#191b22"`, `"aliceblue"`)
+
 ## :mrf_simple
 * `media_removal`: List of instances to remove medias from
 * `media_nsfw`: List of instances to put medias as NSFW(sensitive) from

lib/mix/tasks/pleroma/database.ex

@@ -28,7 +28,7 @@ def run(["remove_embedded_objects" | args]) do
     Logger.info("Removing embedded objects")
 
     Repo.query!(
-      "update activities set data = jsonb_set(data, '{object}'::text[], data->'object'->'id') where data->'object'->>'id' is not null;",
+      "update activities set data = safe_jsonb_set(data, '{object}'::text[], data->'object'->'id') where data->'object'->>'id' is not null;",
       [],
       timeout: :infinity
     )
@@ -126,7 +126,7 @@ def run(["fix_likes_collections"]) do
         set: [
           data:
             fragment(
-              "jsonb_set(?, '{likes}', '[]'::jsonb, true)",
+              "safe_jsonb_set(?, '{likes}', '[]'::jsonb, true)",
               object.data
             )
         ]

lib/mix/tasks/pleroma/emoji.ex

@@ -111,19 +111,21 @@ def run(["get-packs" | args]) do
             file_list: files_to_unzip
           )
 
-        IO.puts(IO.ANSI.format(["Writing emoji.txt for ", :bright, pack_name]))
+        IO.puts(IO.ANSI.format(["Writing pack.json for ", :bright, pack_name]))
 
-        emoji_txt_str =
+        pack_json = %{
-          Enum.map(
+          pack: %{
-            files,
+            "license" => pack["license"],
-            fn {shortcode, path} ->
+            "homepage" => pack["homepage"],
-              emojo_path = Path.join("/emoji/#{pack_name}", path)
+            "description" => pack["description"],
-              "#{shortcode}, #{emojo_path}"
+            "fallback-src" => pack["src"],
-            end
+            "fallback-src-sha256" => pack["src_sha256"],
-          )
+            "share-files" => true
-          |> Enum.join("\n")
+          },
+          files: files
+        }
 
-        File.write!(Path.join(pack_path, "emoji.txt"), emoji_txt_str)
+        File.write!(Path.join(pack_path, "pack.json"), Jason.encode!(pack_json, pretty: true))
       else
         IO.puts(IO.ANSI.format([:bright, :red, "No pack named \"#{pack_name}\" found"]))
       end

lib/mix/tasks/pleroma/relay.ex

@@ -5,7 +5,6 @@
 defmodule Mix.Tasks.Pleroma.Relay do
   use Mix.Task
   import Mix.Pleroma
-  alias Pleroma.User
   alias Pleroma.Web.ActivityPub.Relay
 
   @shortdoc "Manages remote relays"
@@ -36,13 +35,10 @@ def run(["unfollow", target]) do
   def run(["list"]) do
     start_pleroma()
 
-    with %User{following: following} = _user <- Relay.get_actor() do
+    with {:ok, list} <- Relay.list() do
-      following
+      list |> Enum.each(&shell_info(&1))
-      |> Enum.map(fn entry -> URI.parse(entry).host end)
-      |> Enum.uniq()
-      |> Enum.each(&shell_info(&1))
     else
-      e -> shell_error("Error while fetching relay subscription list: #{inspect(e)}")
+      {:error, e} -> shell_error("Error while fetching relay subscription list: #{inspect(e)}")
     end
   end
 end

lib/pleroma/application.ex

@@ -161,11 +161,6 @@ defp task_children(:test) do
         id: :web_push_init,
         start: {Task, :start_link, [&Pleroma.Web.Push.init/0]},
         restart: :temporary
-      },
-      %{
-        id: :federator_init,
-        start: {Task, :start_link, [&Pleroma.Web.Federator.init/0]},
-        restart: :temporary
       }
     ]
   end
@@ -177,11 +172,6 @@ defp task_children(_) do
         start: {Task, :start_link, [&Pleroma.Web.Push.init/0]},
         restart: :temporary
       },
-      %{
-        id: :federator_init,
-        start: {Task, :start_link, [&Pleroma.Web.Federator.init/0]},
-        restart: :temporary
-      },
       %{
         id: :internal_fetch_init,
         start: {Task, :start_link, [&Pleroma.Web.ActivityPub.InternalFetchActor.init/0]},

lib/pleroma/conversation/participation.ex

@@ -69,6 +69,19 @@ def mark_as_read(participation) do
     end
   end
 
+  def mark_all_as_read(user) do
+    {_, participations} =
+      __MODULE__
+      |> where([p], p.user_id == ^user.id)
+      |> where([p], not p.read)
+      |> update([p], set: [read: true])
+      |> select([p], p)
+      |> Repo.update_all([])
+
+    User.set_unread_conversation_count(user)
+    {:ok, participations}
+  end
+
   def mark_as_unread(participation) do
     participation
     |> read_cng(%{read: false})

lib/pleroma/marker.ex

@@ -0,0 +1,74 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Marker do
+  use Ecto.Schema
+
+  import Ecto.Changeset
+  import Ecto.Query
+
+  alias Ecto.Multi
+  alias Pleroma.Repo
+  alias Pleroma.User
+
+  @timelines ["notifications"]
+
+  schema "markers" do
+    field(:last_read_id, :string, default: "")
+    field(:timeline, :string, default: "")
+    field(:lock_version, :integer, default: 0)
+
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
+    timestamps()
+  end
+
+  def get_markers(user, timelines \\ []) do
+    Repo.all(get_query(user, timelines))
+  end
+
+  def upsert(%User{} = user, attrs) do
+    attrs
+    |> Map.take(@timelines)
+    |> Enum.reduce(Multi.new(), fn {timeline, timeline_attrs}, multi ->
+      marker =
+        user
+        |> get_marker(timeline)
+        |> changeset(timeline_attrs)
+
+      Multi.insert(multi, timeline, marker,
+        returning: true,
+        on_conflict: {:replace, [:last_read_id]},
+        conflict_target: [:user_id, :timeline]
+      )
+    end)
+    |> Repo.transaction()
+  end
+
+  defp get_marker(user, timeline) do
+    case Repo.find_resource(get_query(user, timeline)) do
+      {:ok, marker} -> %__MODULE__{marker | user: user}
+      _ -> %__MODULE__{timeline: timeline, user_id: user.id}
+    end
+  end
+
+  @doc false
+  defp changeset(marker, attrs) do
+    marker
+    |> cast(attrs, [:last_read_id])
+    |> validate_required([:user_id, :timeline, :last_read_id])
+    |> validate_inclusion(:timeline, @timelines)
+  end
+
+  defp by_timeline(query, timeline) do
+    from(m in query, where: m.timeline in ^List.wrap(timeline))
+  end
+
+  defp by_user_id(query, id), do: from(m in query, where: m.user_id == ^id)
+
+  defp get_query(user, timelines) do
+    __MODULE__
+    |> by_user_id(user.id)
+    |> by_timeline(timelines)
+  end
+end

lib/pleroma/notification.ex

@@ -55,9 +55,19 @@ def for_user_query(user, opts \\ []) do
     )
     |> preload([n, a, o], activity: {a, object: o})
     |> exclude_muted(user, opts)
+    |> exclude_blocked(user)
     |> exclude_visibility(opts)
   end
 
+  defp exclude_blocked(query, user) do
+    query
+    |> where([n, a], a.actor not in ^user.info.blocks)
+    |> where(
+      [n, a],
+      fragment("substring(? from '.*://([^/]*)')", a.actor) not in ^user.info.domain_blocks
+    )
+  end
+
   defp exclude_muted(query, _, %{with_muted: true}) do
     query
   end
@@ -65,11 +75,6 @@ defp exclude_muted(query, _, %{with_muted: true}) do
   defp exclude_muted(query, user, _opts) do
     query
     |> where([n, a], a.actor not in ^user.info.muted_notifications)
-    |> where([n, a], a.actor not in ^user.info.blocks)
-    |> where(
-      [n, a],
-      fragment("substring(? from '.*://([^/]*)')", a.actor) not in ^user.info.domain_blocks
-    )
     |> join(:left, [n, a], tm in Pleroma.ThreadMute,
       on: tm.user_id == ^user.id and tm.context == fragment("?->>'context'", a.data)
     )

lib/pleroma/object.ex

@@ -181,7 +181,7 @@ def increase_replies_count(ap_id) do
         data:
           fragment(
             """
-            jsonb_set(?, '{repliesCount}',
+            safe_jsonb_set(?, '{repliesCount}',
               (coalesce((?->>'repliesCount')::int, 0) + 1)::varchar::jsonb, true)
             """,
             o.data,
@@ -204,7 +204,7 @@ def decrease_replies_count(ap_id) do
         data:
           fragment(
             """
-            jsonb_set(?, '{repliesCount}',
+            safe_jsonb_set(?, '{repliesCount}',
               (greatest(0, (?->>'repliesCount')::int - 1))::varchar::jsonb, true)
             """,
             o.data,

lib/pleroma/object/containment.ex

@@ -32,6 +32,23 @@ def get_actor(%{"actor" => nil, "attributedTo" => actor}) when not is_nil(actor)
     get_actor(%{"actor" => actor})
   end
 
+  # TODO: We explicitly allow 'tag' URIs through, due to references to legacy OStatus
+  # objects being present in the test suite environment.  Once these objects are
+  # removed, please also remove this.
+  if Mix.env() == :test do
+    defp compare_uris(_, %URI{scheme: "tag"}), do: :ok
+  end
+
+  defp compare_uris(%URI{} = id_uri, %URI{} = other_uri) do
+    if id_uri.host == other_uri.host do
+      :ok
+    else
+      :error
+    end
+  end
+
+  defp compare_uris(_, _), do: :error
+
   @doc """
   Checks that an imported AP object's actor matches the domain it came from.
   """
@@ -41,11 +58,7 @@ def contain_origin(id, %{"actor" => _actor} = params) do
     id_uri = URI.parse(id)
     actor_uri = URI.parse(get_actor(params))
 
-    if id_uri.host == actor_uri.host do
+    compare_uris(actor_uri, id_uri)
-      :ok
-    else
-      :error
-    end
   end
 
   def contain_origin(id, %{"attributedTo" => actor} = params),
@@ -57,11 +70,7 @@ def contain_origin_from_id(id, %{"id" => other_id} = _params) do
     id_uri = URI.parse(id)
     other_uri = URI.parse(other_id)
 
-    if id_uri.host == other_uri.host do
+    compare_uris(id_uri, other_uri)
-      :ok
-    else
-      :error
-    end
   end
 
   def contain_child(%{"object" => %{"id" => id, "attributedTo" => _} = object}),

lib/pleroma/object/fetcher.ex

@@ -10,7 +10,6 @@ defmodule Pleroma.Object.Fetcher do
   alias Pleroma.Signature
   alias Pleroma.Web.ActivityPub.InternalFetchActor
   alias Pleroma.Web.ActivityPub.Transmogrifier
-  alias Pleroma.Web.OStatus
 
   require Logger
   require Pleroma.Constants
@@ -67,7 +66,8 @@ def fetch_object_from_id(id, options \\ []) do
          {:normalize, nil} <- {:normalize, Object.normalize(data, false)},
          params <- prepare_activity_params(data),
          {:containment, :ok} <- {:containment, Containment.contain_origin(id, params)},
-         {:ok, activity} <- Transmogrifier.handle_incoming(params, options),
+         {:transmogrifier, {:ok, activity}} <-
+           {:transmogrifier, Transmogrifier.handle_incoming(params, options)},
          {:object, _data, %Object{} = object} <-
            {:object, data, Object.normalize(activity, false)} do
       {:ok, object}
@@ -75,9 +75,12 @@ def fetch_object_from_id(id, options \\ []) do
       {:containment, _} ->
         {:error, "Object containment failed."}
 
-      {:error, {:reject, nil}} ->
+      {:transmogrifier, {:error, {:reject, nil}}} ->
         {:reject, nil}
 
+      {:transmogrifier, _} ->
+        {:error, "Transmogrifier failure."}
+
       {:object, data, nil} ->
         reinject_object(%Object{}, data)
 
@@ -87,15 +90,8 @@ def fetch_object_from_id(id, options \\ []) do
       {:fetch_object, %Object{} = object} ->
         {:ok, object}
 
-      _e ->
+      e ->
-        # Only fallback when receiving a fetch/normalization error with ActivityPub
+        e
-        Logger.info("Couldn't get object via AP, trying out OStatus fetching...")
-
-        # FIXME: OStatus Object Containment?
-        case OStatus.fetch_activity_from_url(id) do
-          {:ok, [activity | _]} -> {:ok, Object.normalize(activity, false)}
-          e -> e
-        end
     end
   end
 
@@ -114,7 +110,8 @@ def fetch_object_from_id!(id, options \\ []) do
     with {:ok, object} <- fetch_object_from_id(id, options) do
       object
     else
-      _e ->
+      e ->
+        Logger.error("Error while fetching #{id}: #{inspect(e)}")
         nil
     end
   end
@@ -161,7 +158,7 @@ def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
 
     Logger.debug("Fetch headers: #{inspect(headers)}")
 
-    with true <- String.starts_with?(id, "http"),
+    with {:scheme, true} <- {:scheme, String.starts_with?(id, "http")},
          {:ok, %{body: body, status: code}} when code in 200..299 <- HTTP.get(id, headers),
          {:ok, data} <- Jason.decode(body),
          :ok <- Containment.contain_origin_from_id(id, data) do
@@ -170,6 +167,9 @@ def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
       {:ok, %{status: code}} when code in [404, 410] ->
         {:error, "Object has been deleted"}
 
+      {:scheme, _} ->
+        {:error, "Unsupported URI scheme"}
+
       e ->
         {:error, e}
     end

lib/pleroma/user.ex

@@ -26,9 +26,7 @@ defmodule Pleroma.User do
   alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.CommonAPI.Utils, as: CommonUtils
   alias Pleroma.Web.OAuth
-  alias Pleroma.Web.OStatus
   alias Pleroma.Web.RelMe
-  alias Pleroma.Web.Websub
   alias Pleroma.Workers.BackgroundWorker
 
   require Logger
@@ -90,6 +88,9 @@ def superuser?(%User{local: true, info: %User.Info{is_admin: true}}), do: true
   def superuser?(%User{local: true, info: %User.Info{is_moderator: true}}), do: true
   def superuser?(_), do: false
 
+  def invisible?(%User{info: %User.Info{invisible: true}}), do: true
+  def invisible?(_), do: false
+
   def avatar_url(user, options \\ []) do
     case user.avatar do
       %{"url" => [%{"href" => href} | _]} -> href
@@ -437,12 +438,6 @@ def follow(%User{} = follower, %User{info: info} = followed) do
         {:error, "Could not follow user: #{followed.nickname} blocked you."}
 
       true ->
-        benchmark? = Pleroma.Config.get([:env]) == :benchmark
-
-        if !followed.local && follower.local && !ap_enabled?(followed) && !benchmark? do
-          Websub.subscribe(follower, followed)
-        end
-
         q =
           from(u in User,
             where: u.id == ^follower.id,
@@ -616,12 +611,7 @@ def get_cached_user_info(user) do
     Cachex.fetch!(:user_cache, key, fn -> user_info(user) end)
   end
 
-  def fetch_by_nickname(nickname) do
+  def fetch_by_nickname(nickname), do: ActivityPub.make_user_from_nickname(nickname)
-    case ActivityPub.make_user_from_nickname(nickname) do
-      {:ok, user} -> {:ok, user}
-      _ -> OStatus.make_user(nickname)
-    end
-  end
 
   def get_or_fetch_by_nickname(nickname) do
     with %User{} = user <- get_by_nickname(nickname) do
@@ -727,7 +717,7 @@ def increase_note_count(%User{} = user) do
       set: [
         info:
           fragment(
-            "jsonb_set(?, '{note_count}', ((?->>'note_count')::int + 1)::varchar::jsonb, true)",
+            "safe_jsonb_set(?, '{note_count}', ((?->>'note_count')::int + 1)::varchar::jsonb, true)",
             u.info,
             u.info
           )
@@ -748,7 +738,7 @@ def decrease_note_count(%User{} = user) do
       set: [
         info:
           fragment(
-            "jsonb_set(?, '{note_count}', (greatest(0, (?->>'note_count')::int - 1))::varchar::jsonb, true)",
+            "safe_jsonb_set(?, '{note_count}', (greatest(0, (?->>'note_count')::int - 1))::varchar::jsonb, true)",
             u.info,
             u.info
           )
@@ -818,7 +808,7 @@ def update_follower_count(%User{} = user) do
         set: [
           info:
             fragment(
-              "jsonb_set(?, '{follower_count}', ?::varchar::jsonb, true)",
+              "safe_jsonb_set(?, '{follower_count}', ?::varchar::jsonb, true)",
               u.info,
               s.count
             )
@@ -1248,18 +1238,7 @@ def html_filter_policy(%User{info: %{no_rich_text: true}}) do
 
   def html_filter_policy(_), do: Pleroma.Config.get([:markup, :scrub_policy])
 
-  def fetch_by_ap_id(ap_id) do
+  def fetch_by_ap_id(ap_id), do: ActivityPub.make_user_from_ap_id(ap_id)
-    case ActivityPub.make_user_from_ap_id(ap_id) do
-      {:ok, user} ->
-        {:ok, user}
-
-      _ ->
-        case OStatus.make_user(ap_id) do
-          {:ok, user} -> {:ok, user}
-          _ -> {:error, "Could not fetch by AP id"}
-        end
-    end
-  end
 
   def get_or_fetch_by_ap_id(ap_id) do
     user = get_cached_by_ap_id(ap_id)
@@ -1314,11 +1293,6 @@ def public_key_from_info(%{
     {:ok, key}
   end
 
-  # OStatus Magic Key
-  def public_key_from_info(%{magic_key: magic_key}) when not is_nil(magic_key) do
-    {:ok, Pleroma.Web.Salmon.decode_key(magic_key)}
-  end
-
   def public_key_from_info(_), do: {:error, "not found key"}
 
   def get_public_key_for_ap_id(ap_id) do

lib/pleroma/user/info.ex

@@ -39,9 +39,6 @@ defmodule Pleroma.User.Info do
     field(:settings, :map, default: nil)
     field(:magic_key, :string, default: nil)
     field(:uri, :string, default: nil)
-    field(:topic, :string, default: nil)
-    field(:hub, :string, default: nil)
-    field(:salmon, :string, default: nil)
     field(:hide_followers_count, :boolean, default: false)
     field(:hide_follows_count, :boolean, default: false)
     field(:hide_followers, :boolean, default: false)
@@ -56,6 +53,7 @@ defmodule Pleroma.User.Info do
     field(:fields, {:array, :map}, default: nil)
     field(:raw_fields, {:array, :map}, default: [])
     field(:discoverable, :boolean, default: false)
+    field(:invisible, :boolean, default: false)
 
     field(:notification_settings, :map,
       default: %{
@@ -262,9 +260,6 @@ def remote_user_creation(info, params) do
       :locked,
       :magic_key,
       :uri,
-      :hub,
-      :topic,
-      :salmon,
       :hide_followers,
       :hide_follows,
       :hide_followers_count,
@@ -272,7 +267,8 @@ def remote_user_creation(info, params) do
       :follower_count,
       :fields,
       :following_count,
-      :discoverable
+      :discoverable,
+      :invisible
     ])
     |> validate_fields(true)
   end
@@ -399,6 +395,14 @@ def set_source_data(info, source_data) do
     |> validate_required([:source_data])
   end
 
+  def set_invisible(info, invisible) do
+    params = %{invisible: invisible}
+
+    info
+    |> cast(params, [:invisible])
+    |> validate_required([:invisible])
+  end
+
   def admin_api_update(info, params) do
     info
     |> cast(params, [

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -132,7 +132,7 @@ def insert(map, local \\ true, fake \\ false, bypass_actor_check \\ false) when
          {:ok, map} <- MRF.filter(map),
          {recipients, _, _} = get_recipients(map),
          {:fake, false, map, recipients} <- {:fake, fake, map, recipients},
-         :ok <- Containment.contain_child(map),
+         {:containment, :ok} <- {:containment, Containment.contain_child(map)},
          {:ok, map, object} <- insert_full_object(map) do
       {:ok, activity} =
         Repo.insert(%Activity{
@@ -1106,6 +1106,7 @@ defp object_to_user_data(data) do
     locked = data["manuallyApprovesFollowers"] || false
     data = Transmogrifier.maybe_fix_user_object(data)
     discoverable = data["discoverable"] || false
+    invisible = data["invisible"] || false
 
     user_data = %{
       ap_id: data["id"],
@@ -1115,7 +1116,8 @@ defp object_to_user_data(data) do
         banner: banner,
         fields: fields,
         locked: locked,
-        discoverable: discoverable
+        discoverable: discoverable,
+        invisible: invisible
       },
       avatar: avatar,
       name: data["name"],
@@ -1219,7 +1221,9 @@ def fetch_and_prepare_user_from_ap_id(ap_id) do
          data <- maybe_update_follow_information(data) do
       {:ok, data}
     else
-      e -> Logger.error("Could not decode user at fetch #{ap_id}, #{inspect(e)}")
+      e ->
+        Logger.error("Could not decode user at fetch #{ap_id}, #{inspect(e)}")
+        {:error, e}
     end
   end
 

lib/pleroma/web/activity_pub/publisher.ex

@@ -129,7 +129,7 @@ defp recipients(actor, activity) do
           []
       end
 
-    Pleroma.Web.Salmon.remote_users(actor, activity) ++ followers ++ fetchers
+    Pleroma.Web.Federator.Publisher.remote_users(actor, activity) ++ followers ++ fetchers
   end
 
   defp get_cc_ap_ids(ap_id, recipients) do

lib/pleroma/web/activity_pub/relay.ex

@@ -10,8 +10,12 @@ defmodule Pleroma.Web.ActivityPub.Relay do
   require Logger
 
   def get_actor do
+    actor =
       "#{Pleroma.Web.Endpoint.url()}/relay"
       |> User.get_or_create_service_actor_by_ap_id()
+
+    {:ok, actor} = User.update_info(actor, &User.Info.set_invisible(&1, true))
+    actor
   end
 
   @spec follow(String.t()) :: {:ok, Activity.t()} | {:error, any()}
@@ -51,6 +55,20 @@ def publish(%Activity{data: %{"type" => "Create"}} = activity) do
 
   def publish(_), do: {:error, "Not implemented"}
 
+  @spec list() :: {:ok, [String.t()]} | {:error, any()}
+  def list do
+    with %User{following: following} = _user <- get_actor() do
+      list =
+        following
+        |> Enum.map(fn entry -> URI.parse(entry).host end)
+        |> Enum.uniq()
+
+      {:ok, list}
+    else
+      error -> format_error(error)
+    end
+  end
+
   defp format_error({:error, error}), do: format_error(error)
 
   defp format_error(error) do

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -596,13 +596,19 @@ def handle_incoming(
           data,
         _options
       )
-      when object_type in ["Person", "Application", "Service", "Organization"] do
+      when object_type in [
+             "Person",
+             "Application",
+             "Service",
+             "Organization"
+           ] do
     with %User{ap_id: ^actor_id} = actor <- User.get_cached_by_ap_id(object["id"]) do
       {:ok, new_user_data} = ActivityPub.user_data_from_user_object(object)
 
       banner = new_user_data[:info][:banner]
       locked = new_user_data[:info][:locked] || false
       attachment = get_in(new_user_data, [:info, :source_data, "attachment"]) || []
+      invisible = new_user_data[:info][:invisible] || false
 
       fields =
         attachment
@@ -612,7 +618,7 @@ def handle_incoming(
       update_data =
         new_user_data
         |> Map.take([:name, :bio, :avatar])
-        |> Map.put(:info, %{banner: banner, locked: locked, fields: fields})
+        |> Map.put(:info, %{banner: banner, locked: locked, fields: fields, invisible: invisible})
 
       actor
       |> User.upgrade_changeset(update_data, true)
@@ -1073,8 +1079,6 @@ def perform(:user_upgrade, user) do
 
     Repo.update_all(q, [])
 
-    maybe_retire_websub(user.ap_id)
-
     q =
       from(
         a in Activity,
@@ -1117,19 +1121,6 @@ defp upgrade_user(user, data) do
     |> User.update_and_set_cache()
   end
 
-  def maybe_retire_websub(ap_id) do
-    # some sanity checks
-    if is_binary(ap_id) && String.length(ap_id) > 8 do
-      q =
-        from(
-          ws in Pleroma.Web.Websub.WebsubClientSubscription,
-          where: fragment("? like ?", ws.topic, ^"#{ap_id}%")
-        )
-
-      Repo.delete_all(q)
-    end
-  end
-
   def maybe_fix_user_url(%{"url" => url} = data) when is_map(url) do
     Map.put(data, "url", url["href"])
   end

lib/pleroma/web/activity_pub/utils.ex

@@ -491,11 +491,15 @@ def add_announce_to_object(
         %Activity{data: %{"actor" => actor}},
         object
       ) do
+    unless actor |> User.get_cached_by_ap_id() |> User.invisible?() do
       announcements = take_announcements(object)
 
       with announcements <- Enum.uniq([actor | announcements]) do
         update_element_in_object("announcement", announcements, object)
       end
+    else
+      {:ok, object}
+    end
   end
 
   def add_announce_to_object(_, object), do: {:ok, object}

lib/pleroma/web/activity_pub/views/user_view.ex

@@ -55,7 +55,8 @@ def render("service.json", %{user: user}) do
         "owner" => user.ap_id,
         "publicKeyPem" => public_key
       },
-      "endpoints" => endpoints
+      "endpoints" => endpoints,
+      "invisible" => User.invisible?(user)
     }
     |> Map.merge(Utils.make_json_ld_header())
   end

lib/pleroma/web/admin_api/admin_api_controller.ex

@@ -481,6 +481,16 @@ def right_delete(%{assigns: %{user: %{nickname: nickname}}} = conn, %{"nickname"
     render_error(conn, :forbidden, "You can't revoke your own admin status.")
   end
 
+  def relay_list(conn, _params) do
+    with {:ok, list} <- Relay.list() do
+      json(conn, %{relays: list})
+    else
+      _ ->
+        conn
+        |> put_status(500)
+    end
+  end
+
   def relay_follow(%{assigns: %{user: admin}} = conn, %{"relay_url" => target}) do
     with {:ok, _message} <- Relay.follow(target) do
       ModerationLog.insert_log(%{

lib/pleroma/web/federator/federator.ex

@@ -10,19 +10,11 @@ defmodule Pleroma.Web.Federator do
   alias Pleroma.Web.ActivityPub.Transmogrifier
   alias Pleroma.Web.ActivityPub.Utils
   alias Pleroma.Web.Federator.Publisher
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.Websub
   alias Pleroma.Workers.PublisherWorker
   alias Pleroma.Workers.ReceiverWorker
-  alias Pleroma.Workers.SubscriberWorker
 
   require Logger
 
-  def init do
-    # To do: consider removing this call in favor of scheduled execution (`quantum`-based)
-    refresh_subscriptions(schedule_in: 60)
-  end
-
   @doc "Addresses [memory leaks on recursive replies fetching](https://git.pleroma.social/pleroma/pleroma/issues/161)"
   # credo:disable-for-previous-line Credo.Check.Readability.MaxLineLength
   def allowed_incoming_reply_depth?(depth) do
@@ -37,10 +29,6 @@ def allowed_incoming_reply_depth?(depth) do
 
   # Client API
 
-  def incoming_doc(doc) do
-    ReceiverWorker.enqueue("incoming_doc", %{"body" => doc})
-  end
-
   def incoming_ap_doc(params) do
     ReceiverWorker.enqueue("incoming_ap_doc", %{"params" => params})
   end
@@ -53,18 +41,6 @@ def publish(activity) do
     PublisherWorker.enqueue("publish", %{"activity_id" => activity.id})
   end
 
-  def verify_websub(websub) do
-    SubscriberWorker.enqueue("verify_websub", %{"websub_id" => websub.id})
-  end
-
-  def request_subscription(websub) do
-    SubscriberWorker.enqueue("request_subscription", %{"websub_id" => websub.id})
-  end
-
-  def refresh_subscriptions(worker_args \\ []) do
-    SubscriberWorker.enqueue("refresh_subscriptions", %{}, worker_args ++ [max_attempts: 1])
-  end
-
   # Job Worker Callbacks
 
   @spec perform(atom(), module(), any()) :: {:ok, any()} | {:error, any()}
@@ -81,11 +57,6 @@ def perform(:publish, activity) do
     end
   end
 
-  def perform(:incoming_doc, doc) do
-    Logger.info("Got document, trying to parse")
-    OStatus.handle_incoming(doc)
-  end
-
   def perform(:incoming_ap_doc, params) do
     Logger.info("Handling incoming AP activity")
 
@@ -111,29 +82,6 @@ def perform(:incoming_ap_doc, params) do
     end
   end
 
-  def perform(:request_subscription, websub) do
-    Logger.debug("Refreshing #{websub.topic}")
-
-    with {:ok, websub} <- Websub.request_subscription(websub) do
-      Logger.debug("Successfully refreshed #{websub.topic}")
-    else
-      _e -> Logger.debug("Couldn't refresh #{websub.topic}")
-    end
-  end
-
-  def perform(:verify_websub, websub) do
-    Logger.debug(fn ->
-      "Running WebSub verification for #{websub.id} (#{websub.topic}, #{websub.callback})"
-    end)
-
-    Websub.verify(websub)
-  end
-
-  def perform(:refresh_subscriptions) do
-    Logger.debug("Federator running refresh subscriptions")
-    Websub.refresh_subscriptions()
-  end
-
   def ap_enabled_actor(id) do
     user = User.get_cached_by_ap_id(id)
 

lib/pleroma/web/federator/publisher.ex

@@ -80,4 +80,30 @@ def gather_nodeinfo_protocol_names do
       links ++ module.gather_nodeinfo_protocol_names()
     end)
   end
+
+  @doc """
+  Gathers a set of remote users given an IR envelope.
+  """
+  def remote_users(%User{id: user_id}, %{data: %{"to" => to} = data}) do
+    cc = Map.get(data, "cc", [])
+
+    bcc =
+      data
+      |> Map.get("bcc", [])
+      |> Enum.reduce([], fn ap_id, bcc ->
+        case Pleroma.List.get_by_ap_id(ap_id) do
+          %Pleroma.List{user_id: ^user_id} = list ->
+            {:ok, following} = Pleroma.List.get_following(list)
+            bcc ++ Enum.map(following, & &1.ap_id)
+
+          _ ->
+            bcc
+        end
+      end)
+
+    [to, cc, bcc]
+    |> Enum.concat()
+    |> Enum.map(&User.get_cached_by_ap_id/1)
+    |> Enum.filter(fn user -> user && !user.local end)
+  end
 end

lib/pleroma/web/masto_fe_controller.ex

@@ -34,6 +34,12 @@ def index(%{assigns: %{user: user}} = conn, _params) do
     end
   end
 
+  @doc "GET /web/manifest.json"
+  def manifest(conn, _params) do
+    conn
+    |> render("manifest.json")
+  end
+
   @doc "PUT /api/web/settings"
   def put_settings(%{assigns: %{user: user}} = conn, %{"data" => settings} = _params) do
     with {:ok, _} <- User.update_info(user, &User.Info.mastodon_settings_update(&1, settings)) do

lib/pleroma/web/mastodon_api/controllers/marker_controller.ex

@@ -0,0 +1,32 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.MarkerController do
+  use Pleroma.Web, :controller
+  alias Pleroma.Plugs.OAuthScopesPlug
+
+  plug(
+    OAuthScopesPlug,
+    %{scopes: ["read:statuses"]}
+    when action == :index
+  )
+
+  plug(OAuthScopesPlug, %{scopes: ["write:statuses"]} when action == :upsert)
+  plug(Pleroma.Plugs.EnsurePublicOrAuthenticatedPlug)
+  action_fallback(Pleroma.Web.MastodonAPI.FallbackController)
+
+  # GET /api/v1/markers
+  def index(%{assigns: %{user: user}} = conn, params) do
+    markers = Pleroma.Marker.get_markers(user, params["timeline"])
+    render(conn, "markers.json", %{markers: markers})
+  end
+
+  # POST /api/v1/markers
+  def upsert(%{assigns: %{user: user}} = conn, params) do
+    with {:ok, result} <- Pleroma.Marker.upsert(user, params),
+         markers <- Map.values(result) do
+      render(conn, "markers.json", %{markers: markers})
+    end
+  end
+end

lib/pleroma/web/mastodon_api/views/marker_view.ex

@@ -0,0 +1,17 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.MarkerView do
+  use Pleroma.Web, :view
+
+  def render("markers.json", %{markers: markers}) do
+    Enum.reduce(markers, %{}, fn m, acc ->
+      Map.put_new(acc, m.timeline, %{
+        last_read_id: m.last_read_id,
+        version: m.lock_version,
+        updated_at: NaiveDateTime.to_iso8601(m.updated_at)
+      })
+    end)
+  end
+end

lib/pleroma/web/ostatus/activity_representer.ex

@@ -1,313 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.ActivityRepresenter do
-  alias Pleroma.Activity
-  alias Pleroma.Object
-  alias Pleroma.User
-  alias Pleroma.Web.OStatus.UserRepresenter
-
-  require Logger
-  require Pleroma.Constants
-
-  defp get_href(id) do
-    with %Object{data: %{"external_url" => external_url}} <- Object.get_cached_by_ap_id(id) do
-      external_url
-    else
-      _e -> id
-    end
-  end
-
-  defp get_in_reply_to(activity) do
-    with %Object{data: %{"inReplyTo" => in_reply_to}} <- Object.normalize(activity) do
-      [
-        {:"thr:in-reply-to",
-         [ref: to_charlist(in_reply_to), href: to_charlist(get_href(in_reply_to))], []}
-      ]
-    else
-      _ ->
-        []
-    end
-  end
-
-  defp get_mentions(to) do
-    Enum.map(to, fn id ->
-      cond do
-        # Special handling for the AP/Ostatus public collections
-        Pleroma.Constants.as_public() == id ->
-          {:link,
-           [
-             rel: "mentioned",
-             "ostatus:object-type": "http://activitystrea.ms/schema/1.0/collection",
-             href: "http://activityschema.org/collection/public"
-           ], []}
-
-        # Ostatus doesn't handle follower collections, ignore these.
-        Regex.match?(~r/^#{Pleroma.Web.base_url()}.+followers$/, id) ->
-          []
-
-        true ->
-          {:link,
-           [
-             rel: "mentioned",
-             "ostatus:object-type": "http://activitystrea.ms/schema/1.0/person",
-             href: id
-           ], []}
-      end
-    end)
-  end
-
-  defp get_links(%{local: true}, %{"id" => object_id}) do
-    h = fn str -> [to_charlist(str)] end
-
-    [
-      {:link, [type: ['application/atom+xml'], href: h.(object_id), rel: 'self'], []},
-      {:link, [type: ['text/html'], href: h.(object_id), rel: 'alternate'], []}
-    ]
-  end
-
-  defp get_links(%{local: false}, %{"external_url" => external_url}) do
-    h = fn str -> [to_charlist(str)] end
-
-    [
-      {:link, [type: ['text/html'], href: h.(external_url), rel: 'alternate'], []}
-    ]
-  end
-
-  defp get_links(_activity, _object_data), do: []
-
-  defp get_emoji_links(emojis) do
-    Enum.map(emojis, fn {emoji, file} ->
-      {:link, [name: to_charlist(emoji), rel: 'emoji', href: to_charlist(file)], []}
-    end)
-  end
-
-  def to_simple_form(activity, user, with_author \\ false)
-
-  def to_simple_form(%{data: %{"type" => "Create"}} = activity, user, with_author) do
-    h = fn str -> [to_charlist(str)] end
-
-    object = Object.normalize(activity)
-
-    updated_at = object.data["published"]
-    inserted_at = object.data["published"]
-
-    attachments =
-      Enum.map(object.data["attachment"] || [], fn attachment ->
-        url = hd(attachment["url"])
-
-        {:link,
-         [rel: 'enclosure', href: to_charlist(url["href"]), type: to_charlist(url["mediaType"])],
-         []}
-      end)
-
-    in_reply_to = get_in_reply_to(activity)
-    author = if with_author, do: [{:author, UserRepresenter.to_simple_form(user)}], else: []
-    mentions = activity.recipients |> get_mentions
-
-    categories =
-      (object.data["tag"] || [])
-      |> Enum.map(fn tag ->
-        if is_binary(tag) do
-          {:category, [term: to_charlist(tag)], []}
-        else
-          nil
-        end
-      end)
-      |> Enum.filter(& &1)
-
-    emoji_links = get_emoji_links(object.data["emoji"] || %{})
-
-    summary =
-      if object.data["summary"] do
-        [{:summary, [], h.(object.data["summary"])}]
-      else
-        []
-      end
-
-    [
-      {:"activity:object-type", ['http://activitystrea.ms/schema/1.0/note']},
-      {:"activity:verb", ['http://activitystrea.ms/schema/1.0/post']},
-      # For notes, federate the object id.
-      {:id, h.(object.data["id"])},
-      {:title, ['New note by #{user.nickname}']},
-      {:content, [type: 'html'], h.(object.data["content"] |> String.replace(~r/[\n\r]/, ""))},
-      {:published, h.(inserted_at)},
-      {:updated, h.(updated_at)},
-      {:"ostatus:conversation", [ref: h.(activity.data["context"])],
-       h.(activity.data["context"])},
-      {:link, [ref: h.(activity.data["context"]), rel: 'ostatus:conversation'], []}
-    ] ++
-      summary ++
-      get_links(activity, object.data) ++
-      categories ++ attachments ++ in_reply_to ++ author ++ mentions ++ emoji_links
-  end
-
-  def to_simple_form(%{data: %{"type" => "Like"}} = activity, user, with_author) do
-    h = fn str -> [to_charlist(str)] end
-
-    updated_at = activity.data["published"]
-    inserted_at = activity.data["published"]
-
-    author = if with_author, do: [{:author, UserRepresenter.to_simple_form(user)}], else: []
-    mentions = activity.recipients |> get_mentions
-
-    [
-      {:"activity:verb", ['http://activitystrea.ms/schema/1.0/favorite']},
-      {:id, h.(activity.data["id"])},
-      {:title, ['New favorite by #{user.nickname}']},
-      {:content, [type: 'html'], ['#{user.nickname} favorited something']},
-      {:published, h.(inserted_at)},
-      {:updated, h.(updated_at)},
-      {:"activity:object",
-       [
-         {:"activity:object-type", ['http://activitystrea.ms/schema/1.0/note']},
-         # For notes, federate the object id.
-         {:id, h.(activity.data["object"])}
-       ]},
-      {:"ostatus:conversation", [ref: h.(activity.data["context"])],
-       h.(activity.data["context"])},
-      {:link, [ref: h.(activity.data["context"]), rel: 'ostatus:conversation'], []},
-      {:link, [rel: 'self', type: ['application/atom+xml'], href: h.(activity.data["id"])], []},
-      {:"thr:in-reply-to", [ref: to_charlist(activity.data["object"])], []}
-    ] ++ author ++ mentions
-  end
-
-  def to_simple_form(%{data: %{"type" => "Announce"}} = activity, user, with_author) do
-    h = fn str -> [to_charlist(str)] end
-
-    updated_at = activity.data["published"]
-    inserted_at = activity.data["published"]
-
-    author = if with_author, do: [{:author, UserRepresenter.to_simple_form(user)}], else: []
-
-    retweeted_activity = Activity.get_create_by_object_ap_id(activity.data["object"])
-    retweeted_object = Object.normalize(retweeted_activity)
-    retweeted_user = User.get_cached_by_ap_id(retweeted_activity.data["actor"])
-
-    retweeted_xml = to_simple_form(retweeted_activity, retweeted_user, true)
-
-    mentions =
-      ([retweeted_user.ap_id] ++ activity.recipients)
-      |> Enum.uniq()
-      |> get_mentions()
-
-    [
-      {:"activity:object-type", ['http://activitystrea.ms/schema/1.0/activity']},
-      {:"activity:verb", ['http://activitystrea.ms/schema/1.0/share']},
-      {:id, h.(activity.data["id"])},
-      {:title, ['#{user.nickname} repeated a notice']},
-      {:content, [type: 'html'], ['RT #{retweeted_object.data["content"]}']},
-      {:published, h.(inserted_at)},
-      {:updated, h.(updated_at)},
-      {:"ostatus:conversation", [ref: h.(activity.data["context"])],
-       h.(activity.data["context"])},
-      {:link, [ref: h.(activity.data["context"]), rel: 'ostatus:conversation'], []},
-      {:link, [rel: 'self', type: ['application/atom+xml'], href: h.(activity.data["id"])], []},
-      {:"activity:object", retweeted_xml}
-    ] ++ mentions ++ author
-  end
-
-  def to_simple_form(%{data: %{"type" => "Follow"}} = activity, user, with_author) do
-    h = fn str -> [to_charlist(str)] end
-
-    updated_at = activity.data["published"]
-    inserted_at = activity.data["published"]
-
-    author = if with_author, do: [{:author, UserRepresenter.to_simple_form(user)}], else: []
-
-    mentions = (activity.recipients || []) |> get_mentions
-
-    [
-      {:"activity:object-type", ['http://activitystrea.ms/schema/1.0/activity']},
-      {:"activity:verb", ['http://activitystrea.ms/schema/1.0/follow']},
-      {:id, h.(activity.data["id"])},
-      {:title, ['#{user.nickname} started following #{activity.data["object"]}']},
-      {:content, [type: 'html'],
-       ['#{user.nickname} started following #{activity.data["object"]}']},
-      {:published, h.(inserted_at)},
-      {:updated, h.(updated_at)},
-      {:"activity:object",
-       [
-         {:"activity:object-type", ['http://activitystrea.ms/schema/1.0/person']},
-         {:id, h.(activity.data["object"])},
-         {:uri, h.(activity.data["object"])}
-       ]},
-      {:link, [rel: 'self', type: ['application/atom+xml'], href: h.(activity.data["id"])], []}
-    ] ++ mentions ++ author
-  end
-
-  # Only undos of follow for now. Will need to get redone once there are more
-  def to_simple_form(
-        %{data: %{"type" => "Undo", "object" => %{"type" => "Follow"} = follow_activity}} =
-          activity,
-        user,
-        with_author
-      ) do
-    h = fn str -> [to_charlist(str)] end
-
-    updated_at = activity.data["published"]
-    inserted_at = activity.data["published"]
-
-    author = if with_author, do: [{:author, UserRepresenter.to_simple_form(user)}], else: []
-
-    mentions = (activity.recipients || []) |> get_mentions
-    follow_activity = Activity.normalize(follow_activity)
-
-    [
-      {:"activity:object-type", ['http://activitystrea.ms/schema/1.0/activity']},
-      {:"activity:verb", ['http://activitystrea.ms/schema/1.0/unfollow']},
-      {:id, h.(activity.data["id"])},
-      {:title, ['#{user.nickname} stopped following #{follow_activity.data["object"]}']},
-      {:content, [type: 'html'],
-       ['#{user.nickname} stopped following #{follow_activity.data["object"]}']},
-      {:published, h.(inserted_at)},
-      {:updated, h.(updated_at)},
-      {:"activity:object",
-       [
-         {:"activity:object-type", ['http://activitystrea.ms/schema/1.0/person']},
-         {:id, h.(follow_activity.data["object"])},
-         {:uri, h.(follow_activity.data["object"])}
-       ]},
-      {:link, [rel: 'self', type: ['application/atom+xml'], href: h.(activity.data["id"])], []}
-    ] ++ mentions ++ author
-  end
-
-  def to_simple_form(%{data: %{"type" => "Delete"}} = activity, user, with_author) do
-    h = fn str -> [to_charlist(str)] end
-
-    updated_at = activity.data["published"]
-    inserted_at = activity.data["published"]
-
-    author = if with_author, do: [{:author, UserRepresenter.to_simple_form(user)}], else: []
-
-    [
-      {:"activity:object-type", ['http://activitystrea.ms/schema/1.0/activity']},
-      {:"activity:verb", ['http://activitystrea.ms/schema/1.0/delete']},
-      {:id, h.(activity.data["object"])},
-      {:title, ['An object was deleted']},
-      {:content, [type: 'html'], ['An object was deleted']},
-      {:published, h.(inserted_at)},
-      {:updated, h.(updated_at)}
-    ] ++ author
-  end
-
-  def to_simple_form(_, _, _), do: nil
-
-  def wrap_with_entry(simple_form) do
-    [
-      {
-        :entry,
-        [
-          xmlns: 'http://www.w3.org/2005/Atom',
-          "xmlns:thr": 'http://purl.org/syndication/thread/1.0',
-          "xmlns:activity": 'http://activitystrea.ms/spec/1.0/',
-          "xmlns:poco": 'http://portablecontacts.net/spec/1.0',
-          "xmlns:ostatus": 'http://ostatus.org/schema/1.0'
-        ],
-        simple_form
-      }
-    ]
-  end
-end

lib/pleroma/web/ostatus/feed_representer.ex

@@ -1,66 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.FeedRepresenter do
-  alias Pleroma.User
-  alias Pleroma.Web.MediaProxy
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.OStatus.ActivityRepresenter
-  alias Pleroma.Web.OStatus.UserRepresenter
-
-  def to_simple_form(user, activities, _users) do
-    most_recent_update =
-      (List.first(activities) || user).updated_at
-      |> NaiveDateTime.to_iso8601()
-
-    h = fn str -> [to_charlist(str)] end
-
-    last_activity = List.last(activities)
-
-    entries =
-      activities
-      |> Enum.map(fn activity ->
-        {:entry, ActivityRepresenter.to_simple_form(activity, user)}
-      end)
-      |> Enum.filter(fn {_, form} -> form end)
-
-    [
-      {
-        :feed,
-        [
-          xmlns: 'http://www.w3.org/2005/Atom',
-          "xmlns:thr": 'http://purl.org/syndication/thread/1.0',
-          "xmlns:activity": 'http://activitystrea.ms/spec/1.0/',
-          "xmlns:poco": 'http://portablecontacts.net/spec/1.0',
-          "xmlns:ostatus": 'http://ostatus.org/schema/1.0'
-        ],
-        [
-          {:id, h.(OStatus.feed_path(user))},
-          {:title, ['#{user.nickname}\'s timeline']},
-          {:updated, h.(most_recent_update)},
-          {:logo, [to_charlist(User.avatar_url(user) |> MediaProxy.url())]},
-          {:link, [rel: 'hub', href: h.(OStatus.pubsub_path(user))], []},
-          {:link, [rel: 'salmon', href: h.(OStatus.salmon_path(user))], []},
-          {:link, [rel: 'self', href: h.(OStatus.feed_path(user)), type: 'application/atom+xml'],
-           []},
-          {:author, UserRepresenter.to_simple_form(user)}
-        ] ++
-          if last_activity do
-            [
-              {:link,
-               [
-                 rel: 'next',
-                 href:
-                   to_charlist(OStatus.feed_path(user)) ++
-                     '?max_id=' ++ to_charlist(last_activity.id),
-                 type: 'application/atom+xml'
-               ], []}
-            ]
-          else
-            []
-          end ++ entries
-      }
-    ]
-  end
-end

lib/pleroma/web/ostatus/handlers/delete_handler.ex

@@ -1,18 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.DeleteHandler do
-  require Logger
-  alias Pleroma.Object
-  alias Pleroma.Web.ActivityPub.ActivityPub
-  alias Pleroma.Web.XML
-
-  def handle_delete(entry, _doc \\ nil) do
-    with id <- XML.string_from_xpath("//id", entry),
-         %Object{} = object <- Object.normalize(id),
-         {:ok, delete} <- ActivityPub.delete(object, local: false) do
-      delete
-    end
-  end
-end

lib/pleroma/web/ostatus/handlers/follow_handler.ex

@@ -1,26 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.FollowHandler do
-  alias Pleroma.User
-  alias Pleroma.Web.ActivityPub.ActivityPub
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.XML
-
-  def handle(entry, doc) do
-    with {:ok, actor} <- OStatus.find_make_or_update_actor(doc),
-         id when not is_nil(id) <- XML.string_from_xpath("/entry/id", entry),
-         followed_uri when not is_nil(followed_uri) <-
-           XML.string_from_xpath("/entry/activity:object/id", entry),
-         {:ok, followed} <- OStatus.find_or_make_user(followed_uri),
-         {:locked, false} <- {:locked, followed.info.locked},
-         {:ok, activity} <- ActivityPub.follow(actor, followed, id, false) do
-      User.follow(actor, followed)
-      {:ok, activity}
-    else
-      {:locked, true} ->
-        {:error, "It's not possible to follow locked accounts over OStatus"}
-    end
-  end
-end

lib/pleroma/web/ostatus/handlers/note_handler.ex

@@ -1,168 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.NoteHandler do
-  require Logger
-  require Pleroma.Constants
-
-  alias Pleroma.Activity
-  alias Pleroma.Object
-  alias Pleroma.Web.ActivityPub.ActivityPub
-  alias Pleroma.Web.ActivityPub.Utils
-  alias Pleroma.Web.CommonAPI
-  alias Pleroma.Web.Federator
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.XML
-
-  @doc """
-  Get the context for this note. Uses this:
-  1. The context of the parent activity
-  2. The conversation reference in the ostatus xml
-  3. A newly generated context id.
-  """
-  def get_context(entry, in_reply_to) do
-    context =
-      (XML.string_from_xpath("//ostatus:conversation[1]", entry) ||
-         XML.string_from_xpath("//ostatus:conversation[1]/@ref", entry) || "")
-      |> String.trim()
-
-    with %{data: %{"context" => context}} <- Object.get_cached_by_ap_id(in_reply_to) do
-      context
-    else
-      _e ->
-        if String.length(context) > 0 do
-          context
-        else
-          Utils.generate_context_id()
-        end
-    end
-  end
-
-  def get_people_mentions(entry) do
-    :xmerl_xpath.string(
-      '//link[@rel="mentioned" and @ostatus:object-type="http://activitystrea.ms/schema/1.0/person"]',
-      entry
-    )
-    |> Enum.map(fn person -> XML.string_from_xpath("@href", person) end)
-  end
-
-  def get_collection_mentions(entry) do
-    transmogrify = fn
-      "http://activityschema.org/collection/public" ->
-        Pleroma.Constants.as_public()
-
-      group ->
-        group
-    end
-
-    :xmerl_xpath.string(
-      '//link[@rel="mentioned" and @ostatus:object-type="http://activitystrea.ms/schema/1.0/collection"]',
-      entry
-    )
-    |> Enum.map(fn collection -> XML.string_from_xpath("@href", collection) |> transmogrify.() end)
-  end
-
-  def get_mentions(entry) do
-    (get_people_mentions(entry) ++ get_collection_mentions(entry))
-    |> Enum.filter(& &1)
-  end
-
-  def get_emoji(entry) do
-    try do
-      :xmerl_xpath.string('//link[@rel="emoji"]', entry)
-      |> Enum.reduce(%{}, fn emoji, acc ->
-        Map.put(acc, XML.string_from_xpath("@name", emoji), XML.string_from_xpath("@href", emoji))
-      end)
-    rescue
-      _e -> nil
-    end
-  end
-
-  def make_to_list(actor, mentions) do
-    [
-      actor.follower_address
-    ] ++ mentions
-  end
-
-  def add_external_url(note, entry) do
-    url = XML.string_from_xpath("//link[@rel='alternate' and @type='text/html']/@href", entry)
-    Map.put(note, "external_url", url)
-  end
-
-  def fetch_replied_to_activity(entry, in_reply_to, options \\ []) do
-    with %Activity{} = activity <- Activity.get_create_by_object_ap_id(in_reply_to) do
-      activity
-    else
-      _e ->
-        with true <- Federator.allowed_incoming_reply_depth?(options[:depth]),
-             in_reply_to_href when not is_nil(in_reply_to_href) <-
-               XML.string_from_xpath("//thr:in-reply-to[1]/@href", entry),
-             {:ok, [activity | _]} <- OStatus.fetch_activity_from_url(in_reply_to_href, options) do
-          activity
-        else
-          _e -> nil
-        end
-    end
-  end
-
-  # TODO: Clean this up a bit.
-  def handle_note(entry, doc \\ nil, options \\ []) do
-    with id <- XML.string_from_xpath("//id", entry),
-         activity when is_nil(activity) <- Activity.get_create_by_object_ap_id_with_object(id),
-         [author] <- :xmerl_xpath.string('//author[1]', doc),
-         {:ok, actor} <- OStatus.find_make_or_update_actor(author),
-         content_html <- OStatus.get_content(entry),
-         cw <- OStatus.get_cw(entry),
-         in_reply_to <- XML.string_from_xpath("//thr:in-reply-to[1]/@ref", entry),
-         options <- Keyword.put(options, :depth, (options[:depth] || 0) + 1),
-         in_reply_to_activity <- fetch_replied_to_activity(entry, in_reply_to, options),
-         in_reply_to_object <-
-           (in_reply_to_activity && Object.normalize(in_reply_to_activity)) || nil,
-         in_reply_to <- (in_reply_to_object && in_reply_to_object.data["id"]) || in_reply_to,
-         attachments <- OStatus.get_attachments(entry),
-         context <- get_context(entry, in_reply_to),
-         tags <- OStatus.get_tags(entry),
-         mentions <- get_mentions(entry),
-         to <- make_to_list(actor, mentions),
-         date <- XML.string_from_xpath("//published", entry),
-         unlisted <- XML.string_from_xpath("//mastodon:scope", entry) == "unlisted",
-         cc <- if(unlisted, do: [Pleroma.Constants.as_public()], else: []),
-         note <-
-           CommonAPI.Utils.make_note_data(
-             actor.ap_id,
-             to,
-             context,
-             content_html,
-             attachments,
-             in_reply_to_activity,
-             [],
-             cw
-           ),
-         note <- note |> Map.put("id", id) |> Map.put("tag", tags),
-         note <- note |> Map.put("published", date),
-         note <- note |> Map.put("emoji", get_emoji(entry)),
-         note <- add_external_url(note, entry),
-         note <- note |> Map.put("cc", cc),
-         # TODO: Handle this case in make_note_data
-         note <-
-           if(
-             in_reply_to && !in_reply_to_activity,
-             do: note |> Map.put("inReplyTo", in_reply_to),
-             else: note
-           ) do
-      ActivityPub.create(%{
-        to: to,
-        actor: actor,
-        context: context,
-        object: note,
-        published: date,
-        local: false,
-        additional: %{"cc" => cc}
-      })
-    else
-      %Activity{} = activity -> {:ok, activity}
-      e -> {:error, e}
-    end
-  end
-end

lib/pleroma/web/ostatus/handlers/unfollow_handler.ex

@@ -1,22 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.UnfollowHandler do
-  alias Pleroma.User
-  alias Pleroma.Web.ActivityPub.ActivityPub
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.XML
-
-  def handle(entry, doc) do
-    with {:ok, actor} <- OStatus.find_make_or_update_actor(doc),
-         id when not is_nil(id) <- XML.string_from_xpath("/entry/id", entry),
-         followed_uri when not is_nil(followed_uri) <-
-           XML.string_from_xpath("/entry/activity:object/id", entry),
-         {:ok, followed} <- OStatus.find_or_make_user(followed_uri),
-         {:ok, activity} <- ActivityPub.unfollow(actor, followed, id, false) do
-      User.unfollow(actor, followed)
-      {:ok, activity}
-    end
-  end
-end

lib/pleroma/web/ostatus/ostatus.ex

@@ -1,395 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus do
-  import Pleroma.Web.XML
-  require Logger
-
-  alias Pleroma.Activity
-  alias Pleroma.HTTP
-  alias Pleroma.Object
-  alias Pleroma.User
-  alias Pleroma.Web
-  alias Pleroma.Web.ActivityPub.ActivityPub
-  alias Pleroma.Web.ActivityPub.Transmogrifier
-  alias Pleroma.Web.ActivityPub.Visibility
-  alias Pleroma.Web.OStatus.DeleteHandler
-  alias Pleroma.Web.OStatus.FollowHandler
-  alias Pleroma.Web.OStatus.NoteHandler
-  alias Pleroma.Web.OStatus.UnfollowHandler
-  alias Pleroma.Web.WebFinger
-  alias Pleroma.Web.Websub
-
-  def is_representable?(%Activity{} = activity) do
-    object = Object.normalize(activity)
-
-    cond do
-      is_nil(object) ->
-        false
-
-      Visibility.is_public?(activity) && object.data["type"] == "Note" ->
-        true
-
-      true ->
-        false
-    end
-  end
-
-  def feed_path(user), do: "#{user.ap_id}/feed.atom"
-
-  def pubsub_path(user), do: "#{Web.base_url()}/push/hub/#{user.nickname}"
-
-  def salmon_path(user), do: "#{user.ap_id}/salmon"
-
-  def remote_follow_path, do: "#{Web.base_url()}/ostatus_subscribe?acct={uri}"
-
-  def handle_incoming(xml_string, options \\ []) do
-    with doc when doc != :error <- parse_document(xml_string) do
-      with {:ok, actor_user} <- find_make_or_update_actor(doc),
-           do: Pleroma.Instances.set_reachable(actor_user.ap_id)
-
-      entries = :xmerl_xpath.string('//entry', doc)
-
-      activities =
-        Enum.map(entries, fn entry ->
-          {:xmlObj, :string, object_type} =
-            :xmerl_xpath.string('string(/entry/activity:object-type[1])', entry)
-
-          {:xmlObj, :string, verb} = :xmerl_xpath.string('string(/entry/activity:verb[1])', entry)
-          Logger.debug("Handling #{verb}")
-
-          try do
-            case verb do
-              'http://activitystrea.ms/schema/1.0/delete' ->
-                with {:ok, activity} <- DeleteHandler.handle_delete(entry, doc), do: activity
-
-              'http://activitystrea.ms/schema/1.0/follow' ->
-                with {:ok, activity} <- FollowHandler.handle(entry, doc), do: activity
-
-              'http://activitystrea.ms/schema/1.0/unfollow' ->
-                with {:ok, activity} <- UnfollowHandler.handle(entry, doc), do: activity
-
-              'http://activitystrea.ms/schema/1.0/share' ->
-                with {:ok, activity, retweeted_activity} <- handle_share(entry, doc),
-                     do: [activity, retweeted_activity]
-
-              'http://activitystrea.ms/schema/1.0/favorite' ->
-                with {:ok, activity, favorited_activity} <- handle_favorite(entry, doc),
-                     do: [activity, favorited_activity]
-
-              _ ->
-                case object_type do
-                  'http://activitystrea.ms/schema/1.0/note' ->
-                    with {:ok, activity} <- NoteHandler.handle_note(entry, doc, options),
-                         do: activity
-
-                  'http://activitystrea.ms/schema/1.0/comment' ->
-                    with {:ok, activity} <- NoteHandler.handle_note(entry, doc, options),
-                         do: activity
-
-                  _ ->
-                    Logger.error("Couldn't parse incoming document")
-                    nil
-                end
-            end
-          rescue
-            e ->
-              Logger.error("Error occured while handling activity")
-              Logger.error(xml_string)
-              Logger.error(inspect(e))
-              nil
-          end
-        end)
-        |> Enum.filter(& &1)
-
-      {:ok, activities}
-    else
-      _e -> {:error, []}
-    end
-  end
-
-  def make_share(entry, doc, retweeted_activity) do
-    with {:ok, actor} <- find_make_or_update_actor(doc),
-         %Object{} = object <- Object.normalize(retweeted_activity),
-         id when not is_nil(id) <- string_from_xpath("/entry/id", entry),
-         {:ok, activity, _object} = ActivityPub.announce(actor, object, id, false) do
-      {:ok, activity}
-    end
-  end
-
-  def handle_share(entry, doc) do
-    with {:ok, retweeted_activity} <- get_or_build_object(entry),
-         {:ok, activity} <- make_share(entry, doc, retweeted_activity) do
-      {:ok, activity, retweeted_activity}
-    else
-      e -> {:error, e}
-    end
-  end
-
-  def make_favorite(entry, doc, favorited_activity) do
-    with {:ok, actor} <- find_make_or_update_actor(doc),
-         %Object{} = object <- Object.normalize(favorited_activity),
-         id when not is_nil(id) <- string_from_xpath("/entry/id", entry),
-         {:ok, activity, _object} = ActivityPub.like(actor, object, id, false) do
-      {:ok, activity}
-    end
-  end
-
-  def get_or_build_object(entry) do
-    with {:ok, activity} <- get_or_try_fetching(entry) do
-      {:ok, activity}
-    else
-      _e ->
-        with [object] <- :xmerl_xpath.string('/entry/activity:object', entry) do
-          NoteHandler.handle_note(object, object)
-        end
-    end
-  end
-
-  def get_or_try_fetching(entry) do
-    Logger.debug("Trying to get entry from db")
-
-    with id when not is_nil(id) <- string_from_xpath("//activity:object[1]/id", entry),
-         %Activity{} = activity <- Activity.get_create_by_object_ap_id_with_object(id) do
-      {:ok, activity}
-    else
-      _ ->
-        Logger.debug("Couldn't get, will try to fetch")
-
-        with href when not is_nil(href) <-
-               string_from_xpath("//activity:object[1]/link[@type=\"text/html\"]/@href", entry),
-             {:ok, [favorited_activity]} <- fetch_activity_from_url(href) do
-          {:ok, favorited_activity}
-        else
-          e -> Logger.debug("Couldn't find href: #{inspect(e)}")
-        end
-    end
-  end
-
-  def handle_favorite(entry, doc) do
-    with {:ok, favorited_activity} <- get_or_try_fetching(entry),
-         {:ok, activity} <- make_favorite(entry, doc, favorited_activity) do
-      {:ok, activity, favorited_activity}
-    else
-      e -> {:error, e}
-    end
-  end
-
-  def get_attachments(entry) do
-    :xmerl_xpath.string('/entry/link[@rel="enclosure"]', entry)
-    |> Enum.map(fn enclosure ->
-      with href when not is_nil(href) <- string_from_xpath("/link/@href", enclosure),
-           type when not is_nil(type) <- string_from_xpath("/link/@type", enclosure) do
-        %{
-          "type" => "Attachment",
-          "url" => [
-            %{
-              "type" => "Link",
-              "mediaType" => type,
-              "href" => href
-            }
-          ]
-        }
-      end
-    end)
-    |> Enum.filter(& &1)
-  end
-
-  @doc """
-    Gets the content from a an entry.
-  """
-  def get_content(entry) do
-    string_from_xpath("//content", entry)
-  end
-
-  @doc """
-    Get the cw that mastodon uses.
-  """
-  def get_cw(entry) do
-    case string_from_xpath("/*/summary", entry) do
-      cw when not is_nil(cw) -> cw
-      _ -> nil
-    end
-  end
-
-  def get_tags(entry) do
-    :xmerl_xpath.string('//category', entry)
-    |> Enum.map(fn category -> string_from_xpath("/category/@term", category) end)
-    |> Enum.filter(& &1)
-    |> Enum.map(&String.downcase/1)
-  end
-
-  def maybe_update(doc, user) do
-    case string_from_xpath("//author[1]/ap_enabled", doc) do
-      "true" ->
-        Transmogrifier.upgrade_user_from_ap_id(user.ap_id)
-
-      _ ->
-        maybe_update_ostatus(doc, user)
-    end
-  end
-
-  def maybe_update_ostatus(doc, user) do
-    old_data = Map.take(user, [:bio, :avatar, :name])
-
-    with false <- user.local,
-         avatar <- make_avatar_object(doc),
-         bio <- string_from_xpath("//author[1]/summary", doc),
-         name <- string_from_xpath("//author[1]/poco:displayName", doc),
-         new_data <- %{
-           avatar: avatar || old_data.avatar,
-           name: name || old_data.name,
-           bio: bio || old_data.bio
-         },
-         false <- new_data == old_data do
-      change = Ecto.Changeset.change(user, new_data)
-      User.update_and_set_cache(change)
-    else
-      _ ->
-        {:ok, user}
-    end
-  end
-
-  def find_make_or_update_actor(doc) do
-    uri = string_from_xpath("//author/uri[1]", doc)
-
-    with {:ok, %User{} = user} <- find_or_make_user(uri),
-         {:ap_enabled, false} <- {:ap_enabled, User.ap_enabled?(user)} do
-      maybe_update(doc, user)
-    else
-      {:ap_enabled, true} ->
-        {:error, :invalid_protocol}
-
-      _ ->
-        {:error, :unknown_user}
-    end
-  end
-
-  @spec find_or_make_user(String.t()) :: {:ok, User.t()}
-  def find_or_make_user(uri) do
-    case User.get_by_ap_id(uri) do
-      %User{} = user -> {:ok, user}
-      _ -> make_user(uri)
-    end
-  end
-
-  @spec make_user(String.t(), boolean()) :: {:ok, User.t()} | {:error, any()}
-  def make_user(uri, update \\ false) do
-    with {:ok, info} <- gather_user_info(uri) do
-      with false <- update,
-           %User{} = user <- User.get_cached_by_ap_id(info["uri"]) do
-        {:ok, user}
-      else
-        _e -> User.insert_or_update_user(build_user_data(info))
-      end
-    end
-  end
-
-  defp build_user_data(info) do
-    %{
-      name: info["name"],
-      nickname: info["nickname"] <> "@" <> info["host"],
-      ap_id: info["uri"],
-      info: info,
-      avatar: info["avatar"],
-      bio: info["bio"]
-    }
-  end
-
-  # TODO: Just takes the first one for now.
-  def make_avatar_object(author_doc, rel \\ "avatar") do
-    href = string_from_xpath("//author[1]/link[@rel=\"#{rel}\"]/@href", author_doc)
-    type = string_from_xpath("//author[1]/link[@rel=\"#{rel}\"]/@type", author_doc)
-
-    if href do
-      %{
-        "type" => "Image",
-        "url" => [%{"type" => "Link", "mediaType" => type, "href" => href}]
-      }
-    else
-      nil
-    end
-  end
-
-  @spec gather_user_info(String.t()) :: {:ok, map()} | {:error, any()}
-  def gather_user_info(username) do
-    with {:ok, webfinger_data} <- WebFinger.finger(username),
-         {:ok, feed_data} <- Websub.gather_feed_data(webfinger_data["topic"]) do
-      data =
-        webfinger_data
-        |> Map.merge(feed_data)
-        |> Map.put("fqn", username)
-
-      {:ok, data}
-    else
-      e ->
-        Logger.debug(fn -> "Couldn't gather info for #{username}" end)
-        {:error, e}
-    end
-  end
-
-  # Regex-based 'parsing' so we don't have to pull in a full html parser
-  # It's a hack anyway. Maybe revisit this in the future
-  @mastodon_regex ~r/<link href='(.*)' rel='alternate' type='application\/atom\+xml'>/
-  @gs_regex ~r/<link title=.* href="(.*)" type="application\/atom\+xml" rel="alternate">/
-  @gs_classic_regex ~r/<link rel="alternate" href="(.*)" type="application\/atom\+xml" title=.*>/
-  def get_atom_url(body) do
-    cond do
-      Regex.match?(@mastodon_regex, body) ->
-        [[_, match]] = Regex.scan(@mastodon_regex, body)
-        {:ok, match}
-
-      Regex.match?(@gs_regex, body) ->
-        [[_, match]] = Regex.scan(@gs_regex, body)
-        {:ok, match}
-
-      Regex.match?(@gs_classic_regex, body) ->
-        [[_, match]] = Regex.scan(@gs_classic_regex, body)
-        {:ok, match}
-
-      true ->
-        Logger.debug(fn -> "Couldn't find Atom link in #{inspect(body)}" end)
-        {:error, "Couldn't find the Atom link"}
-    end
-  end
-
-  def fetch_activity_from_atom_url(url, options \\ []) do
-    with true <- String.starts_with?(url, "http"),
-         {:ok, %{body: body, status: code}} when code in 200..299 <-
-           HTTP.get(url, [{:Accept, "application/atom+xml"}]) do
-      Logger.debug("Got document from #{url}, handling...")
-      handle_incoming(body, options)
-    else
-      e ->
-        Logger.debug("Couldn't get #{url}: #{inspect(e)}")
-        e
-    end
-  end
-
-  def fetch_activity_from_html_url(url, options \\ []) do
-    Logger.debug("Trying to fetch #{url}")
-
-    with true <- String.starts_with?(url, "http"),
-         {:ok, %{body: body}} <- HTTP.get(url, []),
-         {:ok, atom_url} <- get_atom_url(body) do
-      fetch_activity_from_atom_url(atom_url, options)
-    else
-      e ->
-        Logger.debug("Couldn't get #{url}: #{inspect(e)}")
-        e
-    end
-  end
-
-  def fetch_activity_from_url(url, options \\ []) do
-    with {:ok, [_ | _] = activities} <- fetch_activity_from_atom_url(url, options) do
-      {:ok, activities}
-    else
-      _e -> fetch_activity_from_html_url(url, options)
-    end
-  rescue
-    e ->
-      Logger.debug("Couldn't get #{url}: #{inspect(e)}")
-      {:error, "Couldn't get #{url}: #{inspect(e)}"}
-  end
-end

lib/pleroma/web/ostatus/ostatus_controller.ex

@@ -13,19 +13,14 @@ defmodule Pleroma.Web.OStatus.OStatusController do
   alias Pleroma.Web.ActivityPub.ObjectView
   alias Pleroma.Web.ActivityPub.Visibility
   alias Pleroma.Web.Endpoint
-  alias Pleroma.Web.Federator
   alias Pleroma.Web.Metadata.PlayerView
-  alias Pleroma.Web.OStatus.ActivityRepresenter
   alias Pleroma.Web.Router
-  alias Pleroma.Web.XML
 
   plug(
     Pleroma.Plugs.RateLimiter,
     {:ap_routes, params: ["uuid"]} when action in [:object, :activity]
   )
 
-  plug(Pleroma.Web.FederatingPlug when action in [:salmon_incoming])
-
   plug(
     Pleroma.Plugs.SetFormatPlug
     when action in [:object, :activity, :notice]
@@ -33,32 +28,6 @@ defmodule Pleroma.Web.OStatus.OStatusController do
 
   action_fallback(:errors)
 
-  defp decode_or_retry(body) do
-    with {:ok, magic_key} <- Pleroma.Web.Salmon.fetch_magic_key(body),
-         {:ok, doc} <- Pleroma.Web.Salmon.decode_and_validate(magic_key, body) do
-      {:ok, doc}
-    else
-      _e ->
-        with [decoded | _] <- Pleroma.Web.Salmon.decode(body),
-             doc <- XML.parse_document(decoded),
-             uri when not is_nil(uri) <- XML.string_from_xpath("/entry/author[1]/uri", doc),
-             {:ok, _} <- Pleroma.Web.OStatus.make_user(uri, true),
-             {:ok, magic_key} <- Pleroma.Web.Salmon.fetch_magic_key(body),
-             {:ok, doc} <- Pleroma.Web.Salmon.decode_and_validate(magic_key, body) do
-          {:ok, doc}
-        end
-    end
-  end
-
-  def salmon_incoming(conn, _) do
-    {:ok, body, _conn} = read_body(conn)
-    {:ok, doc} = decode_or_retry(body)
-
-    Federator.incoming_doc(doc)
-
-    send_resp(conn, 200, "")
-  end
-
   def object(%{assigns: %{format: format}} = conn, %{"uuid" => _uuid})
       when format in ["json", "activity+json"] do
     ActivityPubController.call(conn, :object)
@@ -179,23 +148,10 @@ defp represent_activity(
     |> render("object.json", %{object: object})
   end
 
-  defp represent_activity(_conn, "activity+json", _, _) do
+  defp represent_activity(_conn, _, _, _) do
     {:error, :not_found}
   end
 
-  defp represent_activity(conn, _, activity, user) do
-    response =
-      activity
-      |> ActivityRepresenter.to_simple_form(user, true)
-      |> ActivityRepresenter.wrap_with_entry()
-      |> :xmerl.export_simple(:xmerl_xml)
-      |> to_string
-
-    conn
-    |> put_resp_content_type("application/atom+xml")
-    |> send_resp(200, response)
-  end
-
   def errors(conn, {:error, :not_found}) do
     render_error(conn, :not_found, "Not found")
   end

lib/pleroma/web/ostatus/user_representer.ex

@@ -1,41 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.UserRepresenter do
-  alias Pleroma.User
-
-  def to_simple_form(user) do
-    ap_id = to_charlist(user.ap_id)
-    nickname = to_charlist(user.nickname)
-    name = to_charlist(user.name)
-    bio = to_charlist(user.bio)
-    avatar_url = to_charlist(User.avatar_url(user))
-
-    banner =
-      if banner_url = User.banner_url(user) do
-        [{:link, [rel: 'header', href: banner_url], []}]
-      else
-        []
-      end
-
-    ap_enabled =
-      if user.local do
-        [{:ap_enabled, ['true']}]
-      else
-        []
-      end
-
-    [
-      {:id, [ap_id]},
-      {:"activity:object", ['http://activitystrea.ms/schema/1.0/person']},
-      {:uri, [ap_id]},
-      {:"poco:preferredUsername", [nickname]},
-      {:"poco:displayName", [name]},
-      {:"poco:note", [bio]},
-      {:summary, [bio]},
-      {:name, [nickname]},
-      {:link, [rel: 'avatar', href: avatar_url], []}
-    ] ++ banner ++ ap_enabled
-  end
-end

lib/pleroma/web/pleroma_api/controllers/pleroma_api_controller.ex

@@ -79,6 +79,15 @@ def update_conversation(
     end
   end
 
+  def read_conversations(%{assigns: %{user: user}} = conn, _params) do
+    with {:ok, participations} <- Participation.mark_all_as_read(user) do
+      conn
+      |> add_link_headers(participations)
+      |> put_view(ConversationView)
+      |> render("participations.json", participations: participations, for: user)
+    end
+  end
+
   def read_notification(%{assigns: %{user: user}} = conn, %{"id" => notification_id}) do
     with {:ok, notification} <- Notification.read_one(user, notification_id) do
       conn

lib/pleroma/web/push/impl.ex

@@ -125,6 +125,10 @@ def format_body(
     end
   end
 
+  def format_title(%{activity: %{data: %{"directMessage" => true}}}) do
+    "New Direct Message"
+  end
+
   def format_title(%{activity: %{data: %{"type" => type}}}) do
     case type do
       "Create" -> "New Mention"

lib/pleroma/web/router.ex

@@ -161,6 +161,7 @@ defmodule Pleroma.Web.Router do
       :right_delete_multiple
     )
 
+    get("/relay", AdminAPIController, :relay_list)
     post("/relay", AdminAPIController, :relay_follow)
     delete("/relay", AdminAPIController, :relay_unfollow)
 
@@ -265,6 +266,7 @@ defmodule Pleroma.Web.Router do
 
       get("/conversations/:id/statuses", PleromaAPIController, :conversation_statuses)
       get("/conversations/:id", PleromaAPIController, :conversation)
+      post("/conversations/read", PleromaAPIController, :read_conversations)
     end
 
     scope [] do
@@ -403,6 +405,9 @@ defmodule Pleroma.Web.Router do
     get("/push/subscription", SubscriptionController, :get)
     put("/push/subscription", SubscriptionController, :update)
     delete("/push/subscription", SubscriptionController, :delete)
+
+    get("/markers", MarkerController, :index)
+    post("/markers", MarkerController, :upsert)
   end
 
   scope "/api/web", Pleroma.Web do
@@ -508,11 +513,6 @@ defmodule Pleroma.Web.Router do
     get("/users/:nickname/feed", Feed.FeedController, :feed)
     get("/users/:nickname", Feed.FeedController, :feed_redirect)
 
-    post("/users/:nickname/salmon", OStatus.OStatusController, :salmon_incoming)
-    post("/push/hub/:nickname", Websub.WebsubController, :websub_subscription_request)
-    get("/push/subscriptions/:id", Websub.WebsubController, :websub_subscription_confirmation)
-    post("/push/subscriptions/:id", Websub.WebsubController, :websub_incoming)
-
     get("/mailer/unsubscribe/:token", Mailer.SubscriptionController, :unsubscribe)
   end
 
@@ -595,6 +595,12 @@ defmodule Pleroma.Web.Router do
     get("/:version", Nodeinfo.NodeinfoController, :nodeinfo)
   end
 
+  scope "/", Pleroma.Web do
+    pipe_through(:api)
+
+    get("/web/manifest.json", MastoFEController, :manifest)
+  end
+
   scope "/", Pleroma.Web do
     pipe_through(:mastodon_html)
 

lib/pleroma/web/salmon/salmon.ex

@@ -1,254 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.Salmon do
-  @behaviour Pleroma.Web.Federator.Publisher
-
-  use Bitwise
-
-  alias Pleroma.Activity
-  alias Pleroma.HTTP
-  alias Pleroma.Instances
-  alias Pleroma.Keys
-  alias Pleroma.User
-  alias Pleroma.Web.ActivityPub.Visibility
-  alias Pleroma.Web.Federator.Publisher
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.OStatus.ActivityRepresenter
-  alias Pleroma.Web.XML
-
-  require Logger
-
-  def decode(salmon) do
-    doc = XML.parse_document(salmon)
-
-    {:xmlObj, :string, data} = :xmerl_xpath.string('string(//me:data[1])', doc)
-    {:xmlObj, :string, sig} = :xmerl_xpath.string('string(//me:sig[1])', doc)
-    {:xmlObj, :string, alg} = :xmerl_xpath.string('string(//me:alg[1])', doc)
-    {:xmlObj, :string, encoding} = :xmerl_xpath.string('string(//me:encoding[1])', doc)
-    {:xmlObj, :string, type} = :xmerl_xpath.string('string(//me:data[1]/@type)', doc)
-
-    {:ok, data} = Base.url_decode64(to_string(data), ignore: :whitespace)
-    {:ok, sig} = Base.url_decode64(to_string(sig), ignore: :whitespace)
-    alg = to_string(alg)
-    encoding = to_string(encoding)
-    type = to_string(type)
-
-    [data, type, encoding, alg, sig]
-  end
-
-  def fetch_magic_key(salmon) do
-    with [data, _, _, _, _] <- decode(salmon),
-         doc <- XML.parse_document(data),
-         uri when not is_nil(uri) <- XML.string_from_xpath("/entry/author[1]/uri", doc),
-         {:ok, public_key} <- User.get_public_key_for_ap_id(uri),
-         magic_key <- encode_key(public_key) do
-      {:ok, magic_key}
-    end
-  end
-
-  def decode_and_validate(magickey, salmon) do
-    [data, type, encoding, alg, sig] = decode(salmon)
-
-    signed_text =
-      [data, type, encoding, alg]
-      |> Enum.map(&Base.url_encode64/1)
-      |> Enum.join(".")
-
-    key = decode_key(magickey)
-
-    verify = :public_key.verify(signed_text, :sha256, sig, key)
-
-    if verify do
-      {:ok, data}
-    else
-      :error
-    end
-  end
-
-  def decode_key("RSA." <> magickey) do
-    make_integer = fn bin ->
-      list = :erlang.binary_to_list(bin)
-      Enum.reduce(list, 0, fn el, acc -> acc <<< 8 ||| el end)
-    end
-
-    [modulus, exponent] =
-      magickey
-      |> String.split(".")
-      |> Enum.map(fn n -> Base.url_decode64!(n, padding: false) end)
-      |> Enum.map(make_integer)
-
-    {:RSAPublicKey, modulus, exponent}
-  end
-
-  def encode_key({:RSAPublicKey, modulus, exponent}) do
-    modulus_enc = :binary.encode_unsigned(modulus) |> Base.url_encode64()
-    exponent_enc = :binary.encode_unsigned(exponent) |> Base.url_encode64()
-
-    "RSA.#{modulus_enc}.#{exponent_enc}"
-  end
-
-  def encode(private_key, doc) do
-    type = "application/atom+xml"
-    encoding = "base64url"
-    alg = "RSA-SHA256"
-
-    signed_text =
-      [doc, type, encoding, alg]
-      |> Enum.map(&Base.url_encode64/1)
-      |> Enum.join(".")
-
-    signature =
-      signed_text
-      |> :public_key.sign(:sha256, private_key)
-      |> to_string
-      |> Base.url_encode64()
-
-    doc_base64 =
-      doc
-      |> Base.url_encode64()
-
-    # Don't need proper xml building, these strings are safe to leave unescaped
-    salmon = """
-    <?xml version="1.0" encoding="UTF-8"?>
-    <me:env xmlns:me="http://salmon-protocol.org/ns/magic-env">
-      <me:data type="application/atom+xml">#{doc_base64}</me:data>
-      <me:encoding>#{encoding}</me:encoding>
-      <me:alg>#{alg}</me:alg>
-      <me:sig>#{signature}</me:sig>
-    </me:env>
-    """
-
-    {:ok, salmon}
-  end
-
-  def remote_users(%User{id: user_id}, %{data: %{"to" => to} = data}) do
-    cc = Map.get(data, "cc", [])
-
-    bcc =
-      data
-      |> Map.get("bcc", [])
-      |> Enum.reduce([], fn ap_id, bcc ->
-        case Pleroma.List.get_by_ap_id(ap_id) do
-          %Pleroma.List{user_id: ^user_id} = list ->
-            {:ok, following} = Pleroma.List.get_following(list)
-            bcc ++ Enum.map(following, & &1.ap_id)
-
-          _ ->
-            bcc
-        end
-      end)
-
-    [to, cc, bcc]
-    |> Enum.concat()
-    |> Enum.map(&User.get_cached_by_ap_id/1)
-    |> Enum.filter(fn user -> user && !user.local end)
-  end
-
-  @doc "Pushes an activity to remote account."
-  def publish_one(%{recipient: %{info: %{salmon: salmon}}} = params),
-    do: publish_one(Map.put(params, :recipient, salmon))
-
-  def publish_one(%{recipient: url, feed: feed} = params) when is_binary(url) do
-    with {:ok, %{status: code}} when code in 200..299 <-
-           HTTP.post(
-             url,
-             feed,
-             [{"Content-Type", "application/magic-envelope+xml"}]
-           ) do
-      if !Map.has_key?(params, :unreachable_since) || params[:unreachable_since],
-        do: Instances.set_reachable(url)
-
-      Logger.debug(fn -> "Pushed to #{url}, code #{code}" end)
-      {:ok, code}
-    else
-      e ->
-        unless params[:unreachable_since], do: Instances.set_reachable(url)
-        Logger.debug(fn -> "Pushing Salmon to #{url} failed, #{inspect(e)}" end)
-        {:error, "Unreachable instance"}
-    end
-  end
-
-  def publish_one(%{recipient_id: recipient_id} = params) do
-    recipient = User.get_cached_by_id(recipient_id)
-
-    params
-    |> Map.delete(:recipient_id)
-    |> Map.put(:recipient, recipient)
-    |> publish_one()
-  end
-
-  def publish_one(_), do: :noop
-
-  @supported_activities [
-    "Create",
-    "Follow",
-    "Like",
-    "Announce",
-    "Undo",
-    "Delete"
-  ]
-
-  def is_representable?(%Activity{data: %{"type" => type}} = activity)
-      when type in @supported_activities,
-      do: Visibility.is_public?(activity)
-
-  def is_representable?(_), do: false
-
-  @doc """
-  Publishes an activity to remote accounts
-  """
-  @spec publish(User.t(), Pleroma.Activity.t()) :: none
-  def publish(user, activity)
-
-  def publish(%{keys: keys} = user, %{data: %{"type" => type}} = activity)
-      when type in @supported_activities do
-    feed = ActivityRepresenter.to_simple_form(activity, user, true)
-
-    if feed do
-      feed =
-        ActivityRepresenter.wrap_with_entry(feed)
-        |> :xmerl.export_simple(:xmerl_xml)
-        |> to_string
-
-      {:ok, private, _} = Keys.keys_from_pem(keys)
-      {:ok, feed} = encode(private, feed)
-
-      remote_users = remote_users(user, activity)
-
-      salmon_urls = Enum.map(remote_users, & &1.info.salmon)
-      reachable_urls_metadata = Instances.filter_reachable(salmon_urls)
-      reachable_urls = Map.keys(reachable_urls_metadata)
-
-      remote_users
-      |> Enum.filter(&(&1.info.salmon in reachable_urls))
-      |> Enum.each(fn remote_user ->
-        Logger.debug(fn -> "Sending Salmon to #{remote_user.ap_id}" end)
-
-        Publisher.enqueue_one(__MODULE__, %{
-          recipient_id: remote_user.id,
-          feed: feed,
-          unreachable_since: reachable_urls_metadata[remote_user.info.salmon]
-        })
-      end)
-    end
-  end
-
-  def publish(%{id: id}, _), do: Logger.debug(fn -> "Keys missing for user #{id}" end)
-
-  def gather_webfinger_links(%User{} = user) do
-    {:ok, _private, public} = Keys.keys_from_pem(user.keys)
-    magic_key = encode_key(public)
-
-    [
-      %{"rel" => "salmon", "href" => OStatus.salmon_path(user)},
-      %{
-        "rel" => "magic-public-key",
-        "href" => "data:application/magic-public-key,#{magic_key}"
-      }
-    ]
-  end
-
-  def gather_nodeinfo_protocol_names, do: []
-end

lib/pleroma/web/templates/feed/feed/feed.xml.eex

@@ -10,8 +10,6 @@
   <title><%= @user.nickname <> "'s timeline" %></title>
   <updated><%= most_recent_update(@activities, @user) %></updated>
   <logo><%= logo(@user) %></logo>
-  <link rel="hub" href="<%= websub_url(@conn, :websub_subscription_request, @user.nickname) %>"/>
-  <link rel="salmon" href="<%= o_status_url(@conn, :salmon_incoming, @user.nickname) %>"/>
   <link rel="self" href="<%= '#{feed_url(@conn, :feed, @user.nickname)}.atom' %>" type="application/atom+xml"/>
 
   <%= render @view_module, "_author.xml", assigns %>

lib/pleroma/web/templates/masto_fe/index.html.eex

@@ -4,9 +4,13 @@
 <meta charset='utf-8'>
 <meta content='width=device-width, initial-scale=1' name='viewport'>
 <title>
-<%= Pleroma.Config.get([:instance, :name]) %>
+<%= Config.get([:instance, :name]) %>
 </title>
 <link rel="icon" type="image/png" href="/favicon.png"/>
+<link rel="manifest" type="applicaton/manifest+json" href="<%= masto_fe_path(Pleroma.Web.Endpoint, :manifest) %>" />
+
+<meta name="theme-color" content="<%= Config.get([:manifest, :theme_color]) %>" />
+
 <script crossorigin='anonymous' src="/packs/locales.js"></script>
 <script crossorigin='anonymous' src="/packs/locales/glitch/en.js"></script>
 

lib/pleroma/web/views/masto_fe_view.ex

@@ -99,4 +99,23 @@ def initial_state(token, user, custom_emojis) do
   defp present?(nil), do: false
   defp present?(false), do: false
   defp present?(_), do: true
+
+  def render("manifest.json", _params) do
+    %{
+      name: Config.get([:instance, :name]),
+      description: Config.get([:instance, :description]),
+      icons: Config.get([:manifest, :icons]),
+      theme_color: Config.get([:manifest, :theme_color]),
+      background_color: Config.get([:manifest, :background_color]),
+      display: "standalone",
+      scope: Pleroma.Web.base_url(),
+      start_url: masto_fe_path(Pleroma.Web.Endpoint, :index, ["getting-started"]),
+      categories: [
+        "social"
+      ],
+      serviceworker: %{
+        src: "/sw.js"
+      }
+    }
+  end
 end

lib/pleroma/web/web_finger/web_finger.ex

@@ -108,7 +108,6 @@ defp webfinger_from_xml(doc) do
              doc
            ),
          subject <- XML.string_from_xpath("//Subject", doc),
-         salmon <- XML.string_from_xpath(~s{//Link[@rel="salmon"]/@href}, doc),
          subscribe_address <-
            XML.string_from_xpath(
              ~s{//Link[@rel="http://ostatus.org/schema/1.0/subscribe"]/@template},
@@ -123,7 +122,6 @@ defp webfinger_from_xml(doc) do
         "magic_key" => magic_key,
         "topic" => topic,
         "subject" => subject,
-        "salmon" => salmon,
         "subscribe_address" => subscribe_address,
         "ap_id" => ap_id
       }
@@ -148,16 +146,6 @@ defp webfinger_from_json(doc) do
           {"application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"", "self"} ->
             Map.put(data, "ap_id", link["href"])
 
-          {_, "magic-public-key"} ->
-            "data:application/magic-public-key," <> magic_key = link["href"]
-            Map.put(data, "magic_key", magic_key)
-
-          {"application/atom+xml", "http://schemas.google.com/g/2010#updates-from"} ->
-            Map.put(data, "topic", link["href"])
-
-          {_, "salmon"} ->
-            Map.put(data, "salmon", link["href"])
-
           {_, "http://ostatus.org/schema/1.0/subscribe"} ->
             Map.put(data, "subscribe_address", link["template"])
 

lib/pleroma/web/websub/websub.ex

@@ -1,332 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.Websub do
-  alias Ecto.Changeset
-  alias Pleroma.Activity
-  alias Pleroma.HTTP
-  alias Pleroma.Instances
-  alias Pleroma.Repo
-  alias Pleroma.User
-  alias Pleroma.Web.ActivityPub.Visibility
-  alias Pleroma.Web.Endpoint
-  alias Pleroma.Web.Federator
-  alias Pleroma.Web.Federator.Publisher
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.OStatus.FeedRepresenter
-  alias Pleroma.Web.Router.Helpers
-  alias Pleroma.Web.Websub.WebsubClientSubscription
-  alias Pleroma.Web.Websub.WebsubServerSubscription
-  alias Pleroma.Web.XML
-  require Logger
-
-  import Ecto.Query
-
-  @behaviour Pleroma.Web.Federator.Publisher
-
-  def verify(subscription, getter \\ &HTTP.get/3) do
-    challenge = Base.encode16(:crypto.strong_rand_bytes(8))
-    lease_seconds = NaiveDateTime.diff(subscription.valid_until, subscription.updated_at)
-    lease_seconds = lease_seconds |> to_string
-
-    params = %{
-      "hub.challenge": challenge,
-      "hub.lease_seconds": lease_seconds,
-      "hub.topic": subscription.topic,
-      "hub.mode": "subscribe"
-    }
-
-    url = hd(String.split(subscription.callback, "?"))
-    query = URI.parse(subscription.callback).query || ""
-    params = Map.merge(params, URI.decode_query(query))
-
-    with {:ok, response} <- getter.(url, [], params: params),
-         ^challenge <- response.body do
-      changeset = Changeset.change(subscription, %{state: "active"})
-      Repo.update(changeset)
-    else
-      e ->
-        Logger.debug("Couldn't verify subscription")
-        Logger.debug(inspect(e))
-        {:error, subscription}
-    end
-  end
-
-  @supported_activities [
-    "Create",
-    "Follow",
-    "Like",
-    "Announce",
-    "Undo",
-    "Delete"
-  ]
-
-  def is_representable?(%Activity{data: %{"type" => type}} = activity)
-      when type in @supported_activities,
-      do: Visibility.is_public?(activity)
-
-  def is_representable?(_), do: false
-
-  def publish(topic, user, %{data: %{"type" => type}} = activity)
-      when type in @supported_activities do
-    response =
-      user
-      |> FeedRepresenter.to_simple_form([activity], [user])
-      |> :xmerl.export_simple(:xmerl_xml)
-      |> to_string
-
-    query =
-      from(
-        sub in WebsubServerSubscription,
-        where: sub.topic == ^topic and sub.state == "active",
-        where: fragment("? > (NOW() at time zone 'UTC')", sub.valid_until)
-      )
-
-    subscriptions = Repo.all(query)
-
-    callbacks = Enum.map(subscriptions, & &1.callback)
-    reachable_callbacks_metadata = Instances.filter_reachable(callbacks)
-    reachable_callbacks = Map.keys(reachable_callbacks_metadata)
-
-    subscriptions
-    |> Enum.filter(&(&1.callback in reachable_callbacks))
-    |> Enum.each(fn sub ->
-      data = %{
-        xml: response,
-        topic: topic,
-        callback: sub.callback,
-        secret: sub.secret,
-        unreachable_since: reachable_callbacks_metadata[sub.callback]
-      }
-
-      Publisher.enqueue_one(__MODULE__, data)
-    end)
-  end
-
-  def publish(_, _, _), do: ""
-
-  def publish(actor, activity), do: publish(Pleroma.Web.OStatus.feed_path(actor), actor, activity)
-
-  def sign(secret, doc) do
-    :crypto.hmac(:sha, secret, to_string(doc)) |> Base.encode16() |> String.downcase()
-  end
-
-  def incoming_subscription_request(user, %{"hub.mode" => "subscribe"} = params) do
-    with {:ok, topic} <- valid_topic(params, user),
-         {:ok, lease_time} <- lease_time(params),
-         secret <- params["hub.secret"],
-         callback <- params["hub.callback"] do
-      subscription = get_subscription(topic, callback)
-
-      data = %{
-        state: subscription.state || "requested",
-        topic: topic,
-        secret: secret,
-        callback: callback
-      }
-
-      change = Changeset.change(subscription, data)
-      websub = Repo.insert_or_update!(change)
-
-      change =
-        Changeset.change(websub, %{valid_until: NaiveDateTime.add(websub.updated_at, lease_time)})
-
-      websub = Repo.update!(change)
-
-      Federator.verify_websub(websub)
-
-      {:ok, websub}
-    else
-      {:error, reason} ->
-        Logger.debug("Couldn't create subscription")
-        Logger.debug(inspect(reason))
-
-        {:error, reason}
-    end
-  end
-
-  def incoming_subscription_request(user, params) do
-    Logger.info("Unhandled WebSub request for #{user.nickname}: #{inspect(params)}")
-
-    {:error, "Invalid WebSub request"}
-  end
-
-  defp get_subscription(topic, callback) do
-    Repo.get_by(WebsubServerSubscription, topic: topic, callback: callback) ||
-      %WebsubServerSubscription{}
-  end
-
-  # Temp hack for mastodon.
-  defp lease_time(%{"hub.lease_seconds" => ""}) do
-    # three days
-    {:ok, 60 * 60 * 24 * 3}
-  end
-
-  defp lease_time(%{"hub.lease_seconds" => lease_seconds}) do
-    {:ok, String.to_integer(lease_seconds)}
-  end
-
-  defp lease_time(_) do
-    # three days
-    {:ok, 60 * 60 * 24 * 3}
-  end
-
-  defp valid_topic(%{"hub.topic" => topic}, user) do
-    if topic == OStatus.feed_path(user) do
-      {:ok, OStatus.feed_path(user)}
-    else
-      {:error, "Wrong topic requested, expected #{OStatus.feed_path(user)}, got #{topic}"}
-    end
-  end
-
-  def subscribe(subscriber, subscribed, requester \\ &request_subscription/1) do
-    topic = subscribed.info.topic
-    # FIXME: Race condition, use transactions
-    {:ok, subscription} =
-      with subscription when not is_nil(subscription) <-
-             Repo.get_by(WebsubClientSubscription, topic: topic) do
-        subscribers = [subscriber.ap_id | subscription.subscribers] |> Enum.uniq()
-        change = Ecto.Changeset.change(subscription, %{subscribers: subscribers})
-        Repo.update(change)
-      else
-        _e ->
-          subscription = %WebsubClientSubscription{
-            topic: topic,
-            hub: subscribed.info.hub,
-            subscribers: [subscriber.ap_id],
-            state: "requested",
-            secret: :crypto.strong_rand_bytes(8) |> Base.url_encode64(),
-            user: subscribed
-          }
-
-          Repo.insert(subscription)
-      end
-
-    requester.(subscription)
-  end
-
-  def gather_feed_data(topic, getter \\ &HTTP.get/1) do
-    with {:ok, response} <- getter.(topic),
-         status when status in 200..299 <- response.status,
-         body <- response.body,
-         doc <- XML.parse_document(body),
-         uri when not is_nil(uri) <- XML.string_from_xpath("/feed/author[1]/uri", doc),
-         hub when not is_nil(hub) <- XML.string_from_xpath(~S{/feed/link[@rel="hub"]/@href}, doc) do
-      name = XML.string_from_xpath("/feed/author[1]/name", doc)
-      preferred_username = XML.string_from_xpath("/feed/author[1]/poco:preferredUsername", doc)
-      display_name = XML.string_from_xpath("/feed/author[1]/poco:displayName", doc)
-      avatar = OStatus.make_avatar_object(doc)
-      bio = XML.string_from_xpath("/feed/author[1]/summary", doc)
-
-      {:ok,
-       %{
-         "uri" => uri,
-         "hub" => hub,
-         "nickname" => preferred_username || name,
-         "name" => display_name || name,
-         "host" => URI.parse(uri).host,
-         "avatar" => avatar,
-         "bio" => bio
-       }}
-    else
-      e ->
-        {:error, e}
-    end
-  end
-
-  def request_subscription(websub, poster \\ &HTTP.post/3, timeout \\ 10_000) do
-    data = [
-      "hub.mode": "subscribe",
-      "hub.topic": websub.topic,
-      "hub.secret": websub.secret,
-      "hub.callback": Helpers.websub_url(Endpoint, :websub_subscription_confirmation, websub.id)
-    ]
-
-    # This checks once a second if we are confirmed yet
-    websub_checker = fn ->
-      helper = fn helper ->
-        :timer.sleep(1000)
-        websub = Repo.get_by(WebsubClientSubscription, id: websub.id, state: "accepted")
-        if websub, do: websub, else: helper.(helper)
-      end
-
-      helper.(helper)
-    end
-
-    task = Task.async(websub_checker)
-
-    with {:ok, %{status: 202}} <-
-           poster.(websub.hub, {:form, data}, "Content-type": "application/x-www-form-urlencoded"),
-         {:ok, websub} <- Task.yield(task, timeout) do
-      {:ok, websub}
-    else
-      e ->
-        Task.shutdown(task)
-
-        change = Ecto.Changeset.change(websub, %{state: "rejected"})
-        {:ok, websub} = Repo.update(change)
-
-        Logger.debug(fn -> "Couldn't confirm subscription: #{inspect(websub)}" end)
-        Logger.debug(fn -> "error: #{inspect(e)}" end)
-
-        {:error, websub}
-    end
-  end
-
-  def refresh_subscriptions(delta \\ 60 * 60 * 24) do
-    Logger.debug("Refreshing subscriptions")
-
-    cut_off = NaiveDateTime.add(NaiveDateTime.utc_now(), delta)
-
-    query = from(sub in WebsubClientSubscription, where: sub.valid_until < ^cut_off)
-
-    subs = Repo.all(query)
-
-    Enum.each(subs, fn sub ->
-      Federator.request_subscription(sub)
-    end)
-  end
-
-  def publish_one(%{xml: xml, topic: topic, callback: callback, secret: secret} = params) do
-    signature = sign(secret || "", xml)
-    Logger.info(fn -> "Pushing #{topic} to #{callback}" end)
-
-    with {:ok, %{status: code}} when code in 200..299 <-
-           HTTP.post(
-             callback,
-             xml,
-             [
-               {"Content-Type", "application/atom+xml"},
-               {"X-Hub-Signature", "sha1=#{signature}"}
-             ]
-           ) do
-      if !Map.has_key?(params, :unreachable_since) || params[:unreachable_since],
-        do: Instances.set_reachable(callback)
-
-      Logger.info(fn -> "Pushed to #{callback}, code #{code}" end)
-      {:ok, code}
-    else
-      {_post_result, response} ->
-        unless params[:unreachable_since], do: Instances.set_reachable(callback)
-        Logger.debug(fn -> "Couldn't push to #{callback}, #{inspect(response)}" end)
-        {:error, response}
-    end
-  end
-
-  def gather_webfinger_links(%User{} = user) do
-    [
-      %{
-        "rel" => "http://schemas.google.com/g/2010#updates-from",
-        "type" => "application/atom+xml",
-        "href" => OStatus.feed_path(user)
-      },
-      %{
-        "rel" => "http://ostatus.org/schema/1.0/subscribe",
-        "template" => OStatus.remote_follow_path()
-      }
-    ]
-  end
-
-  def gather_nodeinfo_protocol_names, do: ["ostatus"]
-end

lib/pleroma/web/websub/websub_client_subscription.ex

@@ -1,20 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.Websub.WebsubClientSubscription do
-  use Ecto.Schema
-  alias Pleroma.User
-
-  schema "websub_client_subscriptions" do
-    field(:topic, :string)
-    field(:secret, :string)
-    field(:valid_until, :naive_datetime_usec)
-    field(:state, :string)
-    field(:subscribers, {:array, :string}, default: [])
-    field(:hub, :string)
-    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
-
-    timestamps()
-  end
-end

lib/pleroma/web/websub/websub_controller.ex

@@ -1,99 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.Websub.WebsubController do
-  use Pleroma.Web, :controller
-
-  alias Pleroma.Repo
-  alias Pleroma.User
-  alias Pleroma.Web.Federator
-  alias Pleroma.Web.Websub
-  alias Pleroma.Web.Websub.WebsubClientSubscription
-
-  require Logger
-
-  plug(
-    Pleroma.Web.FederatingPlug
-    when action in [
-           :websub_subscription_request,
-           :websub_subscription_confirmation,
-           :websub_incoming
-         ]
-  )
-
-  def websub_subscription_request(conn, %{"nickname" => nickname} = params) do
-    user = User.get_cached_by_nickname(nickname)
-
-    with {:ok, _websub} <- Websub.incoming_subscription_request(user, params) do
-      conn
-      |> send_resp(202, "Accepted")
-    else
-      {:error, reason} ->
-        conn
-        |> send_resp(500, reason)
-    end
-  end
-
-  # TODO: Extract this into the Websub module
-  def websub_subscription_confirmation(
-        conn,
-        %{
-          "id" => id,
-          "hub.mode" => "subscribe",
-          "hub.challenge" => challenge,
-          "hub.topic" => topic
-        } = params
-      ) do
-    Logger.debug("Got WebSub confirmation")
-    Logger.debug(inspect(params))
-
-    lease_seconds =
-      if params["hub.lease_seconds"] do
-        String.to_integer(params["hub.lease_seconds"])
-      else
-        # Guess 3 days
-        60 * 60 * 24 * 3
-      end
-
-    with %WebsubClientSubscription{} = websub <-
-           Repo.get_by(WebsubClientSubscription, id: id, topic: topic) do
-      valid_until = NaiveDateTime.add(NaiveDateTime.utc_now(), lease_seconds)
-      change = Ecto.Changeset.change(websub, %{state: "accepted", valid_until: valid_until})
-      {:ok, _websub} = Repo.update(change)
-
-      conn
-      |> send_resp(200, challenge)
-    else
-      _e ->
-        conn
-        |> send_resp(500, "Error")
-    end
-  end
-
-  def websub_subscription_confirmation(conn, params) do
-    Logger.info("Invalid WebSub confirmation request: #{inspect(params)}")
-
-    conn
-    |> send_resp(500, "Invalid parameters")
-  end
-
-  def websub_incoming(conn, %{"id" => id}) do
-    with "sha1=" <> signature <- hd(get_req_header(conn, "x-hub-signature")),
-         signature <- String.downcase(signature),
-         %WebsubClientSubscription{} = websub <- Repo.get(WebsubClientSubscription, id),
-         {:ok, body, _conn} = read_body(conn),
-         ^signature <- Websub.sign(websub.secret, body) do
-      Federator.incoming_doc(body)
-
-      conn
-      |> send_resp(200, "OK")
-    else
-      _e ->
-        Logger.debug("Can't handle incoming subscription post")
-
-        conn
-        |> send_resp(500, "Error")
-    end
-  end
-end

lib/pleroma/web/websub/websub_server_subscription.ex

@@ -1,17 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.Websub.WebsubServerSubscription do
-  use Ecto.Schema
-
-  schema "websub_server_subscriptions" do
-    field(:topic, :string)
-    field(:callback, :string)
-    field(:secret, :string)
-    field(:valid_until, :naive_datetime)
-    field(:state, :string)
-
-    timestamps()
-  end
-end

lib/pleroma/workers/receiver_worker.ex

@@ -8,10 +8,6 @@ defmodule Pleroma.Workers.ReceiverWorker do
   use Pleroma.Workers.WorkerHelper, queue: "federator_incoming"
 
   @impl Oban.Worker
-  def perform(%{"op" => "incoming_doc", "body" => doc}, _job) do
-    Federator.perform(:incoming_doc, doc)
-  end
-
   def perform(%{"op" => "incoming_ap_doc", "params" => params}, _job) do
     Federator.perform(:incoming_ap_doc, params)
   end

lib/pleroma/workers/subscriber_worker.ex

@@ -1,26 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Workers.SubscriberWorker do
-  alias Pleroma.Repo
-  alias Pleroma.Web.Federator
-  alias Pleroma.Web.Websub
-
-  use Pleroma.Workers.WorkerHelper, queue: "federator_outgoing"
-
-  @impl Oban.Worker
-  def perform(%{"op" => "refresh_subscriptions"}, _job) do
-    Federator.perform(:refresh_subscriptions)
-  end
-
-  def perform(%{"op" => "request_subscription", "websub_id" => websub_id}, _job) do
-    websub = Repo.get(Websub.WebsubClientSubscription, websub_id)
-    Federator.perform(:request_subscription, websub)
-  end
-
-  def perform(%{"op" => "verify_websub", "websub_id" => websub_id}, _job) do
-    websub = Repo.get(Websub.WebsubServerSubscription, websub_id)
-    Federator.perform(:verify_websub, websub)
-  end
-end

mix.exs

@@ -4,7 +4,7 @@ defmodule Pleroma.Mixfile do
   def project do
     [
       app: :pleroma,
-      version: version("1.0.0"),
+      version: version("1.1.50"),
       elixir: "~> 1.8",
       elixirc_paths: elixirc_paths(Mix.env()),
       compilers: [:phoenix, :gettext] ++ Mix.compilers(),

priv/repo/migrations/20190711042021_create_safe_jsonb_set.exs

@@ -0,0 +1,22 @@
+defmodule Pleroma.Repo.Migrations.CreateSafeJsonbSet do
+  use Ecto.Migration
+  alias Pleroma.User
+
+  def change do
+    execute("""
+    create or replace function safe_jsonb_set(target jsonb, path text[], new_value jsonb, create_missing boolean default true) returns jsonb as $$
+    declare
+      result jsonb;
+    begin
+      result := jsonb_set(target, path, coalesce(new_value, 'null'::jsonb), create_missing);
+      if result is NULL then
+        raise 'jsonb_set tried to wipe the object, please report this incindent to Pleroma bug tracker. https://git.pleroma.social/pleroma/pleroma/issues/new';
+        return target;
+      else
+        return result;
+      end if;
+    end;
+    $$ language plpgsql;
+    """)
+  end
+end

priv/repo/migrations/20190711042024_copy_muted_to_muted_notifications.exs

@@ -4,7 +4,7 @@ defmodule Pleroma.Repo.Migrations.CopyMutedToMutedNotifications do
 
   def change do
     execute(
-      "update users set info = jsonb_set(info, '{muted_notifications}', info->'mutes', true) where local = true"
+      "update users set info = safe_jsonb_set(info, '{muted_notifications}', info->'mutes', true) where local = true"
     )
   end
 end

priv/repo/migrations/20191014181019_create_markers.exs

@@ -0,0 +1,15 @@
+defmodule Pleroma.Repo.Migrations.CreateMarkers do
+  use Ecto.Migration
+
+  def change do
+    create_if_not_exists table(:markers) do
+      add(:user_id, references(:users, type: :uuid, on_delete: :delete_all))
+      add(:timeline, :string, default: "", null: false)
+      add(:last_read_id, :string, default: "", null: false)
+      add(:lock_version, :integer, default: 0, null: false)
+      timestamps()
+    end
+
+    create_if_not_exists(unique_index(:markers, [:user_id, :timeline]))
+  end
+end

priv/repo/migrations/20191017225002_drop_websub_tables.exs

@@ -0,0 +1,8 @@
+defmodule Pleroma.Repo.Migrations.DropWebsubTables do
+  use Ecto.Migration
+
+  def change do
+    drop_if_exists(table(:websub_client_subscriptions))
+    drop_if_exists(table(:websub_server_subscriptions))
+  end
+end

priv/static/schemas/litepub-0.1.jsonld

@@ -19,6 +19,7 @@
             "value": "schema:value",
             "sensitive": "as:sensitive",
             "litepub": "http://litepub.social/ns#",
+            "invisible": "litepub:invisible",
             "directMessage": "litepub:directMessage",
             "listMessage": {
                 "@id": "litepub:listMessage",

rel/files/bin/pleroma_ctl

@@ -141,8 +141,8 @@ else
 
 	ACTION="$1"
 	shift
-
+        echo "$1" | grep "^-" >/dev/null
-	if [ "$(echo \"$1\" | grep \"^-\" >/dev/null)" = false ]; then
+	if [ $? -eq 1 ]; then
 		SUBACTION="$1"
 		shift
 	fi

test/conversation/participation_test.exs

@@ -133,6 +133,20 @@ test "it marks a participation as unread" do
     refute participation.read
   end
 
+  test "it marks all the user's participations as read" do
+    user = insert(:user)
+    other_user = insert(:user)
+    participation1 = insert(:participation, %{read: false, user: user})
+    participation2 = insert(:participation, %{read: false, user: user})
+    participation3 = insert(:participation, %{read: false, user: other_user})
+
+    {:ok, [%{read: true}, %{read: true}]} = Participation.mark_all_as_read(user)
+
+    assert Participation.get(participation1.id).read == true
+    assert Participation.get(participation2.id).read == true
+    assert Participation.get(participation3.id).read == false
+  end
+
   test "gets all the participations for a user, ordered by updated at descending" do
     user = insert(:user)
     {:ok, activity_one} = CommonAPI.post(user, %{"status" => "x", "visibility" => "direct"})

test/fixtures/tesla_mock/https___shitposter.club_notice_2827873.json

@@ -0,0 +1 @@
+{"@context":["https://www.w3.org/ns/activitystreams","https://shitposter.club/schemas/litepub-0.1.jsonld",{"@language":"und"}],"actor":"https://shitposter.club/users/moonman","attachment":[],"attributedTo":"https://shitposter.club/users/moonman","cc":["https://shitposter.club/users/moonman/followers"],"content":"@<a href=\"https://shitposter.club/users/9655\" class=\"h-card mention\" title=\"Solidarity for Pigs\">neimzr4luzerz</a> @<a href=\"https://gs.smuglo.li/user/2326\" class=\"h-card mention\" title=\"Dolus_McHonest\">dolus</a> childhood poring over Strong's concordance and a koine Greek dictionary, fast forward to 2017 and some fuckstick who translates japanese jackoff material tells me you just need to make it sound right in English","context":"tag:shitposter.club,2017-05-05:objectType=thread:nonce=3c16e9c2681f6d26","conversation":"tag:shitposter.club,2017-05-05:objectType=thread:nonce=3c16e9c2681f6d26","id":"tag:shitposter.club,2017-05-05:noticeId=2827873:objectType=comment","inReplyTo":"tag:shitposter.club,2017-05-05:noticeId=2827849:objectType=comment","inReplyToStatusId":2827849,"published":"2017-05-05T08:51:48Z","sensitive":false,"summary":null,"tag":[],"to":["https://www.w3.org/ns/activitystreams#Public"],"type":"Note"}

test/fixtures/tesla_mock/moonman@shitposter.club.json

@@ -0,0 +1 @@
+{"@context":["https://www.w3.org/ns/activitystreams","https://shitposter.club/schemas/litepub-0.1.jsonld",{"@language":"und"}],"attachment":[],"endpoints":{"oauthAuthorizationEndpoint":"https://shitposter.club/oauth/authorize","oauthRegistrationEndpoint":"https://shitposter.club/api/v1/apps","oauthTokenEndpoint":"https://shitposter.club/oauth/token","sharedInbox":"https://shitposter.club/inbox"},"followers":"https://shitposter.club/users/moonman/followers","following":"https://shitposter.club/users/moonman/following","icon":{"type":"Image","url":"https://shitposter.club/media/bda6e00074f6a02cbf32ddb0abec08151eb4c795e580927ff7ad638d00cde4c8.jpg?name=blob.jpg"},"id":"https://shitposter.club/users/moonman","image":{"type":"Image","url":"https://shitposter.club/media/4eefb90d-cdb2-2b4f-5f29-7612856a99d2/4eefb90d-cdb2-2b4f-5f29-7612856a99d2.jpeg"},"inbox":"https://shitposter.club/users/moonman/inbox","manuallyApprovesFollowers":false,"name":"Captain Howdy","outbox":"https://shitposter.club/users/moonman/outbox","preferredUsername":"moonman","publicKey":{"id":"https://shitposter.club/users/moonman#main-key","owner":"https://shitposter.club/users/moonman","publicKeyPem":"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnOTitJ19ZqcOZHwSXQUM\nJq9ip4GNblp83LgwG1t5c2h2iaI3fXMsB4EaEBs8XHsoSFyDeDNRSPE3mtVgOnWv\n1eaXWMDerBT06th6DrElD9k5IoEPtZRY4HtZa1xGnte7+6RjuPOzZ1fR9C8WxGgi\nwb9iOUMhazpo85fC3iKCAL5XhiuA3Nas57MDJgueeI9BF+2oFelFZdMSWwG96uch\niDfp8nfpkmzYI6SWbylObjm8RsfZbGTosLHwWyJPEITeYI/5M0XwJe9dgVI1rVNU\n52kplWOGTo1rm6V0AMHaYAd9RpiXxe8xt5OeranrsE/5LvEQUl0fz7SE36YmsOaH\nTwIDAQAB\n-----END PUBLIC KEY-----\n\n"},"summary":"EMAIL:shitposterclub@gmail.com<br>XMPP: moon@talk.shitposter.club<br>PRONOUNS: none of your business<br><br>Purported leftist kike piece of shit","tag":[],"type":"Person","url":"https://shitposter.club/users/moonman"}

test/fixtures/tesla_mock/relay@mastdon.example.org.json

@@ -0,0 +1,55 @@
+{
+  "@context": ["https://www.w3.org/ns/activitystreams", "https://w3id.org/security/v1", {
+    "manuallyApprovesFollowers": "as:manuallyApprovesFollowers",
+    "sensitive": "as:sensitive",
+    "movedTo": "as:movedTo",
+    "Hashtag": "as:Hashtag",
+    "ostatus": "http://ostatus.org#",
+    "atomUri": "ostatus:atomUri",
+    "inReplyToAtomUri": "ostatus:inReplyToAtomUri",
+    "conversation": "ostatus:conversation",
+    "toot": "http://joinmastodon.org/ns#",
+    "Emoji": "toot:Emoji"
+  }],
+  "id": "http://mastodon.example.org/users/admin",
+  "type": "Application",
+  "invisible": true,
+  "following": "http://mastodon.example.org/users/admin/following",
+  "followers": "http://mastodon.example.org/users/admin/followers",
+  "inbox": "http://mastodon.example.org/users/admin/inbox",
+  "outbox": "http://mastodon.example.org/users/admin/outbox",
+  "preferredUsername": "admin",
+  "name": null,
+  "summary": "\u003cp\u003e\u003c/p\u003e",
+  "url": "http://mastodon.example.org/@admin",
+  "manuallyApprovesFollowers": false,
+  "publicKey": {
+    "id": "http://mastodon.example.org/users/admin#main-key",
+    "owner": "http://mastodon.example.org/users/admin",
+    "publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtc4Tir+3ADhSNF6VKrtW\nOU32T01w7V0yshmQei38YyiVwVvFu8XOP6ACchkdxbJ+C9mZud8qWaRJKVbFTMUG\nNX4+6Q+FobyuKrwN7CEwhDALZtaN2IPbaPd6uG1B7QhWorrY+yFa8f2TBM3BxnUy\nI4T+bMIZIEYG7KtljCBoQXuTQmGtuffO0UwJksidg2ffCF5Q+K//JfQagJ3UzrR+\nZXbKMJdAw4bCVJYs4Z5EhHYBwQWiXCyMGTd7BGlmMkY6Av7ZqHKC/owp3/0EWDNz\nNqF09Wcpr3y3e8nA10X40MJqp/wR+1xtxp+YGbq/Cj5hZGBG7etFOmIpVBrDOhry\nBwIDAQAB\n-----END PUBLIC KEY-----\n"
+  },
+  "attachment": [{
+      "type": "PropertyValue",
+      "name": "foo",
+      "value": "bar"
+    },
+    {
+      "type": "PropertyValue",
+      "name": "foo1",
+      "value": "bar1"
+    }
+  ],
+  "endpoints": {
+    "sharedInbox": "http://mastodon.example.org/inbox"
+  },
+  "icon": {
+    "type": "Image",
+    "mediaType": "image/jpeg",
+    "url": "https://cdn.niu.moe/accounts/avatars/000/033/323/original/fd7f8ae0b3ffedc9.jpeg"
+  },
+  "image": {
+    "type": "Image",
+    "mediaType": "image/png",
+    "url": "https://cdn.niu.moe/accounts/headers/000/033/323/original/850b3448fa5fd477.png"
+  }
+}

test/marker_test.exs

@@ -0,0 +1,51 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.MarkerTest do
+  use Pleroma.DataCase
+  alias Pleroma.Marker
+
+  import Pleroma.Factory
+
+  describe "get_markers/2" do
+    test "returns user markers" do
+      user = insert(:user)
+      marker = insert(:marker, user: user)
+      insert(:marker, timeline: "home", user: user)
+      assert Marker.get_markers(user, ["notifications"]) == [refresh_record(marker)]
+    end
+  end
+
+  describe "upsert/2" do
+    test "creates a marker" do
+      user = insert(:user)
+
+      {:ok, %{"notifications" => %Marker{} = marker}} =
+        Marker.upsert(
+          user,
+          %{"notifications" => %{"last_read_id" => "34"}}
+        )
+
+      assert marker.timeline == "notifications"
+      assert marker.last_read_id == "34"
+      assert marker.lock_version == 0
+    end
+
+    test "updates exist marker" do
+      user = insert(:user)
+      marker = insert(:marker, user: user, last_read_id: "8909")
+
+      {:ok, %{"notifications" => %Marker{}}} =
+        Marker.upsert(
+          user,
+          %{"notifications" => %{"last_read_id" => "9909"}}
+        )
+
+      marker = refresh_record(marker)
+      assert marker.timeline == "notifications"
+      assert marker.last_read_id == "9909"
+      assert marker.lock_version == 0
+    end
+  end
+end

test/notification_test.exs

@@ -683,7 +683,7 @@ test "it doesn't return notifications for muted thread" do
       assert Notification.for_user(user) == []
     end
 
-    test "it returns notifications for muted user with notifications and with_muted parameter" do
+    test "it returns notifications from a muted user when with_muted is set" do
       user = insert(:user)
       muted = insert(:user)
       {:ok, user} = User.mute(user, muted)
@@ -693,27 +693,27 @@ test "it returns notifications for muted user with notifications and with_muted
       assert length(Notification.for_user(user, %{with_muted: true})) == 1
     end
 
-    test "it returns notifications for blocked user and with_muted parameter" do
+    test "it doesn't return notifications from a blocked user when with_muted is set" do
       user = insert(:user)
       blocked = insert(:user)
       {:ok, user} = User.block(user, blocked)
 
       {:ok, _activity} = CommonAPI.post(blocked, %{"status" => "hey @#{user.nickname}"})
 
-      assert length(Notification.for_user(user, %{with_muted: true})) == 1
+      assert length(Notification.for_user(user, %{with_muted: true})) == 0
     end
 
-    test "it returns notificatitons for blocked domain and with_muted parameter" do
+    test "it doesn't return notifications from a domain-blocked user when with_muted is set" do
       user = insert(:user)
       blocked = insert(:user, ap_id: "http://some-domain.com")
       {:ok, user} = User.block_domain(user, "some-domain.com")
 
       {:ok, _activity} = CommonAPI.post(blocked, %{"status" => "hey @#{user.nickname}"})
 
-      assert length(Notification.for_user(user, %{with_muted: true})) == 1
+      assert length(Notification.for_user(user, %{with_muted: true})) == 0
     end
 
-    test "it returns notifications for muted thread with_muted parameter" do
+    test "it returns notifications from muted threads when with_muted is set" do
       user = insert(:user)
       another_user = insert(:user)
 

test/object/containment_test.exs

@@ -65,7 +65,7 @@ test "users cannot be collided through fake direction spoofing attempts" do
       assert capture_log(fn ->
                {:error, _} = User.get_or_fetch_by_ap_id("https://n1u.moe/users/rye")
              end) =~
-               "[error] Could not decode user at fetch https://n1u.moe/users/rye, {:error, :error}"
+               "[error] Could not decode user at fetch https://n1u.moe/users/rye"
     end
   end
 

test/object/fetcher_test.exs

@@ -27,31 +27,16 @@ defmodule Pleroma.Object.FetcherTest do
   end
 
   describe "actor origin containment" do
-    test_with_mock "it rejects objects with a bogus origin",
+    test "it rejects objects with a bogus origin" do
-                   Pleroma.Web.OStatus,
-                   [:passthrough],
-                   [] do
       {:error, _} = Fetcher.fetch_object_from_id("https://info.pleroma.site/activity.json")
-
-      refute called(Pleroma.Web.OStatus.fetch_activity_from_url(:_))
     end
 
-    test_with_mock "it rejects objects when attributedTo is wrong (variant 1)",
+    test "it rejects objects when attributedTo is wrong (variant 1)" do
-                   Pleroma.Web.OStatus,
-                   [:passthrough],
-                   [] do
       {:error, _} = Fetcher.fetch_object_from_id("https://info.pleroma.site/activity2.json")
-
-      refute called(Pleroma.Web.OStatus.fetch_activity_from_url(:_))
     end
 
-    test_with_mock "it rejects objects when attributedTo is wrong (variant 2)",
+    test "it rejects objects when attributedTo is wrong (variant 2)" do
-                   Pleroma.Web.OStatus,
-                   [:passthrough],
-                   [] do
       {:error, _} = Fetcher.fetch_object_from_id("https://info.pleroma.site/activity3.json")
-
-      refute called(Pleroma.Web.OStatus.fetch_activity_from_url(:_))
     end
   end
 
@@ -71,24 +56,6 @@ test "it fetches an object" do
 
       assert object == object_again
     end
-
-    test "it works with objects only available via Ostatus" do
-      {:ok, object} = Fetcher.fetch_object_from_id("https://shitposter.club/notice/2827873")
-      assert activity = Activity.get_create_by_object_ap_id(object.data["id"])
-      assert activity.data["id"]
-
-      {:ok, object_again} = Fetcher.fetch_object_from_id("https://shitposter.club/notice/2827873")
-
-      assert object == object_again
-    end
-
-    test "it correctly stitches up conversations between ostatus and ap" do
-      last = "https://mstdn.io/users/mayuutann/statuses/99568293732299394"
-      {:ok, object} = Fetcher.fetch_object_from_id(last)
-
-      object = Object.get_by_ap_id(object.data["inReplyTo"])
-      assert object
-    end
   end
 
   describe "implementation quirks" do

test/safe_jsonb_set_test.exs

@@ -0,0 +1,12 @@
+defmodule Pleroma.SafeJsonbSetTest do
+  use Pleroma.DataCase
+
+  test "it doesn't wipe the object when asked to set the value to NULL" do
+    assert %{rows: [[%{"key" => "value", "test" => nil}]]} =
+             Ecto.Adapters.SQL.query!(
+               Pleroma.Repo,
+               "select safe_jsonb_set('{\"key\": \"value\"}'::jsonb, '{test}', NULL);",
+               []
+             )
+  end
+end

test/signature_test.exs

@@ -69,8 +69,7 @@ test "it returns key" do
 
     test "it returns error when not found user" do
       assert capture_log(fn ->
-               assert Signature.refetch_public_key(make_fake_conn("test-ap_id")) ==
+               {:error, _} = Signature.refetch_public_key(make_fake_conn("test-ap_id"))
-                        {:error, {:error, :ok}}
              end) =~ "[error] Could not decode user"
     end
   end

test/support/factory.ex

@@ -281,26 +281,6 @@ def follow_activity_factory do
     }
   end
 
-  def websub_subscription_factory do
-    %Pleroma.Web.Websub.WebsubServerSubscription{
-      topic: "http://example.org",
-      callback: "http://example.org/callback",
-      secret: "here's a secret",
-      valid_until: NaiveDateTime.add(NaiveDateTime.utc_now(), 100),
-      state: "requested"
-    }
-  end
-
-  def websub_client_subscription_factory do
-    %Pleroma.Web.Websub.WebsubClientSubscription{
-      topic: "http://example.org",
-      secret: "here's a secret",
-      valid_until: nil,
-      state: "requested",
-      subscribers: []
-    }
-  end
-
   def oauth_app_factory do
     %Pleroma.Web.OAuth.App{
       client_name: "Some client",
@@ -397,4 +377,13 @@ def config_factory do
         )
     }
   end
+
+  def marker_factory do
+    %Pleroma.Marker{
+      user: build(:user),
+      timeline: "notifications",
+      lock_version: 0,
+      last_read_id: "1"
+    }
+  end
 end

test/support/http_request_mock.ex

@@ -38,6 +38,14 @@ def get("https://osada.macgirvin.com/channel/mike", _, _, _) do
      }}
   end
 
+  def get("https://shitposter.club/users/moonman", _, _, _) do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/tesla_mock/moonman@shitposter.club.json")
+     }}
+  end
+
   def get("https://mastodon.social/users/emelie/statuses/101849165031453009", _, _, _) do
     {:ok,
      %Tesla.Env{
@@ -340,6 +348,14 @@ def get("http://mastodon.example.org/users/admin", _, _, Accept: "application/ac
      }}
   end
 
+  def get("http://mastodon.example.org/users/relay", _, _, Accept: "application/activity+json") do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/tesla_mock/relay@mastdon.example.org.json")
+     }}
+  end
+
   def get("http://mastodon.example.org/users/gargron", _, _, Accept: "application/activity+json") do
     {:error, :nxdomain}
   end
@@ -620,7 +636,7 @@ def get("https://shitposter.club/notice/2827873", _, _, _) do
     {:ok,
      %Tesla.Env{
        status: 200,
-       body: File.read!("test/fixtures/tesla_mock/https___shitposter.club_notice_2827873.html")
+       body: File.read!("test/fixtures/tesla_mock/https___shitposter.club_notice_2827873.json")
      }}
   end
 

test/user_test.exs

@@ -190,23 +190,6 @@ test "local users do not automatically follow local locked accounts" do
     refute User.following?(follower, followed)
   end
 
-  # This is a somewhat useless test.
-  # test "following a remote user will ensure a websub subscription is present" do
-  #   user = insert(:user)
-  #   {:ok, followed} = OStatus.make_user("shp@social.heldscal.la")
-
-  #   assert followed.local == false
-
-  #   {:ok, user} = User.follow(user, followed)
-  #   assert User.ap_followers(followed) in user.following
-
-  #   query = from w in WebsubClientSubscription,
-  #   where: w.topic == ^followed.info["topic"]
-  #   websub = Repo.one(query)
-
-  #   assert websub
-  # end
-
   describe "unfollow/2" do
     setup do
       setting = Pleroma.Config.get([:instance, :external_user_synchronization])
@@ -474,11 +457,6 @@ test "gets an existing user by fully qualified nickname, case insensitive" do
       assert user == fetched_user
     end
 
-    test "fetches an external user via ostatus if no user exists" do
-      {:ok, fetched_user} = User.get_or_fetch_by_nickname("shp@social.heldscal.la")
-      assert fetched_user.nickname == "shp@social.heldscal.la"
-    end
-
     test "returns nil if no user could be fetched" do
       {:error, fetched_user} = User.get_or_fetch_by_nickname("nonexistant@social.heldscal.la")
       assert fetched_user == "not found nonexistant@social.heldscal.la"
@@ -1254,6 +1232,20 @@ test "returns true for local admins" do
     end
   end
 
+  describe "invisible?/1" do
+    test "returns true for an invisible user" do
+      user = insert(:user, local: true, info: %{invisible: true})
+
+      assert User.invisible?(user)
+    end
+
+    test "returns false for a non-invisible user" do
+      user = insert(:user, local: true)
+
+      refute User.invisible?(user)
+    end
+  end
+
   describe "visible_for?/2" do
     test "returns true when the account is itself" do
       user = insert(:user, local: true)

test/web/activity_pub/activity_pub_test.exs

@@ -179,6 +179,12 @@ test "it returns a user" do
       assert user.follower_address == "http://mastodon.example.org/users/admin/followers"
     end
 
+    test "it returns a user that is invisible" do
+      user_id = "http://mastodon.example.org/users/relay"
+      {:ok, user} = ActivityPub.make_user_from_ap_id(user_id)
+      assert User.invisible?(user)
+    end
+
     test "it fetches the appropriate tag-restricted posts" do
       user = insert(:user)
 

test/web/activity_pub/relay_test.exs

@@ -7,6 +7,7 @@ defmodule Pleroma.Web.ActivityPub.RelayTest do
 
   alias Pleroma.Activity
   alias Pleroma.Object
+  alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.ActivityPub.Relay
 
@@ -19,11 +20,16 @@ test "gets an actor for the relay" do
     assert user.ap_id == "#{Pleroma.Web.Endpoint.url()}/relay"
   end
 
+  test "relay actor is invisible" do
+    user = Relay.get_actor()
+    assert User.invisible?(user)
+  end
+
   describe "follow/1" do
     test "returns errors when user not found" do
       assert capture_log(fn ->
-               assert Relay.follow("test-ap-id") == {:error, "Could not fetch by AP id"}
+               {:error, _} = Relay.follow("test-ap-id")
-             end) =~ "Could not fetch by AP id"
+             end) =~ "Could not decode user at fetch"
     end
 
     test "returns activity" do
@@ -41,8 +47,8 @@ test "returns activity" do
   describe "unfollow/1" do
     test "returns errors when user not found" do
       assert capture_log(fn ->
-               assert Relay.unfollow("test-ap-id") == {:error, "Could not fetch by AP id"}
+               {:error, _} = Relay.unfollow("test-ap-id")
-             end) =~ "Could not fetch by AP id"
+             end) =~ "Could not decode user at fetch"
     end
 
     test "returns activity" do

test/web/activity_pub/transmogrifier_test.exs

@@ -7,14 +7,11 @@ defmodule Pleroma.Web.ActivityPub.TransmogrifierTest do
   alias Pleroma.Activity
   alias Pleroma.Object
   alias Pleroma.Object.Fetcher
-  alias Pleroma.Repo
   alias Pleroma.Tests.ObanHelpers
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.ActivityPub.Transmogrifier
   alias Pleroma.Web.CommonAPI
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.Websub.WebsubClientSubscription
 
   import Mock
   import Pleroma.Factory
@@ -1181,32 +1178,6 @@ test "it sets the 'attributedTo' property to the actor of the object if it doesn
       assert modified["object"]["actor"] == modified["object"]["attributedTo"]
     end
 
-    test "it translates ostatus IDs to external URLs" do
-      incoming = File.read!("test/fixtures/incoming_note_activity.xml")
-      {:ok, [referent_activity]} = OStatus.handle_incoming(incoming)
-
-      user = insert(:user)
-
-      {:ok, activity, _} = CommonAPI.favorite(referent_activity.id, user)
-      {:ok, modified} = Transmogrifier.prepare_outgoing(activity.data)
-
-      assert modified["object"] == "http://gs.example.org:4040/index.php/notice/29"
-    end
-
-    test "it translates ostatus reply_to IDs to external URLs" do
-      incoming = File.read!("test/fixtures/incoming_note_activity.xml")
-      {:ok, [referred_activity]} = OStatus.handle_incoming(incoming)
-
-      user = insert(:user)
-
-      {:ok, activity} =
-        CommonAPI.post(user, %{"status" => "HI!", "in_reply_to_status_id" => referred_activity.id})
-
-      {:ok, modified} = Transmogrifier.prepare_outgoing(activity.data)
-
-      assert modified["object"]["inReplyTo"] == "http://gs.example.org:4040/index.php/notice/29"
-    end
-
     test "it strips internal hashtag data" do
       user = insert(:user)
 
@@ -1371,21 +1342,6 @@ test "it upgrades a user to activitypub" do
     end
   end
 
-  describe "maybe_retire_websub" do
-    test "it deletes all websub client subscripitions with the user as topic" do
-      subscription = %WebsubClientSubscription{topic: "https://niu.moe/users/rye.atom"}
-      {:ok, ws} = Repo.insert(subscription)
-
-      subscription = %WebsubClientSubscription{topic: "https://niu.moe/users/pasty.atom"}
-      {:ok, ws2} = Repo.insert(subscription)
-
-      Transmogrifier.maybe_retire_websub("https://niu.moe/users/rye")
-
-      refute Repo.get(WebsubClientSubscription, ws.id)
-      assert Repo.get(WebsubClientSubscription, ws2.id)
-    end
-  end
-
   describe "actor rewriting" do
     test "it fixes the actor URL property to be a proper URI" do
       data = %{

test/web/activity_pub/views/user_view_test.exs

@@ -76,6 +76,12 @@ test "Does not add an avatar image if the user hasn't set one" do
     assert result["image"]["url"] == "https://somebanner"
   end
 
+  test "renders an invisible user with the invisible property set to true" do
+    user = insert(:user, %{info: %{invisible: true}})
+
+    assert %{"invisible" => true} = UserView.render("service.json", %{user: user})
+  end
+
   describe "endpoints" do
     test "local users have a usable endpoints structure" do
       user = insert(:user)

test/web/admin_api/admin_api_controller_test.exs

@@ -17,6 +17,12 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIControllerTest do
   alias Pleroma.Web.MediaProxy
   import Pleroma.Factory
 
+  setup_all do
+    Tesla.Mock.mock_global(fn env -> apply(HttpRequestMock, :request, [env]) end)
+
+    :ok
+  end
+
   describe "DELETE /api/pleroma/admin/users" do
     test "single user" do
       admin = insert(:user, info: %{is_admin: true})
@@ -2541,6 +2547,74 @@ test "sets password_reset_pending to true", %{admin: admin, user: user} do
       assert User.get_by_id(user.id).info.password_reset_pending == true
     end
   end
+
+  describe "relays" do
+    setup %{conn: conn} do
+      admin = insert(:user, info: %{is_admin: true})
+
+      %{conn: assign(conn, :user, admin), admin: admin}
+    end
+
+    test "POST /relay", %{admin: admin} do
+      conn =
+        build_conn()
+        |> assign(:user, admin)
+        |> post("/api/pleroma/admin/relay", %{
+          relay_url: "http://mastodon.example.org/users/admin"
+        })
+
+      assert json_response(conn, 200) == "http://mastodon.example.org/users/admin"
+
+      log_entry = Repo.one(ModerationLog)
+
+      assert ModerationLog.get_log_entry_message(log_entry) ==
+               "@#{admin.nickname} followed relay: http://mastodon.example.org/users/admin"
+    end
+
+    test "GET /relay", %{admin: admin} do
+      Pleroma.Web.ActivityPub.Relay.get_actor()
+      |> Ecto.Changeset.change(
+        following: [
+          "http://test-app.com/user/test1",
+          "http://test-app.com/user/test1",
+          "http://test-app-42.com/user/test1"
+        ]
+      )
+      |> Pleroma.User.update_and_set_cache()
+
+      conn =
+        build_conn()
+        |> assign(:user, admin)
+        |> get("/api/pleroma/admin/relay")
+
+      assert json_response(conn, 200)["relays"] -- ["test-app.com", "test-app-42.com"] == []
+    end
+
+    test "DELETE /relay", %{admin: admin} do
+      build_conn()
+      |> assign(:user, admin)
+      |> post("/api/pleroma/admin/relay", %{
+        relay_url: "http://mastodon.example.org/users/admin"
+      })
+
+      conn =
+        build_conn()
+        |> assign(:user, admin)
+        |> delete("/api/pleroma/admin/relay", %{
+          relay_url: "http://mastodon.example.org/users/admin"
+        })
+
+      assert json_response(conn, 200) == "http://mastodon.example.org/users/admin"
+
+      [log_entry_one, log_entry_two] = Repo.all(ModerationLog)
+
+      assert ModerationLog.get_log_entry_message(log_entry_one) ==
+               "@#{admin.nickname} followed relay: http://mastodon.example.org/users/admin"
+
+      assert ModerationLog.get_log_entry_message(log_entry_two) ==
+               "@#{admin.nickname} unfollowed relay: http://mastodon.example.org/users/admin"
+    end
+  end
 end
 
 # Needed for testing

test/web/federator_test.exs

@@ -111,93 +111,6 @@ test "it federates only to reachable instances via AP" do
                all_enqueued(worker: PublisherWorker)
              )
     end
-
-    test "it federates only to reachable instances via Websub" do
-      user = insert(:user)
-      websub_topic = Pleroma.Web.OStatus.feed_path(user)
-
-      sub1 =
-        insert(:websub_subscription, %{
-          topic: websub_topic,
-          state: "active",
-          callback: "http://pleroma.soykaf.com/cb"
-        })
-
-      sub2 =
-        insert(:websub_subscription, %{
-          topic: websub_topic,
-          state: "active",
-          callback: "https://pleroma2.soykaf.com/cb"
-        })
-
-      dt = NaiveDateTime.utc_now()
-      Instances.set_unreachable(sub2.callback, dt)
-
-      Instances.set_consistently_unreachable(sub1.callback)
-
-      {:ok, _activity} = CommonAPI.post(user, %{"status" => "HI"})
-
-      expected_callback = sub2.callback
-      expected_dt = NaiveDateTime.to_iso8601(dt)
-
-      ObanHelpers.perform(all_enqueued(worker: PublisherWorker))
-
-      assert ObanHelpers.member?(
-               %{
-                 "op" => "publish_one",
-                 "params" => %{
-                   "callback" => expected_callback,
-                   "unreachable_since" => expected_dt
-                 }
-               },
-               all_enqueued(worker: PublisherWorker)
-             )
-    end
-
-    test "it federates only to reachable instances via Salmon" do
-      user = insert(:user)
-
-      _remote_user1 =
-        insert(:user, %{
-          local: false,
-          nickname: "nick1@domain.com",
-          ap_id: "https://domain.com/users/nick1",
-          info: %{salmon: "https://domain.com/salmon"}
-        })
-
-      remote_user2 =
-        insert(:user, %{
-          local: false,
-          nickname: "nick2@domain2.com",
-          ap_id: "https://domain2.com/users/nick2",
-          info: %{salmon: "https://domain2.com/salmon"}
-        })
-
-      remote_user2_id = remote_user2.id
-
-      dt = NaiveDateTime.utc_now()
-      Instances.set_unreachable(remote_user2.ap_id, dt)
-
-      Instances.set_consistently_unreachable("domain.com")
-
-      {:ok, _activity} =
-        CommonAPI.post(user, %{"status" => "HI @nick1@domain.com, @nick2@domain2.com!"})
-
-      expected_dt = NaiveDateTime.to_iso8601(dt)
-
-      ObanHelpers.perform(all_enqueued(worker: PublisherWorker))
-
-      assert ObanHelpers.member?(
-               %{
-                 "op" => "publish_one",
-                 "params" => %{
-                   "recipient_id" => remote_user2_id,
-                   "unreachable_since" => expected_dt
-                 }
-               },
-               all_enqueued(worker: PublisherWorker)
-             )
-    end
   end
 
   describe "Receive an activity" do

test/web/mastodon_api/controllers/marker_controller_test.exs

@@ -0,0 +1,124 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.MarkerControllerTest do
+  use Pleroma.Web.ConnCase
+
+  import Pleroma.Factory
+
+  describe "GET /api/v1/markers" do
+    test "gets markers with correct scopes", %{conn: conn} do
+      user = insert(:user)
+      token = insert(:oauth_token, user: user, scopes: ["read:statuses"])
+
+      {:ok, %{"notifications" => marker}} =
+        Pleroma.Marker.upsert(
+          user,
+          %{"notifications" => %{"last_read_id" => "69420"}}
+        )
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> assign(:token, token)
+        |> get("/api/v1/markers", %{timeline: ["notifications"]})
+        |> json_response(200)
+
+      assert response == %{
+               "notifications" => %{
+                 "last_read_id" => "69420",
+                 "updated_at" => NaiveDateTime.to_iso8601(marker.updated_at),
+                 "version" => 0
+               }
+             }
+    end
+
+    test "gets markers with missed scopes", %{conn: conn} do
+      user = insert(:user)
+      token = insert(:oauth_token, user: user, scopes: [])
+
+      Pleroma.Marker.upsert(user, %{"notifications" => %{"last_read_id" => "69420"}})
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> assign(:token, token)
+        |> get("/api/v1/markers", %{timeline: ["notifications"]})
+        |> json_response(403)
+
+      assert response == %{"error" => "Insufficient permissions: read:statuses."}
+    end
+  end
+
+  describe "POST /api/v1/markers" do
+    test "creates a marker with correct scopes", %{conn: conn} do
+      user = insert(:user)
+      token = insert(:oauth_token, user: user, scopes: ["write:statuses"])
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> assign(:token, token)
+        |> post("/api/v1/markers", %{
+          home: %{last_read_id: "777"},
+          notifications: %{"last_read_id" => "69420"}
+        })
+        |> json_response(200)
+
+      assert %{
+               "notifications" => %{
+                 "last_read_id" => "69420",
+                 "updated_at" => _,
+                 "version" => 0
+               }
+             } = response
+    end
+
+    test "updates exist marker", %{conn: conn} do
+      user = insert(:user)
+      token = insert(:oauth_token, user: user, scopes: ["write:statuses"])
+
+      {:ok, %{"notifications" => marker}} =
+        Pleroma.Marker.upsert(
+          user,
+          %{"notifications" => %{"last_read_id" => "69477"}}
+        )
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> assign(:token, token)
+        |> post("/api/v1/markers", %{
+          home: %{last_read_id: "777"},
+          notifications: %{"last_read_id" => "69888"}
+        })
+        |> json_response(200)
+
+      assert response == %{
+               "notifications" => %{
+                 "last_read_id" => "69888",
+                 "updated_at" => NaiveDateTime.to_iso8601(marker.updated_at),
+                 "version" => 0
+               }
+             }
+    end
+
+    test "creates a marker with missed scopes", %{conn: conn} do
+      user = insert(:user)
+      token = insert(:oauth_token, user: user, scopes: [])
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> assign(:token, token)
+        |> post("/api/v1/markers", %{
+          home: %{last_read_id: "777"},
+          notifications: %{"last_read_id" => "69420"}
+        })
+        |> json_response(403)
+
+      assert response == %{"error" => "Insufficient permissions: write:statuses."}
+    end
+  end
+end

test/web/mastodon_api/controllers/search_controller_test.exs

@@ -204,17 +204,17 @@ test "search fetches remote accounts", %{conn: conn} do
       conn =
         conn
         |> assign(:user, user)
-        |> get("/api/v1/search", %{"q" => "shp@social.heldscal.la", "resolve" => "true"})
+        |> get("/api/v1/search", %{"q" => "mike@osada.macgirvin.com", "resolve" => "true"})
 
       assert results = json_response(conn, 200)
       [account] = results["accounts"]
-      assert account["acct"] == "shp@social.heldscal.la"
+      assert account["acct"] == "mike@osada.macgirvin.com"
     end
 
     test "search doesn't fetch remote accounts if resolve is false", %{conn: conn} do
       conn =
         conn
-        |> get("/api/v1/search", %{"q" => "shp@social.heldscal.la", "resolve" => "false"})
+        |> get("/api/v1/search", %{"q" => "mike@osada.macgirvin.com", "resolve" => "false"})
 
       assert results = json_response(conn, 200)
       assert [] == results["accounts"]

test/web/mastodon_api/controllers/status_controller_test.exs

@@ -12,12 +12,16 @@ defmodule Pleroma.Web.MastodonAPI.StatusControllerTest do
   alias Pleroma.Object
   alias Pleroma.Repo
   alias Pleroma.ScheduledActivity
+  alias Pleroma.Tests.ObanHelpers
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.CommonAPI
 
   import Pleroma.Factory
 
+  clear_config([:instance, :federating])
+  clear_config([:instance, :allow_relay])
+
   describe "posting statuses" do
     setup do
       user = insert(:user)
@@ -29,6 +33,34 @@ defmodule Pleroma.Web.MastodonAPI.StatusControllerTest do
       [conn: conn]
     end
 
+    test "posting a status does not increment reblog_count when relaying", %{conn: conn} do
+      Pleroma.Config.put([:instance, :federating], true)
+      Pleroma.Config.get([:instance, :allow_relay], true)
+      user = insert(:user)
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> post("api/v1/statuses", %{
+          "content_type" => "text/plain",
+          "source" => "Pleroma FE",
+          "status" => "Hello world",
+          "visibility" => "public"
+        })
+        |> json_response(200)
+
+      assert response["reblogs_count"] == 0
+      ObanHelpers.perform_all()
+
+      response =
+        conn
+        |> assign(:user, user)
+        |> get("api/v1/statuses/#{response["id"]}", %{})
+        |> json_response(200)
+
+      assert response["reblogs_count"] == 0
+    end
+
     test "posting a status", %{conn: conn} do
       idempotency_key = "Pikachu rocks!"
 

test/web/mastodon_api/controllers/timeline_controller_test.exs

@@ -11,7 +11,6 @@ defmodule Pleroma.Web.MastodonAPI.TimelineControllerTest do
   alias Pleroma.Config
   alias Pleroma.User
   alias Pleroma.Web.CommonAPI
-  alias Pleroma.Web.OStatus
 
   clear_config([:instance, :public])
 
@@ -75,8 +74,7 @@ test "the public timeline", %{conn: conn} do
 
       {:ok, _activity} = CommonAPI.post(following, %{"status" => "test"})
 
-      {:ok, [_activity]} =
+      _activity = insert(:note_activity, local: false)
-        OStatus.fetch_activity_from_url("https://shitposter.club/notice/2827873")
 
       conn = get(conn, "/api/v1/timelines/public", %{"local" => "False"})
 
@@ -271,9 +269,6 @@ test "hashtag timeline", %{conn: conn} do
 
       {:ok, activity} = CommonAPI.post(following, %{"status" => "test #2hu"})
 
-      {:ok, [_activity]} =
-        OStatus.fetch_activity_from_url("https://shitposter.club/notice/2827873")
-
       nconn = get(conn, "/api/v1/timelines/tag/2hu")
 
       assert [%{"id" => id}] = json_response(nconn, :ok)

test/web/mastodon_api/views/marker_view_test.exs

@@ -0,0 +1,27 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.MarkerViewTest do
+  use Pleroma.DataCase
+  alias Pleroma.Web.MastodonAPI.MarkerView
+  import Pleroma.Factory
+
+  test "returns markers" do
+    marker1 = insert(:marker, timeline: "notifications", last_read_id: "17")
+    marker2 = insert(:marker, timeline: "home", last_read_id: "42")
+
+    assert MarkerView.render("markers.json", %{markers: [marker1, marker2]}) == %{
+             "home" => %{
+               last_read_id: "42",
+               updated_at: NaiveDateTime.to_iso8601(marker2.updated_at),
+               version: 0
+             },
+             "notifications" => %{
+               last_read_id: "17",
+               updated_at: NaiveDateTime.to_iso8601(marker1.updated_at),
+               version: 0
+             }
+           }
+  end
+end

test/web/mastodon_api/views/status_view_test.exs

@@ -14,7 +14,6 @@ defmodule Pleroma.Web.MastodonAPI.StatusViewTest do
   alias Pleroma.Web.CommonAPI.Utils
   alias Pleroma.Web.MastodonAPI.AccountView
   alias Pleroma.Web.MastodonAPI.StatusView
-  alias Pleroma.Web.OStatus
   import Pleroma.Factory
   import Tesla.Mock
 
@@ -230,17 +229,15 @@ test "a reply" do
   end
 
   test "contains mentions" do
-    incoming = File.read!("test/fixtures/incoming_reply_mastodon.xml")
+    user = insert(:user)
-    # a user with this ap id might be in the cache.
+    mentioned = insert(:user)
-    recipient = "https://pleroma.soykaf.com/users/lain"
-    user = insert(:user, %{ap_id: recipient})
 
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
+    {:ok, activity} = CommonAPI.post(user, %{"status" => "hi @#{mentioned.nickname}"})
 
     status = StatusView.render("show.json", %{activity: activity})
 
     assert status.mentions ==
-             Enum.map([user], fn u -> AccountView.render("mention.json", %{user: u}) end)
+             Enum.map([mentioned], fn u -> AccountView.render("mention.json", %{user: u}) end)
   end
 
   test "create mentions from the 'to' field" do

test/web/ostatus/activity_representer_test.exs

@@ -1,300 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.ActivityRepresenterTest do
-  use Pleroma.DataCase
-
-  alias Pleroma.Activity
-  alias Pleroma.Object
-  alias Pleroma.User
-  alias Pleroma.Web.ActivityPub.ActivityPub
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.OStatus.ActivityRepresenter
-
-  import Pleroma.Factory
-  import Tesla.Mock
-
-  setup do
-    mock(fn env -> apply(HttpRequestMock, :request, [env]) end)
-    :ok
-  end
-
-  test "an external note activity" do
-    incoming = File.read!("test/fixtures/mastodon-note-cw.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-
-    user = User.get_cached_by_ap_id(activity.data["actor"])
-
-    tuple = ActivityRepresenter.to_simple_form(activity, user)
-
-    res = :xmerl.export_simple_content(tuple, :xmerl_xml) |> IO.iodata_to_binary()
-
-    assert String.contains?(
-             res,
-             ~s{<link type="text/html" href="https://mastodon.social/users/lambadalambda/updates/2314748" rel="alternate"/>}
-           )
-  end
-
-  test "a note activity" do
-    note_activity = insert(:note_activity)
-    object_data = Object.normalize(note_activity).data
-
-    user = User.get_cached_by_ap_id(note_activity.data["actor"])
-
-    expected = """
-    <activity:object-type>http://activitystrea.ms/schema/1.0/note</activity:object-type>
-    <activity:verb>http://activitystrea.ms/schema/1.0/post</activity:verb>
-    <id>#{object_data["id"]}</id>
-    <title>New note by #{user.nickname}</title>
-    <content type="html">#{object_data["content"]}</content>
-    <published>#{object_data["published"]}</published>
-    <updated>#{object_data["published"]}</updated>
-    <ostatus:conversation ref="#{note_activity.data["context"]}">#{note_activity.data["context"]}</ostatus:conversation>
-    <link ref="#{note_activity.data["context"]}" rel="ostatus:conversation" />
-    <summary>#{object_data["summary"]}</summary>
-    <link type="application/atom+xml" href="#{object_data["id"]}" rel="self" />
-    <link type="text/html" href="#{object_data["id"]}" rel="alternate" />
-    <category term="2hu"/>
-    <link rel="mentioned" ostatus:object-type="http://activitystrea.ms/schema/1.0/collection" href="http://activityschema.org/collection/public"/>
-    <link name="2hu" rel="emoji" href="corndog.png" />
-    """
-
-    tuple = ActivityRepresenter.to_simple_form(note_activity, user)
-
-    res = :xmerl.export_simple_content(tuple, :xmerl_xml) |> IO.iodata_to_binary()
-
-    assert clean(res) == clean(expected)
-  end
-
-  test "a reply note" do
-    user = insert(:user)
-    note_object = insert(:note)
-    _note = insert(:note_activity, %{note: note_object})
-    object = insert(:note, %{data: %{"inReplyTo" => note_object.data["id"]}})
-    answer = insert(:note_activity, %{note: object})
-
-    Repo.update!(
-      Object.change(note_object, %{data: Map.put(note_object.data, "external_url", "someurl")})
-    )
-
-    expected = """
-    <activity:object-type>http://activitystrea.ms/schema/1.0/note</activity:object-type>
-    <activity:verb>http://activitystrea.ms/schema/1.0/post</activity:verb>
-    <id>#{object.data["id"]}</id>
-    <title>New note by #{user.nickname}</title>
-    <content type="html">#{object.data["content"]}</content>
-    <published>#{object.data["published"]}</published>
-    <updated>#{object.data["published"]}</updated>
-    <ostatus:conversation ref="#{answer.data["context"]}">#{answer.data["context"]}</ostatus:conversation>
-    <link ref="#{answer.data["context"]}" rel="ostatus:conversation" />
-    <summary>2hu</summary>
-    <link type="application/atom+xml" href="#{object.data["id"]}" rel="self" />
-    <link type="text/html" href="#{object.data["id"]}" rel="alternate" />
-    <category term="2hu"/>
-    <thr:in-reply-to ref="#{note_object.data["id"]}" href="someurl" />
-    <link rel="mentioned" ostatus:object-type="http://activitystrea.ms/schema/1.0/collection" href="http://activityschema.org/collection/public"/>
-    <link name="2hu" rel="emoji" href="corndog.png" />
-    """
-
-    tuple = ActivityRepresenter.to_simple_form(answer, user)
-
-    res = :xmerl.export_simple_content(tuple, :xmerl_xml) |> IO.iodata_to_binary()
-
-    assert clean(res) == clean(expected)
-  end
-
-  test "an announce activity" do
-    note = insert(:note_activity)
-    user = insert(:user)
-    object = Object.normalize(note)
-
-    {:ok, announce, _object} = ActivityPub.announce(user, object)
-
-    announce = Activity.get_by_id(announce.id)
-
-    note_user = User.get_cached_by_ap_id(note.data["actor"])
-    note = Activity.get_by_id(note.id)
-
-    note_xml =
-      ActivityRepresenter.to_simple_form(note, note_user, true)
-      |> :xmerl.export_simple_content(:xmerl_xml)
-      |> to_string
-
-    expected = """
-    <activity:object-type>http://activitystrea.ms/schema/1.0/activity</activity:object-type>
-    <activity:verb>http://activitystrea.ms/schema/1.0/share</activity:verb>
-    <id>#{announce.data["id"]}</id>
-    <title>#{user.nickname} repeated a notice</title>
-    <content type="html">RT #{object.data["content"]}</content>
-    <published>#{announce.data["published"]}</published>
-    <updated>#{announce.data["published"]}</updated>
-    <ostatus:conversation ref="#{announce.data["context"]}">#{announce.data["context"]}</ostatus:conversation>
-    <link ref="#{announce.data["context"]}" rel="ostatus:conversation" />
-    <link rel="self" type="application/atom+xml" href="#{announce.data["id"]}"/>
-    <activity:object>
-      #{note_xml}
-    </activity:object>
-    <link rel="mentioned" ostatus:object-type="http://activitystrea.ms/schema/1.0/person" href="#{
-      note.data["actor"]
-    }"/>
-    <link rel="mentioned" ostatus:object-type="http://activitystrea.ms/schema/1.0/collection" href="http://activityschema.org/collection/public"/>
-    """
-
-    announce_xml =
-      ActivityRepresenter.to_simple_form(announce, user)
-      |> :xmerl.export_simple_content(:xmerl_xml)
-      |> to_string
-
-    assert clean(expected) == clean(announce_xml)
-  end
-
-  test "a like activity" do
-    note = insert(:note)
-    user = insert(:user)
-    {:ok, like, _note} = ActivityPub.like(user, note)
-
-    tuple = ActivityRepresenter.to_simple_form(like, user)
-    refute is_nil(tuple)
-
-    res = :xmerl.export_simple_content(tuple, :xmerl_xml) |> IO.iodata_to_binary()
-
-    expected = """
-    <activity:verb>http://activitystrea.ms/schema/1.0/favorite</activity:verb>
-    <id>#{like.data["id"]}</id>
-    <title>New favorite by #{user.nickname}</title>
-    <content type="html">#{user.nickname} favorited something</content>
-    <published>#{like.data["published"]}</published>
-    <updated>#{like.data["published"]}</updated>
-    <activity:object>
-      <activity:object-type>http://activitystrea.ms/schema/1.0/note</activity:object-type>
-      <id>#{note.data["id"]}</id>
-    </activity:object>
-    <ostatus:conversation ref="#{like.data["context"]}">#{like.data["context"]}</ostatus:conversation>
-    <link ref="#{like.data["context"]}" rel="ostatus:conversation" />
-    <link rel="self" type="application/atom+xml" href="#{like.data["id"]}"/>
-    <thr:in-reply-to ref="#{note.data["id"]}" />
-    <link rel="mentioned" ostatus:object-type="http://activitystrea.ms/schema/1.0/person" href="#{
-      note.data["actor"]
-    }"/>
-    <link rel="mentioned" ostatus:object-type="http://activitystrea.ms/schema/1.0/collection" href="http://activityschema.org/collection/public"/>
-    """
-
-    assert clean(res) == clean(expected)
-  end
-
-  test "a follow activity" do
-    follower = insert(:user)
-    followed = insert(:user)
-
-    {:ok, activity} =
-      ActivityPub.insert(%{
-        "type" => "Follow",
-        "actor" => follower.ap_id,
-        "object" => followed.ap_id,
-        "to" => [followed.ap_id]
-      })
-
-    tuple = ActivityRepresenter.to_simple_form(activity, follower)
-
-    refute is_nil(tuple)
-
-    res = :xmerl.export_simple_content(tuple, :xmerl_xml) |> IO.iodata_to_binary()
-
-    expected = """
-    <activity:object-type>http://activitystrea.ms/schema/1.0/activity</activity:object-type>
-    <activity:verb>http://activitystrea.ms/schema/1.0/follow</activity:verb>
-    <id>#{activity.data["id"]}</id>
-    <title>#{follower.nickname} started following #{activity.data["object"]}</title>
-    <content type="html"> #{follower.nickname} started following #{activity.data["object"]}</content>
-    <published>#{activity.data["published"]}</published>
-    <updated>#{activity.data["published"]}</updated>
-    <activity:object>
-      <activity:object-type>http://activitystrea.ms/schema/1.0/person</activity:object-type>
-      <id>#{activity.data["object"]}</id>
-      <uri>#{activity.data["object"]}</uri>
-    </activity:object>
-    <link rel="self" type="application/atom+xml" href="#{activity.data["id"]}"/>
-    <link rel="mentioned" ostatus:object-type="http://activitystrea.ms/schema/1.0/person" href="#{
-      activity.data["object"]
-    }"/>
-    """
-
-    assert clean(res) == clean(expected)
-  end
-
-  test "an unfollow activity" do
-    follower = insert(:user)
-    followed = insert(:user)
-    {:ok, _activity} = ActivityPub.follow(follower, followed)
-    {:ok, activity} = ActivityPub.unfollow(follower, followed)
-
-    tuple = ActivityRepresenter.to_simple_form(activity, follower)
-
-    refute is_nil(tuple)
-
-    res = :xmerl.export_simple_content(tuple, :xmerl_xml) |> IO.iodata_to_binary()
-
-    expected = """
-    <activity:object-type>http://activitystrea.ms/schema/1.0/activity</activity:object-type>
-    <activity:verb>http://activitystrea.ms/schema/1.0/unfollow</activity:verb>
-    <id>#{activity.data["id"]}</id>
-    <title>#{follower.nickname} stopped following #{followed.ap_id}</title>
-    <content type="html"> #{follower.nickname} stopped following #{followed.ap_id}</content>
-    <published>#{activity.data["published"]}</published>
-    <updated>#{activity.data["published"]}</updated>
-    <activity:object>
-      <activity:object-type>http://activitystrea.ms/schema/1.0/person</activity:object-type>
-      <id>#{followed.ap_id}</id>
-      <uri>#{followed.ap_id}</uri>
-    </activity:object>
-    <link rel="self" type="application/atom+xml" href="#{activity.data["id"]}"/>
-    <link rel="mentioned" ostatus:object-type="http://activitystrea.ms/schema/1.0/person" href="#{
-      followed.ap_id
-    }"/>
-    """
-
-    assert clean(res) == clean(expected)
-  end
-
-  test "a delete" do
-    user = insert(:user)
-
-    activity = %Activity{
-      data: %{
-        "id" => "ap_id",
-        "type" => "Delete",
-        "actor" => user.ap_id,
-        "object" => "some_id",
-        "published" => "2017-06-18T12:00:18+00:00"
-      }
-    }
-
-    tuple = ActivityRepresenter.to_simple_form(activity, nil)
-
-    refute is_nil(tuple)
-
-    res = :xmerl.export_simple_content(tuple, :xmerl_xml) |> IO.iodata_to_binary()
-
-    expected = """
-    <activity:object-type>http://activitystrea.ms/schema/1.0/activity</activity:object-type>
-    <activity:verb>http://activitystrea.ms/schema/1.0/delete</activity:verb>
-    <id>#{activity.data["object"]}</id>
-    <title>An object was deleted</title>
-    <content type="html">An object was deleted</content>
-    <published>#{activity.data["published"]}</published>
-    <updated>#{activity.data["published"]}</updated>
-    """
-
-    assert clean(res) == clean(expected)
-  end
-
-  test "an unknown activity" do
-    tuple = ActivityRepresenter.to_simple_form(%Activity{}, nil)
-    assert is_nil(tuple)
-  end
-
-  defp clean(string) do
-    String.replace(string, ~r/\s/, "")
-  end
-end

test/web/ostatus/feed_representer_test.exs

@@ -1,59 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.FeedRepresenterTest do
-  use Pleroma.DataCase
-  import Pleroma.Factory
-  alias Pleroma.User
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.OStatus.ActivityRepresenter
-  alias Pleroma.Web.OStatus.FeedRepresenter
-  alias Pleroma.Web.OStatus.UserRepresenter
-
-  test "returns a feed of the last 20 items of the user" do
-    note_activity = insert(:note_activity)
-    user = User.get_cached_by_ap_id(note_activity.data["actor"])
-
-    tuple = FeedRepresenter.to_simple_form(user, [note_activity], [user])
-
-    most_recent_update =
-      note_activity.updated_at
-      |> NaiveDateTime.to_iso8601()
-
-    res = :xmerl.export_simple_content(tuple, :xmerl_xml) |> to_string
-
-    user_xml =
-      UserRepresenter.to_simple_form(user)
-      |> :xmerl.export_simple_content(:xmerl_xml)
-
-    entry_xml =
-      ActivityRepresenter.to_simple_form(note_activity, user)
-      |> :xmerl.export_simple_content(:xmerl_xml)
-
-    expected = """
-    <feed xmlns="http://www.w3.org/2005/Atom" xmlns:thr="http://purl.org/syndication/thread/1.0" xmlns:activity="http://activitystrea.ms/spec/1.0/" xmlns:poco="http://portablecontacts.net/spec/1.0" xmlns:ostatus="http://ostatus.org/schema/1.0">
-      <id>#{OStatus.feed_path(user)}</id>
-      <title>#{user.nickname}'s timeline</title>
-      <updated>#{most_recent_update}</updated>
-      <logo>#{User.avatar_url(user)}</logo>
-      <link rel="hub" href="#{OStatus.pubsub_path(user)}" />
-      <link rel="salmon" href="#{OStatus.salmon_path(user)}" />
-      <link rel="self" href="#{OStatus.feed_path(user)}" type="application/atom+xml" />
-      <author>
-        #{user_xml}
-      </author>
-      <link rel="next" href="#{OStatus.feed_path(user)}?max_id=#{note_activity.id}" type="application/atom+xml" />
-      <entry>
-        #{entry_xml}
-      </entry>
-    </feed>
-    """
-
-    assert clean(res) == clean(expected)
-  end
-
-  defp clean(string) do
-    String.replace(string, ~r/\s/, "")
-  end
-end

test/web/ostatus/incoming_documents/delete_handling_test.exs

@@ -1,48 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.DeleteHandlingTest do
-  use Pleroma.DataCase
-
-  import Pleroma.Factory
-  import Tesla.Mock
-
-  alias Pleroma.Activity
-  alias Pleroma.Object
-  alias Pleroma.Web.OStatus
-
-  setup do
-    mock(fn env -> apply(HttpRequestMock, :request, [env]) end)
-    :ok
-  end
-
-  describe "deletions" do
-    test "it removes the mentioned activity" do
-      note = insert(:note_activity)
-      second_note = insert(:note_activity)
-      object = Object.normalize(note)
-      second_object = Object.normalize(second_note)
-      user = insert(:user)
-
-      {:ok, like, _object} = Pleroma.Web.ActivityPub.ActivityPub.like(user, object)
-
-      incoming =
-        File.read!("test/fixtures/delete.xml")
-        |> String.replace(
-          "tag:mastodon.sdf.org,2017-06-10:objectId=310513:objectType=Status",
-          object.data["id"]
-        )
-
-      {:ok, [delete]} = OStatus.handle_incoming(incoming)
-
-      refute Activity.get_by_id(note.id)
-      refute Activity.get_by_id(like.id)
-      assert Object.get_by_ap_id(object.data["id"]).data["type"] == "Tombstone"
-      assert Activity.get_by_id(second_note.id)
-      assert Object.get_by_ap_id(second_object.data["id"])
-
-      assert delete.data["type"] == "Delete"
-    end
-  end
-end

test/web/ostatus/ostatus_controller_test.exs

@@ -5,13 +5,11 @@
 defmodule Pleroma.Web.OStatus.OStatusControllerTest do
   use Pleroma.Web.ConnCase
 
-  import ExUnit.CaptureLog
   import Pleroma.Factory
 
   alias Pleroma.Object
   alias Pleroma.User
   alias Pleroma.Web.CommonAPI
-  alias Pleroma.Web.OStatus.ActivityRepresenter
 
   setup_all do
     Tesla.Mock.mock_global(fn env -> apply(HttpRequestMock, :request, [env]) end)
@@ -22,78 +20,7 @@ defmodule Pleroma.Web.OStatus.OStatusControllerTest do
     Pleroma.Config.put([:instance, :federating], true)
   end
 
-  describe "salmon_incoming" do
-    test "decodes a salmon", %{conn: conn} do
-      user = insert(:user)
-      salmon = File.read!("test/fixtures/salmon.xml")
-
-      assert capture_log(fn ->
-               conn =
-                 conn
-                 |> put_req_header("content-type", "application/atom+xml")
-                 |> post("/users/#{user.nickname}/salmon", salmon)
-
-               assert response(conn, 200)
-             end) =~ "[error]"
-    end
-
-    test "decodes a salmon with a changed magic key", %{conn: conn} do
-      user = insert(:user)
-      salmon = File.read!("test/fixtures/salmon.xml")
-
-      assert capture_log(fn ->
-               conn =
-                 conn
-                 |> put_req_header("content-type", "application/atom+xml")
-                 |> post("/users/#{user.nickname}/salmon", salmon)
-
-               assert response(conn, 200)
-             end) =~ "[error]"
-
-      # Wrong key
-      info = %{
-        magic_key:
-          "RSA.pu0s-halox4tu7wmES1FVSx6u-4wc0YrUFXcqWXZG4-27UmbCOpMQftRCldNRfyA-qLbz-eqiwrong1EwUvjsD4cYbAHNGHwTvDOyx5AKthQUP44ykPv7kjKGh3DWKySJvcs9tlUG87hlo7AvnMo9pwRS_Zz2CacQ-MKaXyDepk=.AQAB"
-      }
-
-      # Set a wrong magic-key for a user so it has to refetch
-      "http://gs.example.org:4040/index.php/user/1"
-      |> User.get_cached_by_ap_id()
-      |> User.update_info(&User.Info.remote_user_creation(&1, info))
-
-      assert capture_log(fn ->
-               conn =
-                 build_conn()
-                 |> put_req_header("content-type", "application/atom+xml")
-                 |> post("/users/#{user.nickname}/salmon", salmon)
-
-               assert response(conn, 200)
-             end) =~ "[error]"
-    end
-  end
-
   describe "GET object/2" do
-    test "gets an object", %{conn: conn} do
-      note_activity = insert(:note_activity)
-      object = Object.normalize(note_activity)
-      user = User.get_cached_by_ap_id(note_activity.data["actor"])
-      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
-      url = "/objects/#{uuid}"
-
-      conn =
-        conn
-        |> put_req_header("accept", "application/xml")
-        |> get(url)
-
-      expected =
-        ActivityRepresenter.to_simple_form(note_activity, user, true)
-        |> ActivityRepresenter.wrap_with_entry()
-        |> :xmerl.export_simple(:xmerl_xml)
-        |> to_string
-
-      assert response(conn, 200) == expected
-    end
-
     test "redirects to /notice/id for html format", %{conn: conn} do
       note_activity = insert(:note_activity)
       object = Object.normalize(note_activity)
@@ -143,16 +70,6 @@ test "404s on nonexisting objects", %{conn: conn} do
   end
 
   describe "GET activity/2" do
-    test "gets an activity in xml format", %{conn: conn} do
-      note_activity = insert(:note_activity)
-      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
-
-      conn
-      |> put_req_header("accept", "application/xml")
-      |> get("/activities/#{uuid}")
-      |> response(200)
-    end
-
     test "redirects to /notice/id for html format", %{conn: conn} do
       note_activity = insert(:note_activity)
       [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))
@@ -180,24 +97,6 @@ test "505s when user not found", %{conn: conn} do
       assert response(conn, 500) == ~S({"error":"Something went wrong"})
     end
 
-    test "404s on deleted objects", %{conn: conn} do
-      note_activity = insert(:note_activity)
-      object = Object.normalize(note_activity)
-      [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, object.data["id"]))
-
-      conn
-      |> put_req_header("accept", "application/xml")
-      |> get("/objects/#{uuid}")
-      |> response(200)
-
-      Object.delete(object)
-
-      conn
-      |> put_req_header("accept", "application/xml")
-      |> get("/objects/#{uuid}")
-      |> response(404)
-    end
-
     test "404s on private activities", %{conn: conn} do
       note_activity = insert(:direct_note_activity)
       [_, uuid] = hd(Regex.scan(~r/.+\/([\w-]+)$/, note_activity.data["id"]))

test/web/ostatus/ostatus_test.exs

@@ -1,645 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatusTest do
-  use Pleroma.DataCase
-  alias Pleroma.Activity
-  alias Pleroma.Instances
-  alias Pleroma.Object
-  alias Pleroma.Repo
-  alias Pleroma.User
-  alias Pleroma.Web.OStatus
-  alias Pleroma.Web.XML
-
-  import ExUnit.CaptureLog
-  import Mock
-  import Pleroma.Factory
-
-  setup_all do
-    Tesla.Mock.mock_global(fn env -> apply(HttpRequestMock, :request, [env]) end)
-    :ok
-  end
-
-  test "don't insert create notes twice" do
-    incoming = File.read!("test/fixtures/incoming_note_activity.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    assert {:ok, [activity]} == OStatus.handle_incoming(incoming)
-  end
-
-  test "handle incoming note - GS, Salmon" do
-    incoming = File.read!("test/fixtures/incoming_note_activity.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    object = Object.normalize(activity)
-
-    user = User.get_cached_by_ap_id(activity.data["actor"])
-    assert user.info.note_count == 1
-    assert activity.data["type"] == "Create"
-    assert object.data["type"] == "Note"
-
-    assert object.data["id"] == "tag:gs.example.org:4040,2017-04-23:noticeId=29:objectType=note"
-
-    assert activity.data["published"] == "2017-04-23T14:51:03+00:00"
-    assert object.data["published"] == "2017-04-23T14:51:03+00:00"
-
-    assert activity.data["context"] ==
-             "tag:gs.example.org:4040,2017-04-23:objectType=thread:nonce=f09e22f58abd5c7b"
-
-    assert "http://pleroma.example.org:4000/users/lain3" in activity.data["to"]
-    assert object.data["emoji"] == %{"marko" => "marko.png", "reimu" => "reimu.png"}
-    assert activity.local == false
-  end
-
-  test "handle incoming notes - GS, subscription" do
-    incoming = File.read!("test/fixtures/ostatus_incoming_post.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    object = Object.normalize(activity)
-
-    assert activity.data["type"] == "Create"
-    assert object.data["type"] == "Note"
-    assert object.data["actor"] == "https://social.heldscal.la/user/23211"
-    assert object.data["content"] == "Will it blend?"
-    user = User.get_cached_by_ap_id(activity.data["actor"])
-    assert User.ap_followers(user) in activity.data["to"]
-    assert "https://www.w3.org/ns/activitystreams#Public" in activity.data["to"]
-  end
-
-  test "handle incoming notes with attachments - GS, subscription" do
-    incoming = File.read!("test/fixtures/incoming_websub_gnusocial_attachments.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    object = Object.normalize(activity)
-
-    assert activity.data["type"] == "Create"
-    assert object.data["type"] == "Note"
-    assert object.data["actor"] == "https://social.heldscal.la/user/23211"
-    assert object.data["attachment"] |> length == 2
-    assert object.data["external_url"] == "https://social.heldscal.la/notice/2020923"
-    assert "https://www.w3.org/ns/activitystreams#Public" in activity.data["to"]
-  end
-
-  test "handle incoming notes with tags" do
-    incoming = File.read!("test/fixtures/ostatus_incoming_post_tag.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    object = Object.normalize(activity)
-
-    assert object.data["tag"] == ["nsfw"]
-    assert "https://www.w3.org/ns/activitystreams#Public" in activity.data["to"]
-  end
-
-  test "handle incoming notes - Mastodon, salmon, reply" do
-    # It uses the context of the replied to object
-    Repo.insert!(%Object{
-      data: %{
-        "id" => "https://pleroma.soykaf.com/objects/c237d966-ac75-4fe3-a87a-d89d71a3a7a4",
-        "context" => "2hu"
-      }
-    })
-
-    incoming = File.read!("test/fixtures/incoming_reply_mastodon.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    object = Object.normalize(activity)
-
-    assert activity.data["type"] == "Create"
-    assert object.data["type"] == "Note"
-    assert object.data["actor"] == "https://mastodon.social/users/lambadalambda"
-    assert activity.data["context"] == "2hu"
-    assert "https://www.w3.org/ns/activitystreams#Public" in activity.data["to"]
-  end
-
-  test "handle incoming notes - Mastodon, with CW" do
-    incoming = File.read!("test/fixtures/mastodon-note-cw.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    object = Object.normalize(activity)
-
-    assert activity.data["type"] == "Create"
-    assert object.data["type"] == "Note"
-    assert object.data["actor"] == "https://mastodon.social/users/lambadalambda"
-    assert object.data["summary"] == "technologic"
-    assert "https://www.w3.org/ns/activitystreams#Public" in activity.data["to"]
-  end
-
-  test "handle incoming unlisted messages, put public into cc" do
-    incoming = File.read!("test/fixtures/mastodon-note-unlisted.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    object = Object.normalize(activity)
-
-    refute "https://www.w3.org/ns/activitystreams#Public" in activity.data["to"]
-    assert "https://www.w3.org/ns/activitystreams#Public" in activity.data["cc"]
-    refute "https://www.w3.org/ns/activitystreams#Public" in object.data["to"]
-    assert "https://www.w3.org/ns/activitystreams#Public" in object.data["cc"]
-  end
-
-  test "handle incoming retweets - Mastodon, with CW" do
-    incoming = File.read!("test/fixtures/cw_retweet.xml")
-    {:ok, [[_activity, retweeted_activity]]} = OStatus.handle_incoming(incoming)
-    retweeted_object = Object.normalize(retweeted_activity)
-
-    assert retweeted_object.data["summary"] == "Hey."
-  end
-
-  test "handle incoming notes - GS, subscription, reply" do
-    incoming = File.read!("test/fixtures/ostatus_incoming_reply.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    object = Object.normalize(activity)
-
-    assert activity.data["type"] == "Create"
-    assert object.data["type"] == "Note"
-    assert object.data["actor"] == "https://social.heldscal.la/user/23211"
-
-    assert object.data["content"] ==
-             "@<a href=\"https://gs.archae.me/user/4687\" class=\"h-card u-url p-nickname mention\" title=\"shpbot\">shpbot</a> why not indeed."
-
-    assert object.data["inReplyTo"] ==
-             "tag:gs.archae.me,2017-04-30:noticeId=778260:objectType=note"
-
-    assert "https://www.w3.org/ns/activitystreams#Public" in activity.data["to"]
-  end
-
-  test "handle incoming retweets - GS, subscription" do
-    incoming = File.read!("test/fixtures/share-gs.xml")
-    {:ok, [[activity, retweeted_activity]]} = OStatus.handle_incoming(incoming)
-
-    assert activity.data["type"] == "Announce"
-    assert activity.data["actor"] == "https://social.heldscal.la/user/23211"
-    assert activity.data["object"] == retweeted_activity.data["object"]
-    assert "https://pleroma.soykaf.com/users/lain" in activity.data["to"]
-    refute activity.local
-
-    retweeted_activity = Activity.get_by_id(retweeted_activity.id)
-    retweeted_object = Object.normalize(retweeted_activity)
-    assert retweeted_activity.data["type"] == "Create"
-    assert retweeted_activity.data["actor"] == "https://pleroma.soykaf.com/users/lain"
-    refute retweeted_activity.local
-    assert retweeted_object.data["announcement_count"] == 1
-    assert String.contains?(retweeted_object.data["content"], "mastodon")
-    refute String.contains?(retweeted_object.data["content"], "Test account")
-  end
-
-  test "handle incoming retweets - GS, subscription - local message" do
-    incoming = File.read!("test/fixtures/share-gs-local.xml")
-    note_activity = insert(:note_activity)
-    object = Object.normalize(note_activity)
-    user = User.get_cached_by_ap_id(note_activity.data["actor"])
-
-    incoming =
-      incoming
-      |> String.replace("LOCAL_ID", object.data["id"])
-      |> String.replace("LOCAL_USER", user.ap_id)
-
-    {:ok, [[activity, retweeted_activity]]} = OStatus.handle_incoming(incoming)
-
-    assert activity.data["type"] == "Announce"
-    assert activity.data["actor"] == "https://social.heldscal.la/user/23211"
-    assert activity.data["object"] == object.data["id"]
-    assert user.ap_id in activity.data["to"]
-    refute activity.local
-
-    retweeted_activity = Activity.get_by_id(retweeted_activity.id)
-    assert note_activity.id == retweeted_activity.id
-    assert retweeted_activity.data["type"] == "Create"
-    assert retweeted_activity.data["actor"] == user.ap_id
-    assert retweeted_activity.local
-    assert Object.normalize(retweeted_activity).data["announcement_count"] == 1
-  end
-
-  test "handle incoming retweets - Mastodon, salmon" do
-    incoming = File.read!("test/fixtures/share.xml")
-    {:ok, [[activity, retweeted_activity]]} = OStatus.handle_incoming(incoming)
-    retweeted_object = Object.normalize(retweeted_activity)
-
-    assert activity.data["type"] == "Announce"
-    assert activity.data["actor"] == "https://mastodon.social/users/lambadalambda"
-    assert activity.data["object"] == retweeted_activity.data["object"]
-
-    assert activity.data["id"] ==
-             "tag:mastodon.social,2017-05-03:objectId=4934452:objectType=Status"
-
-    refute activity.local
-    assert retweeted_activity.data["type"] == "Create"
-    assert retweeted_activity.data["actor"] == "https://pleroma.soykaf.com/users/lain"
-    refute retweeted_activity.local
-    refute String.contains?(retweeted_object.data["content"], "Test account")
-  end
-
-  test "handle incoming favorites - GS, websub" do
-    capture_log(fn ->
-      incoming = File.read!("test/fixtures/favorite.xml")
-      {:ok, [[activity, favorited_activity]]} = OStatus.handle_incoming(incoming)
-
-      assert activity.data["type"] == "Like"
-      assert activity.data["actor"] == "https://social.heldscal.la/user/23211"
-      assert activity.data["object"] == favorited_activity.data["object"]
-
-      assert activity.data["id"] ==
-               "tag:social.heldscal.la,2017-05-05:fave:23211:comment:2061643:2017-05-05T09:12:50+00:00"
-
-      refute activity.local
-      assert favorited_activity.data["type"] == "Create"
-      assert favorited_activity.data["actor"] == "https://shitposter.club/user/1"
-
-      assert favorited_activity.data["object"] ==
-               "tag:shitposter.club,2017-05-05:noticeId=2827873:objectType=comment"
-
-      refute favorited_activity.local
-    end)
-  end
-
-  test "handle conversation references" do
-    incoming = File.read!("test/fixtures/mastodon_conversation.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-
-    assert activity.data["context"] ==
-             "tag:mastodon.social,2017-08-28:objectId=7876885:objectType=Conversation"
-  end
-
-  test "handle incoming favorites with locally available object - GS, websub" do
-    note_activity = insert(:note_activity)
-    object = Object.normalize(note_activity)
-
-    incoming =
-      File.read!("test/fixtures/favorite_with_local_note.xml")
-      |> String.replace("localid", object.data["id"])
-
-    {:ok, [[activity, favorited_activity]]} = OStatus.handle_incoming(incoming)
-
-    assert activity.data["type"] == "Like"
-    assert activity.data["actor"] == "https://social.heldscal.la/user/23211"
-    assert activity.data["object"] == object.data["id"]
-    refute activity.local
-    assert note_activity.id == favorited_activity.id
-    assert favorited_activity.local
-  end
-
-  test_with_mock "handle incoming replies, fetching replied-to activities if we don't have them",
-                 OStatus,
-                 [:passthrough],
-                 [] do
-    incoming = File.read!("test/fixtures/incoming_note_activity_answer.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    object = Object.normalize(activity, false)
-
-    assert activity.data["type"] == "Create"
-    assert object.data["type"] == "Note"
-
-    assert object.data["inReplyTo"] ==
-             "http://pleroma.example.org:4000/objects/55bce8fc-b423-46b1-af71-3759ab4670bc"
-
-    assert "http://pleroma.example.org:4000/users/lain5" in activity.data["to"]
-
-    assert object.data["id"] == "tag:gs.example.org:4040,2017-04-25:noticeId=55:objectType=note"
-
-    assert "https://www.w3.org/ns/activitystreams#Public" in activity.data["to"]
-
-    assert called(OStatus.fetch_activity_from_url(object.data["inReplyTo"], :_))
-  end
-
-  test_with_mock "handle incoming replies, not fetching replied-to activities beyond max_replies_depth",
-                 OStatus,
-                 [:passthrough],
-                 [] do
-    incoming = File.read!("test/fixtures/incoming_note_activity_answer.xml")
-
-    with_mock Pleroma.Web.Federator,
-      allowed_incoming_reply_depth?: fn _ -> false end do
-      {:ok, [activity]} = OStatus.handle_incoming(incoming)
-      object = Object.normalize(activity, false)
-
-      refute called(OStatus.fetch_activity_from_url(object.data["inReplyTo"], :_))
-    end
-  end
-
-  test "handle incoming follows" do
-    incoming = File.read!("test/fixtures/follow.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-    assert activity.data["type"] == "Follow"
-
-    assert activity.data["id"] ==
-             "tag:social.heldscal.la,2017-05-07:subscription:23211:person:44803:2017-05-07T09:54:48+00:00"
-
-    assert activity.data["actor"] == "https://social.heldscal.la/user/23211"
-    assert activity.data["object"] == "https://pawoo.net/users/pekorino"
-    refute activity.local
-
-    follower = User.get_cached_by_ap_id(activity.data["actor"])
-    followed = User.get_cached_by_ap_id(activity.data["object"])
-
-    assert User.following?(follower, followed)
-  end
-
-  test "refuse following over OStatus if the followed's account is locked" do
-    incoming = File.read!("test/fixtures/follow.xml")
-    _user = insert(:user, info: %{locked: true}, ap_id: "https://pawoo.net/users/pekorino")
-
-    {:ok, [{:error, "It's not possible to follow locked accounts over OStatus"}]} =
-      OStatus.handle_incoming(incoming)
-  end
-
-  test "handle incoming unfollows with existing follow" do
-    incoming_follow = File.read!("test/fixtures/follow.xml")
-    {:ok, [_activity]} = OStatus.handle_incoming(incoming_follow)
-
-    incoming = File.read!("test/fixtures/unfollow.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-
-    assert activity.data["type"] == "Undo"
-
-    assert activity.data["id"] ==
-             "undo:tag:social.heldscal.la,2017-05-07:subscription:23211:person:44803:2017-05-07T09:54:48+00:00"
-
-    assert activity.data["actor"] == "https://social.heldscal.la/user/23211"
-    embedded_object = activity.data["object"]
-    assert is_map(embedded_object)
-    assert embedded_object["type"] == "Follow"
-    assert embedded_object["object"] == "https://pawoo.net/users/pekorino"
-    refute activity.local
-
-    follower = User.get_cached_by_ap_id(activity.data["actor"])
-    followed = User.get_cached_by_ap_id(embedded_object["object"])
-
-    refute User.following?(follower, followed)
-  end
-
-  test "it clears `unreachable` federation status of the sender" do
-    incoming_reaction_xml = File.read!("test/fixtures/share-gs.xml")
-    doc = XML.parse_document(incoming_reaction_xml)
-    actor_uri = XML.string_from_xpath("//author/uri[1]", doc)
-    reacted_to_author_uri = XML.string_from_xpath("//author/uri[2]", doc)
-
-    Instances.set_consistently_unreachable(actor_uri)
-    Instances.set_consistently_unreachable(reacted_to_author_uri)
-    refute Instances.reachable?(actor_uri)
-    refute Instances.reachable?(reacted_to_author_uri)
-
-    {:ok, _} = OStatus.handle_incoming(incoming_reaction_xml)
-    assert Instances.reachable?(actor_uri)
-    refute Instances.reachable?(reacted_to_author_uri)
-  end
-
-  describe "new remote user creation" do
-    test "returns local users" do
-      local_user = insert(:user)
-      {:ok, user} = OStatus.find_or_make_user(local_user.ap_id)
-
-      assert user == local_user
-    end
-
-    test "tries to use the information in poco fields" do
-      uri = "https://social.heldscal.la/user/23211"
-
-      {:ok, user} = OStatus.find_or_make_user(uri)
-
-      user = User.get_cached_by_id(user.id)
-      assert user.name == "Constance Variable"
-      assert user.nickname == "lambadalambda@social.heldscal.la"
-      assert user.local == false
-      assert user.info.uri == uri
-      assert user.ap_id == uri
-      assert user.bio == "Call me Deacon Blues."
-      assert user.avatar["type"] == "Image"
-
-      {:ok, user_again} = OStatus.find_or_make_user(uri)
-
-      assert user == user_again
-    end
-
-    test "find_or_make_user sets all the nessary input fields" do
-      uri = "https://social.heldscal.la/user/23211"
-      {:ok, user} = OStatus.find_or_make_user(uri)
-
-      assert user.info ==
-               %User.Info{
-                 id: user.info.id,
-                 ap_enabled: false,
-                 background: %{},
-                 banner: %{},
-                 blocks: [],
-                 deactivated: false,
-                 default_scope: "public",
-                 domain_blocks: [],
-                 follower_count: 0,
-                 is_admin: false,
-                 is_moderator: false,
-                 keys: nil,
-                 locked: false,
-                 no_rich_text: false,
-                 note_count: 0,
-                 settings: nil,
-                 source_data: %{},
-                 hub: "https://social.heldscal.la/main/push/hub",
-                 magic_key:
-                   "RSA.uzg6r1peZU0vXGADWxGJ0PE34WvmhjUmydbX5YYdOiXfODVLwCMi1umGoqUDm-mRu4vNEdFBVJU1CpFA7dKzWgIsqsa501i2XqElmEveXRLvNRWFB6nG03Q5OUY2as8eE54BJm0p20GkMfIJGwP6TSFb-ICp3QjzbatuSPJ6xCE=.AQAB",
-                 salmon: "https://social.heldscal.la/main/salmon/user/23211",
-                 topic: "https://social.heldscal.la/api/statuses/user_timeline/23211.atom",
-                 uri: "https://social.heldscal.la/user/23211"
-               }
-    end
-
-    test "find_make_or_update_actor takes an author element and returns an updated user" do
-      uri = "https://social.heldscal.la/user/23211"
-
-      {:ok, user} = OStatus.find_or_make_user(uri)
-      old_name = user.name
-      old_bio = user.bio
-      change = Ecto.Changeset.change(user, %{avatar: nil, bio: nil, name: nil})
-
-      {:ok, user} = Repo.update(change)
-      refute user.avatar
-
-      doc = XML.parse_document(File.read!("test/fixtures/23211.atom"))
-      [author] = :xmerl_xpath.string('//author[1]', doc)
-      {:ok, user} = OStatus.find_make_or_update_actor(author)
-      assert user.avatar["type"] == "Image"
-      assert user.name == old_name
-      assert user.bio == old_bio
-
-      {:ok, user_again} = OStatus.find_make_or_update_actor(author)
-      assert user_again == user
-    end
-
-    test "find_or_make_user disallows protocol downgrade" do
-      user = insert(:user, %{local: true})
-      {:ok, user} = OStatus.find_or_make_user(user.ap_id)
-
-      assert User.ap_enabled?(user)
-
-      user =
-        insert(:user, %{
-          ap_id: "https://social.heldscal.la/user/23211",
-          info: %{ap_enabled: true},
-          local: false
-        })
-
-      assert User.ap_enabled?(user)
-
-      {:ok, user} = OStatus.find_or_make_user(user.ap_id)
-      assert User.ap_enabled?(user)
-    end
-
-    test "find_make_or_update_actor disallows protocol downgrade" do
-      user = insert(:user, %{local: true})
-      {:ok, user} = OStatus.find_or_make_user(user.ap_id)
-
-      assert User.ap_enabled?(user)
-
-      user =
-        insert(:user, %{
-          ap_id: "https://social.heldscal.la/user/23211",
-          info: %{ap_enabled: true},
-          local: false
-        })
-
-      assert User.ap_enabled?(user)
-
-      {:ok, user} = OStatus.find_or_make_user(user.ap_id)
-      assert User.ap_enabled?(user)
-
-      doc = XML.parse_document(File.read!("test/fixtures/23211.atom"))
-      [author] = :xmerl_xpath.string('//author[1]', doc)
-      {:error, :invalid_protocol} = OStatus.find_make_or_update_actor(author)
-    end
-  end
-
-  describe "gathering user info from a user id" do
-    test "it returns user info in a hash" do
-      user = "shp@social.heldscal.la"
-
-      # TODO: make test local
-      {:ok, data} = OStatus.gather_user_info(user)
-
-      expected = %{
-        "hub" => "https://social.heldscal.la/main/push/hub",
-        "magic_key" =>
-          "RSA.wQ3i9UA0qmAxZ0WTIp4a-waZn_17Ez1pEEmqmqoooRsG1_BvpmOvLN0G2tEcWWxl2KOtdQMCiPptmQObeZeuj48mdsDZ4ArQinexY2hCCTcbV8Xpswpkb8K05RcKipdg07pnI7tAgQ0VWSZDImncL6YUGlG5YN8b5TjGOwk2VG8=.AQAB",
-        "name" => "shp",
-        "nickname" => "shp",
-        "salmon" => "https://social.heldscal.la/main/salmon/user/29191",
-        "subject" => "acct:shp@social.heldscal.la",
-        "topic" => "https://social.heldscal.la/api/statuses/user_timeline/29191.atom",
-        "uri" => "https://social.heldscal.la/user/29191",
-        "host" => "social.heldscal.la",
-        "fqn" => user,
-        "bio" => "cofe",
-        "avatar" => %{
-          "type" => "Image",
-          "url" => [
-            %{
-              "href" => "https://social.heldscal.la/avatar/29191-original-20170421154949.jpeg",
-              "mediaType" => "image/jpeg",
-              "type" => "Link"
-            }
-          ]
-        },
-        "subscribe_address" => "https://social.heldscal.la/main/ostatussub?profile={uri}",
-        "ap_id" => nil
-      }
-
-      assert data == expected
-    end
-
-    test "it works with the uri" do
-      user = "https://social.heldscal.la/user/29191"
-
-      # TODO: make test local
-      {:ok, data} = OStatus.gather_user_info(user)
-
-      expected = %{
-        "hub" => "https://social.heldscal.la/main/push/hub",
-        "magic_key" =>
-          "RSA.wQ3i9UA0qmAxZ0WTIp4a-waZn_17Ez1pEEmqmqoooRsG1_BvpmOvLN0G2tEcWWxl2KOtdQMCiPptmQObeZeuj48mdsDZ4ArQinexY2hCCTcbV8Xpswpkb8K05RcKipdg07pnI7tAgQ0VWSZDImncL6YUGlG5YN8b5TjGOwk2VG8=.AQAB",
-        "name" => "shp",
-        "nickname" => "shp",
-        "salmon" => "https://social.heldscal.la/main/salmon/user/29191",
-        "subject" => "https://social.heldscal.la/user/29191",
-        "topic" => "https://social.heldscal.la/api/statuses/user_timeline/29191.atom",
-        "uri" => "https://social.heldscal.la/user/29191",
-        "host" => "social.heldscal.la",
-        "fqn" => user,
-        "bio" => "cofe",
-        "avatar" => %{
-          "type" => "Image",
-          "url" => [
-            %{
-              "href" => "https://social.heldscal.la/avatar/29191-original-20170421154949.jpeg",
-              "mediaType" => "image/jpeg",
-              "type" => "Link"
-            }
-          ]
-        },
-        "subscribe_address" => "https://social.heldscal.la/main/ostatussub?profile={uri}",
-        "ap_id" => nil
-      }
-
-      assert data == expected
-    end
-  end
-
-  describe "fetching a status by it's HTML url" do
-    test "it builds a missing status from an html url" do
-      capture_log(fn ->
-        url = "https://shitposter.club/notice/2827873"
-        {:ok, [activity]} = OStatus.fetch_activity_from_url(url)
-
-        assert activity.data["actor"] == "https://shitposter.club/user/1"
-
-        assert activity.data["object"] ==
-                 "tag:shitposter.club,2017-05-05:noticeId=2827873:objectType=comment"
-      end)
-    end
-
-    test "it works for atom notes, too" do
-      url = "https://social.sakamoto.gq/objects/0ccc1a2c-66b0-4305-b23a-7f7f2b040056"
-      {:ok, [activity]} = OStatus.fetch_activity_from_url(url)
-      assert activity.data["actor"] == "https://social.sakamoto.gq/users/eal"
-      assert activity.data["object"] == url
-    end
-  end
-
-  test "it doesn't add nil in the to field" do
-    incoming = File.read!("test/fixtures/nil_mention_entry.xml")
-    {:ok, [activity]} = OStatus.handle_incoming(incoming)
-
-    assert activity.data["to"] == [
-             "http://localhost:4001/users/atarifrosch@social.stopwatchingus-heidelberg.de/followers",
-             "https://www.w3.org/ns/activitystreams#Public"
-           ]
-  end
-
-  describe "is_representable?" do
-    test "Note objects are representable" do
-      note_activity = insert(:note_activity)
-
-      assert OStatus.is_representable?(note_activity)
-    end
-
-    test "Article objects are not representable" do
-      note_activity = insert(:note_activity)
-      note_object = Object.normalize(note_activity)
-
-      note_data =
-        note_object.data
-        |> Map.put("type", "Article")
-
-      Cachex.clear(:object_cache)
-
-      cs = Object.change(note_object, %{data: note_data})
-      {:ok, _article_object} = Repo.update(cs)
-
-      # the underlying object is now an Article instead of a note, so this should fail
-      refute OStatus.is_representable?(note_activity)
-    end
-  end
-
-  describe "make_user/2" do
-    test "creates new user" do
-      {:ok, user} = OStatus.make_user("https://social.heldscal.la/user/23211")
-
-      created_user =
-        User
-        |> Repo.get_by(ap_id: "https://social.heldscal.la/user/23211")
-        |> Map.put(:last_digest_emailed_at, nil)
-
-      assert user.info
-      assert user == created_user
-    end
-  end
-end

test/web/ostatus/user_representer_test.exs

@@ -1,38 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.OStatus.UserRepresenterTest do
-  use Pleroma.DataCase
-  alias Pleroma.Web.OStatus.UserRepresenter
-
-  import Pleroma.Factory
-  alias Pleroma.User
-
-  test "returns a user with id, uri, name and link" do
-    user = insert(:user, %{nickname: "レイン"})
-    tuple = UserRepresenter.to_simple_form(user)
-
-    res = :xmerl.export_simple_content(tuple, :xmerl_xml) |> to_string
-
-    expected = """
-    <id>#{user.ap_id}</id>
-    <activity:object>http://activitystrea.ms/schema/1.0/person</activity:object>
-    <uri>#{user.ap_id}</uri>
-    <poco:preferredUsername>#{user.nickname}</poco:preferredUsername>
-    <poco:displayName>#{user.name}</poco:displayName>
-    <poco:note>#{user.bio}</poco:note>
-    <summary>#{user.bio}</summary>
-    <name>#{user.nickname}</name>
-    <link rel="avatar" href="#{User.avatar_url(user)}" />
-    <link rel="header" href="#{User.banner_url(user)}" />
-    <ap_enabled>true</ap_enabled>
-    """
-
-    assert clean(res) == clean(expected)
-  end
-
-  defp clean(string) do
-    String.replace(string, ~r/\s/, "")
-  end
-end

test/web/pleroma_api/controllers/pleroma_api_controller_test.exs

@@ -95,6 +95,33 @@ test "PATCH /api/v1/pleroma/conversations/:id", %{conn: conn} do
     assert other_user in participation.recipients
   end
 
+  test "POST /api/v1/pleroma/conversations/read", %{conn: conn} do
+    user = insert(:user)
+    other_user = insert(:user)
+
+    {:ok, _activity} =
+      CommonAPI.post(user, %{"status" => "Hi @#{other_user.nickname}", "visibility" => "direct"})
+
+    {:ok, _activity} =
+      CommonAPI.post(user, %{"status" => "Hi @#{other_user.nickname}", "visibility" => "direct"})
+
+    [participation2, participation1] = Participation.for_user(other_user)
+    assert Participation.get(participation2.id).read == false
+    assert Participation.get(participation1.id).read == false
+    assert User.get_cached_by_id(other_user.id).info.unread_conversation_count == 2
+
+    [%{"unread" => false}, %{"unread" => false}] =
+      conn
+      |> assign(:user, other_user)
+      |> post("/api/v1/pleroma/conversations/read", %{})
+      |> json_response(200)
+
+    [participation2, participation1] = Participation.for_user(other_user)
+    assert Participation.get(participation2.id).read == true
+    assert Participation.get(participation1.id).read == true
+    assert User.get_cached_by_id(other_user.id).info.unread_conversation_count == 0
+  end
+
   describe "POST /api/v1/pleroma/notifications/read" do
     test "it marks a single notification as read", %{conn: conn} do
       user1 = insert(:user)

test/web/push/impl_test.exs

@@ -84,7 +84,7 @@ test "fail message sending" do
            ) == :error
   end
 
-  test "delete subsciption if restult send message between 400..500" do
+  test "delete subscription if result send message between 400..500" do
     subscription = insert(:push_subscription)
 
     assert Impl.push_message(
@@ -97,7 +97,7 @@ test "delete subsciption if restult send message between 400..500" do
     refute Pleroma.Repo.get(Subscription, subscription.id)
   end
 
-  test "renders body for create activity" do
+  test "renders title and body for create activity" do
     user = insert(:user, nickname: "Bob")
 
     {:ok, activity} =
@@ -116,18 +116,24 @@ test "renders body for create activity" do
              object
            ) ==
              "@Bob: Lorem ipsum dolor sit amet, consectetur  adipiscing elit. Fusce sagittis fini..."
+
+    assert Impl.format_title(%{activity: activity}) ==
+             "New Mention"
   end
 
-  test "renders body for follow activity" do
+  test "renders title and body for follow activity" do
     user = insert(:user, nickname: "Bob")
     other_user = insert(:user)
     {:ok, _, _, activity} = CommonAPI.follow(user, other_user)
     object = Object.normalize(activity)
 
     assert Impl.format_body(%{activity: activity}, user, object) == "@Bob has followed you"
+
+    assert Impl.format_title(%{activity: activity}) ==
+             "New Follower"
   end
 
-  test "renders body for announce activity" do
+  test "renders title and body for announce activity" do
     user = insert(:user)
 
     {:ok, activity} =
@@ -141,9 +147,12 @@ test "renders body for announce activity" do
 
     assert Impl.format_body(%{activity: announce_activity}, user, object) ==
              "@#{user.nickname} repeated: Lorem ipsum dolor sit amet, consectetur  adipiscing elit. Fusce sagittis fini..."
+
+    assert Impl.format_title(%{activity: announce_activity}) ==
+             "New Repeat"
   end
 
-  test "renders body for like activity" do
+  test "renders title and body for like activity" do
     user = insert(:user, nickname: "Bob")
 
     {:ok, activity} =
@@ -156,5 +165,21 @@ test "renders body for like activity" do
     object = Object.normalize(activity)
 
     assert Impl.format_body(%{activity: activity}, user, object) == "@Bob has favorited your post"
+
+    assert Impl.format_title(%{activity: activity}) ==
+             "New Favorite"
+  end
+
+  test "renders title for create activity with direct visibility" do
+    user = insert(:user, nickname: "Bob")
+
+    {:ok, activity} =
+      CommonAPI.post(user, %{
+        "visibility" => "direct",
+        "status" => "This is just between you and me, pal"
+      })
+
+    assert Impl.format_title(%{activity: activity}) ==
+             "New Direct Message"
   end
 end

test/web/salmon/salmon_test.exs

@@ -1,101 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.Salmon.SalmonTest do
-  use Pleroma.DataCase
-  alias Pleroma.Activity
-  alias Pleroma.Keys
-  alias Pleroma.Repo
-  alias Pleroma.User
-  alias Pleroma.Web.Federator.Publisher
-  alias Pleroma.Web.Salmon
-  import Mock
-  import Pleroma.Factory
-
-  @magickey "RSA.pu0s-halox4tu7wmES1FVSx6u-4wc0YrUFXcqWXZG4-27UmbCOpMQftRCldNRfyA-qLbz-eqiwQhh-1EwUvjsD4cYbAHNGHwTvDOyx5AKthQUP44ykPv7kjKGh3DWKySJvcs9tlUG87hlo7AvnMo9pwRS_Zz2CacQ-MKaXyDepk=.AQAB"
-
-  @wrong_magickey "RSA.pu0s-halox4tu7wmES1FVSx6u-4wc0YrUFXcqWXZG4-27UmbCOpMQftRCldNRfyA-qLbz-eqiwQhh-1EwUvjsD4cYbAHNGHwTvDOyx5AKthQUP44ykPv7kjKGh3DWKySJvcs9tlUG87hlo7AvnMo9pwRS_Zz2CacQ-MKaXyDepk=.AQAA"
-
-  @magickey_friendica "RSA.AMwa8FUs2fWEjX0xN7yRQgegQffhBpuKNC6fa5VNSVorFjGZhRrlPMn7TQOeihlc9lBz2OsHlIedbYn2uJ7yCs0.AQAB"
-
-  setup_all do
-    Tesla.Mock.mock_global(fn env -> apply(HttpRequestMock, :request, [env]) end)
-    :ok
-  end
-
-  test "decodes a salmon" do
-    {:ok, salmon} = File.read("test/fixtures/salmon.xml")
-    {:ok, doc} = Salmon.decode_and_validate(@magickey, salmon)
-    assert Regex.match?(~r/xml/, doc)
-  end
-
-  test "errors on wrong magic key" do
-    {:ok, salmon} = File.read("test/fixtures/salmon.xml")
-    assert Salmon.decode_and_validate(@wrong_magickey, salmon) == :error
-  end
-
-  test "it encodes a magic key from a public key" do
-    key = Salmon.decode_key(@magickey)
-    magic_key = Salmon.encode_key(key)
-
-    assert @magickey == magic_key
-  end
-
-  test "it decodes a friendica public key" do
-    _key = Salmon.decode_key(@magickey_friendica)
-  end
-
-  test "encodes an xml payload with a private key" do
-    doc = File.read!("test/fixtures/incoming_note_activity.xml")
-    pem = File.read!("test/fixtures/private_key.pem")
-    {:ok, private, public} = Keys.keys_from_pem(pem)
-
-    # Let's try a roundtrip.
-    {:ok, salmon} = Salmon.encode(private, doc)
-    {:ok, decoded_doc} = Salmon.decode_and_validate(Salmon.encode_key(public), salmon)
-
-    assert doc == decoded_doc
-  end
-
-  test "it gets a magic key" do
-    salmon = File.read!("test/fixtures/salmon2.xml")
-    {:ok, key} = Salmon.fetch_magic_key(salmon)
-
-    assert key ==
-             "RSA.uzg6r1peZU0vXGADWxGJ0PE34WvmhjUmydbX5YYdOiXfODVLwCMi1umGoqUDm-mRu4vNEdFBVJU1CpFA7dKzWgIsqsa501i2XqElmEveXRLvNRWFB6nG03Q5OUY2as8eE54BJm0p20GkMfIJGwP6TSFb-ICp3QjzbatuSPJ6xCE=.AQAB"
-  end
-
-  test_with_mock "it pushes an activity to remote accounts it's addressed to",
-                 Publisher,
-                 [:passthrough],
-                 [] do
-    user_data = %{
-      info: %{
-        salmon: "http://test-example.org/salmon"
-      },
-      local: false
-    }
-
-    mentioned_user = insert(:user, user_data)
-    note = insert(:note)
-
-    activity_data = %{
-      "id" => Pleroma.Web.ActivityPub.Utils.generate_activity_id(),
-      "type" => "Create",
-      "actor" => note.data["actor"],
-      "to" => note.data["to"] ++ [mentioned_user.ap_id],
-      "object" => note.data,
-      "published_at" => DateTime.utc_now() |> DateTime.to_iso8601(),
-      "context" => note.data["context"]
-    }
-
-    {:ok, activity} = Repo.insert(%Activity{data: activity_data, recipients: activity_data["to"]})
-    user = User.get_cached_by_ap_id(activity.data["actor"])
-    {:ok, user} = User.ensure_keys_present(user)
-
-    Salmon.publish(user, activity)
-
-    assert called(Publisher.enqueue_one(Salmon, %{recipient_id: mentioned_user.id}))
-  end
-end

test/web/web_finger/web_finger_test.exs

@@ -45,19 +45,6 @@ test "returns error when fails parse xml or json" do
       assert {:error, %Jason.DecodeError{}} = WebFinger.finger(user)
     end
 
-    test "returns the info for an OStatus user" do
-      user = "shp@social.heldscal.la"
-
-      {:ok, data} = WebFinger.finger(user)
-
-      assert data["magic_key"] ==
-               "RSA.wQ3i9UA0qmAxZ0WTIp4a-waZn_17Ez1pEEmqmqoooRsG1_BvpmOvLN0G2tEcWWxl2KOtdQMCiPptmQObeZeuj48mdsDZ4ArQinexY2hCCTcbV8Xpswpkb8K05RcKipdg07pnI7tAgQ0VWSZDImncL6YUGlG5YN8b5TjGOwk2VG8=.AQAB"
-
-      assert data["topic"] == "https://social.heldscal.la/api/statuses/user_timeline/29191.atom"
-      assert data["subject"] == "acct:shp@social.heldscal.la"
-      assert data["salmon"] == "https://social.heldscal.la/main/salmon/user/29191"
-    end
-
     test "returns the ActivityPub actor URI for an ActivityPub user" do
       user = "framasoft@framatube.org"
 
@@ -72,20 +59,6 @@ test "returns the ActivityPub actor URI for an ActivityPub user with the ld+json
       assert data["ap_id"] == "https://gerzilla.de/channel/kaniini"
     end
 
-    test "returns the correctly for json ostatus users" do
-      user = "winterdienst@gnusocial.de"
-
-      {:ok, data} = WebFinger.finger(user)
-
-      assert data["magic_key"] ==
-               "RSA.qfYaxztz7ZELrE4v5WpJrPM99SKI3iv9Y3Tw6nfLGk-4CRljNYqV8IYX2FXjeucC_DKhPNnlF6fXyASpcSmA_qupX9WC66eVhFhZ5OuyBOeLvJ1C4x7Hi7Di8MNBxY3VdQuQR0tTaS_YAZCwASKp7H6XEid3EJpGt0EQZoNzRd8=.AQAB"
-
-      assert data["topic"] == "https://gnusocial.de/api/statuses/user_timeline/249296.atom"
-      assert data["subject"] == "acct:winterdienst@gnusocial.de"
-      assert data["salmon"] == "https://gnusocial.de/main/salmon/user/249296"
-      assert data["subscribe_address"] == "https://gnusocial.de/main/ostatussub?profile={uri}"
-    end
-
     test "it work for AP-only user" do
       user = "kpherox@mstdn.jp"
 

test/web/websub/websub_controller_test.exs

@@ -1,86 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.Websub.WebsubControllerTest do
-  use Pleroma.Web.ConnCase
-  import Pleroma.Factory
-  alias Pleroma.Repo
-  alias Pleroma.Web.Websub
-  alias Pleroma.Web.Websub.WebsubClientSubscription
-
-  clear_config_all([:instance, :federating]) do
-    Pleroma.Config.put([:instance, :federating], true)
-  end
-
-  test "websub subscription request", %{conn: conn} do
-    user = insert(:user)
-
-    path = Pleroma.Web.OStatus.pubsub_path(user)
-
-    data = %{
-      "hub.callback": "http://example.org/sub",
-      "hub.mode": "subscribe",
-      "hub.topic": Pleroma.Web.OStatus.feed_path(user),
-      "hub.secret": "a random secret",
-      "hub.lease_seconds": "100"
-    }
-
-    conn =
-      conn
-      |> post(path, data)
-
-    assert response(conn, 202) == "Accepted"
-  end
-
-  test "websub subscription confirmation", %{conn: conn} do
-    websub = insert(:websub_client_subscription)
-
-    params = %{
-      "hub.mode" => "subscribe",
-      "hub.topic" => websub.topic,
-      "hub.challenge" => "some challenge",
-      "hub.lease_seconds" => "100"
-    }
-
-    conn =
-      conn
-      |> get("/push/subscriptions/#{websub.id}", params)
-
-    websub = Repo.get(WebsubClientSubscription, websub.id)
-
-    assert response(conn, 200) == "some challenge"
-    assert websub.state == "accepted"
-    assert_in_delta NaiveDateTime.diff(websub.valid_until, NaiveDateTime.utc_now()), 100, 5
-  end
-
-  describe "websub_incoming" do
-    test "accepts incoming feed updates", %{conn: conn} do
-      websub = insert(:websub_client_subscription)
-      doc = "some stuff"
-      signature = Websub.sign(websub.secret, doc)
-
-      conn =
-        conn
-        |> put_req_header("x-hub-signature", "sha1=" <> signature)
-        |> put_req_header("content-type", "application/atom+xml")
-        |> post("/push/subscriptions/#{websub.id}", doc)
-
-      assert response(conn, 200) == "OK"
-    end
-
-    test "rejects incoming feed updates with the wrong signature", %{conn: conn} do
-      websub = insert(:websub_client_subscription)
-      doc = "some stuff"
-      signature = Websub.sign("wrong secret", doc)
-
-      conn =
-        conn
-        |> put_req_header("x-hub-signature", "sha1=" <> signature)
-        |> put_req_header("content-type", "application/atom+xml")
-        |> post("/push/subscriptions/#{websub.id}", doc)
-
-      assert response(conn, 500) == "Error"
-    end
-  end
-end

test/web/websub/websub_test.exs

@@ -1,236 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.Web.WebsubTest do
-  use Pleroma.DataCase
-  use Oban.Testing, repo: Pleroma.Repo
-
-  alias Pleroma.Tests.ObanHelpers
-  alias Pleroma.Web.Router.Helpers
-  alias Pleroma.Web.Websub
-  alias Pleroma.Web.Websub.WebsubClientSubscription
-  alias Pleroma.Web.Websub.WebsubServerSubscription
-  alias Pleroma.Workers.SubscriberWorker
-
-  import Pleroma.Factory
-  import Tesla.Mock
-
-  setup do
-    mock(fn env -> apply(HttpRequestMock, :request, [env]) end)
-    :ok
-  end
-
-  test "a verification of a request that is accepted" do
-    sub = insert(:websub_subscription)
-    topic = sub.topic
-
-    getter = fn _path, _headers, options ->
-      %{
-        "hub.challenge": challenge,
-        "hub.lease_seconds": seconds,
-        "hub.topic": ^topic,
-        "hub.mode": "subscribe"
-      } = Keyword.get(options, :params)
-
-      assert String.to_integer(seconds) > 0
-
-      {:ok,
-       %Tesla.Env{
-         status: 200,
-         body: challenge
-       }}
-    end
-
-    {:ok, sub} = Websub.verify(sub, getter)
-    assert sub.state == "active"
-  end
-
-  test "a verification of a request that doesn't return 200" do
-    sub = insert(:websub_subscription)
-
-    getter = fn _path, _headers, _options ->
-      {:ok,
-       %Tesla.Env{
-         status: 500,
-         body: ""
-       }}
-    end
-
-    {:error, sub} = Websub.verify(sub, getter)
-    # Keep the current state.
-    assert sub.state == "requested"
-  end
-
-  test "an incoming subscription request" do
-    user = insert(:user)
-
-    data = %{
-      "hub.callback" => "http://example.org/sub",
-      "hub.mode" => "subscribe",
-      "hub.topic" => Pleroma.Web.OStatus.feed_path(user),
-      "hub.secret" => "a random secret",
-      "hub.lease_seconds" => "100"
-    }
-
-    {:ok, subscription} = Websub.incoming_subscription_request(user, data)
-    assert subscription.topic == Pleroma.Web.OStatus.feed_path(user)
-    assert subscription.state == "requested"
-    assert subscription.secret == "a random secret"
-    assert subscription.callback == "http://example.org/sub"
-  end
-
-  test "an incoming subscription request for an existing subscription" do
-    user = insert(:user)
-
-    sub =
-      insert(:websub_subscription, state: "accepted", topic: Pleroma.Web.OStatus.feed_path(user))
-
-    data = %{
-      "hub.callback" => sub.callback,
-      "hub.mode" => "subscribe",
-      "hub.topic" => Pleroma.Web.OStatus.feed_path(user),
-      "hub.secret" => "a random secret",
-      "hub.lease_seconds" => "100"
-    }
-
-    {:ok, subscription} = Websub.incoming_subscription_request(user, data)
-    assert subscription.topic == Pleroma.Web.OStatus.feed_path(user)
-    assert subscription.state == sub.state
-    assert subscription.secret == "a random secret"
-    assert subscription.callback == sub.callback
-    assert length(Repo.all(WebsubServerSubscription)) == 1
-    assert subscription.id == sub.id
-  end
-
-  def accepting_verifier(subscription) do
-    {:ok, %{subscription | state: "accepted"}}
-  end
-
-  test "initiate a subscription for a given user and topic" do
-    subscriber = insert(:user)
-    user = insert(:user, %{info: %Pleroma.User.Info{topic: "some_topic", hub: "some_hub"}})
-
-    {:ok, websub} = Websub.subscribe(subscriber, user, &accepting_verifier/1)
-    assert websub.subscribers == [subscriber.ap_id]
-    assert websub.topic == "some_topic"
-    assert websub.hub == "some_hub"
-    assert is_binary(websub.secret)
-    assert websub.user == user
-    assert websub.state == "accepted"
-  end
-
-  test "discovers the hub and canonical url" do
-    topic = "https://mastodon.social/users/lambadalambda.atom"
-
-    {:ok, discovered} = Websub.gather_feed_data(topic)
-
-    expected = %{
-      "hub" => "https://mastodon.social/api/push",
-      "uri" => "https://mastodon.social/users/lambadalambda",
-      "nickname" => "lambadalambda",
-      "name" => "Critical Value",
-      "host" => "mastodon.social",
-      "bio" => "a cool dude.",
-      "avatar" => %{
-        "type" => "Image",
-        "url" => [
-          %{
-            "href" =>
-              "https://files.mastodon.social/accounts/avatars/000/000/264/original/1429214160519.gif?1492379244",
-            "mediaType" => "image/gif",
-            "type" => "Link"
-          }
-        ]
-      }
-    }
-
-    assert expected == discovered
-  end
-
-  test "calls the hub, requests topic" do
-    hub = "https://social.heldscal.la/main/push/hub"
-    topic = "https://social.heldscal.la/api/statuses/user_timeline/23211.atom"
-    websub = insert(:websub_client_subscription, %{hub: hub, topic: topic})
-
-    poster = fn ^hub, {:form, data}, _headers ->
-      assert Keyword.get(data, :"hub.mode") == "subscribe"
-
-      assert Keyword.get(data, :"hub.callback") ==
-               Helpers.websub_url(
-                 Pleroma.Web.Endpoint,
-                 :websub_subscription_confirmation,
-                 websub.id
-               )
-
-      {:ok, %{status: 202}}
-    end
-
-    task = Task.async(fn -> Websub.request_subscription(websub, poster) end)
-
-    change = Ecto.Changeset.change(websub, %{state: "accepted"})
-    {:ok, _} = Repo.update(change)
-
-    {:ok, websub} = Task.await(task)
-
-    assert websub.state == "accepted"
-  end
-
-  test "rejects the subscription if it can't be accepted" do
-    hub = "https://social.heldscal.la/main/push/hub"
-    topic = "https://social.heldscal.la/api/statuses/user_timeline/23211.atom"
-    websub = insert(:websub_client_subscription, %{hub: hub, topic: topic})
-
-    poster = fn ^hub, {:form, _data}, _headers ->
-      {:ok, %{status: 202}}
-    end
-
-    {:error, websub} = Websub.request_subscription(websub, poster, 1000)
-    assert websub.state == "rejected"
-
-    websub = insert(:websub_client_subscription, %{hub: hub, topic: topic})
-
-    poster = fn ^hub, {:form, _data}, _headers ->
-      {:ok, %{status: 400}}
-    end
-
-    {:error, websub} = Websub.request_subscription(websub, poster, 1000)
-    assert websub.state == "rejected"
-  end
-
-  test "sign a text" do
-    signed = Websub.sign("secret", "text")
-    assert signed == "B8392C23690CCF871F37EC270BE1582DEC57A503" |> String.downcase()
-
-    _signed = Websub.sign("secret", [["て"], ['す']])
-  end
-
-  describe "renewing subscriptions" do
-    test "it renews subscriptions that have less than a day of time left" do
-      day = 60 * 60 * 24
-      now = NaiveDateTime.utc_now()
-
-      still_good =
-        insert(:websub_client_subscription, %{
-          valid_until: NaiveDateTime.add(now, 2 * day),
-          topic: "http://example.org/still_good",
-          hub: "http://example.org/still_good",
-          state: "accepted"
-        })
-
-      needs_refresh =
-        insert(:websub_client_subscription, %{
-          valid_until: NaiveDateTime.add(now, day - 100),
-          topic: "http://example.org/needs_refresh",
-          hub: "http://example.org/needs_refresh",
-          state: "accepted"
-        })
-
-      _refresh = Websub.refresh_subscriptions()
-      ObanHelpers.perform(all_enqueued(worker: SubscriberWorker))
-
-      assert still_good == Repo.get(WebsubClientSubscription, still_good.id)
-      refute needs_refresh == Repo.get(WebsubClientSubscription, needs_refresh.id)
-    end
-  end
-end