Resolve merge conflicts

CC-BY-NC-ND-4.0

@@ -1,403 +0,0 @@
-Attribution-NonCommercial-NoDerivatives 4.0 International
-
-=======================================================================
-
-Creative Commons Corporation ("Creative Commons") is not a law firm and
-does not provide legal services or legal advice. Distribution of
-Creative Commons public licenses does not create a lawyer-client or
-other relationship. Creative Commons makes its licenses and related
-information available on an "as-is" basis. Creative Commons gives no
-warranties regarding its licenses, any material licensed under their
-terms and conditions, or any related information. Creative Commons
-disclaims all liability for damages resulting from their use to the
-fullest extent possible.
-
-Using Creative Commons Public Licenses
-
-Creative Commons public licenses provide a standard set of terms and
-conditions that creators and other rights holders may use to share
-original works of authorship and other material subject to copyright
-and certain other rights specified in the public license below. The
-following considerations are for informational purposes only, are not
-exhaustive, and do not form part of our licenses.
-
-     Considerations for licensors: Our public licenses are
-     intended for use by those authorized to give the public
-     permission to use material in ways otherwise restricted by
-     copyright and certain other rights. Our licenses are
-     irrevocable. Licensors should read and understand the terms
-     and conditions of the license they choose before applying it.
-     Licensors should also secure all rights necessary before
-     applying our licenses so that the public can reuse the
-     material as expected. Licensors should clearly mark any
-     material not subject to the license. This includes other CC-
-     licensed material, or material used under an exception or
-     limitation to copyright. More considerations for licensors:
-	wiki.creativecommons.org/Considerations_for_licensors
-
-     Considerations for the public: By using one of our public
-     licenses, a licensor grants the public permission to use the
-     licensed material under specified terms and conditions. If
-     the licensor's permission is not necessary for any reason--for
-     example, because of any applicable exception or limitation to
-     copyright--then that use is not regulated by the license. Our
-     licenses grant only permissions under copyright and certain
-     other rights that a licensor has authority to grant. Use of
-     the licensed material may still be restricted for other
-     reasons, including because others have copyright or other
-     rights in the material. A licensor may make special requests,
-     such as asking that all changes be marked or described.
-     Although not required by our licenses, you are encouraged to
-     respect those requests where reasonable. More considerations
-     for the public: 
-	wiki.creativecommons.org/Considerations_for_licensees
-
-=======================================================================
-
-Creative Commons Attribution-NonCommercial-NoDerivatives 4.0
-International Public License
-
-By exercising the Licensed Rights (defined below), You accept and agree
-to be bound by the terms and conditions of this Creative Commons
-Attribution-NonCommercial-NoDerivatives 4.0 International Public
-License ("Public License"). To the extent this Public License may be
-interpreted as a contract, You are granted the Licensed Rights in
-consideration of Your acceptance of these terms and conditions, and the
-Licensor grants You such rights in consideration of benefits the
-Licensor receives from making the Licensed Material available under
-these terms and conditions.
-
-
-Section 1 -- Definitions.
-
-  a. Adapted Material means material subject to Copyright and Similar
-     Rights that is derived from or based upon the Licensed Material
-     and in which the Licensed Material is translated, altered,
-     arranged, transformed, or otherwise modified in a manner requiring
-     permission under the Copyright and Similar Rights held by the
-     Licensor. For purposes of this Public License, where the Licensed
-     Material is a musical work, performance, or sound recording,
-     Adapted Material is always produced where the Licensed Material is
-     synched in timed relation with a moving image.
-
-  b. Copyright and Similar Rights means copyright and/or similar rights
-     closely related to copyright including, without limitation,
-     performance, broadcast, sound recording, and Sui Generis Database
-     Rights, without regard to how the rights are labeled or
-     categorized. For purposes of this Public License, the rights
-     specified in Section 2(b)(1)-(2) are not Copyright and Similar
-     Rights.
-
-  c. Effective Technological Measures means those measures that, in the
-     absence of proper authority, may not be circumvented under laws
-     fulfilling obligations under Article 11 of the WIPO Copyright
-     Treaty adopted on December 20, 1996, and/or similar international
-     agreements.
-
-  d. Exceptions and Limitations means fair use, fair dealing, and/or
-     any other exception or limitation to Copyright and Similar Rights
-     that applies to Your use of the Licensed Material.
-
-  e. Licensed Material means the artistic or literary work, database,
-     or other material to which the Licensor applied this Public
-     License.
-
-  f. Licensed Rights means the rights granted to You subject to the
-     terms and conditions of this Public License, which are limited to
-     all Copyright and Similar Rights that apply to Your use of the
-     Licensed Material and that the Licensor has authority to license.
-
-  g. Licensor means the individual(s) or entity(ies) granting rights
-     under this Public License.
-
-  h. NonCommercial means not primarily intended for or directed towards
-     commercial advantage or monetary compensation. For purposes of
-     this Public License, the exchange of the Licensed Material for
-     other material subject to Copyright and Similar Rights by digital
-     file-sharing or similar means is NonCommercial provided there is
-     no payment of monetary compensation in connection with the
-     exchange.
-
-  i. Share means to provide material to the public by any means or
-     process that requires permission under the Licensed Rights, such
-     as reproduction, public display, public performance, distribution,
-     dissemination, communication, or importation, and to make material
-     available to the public including in ways that members of the
-     public may access the material from a place and at a time
-     individually chosen by them.
-
-  j. Sui Generis Database Rights means rights other than copyright
-     resulting from Directive 96/9/EC of the European Parliament and of
-     the Council of 11 March 1996 on the legal protection of databases,
-     as amended and/or succeeded, as well as other essentially
-     equivalent rights anywhere in the world.
-
-  k. You means the individual or entity exercising the Licensed Rights
-     under this Public License. Your has a corresponding meaning.
-
-
-Section 2 -- Scope.
-
-  a. License grant.
-
-       1. Subject to the terms and conditions of this Public License,
-          the Licensor hereby grants You a worldwide, royalty-free,
-          non-sublicensable, non-exclusive, irrevocable license to
-          exercise the Licensed Rights in the Licensed Material to:
-
-            a. reproduce and Share the Licensed Material, in whole or
-               in part, for NonCommercial purposes only; and
-
-            b. produce and reproduce, but not Share, Adapted Material
-               for NonCommercial purposes only.
-
-       2. Exceptions and Limitations. For the avoidance of doubt, where
-          Exceptions and Limitations apply to Your use, this Public
-          License does not apply, and You do not need to comply with
-          its terms and conditions.
-
-       3. Term. The term of this Public License is specified in Section
-          6(a).
-
-       4. Media and formats; technical modifications allowed. The
-          Licensor authorizes You to exercise the Licensed Rights in
-          all media and formats whether now known or hereafter created,
-          and to make technical modifications necessary to do so. The
-          Licensor waives and/or agrees not to assert any right or
-          authority to forbid You from making technical modifications
-          necessary to exercise the Licensed Rights, including
-          technical modifications necessary to circumvent Effective
-          Technological Measures. For purposes of this Public License,
-          simply making modifications authorized by this Section 2(a)
-          (4) never produces Adapted Material.
-
-       5. Downstream recipients.
-
-            a. Offer from the Licensor -- Licensed Material. Every
-               recipient of the Licensed Material automatically
-               receives an offer from the Licensor to exercise the
-               Licensed Rights under the terms and conditions of this
-               Public License.
-
-            b. No downstream restrictions. You may not offer or impose
-               any additional or different terms or conditions on, or
-               apply any Effective Technological Measures to, the
-               Licensed Material if doing so restricts exercise of the
-               Licensed Rights by any recipient of the Licensed
-               Material.
-
-       6. No endorsement. Nothing in this Public License constitutes or
-          may be construed as permission to assert or imply that You
-          are, or that Your use of the Licensed Material is, connected
-          with, or sponsored, endorsed, or granted official status by,
-          the Licensor or others designated to receive attribution as
-          provided in Section 3(a)(1)(A)(i).
-
-  b. Other rights.
-
-       1. Moral rights, such as the right of integrity, are not
-          licensed under this Public License, nor are publicity,
-          privacy, and/or other similar personality rights; however, to
-          the extent possible, the Licensor waives and/or agrees not to
-          assert any such rights held by the Licensor to the limited
-          extent necessary to allow You to exercise the Licensed
-          Rights, but not otherwise.
-
-       2. Patent and trademark rights are not licensed under this
-          Public License.
-
-       3. To the extent possible, the Licensor waives any right to
-          collect royalties from You for the exercise of the Licensed
-          Rights, whether directly or through a collecting society
-          under any voluntary or waivable statutory or compulsory
-          licensing scheme. In all other cases the Licensor expressly
-          reserves any right to collect such royalties, including when
-          the Licensed Material is used other than for NonCommercial
-          purposes.
-
-
-Section 3 -- License Conditions.
-
-Your exercise of the Licensed Rights is expressly made subject to the
-following conditions.
-
-  a. Attribution.
-
-       1. If You Share the Licensed Material, You must:
-
-            a. retain the following if it is supplied by the Licensor
-               with the Licensed Material:
-
-                 i. identification of the creator(s) of the Licensed
-                    Material and any others designated to receive
-                    attribution, in any reasonable manner requested by
-                    the Licensor (including by pseudonym if
-                    designated);
-
-                ii. a copyright notice;
-
-               iii. a notice that refers to this Public License;
-
-                iv. a notice that refers to the disclaimer of
-                    warranties;
-
-                 v. a URI or hyperlink to the Licensed Material to the
-                    extent reasonably practicable;
-
-            b. indicate if You modified the Licensed Material and
-               retain an indication of any previous modifications; and
-
-            c. indicate the Licensed Material is licensed under this
-               Public License, and include the text of, or the URI or
-               hyperlink to, this Public License.
-
-          For the avoidance of doubt, You do not have permission under
-          this Public License to Share Adapted Material.
-
-       2. You may satisfy the conditions in Section 3(a)(1) in any
-          reasonable manner based on the medium, means, and context in
-          which You Share the Licensed Material. For example, it may be
-          reasonable to satisfy the conditions by providing a URI or
-          hyperlink to a resource that includes the required
-          information.
-
-       3. If requested by the Licensor, You must remove any of the
-          information required by Section 3(a)(1)(A) to the extent
-          reasonably practicable.
-
-
-Section 4 -- Sui Generis Database Rights.
-
-Where the Licensed Rights include Sui Generis Database Rights that
-apply to Your use of the Licensed Material:
-
-  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
-     to extract, reuse, reproduce, and Share all or a substantial
-     portion of the contents of the database for NonCommercial purposes
-     only and provided You do not Share Adapted Material;
-
-  b. if You include all or a substantial portion of the database
-     contents in a database in which You have Sui Generis Database
-     Rights, then the database in which You have Sui Generis Database
-     Rights (but not its individual contents) is Adapted Material; and
-
-  c. You must comply with the conditions in Section 3(a) if You Share
-     all or a substantial portion of the contents of the database.
-
-For the avoidance of doubt, this Section 4 supplements and does not
-replace Your obligations under this Public License where the Licensed
-Rights include other Copyright and Similar Rights.
-
-
-Section 5 -- Disclaimer of Warranties and Limitation of Liability.
-
-  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
-     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
-     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
-     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
-     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
-     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
-     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
-     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
-     KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
-     ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
-
-  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
-     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
-     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
-     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
-     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
-     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
-     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
-     DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
-     IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
-
-  c. The disclaimer of warranties and limitation of liability provided
-     above shall be interpreted in a manner that, to the extent
-     possible, most closely approximates an absolute disclaimer and
-     waiver of all liability.
-
-
-Section 6 -- Term and Termination.
-
-  a. This Public License applies for the term of the Copyright and
-     Similar Rights licensed here. However, if You fail to comply with
-     this Public License, then Your rights under this Public License
-     terminate automatically.
-
-  b. Where Your right to use the Licensed Material has terminated under
-     Section 6(a), it reinstates:
-
-       1. automatically as of the date the violation is cured, provided
-          it is cured within 30 days of Your discovery of the
-          violation; or
-
-       2. upon express reinstatement by the Licensor.
-
-     For the avoidance of doubt, this Section 6(b) does not affect any
-     right the Licensor may have to seek remedies for Your violations
-     of this Public License.
-
-  c. For the avoidance of doubt, the Licensor may also offer the
-     Licensed Material under separate terms or conditions or stop
-     distributing the Licensed Material at any time; however, doing so
-     will not terminate this Public License.
-
-  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
-     License.
-
-
-Section 7 -- Other Terms and Conditions.
-
-  a. The Licensor shall not be bound by any additional or different
-     terms or conditions communicated by You unless expressly agreed.
-
-  b. Any arrangements, understandings, or agreements regarding the
-     Licensed Material not stated herein are separate from and
-     independent of the terms and conditions of this Public License.
-
-
-Section 8 -- Interpretation.
-
-  a. For the avoidance of doubt, this Public License does not, and
-     shall not be interpreted to, reduce, limit, restrict, or impose
-     conditions on any use of the Licensed Material that could lawfully
-     be made without permission under this Public License.
-
-  b. To the extent possible, if any provision of this Public License is
-     deemed unenforceable, it shall be automatically reformed to the
-     minimum extent necessary to make it enforceable. If the provision
-     cannot be reformed, it shall be severed from this Public License
-     without affecting the enforceability of the remaining terms and
-     conditions.
-
-  c. No term or condition of this Public License will be waived and no
-     failure to comply consented to unless expressly agreed to by the
-     Licensor.
-
-  d. Nothing in this Public License constitutes or may be interpreted
-     as a limitation upon, or waiver of, any privileges and immunities
-     that apply to the Licensor or You, including from the legal
-     processes of any jurisdiction or authority.
-
-=======================================================================
-
-Creative Commons is not a party to its public
-licenses. Notwithstanding, Creative Commons may elect to apply one of
-its public licenses to material it publishes and in those instances
-will be considered the “Licensor.” The text of the Creative Commons
-public licenses is dedicated to the public domain under the CC0 Public
-Domain Dedication. Except for the limited purpose of indicating that
-material is shared under a Creative Commons public license or as
-otherwise permitted by the Creative Commons policies published at
-creativecommons.org/policies, Creative Commons does not authorize the
-use of the trademark "Creative Commons" or any other trademark or logo
-of Creative Commons without its prior written consent including,
-without limitation, in connection with any unauthorized modifications
-to any of its public licenses or any other arrangements,
-understandings, or agreements concerning use of licensed material. For
-the avoidance of doubt, this paragraph does not form part of the
-public licenses.
-
-Creative Commons may be contacted at creativecommons.org.
-

CHANGELOG.md

@@ -10,32 +10,53 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
 - Federation: Return 403 errors when trying to request pages from a user's follower/following collections if they have `hide_followers`/`hide_follows` set
 - NodeInfo: Return `skipThreadContainment` in `metadata` for the `skip_thread_containment` option
+- Mastodon API: Unsubscribe followers when they unfollow a user
 
 ### Fixed
 - Not being able to pin unlisted posts
 - Metadata rendering errors resulting in the entire page being inaccessible
+- Federation/MediaProxy not working with instances that have wrong certificate order
 - Mastodon API: Handling of search timeouts (`/api/v1/search` and `/api/v2/search`)
 - Mastodon API: Embedded relationships not being properly rendered in the Account entity of Status entity
 - Mastodon API: Add `account_id`, `type`, `offset`, and `limit` to search API (`/api/v1/search` and `/api/v2/search`)
 - ActivityPub C2S: follower/following collection pages being inaccessible even when authentifucated if `hide_followers`/ `hide_follows` was set
+- Existing user id not being preserved on insert conflict
 
 ### Added
 - MRF: Support for priming the mediaproxy cache (`Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`)
-Configuration: `federation_incoming_replies_max_depth` option
+- MRF: Support for excluding specific domains from Transparency.
+- MRF: Support for filtering posts based on who they mention (`Pleroma.Web.ActivityPub.MRF.MentionPolicy`)
+- Configuration: `federation_incoming_replies_max_depth` option
 - Mastodon API: Support for the [`tagged` filter](https://github.com/tootsuite/mastodon/pull/9755) in [`GET /api/v1/accounts/:id/statuses`](https://docs.joinmastodon.org/api/rest/accounts/#get-api-v1-accounts-id-statuses)
 - Mastodon API, streaming: Add support for passing the token in the `Sec-WebSocket-Protocol` header
 - Mastodon API, extension: Ability to reset avatar, profile banner, and background
+- Mastodon API: Add support for categories for custom emojis by reusing the group feature. <https://github.com/tootsuite/mastodon/pull/11196>
+- Mastodon API: Add support for muting/unmuting notifications
+- Mastodon API: Add support for the `blocked_by` attribute in the relationship API (`GET /api/v1/accounts/relationships`). <https://github.com/tootsuite/mastodon/pull/10373>
+- Mastodon API: Add `pleroma.deactivated` to the Account entity
+- Mastodon API: added `/auth/password` endpoint for password reset with rate limit.
+- Mastodon API: /api/v1/accounts/:id/statuses now supports nicknames or user id
 - Admin API: Return users' tags when querying reports
 - Admin API: Return avatar and display name when querying users
 - Admin API: Allow querying user by ID
 - Admin API: Added support for `tuples`.
 - Added synchronization of following/followers counters for external users
 - Configuration: `enabled` option for `Pleroma.Emails.Mailer`, defaulting to `false`.
-- Mastodon API: Add support for categories for custom emojis by reusing the group feature. <https://github.com/tootsuite/mastodon/pull/11196>
+- Configuration: Pleroma.Plugs.RateLimiter `bucket_name`, `params` options.
+- Addressable lists
+- Twitter API: added rate limit for `/api/account/password_reset` endpoint.
+- ActivityPub: Add an internal service actor for fetching ActivityPub objects.
+- ActivityPub: Optional signing of ActivityPub object fetches.
 
 ### Changed
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
 - Admin API: changed json structure for saving config settings.
+- RichMedia: parsers and their order are configured in `rich_media` config.
+- RichMedia: add the rich media ttl based on image expiration time.
+
+## [1.0.1] - 2019-07-14
+### Security
+- OStatus: fix an object spoofing vulnerability.
 
 ## [1.0.0] - 2019-06-29
 ### Security

config/config.exs

@@ -194,6 +194,8 @@
   send_user_agent: true,
   adapter: [
     ssl_options: [
+      # Workaround for remote server certificate chain issues
+      partial_chain: &:hackney_connect.partial_chain/1,
       # We don't support TLS v1.3 yet
       versions: [:tlsv1, :"tlsv1.1", :"tlsv1.2"]
     ]
@@ -238,6 +240,7 @@
     "text/bbcode"
   ],
   mrf_transparency: true,
+  mrf_transparency_exclusions: [],
   autofollowed_nicknames: [],
   max_pinned_statuses: 1,
   no_attachment_links: false,
@@ -302,7 +305,8 @@
   accept_blocks: true,
   unfollow_blocked: true,
   outgoing_blocks: true,
-  follow_handshake_timeout: 500
+  follow_handshake_timeout: 500,
+  sign_object_fetches: true
 
 config :pleroma, :user, deny_follow_blocked: true
 
@@ -336,7 +340,13 @@
 config :pleroma, :rich_media,
   enabled: true,
   ignore_hosts: [],
-  ignore_tld: ["local", "localdomain", "lan"]
+  ignore_tld: ["local", "localdomain", "lan"],
+  parsers: [
+    Pleroma.Web.RichMedia.Parsers.TwitterCard,
+    Pleroma.Web.RichMedia.Parsers.OGP,
+    Pleroma.Web.RichMedia.Parsers.OEmbed
+  ],
+  ttl_setters: [Pleroma.Web.RichMedia.Parser.TTL.AwsSignedUrl]
 
 config :pleroma, :media_proxy,
   enabled: false,
@@ -519,7 +529,12 @@
 
 config :pleroma, :rate_limit,
   search: [{1000, 10}, {1000, 30}],
-  app_account_creation: {1_800_000, 25}
+  app_account_creation: {1_800_000, 25},
+  relations_actions: {10_000, 10},
+  relation_id_action: {60_000, 2},
+  statuses_actions: {10_000, 15},
+  status_id_action: {60_000, 3},
+  password_reset: {1_800_000, 5}
 
 # Import environment specific config. This must remain at the bottom
 # of this file so it overrides the configuration defined above.

config/test.exs

@@ -32,6 +32,8 @@
   federating: false,
   external_user_synchronization: false
 
+config :pleroma, :activitypub, sign_object_fetches: false
+
 # Configure your database
 config :pleroma, Pleroma.Repo,
   adapter: Ecto.Adapters.Postgres,
@@ -68,7 +70,8 @@
 
 config :pleroma, :rate_limit,
   search: [{1000, 30}, {1000, 30}],
-  app_account_creation: {10_000, 5}
+  app_account_creation: {10_000, 5},
+  password_reset: {1000, 30}
 
 config :pleroma, :http_security, report_uri: "https://endpoint.com"
 

docs/api/differences_in_mastoapi_responses.md

@@ -16,9 +16,11 @@ Adding the parameter `with_muted=true` to the timeline queries will also return
 
 ## Statuses
 
+- `visibility`: has an additional possible value `list`
+
 Has these additional fields under the `pleroma` object:
 
-- `local`: true if the post was made on the local instance.
+- `local`: true if the post was made on the local instance
 - `conversation_id`: the ID of the conversation the status is associated with (if any)
 - `in_reply_to_account_acct`: the `acct` property of User entity for replied user (if any)
 - `content`: a map consisting of alternate representations of the `content` property with the key being it's mimetype. Currently the only alternate representation supported is `text/plain`
@@ -32,7 +34,10 @@ Has these additional fields under the `pleroma` object:
 
 ## Accounts
 
-- `/api/v1/accounts/:id`: The `id` parameter can also be the `nickname` of the user. This only works in this endpoint, not the deeper nested ones for following etc.
+The `id` parameter can also be the `nickname` of the user. This only works in these endpoints, not the deeper nested ones for following etc.
+
+- `/api/v1/accounts/:id`
+- `/api/v1/accounts/:id/statuses`
 
 Has these additional fields under the `pleroma` object:
 
@@ -45,6 +50,7 @@ Has these additional fields under the `pleroma` object:
 - `hide_follows`: boolean, true when the user has follow hiding enabled
 - `settings_store`: A generic map of settings for frontends. Opaque to the backend. Only returned in `verify_credentials` and `update_credentials`
 - `chat_token`: The token needed for Pleroma chat. Only returned in `verify_credentials`
+- `deactivated`: boolean, true when the user is deactivated
 
 ### Source
 
@@ -72,6 +78,7 @@ Additional parameters can be added to the JSON body/Form data:
 - `preview`: boolean, if set to `true` the post won't be actually posted, but the status entitiy would still be rendered back. This could be useful for previewing rich text/custom emoji, for example.
 - `content_type`: string, contain the MIME type of the status, it is transformed into HTML by the backend. You can get the list of the supported MIME types with the nodeinfo endpoint.
 - `to`: A list of nicknames (like `lain@soykaf.club` or `lain` on the local server) that will be used to determine who is going to be addressed by this post. Using this will disable the implicit addressing by mentioned names in the `status` body, only the people in the `to` list will be addressed. The normal rules for for post visibility are not affected by this and will still apply.
+- `visibility`: string, besides standard MastoAPI values (`direct`, `private`, `unlisted` or `public`) it can be used to address a List by setting it to `list:LIST_ID`.
 
 ## PATCH `/api/v1/update_credentials`
 

docs/clients.md

@@ -31,10 +31,11 @@ Feel free to contact us to be added to this list!
 - Features: No Streaming
 
 ### Fedilab
-- Source Code: <https://gitlab.com/tom79/mastalab/>
+- Homepage: <https://fedilab.app/>
-- Contact: [@tom79@mastodon.social](https://mastodon.social/users/tom79)
+- Source Code: <https://framagit.org/tom79/fedilab/>
+- Contact: [@fedilab@framapiaf.org](https://framapiaf.org/users/fedilab)
 - Platforms: Android
-- Features: Streaming Ready
+- Features: Streaming Ready, Moderation, Text Formatting 
 
 ### Nekonium
 - Homepage: [F-Droid Repository](https://repo.gdgd.jp.net/), [Google Play](https://play.google.com/store/apps/details?id=com.apps.nekonium), [Amazon](https://www.amazon.co.jp/dp/B076FXPRBC/)

docs/config.md

@@ -101,11 +101,13 @@ config :pleroma, Pleroma.Emails.Mailer,
   * `Pleroma.Web.ActivityPub.MRF.EnsureRePrepended`: Rewrites posts to ensure that replies to posts with subjects do not have an identical subject and instead begin with re:.
   * `Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy`: Rejects posts from likely spambots by rejecting posts from new users that contain links.
   * `Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`: Crawls attachments using their MediaProxy URLs so that the MediaProxy cache is primed.
+  * `Pleroma.Web.ActivityPub.MRF.MentionPolicy`: Drops posts mentioning configurable users. (see `:mrf_mention` section)
 * `public`: Makes the client API in authentificated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
 * `quarantined_instances`: List of ActivityPub instances where private(DMs, followers-only) activities will not be send.
 * `managed_config`: Whenether the config for pleroma-fe is configured in this config or in ``static/config.json``
 * `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML)
 * `mrf_transparency`: Make the content of your Message Rewrite Facility settings public (via nodeinfo).
+* `mrf_transparency_exclusions`: Exclude specific instance names from MRF transparency.  The use of the exclusions feature will be disclosed in nodeinfo as a boolean value.
 * `scope_copy`: Copy the scope (private/unlisted/public) in replies to posts by default.
 * `subject_line_behavior`: Allows changing the default behaviour of subject lines in replies. Valid values:
   * "email": Copy and preprend re:, as in email.
@@ -270,6 +272,9 @@ config :pleroma, :mrf_subchain,
 * `federated_timeline_removal`: A list of patterns which result in message being removed from federated timelines (a.k.a unlisted), each pattern can be a string or a [regular expression](https://hexdocs.pm/elixir/Regex.html)
 * `replace`: A list of tuples containing `{pattern, replacement}`, `pattern` can be a string or a [regular expression](https://hexdocs.pm/elixir/Regex.html)
 
+## :mrf_mention
+* `actors`: A list of actors, for which to drop any posts mentioning.
+
 ## :media_proxy
 * `enabled`: Enables proxying of remote media to the instance’s proxy
 * `base_url`: The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host/CDN fronts.
@@ -327,6 +332,7 @@ This will make Pleroma listen on `127.0.0.1` port `8080` and generate urls start
 * ``unfollow_blocked``: Whether blocks result in people getting unfollowed
 * ``outgoing_blocks``: Whether to federate blocks to other instances
 * ``deny_follow_blocked``: Whether to disallow following an account that has blocked the user in question
+* ``sign_object_fetches``: Sign object fetches with HTTP signatures
 
 ## :http_security
 * ``enabled``: Whether the managed content security policy is enabled
@@ -424,6 +430,7 @@ This config contains two queues: `federator_incoming` and `federator_outgoing`.
 * `enabled`: if enabled the instance will parse metadata from attached links to generate link previews
 * `ignore_hosts`: list of hosts which will be ignored by the metadata parser. For example `["accounts.google.com", "xss.website"]`, defaults to `[]`.
 * `ignore_tld`: list TLDs (top-level domains) which will ignore for parse metadata. default is ["local", "localdomain", "lan"]
+* `parsers`: list of Rich Media parsers
 
 ## :fetch_initial_posts
 * `enabled`: if enabled, when a new user is federated with, fetch some of their latest posts
@@ -640,3 +647,12 @@ A keyword list of rate limiters where a key is a limiter name and value is the l
 It is also possible to have different limits for unauthenticated and authenticated users: the keyword value must be a list of two tuples where the first one is a config for unauthenticated users and the second one is for authenticated.
 
 See [`Pleroma.Plugs.RateLimiter`](Pleroma.Plugs.RateLimiter.html) documentation for examples.
+
+Supported rate limiters:
+
+* `:search` for the search requests (account & status search etc.)
+* `:app_account_creation` for registering user accounts from the same IP address
+* `:relations_actions` for actions on relations with all users (follow, unfollow)
+* `:relation_id_action` for actions on relation with a specific user (follow, unfollow)
+* `:statuses_actions` for create / delete / fav / unfav / reblog / unreblog actions on any statuses
+* `:status_id_action` for fav / unfav or reblog / unreblog actions on the same status by the same user

docs/config/howto_mediaproxy.md

@@ -24,7 +24,9 @@ If you came here from one of the installation guides, take a look at the example
 ```
 config :pleroma, :media_proxy,
       enabled: true,
+      proxy_opts: [
             redirect_on_failure: true
+      ]
       #base_url: "https://cache.pleroma.social"
 ```
 If you want to use a subdomain to serve the files, uncomment `base_url`, change the url and add a comma after `true` in the previous line.

docs/config/howto_set_richmedia_cache_ttl_based_on_image.md

@@ -0,0 +1,33 @@
+# How to set rich media cache ttl based on image ttl
+## Explanation
+
+Richmedia are cached without the ttl but the rich media may have image which can expire, like aws signed url.
+In such cases the old image url (expired) is returned from the media cache.
+
+So to avoid such situation we can define a module that will set ttl based on image.
+The module must adopt behaviour `Pleroma.Web.RichMedia.Parser.TTL`
+
+### Example
+
+```exs
+defmodule MyModule do
+  @behaviour Pleroma.Web.RichMedia.Parser.TTL
+
+  @impl Pleroma.Web.RichMedia.Parser.TTL
+  def ttl(data, url) do
+    image_url = Map.get(data, :image)
+    # do some parsing in the url and get the ttl of the image
+    # return ttl is unix time
+    parse_ttl_from_url(image_url)
+  end
+end
+```
+
+And update the config
+
+```exs
+config :pleroma, :rich_media,
+  ttl_setters: [Pleroma.Web.RichMedia.Parser.TTL.AwsSignedUrl, MyModule]
+```
+
+> For reference there is a parser for AWS signed URL `Pleroma.Web.RichMedia.Parser.TTL.AwsSignedUrl`, it's enabled by default.

docs/installation/alpine_linux_en.md

@@ -202,7 +202,6 @@ sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress
 
 #### Further reading
 
-* [Admin tasks](Admin tasks)
 * [Backup your instance](backup.html)
 * [Configuration tips](general-tips-for-customizing-pleroma-fe.html)
 * [Hardening your instance](hardening.html)

docs/installation/arch_linux_en.md

@@ -200,7 +200,6 @@ sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress
 
 #### Further reading
 
-* [Admin tasks](Admin tasks)
 * [Backup your instance](backup.html)
 * [Configuration tips](general-tips-for-customizing-pleroma-fe.html)
 * [Hardening your instance](hardening.html)

docs/installation/centos7_en.md

@@ -264,7 +264,6 @@ sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress
 
 #### Further reading
 
-* [Admin tasks](Admin tasks)
 * [Backup your instance](backup.html)
 * [Configuration tips](general-tips-for-customizing-pleroma-fe.html)
 * [Hardening your instance](hardening.html)

docs/installation/debian_based_en.md

@@ -190,7 +190,6 @@ sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress
 
 #### Further reading
 
-* [Admin tasks](Admin tasks)
 * [Backup your instance](backup.html)
 * [Configuration tips](general-tips-for-customizing-pleroma-fe.html)
 * [Hardening your instance](hardening.html)

docs/installation/debian_based_jp.md

@@ -180,7 +180,6 @@ mix set_moderator username [true|false]
 
 #### コンフィギュレーションとカスタマイズ
 
-* [Admin tasks](Admin tasks)
 * [Backup your instance](backup.html)
 * [Configuration tips](general-tips-for-customizing-pleroma-fe.html)
 * [Hardening your instance](hardening.html)

docs/installation/gentoo_en.md

@@ -283,7 +283,6 @@ If you opted to allow sudo for the `pleroma` user but would like to remove the a
 
 #### Further reading
 
-* [Admin tasks](Admin tasks)
 * [Backup your instance](backup.html)
 * [Configuration tips](general-tips-for-customizing-pleroma-fe.html)
 * [Hardening your instance](hardening.html)

docs/installation/otp_en.md

@@ -242,6 +242,14 @@ So for example, if the task is `mix pleroma.user set admin --admin`, you should
 ```sh
 su pleroma -s $SHELL -lc "./bin/pleroma_ctl user set admin --admin"
 ```
+
+## Create your first user and set as admin
+```sh
+cd /opt/pleroma/bin
+su pleroma -s $SHELL -lc "./bin/pleroma_ctl user new joeuser joeuser@sld.tld --admin"
+```
+This will create an account withe the username of 'joeuser' with the email address of joeuser@sld.tld, and set that user's account as an admin. This will result in a link that you can paste into the browser, which logs you in and enables you to set the password.
+
 ### Updating
 Generally, doing the following is enough:
 ```sh

lib/mix/tasks/pleroma/config.ex

@@ -28,6 +28,14 @@ def run(["migrate_to_db"]) do
       |> Enum.reject(fn {k, _v} -> k in [Pleroma.Repo, :env] end)
       |> Enum.each(fn {k, v} ->
         key = to_string(k) |> String.replace("Elixir.", "")
+
+        key =
+          if String.starts_with?(key, "Pleroma.") do
+            key
+          else
+            ":" <> key
+          end
+
         {:ok, _} = Config.update_or_create(%{group: "pleroma", key: key, value: v})
         Mix.shell().info("#{key} is migrated.")
       end)
@@ -53,17 +61,9 @@ def run(["migrate_from_db", env, delete?]) do
 
       Repo.all(Config)
       |> Enum.each(fn config ->
-        mark =
-          if String.starts_with?(config.key, "Pleroma.") or
-               String.starts_with?(config.key, "Ueberauth"),
-             do: ",",
-             else: ":"
-
         IO.write(
           file,
-          "config :#{config.group}, #{config.key}#{mark} #{
+          "config :#{config.group}, #{config.key}, #{inspect(Config.from_binary(config.value))}\r\n\r\n"
-            inspect(Config.from_binary(config.value))
-          }\r\n"
         )
 
         if delete? do

lib/mix/tasks/pleroma/user.ex

@@ -62,6 +62,10 @@ defmodule Mix.Tasks.Pleroma.User do
 
       mix pleroma.user unsubscribe NICKNAME
 
+  ## Unsubscribe local users from an entire instance and deactivate all accounts
+
+      mix pleroma.user unsubscribe_all_from_instance INSTANCE
+
   ## Create a password reset link.
 
       mix pleroma.user reset_password NICKNAME
@@ -246,6 +250,20 @@ def run(["unsubscribe", nickname]) do
     end
   end
 
+  def run(["unsubscribe_all_from_instance", instance]) do
+    start_pleroma()
+
+    Pleroma.User.Query.build(%{nickname: "@#{instance}"})
+    |> Pleroma.RepoStreamer.chunk_stream(500)
+    |> Stream.each(fn users ->
+      users
+      |> Enum.each(fn user ->
+        run(["unsubscribe", user.nickname])
+      end)
+    end)
+    |> Stream.run()
+  end
+
   def run(["set", nickname | rest]) do
     start_pleroma()
 

lib/pleroma/application.ex

@@ -140,6 +140,11 @@ def start(_type, _args) do
             id: :federator_init,
             start: {Task, :start_link, [&Pleroma.Web.Federator.init/0]},
             restart: :temporary
+          },
+          %{
+            id: :internal_fetch_init,
+            start: {Task, :start_link, [&Pleroma.Web.ActivityPub.InternalFetchActor.init/0]},
+            restart: :temporary
           }
         ] ++
         streamer_child() ++

lib/pleroma/config/transfer_task.ex

@@ -35,7 +35,7 @@ defp update_env(setting) do
         if String.starts_with?(setting.key, "Pleroma.") do
           "Elixir." <> setting.key
         else
-          setting.key
+          String.trim_leading(setting.key, ":")
         end
 
       group = String.to_existing_atom(setting.group)

lib/pleroma/http/connection.ex

@@ -29,7 +29,7 @@ def new(opts \\ []) do
 
   # fetch Hackney options
   #
-  defp hackney_options(opts) do
+  def hackney_options(opts) do
     options = Keyword.get(opts, :adapter, [])
     adapter_options = Pleroma.Config.get([:http, :adapter], [])
     proxy_url = Pleroma.Config.get([:http, :proxy_url], nil)

lib/pleroma/http/http.ex

@@ -65,10 +65,7 @@ defp process_sni_options(options, url) do
   end
 
   def process_request_options(options) do
-    case Pleroma.Config.get([:http, :proxy_url]) do
+    Keyword.merge(Pleroma.HTTP.Connection.hackney_options([]), options)
-      nil -> options
-      proxy -> options ++ [proxy: proxy]
-    end
   end
 
   @doc """

lib/pleroma/keys.ex

@@ -35,10 +35,12 @@ def generate_rsa_pem do
   end
 
   def keys_from_pem(pem) do
-    [private_key_code] = :public_key.pem_decode(pem)
+    with [private_key_code] <- :public_key.pem_decode(pem),
-    private_key = :public_key.pem_entry_decode(private_key_code)
+         private_key <- :public_key.pem_entry_decode(private_key_code),
-    {:RSAPrivateKey, _, modulus, exponent, _, _, _, _, _, _, _} = private_key
+         {:RSAPrivateKey, _, modulus, exponent, _, _, _, _, _, _, _} <- private_key do
-    public_key = {:RSAPublicKey, modulus, exponent}
+      {:ok, private_key, {:RSAPublicKey, modulus, exponent}}
-    {:ok, private_key, public_key}
+    else
+      error -> {:error, error}
+    end
   end
 end

lib/pleroma/list.ex

@@ -16,6 +16,7 @@ defmodule Pleroma.List do
     belongs_to(:user, User, type: Pleroma.FlakeId)
     field(:title, :string)
     field(:following, {:array, :string}, default: [])
+    field(:ap_id, :string)
 
     timestamps()
   end
@@ -55,6 +56,10 @@ def get(id, %{id: user_id} = _user) do
     Repo.one(query)
   end
 
+  def get_by_ap_id(ap_id) do
+    Repo.get_by(__MODULE__, ap_id: ap_id)
+  end
+
   def get_following(%Pleroma.List{following: following} = _list) do
     q =
       from(
@@ -105,7 +110,14 @@ def rename(%Pleroma.List{} = list, title) do
 
   def create(title, %User{} = creator) do
     list = %Pleroma.List{user_id: creator.id, title: title}
-    Repo.insert(list)
+
+    Repo.transaction(fn ->
+      list = Repo.insert!(list)
+
+      list
+      |> change(ap_id: "#{creator.ap_id}/lists/#{list.id}")
+      |> Repo.update!()
+    end)
   end
 
   def follow(%Pleroma.List{following: following} = list, %User{} = followed) do
@@ -125,4 +137,19 @@ def update_follows(%Pleroma.List{} = list, attrs) do
     |> follow_changeset(attrs)
     |> Repo.update()
   end
+
+  def memberships(%User{follower_address: follower_address}) do
+    Pleroma.List
+    |> where([l], ^follower_address in l.following)
+    |> select([l], l.ap_id)
+    |> Repo.all()
+  end
+
+  def memberships(_), do: []
+
+  def member?(%Pleroma.List{following: following}, %User{follower_address: follower_address}) do
+    Enum.member?(following, follower_address)
+  end
+
+  def member?(_, _), do: false
 end

lib/pleroma/notification.ex

@@ -11,7 +11,6 @@ defmodule Pleroma.Notification do
   alias Pleroma.Pagination
   alias Pleroma.Repo
   alias Pleroma.User
-  alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.CommonAPI.Utils
   alias Pleroma.Web.Push
   alias Pleroma.Web.Streamer
@@ -32,7 +31,8 @@ def changeset(%Notification{} = notification, attrs) do
     |> cast(attrs, [:seen])
   end
 
-  def for_user_query(user) do
+  def for_user_query(user, opts) do
+    query =
       Notification
       |> where(user_id: ^user.id)
       |> where(
@@ -52,11 +52,26 @@ def for_user_query(user) do
           )
       )
       |> preload([n, a, o], activity: {a, object: o})
+
+    if opts[:with_muted] do
+      query
+    else
+      where(query, [n, a], a.actor not in ^user.info.muted_notifications)
+      |> where([n, a], a.actor not in ^user.info.blocks)
+      |> where(
+        [n, a],
+        fragment("substring(? from '.*://([^/]*)')", a.actor) not in ^user.info.domain_blocks
+      )
+      |> join(:left, [n, a], tm in Pleroma.ThreadMute,
+        on: tm.user_id == ^user.id and tm.context == fragment("?->>'context'", a.data)
+      )
+      |> where([n, a, o, tm], is_nil(tm.user_id))
+    end
   end
 
   def for_user(user, opts \\ %{}) do
     user
-    |> for_user_query()
+    |> for_user_query(opts)
     |> Pagination.fetch_paginated(opts)
   end
 
@@ -179,11 +194,10 @@ def get_notified_from_activity(
 
   def get_notified_from_activity(_, _local_only), do: []
 
+  @spec skip?(Activity.t(), User.t()) :: boolean()
   def skip?(activity, user) do
     [
       :self,
-      :blocked,
-      :muted,
       :followers,
       :follows,
       :non_followers,
@@ -193,21 +207,11 @@ def skip?(activity, user) do
     |> Enum.any?(&skip?(&1, activity, user))
   end
 
+  @spec skip?(atom(), Activity.t(), User.t()) :: boolean()
   def skip?(:self, activity, user) do
     activity.data["actor"] == user.ap_id
   end
 
-  def skip?(:blocked, activity, user) do
-    actor = activity.data["actor"]
-    User.blocks?(user, %{ap_id: actor})
-  end
-
-  def skip?(:muted, activity, user) do
-    actor = activity.data["actor"]
-
-    User.mutes?(user, %{ap_id: actor}) or CommonAPI.thread_muted?(user, activity)
-  end
-
   def skip?(
         :followers,
         activity,

lib/pleroma/object/containment.ex

@@ -48,6 +48,9 @@ def contain_origin(id, %{"actor" => _actor} = params) do
     end
   end
 
+  def contain_origin(id, %{"attributedTo" => actor} = params),
+    do: contain_origin(id, Map.put(params, "actor", actor))
+
   def contain_origin_from_id(_id, %{"id" => nil}), do: :error
 
   def contain_origin_from_id(id, %{"id" => other_id} = _params) do
@@ -60,4 +63,9 @@ def contain_origin_from_id(id, %{"id" => other_id} = _params) do
       :error
     end
   end
+
+  def contain_child(%{"object" => %{"id" => id, "attributedTo" => _} = object}),
+    do: contain_origin(id, object)
+
+  def contain_child(_), do: :ok
 end

lib/pleroma/object/fetcher.ex

@@ -6,6 +6,8 @@ defmodule Pleroma.Object.Fetcher do
   alias Pleroma.HTTP
   alias Pleroma.Object
   alias Pleroma.Object.Containment
+  alias Pleroma.Signature
+  alias Pleroma.Web.ActivityPub.InternalFetchActor
   alias Pleroma.Web.ActivityPub.Transmogrifier
   alias Pleroma.Web.OStatus
 
@@ -32,33 +34,39 @@ def fetch_object_from_id(id, options \\ []) do
     else
       Logger.info("Fetching #{id} via AP")
 
-      with {:ok, data} <- fetch_and_contain_remote_object_from_id(id),
+      with {:fetch, {:ok, data}} <- {:fetch, fetch_and_contain_remote_object_from_id(id)},
-           nil <- Object.normalize(data, false),
+           {:normalize, nil} <- {:normalize, Object.normalize(data, false)},
            params <- %{
              "type" => "Create",
              "to" => data["to"],
              "cc" => data["cc"],
+             # Should we seriously keep this attributedTo thing?
              "actor" => data["actor"] || data["attributedTo"],
              "object" => data
            },
-           :ok <- Containment.contain_origin(id, params),
+           {:containment, :ok} <- {:containment, Containment.contain_origin(id, params)},
            {:ok, activity} <- Transmogrifier.handle_incoming(params, options),
            {:object, _data, %Object{} = object} <-
              {:object, data, Object.normalize(activity, false)} do
         {:ok, object}
       else
+        {:containment, _} ->
+          {:error, "Object containment failed."}
+
         {:error, {:reject, nil}} ->
           {:reject, nil}
 
         {:object, data, nil} ->
           reinject_object(data)
 
-        object = %Object{} ->
+        {:normalize, object = %Object{}} ->
           {:ok, object}
 
         _e ->
+          # Only fallback when receiving a fetch/normalization error with ActivityPub
           Logger.info("Couldn't get object via AP, trying out OStatus fetching...")
 
+          # FIXME: OStatus Object Containment?
           case OStatus.fetch_activity_from_url(id) do
             {:ok, [activity | _]} -> {:ok, Object.normalize(activity, false)}
             e -> e
@@ -76,15 +84,52 @@ def fetch_object_from_id!(id, options \\ []) do
     end
   end
 
+  defp make_signature(id, date) do
+    uri = URI.parse(id)
+
+    signature =
+      InternalFetchActor.get_actor()
+      |> Signature.sign(%{
+        "(request-target)": "get #{uri.path}",
+        host: uri.host,
+        date: date
+      })
+
+    [{:Signature, signature}]
+  end
+
+  defp sign_fetch(headers, id, date) do
+    if Pleroma.Config.get([:activitypub, :sign_object_fetches]) do
+      headers ++ make_signature(id, date)
+    else
+      headers
+    end
+  end
+
+  defp maybe_date_fetch(headers, date) do
+    if Pleroma.Config.get([:activitypub, :sign_object_fetches]) do
+      headers ++ [{:Date, date}]
+    else
+      headers
+    end
+  end
+
   def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
     Logger.info("Fetching object #{id} via AP")
 
-    with true <- String.starts_with?(id, "http"),
+    date =
-         {:ok, %{body: body, status: code}} when code in 200..299 <-
+      NaiveDateTime.utc_now()
-           HTTP.get(
+      |> Timex.format!("{WDshort}, {0D} {Mshort} {YYYY} {h24}:{m}:{s} GMT")
-             id,
+
+    headers =
       [{:Accept, "application/activity+json"}]
-           ),
+      |> maybe_date_fetch(date)
+      |> sign_fetch(id, date)
+
+    Logger.debug("Fetch headers: #{inspect(headers)}")
+
+    with true <- String.starts_with?(id, "http"),
+         {:ok, %{body: body, status: code}} when code in 200..299 <- HTTP.get(id, headers),
          {:ok, data} <- Jason.decode(body),
          :ok <- Containment.contain_origin_from_id(id, data) do
       {:ok, data}
@@ -97,8 +142,8 @@ def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
     end
   end
 
-  def fetch_and_contain_remote_object_from_id(%{"id" => id), do: fetch_and_contain_remote_object_from_id(id)
+  def fetch_and_contain_remote_object_from_id(%{"id" => id}),
+    do: fetch_and_contain_remote_object_from_id(id)
+
   def fetch_and_contain_remote_object_from_id(_id), do: {:error, "id must be a string"}
-    {:error, "id must be a string"}
-  end
 end

lib/pleroma/plugs/authentication_plug.ex

@@ -6,9 +6,21 @@ defmodule Pleroma.Plugs.AuthenticationPlug do
   alias Comeonin.Pbkdf2
   import Plug.Conn
   alias Pleroma.User
+  require Logger
 
-  def init(options) do
+  def init(options), do: options
-    options
+
+  def checkpw(password, "$6" <> _ = password_hash) do
+    :crypt.crypt(password, password_hash) == password_hash
+  end
+
+  def checkpw(password, "$pbkdf2" <> _ = password_hash) do
+    Pbkdf2.checkpw(password, password_hash)
+  end
+
+  def checkpw(_password, _password_hash) do
+    Logger.error("Password hash not recognized")
+    false
   end
 
   def call(%{assigns: %{user: %User{}}} = conn, _), do: conn

lib/pleroma/plugs/http_signature.ex

@@ -3,7 +3,6 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.Plugs.HTTPSignaturePlug do
-  alias Pleroma.Web.ActivityPub.Utils
   import Plug.Conn
   require Logger
 
@@ -16,12 +15,9 @@ def call(%{assigns: %{valid_signature: true}} = conn, _opts) do
   end
 
   def call(conn, _opts) do
-    user = Utils.get_ap_id(conn.params["actor"])
-    Logger.debug("Checking sig for #{user}")
     [signature | _] = get_req_header(conn, "signature")
 
-    cond do
+    if signature do
-      signature && String.contains?(signature, user) ->
       # set (request-target) header to the appropriate value
       # we also replace the digest header with the one we computed
       conn =
@@ -40,12 +36,7 @@ def call(conn, _opts) do
         end
 
       assign(conn, :valid_signature, HTTPSignatures.validate_conn(conn))
-
+    else
-      signature ->
-        Logger.debug("Signature not from actor")
-        assign(conn, :valid_signature, false)
-
-      true ->
       Logger.debug("No signature header!")
       conn
     end

lib/pleroma/plugs/mapped_signature_to_identity_plug.ex

@@ -0,0 +1,70 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.Plugs.MappedSignatureToIdentityPlug do
+  alias Pleroma.Signature
+  alias Pleroma.User
+  alias Pleroma.Web.ActivityPub.Utils
+
+  import Plug.Conn
+  require Logger
+
+  def init(options), do: options
+
+  defp key_id_from_conn(conn) do
+    with %{"keyId" => key_id} <- HTTPSignatures.signature_for_conn(conn) do
+      Signature.key_id_to_actor_id(key_id)
+    else
+      _ ->
+        nil
+    end
+  end
+
+  defp user_from_key_id(conn) do
+    with key_actor_id when is_binary(key_actor_id) <- key_id_from_conn(conn),
+         {:ok, %User{} = user} <- User.get_or_fetch_by_ap_id(key_actor_id) do
+      user
+    else
+      _ ->
+        nil
+    end
+  end
+
+  def call(%{assigns: %{user: _}} = conn, _opts), do: conn
+
+  # if this has payload make sure it is signed by the same actor that made it
+  def call(%{assigns: %{valid_signature: true}, params: %{"actor" => actor}} = conn, _opts) do
+    with actor_id <- Utils.get_ap_id(actor),
+         {:user, %User{} = user} <- {:user, user_from_key_id(conn)},
+         {:user_match, true} <- {:user_match, user.ap_id == actor_id} do
+      assign(conn, :user, user)
+    else
+      {:user_match, false} ->
+        Logger.debug("Failed to map identity from signature (payload actor mismatch)")
+        Logger.debug("key_id=#{key_id_from_conn(conn)}, actor=#{actor}")
+        assign(conn, :valid_signature, false)
+
+      # remove me once testsuite uses mapped capabilities instead of what we do now
+      {:user, nil} ->
+        Logger.debug("Failed to map identity from signature (lookup failure)")
+        Logger.debug("key_id=#{key_id_from_conn(conn)}, actor=#{actor}")
+        conn
+    end
+  end
+
+  # no payload, probably a signed fetch
+  def call(%{assigns: %{valid_signature: true}} = conn, _opts) do
+    with %User{} = user <- user_from_key_id(conn) do
+      assign(conn, :user, user)
+    else
+      _ ->
+        Logger.debug("Failed to map identity from signature (no payload actor mismatch)")
+        Logger.debug("key_id=#{key_id_from_conn(conn)}")
+        assign(conn, :valid_signature, false)
+    end
+  end
+
+  # no signature at all
+  def call(conn, _opts), do: conn
+end

lib/pleroma/plugs/rate_limiter.ex

@@ -31,12 +31,28 @@ defmodule Pleroma.Plugs.RateLimiter do
 
   ## Usage
 
+  AllowedSyntax:
+
+      plug(Pleroma.Plugs.RateLimiter, :limiter_name)
+      plug(Pleroma.Plugs.RateLimiter, {:limiter_name, options})
+
+  Allowed options:
+
+      * `bucket_name` overrides bucket name (e.g. to have a separate limit for a set of actions)
+      * `params` appends values of specified request params (e.g. ["id"]) to bucket name
+
   Inside a controller:
 
       plug(Pleroma.Plugs.RateLimiter, :one when action == :one)
       plug(Pleroma.Plugs.RateLimiter, :two when action in [:two, :three])
 
-  or inside a router pipiline:
+      plug(
+        Pleroma.Plugs.RateLimiter,
+        {:status_id_action, bucket_name: "status_id_action:fav_unfav", params: ["id"]}
+        when action in ~w(fav_status unfav_status)a
+      )
+
+  or inside a router pipeline:
 
       pipeline :api do
         ...
@@ -49,33 +65,56 @@ defmodule Pleroma.Plugs.RateLimiter do
 
   alias Pleroma.User
 
-  def init(limiter_name) do
+  def init(limiter_name) when is_atom(limiter_name) do
+    init({limiter_name, []})
+  end
+
+  def init({limiter_name, opts}) do
     case Pleroma.Config.get([:rate_limit, limiter_name]) do
       nil -> nil
-      config -> {limiter_name, config}
+      config -> {limiter_name, config, opts}
     end
   end
 
-  # do not limit if there is no limiter configuration
+  # Do not limit if there is no limiter configuration
   def call(conn, nil), do: conn
 
-  def call(conn, opts) do
+  def call(conn, settings) do
-    case check_rate(conn, opts) do
+    case check_rate(conn, settings) do
-      {:ok, _count} -> conn
+      {:ok, _count} ->
-      {:error, _count} -> render_throttled_error(conn)
+        conn
+
+      {:error, _count} ->
+        render_throttled_error(conn)
     end
   end
 
-  defp check_rate(%{assigns: %{user: %User{id: user_id}}}, {limiter_name, [_, {scale, limit}]}) do
+  defp bucket_name(conn, limiter_name, opts) do
-    ExRated.check_rate("#{limiter_name}:#{user_id}", scale, limit)
+    bucket_name = opts[:bucket_name] || limiter_name
+
+    if params_names = opts[:params] do
+      params_values = for p <- Enum.sort(params_names), do: conn.params[p]
+      Enum.join([bucket_name] ++ params_values, ":")
+    else
+      bucket_name
+    end
   end
 
-  defp check_rate(conn, {limiter_name, [{scale, limit} | _]}) do
+  defp check_rate(
-    ExRated.check_rate("#{limiter_name}:#{ip(conn)}", scale, limit)
+         %{assigns: %{user: %User{id: user_id}}} = conn,
+         {limiter_name, [_, {scale, limit}], opts}
+       ) do
+    bucket_name = bucket_name(conn, limiter_name, opts)
+    ExRated.check_rate("#{bucket_name}:#{user_id}", scale, limit)
   end
 
-  defp check_rate(conn, {limiter_name, {scale, limit}}) do
+  defp check_rate(conn, {limiter_name, [{scale, limit} | _], opts}) do
-    check_rate(conn, {limiter_name, [{scale, limit}]})
+    bucket_name = bucket_name(conn, limiter_name, opts)
+    ExRated.check_rate("#{bucket_name}:#{ip(conn)}", scale, limit)
+  end
+
+  defp check_rate(conn, {limiter_name, {scale, limit}, opts}) do
+    check_rate(conn, {limiter_name, [{scale, limit}, {scale, limit}], opts})
   end
 
   def ip(%{remote_ip: remote_ip}) do

lib/pleroma/reverse_proxy/reverse_proxy.ex

@@ -61,7 +61,7 @@ defmodule Pleroma.ReverseProxy do
   * `http`: options for [hackney](https://github.com/benoitc/hackney).
 
   """
-  @default_hackney_options []
+  @default_hackney_options [pool: :media]
 
   @inline_content_types [
     "image/gif",
@@ -94,7 +94,8 @@ def call(_conn, _url, _opts \\ [])
 
   def call(conn = %{method: method}, url, opts) when method in @methods do
     hackney_opts =
-      @default_hackney_options
+      Pleroma.HTTP.Connection.hackney_options([])
+      |> Keyword.merge(@default_hackney_options)
       |> Keyword.merge(Keyword.get(opts, :http, []))
       |> HTTP.process_request_options()
 

lib/pleroma/signature.ex

@@ -8,10 +8,16 @@ defmodule Pleroma.Signature do
   alias Pleroma.Keys
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ActivityPub
-  alias Pleroma.Web.ActivityPub.Utils
+
+  def key_id_to_actor_id(key_id) do
+    URI.parse(key_id)
+    |> Map.put(:fragment, nil)
+    |> URI.to_string()
+  end
 
   def fetch_public_key(conn) do
-    with actor_id <- Utils.get_ap_id(conn.params["actor"]),
+    with %{"keyId" => kid} <- HTTPSignatures.signature_for_conn(conn),
+         actor_id <- key_id_to_actor_id(kid),
          {:ok, public_key} <- User.get_public_key_for_ap_id(actor_id) do
       {:ok, public_key}
     else
@@ -21,7 +27,8 @@ def fetch_public_key(conn) do
   end
 
   def refetch_public_key(conn) do
-    with actor_id <- Utils.get_ap_id(conn.params["actor"]),
+    with %{"keyId" => kid} <- HTTPSignatures.signature_for_conn(conn),
+         actor_id <- key_id_to_actor_id(kid),
          {:ok, _user} <- ActivityPub.make_user_from_ap_id(actor_id),
          {:ok, public_key} <- User.get_public_key_for_ap_id(actor_id) do
       {:ok, public_key}

lib/pleroma/upload/filter/dedupe.ex

@@ -6,10 +6,19 @@ defmodule Pleroma.Upload.Filter.Dedupe do
   @behaviour Pleroma.Upload.Filter
   alias Pleroma.Upload
 
-  def filter(%Upload{name: name} = upload) do
+  def filter(%Upload{name: name, tempfile: tempfile} = upload) do
-    extension = String.split(name, ".") |> List.last()
+    extension =
-    shasum = :crypto.hash(:sha256, File.read!(upload.tempfile)) |> Base.encode16(case: :lower)
+      name
+      |> String.split(".")
+      |> List.last()
+
+    shasum =
+      :crypto.hash(:sha256, File.read!(tempfile))
+      |> Base.encode16(case: :lower)
+
     filename = shasum <> "." <> extension
     {:ok, %Upload{upload | id: shasum, path: filename}}
   end
+
+  def filter(_), do: :ok
 end

lib/pleroma/upload/filter/mogrifun.ex

@@ -4,6 +4,7 @@
 
 defmodule Pleroma.Upload.Filter.Mogrifun do
   @behaviour Pleroma.Upload.Filter
+  alias Pleroma.Upload.Filter
 
   @filters [
     {"implode", "1"},
@@ -34,31 +35,10 @@ defmodule Pleroma.Upload.Filter.Mogrifun do
   ]
 
   def filter(%Pleroma.Upload{tempfile: file, content_type: "image" <> _}) do
-    filter = Enum.random(@filters)
+    Filter.Mogrify.do_filter(file, [Enum.random(@filters)])
-
-    file
-    |> Mogrify.open()
-    |> mogrify_filter(filter)
-    |> Mogrify.save(in_place: true)
 
     :ok
   end
 
   def filter(_), do: :ok
-
-  defp mogrify_filter(mogrify, [filter | rest]) do
-    mogrify
-    |> mogrify_filter(filter)
-    |> mogrify_filter(rest)
-  end
-
-  defp mogrify_filter(mogrify, []), do: mogrify
-
-  defp mogrify_filter(mogrify, {action, options}) do
-    Mogrify.custom(mogrify, action, options)
-  end
-
-  defp mogrify_filter(mogrify, string) when is_binary(string) do
-    Mogrify.custom(mogrify, string)
-  end
 end

lib/pleroma/upload/filter/mogrify.ex

@@ -11,16 +11,19 @@ defmodule Pleroma.Upload.Filter.Mogrify do
   def filter(%Pleroma.Upload{tempfile: file, content_type: "image" <> _}) do
     filters = Pleroma.Config.get!([__MODULE__, :args])
 
-    file
+    do_filter(file, filters)
-    |> Mogrify.open()
-    |> mogrify_filter(filters)
-    |> Mogrify.save(in_place: true)
-
     :ok
   end
 
   def filter(_), do: :ok
 
+  def do_filter(file, filters) do
+    file
+    |> Mogrify.open()
+    |> mogrify_filter(filters)
+    |> Mogrify.save(in_place: true)
+  end
+
   defp mogrify_filter(mogrify, nil), do: mogrify
 
   defp mogrify_filter(mogrify, [filter | rest]) do

lib/pleroma/uploaders/uploader.ex

@@ -68,7 +68,14 @@ defp handle_callback(uploader, upload) do
             {:error, error}
         end
     after
-      30_000 -> {:error, dgettext("errors", "Uploader callback timeout")}
+      callback_timeout() -> {:error, dgettext("errors", "Uploader callback timeout")}
+    end
+  end
+
+  defp callback_timeout do
+    case Mix.env() do
+      :test -> 1_000
+      _ -> 30_000
     end
   end
 end

lib/pleroma/user.ex

@@ -707,9 +707,7 @@ def maybe_fetch_follow_information(user) do
       user
     else
       e ->
-        Logger.error(
+        Logger.error("Follower/Following counter update for #{user.ap_id} failed.\n#{inspect(e)}")
-          "Follower/Following counter update for #{user.ap_id} failed.\n#{inspect(e)}"
-        )
 
         user
     end
@@ -798,10 +796,13 @@ def get_recipients_from_activity(%Activity{recipients: to}) do
     |> Repo.all()
   end
 
-  def mute(muter, %User{ap_id: ap_id}) do
+  @spec mute(User.t(), User.t(), boolean()) :: {:ok, User.t()} | {:error, String.t()}
+  def mute(muter, %User{ap_id: ap_id}, notifications? \\ true) do
+    info = muter.info
+
     info_cng =
-      muter.info
+      User.Info.add_to_mutes(info, ap_id)
-      |> User.Info.add_to_mutes(ap_id)
+      |> User.Info.add_to_muted_notifications(info, ap_id, notifications?)
 
     cng =
       change(muter)
@@ -811,9 +812,11 @@ def mute(muter, %User{ap_id: ap_id}) do
   end
 
   def unmute(muter, %{ap_id: ap_id}) do
+    info = muter.info
+
     info_cng =
-      muter.info
+      User.Info.remove_from_mutes(info, ap_id)
-      |> User.Info.remove_from_mutes(ap_id)
+      |> User.Info.remove_from_muted_notifications(info, ap_id)
 
     cng =
       change(muter)
@@ -909,6 +912,12 @@ def unblock(blocker, %{ap_id: ap_id}) do
   def mutes?(nil, _), do: false
   def mutes?(user, %{ap_id: ap_id}), do: Enum.member?(user.info.mutes, ap_id)
 
+  @spec muted_notifications?(User.t() | nil, User.t() | map()) :: boolean()
+  def muted_notifications?(nil, _), do: false
+
+  def muted_notifications?(user, %{ap_id: ap_id}),
+    do: Enum.member?(user.info.muted_notifications, ap_id)
+
   def blocks?(%User{info: info} = _user, %{ap_id: ap_id}) do
     blocks = info.blocks
     domain_blocks = info.domain_blocks
@@ -1195,19 +1204,18 @@ def get_or_fetch_by_ap_id(ap_id) do
     end
   end
 
-  def get_or_create_instance_user do
+  @doc "Creates an internal service actor by URI if missing.  Optionally takes nickname for addressing."
-    relay_uri = "#{Pleroma.Web.Endpoint.url()}/relay"
+  def get_or_create_service_actor_by_ap_id(uri, nickname \\ nil) do
-
+    if user = get_cached_by_ap_id(uri) do
-    if user = get_cached_by_ap_id(relay_uri) do
       user
     else
       changes =
         %User{info: %User.Info{}}
         |> cast(%{}, [:ap_id, :nickname, :local])
-        |> put_change(:ap_id, relay_uri)
+        |> put_change(:ap_id, uri)
-        |> put_change(:nickname, nil)
+        |> put_change(:nickname, nickname)
         |> put_change(:local, true)
-        |> put_change(:follower_address, relay_uri <> "/followers")
+        |> put_change(:follower_address, uri <> "/followers")
 
       {:ok, user} = Repo.insert(changes)
       user
@@ -1228,10 +1236,12 @@ def public_key_from_info(%{
   end
 
   # OStatus Magic Key
-  def public_key_from_info(%{magic_key: magic_key}) do
+  def public_key_from_info(%{magic_key: magic_key}) when not is_nil(magic_key) do
     {:ok, Pleroma.Web.Salmon.decode_key(magic_key)}
   end
 
+  def public_key_from_info(_), do: {:error, "not found key"}
+
   def get_public_key_for_ap_id(ap_id) do
     with {:ok, %User{} = user} <- get_or_fetch_by_ap_id(ap_id),
          {:ok, public_key} <- public_key_from_info(user.info) do
@@ -1248,7 +1258,7 @@ def insert_or_update_user(data) do
     data
     |> Map.put(:name, blank?(data[:name]) || data[:nickname])
     |> remote_user_creation()
-    |> Repo.insert(on_conflict: :replace_all, conflict_target: :nickname)
+    |> Repo.insert(on_conflict: :replace_all_except_primary_key, conflict_target: :nickname)
     |> set_cache()
   end
 
@@ -1417,23 +1427,16 @@ def get_mascot(%{info: %{mascot: mascot}}) when is_nil(mascot) do
     }
   end
 
-  def ensure_keys_present(user) do
+  def ensure_keys_present(%User{info: info} = user) do
-    info = user.info
-
     if info.keys do
       {:ok, user}
     else
       {:ok, pem} = Keys.generate_rsa_pem()
 
-      info_cng =
+      user
-        info
+      |> Ecto.Changeset.change()
-        |> User.Info.set_keys(pem)
+      |> Ecto.Changeset.put_embed(:info, User.Info.set_keys(info, pem))
-
+      |> update_and_set_cache()
-      cng =
-        Ecto.Changeset.change(user)
-        |> Ecto.Changeset.put_embed(:info, info_cng)
-
-      update_and_set_cache(cng)
     end
   end
 
@@ -1454,4 +1457,8 @@ defp put_password_hash(
   end
 
   defp put_password_hash(changeset), do: changeset
+
+  def is_internal_user?(%User{nickname: nil}), do: true
+  def is_internal_user?(%User{local: true, nickname: "internal." <> _}), do: true
+  def is_internal_user?(_), do: false
 end

lib/pleroma/user/info.ex

@@ -26,6 +26,7 @@ defmodule Pleroma.User.Info do
     field(:domain_blocks, {:array, :string}, default: [])
     field(:mutes, {:array, :string}, default: [])
     field(:muted_reblogs, {:array, :string}, default: [])
+    field(:muted_notifications, {:array, :string}, default: [])
     field(:subscribers, {:array, :string}, default: [])
     field(:deactivated, :boolean, default: false)
     field(:no_rich_text, :boolean, default: false)
@@ -122,6 +123,16 @@ def set_mutes(info, mutes) do
     |> validate_required([:mutes])
   end
 
+  @spec set_notification_mutes(Changeset.t(), [String.t()], boolean()) :: Changeset.t()
+  def set_notification_mutes(changeset, muted_notifications, notifications?) do
+    if notifications? do
+      put_change(changeset, :muted_notifications, muted_notifications)
+      |> validate_required([:muted_notifications])
+    else
+      changeset
+    end
+  end
+
   def set_blocks(info, blocks) do
     params = %{blocks: blocks}
 
@@ -138,14 +149,31 @@ def set_subscribers(info, subscribers) do
     |> validate_required([:subscribers])
   end
 
+  @spec add_to_mutes(Info.t(), String.t()) :: Changeset.t()
   def add_to_mutes(info, muted) do
     set_mutes(info, Enum.uniq([muted | info.mutes]))
   end
 
+  @spec add_to_muted_notifications(Changeset.t(), Info.t(), String.t(), boolean()) ::
+          Changeset.t()
+  def add_to_muted_notifications(changeset, info, muted, notifications?) do
+    set_notification_mutes(
+      changeset,
+      Enum.uniq([muted | info.muted_notifications]),
+      notifications?
+    )
+  end
+
+  @spec remove_from_mutes(Info.t(), String.t()) :: Changeset.t()
   def remove_from_mutes(info, muted) do
     set_mutes(info, List.delete(info.mutes, muted))
   end
 
+  @spec remove_from_muted_notifications(Changeset.t(), Info.t(), String.t()) :: Changeset.t()
+  def remove_from_muted_notifications(changeset, info, muted) do
+    set_notification_mutes(changeset, List.delete(info.muted_notifications, muted), true)
+  end
+
   def add_to_block(info, blocked) do
     set_blocks(info, Enum.uniq([blocked | info.blocks]))
   end

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -8,6 +8,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   alias Pleroma.Conversation
   alias Pleroma.Notification
   alias Pleroma.Object
+  alias Pleroma.Object.Containment
   alias Pleroma.Object.Fetcher
   alias Pleroma.Pagination
   alias Pleroma.Repo
@@ -26,19 +27,16 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   # For Announce activities, we filter the recipients based on following status for any actors
   # that match actual users.  See issue #164 for more information about why this is necessary.
   defp get_recipients(%{"type" => "Announce"} = data) do
-    to = data["to"] || []
+    to = Map.get(data, "to", [])
-    cc = data["cc"] || []
+    cc = Map.get(data, "cc", [])
+    bcc = Map.get(data, "bcc", [])
     actor = User.get_cached_by_ap_id(data["actor"])
 
     recipients =
-      (to ++ cc)
+      Enum.filter(Enum.concat([to, cc, bcc]), fn recipient ->
-      |> Enum.filter(fn recipient ->
         case User.get_cached_by_ap_id(recipient) do
-          nil ->
+          nil -> true
-            true
+          user -> User.following?(user, actor)
-
-          user ->
-            User.following?(user, actor)
         end
       end)
 
@@ -46,17 +44,19 @@ defp get_recipients(%{"type" => "Announce"} = data) do
   end
 
   defp get_recipients(%{"type" => "Create"} = data) do
-    to = data["to"] || []
+    to = Map.get(data, "to", [])
-    cc = data["cc"] || []
+    cc = Map.get(data, "cc", [])
-    actor = data["actor"] || []
+    bcc = Map.get(data, "bcc", [])
-    recipients = (to ++ cc ++ [actor]) |> Enum.uniq()
+    actor = Map.get(data, "actor", [])
+    recipients = [to, cc, bcc, [actor]] |> Enum.concat() |> Enum.uniq()
     {recipients, to, cc}
   end
 
   defp get_recipients(data) do
-    to = data["to"] || []
+    to = Map.get(data, "to", [])
-    cc = data["cc"] || []
+    cc = Map.get(data, "cc", [])
-    recipients = to ++ cc
+    bcc = Map.get(data, "bcc", [])
+    recipients = Enum.concat([to, cc, bcc])
     {recipients, to, cc}
   end
 
@@ -126,6 +126,7 @@ def insert(map, local \\ true, fake \\ false) when is_map(map) do
          {:ok, map} <- MRF.filter(map),
          {recipients, _, _} = get_recipients(map),
          {:fake, false, map, recipients} <- {:fake, fake, map, recipients},
+         :ok <- Containment.contain_child(map),
          {:ok, map, object} <- insert_full_object(map) do
       {:ok, activity} =
         Repo.insert(%Activity{
@@ -896,13 +897,11 @@ defp maybe_order(query, %{order: :asc}) do
   defp maybe_order(query, _), do: query
 
   def fetch_activities_query(recipients, opts \\ %{}) do
-    base_query = from(activity in Activity)
-
     config = %{
       skip_thread_containment: Config.get([:instance, :skip_thread_containment])
     }
 
-    base_query
+    Activity
     |> maybe_preload_objects(opts)
     |> maybe_preload_bookmarks(opts)
     |> maybe_set_thread_muted_field(opts)
@@ -931,11 +930,31 @@ def fetch_activities_query(recipients, opts \\ %{}) do
   end
 
   def fetch_activities(recipients, opts \\ %{}) do
-    fetch_activities_query(recipients, opts)
+    list_memberships = Pleroma.List.memberships(opts["user"])
+
+    fetch_activities_query(recipients ++ list_memberships, opts)
     |> Pagination.fetch_paginated(opts)
     |> Enum.reverse()
+    |> maybe_update_cc(list_memberships, opts["user"])
   end
 
+  defp maybe_update_cc(activities, list_memberships, %User{ap_id: user_ap_id})
+       when is_list(list_memberships) and length(list_memberships) > 0 do
+    Enum.map(activities, fn
+      %{data: %{"bcc" => bcc}} = activity when is_list(bcc) and length(bcc) > 0 ->
+        if Enum.any?(bcc, &(&1 in list_memberships)) do
+          update_in(activity.data["cc"], &[user_ap_id | &1])
+        else
+          activity
+        end
+
+      activity ->
+        activity
+    end)
+  end
+
+  defp maybe_update_cc(activities, _, _), do: activities
+
   def fetch_activities_bounded_query(query, recipients, recipients_with_public) do
     from(activity in query,
       where:

lib/pleroma/web/activity_pub/activity_pub_controller.ex

@@ -10,6 +10,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
   alias Pleroma.Object.Fetcher
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ActivityPub
+  alias Pleroma.Web.ActivityPub.InternalFetchActor
   alias Pleroma.Web.ActivityPub.ObjectView
   alias Pleroma.Web.ActivityPub.Relay
   alias Pleroma.Web.ActivityPub.Transmogrifier
@@ -206,9 +207,8 @@ def inbox(conn, params) do
     json(conn, dgettext("errors", "error"))
   end
 
-  def relay(conn, _params) do
+  defp represent_service_actor(%User{} = user, conn) do
-    with %User{} = user <- Relay.get_actor(),
+    with {:ok, user} <- User.ensure_keys_present(user) do
-         {:ok, user} <- User.ensure_keys_present(user) do
       conn
       |> put_resp_header("content-type", "application/activity+json")
       |> json(UserView.render("user.json", %{user: user}))
@@ -217,6 +217,18 @@ def relay(conn, _params) do
     end
   end
 
+  defp represent_service_actor(nil, _), do: {:error, :not_found}
+
+  def relay(conn, _params) do
+    Relay.get_actor()
+    |> represent_service_actor(conn)
+  end
+
+  def internal_fetch(conn, _params) do
+    InternalFetchActor.get_actor()
+    |> represent_service_actor(conn)
+  end
+
   def whoami(%{assigns: %{user: %User{} = user}} = conn, _params) do
     conn
     |> put_resp_header("content-type", "application/activity+json")

lib/pleroma/web/activity_pub/internal_fetch_actor.ex

@@ -0,0 +1,20 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.InternalFetchActor do
+  alias Pleroma.User
+
+  require Logger
+
+  def init do
+    # Wait for everything to settle.
+    Process.sleep(1000 * 5)
+    get_actor()
+  end
+
+  def get_actor do
+    "#{Pleroma.Web.Endpoint.url()}/internal/fetch"
+    |> User.get_or_create_service_actor_by_ap_id("internal.fetch")
+  end
+end

lib/pleroma/web/activity_pub/mrf/mention_policy.ex

@@ -0,0 +1,24 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.MRF.MentionPolicy do
+  @moduledoc "Block messages which mention a user"
+
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
+  @impl true
+  def filter(%{"type" => "Create"} = message) do
+    reject_actors = Pleroma.Config.get([:mrf_mention, :actors], [])
+    recipients = (message["to"] || []) ++ (message["cc"] || [])
+
+    if Enum.any?(recipients, fn recipient -> Enum.member?(reject_actors, recipient) end) do
+      {:reject, nil}
+    else
+      {:ok, message}
+    end
+  end
+
+  @impl true
+  def filter(message), do: {:ok, message}
+end

lib/pleroma/web/activity_pub/publisher.ex

@@ -92,18 +92,68 @@ defp should_federate?(inbox, public) do
     end
   end
 
-  @doc """
+  defp recipients(actor, activity) do
-  Publishes an activity to all relevant peers.
+    followers =
-  """
-  def publish(%User{} = actor, %Activity{} = activity) do
-    remote_followers =
       if actor.follower_address in activity.recipients do
         {:ok, followers} = User.get_followers(actor)
-        followers |> Enum.filter(&(!&1.local))
+        Enum.filter(followers, &(!&1.local))
       else
         []
       end
 
+    Pleroma.Web.Salmon.remote_users(actor, activity) ++ followers
+  end
+
+  defp get_cc_ap_ids(ap_id, recipients) do
+    host = Map.get(URI.parse(ap_id), :host)
+
+    recipients
+    |> Enum.filter(fn %User{ap_id: ap_id} -> Map.get(URI.parse(ap_id), :host) == host end)
+    |> Enum.map(& &1.ap_id)
+  end
+
+  @doc """
+  Publishes an activity with BCC to all relevant peers.
+  """
+
+  def publish(actor, %{data: %{"bcc" => bcc}} = activity) when is_list(bcc) and bcc != [] do
+    public = is_public?(activity)
+    {:ok, data} = Transmogrifier.prepare_outgoing(activity.data)
+
+    recipients = recipients(actor, activity)
+
+    recipients
+    |> Enum.filter(&User.ap_enabled?/1)
+    |> Enum.map(fn %{info: %{source_data: data}} -> data["inbox"] end)
+    |> Enum.filter(fn inbox -> should_federate?(inbox, public) end)
+    |> Instances.filter_reachable()
+    |> Enum.each(fn {inbox, unreachable_since} ->
+      %User{ap_id: ap_id} =
+        Enum.find(recipients, fn %{info: %{source_data: data}} -> data["inbox"] == inbox end)
+
+      # Get all the recipients on the same host and add them to cc. Otherwise, a remote
+      # instance would only accept a first message for the first recipient and ignore the rest.
+      cc = get_cc_ap_ids(ap_id, recipients)
+
+      json =
+        data
+        |> Map.put("cc", cc)
+        |> Jason.encode!()
+
+      Pleroma.Web.Federator.Publisher.enqueue_one(__MODULE__, %{
+        inbox: inbox,
+        json: json,
+        actor: actor,
+        id: activity.data["id"],
+        unreachable_since: unreachable_since
+      })
+    end)
+  end
+
+  @doc """
+  Publishes an activity to all relevant peers.
+  """
+  def publish(%User{} = actor, %Activity{} = activity) do
     public = is_public?(activity)
 
     if public && Config.get([:instance, :allow_relay]) do
@@ -114,7 +164,7 @@ def publish(%User{} = actor, %Activity{} = activity) do
     {:ok, data} = Transmogrifier.prepare_outgoing(activity.data)
     json = Jason.encode!(data)
 
-    (Pleroma.Web.Salmon.remote_users(activity) ++ remote_followers)
+    recipients(actor, activity)
     |> Enum.filter(fn user -> User.ap_enabled?(user) end)
     |> Enum.map(fn %{info: %{source_data: data}} ->
       (is_map(data["endpoints"]) && Map.get(data["endpoints"], "sharedInbox")) || data["inbox"]

lib/pleroma/web/activity_pub/relay.ex

@@ -10,7 +10,8 @@ defmodule Pleroma.Web.ActivityPub.Relay do
   require Logger
 
   def get_actor do
-    User.get_or_create_instance_user()
+    "#{Pleroma.Web.Endpoint.url()}/relay"
+    |> User.get_or_create_service_actor_by_ap_id()
   end
 
   def follow(target_instance) do

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -814,13 +814,16 @@ def prepare_object(object) do
 
   def prepare_outgoing(%{"type" => "Create", "object" => object_id} = data) do
     object =
-      Object.normalize(object_id).data
+      object_id
+      |> Object.normalize()
+      |> Map.get(:data)
       |> prepare_object
 
     data =
       data
       |> Map.put("object", object)
       |> Map.merge(Utils.make_json_ld_header())
+      |> Map.delete("bcc")
 
     {:ok, data}
   end

lib/pleroma/web/activity_pub/utils.ex

@@ -25,12 +25,8 @@ defmodule Pleroma.Web.ActivityPub.Utils do
 
   # Some implementations send the actor URI as the actor field, others send the entire actor object,
   # so figure out what the actor's URI is based on what we have.
-  def get_ap_id(object) do
+  def get_ap_id(%{"id" => id} = _), do: id
-    case object do
+  def get_ap_id(id), do: id
-      %{"id" => id} -> id
-      id -> id
-    end
-  end
 
   def normalize_params(params) do
     Map.put(params, "actor", get_ap_id(params["actor"]))

lib/pleroma/web/activity_pub/views/user_view.ex

@@ -31,8 +31,7 @@ def render("endpoints.json", %{user: %User{local: true} = _user}) do
 
   def render("endpoints.json", _), do: %{}
 
-  # the instance itself is not a Person, but instead an Application
+  def render("service.json", %{user: user}) do
-  def render("user.json", %{user: %{nickname: nil} = user}) do
     {:ok, user} = User.ensure_keys_present(user)
     {:ok, _, public_key} = Keys.keys_from_pem(user.info.keys)
     public_key = :public_key.pem_entry_encode(:SubjectPublicKeyInfo, public_key)
@@ -47,7 +46,8 @@ def render("user.json", %{user: %{nickname: nil} = user}) do
       "followers" => "#{user.ap_id}/followers",
       "inbox" => "#{user.ap_id}/inbox",
       "name" => "Pleroma",
-      "summary" => "Virtual actor for Pleroma relay",
+      "summary" =>
+        "An internal service actor for this Pleroma instance.  No user-serviceable parts inside.",
       "url" => user.ap_id,
       "manuallyApprovesFollowers" => false,
       "publicKey" => %{
@@ -60,6 +60,13 @@ def render("user.json", %{user: %{nickname: nil} = user}) do
     |> Map.merge(Utils.make_json_ld_header())
   end
 
+  # the instance itself is not a Person, but instead an Application
+  def render("user.json", %{user: %User{nickname: nil} = user}),
+    do: render("service.json", %{user: user})
+
+  def render("user.json", %{user: %User{nickname: "internal." <> _} = user}),
+    do: render("service.json", %{user: user})
+
   def render("user.json", %{user: user}) do
     {:ok, user} = User.ensure_keys_present(user)
     {:ok, _, public_key} = Keys.keys_from_pem(user.info.keys)

lib/pleroma/web/activity_pub/visibility.ex

@@ -34,6 +34,20 @@ def is_direct?(activity) do
     !is_public?(activity) && !is_private?(activity)
   end
 
+  def is_list?(%{data: %{"listMessage" => _}}), do: true
+  def is_list?(_), do: false
+
+  def visible_for_user?(%{actor: ap_id}, %User{ap_id: ap_id}), do: true
+
+  def visible_for_user?(%{data: %{"listMessage" => list_ap_id}} = activity, %User{} = user) do
+    user.ap_id in activity.data["to"] ||
+      list_ap_id
+      |> Pleroma.List.get_by_ap_id()
+      |> Pleroma.List.member?(user)
+  end
+
+  def visible_for_user?(%{data: %{"listMessage" => _}}, nil), do: false
+
   def visible_for_user?(activity, nil) do
     is_public?(activity)
   end
@@ -73,6 +87,9 @@ def get_visibility(object) do
       object.data["directMessage"] == true ->
         "direct"
 
+      is_binary(object.data["listMessage"]) ->
+        "list"
+
       length(cc) > 0 ->
         "private"
 

lib/pleroma/web/auth/pleroma_authenticator.ex

@@ -3,7 +3,7 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.Auth.PleromaAuthenticator do
-  alias Comeonin.Pbkdf2
+  alias Pleroma.Plugs.AuthenticationPlug
   alias Pleroma.Registration
   alias Pleroma.Repo
   alias Pleroma.User
@@ -16,7 +16,7 @@ defmodule Pleroma.Web.Auth.PleromaAuthenticator do
   def get_user(%Plug.Conn{} = conn) do
     with {:ok, {name, password}} <- fetch_credentials(conn),
          {_, %User{} = user} <- {:user, fetch_user(name)},
-         {_, true} <- {:checkpw, Pbkdf2.checkpw(password, user.password_hash)} do
+         {_, true} <- {:checkpw, AuthenticationPlug.checkpw(password, user.password_hash)} do
       {:ok, user}
     else
       error ->

lib/pleroma/web/common_api/common_api.ex

@@ -4,7 +4,6 @@
 
 defmodule Pleroma.Web.CommonAPI do
   alias Pleroma.Activity
-  alias Pleroma.Bookmark
   alias Pleroma.Formatter
   alias Pleroma.Object
   alias Pleroma.ThreadMute
@@ -31,7 +30,8 @@ def follow(follower, followed) do
 
   def unfollow(follower, unfollowed) do
     with {:ok, follower, _follow_activity} <- User.unfollow(follower, unfollowed),
-         {:ok, _activity} <- ActivityPub.unfollow(follower, unfollowed) do
+         {:ok, _activity} <- ActivityPub.unfollow(follower, unfollowed),
+         {:ok, _unfollowed} <- User.unsubscribe(follower, unfollowed) do
       {:ok, follower}
     end
   end
@@ -175,6 +175,11 @@ def get_visibility(%{"visibility" => visibility}, in_reply_to)
       when visibility in ~w{public unlisted private direct},
       do: {visibility, get_replied_to_visibility(in_reply_to)}
 
+  def get_visibility(%{"visibility" => "list:" <> list_id}, in_reply_to) do
+    visibility = {:list, String.to_integer(list_id)}
+    {visibility, get_replied_to_visibility(in_reply_to)}
+  end
+
   def get_visibility(_, in_reply_to) when not is_nil(in_reply_to) do
     visibility = get_replied_to_visibility(in_reply_to)
     {visibility, visibility}
@@ -235,19 +240,18 @@ def post(user, %{"status" => status} = data) do
              "emoji",
              Map.merge(Formatter.get_emoji_map(full_payload), poll_emoji)
            ) do
-      res =
+      preview? = Pleroma.Web.ControllerHelper.truthy_param?(data["preview"]) || false
-        ActivityPub.create(
+      direct? = visibility == "direct"
+
       %{
         to: to,
         actor: user,
         context: context,
         object: object,
-            additional: %{"cc" => cc, "directMessage" => visibility == "direct"}
+        additional: %{"cc" => cc, "directMessage" => direct?}
-          },
+      }
-          Pleroma.Web.ControllerHelper.truthy_param?(data["preview"]) || false
+      |> maybe_add_list_data(user, visibility)
-        )
+      |> ActivityPub.create(preview?)
-
-      res
     else
       {:private_to_public, true} ->
         {:error, dgettext("errors", "The message visibility must be direct")}
@@ -351,15 +355,6 @@ def thread_muted?(user, activity) do
     end
   end
 
-  def bookmarked?(user, activity) do
-    with %Bookmark{} <- Bookmark.get(user.id, activity.id) do
-      true
-    else
-      _ ->
-        false
-    end
-  end
-
   def report(user, data) do
     with {:account_id, %{"account_id" => account_id}} <- {:account_id, data},
          {:account, %User{} = account} <- {:account, User.get_cached_by_id(account_id)},

lib/pleroma/web/common_api/utils.ex

@@ -6,11 +6,11 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   import Pleroma.Web.Gettext
 
   alias Calendar.Strftime
-  alias Comeonin.Pbkdf2
   alias Pleroma.Activity
   alias Pleroma.Config
   alias Pleroma.Formatter
   alias Pleroma.Object
+  alias Pleroma.Plugs.AuthenticationPlug
   alias Pleroma.Repo
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.Utils
@@ -100,12 +100,29 @@ def get_to_and_cc(_user, mentioned_users, inReplyTo, "direct") do
     end
   end
 
+  def get_to_and_cc(_user, mentions, _inReplyTo, {:list, _}), do: {mentions, []}
+
   def get_addressed_users(_, to) when is_list(to) do
     User.get_ap_ids_by_nicknames(to)
   end
 
   def get_addressed_users(mentioned_users, _), do: mentioned_users
 
+  def maybe_add_list_data(activity_params, user, {:list, list_id}) do
+    case Pleroma.List.get(list_id, user) do
+      %Pleroma.List{} = list ->
+        activity_params
+        |> put_in([:additional, "bcc"], [list.ap_id])
+        |> put_in([:additional, "listMessage"], list.ap_id)
+        |> put_in([:object, "listMessage"], list.ap_id)
+
+      _ ->
+        activity_params
+    end
+  end
+
+  def maybe_add_list_data(activity_params, _, _), do: activity_params
+
   def make_poll_data(%{"poll" => %{"options" => options, "expires_in" => expires_in}} = data)
       when is_list(options) do
     %{max_expiration: max_expiration, min_expiration: min_expiration} =
@@ -371,7 +388,7 @@ defp shortname(name) do
 
   def confirm_current_password(user, password) do
     with %User{local: true} = db_user <- User.get_cached_by_id(user.id),
-         true <- Pbkdf2.checkpw(password, db_user.password_hash) do
+         true <- AuthenticationPlug.checkpw(password, db_user.password_hash) do
       {:ok, db_user}
     else
       _ -> {:error, dgettext("errors", "Invalid password.")}

lib/pleroma/web/mastodon_api/mastodon_api.ex

@@ -53,7 +53,7 @@ def get_notifications(user, params \\ %{}) do
     options = cast_params(params)
 
     user
-    |> Notification.for_user_query()
+    |> Notification.for_user_query(options)
     |> restrict(:exclude_types, options)
     |> Pagination.fetch_paginated(params)
   end
@@ -67,7 +67,8 @@ def get_scheduled_activities(user, params \\ %{}) do
   defp cast_params(params) do
     param_types = %{
       exclude_types: {:array, :string},
-      reblogs: :boolean
+      reblogs: :boolean,
+      with_muted: :boolean
     }
 
     changeset = cast({%{}, param_types}, params, Map.keys(param_types))

lib/pleroma/web/mastodon_api/mastodon_api_controller.ex

@@ -15,6 +15,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   alias Pleroma.Notification
   alias Pleroma.Object
   alias Pleroma.Pagination
+  alias Pleroma.Plugs.RateLimiter
   alias Pleroma.Repo
   alias Pleroma.ScheduledActivity
   alias Pleroma.Stats
@@ -46,8 +47,33 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
 
   require Logger
 
-  plug(Pleroma.Plugs.RateLimiter, :app_account_creation when action == :account_register)
+  @rate_limited_relations_actions ~w(follow unfollow)a
-  plug(Pleroma.Plugs.RateLimiter, :search when action in [:search, :search2, :account_search])
+
+  @rate_limited_status_actions ~w(reblog_status unreblog_status fav_status unfav_status
+    post_status delete_status)a
+
+  plug(
+    RateLimiter,
+    {:status_id_action, bucket_name: "status_id_action:reblog_unreblog", params: ["id"]}
+    when action in ~w(reblog_status unreblog_status)a
+  )
+
+  plug(
+    RateLimiter,
+    {:status_id_action, bucket_name: "status_id_action:fav_unfav", params: ["id"]}
+    when action in ~w(fav_status unfav_status)a
+  )
+
+  plug(
+    RateLimiter,
+    {:relations_id_action, params: ["id", "uri"]} when action in @rate_limited_relations_actions
+  )
+
+  plug(RateLimiter, :relations_actions when action in @rate_limited_relations_actions)
+  plug(RateLimiter, :statuses_actions when action in @rate_limited_status_actions)
+  plug(RateLimiter, :app_account_creation when action == :account_register)
+  plug(RateLimiter, :search when action in [:search, :search2, :account_search])
+  plug(RateLimiter, :password_reset when action == :password_reset)
 
   @local_mastodon_name "Mastodon-Local"
 
@@ -414,7 +440,7 @@ def public_timeline(%{assigns: %{user: user}} = conn, params) do
   end
 
   def user_statuses(%{assigns: %{user: reading_user}} = conn, params) do
-    with %User{} = user <- User.get_cached_by_id(params["id"]) do
+    with %User{} = user <- User.get_cached_by_nickname_or_id(params["id"]) do
       params =
         params
         |> Map.put("tag", params["tagged"])
@@ -676,11 +702,6 @@ def pin_status(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
       conn
       |> put_view(StatusView)
       |> try_render("status.json", %{activity: activity, for: user, as: :activity})
-    else
-      {:error, reason} ->
-        conn
-        |> put_status(:bad_request)
-        |> json(%{"error" => reason})
     end
   end
 
@@ -721,11 +742,6 @@ def mute_conversation(%{assigns: %{user: user}} = conn, %{"id" => id}) do
       conn
       |> put_view(StatusView)
       |> try_render("status.json", %{activity: activity, for: user, as: :activity})
-    else
-      {:error, reason} ->
-        conn
-        |> put_resp_content_type("application/json")
-        |> send_resp(:bad_request, Jason.encode!(%{"error" => reason}))
     end
   end
 
@@ -864,7 +880,7 @@ def get_mascot(%{assigns: %{user: user}} = conn, _params) do
   end
 
   def favourited_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
-    with %Activity{data: %{"object" => object}} <- Repo.get(Activity, id),
+    with %Activity{data: %{"object" => object}} <- Activity.get_by_id(id),
          %Object{data: %{"likes" => likes}} <- Object.normalize(object) do
       q = from(u in User, where: u.ap_id in ^likes)
       users = Repo.all(q)
@@ -878,7 +894,7 @@ def favourited_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
   end
 
   def reblogged_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
-    with %Activity{data: %{"object" => object}} <- Repo.get(Activity, id),
+    with %Activity{data: %{"object" => object}} <- Activity.get_by_id(id),
          %Object{data: %{"announcements" => announces}} <- Object.normalize(object) do
       q = from(u in User, where: u.ap_id in ^announces)
       users = Repo.all(q)
@@ -1051,9 +1067,14 @@ def unfollow(%{assigns: %{user: follower}} = conn, %{"id" => id}) do
     end
   end
 
-  def mute(%{assigns: %{user: muter}} = conn, %{"id" => id}) do
+  def mute(%{assigns: %{user: muter}} = conn, %{"id" => id} = params) do
+    notifications =
+      if Map.has_key?(params, "notifications"),
+        do: params["notifications"] in [true, "True", "true", "1"],
+        else: true
+
     with %User{} = muted <- User.get_cached_by_id(id),
-         {:ok, muter} <- User.mute(muter, muted) do
+         {:ok, muter} <- User.mute(muter, muted, notifications) do
       conn
       |> put_view(AccountView)
       |> render("relationship.json", %{user: muter, target: muted})
@@ -1629,6 +1650,12 @@ def errors(conn, {:error, :not_found}) do
     render_error(conn, :not_found, "Record not found")
   end
 
+  def errors(conn, {:error, error_message}) do
+    conn
+    |> put_status(:bad_request)
+    |> json(%{error: error_message})
+  end
+
   def errors(conn, _) do
     conn
     |> put_status(:internal_server_error)
@@ -1790,6 +1817,22 @@ def conversation_read(%{assigns: %{user: user}} = conn, %{"id" => participation_
     end
   end
 
+  def password_reset(conn, params) do
+    nickname_or_email = params["email"] || params["nickname"]
+
+    with {:ok, _} <- TwitterAPI.password_reset(nickname_or_email) do
+      conn
+      |> put_status(:no_content)
+      |> json("")
+    else
+      {:error, "unknown user"} ->
+        send_resp(conn, :not_found, "")
+
+      {:error, _} ->
+        send_resp(conn, :bad_request, "")
+    end
+  end
+
   def try_render(conn, target, params)
       when is_binary(target) do
     case render(conn, target, params) do

lib/pleroma/web/mastodon_api/views/account_view.ex

@@ -51,8 +51,9 @@ def render("relationship.json", %{user: %User{} = user, target: %User{} = target
       following: User.following?(user, target),
       followed_by: User.following?(target, user),
       blocking: User.blocks?(user, target),
+      blocked_by: User.blocks?(target, user),
       muting: User.mutes?(user, target),
-      muting_notifications: false,
+      muting_notifications: User.muted_notifications?(user, target),
       subscribing: User.subscribed_to?(user, target),
       requested: requested,
       domain_blocking: false,
@@ -136,6 +137,7 @@ defp do_render("account.json", %{user: user} = opts) do
     |> maybe_put_notification_settings(user, opts[:for])
     |> maybe_put_settings_store(user, opts[:for], opts)
     |> maybe_put_chat_token(user, opts[:for], opts)
+    |> maybe_put_activation_status(user, opts[:for])
   end
 
   defp username_from_nickname(string) when is_binary(string) do
@@ -196,6 +198,12 @@ defp maybe_put_notification_settings(data, %User{id: user_id} = user, %User{id:
 
   defp maybe_put_notification_settings(data, _, _), do: data
 
+  defp maybe_put_activation_status(data, user, %User{info: %{is_admin: true}}) do
+    Kernel.put_in(data, [:pleroma, :deactivated], user.info.deactivated)
+  end
+
+  defp maybe_put_activation_status(data, _, _), do: data
+
   defp image_url(%{"url" => [%{"href" => href} | _]}), do: href
   defp image_url(_), do: nil
 end

lib/pleroma/web/mastodon_api/views/status_view.ex

@@ -382,7 +382,7 @@ def render("poll.json", %{object: object} = opts) do
       %{
         # Mastodon uses separate ids for polls, but an object can't have
         # more than one poll embedded so object id is fine
-        id: object.id,
+        id: to_string(object.id),
         expires_at: Utils.to_masto_date(end_time),
         expired: expired,
         multiple: multiple,

lib/pleroma/web/media_proxy/media_proxy.ex

@@ -3,68 +3,71 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.MediaProxy do
+  alias Pleroma.Config
+  alias Pleroma.Web
+
   @base64_opts [padding: false]
 
-  def url(nil), do: nil
+  def url(url) when is_nil(url) or url == "", do: nil
-
-  def url(""), do: nil
-
   def url("/" <> _ = url), do: url
 
   def url(url) do
-    if !enabled?() or local?(url) or whitelisted?(url) do
+    if disabled?() or local?(url) or whitelisted?(url) do
       url
     else
       encode_url(url)
     end
   end
 
-  defp enabled?, do: Pleroma.Config.get([:media_proxy, :enabled], false)
+  defp disabled?, do: !Config.get([:media_proxy, :enabled], false)
 
   defp local?(url), do: String.starts_with?(url, Pleroma.Web.base_url())
 
   defp whitelisted?(url) do
     %{host: domain} = URI.parse(url)
 
-    Enum.any?(Pleroma.Config.get([:media_proxy, :whitelist]), fn pattern ->
+    Enum.any?(Config.get([:media_proxy, :whitelist]), fn pattern ->
       String.equivalent?(domain, pattern)
     end)
   end
 
   def encode_url(url) do
-    secret = Pleroma.Config.get([Pleroma.Web.Endpoint, :secret_key_base])
     base64 = Base.url_encode64(url, @base64_opts)
-    sig = :crypto.hmac(:sha, secret, base64)
+
-    sig64 = sig |> Base.url_encode64(@base64_opts)
+    sig64 =
+      base64
+      |> signed_url
+      |> Base.url_encode64(@base64_opts)
 
     build_url(sig64, base64, filename(url))
   end
 
   def decode_url(sig, url) do
-    secret = Pleroma.Config.get([Pleroma.Web.Endpoint, :secret_key_base])
+    with {:ok, sig} <- Base.url_decode64(sig, @base64_opts),
-    sig = Base.url_decode64!(sig, @base64_opts)
+         signature when signature == sig <- signed_url(url) do
-    local_sig = :crypto.hmac(:sha, secret, url)
-
-    if local_sig == sig do
       {:ok, Base.url_decode64!(url, @base64_opts)}
     else
-      {:error, :invalid_signature}
+      _ -> {:error, :invalid_signature}
     end
   end
 
+  defp signed_url(url) do
+    :crypto.hmac(:sha, Config.get([Web.Endpoint, :secret_key_base]), url)
+  end
+
   def filename(url_or_path) do
     if path = URI.parse(url_or_path).path, do: Path.basename(path)
   end
 
   def build_url(sig_base64, url_base64, filename \\ nil) do
     [
-      Pleroma.Config.get([:media_proxy, :base_url], Pleroma.Web.base_url()),
+      Pleroma.Config.get([:media_proxy, :base_url], Web.base_url()),
       "proxy",
       sig_base64,
       url_base64,
       filename
     ]
-    |> Enum.filter(fn value -> value end)
+    |> Enum.filter(& &1)
     |> Path.join()
   end
 end

lib/pleroma/web/media_proxy/media_proxy_controller.ex

@@ -13,7 +13,7 @@ def remote(conn, %{"sig" => sig64, "url" => url64} = params) do
     with config <- Pleroma.Config.get([:media_proxy], []),
          true <- Keyword.get(config, :enabled, false),
          {:ok, url} <- MediaProxy.decode_url(sig64, url64),
-         :ok <- filename_matches(Map.has_key?(params, "filename"), conn.request_path, url) do
+         :ok <- filename_matches(params, conn.request_path, url) do
       ReverseProxy.call(conn, url, Keyword.get(config, :proxy_opts, @default_proxy_opts))
     else
       false ->
@@ -27,13 +27,20 @@ def remote(conn, %{"sig" => sig64, "url" => url64} = params) do
     end
   end
 
-  def filename_matches(has_filename, path, url) do
+  def filename_matches(%{"filename" => _} = _, path, url) do
-    filename = url |> MediaProxy.filename()
+    filename = MediaProxy.filename(url)
 
-    if has_filename && filename && Path.basename(path) != filename do
+    if filename && does_not_match(path, filename) do
       {:wrong_filename, filename}
     else
       :ok
     end
   end
+
+  def filename_matches(_, _, _), do: :ok
+
+  defp does_not_match(path, filename) do
+    basename = Path.basename(path)
+    basename != filename and URI.decode(basename) != filename and URI.encode(basename) != filename
+  end
 end

lib/pleroma/web/nodeinfo/nodeinfo_controller.ex

@@ -34,8 +34,11 @@ def schemas(conn, _params) do
   def raw_nodeinfo do
     stats = Stats.get_stats()
 
+    exclusions = Config.get([:instance, :mrf_transparency_exclusions])
+
     mrf_simple =
       Config.get(:mrf_simple)
+      |> Enum.map(fn {k, v} -> {k, Enum.reject(v, fn v -> v in exclusions end)} end)
       |> Enum.into(%{})
 
     # This horror is needed to convert regex sigils to strings
@@ -86,7 +89,8 @@ def raw_nodeinfo do
           mrf_simple: mrf_simple,
           mrf_keyword: mrf_keyword,
           mrf_user_allowlist: mrf_user_allowlist,
-          quarantined_instances: quarantined
+          quarantined_instances: quarantined,
+          exclusions: length(exclusions) > 0
         }
       else
         %{}

lib/pleroma/web/rich_media/parser.ex

@@ -3,12 +3,6 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.RichMedia.Parser do
-  @parsers [
-    Pleroma.Web.RichMedia.Parsers.OGP,
-    Pleroma.Web.RichMedia.Parsers.TwitterCard,
-    Pleroma.Web.RichMedia.Parsers.OEmbed
-  ]
-
   @hackney_options [
     pool: :media,
     recv_timeout: 2_000,
@@ -16,6 +10,10 @@ defmodule Pleroma.Web.RichMedia.Parser do
     with_body: true
   ]
 
+  defp parsers do
+    Pleroma.Config.get([:rich_media, :parsers])
+  end
+
   def parse(nil), do: {:error, "No URL provided"}
 
   if Pleroma.Config.get(:env) == :test do
@@ -26,6 +24,7 @@ def parse(url) do
         Cachex.fetch!(:rich_media_cache, url, fn _ ->
           {:commit, parse_url(url)}
         end)
+        |> set_ttl_based_on_image(url)
       rescue
         e ->
           {:error, "Cachex error: #{inspect(e)}"}
@@ -33,6 +32,50 @@ def parse(url) do
     end
   end
 
+  @doc """
+  Set the rich media cache based on the expiration time of image.
+
+  Adopt behaviour `Pleroma.Web.RichMedia.Parser.TTL`
+
+  ## Example
+
+      defmodule MyModule do
+        @behaviour Pleroma.Web.RichMedia.Parser.TTL
+        def ttl(data, url) do
+          image_url = Map.get(data, :image)
+          # do some parsing in the url and get the ttl of the image
+          # and return ttl is unix time
+          parse_ttl_from_url(image_url)
+        end
+      end
+
+  Define the module in the config
+
+      config :pleroma, :rich_media,
+        ttl_setters: [MyModule]
+  """
+  def set_ttl_based_on_image({:ok, data}, url) do
+    with {:ok, nil} <- Cachex.ttl(:rich_media_cache, url) do
+      ttl = get_ttl_from_image(data, url)
+      Cachex.expire_at(:rich_media_cache, url, ttl * 1000)
+      {:ok, data}
+    else
+      _ ->
+        {:ok, data}
+    end
+  end
+
+  defp get_ttl_from_image(data, url) do
+    Pleroma.Config.get([:rich_media, :ttl_setters])
+    |> Enum.reduce({:ok, nil}, fn
+      module, {:ok, _ttl} ->
+        module.ttl(data, url)
+
+      _, error ->
+        error
+    end)
+  end
+
   defp parse_url(url) do
     try do
       {:ok, %Tesla.Env{body: html}} = Pleroma.HTTP.get(url, [], adapter: @hackney_options)
@@ -48,7 +91,7 @@ defp parse_url(url) do
   end
 
   defp maybe_parse(html) do
-    Enum.reduce_while(@parsers, %{}, fn parser, acc ->
+    Enum.reduce_while(parsers(), %{}, fn parser, acc ->
       case parser.parse(html, acc) do
         {:ok, data} -> {:halt, data}
         {:error, _msg} -> {:cont, acc}

lib/pleroma/web/rich_media/parsers/ttl/aws_signed_url.ex

@@ -0,0 +1,52 @@
+defmodule Pleroma.Web.RichMedia.Parser.TTL.AwsSignedUrl do
+  @behaviour Pleroma.Web.RichMedia.Parser.TTL
+
+  @impl Pleroma.Web.RichMedia.Parser.TTL
+  def ttl(data, _url) do
+    image = Map.get(data, :image)
+
+    if is_aws_signed_url(image) do
+      image
+      |> parse_query_params()
+      |> format_query_params()
+      |> get_expiration_timestamp()
+    end
+  end
+
+  defp is_aws_signed_url(""), do: nil
+  defp is_aws_signed_url(nil), do: nil
+
+  defp is_aws_signed_url(image) when is_binary(image) do
+    %URI{host: host, query: query} = URI.parse(image)
+
+    if String.contains?(host, "amazonaws.com") and
+         String.contains?(query, "X-Amz-Expires") do
+      image
+    else
+      nil
+    end
+  end
+
+  defp is_aws_signed_url(_), do: nil
+
+  defp parse_query_params(image) do
+    %URI{query: query} = URI.parse(image)
+    query
+  end
+
+  defp format_query_params(query) do
+    query
+    |> String.split(~r/&|=/)
+    |> Enum.chunk_every(2)
+    |> Map.new(fn [k, v] -> {k, v} end)
+  end
+
+  defp get_expiration_timestamp(params) when is_map(params) do
+    {:ok, date} =
+      params
+      |> Map.get("X-Amz-Date")
+      |> Timex.parse("{ISO:Basic:Z}")
+
+    Timex.to_unix(date) + String.to_integer(Map.get(params, "X-Amz-Expires"))
+  end
+end

lib/pleroma/web/rich_media/parsers/ttl/ttl.ex

@@ -0,0 +1,3 @@
+defmodule Pleroma.Web.RichMedia.Parser.TTL do
+  @callback ttl(Map.t(), String.t()) :: {:ok, Integer.t()} | {:error, String.t()}
+end

lib/pleroma/web/router.ex

@@ -586,7 +586,7 @@ defmodule Pleroma.Web.Router do
     end
   end
 
-  pipeline :ap_relay do
+  pipeline :ap_service_actor do
     plug(:accepts, ["activity+json", "json"])
   end
 
@@ -617,6 +617,7 @@ defmodule Pleroma.Web.Router do
   pipeline :activitypub do
     plug(:accepts, ["activity+json", "json"])
     plug(Pleroma.Web.Plugs.HTTPSignaturePlug)
+    plug(Pleroma.Web.Plugs.MappedSignatureToIdentityPlug)
   end
 
   scope "/", Pleroma.Web.ActivityPub do
@@ -663,8 +664,17 @@ defmodule Pleroma.Web.Router do
   end
 
   scope "/relay", Pleroma.Web.ActivityPub do
-    pipe_through(:ap_relay)
+    pipe_through(:ap_service_actor)
+
     get("/", ActivityPubController, :relay)
+    post("/inbox", ActivityPubController, :inbox)
+  end
+
+  scope "/internal/fetch", Pleroma.Web.ActivityPub do
+    pipe_through(:ap_service_actor)
+
+    get("/", ActivityPubController, :internal_fetch)
+    post("/inbox", ActivityPubController, :inbox)
   end
 
   scope "/", Pleroma.Web.ActivityPub do
@@ -691,6 +701,8 @@ defmodule Pleroma.Web.Router do
     get("/web/login", MastodonAPIController, :login)
     delete("/auth/sign_out", MastodonAPIController, :logout)
 
+    post("/auth/password", MastodonAPIController, :password_reset)
+
     scope [] do
       pipe_through(:oauth_read_or_public)
       get("/web/*path", MastodonAPIController, :index)

lib/pleroma/web/salmon/salmon.ex

@@ -123,11 +123,26 @@ def encode(private_key, doc) do
     {:ok, salmon}
   end
 
-  def remote_users(%{data: %{"to" => to} = data}) do
+  def remote_users(%User{id: user_id}, %{data: %{"to" => to} = data}) do
-    to = to ++ (data["cc"] || [])
+    cc = Map.get(data, "cc", [])
 
-    to
+    bcc =
-    |> Enum.map(fn id -> User.get_cached_by_ap_id(id) end)
+      data
+      |> Map.get("bcc", [])
+      |> Enum.reduce([], fn ap_id, bcc ->
+        case Pleroma.List.get_by_ap_id(ap_id) do
+          %Pleroma.List{user_id: ^user_id} = list ->
+            {:ok, following} = Pleroma.List.get_following(list)
+            bcc ++ Enum.map(following, & &1.ap_id)
+
+          _ ->
+            bcc
+        end
+      end)
+
+    [to, cc, bcc]
+    |> Enum.concat()
+    |> Enum.map(&User.get_cached_by_ap_id/1)
     |> Enum.filter(fn user -> user && !user.local end)
   end
 
@@ -191,7 +206,7 @@ def publish(%{info: %{keys: keys}} = user, %{data: %{"type" => type}} = activity
       {:ok, private, _} = Keys.keys_from_pem(keys)
       {:ok, feed} = encode(private, feed)
 
-      remote_users = remote_users(activity)
+      remote_users = remote_users(user, activity)
 
       salmon_urls = Enum.map(remote_users, & &1.info.salmon)
       reachable_urls_metadata = Instances.filter_reachable(salmon_urls)

lib/pleroma/web/twitter_api/controllers/util_controller.ex

@@ -7,10 +7,12 @@ defmodule Pleroma.Web.TwitterAPI.UtilController do
 
   require Logger
 
-  alias Comeonin.Pbkdf2
   alias Pleroma.Activity
+  alias Pleroma.Config
   alias Pleroma.Emoji
+  alias Pleroma.Healthcheck
   alias Pleroma.Notification
+  alias Pleroma.Plugs.AuthenticationPlug
   alias Pleroma.User
   alias Pleroma.Web
   alias Pleroma.Web.ActivityPub.ActivityPub
@@ -23,7 +25,8 @@ def help_test(conn, _params) do
   end
 
   def remote_subscribe(conn, %{"nickname" => nick, "profile" => _}) do
-    with %User{} = user <- User.get_cached_by_nickname(nick), avatar = User.avatar_url(user) do
+    with %User{} = user <- User.get_cached_by_nickname(nick),
+         avatar = User.avatar_url(user) do
       conn
       |> render("subscribe.html", %{nickname: nick, avatar: avatar, error: false})
     else
@@ -96,7 +99,7 @@ def do_remote_follow(conn, %{
     name = followee.nickname
 
     with %User{} = user <- User.get_cached_by_nickname(username),
-         true <- Pbkdf2.checkpw(password, user.password_hash),
+         true <- AuthenticationPlug.checkpw(password, user.password_hash),
          %User{} = _followed <- User.get_cached_by_id(id),
          {:ok, follower} <- User.follow(user, followee),
          {:ok, _activity} <- ActivityPub.follow(follower, followee) do
@@ -338,20 +341,21 @@ def captcha(conn, _params) do
   end
 
   def healthcheck(conn, _params) do
-    info =
+    with true <- Config.get([:instance, :healthcheck]),
-      if Pleroma.Config.get([:instance, :healthcheck]) do
+         %{healthy: true} = info <- Healthcheck.system_info() do
-        Pleroma.Healthcheck.system_info()
-      else
-        %{}
-      end
-
-    conn =
-      if info[:healthy] do
-        conn
-      else
-        Plug.Conn.put_status(conn, :service_unavailable)
-      end
-
       json(conn, info)
+    else
+      %{healthy: false} = info ->
+        service_unavailable(conn, info)
+
+      _ ->
+        service_unavailable(conn, %{})
+    end
+  end
+
+  defp service_unavailable(conn, info) do
+    conn
+    |> put_status(:service_unavailable)
+    |> json(info)
   end
 end

lib/pleroma/web/twitter_api/twitter_api.ex

@@ -221,6 +221,8 @@ def password_reset(nickname_or_email) do
       user
       |> UserEmail.password_reset_email(token_record.token)
       |> Mailer.deliver_async()
+
+      {:ok, :enqueued}
     else
       false ->
         {:error, "bad user identifier"}

lib/pleroma/web/twitter_api/twitter_api_controller.ex

@@ -27,6 +27,7 @@ defmodule Pleroma.Web.TwitterAPI.Controller do
 
   require Logger
 
+  plug(Pleroma.Plugs.RateLimiter, :password_reset when action == :password_reset)
   plug(:only_if_public_instance when action in [:public_timeline, :public_and_external_timeline])
   action_fallback(:errors)
 
@@ -192,6 +193,13 @@ def dm_timeline(%{assigns: %{user: user}} = conn, params) do
   end
 
   def notifications(%{assigns: %{user: user}} = conn, params) do
+    params =
+      if Map.has_key?(params, "with_muted") do
+        Map.put(params, :with_muted, params["with_muted"] in [true, "True", "true", "1"])
+      else
+        params
+      end
+
     notifications = Notification.for_user(user, params)
 
     conn
@@ -430,6 +438,12 @@ def password_reset(conn, params) do
 
     with {:ok, _} <- TwitterAPI.password_reset(nickname_or_email) do
       json_response(conn, :no_content, "")
+    else
+      {:error, "unknown user"} ->
+        send_resp(conn, :not_found, "")
+
+      {:error, _} ->
+        send_resp(conn, :bad_request, "")
     end
   end
 

lib/pleroma/web/uploader_controller.ex

@@ -11,10 +11,6 @@ def callback(conn, %{"upload_path" => upload_path} = params) do
     process_callback(conn, :global.whereis_name({Uploader, upload_path}), params)
   end
 
-  def callbacks(conn, _) do
-    render_error(conn, :bad_request, "bad request")
-  end
-
   defp process_callback(conn, pid, params) when is_pid(pid) do
     send(pid, {Uploader, self(), conn, params})
 

lib/pleroma/web/web_finger/web_finger.ex

@@ -32,7 +32,7 @@ def host_meta do
 
   def webfinger(resource, fmt) when fmt in ["XML", "JSON"] do
     host = Pleroma.Web.Endpoint.host()
-    regex = ~r/(acct:)?(?<username>\w+)@#{host}/
+    regex = ~r/(acct:)?(?<username>[a-z0-9A-Z_\.-]+)@#{host}/
 
     with %{"username" => username} <- Regex.named_captures(regex, resource),
          %User{} = user <- User.get_cached_by_nickname(username) do

mix.exs

@@ -14,7 +14,7 @@ def project do
       aliases: aliases(),
       deps: deps(),
       test_coverage: [tool: ExCoveralls],
-
+      preferred_cli_env: ["coveralls.html": :test],
       # Docs
       name: "Pleroma",
       homepage_url: "https://pleroma.social/",
@@ -95,6 +95,7 @@ defp oauth_deps do
   defp deps do
     [
       {:phoenix, "~> 1.4.8"},
+      {:tzdata, "~> 1.0"},
       {:plug_cowboy, "~> 2.0"},
       {:phoenix_pubsub, "~> 1.1"},
       {:phoenix_ecto, "~> 4.0"},
@@ -137,7 +138,7 @@ defp deps do
        ref: "95e8188490e97505c56636c1379ffdf036c1fdde"},
       {:http_signatures,
        git: "https://git.pleroma.social/pleroma/http_signatures.git",
-       ref: "9789401987096ead65646b52b5a2ca6bf52fc531"},
+       ref: "293d77bb6f4a67ac8bde1428735c3b42f22cbb30"},
       {:pleroma_job_queue, "~> 0.2.0"},
       {:telemetry, "~> 0.3"},
       {:prometheus_ex, "~> 3.0"},

mix.lock

@@ -6,7 +6,7 @@
   "benchee": {:hex, :benchee, "1.0.1", "66b211f9bfd84bd97e6d1beaddf8fc2312aaabe192f776e8931cb0c16f53a521", [:mix], [{:deep_merge, "~> 1.0", [hex: :deep_merge, repo: "hexpm", optional: false]}], "hexpm"},
   "bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [:mix], [], "hexpm"},
   "cachex": {:hex, :cachex, "3.0.2", "1351caa4e26e29f7d7ec1d29b53d6013f0447630bbf382b4fb5d5bad0209f203", [:mix], [{:eternal, "~> 1.2", [hex: :eternal, repo: "hexpm", optional: false]}, {:unsafe, "~> 1.0", [hex: :unsafe, repo: "hexpm", optional: false]}], "hexpm"},
-  "calendar": {:hex, :calendar, "0.17.4", "22c5e8d98a4db9494396e5727108dffb820ee0d18fed4b0aa8ab76e4f5bc32f1", [:mix], [{:tzdata, "~> 0.5.8 or ~> 0.1.201603", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm"},
+  "calendar": {:hex, :calendar, "0.17.6", "ec291cb2e4ba499c2e8c0ef5f4ace974e2f9d02ae9e807e711a9b0c7850b9aee", [:mix], [{:tzdata, "~> 0.5.20 or ~> 0.1.201603 or ~> 1.0", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm"},
   "certifi": {:hex, :certifi, "2.5.1", "867ce347f7c7d78563450a18a6a28a8090331e77fa02380b4a21962a65d36ee5", [:rebar3], [{:parse_trans, "~>3.3", [hex: :parse_trans, repo: "hexpm", optional: false]}], "hexpm"},
   "combine": {:hex, :combine, "0.10.0", "eff8224eeb56498a2af13011d142c5e7997a80c8f5b97c499f84c841032e429f", [:mix], [], "hexpm"},
   "comeonin": {:hex, :comeonin, "4.1.1", "c7304fc29b45b897b34142a91122bc72757bc0c295e9e824999d5179ffc08416", [:mix], [{:argon2_elixir, "~> 1.2", [hex: :argon2_elixir, repo: "hexpm", optional: true]}, {:bcrypt_elixir, "~> 0.12.1 or ~> 1.0", [hex: :bcrypt_elixir, repo: "hexpm", optional: true]}, {:pbkdf2_elixir, "~> 0.12", [hex: :pbkdf2_elixir, repo: "hexpm", optional: true]}], "hexpm"},
@@ -38,7 +38,7 @@
   "hackney": {:hex, :hackney, "1.15.1", "9f8f471c844b8ce395f7b6d8398139e26ddca9ebc171a8b91342ee15a19963f4", [:rebar3], [{:certifi, "2.5.1", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "6.0.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "1.0.1", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~>1.1", [hex: :mimerl, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "1.1.4", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}], "hexpm"},
   "html_entities": {:hex, :html_entities, "0.4.0", "f2fee876858cf6aaa9db608820a3209e45a087c5177332799592142b50e89a6b", [:mix], [], "hexpm"},
   "html_sanitize_ex": {:hex, :html_sanitize_ex, "1.3.0", "f005ad692b717691203f940c686208aa3d8ffd9dd4bb3699240096a51fa9564e", [:mix], [{:mochiweb, "~> 2.15", [hex: :mochiweb, repo: "hexpm", optional: false]}], "hexpm"},
-  "http_signatures": {:git, "https://git.pleroma.social/pleroma/http_signatures.git", "9789401987096ead65646b52b5a2ca6bf52fc531", [ref: "9789401987096ead65646b52b5a2ca6bf52fc531"]},
+  "http_signatures": {:git, "https://git.pleroma.social/pleroma/http_signatures.git", "293d77bb6f4a67ac8bde1428735c3b42f22cbb30", [ref: "293d77bb6f4a67ac8bde1428735c3b42f22cbb30"]},
   "httpoison": {:hex, :httpoison, "1.2.0", "2702ed3da5fd7a8130fc34b11965c8cfa21ade2f232c00b42d96d4967c39a3a3", [:mix], [{:hackney, "~> 1.8", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm"},
   "idna": {:hex, :idna, "6.0.0", "689c46cbcdf3524c44d5f3dde8001f364cd7608a99556d8fbd8239a5798d4c10", [:rebar3], [{:unicode_util_compat, "0.4.1", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm"},
   "jason": {:hex, :jason, "1.1.2", "b03dedea67a99223a2eaf9f1264ce37154564de899fd3d8b9a21b1a6fd64afe7", [:mix], [{:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm"},
@@ -82,9 +82,9 @@
   "syslog": {:git, "https://github.com/Vagabond/erlang-syslog.git", "4a6c6f2c996483e86c1320e9553f91d337bcb6aa", [tag: "1.0.5"]},
   "telemetry": {:hex, :telemetry, "0.4.0", "8339bee3fa8b91cb84d14c2935f8ecf399ccd87301ad6da6b71c09553834b2ab", [:rebar3], [], "hexpm"},
   "tesla": {:hex, :tesla, "1.2.1", "864783cc27f71dd8c8969163704752476cec0f3a51eb3b06393b3971dc9733ff", [:mix], [{:exjsx, ">= 3.0.0", [hex: :exjsx, repo: "hexpm", optional: true]}, {:fuse, "~> 2.4", [hex: :fuse, repo: "hexpm", optional: true]}, {:hackney, "~> 1.6", [hex: :hackney, repo: "hexpm", optional: true]}, {:ibrowse, "~> 4.4.0", [hex: :ibrowse, repo: "hexpm", optional: true]}, {:jason, ">= 1.0.0", [hex: :jason, repo: "hexpm", optional: true]}, {:mime, "~> 1.0", [hex: :mime, repo: "hexpm", optional: false]}, {:poison, ">= 1.0.0", [hex: :poison, repo: "hexpm", optional: true]}], "hexpm"},
-  "timex": {:hex, :timex, "3.5.0", "b0a23167da02d0fe4f1a4e104d1f929a00d348502b52432c05de875d0b9cffa5", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.10", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 0.1.8 or ~> 0.5", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm"},
+  "timex": {:hex, :timex, "3.6.1", "efdf56d0e67a6b956cc57774353b0329c8ab7726766a11547e529357ffdc1d56", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.10", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 0.1.8 or ~> 0.5 or ~> 1.0.0", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm"},
   "trailing_format_plug": {:hex, :trailing_format_plug, "0.0.7", "64b877f912cf7273bed03379936df39894149e35137ac9509117e59866e10e45", [:mix], [{:plug, "> 0.12.0", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"},
-  "tzdata": {:hex, :tzdata, "0.5.20", "304b9e98a02840fb32a43ec111ffbe517863c8566eb04a061f1c4dbb90b4d84c", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm"},
+  "tzdata": {:hex, :tzdata, "1.0.1", "f6027a331af7d837471248e62733c6ebee86a72e57c613aa071ebb1f750fc71a", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm"},
   "ueberauth": {:hex, :ueberauth, "0.6.1", "9e90d3337dddf38b1ca2753aca9b1e53d8a52b890191cdc55240247c89230412", [:mix], [{:plug, "~> 1.5", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"},
   "unicode_util_compat": {:hex, :unicode_util_compat, "0.4.1", "d869e4c68901dd9531385bb0c8c40444ebf624e60b6962d95952775cac5e90cd", [:rebar3], [], "hexpm"},
   "unsafe": {:hex, :unsafe, "1.0.0", "7c21742cd05380c7875546b023481d3a26f52df8e5dfedcb9f958f322baae305", [:mix], [], "hexpm"},

priv/repo/migrations/20190516112144_add_ap_id_to_lists.exs

@@ -0,0 +1,26 @@
+defmodule Pleroma.Repo.Migrations.AddApIdToLists do
+  use Ecto.Migration
+
+  def up do
+    alter table(:lists) do
+      add(:ap_id, :string)
+    end
+
+    execute("""
+    UPDATE lists
+    SET ap_id = u.ap_id || '/lists/' || lists.id
+    FROM users AS u
+    WHERE lists.user_id = u.id
+    """)
+
+    create(unique_index(:lists, :ap_id))
+  end
+
+  def down do
+    drop(index(:lists, [:ap_id]))
+
+    alter table(:lists) do
+      remove(:ap_id)
+    end
+  end
+end

priv/repo/migrations/20190711042024_copy_muted_to_muted_notifications.exs

@@ -0,0 +1,24 @@
+defmodule Pleroma.Repo.Migrations.CopyMutedToMutedNotifications do
+  use Ecto.Migration
+  alias Pleroma.User
+
+  def change do
+    query =
+      User.Query.build(%{
+        local: true,
+        active: true,
+        order_by: :id
+      })
+
+    Pleroma.Repo.stream(query)
+    |> Enum.each(fn
+      %{info: %{mutes: mutes} = info} = user ->
+        info_cng =
+          Ecto.Changeset.cast(info, %{muted_notifications: mutes}, [:muted_notifications])
+
+        Ecto.Changeset.change(user)
+        |> Ecto.Changeset.put_embed(:info, info_cng)
+        |> Pleroma.Repo.update()
+    end)
+  end
+end

priv/static/schemas/litepub-0.1.jsonld

@@ -20,6 +20,10 @@
             "sensitive": "as:sensitive",
             "litepub": "http://litepub.social/ns#",
             "directMessage": "litepub:directMessage",
+            "listMessage": {
+                "@id": "litepub:listMessage",
+                "@type": "@id"
+            },
             "oauthRegistrationEndpoint": {
                 "@id": "litepub:oauthRegistrationEndpoint",
                 "@type": "@id"

test/fixtures/rich_media/amz.html

@@ -0,0 +1,5 @@
+<meta name="twitter:card" content="summary" />
+<meta name="twitter:site" content="@flickr" />
+<meta name="twitter:title" content="Small Island Developing States Photo Submission" />
+<meta name="twitter:description" content="View the album on Flickr." />
+<meta name="twitter:image" content="https://pleroma.s3.ap-southeast-1.amazonaws.com/sachin%20%281%29%20_a%20-%25%2Aasdasd%20BNN%20bnnn%20.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIBLWWK6RGDQXDLJQ%2F20190716%2Fap-southeast-1%2Fs3%2Faws4_request&X-Amz-Date=20190716T175105Z&X-Amz-Expires=300000&X-Amz-Signature=04ffd6b98634f4b1bbabc62e0fac4879093cd54a6eed24fe8eb38e8369526bbf&X-Amz-SignedHeaders=host" />

test/list_test.exs

@@ -113,4 +113,30 @@ test "getting own lists a given user belongs to" do
     assert owned_list in lists_2
     refute not_owned_list in lists_2
   end
+
+  test "get by ap_id" do
+    user = insert(:user)
+    {:ok, list} = Pleroma.List.create("foo", user)
+    assert Pleroma.List.get_by_ap_id(list.ap_id) == list
+  end
+
+  test "memberships" do
+    user = insert(:user)
+    member = insert(:user)
+    {:ok, list} = Pleroma.List.create("foo", user)
+    {:ok, list} = Pleroma.List.follow(list, member)
+
+    assert Pleroma.List.memberships(member) == [list.ap_id]
+  end
+
+  test "member?" do
+    user = insert(:user)
+    member = insert(:user)
+
+    {:ok, list} = Pleroma.List.create("foo", user)
+    {:ok, list} = Pleroma.List.follow(list, member)
+
+    assert Pleroma.List.member?(list, member)
+    refute Pleroma.List.member?(list, user)
+  end
 end

test/notification_test.exs

@@ -74,26 +74,37 @@ test "it creates a notification for user and send to the 'user' and the 'user:no
       Task.await(task_user_notification)
     end
 
-    test "it doesn't create a notification for user if the user blocks the activity author" do
+    test "it creates a notification for user if the user blocks the activity author" do
       activity = insert(:note_activity)
       author = User.get_cached_by_ap_id(activity.data["actor"])
       user = insert(:user)
       {:ok, user} = User.block(user, author)
 
-      refute Notification.create_notification(activity, user)
+      assert Notification.create_notification(activity, user)
     end
 
-    test "it doesn't create a notificatin for the user if the user mutes the activity author" do
+    test "it creates a notificatin for the user if the user mutes the activity author" do
       muter = insert(:user)
       muted = insert(:user)
       {:ok, _} = User.mute(muter, muted)
       muter = Repo.get(User, muter.id)
       {:ok, activity} = CommonAPI.post(muted, %{"status" => "Hi @#{muter.nickname}"})
 
-      refute Notification.create_notification(activity, muter)
+      assert Notification.create_notification(activity, muter)
     end
 
-    test "it doesn't create a notification for an activity from a muted thread" do
+    test "notification created if user is muted without notifications" do
+      muter = insert(:user)
+      muted = insert(:user)
+
+      {:ok, muter} = User.mute(muter, muted, false)
+
+      {:ok, activity} = CommonAPI.post(muted, %{"status" => "Hi @#{muter.nickname}"})
+
+      assert Notification.create_notification(activity, muter)
+    end
+
+    test "it creates a notification for an activity from a muted thread" do
       muter = insert(:user)
       other_user = insert(:user)
       {:ok, activity} = CommonAPI.post(muter, %{"status" => "hey"})
@@ -105,7 +116,7 @@ test "it doesn't create a notification for an activity from a muted thread" do
           "in_reply_to_status_id" => activity.id
         })
 
-      refute Notification.create_notification(activity, muter)
+      assert Notification.create_notification(activity, muter)
     end
 
     test "it disables notifications from followers" do
@@ -532,4 +543,98 @@ test "replying to a deleted post without tagging does not generate a notificatio
       assert Enum.empty?(Notification.for_user(user))
     end
   end
+
+  describe "for_user" do
+    test "it returns notifications for muted user without notifications" do
+      user = insert(:user)
+      muted = insert(:user)
+      {:ok, user} = User.mute(user, muted, false)
+
+      {:ok, _activity} = TwitterAPI.create_status(muted, %{"status" => "hey @#{user.nickname}"})
+
+      assert length(Notification.for_user(user)) == 1
+    end
+
+    test "it doesn't return notifications for muted user with notifications" do
+      user = insert(:user)
+      muted = insert(:user)
+      {:ok, user} = User.mute(user, muted)
+
+      {:ok, _activity} = TwitterAPI.create_status(muted, %{"status" => "hey @#{user.nickname}"})
+
+      assert Notification.for_user(user) == []
+    end
+
+    test "it doesn't return notifications for blocked user" do
+      user = insert(:user)
+      blocked = insert(:user)
+      {:ok, user} = User.block(user, blocked)
+
+      {:ok, _activity} = TwitterAPI.create_status(blocked, %{"status" => "hey @#{user.nickname}"})
+
+      assert Notification.for_user(user) == []
+    end
+
+    test "it doesn't return notificatitons for blocked domain" do
+      user = insert(:user)
+      blocked = insert(:user, ap_id: "http://some-domain.com")
+      {:ok, user} = User.block_domain(user, "some-domain.com")
+
+      {:ok, _activity} = TwitterAPI.create_status(blocked, %{"status" => "hey @#{user.nickname}"})
+
+      assert Notification.for_user(user) == []
+    end
+
+    test "it doesn't return notifications for muted thread" do
+      user = insert(:user)
+      another_user = insert(:user)
+
+      {:ok, activity} =
+        TwitterAPI.create_status(another_user, %{"status" => "hey @#{user.nickname}"})
+
+      {:ok, _} = Pleroma.ThreadMute.add_mute(user.id, activity.data["context"])
+      assert Notification.for_user(user) == []
+    end
+
+    test "it returns notifications for muted user with notifications and with_muted parameter" do
+      user = insert(:user)
+      muted = insert(:user)
+      {:ok, user} = User.mute(user, muted)
+
+      {:ok, _activity} = TwitterAPI.create_status(muted, %{"status" => "hey @#{user.nickname}"})
+
+      assert length(Notification.for_user(user, %{with_muted: true})) == 1
+    end
+
+    test "it returns notifications for blocked user and with_muted parameter" do
+      user = insert(:user)
+      blocked = insert(:user)
+      {:ok, user} = User.block(user, blocked)
+
+      {:ok, _activity} = TwitterAPI.create_status(blocked, %{"status" => "hey @#{user.nickname}"})
+
+      assert length(Notification.for_user(user, %{with_muted: true})) == 1
+    end
+
+    test "it returns notificatitons for blocked domain and with_muted parameter" do
+      user = insert(:user)
+      blocked = insert(:user, ap_id: "http://some-domain.com")
+      {:ok, user} = User.block_domain(user, "some-domain.com")
+
+      {:ok, _activity} = TwitterAPI.create_status(blocked, %{"status" => "hey @#{user.nickname}"})
+
+      assert length(Notification.for_user(user, %{with_muted: true})) == 1
+    end
+
+    test "it returns notifications for muted thread with_muted parameter" do
+      user = insert(:user)
+      another_user = insert(:user)
+
+      {:ok, activity} =
+        TwitterAPI.create_status(another_user, %{"status" => "hey @#{user.nickname}"})
+
+      {:ok, _} = Pleroma.ThreadMute.add_mute(user.id, activity.data["context"])
+      assert length(Notification.for_user(user, %{with_muted: true})) == 1
+    end
+  end
 end

test/object/containment_test.exs

@@ -68,4 +68,34 @@ test "users cannot be collided through fake direction spoofing attempts" do
                "[error] Could not decode user at fetch https://n1u.moe/users/rye, {:error, :error}"
     end
   end
+
+  describe "containment of children" do
+    test "contain_child() catches spoofing attempts" do
+      data = %{
+        "id" => "http://example.com/whatever",
+        "type" => "Create",
+        "object" => %{
+          "id" => "http://example.net/~alyssa/activities/1234",
+          "attributedTo" => "http://example.org/~alyssa"
+        },
+        "actor" => "http://example.com/~bob"
+      }
+
+      :error = Containment.contain_child(data)
+    end
+
+    test "contain_child() allows correct origins" do
+      data = %{
+        "id" => "http://example.org/~alyssa/activities/5678",
+        "type" => "Create",
+        "object" => %{
+          "id" => "http://example.org/~alyssa/activities/1234",
+          "attributedTo" => "http://example.org/~alyssa"
+        },
+        "actor" => "http://example.org/~alyssa"
+      }
+
+      :ok = Containment.contain_child(data)
+    end
+  end
 end

test/object/fetcher_test.exs

@@ -9,6 +9,7 @@ defmodule Pleroma.Object.FetcherTest do
   alias Pleroma.Object
   alias Pleroma.Object.Fetcher
   import Tesla.Mock
+  import Mock
 
   setup do
     mock(fn
@@ -26,16 +27,31 @@ defmodule Pleroma.Object.FetcherTest do
   end
 
   describe "actor origin containment" do
-    test "it rejects objects with a bogus origin" do
+    test_with_mock "it rejects objects with a bogus origin",
+                   Pleroma.Web.OStatus,
+                   [:passthrough],
+                   [] do
       {:error, _} = Fetcher.fetch_object_from_id("https://info.pleroma.site/activity.json")
+
+      refute called(Pleroma.Web.OStatus.fetch_activity_from_url(:_))
     end
 
-    test "it rejects objects when attributedTo is wrong (variant 1)" do
+    test_with_mock "it rejects objects when attributedTo is wrong (variant 1)",
+                   Pleroma.Web.OStatus,
+                   [:passthrough],
+                   [] do
       {:error, _} = Fetcher.fetch_object_from_id("https://info.pleroma.site/activity2.json")
+
+      refute called(Pleroma.Web.OStatus.fetch_activity_from_url(:_))
     end
 
-    test "it rejects objects when attributedTo is wrong (variant 2)" do
+    test_with_mock "it rejects objects when attributedTo is wrong (variant 2)",
+                   Pleroma.Web.OStatus,
+                   [:passthrough],
+                   [] do
       {:error, _} = Fetcher.fetch_object_from_id("https://info.pleroma.site/activity3.json")
+
+      refute called(Pleroma.Web.OStatus.fetch_activity_from_url(:_))
     end
   end
 
@@ -134,4 +150,34 @@ test "it can refetch pruned objects" do
       assert object.id != object_two.id
     end
   end
+
+  describe "signed fetches" do
+    test_with_mock "it signs fetches when configured to do so",
+                   Pleroma.Signature,
+                   [:passthrough],
+                   [] do
+      option = Pleroma.Config.get([:activitypub, :sign_object_fetches])
+      Pleroma.Config.put([:activitypub, :sign_object_fetches], true)
+
+      Fetcher.fetch_object_from_id("http://mastodon.example.org/@admin/99541947525187367")
+
+      assert called(Pleroma.Signature.sign(:_, :_))
+
+      Pleroma.Config.put([:activitypub, :sign_object_fetches], option)
+    end
+
+    test_with_mock "it doesn't sign fetches when not configured to do so",
+                   Pleroma.Signature,
+                   [:passthrough],
+                   [] do
+      option = Pleroma.Config.get([:activitypub, :sign_object_fetches])
+      Pleroma.Config.put([:activitypub, :sign_object_fetches], false)
+
+      Fetcher.fetch_object_from_id("http://mastodon.example.org/@admin/99541947525187367")
+
+      refute called(Pleroma.Signature.sign(:_, :_))
+
+      Pleroma.Config.put([:activitypub, :sign_object_fetches], option)
+    end
+  end
 end

test/plugs/authentication_plug_test.exs

@@ -8,6 +8,9 @@ defmodule Pleroma.Plugs.AuthenticationPlugTest do
   alias Pleroma.Plugs.AuthenticationPlug
   alias Pleroma.User
 
+  import ExUnit.CaptureLog
+  import Mock
+
   setup %{conn: conn} do
     user = %User{
       id: 1,
@@ -54,4 +57,32 @@ test "with a wrong password in the credentials, it does nothing", %{conn: conn}
 
     assert conn == ret_conn
   end
+
+  describe "checkpw/2" do
+    test "check pbkdf2 hash" do
+      hash =
+        "$pbkdf2-sha512$160000$loXqbp8GYls43F0i6lEfIw$AY.Ep.2pGe57j2hAPY635sI/6w7l9Q9u9Bp02PkPmF3OrClDtJAI8bCiivPr53OKMF7ph6iHhN68Rom5nEfC2A"
+
+      assert AuthenticationPlug.checkpw("test-password", hash)
+      refute AuthenticationPlug.checkpw("test-password1", hash)
+    end
+
+    test "check sha512-crypt hash" do
+      hash =
+        "$6$9psBWV8gxkGOZWBz$PmfCycChoxeJ3GgGzwvhlgacb9mUoZ.KUXNCssekER4SJ7bOK53uXrHNb2e4i8yPFgSKyzaW9CcmrDXWIEMtD1"
+
+      with_mock :crypt, crypt: fn _password, password_hash -> password_hash end do
+        assert AuthenticationPlug.checkpw("password", hash)
+      end
+    end
+
+    test "it returns false when hash invalid" do
+      hash =
+        "psBWV8gxkGOZWBz$PmfCycChoxeJ3GgGzwvhlgacb9mUoZ.KUXNCssekER4SJ7bOK53uXrHNb2e4i8yPFgSKyzaW9CcmrDXWIEMtD1"
+
+      assert capture_log(fn ->
+               refute Pleroma.Plugs.AuthenticationPlug.checkpw("password", hash)
+             end) =~ "[error] Password hash not recognized"
+    end
+  end
 end

test/plugs/http_signature_plug_test.exs

@@ -26,22 +26,4 @@ test "it call HTTPSignatures to check validity if the actor sighed it" do
       assert called(HTTPSignatures.validate_conn(:_))
     end
   end
-
-  test "bails out early if the signature isn't by the activity actor" do
-    params = %{"actor" => "https://mst3k.interlinked.me/users/luciferMysticus"}
-    conn = build_conn(:get, "/doesntmattter", params)
-
-    with_mock HTTPSignatures, validate_conn: fn _ -> false end do
-      conn =
-        conn
-        |> put_req_header(
-          "signature",
-          "keyId=\"http://mastodon.example.org/users/admin#main-key"
-        )
-        |> HTTPSignaturePlug.call(%{})
-
-      assert conn.assigns.valid_signature == false
-      refute called(HTTPSignatures.validate_conn(:_))
-    end
-  end
 end

test/plugs/mapped_identity_to_signature_plug_test.exs

@@ -0,0 +1,59 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.Plugs.MappedSignatureToIdentityPlugTest do
+  use Pleroma.Web.ConnCase
+  alias Pleroma.Web.Plugs.MappedSignatureToIdentityPlug
+
+  import Tesla.Mock
+  import Plug.Conn
+
+  setup do
+    mock(fn env -> apply(HttpRequestMock, :request, [env]) end)
+    :ok
+  end
+
+  defp set_signature(conn, key_id) do
+    conn
+    |> put_req_header("signature", "keyId=\"#{key_id}\"")
+    |> assign(:valid_signature, true)
+  end
+
+  test "it successfully maps a valid identity with a valid signature" do
+    conn =
+      build_conn(:get, "/doesntmattter")
+      |> set_signature("http://mastodon.example.org/users/admin")
+      |> MappedSignatureToIdentityPlug.call(%{})
+
+    refute is_nil(conn.assigns.user)
+  end
+
+  test "it successfully maps a valid identity with a valid signature with payload" do
+    conn =
+      build_conn(:post, "/doesntmattter", %{"actor" => "http://mastodon.example.org/users/admin"})
+      |> set_signature("http://mastodon.example.org/users/admin")
+      |> MappedSignatureToIdentityPlug.call(%{})
+
+    refute is_nil(conn.assigns.user)
+  end
+
+  test "it considers a mapped identity to be invalid when it mismatches a payload" do
+    conn =
+      build_conn(:post, "/doesntmattter", %{"actor" => "http://mastodon.example.org/users/admin"})
+      |> set_signature("https://niu.moe/users/rye")
+      |> MappedSignatureToIdentityPlug.call(%{})
+
+    assert %{valid_signature: false} == conn.assigns
+  end
+
+  @tag skip: "known breakage; the testsuite presently depends on it"
+  test "it considers a mapped identity to be invalid when the identity cannot be found" do
+    conn =
+      build_conn(:post, "/doesntmattter", %{"actor" => "http://mastodon.example.org/users/admin"})
+      |> set_signature("http://niu.moe/users/rye")
+      |> MappedSignatureToIdentityPlug.call(%{})
+
+    assert %{valid_signature: false} == conn.assigns
+  end
+end

test/plugs/rate_limiter_test.exs

@@ -10,12 +10,13 @@ defmodule Pleroma.Plugs.RateLimiterTest do
 
   import Pleroma.Factory
 
-  @limiter_name :testing
+  # Note: each example must work with separate buckets in order to prevent concurrency issues
 
   test "init/1" do
-    Pleroma.Config.put([:rate_limit, @limiter_name], {1, 1})
+    limiter_name = :test_init
+    Pleroma.Config.put([:rate_limit, limiter_name], {1, 1})
 
-    assert {@limiter_name, {1, 1}} == RateLimiter.init(@limiter_name)
+    assert {limiter_name, {1, 1}, []} == RateLimiter.init(limiter_name)
     assert nil == RateLimiter.init(:foo)
   end
 
@@ -24,14 +25,15 @@ test "ip/1" do
   end
 
   test "it restricts by opts" do
+    limiter_name = :test_opts
     scale = 1000
     limit = 5
 
-    Pleroma.Config.put([:rate_limit, @limiter_name], {scale, limit})
+    Pleroma.Config.put([:rate_limit, limiter_name], {scale, limit})
 
-    opts = RateLimiter.init(@limiter_name)
+    opts = RateLimiter.init(limiter_name)
     conn = conn(:get, "/")
-    bucket_name = "#{@limiter_name}:#{RateLimiter.ip(conn)}"
+    bucket_name = "#{limiter_name}:#{RateLimiter.ip(conn)}"
 
     conn = RateLimiter.call(conn, opts)
     assert {1, 4, _, _, _} = ExRated.inspect_bucket(bucket_name, scale, limit)
@@ -65,18 +67,78 @@ test "it restricts by opts" do
     refute conn.halted
   end
 
+  test "`bucket_name` option overrides default bucket name" do
+    limiter_name = :test_bucket_name
+    scale = 1000
+    limit = 5
+
+    Pleroma.Config.put([:rate_limit, limiter_name], {scale, limit})
+    base_bucket_name = "#{limiter_name}:group1"
+    opts = RateLimiter.init({limiter_name, bucket_name: base_bucket_name})
+
+    conn = conn(:get, "/")
+    default_bucket_name = "#{limiter_name}:#{RateLimiter.ip(conn)}"
+    customized_bucket_name = "#{base_bucket_name}:#{RateLimiter.ip(conn)}"
+
+    RateLimiter.call(conn, opts)
+    assert {1, 4, _, _, _} = ExRated.inspect_bucket(customized_bucket_name, scale, limit)
+    assert {0, 5, _, _, _} = ExRated.inspect_bucket(default_bucket_name, scale, limit)
+  end
+
+  test "`params` option appends specified params' values to bucket name" do
+    limiter_name = :test_params
+    scale = 1000
+    limit = 5
+
+    Pleroma.Config.put([:rate_limit, limiter_name], {scale, limit})
+    opts = RateLimiter.init({limiter_name, params: ["id"]})
+    id = "1"
+
+    conn = conn(:get, "/?id=#{id}")
+    conn = Plug.Conn.fetch_query_params(conn)
+
+    default_bucket_name = "#{limiter_name}:#{RateLimiter.ip(conn)}"
+    parametrized_bucket_name = "#{limiter_name}:#{id}:#{RateLimiter.ip(conn)}"
+
+    RateLimiter.call(conn, opts)
+    assert {1, 4, _, _, _} = ExRated.inspect_bucket(parametrized_bucket_name, scale, limit)
+    assert {0, 5, _, _, _} = ExRated.inspect_bucket(default_bucket_name, scale, limit)
+  end
+
+  test "it supports combination of options modifying bucket name" do
+    limiter_name = :test_options_combo
+    scale = 1000
+    limit = 5
+
+    Pleroma.Config.put([:rate_limit, limiter_name], {scale, limit})
+    base_bucket_name = "#{limiter_name}:group1"
+    opts = RateLimiter.init({limiter_name, bucket_name: base_bucket_name, params: ["id"]})
+    id = "100"
+
+    conn = conn(:get, "/?id=#{id}")
+    conn = Plug.Conn.fetch_query_params(conn)
+
+    default_bucket_name = "#{limiter_name}:#{RateLimiter.ip(conn)}"
+    parametrized_bucket_name = "#{base_bucket_name}:#{id}:#{RateLimiter.ip(conn)}"
+
+    RateLimiter.call(conn, opts)
+    assert {1, 4, _, _, _} = ExRated.inspect_bucket(parametrized_bucket_name, scale, limit)
+    assert {0, 5, _, _, _} = ExRated.inspect_bucket(default_bucket_name, scale, limit)
+  end
+
   test "optional limits for authenticated users" do
+    limiter_name = :test_authenticated
     Ecto.Adapters.SQL.Sandbox.checkout(Pleroma.Repo)
 
     scale = 1000
     limit = 5
-    Pleroma.Config.put([:rate_limit, @limiter_name], [{1, 10}, {scale, limit}])
+    Pleroma.Config.put([:rate_limit, limiter_name], [{1, 10}, {scale, limit}])
 
-    opts = RateLimiter.init(@limiter_name)
+    opts = RateLimiter.init(limiter_name)
 
     user = insert(:user)
     conn = conn(:get, "/") |> assign(:user, user)
-    bucket_name = "#{@limiter_name}:#{user.id}"
+    bucket_name = "#{limiter_name}:#{user.id}"
 
     conn = RateLimiter.call(conn, opts)
     assert {1, 4, _, _, _} = ExRated.inspect_bucket(bucket_name, scale, limit)

test/reverse_proxy_test.exs

@@ -5,7 +5,6 @@
 defmodule Pleroma.ReverseProxyTest do
   use Pleroma.Web.ConnCase, async: true
   import ExUnit.CaptureLog
-  import ExUnit.CaptureLog
   import Mox
   alias Pleroma.ReverseProxy
   alias Pleroma.ReverseProxy.ClientMock

test/signature_test.exs

@@ -0,0 +1,108 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.SignatureTest do
+  use Pleroma.DataCase
+
+  import ExUnit.CaptureLog
+  import Pleroma.Factory
+  import Tesla.Mock
+
+  alias Pleroma.Signature
+
+  setup do
+    mock(fn env -> apply(HttpRequestMock, :request, [env]) end)
+    :ok
+  end
+
+  @private_key "-----BEGIN RSA PRIVATE KEY-----\nMIIEpQIBAAKCAQEA48qb4v6kqigZutO9Ot0wkp27GIF2LiVaADgxQORZozZR63jH\nTaoOrS3Xhngbgc8SSOhfXET3omzeCLqaLNfXnZ8OXmuhJfJSU6mPUvmZ9QdT332j\nfN/g3iWGhYMf/M9ftCKh96nvFVO/tMruzS9xx7tkrfJjehdxh/3LlJMMImPtwcD7\nkFXwyt1qZTAU6Si4oQAJxRDQXHp1ttLl3Ob829VM7IKkrVmY8TD+JSlV0jtVJPj6\n1J19ytKTx/7UaucYvb9HIiBpkuiy5n/irDqKLVf5QEdZoNCdojOZlKJmTLqHhzKP\n3E9TxsUjhrf4/EqegNc/j982RvOxeu4i40zMQwIDAQABAoIBAQDH5DXjfh21i7b4\ncXJuw0cqget617CDUhemdakTDs9yH+rHPZd3mbGDWuT0hVVuFe4vuGpmJ8c+61X0\nRvugOlBlavxK8xvYlsqTzAmPgKUPljyNtEzQ+gz0I+3mH2jkin2rL3D+SksZZgKm\nfiYMPIQWB2WUF04gB46DDb2mRVuymGHyBOQjIx3WC0KW2mzfoFUFRlZEF+Nt8Ilw\nT+g/u0aZ1IWoszbsVFOEdghgZET0HEarum0B2Je/ozcPYtwmU10iBANGMKdLqaP/\nj954BPunrUf6gmlnLZKIKklJj0advx0NA+cL79+zeVB3zexRYSA5o9q0WPhiuTwR\n/aedWHnBAoGBAP0sDWBAM1Y4TRAf8ZI9PcztwLyHPzfEIqzbObJJnx1icUMt7BWi\n+/RMOnhrlPGE1kMhOqSxvXYN3u+eSmWTqai2sSH5Hdw2EqnrISSTnwNUPINX7fHH\njEkgmXQ6ixE48SuBZnb4w1EjdB/BA6/sjL+FNhggOc87tizLTkMXmMtTAoGBAOZV\n+wPuAMBDBXmbmxCuDIjoVmgSlgeRunB1SA8RCPAFAiUo3+/zEgzW2Oz8kgI+xVwM\n33XkLKrWG1Orhpp6Hm57MjIc5MG+zF4/YRDpE/KNG9qU1tiz0UD5hOpIU9pP4bR/\ngxgPxZzvbk4h5BfHWLpjlk8UUpgk6uxqfti48c1RAoGBALBOKDZ6HwYRCSGMjUcg\n3NPEUi84JD8qmFc2B7Tv7h2he2ykIz9iFAGpwCIyETQsJKX1Ewi0OlNnD3RhEEAy\nl7jFGQ+mkzPSeCbadmcpYlgIJmf1KN/x7fDTAepeBpCEzfZVE80QKbxsaybd3Dp8\nCfwpwWUFtBxr4c7J+gNhAGe/AoGAPn8ZyqkrPv9wXtyfqFjxQbx4pWhVmNwrkBPi\nZ2Qh3q4dNOPwTvTO8vjghvzIyR8rAZzkjOJKVFgftgYWUZfM5gE7T2mTkBYq8W+U\n8LetF+S9qAM2gDnaDx0kuUTCq7t87DKk6URuQ/SbI0wCzYjjRD99KxvChVGPBHKo\n1DjqMuECgYEAgJGNm7/lJCS2wk81whfy/ttKGsEIkyhPFYQmdGzSYC5aDc2gp1R3\nxtOkYEvdjfaLfDGEa4UX8CHHF+w3t9u8hBtcdhMH6GYb9iv6z0VBTt4A/11HUR49\n3Z7TQ18Iyh3jAUCzFV9IJlLIExq5Y7P4B3ojWFBN607sDCt8BMPbDYs=\n-----END RSA PRIVATE KEY-----"
+
+  @public_key %{
+    "id" => "https://mastodon.social/users/lambadalambda#main-key",
+    "owner" => "https://mastodon.social/users/lambadalambda",
+    "publicKeyPem" =>
+      "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAw0P/Tq4gb4G/QVuMGbJo\nC/AfMNcv+m7NfrlOwkVzcU47jgESuYI4UtJayissCdBycHUnfVUd9qol+eznSODz\nCJhfJloqEIC+aSnuEPGA0POtWad6DU0E6/Ho5zQn5WAWUwbRQqowbrsm/GHo2+3v\neR5jGenwA6sYhINg/c3QQbksyV0uJ20Umyx88w8+TJuv53twOfmyDWuYNoQ3y5cc\nHKOZcLHxYOhvwg3PFaGfFHMFiNmF40dTXt9K96r7sbzc44iLD+VphbMPJEjkMuf8\nPGEFOBzy8pm3wJZw2v32RNW2VESwMYyqDzwHXGSq1a73cS7hEnc79gXlELsK04L9\nQQIDAQAB\n-----END PUBLIC KEY-----\n"
+  }
+
+  @rsa_public_key {
+    :RSAPublicKey,
+    24_650_000_183_914_698_290_885_268_529_673_621_967_457_234_469_123_179_408_466_269_598_577_505_928_170_923_974_132_111_403_341_217_239_999_189_084_572_368_839_502_170_501_850_920_051_662_384_964_248_315_257_926_552_945_648_828_895_432_624_227_029_881_278_113_244_073_644_360_744_504_606_177_648_469_825_063_267_913_017_309_199_785_535_546_734_904_379_798_564_556_494_962_268_682_532_371_146_333_972_821_570_577_277_375_020_977_087_539_994_500_097_107_935_618_711_808_260_846_821_077_839_605_098_669_707_417_692_791_905_543_116_911_754_774_323_678_879_466_618_738_207_538_013_885_607_095_203_516_030_057_611_111_308_904_599_045_146_148_350_745_339_208_006_497_478_057_622_336_882_506_112_530_056_970_653_403_292_123_624_453_213_574_011_183_684_739_084_105_206_483_178_943_532_208_537_215_396_831_110_268_758_639_826_369_857,
+    # credo:disable-for-previous-line Credo.Check.Readability.MaxLineLength
+    65_537
+  }
+
+  defp make_fake_signature(key_id), do: "keyId=\"#{key_id}\""
+
+  defp make_fake_conn(key_id),
+    do: %Plug.Conn{req_headers: %{"signature" => make_fake_signature(key_id <> "#main-key")}}
+
+  describe "fetch_public_key/1" do
+    test "it returns key" do
+      expected_result = {:ok, @rsa_public_key}
+
+      user = insert(:user, %{info: %{source_data: %{"publicKey" => @public_key}}})
+
+      assert Signature.fetch_public_key(make_fake_conn(user.ap_id)) == expected_result
+    end
+
+    test "it returns error when not found user" do
+      assert capture_log(fn ->
+               assert Signature.fetch_public_key(make_fake_conn("test-ap_id")) ==
+                        {:error, :error}
+             end) =~ "[error] Could not decode user"
+    end
+
+    test "it returns error if public key is empty" do
+      user = insert(:user, %{info: %{source_data: %{"publicKey" => %{}}}})
+
+      assert Signature.fetch_public_key(make_fake_conn(user.ap_id)) ==
+               {:error, :error}
+    end
+  end
+
+  describe "refetch_public_key/1" do
+    test "it returns key" do
+      ap_id = "https://mastodon.social/users/lambadalambda"
+
+      assert Signature.refetch_public_key(make_fake_conn(ap_id)) ==
+               {:ok, @rsa_public_key}
+    end
+
+    test "it returns error when not found user" do
+      assert capture_log(fn ->
+               assert Signature.refetch_public_key(make_fake_conn("test-ap_id")) ==
+                        {:error, {:error, :ok}}
+             end) =~ "[error] Could not decode user"
+    end
+  end
+
+  describe "sign/2" do
+    test "it returns signature headers" do
+      user =
+        insert(:user, %{
+          ap_id: "https://mastodon.social/users/lambadalambda",
+          info: %{keys: @private_key}
+        })
+
+      assert Signature.sign(
+               user,
+               %{
+                 host: "test.test",
+                 "content-length": 100
+               }
+             ) ==
+               "keyId=\"https://mastodon.social/users/lambadalambda#main-key\",algorithm=\"rsa-sha256\",headers=\"content-length host\",signature=\"sibUOoqsFfTDerquAkyprxzDjmJm6erYc42W5w1IyyxusWngSinq5ILTjaBxFvfarvc7ci1xAi+5gkBwtshRMWm7S+Uqix24Yg5EYafXRun9P25XVnYBEIH4XQ+wlnnzNIXQkU3PU9e6D8aajDZVp3hPJNeYt1gIPOA81bROI8/glzb1SAwQVGRbqUHHHKcwR8keiR/W2h7BwG3pVRy4JgnIZRSW7fQogKedDg02gzRXwUDFDk0pr2p3q6bUWHUXNV8cZIzlMK+v9NlyFbVYBTHctAR26GIAN6Hz0eV0mAQAePHDY1mXppbA8Gpp6hqaMuYfwifcXmcc+QFm4e+n3A==\""
+    end
+
+    test "it returns error" do
+      user =
+        insert(:user, %{ap_id: "https://mastodon.social/users/lambadalambda", info: %{keys: ""}})
+
+      assert Signature.sign(
+               user,
+               %{host: "test.test", "content-length": 100}
+             ) == {:error, []}
+    end
+  end
+end

test/support/data_case.ex

@@ -42,19 +42,18 @@ defmodule Pleroma.DataCase do
     :ok
   end
 
-  def ensure_local_uploader(_context) do
+  def ensure_local_uploader(context) do
+    test_uploader = Map.get(context, :uploader, Pleroma.Uploaders.Local)
     uploader = Pleroma.Config.get([Pleroma.Upload, :uploader])
     filters = Pleroma.Config.get([Pleroma.Upload, :filters])
 
-    unless uploader == Pleroma.Uploaders.Local || filters != [] do
+    Pleroma.Config.put([Pleroma.Upload, :uploader], test_uploader)
-      Pleroma.Config.put([Pleroma.Upload, :uploader], Pleroma.Uploaders.Local)
     Pleroma.Config.put([Pleroma.Upload, :filters], [])
 
     on_exit(fn ->
       Pleroma.Config.put([Pleroma.Upload, :uploader], uploader)
       Pleroma.Config.put([Pleroma.Upload, :filters], filters)
     end)
-    end
 
     :ok
   end

test/support/http_request_mock.ex

@@ -879,6 +879,42 @@ def get(
      }}
   end
 
+  def get("https://info.pleroma.site/activity.json", _, _, Accept: "application/activity+json") do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/tesla_mock/https__info.pleroma.site_activity.json")
+     }}
+  end
+
+  def get("https://info.pleroma.site/activity.json", _, _, _) do
+    {:ok, %Tesla.Env{status: 404, body: ""}}
+  end
+
+  def get("https://info.pleroma.site/activity2.json", _, _, Accept: "application/activity+json") do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/tesla_mock/https__info.pleroma.site_activity2.json")
+     }}
+  end
+
+  def get("https://info.pleroma.site/activity2.json", _, _, _) do
+    {:ok, %Tesla.Env{status: 404, body: ""}}
+  end
+
+  def get("https://info.pleroma.site/activity3.json", _, _, Accept: "application/activity+json") do
+    {:ok,
+     %Tesla.Env{
+       status: 200,
+       body: File.read!("test/fixtures/tesla_mock/https__info.pleroma.site_activity3.json")
+     }}
+  end
+
+  def get("https://info.pleroma.site/activity3.json", _, _, _) do
+    {:ok, %Tesla.Env{status: 404, body: ""}}
+  end
+
   def get(url, query, body, headers) do
     {:error,
      "Not implemented the mock response for get #{inspect(url)}, #{query}, #{inspect(body)}, #{

test/tasks/config_test.exs

@@ -34,8 +34,8 @@ test "settings are migrated to db" do
 
     Mix.Tasks.Pleroma.Config.run(["migrate_to_db"])
 
-    first_db = Config.get_by_params(%{group: "pleroma", key: "first_setting"})
+    first_db = Config.get_by_params(%{group: "pleroma", key: ":first_setting"})
-    second_db = Config.get_by_params(%{group: "pleroma", key: "second_setting"})
+    second_db = Config.get_by_params(%{group: "pleroma", key: ":second_setting"})
     refute Config.get_by_params(%{group: "pleroma", key: "Pleroma.Repo"})
 
     assert Config.from_binary(first_db.value) == [key: "value", key2: [Pleroma.Repo]]
@@ -45,13 +45,13 @@ test "settings are migrated to db" do
   test "settings are migrated to file and deleted from db", %{temp_file: temp_file} do
     Config.create(%{
       group: "pleroma",
-      key: "setting_first",
+      key: ":setting_first",
       value: [key: "value", key2: [Pleroma.Activity]]
     })
 
     Config.create(%{
       group: "pleroma",
-      key: "setting_second",
+      key: ":setting_second",
       value: [key: "valu2", key2: [Pleroma.Repo]]
     })
 
@@ -61,7 +61,7 @@ test "settings are migrated to file and deleted from db", %{temp_file: temp_file
     assert File.exists?(temp_file)
     {:ok, file} = File.read(temp_file)
 
-    assert file =~ "config :pleroma, setting_first:"
+    assert file =~ "config :pleroma, :setting_first,"
-    assert file =~ "config :pleroma, setting_second:"
+    assert file =~ "config :pleroma, :setting_second,"
   end
 end

test/upload/filter/dedupe_test.exs

@@ -0,0 +1,31 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Upload.Filter.DedupeTest do
+  use Pleroma.DataCase
+
+  alias Pleroma.Upload
+  alias Pleroma.Upload.Filter.Dedupe
+
+  @shasum "e30397b58d226d6583ab5b8b3c5defb0c682bda5c31ef07a9f57c1c4986e3781"
+
+  test "adds shasum" do
+    File.cp!(
+      "test/fixtures/image.jpg",
+      "test/fixtures/image_tmp.jpg"
+    )
+
+    upload = %Upload{
+      name: "an… image.jpg",
+      content_type: "image/jpg",
+      path: Path.absname("test/fixtures/image_tmp.jpg"),
+      tempfile: Path.absname("test/fixtures/image_tmp.jpg")
+    }
+
+    assert {
+             :ok,
+             %Pleroma.Upload{id: @shasum, path: "#{@shasum}.jpg"}
+           } = Dedupe.filter(upload)
+  end
+end

test/upload/filter/mogrifun_test.exs

@@ -0,0 +1,44 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Upload.Filter.MogrifunTest do
+  use Pleroma.DataCase
+  import Mock
+
+  alias Pleroma.Upload
+  alias Pleroma.Upload.Filter
+
+  test "apply mogrify filter" do
+    File.cp!(
+      "test/fixtures/image.jpg",
+      "test/fixtures/image_tmp.jpg"
+    )
+
+    upload = %Upload{
+      name: "an… image.jpg",
+      content_type: "image/jpg",
+      path: Path.absname("test/fixtures/image_tmp.jpg"),
+      tempfile: Path.absname("test/fixtures/image_tmp.jpg")
+    }
+
+    task =
+      Task.async(fn ->
+        assert_receive {:apply_filter, {}}, 4_000
+      end)
+
+    with_mocks([
+      {Mogrify, [],
+       [
+         open: fn _f -> %Mogrify.Image{} end,
+         custom: fn _m, _a -> send(task.pid, {:apply_filter, {}}) end,
+         custom: fn _m, _a, _o -> send(task.pid, {:apply_filter, {}}) end,
+         save: fn _f, _o -> :ok end
+       ]}
+    ]) do
+      assert Filter.Mogrifun.filter(upload) == :ok
+    end
+
+    Task.await(task)
+  end
+end

test/upload/filter/mogrify_test.exs

@@ -0,0 +1,51 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Upload.Filter.MogrifyTest do
+  use Pleroma.DataCase
+  import Mock
+
+  alias Pleroma.Config
+  alias Pleroma.Upload
+  alias Pleroma.Upload.Filter
+
+  setup do
+    filter = Config.get([Filter.Mogrify, :args])
+
+    on_exit(fn ->
+      Config.put([Filter.Mogrify, :args], filter)
+    end)
+  end
+
+  test "apply mogrify filter" do
+    Config.put([Filter.Mogrify, :args], [{"tint", "40"}])
+
+    File.cp!(
+      "test/fixtures/image.jpg",
+      "test/fixtures/image_tmp.jpg"
+    )
+
+    upload = %Upload{
+      name: "an… image.jpg",
+      content_type: "image/jpg",
+      path: Path.absname("test/fixtures/image_tmp.jpg"),
+      tempfile: Path.absname("test/fixtures/image_tmp.jpg")
+    }
+
+    task =
+      Task.async(fn ->
+        assert_receive {:apply_filter, {_, "tint", "40"}}, 4_000
+      end)
+
+    with_mock Mogrify,
+      open: fn _f -> %Mogrify.Image{} end,
+      custom: fn _m, _a -> :ok end,
+      custom: fn m, a, o -> send(task.pid, {:apply_filter, {m, a, o}}) end,
+      save: fn _f, _o -> :ok end do
+      assert Filter.Mogrify.filter(upload) == :ok
+    end
+
+    Task.await(task)
+  end
+end

test/upload/filter_test.exs

@@ -0,0 +1,39 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Upload.FilterTest do
+  use Pleroma.DataCase
+
+  alias Pleroma.Config
+  alias Pleroma.Upload.Filter
+
+  setup do
+    custom_filename = Config.get([Pleroma.Upload.Filter.AnonymizeFilename, :text])
+
+    on_exit(fn ->
+      Config.put([Pleroma.Upload.Filter.AnonymizeFilename, :text], custom_filename)
+    end)
+  end
+
+  test "applies filters" do
+    Config.put([Pleroma.Upload.Filter.AnonymizeFilename, :text], "custom-file.png")
+
+    File.cp!(
+      "test/fixtures/image.jpg",
+      "test/fixtures/image_tmp.jpg"
+    )
+
+    upload = %Pleroma.Upload{
+      name: "an… image.jpg",
+      content_type: "image/jpg",
+      path: Path.absname("test/fixtures/image_tmp.jpg"),
+      tempfile: Path.absname("test/fixtures/image_tmp.jpg")
+    }
+
+    assert Filter.filter([], upload) == {:ok, upload}
+
+    assert {:ok, upload} = Filter.filter([Pleroma.Upload.Filter.AnonymizeFilename], upload)
+    assert upload.name == "custom-file.png"
+  end
+end

test/upload_test.exs

@@ -3,9 +3,106 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.UploadTest do
-  alias Pleroma.Upload
   use Pleroma.DataCase
 
+  import ExUnit.CaptureLog
+
+  alias Pleroma.Upload
+  alias Pleroma.Uploaders.Uploader
+
+  @upload_file %Plug.Upload{
+    content_type: "image/jpg",
+    path: Path.absname("test/fixtures/image_tmp.jpg"),
+    filename: "image.jpg"
+  }
+
+  defmodule TestUploaderBase do
+    def put_file(%{path: path} = _upload, module_name) do
+      task_pid =
+        Task.async(fn ->
+          :timer.sleep(10)
+
+          {Uploader, path}
+          |> :global.whereis_name()
+          |> send({Uploader, self(), {:test}, %{}})
+
+          assert_receive {Uploader, {:test}}, 4_000
+        end)
+
+      Agent.start(fn -> task_pid end, name: module_name)
+
+      :wait_callback
+    end
+  end
+
+  describe "Tried storing a file when http callback response success result" do
+    defmodule TestUploaderSuccess do
+      def http_callback(conn, _params),
+        do: {:ok, conn, {:file, "post-process-file.jpg"}}
+
+      def put_file(upload), do: TestUploaderBase.put_file(upload, __MODULE__)
+    end
+
+    setup do: [uploader: TestUploaderSuccess]
+    setup [:ensure_local_uploader]
+
+    test "it returns file" do
+      File.cp!("test/fixtures/image.jpg", "test/fixtures/image_tmp.jpg")
+
+      assert Upload.store(@upload_file) ==
+               {:ok,
+                %{
+                  "name" => "image.jpg",
+                  "type" => "Document",
+                  "url" => [
+                    %{
+                      "href" => "http://localhost:4001/media/post-process-file.jpg",
+                      "mediaType" => "image/jpeg",
+                      "type" => "Link"
+                    }
+                  ]
+                }}
+
+      Task.await(Agent.get(TestUploaderSuccess, fn task_pid -> task_pid end))
+    end
+  end
+
+  describe "Tried storing a file when http callback response error" do
+    defmodule TestUploaderError do
+      def http_callback(conn, _params), do: {:error, conn, "Errors"}
+
+      def put_file(upload), do: TestUploaderBase.put_file(upload, __MODULE__)
+    end
+
+    setup do: [uploader: TestUploaderError]
+    setup [:ensure_local_uploader]
+
+    test "it returns error" do
+      File.cp!("test/fixtures/image.jpg", "test/fixtures/image_tmp.jpg")
+
+      assert capture_log(fn ->
+               assert Upload.store(@upload_file) == {:error, "Errors"}
+               Task.await(Agent.get(TestUploaderError, fn task_pid -> task_pid end))
+             end) =~
+               "[error] Elixir.Pleroma.Upload store (using Pleroma.UploadTest.TestUploaderError) failed: \"Errors\""
+    end
+  end
+
+  describe "Tried storing a file when http callback doesn't response by timeout" do
+    defmodule(TestUploader, do: def(put_file(_upload), do: :wait_callback))
+    setup do: [uploader: TestUploader]
+    setup [:ensure_local_uploader]
+
+    test "it returns error" do
+      File.cp!("test/fixtures/image.jpg", "test/fixtures/image_tmp.jpg")
+
+      assert capture_log(fn ->
+               assert Upload.store(@upload_file) == {:error, "Uploader callback timeout"}
+             end) =~
+               "[error] Elixir.Pleroma.Upload store (using Pleroma.UploadTest.TestUploader) failed: \"Uploader callback timeout\""
+    end
+  end
+
   describe "Storing a file with the Local uploader" do
     setup [:ensure_local_uploader]
 

test/user_test.exs

@@ -687,10 +687,12 @@ test "it mutes people" do
       muted_user = insert(:user)
 
       refute User.mutes?(user, muted_user)
+      refute User.muted_notifications?(user, muted_user)
 
       {:ok, user} = User.mute(user, muted_user)
 
       assert User.mutes?(user, muted_user)
+      assert User.muted_notifications?(user, muted_user)
     end
 
     test "it unmutes users" do
@@ -701,6 +703,20 @@ test "it unmutes users" do
       {:ok, user} = User.unmute(user, muted_user)
 
       refute User.mutes?(user, muted_user)
+      refute User.muted_notifications?(user, muted_user)
+    end
+
+    test "it mutes user without notifications" do
+      user = insert(:user)
+      muted_user = insert(:user)
+
+      refute User.mutes?(user, muted_user)
+      refute User.muted_notifications?(user, muted_user)
+
+      {:ok, user} = User.mute(user, muted_user, false)
+
+      assert User.mutes?(user, muted_user)
+      refute User.muted_notifications?(user, muted_user)
     end
   end
 
@@ -1294,4 +1310,21 @@ test "without args", %{user: user} do
       assert following == 0
     end
   end
+
+  describe "is_internal_user?/1" do
+    test "non-internal user returns false" do
+      user = insert(:user)
+      refute User.is_internal_user?(user)
+    end
+
+    test "user with no nickname returns true" do
+      user = insert(:user, %{nickname: nil})
+      assert User.is_internal_user?(user)
+    end
+
+    test "user with internal-prefixed nickname returns true" do
+      user = insert(:user, %{nickname: "internal.test"})
+      assert User.is_internal_user?(user)
+    end
+  end
 end

test/web/activity_pub/activity_pub_controller_test.exs

@@ -48,6 +48,17 @@ test "with the relay disabled, it returns 404", %{conn: conn} do
     end
   end
 
+  describe "/internal/fetch" do
+    test "it returns the internal fetch user", %{conn: conn} do
+      res =
+        conn
+        |> get(activity_pub_path(conn, :internal_fetch))
+        |> json_response(200)
+
+      assert res["id"] =~ "/fetch"
+    end
+  end
+
   describe "/users/:nickname" do
     test "it returns a json representation of the user with accept application/json", %{
       conn: conn

test/web/activity_pub/activity_pub_test.exs

@@ -1190,6 +1190,21 @@ test "it can create a Flag activity" do
     end
   end
 
+  test "fetch_activities/2 returns activities addressed to a list " do
+    user = insert(:user)
+    member = insert(:user)
+    {:ok, list} = Pleroma.List.create("foo", user)
+    {:ok, list} = Pleroma.List.follow(list, member)
+
+    {:ok, activity} =
+      CommonAPI.post(user, %{"status" => "foobar", "visibility" => "list:#{list.id}"})
+
+    activity = Repo.preload(activity, :bookmark)
+    activity = %Activity{activity | thread_muted?: !!activity.thread_muted?}
+
+    assert ActivityPub.fetch_activities([], %{"user" => user}) == [activity]
+  end
+
   def data_uri do
     File.read!("test/fixtures/avatar_data_uri")
   end

test/web/activity_pub/mrf/mention_policy_test.exs

@@ -0,0 +1,92 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.MRF.MentionPolicyTest do
+  use Pleroma.DataCase
+
+  alias Pleroma.Web.ActivityPub.MRF.MentionPolicy
+
+  test "pass filter if allow list is empty" do
+    Pleroma.Config.delete([:mrf_mention])
+
+    message = %{
+      "type" => "Create",
+      "to" => ["https://example.com/ok"],
+      "cc" => ["https://example.com/blocked"]
+    }
+
+    assert MentionPolicy.filter(message) == {:ok, message}
+  end
+
+  describe "allow" do
+    test "empty" do
+      Pleroma.Config.put([:mrf_mention], %{actors: ["https://example.com/blocked"]})
+
+      message = %{
+        "type" => "Create"
+      }
+
+      assert MentionPolicy.filter(message) == {:ok, message}
+    end
+
+    test "to" do
+      Pleroma.Config.put([:mrf_mention], %{actors: ["https://example.com/blocked"]})
+
+      message = %{
+        "type" => "Create",
+        "to" => ["https://example.com/ok"]
+      }
+
+      assert MentionPolicy.filter(message) == {:ok, message}
+    end
+
+    test "cc" do
+      Pleroma.Config.put([:mrf_mention], %{actors: ["https://example.com/blocked"]})
+
+      message = %{
+        "type" => "Create",
+        "cc" => ["https://example.com/ok"]
+      }
+
+      assert MentionPolicy.filter(message) == {:ok, message}
+    end
+
+    test "both" do
+      Pleroma.Config.put([:mrf_mention], %{actors: ["https://example.com/blocked"]})
+
+      message = %{
+        "type" => "Create",
+        "to" => ["https://example.com/ok"],
+        "cc" => ["https://example.com/ok2"]
+      }
+
+      assert MentionPolicy.filter(message) == {:ok, message}
+    end
+  end
+
+  describe "deny" do
+    test "to" do
+      Pleroma.Config.put([:mrf_mention], %{actors: ["https://example.com/blocked"]})
+
+      message = %{
+        "type" => "Create",
+        "to" => ["https://example.com/blocked"]
+      }
+
+      assert MentionPolicy.filter(message) == {:reject, nil}
+    end
+
+    test "cc" do
+      Pleroma.Config.put([:mrf_mention], %{actors: ["https://example.com/blocked"]})
+
+      message = %{
+        "type" => "Create",
+        "to" => ["https://example.com/ok"],
+        "cc" => ["https://example.com/blocked"]
+      }
+
+      assert MentionPolicy.filter(message) == {:reject, nil}
+    end
+  end
+end

test/web/activity_pub/transmogrifier_test.exs

@@ -416,6 +416,7 @@ test "it ensures that as:Public activities make it to their followers collection
         |> Map.put("attributedTo", user.ap_id)
         |> Map.put("to", ["https://www.w3.org/ns/activitystreams#Public"])
         |> Map.put("cc", [])
+        |> Map.put("id", user.ap_id <> "/activities/12345678")
 
       data = Map.put(data, "object", object)
 
@@ -439,6 +440,7 @@ test "it ensures that address fields become lists" do
         |> Map.put("attributedTo", user.ap_id)
         |> Map.put("to", nil)
         |> Map.put("cc", nil)
+        |> Map.put("id", user.ap_id <> "/activities/12345678")
 
       data = Map.put(data, "object", object)
 
@@ -1096,6 +1098,18 @@ test "the directMessage flag is present" do
 
       assert modified["directMessage"] == true
     end
+
+    test "it strips BCC field" do
+      user = insert(:user)
+      {:ok, list} = Pleroma.List.create("foo", user)
+
+      {:ok, activity} =
+        CommonAPI.post(user, %{"status" => "foobar", "visibility" => "list:#{list.id}"})
+
+      {:ok, modified} = Transmogrifier.prepare_outgoing(activity.data)
+
+      assert is_nil(modified["bcc"])
+    end
   end
 
   describe "user upgrade" do

test/web/activity_pub/visibilty_test.exs

@@ -16,6 +16,9 @@ defmodule Pleroma.Web.ActivityPub.VisibilityTest do
     following = insert(:user)
     unrelated = insert(:user)
     {:ok, following} = Pleroma.User.follow(following, user)
+    {:ok, list} = Pleroma.List.create("foo", user)
+
+    Pleroma.List.follow(list, unrelated)
 
     {:ok, public} =
       CommonAPI.post(user, %{"status" => "@#{mentioned.nickname}", "visibility" => "public"})
@@ -29,6 +32,12 @@ defmodule Pleroma.Web.ActivityPub.VisibilityTest do
     {:ok, unlisted} =
       CommonAPI.post(user, %{"status" => "@#{mentioned.nickname}", "visibility" => "unlisted"})
 
+    {:ok, list} =
+      CommonAPI.post(user, %{
+        "status" => "@#{mentioned.nickname}",
+        "visibility" => "list:#{list.id}"
+      })
+
     %{
       public: public,
       private: private,
@@ -37,29 +46,65 @@ defmodule Pleroma.Web.ActivityPub.VisibilityTest do
       user: user,
       mentioned: mentioned,
       following: following,
-      unrelated: unrelated
+      unrelated: unrelated,
+      list: list
     }
   end
 
-  test "is_direct?", %{public: public, private: private, direct: direct, unlisted: unlisted} do
+  test "is_direct?", %{
+    public: public,
+    private: private,
+    direct: direct,
+    unlisted: unlisted,
+    list: list
+  } do
     assert Visibility.is_direct?(direct)
     refute Visibility.is_direct?(public)
     refute Visibility.is_direct?(private)
     refute Visibility.is_direct?(unlisted)
+    assert Visibility.is_direct?(list)
   end
 
-  test "is_public?", %{public: public, private: private, direct: direct, unlisted: unlisted} do
+  test "is_public?", %{
+    public: public,
+    private: private,
+    direct: direct,
+    unlisted: unlisted,
+    list: list
+  } do
     refute Visibility.is_public?(direct)
     assert Visibility.is_public?(public)
     refute Visibility.is_public?(private)
     assert Visibility.is_public?(unlisted)
+    refute Visibility.is_public?(list)
   end
 
-  test "is_private?", %{public: public, private: private, direct: direct, unlisted: unlisted} do
+  test "is_private?", %{
+    public: public,
+    private: private,
+    direct: direct,
+    unlisted: unlisted,
+    list: list
+  } do
     refute Visibility.is_private?(direct)
     refute Visibility.is_private?(public)
     assert Visibility.is_private?(private)
     refute Visibility.is_private?(unlisted)
+    refute Visibility.is_private?(list)
+  end
+
+  test "is_list?", %{
+    public: public,
+    private: private,
+    direct: direct,
+    unlisted: unlisted,
+    list: list
+  } do
+    refute Visibility.is_list?(direct)
+    refute Visibility.is_list?(public)
+    refute Visibility.is_list?(private)
+    refute Visibility.is_list?(unlisted)
+    assert Visibility.is_list?(list)
   end
 
   test "visible_for_user?", %{
@@ -70,7 +115,8 @@ test "visible_for_user?", %{
     user: user,
     mentioned: mentioned,
     following: following,
-    unrelated: unrelated
+    unrelated: unrelated,
+    list: list
   } do
     # All visible to author
 
@@ -78,6 +124,7 @@ test "visible_for_user?", %{
     assert Visibility.visible_for_user?(private, user)
     assert Visibility.visible_for_user?(unlisted, user)
     assert Visibility.visible_for_user?(direct, user)
+    assert Visibility.visible_for_user?(list, user)
 
     # All visible to a mentioned user
 
@@ -85,6 +132,7 @@ test "visible_for_user?", %{
     assert Visibility.visible_for_user?(private, mentioned)
     assert Visibility.visible_for_user?(unlisted, mentioned)
     assert Visibility.visible_for_user?(direct, mentioned)
+    assert Visibility.visible_for_user?(list, mentioned)
 
     # DM not visible for just follower
 
@@ -92,6 +140,7 @@ test "visible_for_user?", %{
     assert Visibility.visible_for_user?(private, following)
     assert Visibility.visible_for_user?(unlisted, following)
     refute Visibility.visible_for_user?(direct, following)
+    refute Visibility.visible_for_user?(list, following)
 
     # Public and unlisted visible for unrelated user
 
@@ -99,6 +148,9 @@ test "visible_for_user?", %{
     assert Visibility.visible_for_user?(unlisted, unrelated)
     refute Visibility.visible_for_user?(private, unrelated)
     refute Visibility.visible_for_user?(direct, unrelated)
+
+    # Visible for a list member
+    assert Visibility.visible_for_user?(list, unrelated)
   end
 
   test "doesn't die when the user doesn't exist",
@@ -115,18 +167,24 @@ test "get_visibility", %{
     public: public,
     private: private,
     direct: direct,
-    unlisted: unlisted
+    unlisted: unlisted,
+    list: list
   } do
     assert Visibility.get_visibility(public) == "public"
     assert Visibility.get_visibility(private) == "private"
     assert Visibility.get_visibility(direct) == "direct"
     assert Visibility.get_visibility(unlisted) == "unlisted"
+    assert Visibility.get_visibility(list) == "list"
   end
 
   test "get_visibility with directMessage flag" do
     assert Visibility.get_visibility(%{data: %{"directMessage" => true}}) == "direct"
   end
 
+  test "get_visibility with listMessage flag" do
+    assert Visibility.get_visibility(%{data: %{"listMessage" => ""}}) == "list"
+  end
+
   describe "entire_thread_visible_for_user?/2" do
     test "returns false if not found activity", %{user: user} do
       refute Visibility.entire_thread_visible_for_user?(%Activity{}, user)

test/web/admin_api/admin_api_controller_test.exs

@@ -1720,7 +1720,7 @@ test "settings with nesting map", %{conn: conn} do
           configs: [
             %{
               "group" => "pleroma",
-              "key" => "key1",
+              "key" => ":key1",
               "value" => [
                 %{"tuple" => [":key2", "some_val"]},
                 %{
@@ -1750,7 +1750,7 @@ test "settings with nesting map", %{conn: conn} do
                  "configs" => [
                    %{
                      "group" => "pleroma",
-                     "key" => "key1",
+                     "key" => ":key1",
                      "value" => [
                        %{"tuple" => [":key2", "some_val"]},
                        %{
@@ -1782,7 +1782,7 @@ test "value as map", %{conn: conn} do
           configs: [
             %{
               "group" => "pleroma",
-              "key" => "key1",
+              "key" => ":key1",
               "value" => %{"key" => "some_val"}
             }
           ]
@@ -1793,7 +1793,7 @@ test "value as map", %{conn: conn} do
                  "configs" => [
                    %{
                      "group" => "pleroma",
-                     "key" => "key1",
+                     "key" => ":key1",
                      "value" => %{"key" => "some_val"}
                    }
                  ]
@@ -1862,6 +1862,45 @@ test "dispatch setting", %{conn: conn} do
                ]
              }
     end
+
+    test "queues key as atom", %{conn: conn} do
+      conn =
+        post(conn, "/api/pleroma/admin/config", %{
+          configs: [
+            %{
+              "group" => "pleroma_job_queue",
+              "key" => ":queues",
+              "value" => [
+                %{"tuple" => [":federator_incoming", 50]},
+                %{"tuple" => [":federator_outgoing", 50]},
+                %{"tuple" => [":web_push", 50]},
+                %{"tuple" => [":mailer", 10]},
+                %{"tuple" => [":transmogrifier", 20]},
+                %{"tuple" => [":scheduled_activities", 10]},
+                %{"tuple" => [":background", 5]}
+              ]
+            }
+          ]
+        })
+
+      assert json_response(conn, 200) == %{
+               "configs" => [
+                 %{
+                   "group" => "pleroma_job_queue",
+                   "key" => ":queues",
+                   "value" => [
+                     %{"tuple" => [":federator_incoming", 50]},
+                     %{"tuple" => [":federator_outgoing", 50]},
+                     %{"tuple" => [":web_push", 50]},
+                     %{"tuple" => [":mailer", 10]},
+                     %{"tuple" => [":transmogrifier", 20]},
+                     %{"tuple" => [":scheduled_activities", 10]},
+                     %{"tuple" => [":background", 5]}
+                   ]
+                 }
+               ]
+             }
+    end
   end
 end