Merge branch 'develop' of https://git.pleroma.social/pleroma/pleroma into develop

CHANGELOG.md

@@ -4,14 +4,25 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
+### Added
+- Refreshing poll results for remote polls
+- Admin API: Add ability to require password reset
+
+### Changed
+- **Breaking:** Elixir >=1.8 is now required (was >= 1.7)
+- Replaced [pleroma_job_queue](https://git.pleroma.social/pleroma/pleroma_job_queue) and `Pleroma.Web.Federator.RetryQueue` with [Oban](https://github.com/sorentwo/oban) (see [`docs/config.md`](docs/config.md) on migrating customized worker / retry settings)
+- Introduced [quantum](https://github.com/quantum-elixir/quantum-core) job scheduler
+- Admin API: Return `total` when querying for reports
+- Mastodon API: Return `pleroma.direct_conversation_id` when creating a direct message (`POST /api/v1/statuses`)
+### Fixed
+- Mastodon API: Fix private and direct statuses not being filtered out from the public timeline for an authenticated user (`GET /api/v1/timelines/public`)
+
+## [1.1.0] - 2019-??-??
 ### Security
-- OStatus: eliminate the possibility of a protocol downgrade attack.
-- OStatus: prevent following locked accounts, bypassing the approval process.
 - Mastodon API: respect post privacy in `/api/v1/statuses/:id/{favourited,reblogged}_by`
 
 ### Removed
 - **Breaking:** GNU Social API with Qvitter extensions support
-- **Breaking:** ActivityPub: The `accept_blocks` configuration setting.
 - Emoji: Remove longfox emojis.
 - Remove `Reply-To` header from report emails for admins.
 
@@ -19,9 +30,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - **Breaking:** Configuration: A setting to explicitly disable the mailer was added, defaulting to true, if you are using a mailer add `config :pleroma, Pleroma.Emails.Mailer, enabled: true` to your config
 - **Breaking:** Configuration: `/media/` is now removed when `base_url` is configured, append `/media/` to your `base_url` config to keep the old behaviour if desired
 - **Breaking:** `/api/pleroma/notifications/read` is moved to `/api/v1/pleroma/notifications/read` and now supports `max_id` and responds with Mastodon API entities.
+- **Breaking:** `/api/pleroma/admin/users/invite_token` now uses `POST`, changed accepted params and returns full invite in json instead of only token string.
+- Configuration: added `config/description.exs`, from which `docs/config.md` is generated
 - Configuration: OpenGraph and TwitterCard providers enabled by default
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
-- Configuration: added `config/description.exs`, from which `docs/config.md` is generated
+- Mastodon API: `pleroma.thread_muted` key in the Status entity
 - Federation: Return 403 errors when trying to request pages from a user's follower/following collections if they have `hide_followers`/`hide_follows` set
 - NodeInfo: Return `skipThreadContainment` in `metadata` for the `skip_thread_containment` option
 - NodeInfo: Return `mailerEnabled` in `metadata`
@@ -30,23 +43,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - AdminAPI: Add "godmode" while fetching user statuses (i.e. admin can see private statuses)
 - Improve digest email template
 – Pagination: (optional) return `total` alongside with `items` when paginating
-- Replaced [pleroma_job_queue](https://git.pleroma.social/pleroma/pleroma_job_queue) and `Pleroma.Web.Federator.RetryQueue` with [Oban](https://github.com/sorentwo/oban) (see [`docs/config.md`](docs/config.md) on migrating customized worker / retry settings)
+- Add `rel="ugc"` to all links in statuses, to prevent SEO spam
-- Introduced [quantum](https://github.com/quantum-elixir/quantum-core) job scheduler
 
 ### Fixed
 - Following from Osada
-- Not being able to pin unlisted posts
-- Objects being re-embedded to activities after being updated (e.g faved/reposted). Running 'mix pleroma.database prune_objects' again is advised.
 - Favorites timeline doing database-intensive queries
 - Metadata rendering errors resulting in the entire page being inaccessible
 - `federation_incoming_replies_max_depth` option being ignored in certain cases
-- Federation/MediaProxy not working with instances that have wrong certificate order
 - Mastodon API: Handling of search timeouts (`/api/v1/search` and `/api/v2/search`)
 - Mastodon API: Misskey's endless polls being unable to render
 - Mastodon API: Embedded relationships not being properly rendered in the Account entity of Status entity
 - Mastodon API: Notifications endpoint crashing if one notification failed to render
-- Mastodon API: follower/following counters not being nullified, when `hide_follows`/`hide_followers` is set
-- Mastodon API: `muted` in the Status entity, using author's account to determine if the tread was muted
 - Mastodon API: Add `account_id`, `type`, `offset`, and `limit` to search API (`/api/v1/search` and `/api/v2/search`)
 - Mastodon API, streaming: Fix filtering of notifications based on blocks/mutes/thread mutes
 - ActivityPub C2S: follower/following collection pages being inaccessible even when authentifucated if `hide_followers`/ `hide_follows` was set
@@ -54,15 +61,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Rich Media: Parser failing when no TTL can be found by image TTL setters
 - Rich Media: The crawled URL is now spliced into the rich media data.
 - ActivityPub S2S: sharedInbox usage has been mostly aligned with the rules in the AP specification.
-- ActivityPub S2S: remote user deletions now work the same as local user deletions.
-- ActivityPub S2S: POST requests are now signed with `(request-target)` pseudo-header.
-- Not being able to access the Mastodon FE login page on private instances
-- Invalid SemVer version generation, when the current branch does not have commits ahead of tag/checked out on a tag
 - Pleroma.Upload base_url was not automatically whitelisted by MediaProxy. Now your custom CDN or file hosting will be accessed directly as expected.
 - Report email not being sent to admins when the reporter is a remote user
-- MRF: ensure that subdomain_match calls are case-insensitive
 - Reverse Proxy limiting `max_body_length` was incorrectly defined and only checked `Content-Length` headers which may not be sufficient in some circumstances
-- MRF: fix use of unserializable keyword lists in describe() implementations
 - ActivityPub: Deactivated user deletion
 - ActivityPub: Fix `/users/:nickname/inbox` crashing without an authenticated user
 - MRF: fix ability to follow a relay when AntiFollowbotPolicy was enabled
@@ -73,16 +74,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: all status JSON responses contain a `pleroma.expires_at` item which states when an activity will expire. The value is only shown to the user who created the activity. To everyone else it's empty.
 - Configuration: `ActivityExpiration.enabled` controls whether expired activites will get deleted at the appropriate time. Enabled by default.
 - Conversations: Add Pleroma-specific conversation endpoints and status posting extensions. Run the `bump_all_conversations` task again to create the necessary data.
-- **Breaking:** MRF describe API, which adds support for exposing configuration information about MRF policies to NodeInfo.
-  Custom modules will need to be updated by adding, at the very least, `def describe, do: {:ok, %{}}` to the MRF policy modules.
 - MRF: Support for priming the mediaproxy cache (`Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`)
 - MRF: Support for excluding specific domains from Transparency.
 - MRF: Support for filtering posts based on who they mention (`Pleroma.Web.ActivityPub.MRF.MentionPolicy`)
-- MRF: Support for filtering posts based on ActivityStreams vocabulary (`Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`)
-- MRF (Simple Policy): Support for wildcard domains.
-- Support for wildcard domains in user domain blocks setting.
-- Configuration: `quarantined_instances` support wildcard domains.
-- Configuration: `federation_incoming_replies_max_depth` option
 - Mastodon API: Support for the [`tagged` filter](https://github.com/tootsuite/mastodon/pull/9755) in [`GET /api/v1/accounts/:id/statuses`](https://docs.joinmastodon.org/api/rest/accounts/#get-api-v1-accounts-id-statuses)
 - Mastodon API, streaming: Add support for passing the token in the `Sec-WebSocket-Protocol` header
 - Mastodon API, extension: Ability to reset avatar, profile banner, and background
@@ -94,6 +88,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: added `/auth/password` endpoint for password reset with rate limit.
 - Mastodon API: /api/v1/accounts/:id/statuses now supports nicknames or user id
 - Mastodon API: Improve support for the user profile custom fields
+- Mastodon API: follower/following counters are nullified when `hide_follows`/`hide_followers` and `hide_follows_count`/`hide_followers_count` are set
 - Admin API: Return users' tags when querying reports
 - Admin API: Return avatar and display name when querying users
 - Admin API: Allow querying user by ID
@@ -110,9 +105,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Admin API: Endpoint for fetching latest user's statuses
 - Pleroma API: Add `/api/v1/pleroma/accounts/confirmation_resend?email=<email>` for resending account confirmation.
 - Pleroma API: Email change endpoint.
-- Relays: Added a task to list relay subscriptions.
-- Mix Tasks: `mix pleroma.database fix_likes_collections`
-- Federation: Remove `likes` from objects.
 - Admin API: Added moderation log
 - Web response cache (currently, enabled for ActivityPub)
 - Mastodon API: Added an endpoint to get multiple statuses by IDs (`GET /api/v1/statuses/?ids[]=1&ids[]=2`)
@@ -123,6 +115,61 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - RichMedia: parsers and their order are configured in `rich_media` config.
 - RichMedia: add the rich media ttl based on image expiration time.
 
+## [1.0.6] - 2019-08-14
+### Fixed
+- MRF: fix use of unserializable keyword lists in describe() implementations
+- ActivityPub S2S: POST requests are now signed with `(request-target)` pseudo-header.
+
+## [1.0.5] - 2019-08-13
+### Fixed
+- Mastodon API: follower/following counters not being nullified, when `hide_follows`/`hide_followers` is set
+- Mastodon API: `muted` in the Status entity, using author's account to determine if the thread was muted
+- Mastodon API: return the actual profile URL in the Account entity's `url` property when appropriate
+- Templates: properly style anchor tags
+- Objects being re-embedded to activities after being updated (e.g faved/reposted). Running 'mix pleroma.database prune_objects' again is advised.
+- Not being able to access the Mastodon FE login page on private instances
+- MRF: ensure that subdomain_match calls are case-insensitive
+- Fix internal server error when using the healthcheck API.
+
+### Added
+- **Breaking:** MRF describe API, which adds support for exposing configuration information about MRF policies to NodeInfo.
+  Custom modules will need to be updated by adding, at the very least, `def describe, do: {:ok, %{}}` to the MRF policy modules.
+- Relays: Added a task to list relay subscriptions.
+- MRF: Support for filtering posts based on ActivityStreams vocabulary (`Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`)
+- MRF (Simple Policy): Support for wildcard domains.
+- Support for wildcard domains in user domain blocks setting.
+- Configuration: `quarantined_instances` support wildcard domains.
+- Mix Tasks: `mix pleroma.database fix_likes_collections`
+- Configuration: `federation_incoming_replies_max_depth` option
+
+### Removed
+- Federation: Remove `likes` from objects.
+- **Breaking:** ActivityPub: The `accept_blocks` configuration setting.
+
+## [1.0.4] - 2019-08-01
+### Fixed
+- Invalid SemVer version generation, when the current branch does not have commits ahead of tag/checked out on a tag
+
+## [1.0.3] - 2019-07-31
+### Security
+- OStatus: eliminate the possibility of a protocol downgrade attack.
+- OStatus: prevent following locked accounts, bypassing the approval process.
+- TwitterAPI: use CommonAPI to handle remote follows instead of OStatus.
+
+## [1.0.2] - 2019-07-28
+### Fixed
+- Not being able to pin unlisted posts
+- Mastodon API: represent poll IDs as strings
+- MediaProxy: fix matching filenames
+- MediaProxy: fix filename encoding
+- Migrations: fix a sporadic migration failure
+- Metadata rendering errors resulting in the entire page being inaccessible
+- Federation/MediaProxy not working with instances that have wrong certificate order
+- ActivityPub S2S: remote user deletions now work the same as local user deletions.
+
+### Changed
+- Configuration: OpenGraph and TwitterCard providers enabled by default
+- Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
 
 ## [1.0.1] - 2019-07-14
 ### Security

config/config.exs

@@ -109,6 +109,7 @@
 
 config :pleroma, Pleroma.Uploaders.S3,
   bucket: nil,
+  streaming_enabled: true,
   public_endpoint: "https://s3.amazonaws.com"
 
 config :pleroma, Pleroma.Uploaders.MDII,
@@ -122,7 +123,8 @@
     # Put groups that have higher priority than defaults here. Example in `docs/config/custom_emoji.md`
     Custom: ["/emoji/*.png", "/emoji/**/*.png"]
   ],
-  default_manifest: "https://git.pleroma.social/pleroma/emoji-index/raw/master/index.json"
+  default_manifest: "https://git.pleroma.social/pleroma/emoji-index/raw/master/index.json",
+  shared_pack_cache_seconds_per_file: 60
 
 config :pleroma, :uri_schemes,
   valid_schemes: [
@@ -507,7 +509,7 @@
     class: false,
     strip_prefix: false,
     new_window: false,
-    rel: false
+    rel: "ugc"
   ]
 
 config :pleroma, :ldap,

config/test.exs

@@ -30,7 +30,8 @@
   notify_email: "noreply@example.com",
   skip_thread_containment: false,
   federating: false,
-  external_user_synchronization: false
+  external_user_synchronization: false,
+  static_dir: "test/instance_static/"
 
 config :pleroma, :activitypub, sign_object_fetches: false
 

docs/api/admin_api.md

@@ -224,15 +224,25 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 
 ## `/api/pleroma/admin/users/invite_token`
 
-### Get an account registration invite token
+### Create an account registration invite token
 
-- Methods: `GET`
+- Methods: `POST`
 - Params:
-  - *optional* `invite` => [
   - *optional* `max_use` (integer)
   - *optional* `expires_at` (date string e.g. "2019-04-07")
-  ]
+- Response:
-- Response: invite token (base64 string)
+
+```json
+{
+  "id": integer,
+  "token": string,
+  "used": boolean,
+  "expires_at": date,
+  "uses": integer,
+  "max_use": integer,
+  "invite_type": string (possible values: `one_time`, `reusable`, `date_limited`, `reusable_date_limited`)
+}
+```
 
 ## `/api/pleroma/admin/users/invites`
 
@@ -300,6 +310,14 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 - Params: none
 - Response: password reset token (base64 string)
 
+## `/api/pleroma/admin/users/:nickname/force_password_reset`
+
+### Force passord reset for a user with a given nickname
+
+- Methods: `PATCH`
+- Params: none
+- Response: none (code `204`)
+
 ## `/api/pleroma/admin/reports`
 ### Get a list of reports
 - Method `GET`
@@ -317,6 +335,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 
 ```json
 {
+  "total" : 1,
   "reports": [
     {
       "account": {
@@ -722,3 +741,10 @@ Compile time settings (need instance reboot):
   }
 ]
 ```
+
+## `POST /api/pleroma/admin/reload_emoji`
+### Reload the instance's custom emoji
+* Method `POST`
+* Authentication: required
+* Params: None
+* Response: JSON, "ok" and 200 status

docs/api/differences_in_mastoapi_responses.md

@@ -21,7 +21,8 @@ Adding the parameter `with_muted=true` to the timeline queries will also return
 Has these additional fields under the `pleroma` object:
 
 - `local`: true if the post was made on the local instance
-- `conversation_id`: the ID of the conversation the status is associated with (if any)
+- `conversation_id`: the ID of the AP context the status is associated with (if any)
+- `direct_conversation_id`: the ID of the Mastodon direct message conversation the status is associated with (if any)
 - `in_reply_to_account_acct`: the `acct` property of User entity for replied user (if any)
 - `content`: a map consisting of alternate representations of the `content` property with the key being it's mimetype. Currently the only alternate representation supported is `text/plain`
 - `spoiler_text`: a map consisting of alternate representations of the `spoiler_text` property with the key being it's mimetype. Currently the only alternate representation supported is `text/plain`
@@ -50,6 +51,8 @@ Has these additional fields under the `pleroma` object:
 - `confirmation_pending`: boolean, true if a new user account is waiting on email confirmation to be activated
 - `hide_followers`: boolean, true when the user has follower hiding enabled
 - `hide_follows`: boolean, true when the user has follow hiding enabled
+- `hide_followers_count`: boolean, true when the user has follower stat hiding enabled
+- `hide_follows_count`: boolean, true when the user has follow stat hiding enabled
 - `settings_store`: A generic map of settings for frontends. Opaque to the backend. Only returned in `verify_credentials` and `update_credentials`
 - `chat_token`: The token needed for Pleroma chat. Only returned in `verify_credentials`
 - `deactivated`: boolean, true when the user is deactivated
@@ -112,6 +115,8 @@ Additional parameters can be added to the JSON body/Form data:
 - `no_rich_text` - if true, html tags are stripped from all statuses requested from the API
 - `hide_followers` - if true, user's followers will be hidden
 - `hide_follows` - if true, user's follows will be hidden
+- `hide_followers_count` - if true, user's follower count will be hidden
+- `hide_follows_count` - if true, user's follow count will be hidden
 - `hide_favorites` - if true, user's favorites timeline will be hidden
 - `show_role` - if true, user's role (e.g admin, moderator) will be exposed to anyone in the API
 - `default_scope` - the scope returned under `privacy` key in Source subentity

docs/api/pleroma_api.md

@@ -365,3 +365,68 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 * Params:
     * `recipients`: A list of ids of users that should receive posts to this conversation. This will replace the current list of recipients, so submit the full list. The owner of owner of the conversation will always be part of the set of recipients, though.
 * Response: JSON, statuses (200 - healthy, 503 unhealthy)
+
+## `GET /api/pleroma/emoji/packs`
+### Lists the custom emoji packs on the server
+* Method `GET`
+* Authentication: not required
+* Params: None
+* Response: JSON, "ok" and 200 status and the JSON hashmap of "pack name" to "pack contents"
+
+## `PUT /api/pleroma/emoji/packs/:name`
+### Creates an empty custom emoji pack
+* Method `PUT`
+* Authentication: required
+* Params: None
+* Response: JSON, "ok" and 200 status or 409 if the pack with that name already exists
+
+## `DELETE /api/pleroma/emoji/packs/:name`
+### Delete a custom emoji pack
+* Method `DELETE`
+* Authentication: required
+* Params: None
+* Response: JSON, "ok" and 200 status or 500 if there was an error deleting the pack
+
+## `POST /api/pleroma/emoji/packs/:name/update_file`
+### Update a file in a custom emoji pack
+* Method `POST`
+* Authentication: required
+* Params: 
+    * if the `action` is `add`, adds an emoji named `shortcode` to the pack `pack_name`,
+      that means that the emoji file needs to be uploaded with the request
+      (thus requiring it to be a multipart request) and be named `file`.
+      There can also be an optional `filename` that will be the new emoji file name
+      (if it's not there, the name will be taken from the uploaded file).
+    * if the `action` is `update`, changes emoji shortcode
+      (from `shortcode` to `new_shortcode` or moves the file (from the current filename to `new_filename`)
+    * if the `action` is `remove`, removes the emoji named `shortcode` and it's associated file
+* Response: JSON, updated "files" section of the pack and 200 status, 409 if the trying to use a shortcode
+  that is already taken, 400 if there was an error with the shortcode, filename or file (additional info
+  in the "error" part of the response JSON)
+
+## `POST /api/pleroma/emoji/packs/:name/update_metadata`
+### Updates (replaces) pack metadata
+* Method `POST`
+* Authentication: required
+* Params: 
+  * `new_data`: new metadata to replace the old one
+* Response: JSON, updated "metadata" section of the pack and 200 status or 400 if there was a
+  problem with the new metadata (the error is specified in the "error" part of the response JSON)
+
+## `POST /api/pleroma/emoji/packs/download_from`
+### Requests the instance to download the pack from another instance
+* Method `POST`
+* Authentication: required
+* Params: 
+  * `instance_address`: the address of the instance to download from
+  * `pack_name`: the pack to download from that instance
+* Response: JSON, "ok" and 200 status if the pack was downloaded, or 500 if there were
+  errors downloading the pack
+
+## `GET /api/pleroma/emoji/packs/:name/download_shared`
+### Requests a local pack from the instance
+* Method `GET`
+* Authentication: not required
+* Params: None
+* Response: the archive of the pack with a 200 status code, 403 if the pack is not set as shared,
+  404 if the pack does not exist

docs/clients.md

@@ -39,7 +39,7 @@ Feel free to contact us to be added to this list!
 
 ### Nekonium
 - Homepage: [F-Droid Repository](https://repo.gdgd.jp.net/), [Google Play](https://play.google.com/store/apps/details?id=com.apps.nekonium), [Amazon](https://www.amazon.co.jp/dp/B076FXPRBC/)
-- Source: <https://git.gdgd.jp.net/lin/nekonium/>
+- Source: <https://gogs.gdgd.jp.net/lin/nekonium>
 - Contact: [@lin@pleroma.gdgd.jp.net](https://pleroma.gdgd.jp.net/users/lin)
 - Platforms: Android
 - Features: Streaming Ready
@@ -67,7 +67,7 @@ Feel free to contact us to be added to this list!
 ## Alternative Web Interfaces
 ### Brutaldon
 - Homepage: <https://jfm.carcosa.net/projects/software/brutaldon/>
-- Source Code: <https://github.com/jfmcbrayer/brutaldon>
+- Source Code: <https://git.carcosa.net/jmcbray/brutaldon>
 - Contact: [@gcupc@glitch.social](https://glitch.social/users/gcupc)
 - Features: No Streaming
 

docs/config.md

@@ -23,6 +23,7 @@ Note: `strip_exif` has been replaced by `Pleroma.Upload.Filter.Mogrify`.
 * `truncated_namespace`: If you use S3 compatible service such as Digital Ocean Spaces or CDN, set folder name or "" etc.
 For example, when using CDN to S3 virtual host format, set "".
 At this time, write CNAME to CDN in public_endpoint.
+* `streaming_enabled`: Enable streaming uploads, when enabled the file will be sent to the server in chunks as it's being read. This may be unsupported by some providers, try disabling this if you have upload problems.
 
 ## Pleroma.Upload.Filter.Mogrify
 
@@ -521,7 +522,7 @@ config :auto_linker,
     class: false,
     strip_prefix: false,
     new_window: false,
-    rel: false
+    rel: "ugc"
   ]
 ```
 
@@ -707,6 +708,8 @@ Configure OAuth 2 provider capabilities:
 * `pack_extensions`: A list of file extensions for emojis, when no emoji.txt for a pack is present. Example `[".png", ".gif"]`
 * `groups`: Emojis are ordered in groups (tags). This is an array of key-value pairs where the key is the groupname and the value the location or array of locations. `*` can be used as a wildcard. Example `[Custom: ["/emoji/*.png", "/emoji/custom/*.png"]]`
 * `default_manifest`: Location of the JSON-manifest. This manifest contains information about the emoji-packs you can download. Currently only one manifest can be added (no arrays).
+* `shared_pack_cache_seconds_per_file`: When an emoji pack is shared, the archive is created and cached in
+  memory for this amount of seconds multiplied by the number of files.
 
 ## Database options
 

installation/pleroma-mongooseim.cfg

@@ -215,7 +215,9 @@
       ]}
   ]},
 
-  { 5222, ejabberd_c2s, [
+  %% If you want dual stack, you have to clone this entire config stanza
+  %% and change the bind to "::"
+  { {5222, "0.0.0.0"}, ejabberd_c2s, [
 
 			%%
 			%% If TLS is compiled in and you installed a SSL
@@ -246,7 +248,9 @@
   %%			{max_stanza_size, 65536}
   %%		       ]},
 
-  { 5269, ejabberd_s2s_in, [
+  %% If you want dual stack, you have to clone this entire config stanza
+  %% and change the bind to "::"
+  { {5269, "0.0.0.0"}, ejabberd_s2s_in, [
 			   {shaper, s2s_shaper},
 			   {max_stanza_size, 131072},
 			   {protocol_options, ["no_sslv3"]}

lib/mix/pleroma.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Pleroma do

lib/mix/tasks/pleroma/database.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Database do

lib/mix/tasks/pleroma/ecto/ecto.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-onl
 
 defmodule Mix.Tasks.Pleroma.Ecto do

lib/mix/tasks/pleroma/ecto/migrate.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-onl
 
 defmodule Mix.Tasks.Pleroma.Ecto.Migrate do

lib/mix/tasks/pleroma/ecto/rollback.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-onl
 
 defmodule Mix.Tasks.Pleroma.Ecto.Rollback do

lib/mix/tasks/pleroma/emoji.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Emoji do
@@ -235,7 +235,7 @@ def run(["gen-pack", src]) do
         cwd: tmp_pack_dir
       )
 
-    emoji_map = Pleroma.Emoji.make_shortcode_to_file_map(tmp_pack_dir, exts)
+    emoji_map = Pleroma.Emoji.Loader.make_shortcode_to_file_map(tmp_pack_dir, exts)
 
     File.write!(files_name, Jason.encode!(emoji_map, pretty: true))
 

lib/mix/tasks/pleroma/instance.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Instance do

lib/mix/tasks/pleroma/relay.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Relay do

lib/mix/tasks/pleroma/uploads.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Uploads do

lib/mix/tasks/pleroma/user.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.User do

lib/pleroma/activity/queries.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Activity.Queries do

lib/pleroma/application.ex

@@ -43,23 +43,9 @@ def start(_type, _args) do
         hackney_pool_children() ++
         [
           Pleroma.Stats,
-          {Oban, Pleroma.Config.get(Oban)},
+          {Oban, Pleroma.Config.get(Oban)}
-          %{
-            id: :web_push_init,
-            start: {Task, :start_link, [&Pleroma.Web.Push.init/0]},
-            restart: :temporary
-          },
-          %{
-            id: :federator_init,
-            start: {Task, :start_link, [&Pleroma.Web.Federator.init/0]},
-            restart: :temporary
-          },
-          %{
-            id: :internal_fetch_init,
-            start: {Task, :start_link, [&Pleroma.Web.ActivityPub.InternalFetchActor.init/0]},
-            restart: :temporary
-          }
         ] ++
+        task_children(@env) ++
         oauth_cleanup_child(oauth_cleanup_enabled?()) ++
         streamer_child(@env) ++
         chat_child(@env, chat_enabled?()) ++
@@ -116,10 +102,14 @@ defp cachex_children do
       build_cachex("rich_media", default_ttl: :timer.minutes(120), limit: 5000),
       build_cachex("scrubber", limit: 2500),
       build_cachex("idempotency", expiration: idempotency_expiration(), limit: 2500),
-      build_cachex("web_resp", limit: 2500)
+      build_cachex("web_resp", limit: 2500),
+      build_cachex("emoji_packs", expiration: emoji_packs_expiration(), limit: 10)
     ]
   end
 
+  defp emoji_packs_expiration,
+    do: expiration(default: :timer.seconds(5 * 60), interval: :timer.seconds(60))
+
   defp idempotency_expiration,
     do: expiration(default: :timer.seconds(6 * 60 * 60), interval: :timer.seconds(60))
 
@@ -163,4 +153,39 @@ defp hackney_pool_children do
       :hackney_pool.child_spec(pool, options)
     end
   end
+
+  defp task_children(:test) do
+    [
+      %{
+        id: :web_push_init,
+        start: {Task, :start_link, [&Pleroma.Web.Push.init/0]},
+        restart: :temporary
+      },
+      %{
+        id: :federator_init,
+        start: {Task, :start_link, [&Pleroma.Web.Federator.init/0]},
+        restart: :temporary
+      }
+    ]
+  end
+
+  defp task_children(_) do
+    [
+      %{
+        id: :web_push_init,
+        start: {Task, :start_link, [&Pleroma.Web.Push.init/0]},
+        restart: :temporary
+      },
+      %{
+        id: :federator_init,
+        start: {Task, :start_link, [&Pleroma.Web.Federator.init/0]},
+        restart: :temporary
+      },
+      %{
+        id: :internal_fetch_init,
+        start: {Task, :start_link, [&Pleroma.Web.ActivityPub.InternalFetchActor.init/0]},
+        restart: :temporary
+      }
+    ]
+  end
 end

lib/pleroma/constants.ex

@@ -6,4 +6,16 @@ defmodule Pleroma.Constants do
   use Const
 
   const(as_public, do: "https://www.w3.org/ns/activitystreams#Public")
+
+  const(object_internal_fields,
+    do: [
+      "likes",
+      "like_count",
+      "announcements",
+      "announcement_count",
+      "emoji",
+      "context_id",
+      "deleted_activity_id"
+    ]
+  )
 end

lib/pleroma/docs/markdown.ex

@@ -23,7 +23,7 @@ def process(descriptions) do
 
       IO.write(file, "#{group[:description]}\n")
 
-      for child <- group[:children] do
+      for child <- group[:children] || [] do
         print_child_header(file, child)
 
         print_suggestions(file, child[:suggestions])
@@ -44,6 +44,17 @@ def process(descriptions) do
     {:ok, config_path}
   end
 
+  defp print_child_header(file, %{key: key, type: type, description: description} = _child) do
+    IO.write(
+      file,
+      "- `#{inspect(key)}` (`#{inspect(type)}`): #{description}  \n"
+    )
+  end
+
+  defp print_child_header(file, %{key: key, type: type} = _child) do
+    IO.write(file, "- `#{inspect(key)}` (`#{inspect(type)}`)  \n")
+  end
+
   defp print_suggestion(file, suggestion) when is_list(suggestion) do
     IO.write(file, "  `#{inspect(suggestion)}`\n")
   end
@@ -59,20 +70,19 @@ defp print_suggestion(file, suggestion, as_list \\ false) do
 
   defp print_suggestions(_file, nil), do: nil
 
+  defp print_suggestions(_file, ""), do: nil
+
   defp print_suggestions(file, suggestions) do
+    if length(suggestions) > 1 do
       IO.write(file, "Suggestions:\n")
 
-    if length(suggestions) > 1 do
       for suggestion <- suggestions do
         print_suggestion(file, suggestion, true)
       end
     else
+      IO.write(file, "  Suggestion: ")
+
       print_suggestion(file, List.first(suggestions))
     end
   end
-
-  defp print_child_header(file, child) do
-    IO.write(file, "- `#{inspect(child[:key])}` -`#{inspect(child[:type])}`  \n")
-    IO.write(file, "#{child[:description]}  \n")
-  end
 end

lib/pleroma/emoji.ex

@@ -4,24 +4,37 @@
 
 defmodule Pleroma.Emoji do
   @moduledoc """
-  The emojis are loaded from:
+  This GenServer stores in an ETS table the list of the loaded emojis,
-
+  and also allows to reload the list at runtime.
-    * emoji packs in INSTANCE-DIR/emoji
-    * the files: `config/emoji.txt` and `config/custom_emoji.txt`
-    * glob paths, nested folder is used as tag name for grouping e.g. priv/static/emoji/custom/nested_folder
-
-  This GenServer stores in an ETS table the list of the loaded emojis, and also allows to reload the list at runtime.
   """
   use GenServer
 
+  alias Pleroma.Emoji.Loader
+
   require Logger
 
-  @type pattern :: Regex.t() | module() | String.t()
-  @type patterns :: pattern() | [pattern()]
-  @type group_patterns :: keyword(patterns())
-
   @ets __MODULE__.Ets
-  @ets_options [:ordered_set, :protected, :named_table, {:read_concurrency, true}]
+  @ets_options [
+    :ordered_set,
+    :protected,
+    :named_table,
+    {:read_concurrency, true}
+  ]
+
+  defstruct [:code, :file, :tags, :safe_code, :safe_file]
+
+  @doc "Build emoji struct"
+  def build({code, file, tags}) do
+    %__MODULE__{
+      code: code,
+      file: file,
+      tags: tags,
+      safe_code: Pleroma.HTML.strip_tags(code),
+      safe_file: Pleroma.HTML.strip_tags(file)
+    }
+  end
+
+  def build({code, file}), do: build({code, file, []})
 
   @doc false
   def start_link(_) do
@@ -44,11 +57,14 @@ def get(name) do
   end
 
   @doc "Returns all the emojos!!"
-  @spec get_all() :: [{String.t(), String.t()}, ...]
+  @spec get_all() :: list({String.t(), String.t(), String.t()})
   def get_all do
     :ets.tab2list(@ets)
   end
 
+  @doc "Clear out old emojis"
+  def clear_all, do: :ets.delete_all_objects(@ets)
+
   @doc false
   def init(_) do
     @ets = :ets.new(@ets, @ets_options)
@@ -58,13 +74,13 @@ def init(_) do
 
   @doc false
   def handle_cast(:reload, state) do
-    load()
+    update_emojis(Loader.load())
     {:noreply, state}
   end
 
   @doc false
   def handle_call(:reload, _from, state) do
-    load()
+    update_emojis(Loader.load())
     {:reply, :ok, state}
   end
 
@@ -75,189 +91,11 @@ def terminate(_, _) do
 
   @doc false
   def code_change(_old_vsn, state, _extra) do
-    load()
+    update_emojis(Loader.load())
     {:ok, state}
   end
 
-  defp load do
+  defp update_emojis(emojis) do
-    emoji_dir_path =
+    :ets.insert(@ets, emojis)
-      Path.join(
-        Pleroma.Config.get!([:instance, :static_dir]),
-        "emoji"
-      )
-
-    emoji_groups = Pleroma.Config.get([:emoji, :groups])
-
-    case File.ls(emoji_dir_path) do
-      {:error, :enoent} ->
-        # The custom emoji directory doesn't exist,
-        # don't do anything
-        nil
-
-      {:error, e} ->
-        # There was some other error
-        Logger.error("Could not access the custom emoji directory #{emoji_dir_path}: #{e}")
-
-      {:ok, results} ->
-        grouped =
-          Enum.group_by(results, fn file -> File.dir?(Path.join(emoji_dir_path, file)) end)
-
-        packs = grouped[true] || []
-        files = grouped[false] || []
-
-        # Print the packs we've found
-        Logger.info("Found emoji packs: #{Enum.join(packs, ", ")}")
-
-        if not Enum.empty?(files) do
-          Logger.warn(
-            "Found files in the emoji folder. These will be ignored, please move them to a subdirectory\nFound files: #{
-              Enum.join(files, ", ")
-            }"
-          )
-        end
-
-        emojis =
-          Enum.flat_map(
-            packs,
-            fn pack -> load_pack(Path.join(emoji_dir_path, pack), emoji_groups) end
-          )
-
-        true = :ets.insert(@ets, emojis)
-    end
-
-    # Compat thing for old custom emoji handling & default emoji,
-    # it should run even if there are no emoji packs
-    shortcode_globs = Pleroma.Config.get([:emoji, :shortcode_globs], [])
-
-    emojis =
-      (load_from_file("config/emoji.txt", emoji_groups) ++
-         load_from_file("config/custom_emoji.txt", emoji_groups) ++
-         load_from_globs(shortcode_globs, emoji_groups))
-      |> Enum.reject(fn value -> value == nil end)
-
-    true = :ets.insert(@ets, emojis)
-
-    :ok
-  end
-
-  defp load_pack(pack_dir, emoji_groups) do
-    pack_name = Path.basename(pack_dir)
-
-    emoji_txt = Path.join(pack_dir, "emoji.txt")
-
-    if File.exists?(emoji_txt) do
-      load_from_file(emoji_txt, emoji_groups)
-    else
-      extensions = Pleroma.Config.get([:emoji, :pack_extensions])
-
-      Logger.info(
-        "No emoji.txt found for pack \"#{pack_name}\", assuming all #{Enum.join(extensions, ", ")} files are emoji"
-      )
-
-      make_shortcode_to_file_map(pack_dir, extensions)
-      |> Enum.map(fn {shortcode, rel_file} ->
-        filename = Path.join("/emoji/#{pack_name}", rel_file)
-
-        {shortcode, filename, [to_string(match_extra(emoji_groups, filename))]}
-      end)
-    end
-  end
-
-  def make_shortcode_to_file_map(pack_dir, exts) do
-    find_all_emoji(pack_dir, exts)
-    |> Enum.map(&Path.relative_to(&1, pack_dir))
-    |> Enum.map(fn f -> {f |> Path.basename() |> Path.rootname(), f} end)
-    |> Enum.into(%{})
-  end
-
-  def find_all_emoji(dir, exts) do
-    Enum.reduce(
-      File.ls!(dir),
-      [],
-      fn f, acc ->
-        filepath = Path.join(dir, f)
-
-        if File.dir?(filepath) do
-          acc ++ find_all_emoji(filepath, exts)
-        else
-          acc ++ [filepath]
-        end
-      end
-    )
-    |> Enum.filter(fn f -> Path.extname(f) in exts end)
-  end
-
-  defp load_from_file(file, emoji_groups) do
-    if File.exists?(file) do
-      load_from_file_stream(File.stream!(file), emoji_groups)
-    else
-      []
-    end
-  end
-
-  defp load_from_file_stream(stream, emoji_groups) do
-    stream
-    |> Stream.map(&String.trim/1)
-    |> Stream.map(fn line ->
-      case String.split(line, ~r/,\s*/) do
-        [name, file] ->
-          {name, file, [to_string(match_extra(emoji_groups, file))]}
-
-        [name, file | tags] ->
-          {name, file, tags}
-
-        _ ->
-          nil
-      end
-    end)
-    |> Enum.to_list()
-  end
-
-  defp load_from_globs(globs, emoji_groups) do
-    static_path = Path.join(:code.priv_dir(:pleroma), "static")
-
-    paths =
-      Enum.map(globs, fn glob ->
-        Path.join(static_path, glob)
-        |> Path.wildcard()
-      end)
-      |> Enum.concat()
-
-    Enum.map(paths, fn path ->
-      tag = match_extra(emoji_groups, Path.join("/", Path.relative_to(path, static_path)))
-      shortcode = Path.basename(path, Path.extname(path))
-      external_path = Path.join("/", Path.relative_to(path, static_path))
-      {shortcode, external_path, [to_string(tag)]}
-    end)
-  end
-
-  @doc """
-  Finds a matching group for the given emoji filename
-  """
-  @spec match_extra(group_patterns(), String.t()) :: atom() | nil
-  def match_extra(group_patterns, filename) do
-    match_group_patterns(group_patterns, fn pattern ->
-      case pattern do
-        %Regex{} = regex -> Regex.match?(regex, filename)
-        string when is_binary(string) -> filename == string
-      end
-    end)
-  end
-
-  defp match_group_patterns(group_patterns, matcher) do
-    Enum.find_value(group_patterns, fn {group, patterns} ->
-      patterns =
-        patterns
-        |> List.wrap()
-        |> Enum.map(fn pattern ->
-          if String.contains?(pattern, "*") do
-            ~r(#{String.replace(pattern, "*", ".*")})
-          else
-            pattern
-          end
-        end)
-
-      Enum.any?(patterns, matcher) && group
-    end)
   end
 end

lib/pleroma/emoji/formatter.ex

@@ -0,0 +1,59 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Emoji.Formatter do
+  alias Pleroma.Emoji
+  alias Pleroma.HTML
+  alias Pleroma.Web.MediaProxy
+
+  def emojify(text) do
+    emojify(text, Emoji.get_all())
+  end
+
+  def emojify(text, nil), do: text
+
+  def emojify(text, emoji, strip \\ false) do
+    Enum.reduce(emoji, text, fn
+      {_, %Emoji{safe_code: emoji, safe_file: file}}, text ->
+        String.replace(text, ":#{emoji}:", prepare_emoji_html(emoji, file, strip))
+
+      {unsafe_emoji, unsafe_file}, text ->
+        emoji = HTML.strip_tags(unsafe_emoji)
+        file = HTML.strip_tags(unsafe_file)
+        String.replace(text, ":#{emoji}:", prepare_emoji_html(emoji, file, strip))
+    end)
+    |> HTML.filter_tags()
+  end
+
+  defp prepare_emoji_html(_emoji, _file, true), do: ""
+
+  defp prepare_emoji_html(emoji, file, _strip) do
+    "<img class='emoji' alt='#{emoji}' title='#{emoji}' src='#{MediaProxy.url(file)}' />"
+  end
+
+  def demojify(text) do
+    emojify(text, Emoji.get_all(), true)
+  end
+
+  def demojify(text, nil), do: text
+
+  @doc "Outputs a list of the emoji-shortcodes in a text"
+  def get_emoji(text) when is_binary(text) do
+    Enum.filter(Emoji.get_all(), fn {emoji, %Emoji{}} ->
+      String.contains?(text, ":#{emoji}:")
+    end)
+  end
+
+  def get_emoji(_), do: []
+
+  @doc "Outputs a list of the emoji-Maps in a text"
+  def get_emoji_map(text) when is_binary(text) do
+    get_emoji(text)
+    |> Enum.reduce(%{}, fn {name, %Emoji{file: file}}, acc ->
+      Map.put(acc, name, "#{Pleroma.Web.Endpoint.static_url()}#{file}")
+    end)
+  end
+
+  def get_emoji_map(_), do: []
+end

lib/pleroma/emoji/loader.ex

@@ -0,0 +1,224 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Emoji.Loader do
+  @moduledoc """
+  The Loader emoji from:
+
+    * emoji packs in INSTANCE-DIR/emoji
+    * the files: `config/emoji.txt` and `config/custom_emoji.txt`
+    * glob paths, nested folder is used as tag name for grouping e.g. priv/static/emoji/custom/nested_folder
+  """
+  alias Pleroma.Config
+  alias Pleroma.Emoji
+
+  require Logger
+
+  @type pattern :: Regex.t() | module() | String.t()
+  @type patterns :: pattern() | [pattern()]
+  @type group_patterns :: keyword(patterns())
+  @type emoji :: {String.t(), Emoji.t()}
+
+  @doc """
+  Loads emojis from files/packs.
+
+  returns list emojis in format:
+  `{"000", "/emoji/freespeechextremist.com/000.png", ["Custom"]}`
+  """
+  @spec load() :: list(emoji)
+  def load do
+    emoji_dir_path = Path.join(Config.get!([:instance, :static_dir]), "emoji")
+
+    emoji_groups = Config.get([:emoji, :groups])
+
+    emojis =
+      case File.ls(emoji_dir_path) do
+        {:error, :enoent} ->
+          # The custom emoji directory doesn't exist,
+          # don't do anything
+          []
+
+        {:error, e} ->
+          # There was some other error
+          Logger.error("Could not access the custom emoji directory #{emoji_dir_path}: #{e}")
+          []
+
+        {:ok, results} ->
+          grouped =
+            Enum.group_by(results, fn file ->
+              File.dir?(Path.join(emoji_dir_path, file))
+            end)
+
+          packs = grouped[true] || []
+          files = grouped[false] || []
+
+          # Print the packs we've found
+          Logger.info("Found emoji packs: #{Enum.join(packs, ", ")}")
+
+          if not Enum.empty?(files) do
+            Logger.warn(
+              "Found files in the emoji folder. These will be ignored, please move them to a subdirectory\nFound files: #{
+                Enum.join(files, ", ")
+              }"
+            )
+          end
+
+          emojis =
+            Enum.flat_map(packs, fn pack ->
+              load_pack(Path.join(emoji_dir_path, pack), emoji_groups)
+            end)
+
+          Emoji.clear_all()
+          emojis
+      end
+
+    # Compat thing for old custom emoji handling & default emoji,
+    # it should run even if there are no emoji packs
+    shortcode_globs = Config.get([:emoji, :shortcode_globs], [])
+
+    emojis_txt =
+      (load_from_file("config/emoji.txt", emoji_groups) ++
+         load_from_file("config/custom_emoji.txt", emoji_groups) ++
+         load_from_globs(shortcode_globs, emoji_groups))
+      |> Enum.reject(fn value -> value == nil end)
+
+    Enum.map(emojis ++ emojis_txt, &prepare_emoji/1)
+  end
+
+  defp prepare_emoji({code, _, _} = emoji), do: {code, Emoji.build(emoji)}
+
+  defp load_pack(pack_dir, emoji_groups) do
+    pack_name = Path.basename(pack_dir)
+
+    pack_file = Path.join(pack_dir, "pack.json")
+
+    if File.exists?(pack_file) do
+      contents = Jason.decode!(File.read!(pack_file))
+
+      contents["files"]
+      |> Enum.map(fn {name, rel_file} ->
+        filename = Path.join("/emoji/#{pack_name}", rel_file)
+        {name, filename, pack_name}
+      end)
+    else
+      # Load from emoji.txt / all files
+      emoji_txt = Path.join(pack_dir, "emoji.txt")
+
+      if File.exists?(emoji_txt) do
+        load_from_file(emoji_txt, emoji_groups)
+      else
+        extensions = Pleroma.Config.get([:emoji, :pack_extensions])
+
+        Logger.info(
+          "No emoji.txt found for pack \"#{pack_name}\", assuming all #{
+            Enum.join(extensions, ", ")
+          } files are emoji"
+        )
+
+        make_shortcode_to_file_map(pack_dir, extensions)
+        |> Enum.map(fn {shortcode, rel_file} ->
+          filename = Path.join("/emoji/#{pack_name}", rel_file)
+
+          {shortcode, filename, [to_string(match_extra(emoji_groups, filename))]}
+        end)
+      end
+    end
+  end
+
+  def make_shortcode_to_file_map(pack_dir, exts) do
+    find_all_emoji(pack_dir, exts)
+    |> Enum.map(&Path.relative_to(&1, pack_dir))
+    |> Enum.map(fn f -> {f |> Path.basename() |> Path.rootname(), f} end)
+    |> Enum.into(%{})
+  end
+
+  def find_all_emoji(dir, exts) do
+    dir
+    |> File.ls!()
+    |> Enum.flat_map(fn f ->
+      filepath = Path.join(dir, f)
+
+      if File.dir?(filepath) do
+        find_all_emoji(filepath, exts)
+      else
+        [filepath]
+      end
+    end)
+    |> Enum.filter(fn f -> Path.extname(f) in exts end)
+  end
+
+  defp load_from_file(file, emoji_groups) do
+    if File.exists?(file) do
+      load_from_file_stream(File.stream!(file), emoji_groups)
+    else
+      []
+    end
+  end
+
+  defp load_from_file_stream(stream, emoji_groups) do
+    stream
+    |> Stream.map(&String.trim/1)
+    |> Stream.map(fn line ->
+      case String.split(line, ~r/,\s*/) do
+        [name, file] ->
+          {name, file, [to_string(match_extra(emoji_groups, file))]}
+
+        [name, file | tags] ->
+          {name, file, tags}
+
+        _ ->
+          nil
+      end
+    end)
+    |> Enum.to_list()
+  end
+
+  defp load_from_globs(globs, emoji_groups) do
+    static_path = Path.join(:code.priv_dir(:pleroma), "static")
+
+    paths =
+      Enum.map(globs, fn glob ->
+        Path.join(static_path, glob)
+        |> Path.wildcard()
+      end)
+      |> Enum.concat()
+
+    Enum.map(paths, fn path ->
+      tag = match_extra(emoji_groups, Path.join("/", Path.relative_to(path, static_path)))
+      shortcode = Path.basename(path, Path.extname(path))
+      external_path = Path.join("/", Path.relative_to(path, static_path))
+      {shortcode, external_path, [to_string(tag)]}
+    end)
+  end
+
+  @doc """
+  Finds a matching group for the given emoji filename
+  """
+  @spec match_extra(group_patterns(), String.t()) :: atom() | nil
+  def match_extra(group_patterns, filename) do
+    match_group_patterns(group_patterns, fn pattern ->
+      case pattern do
+        %Regex{} = regex -> Regex.match?(regex, filename)
+        string when is_binary(string) -> filename == string
+      end
+    end)
+  end
+
+  defp match_group_patterns(group_patterns, matcher) do
+    Enum.find_value(group_patterns, fn {group, patterns} ->
+      patterns =
+        patterns
+        |> List.wrap()
+        |> Enum.map(fn pattern ->
+          if String.contains?(pattern, "*") do
+            ~r(#{String.replace(pattern, "*", ".*")})
+          else
+            pattern
+          end
+        end)
+
+      Enum.any?(patterns, matcher) && group
+    end)
+  end
+end

lib/pleroma/formatter.ex

@@ -3,10 +3,8 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Formatter do
-  alias Pleroma.Emoji
   alias Pleroma.HTML
   alias Pleroma.User
-  alias Pleroma.Web.MediaProxy
 
   @safe_mention_regex ~r/^(\s*(?<mentions>(@.+?\s+){1,})+)(?<rest>.*)/s
   @link_regex ~r"((?:http(s)?:\/\/)?[\w.-]+(?:\.[\w\.-]+)+[\w\-\._~%:/?#[\]@!\$&'\(\)\*\+,;=.]+)|[0-9a-z+\-\.]+:[0-9a-z$-_.+!*'(),]+"ui
@@ -36,9 +34,9 @@ def mention_handler("@" <> nickname, buffer, opts, acc) do
         nickname_text = get_nickname_text(nickname, opts)
 
         link =
-          "<span class='h-card'><a data-user='#{id}' class='u-url mention' href='#{ap_id}'>@<span>#{
+          ~s(<span class="h-card"><a data-user="#{id}" class="u-url mention" href="#{ap_id}" rel="ugc">@<span>#{
             nickname_text
-          }</span></a></span>"
+          }</span></a></span>)
 
         {link, %{acc | mentions: MapSet.put(acc.mentions, {"@" <> nickname, user})}}
 
@@ -50,7 +48,7 @@ def mention_handler("@" <> nickname, buffer, opts, acc) do
   def hashtag_handler("#" <> tag = tag_text, _buffer, _opts, acc) do
     tag = String.downcase(tag)
     url = "#{Pleroma.Web.base_url()}/tag/#{tag}"
-    link = "<a class='hashtag' data-tag='#{tag}' href='#{url}' rel='tag'>#{tag_text}</a>"
+    link = ~s(<a class="hashtag" data-tag="#{tag}" href="#{url}" rel="tag ugc">#{tag_text}</a>)
 
     {link, %{acc | tags: MapSet.put(acc.tags, {tag_text, tag})}}
   end
@@ -100,51 +98,6 @@ def mentions_escape(text, options \\ []) do
     end
   end
 
-  def emojify(text) do
-    emojify(text, Emoji.get_all())
-  end
-
-  def emojify(text, nil), do: text
-
-  def emojify(text, emoji, strip \\ false) do
-    Enum.reduce(emoji, text, fn emoji_data, text ->
-      emoji = HTML.strip_tags(elem(emoji_data, 0))
-      file = HTML.strip_tags(elem(emoji_data, 1))
-
-      html =
-        if not strip do
-          "<img class='emoji' alt='#{emoji}' title='#{emoji}' src='#{MediaProxy.url(file)}' />"
-        else
-          ""
-        end
-
-      String.replace(text, ":#{emoji}:", html) |> HTML.filter_tags()
-    end)
-  end
-
-  def demojify(text) do
-    emojify(text, Emoji.get_all(), true)
-  end
-
-  def demojify(text, nil), do: text
-
-  @doc "Outputs a list of the emoji-shortcodes in a text"
-  def get_emoji(text) when is_binary(text) do
-    Enum.filter(Emoji.get_all(), fn {emoji, _, _} -> String.contains?(text, ":#{emoji}:") end)
-  end
-
-  def get_emoji(_), do: []
-
-  @doc "Outputs a list of the emoji-Maps in a text"
-  def get_emoji_map(text) when is_binary(text) do
-    get_emoji(text)
-    |> Enum.reduce(%{}, fn {name, file, _group}, acc ->
-      Map.put(acc, name, "#{Pleroma.Web.Endpoint.static_url()}#{file}")
-    end)
-  end
-
-  def get_emoji_map(_), do: []
-
   def html_escape({text, mentions, hashtags}, type) do
     {html_escape(text, type), mentions, hashtags}
   end

lib/pleroma/html.ex

@@ -184,7 +184,8 @@ defmodule Pleroma.HTML.Scrubber.Default do
     "tag",
     "nofollow",
     "noopener",
-    "noreferrer"
+    "noreferrer",
+    "ugc"
   ])
 
   Meta.allow_tag_with_these_attributes("a", ["name", "title"])
@@ -304,7 +305,8 @@ defmodule Pleroma.HTML.Scrubber.LinksOnly do
     "nofollow",
     "noopener",
     "noreferrer",
-    "me"
+    "me",
+    "ugc"
   ])
 
   Meta.allow_tag_with_these_attributes("a", ["name", "title"])

lib/pleroma/object.ex

@@ -38,6 +38,24 @@ def change(struct, params \\ %{}) do
   def get_by_id(nil), do: nil
   def get_by_id(id), do: Repo.get(Object, id)
 
+  def get_by_id_and_maybe_refetch(id, opts \\ []) do
+    %{updated_at: updated_at} = object = get_by_id(id)
+
+    if opts[:interval] &&
+         NaiveDateTime.diff(NaiveDateTime.utc_now(), updated_at) > opts[:interval] do
+      case Fetcher.refetch_object(object) do
+        {:ok, %Object{} = object} ->
+          object
+
+        e ->
+          Logger.error("Couldn't refresh #{object.data["id"]}:\n#{inspect(e)}")
+          object
+      end
+    else
+      object
+    end
+  end
+
   def get_by_ap_id(nil), do: nil
 
   def get_by_ap_id(ap_id) do

lib/pleroma/object/fetcher.ex

@@ -6,18 +6,40 @@ defmodule Pleroma.Object.Fetcher do
   alias Pleroma.HTTP
   alias Pleroma.Object
   alias Pleroma.Object.Containment
+  alias Pleroma.Repo
   alias Pleroma.Signature
   alias Pleroma.Web.ActivityPub.InternalFetchActor
   alias Pleroma.Web.ActivityPub.Transmogrifier
   alias Pleroma.Web.OStatus
 
   require Logger
+  require Pleroma.Constants
 
-  defp reinject_object(data) do
+  defp touch_changeset(changeset) do
+    updated_at =
+      NaiveDateTime.utc_now()
+      |> NaiveDateTime.truncate(:second)
+
+    Ecto.Changeset.put_change(changeset, :updated_at, updated_at)
+  end
+
+  defp maybe_reinject_internal_fields(data, %{data: %{} = old_data}) do
+    internal_fields = Map.take(old_data, Pleroma.Constants.object_internal_fields())
+
+    Map.merge(data, internal_fields)
+  end
+
+  defp maybe_reinject_internal_fields(data, _), do: data
+
+  @spec reinject_object(struct(), map()) :: {:ok, Object.t()} | {:error, any()}
+  defp reinject_object(struct, data) do
     Logger.debug("Reinjecting object #{data["id"]}")
 
     with data <- Transmogrifier.fix_object(data),
-         {:ok, object} <- Object.create(data) do
+         data <- maybe_reinject_internal_fields(data, struct),
+         changeset <- Object.change(struct, %{data: data}),
+         changeset <- touch_changeset(changeset),
+         {:ok, object} <- Repo.insert_or_update(changeset) do
       {:ok, object}
     else
       e ->
@@ -26,24 +48,24 @@ defp reinject_object(data) do
     end
   end
 
+  def refetch_object(%Object{data: %{"id" => id}} = object) do
+    with {:local, false} <- {:local, String.starts_with?(id, Pleroma.Web.base_url() <> "/")},
+         {:ok, data} <- fetch_and_contain_remote_object_from_id(id),
+         {:ok, object} <- reinject_object(object, data) do
+      {:ok, object}
+    else
+      {:local, true} -> object
+      e -> {:error, e}
+    end
+  end
+
   # TODO:
   # This will create a Create activity, which we need internally at the moment.
   def fetch_object_from_id(id, options \\ []) do
-    if object = Object.get_cached_by_ap_id(id) do
+    with {:fetch_object, nil} <- {:fetch_object, Object.get_cached_by_ap_id(id)},
-      {:ok, object}
+         {:fetch, {:ok, data}} <- {:fetch, fetch_and_contain_remote_object_from_id(id)},
-    else
-      Logger.info("Fetching #{id} via AP")
-
-      with {:fetch, {:ok, data}} <- {:fetch, fetch_and_contain_remote_object_from_id(id)},
          {:normalize, nil} <- {:normalize, Object.normalize(data, false)},
-           params <- %{
+         params <- prepare_activity_params(data),
-             "type" => "Create",
-             "to" => data["to"],
-             "cc" => data["cc"],
-             # Should we seriously keep this attributedTo thing?
-             "actor" => data["actor"] || data["attributedTo"],
-             "object" => data
-           },
          {:containment, :ok} <- {:containment, Containment.contain_origin(id, params)},
          {:ok, activity} <- Transmogrifier.handle_incoming(params, options),
          {:object, _data, %Object{} = object} <-
@@ -57,11 +79,14 @@ def fetch_object_from_id(id, options \\ []) do
         {:reject, nil}
 
       {:object, data, nil} ->
-          reinject_object(data)
+        reinject_object(%Object{}, data)
 
       {:normalize, object = %Object{}} ->
         {:ok, object}
 
+      {:fetch_object, %Object{} = object} ->
+        {:ok, object}
+
       _e ->
         # Only fallback when receiving a fetch/normalization error with ActivityPub
         Logger.info("Couldn't get object via AP, trying out OStatus fetching...")
@@ -73,6 +98,16 @@ def fetch_object_from_id(id, options \\ []) do
         end
     end
   end
+
+  defp prepare_activity_params(data) do
+    %{
+      "type" => "Create",
+      "to" => data["to"],
+      "cc" => data["cc"],
+      # Should we seriously keep this attributedTo thing?
+      "actor" => data["actor"] || data["attributedTo"],
+      "object" => data
+    }
   end
 
   def fetch_object_from_id!(id, options \\ []) do

lib/pleroma/uploaders/s3.ex

@@ -38,16 +38,26 @@ def get_file(file) do
   def put_file(%Pleroma.Upload{} = upload) do
     config = Config.get([__MODULE__])
     bucket = Keyword.get(config, :bucket)
+    streaming = Keyword.get(config, :streaming_enabled)
 
     s3_name = strict_encode(upload.path)
 
     op =
+      if streaming do
         upload.tempfile
         |> ExAws.S3.Upload.stream_file()
         |> ExAws.S3.upload(bucket, s3_name, [
           {:acl, :public_read},
           {:content_type, upload.content_type}
         ])
+      else
+        {:ok, file_data} = File.read(upload.tempfile)
+
+        ExAws.S3.put_object(bucket, s3_name, file_data, [
+          {:acl, :public_read},
+          {:content_type, upload.content_type}
+        ])
+      end
 
     case ExAws.request(op) do
       {:ok, _} ->

lib/pleroma/user.ex

@@ -150,6 +150,7 @@ def get_cached_follow_state(user, target) do
     Cachex.fetch!(:user_cache, key, fn _ -> {:commit, follow_state(user, target)} end)
   end
 
+  @spec set_follow_state_cache(String.t(), String.t(), String.t()) :: {:ok | :error, boolean()}
   def set_follow_state_cache(user_ap_id, target_ap_id, state) do
     Cachex.put(
       :user_cache,
@@ -268,6 +269,7 @@ def password_update_changeset(struct, params) do
     |> validate_required([:password, :password_confirmation])
     |> validate_confirmation(:password)
     |> put_password_hash
+    |> put_embed(:info, User.Info.set_password_reset_pending(struct.info, false))
   end
 
   @spec reset_password(User.t(), map) :: {:ok, User.t()} | {:error, Ecto.Changeset.t()}
@@ -284,6 +286,20 @@ def reset_password(%User{id: user_id} = user, data) do
     end
   end
 
+  def force_password_reset_async(user) do
+    BackgroundWorker.enqueue("force_password_reset", %{"user_id" => user.id})
+  end
+
+  @spec force_password_reset(User.t()) :: {:ok, User.t()} | {:error, Ecto.Changeset.t()}
+  def force_password_reset(user) do
+    info_cng = User.Info.set_password_reset_pending(user.info, true)
+
+    user
+    |> change()
+    |> put_embed(:info, info_cng)
+    |> update_and_set_cache()
+  end
+
   def register_changeset(struct, params \\ %{}, opts \\ []) do
     bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
     name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
@@ -1114,6 +1130,8 @@ def delete(%User{} = user) do
     BackgroundWorker.enqueue("delete_user", %{"user_id" => user.id})
   end
 
+  def perform(:force_password_reset, user), do: force_password_reset(user)
+
   @spec perform(atom(), User.t()) :: {:ok, User.t()}
   def perform(:delete, %User{} = user) do
     {:ok, _user} = ActivityPub.delete(user)

lib/pleroma/user/info.ex

@@ -20,6 +20,7 @@ defmodule Pleroma.User.Info do
     field(:following_count, :integer, default: nil)
     field(:locked, :boolean, default: false)
     field(:confirmation_pending, :boolean, default: false)
+    field(:password_reset_pending, :boolean, default: false)
     field(:confirmation_token, :string, default: nil)
     field(:default_scope, :string, default: "public")
     field(:blocks, {:array, :string}, default: [])
@@ -41,6 +42,8 @@ defmodule Pleroma.User.Info do
     field(:topic, :string, default: nil)
     field(:hub, :string, default: nil)
     field(:salmon, :string, default: nil)
+    field(:hide_followers_count, :boolean, default: false)
+    field(:hide_follows_count, :boolean, default: false)
     field(:hide_followers, :boolean, default: false)
     field(:hide_follows, :boolean, default: false)
     field(:hide_favorites, :boolean, default: true)
@@ -80,6 +83,14 @@ def set_activation_status(info, deactivated) do
     |> validate_required([:deactivated])
   end
 
+  def set_password_reset_pending(info, pending) do
+    params = %{password_reset_pending: pending}
+
+    info
+    |> cast(params, [:password_reset_pending])
+    |> validate_required([:password_reset_pending])
+  end
+
   def update_notification_settings(info, settings) do
     settings =
       settings
@@ -262,6 +273,8 @@ def remote_user_creation(info, params) do
       :salmon,
       :hide_followers,
       :hide_follows,
+      :hide_followers_count,
+      :hide_follows_count,
       :follower_count,
       :fields,
       :following_count
@@ -281,7 +294,9 @@ def user_upgrade(info, params, remote? \\ false) do
       :following_count,
       :hide_follows,
       :fields,
-      :hide_followers
+      :hide_followers,
+      :hide_followers_count,
+      :hide_follows_count
     ])
     |> validate_fields(remote?)
   end
@@ -295,6 +310,8 @@ def profile_update(info, params) do
       :banner,
       :hide_follows,
       :hide_followers,
+      :hide_followers_count,
+      :hide_follows_count,
       :hide_favorites,
       :background,
       :show_role,
@@ -458,7 +475,9 @@ def follow_information_update(info, params) do
       :hide_followers,
       :hide_follows,
       :follower_count,
-      :following_count
+      :following_count,
+      :hide_followers_count,
+      :hide_follows_count
     ])
   end
 end

lib/pleroma/user/query.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.User.Query do

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -410,6 +410,7 @@ def delete(%Object{data: %{"id" => id, "actor" => actor}} = object, local \\ tru
     end
   end
 
+  @spec block(User.t(), User.t(), String.t() | nil, boolean) :: {:ok, Activity.t() | nil}
   def block(blocker, blocked, activity_id \\ nil, local \\ true) do
     outgoing_blocks = Config.get([:activitypub, :outgoing_blocks])
     unfollow_blocked = Config.get([:activitypub, :unfollow_blocked])
@@ -438,10 +439,11 @@ def unblock(blocker, blocked, activity_id \\ nil, local \\ true) do
     end
   end
 
+  @spec flag(map()) :: {:ok, Activity.t()} | any
   def flag(
         %{
           actor: actor,
-          context: context,
+          context: _context,
           account: account,
           statuses: statuses,
           content: content
@@ -453,14 +455,6 @@ def flag(
 
     additional = params[:additional] || %{}
 
-    params = %{
-      actor: actor,
-      context: context,
-      account: account,
-      statuses: statuses,
-      content: content
-    }
-
     additional =
       if forward do
         Map.merge(additional, %{"to" => [], "cc" => [account.ap_id]})
@@ -526,9 +520,10 @@ def fetch_latest_activity_id_for_context(context, opts \\ %{}) do
   end
 
   def fetch_public_activities(opts \\ %{}) do
-    q = fetch_activities_query([Pleroma.Constants.as_public()], opts)
+    opts = Map.drop(opts, ["user"])
 
-    q
+    [Pleroma.Constants.as_public()]
+    |> fetch_activities_query(opts)
     |> restrict_unlisted()
     |> Pagination.fetch_paginated(opts)
     |> Enum.reverse()

lib/pleroma/web/activity_pub/activity_pub_controller.ex

@@ -49,7 +49,8 @@ def user(conn, %{"nickname" => nickname}) do
          {:ok, user} <- User.ensure_keys_present(user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("user.json", %{user: user}))
+      |> put_view(UserView)
+      |> render("user.json", %{user: user})
     else
       nil -> {:error, :not_found}
     end
@@ -90,7 +91,8 @@ def object_likes(conn, %{"uuid" => uuid, "page" => page}) do
 
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(ObjectView.render("likes.json", ap_id, likes, page))
+      |> put_view(ObjectView)
+      |> render("likes.json", %{ap_id: ap_id, likes: likes, page: page})
     else
       {:public?, false} ->
         {:error, :not_found}
@@ -104,7 +106,8 @@ def object_likes(conn, %{"uuid" => uuid}) do
          likes <- Utils.get_object_likes(object) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(ObjectView.render("likes.json", ap_id, likes))
+      |> put_view(ObjectView)
+      |> render("likes.json", %{ap_id: ap_id, likes: likes})
     else
       {:public?, false} ->
         {:error, :not_found}
@@ -158,7 +161,8 @@ defp set_cache_ttl_for(conn, entity) do
   def following(%{assigns: %{relay: true}} = conn, _params) do
     conn
     |> put_resp_content_type("application/activity+json")
-    |> json(UserView.render("following.json", %{user: Relay.get_actor()}))
+    |> put_view(UserView)
+    |> render("following.json", %{user: Relay.get_actor()})
   end
 
   def following(%{assigns: %{user: for_user}} = conn, %{"nickname" => nickname, "page" => page}) do
@@ -170,7 +174,8 @@ def following(%{assigns: %{user: for_user}} = conn, %{"nickname" => nickname, "p
 
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("following.json", %{user: user, page: page, for: for_user}))
+      |> put_view(UserView)
+      |> render("following.json", %{user: user, page: page, for: for_user})
     else
       {:show_follows, _} ->
         conn
@@ -184,7 +189,8 @@ def following(%{assigns: %{user: for_user}} = conn, %{"nickname" => nickname}) d
          {user, for_user} <- ensure_user_keys_present_and_maybe_refresh_for_user(user, for_user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("following.json", %{user: user, for: for_user}))
+      |> put_view(UserView)
+      |> render("following.json", %{user: user, for: for_user})
     end
   end
 
@@ -192,7 +198,8 @@ def following(%{assigns: %{user: for_user}} = conn, %{"nickname" => nickname}) d
   def followers(%{assigns: %{relay: true}} = conn, _params) do
     conn
     |> put_resp_content_type("application/activity+json")
-    |> json(UserView.render("followers.json", %{user: Relay.get_actor()}))
+    |> put_view(UserView)
+    |> render("followers.json", %{user: Relay.get_actor()})
   end
 
   def followers(%{assigns: %{user: for_user}} = conn, %{"nickname" => nickname, "page" => page}) do
@@ -204,7 +211,8 @@ def followers(%{assigns: %{user: for_user}} = conn, %{"nickname" => nickname, "p
 
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("followers.json", %{user: user, page: page, for: for_user}))
+      |> put_view(UserView)
+      |> render("followers.json", %{user: user, page: page, for: for_user})
     else
       {:show_followers, _} ->
         conn
@@ -218,7 +226,8 @@ def followers(%{assigns: %{user: for_user}} = conn, %{"nickname" => nickname}) d
          {user, for_user} <- ensure_user_keys_present_and_maybe_refresh_for_user(user, for_user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("followers.json", %{user: user, for: for_user}))
+      |> put_view(UserView)
+      |> render("followers.json", %{user: user, for: for_user})
     end
   end
 
@@ -227,7 +236,8 @@ def outbox(conn, %{"nickname" => nickname} = params) do
          {:ok, user} <- User.ensure_keys_present(user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("outbox.json", %{user: user, max_id: params["max_id"]}))
+      |> put_view(UserView)
+      |> render("outbox.json", %{user: user, max_id: params["max_id"]})
     end
   end
 
@@ -275,7 +285,8 @@ defp represent_service_actor(%User{} = user, conn) do
     with {:ok, user} <- User.ensure_keys_present(user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("user.json", %{user: user}))
+      |> put_view(UserView)
+      |> render("user.json", %{user: user})
     else
       nil -> {:error, :not_found}
     end
@@ -296,7 +307,8 @@ def internal_fetch(conn, _params) do
   def whoami(%{assigns: %{user: %User{} = user}} = conn, _params) do
     conn
     |> put_resp_content_type("application/activity+json")
-    |> json(UserView.render("user.json", %{user: user}))
+    |> put_view(UserView)
+    |> render("user.json", %{user: user})
   end
 
   def whoami(_conn, _params), do: {:error, :not_found}

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -42,8 +42,7 @@ def fix_object(object, options \\ []) do
   end
 
   def fix_summary(%{"summary" => nil} = object) do
-    object
+    Map.put(object, "summary", "")
-    |> Map.put("summary", "")
   end
 
   def fix_summary(%{"summary" => _} = object) do
@@ -51,10 +50,7 @@ def fix_summary(%{"summary" => _} = object) do
     object
   end
 
-  def fix_summary(object) do
+  def fix_summary(object), do: Map.put(object, "summary", "")
-    object
-    |> Map.put("summary", "")
-  end
 
   def fix_addressing_list(map, field) do
     cond do
@@ -74,13 +70,9 @@ def fix_explicit_addressing(
         explicit_mentions,
         follower_collection
       ) do
-    explicit_to =
+    explicit_to = Enum.filter(to, fn x -> x in explicit_mentions end)
-      to
-      |> Enum.filter(fn x -> x in explicit_mentions end)
 
-    explicit_cc =
+    explicit_cc = Enum.filter(to, fn x -> x not in explicit_mentions end)
-      to
-      |> Enum.filter(fn x -> x not in explicit_mentions end)
 
     final_cc =
       (cc ++ explicit_cc)
@@ -98,13 +90,19 @@ def fix_explicit_addressing(object, _explicit_mentions, _followers_collection),
   def fix_explicit_addressing(%{"directMessage" => true} = object), do: object
 
   def fix_explicit_addressing(object) do
-    explicit_mentions =
+    explicit_mentions = Utils.determine_explicit_mentions(object)
+
+    %User{follower_address: follower_collection} =
       object
-      |> Utils.determine_explicit_mentions()
+      |> Containment.get_actor()
+      |> User.get_cached_by_ap_id()
 
-    follower_collection = User.get_cached_by_ap_id(Containment.get_actor(object)).follower_address
+    explicit_mentions =
-
+      explicit_mentions ++
-    explicit_mentions = explicit_mentions ++ [Pleroma.Constants.as_public(), follower_collection]
+        [
+          Pleroma.Constants.as_public(),
+          follower_collection
+        ]
 
     fix_explicit_addressing(object, explicit_mentions, follower_collection)
   end
@@ -148,37 +146,19 @@ def fix_addressing(object) do
   end
 
   def fix_actor(%{"attributedTo" => actor} = object) do
-    object
+    Map.put(object, "actor", Containment.get_actor(%{"actor" => actor}))
-    |> Map.put("actor", Containment.get_actor(%{"actor" => actor}))
   end
 
   def fix_in_reply_to(object, options \\ [])
 
   def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
       when not is_nil(in_reply_to) do
-    in_reply_to_id =
+    in_reply_to_id = prepare_in_reply_to(in_reply_to)
-      cond do
-        is_bitstring(in_reply_to) ->
-          in_reply_to
-
-        is_map(in_reply_to) && is_bitstring(in_reply_to["id"]) ->
-          in_reply_to["id"]
-
-        is_list(in_reply_to) && is_bitstring(Enum.at(in_reply_to, 0)) ->
-          Enum.at(in_reply_to, 0)
-
-        # Maybe I should output an error too?
-        true ->
-          ""
-      end
-
     object = Map.put(object, "inReplyToAtomUri", in_reply_to_id)
 
     if Federator.allowed_incoming_reply_depth?(options[:depth]) do
-      case get_obj_helper(in_reply_to_id, options) do
+      with {:ok, replied_object} <- get_obj_helper(in_reply_to_id, options),
-        {:ok, replied_object} ->
+           %Activity{} = _ <- Activity.get_create_by_object_ap_id(replied_object.data["id"]) do
-          with %Activity{} = _activity <-
-                 Activity.get_create_by_object_ap_id(replied_object.data["id"]) do
         object
         |> Map.put("inReplyTo", replied_object.data["id"])
         |> Map.put("inReplyToAtomUri", object["inReplyToAtomUri"] || in_reply_to_id)
@@ -189,11 +169,6 @@ def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
           Logger.error("Couldn't fetch #{inspect(in_reply_to_id)}, error: #{inspect(e)}")
           object
       end
-
-        e ->
-          Logger.error("Couldn't fetch #{inspect(in_reply_to_id)}, error: #{inspect(e)}")
-          object
-      end
     else
       object
     end
@@ -201,6 +176,22 @@ def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
 
   def fix_in_reply_to(object, _options), do: object
 
+  defp prepare_in_reply_to(in_reply_to) do
+    cond do
+      is_bitstring(in_reply_to) ->
+        in_reply_to
+
+      is_map(in_reply_to) && is_bitstring(in_reply_to["id"]) ->
+        in_reply_to["id"]
+
+      is_list(in_reply_to) && is_bitstring(Enum.at(in_reply_to, 0)) ->
+        Enum.at(in_reply_to, 0)
+
+      true ->
+        ""
+    end
+  end
+
   def fix_context(object) do
     context = object["context"] || object["conversation"] || Utils.generate_context_id()
 
@@ -211,11 +202,9 @@ def fix_context(object) do
 
   def fix_attachments(%{"attachment" => attachment} = object) when is_list(attachment) do
     attachments =
-      attachment
+      Enum.map(attachment, fn data ->
-      |> Enum.map(fn data ->
         media_type = data["mediaType"] || data["mimeType"]
         href = data["url"] || data["href"]
-
         url = [%{"type" => "Link", "mediaType" => media_type, "href" => href}]
 
         data
@@ -223,30 +212,25 @@ def fix_attachments(%{"attachment" => attachment} = object) when is_list(attachm
         |> Map.put("url", url)
       end)
 
-    object
+    Map.put(object, "attachment", attachments)
-    |> Map.put("attachment", attachments)
   end
 
   def fix_attachments(%{"attachment" => attachment} = object) when is_map(attachment) do
-    Map.put(object, "attachment", [attachment])
+    object
+    |> Map.put("attachment", [attachment])
     |> fix_attachments()
   end
 
   def fix_attachments(object), do: object
 
   def fix_url(%{"url" => url} = object) when is_map(url) do
-    object
+    Map.put(object, "url", url["href"])
-    |> Map.put("url", url["href"])
   end
 
   def fix_url(%{"type" => "Video", "url" => url} = object) when is_list(url) do
     first_element = Enum.at(url, 0)
 
-    link_element =
+    link_element = Enum.find(url, fn x -> is_map(x) and x["mimeType"] == "text/html" end)
-      url
-      |> Enum.filter(fn x -> is_map(x) end)
-      |> Enum.filter(fn x -> x["mimeType"] == "text/html" end)
-      |> Enum.at(0)
 
     object
     |> Map.put("attachment", [first_element])
@@ -264,36 +248,32 @@ def fix_url(%{"type" => object_type, "url" => url} = object)
         true -> ""
       end
 
-    object
+    Map.put(object, "url", url_string)
-    |> Map.put("url", url_string)
   end
 
   def fix_url(object), do: object
 
   def fix_emoji(%{"tag" => tags} = object) when is_list(tags) do
-    emoji = tags |> Enum.filter(fn data -> data["type"] == "Emoji" and data["icon"] end)
-
     emoji =
-      emoji
+      tags
+      |> Enum.filter(fn data -> data["type"] == "Emoji" and data["icon"] end)
       |> Enum.reduce(%{}, fn data, mapping ->
         name = String.trim(data["name"], ":")
 
-        mapping |> Map.put(name, data["icon"]["url"])
+        Map.put(mapping, name, data["icon"]["url"])
       end)
 
     # we merge mastodon and pleroma emoji into a single mapping, to allow for both wire formats
     emoji = Map.merge(object["emoji"] || %{}, emoji)
 
-    object
+    Map.put(object, "emoji", emoji)
-    |> Map.put("emoji", emoji)
   end
 
   def fix_emoji(%{"tag" => %{"type" => "Emoji"} = tag} = object) do
     name = String.trim(tag["name"], ":")
     emoji = %{name => tag["icon"]["url"]}
 
-    object
+    Map.put(object, "emoji", emoji)
-    |> Map.put("emoji", emoji)
   end
 
   def fix_emoji(object), do: object
@@ -304,17 +284,13 @@ def fix_tag(%{"tag" => tag} = object) when is_list(tag) do
       |> Enum.filter(fn data -> data["type"] == "Hashtag" and data["name"] end)
       |> Enum.map(fn data -> String.slice(data["name"], 1..-1) end)
 
-    combined = tag ++ tags
+    Map.put(object, "tag", tag ++ tags)
-
-    object
-    |> Map.put("tag", combined)
   end
 
   def fix_tag(%{"tag" => %{"type" => "Hashtag", "name" => hashtag} = tag} = object) do
     combined = [tag, String.slice(hashtag, 1..-1)]
 
-    object
+    Map.put(object, "tag", combined)
-    |> Map.put("tag", combined)
   end
 
   def fix_tag(%{"tag" => %{} = tag} = object), do: Map.put(object, "tag", [tag])
@@ -326,8 +302,7 @@ def fix_content_map(%{"contentMap" => content_map} = object) do
     content_groups = Map.to_list(content_map)
     {_, content} = Enum.at(content_groups, 0)
 
-    object
+    Map.put(object, "content", content)
-    |> Map.put("content", content)
   end
 
   def fix_content_map(object), do: object
@@ -336,16 +311,11 @@ def fix_type(object, options \\ [])
 
   def fix_type(%{"inReplyTo" => reply_id, "name" => _} = object, options)
       when is_binary(reply_id) do
-    reply =
     with true <- Federator.allowed_incoming_reply_depth?(options[:depth]),
-           {:ok, object} <- get_obj_helper(reply_id, options) do
+         {:ok, %{data: %{"type" => "Question"} = _} = _} <- get_obj_helper(reply_id, options) do
-        object
-      end
-
-    if reply && reply.data["type"] == "Question" do
       Map.put(object, "type", "Answer")
     else
-      object
+      _ -> object
     end
   end
 
@@ -377,6 +347,17 @@ defp get_follow_activity(follow_object, followed) do
     end
   end
 
+  # Reduce the object list to find the reported user.
+  defp get_reported(objects) do
+    Enum.reduce_while(objects, nil, fn ap_id, _ ->
+      with %User{} = user <- User.get_cached_by_ap_id(ap_id) do
+        {:halt, user}
+      else
+        _ -> {:cont, nil}
+      end
+    end)
+  end
+
   def handle_incoming(data, options \\ [])
 
   # Flag objects are placed ahead of the ID check because Mastodon 2.8 and earlier send them
@@ -385,31 +366,19 @@ def handle_incoming(%{"type" => "Flag", "object" => objects, "actor" => actor} =
     with context <- data["context"] || Utils.generate_context_id(),
          content <- data["content"] || "",
          %User{} = actor <- User.get_cached_by_ap_id(actor),
-
          # Reduce the object list to find the reported user.
-         %User{} = account <-
+         %User{} = account <- get_reported(objects),
-           Enum.reduce_while(objects, nil, fn ap_id, _ ->
-             with %User{} = user <- User.get_cached_by_ap_id(ap_id) do
-               {:halt, user}
-             else
-               _ -> {:cont, nil}
-             end
-           end),
-
          # Remove the reported user from the object list.
          statuses <- Enum.filter(objects, fn ap_id -> ap_id != account.ap_id end) do
-      params = %{
+      %{
         actor: actor,
         context: context,
         account: account,
         statuses: statuses,
         content: content,
-        additional: %{
+        additional: %{"cc" => [account.ap_id]}
-          "cc" => [account.ap_id]
       }
-      }
+      |> ActivityPub.flag()
-
-      ActivityPub.flag(params)
     end
   end
 
@@ -756,8 +725,12 @@ def handle_incoming(
 
   def handle_incoming(_, _), do: :error
 
+  @spec get_obj_helper(String.t(), Keyword.t()) :: {:ok, Object.t()} | nil
   def get_obj_helper(id, options \\ []) do
-    if object = Object.normalize(id, true, options), do: {:ok, object}, else: nil
+    case Object.normalize(id, true, options) do
+      %Object{} = object -> {:ok, object}
+      _ -> nil
+    end
   end
 
   def set_reply_to_uri(%{"inReplyTo" => in_reply_to} = object) when is_binary(in_reply_to) do
@@ -856,26 +829,23 @@ def prepare_outgoing(%{"type" => _type} = data) do
     {:ok, data}
   end
 
-  def maybe_fix_object_url(data) do
+  def maybe_fix_object_url(%{"object" => object} = data) when is_binary(object) do
-    if is_binary(data["object"]) and not String.starts_with?(data["object"], "http") do
+    with false <- String.starts_with?(object, "http"),
-      case get_obj_helper(data["object"]) do
+         {:fetch, {:ok, relative_object}} <- {:fetch, get_obj_helper(object)},
-        {:ok, relative_object} ->
+         %{data: %{"external_url" => external_url}} when not is_nil(external_url) <-
-          if relative_object.data["external_url"] do
+           relative_object do
-            _data =
+      Map.put(data, "object", external_url)
-              data
-              |> Map.put("object", relative_object.data["external_url"])
     else
+      {:fetch, e} ->
+        Logger.error("Couldn't fetch #{object} #{inspect(e)}")
         data
+
+      _ ->
+        data
+    end
   end
 
-        e ->
+  def maybe_fix_object_url(data), do: data
-          Logger.error("Couldn't fetch #{data["object"]} #{inspect(e)}")
-          data
-      end
-    else
-      data
-    end
-  end
 
   def add_hashtags(object) do
     tags =
@@ -894,38 +864,42 @@ def add_hashtags(object) do
           tag
       end)
 
-    object
+    Map.put(object, "tag", tags)
-    |> Map.put("tag", tags)
   end
 
   def add_mention_tags(object) do
     mentions =
       object
       |> Utils.get_notified_from_object()
-      |> Enum.map(fn user ->
+      |> Enum.map(&build_mention_tag/1)
-        %{"type" => "Mention", "href" => user.ap_id, "name" => "@#{user.nickname}"}
-      end)
 
     tags = object["tag"] || []
 
-    object
+    Map.put(object, "tag", tags ++ mentions)
-    |> Map.put("tag", tags ++ mentions)
   end
 
-  def add_emoji_tags(%User{info: %{"emoji" => _emoji} = user_info} = object) do
+  defp build_mention_tag(%{ap_id: ap_id, nickname: nickname} = _) do
-    user_info = add_emoji_tags(user_info)
+    %{"type" => "Mention", "href" => ap_id, "name" => "@#{nickname}"}
+  end
 
-    object
+  def take_emoji_tags(%User{info: %{emoji: emoji} = _user_info} = _user) do
-    |> Map.put(:info, user_info)
+    emoji
+    |> Enum.flat_map(&Map.to_list/1)
+    |> Enum.map(&build_emoji_tag/1)
   end
 
   # TODO: we should probably send mtime instead of unix epoch time for updated
   def add_emoji_tags(%{"emoji" => emoji} = object) do
     tags = object["tag"] || []
 
-    out =
+    out = Enum.map(emoji, &build_emoji_tag/1)
-      emoji
+
-      |> Enum.map(fn {name, url} ->
+    Map.put(object, "tag", tags ++ out)
+  end
+
+  def add_emoji_tags(object), do: object
+
+  defp build_emoji_tag({name, url}) do
     %{
       "icon" => %{"url" => url, "type" => "Image"},
       "name" => ":" <> name <> ":",
@@ -933,14 +907,6 @@ def add_emoji_tags(%{"emoji" => emoji} = object) do
       "updated" => "1970-01-01T00:00:00Z",
       "id" => url
     }
-      end)
-
-    object
-    |> Map.put("tag", tags ++ out)
-  end
-
-  def add_emoji_tags(object) do
-    object
   end
 
   def set_conversation(object) do
@@ -960,9 +926,7 @@ def set_type(object), do: object
 
   def add_attributed_to(object) do
     attributed_to = object["attributedTo"] || object["actor"]
-
+    Map.put(object, "attributedTo", attributed_to)
-    object
-    |> Map.put("attributedTo", attributed_to)
   end
 
   def prepare_attachments(object) do
@@ -973,30 +937,18 @@ def prepare_attachments(object) do
         %{"url" => href, "mediaType" => media_type, "name" => data["name"], "type" => "Document"}
       end)
 
-    object
+    Map.put(object, "attachment", attachments)
-    |> Map.put("attachment", attachments)
   end
 
   defp strip_internal_fields(object) do
     object
-    |> Map.drop([
+    |> Map.drop(Pleroma.Constants.object_internal_fields())
-      "likes",
-      "like_count",
-      "announcements",
-      "announcement_count",
-      "emoji",
-      "context_id",
-      "deleted_activity_id"
-    ])
   end
 
   defp strip_internal_tags(%{"tag" => tags} = object) do
-    tags =
+    tags = Enum.filter(tags, fn x -> is_map(x) end)
-      tags
-      |> Enum.filter(fn x -> is_map(x) end)
 
-    object
+    Map.put(object, "tag", tags)
-    |> Map.put("tag", tags)
   end
 
   defp strip_internal_tags(object), do: object
@@ -1050,8 +1002,8 @@ def upgrade_user_from_ap_id(ap_id) do
     with %User{local: false} = user <- User.get_cached_by_ap_id(ap_id),
          {:ok, data} <- ActivityPub.fetch_and_prepare_user_from_ap_id(ap_id),
          already_ap <- User.ap_enabled?(user),
-         {:ok, user} <- user |> User.upgrade_changeset(data) |> User.update_and_set_cache() do
+         {:ok, user} <- upgrade_user(user, data) do
-      unless already_ap do
+      if not already_ap do
         TransmogrifierWorker.enqueue("user_upgrade", %{"user_id" => user.id})
       end
 
@@ -1062,6 +1014,12 @@ def upgrade_user_from_ap_id(ap_id) do
     end
   end
 
+  defp upgrade_user(user, data) do
+    user
+    |> User.upgrade_changeset(data, true)
+    |> User.update_and_set_cache()
+  end
+
   def maybe_retire_websub(ap_id) do
     # some sanity checks
     if is_binary(ap_id) && String.length(ap_id) > 8 do
@@ -1075,16 +1033,11 @@ def maybe_retire_websub(ap_id) do
     end
   end
 
-  def maybe_fix_user_url(data) do
+  def maybe_fix_user_url(%{"url" => url} = data) when is_map(url) do
-    if is_map(data["url"]) do
+    Map.put(data, "url", url["href"])
-      Map.put(data, "url", data["url"]["href"])
-    else
-      data
-    end
   end
 
-  def maybe_fix_user_object(data) do
+  def maybe_fix_user_url(data), do: data
-    data
+
-    |> maybe_fix_user_url
+  def maybe_fix_user_object(data), do: maybe_fix_user_url(data)
-  end
 end

lib/pleroma/web/activity_pub/utils.ex

@@ -33,50 +33,40 @@ def normalize_params(params) do
     Map.put(params, "actor", get_ap_id(params["actor"]))
   end
 
-  def determine_explicit_mentions(%{"tag" => tag} = _object) when is_list(tag) do
+  @spec determine_explicit_mentions(map()) :: map()
-    tag
+  def determine_explicit_mentions(%{"tag" => tag} = _) when is_list(tag) do
-    |> Enum.filter(fn x -> is_map(x) end)
+    Enum.flat_map(tag, fn
-    |> Enum.filter(fn x -> x["type"] == "Mention" end)
+      %{"type" => "Mention", "href" => href} -> [href]
-    |> Enum.map(fn x -> x["href"] end)
+      _ -> []
+    end)
   end
 
   def determine_explicit_mentions(%{"tag" => tag} = object) when is_map(tag) do
-    Map.put(object, "tag", [tag])
+    object
+    |> Map.put("tag", [tag])
     |> determine_explicit_mentions()
   end
 
   def determine_explicit_mentions(_), do: []
 
+  @spec recipient_in_collection(any(), any()) :: boolean()
   defp recipient_in_collection(ap_id, coll) when is_binary(coll), do: ap_id == coll
   defp recipient_in_collection(ap_id, coll) when is_list(coll), do: ap_id in coll
   defp recipient_in_collection(_, _), do: false
 
+  @spec recipient_in_message(User.t(), User.t(), map()) :: boolean()
   def recipient_in_message(%User{ap_id: ap_id} = recipient, %User{} = actor, params) do
+    addresses = [params["to"], params["cc"], params["bto"], params["bcc"]]
+
     cond do
-      recipient_in_collection(ap_id, params["to"]) ->
+      Enum.any?(addresses, &recipient_in_collection(ap_id, &1)) -> true
-        true
-
-      recipient_in_collection(ap_id, params["cc"]) ->
-        true
-
-      recipient_in_collection(ap_id, params["bto"]) ->
-        true
-
-      recipient_in_collection(ap_id, params["bcc"]) ->
-        true
-
       # if the message is unaddressed at all, then assume it is directly addressed
       # to the recipient
-      !params["to"] && !params["cc"] && !params["bto"] && !params["bcc"] ->
+      Enum.all?(addresses, &is_nil(&1)) -> true
-        true
-
       # if the message is sent from somebody the user is following, then assume it
       # is addressed to the recipient
-      User.following?(recipient, actor) ->
+      User.following?(recipient, actor) -> true
-        true
+      true -> false
-
-      true ->
-        false
     end
   end
 
@@ -179,9 +169,19 @@ def maybe_federate(_), do: :ok
   Adds an id and a published data if they aren't there,
   also adds it to an included object
   """
-  def lazy_put_activity_defaults(map, fake? \\ false) do
+  @spec lazy_put_activity_defaults(map(), boolean) :: map()
-    map =
+  def lazy_put_activity_defaults(map, fake? \\ false)
-      if not fake? do
+
+  def lazy_put_activity_defaults(map, true) do
+    map
+    |> Map.put_new("id", "pleroma:fakeid")
+    |> Map.put_new_lazy("published", &make_date/0)
+    |> Map.put_new("context", "pleroma:fakecontext")
+    |> Map.put_new("context_id", -1)
+    |> lazy_put_object_defaults(true)
+  end
+
+  def lazy_put_activity_defaults(map, _fake?) do
     %{data: %{"id" => context}, id: context_id} = create_context(map["context"])
 
     map
@@ -189,44 +189,39 @@ def lazy_put_activity_defaults(map, fake? \\ false) do
     |> Map.put_new_lazy("published", &make_date/0)
     |> Map.put_new("context", context)
     |> Map.put_new("context_id", context_id)
-      else
+    |> lazy_put_object_defaults(false)
-        map
-        |> Map.put_new("id", "pleroma:fakeid")
-        |> Map.put_new_lazy("published", &make_date/0)
-        |> Map.put_new("context", "pleroma:fakecontext")
-        |> Map.put_new("context_id", -1)
   end
 
-    if is_map(map["object"]) do
+  # Adds an id and published date if they aren't there.
-      object = lazy_put_object_defaults(map["object"], map, fake?)
+  #
-      %{map | "object" => object}
+  @spec lazy_put_object_defaults(map(), boolean()) :: map()
-    else
+  defp lazy_put_object_defaults(%{"object" => map} = activity, true)
+       when is_map(map) do
+    object =
       map
-    end
-  end
-
-  @doc """
-  Adds an id and published date if they aren't there.
-  """
-  def lazy_put_object_defaults(map, activity \\ %{}, fake?)
-
-  def lazy_put_object_defaults(map, activity, true = _fake?) do
-    map
-    |> Map.put_new_lazy("published", &make_date/0)
       |> Map.put_new("id", "pleroma:fake_object_id")
+      |> Map.put_new_lazy("published", &make_date/0)
       |> Map.put_new("context", activity["context"])
-    |> Map.put_new("fake", true)
       |> Map.put_new("context_id", activity["context_id"])
+      |> Map.put_new("fake", true)
+
+    %{activity | "object" => object}
   end
 
-  def lazy_put_object_defaults(map, activity, _fake?) do
+  defp lazy_put_object_defaults(%{"object" => map} = activity, _)
+       when is_map(map) do
+    object =
       map
       |> Map.put_new_lazy("id", &generate_object_id/0)
       |> Map.put_new_lazy("published", &make_date/0)
       |> Map.put_new("context", activity["context"])
       |> Map.put_new("context_id", activity["context_id"])
+
+    %{activity | "object" => object}
   end
 
+  defp lazy_put_object_defaults(activity, _), do: activity
+
   @doc """
   Inserts a full object if it is contained in an activity.
   """
@@ -345,24 +340,24 @@ defp fetch_likes(object) do
   @doc """
   Updates a follow activity's state (for locked accounts).
   """
+  @spec update_follow_state_for_all(Activity.t(), String.t()) :: {:ok, Activity} | {:error, any()}
   def update_follow_state_for_all(
         %Activity{data: %{"actor" => actor, "object" => object}} = activity,
         state
       ) do
-    try do
+    "Follow"
-      Ecto.Adapters.SQL.query!(
+    |> Activity.Queries.by_type()
-        Repo,
+    |> Activity.Queries.by_actor(actor)
-        "UPDATE activities SET data = jsonb_set(data, '{state}', $1) WHERE data->>'type' = 'Follow' AND data->>'actor' = $2 AND data->>'object' = $3 AND data->>'state' = 'pending'",
+    |> Activity.Queries.by_object_id(object)
-        [state, actor, object]
+    |> where(fragment("data->>'state' = 'pending'"))
-      )
+    |> update(set: [data: fragment("jsonb_set(data, '{state}', ?)", ^state)])
+    |> Repo.update_all([])
 
     User.set_follow_state_cache(actor, object, state)
+
     activity = Activity.get_by_id(activity.id)
+
     {:ok, activity}
-    rescue
-      e ->
-        {:error, e}
-    end
   end
 
   def update_follow_state(
@@ -413,6 +408,7 @@ def fetch_latest_follow(%User{ap_id: follower_id}, %User{ap_id: followed_id}) do
   @doc """
   Retruns an existing announce activity if the notice has already been announced
   """
+  @spec get_existing_announce(String.t(), map()) :: Activity.t() | nil
   def get_existing_announce(actor, %{data: %{"id" => ap_id}}) do
     "Announce"
     |> Activity.Queries.by_type()
@@ -495,33 +491,35 @@ def make_unlike_data(
     |> maybe_put("id", activity_id)
   end
 
+  @spec add_announce_to_object(Activity.t(), Object.t()) ::
+          {:ok, Object.t()} | {:error, Ecto.Changeset.t()}
   def add_announce_to_object(
-        %Activity{
+        %Activity{data: %{"actor" => actor, "cc" => [Pleroma.Constants.as_public()]}},
-          data: %{"actor" => actor, "cc" => [Pleroma.Constants.as_public()]}
-        },
         object
       ) do
-    announcements =
+    announcements = take_announcements(object)
-      if is_list(object.data["announcements"]) do
-        Enum.uniq([actor | object.data["announcements"]])
-      else
-        [actor]
-      end
 
+    with announcements <- Enum.uniq([actor | announcements]) do
       update_element_in_object("announcement", announcements, object)
     end
+  end
 
   def add_announce_to_object(_, object), do: {:ok, object}
 
+  @spec remove_announce_from_object(Activity.t(), Object.t()) ::
+          {:ok, Object.t()} | {:error, Ecto.Changeset.t()}
   def remove_announce_from_object(%Activity{data: %{"actor" => actor}}, object) do
-    announcements =
+    with announcements <- List.delete(take_announcements(object), actor) do
-      if is_list(object.data["announcements"]), do: object.data["announcements"], else: []
-
-    with announcements <- announcements |> List.delete(actor) do
       update_element_in_object("announcement", announcements, object)
     end
   end
 
+  defp take_announcements(%{data: %{"announcements" => announcements}} = _)
+       when is_list(announcements),
+       do: announcements
+
+  defp take_announcements(_), do: []
+
   #### Unfollow-related helpers
 
   def make_unfollow_data(follower, followed, follow_activity, activity_id) do
@@ -535,6 +533,7 @@ def make_unfollow_data(follower, followed, follow_activity, activity_id) do
   end
 
   #### Block-related helpers
+  @spec fetch_latest_block(User.t(), User.t()) :: Activity.t() | nil
   def fetch_latest_block(%User{ap_id: blocker_id}, %User{ap_id: blocked_id}) do
     "Block"
     |> Activity.Queries.by_type()
@@ -583,28 +582,32 @@ def make_create_data(params, additional) do
   end
 
   #### Flag-related helpers
-
+  @spec make_flag_data(map(), map()) :: map()
-  def make_flag_data(params, additional) do
+  def make_flag_data(%{actor: actor, context: context, content: content} = params, additional) do
-    status_ap_ids =
-      Enum.map(params.statuses || [], fn
-        %Activity{} = act -> act.data["id"]
-        act when is_map(act) -> act["id"]
-        act when is_binary(act) -> act
-      end)
-
-    object = [params.account.ap_id] ++ status_ap_ids
-
     %{
       "type" => "Flag",
-      "actor" => params.actor.ap_id,
+      "actor" => actor.ap_id,
-      "content" => params.content,
+      "content" => content,
-      "object" => object,
+      "object" => build_flag_object(params),
-      "context" => params.context,
+      "context" => context,
       "state" => "open"
     }
     |> Map.merge(additional)
   end
 
+  def make_flag_data(_, _), do: %{}
+
+  defp build_flag_object(%{account: account, statuses: statuses} = _) do
+    [account.ap_id] ++
+      Enum.map(statuses || [], fn
+        %Activity{} = act -> act.data["id"]
+        act when is_map(act) -> act["id"]
+        act when is_binary(act) -> act
+      end)
+  end
+
+  defp build_flag_object(_), do: []
+
   @doc """
   Fetches the OrderedCollection/OrderedCollectionPage from `from`, limiting the amount of pages fetched after
   the first one to `pages_left` pages.

lib/pleroma/web/activity_pub/views/object_view.ex

@@ -37,12 +37,12 @@ def render("object.json", %{object: %Activity{} = activity}) do
     Map.merge(base, additional)
   end
 
-  def render("likes.json", ap_id, likes, page) do
+  def render("likes.json", %{ap_id: ap_id, likes: likes, page: page}) do
     collection(likes, "#{ap_id}/likes", page)
     |> Map.merge(Pleroma.Web.ActivityPub.Utils.make_json_ld_header())
   end
 
-  def render("likes.json", ap_id, likes) do
+  def render("likes.json", %{ap_id: ap_id, likes: likes}) do
     %{
       "id" => "#{ap_id}/likes",
       "type" => "OrderedCollection",

lib/pleroma/web/activity_pub/views/user_view.ex

@@ -75,10 +75,7 @@ def render("user.json", %{user: user}) do
 
     endpoints = render("endpoints.json", %{user: user})
 
-    user_tags =
+    emoji_tags = Transmogrifier.take_emoji_tags(user)
-      user
-      |> Transmogrifier.add_emoji_tags()
-      |> Map.get("tag", [])
 
     fields =
       user.info
@@ -110,7 +107,7 @@ def render("user.json", %{user: user}) do
       },
       "endpoints" => endpoints,
       "attachment" => fields,
-      "tag" => (user.info.source_data["tag"] || []) ++ user_tags
+      "tag" => (user.info.source_data["tag"] || []) ++ emoji_tags
     }
     |> Map.merge(maybe_make_image(&User.avatar_url/2, "icon", user))
     |> Map.merge(maybe_make_image(&User.banner_url/2, "image", user))
@@ -118,30 +115,34 @@ def render("user.json", %{user: user}) do
   end
 
   def render("following.json", %{user: user, page: page} = opts) do
-    showing = (opts[:for] && opts[:for] == user) || !user.info.hide_follows
+    showing_items = (opts[:for] && opts[:for] == user) || !user.info.hide_follows
+    showing_count = showing_items || !user.info.hide_follows_count
+
     query = User.get_friends_query(user)
     query = from(user in query, select: [:ap_id])
     following = Repo.all(query)
 
     total =
-      if showing do
+      if showing_count do
         length(following)
       else
         0
       end
 
-    collection(following, "#{user.ap_id}/following", page, showing, total)
+    collection(following, "#{user.ap_id}/following", page, showing_items, total)
     |> Map.merge(Utils.make_json_ld_header())
   end
 
   def render("following.json", %{user: user} = opts) do
-    showing = (opts[:for] && opts[:for] == user) || !user.info.hide_follows
+    showing_items = (opts[:for] && opts[:for] == user) || !user.info.hide_follows
+    showing_count = showing_items || !user.info.hide_follows_count
+
     query = User.get_friends_query(user)
     query = from(user in query, select: [:ap_id])
     following = Repo.all(query)
 
     total =
-      if showing do
+      if showing_count do
         length(following)
       else
         0
@@ -152,7 +153,7 @@ def render("following.json", %{user: user} = opts) do
       "type" => "OrderedCollection",
       "totalItems" => total,
       "first" =>
-        if showing do
+        if showing_items do
           collection(following, "#{user.ap_id}/following", 1, !user.info.hide_follows)
         else
           "#{user.ap_id}/following?page=1"
@@ -162,32 +163,34 @@ def render("following.json", %{user: user} = opts) do
   end
 
   def render("followers.json", %{user: user, page: page} = opts) do
-    showing = (opts[:for] && opts[:for] == user) || !user.info.hide_followers
+    showing_items = (opts[:for] && opts[:for] == user) || !user.info.hide_followers
+    showing_count = showing_items || !user.info.hide_followers_count
 
     query = User.get_followers_query(user)
     query = from(user in query, select: [:ap_id])
     followers = Repo.all(query)
 
     total =
-      if showing do
+      if showing_count do
         length(followers)
       else
         0
       end
 
-    collection(followers, "#{user.ap_id}/followers", page, showing, total)
+    collection(followers, "#{user.ap_id}/followers", page, showing_items, total)
     |> Map.merge(Utils.make_json_ld_header())
   end
 
   def render("followers.json", %{user: user} = opts) do
-    showing = (opts[:for] && opts[:for] == user) || !user.info.hide_followers
+    showing_items = (opts[:for] && opts[:for] == user) || !user.info.hide_followers
+    showing_count = showing_items || !user.info.hide_followers_count
 
     query = User.get_followers_query(user)
     query = from(user in query, select: [:ap_id])
     followers = Repo.all(query)
 
     total =
-      if showing do
+      if showing_count do
         length(followers)
       else
         0
@@ -198,8 +201,8 @@ def render("followers.json", %{user: user} = opts) do
       "type" => "OrderedCollection",
       "totalItems" => total,
       "first" =>
-        if showing do
+        if showing_items do
-          collection(followers, "#{user.ap_id}/followers", 1, showing, total)
+          collection(followers, "#{user.ap_id}/followers", 1, showing_items, total)
         else
           "#{user.ap_id}/followers?page=1"
         end
@@ -221,11 +224,12 @@ def render("outbox.json", %{user: user, max_id: max_qid}) do
 
     activities = ActivityPub.fetch_user_activities(user, nil, params)
 
+    # this is sorted chronologically, so first activity is the newest (max)
     {max_id, min_id, collection} =
       if length(activities) > 0 do
         {
-          Enum.at(Enum.reverse(activities), 0).id,
           Enum.at(activities, 0).id,
+          Enum.at(Enum.reverse(activities), 0).id,
           Enum.map(activities, fn act ->
             {:ok, data} = Transmogrifier.prepare_outgoing(act.data)
             data

lib/pleroma/web/admin_api/admin_api_controller.ex

@@ -14,6 +14,7 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
   alias Pleroma.Web.AdminAPI.Config
   alias Pleroma.Web.AdminAPI.ConfigView
   alias Pleroma.Web.AdminAPI.ModerationLogView
+  alias Pleroma.Web.AdminAPI.Report
   alias Pleroma.Web.AdminAPI.ReportView
   alias Pleroma.Web.AdminAPI.Search
   alias Pleroma.Web.CommonAPI
@@ -139,7 +140,8 @@ def users_create(%{assigns: %{user: admin}} = conn, %{"users" => users}) do
   def user_show(conn, %{"nickname" => nickname}) do
     with %User{} = user <- User.get_cached_by_nickname_or_id(nickname) do
       conn
-      |> json(AccountView.render("show.json", %{user: user}))
+      |> put_view(AccountView)
+      |> render("show.json", %{user: user})
     else
       _ -> {:error, :not_found}
     end
@@ -158,7 +160,8 @@ def list_user_statuses(conn, %{"nickname" => nickname} = params) do
         })
 
       conn
-      |> json(StatusView.render("index.json", %{activities: activities, as: :activity}))
+      |> put_view(StatusView)
+      |> render("index.json", %{activities: activities, as: :activity})
     else
       _ -> {:error, :not_found}
     end
@@ -178,7 +181,8 @@ def user_toggle_activation(%{assigns: %{user: admin}} = conn, %{"nickname" => ni
     })
 
     conn
-    |> json(AccountView.render("show.json", %{user: updated_user}))
+    |> put_view(AccountView)
+    |> render("show.json", %{user: updated_user})
   end
 
   def tag_users(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames, "tags" => tags}) do
@@ -400,13 +404,23 @@ def email_invite(%{assigns: %{user: user}} = conn, %{"email" => email} = params)
     end
   end
 
-  @doc "Get a account registeration invite token (base64 string)"
+  @doc "Create an account registration invite token"
-  def get_invite_token(conn, params) do
+  def create_invite_token(conn, params) do
-    options = params["invite"] || %{}
+    opts = %{}
-    {:ok, invite} = UserInviteToken.create_invite(options)
 
-    conn
+    opts =
-    |> json(invite.token)
+      if params["max_use"],
+        do: Map.put(opts, :max_use, params["max_use"]),
+        else: opts
+
+    opts =
+      if params["expires_at"],
+        do: Map.put(opts, :expires_at, params["expires_at"]),
+        else: opts
+
+    {:ok, invite} = UserInviteToken.create_invite(opts)
+
+    json(conn, AccountView.render("invite.json", %{invite: invite}))
   end
 
   @doc "Get list of created invites"
@@ -414,7 +428,8 @@ def invites(conn, _params) do
     invites = UserInviteToken.list_invites()
 
     conn
-    |> json(AccountView.render("invites.json", %{invites: invites}))
+    |> put_view(AccountView)
+    |> render("invites.json", %{invites: invites})
   end
 
   @doc "Revokes invite by token"
@@ -422,7 +437,8 @@ def revoke_invite(conn, %{"token" => token}) do
     with {:ok, invite} <- UserInviteToken.find_by_token(token),
          {:ok, updated_invite} = UserInviteToken.update_invite(invite, %{used: true}) do
       conn
-      |> json(AccountView.render("invite.json", %{invite: updated_invite}))
+      |> put_view(AccountView)
+      |> render("invite.json", %{invite: updated_invite})
     else
       nil -> {:error, :not_found}
     end
@@ -437,16 +453,23 @@ def get_password_reset(conn, %{"nickname" => nickname}) do
     |> json(token.token)
   end
 
+  @doc "Force password reset for a given user"
+  def force_password_reset(conn, %{"nickname" => nickname}) do
+    (%User{local: true} = user) = User.get_cached_by_nickname(nickname)
+
+    User.force_password_reset_async(user)
+
+    json_response(conn, :no_content, "")
+  end
+
   def list_reports(conn, params) do
     params =
       params
       |> Map.put("type", "Flag")
       |> Map.put("skip_preload", true)
+      |> Map.put("total", true)
 
-    reports =
+    reports = ActivityPub.fetch_activities([], params)
-      []
-      |> ActivityPub.fetch_activities(params)
-      |> Enum.reverse()
 
     conn
     |> put_view(ReportView)
@@ -457,7 +480,7 @@ def report_show(conn, %{"id" => id}) do
     with %Activity{} = report <- Activity.get_by_id(id) do
       conn
       |> put_view(ReportView)
-      |> render("show.json", %{report: report})
+      |> render("show.json", Report.extract_report_info(report))
     else
       _ -> {:error, :not_found}
     end
@@ -473,7 +496,7 @@ def report_update_state(%{assigns: %{user: admin}} = conn, %{"id" => id, "state"
 
       conn
       |> put_view(ReportView)
-      |> render("show.json", %{report: report})
+      |> render("show.json", Report.extract_report_info(report))
     end
   end
 
@@ -591,6 +614,12 @@ def config_update(conn, %{"configs" => configs}) do
     |> render("index.json", %{configs: updated})
   end
 
+  def reload_emoji(conn, _params) do
+    Pleroma.Emoji.reload()
+
+    conn |> json("ok")
+  end
+
   def errors(conn, {:error, :not_found}) do
     conn
     |> put_status(:not_found)

lib/pleroma/web/admin_api/report.ex

@@ -0,0 +1,22 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.AdminAPI.Report do
+  alias Pleroma.Activity
+  alias Pleroma.User
+
+  def extract_report_info(
+        %{data: %{"actor" => actor, "object" => [account_ap_id | status_ap_ids]}} = report
+      ) do
+    user = User.get_cached_by_ap_id(actor)
+    account = User.get_cached_by_ap_id(account_ap_id)
+
+    statuses =
+      Enum.map(status_ap_ids, fn ap_id ->
+        Activity.get_by_ap_id_with_object(ap_id)
+      end)
+
+    %{report: report, user: user, account: account, statuses: statuses}
+  end
+end

lib/pleroma/web/admin_api/views/report_view.ex

@@ -4,25 +4,26 @@
 
 defmodule Pleroma.Web.AdminAPI.ReportView do
   use Pleroma.Web, :view
-  alias Pleroma.Activity
   alias Pleroma.HTML
   alias Pleroma.User
+  alias Pleroma.Web.AdminAPI.Report
   alias Pleroma.Web.CommonAPI.Utils
   alias Pleroma.Web.MastodonAPI.StatusView
 
   def render("index.json", %{reports: reports}) do
     %{
-      reports: render_many(reports, __MODULE__, "show.json", as: :report)
+      reports:
+        reports[:items]
+        |> Enum.map(&Report.extract_report_info(&1))
+        |> Enum.map(&render(__MODULE__, "show.json", &1))
+        |> Enum.reverse(),
+      total: reports[:total]
     }
   end
 
-  def render("show.json", %{report: report}) do
+  def render("show.json", %{report: report, user: user, account: account, statuses: statuses}) do
-    user = User.get_cached_by_ap_id(report.data["actor"])
     created_at = Utils.to_masto_date(report.data["published"])
 
-    [account_ap_id | status_ap_ids] = report.data["object"]
-    account = User.get_cached_by_ap_id(account_ap_id)
-
     content =
       unless is_nil(report.data["content"]) do
         HTML.filter_tags(report.data["content"])
@@ -30,11 +31,6 @@ def render("show.json", %{report: report}) do
         nil
       end
 
-    statuses =
-      Enum.map(status_ap_ids, fn ap_id ->
-        Activity.get_by_ap_id_with_object(ap_id)
-      end)
-
     %{
       id: report.id,
       account: merge_account_views(account),

lib/pleroma/web/common_api/common_api.ex

@@ -6,7 +6,7 @@ defmodule Pleroma.Web.CommonAPI do
   alias Pleroma.Activity
   alias Pleroma.ActivityExpiration
   alias Pleroma.Conversation.Participation
-  alias Pleroma.Formatter
+  alias Pleroma.Emoji
   alias Pleroma.Object
   alias Pleroma.ThreadMute
   alias Pleroma.User
@@ -261,12 +261,7 @@ def post(user, %{"status" => status} = data) do
              sensitive,
              poll
            ),
-         object <-
+         object <- put_emoji(object, full_payload, poll_emoji) do
-           Map.put(
-             object,
-             "emoji",
-             Map.merge(Formatter.get_emoji_map(full_payload), poll_emoji)
-           ) do
       preview? = Pleroma.Web.ControllerHelper.truthy_param?(data["preview"]) || false
       direct? = visibility == "direct"
 
@@ -300,6 +295,15 @@ def post(user, %{"status" => status} = data) do
     end
   end
 
+  # parse and put emoji to object data
+  defp put_emoji(map, text, emojis) do
+    Map.put(
+      map,
+      "emoji",
+      Map.merge(Emoji.Formatter.get_emoji_map(text), emojis)
+    )
+  end
+
   # Updates the emojis for a user based on their profile
   def update(user) do
     user =

lib/pleroma/web/common_api/utils.ex

@@ -9,6 +9,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   alias Pleroma.Activity
   alias Pleroma.Config
   alias Pleroma.Conversation.Participation
+  alias Pleroma.Emoji
   alias Pleroma.Formatter
   alias Pleroma.Object
   alias Pleroma.Plugs.AuthenticationPlug
@@ -184,7 +185,7 @@ def make_poll_data(%{"poll" => %{"options" => options, "expires_in" => expires_i
              "name" => option,
              "type" => "Note",
              "replies" => %{"type" => "Collection", "totalItems" => 0}
-           }, Map.merge(emoji, Formatter.get_emoji_map(option))}
+           }, Map.merge(emoji, Emoji.Formatter.get_emoji_map(option))}
         end)
 
       case expires_in do
@@ -434,8 +435,8 @@ def confirm_current_password(user, password) do
   end
 
   def emoji_from_profile(%{info: _info} = user) do
-    (Formatter.get_emoji(user.bio) ++ Formatter.get_emoji(user.name))
+    (Emoji.Formatter.get_emoji(user.bio) ++ Emoji.Formatter.get_emoji(user.name))
-    |> Enum.map(fn {shortcode, url, _} ->
+    |> Enum.map(fn {shortcode, %Emoji{file: url}} ->
       %{
         "type" => "Emoji",
         "icon" => %{"type" => "Image", "url" => "#{Endpoint.url()}#{url}"},

lib/pleroma/web/mastodon_api/controllers/mastodon_api_controller.ex

@@ -13,8 +13,8 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   alias Pleroma.Bookmark
   alias Pleroma.Config
   alias Pleroma.Conversation.Participation
+  alias Pleroma.Emoji
   alias Pleroma.Filter
-  alias Pleroma.Formatter
   alias Pleroma.HTTP
   alias Pleroma.Notification
   alias Pleroma.Object
@@ -140,13 +140,15 @@ def update_credentials(%{assigns: %{user: user}} = conn, params) do
     user_info_emojis =
       user.info
       |> Map.get(:emoji, [])
-      |> Enum.concat(Formatter.get_emoji_map(emojis_text))
+      |> Enum.concat(Emoji.Formatter.get_emoji_map(emojis_text))
       |> Enum.dedup()
 
     info_params =
       [
         :no_rich_text,
         :locked,
+        :hide_followers_count,
+        :hide_follows_count,
         :hide_followers,
         :hide_follows,
         :hide_favorites,
@@ -331,7 +333,7 @@ def peers(conn, _params) do
 
   defp mastodonized_emoji do
     Pleroma.Emoji.get_all()
-    |> Enum.map(fn {shortcode, relative_url, tags} ->
+    |> Enum.map(fn {shortcode, %Pleroma.Emoji{file: relative_url, tags: tags}} ->
       url = to_string(URI.merge(Web.base_url(), relative_url))
 
       %{
@@ -379,7 +381,6 @@ def public_timeline(%{assigns: %{user: user}} = conn, params) do
       |> Map.put("local_only", local_only)
       |> Map.put("blocking_user", user)
       |> Map.put("muting_user", user)
-      |> Map.put("user", user)
       |> ActivityPub.fetch_public_activities()
       |> Enum.reverse()
 
@@ -485,7 +486,7 @@ def get_context(%{assigns: %{user: user}} = conn, %{"id" => id}) do
   end
 
   def get_poll(%{assigns: %{user: user}} = conn, %{"id" => id}) do
-    with %Object{} = object <- Object.get_by_id(id),
+    with %Object{} = object <- Object.get_by_id_and_maybe_refetch(id, interval: 60),
          %Activity{} = activity <- Activity.get_create_by_object_ap_id(object.data["id"]),
          true <- Visibility.visible_for_user?(activity, user) do
       conn
@@ -609,7 +610,12 @@ def post_status(%{assigns: %{user: user}} = conn, %{"status" => _} = params) do
         {:ok, activity} ->
           conn
           |> put_view(StatusView)
-          |> try_render("status.json", %{activity: activity, for: user, as: :activity})
+          |> try_render("status.json", %{
+            activity: activity,
+            for: user,
+            as: :activity,
+            with_direct_conversation_id: true
+          })
       end
     end
   end

lib/pleroma/web/mastodon_api/controllers/search_controller.ex

@@ -19,9 +19,10 @@ defmodule Pleroma.Web.MastodonAPI.SearchController do
 
   def account_search(%{assigns: %{user: user}} = conn, %{"q" => query} = params) do
     accounts = User.search(query, search_options(params, user))
-    res = AccountView.render("accounts.json", users: accounts, for: user, as: :user)
 
-    json(conn, res)
+    conn
+    |> put_view(AccountView)
+    |> render("accounts.json", users: accounts, for: user, as: :user)
   end
 
   def search2(conn, params), do: do_search(:v2, conn, params)

lib/pleroma/web/mastodon_api/views/account_view.ex

@@ -74,10 +74,18 @@ defp do_render("account.json", %{user: user} = opts) do
     user_info = User.get_cached_user_info(user)
 
     following_count =
-      ((!user.info.hide_follows or opts[:for] == user) && user_info.following_count) || 0
+      if !user.info.hide_follows_count or !user.info.hide_follows or opts[:for] == user do
+        user_info.following_count
+      else
+        0
+      end
 
     followers_count =
-      ((!user.info.hide_followers or opts[:for] == user) && user_info.follower_count) || 0
+      if !user.info.hide_followers_count or !user.info.hide_followers or opts[:for] == user do
+        user_info.follower_count
+      else
+        0
+      end
 
     bot = (user.info.source_data["type"] || "Person") in ["Application", "Service"]
 
@@ -138,6 +146,8 @@ defp do_render("account.json", %{user: user} = opts) do
       pleroma: %{
         confirmation_pending: user_info.confirmation_pending,
         tags: user.tags,
+        hide_followers_count: user.info.hide_followers_count,
+        hide_follows_count: user.info.hide_follows_count,
         hide_followers: user.info.hide_followers,
         hide_follows: user.info.hide_follows,
         hide_favorites: user.info.hide_favorites,

lib/pleroma/web/metadata/utils.ex

@@ -3,6 +3,7 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.Metadata.Utils do
+  alias Pleroma.Emoji
   alias Pleroma.Formatter
   alias Pleroma.HTML
   alias Pleroma.Web.MediaProxy
@@ -13,7 +14,7 @@ def scrub_html_and_truncate(%{data: %{"content" => content}} = object) do
     |> HtmlEntities.decode()
     |> String.replace(~r/<br\s?\/?>/, " ")
     |> HTML.get_cached_stripped_html_for_activity(object, "metadata")
-    |> Formatter.demojify()
+    |> Emoji.Formatter.demojify()
     |> Formatter.truncate()
   end
 
@@ -23,7 +24,7 @@ def scrub_html_and_truncate(content, max_length \\ 200) when is_binary(content)
     |> HtmlEntities.decode()
     |> String.replace(~r/<br\s?\/?>/, " ")
     |> HTML.strip_tags()
-    |> Formatter.demojify()
+    |> Emoji.Formatter.demojify()
     |> Formatter.truncate(max_length)
   end
 

lib/pleroma/web/nodeinfo/nodeinfo_controller.ex

@@ -57,6 +57,7 @@ def raw_nodeinfo do
         "mastodon_api_streaming",
         "polls",
         "pleroma_explicit_addressing",
+        "shareable_emoji_packs",
         if Config.get([:media_proxy, :enabled]) do
           "media_proxy"
         end,

lib/pleroma/web/oauth/oauth_controller.ex

@@ -202,6 +202,8 @@ def token_exchange(
          {:ok, app} <- Token.Utils.fetch_app(conn),
          {:auth_active, true} <- {:auth_active, User.auth_active?(user)},
          {:user_active, true} <- {:user_active, !user.info.deactivated},
+         {:password_reset_pending, false} <-
+           {:password_reset_pending, user.info.password_reset_pending},
          {:ok, scopes} <- validate_scopes(app, params),
          {:ok, auth} <- Authorization.create_authorization(app, user, scopes),
          {:ok, token} <- Token.exchange_token(app, auth) do
@@ -215,6 +217,9 @@ def token_exchange(
       {:user_active, false} ->
         render_error(conn, :forbidden, "Your account is currently disabled")
 
+      {:password_reset_pending, true} ->
+        render_error(conn, :forbidden, "Password reset is required")
+
       _error ->
         render_invalid_credentials_error(conn)
     end

lib/pleroma/web/oauth/token/clean_worker.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.OAuth.Token.CleanWorker do

lib/pleroma/web/oauth/token/query.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.OAuth.Token.Query do

lib/pleroma/web/ostatus/ostatus.ex

@@ -3,14 +3,12 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.OStatus do
-  import Ecto.Query
   import Pleroma.Web.XML
   require Logger
 
   alias Pleroma.Activity
   alias Pleroma.HTTP
   alias Pleroma.Object
-  alias Pleroma.Repo
   alias Pleroma.User
   alias Pleroma.Web
   alias Pleroma.Web.ActivityPub.ActivityPub
@@ -38,21 +36,13 @@ def is_representable?(%Activity{} = activity) do
     end
   end
 
-  def feed_path(user) do
+  def feed_path(user), do: "#{user.ap_id}/feed.atom"
-    "#{user.ap_id}/feed.atom"
-  end
 
-  def pubsub_path(user) do
+  def pubsub_path(user), do: "#{Web.base_url()}/push/hub/#{user.nickname}"
-    "#{Web.base_url()}/push/hub/#{user.nickname}"
-  end
 
-  def salmon_path(user) do
+  def salmon_path(user), do: "#{user.ap_id}/salmon"
-    "#{user.ap_id}/salmon"
-  end
 
-  def remote_follow_path do
+  def remote_follow_path, do: "#{Web.base_url()}/ostatus_subscribe?acct={uri}"
-    "#{Web.base_url()}/ostatus_subscribe?acct={uri}"
-  end
 
   def handle_incoming(xml_string, options \\ []) do
     with doc when doc != :error <- parse_document(xml_string) do
@@ -217,10 +207,9 @@ def get_content(entry) do
     Get the cw that mastodon uses.
   """
   def get_cw(entry) do
-    with cw when not is_nil(cw) <- string_from_xpath("/*/summary", entry) do
+    case string_from_xpath("/*/summary", entry) do
-      cw
+      cw when not is_nil(cw) -> cw
-    else
+      _ -> nil
-      _e -> nil
     end
   end
 
@@ -232,19 +221,17 @@ def get_tags(entry) do
   end
 
   def maybe_update(doc, user) do
-    if "true" == string_from_xpath("//author[1]/ap_enabled", doc) do
+    case string_from_xpath("//author[1]/ap_enabled", doc) do
+      "true" ->
         Transmogrifier.upgrade_user_from_ap_id(user.ap_id)
-    else
+
+      _ ->
         maybe_update_ostatus(doc, user)
     end
   end
 
   def maybe_update_ostatus(doc, user) do
-    old_data = %{
+    old_data = Map.take(user, [:bio, :avatar, :name])
-      avatar: user.avatar,
-      bio: user.bio,
-      name: user.name
-    }
 
     with false <- user.local,
          avatar <- make_avatar_object(doc),
@@ -279,21 +266,28 @@ def find_make_or_update_actor(doc) do
     end
   end
 
+  @spec find_or_make_user(String.t()) :: {:ok, User.t()}
   def find_or_make_user(uri) do
-    query = from(user in User, where: user.ap_id == ^uri)
+    case User.get_by_ap_id(uri) do
-
+      %User{} = user -> {:ok, user}
-    user = Repo.one(query)
+      _ -> make_user(uri)
-
-    if is_nil(user) do
-      make_user(uri)
-    else
-      {:ok, user}
     end
   end
 
+  @spec make_user(String.t(), boolean()) :: {:ok, User.t()} | {:error, any()}
   def make_user(uri, update \\ false) do
     with {:ok, info} <- gather_user_info(uri) do
-      data = %{
+      with false <- update,
+           %User{} = user <- User.get_cached_by_ap_id(info["uri"]) do
+        {:ok, user}
+      else
+        _e -> User.insert_or_update_user(build_user_data(info))
+      end
+    end
+  end
+
+  defp build_user_data(info) do
+    %{
       name: info["name"],
       nickname: info["nickname"] <> "@" <> info["host"],
       ap_id: info["uri"],
@@ -301,14 +295,6 @@ def make_user(uri, update \\ false) do
       avatar: info["avatar"],
       bio: info["bio"]
     }
-
-      with false <- update,
-           %User{} = user <- User.get_cached_by_ap_id(data.ap_id) do
-        {:ok, user}
-      else
-        _e -> User.insert_or_update_user(data)
-      end
-    end
   end
 
   # TODO: Just takes the first one for now.
@@ -319,23 +305,23 @@ def make_avatar_object(author_doc, rel \\ "avatar") do
     if href do
       %{
         "type" => "Image",
-        "url" => [
+        "url" => [%{"type" => "Link", "mediaType" => type, "href" => href}]
-          %{
-            "type" => "Link",
-            "mediaType" => type,
-            "href" => href
-          }
-        ]
       }
     else
       nil
     end
   end
 
+  @spec gather_user_info(String.t()) :: {:ok, map()} | {:error, any()}
   def gather_user_info(username) do
     with {:ok, webfinger_data} <- WebFinger.finger(username),
          {:ok, feed_data} <- Websub.gather_feed_data(webfinger_data["topic"]) do
-      {:ok, Map.merge(webfinger_data, feed_data) |> Map.put("fqn", username)}
+      data =
+        webfinger_data
+        |> Map.merge(feed_data)
+        |> Map.put("fqn", username)
+
+      {:ok, data}
     else
       e ->
         Logger.debug(fn -> "Couldn't gather info for #{username}" end)
@@ -371,10 +357,7 @@ def get_atom_url(body) do
   def fetch_activity_from_atom_url(url, options \\ []) do
     with true <- String.starts_with?(url, "http"),
          {:ok, %{body: body, status: code}} when code in 200..299 <-
-           HTTP.get(
+           HTTP.get(url, [{:Accept, "application/atom+xml"}]) do
-             url,
-             [{:Accept, "application/atom+xml"}]
-           ) do
       Logger.debug("Got document from #{url}, handling...")
       handle_incoming(body, options)
     else

lib/pleroma/web/ostatus/ostatus_controller.ex

@@ -55,12 +55,11 @@ def feed_redirect(conn, %{"nickname" => nickname}) do
 
   def feed(conn, %{"nickname" => nickname} = params) do
     with {_, %User{} = user} <- {:fetch_user, User.get_cached_by_nickname(nickname)} do
-      query_params =
-        Map.take(params, ["max_id"])
-        |> Map.merge(%{"whole_db" => true, "actor_id" => user.ap_id})
-
       activities =
-        ActivityPub.fetch_public_activities(query_params)
+        params
+        |> Map.take(["max_id"])
+        |> Map.merge(%{"whole_db" => true, "actor_id" => user.ap_id})
+        |> ActivityPub.fetch_public_activities()
         |> Enum.reverse()
 
       response =
@@ -98,8 +97,7 @@ def salmon_incoming(conn, _) do
 
     Federator.incoming_doc(doc)
 
-    conn
+    send_resp(conn, 200, "")
-    |> send_resp(200, "")
   end
 
   def object(%{assigns: %{format: format}} = conn, %{"uuid" => _uuid})
@@ -218,7 +216,8 @@ defp represent_activity(
 
     conn
     |> put_resp_header("content-type", "application/activity+json")
-    |> json(ObjectView.render("object.json", %{object: object}))
+    |> put_view(ObjectView)
+    |> render("object.json", %{object: object})
   end
 
   defp represent_activity(_conn, "activity+json", _, _) do

lib/pleroma/web/pleroma_api/controllers/emoji_api_controller.ex

@@ -0,0 +1,575 @@
+defmodule Pleroma.Web.PleromaAPI.EmojiAPIController do
+  use Pleroma.Web, :controller
+
+  require Logger
+
+  @emoji_dir_path Path.join(
+                    Pleroma.Config.get!([:instance, :static_dir]),
+                    "emoji"
+                  )
+
+  @cache_seconds_per_file Pleroma.Config.get!([:emoji, :shared_pack_cache_seconds_per_file])
+
+  @doc """
+  Lists the packs available on the instance as JSON.
+
+  The information is public and does not require authentification. The format is
+  a map of "pack directory name" to pack.json contents.
+  """
+  def list_packs(conn, _params) do
+    with {:ok, results} <- File.ls(@emoji_dir_path) do
+      pack_infos =
+        results
+        |> Enum.filter(&has_pack_json?/1)
+        |> Enum.map(&load_pack/1)
+        # Check if all the files are in place and can be sent
+        |> Enum.map(&validate_pack/1)
+        # Transform into a map of pack-name => pack-data
+        |> Enum.into(%{})
+
+      json(conn, pack_infos)
+    end
+  end
+
+  defp has_pack_json?(file) do
+    dir_path = Path.join(@emoji_dir_path, file)
+    # Filter to only use the pack.json packs
+    File.dir?(dir_path) and File.exists?(Path.join(dir_path, "pack.json"))
+  end
+
+  defp load_pack(pack_name) do
+    pack_path = Path.join(@emoji_dir_path, pack_name)
+    pack_file = Path.join(pack_path, "pack.json")
+
+    {pack_name, Jason.decode!(File.read!(pack_file))}
+  end
+
+  defp validate_pack({name, pack}) do
+    pack_path = Path.join(@emoji_dir_path, name)
+
+    if can_download?(pack, pack_path) do
+      archive_for_sha = make_archive(name, pack, pack_path)
+      archive_sha = :crypto.hash(:sha256, archive_for_sha) |> Base.encode16()
+
+      pack =
+        pack
+        |> put_in(["pack", "can-download"], true)
+        |> put_in(["pack", "download-sha256"], archive_sha)
+
+      {name, pack}
+    else
+      {name, put_in(pack, ["pack", "can-download"], false)}
+    end
+  end
+
+  defp can_download?(pack, pack_path) do
+    # If the pack is set as shared, check if it can be downloaded
+    # That means that when asked, the pack can be packed and sent to the remote
+    # Otherwise, they'd have to download it from external-src
+    pack["pack"]["share-files"] &&
+      Enum.all?(pack["files"], fn {_, path} ->
+        File.exists?(Path.join(pack_path, path))
+      end)
+  end
+
+  defp create_archive_and_cache(name, pack, pack_dir, md5) do
+    files =
+      ['pack.json'] ++
+        (pack["files"] |> Enum.map(fn {_, path} -> to_charlist(path) end))
+
+    {:ok, {_, zip_result}} = :zip.zip('#{name}.zip', files, [:memory, cwd: to_charlist(pack_dir)])
+
+    cache_ms = :timer.seconds(@cache_seconds_per_file * Enum.count(files))
+
+    Cachex.put!(
+      :emoji_packs_cache,
+      name,
+      # if pack.json MD5 changes, the cache is not valid anymore
+      %{pack_json_md5: md5, pack_data: zip_result},
+      # Add a minute to cache time for every file in the pack
+      ttl: cache_ms
+    )
+
+    Logger.debug("Created an archive for the '#{name}' emoji pack, \
+keeping it in cache for #{div(cache_ms, 1000)}s")
+
+    zip_result
+  end
+
+  defp make_archive(name, pack, pack_dir) do
+    # Having a different pack.json md5 invalidates cache
+    pack_file_md5 = :crypto.hash(:md5, File.read!(Path.join(pack_dir, "pack.json")))
+
+    case Cachex.get!(:emoji_packs_cache, name) do
+      %{pack_file_md5: ^pack_file_md5, pack_data: zip_result} ->
+        Logger.debug("Using cache for the '#{name}' shared emoji pack")
+        zip_result
+
+      _ ->
+        create_archive_and_cache(name, pack, pack_dir, pack_file_md5)
+    end
+  end
+
+  @doc """
+  An endpoint for other instances (via admin UI) or users (via browser)
+  to download packs that the instance shares.
+  """
+  def download_shared(conn, %{"name" => name}) do
+    pack_dir = Path.join(@emoji_dir_path, name)
+    pack_file = Path.join(pack_dir, "pack.json")
+
+    with {_, true} <- {:exists?, File.exists?(pack_file)},
+         pack = Jason.decode!(File.read!(pack_file)),
+         {_, true} <- {:can_download?, can_download?(pack, pack_dir)} do
+      zip_result = make_archive(name, pack, pack_dir)
+      send_download(conn, {:binary, zip_result}, filename: "#{name}.zip")
+    else
+      {:can_download?, _} ->
+        conn
+        |> put_status(:forbidden)
+        |> json(%{
+          error: "Pack #{name} cannot be downloaded from this instance, either pack sharing\
+           was disabled for this pack or some files are missing"
+        })
+
+      {:exists?, _} ->
+        conn
+        |> put_status(:not_found)
+        |> json(%{error: "Pack #{name} does not exist"})
+    end
+  end
+
+  @doc """
+  An admin endpoint to request downloading a pack named `pack_name` from the instance
+  `instance_address`.
+
+  If the requested instance's admin chose to share the pack, it will be downloaded
+  from that instance, otherwise it will be downloaded from the fallback source, if there is one.
+  """
+  def download_from(conn, %{"instance_address" => address, "pack_name" => name} = data) do
+    shareable_packs_available =
+      "#{address}/.well-known/nodeinfo"
+      |> Tesla.get!()
+      |> Map.get(:body)
+      |> Jason.decode!()
+      |> List.last()
+      |> Map.get("href")
+      # Get the actual nodeinfo address and fetch it
+      |> Tesla.get!()
+      |> Map.get(:body)
+      |> Jason.decode!()
+      |> get_in(["metadata", "features"])
+      |> Enum.member?("shareable_emoji_packs")
+
+    if shareable_packs_available do
+      full_pack =
+        "#{address}/api/pleroma/emoji/packs/list"
+        |> Tesla.get!()
+        |> Map.get(:body)
+        |> Jason.decode!()
+        |> Map.get(name)
+
+      pack_info_res =
+        case full_pack["pack"] do
+          %{"share-files" => true, "can-download" => true, "download-sha256" => sha} ->
+            {:ok,
+             %{
+               sha: sha,
+               uri: "#{address}/api/pleroma/emoji/packs/download_shared/#{name}"
+             }}
+
+          %{"fallback-src" => src, "fallback-src-sha256" => sha} when is_binary(src) ->
+            {:ok,
+             %{
+               sha: sha,
+               uri: src,
+               fallback: true
+             }}
+
+          _ ->
+            {:error,
+             "The pack was not set as shared and there is no fallback src to download from"}
+        end
+
+      with {:ok, %{sha: sha, uri: uri} = pinfo} <- pack_info_res,
+           %{body: emoji_archive} <- Tesla.get!(uri),
+           {_, true} <- {:checksum, Base.decode16!(sha) == :crypto.hash(:sha256, emoji_archive)} do
+        local_name = data["as"] || name
+        pack_dir = Path.join(@emoji_dir_path, local_name)
+        File.mkdir_p!(pack_dir)
+
+        files = Enum.map(full_pack["files"], fn {_, path} -> to_charlist(path) end)
+        # Fallback cannot contain a pack.json file
+        files = if pinfo[:fallback], do: files, else: ['pack.json'] ++ files
+
+        {:ok, _} = :zip.unzip(emoji_archive, cwd: to_charlist(pack_dir), file_list: files)
+
+        # Fallback can't contain a pack.json file, since that would cause the fallback-src-sha256
+        # in it to depend on itself
+        if pinfo[:fallback] do
+          pack_file_path = Path.join(pack_dir, "pack.json")
+
+          File.write!(pack_file_path, Jason.encode!(full_pack, pretty: true))
+        end
+
+        json(conn, "ok")
+      else
+        {:error, e} ->
+          conn |> put_status(:internal_server_error) |> json(%{error: e})
+
+        {:checksum, _} ->
+          conn
+          |> put_status(:internal_server_error)
+          |> json(%{error: "SHA256 for the pack doesn't match the one sent by the server"})
+      end
+    else
+      conn
+      |> put_status(:internal_server_error)
+      |> json(%{error: "The requested instance does not support sharing emoji packs"})
+    end
+  end
+
+  @doc """
+  Creates an empty pack named `name` which then can be updated via the admin UI.
+  """
+  def create(conn, %{"name" => name}) do
+    pack_dir = Path.join(@emoji_dir_path, name)
+
+    if not File.exists?(pack_dir) do
+      File.mkdir_p!(pack_dir)
+
+      pack_file_p = Path.join(pack_dir, "pack.json")
+
+      File.write!(
+        pack_file_p,
+        Jason.encode!(%{pack: %{}, files: %{}}, pretty: true)
+      )
+
+      conn |> json("ok")
+    else
+      conn
+      |> put_status(:conflict)
+      |> json(%{error: "A pack named \"#{name}\" already exists"})
+    end
+  end
+
+  @doc """
+  Deletes the pack `name` and all it's files.
+  """
+  def delete(conn, %{"name" => name}) do
+    pack_dir = Path.join(@emoji_dir_path, name)
+
+    case File.rm_rf(pack_dir) do
+      {:ok, _} ->
+        conn |> json("ok")
+
+      {:error, _} ->
+        conn
+        |> put_status(:internal_server_error)
+        |> json(%{error: "Couldn't delete the pack #{name}"})
+    end
+  end
+
+  @doc """
+  An endpoint to update `pack_names`'s metadata.
+
+  `new_data` is the new metadata for the pack, that will replace the old metadata.
+  """
+  def update_metadata(conn, %{"pack_name" => name, "new_data" => new_data}) do
+    pack_file_p = Path.join([@emoji_dir_path, name, "pack.json"])
+
+    full_pack = Jason.decode!(File.read!(pack_file_p))
+
+    # The new fallback-src is in the new data and it's not the same as it was in the old data
+    should_update_fb_sha =
+      not is_nil(new_data["fallback-src"]) and
+        new_data["fallback-src"] != full_pack["pack"]["fallback-src"]
+
+    with {_, true} <- {:should_update?, should_update_fb_sha},
+         %{body: pack_arch} <- Tesla.get!(new_data["fallback-src"]),
+         {:ok, flist} <- :zip.unzip(pack_arch, [:memory]),
+         {_, true} <- {:has_all_files?, has_all_files?(full_pack, flist)} do
+      fallback_sha = :crypto.hash(:sha256, pack_arch) |> Base.encode16()
+
+      new_data = Map.put(new_data, "fallback-src-sha256", fallback_sha)
+      update_metadata_and_send(conn, full_pack, new_data, pack_file_p)
+    else
+      {:should_update?, _} ->
+        update_metadata_and_send(conn, full_pack, new_data, pack_file_p)
+
+      {:has_all_files?, _} ->
+        conn
+        |> put_status(:bad_request)
+        |> json(%{error: "The fallback archive does not have all files specified in pack.json"})
+    end
+  end
+
+  # Check if all files from the pack.json are in the archive
+  defp has_all_files?(%{"files" => files}, flist) do
+    Enum.all?(files, fn {_, from_manifest} ->
+      Enum.find(flist, fn {from_archive, _} ->
+        to_string(from_archive) == from_manifest
+      end)
+    end)
+  end
+
+  defp update_metadata_and_send(conn, full_pack, new_data, pack_file_p) do
+    full_pack = Map.put(full_pack, "pack", new_data)
+    File.write!(pack_file_p, Jason.encode!(full_pack, pretty: true))
+
+    # Send new data back with fallback sha filled
+    json(conn, new_data)
+  end
+
+  defp get_filename(%{"filename" => filename}), do: filename
+
+  defp get_filename(%{"file" => file}) do
+    case file do
+      %Plug.Upload{filename: filename} -> filename
+      url when is_binary(url) -> Path.basename(url)
+    end
+  end
+
+  defp empty?(str), do: String.trim(str) == ""
+
+  defp update_file_and_send(conn, updated_full_pack, pack_file_p) do
+    # Write the emoji pack file
+    File.write!(pack_file_p, Jason.encode!(updated_full_pack, pretty: true))
+
+    # Return the modified file list
+    json(conn, updated_full_pack["files"])
+  end
+
+  @doc """
+  Updates a file in a pack.
+
+  Updating can mean three things:
+
+  - `add` adds an emoji named `shortcode` to the pack `pack_name`,
+    that means that the emoji file needs to be uploaded with the request
+    (thus requiring it to be a multipart request) and be named `file`.
+    There can also be an optional `filename` that will be the new emoji file name
+    (if it's not there, the name will be taken from the uploaded file).
+  - `update` changes emoji shortcode (from `shortcode` to `new_shortcode` or moves the file
+    (from the current filename to `new_filename`)
+  - `remove` removes the emoji named `shortcode` and it's associated file
+  """
+
+  # Add
+  def update_file(
+        conn,
+        %{"pack_name" => pack_name, "action" => "add", "shortcode" => shortcode} = params
+      ) do
+    pack_dir = Path.join(@emoji_dir_path, pack_name)
+    pack_file_p = Path.join(pack_dir, "pack.json")
+
+    full_pack = Jason.decode!(File.read!(pack_file_p))
+
+    with {_, false} <- {:has_shortcode, Map.has_key?(full_pack["files"], shortcode)},
+         filename <- get_filename(params),
+         false <- empty?(shortcode),
+         false <- empty?(filename) do
+      file_path = Path.join(pack_dir, filename)
+
+      # If the name contains directories, create them
+      if String.contains?(file_path, "/") do
+        File.mkdir_p!(Path.dirname(file_path))
+      end
+
+      case params["file"] do
+        %Plug.Upload{path: upload_path} ->
+          # Copy the uploaded file from the temporary directory
+          File.copy!(upload_path, file_path)
+
+        url when is_binary(url) ->
+          # Download and write the file
+          file_contents = Tesla.get!(url).body
+          File.write!(file_path, file_contents)
+      end
+
+      updated_full_pack = put_in(full_pack, ["files", shortcode], filename)
+      update_file_and_send(conn, updated_full_pack, pack_file_p)
+    else
+      {:has_shortcode, _} ->
+        conn
+        |> put_status(:conflict)
+        |> json(%{error: "An emoji with the \"#{shortcode}\" shortcode already exists"})
+
+      true ->
+        conn
+        |> put_status(:bad_request)
+        |> json(%{error: "shortcode or filename cannot be empty"})
+    end
+  end
+
+  # Remove
+  def update_file(conn, %{
+        "pack_name" => pack_name,
+        "action" => "remove",
+        "shortcode" => shortcode
+      }) do
+    pack_dir = Path.join(@emoji_dir_path, pack_name)
+    pack_file_p = Path.join(pack_dir, "pack.json")
+
+    full_pack = Jason.decode!(File.read!(pack_file_p))
+
+    if Map.has_key?(full_pack["files"], shortcode) do
+      {emoji_file_path, updated_full_pack} = pop_in(full_pack, ["files", shortcode])
+
+      emoji_file_path = Path.join(pack_dir, emoji_file_path)
+
+      # Delete the emoji file
+      File.rm!(emoji_file_path)
+
+      # If the old directory has no more files, remove it
+      if String.contains?(emoji_file_path, "/") do
+        dir = Path.dirname(emoji_file_path)
+
+        if Enum.empty?(File.ls!(dir)) do
+          File.rmdir!(dir)
+        end
+      end
+
+      update_file_and_send(conn, updated_full_pack, pack_file_p)
+    else
+      conn
+      |> put_status(:bad_request)
+      |> json(%{error: "Emoji \"#{shortcode}\" does not exist"})
+    end
+  end
+
+  # Update
+  def update_file(
+        conn,
+        %{"pack_name" => pack_name, "action" => "update", "shortcode" => shortcode} = params
+      ) do
+    pack_dir = Path.join(@emoji_dir_path, pack_name)
+    pack_file_p = Path.join(pack_dir, "pack.json")
+
+    full_pack = Jason.decode!(File.read!(pack_file_p))
+
+    with {_, true} <- {:has_shortcode, Map.has_key?(full_pack["files"], shortcode)},
+         %{"new_shortcode" => new_shortcode, "new_filename" => new_filename} <- params,
+         false <- empty?(new_shortcode),
+         false <- empty?(new_filename) do
+      # First, remove the old shortcode, saving the old path
+      {old_emoji_file_path, updated_full_pack} = pop_in(full_pack, ["files", shortcode])
+      old_emoji_file_path = Path.join(pack_dir, old_emoji_file_path)
+      new_emoji_file_path = Path.join(pack_dir, new_filename)
+
+      # If the name contains directories, create them
+      if String.contains?(new_emoji_file_path, "/") do
+        File.mkdir_p!(Path.dirname(new_emoji_file_path))
+      end
+
+      # Move/Rename the old filename to a new filename
+      # These are probably on the same filesystem, so just rename should work
+      :ok = File.rename(old_emoji_file_path, new_emoji_file_path)
+
+      # If the old directory has no more files, remove it
+      if String.contains?(old_emoji_file_path, "/") do
+        dir = Path.dirname(old_emoji_file_path)
+
+        if Enum.empty?(File.ls!(dir)) do
+          File.rmdir!(dir)
+        end
+      end
+
+      # Then, put in the new shortcode with the new path
+      updated_full_pack = put_in(updated_full_pack, ["files", new_shortcode], new_filename)
+      update_file_and_send(conn, updated_full_pack, pack_file_p)
+    else
+      {:has_shortcode, _} ->
+        conn
+        |> put_status(:bad_request)
+        |> json(%{error: "Emoji \"#{shortcode}\" does not exist"})
+
+      true ->
+        conn
+        |> put_status(:bad_request)
+        |> json(%{error: "new_shortcode or new_filename cannot be empty"})
+
+      _ ->
+        conn
+        |> put_status(:bad_request)
+        |> json(%{error: "new_shortcode or new_file were not specified"})
+    end
+  end
+
+  def update_file(conn, %{"action" => action}) do
+    conn
+    |> put_status(:bad_request)
+    |> json(%{error: "Unknown action: #{action}"})
+  end
+
+  @doc """
+  Imports emoji from the filesystem.
+
+  Importing means checking all the directories in the
+  `$instance_static/emoji/` for directories which do not have
+  `pack.json`. If one has an emoji.txt file, that file will be used
+  to create a `pack.json` file with it's contents. If the directory has
+  neither, all the files with specific configured extenstions will be
+  assumed to be emojis and stored in the new `pack.json` file.
+  """
+  def import_from_fs(conn, _params) do
+    with {:ok, results} <- File.ls(@emoji_dir_path) do
+      imported_pack_names =
+        results
+        |> Enum.filter(fn file ->
+          dir_path = Path.join(@emoji_dir_path, file)
+          # Find the directories that do NOT have pack.json
+          File.dir?(dir_path) and not File.exists?(Path.join(dir_path, "pack.json"))
+        end)
+        |> Enum.map(&write_pack_json_contents/1)
+
+      json(conn, imported_pack_names)
+    else
+      {:error, _} ->
+        conn
+        |> put_status(:internal_server_error)
+        |> json(%{error: "Error accessing emoji pack directory"})
+    end
+  end
+
+  defp write_pack_json_contents(dir) do
+    dir_path = Path.join(@emoji_dir_path, dir)
+    emoji_txt_path = Path.join(dir_path, "emoji.txt")
+
+    files_for_pack = files_for_pack(emoji_txt_path, dir_path)
+    pack_json_contents = Jason.encode!(%{pack: %{}, files: files_for_pack})
+
+    File.write!(Path.join(dir_path, "pack.json"), pack_json_contents)
+
+    dir
+  end
+
+  defp files_for_pack(emoji_txt_path, dir_path) do
+    if File.exists?(emoji_txt_path) do
+      # There's an emoji.txt file, it's likely from a pack installed by the pack manager.
+      # Make a pack.json file from the contents of that emoji.txt fileh
+
+      # FIXME: Copy-pasted from Pleroma.Emoji/load_from_file_stream/2
+
+      # Create a map of shortcodes to filenames from emoji.txt
+      File.read!(emoji_txt_path)
+      |> String.split("\n")
+      |> Enum.map(&String.trim/1)
+      |> Enum.map(fn line ->
+        case String.split(line, ~r/,\s*/) do
+          # This matches both strings with and without tags
+          # and we don't care about tags here
+          [name, file | _] -> {name, file}
+          _ -> nil
+        end
+      end)
+      |> Enum.filter(fn x -> not is_nil(x) end)
+      |> Enum.into(%{})
+    else
+      # If there's no emoji.txt, assume all files
+      # that are of certain extensions from the config are emojis and import them all
+      pack_extensions = Pleroma.Config.get!([:emoji, :pack_extensions])
+      Pleroma.Emoji.Loader.make_shortcode_to_file_map(dir_path, pack_extensions)
+    end
+  end
+end

lib/pleroma/web/router.ex

@@ -180,12 +180,13 @@ defmodule Pleroma.Web.Router do
     post("/relay", AdminAPIController, :relay_follow)
     delete("/relay", AdminAPIController, :relay_unfollow)
 
-    get("/users/invite_token", AdminAPIController, :get_invite_token)
+    post("/users/invite_token", AdminAPIController, :create_invite_token)
     get("/users/invites", AdminAPIController, :invites)
     post("/users/revoke_invite", AdminAPIController, :revoke_invite)
     post("/users/email_invite", AdminAPIController, :email_invite)
 
     get("/users/:nickname/password_reset", AdminAPIController, :get_password_reset)
+    patch("/users/:nickname/force_password_reset", AdminAPIController, :force_password_reset)
 
     get("/users", AdminAPIController, :list_users)
     get("/users/:nickname", AdminAPIController, :user_show)
@@ -205,6 +206,29 @@ defmodule Pleroma.Web.Router do
     get("/config/migrate_from_db", AdminAPIController, :migrate_from_db)
 
     get("/moderation_log", AdminAPIController, :list_log)
+
+    post("/reload_emoji", AdminAPIController, :reload_emoji)
+  end
+
+  scope "/api/pleroma/emoji", Pleroma.Web.PleromaAPI do
+    scope "/packs" do
+      # Modifying packs
+      pipe_through([:admin_api, :oauth_write])
+
+      post("/import_from_fs", EmojiAPIController, :import_from_fs)
+
+      post("/:pack_name/update_file", EmojiAPIController, :update_file)
+      post("/:pack_name/update_metadata", EmojiAPIController, :update_metadata)
+      put("/:name", EmojiAPIController, :create)
+      delete("/:name", EmojiAPIController, :delete)
+      post("/download_from", EmojiAPIController, :download_from)
+    end
+
+    scope "/packs" do
+      # Pack info / downloading
+      get("/", EmojiAPIController, :list_packs)
+      get("/:name/download_shared/", EmojiAPIController, :download_shared)
+    end
   end
 
   scope "/", Pleroma.Web.TwitterAPI do

lib/pleroma/web/streamer/state.ex

@@ -4,16 +4,18 @@ defmodule Pleroma.Web.Streamer.State do
 
   alias Pleroma.Web.Streamer.StreamerSocket
 
+  @env Mix.env()
+
   def start_link(_) do
     GenServer.start_link(__MODULE__, %{sockets: %{}}, name: __MODULE__)
   end
 
   def add_socket(topic, socket) do
-    GenServer.call(__MODULE__, {:add, socket, topic})
+    GenServer.call(__MODULE__, {:add, topic, socket})
   end
 
   def remove_socket(topic, socket) do
-    GenServer.call(__MODULE__, {:remove, socket, topic})
+    do_remove_socket(@env, topic, socket)
   end
 
   def get_sockets do
@@ -29,7 +31,7 @@ def handle_call(:get_state, _from, state) do
     {:reply, state, state}
   end
 
-  def handle_call({:add, socket, topic}, _from, %{sockets: sockets} = state) do
+  def handle_call({:add, topic, socket}, _from, %{sockets: sockets} = state) do
     internal_topic = internal_topic(topic, socket)
     stream_socket = StreamerSocket.from_socket(socket)
 
@@ -44,7 +46,7 @@ def handle_call({:add, socket, topic}, _from, %{sockets: sockets} = state) do
     {:reply, state, state}
   end
 
-  def handle_call({:remove, socket, topic}, _from, %{sockets: sockets} = state) do
+  def handle_call({:remove, topic, socket}, _from, %{sockets: sockets} = state) do
     internal_topic = internal_topic(topic, socket)
     stream_socket = StreamerSocket.from_socket(socket)
 
@@ -57,6 +59,14 @@ def handle_call({:remove, socket, topic}, _from, %{sockets: sockets} = state) do
     {:reply, state, state}
   end
 
+  defp do_remove_socket(:test, _, _) do
+    :ok
+  end
+
+  defp do_remove_socket(_env, topic, socket) do
+    GenServer.call(__MODULE__, {:remove, topic, socket})
+  end
+
   defp internal_topic(topic, socket)
        when topic in ~w[user user:notification direct] do
     "#{topic}:#{socket.assigns[:user].id}"

lib/pleroma/web/twitter_api/controllers/util_controller.ex

@@ -239,11 +239,9 @@ def version(conn, _params) do
 
   def emoji(conn, _params) do
     emoji =
-      Emoji.get_all()
+      Enum.reduce(Emoji.get_all(), %{}, fn {code, %Emoji{file: file, tags: tags}}, acc ->
-      |> Enum.map(fn {short_code, path, tags} ->
+        Map.put(acc, code, %{image_url: file, tags: tags})
-        {short_code, %{image_url: path, tags: tags}}
       end)
-      |> Enum.into(%{})
 
     json(conn, emoji)
   end

lib/pleroma/workers/background_worker.ex

@@ -26,6 +26,11 @@ def perform(%{"op" => "delete_user", "user_id" => user_id}, _job) do
     User.perform(:delete, user)
   end
 
+  def perform(%{"op" => "force_password_reset", "user_id" => user_id}, _job) do
+    user = User.get_cached_by_id(user_id)
+    User.perform(:force_password_reset, user)
+  end
+
   def perform(
         %{
           "op" => "blocks_import",

lib/pleroma/workers/web_pusher_worker.ex

@@ -10,7 +10,11 @@ defmodule Pleroma.Workers.WebPusherWorker do
 
   @impl Oban.Worker
   def perform(%{"op" => "web_push", "notification_id" => notification_id}, _job) do
-    notification = Repo.get(Notification, notification_id)
+    notification =
+      Notification
+      |> Repo.get(notification_id)
+      |> Repo.preload([:activity])
+
     Pleroma.Web.Push.Impl.perform(notification)
   end
 end

mix.exs

@@ -5,7 +5,7 @@ def project do
     [
       app: :pleroma,
       version: version("1.0.0"),
-      elixir: "~> 1.7",
+      elixir: "~> 1.8",
       elixirc_paths: elixirc_paths(Mix.env()),
       compilers: [:phoenix, :gettext] ++ Mix.compilers(),
       elixirc_options: [warnings_as_errors: true],
@@ -99,7 +99,7 @@ defp deps do
       {:plug_cowboy, "~> 2.0"},
       {:phoenix_pubsub, "~> 1.1"},
       {:phoenix_ecto, "~> 4.0"},
-      {:ecto_sql, "~> 3.1"},
+      {:ecto_sql, "~> 3.2"},
       {:postgrex, ">= 0.13.5"},
       {:oban, "~> 0.8.1"},
       {:quantum, "~> 2.3"},
@@ -113,7 +113,7 @@ defp deps do
       {:calendar, "~> 0.17.4"},
       {:cachex, "~> 3.0.2"},
       {:poison, "~> 3.0", override: true},
-      {:tesla, "~> 1.2"},
+      {:tesla, "~> 1.3", override: true},
       {:jason, "~> 1.0"},
       {:mogrify, "~> 0.6.1"},
       {:ex_aws, "~> 2.1"},
@@ -174,7 +174,8 @@ defp aliases do
       "ecto.rollback": ["pleroma.ecto.rollback"],
       "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"],
       "ecto.reset": ["ecto.drop", "ecto.setup"],
-      test: ["ecto.create --quiet", "ecto.migrate", "test"]
+      test: ["ecto.create --quiet", "ecto.migrate", "test"],
+      docs: ["pleroma.docs", "docs"]
     ]
   end
 

mix.lock

@@ -90,7 +90,7 @@
   "swoosh": {:hex, :swoosh, "0.23.2", "7dda95ff0bf54a2298328d6899c74dae1223777b43563ccebebb4b5d2b61df38", [:mix], [{:cowboy, "~> 1.0.1 or ~> 1.1 or ~> 2.4", [hex: :cowboy, repo: "hexpm", optional: true]}, {:gen_smtp, "~> 0.13", [hex: :gen_smtp, repo: "hexpm", optional: true]}, {:hackney, "~> 1.9", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}, {:mail, "~> 0.2", [hex: :mail, repo: "hexpm", optional: true]}, {:mime, "~> 1.1", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_cowboy, ">= 1.0.0", [hex: :plug_cowboy, repo: "hexpm", optional: true]}], "hexpm"},
   "syslog": {:git, "https://github.com/Vagabond/erlang-syslog.git", "4a6c6f2c996483e86c1320e9553f91d337bcb6aa", [tag: "1.0.5"]},
   "telemetry": {:hex, :telemetry, "0.4.0", "8339bee3fa8b91cb84d14c2935f8ecf399ccd87301ad6da6b71c09553834b2ab", [:rebar3], [], "hexpm"},
-  "tesla": {:hex, :tesla, "1.2.1", "864783cc27f71dd8c8969163704752476cec0f3a51eb3b06393b3971dc9733ff", [:mix], [{:exjsx, ">= 3.0.0", [hex: :exjsx, repo: "hexpm", optional: true]}, {:fuse, "~> 2.4", [hex: :fuse, repo: "hexpm", optional: true]}, {:hackney, "~> 1.6", [hex: :hackney, repo: "hexpm", optional: true]}, {:ibrowse, "~> 4.4.0", [hex: :ibrowse, repo: "hexpm", optional: true]}, {:jason, ">= 1.0.0", [hex: :jason, repo: "hexpm", optional: true]}, {:mime, "~> 1.0", [hex: :mime, repo: "hexpm", optional: false]}, {:poison, ">= 1.0.0", [hex: :poison, repo: "hexpm", optional: true]}], "hexpm"},
+  "tesla": {:hex, :tesla, "1.3.0", "f35d72f029e608f9cdc6f6d6fcc7c66cf6d6512a70cfef9206b21b8bd0203a30", [:mix], [{:castore, "~> 0.1", [hex: :castore, repo: "hexpm", optional: true]}, {:exjsx, ">= 3.0.0", [hex: :exjsx, repo: "hexpm", optional: true]}, {:fuse, "~> 2.4", [hex: :fuse, repo: "hexpm", optional: true]}, {:gun, "~> 1.3", [hex: :gun, repo: "hexpm", optional: true]}, {:hackney, "~> 1.6", [hex: :hackney, repo: "hexpm", optional: true]}, {:ibrowse, "~> 4.4.0", [hex: :ibrowse, repo: "hexpm", optional: true]}, {:jason, ">= 1.0.0", [hex: :jason, repo: "hexpm", optional: true]}, {:mime, "~> 1.0", [hex: :mime, repo: "hexpm", optional: false]}, {:mint, "~> 0.4", [hex: :mint, repo: "hexpm", optional: true]}, {:poison, ">= 1.0.0", [hex: :poison, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.3", [hex: :telemetry, repo: "hexpm", optional: true]}], "hexpm"},
   "timex": {:hex, :timex, "3.6.1", "efdf56d0e67a6b956cc57774353b0329c8ab7726766a11547e529357ffdc1d56", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.10", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 0.1.8 or ~> 0.5 or ~> 1.0.0", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm"},
   "trailing_format_plug": {:hex, :trailing_format_plug, "0.0.7", "64b877f912cf7273bed03379936df39894149e35137ac9509117e59866e10e45", [:mix], [{:plug, "> 0.12.0", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"},
   "tzdata": {:hex, :tzdata, "0.5.21", "8cbf3607fcce69636c672d5be2bbb08687fe26639a62bdcc283d267277db7cf0", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm"},

priv/static/adminfe/index.html

@@ -1 +1 @@
-<!DOCTYPE html><html><head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge,chrome=1"><meta name=renderer content=webkit><meta name=viewport content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no"><title>Admin FE</title><link rel="shortcut icon" href=favicon.ico><link href=chunk-elementUI.e5cd8da6.css rel=stylesheet><link href=chunk-libs.4e8c4664.css rel=stylesheet><link href=app.34fc670f.css rel=stylesheet></head><body><script src=/pleroma/admin/static/tinymce4.7.5/tinymce.min.js></script><div id=app></div><script type=text/javascript src=static/js/runtime.f40c8ec4.js></script><script type=text/javascript src=static/js/chunk-elementUI.1911151b.js></script><script type=text/javascript src=static/js/chunk-libs.fb0b7f4a.js></script><script type=text/javascript src=static/js/app.8e186193.js></script></body></html>
+<!DOCTYPE html><html><head><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge,chrome=1"><meta name=renderer content=webkit><meta name=viewport content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no"><title>Admin FE</title><link rel="shortcut icon" href=favicon.ico><link href=chunk-elementUI.f35d8ab1.css rel=stylesheet><link href=chunk-libs.00388c73.css rel=stylesheet><link href=app.40438ff5.css rel=stylesheet></head><body><script src=/pleroma/admin/static/tinymce4.7.5/tinymce.min.js></script><div id=app></div><script type=text/javascript src=static/js/runtime.e85850af.js></script><script type=text/javascript src=static/js/chunk-elementUI.708d6b68.js></script><script type=text/javascript src=static/js/chunk-libs.14514767.js></script><script type=text/javascript src=static/js/app.90c455c5.js></script></body></html>