[#1260] Merge remote-tracking branch 'remotes/upstream/develop' into 1260-rate-limited-auth-actions

# Conflicts:
#	CHANGELOG.md

CHANGELOG.md

@@ -4,11 +4,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
+### Added
+- Refreshing poll results for remote polls
+- Admin API: Add ability to require password reset
+- Mastodon API: Account entities now include `follow_requests_count` (planned Mastodon 3.x addition)
+- Pleroma API: `GET /api/v1/pleroma/accounts/:id/scrobbles` to get a list of recently scrobbled items
+- Pleroma API: `POST /api/v1/pleroma/scrobble` to scrobble a media item
+
 ### Changed
 - **Breaking:** Elixir >=1.8 is now required (was >= 1.7)
+- **Breaking:** Admin API: Return link alongside with token on password reset
 - Replaced [pleroma_job_queue](https://git.pleroma.social/pleroma/pleroma_job_queue) and `Pleroma.Web.Federator.RetryQueue` with [Oban](https://github.com/sorentwo/oban) (see [`docs/config.md`](docs/config.md) on migrating customized worker / retry settings)
 - Introduced [quantum](https://github.com/quantum-elixir/quantum-core) job scheduler
 - Admin API: Return `total` when querying for reports
+- Mastodon API: Return `pleroma.direct_conversation_id` when creating a direct message (`POST /api/v1/statuses`)
+- Admin API: Return link alongside with token on password reset
+
+### Fixed
+- Mastodon API: Fix private and direct statuses not being filtered out from the public timeline for an authenticated user (`GET /api/v1/timelines/public`)
 
 ## [1.1.0] - 2019-??-??
 ### Security
@@ -23,7 +36,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - **Breaking:** Configuration: A setting to explicitly disable the mailer was added, defaulting to true, if you are using a mailer add `config :pleroma, Pleroma.Emails.Mailer, enabled: true` to your config
 - **Breaking:** Configuration: `/media/` is now removed when `base_url` is configured, append `/media/` to your `base_url` config to keep the old behaviour if desired
 - **Breaking:** `/api/pleroma/notifications/read` is moved to `/api/v1/pleroma/notifications/read` and now supports `max_id` and responds with Mastodon API entities.
+- **Breaking:** `/api/pleroma/admin/users/invite_token` now uses `POST`, changed accepted params and returns full invite in json instead of only token string.
 - Configuration: added `config/description.exs`, from which `docs/config.md` is generated
+- Configuration: OpenGraph and TwitterCard providers enabled by default
+- Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
+- Mastodon API: `pleroma.thread_muted` key in the Status entity
 - Federation: Return 403 errors when trying to request pages from a user's follower/following collections if they have `hide_followers`/`hide_follows` set
 - NodeInfo: Return `skipThreadContainment` in `metadata` for the `skip_thread_containment` option
 - NodeInfo: Return `mailerEnabled` in `metadata`
@@ -32,6 +49,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - AdminAPI: Add "godmode" while fetching user statuses (i.e. admin can see private statuses)
 - Improve digest email template
 – Pagination: (optional) return `total` alongside with `items` when paginating
+- Add `rel="ugc"` to all links in statuses, to prevent SEO spam
 
 ### Fixed
 - Following from Osada
@@ -76,6 +94,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: added `/auth/password` endpoint for password reset with rate limit.
 - Mastodon API: /api/v1/accounts/:id/statuses now supports nicknames or user id
 - Mastodon API: Improve support for the user profile custom fields
+- Mastodon API: follower/following counters are nullified when `hide_follows`/`hide_followers` and `hide_follows_count`/`hide_followers_count` are set
 - Admin API: Return users' tags when querying reports
 - Admin API: Return avatar and display name when querying users
 - Admin API: Allow querying user by ID
@@ -93,8 +112,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Pleroma API: Add `/api/v1/pleroma/accounts/confirmation_resend?email=<email>` for resending account confirmation.
 - Pleroma API: Email change endpoint.
 - Admin API: Added moderation log
+- Support for `X-Forwarded-For` and similar HTTP headers which used by reverse proxies to pass a real user IP address to the backend. Must not be enabled unless your instance is behind at least one reverse proxy (such as Nginx, Apache HTTPD or Varnish Cache).
 - Web response cache (currently, enabled for ActivityPub)
 - Mastodon API: Added an endpoint to get multiple statuses by IDs (`GET /api/v1/statuses/?ids[]=1&ids[]=2`)
+- ActivityPub: Add ActivityPub actor's `discoverable` parameter.
+- Admin API: Added moderation log filters (user/start date/end date/search/pagination)
 - Authentication: Added rate limit for password-authorized actions / login existence checks. 
 
 ### Changed
@@ -103,6 +125,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - RichMedia: parsers and their order are configured in `rich_media` config.
 - RichMedia: add the rich media ttl based on image expiration time.
 
+## [1.0.7] - 2019-09-26
+### Fixed
+- Broken federation on Erlang 22 (previous versions of hackney http client were using an option that got deprecated)
+### Changed
+- ActivityPub: The first page in inboxes/outboxes is no longer embedded.
+
 ## [1.0.6] - 2019-08-14
 ### Fixed
 - MRF: fix use of unserializable keyword lists in describe() implementations

config/config.exs

@@ -109,6 +109,7 @@ config :pleroma, Pleroma.Uploaders.Local, uploads: "uploads"
 
 config :pleroma, Pleroma.Uploaders.S3,
   bucket: nil,
+  streaming_enabled: true,
   public_endpoint: "https://s3.amazonaws.com"
 
 config :pleroma, Pleroma.Uploaders.MDII,
@@ -122,7 +123,8 @@ config :pleroma, :emoji,
     # Put groups that have higher priority than defaults here. Example in `docs/config/custom_emoji.md`
     Custom: ["/emoji/*.png", "/emoji/**/*.png"]
   ],
-  default_manifest: "https://git.pleroma.social/pleroma/emoji-index/raw/master/index.json"
+  default_manifest: "https://git.pleroma.social/pleroma/emoji-index/raw/master/index.json",
+  shared_pack_cache_seconds_per_file: 60
 
 config :pleroma, :uri_schemes,
   valid_schemes: [
@@ -507,7 +509,7 @@ config :auto_linker,
     class: false,
     strip_prefix: false,
     new_window: false,
-    rel: false
+    rel: "ugc"
   ]
 
 config :pleroma, :ldap,
@@ -589,6 +591,8 @@ config :pleroma, :rate_limit, authentication: {60_000, 15}
 
 config :pleroma, Pleroma.ActivityExpiration, enabled: true
 
+config :pleroma, Pleroma.Plugs.RemoteIp, enabled: false
+
 config :pleroma, :web_cache_ttl,
   activity_pub: nil,
   activity_pub_question: 30_000

config/test.exs

@@ -30,7 +30,8 @@ config :pleroma, :instance,
   notify_email: "noreply@example.com",
   skip_thread_containment: false,
   federating: false,
-  external_user_synchronization: false
+  external_user_synchronization: false,
+  static_dir: "test/instance_static/"
 
 config :pleroma, :activitypub, sign_object_fetches: false
 

docs/api/admin_api.md

@@ -224,15 +224,25 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 
 ## `/api/pleroma/admin/users/invite_token`
 
-### Get an account registration invite token
+### Create an account registration invite token
 
-- Methods: `GET`
+- Methods: `POST`
 - Params:
-  - *optional* `invite` => [
+  - *optional* `max_use` (integer)
-    - *optional* `max_use` (integer)
+  - *optional* `expires_at` (date string e.g. "2019-04-07")
-    - *optional* `expires_at` (date string e.g. "2019-04-07")
+- Response:
-  ]
+
-- Response: invite token (base64 string)
+```json
+{
+  "id": integer,
+  "token": string,
+  "used": boolean,
+  "expires_at": date,
+  "uses": integer,
+  "max_use": integer,
+  "invite_type": string (possible values: `one_time`, `reusable`, `date_limited`, `reusable_date_limited`)
+}
+```
 
 ## `/api/pleroma/admin/users/invites`
 
@@ -298,16 +308,32 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 
 - Methods: `GET`
 - Params: none
-- Response: password reset token (base64 string)
+- Response:
+
+```json
+{
+  "token": "base64 reset token",
+  "link": "https://pleroma.social/api/pleroma/password_reset/url-encoded-base64-token"
+}
+```
+
+
+## `/api/pleroma/admin/users/:nickname/force_password_reset`
+
+### Force passord reset for a user with a given nickname
+
+- Methods: `PATCH`
+- Params: none
+- Response: none (code `204`)
 
 ## `/api/pleroma/admin/reports`
 ### Get a list of reports
 - Method `GET`
 - Params:
-  - `state`: optional, the state of reports. Valid values are `open`, `closed` and `resolved`
+  - *optional* `state`: **string** the state of reports. Valid values are `open`, `closed` and `resolved`
-  - `limit`: optional, the number of records to retrieve
+  - *optional* `limit`: **integer** the number of records to retrieve
-  - `since_id`: optional, returns results that are more recent than the specified id
+  - *optional* `page`: **integer** page number
-  - `max_id`: optional, returns results that are older than the specified id
+  - *optional* `page_size`: **integer** number of log entries per page (default is `50`)
 - Response:
   - On failure: 403 Forbidden error `{"error": "error_msg"}` when requested by anonymous or non-admin
   - On success: JSON, returns a list of reports, where:
@@ -685,6 +711,7 @@ Compile time settings (need instance reboot):
     }
   ]
 }
+```
 
 - Response:
 
@@ -705,7 +732,11 @@ Compile time settings (need instance reboot):
 - Method `GET`
 - Params:
   - *optional* `page`: **integer** page number
-  - *optional* `page_size`: **integer** number of users per page (default is `50`)
+  - *optional* `page_size`: **integer** number of log entries per page (default is `50`)
+  - *optional* `start_date`: **datetime (ISO 8601)** filter logs by creation date, start from `start_date`. Accepts datetime in ISO 8601 format (YYYY-MM-DDThh:mm:ss), e.g. `2005-08-09T18:31:42`
+  - *optional* `end_date`: **datetime (ISO 8601)** filter logs by creation date, end by from `end_date`. Accepts datetime in ISO 8601 format (YYYY-MM-DDThh:mm:ss), e.g. 2005-08-09T18:31:42
+  - *optional* `user_id`: **integer** filter logs by actor's id
+  - *optional* `search`: **string** search logs by the log message
 - Response:
 
 ```json
@@ -723,3 +754,10 @@ Compile time settings (need instance reboot):
   }
 ]
 ```
+
+## `POST /api/pleroma/admin/reload_emoji`
+### Reload the instance's custom emoji
+* Method `POST`
+* Authentication: required
+* Params: None
+* Response: JSON, "ok" and 200 status

docs/api/differences_in_mastoapi_responses.md

@@ -21,7 +21,8 @@ Adding the parameter `with_muted=true` to the timeline queries will also return
 Has these additional fields under the `pleroma` object:
 
 - `local`: true if the post was made on the local instance
-- `conversation_id`: the ID of the conversation the status is associated with (if any)
+- `conversation_id`: the ID of the AP context the status is associated with (if any)
+- `direct_conversation_id`: the ID of the Mastodon direct message conversation the status is associated with (if any)
 - `in_reply_to_account_acct`: the `acct` property of User entity for replied user (if any)
 - `content`: a map consisting of alternate representations of the `content` property with the key being it's mimetype. Currently the only alternate representation supported is `text/plain`
 - `spoiler_text`: a map consisting of alternate representations of the `spoiler_text` property with the key being it's mimetype. Currently the only alternate representation supported is `text/plain`
@@ -50,6 +51,8 @@ Has these additional fields under the `pleroma` object:
 - `confirmation_pending`: boolean, true if a new user account is waiting on email confirmation to be activated
 - `hide_followers`: boolean, true when the user has follower hiding enabled
 - `hide_follows`: boolean, true when the user has follow hiding enabled
+- `hide_followers_count`: boolean, true when the user has follower stat hiding enabled
+- `hide_follows_count`: boolean, true when the user has follow stat hiding enabled
 - `settings_store`: A generic map of settings for frontends. Opaque to the backend. Only returned in `verify_credentials` and `update_credentials`
 - `chat_token`: The token needed for Pleroma chat. Only returned in `verify_credentials`
 - `deactivated`: boolean, true when the user is deactivated
@@ -112,6 +115,8 @@ Additional parameters can be added to the JSON body/Form data:
 - `no_rich_text` - if true, html tags are stripped from all statuses requested from the API
 - `hide_followers` - if true, user's followers will be hidden
 - `hide_follows` - if true, user's follows will be hidden
+- `hide_followers_count` - if true, user's follower count will be hidden
+- `hide_follows_count` - if true, user's follow count will be hidden
 - `hide_favorites` - if true, user's favorites timeline will be hidden
 - `show_role` - if true, user's role (e.g admin, moderator) will be exposed to anyone in the API
 - `default_scope` - the scope returned under `privacy` key in Source subentity

docs/api/pleroma_api.md

@@ -365,3 +365,109 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 * Params:
     * `recipients`: A list of ids of users that should receive posts to this conversation. This will replace the current list of recipients, so submit the full list. The owner of owner of the conversation will always be part of the set of recipients, though.
 * Response: JSON, statuses (200 - healthy, 503 unhealthy)
+
+## `GET /api/pleroma/emoji/packs`
+### Lists the custom emoji packs on the server
+* Method `GET`
+* Authentication: not required
+* Params: None
+* Response: JSON, "ok" and 200 status and the JSON hashmap of "pack name" to "pack contents"
+
+## `PUT /api/pleroma/emoji/packs/:name`
+### Creates an empty custom emoji pack
+* Method `PUT`
+* Authentication: required
+* Params: None
+* Response: JSON, "ok" and 200 status or 409 if the pack with that name already exists
+
+## `DELETE /api/pleroma/emoji/packs/:name`
+### Delete a custom emoji pack
+* Method `DELETE`
+* Authentication: required
+* Params: None
+* Response: JSON, "ok" and 200 status or 500 if there was an error deleting the pack
+
+## `POST /api/pleroma/emoji/packs/:name/update_file`
+### Update a file in a custom emoji pack
+* Method `POST`
+* Authentication: required
+* Params: 
+    * if the `action` is `add`, adds an emoji named `shortcode` to the pack `pack_name`,
+      that means that the emoji file needs to be uploaded with the request
+      (thus requiring it to be a multipart request) and be named `file`.
+      There can also be an optional `filename` that will be the new emoji file name
+      (if it's not there, the name will be taken from the uploaded file).
+    * if the `action` is `update`, changes emoji shortcode
+      (from `shortcode` to `new_shortcode` or moves the file (from the current filename to `new_filename`)
+    * if the `action` is `remove`, removes the emoji named `shortcode` and it's associated file
+* Response: JSON, updated "files" section of the pack and 200 status, 409 if the trying to use a shortcode
+  that is already taken, 400 if there was an error with the shortcode, filename or file (additional info
+  in the "error" part of the response JSON)
+
+## `POST /api/pleroma/emoji/packs/:name/update_metadata`
+### Updates (replaces) pack metadata
+* Method `POST`
+* Authentication: required
+* Params: 
+  * `new_data`: new metadata to replace the old one
+* Response: JSON, updated "metadata" section of the pack and 200 status or 400 if there was a
+  problem with the new metadata (the error is specified in the "error" part of the response JSON)
+
+## `POST /api/pleroma/emoji/packs/download_from`
+### Requests the instance to download the pack from another instance
+* Method `POST`
+* Authentication: required
+* Params: 
+  * `instance_address`: the address of the instance to download from
+  * `pack_name`: the pack to download from that instance
+* Response: JSON, "ok" and 200 status if the pack was downloaded, or 500 if there were
+  errors downloading the pack
+
+## `POST /api/pleroma/emoji/packs/list_from`
+### Requests the instance to list the packs from another instance
+* Method `POST`
+* Authentication: required
+* Params:
+  * `instance_address`: the address of the instance to download from
+* Response: JSON with the pack list, same as if the request was made to that instance's
+  list endpoint directly + 200 status
+
+## `GET /api/pleroma/emoji/packs/:name/download_shared`
+### Requests a local pack from the instance
+* Method `GET`
+* Authentication: not required
+* Params: None
+* Response: the archive of the pack with a 200 status code, 403 if the pack is not set as shared,
+  404 if the pack does not exist
+
+## `GET /api/v1/pleroma/accounts/:id/scrobbles`
+### Requests a list of current and recent Listen activities for an account
+* Method `GET`
+* Authentication: not required
+* Params: None
+* Response: An array of media metadata entities.
+* Example response:
+```json
+[
+   {
+       "account": {...},
+       "id": "1234",
+       "title": "Some Title",
+       "artist": "Some Artist",
+       "album": "Some Album",
+       "length": 180000,
+       "created_at": "2019-09-28T12:40:45.000Z"
+   }
+]
+```
+
+## `POST /api/v1/pleroma/scrobble`
+### Creates a new Listen activity for an account
+* Method `POST`
+* Authentication: required
+* Params:
+  * `title`: the title of the media playing
+  * `album`: the album of the media playing [optional]
+  * `artist`: the artist of the media playing [optional]
+  * `length`: the length of the media playing [optional]
+* Response: the newly created media metadata entity representing the Listen activity

docs/clients.md

@@ -39,7 +39,7 @@ Feel free to contact us to be added to this list!
 
 ### Nekonium
 - Homepage: [F-Droid Repository](https://repo.gdgd.jp.net/), [Google Play](https://play.google.com/store/apps/details?id=com.apps.nekonium), [Amazon](https://www.amazon.co.jp/dp/B076FXPRBC/)
-- Source: <https://git.gdgd.jp.net/lin/nekonium/>
+- Source: <https://gogs.gdgd.jp.net/lin/nekonium>
 - Contact: [@lin@pleroma.gdgd.jp.net](https://pleroma.gdgd.jp.net/users/lin)
 - Platforms: Android
 - Features: Streaming Ready
@@ -67,7 +67,7 @@ Feel free to contact us to be added to this list!
 ## Alternative Web Interfaces
 ### Brutaldon
 - Homepage: <https://jfm.carcosa.net/projects/software/brutaldon/>
-- Source Code: <https://github.com/jfmcbrayer/brutaldon>
+- Source Code: <https://git.carcosa.net/jmcbray/brutaldon>
 - Contact: [@gcupc@glitch.social](https://glitch.social/users/gcupc)
 - Features: No Streaming
 

docs/config.md

@@ -23,6 +23,7 @@ Note: `strip_exif` has been replaced by `Pleroma.Upload.Filter.Mogrify`.
 * `truncated_namespace`: If you use S3 compatible service such as Digital Ocean Spaces or CDN, set folder name or "" etc.
 For example, when using CDN to S3 virtual host format, set "".
 At this time, write CNAME to CDN in public_endpoint.
+* `streaming_enabled`: Enable streaming uploads, when enabled the file will be sent to the server in chunks as it's being read. This may be unsupported by some providers, try disabling this if you have upload problems.
 
 ## Pleroma.Upload.Filter.Mogrify
 
@@ -521,7 +522,7 @@ config :auto_linker,
     class: false,
     strip_prefix: false,
     new_window: false,
-    rel: false
+    rel: "ugc"
   ]
 ```
 
@@ -707,6 +708,8 @@ Configure OAuth 2 provider capabilities:
 * `pack_extensions`: A list of file extensions for emojis, when no emoji.txt for a pack is present. Example `[".png", ".gif"]`
 * `groups`: Emojis are ordered in groups (tags). This is an array of key-value pairs where the key is the groupname and the value the location or array of locations. `*` can be used as a wildcard. Example `[Custom: ["/emoji/*.png", "/emoji/custom/*.png"]]`
 * `default_manifest`: Location of the JSON-manifest. This manifest contains information about the emoji-packs you can download. Currently only one manifest can be added (no arrays).
+* `shared_pack_cache_seconds_per_file`: When an emoji pack is shared, the archive is created and cached in
+  memory for this amount of seconds multiplied by the number of files.
 
 ## Database options
 
@@ -727,6 +730,8 @@ This will probably take a long time.
 
 This is an advanced feature and disabled by default.
 
+If your instance is behind a reverse proxy you must enable and configure [`Pleroma.Plugs.RemoteIp`](#pleroma-plugs-remoteip).
+
 A keyword list of rate limiters where a key is a limiter name and value is the limiter configuration. The basic configuration is a tuple where:
 
 * The first element: `scale` (Integer). The time scale in milliseconds.
@@ -753,3 +758,16 @@ Available caches:
 
 * `:activity_pub` - activity pub routes (except question activities). Defaults to `nil` (no expiration).
 * `:activity_pub_question` - activity pub routes (question activities). Defaults to `30_000` (30 seconds).
+
+## Pleroma.Plugs.RemoteIp
+
+**If your instance is not behind at least one reverse proxy, you should not enable this plug.**
+
+`Pleroma.Plugs.RemoteIp` is a shim to call [`RemoteIp`](https://git.pleroma.social/pleroma/remote_ip) but with runtime configuration.
+
+Available options:
+
+* `enabled` - Enable/disable the plug. Defaults to `false`.
+* `headers` - A list of strings naming the `req_headers` to use when deriving the `remote_ip`. Order does not matter. Defaults to `~w[forwarded x-forwarded-for x-client-ip x-real-ip]`.
+* `proxies` - A list of strings in [CIDR](https://en.wikipedia.org/wiki/CIDR) notation specifying the IPs of known proxies. Defaults to `[]`.
+* `reserved` - Defaults to [localhost](https://en.wikipedia.org/wiki/Localhost) and [private network](https://en.wikipedia.org/wiki/Private_network).

docs/installation/alpine_linux_en.md

@@ -1,7 +1,9 @@
 # Installing on Alpine Linux
 ## Installation
 
-This guide is a step-by-step installation guide for Alpine Linux. It also assumes that you have administrative rights, either as root or a user with [sudo permissions](https://www.linode.com/docs/tools-reference/custom-kernels-distros/install-alpine-linux-on-your-linode/#configuration). If you want to run this guide with root, ignore the `sudo` at the beginning of the lines, unless it calls a user like `sudo -Hu pleroma`; in this case, use `su -l <username> -s $SHELL -c 'command'` instead.
+This guide is a step-by-step installation guide for Alpine Linux. The instructions were verified against Alpine v3.10 standard image. You might miss additional dependencies if you use `netboot` instead.
+
+It assumes that you have administrative rights, either as root or a user with [sudo permissions](https://www.linode.com/docs/tools-reference/custom-kernels-distros/install-alpine-linux-on-your-linode/#configuration). If you want to run this guide with root, ignore the `sudo` at the beginning of the lines, unless it calls a user like `sudo -Hu pleroma`; in this case, use `su -l <username> -s $SHELL -c 'command'` instead.
 
 ### Required packages
 
@@ -20,12 +22,13 @@ This guide is a step-by-step installation guide for Alpine Linux. It also assume
 
 ### Prepare the system
 
-* First make sure to have the community repository enabled:
+* The community repository must be enabled in `/etc/apk/repositories`. Depending on which version and mirror you use this looks like `http://alpine.42.fr/v3.10/community`. If you autogenerated the mirror during installation:
 
 ```shell
-echo "https://nl.alpinelinux.org/alpine/latest-stable/community" | sudo tee -a /etc/apk/repository
+awk 'NR==2' /etc/apk/repositories | sed 's/main/community/' | tee -a /etc/apk/repositories
 ```
 
+
 * Then update the system, if not already done:
 
 ```shell
@@ -77,7 +80,8 @@ sudo rc-update add postgresql
 * Add a new system user for the Pleroma service:
 
 ```shell
-sudo adduser -S -s /bin/false -h /opt/pleroma -H pleroma
+sudo addgroup pleroma
+sudo adduser -S -s /bin/false -h /opt/pleroma -H -G pleroma pleroma
 ```
 
 **Note**: To execute a single command as the Pleroma system user, use `sudo -Hu pleroma command`. You can also switch to a shell by using `sudo -Hu pleroma $SHELL`. If you don’t have and want `sudo` on your system, you can use `su` as root user (UID 0) for a single command by using `su -l pleroma -s $SHELL -c 'command'` and `su -l pleroma -s $SHELL` for starting a shell.
@@ -164,7 +168,26 @@ If that doesn’t work, make sure, that nginx is not already running. If it stil
 sudo cp /opt/pleroma/installation/pleroma.nginx /etc/nginx/conf.d/pleroma.conf
 ```
 
-* Before starting nginx edit the configuration and change it to your needs (e.g. change servername, change cert paths)
+* Before starting nginx edit the configuration and change it to your needs. You must change change `server_name` and the paths to the certificates. You can use `nano` (install with `apk add nano` if missing).
+
+```
+server {
+    server_name    your.domain;
+    listen         80;
+    ...
+}
+
+server {
+    server_name your.domain;
+    listen 443 ssl http2;
+    ...
+    ssl_trusted_certificate   /etc/letsencrypt/live/your.domain/chain.pem;
+    ssl_certificate           /etc/letsencrypt/live/your.domain/fullchain.pem;
+    ssl_certificate_key       /etc/letsencrypt/live/your.domain/privkey.pem;
+    ...
+}
+```
+
 * Enable and start nginx:
 
 ```shell

docs/installation/debian_based_jp.md

@@ -5,180 +5,179 @@
 
 ## インストール
 
-このガイドはDebian Stretchを仮定しています。Ubuntu 16.04でも可能です。
+このガイドはDebian Stretchを利用することを想定しています。Ubuntu 16.04や18.04でもおそらく動作します。また、ユーザはrootもしくはsudoにより管理者権限を持っていることを前提とします。もし、以下の操作をrootユーザで行う場合は、 `sudo` を無視してください。ただし、`sudo -Hu pleroma` のようにユーザを指定している場合には `su <username> -s $SHELL -c 'command'` を代わりに使ってください。
 
 ### 必要なソフトウェア
 
-- PostgreSQL 9.6+ (postgresql-contrib-9.6 または他のバージョンの PSQL をインストールしてください)
+- PostgreSQL 9.6以上 (Ubuntu16.04では9.5しか提供されていないので，[](https://www.postgresql.org/download/linux/ubuntu/)こちらから新しいバージョンを入手してください)
-- Elixir 1.5 以上 ([Debianのリポジトリからインストールしないこと！！！ ここからインストールすること！](https://elixir-lang.org/install.html#unix-and-unix-like))。または [asdf](https://github.com/asdf-vm/asdf) を pleroma ユーザーでインストール。
+- postgresql-contrib 9.6以上 (同上)
-- erlang-dev
+- Elixir 1.5 以上 ([Debianのリポジトリからインストールしないこと！！！ ここからインストールすること！](https://elixir-lang.org/install.html#unix-and-unix-like)。または [asdf](https://github.com/asdf-vm/asdf) をpleromaユーザーでインストールしてください)
+  - erlang-dev
 - erlang-tools
 - erlang-parsetools
+- erlang-eldap (LDAP認証を有効化するときのみ必要)
 - erlang-ssh
-- erlang-xmerl (Jessieではバックポートからインストールすること！)
+- erlang-xmerl
 - git
 - build-essential
-- openssh
+
-- openssl
+#### このガイドで利用している追加パッケージ
-- nginx prefered (Apacheも動くかもしれませんが、誰もテストしていません！)
+
-- certbot (または何らかのACME Let's encryptクライアント)
+- nginx (おすすめです。他のリバースプロキシを使う場合は、参考となる設定をこのリポジトリから探してください)
+- certbot (または何らかのLet's Encrypt向けACMEクライアント)
 
 ### システムを準備する
 
 * まずシステムをアップデートしてください。
 ```
-apt update && apt dist-upgrade
+sudo apt update
+sudo apt full-upgrade
 ```
 
-* 複数のツールとpostgresqlをインストールします。あとで必要になるので。
+* 上記に挙げたパッケージをインストールしておきます。
 ```
-apt install git build-essential openssl ssh sudo postgresql-9.6 postgresql-contrib-9.6
+sudo apt install git build-essential postgresql postgresql-contrib
 ```
-(postgresqlのバージョンは、あなたのディストロにあわせて変えてください。または、バージョン番号がいらないかもしれません。)
+
 
 ### ElixirとErlangをインストールします
 
 * Erlangのリポジトリをダウンロードおよびインストールします。
 ```
-wget -P /tmp/ https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb && sudo dpkg -i /tmp/erlang-solutions_1.0_all.deb
+wget -P /tmp/ https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
+sudo dpkg -i /tmp/erlang-solutions_1.0_all.deb
 ```
 
 * ElixirとErlangをインストールします、
 ```
-apt update && apt install elixir erlang-dev erlang-parsetools erlang-xmerl erlang-tools erlang-ssh
+sudo apt update
+sudo apt install elixir erlang-dev erlang-parsetools erlang-xmerl erlang-tools erlang-ssh
 ```
 
 ### Pleroma BE (バックエンド) をインストールします
 
-*  新しいユーザーを作ります。
+*  Pleroma用に新しいユーザーを作ります。
-```
-adduser pleroma
-``` 
-(Give it any password you want, make it STRONG)
 
-*  新しいユーザーをsudoグループに入れます。
 ```
-usermod -aG sudo pleroma
+sudo useradd -r -s /bin/false -m -d /var/lib/pleroma -U pleroma
 ```
 
-*  新しいユーザーに変身し、ホームディレクトリに移動します。
+**注意**: Pleromaユーザとして単発のコマンドを実行したい場合はは、`sudo -Hu pleroma command` を使ってください。シェルを使いたい場合は `sudo -Hu pleroma $SHELL`です。もし `sudo` を使わない場合は、rootユーザで `su -l pleroma -s $SHELL -c 'command'` とすることでコマンドを、`su -l pleroma -s $SHELL` とすることでシェルを開始できます。
-```
-su pleroma
-cd ~
-```
 
 *  Gitリポジトリをクローンします。
 ```
-git clone -b master https://git.pleroma.social/pleroma/pleroma
+sudo mkdir -p /opt/pleroma
+sudo chown -R pleroma:pleroma /opt/pleroma
+sudo -Hu pleroma git clone -b master https://git.pleroma.social/pleroma/pleroma /opt/pleroma
 ```
 
 *  新しいディレクトリに移動します。
 ```
-cd pleroma/
+cd /opt/pleroma
 ```
 
 * Pleromaが依存するパッケージをインストールします。Hexをインストールしてもよいか聞かれたら、yesを入力してください。
 ```
-mix deps.get
+sudo -Hu pleroma mix deps.get
 ```
 
 * コンフィギュレーションを生成します。
 ```
-mix pleroma.instance gen
+sudo -Hu pleroma mix pleroma.instance gen
 ```
     * rebar3をインストールしてもよいか聞かれたら、yesを入力してください。
-    * この処理には時間がかかります。私もよく分かりませんが、何らかのコンパイルが行われているようです。
+    * このときにpleromaの一部がコンパイルされるため、この処理には時間がかかります。
-    * あなたのインスタンスについて、いくつかの質問があります。その回答は `config/generated_config.exs` というコンフィギュレーションファイルに保存されます。
+    * あなたのインスタンスについて、いくつかの質問されます。この質問により `config/generated_config.exs` という設定ファイルが生成されます。
 
-**注意**: メディアプロクシを有効にすると回答して、なおかつ、キャッシュのURLは空欄のままにしている場合は、`generated_config.exs` を編集して、`base_url` で始まる行をコメントアウトまたは削除してください。そして、上にある行の `true` の後にあるコンマを消してください。
 
 * コンフィギュレーションを確認して、もし問題なければ、ファイル名を変更してください。
 ```
 mv config/{generated_config.exs,prod.secret.exs}
 ```
 
-* これまでのコマンドで、すでに `config/setup_db.psql` というファイルが作られています。このファイルをもとに、データベースを作成します。
+* 先程のコマンドで、すでに `config/setup_db.psql` というファイルが作られています。このファイルをもとに、データベースを作成します。
 ```
-sudo su postgres -c 'psql -f config/setup_db.psql'
+sudo -Hu pleroma mix pleroma.instance gen
 ```
 
-* そして、データベースのミグレーションを実行します。
+* そして、データベースのマイグレーションを実行します。
 ```
-MIX_ENV=prod mix ecto.migrate
+sudo -Hu pleroma MIX_ENV=prod mix ecto.migrate
 ```
 
-* Pleromaを起動できるようになりました。
+* これでPleromaを起動できるようになりました。
 ```
-MIX_ENV=prod mix phx.server
+sudo -Hu pleroma MIX_ENV=prod mix phx.server
 ```
 
-### インストールを終わらせる
+### インストールの最終段階
 
-あなたの新しいインスタンスを世界に向けて公開するには、nginxまたは何らかのウェブサーバー (プロクシ) を使用する必要があります。また、Pleroma のためにシステムサービスファイルを作成する必要があります。
+あなたの新しいインスタンスを世界に向けて公開するには、nginx等のWebサーバやプロキシサーバをPleromaの前段に使用する必要があります。また、Pleroma のためにシステムサービスファイルを作成する必要があります。
 
 #### Nginx
 
 * まだインストールしていないなら、nginxをインストールします。
 ```
-apt install nginx
+sudo apt install nginx
 ```
 
 * SSLをセットアップします。他の方法でもよいですが、ここではcertbotを説明します。
 certbotを使うならば、まずそれをインストールします。
 ```
-apt install certbot
+sudo apt install certbot
 ```
 そしてセットアップします。
 ```
-mkdir -p /var/lib/letsencrypt/.well-known
+sudo mkdir -p /var/lib/letsencrypt/
-% certbot certonly --email your@emailaddress --webroot -w /var/lib/letsencrypt/ -d yourdomain
+sudo certbot certonly --email <your@emailaddress> -d <yourdomain> --standalone
 ```
-もしうまくいかないときは、先にnginxを設定してください。ssl "on" を "off" に変えてから再試行してください。
+もしうまくいかないときは、nginxが正しく動いていない可能性があります。先にnginxを設定してください。ssl "on" を "off" に変えてから再試行してください。
 
 ---
 
-* nginxコンフィギュレーションの例をnginxフォルダーにコピーします。
+* nginxの設定ファイルサンプルをnginxフォルダーにコピーします。
 ```
-cp /home/pleroma/pleroma/installation/pleroma.nginx /etc/nginx/sites-enabled/pleroma.nginx
+sudo cp /opt/pleroma/installation/pleroma.nginx /etc/nginx/sites-available/pleroma.nginx
+sudo ln -s /etc/nginx/sites-available/pleroma.nginx /etc/nginx/sites-enabled/pleroma.nginx
 ```
 
-* nginxを起動する前に、コンフィギュレーションを編集してください。例えば、サーバー名、証明書のパスなどを変更する必要があります。
+* nginxを起動する前に、設定ファイルを編集してください。例えば、サーバー名、証明書のパスなどを変更する必要があります。
 * nginxを再起動します。
 ```
-systemctl reload nginx.service
+sudo systemctl enable --now nginx.service
 ```
 
+もし証明書を更新する必要が出てきた場合には、nginxの関連するlocationブロックのコメントアウトを外し、以下のコマンドを動かします。
+
+```
+sudo certbot certonly --email <your@emailaddress> -d <yourdomain> --webroot -w /var/lib/letsencrypt/
+```
+
+#### 他のWebサーバやプロキシ
+これに関してはサンプルが `/opt/pleroma/installation/` にあるので、探してみてください。
+
 #### Systemd サービス
 
-* サービスファイルの例をコピーします。
+* サービスファイルのサンプルをコピーします。
 ```
-cp /home/pleroma/pleroma/installation/pleroma.service /usr/lib/systemd/system/pleroma.service
+sudo cp /opt/pleroma/installation/pleroma.service /etc/systemd/system/pleroma.service
 ```
 
-* サービスファイルを変更します。すべてのパスが正しいことを確認してください。また、`[Service]` セクションに以下の行があることを確認してください。
+* サービスファイルを変更します。すべてのパスが正しいことを確認してください
+* サービスを有効化し `pleroma.service` を開始してください
 ```
-Environment="MIX_ENV=prod"
+sudo systemctl enable --now pleroma.service
 ```
 
-* `pleroma.service` を enable および start してください。
+#### 初期ユーザの作成
+
+新たにインスタンスを作成したら、以下のコマンドにより管理者権限を持った初期ユーザを作成できます。
+
 ```
-systemctl enable --now pleroma.service
+sudo -Hu pleroma MIX_ENV=prod mix pleroma.user new <username> <your@emailaddress> --admin
 ```
 
-#### モデレーターを作る
+#### その他の設定とカスタマイズ
-
-新たにユーザーを作ったら、モデレーター権限を与えたいかもしれません。以下のタスクで可能です。
-```
-mix set_moderator username [true|false]
-```
-
-モデレーターはすべてのポストを消すことができます。将来的には他のことも可能になるかもしれません。
-
-#### メディアプロクシを有効にする
-
-`generate_config` でメディアプロクシを有効にしているなら、すでにメディアプロクシが動作しています。あとから設定を変更したいなら、[How to activate mediaproxy](How-to-activate-mediaproxy) を見てください。
-
-#### コンフィギュレーションとカスタマイズ
 
 * [Backup your instance](backup.html)
 * [Configuration tips](general-tips-for-customizing-pleroma-fe.html)

installation/pleroma-mongooseim.cfg

@@ -215,7 +215,9 @@
       ]}
   ]},
 
-  { 5222, ejabberd_c2s, [
+  %% If you want dual stack, you have to clone this entire config stanza
+  %% and change the bind to "::"
+  { {5222, "0.0.0.0"}, ejabberd_c2s, [
 
 			%%
 			%% If TLS is compiled in and you installed a SSL
@@ -246,7 +248,9 @@
   %%			{max_stanza_size, 65536}
   %%		       ]},
 
-  { 5269, ejabberd_s2s_in, [
+  %% If you want dual stack, you have to clone this entire config stanza
+  %% and change the bind to "::"
+  { {5269, "0.0.0.0"}, ejabberd_s2s_in, [
 			   {shaper, s2s_shaper},
 			   {max_stanza_size, 131072},
 			   {protocol_options, ["no_sslv3"]}

installation/pleroma.nginx

@@ -70,6 +70,7 @@ server {
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "upgrade";
         proxy_set_header Host $http_host;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
         # this is explicitly IPv4 since Pleroma.Web.Endpoint binds on IPv4 only
         # and `localhost.` resolves to [::0] on some systems: see issue #930

lib/mix/pleroma.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Pleroma do

lib/mix/tasks/pleroma/database.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Database do

lib/mix/tasks/pleroma/ecto/ecto.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-onl
 
 defmodule Mix.Tasks.Pleroma.Ecto do

lib/mix/tasks/pleroma/ecto/migrate.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-onl
 
 defmodule Mix.Tasks.Pleroma.Ecto.Migrate do

lib/mix/tasks/pleroma/ecto/rollback.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-onl
 
 defmodule Mix.Tasks.Pleroma.Ecto.Rollback do

lib/mix/tasks/pleroma/emoji.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Emoji do
@@ -235,7 +235,7 @@ defmodule Mix.Tasks.Pleroma.Emoji do
         cwd: tmp_pack_dir
       )
 
-    emoji_map = Pleroma.Emoji.make_shortcode_to_file_map(tmp_pack_dir, exts)
+    emoji_map = Pleroma.Emoji.Loader.make_shortcode_to_file_map(tmp_pack_dir, exts)
 
     File.write!(files_name, Jason.encode!(emoji_map, pretty: true))
 

lib/mix/tasks/pleroma/instance.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Instance do

lib/mix/tasks/pleroma/relay.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Relay do

lib/mix/tasks/pleroma/uploads.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.Uploads do

lib/mix/tasks/pleroma/user.ex

@@ -1,10 +1,9 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Mix.Tasks.Pleroma.User do
   use Mix.Task
-  import Ecto.Changeset
   import Mix.Pleroma
   alias Pleroma.User
   alias Pleroma.UserInviteToken
@@ -228,9 +227,9 @@ defmodule Mix.Tasks.Pleroma.User do
       shell_info("Deactivating #{user.nickname}")
       User.deactivate(user)
 
-      {:ok, friends} = User.get_friends(user)
+      user
-
+      |> User.get_friends()
-      Enum.each(friends, fn friend ->
+      |> Enum.each(fn friend ->
         user = User.get_cached_by_id(user.id)
 
         shell_info("Unsubscribing #{friend.nickname} from #{user.nickname}")
@@ -405,7 +404,7 @@ defmodule Mix.Tasks.Pleroma.User do
     start_pleroma()
 
     with %User{local: true} = user <- User.get_cached_by_nickname(nickname) do
-      {:ok, _} = User.delete_user_activities(user)
+      User.delete_user_activities(user)
       shell_info("User #{nickname} statuses deleted.")
     else
       _ ->
@@ -443,39 +442,21 @@ defmodule Mix.Tasks.Pleroma.User do
   end
 
   defp set_moderator(user, value) do
-    info_cng = User.Info.admin_api_update(user.info, %{is_moderator: value})
+    {:ok, user} = User.update_info(user, &User.Info.admin_api_update(&1, %{is_moderator: value}))
-
-    user_cng =
-      Ecto.Changeset.change(user)
-      |> put_embed(:info, info_cng)
-
-    {:ok, user} = User.update_and_set_cache(user_cng)
 
     shell_info("Moderator status of #{user.nickname}: #{user.info.is_moderator}")
     user
   end
 
   defp set_admin(user, value) do
-    info_cng = User.Info.admin_api_update(user.info, %{is_admin: value})
+    {:ok, user} = User.update_info(user, &User.Info.admin_api_update(&1, %{is_admin: value}))
-
-    user_cng =
-      Ecto.Changeset.change(user)
-      |> put_embed(:info, info_cng)
-
-    {:ok, user} = User.update_and_set_cache(user_cng)
 
     shell_info("Admin status of #{user.nickname}: #{user.info.is_admin}")
     user
   end
 
   defp set_locked(user, value) do
-    info_cng = User.Info.user_upgrade(user.info, %{locked: value})
+    {:ok, user} = User.update_info(user, &User.Info.user_upgrade(&1, %{locked: value}))
-
-    user_cng =
-      Ecto.Changeset.change(user)
-      |> put_embed(:info, info_cng)
-
-    {:ok, user} = User.update_and_set_cache(user_cng)
 
     shell_info("Locked status of #{user.nickname}: #{user.info.locked}")
     user

lib/pleroma/activity.ex

@@ -21,7 +21,7 @@ defmodule Pleroma.Activity do
   @type t :: %__MODULE__{}
   @type actor :: String.t()
 
-  @primary_key {:id, Pleroma.FlakeId, autogenerate: true}
+  @primary_key {:id, FlakeId.Ecto.CompatType, autogenerate: true}
 
   # https://github.com/tootsuite/mastodon/blob/master/app/models/notification.rb#L19
   @mastodon_notification_types %{
@@ -137,11 +137,18 @@ defmodule Pleroma.Activity do
     |> Repo.one()
   end
 
+  @spec get_by_id(String.t()) :: Activity.t() | nil
   def get_by_id(id) do
-    Activity
+    case FlakeId.flake_id?(id) do
-    |> where([a], a.id == ^id)
+      true ->
-    |> restrict_deactivated_users()
+        Activity
-    |> Repo.one()
+        |> where([a], a.id == ^id)
+        |> restrict_deactivated_users()
+        |> Repo.one()
+
+      _ ->
+        nil
+    end
   end
 
   def get_by_id_with_object(id) do

lib/pleroma/activity/queries.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Activity.Queries do

lib/pleroma/activity_expiration.ex

@@ -7,7 +7,6 @@ defmodule Pleroma.ActivityExpiration do
 
   alias Pleroma.Activity
   alias Pleroma.ActivityExpiration
-  alias Pleroma.FlakeId
   alias Pleroma.Repo
 
   import Ecto.Changeset
@@ -17,7 +16,7 @@ defmodule Pleroma.ActivityExpiration do
   @min_activity_lifetime :timer.hours(1)
 
   schema "activity_expirations" do
-    belongs_to(:activity, Activity, type: FlakeId)
+    belongs_to(:activity, Activity, type: FlakeId.Ecto.CompatType)
     field(:scheduled_at, :naive_datetime)
   end
 

lib/pleroma/application.ex

@@ -35,7 +35,6 @@ defmodule Pleroma.Application do
         Pleroma.Config.TransferTask,
         Pleroma.Emoji,
         Pleroma.Captcha,
-        Pleroma.FlakeId,
         Pleroma.Daemons.ScheduledActivityDaemon,
         Pleroma.Daemons.ActivityExpirationDaemon
       ] ++
@@ -102,10 +101,14 @@ defmodule Pleroma.Application do
       build_cachex("rich_media", default_ttl: :timer.minutes(120), limit: 5000),
       build_cachex("scrubber", limit: 2500),
       build_cachex("idempotency", expiration: idempotency_expiration(), limit: 2500),
-      build_cachex("web_resp", limit: 2500)
+      build_cachex("web_resp", limit: 2500),
+      build_cachex("emoji_packs", expiration: emoji_packs_expiration(), limit: 10)
     ]
   end
 
+  defp emoji_packs_expiration,
+    do: expiration(default: :timer.seconds(5 * 60), interval: :timer.seconds(60))
+
   defp idempotency_expiration,
     do: expiration(default: :timer.seconds(6 * 60 * 60), interval: :timer.seconds(60))
 

lib/pleroma/bbs/handler.ex

@@ -42,7 +42,7 @@ defmodule Pleroma.BBS.Handler do
   end
 
   def puts_activity(activity) do
-    status = Pleroma.Web.MastodonAPI.StatusView.render("status.json", %{activity: activity})
+    status = Pleroma.Web.MastodonAPI.StatusView.render("show.json", %{activity: activity})
     IO.puts("-- #{status.id} by #{status.account.display_name} (#{status.account.acct})")
     IO.puts(HtmlSanitizeEx.strip_tags(status.content))
     IO.puts("")

lib/pleroma/bookmark.ex

@@ -10,20 +10,20 @@ defmodule Pleroma.Bookmark do
 
   alias Pleroma.Activity
   alias Pleroma.Bookmark
-  alias Pleroma.FlakeId
   alias Pleroma.Repo
   alias Pleroma.User
 
   @type t :: %__MODULE__{}
 
   schema "bookmarks" do
-    belongs_to(:user, User, type: FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
-    belongs_to(:activity, Activity, type: FlakeId)
+    belongs_to(:activity, Activity, type: FlakeId.Ecto.CompatType)
 
     timestamps()
   end
 
-  @spec create(FlakeId.t(), FlakeId.t()) :: {:ok, Bookmark.t()} | {:error, Changeset.t()}
+  @spec create(FlakeId.Ecto.CompatType.t(), FlakeId.Ecto.CompatType.t()) ::
+          {:ok, Bookmark.t()} | {:error, Changeset.t()}
   def create(user_id, activity_id) do
     attrs = %{
       user_id: user_id,
@@ -37,7 +37,7 @@ defmodule Pleroma.Bookmark do
     |> Repo.insert()
   end
 
-  @spec for_user_query(FlakeId.t()) :: Ecto.Query.t()
+  @spec for_user_query(FlakeId.Ecto.CompatType.t()) :: Ecto.Query.t()
   def for_user_query(user_id) do
     Bookmark
     |> where(user_id: ^user_id)
@@ -52,7 +52,8 @@ defmodule Pleroma.Bookmark do
     |> Repo.one()
   end
 
-  @spec destroy(FlakeId.t(), FlakeId.t()) :: {:ok, Bookmark.t()} | {:error, Changeset.t()}
+  @spec destroy(FlakeId.Ecto.CompatType.t(), FlakeId.Ecto.CompatType.t()) ::
+          {:ok, Bookmark.t()} | {:error, Changeset.t()}
   def destroy(user_id, activity_id) do
     from(b in Bookmark,
       where: b.user_id == ^user_id,

lib/pleroma/constants.ex

@@ -6,4 +6,16 @@ defmodule Pleroma.Constants do
   use Const
 
   const(as_public, do: "https://www.w3.org/ns/activitystreams#Public")
+
+  const(object_internal_fields,
+    do: [
+      "likes",
+      "like_count",
+      "announcements",
+      "announcement_count",
+      "emoji",
+      "context_id",
+      "deleted_activity_id"
+    ]
+  )
 end

lib/pleroma/conversation/participation.ex

@@ -13,10 +13,10 @@ defmodule Pleroma.Conversation.Participation do
   import Ecto.Query
 
   schema "conversation_participations" do
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     belongs_to(:conversation, Conversation)
     field(:read, :boolean, default: false)
-    field(:last_activity_id, Pleroma.FlakeId, virtual: true)
+    field(:last_activity_id, FlakeId.Ecto.CompatType, virtual: true)
 
     has_many(:recipient_ships, RecipientShip)
     has_many(:recipients, through: [:recipient_ships, :user])

lib/pleroma/conversation/participation_recipient_ship.ex

@@ -12,7 +12,7 @@ defmodule Pleroma.Conversation.Participation.RecipientShip do
   import Ecto.Changeset
 
   schema "conversation_participation_recipient_ships" do
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     belongs_to(:participation, Participation)
   end
 

lib/pleroma/delivery.ex

@@ -6,7 +6,6 @@ defmodule Pleroma.Delivery do
   use Ecto.Schema
 
   alias Pleroma.Delivery
-  alias Pleroma.FlakeId
   alias Pleroma.Object
   alias Pleroma.Repo
   alias Pleroma.User
@@ -16,7 +15,7 @@ defmodule Pleroma.Delivery do
   import Ecto.Query
 
   schema "deliveries" do
-    belongs_to(:user, User, type: FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     belongs_to(:object, Object)
   end
 

lib/pleroma/docs/markdown.ex

@@ -23,7 +23,7 @@ defmodule Pleroma.Docs.Markdown do
 
       IO.write(file, "#{group[:description]}\n")
 
-      for child <- group[:children] do
+      for child <- group[:children] || [] do
         print_child_header(file, child)
 
         print_suggestions(file, child[:suggestions])
@@ -44,6 +44,17 @@ defmodule Pleroma.Docs.Markdown do
     {:ok, config_path}
   end
 
+  defp print_child_header(file, %{key: key, type: type, description: description} = _child) do
+    IO.write(
+      file,
+      "- `#{inspect(key)}` (`#{inspect(type)}`): #{description}  \n"
+    )
+  end
+
+  defp print_child_header(file, %{key: key, type: type} = _child) do
+    IO.write(file, "- `#{inspect(key)}` (`#{inspect(type)}`)  \n")
+  end
+
   defp print_suggestion(file, suggestion) when is_list(suggestion) do
     IO.write(file, "  `#{inspect(suggestion)}`\n")
   end
@@ -59,20 +70,19 @@ defmodule Pleroma.Docs.Markdown do
 
   defp print_suggestions(_file, nil), do: nil
 
-  defp print_suggestions(file, suggestions) do
+  defp print_suggestions(_file, ""), do: nil
-    IO.write(file, "Suggestions:\n")
 
+  defp print_suggestions(file, suggestions) do
     if length(suggestions) > 1 do
+      IO.write(file, "Suggestions:\n")
+
       for suggestion <- suggestions do
         print_suggestion(file, suggestion, true)
       end
     else
+      IO.write(file, "  Suggestion: ")
+
       print_suggestion(file, List.first(suggestions))
     end
   end
-
-  defp print_child_header(file, child) do
-    IO.write(file, "- `#{inspect(child[:key])}` -`#{inspect(child[:type])}`  \n")
-    IO.write(file, "#{child[:description]}  \n")
-  end
 end

lib/pleroma/emoji.ex

@@ -4,24 +4,37 @@
 
 defmodule Pleroma.Emoji do
   @moduledoc """
-  The emojis are loaded from:
+  This GenServer stores in an ETS table the list of the loaded emojis,
-
+  and also allows to reload the list at runtime.
-    * emoji packs in INSTANCE-DIR/emoji
-    * the files: `config/emoji.txt` and `config/custom_emoji.txt`
-    * glob paths, nested folder is used as tag name for grouping e.g. priv/static/emoji/custom/nested_folder
-
-  This GenServer stores in an ETS table the list of the loaded emojis, and also allows to reload the list at runtime.
   """
   use GenServer
 
+  alias Pleroma.Emoji.Loader
+
   require Logger
 
-  @type pattern :: Regex.t() | module() | String.t()
-  @type patterns :: pattern() | [pattern()]
-  @type group_patterns :: keyword(patterns())
-
   @ets __MODULE__.Ets
-  @ets_options [:ordered_set, :protected, :named_table, {:read_concurrency, true}]
+  @ets_options [
+    :ordered_set,
+    :protected,
+    :named_table,
+    {:read_concurrency, true}
+  ]
+
+  defstruct [:code, :file, :tags, :safe_code, :safe_file]
+
+  @doc "Build emoji struct"
+  def build({code, file, tags}) do
+    %__MODULE__{
+      code: code,
+      file: file,
+      tags: tags,
+      safe_code: Pleroma.HTML.strip_tags(code),
+      safe_file: Pleroma.HTML.strip_tags(file)
+    }
+  end
+
+  def build({code, file}), do: build({code, file, []})
 
   @doc false
   def start_link(_) do
@@ -44,11 +57,14 @@ defmodule Pleroma.Emoji do
   end
 
   @doc "Returns all the emojos!!"
-  @spec get_all() :: [{String.t(), String.t()}, ...]
+  @spec get_all() :: list({String.t(), String.t(), String.t()})
   def get_all do
     :ets.tab2list(@ets)
   end
 
+  @doc "Clear out old emojis"
+  def clear_all, do: :ets.delete_all_objects(@ets)
+
   @doc false
   def init(_) do
     @ets = :ets.new(@ets, @ets_options)
@@ -58,13 +74,13 @@ defmodule Pleroma.Emoji do
 
   @doc false
   def handle_cast(:reload, state) do
-    load()
+    update_emojis(Loader.load())
     {:noreply, state}
   end
 
   @doc false
   def handle_call(:reload, _from, state) do
-    load()
+    update_emojis(Loader.load())
     {:reply, :ok, state}
   end
 
@@ -75,189 +91,11 @@ defmodule Pleroma.Emoji do
 
   @doc false
   def code_change(_old_vsn, state, _extra) do
-    load()
+    update_emojis(Loader.load())
     {:ok, state}
   end
 
-  defp load do
+  defp update_emojis(emojis) do
-    emoji_dir_path =
+    :ets.insert(@ets, emojis)
-      Path.join(
-        Pleroma.Config.get!([:instance, :static_dir]),
-        "emoji"
-      )
-
-    emoji_groups = Pleroma.Config.get([:emoji, :groups])
-
-    case File.ls(emoji_dir_path) do
-      {:error, :enoent} ->
-        # The custom emoji directory doesn't exist,
-        # don't do anything
-        nil
-
-      {:error, e} ->
-        # There was some other error
-        Logger.error("Could not access the custom emoji directory #{emoji_dir_path}: #{e}")
-
-      {:ok, results} ->
-        grouped =
-          Enum.group_by(results, fn file -> File.dir?(Path.join(emoji_dir_path, file)) end)
-
-        packs = grouped[true] || []
-        files = grouped[false] || []
-
-        # Print the packs we've found
-        Logger.info("Found emoji packs: #{Enum.join(packs, ", ")}")
-
-        if not Enum.empty?(files) do
-          Logger.warn(
-            "Found files in the emoji folder. These will be ignored, please move them to a subdirectory\nFound files: #{
-              Enum.join(files, ", ")
-            }"
-          )
-        end
-
-        emojis =
-          Enum.flat_map(
-            packs,
-            fn pack -> load_pack(Path.join(emoji_dir_path, pack), emoji_groups) end
-          )
-
-        true = :ets.insert(@ets, emojis)
-    end
-
-    # Compat thing for old custom emoji handling & default emoji,
-    # it should run even if there are no emoji packs
-    shortcode_globs = Pleroma.Config.get([:emoji, :shortcode_globs], [])
-
-    emojis =
-      (load_from_file("config/emoji.txt", emoji_groups) ++
-         load_from_file("config/custom_emoji.txt", emoji_groups) ++
-         load_from_globs(shortcode_globs, emoji_groups))
-      |> Enum.reject(fn value -> value == nil end)
-
-    true = :ets.insert(@ets, emojis)
-
-    :ok
-  end
-
-  defp load_pack(pack_dir, emoji_groups) do
-    pack_name = Path.basename(pack_dir)
-
-    emoji_txt = Path.join(pack_dir, "emoji.txt")
-
-    if File.exists?(emoji_txt) do
-      load_from_file(emoji_txt, emoji_groups)
-    else
-      extensions = Pleroma.Config.get([:emoji, :pack_extensions])
-
-      Logger.info(
-        "No emoji.txt found for pack \"#{pack_name}\", assuming all #{Enum.join(extensions, ", ")} files are emoji"
-      )
-
-      make_shortcode_to_file_map(pack_dir, extensions)
-      |> Enum.map(fn {shortcode, rel_file} ->
-        filename = Path.join("/emoji/#{pack_name}", rel_file)
-
-        {shortcode, filename, [to_string(match_extra(emoji_groups, filename))]}
-      end)
-    end
-  end
-
-  def make_shortcode_to_file_map(pack_dir, exts) do
-    find_all_emoji(pack_dir, exts)
-    |> Enum.map(&Path.relative_to(&1, pack_dir))
-    |> Enum.map(fn f -> {f |> Path.basename() |> Path.rootname(), f} end)
-    |> Enum.into(%{})
-  end
-
-  def find_all_emoji(dir, exts) do
-    Enum.reduce(
-      File.ls!(dir),
-      [],
-      fn f, acc ->
-        filepath = Path.join(dir, f)
-
-        if File.dir?(filepath) do
-          acc ++ find_all_emoji(filepath, exts)
-        else
-          acc ++ [filepath]
-        end
-      end
-    )
-    |> Enum.filter(fn f -> Path.extname(f) in exts end)
-  end
-
-  defp load_from_file(file, emoji_groups) do
-    if File.exists?(file) do
-      load_from_file_stream(File.stream!(file), emoji_groups)
-    else
-      []
-    end
-  end
-
-  defp load_from_file_stream(stream, emoji_groups) do
-    stream
-    |> Stream.map(&String.trim/1)
-    |> Stream.map(fn line ->
-      case String.split(line, ~r/,\s*/) do
-        [name, file] ->
-          {name, file, [to_string(match_extra(emoji_groups, file))]}
-
-        [name, file | tags] ->
-          {name, file, tags}
-
-        _ ->
-          nil
-      end
-    end)
-    |> Enum.to_list()
-  end
-
-  defp load_from_globs(globs, emoji_groups) do
-    static_path = Path.join(:code.priv_dir(:pleroma), "static")
-
-    paths =
-      Enum.map(globs, fn glob ->
-        Path.join(static_path, glob)
-        |> Path.wildcard()
-      end)
-      |> Enum.concat()
-
-    Enum.map(paths, fn path ->
-      tag = match_extra(emoji_groups, Path.join("/", Path.relative_to(path, static_path)))
-      shortcode = Path.basename(path, Path.extname(path))
-      external_path = Path.join("/", Path.relative_to(path, static_path))
-      {shortcode, external_path, [to_string(tag)]}
-    end)
-  end
-
-  @doc """
-  Finds a matching group for the given emoji filename
-  """
-  @spec match_extra(group_patterns(), String.t()) :: atom() | nil
-  def match_extra(group_patterns, filename) do
-    match_group_patterns(group_patterns, fn pattern ->
-      case pattern do
-        %Regex{} = regex -> Regex.match?(regex, filename)
-        string when is_binary(string) -> filename == string
-      end
-    end)
-  end
-
-  defp match_group_patterns(group_patterns, matcher) do
-    Enum.find_value(group_patterns, fn {group, patterns} ->
-      patterns =
-        patterns
-        |> List.wrap()
-        |> Enum.map(fn pattern ->
-          if String.contains?(pattern, "*") do
-            ~r(#{String.replace(pattern, "*", ".*")})
-          else
-            pattern
-          end
-        end)
-
-      Enum.any?(patterns, matcher) && group
-    end)
   end
 end

lib/pleroma/emoji/formatter.ex

@@ -0,0 +1,59 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Emoji.Formatter do
+  alias Pleroma.Emoji
+  alias Pleroma.HTML
+  alias Pleroma.Web.MediaProxy
+
+  def emojify(text) do
+    emojify(text, Emoji.get_all())
+  end
+
+  def emojify(text, nil), do: text
+
+  def emojify(text, emoji, strip \\ false) do
+    Enum.reduce(emoji, text, fn
+      {_, %Emoji{safe_code: emoji, safe_file: file}}, text ->
+        String.replace(text, ":#{emoji}:", prepare_emoji_html(emoji, file, strip))
+
+      {unsafe_emoji, unsafe_file}, text ->
+        emoji = HTML.strip_tags(unsafe_emoji)
+        file = HTML.strip_tags(unsafe_file)
+        String.replace(text, ":#{emoji}:", prepare_emoji_html(emoji, file, strip))
+    end)
+    |> HTML.filter_tags()
+  end
+
+  defp prepare_emoji_html(_emoji, _file, true), do: ""
+
+  defp prepare_emoji_html(emoji, file, _strip) do
+    "<img class='emoji' alt='#{emoji}' title='#{emoji}' src='#{MediaProxy.url(file)}' />"
+  end
+
+  def demojify(text) do
+    emojify(text, Emoji.get_all(), true)
+  end
+
+  def demojify(text, nil), do: text
+
+  @doc "Outputs a list of the emoji-shortcodes in a text"
+  def get_emoji(text) when is_binary(text) do
+    Enum.filter(Emoji.get_all(), fn {emoji, %Emoji{}} ->
+      String.contains?(text, ":#{emoji}:")
+    end)
+  end
+
+  def get_emoji(_), do: []
+
+  @doc "Outputs a list of the emoji-Maps in a text"
+  def get_emoji_map(text) when is_binary(text) do
+    get_emoji(text)
+    |> Enum.reduce(%{}, fn {name, %Emoji{file: file}}, acc ->
+      Map.put(acc, name, "#{Pleroma.Web.Endpoint.static_url()}#{file}")
+    end)
+  end
+
+  def get_emoji_map(_), do: []
+end

lib/pleroma/emoji/loader.ex

@@ -0,0 +1,224 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Emoji.Loader do
+  @moduledoc """
+  The Loader emoji from:
+
+    * emoji packs in INSTANCE-DIR/emoji
+    * the files: `config/emoji.txt` and `config/custom_emoji.txt`
+    * glob paths, nested folder is used as tag name for grouping e.g. priv/static/emoji/custom/nested_folder
+  """
+  alias Pleroma.Config
+  alias Pleroma.Emoji
+
+  require Logger
+
+  @type pattern :: Regex.t() | module() | String.t()
+  @type patterns :: pattern() | [pattern()]
+  @type group_patterns :: keyword(patterns())
+  @type emoji :: {String.t(), Emoji.t()}
+
+  @doc """
+  Loads emojis from files/packs.
+
+  returns list emojis in format:
+  `{"000", "/emoji/freespeechextremist.com/000.png", ["Custom"]}`
+  """
+  @spec load() :: list(emoji)
+  def load do
+    emoji_dir_path = Path.join(Config.get!([:instance, :static_dir]), "emoji")
+
+    emoji_groups = Config.get([:emoji, :groups])
+
+    emojis =
+      case File.ls(emoji_dir_path) do
+        {:error, :enoent} ->
+          # The custom emoji directory doesn't exist,
+          # don't do anything
+          []
+
+        {:error, e} ->
+          # There was some other error
+          Logger.error("Could not access the custom emoji directory #{emoji_dir_path}: #{e}")
+          []
+
+        {:ok, results} ->
+          grouped =
+            Enum.group_by(results, fn file ->
+              File.dir?(Path.join(emoji_dir_path, file))
+            end)
+
+          packs = grouped[true] || []
+          files = grouped[false] || []
+
+          # Print the packs we've found
+          Logger.info("Found emoji packs: #{Enum.join(packs, ", ")}")
+
+          if not Enum.empty?(files) do
+            Logger.warn(
+              "Found files in the emoji folder. These will be ignored, please move them to a subdirectory\nFound files: #{
+                Enum.join(files, ", ")
+              }"
+            )
+          end
+
+          emojis =
+            Enum.flat_map(packs, fn pack ->
+              load_pack(Path.join(emoji_dir_path, pack), emoji_groups)
+            end)
+
+          Emoji.clear_all()
+          emojis
+      end
+
+    # Compat thing for old custom emoji handling & default emoji,
+    # it should run even if there are no emoji packs
+    shortcode_globs = Config.get([:emoji, :shortcode_globs], [])
+
+    emojis_txt =
+      (load_from_file("config/emoji.txt", emoji_groups) ++
+         load_from_file("config/custom_emoji.txt", emoji_groups) ++
+         load_from_globs(shortcode_globs, emoji_groups))
+      |> Enum.reject(fn value -> value == nil end)
+
+    Enum.map(emojis ++ emojis_txt, &prepare_emoji/1)
+  end
+
+  defp prepare_emoji({code, _, _} = emoji), do: {code, Emoji.build(emoji)}
+
+  defp load_pack(pack_dir, emoji_groups) do
+    pack_name = Path.basename(pack_dir)
+
+    pack_file = Path.join(pack_dir, "pack.json")
+
+    if File.exists?(pack_file) do
+      contents = Jason.decode!(File.read!(pack_file))
+
+      contents["files"]
+      |> Enum.map(fn {name, rel_file} ->
+        filename = Path.join("/emoji/#{pack_name}", rel_file)
+        {name, filename, ["pack:#{pack_name}"]}
+      end)
+    else
+      # Load from emoji.txt / all files
+      emoji_txt = Path.join(pack_dir, "emoji.txt")
+
+      if File.exists?(emoji_txt) do
+        load_from_file(emoji_txt, emoji_groups)
+      else
+        extensions = Pleroma.Config.get([:emoji, :pack_extensions])
+
+        Logger.info(
+          "No emoji.txt found for pack \"#{pack_name}\", assuming all #{
+            Enum.join(extensions, ", ")
+          } files are emoji"
+        )
+
+        make_shortcode_to_file_map(pack_dir, extensions)
+        |> Enum.map(fn {shortcode, rel_file} ->
+          filename = Path.join("/emoji/#{pack_name}", rel_file)
+
+          {shortcode, filename, [to_string(match_extra(emoji_groups, filename))]}
+        end)
+      end
+    end
+  end
+
+  def make_shortcode_to_file_map(pack_dir, exts) do
+    find_all_emoji(pack_dir, exts)
+    |> Enum.map(&Path.relative_to(&1, pack_dir))
+    |> Enum.map(fn f -> {f |> Path.basename() |> Path.rootname(), f} end)
+    |> Enum.into(%{})
+  end
+
+  def find_all_emoji(dir, exts) do
+    dir
+    |> File.ls!()
+    |> Enum.flat_map(fn f ->
+      filepath = Path.join(dir, f)
+
+      if File.dir?(filepath) do
+        find_all_emoji(filepath, exts)
+      else
+        [filepath]
+      end
+    end)
+    |> Enum.filter(fn f -> Path.extname(f) in exts end)
+  end
+
+  defp load_from_file(file, emoji_groups) do
+    if File.exists?(file) do
+      load_from_file_stream(File.stream!(file), emoji_groups)
+    else
+      []
+    end
+  end
+
+  defp load_from_file_stream(stream, emoji_groups) do
+    stream
+    |> Stream.map(&String.trim/1)
+    |> Stream.map(fn line ->
+      case String.split(line, ~r/,\s*/) do
+        [name, file] ->
+          {name, file, [to_string(match_extra(emoji_groups, file))]}
+
+        [name, file | tags] ->
+          {name, file, tags}
+
+        _ ->
+          nil
+      end
+    end)
+    |> Enum.to_list()
+  end
+
+  defp load_from_globs(globs, emoji_groups) do
+    static_path = Path.join(:code.priv_dir(:pleroma), "static")
+
+    paths =
+      Enum.map(globs, fn glob ->
+        Path.join(static_path, glob)
+        |> Path.wildcard()
+      end)
+      |> Enum.concat()
+
+    Enum.map(paths, fn path ->
+      tag = match_extra(emoji_groups, Path.join("/", Path.relative_to(path, static_path)))
+      shortcode = Path.basename(path, Path.extname(path))
+      external_path = Path.join("/", Path.relative_to(path, static_path))
+      {shortcode, external_path, [to_string(tag)]}
+    end)
+  end
+
+  @doc """
+  Finds a matching group for the given emoji filename
+  """
+  @spec match_extra(group_patterns(), String.t()) :: atom() | nil
+  def match_extra(group_patterns, filename) do
+    match_group_patterns(group_patterns, fn pattern ->
+      case pattern do
+        %Regex{} = regex -> Regex.match?(regex, filename)
+        string when is_binary(string) -> filename == string
+      end
+    end)
+  end
+
+  defp match_group_patterns(group_patterns, matcher) do
+    Enum.find_value(group_patterns, fn {group, patterns} ->
+      patterns =
+        patterns
+        |> List.wrap()
+        |> Enum.map(fn pattern ->
+          if String.contains?(pattern, "*") do
+            ~r(#{String.replace(pattern, "*", ".*")})
+          else
+            pattern
+          end
+        end)
+
+      Enum.any?(patterns, matcher) && group
+    end)
+  end
+end

lib/pleroma/filter.ex

@@ -12,7 +12,7 @@ defmodule Pleroma.Filter do
   alias Pleroma.User
 
   schema "filters" do
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     field(:filter_id, :integer)
     field(:hide, :boolean, default: false)
     field(:whole_word, :boolean, default: true)

lib/pleroma/flake_id.ex

@@ -1,182 +0,0 @@
-# Pleroma: A lightweight social networking server
-# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
-# SPDX-License-Identifier: AGPL-3.0-only
-
-defmodule Pleroma.FlakeId do
-  @moduledoc """
-  Flake is a decentralized, k-ordered id generation service.
-
-  Adapted from:
-
-  * [flaky](https://github.com/nirvana/flaky), released under the terms of the Truly Free License,
-  * [Flake](https://github.com/boundary/flake), Copyright 2012, Boundary, Apache License, Version 2.0
-  """
-
-  @type t :: binary
-
-  use Ecto.Type
-  use GenServer
-  require Logger
-  alias __MODULE__
-  import Kernel, except: [to_string: 1]
-
-  defstruct node: nil, time: 0, sq: 0
-
-  @doc "Converts a binary Flake to a String"
-  def to_string(<<0::integer-size(64), id::integer-size(64)>>) do
-    Kernel.to_string(id)
-  end
-
-  def to_string(<<_::integer-size(64), _::integer-size(48), _::integer-size(16)>> = flake) do
-    encode_base62(flake)
-  end
-
-  def to_string(s), do: s
-
-  def from_string(int) when is_integer(int) do
-    from_string(Kernel.to_string(int))
-  end
-
-  for i <- [-1, 0] do
-    def from_string(unquote(i)), do: <<0::integer-size(128)>>
-    def from_string(unquote(Kernel.to_string(i))), do: <<0::integer-size(128)>>
-  end
-
-  def from_string(<<_::integer-size(128)>> = flake), do: flake
-
-  def from_string(string) when is_binary(string) and byte_size(string) < 18 do
-    case Integer.parse(string) do
-      {id, ""} -> <<0::integer-size(64), id::integer-size(64)>>
-      _ -> nil
-    end
-  end
-
-  def from_string(string) do
-    string |> decode_base62 |> from_integer
-  end
-
-  def to_integer(<<integer::integer-size(128)>>), do: integer
-
-  def from_integer(integer) do
-    <<_time::integer-size(64), _node::integer-size(48), _seq::integer-size(16)>> =
-      <<integer::integer-size(128)>>
-  end
-
-  @doc "Generates a Flake"
-  @spec get :: binary
-  def get, do: to_string(:gen_server.call(:flake, :get))
-
-  # checks that ID is is valid FlakeID
-  #
-  @spec is_flake_id?(String.t()) :: boolean
-  def is_flake_id?(id), do: is_flake_id?(String.to_charlist(id), true)
-  defp is_flake_id?([c | cs], true) when c >= ?0 and c <= ?9, do: is_flake_id?(cs, true)
-  defp is_flake_id?([c | cs], true) when c >= ?A and c <= ?Z, do: is_flake_id?(cs, true)
-  defp is_flake_id?([c | cs], true) when c >= ?a and c <= ?z, do: is_flake_id?(cs, true)
-  defp is_flake_id?([], true), do: true
-  defp is_flake_id?(_, _), do: false
-
-  # -- Ecto.Type API
-  @impl Ecto.Type
-  def type, do: :uuid
-
-  @impl Ecto.Type
-  def cast(value) do
-    {:ok, FlakeId.to_string(value)}
-  end
-
-  @impl Ecto.Type
-  def load(value) do
-    {:ok, FlakeId.to_string(value)}
-  end
-
-  @impl Ecto.Type
-  def dump(value) do
-    {:ok, FlakeId.from_string(value)}
-  end
-
-  def autogenerate, do: get()
-
-  # -- GenServer API
-  def start_link(_) do
-    :gen_server.start_link({:local, :flake}, __MODULE__, [], [])
-  end
-
-  @impl GenServer
-  def init([]) do
-    {:ok, %FlakeId{node: worker_id(), time: time()}}
-  end
-
-  @impl GenServer
-  def handle_call(:get, _from, state) do
-    {flake, new_state} = get(time(), state)
-    {:reply, flake, new_state}
-  end
-
-  # Matches when the calling time is the same as the state time. Incr. sq
-  defp get(time, %FlakeId{time: time, node: node, sq: seq}) do
-    new_state = %FlakeId{time: time, node: node, sq: seq + 1}
-    {gen_flake(new_state), new_state}
-  end
-
-  # Matches when the times are different, reset sq
-  defp get(newtime, %FlakeId{time: time, node: node}) when newtime > time do
-    new_state = %FlakeId{time: newtime, node: node, sq: 0}
-    {gen_flake(new_state), new_state}
-  end
-
-  # Error when clock is running backwards
-  defp get(newtime, %FlakeId{time: time}) when newtime < time do
-    {:error, :clock_running_backwards}
-  end
-
-  defp gen_flake(%FlakeId{time: time, node: node, sq: seq}) do
-    <<time::integer-size(64), node::integer-size(48), seq::integer-size(16)>>
-  end
-
-  defp nthchar_base62(n) when n <= 9, do: ?0 + n
-  defp nthchar_base62(n) when n <= 35, do: ?A + n - 10
-  defp nthchar_base62(n), do: ?a + n - 36
-
-  defp encode_base62(<<integer::integer-size(128)>>) do
-    integer
-    |> encode_base62([])
-    |> List.to_string()
-  end
-
-  defp encode_base62(int, acc) when int < 0, do: encode_base62(-int, acc)
-  defp encode_base62(int, []) when int == 0, do: '0'
-  defp encode_base62(int, acc) when int == 0, do: acc
-
-  defp encode_base62(int, acc) do
-    r = rem(int, 62)
-    id = div(int, 62)
-    acc = [nthchar_base62(r) | acc]
-    encode_base62(id, acc)
-  end
-
-  defp decode_base62(s) do
-    decode_base62(String.to_charlist(s), 0)
-  end
-
-  defp decode_base62([c | cs], acc) when c >= ?0 and c <= ?9,
-    do: decode_base62(cs, 62 * acc + (c - ?0))
-
-  defp decode_base62([c | cs], acc) when c >= ?A and c <= ?Z,
-    do: decode_base62(cs, 62 * acc + (c - ?A + 10))
-
-  defp decode_base62([c | cs], acc) when c >= ?a and c <= ?z,
-    do: decode_base62(cs, 62 * acc + (c - ?a + 36))
-
-  defp decode_base62([], acc), do: acc
-
-  defp time do
-    {mega_seconds, seconds, micro_seconds} = :erlang.timestamp()
-    1_000_000_000 * mega_seconds + seconds * 1000 + :erlang.trunc(micro_seconds / 1000)
-  end
-
-  defp worker_id do
-    <<worker::integer-size(48)>> = :crypto.strong_rand_bytes(6)
-    worker
-  end
-end

lib/pleroma/formatter.ex

@@ -3,10 +3,8 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Formatter do
-  alias Pleroma.Emoji
   alias Pleroma.HTML
   alias Pleroma.User
-  alias Pleroma.Web.MediaProxy
 
   @safe_mention_regex ~r/^(\s*(?<mentions>(@.+?\s+){1,})+)(?<rest>.*)/s
   @link_regex ~r"((?:http(s)?:\/\/)?[\w.-]+(?:\.[\w\.-]+)+[\w\-\._~%:/?#[\]@!\$&'\(\)\*\+,;=.]+)|[0-9a-z+\-\.]+:[0-9a-z$-_.+!*'(),]+"ui
@@ -36,9 +34,9 @@ defmodule Pleroma.Formatter do
         nickname_text = get_nickname_text(nickname, opts)
 
         link =
-          "<span class='h-card'><a data-user='#{id}' class='u-url mention' href='#{ap_id}'>@<span>#{
+          ~s(<span class="h-card"><a data-user="#{id}" class="u-url mention" href="#{ap_id}" rel="ugc">@<span>#{
             nickname_text
-          }</span></a></span>"
+          }</span></a></span>)
 
         {link, %{acc | mentions: MapSet.put(acc.mentions, {"@" <> nickname, user})}}
 
@@ -50,7 +48,7 @@ defmodule Pleroma.Formatter do
   def hashtag_handler("#" <> tag = tag_text, _buffer, _opts, acc) do
     tag = String.downcase(tag)
     url = "#{Pleroma.Web.base_url()}/tag/#{tag}"
-    link = "<a class='hashtag' data-tag='#{tag}' href='#{url}' rel='tag'>#{tag_text}</a>"
+    link = ~s(<a class="hashtag" data-tag="#{tag}" href="#{url}" rel="tag ugc">#{tag_text}</a>)
 
     {link, %{acc | tags: MapSet.put(acc.tags, {tag_text, tag})}}
   end
@@ -100,51 +98,6 @@ defmodule Pleroma.Formatter do
     end
   end
 
-  def emojify(text) do
-    emojify(text, Emoji.get_all())
-  end
-
-  def emojify(text, nil), do: text
-
-  def emojify(text, emoji, strip \\ false) do
-    Enum.reduce(emoji, text, fn emoji_data, text ->
-      emoji = HTML.strip_tags(elem(emoji_data, 0))
-      file = HTML.strip_tags(elem(emoji_data, 1))
-
-      html =
-        if not strip do
-          "<img class='emoji' alt='#{emoji}' title='#{emoji}' src='#{MediaProxy.url(file)}' />"
-        else
-          ""
-        end
-
-      String.replace(text, ":#{emoji}:", html) |> HTML.filter_tags()
-    end)
-  end
-
-  def demojify(text) do
-    emojify(text, Emoji.get_all(), true)
-  end
-
-  def demojify(text, nil), do: text
-
-  @doc "Outputs a list of the emoji-shortcodes in a text"
-  def get_emoji(text) when is_binary(text) do
-    Enum.filter(Emoji.get_all(), fn {emoji, _, _} -> String.contains?(text, ":#{emoji}:") end)
-  end
-
-  def get_emoji(_), do: []
-
-  @doc "Outputs a list of the emoji-Maps in a text"
-  def get_emoji_map(text) when is_binary(text) do
-    get_emoji(text)
-    |> Enum.reduce(%{}, fn {name, file, _group}, acc ->
-      Map.put(acc, name, "#{Pleroma.Web.Endpoint.static_url()}#{file}")
-    end)
-  end
-
-  def get_emoji_map(_), do: []
-
   def html_escape({text, mentions, hashtags}, type) do
     {html_escape(text, type), mentions, hashtags}
   end

lib/pleroma/html.ex

@@ -184,7 +184,8 @@ defmodule Pleroma.HTML.Scrubber.Default do
     "tag",
     "nofollow",
     "noopener",
-    "noreferrer"
+    "noreferrer",
+    "ugc"
   ])
 
   Meta.allow_tag_with_these_attributes("a", ["name", "title"])
@@ -304,7 +305,8 @@ defmodule Pleroma.HTML.Scrubber.LinksOnly do
     "nofollow",
     "noopener",
     "noreferrer",
-    "me"
+    "me",
+    "ugc"
   ])
 
   Meta.allow_tag_with_these_attributes("a", ["name", "title"])

lib/pleroma/list.ex

@@ -13,7 +13,7 @@ defmodule Pleroma.List do
   alias Pleroma.User
 
   schema "lists" do
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     field(:title, :string)
     field(:following, {:array, :string}, default: [])
     field(:ap_id, :string)
@@ -84,22 +84,11 @@ defmodule Pleroma.List do
   end
 
   # Get lists to which the account belongs.
-  def get_lists_account_belongs(%User{} = owner, account_id) do
+  def get_lists_account_belongs(%User{} = owner, user) do
-    user = User.get_cached_by_id(account_id)
+    Pleroma.List
-
+    |> where([l], l.user_id == ^owner.id)
-    query =
+    |> where([l], fragment("? = ANY(?)", ^user.follower_address, l.following))
-      from(
+    |> Repo.all()
-        l in Pleroma.List,
-        where:
-          l.user_id == ^owner.id and
-            fragment(
-              "? = ANY(?)",
-              ^user.follower_address,
-              l.following
-            )
-      )
-
-    Repo.all(query)
   end
 
   def rename(%Pleroma.List{} = list, title) do

lib/pleroma/moderation_log.ex

@@ -14,61 +14,143 @@ defmodule Pleroma.ModerationLog do
     timestamps()
   end
 
-  def get_all(page, page_size) do
+  def get_all(params) do
-    from(q in __MODULE__,
+    base_query =
-      order_by: [desc: q.inserted_at],
+      get_all_query()
+      |> maybe_filter_by_date(params)
+      |> maybe_filter_by_user(params)
+      |> maybe_filter_by_search(params)
+
+    query_with_pagination = base_query |> paginate_query(params)
+
+    %{
+      items: Repo.all(query_with_pagination),
+      count: Repo.aggregate(base_query, :count, :id)
+    }
+  end
+
+  defp maybe_filter_by_date(query, %{start_date: nil, end_date: nil}), do: query
+
+  defp maybe_filter_by_date(query, %{start_date: start_date, end_date: nil}) do
+    from(q in query,
+      where: q.inserted_at >= ^parse_datetime(start_date)
+    )
+  end
+
+  defp maybe_filter_by_date(query, %{start_date: nil, end_date: end_date}) do
+    from(q in query,
+      where: q.inserted_at <= ^parse_datetime(end_date)
+    )
+  end
+
+  defp maybe_filter_by_date(query, %{start_date: start_date, end_date: end_date}) do
+    from(q in query,
+      where: q.inserted_at >= ^parse_datetime(start_date),
+      where: q.inserted_at <= ^parse_datetime(end_date)
+    )
+  end
+
+  defp maybe_filter_by_user(query, %{user_id: nil}), do: query
+
+  defp maybe_filter_by_user(query, %{user_id: user_id}) do
+    from(q in query,
+      where: fragment("(?)->'actor'->>'id' = ?", q.data, ^user_id)
+    )
+  end
+
+  defp maybe_filter_by_search(query, %{search: search}) when is_nil(search) or search == "",
+    do: query
+
+  defp maybe_filter_by_search(query, %{search: search}) do
+    from(q in query,
+      where: fragment("(?)->>'message' ILIKE ?", q.data, ^"%#{search}%")
+    )
+  end
+
+  defp paginate_query(query, %{page: page, page_size: page_size}) do
+    from(q in query,
       limit: ^page_size,
       offset: ^((page - 1) * page_size)
     )
-    |> Repo.all()
   end
 
+  defp get_all_query do
+    from(q in __MODULE__,
+      order_by: [desc: q.inserted_at]
+    )
+  end
+
+  defp parse_datetime(datetime) do
+    {:ok, parsed_datetime, _} = DateTime.from_iso8601(datetime)
+
+    parsed_datetime
+  end
+
+  @spec insert_log(%{actor: User, subject: User, action: String.t(), permission: String.t()}) ::
+          {:ok, ModerationLog} | {:error, any}
   def insert_log(%{
         actor: %User{} = actor,
         subject: %User{} = subject,
         action: action,
         permission: permission
       }) do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        subject: user_to_map(subject),
+        "subject" => user_to_map(subject),
-        action: action,
+        "action" => action,
-        permission: permission
+        "permission" => permission,
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
+  @spec insert_log(%{actor: User, subject: User, action: String.t()}) ::
+          {:ok, ModerationLog} | {:error, any}
   def insert_log(%{
         actor: %User{} = actor,
         action: "report_update",
         subject: %Activity{data: %{"type" => "Flag"}} = subject
       }) do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        action: "report_update",
+        "action" => "report_update",
-        subject: report_to_map(subject)
+        "subject" => report_to_map(subject),
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
+  @spec insert_log(%{actor: User, subject: Activity, action: String.t(), text: String.t()}) ::
+          {:ok, ModerationLog} | {:error, any}
   def insert_log(%{
         actor: %User{} = actor,
         action: "report_response",
         subject: %Activity{} = subject,
         text: text
       }) do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        action: "report_response",
+        "action" => "report_response",
-        subject: report_to_map(subject),
+        "subject" => report_to_map(subject),
-        text: text
+        "text" => text,
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
+  @spec insert_log(%{
+          actor: User,
+          subject: Activity,
+          action: String.t(),
+          sensitive: String.t(),
+          visibility: String.t()
+        }) :: {:ok, ModerationLog} | {:error, any}
   def insert_log(%{
         actor: %User{} = actor,
         action: "status_update",
@@ -76,41 +158,49 @@ defmodule Pleroma.ModerationLog do
         sensitive: sensitive,
         visibility: visibility
       }) do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        action: "status_update",
+        "action" => "status_update",
-        subject: status_to_map(subject),
+        "subject" => status_to_map(subject),
-        sensitive: sensitive,
+        "sensitive" => sensitive,
-        visibility: visibility
+        "visibility" => visibility,
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
+  @spec insert_log(%{actor: User, action: String.t(), subject_id: String.t()}) ::
+          {:ok, ModerationLog} | {:error, any}
   def insert_log(%{
         actor: %User{} = actor,
         action: "status_delete",
         subject_id: subject_id
       }) do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        action: "status_delete",
+        "action" => "status_delete",
-        subject_id: subject_id
+        "subject_id" => subject_id,
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
   @spec insert_log(%{actor: User, subject: User, action: String.t()}) ::
           {:ok, ModerationLog} | {:error, any}
   def insert_log(%{actor: %User{} = actor, subject: subject, action: action}) do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        action: action,
+        "action" => action,
-        subject: user_to_map(subject)
+        "subject" => user_to_map(subject),
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
   @spec insert_log(%{actor: User, subjects: [User], action: String.t()}) ::
@@ -118,97 +208,128 @@ defmodule Pleroma.ModerationLog do
   def insert_log(%{actor: %User{} = actor, subjects: subjects, action: action}) do
     subjects = Enum.map(subjects, &user_to_map/1)
 
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        action: action,
+        "action" => action,
-        subjects: subjects
+        "subjects" => subjects,
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
+  @spec insert_log(%{actor: User, action: String.t(), followed: User, follower: User}) ::
+          {:ok, ModerationLog} | {:error, any}
   def insert_log(%{
         actor: %User{} = actor,
         followed: %User{} = followed,
         follower: %User{} = follower,
         action: "follow"
       }) do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        action: "follow",
+        "action" => "follow",
-        followed: user_to_map(followed),
+        "followed" => user_to_map(followed),
-        follower: user_to_map(follower)
+        "follower" => user_to_map(follower),
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
+  @spec insert_log(%{actor: User, action: String.t(), followed: User, follower: User}) ::
+          {:ok, ModerationLog} | {:error, any}
   def insert_log(%{
         actor: %User{} = actor,
         followed: %User{} = followed,
         follower: %User{} = follower,
         action: "unfollow"
       }) do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        action: "unfollow",
+        "action" => "unfollow",
-        followed: user_to_map(followed),
+        "followed" => user_to_map(followed),
-        follower: user_to_map(follower)
+        "follower" => user_to_map(follower),
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
+  @spec insert_log(%{
+          actor: User,
+          action: String.t(),
+          nicknames: [String.t()],
+          tags: [String.t()]
+        }) :: {:ok, ModerationLog} | {:error, any}
   def insert_log(%{
         actor: %User{} = actor,
         nicknames: nicknames,
         tags: tags,
         action: action
       }) do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        nicknames: nicknames,
+        "nicknames" => nicknames,
-        tags: tags,
+        "tags" => tags,
-        action: action
+        "action" => action,
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
   end
 
+  @spec insert_log(%{actor: User, action: String.t(), target: String.t()}) ::
+          {:ok, ModerationLog} | {:error, any}
   def insert_log(%{
         actor: %User{} = actor,
         action: action,
         target: target
       })
       when action in ["relay_follow", "relay_unfollow"] do
-    Repo.insert(%ModerationLog{
+    %ModerationLog{
       data: %{
-        actor: user_to_map(actor),
+        "actor" => user_to_map(actor),
-        action: action,
+        "action" => action,
-        target: target
+        "target" => target,
+        "message" => ""
       }
-    })
+    }
+    |> insert_log_entry_with_message()
+  end
+
+  @spec insert_log_entry_with_message(ModerationLog) :: {:ok, ModerationLog} | {:error, any}
+
+  defp insert_log_entry_with_message(entry) do
+    entry.data["message"]
+    |> put_in(get_log_entry_message(entry))
+    |> Repo.insert()
   end
 
   defp user_to_map(%User{} = user) do
     user
     |> Map.from_struct()
     |> Map.take([:id, :nickname])
-    |> Map.put(:type, "user")
+    |> Map.new(fn {k, v} -> {Atom.to_string(k), v} end)
+    |> Map.put("type", "user")
   end
 
   defp report_to_map(%Activity{} = report) do
     %{
-      type: "report",
+      "type" => "report",
-      id: report.id,
+      "id" => report.id,
-      state: report.data["state"]
+      "state" => report.data["state"]
     }
   end
 
   defp status_to_map(%Activity{} = status) do
     %{
-      type: "status",
+      "type" => "status",
-      id: status.id
+      "id" => status.id
     }
   end
 

lib/pleroma/notification.ex

@@ -22,8 +22,8 @@ defmodule Pleroma.Notification do
 
   schema "notifications" do
     field(:seen, :boolean, default: false)
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
-    belongs_to(:activity, Activity, type: Pleroma.FlakeId)
+    belongs_to(:activity, Activity, type: FlakeId.Ecto.CompatType)
 
     timestamps()
   end

lib/pleroma/object.ex

@@ -38,6 +38,24 @@ defmodule Pleroma.Object do
   def get_by_id(nil), do: nil
   def get_by_id(id), do: Repo.get(Object, id)
 
+  def get_by_id_and_maybe_refetch(id, opts \\ []) do
+    %{updated_at: updated_at} = object = get_by_id(id)
+
+    if opts[:interval] &&
+         NaiveDateTime.diff(NaiveDateTime.utc_now(), updated_at) > opts[:interval] do
+      case Fetcher.refetch_object(object) do
+        {:ok, %Object{} = object} ->
+          object
+
+        e ->
+          Logger.error("Couldn't refresh #{object.data["id"]}:\n#{inspect(e)}")
+          object
+      end
+    else
+      object
+    end
+  end
+
   def get_by_ap_id(nil), do: nil
 
   def get_by_ap_id(ap_id) do
@@ -230,4 +248,11 @@ defmodule Pleroma.Object do
       _ -> :noop
     end
   end
+
+  @doc "Updates data field of an object"
+  def update_data(%Object{data: data} = object, attrs \\ %{}) do
+    object
+    |> Object.change(%{data: Map.merge(data || %{}, attrs)})
+    |> Repo.update()
+  end
 end

lib/pleroma/object/fetcher.ex

@@ -6,18 +6,40 @@ defmodule Pleroma.Object.Fetcher do
   alias Pleroma.HTTP
   alias Pleroma.Object
   alias Pleroma.Object.Containment
+  alias Pleroma.Repo
   alias Pleroma.Signature
   alias Pleroma.Web.ActivityPub.InternalFetchActor
   alias Pleroma.Web.ActivityPub.Transmogrifier
   alias Pleroma.Web.OStatus
 
   require Logger
+  require Pleroma.Constants
 
-  defp reinject_object(data) do
+  defp touch_changeset(changeset) do
+    updated_at =
+      NaiveDateTime.utc_now()
+      |> NaiveDateTime.truncate(:second)
+
+    Ecto.Changeset.put_change(changeset, :updated_at, updated_at)
+  end
+
+  defp maybe_reinject_internal_fields(data, %{data: %{} = old_data}) do
+    internal_fields = Map.take(old_data, Pleroma.Constants.object_internal_fields())
+
+    Map.merge(data, internal_fields)
+  end
+
+  defp maybe_reinject_internal_fields(data, _), do: data
+
+  @spec reinject_object(struct(), map()) :: {:ok, Object.t()} | {:error, any()}
+  defp reinject_object(struct, data) do
     Logger.debug("Reinjecting object #{data["id"]}")
 
     with data <- Transmogrifier.fix_object(data),
-         {:ok, object} <- Object.create(data) do
+         data <- maybe_reinject_internal_fields(data, struct),
+         changeset <- Object.change(struct, %{data: data}),
+         changeset <- touch_changeset(changeset),
+         {:ok, object} <- Repo.insert_or_update(changeset) do
       {:ok, object}
     else
       e ->
@@ -26,55 +48,68 @@ defmodule Pleroma.Object.Fetcher do
     end
   end
 
+  def refetch_object(%Object{data: %{"id" => id}} = object) do
+    with {:local, false} <- {:local, String.starts_with?(id, Pleroma.Web.base_url() <> "/")},
+         {:ok, data} <- fetch_and_contain_remote_object_from_id(id),
+         {:ok, object} <- reinject_object(object, data) do
+      {:ok, object}
+    else
+      {:local, true} -> object
+      e -> {:error, e}
+    end
+  end
+
   # TODO:
   # This will create a Create activity, which we need internally at the moment.
   def fetch_object_from_id(id, options \\ []) do
-    if object = Object.get_cached_by_ap_id(id) do
+    with {:fetch_object, nil} <- {:fetch_object, Object.get_cached_by_ap_id(id)},
+         {:fetch, {:ok, data}} <- {:fetch, fetch_and_contain_remote_object_from_id(id)},
+         {:normalize, nil} <- {:normalize, Object.normalize(data, false)},
+         params <- prepare_activity_params(data),
+         {:containment, :ok} <- {:containment, Containment.contain_origin(id, params)},
+         {:ok, activity} <- Transmogrifier.handle_incoming(params, options),
+         {:object, _data, %Object{} = object} <-
+           {:object, data, Object.normalize(activity, false)} do
       {:ok, object}
     else
-      Logger.info("Fetching #{id} via AP")
+      {:containment, _} ->
+        {:error, "Object containment failed."}
 
-      with {:fetch, {:ok, data}} <- {:fetch, fetch_and_contain_remote_object_from_id(id)},
+      {:error, {:reject, nil}} ->
-           {:normalize, nil} <- {:normalize, Object.normalize(data, false)},
+        {:reject, nil}
-           params <- %{
+
-             "type" => "Create",
+      {:object, data, nil} ->
-             "to" => data["to"],
+        reinject_object(%Object{}, data)
-             "cc" => data["cc"],
+
-             # Should we seriously keep this attributedTo thing?
+      {:normalize, object = %Object{}} ->
-             "actor" => data["actor"] || data["attributedTo"],
-             "object" => data
-           },
-           {:containment, :ok} <- {:containment, Containment.contain_origin(id, params)},
-           {:ok, activity} <- Transmogrifier.handle_incoming(params, options),
-           {:object, _data, %Object{} = object} <-
-             {:object, data, Object.normalize(activity, false)} do
         {:ok, object}
-      else
-        {:containment, _} ->
-          {:error, "Object containment failed."}
 
-        {:error, {:reject, nil}} ->
+      {:fetch_object, %Object{} = object} ->
-          {:reject, nil}
+        {:ok, object}
 
-        {:object, data, nil} ->
+      _e ->
-          reinject_object(data)
+        # Only fallback when receiving a fetch/normalization error with ActivityPub
+        Logger.info("Couldn't get object via AP, trying out OStatus fetching...")
 
-        {:normalize, object = %Object{}} ->
+        # FIXME: OStatus Object Containment?
-          {:ok, object}
+        case OStatus.fetch_activity_from_url(id) do
-
+          {:ok, [activity | _]} -> {:ok, Object.normalize(activity, false)}
-        _e ->
+          e -> e
-          # Only fallback when receiving a fetch/normalization error with ActivityPub
+        end
-          Logger.info("Couldn't get object via AP, trying out OStatus fetching...")
-
-          # FIXME: OStatus Object Containment?
-          case OStatus.fetch_activity_from_url(id) do
-            {:ok, [activity | _]} -> {:ok, Object.normalize(activity, false)}
-            e -> e
-          end
-      end
     end
   end
 
+  defp prepare_activity_params(data) do
+    %{
+      "type" => "Create",
+      "to" => data["to"],
+      "cc" => data["cc"],
+      # Should we seriously keep this attributedTo thing?
+      "actor" => data["actor"] || data["attributedTo"],
+      "object" => data
+    }
+  end
+
   def fetch_object_from_id!(id, options \\ []) do
     with {:ok, object} <- fetch_object_from_id(id, options) do
       object

lib/pleroma/pagination.ex

@@ -64,6 +64,7 @@ defmodule Pleroma.Pagination do
 
   def paginate(query, options, :offset) do
     query
+    |> restrict(:order, options)
     |> restrict(:offset, options)
     |> restrict(:limit, options)
   end

lib/pleroma/password_reset_token.ex

@@ -12,7 +12,7 @@ defmodule Pleroma.PasswordResetToken do
   alias Pleroma.User
 
   schema "password_reset_tokens" do
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     field(:token, :string)
     field(:used, :boolean, default: false)
 

lib/pleroma/plugs/remote_ip.ex

@@ -0,0 +1,54 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Plugs.RemoteIp do
+  @moduledoc """
+  This is a shim to call [`RemoteIp`](https://git.pleroma.social/pleroma/remote_ip) but with runtime configuration.
+  """
+
+  @behaviour Plug
+
+  @headers ~w[
+    forwarded
+    x-forwarded-for
+    x-client-ip
+    x-real-ip
+  ]
+
+  # https://en.wikipedia.org/wiki/Localhost
+  # https://en.wikipedia.org/wiki/Private_network
+  @reserved ~w[
+    127.0.0.0/8
+    ::1/128
+    fc00::/7
+    10.0.0.0/8
+    172.16.0.0/12
+    192.168.0.0/16
+  ]
+
+  def init(_), do: nil
+
+  def call(conn, _) do
+    config = Pleroma.Config.get(__MODULE__, [])
+
+    if Keyword.get(config, :enabled, false) do
+      RemoteIp.call(conn, remote_ip_opts(config))
+    else
+      conn
+    end
+  end
+
+  defp remote_ip_opts(config) do
+    headers = config |> Keyword.get(:headers, @headers) |> MapSet.new()
+    reserved = Keyword.get(config, :reserved, @reserved)
+
+    proxies =
+      config
+      |> Keyword.get(:proxies, [])
+      |> Enum.concat(reserved)
+      |> Enum.map(&InetCidr.parse/1)
+
+    {headers, proxies}
+  end
+end

lib/pleroma/registration.ex

@@ -11,10 +11,10 @@ defmodule Pleroma.Registration do
   alias Pleroma.Repo
   alias Pleroma.User
 
-  @primary_key {:id, Pleroma.FlakeId, autogenerate: true}
+  @primary_key {:id, FlakeId.Ecto.CompatType, autogenerate: true}
 
   schema "registrations" do
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     field(:provider, :string)
     field(:uid, :string)
     field(:info, :map, default: %{})

lib/pleroma/scheduled_activity.ex

@@ -17,7 +17,7 @@ defmodule Pleroma.ScheduledActivity do
   @min_offset :timer.minutes(5)
 
   schema "scheduled_activities" do
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     field(:scheduled_at, :naive_datetime)
     field(:params, :map)
 

lib/pleroma/thread_mute.ex

@@ -12,7 +12,7 @@ defmodule Pleroma.ThreadMute do
   require Ecto.Query
 
   schema "thread_mutes" do
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     field(:context, :string)
   end
 
@@ -24,7 +24,7 @@ defmodule Pleroma.ThreadMute do
   end
 
   def query(user_id, context) do
-    user_id = Pleroma.FlakeId.from_string(user_id)
+    {:ok, user_id} = FlakeId.Ecto.CompatType.dump(user_id)
 
     ThreadMute
     |> Ecto.Query.where(user_id: ^user_id)

lib/pleroma/uploaders/s3.ex

@@ -38,16 +38,26 @@ defmodule Pleroma.Uploaders.S3 do
   def put_file(%Pleroma.Upload{} = upload) do
     config = Config.get([__MODULE__])
     bucket = Keyword.get(config, :bucket)
+    streaming = Keyword.get(config, :streaming_enabled)
 
     s3_name = strict_encode(upload.path)
 
     op =
-      upload.tempfile
+      if streaming do
-      |> ExAws.S3.Upload.stream_file()
+        upload.tempfile
-      |> ExAws.S3.upload(bucket, s3_name, [
+        |> ExAws.S3.Upload.stream_file()
-        {:acl, :public_read},
+        |> ExAws.S3.upload(bucket, s3_name, [
-        {:content_type, upload.content_type}
+          {:acl, :public_read},
-      ])
+          {:content_type, upload.content_type}
+        ])
+      else
+        {:ok, file_data} = File.read(upload.tempfile)
+
+        ExAws.S3.put_object(bucket, s3_name, file_data, [
+          {:acl, :public_read},
+          {:content_type, upload.content_type}
+        ])
+      end
 
     case ExAws.request(op) do
       {:ok, _} ->

lib/pleroma/user.ex

@@ -34,7 +34,7 @@ defmodule Pleroma.User do
 
   @type t :: %__MODULE__{}
 
-  @primary_key {:id, Pleroma.FlakeId, autogenerate: true}
+  @primary_key {:id, FlakeId.Ecto.CompatType, autogenerate: true}
 
   # credo:disable-for-next-line Credo.Check.Readability.MaxLineLength
   @email_regex ~r/^[a-zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/
@@ -106,9 +106,7 @@ defmodule Pleroma.User do
   def profile_url(%User{ap_id: ap_id}), do: ap_id
   def profile_url(_), do: nil
 
-  def ap_id(%User{nickname: nickname}) do
+  def ap_id(%User{nickname: nickname}), do: "#{Web.base_url()}/users/#{nickname}"
-    "#{Web.base_url()}/users/#{nickname}"
-  end
 
   def ap_followers(%User{follower_address: fa}) when is_binary(fa), do: fa
   def ap_followers(%User{} = user), do: "#{ap_id(user)}/followers"
@@ -119,12 +117,9 @@ defmodule Pleroma.User do
 
   def user_info(%User{} = user, args \\ %{}) do
     following_count =
-      if args[:following_count],
+      Map.get(args, :following_count, user.info.following_count || following_count(user))
-        do: args[:following_count],
-        else: user.info.following_count || following_count(user)
 
-    follower_count =
+    follower_count = Map.get(args, :follower_count, user.info.follower_count)
-      if args[:follower_count], do: args[:follower_count], else: user.info.follower_count
 
     %{
       note_count: user.info.note_count,
@@ -137,12 +132,11 @@ defmodule Pleroma.User do
   end
 
   def follow_state(%User{} = user, %User{} = target) do
-    follow_activity = Utils.fetch_latest_follow(user, target)
+    case Utils.fetch_latest_follow(user, target) do
-
+      %{data: %{"state" => state}} -> state
-    if follow_activity,
-      do: follow_activity.data["state"],
       # Ideally this would be nil, but then Cachex does not commit the value
-      else: false
+      _ -> false
+    end
   end
 
   def get_cached_follow_state(user, target) do
@@ -152,11 +146,7 @@ defmodule Pleroma.User do
 
   @spec set_follow_state_cache(String.t(), String.t(), String.t()) :: {:ok | :error, boolean()}
   def set_follow_state_cache(user_ap_id, target_ap_id, state) do
-    Cachex.put(
+    Cachex.put(:user_cache, "follow_state:#{user_ap_id}|#{target_ap_id}", state)
-      :user_cache,
-      "follow_state:#{user_ap_id}|#{target_ap_id}",
-      state
-    )
   end
 
   def set_info_cache(user, args) do
@@ -197,34 +187,25 @@ defmodule Pleroma.User do
       |> truncate_if_exists(:name, name_limit)
       |> truncate_if_exists(:bio, bio_limit)
 
-    info_cng = User.Info.remote_user_creation(%User.Info{}, params[:info])
+    changeset =
-
+      %User{local: false}
-    changes =
-      %User{}
       |> cast(params, [:bio, :name, :ap_id, :nickname, :avatar])
       |> validate_required([:name, :ap_id])
       |> unique_constraint(:nickname)
       |> validate_format(:nickname, @email_regex)
       |> validate_length(:bio, max: bio_limit)
       |> validate_length(:name, max: name_limit)
-      |> put_change(:local, false)
+      |> change_info(&User.Info.remote_user_creation(&1, params[:info]))
-      |> put_embed(:info, info_cng)
 
-    if changes.valid? do
+    case params[:info][:source_data] do
-      case info_cng.changes[:source_data] do
+      %{"followers" => followers, "following" => following} ->
-        %{"followers" => followers, "following" => following} ->
+        changeset
-          changes
+        |> put_change(:follower_address, followers)
-          |> put_change(:follower_address, followers)
+        |> put_change(:following_address, following)
-          |> put_change(:following_address, following)
 
-        _ ->
+      _ ->
-          followers = User.ap_followers(%User{nickname: changes.changes[:nickname]})
+        followers = ap_followers(%User{nickname: get_field(changeset, :nickname)})
-
+        put_change(changeset, :follower_address, followers)
-          changes
-          |> put_change(:follower_address, followers)
-      end
-    else
-      changes
     end
   end
 
@@ -245,7 +226,6 @@ defmodule Pleroma.User do
     name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
 
     params = Map.put(params, :last_refreshed_at, NaiveDateTime.utc_now())
-    info_cng = User.Info.user_upgrade(struct.info, params[:info], remote?)
 
     struct
     |> cast(params, [
@@ -260,7 +240,7 @@ defmodule Pleroma.User do
     |> validate_format(:nickname, local_nickname_regex())
     |> validate_length(:bio, max: bio_limit)
     |> validate_length(:name, max: name_limit)
-    |> put_embed(:info, info_cng)
+    |> change_info(&User.Info.user_upgrade(&1, params[:info], remote?))
   end
 
   def password_update_changeset(struct, params) do
@@ -269,6 +249,7 @@ defmodule Pleroma.User do
     |> validate_required([:password, :password_confirmation])
     |> validate_confirmation(:password)
     |> put_password_hash
+    |> put_embed(:info, User.Info.set_password_reset_pending(struct.info, false))
   end
 
   @spec reset_password(User.t(), map) :: {:ok, User.t()} | {:error, Ecto.Changeset.t()}
@@ -285,6 +266,20 @@ defmodule Pleroma.User do
     end
   end
 
+  def force_password_reset_async(user) do
+    BackgroundWorker.enqueue("force_password_reset", %{"user_id" => user.id})
+  end
+
+  @spec force_password_reset(User.t()) :: {:ok, User.t()} | {:error, Ecto.Changeset.t()}
+  def force_password_reset(user) do
+    info_cng = User.Info.set_password_reset_pending(user.info, true)
+
+    user
+    |> change()
+    |> put_embed(:info, info_cng)
+    |> update_and_set_cache()
+  end
+
   def register_changeset(struct, params \\ %{}, opts \\ []) do
     bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
     name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
@@ -296,43 +291,39 @@ defmodule Pleroma.User do
         opts[:need_confirmation]
       end
 
-    info_change =
+    struct
-      User.Info.confirmation_changeset(%User.Info{}, need_confirmation: need_confirmation?)
+    |> cast(params, [:bio, :email, :name, :nickname, :password, :password_confirmation])
+    |> validate_required([:name, :nickname, :password, :password_confirmation])
+    |> validate_confirmation(:password)
+    |> unique_constraint(:email)
+    |> unique_constraint(:nickname)
+    |> validate_exclusion(:nickname, Pleroma.Config.get([User, :restricted_nicknames]))
+    |> validate_format(:nickname, local_nickname_regex())
+    |> validate_format(:email, @email_regex)
+    |> validate_length(:bio, max: bio_limit)
+    |> validate_length(:name, min: 1, max: name_limit)
+    |> change_info(&User.Info.confirmation_changeset(&1, need_confirmation: need_confirmation?))
+    |> maybe_validate_required_email(opts[:external])
+    |> put_password_hash
+    |> put_ap_id()
+    |> unique_constraint(:ap_id)
+    |> put_following_and_follower_address()
+  end
 
-    changeset =
+  def maybe_validate_required_email(changeset, true), do: changeset
-      struct
+  def maybe_validate_required_email(changeset, _), do: validate_required(changeset, [:email])
-      |> cast(params, [:bio, :email, :name, :nickname, :password, :password_confirmation])
-      |> validate_required([:name, :nickname, :password, :password_confirmation])
-      |> validate_confirmation(:password)
-      |> unique_constraint(:email)
-      |> unique_constraint(:nickname)
-      |> validate_exclusion(:nickname, Pleroma.Config.get([User, :restricted_nicknames]))
-      |> validate_format(:nickname, local_nickname_regex())
-      |> validate_format(:email, @email_regex)
-      |> validate_length(:bio, max: bio_limit)
-      |> validate_length(:name, min: 1, max: name_limit)
-      |> put_change(:info, info_change)
 
-    changeset =
+  defp put_ap_id(changeset) do
-      if opts[:external] do
+    ap_id = ap_id(%User{nickname: get_field(changeset, :nickname)})
-        changeset
+    put_change(changeset, :ap_id, ap_id)
-      else
+  end
-        validate_required(changeset, [:email])
-      end
 
-    if changeset.valid? do
+  defp put_following_and_follower_address(changeset) do
-      ap_id = User.ap_id(%User{nickname: changeset.changes[:nickname]})
+    followers = ap_followers(%User{nickname: get_field(changeset, :nickname)})
-      followers = User.ap_followers(%User{nickname: changeset.changes[:nickname]})
 
-      changeset
+    changeset
-      |> put_password_hash
+    |> put_change(:following, [followers])
-      |> put_change(:ap_id, ap_id)
+    |> put_change(:follower_address, followers)
-      |> unique_constraint(:ap_id)
-      |> put_change(:following, [followers])
-      |> put_change(:follower_address, followers)
-    else
-      changeset
-    end
   end
 
   defp autofollow_users(user) do
@@ -347,9 +338,8 @@ defmodule Pleroma.User do
 
   @doc "Inserts provided changeset, performs post-registration actions (confirmation email sending etc.)"
   def register(%Ecto.Changeset{} = changeset) do
-    with {:ok, user} <- Repo.insert(changeset),
+    with {:ok, user} <- Repo.insert(changeset) do
-         {:ok, user} <- post_register_action(user) do
+      post_register_action(user)
-      {:ok, user}
     end
   end
 
@@ -395,7 +385,7 @@ defmodule Pleroma.User do
   end
 
   def maybe_direct_follow(%User{} = follower, %User{} = followed) do
-    if not User.ap_enabled?(followed) do
+    if not ap_enabled?(followed) do
       follow(follower, followed)
     else
       {:ok, follower}
@@ -428,9 +418,7 @@ defmodule Pleroma.User do
 
     {1, [follower]} = Repo.update_all(q, [])
 
-    Enum.each(followeds, fn followed ->
+    Enum.each(followeds, &update_follower_count/1)
-      update_follower_count(followed)
-    end)
 
     set_cache(follower)
   end
@@ -517,6 +505,11 @@ defmodule Pleroma.User do
     |> Repo.all()
   end
 
+  def get_all_by_ids(ids) do
+    from(u in __MODULE__, where: u.id in ^ids)
+    |> Repo.all()
+  end
+
   # This is mostly an SPC migration fix. This guesses the user nickname by taking the last part
   # of the ap_id and the domain and tries to get that user
   def get_by_guessed_nickname(ap_id) do
@@ -540,8 +533,6 @@ defmodule Pleroma.User do
   def update_and_set_cache(changeset) do
     with {:ok, user} <- Repo.update(changeset, stale_error_field: :id) do
       set_cache(user)
-    else
-      e -> e
     end
   end
 
@@ -578,9 +569,7 @@ defmodule Pleroma.User do
     key = "nickname:#{nickname}"
 
     Cachex.fetch!(:user_cache, key, fn ->
-      user_result = get_or_fetch_by_nickname(nickname)
+      case get_or_fetch_by_nickname(nickname) do
-
-      case user_result do
         {:ok, user} -> {:commit, user}
         {:error, _error} -> {:ignore, nil}
       end
@@ -591,7 +580,7 @@ defmodule Pleroma.User do
     restrict_to_local = Pleroma.Config.get([:instance, :limit_to_local_content])
 
     cond do
-      is_integer(nickname_or_id) or Pleroma.FlakeId.is_flake_id?(nickname_or_id) ->
+      is_integer(nickname_or_id) or FlakeId.flake_id?(nickname_or_id) ->
         get_cached_by_id(nickname_or_id) || get_cached_by_nickname(nickname_or_id)
 
       restrict_to_local == false ->
@@ -620,13 +609,11 @@ defmodule Pleroma.User do
 
   def get_cached_user_info(user) do
     key = "user_info:#{user.id}"
-    Cachex.fetch!(:user_cache, key, fn _ -> user_info(user) end)
+    Cachex.fetch!(:user_cache, key, fn -> user_info(user) end)
   end
 
   def fetch_by_nickname(nickname) do
-    ap_try = ActivityPub.make_user_from_nickname(nickname)
+    case ActivityPub.make_user_from_nickname(nickname) do
-
-    case ap_try do
       {:ok, user} -> {:ok, user}
       _ -> OStatus.make_user(nickname)
     end
@@ -661,7 +648,8 @@ defmodule Pleroma.User do
   end
 
   def get_followers_query(user, page) do
-    from(u in get_followers_query(user, nil))
+    user
+    |> get_followers_query(nil)
     |> User.Query.paginate(page, 20)
   end
 
@@ -670,25 +658,24 @@ defmodule Pleroma.User do
 
   @spec get_followers(User.t(), pos_integer()) :: {:ok, list(User.t())}
   def get_followers(user, page \\ nil) do
-    q = get_followers_query(user, page)
+    user
-
+    |> get_followers_query(page)
-    {:ok, Repo.all(q)}
+    |> Repo.all()
   end
 
   @spec get_external_followers(User.t(), pos_integer()) :: {:ok, list(User.t())}
   def get_external_followers(user, page \\ nil) do
-    q =
+    user
-      user
+    |> get_followers_query(page)
-      |> get_followers_query(page)
+    |> User.Query.build(%{external: true})
-      |> User.Query.build(%{external: true})
+    |> Repo.all()
-
-    {:ok, Repo.all(q)}
   end
 
   def get_followers_ids(user, page \\ nil) do
-    q = get_followers_query(user, page)
+    user
-
+    |> get_followers_query(page)
-    Repo.all(from(u in q, select: u.id))
+    |> select([u], u.id)
+    |> Repo.all()
   end
 
   @spec get_friends_query(User.t(), pos_integer() | nil) :: Ecto.Query.t()
@@ -697,7 +684,8 @@ defmodule Pleroma.User do
   end
 
   def get_friends_query(user, page) do
-    from(u in get_friends_query(user, nil))
+    user
+    |> get_friends_query(nil)
     |> User.Query.paginate(page, 20)
   end
 
@@ -705,28 +693,27 @@ defmodule Pleroma.User do
   def get_friends_query(user), do: get_friends_query(user, nil)
 
   def get_friends(user, page \\ nil) do
-    q = get_friends_query(user, page)
+    user
-
+    |> get_friends_query(page)
-    {:ok, Repo.all(q)}
+    |> Repo.all()
   end
 
   def get_friends_ids(user, page \\ nil) do
-    q = get_friends_query(user, page)
+    user
-
+    |> get_friends_query(page)
-    Repo.all(from(u in q, select: u.id))
+    |> select([u], u.id)
+    |> Repo.all()
   end
 
   @spec get_follow_requests(User.t()) :: {:ok, [User.t()]}
   def get_follow_requests(%User{} = user) do
-    users =
+    user
-      Activity.follow_requests_for_actor(user)
+    |> Activity.follow_requests_for_actor()
-      |> join(:inner, [a], u in User, on: a.actor == u.ap_id)
+    |> join(:inner, [a], u in User, on: a.actor == u.ap_id)
-      |> where([a, u], not fragment("? @> ?", u.following, ^[user.follower_address]))
+    |> where([a, u], not fragment("? @> ?", u.following, ^[user.follower_address]))
-      |> group_by([a, u], u.id)
+    |> group_by([a, u], u.id)
-      |> select([a, u], u)
+    |> select([a, u], u)
-      |> Repo.all()
+    |> Repo.all()
-
-    {:ok, users}
   end
 
   def increase_note_count(%User{} = user) do
@@ -772,20 +759,27 @@ defmodule Pleroma.User do
   end
 
   def update_note_count(%User{} = user) do
-    note_count_query =
+    note_count =
       from(
         a in Object,
         where: fragment("?->>'actor' = ? and ?->>'type' = 'Note'", a.data, ^user.ap_id, a.data),
         select: count(a.id)
       )
+      |> Repo.one()
 
-    note_count = Repo.one(note_count_query)
+    update_info(user, &User.Info.set_note_count(&1, note_count))
+  end
 
-    info_cng = User.Info.set_note_count(user.info, note_count)
+  def update_mascot(user, url) do
+    info_changeset =
+      User.Info.mascot_update(
+        user.info,
+        url
+      )
 
     user
     |> change()
-    |> put_embed(:info, info_cng)
+    |> put_embed(:info, info_changeset)
     |> update_and_set_cache()
   end
 
@@ -803,17 +797,7 @@ defmodule Pleroma.User do
 
   def fetch_follow_information(user) do
     with {:ok, info} <- ActivityPub.fetch_follow_information_for_user(user) do
-      info_cng = User.Info.follow_information_update(user.info, info)
+      update_info(user, &User.Info.follow_information_update(&1, info))
-
-      changeset =
-        user
-        |> change()
-        |> put_embed(:info, info_cng)
-
-      update_and_set_cache(changeset)
-    else
-      {:error, _} = e -> e
-      e -> {:error, e}
     end
   end
 
@@ -887,62 +871,28 @@ defmodule Pleroma.User do
 
   @spec mute(User.t(), User.t(), boolean()) :: {:ok, User.t()} | {:error, String.t()}
   def mute(muter, %User{ap_id: ap_id}, notifications? \\ true) do
-    info = muter.info
+    update_info(muter, &User.Info.add_to_mutes(&1, ap_id, notifications?))
-
-    info_cng =
-      User.Info.add_to_mutes(info, ap_id)
-      |> User.Info.add_to_muted_notifications(info, ap_id, notifications?)
-
-    cng =
-      change(muter)
-      |> put_embed(:info, info_cng)
-
-    update_and_set_cache(cng)
   end
 
   def unmute(muter, %{ap_id: ap_id}) do
-    info = muter.info
+    update_info(muter, &User.Info.remove_from_mutes(&1, ap_id))
-
-    info_cng =
-      User.Info.remove_from_mutes(info, ap_id)
-      |> User.Info.remove_from_muted_notifications(info, ap_id)
-
-    cng =
-      change(muter)
-      |> put_embed(:info, info_cng)
-
-    update_and_set_cache(cng)
   end
 
   def subscribe(subscriber, %{ap_id: ap_id}) do
-    deny_follow_blocked = Pleroma.Config.get([:user, :deny_follow_blocked])
-
     with %User{} = subscribed <- get_cached_by_ap_id(ap_id) do
-      blocked = blocks?(subscribed, subscriber) and deny_follow_blocked
+      deny_follow_blocked = Pleroma.Config.get([:user, :deny_follow_blocked])
 
-      if blocked do
+      if blocks?(subscribed, subscriber) and deny_follow_blocked do
         {:error, "Could not subscribe: #{subscribed.nickname} is blocking you"}
       else
-        info_cng =
+        update_info(subscribed, &User.Info.add_to_subscribers(&1, subscriber.ap_id))
-          subscribed.info
-          |> User.Info.add_to_subscribers(subscriber.ap_id)
-
-        change(subscribed)
-        |> put_embed(:info, info_cng)
-        |> update_and_set_cache()
       end
     end
   end
 
   def unsubscribe(unsubscriber, %{ap_id: ap_id}) do
     with %User{} = user <- get_cached_by_ap_id(ap_id) do
-      info_cng =
+      update_info(user, &User.Info.remove_from_subscribers(&1, unsubscriber.ap_id))
-        user.info
-        |> User.Info.remove_from_subscribers(unsubscriber.ap_id)
-
-      change(user)
-      |> put_embed(:info, info_cng)
-      |> update_and_set_cache()
     end
   end
 
@@ -971,21 +921,11 @@ defmodule Pleroma.User do
         blocker
       end
 
-    if following?(blocked, blocker) do
+    if following?(blocked, blocker), do: unfollow(blocked, blocker)
-      unfollow(blocked, blocker)
-    end
 
     {:ok, blocker} = update_follower_count(blocker)
 
-    info_cng =
+    update_info(blocker, &User.Info.add_to_block(&1, ap_id))
-      blocker.info
-      |> User.Info.add_to_block(ap_id)
-
-    cng =
-      change(blocker)
-      |> put_embed(:info, info_cng)
-
-    update_and_set_cache(cng)
   end
 
   # helper to handle the block given only an actor's AP id
@@ -994,15 +934,7 @@ defmodule Pleroma.User do
   end
 
   def unblock(blocker, %{ap_id: ap_id}) do
-    info_cng =
+    update_info(blocker, &User.Info.remove_from_block(&1, ap_id))
-      blocker.info
-      |> User.Info.remove_from_block(ap_id)
-
-    cng =
-      change(blocker)
-      |> put_embed(:info, info_cng)
-
-    update_and_set_cache(cng)
   end
 
   def mutes?(nil, _), do: false
@@ -1059,27 +991,11 @@ defmodule Pleroma.User do
   end
 
   def block_domain(user, domain) do
-    info_cng =
+    update_info(user, &User.Info.add_to_domain_block(&1, domain))
-      user.info
-      |> User.Info.add_to_domain_block(domain)
-
-    cng =
-      change(user)
-      |> put_embed(:info, info_cng)
-
-    update_and_set_cache(cng)
   end
 
   def unblock_domain(user, domain) do
-    info_cng =
+    update_info(user, &User.Info.remove_from_domain_block(&1, domain))
-      user.info
-      |> User.Info.remove_from_domain_block(domain)
-
-    cng =
-      change(user)
-      |> put_embed(:info, info_cng)
-
-    update_and_set_cache(cng)
   end
 
   def deactivate_async(user, status \\ true) do
@@ -1087,51 +1003,41 @@ defmodule Pleroma.User do
   end
 
   def deactivate(%User{} = user, status \\ true) do
-    info_cng = User.Info.set_activation_status(user.info, status)
+    with {:ok, user} <- update_info(user, &User.Info.set_activation_status(&1, status)) do
-
+      Enum.each(get_followers(user), &invalidate_cache/1)
-    with {:ok, friends} <- User.get_friends(user),
+      Enum.each(get_friends(user), &update_follower_count/1)
-         {:ok, followers} <- User.get_followers(user),
-         {:ok, user} <-
-           user
-           |> change()
-           |> put_embed(:info, info_cng)
-           |> update_and_set_cache() do
-      Enum.each(followers, &invalidate_cache(&1))
-      Enum.each(friends, &update_follower_count(&1))
 
       {:ok, user}
     end
   end
 
   def update_notification_settings(%User{} = user, settings \\ %{}) do
-    info_changeset = User.Info.update_notification_settings(user.info, settings)
+    update_info(user, &User.Info.update_notification_settings(&1, settings))
-
-    change(user)
-    |> put_embed(:info, info_changeset)
-    |> update_and_set_cache()
   end
 
   def delete(%User{} = user) do
     BackgroundWorker.enqueue("delete_user", %{"user_id" => user.id})
   end
 
+  def perform(:force_password_reset, user), do: force_password_reset(user)
+
   @spec perform(atom(), User.t()) :: {:ok, User.t()}
   def perform(:delete, %User{} = user) do
     {:ok, _user} = ActivityPub.delete(user)
 
     # Remove all relationships
-    {:ok, followers} = User.get_followers(user)
+    user
-
+    |> get_followers()
-    Enum.each(followers, fn follower ->
+    |> Enum.each(fn follower ->
       ActivityPub.unfollow(follower, user)
-      User.unfollow(follower, user)
+      unfollow(follower, user)
     end)
 
-    {:ok, friends} = User.get_friends(user)
+    user
-
+    |> get_friends()
-    Enum.each(friends, fn followed ->
+    |> Enum.each(fn followed ->
       ActivityPub.unfollow(user, followed)
-      User.unfollow(user, followed)
+      unfollow(user, followed)
     end)
 
     delete_user_activities(user)
@@ -1143,13 +1049,11 @@ defmodule Pleroma.User do
   def perform(:fetch_initial_posts, %User{} = user) do
     pages = Pleroma.Config.get!([:fetch_initial_posts, :pages])
 
-    Enum.each(
+    # Insert all the posts in reverse order, so they're in the right order on the timeline
-      # Insert all the posts in reverse order, so they're in the right order on the timeline
+    user.info.source_data["outbox"]
-      Enum.reverse(Utils.fetch_ordered_collection(user.info.source_data["outbox"], pages)),
+    |> Utils.fetch_ordered_collection(pages)
-      &Pleroma.Web.Federator.incoming_ap_doc/1
+    |> Enum.reverse()
-    )
+    |> Enum.each(&Pleroma.Web.Federator.incoming_ap_doc/1)
-
-    {:ok, user}
   end
 
   def perform(:deactivate_async, user, status), do: deactivate(user, status)
@@ -1235,16 +1139,12 @@ defmodule Pleroma.User do
     })
   end
 
-  def delete_user_activities(%User{ap_id: ap_id} = user) do
+  def delete_user_activities(%User{ap_id: ap_id}) do
     ap_id
     |> Activity.Queries.by_actor()
     |> RepoStreamer.chunk_stream(50)
-    |> Stream.each(fn activities ->
+    |> Stream.each(fn activities -> Enum.each(activities, &delete_activity/1) end)
-      Enum.each(activities, &delete_activity(&1))
-    end)
     |> Stream.run()
-
-    {:ok, user}
   end
 
   defp delete_activity(%{data: %{"type" => "Create"}} = activity) do
@@ -1254,17 +1154,19 @@ defmodule Pleroma.User do
   end
 
   defp delete_activity(%{data: %{"type" => "Like"}} = activity) do
-    user = get_cached_by_ap_id(activity.actor)
     object = Object.normalize(activity)
 
-    ActivityPub.unlike(user, object)
+    activity.actor
+    |> get_cached_by_ap_id()
+    |> ActivityPub.unlike(object)
   end
 
   defp delete_activity(%{data: %{"type" => "Announce"}} = activity) do
-    user = get_cached_by_ap_id(activity.actor)
     object = Object.normalize(activity)
 
-    ActivityPub.unannounce(user, object)
+    activity.actor
+    |> get_cached_by_ap_id()
+    |> ActivityPub.unannounce(object)
   end
 
   defp delete_activity(_activity), do: "Doing nothing"
@@ -1276,9 +1178,7 @@ defmodule Pleroma.User do
   def html_filter_policy(_), do: Pleroma.Config.get([:markup, :scrub_policy])
 
   def fetch_by_ap_id(ap_id) do
-    ap_try = ActivityPub.make_user_from_ap_id(ap_id)
+    case ActivityPub.make_user_from_ap_id(ap_id) do
-
-    case ap_try do
       {:ok, user} ->
         {:ok, user}
 
@@ -1293,7 +1193,7 @@ defmodule Pleroma.User do
   def get_or_fetch_by_ap_id(ap_id) do
     user = get_cached_by_ap_id(ap_id)
 
-    if !is_nil(user) and !User.needs_update?(user) do
+    if !is_nil(user) and !needs_update?(user) do
       {:ok, user}
     else
       # Whether to fetch initial posts for the user (if it's a new user & the fetching is enabled)
@@ -1313,19 +1213,20 @@ defmodule Pleroma.User do
 
   @doc "Creates an internal service actor by URI if missing.  Optionally takes nickname for addressing."
   def get_or_create_service_actor_by_ap_id(uri, nickname \\ nil) do
-    if user = get_cached_by_ap_id(uri) do
+    with %User{} = user <- get_cached_by_ap_id(uri) do
       user
     else
-      changes =
+      _ ->
-        %User{info: %User.Info{}}
+        {:ok, user} =
-        |> cast(%{}, [:ap_id, :nickname, :local])
+          %User{info: %User.Info{}}
-        |> put_change(:ap_id, uri)
+          |> cast(%{}, [:ap_id, :nickname, :local])
-        |> put_change(:nickname, nickname)
+          |> put_change(:ap_id, uri)
-        |> put_change(:local, true)
+          |> put_change(:nickname, nickname)
-        |> put_change(:follower_address, uri <> "/followers")
+          |> put_change(:local, true)
+          |> put_change(:follower_address, uri <> "/followers")
+          |> Repo.insert()
 
-      {:ok, user} = Repo.insert(changes)
+        user
-      user
     end
   end
 
@@ -1382,23 +1283,21 @@ defmodule Pleroma.User do
   # this is because we have synchronous follow APIs and need to simulate them
   # with an async handshake
   def wait_and_refresh(_, %User{local: true} = a, %User{local: true} = b) do
-    with %User{} = a <- User.get_cached_by_id(a.id),
+    with %User{} = a <- get_cached_by_id(a.id),
-         %User{} = b <- User.get_cached_by_id(b.id) do
+         %User{} = b <- get_cached_by_id(b.id) do
       {:ok, a, b}
     else
-      _e ->
+      nil -> :error
-        :error
     end
   end
 
   def wait_and_refresh(timeout, %User{} = a, %User{} = b) do
     with :ok <- :timer.sleep(timeout),
-         %User{} = a <- User.get_cached_by_id(a.id),
+         %User{} = a <- get_cached_by_id(a.id),
-         %User{} = b <- User.get_cached_by_id(b.id) do
+         %User{} = b <- get_cached_by_id(b.id) do
       {:ok, a, b}
     else
-      _e ->
+      nil -> :error
-        :error
     end
   end
 
@@ -1460,7 +1359,7 @@ defmodule Pleroma.User do
   defp normalize_tags(tags) do
     [tags]
     |> List.flatten()
-    |> Enum.map(&String.downcase(&1))
+    |> Enum.map(&String.downcase/1)
   end
 
   defp local_nickname_regex do
@@ -1553,11 +1452,7 @@ defmodule Pleroma.User do
   @spec switch_email_notifications(t(), String.t(), boolean()) ::
           {:ok, t()} | {:error, Ecto.Changeset.t()}
   def switch_email_notifications(user, type, status) do
-    info = Pleroma.User.Info.update_email_notifications(user.info, %{type => status})
+    update_info(user, &User.Info.update_email_notifications(&1, %{type => status}))
-
-    change(user)
-    |> put_embed(:info, info)
-    |> update_and_set_cache()
   end
 
   @doc """
@@ -1579,13 +1474,8 @@ defmodule Pleroma.User do
   def toggle_confirmation(%User{} = user) do
     need_confirmation? = !user.info.confirmation_pending
 
-    info_changeset =
-      User.Info.confirmation_changeset(user.info, need_confirmation: need_confirmation?)
-
     user
-    |> change()
+    |> update_info(&User.Info.confirmation_changeset(&1, need_confirmation: need_confirmation?))
-    |> put_embed(:info, info_changeset)
-    |> update_and_set_cache()
   end
 
   def get_mascot(%{info: %{mascot: %{} = mascot}}) when not is_nil(mascot) do
@@ -1608,16 +1498,11 @@ defmodule Pleroma.User do
     }
   end
 
-  def ensure_keys_present(%User{info: info} = user) do
+  def ensure_keys_present(%{info: %{keys: keys}} = user) when not is_nil(keys), do: {:ok, user}
-    if info.keys do
-      {:ok, user}
-    else
-      {:ok, pem} = Keys.generate_rsa_pem()
 
-      user
+  def ensure_keys_present(%User{} = user) do
-      |> Ecto.Changeset.change()
+    with {:ok, pem} <- Keys.generate_rsa_pem() do
-      |> Ecto.Changeset.put_embed(:info, User.Info.set_keys(info, pem))
+      update_info(user, &User.Info.set_keys(&1, pem))
-      |> update_and_set_cache()
     end
   end
 
@@ -1663,4 +1548,26 @@ defmodule Pleroma.User do
     |> validate_format(:email, @email_regex)
     |> update_and_set_cache()
   end
+
+  @doc """
+  Changes `user.info` and returns the user changeset.
+
+  `fun` is called with the `user.info`.
+  """
+  def change_info(user, fun) do
+    changeset = change(user)
+    info = get_field(changeset, :info) || %User.Info{}
+    put_embed(changeset, :info, fun.(info))
+  end
+
+  @doc """
+  Updates `user.info` and sets cache.
+
+  `fun` is called with the `user.info`.
+  """
+  def update_info(user, fun) do
+    user
+    |> change_info(fun)
+    |> update_and_set_cache()
+  end
 end

lib/pleroma/user/info.ex

@@ -20,6 +20,7 @@ defmodule Pleroma.User.Info do
     field(:following_count, :integer, default: nil)
     field(:locked, :boolean, default: false)
     field(:confirmation_pending, :boolean, default: false)
+    field(:password_reset_pending, :boolean, default: false)
     field(:confirmation_token, :string, default: nil)
     field(:default_scope, :string, default: "public")
     field(:blocks, {:array, :string}, default: [])
@@ -41,6 +42,8 @@ defmodule Pleroma.User.Info do
     field(:topic, :string, default: nil)
     field(:hub, :string, default: nil)
     field(:salmon, :string, default: nil)
+    field(:hide_followers_count, :boolean, default: false)
+    field(:hide_follows_count, :boolean, default: false)
     field(:hide_followers, :boolean, default: false)
     field(:hide_follows, :boolean, default: false)
     field(:hide_favorites, :boolean, default: true)
@@ -51,6 +54,7 @@ defmodule Pleroma.User.Info do
     field(:pleroma_settings_store, :map, default: %{})
     field(:fields, {:array, :map}, default: nil)
     field(:raw_fields, {:array, :map}, default: [])
+    field(:discoverable, :boolean, default: false)
 
     field(:notification_settings, :map,
       default: %{
@@ -80,6 +84,14 @@ defmodule Pleroma.User.Info do
     |> validate_required([:deactivated])
   end
 
+  def set_password_reset_pending(info, pending) do
+    params = %{password_reset_pending: pending}
+
+    info
+    |> cast(params, [:password_reset_pending])
+    |> validate_required([:password_reset_pending])
+  end
+
   def update_notification_settings(info, settings) do
     settings =
       settings
@@ -176,16 +188,11 @@ defmodule Pleroma.User.Info do
     |> validate_required([:subscribers])
   end
 
-  @spec add_to_mutes(Info.t(), String.t()) :: Changeset.t()
+  @spec add_to_mutes(Info.t(), String.t(), boolean()) :: Changeset.t()
-  def add_to_mutes(info, muted) do
+  def add_to_mutes(info, muted, notifications?) do
-    set_mutes(info, Enum.uniq([muted | info.mutes]))
+    info
-  end
+    |> set_mutes(Enum.uniq([muted | info.mutes]))
-
+    |> set_notification_mutes(
-  @spec add_to_muted_notifications(Changeset.t(), Info.t(), String.t(), boolean()) ::
-          Changeset.t()
-  def add_to_muted_notifications(changeset, info, muted, notifications?) do
-    set_notification_mutes(
-      changeset,
       Enum.uniq([muted | info.muted_notifications]),
       notifications?
     )
@@ -193,12 +200,9 @@ defmodule Pleroma.User.Info do
 
   @spec remove_from_mutes(Info.t(), String.t()) :: Changeset.t()
   def remove_from_mutes(info, muted) do
-    set_mutes(info, List.delete(info.mutes, muted))
+    info
-  end
+    |> set_mutes(List.delete(info.mutes, muted))
-
+    |> set_notification_mutes(List.delete(info.muted_notifications, muted), true)
-  @spec remove_from_muted_notifications(Changeset.t(), Info.t(), String.t()) :: Changeset.t()
-  def remove_from_muted_notifications(changeset, info, muted) do
-    set_notification_mutes(changeset, List.delete(info.muted_notifications, muted), true)
   end
 
   def add_to_block(info, blocked) do
@@ -262,9 +266,12 @@ defmodule Pleroma.User.Info do
       :salmon,
       :hide_followers,
       :hide_follows,
+      :hide_followers_count,
+      :hide_follows_count,
       :follower_count,
       :fields,
-      :following_count
+      :following_count,
+      :discoverable
     ])
     |> validate_fields(true)
   end
@@ -281,7 +288,10 @@ defmodule Pleroma.User.Info do
       :following_count,
       :hide_follows,
       :fields,
-      :hide_followers
+      :hide_followers,
+      :discoverable,
+      :hide_followers_count,
+      :hide_follows_count
     ])
     |> validate_fields(remote?)
   end
@@ -295,13 +305,16 @@ defmodule Pleroma.User.Info do
       :banner,
       :hide_follows,
       :hide_followers,
+      :hide_followers_count,
+      :hide_follows_count,
       :hide_favorites,
       :background,
       :show_role,
       :skip_thread_containment,
       :fields,
       :raw_fields,
-      :pleroma_settings_store
+      :pleroma_settings_store,
+      :discoverable
     ])
     |> validate_fields()
   end
@@ -325,9 +338,7 @@ defmodule Pleroma.User.Info do
     name_limit = Pleroma.Config.get([:instance, :account_field_name_length], 255)
     value_limit = Pleroma.Config.get([:instance, :account_field_value_length], 255)
 
-    is_binary(name) &&
+    is_binary(name) && is_binary(value) && String.length(name) <= name_limit &&
-      is_binary(value) &&
-      String.length(name) <= name_limit &&
       String.length(value) <= value_limit
   end
 
@@ -458,7 +469,9 @@ defmodule Pleroma.User.Info do
       :hide_followers,
       :hide_follows,
       :follower_count,
-      :following_count
+      :following_count,
+      :hide_followers_count,
+      :hide_follows_count
     ])
   end
 end

lib/pleroma/user/query.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.User.Query do

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -248,6 +248,26 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     end
   end
 
+  def listen(%{to: to, actor: actor, context: context, object: object} = params) do
+    additional = params[:additional] || %{}
+    # only accept false as false value
+    local = !(params[:local] == false)
+    published = params[:published]
+
+    with listen_data <-
+           make_listen_data(
+             %{to: to, actor: actor, published: published, context: context, object: object},
+             additional
+           ),
+         {:ok, activity} <- insert(listen_data, local),
+         :ok <- maybe_federate(activity) do
+      {:ok, activity}
+    else
+      {:error, message} ->
+        {:error, message}
+    end
+  end
+
   def accept(%{to: to, actor: actor, object: object} = params) do
     # only accept false as false value
     local = !(params[:local] == false)
@@ -510,7 +530,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   end
 
   @spec fetch_latest_activity_id_for_context(String.t(), keyword() | map()) ::
-          Pleroma.FlakeId.t() | nil
+          FlakeId.Ecto.CompatType.t() | nil
   def fetch_latest_activity_id_for_context(context, opts \\ %{}) do
     context
     |> fetch_activities_for_context_query(Map.merge(%{"skip_preload" => true}, opts))
@@ -519,12 +539,13 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     |> Repo.one()
   end
 
-  def fetch_public_activities(opts \\ %{}) do
+  def fetch_public_activities(opts \\ %{}, pagination \\ :keyset) do
-    q = fetch_activities_query([Pleroma.Constants.as_public()], opts)
+    opts = Map.drop(opts, ["user"])
 
-    q
+    [Pleroma.Constants.as_public()]
+    |> fetch_activities_query(opts)
     |> restrict_unlisted()
-    |> Pagination.fetch_paginated(opts)
+    |> Pagination.fetch_paginated(opts, pagination)
     |> Enum.reverse()
   end
 
@@ -587,6 +608,23 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp restrict_thread_visibility(query, _, _), do: query
 
+  def fetch_user_abstract_activities(user, reading_user, params \\ %{}) do
+    params =
+      params
+      |> Map.put("user", reading_user)
+      |> Map.put("actor_id", user.ap_id)
+      |> Map.put("whole_db", true)
+
+    recipients =
+      user_activities_recipients(%{
+        "godmode" => params["godmode"],
+        "reading_user" => reading_user
+      })
+
+    fetch_activities(recipients, params)
+    |> Enum.reverse()
+  end
+
   def fetch_user_activities(user, reading_user, params \\ %{}) do
     params =
       params
@@ -833,7 +871,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp restrict_muted_reblogs(query, _), do: query
 
-  defp exclude_poll_votes(query, %{"include_poll_votes" => "true"}), do: query
+  defp exclude_poll_votes(query, %{"include_poll_votes" => true}), do: query
 
   defp exclude_poll_votes(query, _) do
     if has_named_binding?(query, :object) do
@@ -917,11 +955,11 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     |> exclude_poll_votes(opts)
   end
 
-  def fetch_activities(recipients, opts \\ %{}) do
+  def fetch_activities(recipients, opts \\ %{}, pagination \\ :keyset) do
     list_memberships = Pleroma.List.memberships(opts["user"])
 
     fetch_activities_query(recipients ++ list_memberships, opts)
-    |> Pagination.fetch_paginated(opts)
+    |> Pagination.fetch_paginated(opts, pagination)
     |> Enum.reverse()
     |> maybe_update_cc(list_memberships, opts["user"])
   end
@@ -952,10 +990,15 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     )
   end
 
-  def fetch_activities_bounded(recipients, recipients_with_public, opts \\ %{}) do
+  def fetch_activities_bounded(
+        recipients,
+        recipients_with_public,
+        opts \\ %{},
+        pagination \\ :keyset
+      ) do
     fetch_activities_query([], opts)
     |> fetch_activities_bounded_query(recipients, recipients_with_public)
-    |> Pagination.fetch_paginated(opts)
+    |> Pagination.fetch_paginated(opts, pagination)
     |> Enum.reverse()
   end
 
@@ -995,6 +1038,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
     locked = data["manuallyApprovesFollowers"] || false
     data = Transmogrifier.maybe_fix_user_object(data)
+    discoverable = data["discoverable"] || false
 
     user_data = %{
       ap_id: data["id"],
@@ -1003,7 +1047,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
         source_data: data,
         banner: banner,
         fields: fields,
-        locked: locked
+        locked: locked,
+        discoverable: discoverable
       },
       avatar: avatar,
       name: data["name"],

lib/pleroma/web/activity_pub/activity_pub_controller.ex

@@ -49,7 +49,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
          {:ok, user} <- User.ensure_keys_present(user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("user.json", %{user: user}))
+      |> put_view(UserView)
+      |> render("user.json", %{user: user})
     else
       nil -> {:error, :not_found}
     end
@@ -90,7 +91,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
 
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(ObjectView.render("likes.json", ap_id, likes, page))
+      |> put_view(ObjectView)
+      |> render("likes.json", %{ap_id: ap_id, likes: likes, page: page})
     else
       {:public?, false} ->
         {:error, :not_found}
@@ -104,7 +106,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
          likes <- Utils.get_object_likes(object) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(ObjectView.render("likes.json", ap_id, likes))
+      |> put_view(ObjectView)
+      |> render("likes.json", %{ap_id: ap_id, likes: likes})
     else
       {:public?, false} ->
         {:error, :not_found}
@@ -158,7 +161,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
   def following(%{assigns: %{relay: true}} = conn, _params) do
     conn
     |> put_resp_content_type("application/activity+json")
-    |> json(UserView.render("following.json", %{user: Relay.get_actor()}))
+    |> put_view(UserView)
+    |> render("following.json", %{user: Relay.get_actor()})
   end
 
   def following(%{assigns: %{user: for_user}} = conn, %{"nickname" => nickname, "page" => page}) do
@@ -170,7 +174,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
 
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("following.json", %{user: user, page: page, for: for_user}))
+      |> put_view(UserView)
+      |> render("following.json", %{user: user, page: page, for: for_user})
     else
       {:show_follows, _} ->
         conn
@@ -184,7 +189,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
          {user, for_user} <- ensure_user_keys_present_and_maybe_refresh_for_user(user, for_user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("following.json", %{user: user, for: for_user}))
+      |> put_view(UserView)
+      |> render("following.json", %{user: user, for: for_user})
     end
   end
 
@@ -192,7 +198,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
   def followers(%{assigns: %{relay: true}} = conn, _params) do
     conn
     |> put_resp_content_type("application/activity+json")
-    |> json(UserView.render("followers.json", %{user: Relay.get_actor()}))
+    |> put_view(UserView)
+    |> render("followers.json", %{user: Relay.get_actor()})
   end
 
   def followers(%{assigns: %{user: for_user}} = conn, %{"nickname" => nickname, "page" => page}) do
@@ -204,7 +211,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
 
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("followers.json", %{user: user, page: page, for: for_user}))
+      |> put_view(UserView)
+      |> render("followers.json", %{user: user, page: page, for: for_user})
     else
       {:show_followers, _} ->
         conn
@@ -218,16 +226,48 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
          {user, for_user} <- ensure_user_keys_present_and_maybe_refresh_for_user(user, for_user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("followers.json", %{user: user, for: for_user}))
+      |> put_view(UserView)
+      |> render("followers.json", %{user: user, for: for_user})
     end
   end
 
-  def outbox(conn, %{"nickname" => nickname} = params) do
+  def outbox(conn, %{"nickname" => nickname, "page" => page?} = params)
+      when page? in [true, "true"] do
+    with %User{} = user <- User.get_cached_by_nickname(nickname),
+         {:ok, user} <- User.ensure_keys_present(user) do
+      activities =
+        if params["max_id"] do
+          ActivityPub.fetch_user_activities(user, nil, %{
+            "max_id" => params["max_id"],
+            # This is a hack because postgres generates inefficient queries when filtering by
+            # 'Answer', poll votes will be hidden by the visibility filter in this case anyway
+            "include_poll_votes" => true,
+            "limit" => 10
+          })
+        else
+          ActivityPub.fetch_user_activities(user, nil, %{
+            "limit" => 10,
+            "include_poll_votes" => true
+          })
+        end
+
+      conn
+      |> put_resp_content_type("application/activity+json")
+      |> put_view(UserView)
+      |> render("activity_collection_page.json", %{
+        activities: activities,
+        iri: "#{user.ap_id}/outbox"
+      })
+    end
+  end
+
+  def outbox(conn, %{"nickname" => nickname}) do
     with %User{} = user <- User.get_cached_by_nickname(nickname),
          {:ok, user} <- User.ensure_keys_present(user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("outbox.json", %{user: user, max_id: params["max_id"]}))
+      |> put_view(UserView)
+      |> render("activity_collection.json", %{iri: "#{user.ap_id}/outbox"})
     end
   end
 
@@ -275,7 +315,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
     with {:ok, user} <- User.ensure_keys_present(user) do
       conn
       |> put_resp_content_type("application/activity+json")
-      |> json(UserView.render("user.json", %{user: user}))
+      |> put_view(UserView)
+      |> render("user.json", %{user: user})
     else
       nil -> {:error, :not_found}
     end
@@ -293,22 +334,49 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
     |> represent_service_actor(conn)
   end
 
+  @doc "Returns the authenticated user's ActivityPub User object or a 404 Not Found if non-authenticated"
   def whoami(%{assigns: %{user: %User{} = user}} = conn, _params) do
     conn
     |> put_resp_content_type("application/activity+json")
-    |> json(UserView.render("user.json", %{user: user}))
+    |> put_view(UserView)
+    |> render("user.json", %{user: user})
   end
 
   def whoami(_conn, _params), do: {:error, :not_found}
 
   def read_inbox(
         %{assigns: %{user: %{nickname: nickname} = user}} = conn,
-        %{"nickname" => nickname} = params
+        %{"nickname" => nickname, "page" => page?} = params
-      ) do
+      )
+      when page? in [true, "true"] do
+    activities =
+      if params["max_id"] do
+        ActivityPub.fetch_activities([user.ap_id | user.following], %{
+          "max_id" => params["max_id"],
+          "limit" => 10
+        })
+      else
+        ActivityPub.fetch_activities([user.ap_id | user.following], %{"limit" => 10})
+      end
+
     conn
     |> put_resp_content_type("application/activity+json")
     |> put_view(UserView)
-    |> render("inbox.json", user: user, max_id: params["max_id"])
+    |> render("activity_collection_page.json", %{
+      activities: activities,
+      iri: "#{user.ap_id}/inbox"
+    })
+  end
+
+  def read_inbox(%{assigns: %{user: %{nickname: nickname} = user}} = conn, %{
+        "nickname" => nickname
+      }) do
+    with {:ok, user} <- User.ensure_keys_present(user) do
+      conn
+      |> put_resp_content_type("application/activity+json")
+      |> put_view(UserView)
+      |> render("activity_collection.json", %{iri: "#{user.ap_id}/inbox"})
+    end
   end
 
   def read_inbox(%{assigns: %{user: nil}} = conn, %{"nickname" => nickname}) do
@@ -442,4 +510,31 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
 
     {new_user, for_user}
   end
+
+  # TODO: Add support for "object" field
+  @doc """
+  Endpoint based on <https://www.w3.org/wiki/SocialCG/ActivityPub/MediaUpload>
+
+  Parameters:
+  - (required) `file`: data of the media
+  - (optionnal) `description`: description of the media, intended for accessibility
+
+  Response:
+  - HTTP Code: 201 Created
+  - HTTP Body: ActivityPub object to be inserted into another's `attachment` field
+  """
+  def upload_media(%{assigns: %{user: user}} = conn, %{"file" => file} = data) do
+    with {:ok, object} <-
+           ActivityPub.upload(
+             file,
+             actor: User.ap_id(user),
+             description: Map.get(data, "description")
+           ) do
+      Logger.debug(inspect(object))
+
+      conn
+      |> put_status(:created)
+      |> json(object.data)
+    end
+  end
 end

lib/pleroma/web/activity_pub/publisher.ex

@@ -111,11 +111,11 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
 
   @spec recipients(User.t(), Activity.t()) :: list(User.t()) | []
   defp recipients(actor, activity) do
-    {:ok, followers} =
+    followers =
       if actor.follower_address in activity.recipients do
         User.get_external_followers(actor)
       else
-        {:ok, []}
+        []
       end
 
     fetchers =

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -42,8 +42,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   end
 
   def fix_summary(%{"summary" => nil} = object) do
-    object
+    Map.put(object, "summary", "")
-    |> Map.put("summary", "")
   end
 
   def fix_summary(%{"summary" => _} = object) do
@@ -51,10 +50,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     object
   end
 
-  def fix_summary(object) do
+  def fix_summary(object), do: Map.put(object, "summary", "")
-    object
-    |> Map.put("summary", "")
-  end
 
   def fix_addressing_list(map, field) do
     cond do
@@ -74,13 +70,9 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
         explicit_mentions,
         follower_collection
       ) do
-    explicit_to =
+    explicit_to = Enum.filter(to, fn x -> x in explicit_mentions end)
-      to
-      |> Enum.filter(fn x -> x in explicit_mentions end)
 
-    explicit_cc =
+    explicit_cc = Enum.filter(to, fn x -> x not in explicit_mentions end)
-      to
-      |> Enum.filter(fn x -> x not in explicit_mentions end)
 
     final_cc =
       (cc ++ explicit_cc)
@@ -98,13 +90,19 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   def fix_explicit_addressing(%{"directMessage" => true} = object), do: object
 
   def fix_explicit_addressing(object) do
-    explicit_mentions =
+    explicit_mentions = Utils.determine_explicit_mentions(object)
+
+    %User{follower_address: follower_collection} =
       object
-      |> Utils.determine_explicit_mentions()
+      |> Containment.get_actor()
+      |> User.get_cached_by_ap_id()
 
-    follower_collection = User.get_cached_by_ap_id(Containment.get_actor(object)).follower_address
+    explicit_mentions =
-
+      explicit_mentions ++
-    explicit_mentions = explicit_mentions ++ [Pleroma.Constants.as_public(), follower_collection]
+        [
+          Pleroma.Constants.as_public(),
+          follower_collection
+        ]
 
     fix_explicit_addressing(object, explicit_mentions, follower_collection)
   end
@@ -148,48 +146,25 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   end
 
   def fix_actor(%{"attributedTo" => actor} = object) do
-    object
+    Map.put(object, "actor", Containment.get_actor(%{"actor" => actor}))
-    |> Map.put("actor", Containment.get_actor(%{"actor" => actor}))
   end
 
   def fix_in_reply_to(object, options \\ [])
 
   def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
       when not is_nil(in_reply_to) do
-    in_reply_to_id =
+    in_reply_to_id = prepare_in_reply_to(in_reply_to)
-      cond do
-        is_bitstring(in_reply_to) ->
-          in_reply_to
-
-        is_map(in_reply_to) && is_bitstring(in_reply_to["id"]) ->
-          in_reply_to["id"]
-
-        is_list(in_reply_to) && is_bitstring(Enum.at(in_reply_to, 0)) ->
-          Enum.at(in_reply_to, 0)
-
-        # Maybe I should output an error too?
-        true ->
-          ""
-      end
-
     object = Map.put(object, "inReplyToAtomUri", in_reply_to_id)
 
     if Federator.allowed_incoming_reply_depth?(options[:depth]) do
-      case get_obj_helper(in_reply_to_id, options) do
+      with {:ok, replied_object} <- get_obj_helper(in_reply_to_id, options),
-        {:ok, replied_object} ->
+           %Activity{} = _ <- Activity.get_create_by_object_ap_id(replied_object.data["id"]) do
-          with %Activity{} = _activity <-
+        object
-                 Activity.get_create_by_object_ap_id(replied_object.data["id"]) do
+        |> Map.put("inReplyTo", replied_object.data["id"])
-            object
+        |> Map.put("inReplyToAtomUri", object["inReplyToAtomUri"] || in_reply_to_id)
-            |> Map.put("inReplyTo", replied_object.data["id"])
+        |> Map.put("conversation", replied_object.data["context"] || object["conversation"])
-            |> Map.put("inReplyToAtomUri", object["inReplyToAtomUri"] || in_reply_to_id)
+        |> Map.put("context", replied_object.data["context"] || object["conversation"])
-            |> Map.put("conversation", replied_object.data["context"] || object["conversation"])
+      else
-            |> Map.put("context", replied_object.data["context"] || object["conversation"])
-          else
-            e ->
-              Logger.error("Couldn't fetch #{inspect(in_reply_to_id)}, error: #{inspect(e)}")
-              object
-          end
-
         e ->
           Logger.error("Couldn't fetch #{inspect(in_reply_to_id)}, error: #{inspect(e)}")
           object
@@ -201,6 +176,22 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
   def fix_in_reply_to(object, _options), do: object
 
+  defp prepare_in_reply_to(in_reply_to) do
+    cond do
+      is_bitstring(in_reply_to) ->
+        in_reply_to
+
+      is_map(in_reply_to) && is_bitstring(in_reply_to["id"]) ->
+        in_reply_to["id"]
+
+      is_list(in_reply_to) && is_bitstring(Enum.at(in_reply_to, 0)) ->
+        Enum.at(in_reply_to, 0)
+
+      true ->
+        ""
+    end
+  end
+
   def fix_context(object) do
     context = object["context"] || object["conversation"] || Utils.generate_context_id()
 
@@ -211,11 +202,9 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
   def fix_attachments(%{"attachment" => attachment} = object) when is_list(attachment) do
     attachments =
-      attachment
+      Enum.map(attachment, fn data ->
-      |> Enum.map(fn data ->
         media_type = data["mediaType"] || data["mimeType"]
         href = data["url"] || data["href"]
-
         url = [%{"type" => "Link", "mediaType" => media_type, "href" => href}]
 
         data
@@ -223,30 +212,25 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
         |> Map.put("url", url)
       end)
 
-    object
+    Map.put(object, "attachment", attachments)
-    |> Map.put("attachment", attachments)
   end
 
   def fix_attachments(%{"attachment" => attachment} = object) when is_map(attachment) do
-    Map.put(object, "attachment", [attachment])
+    object
+    |> Map.put("attachment", [attachment])
     |> fix_attachments()
   end
 
   def fix_attachments(object), do: object
 
   def fix_url(%{"url" => url} = object) when is_map(url) do
-    object
+    Map.put(object, "url", url["href"])
-    |> Map.put("url", url["href"])
   end
 
   def fix_url(%{"type" => "Video", "url" => url} = object) when is_list(url) do
     first_element = Enum.at(url, 0)
 
-    link_element =
+    link_element = Enum.find(url, fn x -> is_map(x) and x["mimeType"] == "text/html" end)
-      url
-      |> Enum.filter(fn x -> is_map(x) end)
-      |> Enum.filter(fn x -> x["mimeType"] == "text/html" end)
-      |> Enum.at(0)
 
     object
     |> Map.put("attachment", [first_element])
@@ -264,36 +248,32 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
         true -> ""
       end
 
-    object
+    Map.put(object, "url", url_string)
-    |> Map.put("url", url_string)
   end
 
   def fix_url(object), do: object
 
   def fix_emoji(%{"tag" => tags} = object) when is_list(tags) do
-    emoji = tags |> Enum.filter(fn data -> data["type"] == "Emoji" and data["icon"] end)
-
     emoji =
-      emoji
+      tags
+      |> Enum.filter(fn data -> data["type"] == "Emoji" and data["icon"] end)
       |> Enum.reduce(%{}, fn data, mapping ->
         name = String.trim(data["name"], ":")
 
-        mapping |> Map.put(name, data["icon"]["url"])
+        Map.put(mapping, name, data["icon"]["url"])
       end)
 
     # we merge mastodon and pleroma emoji into a single mapping, to allow for both wire formats
     emoji = Map.merge(object["emoji"] || %{}, emoji)
 
-    object
+    Map.put(object, "emoji", emoji)
-    |> Map.put("emoji", emoji)
   end
 
   def fix_emoji(%{"tag" => %{"type" => "Emoji"} = tag} = object) do
     name = String.trim(tag["name"], ":")
     emoji = %{name => tag["icon"]["url"]}
 
-    object
+    Map.put(object, "emoji", emoji)
-    |> Map.put("emoji", emoji)
   end
 
   def fix_emoji(object), do: object
@@ -304,17 +284,13 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
       |> Enum.filter(fn data -> data["type"] == "Hashtag" and data["name"] end)
       |> Enum.map(fn data -> String.slice(data["name"], 1..-1) end)
 
-    combined = tag ++ tags
+    Map.put(object, "tag", tag ++ tags)
-
-    object
-    |> Map.put("tag", combined)
   end
 
   def fix_tag(%{"tag" => %{"type" => "Hashtag", "name" => hashtag} = tag} = object) do
     combined = [tag, String.slice(hashtag, 1..-1)]
 
-    object
+    Map.put(object, "tag", combined)
-    |> Map.put("tag", combined)
   end
 
   def fix_tag(%{"tag" => %{} = tag} = object), do: Map.put(object, "tag", [tag])
@@ -326,8 +302,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     content_groups = Map.to_list(content_map)
     {_, content} = Enum.at(content_groups, 0)
 
-    object
+    Map.put(object, "content", content)
-    |> Map.put("content", content)
   end
 
   def fix_content_map(object), do: object
@@ -336,16 +311,11 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
   def fix_type(%{"inReplyTo" => reply_id, "name" => _} = object, options)
       when is_binary(reply_id) do
-    reply =
+    with true <- Federator.allowed_incoming_reply_depth?(options[:depth]),
-      with true <- Federator.allowed_incoming_reply_depth?(options[:depth]),
+         {:ok, %{data: %{"type" => "Question"} = _} = _} <- get_obj_helper(reply_id, options) do
-           {:ok, object} <- get_obj_helper(reply_id, options) do
-        object
-      end
-
-    if reply && reply.data["type"] == "Question" do
       Map.put(object, "type", "Answer")
     else
-      object
+      _ -> object
     end
   end
 
@@ -377,6 +347,17 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     end
   end
 
+  # Reduce the object list to find the reported user.
+  defp get_reported(objects) do
+    Enum.reduce_while(objects, nil, fn ap_id, _ ->
+      with %User{} = user <- User.get_cached_by_ap_id(ap_id) do
+        {:halt, user}
+      else
+        _ -> {:cont, nil}
+      end
+    end)
+  end
+
   def handle_incoming(data, options \\ [])
 
   # Flag objects are placed ahead of the ID check because Mastodon 2.8 and earlier send them
@@ -385,31 +366,19 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     with context <- data["context"] || Utils.generate_context_id(),
          content <- data["content"] || "",
          %User{} = actor <- User.get_cached_by_ap_id(actor),
-
          # Reduce the object list to find the reported user.
-         %User{} = account <-
+         %User{} = account <- get_reported(objects),
-           Enum.reduce_while(objects, nil, fn ap_id, _ ->
-             with %User{} = user <- User.get_cached_by_ap_id(ap_id) do
-               {:halt, user}
-             else
-               _ -> {:cont, nil}
-             end
-           end),
-
          # Remove the reported user from the object list.
          statuses <- Enum.filter(objects, fn ap_id -> ap_id != account.ap_id end) do
-      params = %{
+      %{
         actor: actor,
         context: context,
         account: account,
         statuses: statuses,
         content: content,
-        additional: %{
+        additional: %{"cc" => [account.ap_id]}
-          "cc" => [account.ap_id]
-        }
       }
-
+      |> ActivityPub.flag()
-      ActivityPub.flag(params)
     end
   end
 
@@ -461,6 +430,36 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     end
   end
 
+  def handle_incoming(
+        %{"type" => "Listen", "object" => %{"type" => "Audio"} = object} = data,
+        options
+      ) do
+    actor = Containment.get_actor(data)
+
+    data =
+      Map.put(data, "actor", actor)
+      |> fix_addressing
+
+    with {:ok, %User{} = user} <- User.get_or_fetch_by_ap_id(data["actor"]) do
+      options = Keyword.put(options, :depth, (options[:depth] || 0) + 1)
+      object = fix_object(object, options)
+
+      params = %{
+        to: data["to"],
+        object: object,
+        actor: user,
+        context: nil,
+        local: false,
+        published: data["published"],
+        additional: Map.take(data, ["cc", "id"])
+      }
+
+      ActivityPub.listen(params)
+    else
+      _e -> :error
+    end
+  end
+
   def handle_incoming(
         %{"type" => "Follow", "object" => followed, "actor" => follower, "id" => id} = data,
         _options
@@ -756,8 +755,12 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
   def handle_incoming(_, _), do: :error
 
+  @spec get_obj_helper(String.t(), Keyword.t()) :: {:ok, Object.t()} | nil
   def get_obj_helper(id, options \\ []) do
-    if object = Object.normalize(id, true, options), do: {:ok, object}, else: nil
+    case Object.normalize(id, true, options) do
+      %Object{} = object -> {:ok, object}
+      _ -> nil
+    end
   end
 
   def set_reply_to_uri(%{"inReplyTo" => in_reply_to} = object) when is_binary(in_reply_to) do
@@ -792,7 +795,8 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   #  internal -> Mastodon
   #  """
 
-  def prepare_outgoing(%{"type" => "Create", "object" => object_id} = data) do
+  def prepare_outgoing(%{"type" => activity_type, "object" => object_id} = data)
+      when activity_type in ["Create", "Listen"] do
     object =
       object_id
       |> Object.normalize()
@@ -856,27 +860,24 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     {:ok, data}
   end
 
-  def maybe_fix_object_url(data) do
+  def maybe_fix_object_url(%{"object" => object} = data) when is_binary(object) do
-    if is_binary(data["object"]) and not String.starts_with?(data["object"], "http") do
+    with false <- String.starts_with?(object, "http"),
-      case get_obj_helper(data["object"]) do
+         {:fetch, {:ok, relative_object}} <- {:fetch, get_obj_helper(object)},
-        {:ok, relative_object} ->
+         %{data: %{"external_url" => external_url}} when not is_nil(external_url) <-
-          if relative_object.data["external_url"] do
+           relative_object do
-            _data =
+      Map.put(data, "object", external_url)
-              data
-              |> Map.put("object", relative_object.data["external_url"])
-          else
-            data
-          end
-
-        e ->
-          Logger.error("Couldn't fetch #{data["object"]} #{inspect(e)}")
-          data
-      end
     else
-      data
+      {:fetch, e} ->
+        Logger.error("Couldn't fetch #{object} #{inspect(e)}")
+        data
+
+      _ ->
+        data
     end
   end
 
+  def maybe_fix_object_url(data), do: data
+
   def add_hashtags(object) do
     tags =
       (object["tag"] || [])
@@ -894,53 +895,49 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
           tag
       end)
 
-    object
+    Map.put(object, "tag", tags)
-    |> Map.put("tag", tags)
   end
 
   def add_mention_tags(object) do
     mentions =
       object
       |> Utils.get_notified_from_object()
-      |> Enum.map(fn user ->
+      |> Enum.map(&build_mention_tag/1)
-        %{"type" => "Mention", "href" => user.ap_id, "name" => "@#{user.nickname}"}
-      end)
 
     tags = object["tag"] || []
 
-    object
+    Map.put(object, "tag", tags ++ mentions)
-    |> Map.put("tag", tags ++ mentions)
   end
 
-  def add_emoji_tags(%User{info: %{"emoji" => _emoji} = user_info} = object) do
+  defp build_mention_tag(%{ap_id: ap_id, nickname: nickname} = _) do
-    user_info = add_emoji_tags(user_info)
+    %{"type" => "Mention", "href" => ap_id, "name" => "@#{nickname}"}
+  end
 
-    object
+  def take_emoji_tags(%User{info: %{emoji: emoji} = _user_info} = _user) do
-    |> Map.put(:info, user_info)
+    emoji
+    |> Enum.flat_map(&Map.to_list/1)
+    |> Enum.map(&build_emoji_tag/1)
   end
 
   # TODO: we should probably send mtime instead of unix epoch time for updated
   def add_emoji_tags(%{"emoji" => emoji} = object) do
     tags = object["tag"] || []
 
-    out =
+    out = Enum.map(emoji, &build_emoji_tag/1)
-      emoji
-      |> Enum.map(fn {name, url} ->
-        %{
-          "icon" => %{"url" => url, "type" => "Image"},
-          "name" => ":" <> name <> ":",
-          "type" => "Emoji",
-          "updated" => "1970-01-01T00:00:00Z",
-          "id" => url
-        }
-      end)
 
-    object
+    Map.put(object, "tag", tags ++ out)
-    |> Map.put("tag", tags ++ out)
   end
 
-  def add_emoji_tags(object) do
+  def add_emoji_tags(object), do: object
-    object
+
+  defp build_emoji_tag({name, url}) do
+    %{
+      "icon" => %{"url" => url, "type" => "Image"},
+      "name" => ":" <> name <> ":",
+      "type" => "Emoji",
+      "updated" => "1970-01-01T00:00:00Z",
+      "id" => url
+    }
   end
 
   def set_conversation(object) do
@@ -960,9 +957,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
   def add_attributed_to(object) do
     attributed_to = object["attributedTo"] || object["actor"]
-
+    Map.put(object, "attributedTo", attributed_to)
-    object
-    |> Map.put("attributedTo", attributed_to)
   end
 
   def prepare_attachments(object) do
@@ -973,30 +968,18 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
         %{"url" => href, "mediaType" => media_type, "name" => data["name"], "type" => "Document"}
       end)
 
-    object
+    Map.put(object, "attachment", attachments)
-    |> Map.put("attachment", attachments)
   end
 
   defp strip_internal_fields(object) do
     object
-    |> Map.drop([
+    |> Map.drop(Pleroma.Constants.object_internal_fields())
-      "likes",
-      "like_count",
-      "announcements",
-      "announcement_count",
-      "emoji",
-      "context_id",
-      "deleted_activity_id"
-    ])
   end
 
   defp strip_internal_tags(%{"tag" => tags} = object) do
-    tags =
+    tags = Enum.filter(tags, fn x -> is_map(x) end)
-      tags
-      |> Enum.filter(fn x -> is_map(x) end)
 
-    object
+    Map.put(object, "tag", tags)
-    |> Map.put("tag", tags)
   end
 
   defp strip_internal_tags(object), do: object
@@ -1050,8 +1033,8 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     with %User{local: false} = user <- User.get_cached_by_ap_id(ap_id),
          {:ok, data} <- ActivityPub.fetch_and_prepare_user_from_ap_id(ap_id),
          already_ap <- User.ap_enabled?(user),
-         {:ok, user} <- user |> User.upgrade_changeset(data, true) |> User.update_and_set_cache() do
+         {:ok, user} <- upgrade_user(user, data) do
-      unless already_ap do
+      if not already_ap do
         TransmogrifierWorker.enqueue("user_upgrade", %{"user_id" => user.id})
       end
 
@@ -1062,6 +1045,12 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     end
   end
 
+  defp upgrade_user(user, data) do
+    user
+    |> User.upgrade_changeset(data, true)
+    |> User.update_and_set_cache()
+  end
+
   def maybe_retire_websub(ap_id) do
     # some sanity checks
     if is_binary(ap_id) && String.length(ap_id) > 8 do
@@ -1075,16 +1064,11 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     end
   end
 
-  def maybe_fix_user_url(data) do
+  def maybe_fix_user_url(%{"url" => url} = data) when is_map(url) do
-    if is_map(data["url"]) do
+    Map.put(data, "url", url["href"])
-      Map.put(data, "url", data["url"]["href"])
-    else
-      data
-    end
   end
 
-  def maybe_fix_user_object(data) do
+  def maybe_fix_user_url(data), do: data
-    data
+
-    |> maybe_fix_user_url
+  def maybe_fix_user_object(data), do: maybe_fix_user_url(data)
-  end
 end

lib/pleroma/web/activity_pub/utils.ex

@@ -20,7 +20,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
   require Logger
   require Pleroma.Constants
 
-  @supported_object_types ["Article", "Note", "Video", "Page", "Question", "Answer"]
+  @supported_object_types ["Article", "Note", "Video", "Page", "Question", "Answer", "Audio"]
   @supported_report_states ~w(open closed resolved)
   @valid_visibilities ~w(public unlisted private direct)
 
@@ -581,6 +581,21 @@ defmodule Pleroma.Web.ActivityPub.Utils do
     |> Map.merge(additional)
   end
 
+  #### Listen-related helpers
+  def make_listen_data(params, additional) do
+    published = params.published || make_date()
+
+    %{
+      "type" => "Listen",
+      "to" => params.to |> Enum.uniq(),
+      "actor" => params.actor.ap_id,
+      "object" => params.object,
+      "published" => published,
+      "context" => params.context
+    }
+    |> Map.merge(additional)
+  end
+
   #### Flag-related helpers
   @spec make_flag_data(map(), map()) :: map()
   def make_flag_data(%{actor: actor, context: context, content: content} = params, additional) do

lib/pleroma/web/activity_pub/views/object_view.ex

@@ -15,7 +15,8 @@ defmodule Pleroma.Web.ActivityPub.ObjectView do
     Map.merge(base, additional)
   end
 
-  def render("object.json", %{object: %Activity{data: %{"type" => "Create"}} = activity}) do
+  def render("object.json", %{object: %Activity{data: %{"type" => activity_type}} = activity})
+      when activity_type in ["Create", "Listen"] do
     base = Pleroma.Web.ActivityPub.Utils.make_json_ld_header()
     object = Object.normalize(activity)
 
@@ -37,12 +38,12 @@ defmodule Pleroma.Web.ActivityPub.ObjectView do
     Map.merge(base, additional)
   end
 
-  def render("likes.json", ap_id, likes, page) do
+  def render("likes.json", %{ap_id: ap_id, likes: likes, page: page}) do
     collection(likes, "#{ap_id}/likes", page)
     |> Map.merge(Pleroma.Web.ActivityPub.Utils.make_json_ld_header())
   end
 
-  def render("likes.json", ap_id, likes) do
+  def render("likes.json", %{ap_id: ap_id, likes: likes}) do
     %{
       "id" => "#{ap_id}/likes",
       "type" => "OrderedCollection",

lib/pleroma/web/activity_pub/views/user_view.ex

@@ -8,7 +8,6 @@ defmodule Pleroma.Web.ActivityPub.UserView do
   alias Pleroma.Keys
   alias Pleroma.Repo
   alias Pleroma.User
-  alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.ActivityPub.Transmogrifier
   alias Pleroma.Web.ActivityPub.Utils
   alias Pleroma.Web.Endpoint
@@ -25,7 +24,8 @@ defmodule Pleroma.Web.ActivityPub.UserView do
       "oauthAuthorizationEndpoint" => Helpers.o_auth_url(Endpoint, :authorize),
       "oauthRegistrationEndpoint" => Helpers.mastodon_api_url(Endpoint, :create_app),
       "oauthTokenEndpoint" => Helpers.o_auth_url(Endpoint, :token_exchange),
-      "sharedInbox" => Helpers.activity_pub_url(Endpoint, :inbox)
+      "sharedInbox" => Helpers.activity_pub_url(Endpoint, :inbox),
+      "uploadMedia" => Helpers.activity_pub_url(Endpoint, :upload_media)
     }
   end
 
@@ -75,10 +75,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
 
     endpoints = render("endpoints.json", %{user: user})
 
-    user_tags =
+    emoji_tags = Transmogrifier.take_emoji_tags(user)
-      user
-      |> Transmogrifier.add_emoji_tags()
-      |> Map.get("tag", [])
 
     fields =
       user.info
@@ -110,7 +107,8 @@ defmodule Pleroma.Web.ActivityPub.UserView do
       },
       "endpoints" => endpoints,
       "attachment" => fields,
-      "tag" => (user.info.source_data["tag"] || []) ++ user_tags
+      "tag" => (user.info.source_data["tag"] || []) ++ emoji_tags,
+      "discoverable" => user.info.discoverable
     }
     |> Map.merge(maybe_make_image(&User.avatar_url/2, "icon", user))
     |> Map.merge(maybe_make_image(&User.banner_url/2, "image", user))
@@ -118,30 +116,34 @@ defmodule Pleroma.Web.ActivityPub.UserView do
   end
 
   def render("following.json", %{user: user, page: page} = opts) do
-    showing = (opts[:for] && opts[:for] == user) || !user.info.hide_follows
+    showing_items = (opts[:for] && opts[:for] == user) || !user.info.hide_follows
+    showing_count = showing_items || !user.info.hide_follows_count
+
     query = User.get_friends_query(user)
     query = from(user in query, select: [:ap_id])
     following = Repo.all(query)
 
     total =
-      if showing do
+      if showing_count do
         length(following)
       else
         0
       end
 
-    collection(following, "#{user.ap_id}/following", page, showing, total)
+    collection(following, "#{user.ap_id}/following", page, showing_items, total)
     |> Map.merge(Utils.make_json_ld_header())
   end
 
   def render("following.json", %{user: user} = opts) do
-    showing = (opts[:for] && opts[:for] == user) || !user.info.hide_follows
+    showing_items = (opts[:for] && opts[:for] == user) || !user.info.hide_follows
+    showing_count = showing_items || !user.info.hide_follows_count
+
     query = User.get_friends_query(user)
     query = from(user in query, select: [:ap_id])
     following = Repo.all(query)
 
     total =
-      if showing do
+      if showing_count do
         length(following)
       else
         0
@@ -152,7 +154,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
       "type" => "OrderedCollection",
       "totalItems" => total,
       "first" =>
-        if showing do
+        if showing_items do
           collection(following, "#{user.ap_id}/following", 1, !user.info.hide_follows)
         else
           "#{user.ap_id}/following?page=1"
@@ -162,32 +164,34 @@ defmodule Pleroma.Web.ActivityPub.UserView do
   end
 
   def render("followers.json", %{user: user, page: page} = opts) do
-    showing = (opts[:for] && opts[:for] == user) || !user.info.hide_followers
+    showing_items = (opts[:for] && opts[:for] == user) || !user.info.hide_followers
+    showing_count = showing_items || !user.info.hide_followers_count
 
     query = User.get_followers_query(user)
     query = from(user in query, select: [:ap_id])
     followers = Repo.all(query)
 
     total =
-      if showing do
+      if showing_count do
         length(followers)
       else
         0
       end
 
-    collection(followers, "#{user.ap_id}/followers", page, showing, total)
+    collection(followers, "#{user.ap_id}/followers", page, showing_items, total)
     |> Map.merge(Utils.make_json_ld_header())
   end
 
   def render("followers.json", %{user: user} = opts) do
-    showing = (opts[:for] && opts[:for] == user) || !user.info.hide_followers
+    showing_items = (opts[:for] && opts[:for] == user) || !user.info.hide_followers
+    showing_count = showing_items || !user.info.hide_followers_count
 
     query = User.get_followers_query(user)
     query = from(user in query, select: [:ap_id])
     followers = Repo.all(query)
 
     total =
-      if showing do
+      if showing_count do
         length(followers)
       else
         0
@@ -198,8 +202,8 @@ defmodule Pleroma.Web.ActivityPub.UserView do
       "type" => "OrderedCollection",
       "totalItems" => total,
       "first" =>
-        if showing do
+        if showing_items do
-          collection(followers, "#{user.ap_id}/followers", 1, showing, total)
+          collection(followers, "#{user.ap_id}/followers", 1, showing_items, total)
         else
           "#{user.ap_id}/followers?page=1"
         end
@@ -207,25 +211,22 @@ defmodule Pleroma.Web.ActivityPub.UserView do
     |> Map.merge(Utils.make_json_ld_header())
   end
 
-  def render("outbox.json", %{user: user, max_id: max_qid}) do
+  def render("activity_collection.json", %{iri: iri}) do
-    params = %{
+    %{
-      "limit" => "10"
+      "id" => iri,
+      "type" => "OrderedCollection",
+      "first" => "#{iri}?page=true"
     }
+    |> Map.merge(Utils.make_json_ld_header())
+  end
 
-    params =
+  def render("activity_collection_page.json", %{activities: activities, iri: iri}) do
-      if max_qid != nil do
+    # this is sorted chronologically, so first activity is the newest (max)
-        Map.put(params, "max_id", max_qid)
-      else
-        params
-      end
-
-    activities = ActivityPub.fetch_user_activities(user, nil, params)
-
     {max_id, min_id, collection} =
       if length(activities) > 0 do
         {
-          Enum.at(Enum.reverse(activities), 0).id,
           Enum.at(activities, 0).id,
+          Enum.at(Enum.reverse(activities), 0).id,
           Enum.map(activities, fn act ->
             {:ok, data} = Transmogrifier.prepare_outgoing(act.data)
             data
@@ -239,71 +240,14 @@ defmodule Pleroma.Web.ActivityPub.UserView do
         }
       end
 
-    iri = "#{user.ap_id}/outbox"
+    %{
-
+      "id" => "#{iri}?max_id=#{max_id}&page=true",
-    page = %{
-      "id" => "#{iri}?max_id=#{max_id}",
       "type" => "OrderedCollectionPage",
       "partOf" => iri,
       "orderedItems" => collection,
-      "next" => "#{iri}?max_id=#{min_id}"
+      "next" => "#{iri}?max_id=#{min_id}&page=true"
     }
-
+    |> Map.merge(Utils.make_json_ld_header())
-    if max_qid == nil do
-      %{
-        "id" => iri,
-        "type" => "OrderedCollection",
-        "first" => page
-      }
-      |> Map.merge(Utils.make_json_ld_header())
-    else
-      page |> Map.merge(Utils.make_json_ld_header())
-    end
-  end
-
-  def render("inbox.json", %{user: user, max_id: max_qid}) do
-    params = %{
-      "limit" => "10"
-    }
-
-    params =
-      if max_qid != nil do
-        Map.put(params, "max_id", max_qid)
-      else
-        params
-      end
-
-    activities = ActivityPub.fetch_activities([user.ap_id | user.following], params)
-
-    min_id = Enum.at(Enum.reverse(activities), 0).id
-    max_id = Enum.at(activities, 0).id
-
-    collection =
-      Enum.map(activities, fn act ->
-        {:ok, data} = Transmogrifier.prepare_outgoing(act.data)
-        data
-      end)
-
-    iri = "#{user.ap_id}/inbox"
-
-    page = %{
-      "id" => "#{iri}?max_id=#{max_id}",
-      "type" => "OrderedCollectionPage",
-      "partOf" => iri,
-      "orderedItems" => collection,
-      "next" => "#{iri}?max_id=#{min_id}"
-    }
-
-    if max_qid == nil do
-      %{
-        "id" => iri,
-        "type" => "OrderedCollection",
-        "first" => page
-      }
-      |> Map.merge(Utils.make_json_ld_header())
-    else
-      page |> Map.merge(Utils.make_json_ld_header())
-    end
   end
 
   def collection(collection, iri, page, show_items \\ true, total \\ nil) do

lib/pleroma/web/admin_api/admin_api_controller.ex

@@ -14,10 +14,13 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
   alias Pleroma.Web.AdminAPI.Config
   alias Pleroma.Web.AdminAPI.ConfigView
   alias Pleroma.Web.AdminAPI.ModerationLogView
+  alias Pleroma.Web.AdminAPI.Report
   alias Pleroma.Web.AdminAPI.ReportView
   alias Pleroma.Web.AdminAPI.Search
   alias Pleroma.Web.CommonAPI
+  alias Pleroma.Web.Endpoint
   alias Pleroma.Web.MastodonAPI.StatusView
+  alias Pleroma.Web.Router
 
   import Pleroma.Web.ControllerHelper, only: [json_response: 3]
 
@@ -139,7 +142,8 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
   def user_show(conn, %{"nickname" => nickname}) do
     with %User{} = user <- User.get_cached_by_nickname_or_id(nickname) do
       conn
-      |> json(AccountView.render("show.json", %{user: user}))
+      |> put_view(AccountView)
+      |> render("show.json", %{user: user})
     else
       _ -> {:error, :not_found}
     end
@@ -158,7 +162,8 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
         })
 
       conn
-      |> json(StatusView.render("index.json", %{activities: activities, as: :activity}))
+      |> put_view(StatusView)
+      |> render("index.json", %{activities: activities, as: :activity})
     else
       _ -> {:error, :not_found}
     end
@@ -178,7 +183,8 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     })
 
     conn
-    |> json(AccountView.render("show.json", %{user: updated_user}))
+    |> put_view(AccountView)
+    |> render("show.json", %{user: updated_user})
   end
 
   def tag_users(%{assigns: %{user: admin}} = conn, %{"nicknames" => nicknames, "tags" => tags}) do
@@ -250,18 +256,12 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
         "nickname" => nickname
       })
       when permission_group in ["moderator", "admin"] do
-    user = User.get_cached_by_nickname(nickname)
+    info = Map.put(%{}, "is_" <> permission_group, true)
 
-    info =
+    {:ok, user} =
-      %{}
+      nickname
-      |> Map.put("is_" <> permission_group, true)
+      |> User.get_cached_by_nickname()
-
+      |> User.update_info(&User.Info.admin_api_update(&1, info))
-    info_cng = User.Info.admin_api_update(user.info, info)
-
-    cng =
-      user
-      |> Ecto.Changeset.change()
-      |> Ecto.Changeset.put_embed(:info, info_cng)
 
     ModerationLog.insert_log(%{
       action: "grant",
@@ -270,8 +270,6 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
       permission: permission_group
     })
 
-    {:ok, _user} = User.update_and_set_cache(cng)
-
     json(conn, info)
   end
 
@@ -289,40 +287,33 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     })
   end
 
+  def right_delete(%{assigns: %{user: %{nickname: nickname}}} = conn, %{"nickname" => nickname}) do
+    render_error(conn, :forbidden, "You can't revoke your own admin status.")
+  end
+
   def right_delete(
-        %{assigns: %{user: %User{:nickname => admin_nickname} = admin}} = conn,
+        %{assigns: %{user: admin}} = conn,
         %{
           "permission_group" => permission_group,
           "nickname" => nickname
         }
       )
       when permission_group in ["moderator", "admin"] do
-    if admin_nickname == nickname do
+    info = Map.put(%{}, "is_" <> permission_group, false)
-      render_error(conn, :forbidden, "You can't revoke your own admin status.")
-    else
-      user = User.get_cached_by_nickname(nickname)
 
-      info =
+    {:ok, user} =
-        %{}
+      nickname
-        |> Map.put("is_" <> permission_group, false)
+      |> User.get_cached_by_nickname()
+      |> User.update_info(&User.Info.admin_api_update(&1, info))
 
-      info_cng = User.Info.admin_api_update(user.info, info)
+    ModerationLog.insert_log(%{
+      action: "revoke",
+      actor: admin,
+      subject: user,
+      permission: permission_group
+    })
 
-      cng =
+    json(conn, info)
-        Ecto.Changeset.change(user)
-        |> Ecto.Changeset.put_embed(:info, info_cng)
-
-      {:ok, _user} = User.update_and_set_cache(cng)
-
-      ModerationLog.insert_log(%{
-        action: "revoke",
-        actor: admin,
-        subject: user,
-        permission: permission_group
-      })
-
-      json(conn, info)
-    end
   end
 
   def right_delete(conn, _) do
@@ -400,13 +391,23 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     end
   end
 
-  @doc "Get a account registeration invite token (base64 string)"
+  @doc "Create an account registration invite token"
-  def get_invite_token(conn, params) do
+  def create_invite_token(conn, params) do
-    options = params["invite"] || %{}
+    opts = %{}
-    {:ok, invite} = UserInviteToken.create_invite(options)
 
-    conn
+    opts =
-    |> json(invite.token)
+      if params["max_use"],
+        do: Map.put(opts, :max_use, params["max_use"]),
+        else: opts
+
+    opts =
+      if params["expires_at"],
+        do: Map.put(opts, :expires_at, params["expires_at"]),
+        else: opts
+
+    {:ok, invite} = UserInviteToken.create_invite(opts)
+
+    json(conn, AccountView.render("invite.json", %{invite: invite}))
   end
 
   @doc "Get list of created invites"
@@ -414,7 +415,8 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     invites = UserInviteToken.list_invites()
 
     conn
-    |> json(AccountView.render("invites.json", %{invites: invites}))
+    |> put_view(AccountView)
+    |> render("invites.json", %{invites: invites})
   end
 
   @doc "Revokes invite by token"
@@ -422,7 +424,8 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     with {:ok, invite} <- UserInviteToken.find_by_token(token),
          {:ok, updated_invite} = UserInviteToken.update_invite(invite, %{used: true}) do
       conn
-      |> json(AccountView.render("invite.json", %{invite: updated_invite}))
+      |> put_view(AccountView)
+      |> render("invite.json", %{invite: updated_invite})
     else
       nil -> {:error, :not_found}
     end
@@ -434,17 +437,33 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     {:ok, token} = Pleroma.PasswordResetToken.create_token(user)
 
     conn
-    |> json(token.token)
+    |> json(%{
+      token: token.token,
+      link: Router.Helpers.reset_password_url(Endpoint, :reset, token.token)
+    })
+  end
+
+  @doc "Force password reset for a given user"
+  def force_password_reset(conn, %{"nickname" => nickname}) do
+    (%User{local: true} = user) = User.get_cached_by_nickname(nickname)
+
+    User.force_password_reset_async(user)
+
+    json_response(conn, :no_content, "")
   end
 
   def list_reports(conn, params) do
+    {page, page_size} = page_params(params)
+
     params =
       params
       |> Map.put("type", "Flag")
       |> Map.put("skip_preload", true)
       |> Map.put("total", true)
+      |> Map.put("limit", page_size)
+      |> Map.put("offset", (page - 1) * page_size)
 
-    reports = ActivityPub.fetch_activities([], params)
+    reports = ActivityPub.fetch_activities([], params, :offset)
 
     conn
     |> put_view(ReportView)
@@ -455,7 +474,7 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     with %Activity{} = report <- Activity.get_by_id(id) do
       conn
       |> put_view(ReportView)
-      |> render("show.json", %{report: report})
+      |> render("show.json", Report.extract_report_info(report))
     else
       _ -> {:error, :not_found}
     end
@@ -471,7 +490,7 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
 
       conn
       |> put_view(ReportView)
-      |> render("show.json", %{report: report})
+      |> render("show.json", Report.extract_report_info(report))
     end
   end
 
@@ -494,7 +513,7 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
 
       conn
       |> put_view(StatusView)
-      |> render("status.json", %{activity: activity})
+      |> render("show.json", %{activity: activity})
     else
       true ->
         {:param_cast, nil}
@@ -518,7 +537,7 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
 
       conn
       |> put_view(StatusView)
-      |> render("status.json", %{activity: activity})
+      |> render("show.json", %{activity: activity})
     end
   end
 
@@ -537,7 +556,15 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
   def list_log(conn, params) do
     {page, page_size} = page_params(params)
 
-    log = ModerationLog.get_all(page, page_size)
+    log =
+      ModerationLog.get_all(%{
+        page: page,
+        page_size: page_size,
+        start_date: params["start_date"],
+        end_date: params["end_date"],
+        user_id: params["user_id"],
+        search: params["search"]
+      })
 
     conn
     |> put_view(ModerationLogView)
@@ -589,6 +616,12 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     |> render("index.json", %{configs: updated})
   end
 
+  def reload_emoji(conn, _params) do
+    Pleroma.Emoji.reload()
+
+    conn |> json("ok")
+  end
+
   def errors(conn, {:error, :not_found}) do
     conn
     |> put_status(:not_found)

lib/pleroma/web/admin_api/report.ex

@@ -0,0 +1,22 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.AdminAPI.Report do
+  alias Pleroma.Activity
+  alias Pleroma.User
+
+  def extract_report_info(
+        %{data: %{"actor" => actor, "object" => [account_ap_id | status_ap_ids]}} = report
+      ) do
+    user = User.get_cached_by_ap_id(actor)
+    account = User.get_cached_by_ap_id(account_ap_id)
+
+    statuses =
+      Enum.map(status_ap_ids, fn ap_id ->
+        Activity.get_by_ap_id_with_object(ap_id)
+      end)
+
+    %{report: report, user: user, account: account, statuses: statuses}
+  end
+end

lib/pleroma/web/admin_api/views/moderation_log_view.ex

@@ -8,7 +8,10 @@ defmodule Pleroma.Web.AdminAPI.ModerationLogView do
   alias Pleroma.ModerationLog
 
   def render("index.json", %{log: log}) do
-    render_many(log, __MODULE__, "show.json", as: :log_entry)
+    %{
+      items: render_many(log.items, __MODULE__, "show.json", as: :log_entry),
+      total: log.count
+    }
   end
 
   def render("show.json", %{log_entry: log_entry}) do

lib/pleroma/web/admin_api/views/report_view.ex

@@ -4,27 +4,26 @@
 
 defmodule Pleroma.Web.AdminAPI.ReportView do
   use Pleroma.Web, :view
-  alias Pleroma.Activity
   alias Pleroma.HTML
   alias Pleroma.User
+  alias Pleroma.Web.AdminAPI.Report
   alias Pleroma.Web.CommonAPI.Utils
   alias Pleroma.Web.MastodonAPI.StatusView
 
   def render("index.json", %{reports: reports}) do
     %{
       reports:
-        render_many(reports[:items], __MODULE__, "show.json", as: :report) |> Enum.reverse(),
+        reports[:items]
+        |> Enum.map(&Report.extract_report_info(&1))
+        |> Enum.map(&render(__MODULE__, "show.json", &1))
+        |> Enum.reverse(),
       total: reports[:total]
     }
   end
 
-  def render("show.json", %{report: report}) do
+  def render("show.json", %{report: report, user: user, account: account, statuses: statuses}) do
-    user = User.get_cached_by_ap_id(report.data["actor"])
     created_at = Utils.to_masto_date(report.data["published"])
 
-    [account_ap_id | status_ap_ids] = report.data["object"]
-    account = User.get_cached_by_ap_id(account_ap_id)
-
     content =
       unless is_nil(report.data["content"]) do
         HTML.filter_tags(report.data["content"])
@@ -32,11 +31,6 @@ defmodule Pleroma.Web.AdminAPI.ReportView do
         nil
       end
 
-    statuses =
-      Enum.map(status_ap_ids, fn ap_id ->
-        Activity.get_by_ap_id_with_object(ap_id)
-      end)
-
     %{
       id: report.id,
       account: merge_account_views(account),
@@ -49,7 +43,7 @@ defmodule Pleroma.Web.AdminAPI.ReportView do
   end
 
   defp merge_account_views(%User{} = user) do
-    Pleroma.Web.MastodonAPI.AccountView.render("account.json", %{user: user})
+    Pleroma.Web.MastodonAPI.AccountView.render("show.json", %{user: user})
     |> Map.merge(Pleroma.Web.AdminAPI.AccountView.render("show.json", %{user: user}))
   end
 

lib/pleroma/web/chat_channel.ex

@@ -22,7 +22,7 @@ defmodule Pleroma.Web.ChatChannel do
 
     if String.length(text) > 0 do
       author = User.get_cached_by_nickname(user_name)
-      author = Pleroma.Web.MastodonAPI.AccountView.render("account.json", user: author)
+      author = Pleroma.Web.MastodonAPI.AccountView.render("show.json", user: author)
       message = ChatChannelState.add_message(%{text: text, author: author})
 
       broadcast!(socket, "new_msg", message)

lib/pleroma/web/common_api/activity_draft.ex

@@ -0,0 +1,219 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.CommonAPI.ActivityDraft do
+  alias Pleroma.Activity
+  alias Pleroma.Conversation.Participation
+  alias Pleroma.Web.CommonAPI
+  alias Pleroma.Web.CommonAPI.Utils
+
+  import Pleroma.Web.Gettext
+
+  defstruct valid?: true,
+            errors: [],
+            user: nil,
+            params: %{},
+            status: nil,
+            summary: nil,
+            full_payload: nil,
+            attachments: [],
+            in_reply_to: nil,
+            in_reply_to_conversation: nil,
+            visibility: nil,
+            expires_at: nil,
+            poll: nil,
+            emoji: %{},
+            content_html: nil,
+            mentions: [],
+            tags: [],
+            to: [],
+            cc: [],
+            context: nil,
+            sensitive: false,
+            object: nil,
+            preview?: false,
+            changes: %{}
+
+  def create(user, params) do
+    %__MODULE__{user: user}
+    |> put_params(params)
+    |> status()
+    |> summary()
+    |> with_valid(&attachments/1)
+    |> full_payload()
+    |> expires_at()
+    |> poll()
+    |> with_valid(&in_reply_to/1)
+    |> with_valid(&in_reply_to_conversation/1)
+    |> with_valid(&visibility/1)
+    |> content()
+    |> with_valid(&to_and_cc/1)
+    |> with_valid(&context/1)
+    |> sensitive()
+    |> with_valid(&object/1)
+    |> preview?()
+    |> with_valid(&changes/1)
+    |> validate()
+  end
+
+  defp put_params(draft, params) do
+    params = Map.put_new(params, "in_reply_to_status_id", params["in_reply_to_id"])
+    %__MODULE__{draft | params: params}
+  end
+
+  defp status(%{params: %{"status" => status}} = draft) do
+    %__MODULE__{draft | status: String.trim(status)}
+  end
+
+  defp summary(%{params: params} = draft) do
+    %__MODULE__{draft | summary: Map.get(params, "spoiler_text", "")}
+  end
+
+  defp full_payload(%{status: status, summary: summary} = draft) do
+    full_payload = String.trim(status <> summary)
+
+    case Utils.validate_character_limit(full_payload, draft.attachments) do
+      :ok -> %__MODULE__{draft | full_payload: full_payload}
+      {:error, message} -> add_error(draft, message)
+    end
+  end
+
+  defp attachments(%{params: params} = draft) do
+    attachments = Utils.attachments_from_ids(params)
+    %__MODULE__{draft | attachments: attachments}
+  end
+
+  defp in_reply_to(draft) do
+    case Map.get(draft.params, "in_reply_to_status_id") do
+      "" -> draft
+      nil -> draft
+      id -> %__MODULE__{draft | in_reply_to: Activity.get_by_id(id)}
+    end
+  end
+
+  defp in_reply_to_conversation(draft) do
+    in_reply_to_conversation = Participation.get(draft.params["in_reply_to_conversation_id"])
+    %__MODULE__{draft | in_reply_to_conversation: in_reply_to_conversation}
+  end
+
+  defp visibility(%{params: params} = draft) do
+    case CommonAPI.get_visibility(params, draft.in_reply_to, draft.in_reply_to_conversation) do
+      {visibility, "direct"} when visibility != "direct" ->
+        add_error(draft, dgettext("errors", "The message visibility must be direct"))
+
+      {visibility, _} ->
+        %__MODULE__{draft | visibility: visibility}
+    end
+  end
+
+  defp expires_at(draft) do
+    case CommonAPI.check_expiry_date(draft.params["expires_in"]) do
+      {:ok, expires_at} -> %__MODULE__{draft | expires_at: expires_at}
+      {:error, message} -> add_error(draft, message)
+    end
+  end
+
+  defp poll(draft) do
+    case Utils.make_poll_data(draft.params) do
+      {:ok, {poll, poll_emoji}} ->
+        %__MODULE__{draft | poll: poll, emoji: Map.merge(draft.emoji, poll_emoji)}
+
+      {:error, message} ->
+        add_error(draft, message)
+    end
+  end
+
+  defp content(draft) do
+    {content_html, mentions, tags} =
+      Utils.make_content_html(
+        draft.status,
+        draft.attachments,
+        draft.params,
+        draft.visibility
+      )
+
+    %__MODULE__{draft | content_html: content_html, mentions: mentions, tags: tags}
+  end
+
+  defp to_and_cc(draft) do
+    addressed_users =
+      draft.mentions
+      |> Enum.map(fn {_, mentioned_user} -> mentioned_user.ap_id end)
+      |> Utils.get_addressed_users(draft.params["to"])
+
+    {to, cc} =
+      Utils.get_to_and_cc(
+        draft.user,
+        addressed_users,
+        draft.in_reply_to,
+        draft.visibility,
+        draft.in_reply_to_conversation
+      )
+
+    %__MODULE__{draft | to: to, cc: cc}
+  end
+
+  defp context(draft) do
+    context = Utils.make_context(draft.in_reply_to, draft.in_reply_to_conversation)
+    %__MODULE__{draft | context: context}
+  end
+
+  defp sensitive(draft) do
+    sensitive = draft.params["sensitive"] || Enum.member?(draft.tags, {"#nsfw", "nsfw"})
+    %__MODULE__{draft | sensitive: sensitive}
+  end
+
+  defp object(draft) do
+    emoji = Map.merge(Pleroma.Emoji.Formatter.get_emoji_map(draft.full_payload), draft.emoji)
+
+    object =
+      Utils.make_note_data(
+        draft.user.ap_id,
+        draft.to,
+        draft.context,
+        draft.content_html,
+        draft.attachments,
+        draft.in_reply_to,
+        draft.tags,
+        draft.summary,
+        draft.cc,
+        draft.sensitive,
+        draft.poll
+      )
+      |> Map.put("emoji", emoji)
+
+    %__MODULE__{draft | object: object}
+  end
+
+  defp preview?(draft) do
+    preview? = Pleroma.Web.ControllerHelper.truthy_param?(draft.params["preview"]) || false
+    %__MODULE__{draft | preview?: preview?}
+  end
+
+  defp changes(draft) do
+    direct? = draft.visibility == "direct"
+
+    changes =
+      %{
+        to: draft.to,
+        actor: draft.user,
+        context: draft.context,
+        object: draft.object,
+        additional: %{"cc" => draft.cc, "directMessage" => direct?}
+      }
+      |> Utils.maybe_add_list_data(draft.user, draft.visibility)
+
+    %__MODULE__{draft | changes: changes}
+  end
+
+  defp with_valid(%{valid?: true} = draft, func), do: func.(draft)
+  defp with_valid(draft, _func), do: draft
+
+  defp add_error(draft, message) do
+    %__MODULE__{draft | valid?: false, errors: [message | draft.errors]}
+  end
+
+  defp validate(%{valid?: true} = draft), do: {:ok, draft}
+  defp validate(%{errors: [message | _]}), do: {:error, message}
+end

lib/pleroma/web/common_api/common_api.ex

@@ -6,7 +6,6 @@ defmodule Pleroma.Web.CommonAPI do
   alias Pleroma.Activity
   alias Pleroma.ActivityExpiration
   alias Pleroma.Conversation.Participation
-  alias Pleroma.Formatter
   alias Pleroma.Object
   alias Pleroma.ThreadMute
   alias Pleroma.User
@@ -18,14 +17,11 @@ defmodule Pleroma.Web.CommonAPI do
   import Pleroma.Web.CommonAPI.Utils
 
   def follow(follower, followed) do
+    timeout = Pleroma.Config.get([:activitypub, :follow_handshake_timeout])
+
     with {:ok, follower} <- User.maybe_direct_follow(follower, followed),
          {:ok, activity} <- ActivityPub.follow(follower, followed),
-         {:ok, follower, followed} <-
+         {:ok, follower, followed} <- User.wait_and_refresh(timeout, follower, followed) do
-           User.wait_and_refresh(
-             Pleroma.Config.get([:activitypub, :follow_handshake_timeout]),
-             follower,
-             followed
-           ) do
       {:ok, follower, followed, activity}
     end
   end
@@ -76,8 +72,7 @@ defmodule Pleroma.Web.CommonAPI do
          {:ok, delete} <- ActivityPub.delete(object) do
       {:ok, delete}
     else
-      _ ->
+      _ -> {:error, dgettext("errors", "Could not delete")}
-        {:error, dgettext("errors", "Could not delete")}
     end
   end
 
@@ -87,18 +82,16 @@ defmodule Pleroma.Web.CommonAPI do
          nil <- Utils.get_existing_announce(user.ap_id, object) do
       ActivityPub.announce(user, object)
     else
-      _ ->
+      _ -> {:error, dgettext("errors", "Could not repeat")}
-        {:error, dgettext("errors", "Could not repeat")}
     end
   end
 
   def unrepeat(id_or_ap_id, user) do
-    with %Activity{} = activity <- get_by_id_or_ap_id(id_or_ap_id),
+    with %Activity{} = activity <- get_by_id_or_ap_id(id_or_ap_id) do
-         object <- Object.normalize(activity) do
+      object = Object.normalize(activity)
       ActivityPub.unannounce(user, object)
     else
-      _ ->
+      _ -> {:error, dgettext("errors", "Could not unrepeat")}
-        {:error, dgettext("errors", "Could not unrepeat")}
     end
   end
 
@@ -108,30 +101,23 @@ defmodule Pleroma.Web.CommonAPI do
          nil <- Utils.get_existing_like(user.ap_id, object) do
       ActivityPub.like(user, object)
     else
-      _ ->
+      _ -> {:error, dgettext("errors", "Could not favorite")}
-        {:error, dgettext("errors", "Could not favorite")}
     end
   end
 
   def unfavorite(id_or_ap_id, user) do
-    with %Activity{} = activity <- get_by_id_or_ap_id(id_or_ap_id),
+    with %Activity{} = activity <- get_by_id_or_ap_id(id_or_ap_id) do
-         object <- Object.normalize(activity) do
+      object = Object.normalize(activity)
       ActivityPub.unlike(user, object)
     else
-      _ ->
+      _ -> {:error, dgettext("errors", "Could not unfavorite")}
-        {:error, dgettext("errors", "Could not unfavorite")}
     end
   end
 
-  def vote(user, object, choices) do
+  def vote(user, %{data: %{"type" => "Question"}} = object, choices) do
-    with "Question" <- object.data["type"],
+    with :ok <- validate_not_author(object, user),
-         {:author, false} <- {:author, object.data["actor"] == user.ap_id},
+         :ok <- validate_existing_votes(user, object),
-         {:existing_votes, []} <- {:existing_votes, Utils.get_existing_votes(user.ap_id, object)},
+         {:ok, options, choices} <- normalize_and_validate_choices(choices, object) do
-         {options, max_count} <- get_options_and_max_count(object),
-         option_count <- Enum.count(options),
-         {:choice_check, {choices, true}} <-
-           {:choice_check, normalize_and_validate_choice_indices(choices, option_count)},
-         {:count_check, true} <- {:count_check, Enum.count(choices) <= max_count} do
       answer_activities =
         Enum.map(choices, fn index ->
           answer_data = make_answer_data(user, object, Enum.at(options, index)["name"])
@@ -150,32 +136,40 @@ defmodule Pleroma.Web.CommonAPI do
 
       object = Object.get_cached_by_ap_id(object.data["id"])
       {:ok, answer_activities, object}
-    else
-      {:author, _} -> {:error, dgettext("errors", "Poll's author can't vote")}
-      {:existing_votes, _} -> {:error, dgettext("errors", "Already voted")}
-      {:choice_check, {_, false}} -> {:error, dgettext("errors", "Invalid indices")}
-      {:count_check, false} -> {:error, dgettext("errors", "Too many choices")}
     end
   end
 
-  defp get_options_and_max_count(object) do
+  defp validate_not_author(%{data: %{"actor" => ap_id}}, %{ap_id: ap_id}),
-    if Map.has_key?(object.data, "anyOf") do
+    do: {:error, dgettext("errors", "Poll's author can't vote")}
-      {object.data["anyOf"], Enum.count(object.data["anyOf"])}
+
+  defp validate_not_author(_, _), do: :ok
+
+  defp validate_existing_votes(%{ap_id: ap_id}, object) do
+    if Utils.get_existing_votes(ap_id, object) == [] do
+      :ok
     else
-      {object.data["oneOf"], 1}
+      {:error, dgettext("errors", "Already voted")}
     end
   end
 
-  defp normalize_and_validate_choice_indices(choices, count) do
+  defp get_options_and_max_count(%{data: %{"anyOf" => any_of}}), do: {any_of, Enum.count(any_of)}
-    Enum.map_reduce(choices, true, fn index, valid ->
+  defp get_options_and_max_count(%{data: %{"oneOf" => one_of}}), do: {one_of, 1}
-      index = if is_binary(index), do: String.to_integer(index), else: index
+
-      {index, if(valid, do: index < count, else: valid)}
+  defp normalize_and_validate_choices(choices, object) do
-    end)
+    choices = Enum.map(choices, fn i -> if is_binary(i), do: String.to_integer(i), else: i end)
+    {options, max_count} = get_options_and_max_count(object)
+    count = Enum.count(options)
+
+    with {_, true} <- {:valid_choice, Enum.all?(choices, &(&1 < count))},
+         {_, true} <- {:count_check, Enum.count(choices) <= max_count} do
+      {:ok, options, choices}
+    else
+      {:valid_choice, _} -> {:error, dgettext("errors", "Invalid indices")}
+      {:count_check, _} -> {:error, dgettext("errors", "Too many choices")}
+    end
   end
 
-  def get_visibility(_, _, %Participation{}) do
+  def get_visibility(_, _, %Participation{}), do: {"direct", "direct"}
-    {"direct", "direct"}
-  end
 
   def get_visibility(%{"visibility" => visibility}, in_reply_to, _)
       when visibility in ~w{public unlisted private direct},
@@ -197,13 +191,13 @@ defmodule Pleroma.Web.CommonAPI do
 
   def get_replied_to_visibility(activity) do
     with %Object{} = object <- Object.normalize(activity) do
-      Pleroma.Web.ActivityPub.Visibility.get_visibility(object)
+      Visibility.get_visibility(object)
     end
   end
 
-  defp check_expiry_date({:ok, nil} = res), do: res
+  def check_expiry_date({:ok, nil} = res), do: res
 
-  defp check_expiry_date({:ok, in_seconds}) do
+  def check_expiry_date({:ok, in_seconds}) do
     expiry = NaiveDateTime.utc_now() |> NaiveDateTime.add(in_seconds)
 
     if ActivityExpiration.expires_late_enough?(expiry) do
@@ -213,105 +207,57 @@ defmodule Pleroma.Web.CommonAPI do
     end
   end
 
-  defp check_expiry_date(expiry_str) do
+  def check_expiry_date(expiry_str) do
     Ecto.Type.cast(:integer, expiry_str)
     |> check_expiry_date()
   end
 
-  def post(user, %{"status" => status} = data) do
+  def listen(user, %{"title" => _} = data) do
-    limit = Pleroma.Config.get([:instance, :limit])
+    with visibility <- data["visibility"] || "public",
-
+         {to, cc} <- get_to_and_cc(user, [], nil, visibility, nil),
-    with status <- String.trim(status),
+         listen_data <-
-         attachments <- attachments_from_ids(data),
+           Map.take(data, ["album", "artist", "title", "length"])
-         in_reply_to <- get_replied_to_activity(data["in_reply_to_status_id"]),
+           |> Map.put("type", "Audio")
-         in_reply_to_conversation <- Participation.get(data["in_reply_to_conversation_id"]),
+           |> Map.put("to", to)
-         {visibility, in_reply_to_visibility} <-
+           |> Map.put("cc", cc)
-           get_visibility(data, in_reply_to, in_reply_to_conversation),
+           |> Map.put("actor", user.ap_id),
-         {_, false} <-
+         {:ok, activity} <-
-           {:private_to_public, in_reply_to_visibility == "direct" && visibility != "direct"},
+           ActivityPub.listen(%{
-         {content_html, mentions, tags} <-
+             actor: user,
-           make_content_html(
+             to: to,
-             status,
+             object: listen_data,
-             attachments,
+             context: Utils.generate_context_id(),
-             data,
+             additional: %{"cc" => cc}
-             visibility
+           }) do
-           ),
+      {:ok, activity}
-         mentioned_users <- for({_, mentioned_user} <- mentions, do: mentioned_user.ap_id),
-         addressed_users <- get_addressed_users(mentioned_users, data["to"]),
-         {poll, poll_emoji} <- make_poll_data(data),
-         {to, cc} <-
-           get_to_and_cc(user, addressed_users, in_reply_to, visibility, in_reply_to_conversation),
-         context <- make_context(in_reply_to, in_reply_to_conversation),
-         cw <- data["spoiler_text"] || "",
-         sensitive <- data["sensitive"] || Enum.member?(tags, {"#nsfw", "nsfw"}),
-         {:ok, expires_at} <- check_expiry_date(data["expires_in"]),
-         full_payload <- String.trim(status <> cw),
-         :ok <- validate_character_limit(full_payload, attachments, limit),
-         object <-
-           make_note_data(
-             user.ap_id,
-             to,
-             context,
-             content_html,
-             attachments,
-             in_reply_to,
-             tags,
-             cw,
-             cc,
-             sensitive,
-             poll
-           ),
-         object <-
-           Map.put(
-             object,
-             "emoji",
-             Map.merge(Formatter.get_emoji_map(full_payload), poll_emoji)
-           ) do
-      preview? = Pleroma.Web.ControllerHelper.truthy_param?(data["preview"]) || false
-      direct? = visibility == "direct"
-
-      result =
-        %{
-          to: to,
-          actor: user,
-          context: context,
-          object: object,
-          additional: %{"cc" => cc, "directMessage" => direct?}
-        }
-        |> maybe_add_list_data(user, visibility)
-        |> ActivityPub.create(preview?)
-
-      if expires_at do
-        with {:ok, activity} <- result do
-          {:ok, _} = ActivityExpiration.create(activity, expires_at)
-        end
-      end
-
-      result
-    else
-      {:private_to_public, true} ->
-        {:error, dgettext("errors", "The message visibility must be direct")}
-
-      {:error, _} = e ->
-        e
-
-      e ->
-        {:error, e}
     end
   end
 
+  def post(user, %{"status" => _} = data) do
+    with {:ok, draft} <- Pleroma.Web.CommonAPI.ActivityDraft.create(user, data) do
+      draft.changes
+      |> ActivityPub.create(draft.preview?)
+      |> maybe_create_activity_expiration(draft.expires_at)
+    end
+  end
+
+  defp maybe_create_activity_expiration({:ok, activity}, %NaiveDateTime{} = expires_at) do
+    with {:ok, _} <- ActivityExpiration.create(activity, expires_at) do
+      {:ok, activity}
+    end
+  end
+
+  defp maybe_create_activity_expiration(result, _), do: result
+
   # Updates the emojis for a user based on their profile
   def update(user) do
+    emoji = emoji_from_profile(user)
+    source_data = user.info |> Map.get(:source_data, %{}) |> Map.put("tag", emoji)
+
     user =
-      with emoji <- emoji_from_profile(user),
+      case User.update_info(user, &User.Info.set_source_data(&1, source_data)) do
-           source_data <- (user.info.source_data || %{}) |> Map.put("tag", emoji),
+        {:ok, user} -> user
-           info_cng <- User.Info.set_source_data(user.info, source_data),
+        _ -> user
-           change <- Ecto.Changeset.change(user) |> Ecto.Changeset.put_embed(:info, info_cng),
-           {:ok, user} <- User.update_and_set_cache(change) do
-        user
-      else
-        _e ->
-          user
       end
 
     ActivityPub.update(%{
@@ -326,44 +272,25 @@ defmodule Pleroma.Web.CommonAPI do
   def pin(id_or_ap_id, %{ap_id: user_ap_id} = user) do
     with %Activity{
            actor: ^user_ap_id,
-           data: %{
+           data: %{"type" => "Create"},
-             "type" => "Create"
+           object: %Object{data: %{"type" => "Note"}}
-           },
-           object: %Object{
-             data: %{
-               "type" => "Note"
-             }
-           }
          } = activity <- get_by_id_or_ap_id(id_or_ap_id),
          true <- Visibility.is_public?(activity),
-         %{valid?: true} = info_changeset <- User.Info.add_pinnned_activity(user.info, activity),
+         {:ok, _user} <- User.update_info(user, &User.Info.add_pinnned_activity(&1, activity)) do
-         changeset <-
-           Ecto.Changeset.change(user) |> Ecto.Changeset.put_embed(:info, info_changeset),
-         {:ok, _user} <- User.update_and_set_cache(changeset) do
       {:ok, activity}
     else
-      %{errors: [pinned_activities: {err, _}]} ->
+      {:error, %{changes: %{info: %{errors: [pinned_activities: {err, _}]}}}} -> {:error, err}
-        {:error, err}
+      _ -> {:error, dgettext("errors", "Could not pin")}
-
-      _ ->
-        {:error, dgettext("errors", "Could not pin")}
     end
   end
 
   def unpin(id_or_ap_id, user) do
     with %Activity{} = activity <- get_by_id_or_ap_id(id_or_ap_id),
-         %{valid?: true} = info_changeset <-
+         {:ok, _user} <- User.update_info(user, &User.Info.remove_pinnned_activity(&1, activity)) do
-           User.Info.remove_pinnned_activity(user.info, activity),
-         changeset <-
-           Ecto.Changeset.change(user) |> Ecto.Changeset.put_embed(:info, info_changeset),
-         {:ok, _user} <- User.update_and_set_cache(changeset) do
       {:ok, activity}
     else
-      %{errors: [pinned_activities: {err, _}]} ->
+      %{errors: [pinned_activities: {err, _}]} -> {:error, err}
-        {:error, err}
+      _ -> {:error, dgettext("errors", "Could not unpin")}
-
-      _ ->
-        {:error, dgettext("errors", "Could not unpin")}
     end
   end
 
@@ -383,51 +310,46 @@ defmodule Pleroma.Web.CommonAPI do
   def thread_muted?(%{id: nil} = _user, _activity), do: false
 
   def thread_muted?(user, activity) do
-    with [] <- ThreadMute.check_muted(user.id, activity.data["context"]) do
+    ThreadMute.check_muted(user.id, activity.data["context"]) != []
-      false
+  end
-    else
+
-      _ -> true
+  def report(user, %{"account_id" => account_id} = data) do
+    with {:ok, account} <- get_reported_account(account_id),
+         {:ok, {content_html, _, _}} <- make_report_content_html(data["comment"]),
+         {:ok, statuses} <- get_report_statuses(account, data) do
+      ActivityPub.flag(%{
+        context: Utils.generate_context_id(),
+        actor: user,
+        account: account,
+        statuses: statuses,
+        content: content_html,
+        forward: data["forward"] || false
+      })
     end
   end
 
-  def report(user, data) do
+  def report(_user, _params), do: {:error, dgettext("errors", "Valid `account_id` required")}
-    with {:account_id, %{"account_id" => account_id}} <- {:account_id, data},
+
-         {:account, %User{} = account} <- {:account, User.get_cached_by_id(account_id)},
+  defp get_reported_account(account_id) do
-         {:ok, {content_html, _, _}} <- make_report_content_html(data["comment"]),
+    case User.get_cached_by_id(account_id) do
-         {:ok, statuses} <- get_report_statuses(account, data),
+      %User{} = account -> {:ok, account}
-         {:ok, activity} <-
+      _ -> {:error, dgettext("errors", "Account not found")}
-           ActivityPub.flag(%{
-             context: Utils.generate_context_id(),
-             actor: user,
-             account: account,
-             statuses: statuses,
-             content: content_html,
-             forward: data["forward"] || false
-           }) do
-      {:ok, activity}
-    else
-      {:error, err} -> {:error, err}
-      {:account_id, %{}} -> {:error, dgettext("errors", "Valid `account_id` required")}
-      {:account, nil} -> {:error, dgettext("errors", "Account not found")}
     end
   end
 
   def update_report_state(activity_id, state) do
-    with %Activity{} = activity <- Activity.get_by_id(activity_id),
+    with %Activity{} = activity <- Activity.get_by_id(activity_id) do
-         {:ok, activity} <- Utils.update_report_state(activity, state) do
+      Utils.update_report_state(activity, state)
-      {:ok, activity}
     else
       nil -> {:error, :not_found}
-      {:error, reason} -> {:error, reason}
       _ -> {:error, dgettext("errors", "Could not update state")}
     end
   end
 
   def update_activity_scope(activity_id, opts \\ %{}) do
     with %Activity{} = activity <- Activity.get_by_id_with_object(activity_id),
-         {:ok, activity} <- toggle_sensitive(activity, opts),
+         {:ok, activity} <- toggle_sensitive(activity, opts) do
-         {:ok, activity} <- set_visibility(activity, opts) do
+      set_visibility(activity, opts)
-      {:ok, activity}
     else
       nil -> {:error, :not_found}
       {:error, reason} -> {:error, reason}
@@ -458,23 +380,15 @@ defmodule Pleroma.Web.CommonAPI do
 
   defp set_visibility(activity, _), do: {:ok, activity}
 
-  def hide_reblogs(user, muted) do
+  def hide_reblogs(user, %{ap_id: ap_id} = _muted) do
-    ap_id = muted.ap_id
-
     if ap_id not in user.info.muted_reblogs do
-      info_changeset = User.Info.add_reblog_mute(user.info, ap_id)
+      User.update_info(user, &User.Info.add_reblog_mute(&1, ap_id))
-      changeset = Ecto.Changeset.change(user) |> Ecto.Changeset.put_embed(:info, info_changeset)
-      User.update_and_set_cache(changeset)
     end
   end
 
-  def show_reblogs(user, muted) do
+  def show_reblogs(user, %{ap_id: ap_id} = _muted) do
-    ap_id = muted.ap_id
-
     if ap_id in user.info.muted_reblogs do
-      info_changeset = User.Info.remove_reblog_mute(user.info, ap_id)
+      User.update_info(user, &User.Info.remove_reblog_mute(&1, ap_id))
-      changeset = Ecto.Changeset.change(user) |> Ecto.Changeset.put_embed(:info, info_changeset)
-      User.update_and_set_cache(changeset)
     end
   end
 end

lib/pleroma/web/common_api/utils.ex

@@ -4,11 +4,13 @@
 
 defmodule Pleroma.Web.CommonAPI.Utils do
   import Pleroma.Web.Gettext
+  import Pleroma.Web.ControllerHelper, only: [truthy_param?: 1]
 
   alias Calendar.Strftime
   alias Pleroma.Activity
   alias Pleroma.Config
   alias Pleroma.Conversation.Participation
+  alias Pleroma.Emoji
   alias Pleroma.Formatter
   alias Pleroma.Object
   alias Pleroma.Plugs.AuthenticationPlug
@@ -25,7 +27,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   # This is a hack for twidere.
   def get_by_id_or_ap_id(id) do
     activity =
-      with true <- Pleroma.FlakeId.is_flake_id?(id),
+      with true <- FlakeId.flake_id?(id),
            %Activity{} = activity <- Activity.get_by_id_with_object(id) do
         activity
       else
@@ -40,14 +42,6 @@ defmodule Pleroma.Web.CommonAPI.Utils do
       end
   end
 
-  def get_replied_to_activity(""), do: nil
-
-  def get_replied_to_activity(id) when not is_nil(id) do
-    Activity.get_by_id(id)
-  end
-
-  def get_replied_to_activity(_), do: nil
-
   def attachments_from_ids(%{"media_ids" => ids, "descriptions" => desc} = _) do
     attachments_from_ids_descs(ids, desc)
   end
@@ -158,70 +152,74 @@ defmodule Pleroma.Web.CommonAPI.Utils do
 
   def maybe_add_list_data(activity_params, _, _), do: activity_params
 
+  def make_poll_data(%{"poll" => %{"expires_in" => expires_in}} = data)
+      when is_binary(expires_in) do
+    # In some cases mastofe sends out strings instead of integers
+    data
+    |> put_in(["poll", "expires_in"], String.to_integer(expires_in))
+    |> make_poll_data()
+  end
+
   def make_poll_data(%{"poll" => %{"options" => options, "expires_in" => expires_in}} = data)
       when is_list(options) do
-    %{max_expiration: max_expiration, min_expiration: min_expiration} =
+    limits = Pleroma.Config.get([:instance, :poll_limits])
-      limits = Pleroma.Config.get([:instance, :poll_limits])
 
-    # XXX: There is probably a cleaner way of doing this
+    with :ok <- validate_poll_expiration(expires_in, limits),
-    try do
+         :ok <- validate_poll_options_amount(options, limits),
-      # In some cases mastofe sends out strings instead of integers
+         :ok <- validate_poll_options_length(options, limits) do
-      expires_in = if is_binary(expires_in), do: String.to_integer(expires_in), else: expires_in
+      {option_notes, emoji} =
-
-      if Enum.count(options) > limits.max_options do
-        raise ArgumentError, message: "Poll can't contain more than #{limits.max_options} options"
-      end
-
-      {poll, emoji} =
         Enum.map_reduce(options, %{}, fn option, emoji ->
-          if String.length(option) > limits.max_option_chars do
+          note = %{
-            raise ArgumentError,
+            "name" => option,
-              message:
+            "type" => "Note",
-                "Poll options cannot be longer than #{limits.max_option_chars} characters each"
+            "replies" => %{"type" => "Collection", "totalItems" => 0}
-          end
+          }
 
-          {%{
+          {note, Map.merge(emoji, Emoji.Formatter.get_emoji_map(option))}
-             "name" => option,
-             "type" => "Note",
-             "replies" => %{"type" => "Collection", "totalItems" => 0}
-           }, Map.merge(emoji, Formatter.get_emoji_map(option))}
         end)
 
-      case expires_in do
-        expires_in when expires_in > max_expiration ->
-          raise ArgumentError, message: "Expiration date is too far in the future"
-
-        expires_in when expires_in < min_expiration ->
-          raise ArgumentError, message: "Expiration date is too soon"
-
-        _ ->
-          :noop
-      end
-
       end_time =
         NaiveDateTime.utc_now()
         |> NaiveDateTime.add(expires_in)
         |> NaiveDateTime.to_iso8601()
 
-      poll =
+      key = if truthy_param?(data["poll"]["multiple"]), do: "anyOf", else: "oneOf"
-        if Pleroma.Web.ControllerHelper.truthy_param?(data["poll"]["multiple"]) do
+      poll = %{"type" => "Question", key => option_notes, "closed" => end_time}
-          %{"type" => "Question", "anyOf" => poll, "closed" => end_time}
-        else
-          %{"type" => "Question", "oneOf" => poll, "closed" => end_time}
-        end
 
-      {poll, emoji}
+      {:ok, {poll, emoji}}
-    rescue
-      e in ArgumentError -> e.message
     end
   end
 
   def make_poll_data(%{"poll" => poll}) when is_map(poll) do
-    "Invalid poll"
+    {:error, "Invalid poll"}
   end
 
   def make_poll_data(_data) do
-    {%{}, %{}}
+    {:ok, {%{}, %{}}}
+  end
+
+  defp validate_poll_options_amount(options, %{max_options: max_options}) do
+    if Enum.count(options) > max_options do
+      {:error, "Poll can't contain more than #{max_options} options"}
+    else
+      :ok
+    end
+  end
+
+  defp validate_poll_options_length(options, %{max_option_chars: max_option_chars}) do
+    if Enum.any?(options, &(String.length(&1) > max_option_chars)) do
+      {:error, "Poll options cannot be longer than #{max_option_chars} characters each"}
+    else
+      :ok
+    end
+  end
+
+  defp validate_poll_expiration(expires_in, %{min_expiration: min, max_expiration: max}) do
+    cond do
+      expires_in > max -> {:error, "Expiration date is too far in the future"}
+      expires_in < min -> {:error, "Expiration date is too soon"}
+      true -> :ok
+    end
   end
 
   def make_content_html(
@@ -233,7 +231,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     no_attachment_links =
       data
       |> Map.get("no_attachment_links", Config.get([:instance, :no_attachment_links]))
-      |> Kernel.in([true, "true"])
+      |> truthy_param?()
 
     content_type = get_content_type(data["content_type"])
 
@@ -346,25 +344,25 @@ defmodule Pleroma.Web.CommonAPI.Utils do
         attachments,
         in_reply_to,
         tags,
-        cw \\ nil,
+        summary \\ nil,
         cc \\ [],
         sensitive \\ false,
-        merge \\ %{}
+        extra_params \\ %{}
       ) do
     %{
       "type" => "Note",
       "to" => to,
       "cc" => cc,
       "content" => content_html,
-      "summary" => cw,
+      "summary" => summary,
-      "sensitive" => !Enum.member?(["false", "False", "0", false], sensitive),
+      "sensitive" => truthy_param?(sensitive),
       "context" => context,
       "attachment" => attachments,
       "actor" => actor,
       "tag" => Keyword.values(tags) |> Enum.uniq()
     }
     |> add_in_reply_to(in_reply_to)
-    |> Map.merge(merge)
+    |> Map.merge(extra_params)
   end
 
   defp add_in_reply_to(object, nil), do: object
@@ -433,12 +431,14 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     end
   end
 
-  def emoji_from_profile(%{info: _info} = user) do
+  def emoji_from_profile(%User{bio: bio, name: name}) do
-    (Formatter.get_emoji(user.bio) ++ Formatter.get_emoji(user.name))
+    [bio, name]
-    |> Enum.map(fn {shortcode, url, _} ->
+    |> Enum.map(&Emoji.Formatter.get_emoji/1)
+    |> Enum.concat()
+    |> Enum.map(fn {shortcode, %Emoji{file: path}} ->
       %{
         "type" => "Emoji",
-        "icon" => %{"type" => "Image", "url" => "#{Endpoint.url()}#{url}"},
+        "icon" => %{"type" => "Image", "url" => "#{Endpoint.url()}#{path}"},
         "name" => ":#{shortcode}:"
       }
     end)
@@ -570,15 +570,16 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     }
   end
 
-  def validate_character_limit(full_payload, attachments, limit) do
+  def validate_character_limit("" = _full_payload, [] = _attachments) do
+    {:error, dgettext("errors", "Cannot post an empty status without attachments")}
+  end
+
+  def validate_character_limit(full_payload, _attachments) do
+    limit = Pleroma.Config.get([:instance, :limit])
     length = String.length(full_payload)
 
     if length < limit do
-      if length > 0 or Enum.count(attachments) > 0 do
+      :ok
-        :ok
-      else
-        {:error, dgettext("errors", "Cannot post an empty status without attachments")}
-      end
     else
       {:error, dgettext("errors", "The status is over the character limit")}
     end

lib/pleroma/web/controller_helper.ex

@@ -6,7 +6,7 @@ defmodule Pleroma.Web.ControllerHelper do
   use Pleroma.Web, :controller
 
   # As in MastoAPI, per https://api.rubyonrails.org/classes/ActiveModel/Type/Boolean.html
-  @falsy_param_values [false, 0, "0", "f", "F", "false", "FALSE", "off", "OFF"]
+  @falsy_param_values [false, 0, "0", "f", "F", "false", "False", "FALSE", "off", "OFF"]
   def truthy_param?(blank_value) when blank_value in [nil, ""], do: nil
   def truthy_param?(value), do: value not in @falsy_param_values
 
@@ -68,4 +68,11 @@ defmodule Pleroma.Web.ControllerHelper do
         conn
     end
   end
+
+  def assign_account_by_id(%{params: %{"id" => id}} = conn, _) do
+    case Pleroma.User.get_cached_by_id(id) do
+      %Pleroma.User{} = account -> assign(conn, :account, account)
+      nil -> Pleroma.Web.MastodonAPI.FallbackController.call(conn, {:error, :not_found}) |> halt()
+    end
+  end
 end

lib/pleroma/web/endpoint.ex

@@ -97,10 +97,7 @@ defmodule Pleroma.Web.Endpoint do
     extra: extra
   )
 
-  # Note: the plug and its configuration is compile-time this can't be upstreamed yet
+  plug(Pleroma.Plugs.RemoteIp)
-  if proxies = Pleroma.Config.get([__MODULE__, :reverse_proxies]) do
-    plug(RemoteIp, proxies: proxies)
-  end
 
   defmodule Instrumenter do
     use Prometheus.PhoenixInstrumenter

lib/pleroma/web/mastodon_api/controllers/account_controller.ex

@@ -0,0 +1,304 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.AccountController do
+  use Pleroma.Web, :controller
+
+  import Pleroma.Web.ControllerHelper,
+    only: [add_link_headers: 2, truthy_param?: 1, assign_account_by_id: 2, json_response: 3]
+
+  alias Pleroma.Emoji
+  alias Pleroma.Plugs.RateLimiter
+  alias Pleroma.User
+  alias Pleroma.Web.ActivityPub.ActivityPub
+  alias Pleroma.Web.CommonAPI
+  alias Pleroma.Web.MastodonAPI.ListView
+  alias Pleroma.Web.MastodonAPI.MastodonAPI
+  alias Pleroma.Web.MastodonAPI.StatusView
+  alias Pleroma.Web.OAuth.Token
+  alias Pleroma.Web.TwitterAPI.TwitterAPI
+
+  @relations [:follow, :unfollow]
+  @needs_account ~W(followers following lists follow unfollow mute unmute block unblock)a
+
+  plug(RateLimiter, {:relations_id_action, params: ["id", "uri"]} when action in @relations)
+  plug(RateLimiter, :relations_actions when action in @relations)
+  plug(RateLimiter, :app_account_creation when action == :create)
+  plug(:assign_account_by_id when action in @needs_account)
+
+  action_fallback(Pleroma.Web.MastodonAPI.FallbackController)
+
+  @doc "POST /api/v1/accounts"
+  def create(
+        %{assigns: %{app: app}} = conn,
+        %{"username" => nickname, "email" => _, "password" => _, "agreement" => true} = params
+      ) do
+    params =
+      params
+      |> Map.take([
+        "email",
+        "captcha_solution",
+        "captcha_token",
+        "captcha_answer_data",
+        "token",
+        "password"
+      ])
+      |> Map.put("nickname", nickname)
+      |> Map.put("fullname", params["fullname"] || nickname)
+      |> Map.put("bio", params["bio"] || "")
+      |> Map.put("confirm", params["password"])
+
+    with {:ok, user} <- TwitterAPI.register_user(params, need_confirmation: true),
+         {:ok, token} <- Token.create_token(app, user, %{scopes: app.scopes}) do
+      json(conn, %{
+        token_type: "Bearer",
+        access_token: token.token,
+        scope: app.scopes,
+        created_at: Token.Utils.format_created_at(token)
+      })
+    else
+      {:error, errors} -> json_response(conn, :bad_request, errors)
+    end
+  end
+
+  def create(%{assigns: %{app: _app}} = conn, _) do
+    render_error(conn, :bad_request, "Missing parameters")
+  end
+
+  def create(conn, _) do
+    render_error(conn, :forbidden, "Invalid credentials")
+  end
+
+  @doc "GET /api/v1/accounts/verify_credentials"
+  def verify_credentials(%{assigns: %{user: user}} = conn, _) do
+    chat_token = Phoenix.Token.sign(conn, "user socket", user.id)
+
+    render(conn, "show.json",
+      user: user,
+      for: user,
+      with_pleroma_settings: true,
+      with_chat_token: chat_token
+    )
+  end
+
+  @doc "PATCH /api/v1/accounts/update_credentials"
+  def update_credentials(%{assigns: %{user: original_user}} = conn, params) do
+    user = original_user
+
+    user_params =
+      %{}
+      |> add_if_present(params, "display_name", :name)
+      |> add_if_present(params, "note", :bio, fn value -> {:ok, User.parse_bio(value, user)} end)
+      |> add_if_present(params, "avatar", :avatar, fn value ->
+        with %Plug.Upload{} <- value,
+             {:ok, object} <- ActivityPub.upload(value, type: :avatar) do
+          {:ok, object.data}
+        end
+      end)
+
+    emojis_text = (user_params["display_name"] || "") <> (user_params["note"] || "")
+
+    user_info_emojis =
+      user.info
+      |> Map.get(:emoji, [])
+      |> Enum.concat(Emoji.Formatter.get_emoji_map(emojis_text))
+      |> Enum.dedup()
+
+    info_params =
+      [
+        :no_rich_text,
+        :locked,
+        :hide_followers_count,
+        :hide_follows_count,
+        :hide_followers,
+        :hide_follows,
+        :hide_favorites,
+        :show_role,
+        :skip_thread_containment,
+        :discoverable
+      ]
+      |> Enum.reduce(%{}, fn key, acc ->
+        add_if_present(acc, params, to_string(key), key, &{:ok, truthy_param?(&1)})
+      end)
+      |> add_if_present(params, "default_scope", :default_scope)
+      |> add_if_present(params, "fields", :fields, fn fields ->
+        fields = Enum.map(fields, fn f -> Map.update!(f, "value", &AutoLinker.link(&1)) end)
+
+        {:ok, fields}
+      end)
+      |> add_if_present(params, "fields", :raw_fields)
+      |> add_if_present(params, "pleroma_settings_store", :pleroma_settings_store, fn value ->
+        {:ok, Map.merge(user.info.pleroma_settings_store, value)}
+      end)
+      |> add_if_present(params, "header", :banner, fn value ->
+        with %Plug.Upload{} <- value,
+             {:ok, object} <- ActivityPub.upload(value, type: :banner) do
+          {:ok, object.data}
+        end
+      end)
+      |> add_if_present(params, "pleroma_background_image", :background, fn value ->
+        with %Plug.Upload{} <- value,
+             {:ok, object} <- ActivityPub.upload(value, type: :background) do
+          {:ok, object.data}
+        end
+      end)
+      |> Map.put(:emoji, user_info_emojis)
+
+    changeset =
+      user
+      |> User.update_changeset(user_params)
+      |> User.change_info(&User.Info.profile_update(&1, info_params))
+
+    with {:ok, user} <- User.update_and_set_cache(changeset) do
+      if original_user != user, do: CommonAPI.update(user)
+
+      render(conn, "show.json", user: user, for: user, with_pleroma_settings: true)
+    else
+      _e -> render_error(conn, :forbidden, "Invalid request")
+    end
+  end
+
+  defp add_if_present(map, params, params_field, map_field, value_function \\ &{:ok, &1}) do
+    with true <- Map.has_key?(params, params_field),
+         {:ok, new_value} <- value_function.(params[params_field]) do
+      Map.put(map, map_field, new_value)
+    else
+      _ -> map
+    end
+  end
+
+  @doc "GET /api/v1/accounts/relationships"
+  def relationships(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    targets = User.get_all_by_ids(List.wrap(id))
+
+    render(conn, "relationships.json", user: user, targets: targets)
+  end
+
+  # Instead of returning a 400 when no "id" params is present, Mastodon returns an empty array.
+  def relationships(%{assigns: %{user: _user}} = conn, _), do: json(conn, [])
+
+  @doc "GET /api/v1/accounts/:id"
+  def show(%{assigns: %{user: for_user}} = conn, %{"id" => nickname_or_id}) do
+    with %User{} = user <- User.get_cached_by_nickname_or_id(nickname_or_id, for: for_user),
+         true <- User.auth_active?(user) || user.id == for_user.id || User.superuser?(for_user) do
+      render(conn, "show.json", user: user, for: for_user)
+    else
+      _e -> render_error(conn, :not_found, "Can't find user")
+    end
+  end
+
+  @doc "GET /api/v1/accounts/:id/statuses"
+  def statuses(%{assigns: %{user: reading_user}} = conn, params) do
+    with %User{} = user <- User.get_cached_by_nickname_or_id(params["id"], for: reading_user) do
+      params = Map.put(params, "tag", params["tagged"])
+      activities = ActivityPub.fetch_user_activities(user, reading_user, params)
+
+      conn
+      |> add_link_headers(activities)
+      |> put_view(StatusView)
+      |> render("index.json", activities: activities, for: reading_user, as: :activity)
+    end
+  end
+
+  @doc "GET /api/v1/accounts/:id/followers"
+  def followers(%{assigns: %{user: for_user, account: user}} = conn, params) do
+    followers =
+      cond do
+        for_user && user.id == for_user.id -> MastodonAPI.get_followers(user, params)
+        user.info.hide_followers -> []
+        true -> MastodonAPI.get_followers(user, params)
+      end
+
+    conn
+    |> add_link_headers(followers)
+    |> render("index.json", for: for_user, users: followers, as: :user)
+  end
+
+  @doc "GET /api/v1/accounts/:id/following"
+  def following(%{assigns: %{user: for_user, account: user}} = conn, params) do
+    followers =
+      cond do
+        for_user && user.id == for_user.id -> MastodonAPI.get_friends(user, params)
+        user.info.hide_follows -> []
+        true -> MastodonAPI.get_friends(user, params)
+      end
+
+    conn
+    |> add_link_headers(followers)
+    |> render("index.json", for: for_user, users: followers, as: :user)
+  end
+
+  @doc "GET /api/v1/accounts/:id/lists"
+  def lists(%{assigns: %{user: user, account: account}} = conn, _params) do
+    lists = Pleroma.List.get_lists_account_belongs(user, account)
+
+    conn
+    |> put_view(ListView)
+    |> render("index.json", lists: lists)
+  end
+
+  @doc "POST /api/v1/accounts/:id/follow"
+  def follow(%{assigns: %{user: %{id: id}, account: %{id: id}}}, _params) do
+    {:error, :not_found}
+  end
+
+  def follow(%{assigns: %{user: follower, account: followed}} = conn, _params) do
+    with {:ok, follower} <- MastodonAPI.follow(follower, followed, conn.params) do
+      render(conn, "relationship.json", user: follower, target: followed)
+    else
+      {:error, message} -> json_response(conn, :forbidden, %{error: message})
+    end
+  end
+
+  @doc "POST /api/v1/accounts/:id/unfollow"
+  def unfollow(%{assigns: %{user: %{id: id}, account: %{id: id}}}, _params) do
+    {:error, :not_found}
+  end
+
+  def unfollow(%{assigns: %{user: follower, account: followed}} = conn, _params) do
+    with {:ok, follower} <- CommonAPI.unfollow(follower, followed) do
+      render(conn, "relationship.json", user: follower, target: followed)
+    end
+  end
+
+  @doc "POST /api/v1/accounts/:id/mute"
+  def mute(%{assigns: %{user: muter, account: muted}} = conn, params) do
+    notifications? = params |> Map.get("notifications", true) |> truthy_param?()
+
+    with {:ok, muter} <- User.mute(muter, muted, notifications?) do
+      render(conn, "relationship.json", user: muter, target: muted)
+    else
+      {:error, message} -> json_response(conn, :forbidden, %{error: message})
+    end
+  end
+
+  @doc "POST /api/v1/accounts/:id/unmute"
+  def unmute(%{assigns: %{user: muter, account: muted}} = conn, _params) do
+    with {:ok, muter} <- User.unmute(muter, muted) do
+      render(conn, "relationship.json", user: muter, target: muted)
+    else
+      {:error, message} -> json_response(conn, :forbidden, %{error: message})
+    end
+  end
+
+  @doc "POST /api/v1/accounts/:id/block"
+  def block(%{assigns: %{user: blocker, account: blocked}} = conn, _params) do
+    with {:ok, blocker} <- User.block(blocker, blocked),
+         {:ok, _activity} <- ActivityPub.block(blocker, blocked) do
+      render(conn, "relationship.json", user: blocker, target: blocked)
+    else
+      {:error, message} -> json_response(conn, :forbidden, %{error: message})
+    end
+  end
+
+  @doc "POST /api/v1/accounts/:id/unblock"
+  def unblock(%{assigns: %{user: blocker, account: blocked}} = conn, _params) do
+    with {:ok, blocker} <- User.unblock(blocker, blocked),
+         {:ok, _activity} <- ActivityPub.unblock(blocker, blocked) do
+      render(conn, "relationship.json", user: blocker, target: blocked)
+    else
+      {:error, message} -> json_response(conn, :forbidden, %{error: message})
+    end
+  end
+end

lib/pleroma/web/mastodon_api/controllers/conversation_controller.ex

@@ -0,0 +1,32 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.ConversationController do
+  use Pleroma.Web, :controller
+
+  import Pleroma.Web.ControllerHelper, only: [add_link_headers: 2]
+
+  alias Pleroma.Conversation.Participation
+  alias Pleroma.Repo
+
+  action_fallback(Pleroma.Web.MastodonAPI.FallbackController)
+
+  @doc "GET /api/v1/conversations"
+  def index(%{assigns: %{user: user}} = conn, params) do
+    participations = Participation.for_user_with_last_activity_id(user, params)
+
+    conn
+    |> add_link_headers(participations)
+    |> render("participations.json", participations: participations, for: user)
+  end
+
+  @doc "POST /api/v1/conversations/:id/read"
+  def read(%{assigns: %{user: user}} = conn, %{"id" => participation_id}) do
+    with %Participation{} = participation <-
+           Repo.get_by(Participation, id: participation_id, user_id: user.id),
+         {:ok, participation} <- Participation.mark_as_read(participation) do
+      render(conn, "participation.json", participation: participation, for: user)
+    end
+  end
+end

lib/pleroma/web/mastodon_api/controllers/domain_block_controller.ex

@@ -0,0 +1,26 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.DomainBlockController do
+  use Pleroma.Web, :controller
+
+  alias Pleroma.User
+
+  @doc "GET /api/v1/domain_blocks"
+  def index(%{assigns: %{user: %{info: info}}} = conn, _) do
+    json(conn, Map.get(info, :domain_blocks, []))
+  end
+
+  @doc "POST /api/v1/domain_blocks"
+  def create(%{assigns: %{user: blocker}} = conn, %{"domain" => domain}) do
+    User.block_domain(blocker, domain)
+    json(conn, %{})
+  end
+
+  @doc "DELETE /api/v1/domain_blocks"
+  def delete(%{assigns: %{user: blocker}} = conn, %{"domain" => domain}) do
+    User.unblock_domain(blocker, domain)
+    json(conn, %{})
+  end
+end

lib/pleroma/web/mastodon_api/controllers/filter_controller.ex

@@ -0,0 +1,72 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.FilterController do
+  use Pleroma.Web, :controller
+
+  alias Pleroma.Filter
+
+  @doc "GET /api/v1/filters"
+  def index(%{assigns: %{user: user}} = conn, _) do
+    filters = Filter.get_filters(user)
+
+    render(conn, "filters.json", filters: filters)
+  end
+
+  @doc "POST /api/v1/filters"
+  def create(
+        %{assigns: %{user: user}} = conn,
+        %{"phrase" => phrase, "context" => context} = params
+      ) do
+    query = %Filter{
+      user_id: user.id,
+      phrase: phrase,
+      context: context,
+      hide: Map.get(params, "irreversible", false),
+      whole_word: Map.get(params, "boolean", true)
+      # expires_at
+    }
+
+    {:ok, response} = Filter.create(query)
+
+    render(conn, "filter.json", filter: response)
+  end
+
+  @doc "GET /api/v1/filters/:id"
+  def show(%{assigns: %{user: user}} = conn, %{"id" => filter_id}) do
+    filter = Filter.get(filter_id, user)
+
+    render(conn, "filter.json", filter: filter)
+  end
+
+  @doc "PUT /api/v1/filters/:id"
+  def update(
+        %{assigns: %{user: user}} = conn,
+        %{"phrase" => phrase, "context" => context, "id" => filter_id} = params
+      ) do
+    query = %Filter{
+      user_id: user.id,
+      filter_id: filter_id,
+      phrase: phrase,
+      context: context,
+      hide: Map.get(params, "irreversible", nil),
+      whole_word: Map.get(params, "boolean", true)
+      # expires_at
+    }
+
+    {:ok, response} = Filter.update(query)
+    render(conn, "filter.json", filter: response)
+  end
+
+  @doc "DELETE /api/v1/filters/:id"
+  def delete(%{assigns: %{user: user}} = conn, %{"id" => filter_id}) do
+    query = %Filter{
+      user_id: user.id,
+      filter_id: filter_id
+    }
+
+    {:ok, _} = Filter.delete(query)
+    json(conn, %{})
+  end
+end

lib/pleroma/web/mastodon_api/controllers/follow_request_controller.ex

@@ -0,0 +1,49 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.FollowRequestController do
+  use Pleroma.Web, :controller
+
+  alias Pleroma.User
+  alias Pleroma.Web.CommonAPI
+
+  plug(:put_view, Pleroma.Web.MastodonAPI.AccountView)
+  plug(:assign_follower when action != :index)
+
+  action_fallback(:errors)
+
+  @doc "GET /api/v1/follow_requests"
+  def index(%{assigns: %{user: followed}} = conn, _params) do
+    follow_requests = User.get_follow_requests(followed)
+
+    render(conn, "index.json", for: followed, users: follow_requests, as: :user)
+  end
+
+  @doc "POST /api/v1/follow_requests/:id/authorize"
+  def authorize(%{assigns: %{user: followed, follower: follower}} = conn, _params) do
+    with {:ok, follower} <- CommonAPI.accept_follow_request(follower, followed) do
+      render(conn, "relationship.json", user: followed, target: follower)
+    end
+  end
+
+  @doc "POST /api/v1/follow_requests/:id/reject"
+  def reject(%{assigns: %{user: followed, follower: follower}} = conn, _params) do
+    with {:ok, follower} <- CommonAPI.reject_follow_request(follower, followed) do
+      render(conn, "relationship.json", user: followed, target: follower)
+    end
+  end
+
+  defp assign_follower(%{params: %{"id" => id}} = conn, _) do
+    case User.get_cached_by_id(id) do
+      %User{} = follower -> assign(conn, :follower, follower)
+      nil -> Pleroma.Web.MastodonAPI.FallbackController.call(conn, {:error, :not_found}) |> halt()
+    end
+  end
+
+  defp errors(conn, {:error, message}) do
+    conn
+    |> put_status(:forbidden)
+    |> json(%{error: message})
+  end
+end

lib/pleroma/web/mastodon_api/controllers/list_controller.ex

@@ -49,7 +49,7 @@ defmodule Pleroma.Web.MastodonAPI.ListController do
     with {:ok, users} <- Pleroma.List.get_following(list) do
       conn
       |> put_view(AccountView)
-      |> render("accounts.json", for: user, users: users, as: :user)
+      |> render("index.json", for: user, users: users, as: :user)
     end
   end
 

lib/pleroma/web/mastodon_api/controllers/notification_controller.ex

@@ -0,0 +1,57 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.NotificationController do
+  use Pleroma.Web, :controller
+
+  import Pleroma.Web.ControllerHelper, only: [add_link_headers: 2]
+
+  alias Pleroma.Notification
+  alias Pleroma.Web.MastodonAPI.MastodonAPI
+
+  # GET /api/v1/notifications
+  def index(%{assigns: %{user: user}} = conn, params) do
+    notifications = MastodonAPI.get_notifications(user, params)
+
+    conn
+    |> add_link_headers(notifications)
+    |> render("index.json", notifications: notifications, for: user)
+  end
+
+  # GET /api/v1/notifications/:id
+  def show(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with {:ok, notification} <- Notification.get(user, id) do
+      render(conn, "show.json", notification: notification, for: user)
+    else
+      {:error, reason} ->
+        conn
+        |> put_status(:forbidden)
+        |> json(%{"error" => reason})
+    end
+  end
+
+  # POST /api/v1/notifications/clear
+  def clear(%{assigns: %{user: user}} = conn, _params) do
+    Notification.clear(user)
+    json(conn, %{})
+  end
+
+  # POST /api/v1/notifications/dismiss
+  def dismiss(%{assigns: %{user: user}} = conn, %{"id" => id} = _params) do
+    with {:ok, _notif} <- Notification.dismiss(user, id) do
+      json(conn, %{})
+    else
+      {:error, reason} ->
+        conn
+        |> put_status(:forbidden)
+        |> json(%{"error" => reason})
+    end
+  end
+
+  # DELETE /api/v1/notifications/destroy_multiple
+  def destroy_multiple(%{assigns: %{user: user}} = conn, %{"ids" => ids} = _params) do
+    Notification.destroy_multiple(user, ids)
+    json(conn, %{})
+  end
+end

lib/pleroma/web/mastodon_api/controllers/report_controller.ex

@@ -0,0 +1,16 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.ReportController do
+  use Pleroma.Web, :controller
+
+  action_fallback(Pleroma.Web.MastodonAPI.FallbackController)
+
+  @doc "POST /api/v1/reports"
+  def create(%{assigns: %{user: user}} = conn, params) do
+    with {:ok, activity} <- Pleroma.Web.CommonAPI.report(user, params) do
+      render(conn, "show.json", activity: activity)
+    end
+  end
+end

lib/pleroma/web/mastodon_api/controllers/scheduled_activity_controller.ex

@@ -0,0 +1,51 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.ScheduledActivityController do
+  use Pleroma.Web, :controller
+
+  import Pleroma.Web.ControllerHelper, only: [add_link_headers: 2]
+
+  alias Pleroma.ScheduledActivity
+  alias Pleroma.Web.MastodonAPI.MastodonAPI
+
+  plug(:assign_scheduled_activity when action != :index)
+
+  action_fallback(Pleroma.Web.MastodonAPI.FallbackController)
+
+  @doc "GET /api/v1/scheduled_statuses"
+  def index(%{assigns: %{user: user}} = conn, params) do
+    with scheduled_activities <- MastodonAPI.get_scheduled_activities(user, params) do
+      conn
+      |> add_link_headers(scheduled_activities)
+      |> render("index.json", scheduled_activities: scheduled_activities)
+    end
+  end
+
+  @doc "GET /api/v1/scheduled_statuses/:id"
+  def show(%{assigns: %{scheduled_activity: scheduled_activity}} = conn, _params) do
+    render(conn, "show.json", scheduled_activity: scheduled_activity)
+  end
+
+  @doc "PUT /api/v1/scheduled_statuses/:id"
+  def update(%{assigns: %{scheduled_activity: scheduled_activity}} = conn, params) do
+    with {:ok, scheduled_activity} <- ScheduledActivity.update(scheduled_activity, params) do
+      render(conn, "show.json", scheduled_activity: scheduled_activity)
+    end
+  end
+
+  @doc "DELETE /api/v1/scheduled_statuses/:id"
+  def delete(%{assigns: %{scheduled_activity: scheduled_activity}} = conn, _params) do
+    with {:ok, scheduled_activity} <- ScheduledActivity.delete(scheduled_activity) do
+      render(conn, "show.json", scheduled_activity: scheduled_activity)
+    end
+  end
+
+  defp assign_scheduled_activity(%{assigns: %{user: user}, params: %{"id" => id}} = conn, _) do
+    case ScheduledActivity.get(user, id) do
+      %ScheduledActivity{} = activity -> assign(conn, :scheduled_activity, activity)
+      nil -> Pleroma.Web.MastodonAPI.FallbackController.call(conn, {:error, :not_found}) |> halt()
+    end
+  end
+end

lib/pleroma/web/mastodon_api/controllers/search_controller.ex

@@ -19,9 +19,10 @@ defmodule Pleroma.Web.MastodonAPI.SearchController do
 
   def account_search(%{assigns: %{user: user}} = conn, %{"q" => query} = params) do
     accounts = User.search(query, search_options(params, user))
-    res = AccountView.render("accounts.json", users: accounts, for: user, as: :user)
 
-    json(conn, res)
+    conn
+    |> put_view(AccountView)
+    |> render("index.json", users: accounts, for: user, as: :user)
   end
 
   def search2(conn, params), do: do_search(:v2, conn, params)
@@ -71,7 +72,7 @@ defmodule Pleroma.Web.MastodonAPI.SearchController do
 
   defp resource_search(_, "accounts", query, options) do
     accounts = with_fallback(fn -> User.search(query, options) end)
-    AccountView.render("accounts.json", users: accounts, for: options[:for_user], as: :user)
+    AccountView.render("index.json", users: accounts, for: options[:for_user], as: :user)
   end
 
   defp resource_search(_, "statuses", query, options) do

lib/pleroma/web/mastodon_api/controllers/status_controller.ex

@@ -0,0 +1,274 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.StatusController do
+  use Pleroma.Web, :controller
+
+  import Pleroma.Web.MastodonAPI.MastodonAPIController, only: [try_render: 3]
+
+  require Ecto.Query
+
+  alias Pleroma.Activity
+  alias Pleroma.Bookmark
+  alias Pleroma.Object
+  alias Pleroma.Plugs.RateLimiter
+  alias Pleroma.Repo
+  alias Pleroma.ScheduledActivity
+  alias Pleroma.User
+  alias Pleroma.Web.ActivityPub.ActivityPub
+  alias Pleroma.Web.ActivityPub.Visibility
+  alias Pleroma.Web.CommonAPI
+  alias Pleroma.Web.MastodonAPI.AccountView
+  alias Pleroma.Web.MastodonAPI.ScheduledActivityView
+
+  @rate_limited_status_actions ~w(reblog unreblog favourite unfavourite create delete)a
+
+  plug(
+    RateLimiter,
+    {:status_id_action, bucket_name: "status_id_action:reblog_unreblog", params: ["id"]}
+    when action in ~w(reblog unreblog)a
+  )
+
+  plug(
+    RateLimiter,
+    {:status_id_action, bucket_name: "status_id_action:fav_unfav", params: ["id"]}
+    when action in ~w(favourite unfavourite)a
+  )
+
+  plug(RateLimiter, :statuses_actions when action in @rate_limited_status_actions)
+
+  action_fallback(Pleroma.Web.MastodonAPI.FallbackController)
+
+  @doc """
+  GET `/api/v1/statuses?ids[]=1&ids[]=2`
+
+  `ids` query param is required
+  """
+  def index(%{assigns: %{user: user}} = conn, %{"ids" => ids}) do
+    limit = 100
+
+    activities =
+      ids
+      |> Enum.take(limit)
+      |> Activity.all_by_ids_with_object()
+      |> Enum.filter(&Visibility.visible_for_user?(&1, user))
+
+    render(conn, "index.json", activities: activities, for: user, as: :activity)
+  end
+
+  @doc """
+  POST /api/v1/statuses
+
+  Creates a scheduled status when `scheduled_at` param is present and it's far enough
+  """
+  def create(
+        %{assigns: %{user: user}} = conn,
+        %{"status" => _, "scheduled_at" => scheduled_at} = params
+      ) do
+    params = Map.put(params, "in_reply_to_status_id", params["in_reply_to_id"])
+
+    if ScheduledActivity.far_enough?(scheduled_at) do
+      with {:ok, scheduled_activity} <-
+             ScheduledActivity.create(user, %{"params" => params, "scheduled_at" => scheduled_at}) do
+        conn
+        |> put_view(ScheduledActivityView)
+        |> render("show.json", scheduled_activity: scheduled_activity)
+      end
+    else
+      create(conn, Map.drop(params, ["scheduled_at"]))
+    end
+  end
+
+  @doc """
+  POST /api/v1/statuses
+
+  Creates a regular status
+  """
+  def create(%{assigns: %{user: user}} = conn, %{"status" => _} = params) do
+    params = Map.put(params, "in_reply_to_status_id", params["in_reply_to_id"])
+
+    with {:ok, activity} <- CommonAPI.post(user, params) do
+      try_render(conn, "show.json",
+        activity: activity,
+        for: user,
+        as: :activity,
+        with_direct_conversation_id: true
+      )
+    else
+      {:error, message} ->
+        conn
+        |> put_status(:unprocessable_entity)
+        |> json(%{error: message})
+    end
+  end
+
+  def create(%{assigns: %{user: _user}} = conn, %{"media_ids" => _} = params) do
+    create(conn, Map.put(params, "status", ""))
+  end
+
+  @doc "GET /api/v1/statuses/:id"
+  def show(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
+         true <- Visibility.visible_for_user?(activity, user) do
+      try_render(conn, "show.json", activity: activity, for: user)
+    end
+  end
+
+  @doc "DELETE /api/v1/statuses/:id"
+  def delete(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with {:ok, %Activity{}} <- CommonAPI.delete(id, user) do
+      json(conn, %{})
+    else
+      _e -> render_error(conn, :forbidden, "Can't delete this post")
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/reblog"
+  def reblog(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
+    with {:ok, announce, _activity} <- CommonAPI.repeat(ap_id_or_id, user),
+         %Activity{} = announce <- Activity.normalize(announce.data) do
+      try_render(conn, "show.json", %{activity: announce, for: user, as: :activity})
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/unreblog"
+  def unreblog(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
+    with {:ok, _unannounce, %{data: %{"id" => id}}} <- CommonAPI.unrepeat(ap_id_or_id, user),
+         %Activity{} = activity <- Activity.get_create_by_object_ap_id_with_object(id) do
+      try_render(conn, "show.json", %{activity: activity, for: user, as: :activity})
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/favourite"
+  def favourite(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
+    with {:ok, _fav, %{data: %{"id" => id}}} <- CommonAPI.favorite(ap_id_or_id, user),
+         %Activity{} = activity <- Activity.get_create_by_object_ap_id(id) do
+      try_render(conn, "show.json", activity: activity, for: user, as: :activity)
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/unfavourite"
+  def unfavourite(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
+    with {:ok, _, _, %{data: %{"id" => id}}} <- CommonAPI.unfavorite(ap_id_or_id, user),
+         %Activity{} = activity <- Activity.get_create_by_object_ap_id(id) do
+      try_render(conn, "show.json", activity: activity, for: user, as: :activity)
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/pin"
+  def pin(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
+    with {:ok, activity} <- CommonAPI.pin(ap_id_or_id, user) do
+      try_render(conn, "show.json", activity: activity, for: user, as: :activity)
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/unpin"
+  def unpin(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
+    with {:ok, activity} <- CommonAPI.unpin(ap_id_or_id, user) do
+      try_render(conn, "show.json", activity: activity, for: user, as: :activity)
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/bookmark"
+  def bookmark(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
+         %User{} = user <- User.get_cached_by_nickname(user.nickname),
+         true <- Visibility.visible_for_user?(activity, user),
+         {:ok, _bookmark} <- Bookmark.create(user.id, activity.id) do
+      try_render(conn, "show.json", activity: activity, for: user, as: :activity)
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/unbookmark"
+  def unbookmark(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
+         %User{} = user <- User.get_cached_by_nickname(user.nickname),
+         true <- Visibility.visible_for_user?(activity, user),
+         {:ok, _bookmark} <- Bookmark.destroy(user.id, activity.id) do
+      try_render(conn, "show.json", activity: activity, for: user, as: :activity)
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/mute"
+  def mute_conversation(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with %Activity{} = activity <- Activity.get_by_id(id),
+         {:ok, activity} <- CommonAPI.add_mute(user, activity) do
+      try_render(conn, "show.json", activity: activity, for: user, as: :activity)
+    end
+  end
+
+  @doc "POST /api/v1/statuses/:id/unmute"
+  def unmute_conversation(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with %Activity{} = activity <- Activity.get_by_id(id),
+         {:ok, activity} <- CommonAPI.remove_mute(user, activity) do
+      try_render(conn, "show.json", activity: activity, for: user, as: :activity)
+    end
+  end
+
+  @doc "GET /api/v1/statuses/:id/card"
+  @deprecated "https://github.com/tootsuite/mastodon/pull/11213"
+  def card(%{assigns: %{user: user}} = conn, %{"id" => status_id}) do
+    with %Activity{} = activity <- Activity.get_by_id(status_id),
+         true <- Visibility.visible_for_user?(activity, user) do
+      data = Pleroma.Web.RichMedia.Helpers.fetch_data_for_activity(activity)
+      render(conn, "card.json", data)
+    else
+      _ -> render_error(conn, :not_found, "Record not found")
+    end
+  end
+
+  @doc "GET /api/v1/statuses/:id/favourited_by"
+  def favourited_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
+         {:visible, true} <- {:visible, Visibility.visible_for_user?(activity, user)},
+         %Object{data: %{"likes" => likes}} <- Object.normalize(activity) do
+      users =
+        User
+        |> Ecto.Query.where([u], u.ap_id in ^likes)
+        |> Repo.all()
+        |> Enum.filter(&(not User.blocks?(user, &1)))
+
+      conn
+      |> put_view(AccountView)
+      |> render("index.json", for: user, users: users, as: :user)
+    else
+      {:visible, false} -> {:error, :not_found}
+      _ -> json(conn, [])
+    end
+  end
+
+  @doc "GET /api/v1/statuses/:id/reblogged_by"
+  def reblogged_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
+         {:visible, true} <- {:visible, Visibility.visible_for_user?(activity, user)},
+         %Object{data: %{"announcements" => announces}} <- Object.normalize(activity) do
+      users =
+        User
+        |> Ecto.Query.where([u], u.ap_id in ^announces)
+        |> Repo.all()
+        |> Enum.filter(&(not User.blocks?(user, &1)))
+
+      conn
+      |> put_view(AccountView)
+      |> render("index.json", for: user, users: users, as: :user)
+    else
+      {:visible, false} -> {:error, :not_found}
+      _ -> json(conn, [])
+    end
+  end
+
+  @doc "GET /api/v1/statuses/:id/context"
+  def context(%{assigns: %{user: user}} = conn, %{"id" => id}) do
+    with %Activity{} = activity <- Activity.get_by_id(id) do
+      activities =
+        ActivityPub.fetch_activities_for_context(activity.data["context"], %{
+          "blocking_user" => user,
+          "user" => user,
+          "exclude_id" => activity.id
+        })
+
+      render(conn, "context.json", activity: activity, activities: activities, user: user)
+    end
+  end
+end

lib/pleroma/web/mastodon_api/controllers/timeline_controller.ex

@@ -0,0 +1,136 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.MastodonAPI.TimelineController do
+  use Pleroma.Web, :controller
+
+  import Pleroma.Web.ControllerHelper,
+    only: [add_link_headers: 2, add_link_headers: 3, truthy_param?: 1]
+
+  alias Pleroma.Pagination
+  alias Pleroma.Web.ActivityPub.ActivityPub
+
+  plug(:put_view, Pleroma.Web.MastodonAPI.StatusView)
+
+  # GET /api/v1/timelines/home
+  def home(%{assigns: %{user: user}} = conn, params) do
+    params =
+      params
+      |> Map.put("type", ["Create", "Announce"])
+      |> Map.put("blocking_user", user)
+      |> Map.put("muting_user", user)
+      |> Map.put("user", user)
+
+    recipients = [user.ap_id | user.following]
+
+    activities =
+      recipients
+      |> ActivityPub.fetch_activities(params)
+      |> Enum.reverse()
+
+    conn
+    |> add_link_headers(activities)
+    |> render("index.json", activities: activities, for: user, as: :activity)
+  end
+
+  # GET /api/v1/timelines/direct
+  def direct(%{assigns: %{user: user}} = conn, params) do
+    params =
+      params
+      |> Map.put("type", "Create")
+      |> Map.put("blocking_user", user)
+      |> Map.put("user", user)
+      |> Map.put(:visibility, "direct")
+
+    activities =
+      [user.ap_id]
+      |> ActivityPub.fetch_activities_query(params)
+      |> Pagination.fetch_paginated(params)
+
+    conn
+    |> add_link_headers(activities)
+    |> render("index.json", activities: activities, for: user, as: :activity)
+  end
+
+  # GET /api/v1/timelines/public
+  def public(%{assigns: %{user: user}} = conn, params) do
+    local_only = truthy_param?(params["local"])
+
+    activities =
+      params
+      |> Map.put("type", ["Create", "Announce"])
+      |> Map.put("local_only", local_only)
+      |> Map.put("blocking_user", user)
+      |> Map.put("muting_user", user)
+      |> ActivityPub.fetch_public_activities()
+      |> Enum.reverse()
+
+    conn
+    |> add_link_headers(activities, %{"local" => local_only})
+    |> render("index.json", activities: activities, for: user, as: :activity)
+  end
+
+  # GET /api/v1/timelines/tag/:tag
+  def hashtag(%{assigns: %{user: user}} = conn, params) do
+    local_only = truthy_param?(params["local"])
+
+    tags =
+      [params["tag"], params["any"]]
+      |> List.flatten()
+      |> Enum.uniq()
+      |> Enum.filter(& &1)
+      |> Enum.map(&String.downcase(&1))
+
+    tag_all =
+      params
+      |> Map.get("all", [])
+      |> Enum.map(&String.downcase(&1))
+
+    tag_reject =
+      params
+      |> Map.get("none", [])
+      |> Enum.map(&String.downcase(&1))
+
+    activities =
+      params
+      |> Map.put("type", "Create")
+      |> Map.put("local_only", local_only)
+      |> Map.put("blocking_user", user)
+      |> Map.put("muting_user", user)
+      |> Map.put("user", user)
+      |> Map.put("tag", tags)
+      |> Map.put("tag_all", tag_all)
+      |> Map.put("tag_reject", tag_reject)
+      |> ActivityPub.fetch_public_activities()
+      |> Enum.reverse()
+
+    conn
+    |> add_link_headers(activities, %{"local" => local_only})
+    |> render("index.json", activities: activities, for: user, as: :activity)
+  end
+
+  # GET /api/v1/timelines/list/:list_id
+  def list(%{assigns: %{user: user}} = conn, %{"list_id" => id} = params) do
+    with %Pleroma.List{title: _title, following: following} <- Pleroma.List.get(id, user) do
+      params =
+        params
+        |> Map.put("type", "Create")
+        |> Map.put("blocking_user", user)
+        |> Map.put("user", user)
+        |> Map.put("muting_user", user)
+
+      # we must filter the following list for the user to avoid leaking statuses the user
+      # does not actually have permission to see (for more info, peruse security issue #270).
+      activities =
+        following
+        |> Enum.filter(fn x -> x in user.following end)
+        |> ActivityPub.fetch_activities_bounded(following, params)
+        |> Enum.reverse()
+
+      render(conn, "index.json", activities: activities, for: user, as: :activity)
+    else
+      _e -> render_error(conn, :forbidden, "Error.")
+    end
+  end
+end

lib/pleroma/web/mastodon_api/views/account_view.ex

@@ -11,15 +11,15 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
   alias Pleroma.Web.MastodonAPI.AccountView
   alias Pleroma.Web.MediaProxy
 
-  def render("accounts.json", %{users: users} = opts) do
+  def render("index.json", %{users: users} = opts) do
     users
-    |> render_many(AccountView, "account.json", opts)
+    |> render_many(AccountView, "show.json", opts)
     |> Enum.filter(&Enum.any?/1)
   end
 
-  def render("account.json", %{user: user} = opts) do
+  def render("show.json", %{user: user} = opts) do
     if User.visible_for?(user, opts[:for]),
-      do: do_render("account.json", opts),
+      do: do_render("show.json", opts),
       else: %{}
   end
 
@@ -66,7 +66,7 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
     render_many(targets, AccountView, "relationship.json", user: user, as: :target)
   end
 
-  defp do_render("account.json", %{user: user} = opts) do
+  defp do_render("show.json", %{user: user} = opts) do
     display_name = HTML.strip_tags(user.name || user.nickname)
 
     image = User.avatar_url(user) |> MediaProxy.url()
@@ -74,10 +74,18 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
     user_info = User.get_cached_user_info(user)
 
     following_count =
-      ((!user.info.hide_follows or opts[:for] == user) && user_info.following_count) || 0
+      if !user.info.hide_follows_count or !user.info.hide_follows or opts[:for] == user do
+        user_info.following_count
+      else
+        0
+      end
 
     followers_count =
-      ((!user.info.hide_followers or opts[:for] == user) && user_info.follower_count) || 0
+      if !user.info.hide_followers_count or !user.info.hide_followers or opts[:for] == user do
+        user_info.follower_count
+      else
+        0
+      end
 
     bot = (user.info.source_data["type"] || "Person") in ["Application", "Service"]
 
@@ -108,6 +116,8 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
     bio = HTML.filter_tags(user.bio, User.html_filter_policy(opts[:for]))
     relationship = render("relationship.json", %{user: opts[:for], target: user})
 
+    discoverable = user.info.discoverable
+
     %{
       id: to_string(user.id),
       username: username_from_nickname(user.nickname),
@@ -131,13 +141,17 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
         note: HTML.strip_tags((user.bio || "") |> String.replace("<br>", "\n")),
         sensitive: false,
         fields: raw_fields,
-        pleroma: %{}
+        pleroma: %{
+          discoverable: discoverable
+        }
       },
 
       # Pleroma extension
       pleroma: %{
         confirmation_pending: user_info.confirmation_pending,
         tags: user.tags,
+        hide_followers_count: user.info.hide_followers_count,
+        hide_follows_count: user.info.hide_follows_count,
         hide_followers: user.info.hide_followers,
         hide_follows: user.info.hide_follows,
         hide_favorites: user.info.hide_favorites,
@@ -152,6 +166,7 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
     |> maybe_put_settings_store(user, opts[:for], opts)
     |> maybe_put_chat_token(user, opts[:for], opts)
     |> maybe_put_activation_status(user, opts[:for])
+    |> maybe_put_follow_requests_count(user, opts[:for])
   end
 
   defp username_from_nickname(string) when is_binary(string) do
@@ -160,6 +175,21 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
 
   defp username_from_nickname(_), do: nil
 
+  defp maybe_put_follow_requests_count(
+         data,
+         %User{id: user_id} = user,
+         %User{id: user_id}
+       ) do
+    count =
+      User.get_follow_requests(user)
+      |> length()
+
+    data
+    |> Kernel.put_in([:follow_requests_count], count)
+  end
+
+  defp maybe_put_follow_requests_count(data, _, _), do: data
+
   defp maybe_put_settings(
          data,
          %User{id: user_id} = user,

lib/pleroma/web/mastodon_api/views/conversation_view.ex

@@ -11,6 +11,10 @@ defmodule Pleroma.Web.MastodonAPI.ConversationView do
   alias Pleroma.Web.MastodonAPI.AccountView
   alias Pleroma.Web.MastodonAPI.StatusView
 
+  def render("participations.json", %{participations: participations, for: user}) do
+    render_many(participations, __MODULE__, "participation.json", as: :participation, for: user)
+  end
+
   def render("participation.json", %{participation: participation, for: user}) do
     participation = Repo.preload(participation, conversation: [], recipients: [])
 
@@ -23,25 +27,14 @@ defmodule Pleroma.Web.MastodonAPI.ConversationView do
       end
 
     activity = Activity.get_by_id_with_object(last_activity_id)
-
-    last_status = StatusView.render("status.json", %{activity: activity, for: user})
-
     # Conversations return all users except the current user.
-    users =
+    users = Enum.reject(participation.recipients, &(&1.id == user.id))
-      participation.recipients
-      |> Enum.reject(&(&1.id == user.id))
-
-    accounts =
-      AccountView.render("accounts.json", %{
-        users: users,
-        as: :user
-      })
 
     %{
       id: participation.id |> to_string(),
-      accounts: accounts,
+      accounts: render(AccountView, "index.json", users: users, as: :user),
       unread: !participation.read,
-      last_status: last_status
+      last_status: render(StatusView, "show.json", activity: activity, for: user)
     }
   end
 end

lib/pleroma/web/mastodon_api/views/notification_view.ex

@@ -29,7 +29,7 @@ defmodule Pleroma.Web.MastodonAPI.NotificationView do
       id: to_string(notification.id),
       type: mastodon_type,
       created_at: CommonAPI.Utils.to_masto_date(notification.inserted_at),
-      account: AccountView.render("account.json", %{user: actor, for: user}),
+      account: AccountView.render("show.json", %{user: actor, for: user}),
       pleroma: %{
         is_seen: notification.seen
       }
@@ -39,19 +39,19 @@ defmodule Pleroma.Web.MastodonAPI.NotificationView do
       "mention" ->
         response
         |> Map.merge(%{
-          status: StatusView.render("status.json", %{activity: activity, for: user})
+          status: StatusView.render("show.json", %{activity: activity, for: user})
         })
 
       "favourite" ->
         response
         |> Map.merge(%{
-          status: StatusView.render("status.json", %{activity: parent_activity, for: user})
+          status: StatusView.render("show.json", %{activity: parent_activity, for: user})
         })
 
       "reblog" ->
         response
         |> Map.merge(%{
-          status: StatusView.render("status.json", %{activity: parent_activity, for: user})
+          status: StatusView.render("show.json", %{activity: parent_activity, for: user})
         })
 
       "follow" ->

lib/pleroma/web/mastodon_api/views/report_view.ex

@@ -5,7 +5,7 @@
 defmodule Pleroma.Web.MastodonAPI.ReportView do
   use Pleroma.Web, :view
 
-  def render("report.json", %{activity: activity}) do
+  def render("show.json", %{activity: activity}) do
     %{
       id: to_string(activity.id),
       action_taken: false

lib/pleroma/web/mastodon_api/views/scheduled_activity_view.ex

@@ -7,11 +7,10 @@ defmodule Pleroma.Web.MastodonAPI.ScheduledActivityView do
 
   alias Pleroma.ScheduledActivity
   alias Pleroma.Web.CommonAPI
-  alias Pleroma.Web.MastodonAPI.ScheduledActivityView
   alias Pleroma.Web.MastodonAPI.StatusView
 
   def render("index.json", %{scheduled_activities: scheduled_activities}) do
-    render_many(scheduled_activities, ScheduledActivityView, "show.json")
+    render_many(scheduled_activities, __MODULE__, "show.json")
   end
 
   def render("show.json", %{scheduled_activity: %ScheduledActivity{} = scheduled_activity}) do
@@ -24,12 +23,8 @@ defmodule Pleroma.Web.MastodonAPI.ScheduledActivityView do
   end
 
   defp with_media_attachments(data, %{params: %{"media_attachments" => media_attachments}}) do
-    try do
+    attachments = render_many(media_attachments, StatusView, "attachment.json", as: :attachment)
-      attachments = render_many(media_attachments, StatusView, "attachment.json", as: :attachment)
+    Map.put(data, :media_attachments, attachments)
-      Map.put(data, :media_attachments, attachments)
-    rescue
-      _ -> data
-    end
   end
 
   defp with_media_attachments(data, _), do: data
@@ -45,13 +40,9 @@ defmodule Pleroma.Web.MastodonAPI.ScheduledActivityView do
       in_reply_to_id: params["in_reply_to_id"]
     }
 
-    data =
+    case params["media_ids"] do
-      if media_ids = params["media_ids"] do
+      nil -> data
-        Map.put(data, :media_ids, media_ids)
+      media_ids -> Map.put(data, :media_ids, media_ids)
-      else
+    end
-        data
-      end
-
-    data
   end
 end

lib/pleroma/web/mastodon_api/views/status_view.ex

@@ -73,17 +73,13 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
 
   def render("index.json", opts) do
     replied_to_activities = get_replied_to_activities(opts.activities)
+    opts = Map.put(opts, :replied_to_activities, replied_to_activities)
 
-    opts.activities
+    safe_render_many(opts.activities, StatusView, "show.json", opts)
-    |> safe_render_many(
-      StatusView,
-      "status.json",
-      Map.put(opts, :replied_to_activities, replied_to_activities)
-    )
   end
 
   def render(
-        "status.json",
+        "show.json",
         %{activity: %{data: %{"type" => "Announce", "object" => _object}} = activity} = opts
       ) do
     user = get_user(activity.data["actor"])
@@ -96,7 +92,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
       |> Activity.with_set_thread_muted_field(opts[:for])
       |> Repo.one()
 
-    reblogged = render("status.json", Map.put(opts, :activity, reblogged_activity))
+    reblogged = render("show.json", Map.put(opts, :activity, reblogged_activity))
 
     favorited = opts[:for] && opts[:for].ap_id in (activity_object.data["likes"] || [])
 
@@ -112,7 +108,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
       id: to_string(activity.id),
       uri: activity_object.data["id"],
       url: activity_object.data["id"],
-      account: AccountView.render("account.json", %{user: user, for: opts[:for]}),
+      account: AccountView.render("show.json", %{user: user, for: opts[:for]}),
       in_reply_to_id: nil,
       in_reply_to_account_id: nil,
       reblog: reblogged,
@@ -144,7 +140,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     }
   end
 
-  def render("status.json", %{activity: %{data: %{"object" => _object}} = activity} = opts) do
+  def render("show.json", %{activity: %{data: %{"object" => _object}} = activity} = opts) do
     object = Object.normalize(activity)
 
     user = get_user(activity.data["actor"])
@@ -262,7 +258,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
       id: to_string(activity.id),
       uri: object.data["id"],
       url: url,
-      account: AccountView.render("account.json", %{user: user, for: opts[:for]}),
+      account: AccountView.render("show.json", %{user: user, for: opts[:for]}),
       in_reply_to_id: reply_to && to_string(reply_to.id),
       in_reply_to_account_id: reply_to_user && to_string(reply_to_user.id),
       reblog: nil,
@@ -303,7 +299,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     }
   end
 
-  def render("status.json", _) do
+  def render("show.json", _) do
     nil
   end
 
@@ -343,9 +339,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     }
   end
 
-  def render("card.json", _) do
+  def render("card.json", _), do: nil
-    nil
-  end
 
   def render("attachment.json", %{attachment: attachment}) do
     [attachment_url | _] = attachment["url"]
@@ -374,6 +368,27 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     }
   end
 
+  def render("listen.json", %{activity: %Activity{data: %{"type" => "Listen"}} = activity} = opts) do
+    object = Object.normalize(activity)
+
+    user = get_user(activity.data["actor"])
+    created_at = Utils.to_masto_date(activity.data["published"])
+
+    %{
+      id: activity.id,
+      account: AccountView.render("show.json", %{user: user, for: opts[:for]}),
+      created_at: created_at,
+      title: object.data["title"] |> HTML.strip_tags(),
+      artist: object.data["artist"] |> HTML.strip_tags(),
+      album: object.data["album"] |> HTML.strip_tags(),
+      length: object.data["length"]
+    }
+  end
+
+  def render("listens.json", opts) do
+    safe_render_many(opts.activities, StatusView, "listen.json", opts)
+  end
+
   def render("poll.json", %{object: object} = opts) do
     {multiple, options} =
       case object.data do
@@ -443,6 +458,20 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     end
   end
 
+  def render("context.json", %{activity: activity, activities: activities, user: user}) do
+    %{ancestors: ancestors, descendants: descendants} =
+      activities
+      |> Enum.reverse()
+      |> Enum.group_by(fn %{id: id} -> if id < activity.id, do: :ancestors, else: :descendants end)
+      |> Map.put_new(:ancestors, [])
+      |> Map.put_new(:descendants, [])
+
+    %{
+      ancestors: render("index.json", for: user, activities: ancestors, as: :activity),
+      descendants: render("index.json", for: user, activities: descendants, as: :activity)
+    }
+  end
+
   def get_reply_to(activity, %{replied_to_activities: replied_to_activities}) do
     object = Object.normalize(activity)
 

lib/pleroma/web/metadata/utils.ex

@@ -3,6 +3,7 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.Metadata.Utils do
+  alias Pleroma.Emoji
   alias Pleroma.Formatter
   alias Pleroma.HTML
   alias Pleroma.Web.MediaProxy
@@ -13,7 +14,7 @@ defmodule Pleroma.Web.Metadata.Utils do
     |> HtmlEntities.decode()
     |> String.replace(~r/<br\s?\/?>/, " ")
     |> HTML.get_cached_stripped_html_for_activity(object, "metadata")
-    |> Formatter.demojify()
+    |> Emoji.Formatter.demojify()
     |> Formatter.truncate()
   end
 
@@ -23,7 +24,7 @@ defmodule Pleroma.Web.Metadata.Utils do
     |> HtmlEntities.decode()
     |> String.replace(~r/<br\s?\/?>/, " ")
     |> HTML.strip_tags()
-    |> Formatter.demojify()
+    |> Emoji.Formatter.demojify()
     |> Formatter.truncate(max_length)
   end
 

lib/pleroma/web/nodeinfo/nodeinfo_controller.ex

@@ -57,6 +57,7 @@ defmodule Pleroma.Web.Nodeinfo.NodeinfoController do
         "mastodon_api_streaming",
         "polls",
         "pleroma_explicit_addressing",
+        "shareable_emoji_packs",
         if Config.get([:media_proxy, :enabled]) do
           "media_proxy"
         end,

lib/pleroma/web/oauth/app.ex

@@ -5,6 +5,7 @@
 defmodule Pleroma.Web.OAuth.App do
   use Ecto.Schema
   import Ecto.Changeset
+  alias Pleroma.Repo
 
   @type t :: %__MODULE__{}
 
@@ -39,4 +40,29 @@ defmodule Pleroma.Web.OAuth.App do
       changeset
     end
   end
+
+  @doc """
+  Gets app by attrs or create new  with attrs.
+  And updates the scopes if need.
+  """
+  @spec get_or_make(map(), list(String.t())) :: {:ok, App.t()} | {:error, Ecto.Changeset.t()}
+  def get_or_make(attrs, scopes) do
+    with %__MODULE__{} = app <- Repo.get_by(__MODULE__, attrs) do
+      update_scopes(app, scopes)
+    else
+      _e ->
+        %__MODULE__{}
+        |> register_changeset(Map.put(attrs, :scopes, scopes))
+        |> Repo.insert()
+    end
+  end
+
+  defp update_scopes(%__MODULE__{} = app, []), do: {:ok, app}
+  defp update_scopes(%__MODULE__{scopes: scopes} = app, scopes), do: {:ok, app}
+
+  defp update_scopes(%__MODULE__{} = app, scopes) do
+    app
+    |> change(%{scopes: scopes})
+    |> Repo.update()
+  end
 end

lib/pleroma/web/oauth/authorization.ex

@@ -20,7 +20,7 @@ defmodule Pleroma.Web.OAuth.Authorization do
     field(:scopes, {:array, :string}, default: [])
     field(:valid_until, :naive_datetime_usec)
     field(:used, :boolean, default: false)
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     belongs_to(:app, App)
 
     timestamps()

lib/pleroma/web/oauth/oauth_controller.ex

@@ -203,6 +203,8 @@ defmodule Pleroma.Web.OAuth.OAuthController do
          {:ok, app} <- Token.Utils.fetch_app(conn),
          {:auth_active, true} <- {:auth_active, User.auth_active?(user)},
          {:user_active, true} <- {:user_active, !user.info.deactivated},
+         {:password_reset_pending, false} <-
+           {:password_reset_pending, user.info.password_reset_pending},
          {:ok, scopes} <- validate_scopes(app, params),
          {:ok, auth} <- Authorization.create_authorization(app, user, scopes),
          {:ok, token} <- Token.exchange_token(app, auth) do
@@ -216,6 +218,9 @@ defmodule Pleroma.Web.OAuth.OAuthController do
       {:user_active, false} ->
         render_error(conn, :forbidden, "Your account is currently disabled")
 
+      {:password_reset_pending, true} ->
+        render_error(conn, :forbidden, "Password reset is required")
+
       _error ->
         render_invalid_credentials_error(conn)
     end

lib/pleroma/web/oauth/token.ex

@@ -21,7 +21,7 @@ defmodule Pleroma.Web.OAuth.Token do
     field(:refresh_token, :string)
     field(:scopes, {:array, :string}, default: [])
     field(:valid_until, :naive_datetime_usec)
-    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:user, User, type: FlakeId.Ecto.CompatType)
     belongs_to(:app, App)
 
     timestamps()

lib/pleroma/web/oauth/token/clean_worker.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.OAuth.Token.CleanWorker do

lib/pleroma/web/oauth/token/query.ex

@@ -1,5 +1,5 @@
 # Pleroma: A lightweight social networking server
-# Copyright © 2017-2018 Pleroma Authors <https://pleroma.social/>
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.OAuth.Token.Query do