Commit graph

115 commits

Author SHA1 Message Date
Eugen Rochko
f52c988e12
Add announcements (#12662)
* Add announcements

Fix #11006

* Add reactions to announcements

* Add admin UI for announcements

* Add unit tests

* Fix issues

- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"

* Fix scheduler unpublishing announcements before they are due

* Fix filter params not being passed to announcements filter
2020-01-23 22:00:13 +01:00
Yamagishi Kazutoshi
172eaeba3f Add config of multipart threshold for S3 (#11924) 2019-09-23 15:37:45 +02:00
Eugen Rochko
9361981388
Add missing locale file for ga and add rake task to check for it (#11813)
* Add missing locale file for ga and add rake task to check for it

* Update lib/tasks/repo.rake

Co-Authored-By: Yamagishi Kazutoshi <ykzts@desire.sh>

* Fix check-i18n build
2019-09-12 04:58:33 +02:00
Eugen Rochko
9fd6443487
Add translators to AUTHORS.md (#9865) 2019-01-19 22:01:12 +01:00
Eugen Rochko
7c3fddeab4
Change CHANGELOG.md references from pull requests to authors (#9864)
Add `rake repo:changelog` to do that automatically
2019-01-19 16:25:20 +01:00
Eugen Rochko
2ee779dcd3
Update emoji codepoint mappings to v11.0 (#9618) 2018-12-24 19:12:00 +01:00
Eugen Rochko
186024a058
Add tootctl settings registrations open (#8829) 2018-10-04 16:05:54 +02:00
Eugen Rochko
6a3f9b7e53
Move more tasks to tootctl (#8675)
* Move more tasks to tootctl

- tootctl feeds build
- tootctl feeds clear
- tootctl accounts refresh

Clean up exit codes and help messages

* Move user modifying to tootctl

* Improve user modification through CLI, rename commands

add -> create
mod -> modify
del -> delete

To remove ambiguity

* Fix code style issues

* Fix not being able to unset admin/mod role
2018-09-14 17:42:22 +02:00
rinsuki
25dd523887 using mailto scheme in AUTHORS.md (#8663) 2018-09-10 02:01:03 +02:00
nightpool
b0b484ba18 Add rake task for generating AUTHORS.md (#8661)
* add rake task for generating AUTHORS.md

* update AUTHORS.md

* rubocop
2018-09-10 01:31:42 +02:00
Quint Guvernator
da13fa5021 Fix low-hanging rubocop gripes (#8458)
* rubocop: quit being so picky

* rubocop: miscellany

* rubocop: prefer present to blank
2018-08-26 19:22:46 +02:00
Eugen Rochko
b9c3788485
Fix NUM_DAYS in rake mastodon:media:remove_remote and print warning (#8462)
Remove old deprecated rake tasks
2018-08-26 17:19:47 +02:00
M Somerville
2bba6e582d Rename S3_CLOUDFRONT_HOST to S3_ALIAS_HOST. (#8423)
Still check for S3_CLOUDFRONT_HOST for existing installs.
2018-08-25 13:27:08 +02:00
Eugen Rochko
793eea2982
Add improved CLI interface for removing remote media (#8411)
./bin/tootctl media remove --days 7 --background

Make the old rake task point to it
2018-08-25 13:25:39 +02:00
Eugen Rochko
d98de8ada7
Get rid of all batch order warnings (#8334) 2018-08-21 12:25:50 +02:00
ThibG
110b3f6335 Add some feedback to maintenance rake tasks (#8173) 2018-08-11 22:02:55 +02:00
Eugen Rochko
c7405fda11
Fix missing column in select in mastodon:feeds:build task (#7720) 2018-06-04 04:21:15 +02:00
Eugen Rochko
ad40bf5e0c
Speed up some rake tasks by moving execution to Sidekiq (#7678)
* Speed up some rake tasks by moving execution to Sidekiq

mastodon:media:remove_silenced
mastodon:media:remove_remote
mastodon:media:redownload_avatars
mastodon:feeds:build

* Fix code style issue
2018-05-30 21:09:30 +02:00
Eugen Rochko
cb5b5cb5f7
Slightly reduce RAM usage (#7301)
* No need to re-require sidekiq plugins, they are required via Gemfile

* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile

* Replace ruby-oembed with FetchOEmbedService

Reduce startup by 45382 allocated objects

* Remove preloaded JSON-LD in favour of caching HTTP responses

Reduce boot RAM by about 6 MiB

* Fix tests

* Fix test suite by stubbing out JSON-LD contexts
2018-05-02 18:58:48 +02:00
Alejandro Martinez Ruiz
06817b3c1f tasks/mastodon: fix prompt for Redis password (#7241) 2018-04-23 16:03:58 +02:00
David Baucum
ca9192d9ba Ability to specify Redis passwd on mastodon:setup (#7222)
Closes #7221
2018-04-22 11:49:16 +02:00
Eugen Rochko
36eac8ba90
Do not default SMTP verify mode to "peer", default to "none" (#6996) 2018-04-02 19:19:51 +02:00
Eugen Rochko
2a90da1837
Fix UniqueUsernameValidator comparison (#6926)
Comparison was downcasing only one side, therefore if previously
existing account had a non-lowercase spelling, it would be ignored
when checking for duplicates.

New rake task `mastodon:maintenance:find_duplicate_usernames` will
help find constraint violations that might have occured from the
presence of this bug.

Bump version to 2.3.3
2018-03-27 04:33:57 +02:00
Akihiko Odaki
54b273bf99 Close http connection in perform method of Request class (#6889)
HTTP connections must be explicitly closed in many cases, and letting
perform method close connections makes its callers less redundant and
prevent them from forgetting to close connections.
2018-03-24 12:49:54 +01:00
Akihiko Odaki
338bff8b93 Correct the reference to user's password in mastodon:add_user task (#6800) 2018-03-17 13:27:19 +01:00
Ushitora Anqou
051b649628 Detailed SMTP setup (#6759)
* add detailed SMTP settings setup in mastodon:setup

* add localhost SMTP settings setup in mastodon:setup

* SMTP settings setup should exit after successful delivery of test mail
2018-03-12 21:41:26 +01:00
艮 鮟鱇
f5f165a5eb set SAFETY_ASSURED=1 of db:setup in mastodon:setup (#6758) 2018-03-12 16:21:48 +01:00
Eugen Rochko
e5f18ace2a
When inside Docker, output saved configuration during mastodon:setup (#6711) 2018-03-09 11:52:18 +01:00
Eugen Rochko
64db9ed5f6
After blocking domain with reject_media, invalidate cache (#6679)
Media attachments are part of the association cache of statuses,
since they are presumed to be immutable. Unless this cache is
cleared manually, the statuses will continue to look like they
have media embedded.
2018-03-08 06:59:42 +01:00
Akihiko Odaki
778b37790b Do not fetch environment variables to determine default locale (#6618)
The default locale is now set by config.
2018-03-04 10:00:46 +01:00
Thomas Leister
ecd36c1ede Fixes #6584 (#6585) 2018-03-01 23:30:06 +01:00
Paul Woolcock
76198c63b6 Some images can cause convert to fail, which crashes this whole task (#6565)
* Some images can cause `convert` to fail, which crashes this whole task

* Add more specific exception
2018-02-26 22:01:49 +01:00
Eugen Rochko
5cc716688a
Ensure the app does not even start if OTP_SECRET is not set (#6557)
* Ensure the app does not even start if OTP_SECRET is not set

* Remove PAPERCLIP_SECRET (it's not used by anything, actually)

Imports are for internal consumption and the url option isn't even
used correctly, so we can remove the hash stuff from them
2018-02-26 01:31:44 +01:00
takayamaki
f69d7cb43b fix purge_removed_accounts task should suspend account before delete it (#6521) 2018-02-21 16:30:46 +01:00
Eugen Rochko
cf36d184f4
Interactive rake mastodon:setup task (#6451)
* Add better CLI prompt

* Add rake mastodon:setup interactive wizard

* Test db/redis/smtp configurations and add admin user at the end

* Test database connection even when database does not exist yet
2018-02-11 18:40:57 +01:00
Akihiko Odaki
f7bf36d8fc Require environment for generate_static_pages (#6420)
It is required for ApplicationController.
2018-02-03 18:41:01 +01:00
Eugen Rochko
9b3b40df66
Fix regeneration marker not expiring (#6290)
* Fix regeneration key not getting expired

* Add rake task to remove old regeneration markers
2018-01-18 20:29:56 +01:00
Eugen Rochko
ed867eca9d
Move e-mail digest task to sidekiq, reduce workload, improve hint (#6252) 2018-01-15 04:34:28 +01:00
Eugen Rochko
cafe27fb29
Add rake task to check and purge accounts that are missing in origin (#6085)
* Add rake task to check and purge accounts that are missing in origin

* Add progress bar and --force options to mastodon:maintenance:purge_removed_accounts
2017-12-24 16:14:33 +01:00
Akihiko Odaki
c083816c24 Add embed_url to preview cards (#5775) 2017-12-07 03:37:43 +01:00
Eugen Rochko
7bb8b0b2fc
Add moderator role and add pundit policies for admin actions (#5635)
* Add moderator role and add pundit policies for admin actions

* Add rake task for turning user into mod and revoking it again

* Fix handling of unauthorized exception

* Deliver new report e-mails to staff, not just admins

* Add promote/demote to admin UI, hide some actions conditionally

* Fix unused i18n
2017-11-11 20:23:33 +01:00
Naoki Kosaka
4c0a85ef9b In remove_remote, exclude removed media attachments. (#5626) 2017-11-08 15:19:22 +01:00
Eugen Rochko
92e7815d1d Fix #5274 - Create symlink from public/500.html to public/assets/500.html (#5288) 2017-10-09 20:51:24 +02:00
Eugen Rochko
0717d9b3e6 Set snowflake IDs for backdated statuses (#5260)
- Rename Mastodon::TimestampIds into Mastodon::Snowflake for clarity
- Skip for statuses coming from inbox, aka delivered in real-time
- Skip for statuses that claim to be from the future
2017-10-08 17:34:34 +02:00
Nolan Lawson
fd7f0732fe Compress and combine emoji data (#5229) 2017-10-06 03:42:34 +02:00
Eugen Rochko
eb5ac23434 Clean up code style of Mastodon::TimestampId module (#5232)
* Clean up code style of Mastodon::TimestampId module

* Update brakeman config
2017-10-06 03:42:21 +02:00
aschmitz
468523f4ad Non-Serial ("Snowflake") IDs (#4801)
* Use non-serial IDs

This change makes a number of nontrivial tweaks to the data model in
Mastodon:

* All IDs are now 8 byte integers (rather than mixed 4- and 8-byte)
* IDs are now assigned as:
  * Top 6 bytes: millisecond-resolution time from epoch
  * Bottom 2 bytes: serial (within the millisecond) sequence number
  * See /lib/tasks/db.rake's `define_timestamp_id` for details, but
    note that the purpose of these changes is to make it difficult to
    determine the number of objects in a table from the ID of any
    object.
* The Redis sorted set used for the feed will have values used to look
  up toots, rather than scores. This is almost always the same as the
  existing behavior, except in the case of boosted toots. This change
  was made because Redis stores scores as double-precision floats,
  which cannot store the new ID format exactly. Note that this doesn't
  cause problems with sorting/pagination, because ZREVRANGEBYSCORE
  sorts lexicographically when scores are tied. (This will still cause
  sorting issues when the ID gains a new significant digit, but that's
  extraordinarily uncommon.)

Note a couple of tradeoffs have been made in this commit:

* lib/tasks/db.rake is used to enforce many/most column constraints,
  because this commit seems likely to take a while to bring upstream.
  Enforcing a post-migrate hook is an easier way to maintain the code
  in the interim.
* Boosted toots will appear in the timeline as many times as they have
  been boosted. This is a tradeoff due to the way the feed is saved in
  Redis at the moment, but will be handled by a future commit.

This would effectively close Mastodon's #1059, as it is a
snowflake-like system of generating IDs. However, given how involved
the changes were simply within Mastodon, it may have unexpected
interactions with some clients, if they store IDs as doubles
(or as 4-byte integers). This was a problem that Twitter ran into with
their "snowflake" transition, particularly in JavaScript clients that
treated IDs as JS integers, rather than strings. It therefore would be
useful to test these changes at least in the web interface and popular
clients before pushing them to all users.

* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Restructure feed pushes/unpushes

This was necessary because the previous behavior used Redis zset scores
to identify statuses, but those are IEEE double-precision floats, so we
can't actually use them to identify all 64-bit IDs. However, it leaves
the code in a much better state for refactoring reblog handling /
coalescing.

Feed-management code has been consolidated in FeedManager, including:

* BatchedRemoveStatusService no longer directly manipulates feed zsets
* RemoveStatusService no longer directly manipulates feed zsets
* PrecomputeFeedService has moved its logic to FeedManager#populate_feed

(PrecomputeFeedService largely made lots of calls to FeedManager, but
didn't follow the normal adding-to-feed process.)

This has the effect of unifying all of the feed push/unpush logic in
FeedManager, making it much more tractable to update it in the future.

Due to some additional checks that must be made during, for example,
batch status removals, some Redis pipelining has been removed. It does
not appear that this should cause significantly increased load, but if
necessary, some optimizations are possible in batch cases. These were
omitted in the pursuit of simplicity, but a batch_push and batch_unpush
would be possible in the future.

Tests were added to verify that pushes happen under expected conditions,
and to verify reblog behavior (both on pushing and unpushing). In the
case of unpushing, this includes testing behavior that currently leads
to confusion such as Mastodon's #2817, but this codifies that the
behavior is currently expected.

* Rubocop fixes

I could swear I made these changes already, but I must have lost them
somewhere along the line.

* Address review comments

This addresses the first two comments from review of this feature:

https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336735
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336931

This adds an optional argument to FeedManager#key, the subtype of feed
key to generate. It also tests to ensure that FeedManager's settings are
such that reblogs won't be tracked forever.

* Hardcode IdToBigints migration columns

This addresses a comment during review:
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139337452

This means we'll need to make sure that all _id columns going forward
are bigints, but that should happen automatically in most cases.

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Only implement timestamp IDs for Status IDs

Per discussion in #4801, this is only being merged in for Status IDs at
this point. We do this in a migration, as there is no longer use for
a post-migration hook. We keep the initialization of the timestamp_id
function as a Rake task, as it is also needed after db:schema:load (as
db/schema.rb doesn't store Postgres functions).

* Change internal streaming payloads to stringified IDs as well

This is equivalent to 591a9af356faf2d5c7e66e3ec715502796c875cd from
#5019, with an extra change for the addition to FeedManager#unpush.

* Ensure we have a status_id_seq sequence

Apparently this is not a given when specifying a custom ID function,
so now we ensure it gets created. This uses the generic version of this
function to more easily support adding additional tables with timestamp
IDs in the future, although it would be possible to cut this down to a
less generic version if necessary. It is only run during db:schema:load
or the relevant migration, so the overhead is extraordinarily minimal.

* Transition reblogs to new Redis format

This provides a one-way migration to transition old Redis reblog entries
into the new format, with a separate tracking entry for reblogs.

It is not invertible because doing so could (if timestamp IDs are used)
require a database query for each status in each users' feed, which is
likely to be a significant toll on major instances.

* Address review comments from @akihikodaki

No functional changes.

* Additional review changes

* Heredoc cleanup

* Run db:schema:load hooks for test in development

This matches the behavior in Rails'
ActiveRecord::Tasks::DatabaseTasks.each_current_configuration, which
would otherwise break `rake db:setup` in development.

It also moves some functionality out to a library, which will be a good
place to put additional related functionality in the near future.
2017-10-04 09:56:37 +02:00
Yamagishi Kazutoshi
901fc48aae Upgrade Webpacker to version 3.0.1 (#5122) 2017-09-27 14:41:54 +02:00
nullkal
b32a1d5754 Refactor 500 file generation for future extension (#5105) 2017-09-26 11:55:33 +02:00
Eugen Rochko
3b60832214 New error page graphic. Other error page improvements (#5099)
- 500.html generated with admin-set default locale if set
- Error page `<title>` includes Mastodon site title
- 500 title changed to "This page is not
  correct" (ref: <https://www.youtube.com/watch?v=2VCAP_seh1A>)
- 500 content appended with "on our end" to make clear it's
  not user's fault
2017-09-25 23:05:54 +02:00