Previously, if an event stream consumer hit an error when processing a
message, it would crash and restart, and the message that caused the
error would be left in "pending" status for that consumer forever while
the consumer continued processing new messages.
This commit adds some more deliberate handling of messages that cause
errors:
* When a consumer starts, it will try to read pending messages first. In
a case where an error was transient, this should mean that the message
that previously caused a crash will be processed on retry.
* If a particular message causes the consumer to crash 3 times, it will
be considered "dead" and moved out of the consumer's pending list into
one specifically for dead messages. These dead queues can be monitored
and inspected manually to look into failures, while the consumer can
still continue processing new messages.
* After clearing or processing all pending messages, consumers go back
to waiting for and processing new messages.
This should be a little more clear about what the Ignore function does
(as opposed to thinking that it might ignore the user that posted the
topic, one of the topic's tags, etc.).
Not a huge fan of this implementation, but it seems to work okay.
This removes RabbitMQ as well as everything else attached to it:
Erlang; the Prometheus collector; the pg-amqp-bridge and all PostgreSQL
functions and triggers; and the amqpy Python package and the Tildes code
that used it.
Note that this commit does not actually uninstall or delete any of these
packages or services, so if you have a running instance that you want to
keep (instead of re-provisioning from scratch), you will need to
manually remove them if you want them completely gone.
RabbitMQ was used to support asynchronous/background processing tasks,
such as determining word count for text topics and scraping the
destinations or relevant APIs for link topics. This commit replaces
RabbitMQ's role (as the message broker) with Redis streams.
This included building a new "PostgreSQL to Redis bridge" that takes
over the previous role of pg-amqp-bridge: listening for NOTIFY messages
on a particular PostgreSQL channel and translating them to messages in
appropriate Redis streams.
One particular change of note is that the names of message "sources"
were adjusted a little and standardized. For example, the routing key
for a message caused by a new comment was previously "comment.created",
but is now "comments.insert". Similarly, "comment.edited" became
"comments.update.markdown". The new naming scheme uses the table name,
proper name for the SQL operation, and column name instead of the
previous unpredictable terms.
The "keyset"-style pagination that Tildes uses for topic listings uses
WHERE and ORDER BY clauses that involve multiple columns to keep a
deterministic ordering even when the values in the main sort column are
equal. For example, when sorting by number of votes, you're actually
ordering by num_votes DESC, topic_id DESC. The previous single-column
indexes were a little inefficient for this and couldn't always be used
well.
This commit extends all of the relevant indexes to composite ones that
contain topic_id as well, and drops all of the original ones. This
should be more efficient, and should probably be done to indexes on the
comments table as well.
This generates a significantly better execution plan for the query - I
think using one of the columns from the join condition helps the
query-planner do this properly.
If the footer stretches to two lines (should only happen on mobile and
when there are new comments), this aligns the dropdown button more
towards the bottom of the topic, which looks better.
This also removes the attempt to close the dropdown if you the toggle
again while it's already open, since it seems like that isn't working
correctly anyway.
Currently, this dropdown contains Bookmark and Ignore.
As part of this, Spectre.css's icons stylesheet is now included, for the
caret icon used on the dropdown button. This is excessive since it's the
only icon being used for now, but the whole thing only adds about 1.3kb
to the gzipped size of tildes.css.
This adds a simple Ignore functionality for topics, with the only effect
being to exclude ignored topics from listing pages.
It also adds a "Your ignored topics" page (linked through the user menu)
that lists all of a user's ignored topics, so that they can access them
if needed (to be able to un-ignore).
gunicorn 20.0.0 included a change so that it will no longer read server
configuration out of Paster files. Because of this, the settings for it
in development.ini and production.ini were no longer being used. This
resulted in the auto-reloading no longer working in dev, and the number
of workers being reduced back down to 1 in production. The socket/PID
may have been impacted as well.
This commit moves the configuration into command-line args used to
launch gunicorn, and uses a pillar variable to handle the args different
between dev and prod.
This required a few minor changes/fixes:
* Change the name of an ignored pylint check about logging interpolation
* Add check=True to all subprocess.run() calls - this probably always
should have been used so the scripts will crash if a command fails
* Remove a couple of unnecessary list comprehensions
* Ignore some warnings caused by mypy @hybrid_property workaround
Previously, scheduled topics just had "Tildes" in place of the name of
the user that posted them, which wasn't very clear. This is a little
more explicit, and uses some different styling for that info on topics
when shown in listings.
As expected, these updates ended up requiring quite a few changes. I was
initially going to update only Marshmallow, but the older version of
webargs couldn't work with an updated Marshmallow, so I ended up needing
to do both at the same time.
The main changes required were:
* Schemas don't need to be specified as "strict" any more, so that could
be removed from constructors and Meta classes.
* .validate() now returns a dict of errors (if any) instead of raising a
ValidationError if anything goes wrong. This meant that I either need
to check the returned dict, or switch to .load() to still get raised
errors.
* Marshmallow doesn't support loading from two different field names as
easily (and changed the name of that to data_key), so all the routes
using "group_path" needed to be changed to just "path".
* Some of the Field methods and some decorated schema ones like
@pre_load receive new arguments and needed to be updated to handle
them and/or pass them on.
* webargs will no longer send a keyword argument for any fields that
aren't specified and don't have a default value when using
@use_kwargs. Because of this, I added missing= values for most
optional fields, but still required some special treatment for the
order query variable in a couple of topic listing views.
And finally, there is some strange behavior in webargs by default when a
form doesn't send any data for a field (due to the input not being
included or similar). When it doesn't find the field in form data, it
tries to fall back to checking for JSON data, but then crashes because
the request doesn't have any JSON data attached. I had to specify only
to look in the form data in a few places to fix this, but I've also
registered an issue against webargs related to it:
https://github.com/marshmallow-code/webargs/issues/444
This will allow people to mouseover a site name/icon if they want to
check which domain its from. Mousing over the title and checking the
link in the status bar would generally accomplish the same thing, but I
think this might feel a bit more intuitive.
This switches comment voting to use the "toggle buttons" that most other
actions are using (e.g. Bookmark). This makes it so that only the vote
button is replaced, instead of the entire comment's contents. This is
mostly significant when the comment contains a <details> block, because
previously voting/unvoting while one of those was expanded would cause
it to collapse when the comment was replaced.
As another side-effect of this, voting will now always appear to
increase the count by 1, and unvoting will always appear to decrease it
by 1. Previously it would re-query the comment, which could result in
larger jumps in the vote count if other people were voting at the same
time. That confused some people, so this will probably be better.
Deimos
cfabbro
Ivan Fonseca
yabai
Timo