This makes it so that posts (both topics and comments) can no longer be
voted on after they're over 30 days old. An hourly cronjob makes this
"official" by updating a flag on the post indicating that voting is
closed. The daily clean_private_data script then deletes all individual
vote records for posts with closed voting, and the triggers on the
voting tables have been updated to not decrement the vote totals when
these deletions happen.
The net result of this is that Tildes only stores users' votes for a
maximum of 30 days, removing a lot of sensitive/private data that builds
up over the long term.
Creates an effect of some "extra bar" (in the same color as Exemplary
labels) expanding on the right end of the donation goal meter when it's
above 100%.
On small screens, when the bottom row with a topic's comments, source,
and age starts wrapping, there was some weird behavior. This improves it
so that:
* The number(s) for comments won't wrap separately
* If the row gets taller due to wrapping, text will align to the bottom
The way I was doing this before had the colon as part of the metadata,
which meant that it could wrap onto the next line in thin windows,
giving a result like:
Article
: 1500 words
This changes it so that it's added with CSS after the content type when
needed, so it won't wrap separately.
Previously, the content metadata displayed next to a topic's content
type (like "Article: 1800 words") was fairly generic and could result in
strange data being displayed if a scraper fetched it for an
inappropriate type (for example, displaying word count for videos).
This creates an enum to hold all the different content metadata fields,
and moves some logic into that class to handle deciding which fields to
show for different types, and the formatting logic for values.
Previously, the warning would only ever say "over a week old", even when
the topic/comment was much older than that. This adds a new function to
create a vague timedelta description for longer periods, and also
enables the Javascript to use it as well through adding the description
as a data attr on the reply button when a warning is needed, instead of
duplicating the logic in JS.
This is just a small reduction to help compensate for having the list
pushed down by the donation goal. This list isn't intended to stay here
forever anyway.
This uses the data in the financials table to generate a donation goal
meter at the top of the home page's sidebar. It uses the new-ish HTML
<meter> element, which will automatically change colors as it hits
different thresholds.
This enables (manually, so far) setting "important tags" for individual
groups. Topics in the group with any of these tags will have them shown
in listings next to the group name (similar to how spoiler and nsfw work
globally).
This didn't get updated when boussole was split out to its own
virtualenv, and was still being linked to the pip installs from the
application succeeding.
Previously, the virtualenvs were owned by root and the pip installs were
done as root as well. This worked fine, but it meant that I can't use
pip-tools' pip-sync function without sudo. This makes it simpler by
giving ownership to the app user (tildes in prod, vagrant in dev).
I'm going to start using pip-tools to manage dependencies:
https://github.com/jazzband/pip-tools
This makes updating the dependencies and virtualenv easier in a few
ways, and makes it simple to keep dev dependencies split out (so I can
stop installing them in production).
Now, to do a check and update all packages to their newest versions, the
main command is:
pip-compile --no-header --upgrade requirements.in
and again with requirements-dev.in to update that one as well. This will
update all the package versions in requirements.txt and
requirements-dev.txt. The virtualenv can then be updated to match those
versions by running:
pip-sync requirements.txt
(or requirements-dev.txt for dev environment). This currently needs to
be run with sudo, but I'm going to try to fix that shortly.
Now that people can use quotes inside searches, it's awkward to be
displaying them with quotes already around them. This should remove
everywhere that could have been happening.
I don't think this has ever affected anything, but this function
shouldn't have been defined as STABLE. That enables the database to
potentially optimize away some calls to it, and since this has side
effects, we never want that to happen. Removing this defaults it back to
classifying as VOLATILE, which is what we want.
This installs PL/Python (specifically plpython3u), enables it in the
database, and creates a function id36_to_id that calls the Python
function with the same name inside the tildes.lib.id module. This will
enable doing queries similar to this, when I have a topic's ID36 from
the site:
SELECT * FROM topics WHERE topic_id = id36_to_id('asdf');
The fact that this was possible to set up without having to port the
id36_to_id logic to a different language is blowing my mind a little.
There are some really interesting possibilities from being able to
import all of the Python code into the database itself.
Now that PostgreSQL has been upgraded to version 12 (from 10), the
websearch_to_tsquery() function is available. This is much better for
general search purposes, and will add multiple capabilities.
I'm not sure if there's a full specification of what it supports
anywhere, but this is the main relevant section in the docs:
https://www.postgresql.org/docs/12/textsearch-controls.html
Changing these pillar values are the only actual changes to Tildes
code/config needed, but if you're upgrading an existing version from 10
to 12 you will need to do some manual steps. The below should cover it -
lines starting with a * are descriptions of things you need to do, while
the rest are actual commands to run:
sudo apt-get install postgresql-12
sudo systemctl stop postgresql@10-main.service
sudo systemctl stop postgresql@12-main.service
cd /var/lib/postgresql
sudo -u postgres /usr/lib/postgresql/12/bin/pg_upgrade -b /usr/lib/postgresql/10/bin/ -B /usr/lib/postgresql/12/bin/ -d /var/lib/postgresql/10/main/ -D /var/lib/postgresql/12/main/ -o '-c config_file=/etc/postgresql/10/main/postgresql.conf' -O '-c config_file=/etc/postgresql/12/main/postgresql.conf'
* Change pillar value to 12, and run salt
sudo systemctl stop postgresql@10-main.service
* Edit /etc/postgresql/12/main/postgresql.conf and change port to 5432
sudo systemctl restart postgresql@12-main.service
sudo -u postgres ./analyze_new_cluster.sh
* After verifying the new version seems to be working, clean up the old version:
sudo apt-get remove postgresql-10
sudo rm -rf /usr/lib/postgresql/10/
sudo rm -rf /var/lib/postgresql/10/
sudo rm -rf /etc/postgresql/10/
The creation of the Theme Previews page showed that the way I was doing
metrics for database model objects being created wasn't very good.
Whenever someone loaded the Theme Previews page, the "total topics"
metric would increase by 2, "total comments" by 4, and "exemplary
labels" by 1, because the page is creating that many fake objects and
the metrics were being sent in each class's __init__ method.
This changes to take advantage of SQLAlchemy's object lifecycle event
for "pending to persistent", which only triggers when an object is
actually persisted to the database. When this event happens, the
object's _update_creation_metric method is called, and all metric
updates have been moved into that method now.
This is mostly just rearranging, but a couple of functional changes:
* The "preview blocks" can now be clicked to switch themes, instead of
using the dropdown menu.
* Click events should be disabled in the fake posts, so we don't need to
worry about voting/labels/etc.
This actually involves the necessary backend changes for full comment
search, but I'm a little nervous about the privacy implications of that,
and don't want to enable it for everyone out of nowhere. So for now,
this just allows users to search *their own* comments or topics.
These views and templates are starting to get real ugly, and probably
need a major re-thinking soon.
Previously a topic with an unknown content type would display "Article".
This switches to "Link" which will have some different idiosyncracies,
but should be a little more correct in general.
Deimos
deing