Previously, the content metadata displayed next to a topic's content
type (like "Article: 1800 words") was fairly generic and could result in
strange data being displayed if a scraper fetched it for an
inappropriate type (for example, displaying word count for videos).
This creates an enum to hold all the different content metadata fields,
and moves some logic into that class to handle deciding which fields to
show for different types, and the formatting logic for values.
Previously, the warning would only ever say "over a week old", even when
the topic/comment was much older than that. This adds a new function to
create a vague timedelta description for longer periods, and also
enables the Javascript to use it as well through adding the description
as a data attr on the reply button when a warning is needed, instead of
duplicating the logic in JS.
This is just a small reduction to help compensate for having the list
pushed down by the donation goal. This list isn't intended to stay here
forever anyway.
This uses the data in the financials table to generate a donation goal
meter at the top of the home page's sidebar. It uses the new-ish HTML
<meter> element, which will automatically change colors as it hits
different thresholds.
This enables (manually, so far) setting "important tags" for individual
groups. Topics in the group with any of these tags will have them shown
in listings next to the group name (similar to how spoiler and nsfw work
globally).
This didn't get updated when boussole was split out to its own
virtualenv, and was still being linked to the pip installs from the
application succeeding.
Previously, the virtualenvs were owned by root and the pip installs were
done as root as well. This worked fine, but it meant that I can't use
pip-tools' pip-sync function without sudo. This makes it simpler by
giving ownership to the app user (tildes in prod, vagrant in dev).
I'm going to start using pip-tools to manage dependencies:
https://github.com/jazzband/pip-tools
This makes updating the dependencies and virtualenv easier in a few
ways, and makes it simple to keep dev dependencies split out (so I can
stop installing them in production).
Now, to do a check and update all packages to their newest versions, the
main command is:
pip-compile --no-header --upgrade requirements.in
and again with requirements-dev.in to update that one as well. This will
update all the package versions in requirements.txt and
requirements-dev.txt. The virtualenv can then be updated to match those
versions by running:
pip-sync requirements.txt
(or requirements-dev.txt for dev environment). This currently needs to
be run with sudo, but I'm going to try to fix that shortly.
Now that people can use quotes inside searches, it's awkward to be
displaying them with quotes already around them. This should remove
everywhere that could have been happening.
I don't think this has ever affected anything, but this function
shouldn't have been defined as STABLE. That enables the database to
potentially optimize away some calls to it, and since this has side
effects, we never want that to happen. Removing this defaults it back to
classifying as VOLATILE, which is what we want.
This installs PL/Python (specifically plpython3u), enables it in the
database, and creates a function id36_to_id that calls the Python
function with the same name inside the tildes.lib.id module. This will
enable doing queries similar to this, when I have a topic's ID36 from
the site:
SELECT * FROM topics WHERE topic_id = id36_to_id('asdf');
The fact that this was possible to set up without having to port the
id36_to_id logic to a different language is blowing my mind a little.
There are some really interesting possibilities from being able to
import all of the Python code into the database itself.
Now that PostgreSQL has been upgraded to version 12 (from 10), the
websearch_to_tsquery() function is available. This is much better for
general search purposes, and will add multiple capabilities.
I'm not sure if there's a full specification of what it supports
anywhere, but this is the main relevant section in the docs:
https://www.postgresql.org/docs/12/textsearch-controls.html
Changing these pillar values are the only actual changes to Tildes
code/config needed, but if you're upgrading an existing version from 10
to 12 you will need to do some manual steps. The below should cover it -
lines starting with a * are descriptions of things you need to do, while
the rest are actual commands to run:
sudo apt-get install postgresql-12
sudo systemctl stop postgresql@10-main.service
sudo systemctl stop postgresql@12-main.service
cd /var/lib/postgresql
sudo -u postgres /usr/lib/postgresql/12/bin/pg_upgrade -b /usr/lib/postgresql/10/bin/ -B /usr/lib/postgresql/12/bin/ -d /var/lib/postgresql/10/main/ -D /var/lib/postgresql/12/main/ -o '-c config_file=/etc/postgresql/10/main/postgresql.conf' -O '-c config_file=/etc/postgresql/12/main/postgresql.conf'
* Change pillar value to 12, and run salt
sudo systemctl stop postgresql@10-main.service
* Edit /etc/postgresql/12/main/postgresql.conf and change port to 5432
sudo systemctl restart postgresql@12-main.service
sudo -u postgres ./analyze_new_cluster.sh
* After verifying the new version seems to be working, clean up the old version:
sudo apt-get remove postgresql-10
sudo rm -rf /usr/lib/postgresql/10/
sudo rm -rf /var/lib/postgresql/10/
sudo rm -rf /etc/postgresql/10/
The creation of the Theme Previews page showed that the way I was doing
metrics for database model objects being created wasn't very good.
Whenever someone loaded the Theme Previews page, the "total topics"
metric would increase by 2, "total comments" by 4, and "exemplary
labels" by 1, because the page is creating that many fake objects and
the metrics were being sent in each class's __init__ method.
This changes to take advantage of SQLAlchemy's object lifecycle event
for "pending to persistent", which only triggers when an object is
actually persisted to the database. When this event happens, the
object's _update_creation_metric method is called, and all metric
updates have been moved into that method now.
This is mostly just rearranging, but a couple of functional changes:
* The "preview blocks" can now be clicked to switch themes, instead of
using the dropdown menu.
* Click events should be disabled in the fake posts, so we don't need to
worry about voting/labels/etc.
This actually involves the necessary backend changes for full comment
search, but I'm a little nervous about the privacy implications of that,
and don't want to enable it for everyone out of nowhere. So for now,
this just allows users to search *their own* comments or topics.
These views and templates are starting to get real ugly, and probably
need a major re-thinking soon.
Previously a topic with an unknown content type would display "Article".
This switches to "Link" which will have some different idiosyncracies,
but should be a little more correct in general.
When a chain of comments has been fully removed, this will display a
single marker like "Removed by admin: 8 comments by 2 users" instead of
individual "Comment removed by site admin" markers for each one.
This should be stored in the database so that we can take advantage of
it for searching/filtering, but I want to test it on the live site first
and get input about what other types of content get posted. It'll be
simpler to leave it in code only for that, and I can add it to the
database later.
I've always been pretty unhappy with the ugly way tags were implemented,
using @hybrid_property and needing to do strange things all over the
place to deal with converting underscores to spaces and vice versa, as
well as other idiosyncracies.
There are still a few oddities here and there, but overall this is much
better.
Now that there are topics being posted automatically by the "fake" user,
we don't want to be generating reply notifications for that user
whenever someone posts a comment in them. These serve no purpose and
would just pile up forever without being read. The same problem could
have already been happening if people replied back to a comment with the
"unknown user" author.
This was always inconsistent - I've been setting objects in some models,
and ids in other ones. Using ids can cause various errors, especially in
cases where one of the objects involved hasn't been committed to the
database yet and so hasn't been assigned its id.
Deimos
deing