This way, instead of needing to know that you run "pytest" and knowing
tricks like "pytest -m ''" to run webtests and HTML validation, you can
now just run "invoke test", with more intuitive flags. This also reduces
the output in quiet mode even more.
After adding invoke tasks for some of the other tools/checks, I'll be
able to switch the git hooks to use these instead.
First invoke task: uses pip-compile to update the versions of all the
pip packages in requirements.txt and requirements-dev.txt. It also
post-processes the output file and removes any comments that have a "-r"
reference in them, since those currently cause Salt to break (and are
kind of redundant anyway).
Unfortunately, as part of writing this I discovered that invoke can't
handle type annotations in the definitions of its task functions, so I
had to exclude tasks.py from being checked by mypy. That makes me a
little nervous about whether invoke is still being maintained. Relevant
issue (over 4 years old): https://github.com/pyinvoke/invoke/issues/357
This isn't perfectly equivalent in some cases, but it's a barely
noticeable difference, and it's nice to not have all of these extra
custom properties like "--button-darkened-8-color" for an extremely
niche usage.
Now that we've switched to CSS custom properties, all the color rules
don't need to be repeated for each theme via a mixin, so the
_theme_base.scss could be split up with all its rules going into the
expected modules/locations along with all the other associated styles.
I think it's best to be specific that all of these are colors, otherwise
there could be some confusing usages (and potential collisions) with
ones like --border.
Sorry @Bauke (and probably some others), I know this will most likely
mess with any changes you've already made to override these properties,
but I wanted to do it eventually and it's only going to get worse the
longer I wait.
This should allow users with browsers that don't support CSS custom
properties to still have some minimal theme support. There will be
various issues with the themes (and that's fine), but it will at least
set the main colors for their chosen theme.
With the switch to CSS custom properties for the themes, old browsers
with no support are ending up with a transparent background on the
sidebar. This makes the site especially difficult to use on mobile.
I'm going to do something more extensive to allow browsers with no
support for custom properties to still get basic theming, but it's
dependent on a @supports query. For browsers that don't support that
query either, we need this line to give the sidebar a background.
This is probably just temporary, but I'm going to leave the donation
goal meter off the sidebar for now, so I don't want the confusing
section in the middle of the Financials page saying that the goal is $0
and so on.
This starts showing the Exemplary badge to all users again (but only the
author can see the count still).
It also changes the "priority" of the .is-comment-exemplary and
.is-comment-new classes so that the stripe will show the new color when
a comment is both new and exemplary.
We've been using pts_lbsearch on the text file for a few weeks now, and
it's working fine. Checks generally seem to take about 10 ms, and that's
totally fine for the relatively uncommon events of registrations and
password changes.
This removes everything related to the previous Redis-based method,
which means we no longer need the second Redis server or the ReBloom
module.
The "last comment posted" link in the sidebar on a topic's comments page
was still considering removed comments, so if the last comment in a
topic was removed it would link to that one. That's not very useful for
anyone, so this excludes removed comments the same way that deleted ones
were already excluded.
This replaces the current method of using a Bloom filter in Redis to
check for breached passwords with searching the text file directly using
pts_lbsearch (https://github.com/pts/pts-line-bisect/).
I'm not removing the Redis-based method yet because I want to test the
performance of this first, but this is *far* simpler and doesn't have
the possibility for false positives like the Bloom filter does.
When a ValidationError comes up for a reason unrelated to webargs (for
example, if a user tries to set a password that's in the breached list),
this crashes when trying to unnest it, since it doesn't have the extra
level that webargs adds.
This is a bit ugly, but checks to see whether the extra level is there
first.
Previously, there wasn't any defined list of which permissions were
valid or not. You basically had to look through each model's __acl__
method to see what the possibilities were.
Using an enum will be less convenient when adding new permissions or
changing existing ones (since it will require a database migration), but
it makes it much easier to see what the valid options are, and will
prevent invalid permissions from being set up in the database.
This uses pytest's "markers" system to add markers to two special types
of tests:
* webtest - ones that use the WebTest library and are testing the actual
HTTP app, instead of executing code/functions directly
* html_validation - ones that are generating HTML output (via webtest)
and running it through the Nu HTML Checker to validate it.
The "webtest" marker is added automatically by checking whether a test
uses either of the webtest fixtures, and the html_validation one is
currently added manually to the only module that has those tests. In the
future, we could probably put HTML validation tests in their own folder
and mark them automatically based on the module's path or something
similar.
This also changes the default arguments for pytest to exclude these two
marked types of tests, and updates the git hooks so that webtests are
run pre-commit (but not HTML validation), and all tests are run
pre-push. Similar to the way we use prospector, this makes it so that
the very slow tests are only run before pushing.
Installs the Nu Html Checker and starts using it to validate the home
page's HTML: https://validator.github.io/validator/
Also includes fixes to some lists that were nested in an invalid way.
I mistakenly assumed that not setting the cookiejar argument when
creating a webtest TestApp would mean that no cookies would be retained
between requests, but that's wrong. If you don't pass a cookiejar, it
just automatically creates one for you. Because of this, logged-out
webtests would end up being logged-in after any test logged in.
This reduces the webtest_loggedout fixture's scope to function-level so
that it will be re-initiated on every test instead. It also stops
passing a cookiejar for the logged-in webtest, since that's unnecessary.
This reverts commit cb7be83877.
HTML Tidy seems to have various gaps in its validation that we've found
already, including one that's pretty much a deal-breaker for Tildes's
HTML: it doesn't think that <menu> is a valid parent for <li>.
We're looking at alternative validators still.
Adds the HTML Tidy library to the dev version, along with the pytidylib
wrapper for it, and a couple of tests that use it to validate the HTML
of the home page.
Includes a fix to the GitLab "Planned features" link that Tidy considers
invalid because it includes some un-encoded characters.
This was not a fun upgrade. webargs made some major changes to its
approaches in 6.0, which are mostly covered here:
https://webargs.readthedocs.io/en/latest/upgrading.html
To keep using it on Tildes, this commit had to make the following
changes:
- Write my own wrapper for use_kwargs that changes some of the default
behavior. Specifically, we want the location that data is being
loaded from to default to "query" (the query string) instead of
webargs' default of "json". We also needed to set the "unknown"
behavior on every schema to "exclude" so that the schemas would
ignore any data fields they didn't need, since the default behavior
is to throw an error, which happens almost everywhere because of
Intercooler variables and/or multiple use_kwargs calls for different
subsets of the data.
- All @pre_load hooks in schemas needed to be rewritten so that they
weren't modifying data in-place (copy to a new data dict first).
Because webargs is now passing all data through all schemas,
modifying in-place could result in an earlier schema modifying data
that would then be passed in modified form to the later ones.
Specifically, this caused an issue with tags on posting a new topic,
where we just wanted to treat the tags as a string, but TopicSchema
would convert it to a list in @pre_load.
- use_kwargs on every endpoint using non-query data needed to be
updated to support the new single-location approach, either replacing
an existing locations= with location=, or adding location="form",
since form data was no longer used by default.
- The code that parsed the errors returned by webargs/Marshmallow
ValidationErrors needed to update to handle the additional "level"
in the dict of errors, where errors are now split out by location
and then field, instead of only by field.
- A few other minor updates, like always passing a schema object
instead of a class, and never passing a callable (mostly just for
simplicity in the wrapper).
I thought this would be a larger task due to so many of the tools
updating to new versions, but the only thing necessary for this upgrade
was updating the name of one of the disabled pylint errors.
I temporarily pinned two packages that will require more significant
updates (webargs in requirements and prospector in requirements-dev).
Other than those, everything seemed to upgrade cleanly, except for an
issue with mypy that needed a "type: ignore" comment to circumvent.
Note that there is currently an issue with Salt's pip module being
unable to handle comments in a requirements file that include "-r", so I
had to manually edit the two .txt files after using pip-tools to remove
all lines with "via -r" comments in them. I've commented about this in
an issue on Salt's repo here:
https://github.com/saltstack/salt/issues/56514#issuecomment-665947887
Previously, when checking if a link had been posted before, there was no
restriction on the time limit, so even posts from years ago would come
up. This restricts it to only the last 6 months, which I think is a
pretty reasonable time period for reposting.
This isn't great, but will fix an error that's actively occurring when
someone filters to a single tag (tag= query var) and also has a filtered
topic tag with a space in it.
Deimos
Bauke
Andrew Shu