A lot of the code in common between this and the EmbedlyScraper should
probably be generalized out to a base class soon, but let's make sure
this works first.
The monitoring server needs Redis, but not the separate server that's
used for the breached-passwords bloom filter in dev/prod. This splits
that server out to its own state, so that it doesn't need to be set up
on the monitoring server.
Some of these states were built entirely around a single-server approach
(Prometheus + monitoring being on the same server as the site), and the
files have needed modifications to work with a separate monitoring
server.
This updates the states so that it should all happen as expected in all
types of environments.
Previously I was using Salt to install the Sentry SDK (previously known
as "raven") only on the production server, but that's not really
necessary. This will just install it everywhere, and then we'll only
actually integrate it in production.
Links aren't displayed/used anywhere yet, but this should be the basic
setup needed for a simple link-shortener on the tild.es domain.
Currently, it will support two uses:
* https://tild.es/asdf - redirect to topic with id "asdf"
* https://tild.es/~asdf - redirect to group "~asdf"
Previously I was using pyenv to build Python, but that's mostly
unnecessary and has some other side effects (like needing to install a
lot of packages as dependencies).
This switches to using the deadsnakes PPA instead, which also has the
effect of upgrading to the most recent version of Python 3.6 (currently,
3.6.7 instead of 3.6.5).
Of course right after I did that last update, they released two more new
versions of cmark-gfm, so here we go again.
The .18 update defaults to "safe" mode, but I want to disable that and
leave sanitization up to Bleach, so this required changing the options.
Automatically runs the generate-site-icons script every 5 minutes. The
script was also updated to use checksum-based rsync instead of cp, so
that the file won't be replaced (and need to be redownloaded by users)
unless it actually changes.
This adds a trigger to the scraper_results table which will add rabbitmq
messages whenever a scrape finishes, as well as a consumer that picks up
these messages, and uses Embedly data to download (and resize if
necessary) the favicons from any sites that are scraped. These are
downloaded into the input folder for the site-icons-spriter, so it
should be able to use these to generate spritesheets.
This adds a consumer (in prod only) that uses Embedly's Extract API to
scrape the links from all new link topics and stores some of the data in
the topic's content_metadata column.
Boussole (which watches the SCSS files for changes and compiles them)
has just been running in the same venv as the main app until now, but
it's holding back the version of the click package. There's no real
reason that it needs to be in the app venv, so this moves it to its own
one, which also eliminates quite a few other packages that were only
being installed because of Boussole.
This detects mentions of users in comments using the same pattern as the
markdown parsing uses to generate user links. Mentioned users are sent a
notification, and mentions are added/deleted if needed on comment edits.
As part of this, setup was done to generate rabbitmq messages for
comment creation and edits, and the mentions are handled by an async
consumer of these messages.
Deimos
Andrew Shu
Celeo