Sometimes the database initialization fails, generally due to some
earlier step in the setup having an issue. Even if a re-provision
resolves that issue, the database init wouldn't be re-run since it was
set up to only happen after the database was created.
This changes it so that it will try to select from the users table, and
if that fails it will re-run the initialization.
This allows groups to have wiki pages. The rendered form of the page is
stored in the database, but the markdown source is kept on the
filesystem, using git to maintain the history (by doing a commit on
every edit).
A lot of aspects of this are still quite rough, but it should be a
decent start.
This sets up a cronjob that will run every hour to select the most
common tags used in a group (up to 100), and store them in a new column
in the groups table. This will be used to populate the list of tags to
use for autocompletion.
This redirect being first in the file meant that if someone tried to
access a dev version through any method except using "localhost" (such
as via the IP address), no server block would be matched, which causes
nginx to use the first one. That resulted in a 301 redirect to
tildes.net, which definitely shouldn't happen for a dev version.
This change both moves the redirect to the bottom, as well as only
adding it if it's the "prod" environment, since it's not needed in the
dev environment at all.
Starting with psycopg2 version 2.8, the package on pypi no longer
contains a binary version and must be compiled from source. These two
packages are required for this to be possible.
It would have been simpler to just switch to the psycopg2-binary
package, however that isn't a very good solution overall since many
other packages treat "psycopg2" as the dependency that they want
installed, not "psycopg2-binary". Overall, this situation is pretty
messy and I'm not sure what will end up being the final state, but this
should work for now.
More info about the source-only change:
http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/
Previously, this was set as "same-origin" which will only send a
referrer to Tildes itself. This changes so that it will continue sending
the full referrer to Tildes, but will send only the domain to external
sites if they use HTTPS (and no referer to HTTP ones).
This can be useful because there are often situations where an article
author sees traffic coming from a site and will come to check it out and
be able to participate in the discussion.
The site-icons spritesheet has already become unwieldy - it's almost
1MB, is mostly rarely-needed icons, and needs to be fully replaced and
re-downloaded whenever a new icon is added. With HTTP/2 now being widely
supported, spritesheets seem to be mostly obsolete, and I probably never
should have done it that way in the first place.
This commit changes over to simply using individual icon images, and
rebuilds the CSS file whenever new icons are downloaded. This new CSS
file will probably be somewhat large, but should gzip extremely well.
This probably still needs some work to support cache-busting on the CSS
file.
I've been reading a little about PostgreSQL transaction ID wraparound
today, and how it's knocked multiple companies out of commission for
days to get it resolved. It should have almost no chance of happening on
Tildes for years, but this will let me set up some monitoring for it
now, while I'm thinking about it.
For more info:
https://blog.sentry.io/2015/07/23/transaction-id-wraparound-in-postgres.html
A lot of the code in common between this and the EmbedlyScraper should
probably be generalized out to a base class soon, but let's make sure
this works first.
The monitoring server needs Redis, but not the separate server that's
used for the breached-passwords bloom filter in dev/prod. This splits
that server out to its own state, so that it doesn't need to be set up
on the monitoring server.
Some of these states were built entirely around a single-server approach
(Prometheus + monitoring being on the same server as the site), and the
files have needed modifications to work with a separate monitoring
server.
This updates the states so that it should all happen as expected in all
types of environments.
Previously I was using Salt to install the Sentry SDK (previously known
as "raven") only on the production server, but that's not really
necessary. This will just install it everywhere, and then we'll only
actually integrate it in production.
Links aren't displayed/used anywhere yet, but this should be the basic
setup needed for a simple link-shortener on the tild.es domain.
Currently, it will support two uses:
* https://tild.es/asdf - redirect to topic with id "asdf"
* https://tild.es/~asdf - redirect to group "~asdf"
Previously I was using pyenv to build Python, but that's mostly
unnecessary and has some other side effects (like needing to install a
lot of packages as dependencies).
This switches to using the deadsnakes PPA instead, which also has the
effect of upgrading to the most recent version of Python 3.6 (currently,
3.6.7 instead of 3.6.5).
Of course right after I did that last update, they released two more new
versions of cmark-gfm, so here we go again.
The .18 update defaults to "safe" mode, but I want to disable that and
leave sanitization up to Bleach, so this required changing the options.
Automatically runs the generate-site-icons script every 5 minutes. The
script was also updated to use checksum-based rsync instead of cp, so
that the file won't be replaced (and need to be redownloaded by users)
unless it actually changes.
This adds a trigger to the scraper_results table which will add rabbitmq
messages whenever a scrape finishes, as well as a consumer that picks up
these messages, and uses Embedly data to download (and resize if
necessary) the favicons from any sites that are scraped. These are
downloaded into the input folder for the site-icons-spriter, so it
should be able to use these to generate spritesheets.
This adds a consumer (in prod only) that uses Embedly's Extract API to
scrape the links from all new link topics and stores some of the data in
the topic's content_metadata column.
Boussole (which watches the SCSS files for changes and compiles them)
has just been running in the same venv as the main app until now, but
it's holding back the version of the click package. There's no real
reason that it needs to be in the app venv, so this moves it to its own
one, which also eliminates quite a few other packages that were only
being installed because of Boussole.
This detects mentions of users in comments using the same pattern as the
markdown parsing uses to generate user links. Mentioned users are sent a
notification, and mentions are added/deleted if needed on comment edits.
As part of this, setup was done to generate rabbitmq messages for
comment creation and edits, and the mentions are handled by an async
consumer of these messages.
Deimos
Andrew Shu
Celeo