The previous method of doing this could cause redis to try to start up
(via restart) earlier than it should. By using require_in and watch_in,
it should now only start up in the first place once this service has
been started first, and it will also cause redis to restart if it ever
needs to run again in the future.
Vagrant on Windows has issues with creating symlinks inside shared
folders - it requires a permission that isn't granted to a user by
default. This can be fixed by changing security policies, but for our
purposes we don't need the symlinks anyway, and can run the tools
manually like this, instead of using the .bin/ symlinks.
Previously tild.es urls would proxy_pass through to the views inside the
Pyramid app, but this caused strange behavior in some cases. For
example, anything that caused a 404 response would end up in a broken
page that still appeared to be on the tild.es domain, but would be an
HTML-only page coming from the app, since the CSS and JS would not be
available.
This method is still a bit weird in some ways (now you'll end up on a
404 page at https://tildes.net/shortener/... instead), but I think it's
an improvement overall.
This changes the "activity" topic-sorting method to look for
"interesting" activity instead of everything, and adds a new "All
activity" method that retains the previous behavior.
Currently, "interesting activity" excludes any comments that have active
Noise, Offtopic, or Malice labels, or any of their children. These
checks are also done based on labeling activity, so for example if
someone posts a new comment it will bump the thread initially, but if
that comment is then labeled as Noise, the thread will "un-bump" and go
back to its previous position in the Activity sort.
There were also some other minor changes made to appearance to support
adding another sorting option, such as shortening the displayed names on
the "tabs", like showing "Votes" instead of "Most votes". This probably
needs some further work, but is okay for now.
This won't affect requests for static files or anything except ones that
get proxied to the app.
The current configuration is based on IP, and allows a rate of 4/sec,
with an additional burst of 5 above the limit permitted, and burst
requests allowed to go through immediately (nodelay). For more info:
https://www.nginx.com/blog/rate-limiting-nginx/
Sometimes the database initialization fails, generally due to some
earlier step in the setup having an issue. Even if a re-provision
resolves that issue, the database init wouldn't be re-run since it was
set up to only happen after the database was created.
This changes it so that it will try to select from the users table, and
if that fails it will re-run the initialization.
This allows groups to have wiki pages. The rendered form of the page is
stored in the database, but the markdown source is kept on the
filesystem, using git to maintain the history (by doing a commit on
every edit).
A lot of aspects of this are still quite rough, but it should be a
decent start.
This sets up a cronjob that will run every hour to select the most
common tags used in a group (up to 100), and store them in a new column
in the groups table. This will be used to populate the list of tags to
use for autocompletion.
This redirect being first in the file meant that if someone tried to
access a dev version through any method except using "localhost" (such
as via the IP address), no server block would be matched, which causes
nginx to use the first one. That resulted in a 301 redirect to
tildes.net, which definitely shouldn't happen for a dev version.
This change both moves the redirect to the bottom, as well as only
adding it if it's the "prod" environment, since it's not needed in the
dev environment at all.
Starting with psycopg2 version 2.8, the package on pypi no longer
contains a binary version and must be compiled from source. These two
packages are required for this to be possible.
It would have been simpler to just switch to the psycopg2-binary
package, however that isn't a very good solution overall since many
other packages treat "psycopg2" as the dependency that they want
installed, not "psycopg2-binary". Overall, this situation is pretty
messy and I'm not sure what will end up being the final state, but this
should work for now.
More info about the source-only change:
http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/
Previously, this was set as "same-origin" which will only send a
referrer to Tildes itself. This changes so that it will continue sending
the full referrer to Tildes, but will send only the domain to external
sites if they use HTTPS (and no referer to HTTP ones).
This can be useful because there are often situations where an article
author sees traffic coming from a site and will come to check it out and
be able to participate in the discussion.
The site-icons spritesheet has already become unwieldy - it's almost
1MB, is mostly rarely-needed icons, and needs to be fully replaced and
re-downloaded whenever a new icon is added. With HTTP/2 now being widely
supported, spritesheets seem to be mostly obsolete, and I probably never
should have done it that way in the first place.
This commit changes over to simply using individual icon images, and
rebuilds the CSS file whenever new icons are downloaded. This new CSS
file will probably be somewhat large, but should gzip extremely well.
This probably still needs some work to support cache-busting on the CSS
file.
I've been reading a little about PostgreSQL transaction ID wraparound
today, and how it's knocked multiple companies out of commission for
days to get it resolved. It should have almost no chance of happening on
Tildes for years, but this will let me set up some monitoring for it
now, while I'm thinking about it.
For more info:
https://blog.sentry.io/2015/07/23/transaction-id-wraparound-in-postgres.html
A lot of the code in common between this and the EmbedlyScraper should
probably be generalized out to a base class soon, but let's make sure
this works first.
The monitoring server needs Redis, but not the separate server that's
used for the breached-passwords bloom filter in dev/prod. This splits
that server out to its own state, so that it doesn't need to be set up
on the monitoring server.
Some of these states were built entirely around a single-server approach
(Prometheus + monitoring being on the same server as the site), and the
files have needed modifications to work with a separate monitoring
server.
This updates the states so that it should all happen as expected in all
types of environments.
Previously I was using Salt to install the Sentry SDK (previously known
as "raven") only on the production server, but that's not really
necessary. This will just install it everywhere, and then we'll only
actually integrate it in production.
Links aren't displayed/used anywhere yet, but this should be the basic
setup needed for a simple link-shortener on the tild.es domain.
Currently, it will support two uses:
* https://tild.es/asdf - redirect to topic with id "asdf"
* https://tild.es/~asdf - redirect to group "~asdf"
Previously I was using pyenv to build Python, but that's mostly
unnecessary and has some other side effects (like needing to install a
lot of packages as dependencies).
This switches to using the deadsnakes PPA instead, which also has the
effect of upgrading to the most recent version of Python 3.6 (currently,
3.6.7 instead of 3.6.5).
Of course right after I did that last update, they released two more new
versions of cmark-gfm, so here we go again.
The .18 update defaults to "safe" mode, but I want to disable that and
leave sanitization up to Bleach, so this required changing the options.
Automatically runs the generate-site-icons script every 5 minutes. The
script was also updated to use checksum-based rsync instead of cp, so
that the file won't be replaced (and need to be redownloaded by users)
unless it actually changes.
This adds a trigger to the scraper_results table which will add rabbitmq
messages whenever a scrape finishes, as well as a consumer that picks up
these messages, and uses Embedly data to download (and resize if
necessary) the favicons from any sites that are scraped. These are
downloaded into the input folder for the site-icons-spriter, so it
should be able to use these to generate spritesheets.
This adds a consumer (in prod only) that uses Embedly's Extract API to
scrape the links from all new link topics and stores some of the data in
the topic's content_metadata column.
Boussole (which watches the SCSS files for changes and compiles them)
has just been running in the same venv as the main app until now, but
it's holding back the version of the click package. There's no real
reason that it needs to be in the app venv, so this moves it to its own
one, which also eliminates quite a few other packages that were only
being installed because of Boussole.
This detects mentions of users in comments using the same pattern as the
markdown parsing uses to generate user links. Mentioned users are sent a
notification, and mentions are added/deleted if needed on comment edits.
As part of this, setup was done to generate rabbitmq messages for
comment creation and edits, and the mentions are handled by an async
consumer of these messages.
Deimos
Andrew Shu
Celeo