Kegan Dougal
e35ea2f27c
Use the NextPollTimestampSecs or else we'll resend on /configureService due to JSON key name change
8 years ago
Kegsay
1cb7a3b565
Merge pull request #94 from matrix-org/kegan/rss-notify-on-failures
Add a flag `IsFailing` to let clients know if their RSS feeds are down
8 years ago
Kegan Dougal
bc1e6283d8
Make some fields 'public'
8 years ago
Kegan Dougal
fae997fdb7
Add a flag `IsFailing` to let clients know if their RSS feeds are down
8 years ago
Kegan Dougal
299164d525
Fix RSS feeds which do not return GUIDs and do not have published dates
We previously relied on the published date and GUIDs to determine which items
were new. We now only rely on the GUID and not the published date as a lot of
RSS feeds don't have published dates. A lot of feeds don't have GUIDs either,
so we now fallback to the HTTP `Link` field, or worst-case, the item `Title`.
8 years ago
Kegsay
1f47c5a202
Merge pull request #92 from matrix-org/kegan/rss-user-agent
Set "Go-NEB" as the User-Agent on RSS feed polling
8 years ago
Kegan Dougal
5b5de2dc4b
Set "Go-NEB" as the User-Agent on RSS feed polling
Some services, notably Reddit, basically force you to set a custom UA in order
to use their RSS feeds. All the common default non-browser UAs are HEAVILY
rate-limited such that you can't realistically use them.
8 years ago
Kegan Dougal
5c8cc4a81c
strconv to convert int to string
8 years ago
Kegan Dougal
433331ca20
Metrics: Add auth session counter
8 years ago
Kegsay
3f31f63b19
Merge pull request #91 from matrix-org/kegan/neb-metrics
Add metrics to NEB
8 years ago
Kegan Dougal
f493a1933e
s/sendMetric/incrementMetrics/
8 years ago
Kegan Dougal
2e5c8e4d79
Only send the domain part of the RSS feed as that's more useful when doing aggregate metrics
8 years ago
Kegan Dougal
b4b0a661c4
Add more metrics
8 years ago
Kegan Dougal
cdb49fa60b
Add command type as a label
8 years ago
Kegan Dougal
32e8a5b6cc
Docs
8 years ago
Kegan Dougal
19eeb96610
Use enums instead
8 years ago
Kegan Dougal
219da01e08
Add extremely noddy counter metrics for !commands
Missing the command type currently. Also, not sure how "nice" this is: we could
probably blob the 3 functions into 1 and use an enum but, meh?
8 years ago
Kegan Dougal
729b80c6bc
Set max open conns to 1 on sqlite3
8 years ago
Kegsay
e875b9c251
Merge pull request #87 from matrix-org/kegan/metrics
Instrument all HTTP handlers
8 years ago
Kegan Dougal
c9de6eaf06
Instrument all HTTP handlers
8 years ago
Kegan Dougal
d169b5560f
Add prometheus dep for metrics
8 years ago
Kegan Dougal
248822a3d6
Fix #86 - set a 20s min threshold before re-polling
8 years ago
Kegan Dougal
ef6fc0d3bf
Remove useless field
8 years ago
Kegsay
7bcffa4391
Merge pull request #85 from matrix-org/kegan/database-is-locked
Edit README to put `_busy_timeout` to fix database is locked errors (#63 )
8 years ago
Kegsay
d9da3caac4
Merge pull request #84 from matrix-org/kegan/rss-guids
Maintain a list of GUIDs per feed and use them to de-dupe
8 years ago
Kegan Dougal
95521a08f2
Edit README to put `_busy_timeout` to fix database is locked errors ( #63 )
8 years ago
Kegan Dougal
8316ac21ab
Maintain a list of GUIDs per feed and use them to de-dupe
Some RSS feeds will edit the published time of an item AFTER putting it out,
which resulted in RSS Bot sending the same article twice. We now remember the
"GUID" field for each item and de-dupe based on that. Normal timestamp
algorithm still applies.
8 years ago
Kegan Dougal
77b729d4ba
Remove spammy logs
8 years ago
Kegan Dougal
5f015e26ee
Join rooms when registering rss service
8 years ago
Kegan Dougal
b903860af9
Bump min to 5min to help not spam feeds with no cache control
8 years ago
Kegan Dougal
1640728857
Add lrucache
8 years ago
Kegsay
2f0a2c7f10
Merge pull request #83 from matrix-org/kegan/cache-rss-feeds
Add a caching HTTP client
8 years ago
Kegan Dougal
e088e11821
Swap to an LRU cache set to 20MB
8 years ago
Kegan Dougal
e61356431a
Merge branch 'master' into kegan/cache-rss-feeds
8 years ago
Kegan Dougal
7587dbc9c4
Add lrucache
8 years ago
Kegan Dougal
b0eb284347
Add a caching HTTP client
8 years ago
Kegan Dougal
223ccac851
Add HTTP caching library
8 years ago
Kegsay
f63e5ce7c1
Merge pull request #82 from matrix-org/kegan/per-service-poll
Move OnPoll into the Service method set
8 years ago
Kegan Dougal
fd54eb1a55
Move OnPoll into the Service method set
This feels a lot better because now `OnPoll` works in a similar way to
`OnReceiveWebhook` (called on a `Service`) rather than have this strange
global-per-service-type struct.
8 years ago
Kegan Dougal
7518ed1995
Fix NPE if oldService is nil
8 years ago
Kegsay
5fe9743cba
Merge pull request #81 from matrix-org/kegan/rssbot
s/feedreader/rssbot/g
8 years ago
Kegan Dougal
cd39219113
s/feedreader/rssbot/g
Because apparently "Feed Reader" is the name of a thing popular enough that we
don't want to step on toes, so let's call it the name of a less popular thing.
8 years ago
Kegsay
ccb1c6c779
Merge pull request #80 from matrix-org/kegan/feedreader-struct
Rejig how feedreaders stores rooms to send updates to
8 years ago
Kegan Dougal
185a7d5678
Rejig how feedreaders stores rooms to send updates to
8 years ago
Kegsay
7971b13368
Merge pull request #78 from matrix-org/kegan/feedreader
Implement feedreader service
8 years ago
Kegan Dougal
8684b671e2
Bump sqlite3 dep to make it compile on OSX
8 years ago
Kegan Dougal
7b4448957b
Review comments
8 years ago
Kegan Dougal
98e7533469
Send feed items in reverse-order for chronological times
Also add checks on Register() to make sure the feeds are valid
8 years ago
Kegan Dougal
d92a95ff4e
Linting
8 years ago
Kegan Dougal
0b09fcd86d
First working feed reader version
8 years ago