Description:
I like news. I receive variety of news everyday. All news fall into email and sorted to different folders. Few years ago many news sites began to disable RSS support and now I have to open full version of sites to read the news. But these sites also posted their news inside Twitter, Facebook, Telegram. Because of this I wrote a tool which allow me grab news from these new data sources ... and fill my Elasticsearch with a new data too.
Quick start:
Clone and install:
git clone https://github.com/livelace/mosquito.git
cd mosquito
apt-get install phantomjs
pip install -r requirements.txt
pip install .
Save the configuration file (~/.mosquito.ini):
[main]
grab_timeout = 60
smtp_server = mail.example.com
smtp_port = 25
smtp_usessl = true
smtp_auth = true
smtp_from = mosquito@example.com
smtp_username = user@example.com
smtp_password = Passw0rD
verbose = info
[twitter]
consumer_key = <CONSUMER_KEY>
consumer_secret = <CONSUMER_SECRET>
access_token_key = <ACCESS_KEY>
access_token_secret = <ACCESS_SECRET>
List virtual machines:
mosquito list
+----+---------+---------+--------+---------------------+-----------------+-------------+--------+--------------------------+---------------------+
| ID | Enabled | Plugin | Source | Destination | Update interval | Description | Regexp | Regexp action | Last update |
+----+---------+---------+--------+---------------------+-----------------+-------------+--------+--------------------------+---------------------+
| 1 | True | twitter | rianru | o.popov@livelace.ru | 1s | grab | .* | grab=full | 2016-09-16 12:27:37 |
| | | | | | | | | subject=Twitter | |
| | | | | | | | | priority=high | |
| | | | | | | | | header=X-mosquito:rianru | |
+----+---------+---------+--------+---------------------+-----------------+-------------+--------+--------------------------+---------------------+