Abusehelper
Encyclopedia
AbuseHelper is an open-source project initiated by CERT.FI (Finland) and CERT.EE (Estonia) with ClarifiedNetworks to automatically process incidents notifications.
This tool is being developed for CERTs) and ISP
s) to help them in their daily job of following and treating a wide range of high-volume information sources. It is interesting to note that the framework can also be used for automatically processing (standardised) information from a wide range of sources.
, BotNet Botnet
, ...). These notifications are often normalized per feed (each feed typically uses different formats to report). There is also a lot of information available on some information provider on internet (Zone-H Zone-H
http://www.zone-h.org/, DShield Dshield
http://www.dshield.org/, Zeus Tracker Zeus (trojan horse)
https://zeustracker.abuse.ch/...). The amount of information is too high for manual processing The goal of AbuseHelper is to take all these sources and try to produce useful report and dashboard for the people that need to treat all these notifications. There is also a try to automate as much as possible repetitive process like founding the owner of some IP address with public databaseses (like WHOIS Whois
).
The community is working on being able to handle more type of input formats. Each type of input is handle by a dedicated bot.
This tool is being developed for CERTs) and ISP
Internet service provider
An Internet service provider is a company that provides access to the Internet. Access ISPs directly connect customers to the Internet using copper wires, wireless or fiber-optic connections. Hosting ISPs lease server space for smaller businesses and host other people servers...
s) to help them in their daily job of following and treating a wide range of high-volume information sources. It is interesting to note that the framework can also be used for automatically processing (standardised) information from a wide range of sources.
Context
CERTs and ISP have to handle really high-volume of events notifications (SPAM E-mail spamE-mail spam
Email spam, also known as junk email or unsolicited bulk email , is a subset of spam that involves nearly identical messages sent to numerous recipients by email. Definitions of spam usually include the aspects that email is unsolicited and sent in bulk. One subset of UBE is UCE...
, BotNet Botnet
Botnet
A botnet is a collection of compromised computers connected to the Internet. Termed "bots," they are generally used for malicious purposes. When a computer becomes compromised, it becomes a part of a botnet...
, ...). These notifications are often normalized per feed (each feed typically uses different formats to report). There is also a lot of information available on some information provider on internet (Zone-H Zone-H
Zone-H
Zone-H is an archive of defaced websites. Once a defaced website is submitted to Zone-H, it is mirrored on the Zone-H servers, it is then moderated by the Zone-H staff to check if the defacement was fake....
http://www.zone-h.org/, DShield Dshield
DShield
DShield is a community-based collaborative firewall log correlation system. It receives logs from volunteers world wide and uses them to analyze attack trends. It is used as the data collection engine behind the SANS Internet Storm Center . It was officially launched end of November 2000 by...
http://www.dshield.org/, Zeus Tracker Zeus (trojan horse)
Zeus (trojan horse)
Zeus is a Trojan horse that steals banking information by keystroke logging and Form Grabbing. Zeus is spread mainly through drive-by downloads and phishing schemes. First identified in July 2007 when it was used to steal information from the United States Department of Transportation, it became...
https://zeustracker.abuse.ch/...). The amount of information is too high for manual processing The goal of AbuseHelper is to take all these sources and try to produce useful report and dashboard for the people that need to treat all these notifications. There is also a try to automate as much as possible repetitive process like founding the owner of some IP address with public databaseses (like WHOIS Whois
WHOIS
WHOIS is a query and response protocol that is widely used for querying databases that store the registered users or assignees of an Internet resource, such as a domain name, an IP address block, or an autonomous system, but is also used for a wider range of other information. The protocol stores...
).
History
Technical developments that led to collaborative effort on solving the automated collection of Abuse Information- 2005 CERT-FI Autoreporter gen1, implemented with Perl
- 2006-2007 CERT-FI Autoreporter generations 2&3 (incremental updates to gen1). Plans to rewirte
- 2008-2009 CERT-FI Autoreporter gen4, proof-of-concept implementation using sh. Paper describing the prototype won a reward
- 2009 CERT-FI gen5, implemented with Python. Full rewrite
- 2009-10 Clarified Networks & CERT-EE Abusehelper collaboration starts
- 2009-11 CERT-FI joins.
- 2010-01 AbuseHelper first public release
- 2010-01 First training @ TF-CSIRT event in Germany
- 2010-03 CERT.BE (Belgium) / BELNET CERT join.
Architecture
AbuseHelper is written in Python and developed relying on XMPP protocol (not mandatory) and agents. The base principle is to control agent via a central chat room where all bots are listening. Agents are exchanging information in subrooms. AbuseHelper is then scalable and each agent follows a KISS (Keep it Simple and Stupid) approach. Each user is able to produce the perfect workflow for his business. The user just needs to take the agents he needs and connect them together.Sources
The goal of AbuseHelper is to be able to handle a large panel of sources and try to extract useful information for event follow-up. Currently, AbuseHelper is able to parse the following types of sources:- email containing an ARF attachment (Abuse Report Format) (like CleanMX http://www.clean-mx.de/ or AbusIX http://abusix.org/);
- email with a CSV (could be zipped) attachment or an URL to a CSV file (like ShadowServer http://www.shadowserver.org/wiki/) ;
- IRC events (live feed);
- XMPP events (live feed);
- DShield events (http://www.dshield.org/).
The community is working on being able to handle more type of input formats. Each type of input is handle by a dedicated bot.
Internal information processing
AbuseHelper is more than a pipe. In the workflow, it could be decided to add extra informations coming from other sources like:- Whois to retrieve abuse contact (typically people you have to contact when something security related happened);
- CRM (Customer Relation Management) to retrieve the same kind of info than for Whois.
Output
As AbuseHelper should help to handle incidents, a large panels of output has also to be handled. Per default, AbuseHelper could produces the following kind of reports:- Mail reports with digests of events and a CSV attachments with all observed events for a time frame following some conditions;
- Wiki report - AbuseHelper wrote the incidents to a wiki;
- SQL report - AbuseHelper writes all events to a SQL database.
Generic agents
At all steps, there is some standards agents:- Roomgraph to transport events for one chat room to another based on some rules;
- Historian to log all events observed in each chat room.
Community
AbuseHelper is developed by a open-source community composed by:- Clarified Networks https://www.clarifiednetworks.com/
- CERT.FI (Finland) http://www.cert.fi/en/index.html
- CERT.EE (Estonia) http://www.cert.ee/
- CERT.BE (Belgium) / BELNET CERT https://www.cert.be/ / http://www.belnet.be
- CSC/FUNET (Finland)
- University of Oulu (OUSPG)