Using CIF to create content for ArcSight – Part 1
April 29, 2012 5 Comments
If you use ArcSight hopefully by now you have come across the great ArcOSI Project for generating content for use within ArcSight. I have used it in the past and liked it but I found myself having to look for more context around the alerts it generated. I recently came across the Collective Intelligence Framework (CIF) and really like how many intel sources it aggregates like ArcOSI does and how it stores the data from the intel source and I think this too can be a great source of content for ArcSight. I have previously blogged about integrating CIF and ArcSight, but that was just using CIF as a tool for looking up data with in ArcSight not using CIF to create content to be used by ArcSight.
EDIT: 6/10/2012 if you haven’t seen @kylemaxwell ‘s Post Introduction to the Collective Intelligence Framework I highly recommend check it out!
I think the content CIF can provide could be great for ActiveLists and Correlation rules on those active lists. I came up with a few possible scenarios on how this content could be used:
- Malicious Domain Queries – DNS Logs
- Malicious Domain Web Traffic – Proxy Logs
- Malicious IP Traffic – Firewall/Proxy Logs
- Scanner Traffic – SSH/Firewall Logs
For the Scanner Traffic maybe instead of reporting on the noise of someone knocking on your door, you report on any traffic that was accepted (meaning authentication happened) but that is up to you.
I have been working on a python script that assumes you are using the CIF Perl client to generate feed data in csv format, then the script will parse the csv files and send them like ArcOSI does to ArcSight via CEF over syslog. I have posted the script and a quick tutorial on it over at the Google Code Project cif-csv-parse-to-cef.
A quick example for this post will be to generate the domain/malware feed using the medium severity and confidence level of 85, send it to ArcSight and have it add the feed data to an Active List. Part 2 of this post will cover writing a correlation rule to monitor the Active List for actionable data.
Let’s start by first creating the Active List and the Correlation Rule to populate the Active List:
In the Navigation Panel go to Active Lists and right click your personal folder and select New Active List
Next in the Inspect/Edit Panel modify the Active List to meet your needs but in this example it will have the name “Malicious Domains”, it will not expire, 100,000 entries allowed (these settings can be changed later) Now set the fields the Active List will use. I have entered:
Domain, Source, Confidence, Description
Click Apply and all that is left is to create the correlation rule to populate the Active List.
Next add a name for your rule
Then click on the Conditions field and create the following filter.
Next click on the Actions tab and make sure you De-Activate the Trigger for On First Event| Action. Then activate the On Every Event Trigger
After activating the On Every Event Trigger right click and select Add -> Active List -> Add to Active List
Select the active list you previously created in this case. Malicious Domains
Once you click Ok, you will most likely get a pop up message similar to this that asks if you want to add all the ArcSight Fields you mapped in the previous step to the aggregation tab. Click yes, if you don’t then your active list will be blank after the rule fires.
Now deploy the rule as a real time rule. Your account will need privileges to do that, If you don’t have them ask your ArcSight Admin to deploy the rule for you.
Now the rule and active list have been created let’s generate content for the rule to populate the active list with.
Let’s start by generating the csv:
$ cif -q domain/malware -s medium -c 85 -p csv > dom_malware.csv
Now run the cifcsv.py script
$./cifcsv.py -f dom_malware.csv -s 192.168.100.154 -p 514 -t Domain
You will see output on the screen similar to this
<29>CEF:0|CIF|CIF 0.1|100|1|CIF Malicious Domain|1|shost=7daily-homebusiness7.net cs1=www.spamhaus.org/sbl/sbl.lasso?query=sbl112756 cs1Label=Source cs2=85 cs2Label=ConfidenceLevel cs3=malware cs3Label=Description
Now if you have an Active Channel up and running with a filter for Device Vendor = CIF and Name = CIF Malicious Domain you should see something similar to this.
Now if you right click your active list and show entries you should also see that your Active List is being populated with data.
This concludes Part 1 – Part 2 will cover writing a correlation rule to monitor the Active List for actionable data.