Using CIF to create content for ArcSight – Part 1

If you use ArcSight hopefully by now you have come across the great ArcOSI Project for generating content for use within ArcSight. I have used it in the past and liked it but I found myself having to look for more context around the alerts it generated. I recently came across the Collective Intelligence Framework (CIF) and really like how many intel sources it aggregates like ArcOSI does and how it stores the data from the intel source and I think this too can be a great source of content for ArcSight. I have previously blogged about integrating CIF and ArcSight, but that was just using CIF as a tool for looking up data with in ArcSight not using CIF to create content to be used by ArcSight.

EDIT: 6/10/2012 if you haven’t seen @kylemaxwell ‘s Post Introduction to the Collective Intelligence Framework I highly recommend check it out!


I think the content CIF can provide could be great for ActiveLists and Correlation rules on those active lists. I came up with a few possible scenarios on how this content could be used:

  • Malicious Domain Queries – DNS Logs
  • Malicious Domain Web Traffic – Proxy Logs
  • Malicious IP Traffic – Firewall/Proxy Logs
  • Scanner Traffic – SSH/Firewall Logs

For the Scanner Traffic maybe instead of reporting on the noise of someone knocking on your door, you report on any traffic that was accepted (meaning authentication happened) but that is up to you.

I have been working on a python script that assumes you are using the CIF Perl client to generate feed data in csv format, then the script will parse the csv files and send them like ArcOSI does to ArcSight via  CEF over syslog. I have posted the script and a quick tutorial on it over at the Google Code Project cif-csv-parse-to-cef.

A quick example for this post will be to generate the domain/malware feed using the medium severity and confidence level of 85, send it to ArcSight and have it add the feed data to an Active List. Part 2 of this post will cover writing a correlation rule to monitor the Active List for actionable data.

Let’s start by first creating the Active List and the Correlation Rule to populate the Active List:

In the Navigation Panel go to Active Lists and right click your personal folder and select New Active List

New Active List

Next in the Inspect/Edit Panel modify the Active List to meet your needs but in this example it will have the name “Malicious Domains”, it will not expire, 100,000 entries allowed (these settings can be changed later) Now set the fields the Active List will use. I have entered:
Domain, Source, Confidence, Description

Active List Edit Panel

Click Apply and all that is left is to create the correlation rule to populate the Active List.

New Rule

Next add a name for your rule

Rule Name

Then click on the Conditions field and create the following filter.

Rule Conditions

Next click on the Actions tab and make sure you De-Activate the Trigger for On First Event| Action. Then activate the On Every Event Trigger

Deactivate Trigger

After activating the On Every Event Trigger right click and select Add -> Active List -> Add to Active List

Select the active list you previously created in this case. Malicious Domains

Select Active ListAfter selecting the Active List you will have to map ArcSight event Fields to the corresponding Active list fields.

Active List Action

Once you click Ok, you will most likely get a pop up message similar to this that asks if you want to add all the ArcSight Fields you mapped in the previous step to the aggregation tab. Click yes, if you don’t then your active list will be blank after the rule fires.

Aggregation Question

Now deploy the rule as a real time rule. Your account will need privileges to do that, If you don’t have them ask your ArcSight Admin to deploy the rule for you.

Now the rule and active list have been created let’s generate content for the rule to populate the active list with.

Let’s start by generating the csv:

$ cif -q domain/malware -s medium -c 85 -p csv > dom_malware.csv

Now run the script

$./ -f dom_malware.csv -s -p 514 -t Domain

You will see output on the screen similar to this

<29>CEF:0|CIF|CIF 0.1|100|1|CIF Malicious Domain|1| cs1Label=Source cs2=85 cs2Label=ConfidenceLevel cs3=malware cs3Label=Description

Now if you have an Active Channel up and running with a filter for Device Vendor = CIF and  Name = CIF Malicious Domain you should see something similar to this.

CIF Active Channel

Now if you right click your active list and show entries you should also see that your Active List is being populated with data.

Populated Active List

This concludes Part 1 – Part 2 will cover writing a correlation rule to monitor the Active List for actionable data.

Happy Hunting!

5 Responses to Using CIF to create content for ArcSight – Part 1

  1. Pingback: Using CIF to create content for ArcSight – Part 2 « DFIR Journal

  2. elhoim says:

    ArcOSI is an interesting related open-source project:

  3. Pingback: ArcOSI / BadHarvest « insecuriosity

  4. Red says:

    What type of connector is the CEF messages being sent to in your environment? I have tried a dozen different ways of getting the CIF database entries into Arcsight Express without any luck. I also had to modify the message type line in the parser section of the python script to get it to run on any of my systems. Any help would be greatly appreciated.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: