"Your security operations or network operations tools cannot run in isolation anymore. True SOC operations is about cohesion of data."
The common need I get from many Network & Security Operations Centers (NOC & SOC) analysts is their wish to integrate data from their various feeds into UI frameworks and dashboards. How about NetFlow data itself, NetFlow basically is (at a minimum) a 5-tuple summary of network traffic that crosses your perimeter (firewall or router). If your NetFlow data goes into a database there are a few tools to translate your database queries to JSON/XML and make these available over webservices. In this blog I am introducing a project that will help expose SiLK NetFlow data over webservices JSON/XML/CSV to able to integrate SiLK data into your NOC/SOC dashboard with some basic examples. The project is in GitHub called silkweb.
The project silkweb uses SiLK's python API to provide the typical NetFlow analysis capabilities over a simple webservices. Silkweb allows you to query SiLK data repository using a web query with GET or POST parameter to ask your typical questions and gather results in JSON/XML or CSV format. Using any number of web visualization tools like D3 or Bootstrap one can build useful visualization and data representation UI that can integrate to any of the UI frameworks.
Some examples of the analysis are show below
1. Search for a communication between to a bad IP Address 39.153.26.14 for Sept 19th
Query
https://demo.org/cgi-bin/silk/silkapi.py?out_type=json&types=out,outweb&start_date=2015/09/19:00&end_date=2015/09/19:23&istart=0&iend=1000&dip=39.153.26.14
Results
{"rows_searched": 19756, "header": {"timestamp": "1442682747", "version": "1.57"}, "gdata": [{"sensor_id": 6, "duration_secs": 0.15, "rowid": 1, "packets": 1, "classname": "all", "tcpflags": "", "classtype_id": 1, "initflags": null, "duration": "0:00:00.150000", "nhip": "0.0.0.0", "sport": 49472, "uniform_packets": false, "sip": "10.168.5.186", "stime_epoch_secs": 1442620801.414, "timeout_killed": false, "timeout_started": false, "session_tcpflags": null, "protocol": 17, "etime": "2015-09-19 00:00:01.564000", "application": 0, "initial_tcpflags": null, "input": 3, "icmpcode": 53, "finnoack": false, "icmptype": 0, "stime": "2015-09-19 00:00:01.414000", "restflags": null, "sensor": "asa-north", "classtype": ["all", "out"], "bytes": 30, "typename": "out", "etime_epoch_secs": 1442620801.564, "dport": 53, "output": 3, "dip": "39.153.26.14"}],"rows": "1", "query_conditions": {"end": "2015/09/19:23", "istart": "0", "out_type": "json", "classname": "all", "start": "2015/09/19:00", "sortby": "bytes", "stats": null, "sensors": ["asa-north", "juniper-edge", "extreme-p"], "iend": "1000", "types": ["out", "outweb"]}}
2. Search for internal IP address violating DNS policy to query DNS servers (not our DNS servers 172.22.0.0/24 subnet) for Sept 19th 1.30 AM when IDS triggered large DNS exfil.
https://demo.org/cgi-bin/silk/silkapi.py?out_type=json&types=out&dport=53&dip=!172.22.0.0/24&start_date=2015/09/19:01&end_date=2015/09/19:02&istart=0&iend=1000
{"rows_searched": 4000, "header": {"timestamp": "1442683883", "version": "1.57"}, "gdata": [{"sensor_id": 6, "duration_secs": 0.03, "rowid": 4, "packets": 1, "classname": "all", "tcpflags": "", "classtype_id": 1, "initflags": null, "duration": "0:00:00.030000", "nhip": "0.0.0.0", "sport": 58255, "uniform_packets": false, "sip": "10.168.5.80", "stime_epoch_secs": 1442626052.852, "timeout_killed": false, "timeout_started": false, "session_tcpflags": null, "protocol": 17, "etime": "2015-09-19 01:27:32.882000", "application": 0, "initial_tcpflags": null, "input": 4, "icmpcode": 53, "finnoack": false, "icmptype": 0, "stime": "2015-09-19 01:27:32.852000", "restflags": null, "sensor": "asa-north", "classtype": ["all", "out"], "bytes": 40, "typename": "out", "etime_epoch_secs": 1442626052.882, "dport": 53, "output": 4, "dip": "203.18.98.8"}], "rows": "1", "query_conditions": {"end": "2015/09/19:02", "istart": "0", "iend": "1000", "out_type": "json", "classname": "all", "start": "2015/09/19:01", "sortby": "bytes", "dport": "53", "stats": null, "sensors": ["asa-north", "juniper-edge", "extreme-p"], "dip": "!172.22.0.0/24", "types": ["out"]}}
3. Show top-10 stats of users (by bytes used) attempting to bypass proxy and go directly to external HTTPS sites.
http://demo.org/cgi-bin/silk/silkapi.py?out_type=json&types=outweb,out&dport=443,8443&dip=!172.19.11.1&sortby=bytes&stats=sip,dip&start_date=2015/09/19:00&end_date=2015/09/19:01&istart=0&iend=10
{"rows_searched": 120039, "header": {"timestamp": "1442684442", "version": "1.57"}, "gdata": [{"sip": "10.168.5.67", "bytes": 16371638, "packets": 1, "records": 1, "rowid": 1, "dip": "216.58.219.80"}, {"sip": "10.168.5.73", "bytes": 2803064, "packets": 6, "records": 6, "rowid": 2, "dip": "216.58.192.101"}], "rows": "10", "query_conditions": {"end": "2015/09/19:01", "istart": "0", "iend": "2", "out_type": "json", "classname": "all", "start": "2015/09/19:00", "sortby": "bytes", "dport": "80,443,8080,8443", "stats": "sip,dip", "sensors": ["asa-north", "juniper-edge"], "dip": "!172.19.11.1", "types": ["outweb", "out"]}}
3. Show top-10 stats of internal subnets /24 (by bytes used) attempting to bypass proxy and go directly to external HTTPS sites.
http://demo.org/cgi-bin/silk/silkapi.py?out_type=json&types=outweb,out&dport=443,8443&dip=!172.19.11.1&sortby=bytes&stats=sip/24,dip&start_date=2015/09/19:00&end_date=2015/09/19:01&istart=0&iend=10
{"rows_searched": 120039, "header": {"timestamp": "1442684442", "version": "1.57"}, "gdata": [{"sip/24": "10.168.5.0/24", "bytes": 16371638, "packets": 1, "records": 1, "rowid": 1, "dip": "216.58.219.80"}, {"sip": "10.168.5.73", "bytes": 2803064, "packets": 6, "records": 6, "rowid": 2, "dip": "216.58.192.101"}], "rows": "10", "query_conditions": {"end": "2015/09/19:01", "istart": "0", "iend": "10", "out_type": "json", "classname": "all", "start": "2015/09/19:00", "sortby": "bytes", "dport": "80,443,8080,8443", "stats": "sip,dip", "sensors": ["asa-north", "juniper-edge"], "dip": "!172.19.11.1", "types": ["outweb", "out"]}}
4. Product an MRTG looking 5-minute bin traffic graph for Sep. 19th 8.00 AM to 10:00 AM of only UDP traffic through firewall asa-north
http://demo.org/cgi-bin/silk/silkapi.py?out_type=json&sensor=asa-north&protocol=17&sortby=stime&stats=stime/300&start_date=2015/09/19:08&end_date=2015/09/19:10&istart=0&iend=1000
{"rows_searched": 2392876, "header": {"timestamp": "1442687429", "version": "1.57"}, "gdata": [{"bytes": 108698, "packets": 281, "rowid": 1, "records": 281, "stime/300": "2015-09-19 10:55:00-2015-09-19 11:00:00"}, {"bytes": 131395, "packets": 271, "rowid": 2, "records": 271, "stime/300": "2015-09-19 10:50:00-2015-09-19 10:55:00"}, {"bytes": 209696, "packets": 457, "rowid": 3, "records": 457, "stime/300": "2015-09-19 10:45:00-2015-09-19 10:50:00"}, {"bytes": 202554, "packets": 395, "rowid": 4, "records": 395, "stime/300": "2015-09-19 10:40:00-2015-09-19 10:45:00"}, {"bytes": 73887, "packets": 170, "rowid": 5, "records": 170, "stime/300": "2015-09-19 10:35:00-2015-09-19 10:40:00"}, {"bytes": 134565, "packets": 307, "rowid": 6, "records": 307, "stime/300": "2015-09-19 10:30:00-2015-09-19 10:35:00"},..],"rows": "36", "query_conditions": {"stats": "stime/300", "end": "2015/09/19:10", "istart": "0", "iend": "1000", "out_type": "json", "classname": "all", "start": "2015/09/19:08", "sortby": "time", "protocol": "17", "sensors": ["asa-north"], "sensor": "asa-north", "types": ["in", "inweb", "inicmp"]}}
You can get a D3 bar chart of the data as seen below:
Note: any of the numeric queries can include a range, greater than or less than for e.g., port=137-139, port=-25 (less than or equal to 25 or 0-25) or port=1023- (greater than or equal to 1023).
The common need I get from many Network & Security Operations Centers (NOC & SOC) analysts is their wish to integrate data from their various feeds into UI frameworks and dashboards. How about NetFlow data itself, NetFlow basically is (at a minimum) a 5-tuple summary of network traffic that crosses your perimeter (firewall or router). If your NetFlow data goes into a database there are a few tools to translate your database queries to JSON/XML and make these available over webservices. In this blog I am introducing a project that will help expose SiLK NetFlow data over webservices JSON/XML/CSV to able to integrate SiLK data into your NOC/SOC dashboard with some basic examples. The project is in GitHub called silkweb.
The project silkweb uses SiLK's python API to provide the typical NetFlow analysis capabilities over a simple webservices. Silkweb allows you to query SiLK data repository using a web query with GET or POST parameter to ask your typical questions and gather results in JSON/XML or CSV format. Using any number of web visualization tools like D3 or Bootstrap one can build useful visualization and data representation UI that can integrate to any of the UI frameworks.
Some examples of the analysis are show below
1. Search for a communication between to a bad IP Address 39.153.26.14 for Sept 19th
Query
https://demo.org/cgi-bin/silk/silkapi.py?out_type=json&types=out,outweb&start_date=2015/09/19:00&end_date=2015/09/19:23&istart=0&iend=1000&dip=39.153.26.14
Results
{"rows_searched": 19756, "header": {"timestamp": "1442682747", "version": "1.57"}, "gdata": [{"sensor_id": 6, "duration_secs": 0.15, "rowid": 1, "packets": 1, "classname": "all", "tcpflags": "", "classtype_id": 1, "initflags": null, "duration": "0:00:00.150000", "nhip": "0.0.0.0", "sport": 49472, "uniform_packets": false, "sip": "10.168.5.186", "stime_epoch_secs": 1442620801.414, "timeout_killed": false, "timeout_started": false, "session_tcpflags": null, "protocol": 17, "etime": "2015-09-19 00:00:01.564000", "application": 0, "initial_tcpflags": null, "input": 3, "icmpcode": 53, "finnoack": false, "icmptype": 0, "stime": "2015-09-19 00:00:01.414000", "restflags": null, "sensor": "asa-north", "classtype": ["all", "out"], "bytes": 30, "typename": "out", "etime_epoch_secs": 1442620801.564, "dport": 53, "output": 3, "dip": "39.153.26.14"}],"rows": "1", "query_conditions": {"end": "2015/09/19:23", "istart": "0", "out_type": "json", "classname": "all", "start": "2015/09/19:00", "sortby": "bytes", "stats": null, "sensors": ["asa-north", "juniper-edge", "extreme-p"], "iend": "1000", "types": ["out", "outweb"]}}
2. Search for internal IP address violating DNS policy to query DNS servers (not our DNS servers 172.22.0.0/24 subnet) for Sept 19th 1.30 AM when IDS triggered large DNS exfil.
https://demo.org/cgi-bin/silk/silkapi.py?out_type=json&types=out&dport=53&dip=!172.22.0.0/24&start_date=2015/09/19:01&end_date=2015/09/19:02&istart=0&iend=1000
{"rows_searched": 4000, "header": {"timestamp": "1442683883", "version": "1.57"}, "gdata": [{"sensor_id": 6, "duration_secs": 0.03, "rowid": 4, "packets": 1, "classname": "all", "tcpflags": "", "classtype_id": 1, "initflags": null, "duration": "0:00:00.030000", "nhip": "0.0.0.0", "sport": 58255, "uniform_packets": false, "sip": "10.168.5.80", "stime_epoch_secs": 1442626052.852, "timeout_killed": false, "timeout_started": false, "session_tcpflags": null, "protocol": 17, "etime": "2015-09-19 01:27:32.882000", "application": 0, "initial_tcpflags": null, "input": 4, "icmpcode": 53, "finnoack": false, "icmptype": 0, "stime": "2015-09-19 01:27:32.852000", "restflags": null, "sensor": "asa-north", "classtype": ["all", "out"], "bytes": 40, "typename": "out", "etime_epoch_secs": 1442626052.882, "dport": 53, "output": 4, "dip": "203.18.98.8"}], "rows": "1", "query_conditions": {"end": "2015/09/19:02", "istart": "0", "iend": "1000", "out_type": "json", "classname": "all", "start": "2015/09/19:01", "sortby": "bytes", "dport": "53", "stats": null, "sensors": ["asa-north", "juniper-edge", "extreme-p"], "dip": "!172.22.0.0/24", "types": ["out"]}}
3. Show top-10 stats of users (by bytes used) attempting to bypass proxy and go directly to external HTTPS sites.
http://demo.org/cgi-bin/silk/silkapi.py?out_type=json&types=outweb,out&dport=443,8443&dip=!172.19.11.1&sortby=bytes&stats=sip,dip&start_date=2015/09/19:00&end_date=2015/09/19:01&istart=0&iend=10
{"rows_searched": 120039, "header": {"timestamp": "1442684442", "version": "1.57"}, "gdata": [{"sip": "10.168.5.67", "bytes": 16371638, "packets": 1, "records": 1, "rowid": 1, "dip": "216.58.219.80"}, {"sip": "10.168.5.73", "bytes": 2803064, "packets": 6, "records": 6, "rowid": 2, "dip": "216.58.192.101"}], "rows": "10", "query_conditions": {"end": "2015/09/19:01", "istart": "0", "iend": "2", "out_type": "json", "classname": "all", "start": "2015/09/19:00", "sortby": "bytes", "dport": "80,443,8080,8443", "stats": "sip,dip", "sensors": ["asa-north", "juniper-edge"], "dip": "!172.19.11.1", "types": ["outweb", "out"]}}
3. Show top-10 stats of internal subnets /24 (by bytes used) attempting to bypass proxy and go directly to external HTTPS sites.
http://demo.org/cgi-bin/silk/silkapi.py?out_type=json&types=outweb,out&dport=443,8443&dip=!172.19.11.1&sortby=bytes&stats=sip/24,dip&start_date=2015/09/19:00&end_date=2015/09/19:01&istart=0&iend=10
{"rows_searched": 120039, "header": {"timestamp": "1442684442", "version": "1.57"}, "gdata": [{"sip/24": "10.168.5.0/24", "bytes": 16371638, "packets": 1, "records": 1, "rowid": 1, "dip": "216.58.219.80"}, {"sip": "10.168.5.73", "bytes": 2803064, "packets": 6, "records": 6, "rowid": 2, "dip": "216.58.192.101"}], "rows": "10", "query_conditions": {"end": "2015/09/19:01", "istart": "0", "iend": "10", "out_type": "json", "classname": "all", "start": "2015/09/19:00", "sortby": "bytes", "dport": "80,443,8080,8443", "stats": "sip,dip", "sensors": ["asa-north", "juniper-edge"], "dip": "!172.19.11.1", "types": ["outweb", "out"]}}
4. Product an MRTG looking 5-minute bin traffic graph for Sep. 19th 8.00 AM to 10:00 AM of only UDP traffic through firewall asa-north
http://demo.org/cgi-bin/silk/silkapi.py?out_type=json&sensor=asa-north&protocol=17&sortby=stime&stats=stime/300&start_date=2015/09/19:08&end_date=2015/09/19:10&istart=0&iend=1000
{"rows_searched": 2392876, "header": {"timestamp": "1442687429", "version": "1.57"}, "gdata": [{"bytes": 108698, "packets": 281, "rowid": 1, "records": 281, "stime/300": "2015-09-19 10:55:00-2015-09-19 11:00:00"}, {"bytes": 131395, "packets": 271, "rowid": 2, "records": 271, "stime/300": "2015-09-19 10:50:00-2015-09-19 10:55:00"}, {"bytes": 209696, "packets": 457, "rowid": 3, "records": 457, "stime/300": "2015-09-19 10:45:00-2015-09-19 10:50:00"}, {"bytes": 202554, "packets": 395, "rowid": 4, "records": 395, "stime/300": "2015-09-19 10:40:00-2015-09-19 10:45:00"}, {"bytes": 73887, "packets": 170, "rowid": 5, "records": 170, "stime/300": "2015-09-19 10:35:00-2015-09-19 10:40:00"}, {"bytes": 134565, "packets": 307, "rowid": 6, "records": 307, "stime/300": "2015-09-19 10:30:00-2015-09-19 10:35:00"},..],"rows": "36", "query_conditions": {"stats": "stime/300", "end": "2015/09/19:10", "istart": "0", "iend": "1000", "out_type": "json", "classname": "all", "start": "2015/09/19:08", "sortby": "time", "protocol": "17", "sensors": ["asa-north"], "sensor": "asa-north", "types": ["in", "inweb", "inicmp"]}}
You can get a D3 bar chart of the data as seen below:
Note: any of the numeric queries can include a range, greater than or less than for e.g., port=137-139, port=-25 (less than or equal to 25 or 0-25) or port=1023- (greater than or equal to 1023).
No comments:
Post a Comment