tcpflow/0000755000175000017500000000000013375371167011157 5ustar dimadimatcpflow/CONFIGURE_FEDORA_26.sh0000755000175000017500000000040513342641114014347 0ustar dimadimasudo dnf -y install emacs sudo dnf -y install zlib sudo dnf -y install zlib-static sudo dnf -y install openssl-devel sudo dnf -y install boost-devel sudo dnf -y install cairo-devel libpcap-devel sudo dnf -y install libpcap-devel sudo dnf -y install cairo-devel tcpflow/doc/0000755000175000017500000000000013347320502011706 5ustar dimadimatcpflow/doc/Planning-1page.txt0000644000175000017500000000611113342626673015224 0ustar dimadima[mockup of 1 page report] tcpflow input: Date range: ISO-8601 to ISO-8601 Packets Analyzed: COUNT (XXX MB) Protocols: IPv4 (%); IPv6 (%); ARP (%); Other (?) +-------------------------------------------------------------+ | | | | | Bandwidth Histogram | | | +-------------------------------------------------------------+ +-------------------------------------------------------------+ | | | | | MAP | | | +-------------------------------------------------------------+ +-----------------------+ +-----------------------+ | | | | | | | | | | | | | Top Server IPs | -> | Top Client IPs | | | | | | | | | +-----------------------+ +-----------------------+ Top #1: IP (%) Top #1: IP (%) Top #2: IP (%) Top #2: IP (%) Top #3: IP (%) Top #3: IP (%) +-----------------------+ +-----------------------+ | | | | | | | | | | | | | Top Server Ports | -> | Top Client Ports | | | | | | | | | +-----------------------+ +-----------------------+ Top #1: Port (%) Top #1: Port (%) Top #2: Port (%) Top #2: Port (%) Top #3: Port (%) Top #3: Port (%) +-----------------------+ +-----------------------+ | | | | | | | | | | | | | | | Observed Downloaded | | | | Types | | | | | +-----------------------+ +-----------------------+ ==== Notes: 1 - DNS Resolution: 1.1 - IP addresses only. 1.2 - Passive DNS 1.3 - Passive DNS augmented by a list of DNS records (in PCAP format) 1.4 - Generate a list of IP addresses requiring resolution tcpflow/doc/announce_1_3.txt0000644000175000017500000000061713342626673014740 0ustar dimadimaI'm pleased to announce the release of tcpflow version 1.3.0. Key elements in 1.3.0 include: - Compiles with mingw32 and mingw64 for 32-bit and 64-bit windows. I am now distributing pre-compiled binaries of some releases. - Better support for DFXML (fixed some bugs) You can download version 1.3.0, both source and precompiled windows binaries, from: https://github.com/simsong/tcpflow/downloads tcpflow/doc/announce_1_4.txt0000644000175000017500000000725213342626673014743 0ustar dimadimaI'm pleased to announce the release of tcpflow version 1.4.0. Key elements in 1.4.0 include: Completely rewritten TCP implementation that: * Handles TCP flows larger than 4GiB * Handles TCP packets sent after a connection is closed. Such packets are compared with the packets from the connection that were already received. If the packets match, they are discarded as retransmissions. If they do not match they are put in new transcript files. Incompatiable Changes: * -e (alternate colors of console output) has been renamed -J so that -e and -E can be used in a manner consistent with bulk_extractor Completely rewritten HTTP parser * Handles multiple HTTP objects per connections * Optional http_cmd will run a program or script for each HTTP attachment received as received. For example, to run the program /bin/echo and provide the filename of the attachment for each attachment, specify: tcpflow -Shttp_cmd=/bin/echo ... * Optional timeout on tcp connections, causes processing of HTTP objects before the HTTP connection closes: -Stcp_timeout=10 * Optional alert file descriptor causes http parser to alert as files are opened and closed by the HTTP parser. e.g.: $ ./tcpflow -o out -a -E http -S http_alert_fd=1 -r ../tests/multifile_25_21.pcap open out/038.122.002.045.00080-192.168.123.101.04634-HTTPBODY-001.png close out/038.122.002.045.00080-192.168.123.101.04634-HTTPBODY-001.png open out/038.122.002.045.00080-192.168.123.101.04637-HTTPBODY-001.png close out/038.122.002.045.00080-192.168.123.101.04637-HTTPBODY-001.png open out/038.122.002.045.00080-192.168.123.101.04637-HTTPBODY-002.png close out/038.122.002.045.00080-192.168.123.101.04637-HTTPBODY-002.png open out/038.122.002.045.00080-192.168.123.101.04648-HTTPBODY-001.png close out/038.122.002.045.00080-192.168.123.101.04648-HTTPBODY-001.png open out/038.122.002.045.00080-192.168.123.101.04649-HTTPBODY-001.png close out/038.122.002.045.00080-192.168.123.101.04649-HTTPBODY-001.png open out/038.122.002.045.00080-192.168.123.101.04654-HTTPBODY-001 close out/038.122.002.045.00080-192.168.123.101.04654-HTTPBODY-001 open out/038.122.002.045.00080-192.168.123.101.04655-HTTPBODY-001 close out/038.122.002.045.00080-192.168.123.101.04655-HTTPBODY-001 open out/038.122.002.045.00080-192.168.123.101.04655-HTTPBODY-002 close out/038.122.002.045.00080-192.168.123.101.04655-HTTPBODY-002 ... New Scalability features: * Automatically creates new directories as necessary for output filenames that include forward slashes ("/"). * -Fk option automatically bins up to a thousand flows in a thousand directories in one directory layer, easily handling up to million flows. * -Fm option automatically bins up to a thousand flows in a million directories (two directory layers), easily handling up to billion flows. * -Fg option automatically bins up to a thousand flows in a billion directories (three directory layers), easily handling up to trillion flows. Additional features: * Produces a one-page visualization (report.pdf) of the packets that were analyzed. * New -w option writes a PCAP file of packets not processed by tcpflow. * Better support for decoding and decompressing HTTP objects, including multiple objects sent over a single HTTP stream. * Full support for the bulk_extractor plug-in system You can download version 1.4.0, both source and precompiled windows binaries, from: https://github.com/simsong/tcpflow/downloads (what we didn't get to) The following is scheduled for release 1.5: * Full handling of radiotap files * Passive DNS implementation * top-100 connection map tcpflow/doc/make_web.sh0000644000175000017500000000146113342626673014033 0ustar dimadima#!/bin/bash # # Create the files for the tcpflow website CORP=/corp/ DEST=/var/www/digitalcorpora/tcpflow/demo TCPFLOW=../src/tcpflow TMP=/tmp/out$$ if [ ! -d $DEST ]; then mkdir -p $DEST ; fi if [ ! -x $TCPFLOW ]; then (cd .. ; make ) ; fi run() { DPDF=$DEST/$2 DPNG=${DPDF%pdf}png echo DPDF=$DPDF echo DPNG=$DPNG echo $TCPFLOW -o $TMP -x tcpdemux -E netviz $1 $TCPFLOW -o $TMP -x tcpdemux -E netviz $1 if [ ! -r $TMP/report.pdf ]; then echo tcpflow failed exit 1 fi mv $TMP/report.pdf $DPDF /bin/rm -rf $TMP convert -scale 300 $DPDF $DPNG ls -l $DPDF $DPNG } run "-r $CORP/nps/packets/2008-nitroba/nitroba.pcap" nitroba.pdf run "-l $CORP/nps/packets/2009-m57-patents/net-2009*.gz" m57-net.pdf run "-l $CORP/mitll/packets/ideval99/week?/*/outside*gz" id99-outside.pdf tcpflow/doc/announce_1_5.txt0000644000175000017500000000212613342641114014722 0ustar dimadimaI'm pleased to announce the release of tcpflow version 1.5.0. This version was going to be called version 1.4.7, but so much has changed that it seemed like a good time to bump the minor version number. It's more than just a bug-fix release! Changes from 1.4.6: * We now pin to simsong/http-parser rather than nodejs/http-parser, so that we have more control over http-parser's .gitignore file. * support for cmake. (May not be current, but it's there for you cmake fans!) * CONFIGURE_FEDORA_26.sh script installs all needed Fedora 26 packages. * CONFIGURE_UBUNTU_16_04.sh script installs all needed Ubuntu 16.04 packages * CONFIGURE_ARCH_17_8.sh installs all needed Arch Linux 17.8 packages! Yes, we now support Arch! * getopt_long() support. Now you can go to town and create long-versions of all your favorite, one-character tcpflow options (provided you know C++, of course. Please submit pull requests!) * tcpflow man page updated * There's a Python post-processor option as well. * chroot() and drop root, allowing better control of privledge. * Support for libcap (capability library). tcpflow/doc/Makefile.am0000644000175000017500000000005413043540601013736 0ustar dimadimaman_MANS = tcpflow.1 CLEANFILES = tcpflow.1 tcpflow/doc/tcpflow.1.in0000644000175000017500000004341513342641114014062 0ustar dimadima.\"edit the file tcpflow.1.in, not tcpflow.1" .\"" .\"" .TH tcpflow 1 "2013-04-13" "tcpflow @VERSION@" "tcpflow @VERSION@" .SH NAME tcpflow \- TCP flow recorder .SH SYNOPSIS .na .B tcpflow [\c .BI \-aBcCDIpsZ\fR\c ] [\c .BI \-b \ max_bytes\fR\c ] [\c .BI \-d \ debug_level\fR\c ] [\c .BI \-[eE] \ scanner\fR\c ] [\c .BI \-f \ max_fds\fR\c ] [\c .BI \-F[ctTXMkmg]\fR\c ] [\c .BI \-h\fR\c |\c .BI \--help\fR\c ] [\c .BI \-i \ iface\fR\c ] [\c .BI \-l \ file1.pcap\ file2.pcap...\fR\c ] [\c .BI \-L \ semlock\fR\c ] [\c .BI \-m \ min_bytes\fR\c ] [\c .BI \-o \ outdir\fR\c ] [\c .BI \-r \ file1.pcap\fR\c ] [\c .BI \-R \ file0.pcap\fR\c ] [\c .BI \-S \ name=value\fR\c ] [\c .BI \-T[filename\ template]\fR\c ] [\c .BI \-U\fR\c |\c .BI \--relinquish-privileges \ username\fR\c ] [\c .BI \-v\fR\c |\c .BI \--verbose\fR\c ] [\c .BI \-V\fR\c |\c .BI \--version\fR\c ] [\c .BI \-w \ file\fR\c ] [\c .BI \-x \ scanner\fR\c ] [\c .BI \-X \ file.xml\fR\c ] [\c .BI \-z\fR\c |\c .BI \--chroot \ directory\fR\c ] [\c .BI expression\fR\c ] .SH DESCRIPTION .LP .B tcpflow is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis or debugging. Rather than showing packet-by-packet information, \fBtcpflow\fP reconstructs the actual data streams and stores each flow in a separate file for later analysis. \fBtcpflow\fP understands TCP sequence numbers and will correctly reconstruct data streams regardless of retransmissions or out-of-order delivery. \fBtcpflow\fP provides control over filenames for automatic binning of connections by protocol, IP address or connection number, and has a sophisticated plug-in system for decompressing compressed HTTP connections, undoing MIME encoding, or calling user-provided programs for post-processing. .LP By default \fBtcpflow\fP stores all captured data in files that have names of the form: .in +.5i .nf \fB192.168.101.102.02345-010.011.012.013.45103\fP .fi .in -.5i \,...where the contents of the above file would be data transmitted from host 192.168.101.102 port 2345, to host 10.11.12.13 port 45103. .LP If you want to simply process a few hundred thousand packets and see what you have, try this: .in +.5i .nf \fBtcpflow -a -o outdir -Fk -r packets.pcap\fP .fi .in -.5i This will cause \fBtcpflow\fP to perform (-a) all processing, store the output in a directory called .BI outdir, bin the output in directories of 1000 connections each, and read its input from the file \fBpackets.pcap\fP. More sophisticated processing is possible, of course. .SH OPTIONS .TP .B \-a Enable all processing. Same as .B \-e all. .TP .B \-B Force binary output even when printing to console with .B -C or .B -c. .TP .B \-b \fImax_bytes\fP Specifies the maximum size of a captured flow. Any bytes beyond \fImax_bytes\fP from the first byte captured will be discarded. The default is to store an unlimited number of bytes per flow. \fBNote:\fP before version 1.4, \fBtcpflow\fP could only store a maximum of 4GiB per flow. .TP .B \-c Console print. Print the contents of packets to stdout as they are received, without storing any captured data to files (implies \fB\-s\fP). .TP .B \-C Console print without the packet source and destination details being printed. Print the contents of packets to stdout as they are received, without storing any captured data to files (implies \fB\-s\fP). .TP .B \-D Console output should be in hex. .TP .B \-d Debug level. Set the level of debugging messages printed to stderr to \fIdebug_level\fP. Higher numbers produce more messages. .B \-d 0 causes completely silent operation. .B \-d 1 , the default, produces minimal status messages. .B \-d 10 produces verbose output equivalent to .B \-v . Numbers higher than 10 can produce a large amount of debugging information useful only to developers. .TP .B \-E name Disable all scanners and then enable scanner .B name .TP .B \-e name Enable scanner .B name. .TP .B \-e all Enables all scanners. Same as .B \-a .TP .B \-e http Perform HTTP post-processing ("After" processing). If the output file is .in +.5i .nf \fB208.111.153.175.00080-192.168.001.064.37314,\fP .fi .in -.5i Then the post-processing will create the files: .in +.5i .nf \fB208.111.153.175.00080-192.168.001.064.37314-HTTP\fP \fB208.111.153.175.00080-192.168.001.064.37314-HTTPBODY\fP .fi .in -.5i If the HTTPBODY was compressed with GZIP, you may get a third file as well: .in +.5i .nf \fB208.111.153.175.00080-192.168.001.064.37314-HTTPBODY-GZIP\fP .fi .in -.5i Additional information about these streams, such as their MD5 hash value, is also written to the .B DFXML report file. .TP .B \-e python \-S py_path=path \-S py_module=module \-S py_function=foo Post-process TCP payload by an external python function. .RS .PP The python function must take a single string parameter. The python function can return a string (else the function does must not return). The returned string (if any) is written in the .B DFXML report file inside the XML tag \fB...\fP. A sample python script is available within the tcpflow source code in directory \fBpython/plugins\fP. .PP Example: .PP .nf \fBtcpflow -r my.cap -e python -S py_path=python/plugins -S py_module=samplePlugin -S py_function=sampleFunction\fP .fi .RE .TP .B \-F[format] Specifies format for output filenames. .RS .TP Format specifiers: .TP .B c Appends the connection counter to ALL filenames. .TP .B t Prepends each filename with a Unix timestamp (seconds since epoch). .TP .B T Prepends each filename with an ISO-8601 timestamp. .TP .B X Do not output any files (other than the .B DFXML report file). .RE .TP .B \-FM Include MD5 of each flow in the .B DFXML report file. .TP .B \-FX Suppresses file output entirely, .B DFXML report file is still produced. .TP .B \-Fk bin output in 1K directories .TP .B \-Fm bin output in 1M directories (2 levels) .TP .B \-Fg bin output in 1G directories (3 levels) .TP .B \-f\fImax_fds\fP Max file descriptors used. Limit the number of file descriptors used by \fBtcpflow\fP to \fImax_fds\fP. Higher numbers use more system resources, but usually perform better. If the underlying operating system supports the .B setrlimit() system call, the OS will be asked to enforce the requested limit. The default is for \fBtcpflow\fP to use the maximum number of file descriptors allowed by the OS. The .B \-v option will report how many file descriptors \fBtcpflow\fP is using. .TP .B \-g Output flow information to console in multiple colors. (Blue for client to server flows, red for server to client flows, green for undecided flows.) Note: This option was different from \fBtcpflow\fP 1.3 (-e) and 1.4.4 (-J). .TP .B \-h \--help Help. Print usage information and exit. .TP .B \-hh More help. Print more usage information and exit. .TP .B \-i \fIiface\fP Interface name. Capture packets from the network interface named \fIiface\fP. If no interface is specified with .B \-i , a reasonable default will be used by libpcap automatically. .TP .B \-I Store the reception timestamps (of TCP packets) in a companion file \fB*.findx\fP. Therefore each flow will have two files: (1) the usual file containing payload bytes and (2) the text file containing the corresponding timestamps. This last file \fB*.findx\fP has three columns using the pipe \fB'|'\fP as separator: .nf \fBbyte-index|timestamp|length\fP .fi The \fBbyte-index\fP column is the postion within the file containing the payload bytes. The \fBtimestamp\fP column represents the number of seconds since epoch as a floating point number. The precision is the microsecond but may also be the nanosecond in a future \fBtcpflow\fP version. The \fBlength\fP column is the number of successive bytes concerned by \fBtimestamp\fP and can include several TCP frames (TCP packets). The extension \fBfindx\fP may become from the fact that the timestamps are \fBframe indexed\fP. .TP .B \-L \fIsemlock_name\fP Specifies that \fIsemlock_name\fP should be used as a Unix semaphore to prevent two different copies of \fBtcpflow\fP running in two different processes but outputting to the same standard output from printing on top of each other. This is an application of Unix named semaphores; bet you have never seen one before. .TP .B \-l Treat the following arguments as filenames with an assumed \fB-r\fP command before each one. This allows you to read a lot of files at once with shell globbing. For example, to process all of the pcap files in the current directory, use this: .in +.5i .nf \fBtcpflow -o out -a -l *.pcap\fP .fi .in -.5i .TP .B \-m \fImin_size\fP Forces a new connection output file when there is a skip in the TCP session of \fImin_size\fP bytes or more. .TP .B \-o \fIoutdir\fP Specifies the output directory where the transcript files will be written. .TP .B \-P No purge. Normally \fBtcpflow\fP removes connections from the hash table after the connection is closed with a FIN. This conserves memory but takes additional CPU time. Selecting this option causes the std::tr1:unordered_map to grow without bounds, as \fBtcpflow\fP did prior to version 1.1. That makes \fBtcpflow\fP run faster if there are less than 10 million connections, but can lead to out-of-memory errors. .TP .B \-p No promiscuous mode. Normally, \fBtcpflow\fP attempts to put the network interface into promiscuous mode before capturing packets. The \fB-p\fP option tells \fBtcpflow\fP \fInot\fP to put the interface into promiscuous mode. Note that it might already be in promiscuous mode for some other reason. .TP .B \-q Quiet mode --- don't print warnings. Currently the only warning that \fBtcpflow\fP prints is a warning when more than 10,000 files are created that the user should have provided the \fB-Fk\fP, \fB-Fm\fP, or \fB-Fg\fP options. We might have other warnings in the future. .TP .B \--relinquish-privileges\fB=\fIusername\fP When \fBtcpflow\fP is run as root, this option changes the user ID and group ID to write files owned by \fIusername\fP. The group ID is the first one from the \fIusername\fP groups list. This operation is performed just after opening the capture device or just after opening the first input PCAP file. This option does not support multi root-only readable input files as the root privileges are dropped after opening the first file (e.g. .B \-r \fIroot-only-access.pcap\fP .B \-R \fIroot-only.pcap\fP .B \-l \fIroot-only*.pcap\fP\c ). This option has the same behaviour as the .IR tcpdump (1) option having the same name .B \--relinquish-privileges\fB . .TP .B \-r Read from file. Read packets from \fIfile\fP, which was created using the .B \-w option of .IR tcpdump (1). This option may be repeated any number of times. Standard input is used if \fIfile\fP is "-". Note that for this option to be useful, tcpdump's .B \-s option should be used to set the snaplen to the MTU of the interface (e.g., 1500) while capturing packets. .TP .B \-R Read from a file, but only to complete TCP flows. This option is used when .IR tcpflow is used to process a series of files that are captured over time. For each time period \fIn,\fP file \fIfile(n).pcap\fP should be processed with \fB-R \fIfile(n).pcap\fP, while \fIfile(n-1).pcap\fP should be processed with \fI-r file(n-1).pcap.\fP .TP .B \-S\fIname\fB=\fIvalue\fP Sets a \fIname\fP parameter to be equal to \fIvalue\fP for a plug-in. Use \fB-hh\fP to find out all of the settable parameters. .TP .B \-s Strip non-printables. Convert all non-printable characters to the "." character before printing packets to the console or storing them to a file. .TP .B \-T[format] Specifies an arbitrary template for filenames. .RS .TP .B %A expands to source IP address. .TP .B %a expands to source IP port. .TP .B %B expands to destination IP address. .TP .B %b expands to destination IP port. .TP .B %T expands to timestamp in ISO8601 format. .TP .B %t expands to timestamp in Unix time_t format. .TP .B %V expands to "--" if a VLAN is present. .TP .B %v expands to the VLAN number if a VLAN is present. .TP .B %C expands to "c" if the connection count>0. .TP .B %c expands to the connection count if the connection count>0. .TP .B %# always expands to the connection count. .TP .B %N (connection_number ) % 1000 .TP .B %K (connection_number / 1000) % 1000 .TP .B %M (connection_number / 1000000) % 1000 .TP .B %G (connection_number / 1000000000) % 1000 .TP .B %% prints a "%". .TP .RE When the option \fB\-T\fP is used, \fBtcpflow\fP ignores options \fB\-Fk\fP, \fB\-Fm\fP and \fB\-Fg\fP. However, the option \fB\-T\fP handles \fB'/'\fP within the filename template patern to create sub-directories. For example the following line will create a directory tree \fBout/IP-src/port-src/IP-dst/port-dst\fP. .nf \fBtcpflow -r packets.pcap -o out -T %A/%a/%B/%b/%c%N.flow\fP .fi .TP .B \-V \--version Print the version number and exit. .TP .B \-v \--verbose Verbose operation. Verbosely describe \fBtcpflow\fP's operation. Equivalent to \fB \-d 10\fP. .TP .B \-w \fIfilename.pcap\fP Write packets that were not processed to \fIfilename.pcap\fP. Typically this will be UDP packets. .TP .B \-X \fIfilename.xml\fP Write a .B DFXML report to \fIfilename.xml\fP. The file contains a record of every tcp connection, how the \fBtcpflow\fP program was compiled, and the computer on which \fBtcpflow\fP was run. By default .B tcpflow writes the .B DFXML report in file \fIreport.xml\fP. .TP .B \-Z Don't decompress gzip-compressed streams. .\"START -- tcpdump excerpt" .TP \fIexpression\fP selects which packets will be captured. If no \fIexpression\fP is given, all packets on the net will be captured. Otherwise, only packets for which \fIexpression\fP is `true' will be captured. .IP For the \fIexpression\fP syntax, see .BR pcap-filter (7). .IP The \fIexpression\fP argument can be passed to \fItcpflow\fP as either a single Shell argument, or as multiple Shell arguments, whichever is more convenient. Generally, if the expression contains Shell metacharacters, such as backslashes used to escape protocol names, it is easier to pass it as a single, quoted argument rather than to escape the Shell metacharacters. Multiple arguments are concatenated with spaces before being parsed. .SH DFXML report .LP The .B DFXML report is the XML file written by .B tcpflow to provide .B tcpflow build details, command line arguments and information about processed flows. .LP By default the .B DFXML file is named \fIreport.xml\fP. But this filename can be changed using command line option \fB\-X\fP. .LP .B DFXML file respects the .B DFXML schema defined by project \fIhttps://github.com/dfxml-working-group/dfxml_schema\fP. .br Moreover .B tcpflow adds two extra XML tags, as illustrated by the following example: .RS .nf \fB bla bla bla \fP .fi .RE .LP The first XML tag \fB\fP provide information about the captured flow. This tag should be renamed \fB\fP in a future version in order to conform better to \fBDFXML schema\fP. .LP The second XML tag \fB\fP collects processing results. For the moment, only the scanner \fIpython\fP uses this feature. .PP The XML attributes of \fB\fP are: .IP \(bu \fBstartime\fP Reception time of first packet .IP \(bu \fBendtime\fP Reception time of last packet .IP \(bu \fBfamily\fP .IP \(bu \fBmac_daddr\fP Destination MAC address of first packet (printed if any) .IP \(bu \fBmac_saddr\fP Source MAC address of first packet (printed if any) .IP \(bu \fBsrc_ipn\fP IP source .IP \(bu \fBdst_ipn\fP IP destination .IP \(bu \fBsrcport\fP TCP port source .IP \(bu \fBdstport\fP TCP port destination .IP \(bu \fBpackets\fP Nummber of packets .IP \(bu \fBout_of_order_count\fP Number of times .B tcpflow has replaced missing payload by zeros in the flow file, for example when capture does not contain the TCP session begin (printed if any) .IP \(bu \fBviolations\fP Number of protocol violations (printed if any) .IP \(bu \fBlen\fP Sum of un-truncated length of all packet data (including headers, see https://stackoverflow.com/q/1491660) .IP \(bu \fBcaplen\fP Sum of captured bytes of all packet data (including headers, printed if different from \fBlen\fP) .PP The XML attributes of \fB\fP are: .IP \(bu \fBscanner\fP Name of the scanner .IP \(bu \fBpath\fP Directory of the scanner module (printed if relevant) .IP \(bu \fBmodule\fP Module name (printed if relevant, used to indicate the python script) .IP \(bu \fBfunction\fP Function name (printed if relevant, used to indicate the function within the python module) .SH EXAMPLES .LP To record all packets arriving at or departing from \fIsundown\fP and extract all of the HTTP attachments: .RS .nf \fBtcpflow -e scan_http -o outdir host sundown\fP .fi .RE .LP To record traffic between \fIhelios\fR and either \fIhot\fR or \fIace\fR and bin the results into 1000 files per directory and calculate the MD5 of each flow: .RS .nf \fBtcpflow -X report.xml -e scan_md5 -o outdir -Fk host helios and \\( hot or ace \\)\fP .fi .SH BUGS Please send bug reports to simsong@acm.org. .LP \fBtcpflow\fP currently does not understand IP fragments. Flows containing IP fragments will not be recorded correctly. .SH AUTHORS Originally by Jeremy Elson . Substantially modified and maintained by Simson L. Garfinkel . Network visualization code by Michael Shick .LP The current version of this software is available at .RS .I http://digitalcorpora.org/downloads/tcpflow/ .LP .RE An announcement mailing list for this program is at: .RS .I http://groups.google.com/group/tcpflow-users .RE .SH "SEE ALSO" tcpdump(1), nit(4P), bpf(4), pcap(3), pcap-savefile(5), pcap-filter(7) tcpflow/doc/timeline_1.4.txt0000644000175000017500000000602013342626673014652 0ustar dimadimaTimeline for 1.4 ship: + mfs + Document the refactored class hiearchy for one-page-report. + slg will look at before mfs begins refactoring + plot becomes abstract + pure virtual destructor (protected constructor too?) + concrete render(cairo_t, bounds_t) + pure virtual render_data(cairo_t, bounds_t) - This is called by render, which will calculate the bounds within the axes, labels etc. - sublcasses need only override render_data, plot() owns the space given to it - time_histogram split into time_histogram and time_histogram_plot - time_histogram_plot is a concrete subclass of plot - contains pointer to const time_histogram and probably not too much else - time_histogram replaces dyn_time_histogram - time_histogram now contains a vector of maps and a const pointer to the best fit histogram (best fit map) which starts as the most granular histogram and is updated as histograms are dropped for overflow - implement [], size() directly on time_histogram to hide implementation and selection - port_histogram and address_histogram are similarly split - histogram_bar class is added - map (or fixed bucket uints?) values to counts - render(cairo_t, bounds_t, color_map) will fill region with a proportional bar by counts - with a flexible enough histogram_bar class, time_histogram_plot, port_histogram_plot, and address_histogram_plot can be merged into templated histogram_plot concrete subclass of plot if desired - packet intestion logic is moved to one_page_report; histograms are simply data structures - could create ingester class instead too Features needed: ================ - Packet Grid - Documentation (NPS Report) What we are not doing: ====================== - Traffic Map - Language identification - Keyword extraction & clustering - Passive DNS Test Plan: ========== - Performance testing Packaging: ---------- - Put relevant boost headers in boost subdirectory Compile testing: ---------------- Make sure that it compiles on these platforms: - FC17 - Ubuntu - OS10.6, 10.8 - cygwin - mingw - Centos 5.8 - Centos 6.0 - SUSE Reliability testing: -------------------- Test for crashing with all scanners on with specific data sets: - no packets - Lincoln Labs ID98 - One day - All of the packets concatenated together - Lincoln Labs ID99 (All of the packets concatenated together) - One day - All of the packets concatenated together - One day from ID98 & One day from ID99 - M57 Patents - One day - All packets concatenated together - NGDC 2012 - All packets concatenated together - Cada? Correctness testing: -------------------- Create a file of all MD5s of all TCP streams. Sort the file. Use "diff" Data Sets for comparison: - ID98 one day - M57 one day Compare results of: - tcpflow 1.4 with tcpflow 1.0 - Question: Can Suricata provide the MD5 of tcp streams? Packaging: ---------- - .tar.gz file distributed on digitalcorpora. - windows executables on digitalcorpora - downloadable tag from github Announcements: ------------- tcpflow/doc/tcpflow-logo.pdf0000644000175000017500000044576513342626673015057 0ustar dimadima%PDF-1.3 % 4 0 obj << /Length 5 0 R /Filter /FlateDecode >> stream xAO wl -݄ă`Hi]jiAH`^#x RZ<~&/Slbش}z(3+:cq5?;Sh?_{Lo^SFav(! ()ɺ6T&΍RZmTx@=1wÿ'\&;Gv ljV Șgd]CI?9^8/3c, endstream endobj 5 0 obj 253 endobj 2 0 obj << /Type /Page /Parent 3 0 R /Resources 6 0 R /Contents 4 0 R /MediaBox [0 0 122 110] /CropBox [0 3.090372 122 110] /BleedBox [0 0 122 110] /TrimBox [0 0 122 110] /ArtBox [0 0 122 110] >> endobj 6 0 obj << /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ] /ColorSpace << /Cs1 7 0 R >> /ExtGState << /Gs1 12 0 R >> /Font << /TT2 11 0 R /TT1 10 0 R >> /XObject << /Im1 8 0 R >> >> endobj 8 0 obj << /Length 9 0 R /Type /XObject /Subtype /Image /Width 794 /Height 567 /Interpolate true /ColorSpace 13 0 R /Intent /Perceptual /SMask 14 0 R /BitsPerComponent 8 /Filter /FlateDecode >> stream xydurlX&iR2EKHV!/RdHvڊ"PȌ?[&1f0 @FPztR{U;wUՃaOq';*Ys <"%'VzD8>8h4;ǣH/x!   -%W*7slBҿH$Z+P'yM$@$@$@$0:͖;P*X|T"1I nP!   *(TۥZV]T33Y,lJJ'?$   &E3j=UZޮzh4BD7ոIHHH`@49)7 AUQ)z(i6*fS! 8D2W)6j%KB+f5PA\[ɍHHHH@ ] aT5WrV-6 P]~DG$@$@$@H R2= Ӥ@:ɲ JWL{F!JDh¿p7r   2aGY+_(MP=H*nW;Z >g&!.%o2epIHHH)!O!Q)Oex BUlN7|B+HHH` H3$ndR7Yb_7lSzaNքzOb|$@$@$@L@l _B)%6FiUPaB*T] :"Cѧ ; hT$VLk)T:hv1 L=#Q$.e sQGeP|WT}%P(bk*wp Tu. 𾘙[ehE k !`ic Q7׫X9o.ӹU_ʢu[wV <rgx8a'C*HOdS HJ-~G"  |Ə,oNIe[b[y*y֦2.lMeDcQMMˍHHɋ3YqLȭ;[;&B*,^*Ӌ^jcv3zoSFȦPx SFO:JXHHHDq4Kd^+>M}Mīj^g|$@$@$pt M;kkm zu.In*6cB v?Gz}5 4Ja`Atf։\ڼ޺voױ>=l Ћh$^tJHUĦS:*g͘2'ݞ@4PB$@$@$@G@̉&fg0T~}ͽ޺ nط#2c{1{epyvwuqm[[2xJt^5f%mDϵCe6upfkJ{Q3:l#}@sDc:{K$@$@$0%4(ӇlNx}|7 W+oWPJM76u'%BTU)=yM3J)C@ {XB5%O& @ę=.03&ȱfǜ{k!w~ԥ&//JmE76^tdS*f(:sl*hQ Pk6mk<$gOf^B$@$@$@O&h"~\ j88_/\/]p\\XcC;IEHPh!EOcT[d#TբIOiDt.:+4foShTX|ҚmfSo 33mxސ D?[DJLş~g{ūŕƅ*/K0el%S:?9Nxg߄M\TS8OayިuܾHK_$@$@$@O6Be©h‰%x:ϣط޺PD/dW62s}2̴2bSX(բ!gM)3WæujPmaNBOuFqoDP|x~ߓ#  &= db.O]+CdCRES)0rj¦ Tl!֦3\)E?MQ^wJ:y!   '@4)'܉~╅ڥ/...-U//װQú^Snmo'$S9ig[*46JoV}!CզSᬙ U; A.bJ*Cꉅ;F$@$@$0E"D<8/AQHoO\ lJfxjMasu6Uibߠ Y3P [)z JQ9k7/.7-V}x6sͨ6)kSڋœ)SXTp Rf"   P)(JPx/|UXlMZ+}ZC6R9 ;SjS JJ>WlЎ1ZWb梛FAṃ8U  G&}Cx IHH`|J*kwo..1S 6uq‚MuJmJxZRQ jSkbg{NQ ]=h5mj]#T~OTl wHHH s%!KgJ)~6BOPaNBpʌ?c<Oɦvڔ w\t6BeACTGQކP*; y.G#  xVc^eSPCT0ɳxJ}RZζO6O<ͦpmP{ BO e%I$@$@G@4rg7o-W:PµfSO̦.xj9놋}!Ȟ9ٷ)OyOA*ml 6)Ji /Tw_|h$@$@$dhQ2C[;iL|v To,T_+9_Յ}FdN] }fTٔS[[M ^t61*a1ҹ+tOQ z/b[jRX<Րߠal{J*T Jޑ !:gHU ;1=1g./7+ N;8t[l c<澆|gOCT7Z8u">   'B#x‰`^Z,aٻ7*o/^[@|(lj¦bM̤:s1#geSRS2BOMa5cy<"  x$Eb%\42f#1WC6uayvbm BJ-ይMSaw6^H!TƦ\TԎb)LB>)ݡtG`}NyHHHdIu"LE_.4;wJ!^/ x*lSv<}CPaTRօPMOPOSOmS¨=nF[?4   K e>3ĝ臯.1Z%ѧsKWL!‡WO)}BOؔ*y:'AmJJG% v!GMT؇( 9q&xxHHH3θl*zBvQ;EtLRݓ6*= 1~oelJ)9bfSjS:=Y*WcSMa.zOΚ7Zcq] 1lIHH+ A3z|浻;>>$*3eԦTҧSPڔ0SmJz'OI<%S-̦:%' Cl)\t<TO   N&$_d j락_{FtO) SXE̴k:usvkkSffs_kY+:xSfs_]NѡJBPkw{S?~J4ѿ* ?IHH%1H _^*~~RmoboS*T{ڔ)3;m)`jE7؇ hDWDi ǎ =n$   GC@Βqo/q|o.71½mJ *;m&yCbg)lNȦR8>Õ QmPw!   @46߹)l;ܑ;+ M8tE>.;jSfTdS֦Tlڔ)<5aS4Z$SfMuM:5aSj0 Tp(tMaZQFi0fp HHH=ً#]_{'}~ڒlJV碿ܑM6>ͦJ~ئ0v2*Ju~0h mD7S=z$@$@$@G@ c"H4۳OΑ T,̌b8 sfe}:BMb5E7k9eN57`S77ZX8hFvwC糭\{12Sl;r89[sv}rS0x bssG$@$@$ ȌNLH(>z+'Ʀ:6UԦRFGTئWi)4;*>0ђcoO$@$@$pd jlLlqⷖqa)$TMlZݞه{A੅dS(SMMum6`)99MagT 㧆mc}HHH;(xzeT2:ϭMB:e+}PJ_Pξطæ)=YxBLهljѹE7f'}0@<FtOM)PAcg@$@$@$p Ho&bs?X^X(_`SFdfئ&` CgU14MIxM-g*4k<S)!!}dSX)Z?/|2ZQȦ؇ϏKHH#bSZsư9xyM=PBEmSbST$xjO*E4ϦM6ٔqH{R!  #J@lJj|ѧ-0j|QٔOeSA/cgBؔE76؇xjS梣uK=SȦ)L ?"VkNA!}?161YwoHT`SlJm BSXƣ T)3G}žLs12gOMͲ'SbߤM;X< Tw(>g%S|z~xS ! {4Ay؝;lwqx}6Bg6em NuJ_hg)dbrFI3-Mb_GmjB^Nx+32ž OSް?0!;m0x V>`({c^ |$@$@$@'f=)ǝ,۪Z4{kv+E#<(UOm:žMlD߯ojO #fO T89OG%ۚMM!ZͷզsLx.mSu :'AzBd6A$@$@SJ@R)hԌINdVZy.nlH tlؔ tH6тYe$^titmSdS<<mؔ 'a!ȡSfĦ~, Cp޶/ 2xHHH` DR3f7:ѪD譥 lVU>;dSAz  Iu=+}c [E'cꝤ{suct.mT{buSl6}֦2u׏.JMA4S>9y{4[mow]Q;_7HHH)XʴO[H5nKoƔ',4,!Z.& nCܸҷsOΞwO}z)SLd'hiOaK5OCwJPq/M SVwU؇)2ʾzxHHHm籈R7aS"Tk #)8*:5aSbߤM}Ʀ؇}aJU\sʌT. :xJZ`Ma.:l p-~xi| @@M@o*q"Ly!ӄMa&)~6eG%h6k۾4 ۔m ۔k#S/Ȧ&lʌJxJ[RvTBE6ewMSfĦPSP!x*ȦMu.T ٔ9k 7P ' T2E>':kBu@0*TaP&QzfS֦!T&Aևdj|rئbS)9ecbT NQRƷYh\ʬ9ɦ0 "z27iS HwbP;OhJf)tg)<.6xJl*هctT.>cS̾=e )UئģF"TߧNHHH` `y$fkLs2T ҂okS:ۦpTJeS޷MkS{GU`SOa:e)=6Bl 6ߦTPx*M.Ybldl**)MafSw6[w੕ZSjS[UT1"T݁fSjSI ds63z!vI$@$@$0{ԝTiȦPò)ͦ&ld!m ž}m !{6U"mG