From 1b82eda9ebb72a4ac5198a44072f8694ffc1e060 Mon Sep 17 00:00:00 2001 From: John Thacker Date: Mon, 1 May 2023 08:02:18 -0400 Subject: [PATCH] epan: Register dynamic column fields and make them filterable Make the text of each registered column a FT_STRING field that can be filtered, prefixed with _ws.col - these work in display filters, filters in taps, coloring rules, Wireshark read filters, and in the -Y, -R, -e, and -j options to tshark. Use them as the default "Apply as Filter" value for the columns that aren't handled by anything else currently. Because only the columns formats that actually correspond to columns get filled in (invisible columns work), register and deregister the fields when the columns change. Use the lower case version of the rest of the COL_* define for each column as the field name. This adds a number of conditions to "when are the columns needed", including when the main display filter or any filter on a tap is using one of these fields. Custom columns are currently not implemented. For custom columns, the tree then has to be further primed with any fields used by the custom columns as well. (Perhaps that should happen in epan_dissect_run() - are there any cases where we construct the columns and don't want to prime with any field that custom columns contains? Possibly in taps that we know only use build in columns.) Thus, for performance reasons, you're better off matching an ordinary field if possible; it takes extra time to generate the columns and many of them are numeric types. (Note that you can always convert a non-string field to a string field if you want regex matching, consult the *wireshark-filter(4)* man page.) It does save a bit on typing (especially for a multifield custom column) and remembering the column title might be easier in some cases. The columns are set before the color filters, which means that you can have a color filter that depends on a built-in column like Info or Protocol. Remove the special handling for the -e option to tshark. Note that the behavior is a little different now, because fixed field names are used instead of the titles (using the titles allowed illegal filter names, because it wasn't going through the filter engine.) For default names, this means that they're no longer capitalized, so "_ws.col.info" instead of "_ws.col.Info" - hopefully a small price in exchange for the filters working everywhere. The output format for -T fields remains the same; all that special handling is removed (except for remembering if someone asked for a column field to know that columns should be constructed.) They're also set before the postdissectors, so postdissectors can have access. Anything that depends on whether a packet and previous packets are displayed (COL_DELTA_TIME_DIS or COL_CUMULATIVE_BYTES) doesn't work the way most people expect, so don't register fields for those. (The same is already true of color filters that use those, along with color filters that use the color filter fields.) Fix #16576. Fix #17971. Fix #4684. Fix #13491. Fix #13941. --- doc/tshark.adoc | 4 +- docbook/release-notes.adoc | 19 +++- epan/column-info.h | 11 +++ epan/column-utils.c | 95 +++++++++++++++++- epan/column.c | 143 +++++++++++++++++++++++++++- epan/column.h | 5 + epan/dfilter/dfilter.c | 17 ++++ epan/dfilter/dfilter.h | 4 + epan/dissectors/packet-frame.c | 13 +++ epan/prefs.c | 3 + epan/print.c | 26 +---- epan/proto.c | 3 +- epan/tap.c | 23 +++++ epan/tap.h | 6 ++ file.c | 35 +++++-- sharkd.c | 2 +- test/suite_dfilter/dfiltertest.py | 31 +++++- test/suite_dfilter/group_columns.py | 54 +++++++++++ test/suite_dissection.py | 2 +- tfshark.c | 18 ++-- tshark.c | 25 +++-- ui/qt/packet_list.cpp | 2 + 22 files changed, 472 insertions(+), 69 deletions(-) create mode 100644 test/suite_dfilter/group_columns.py diff --git a/doc/tshark.adoc b/doc/tshark.adoc index a6eb2f6242..db5a1b0cad 100644 --- a/doc/tshark.adoc +++ b/doc/tshark.adoc @@ -312,9 +312,9 @@ contain a GUID. Add a field to the list of fields to display if *-T ek|fields|json|pdml* is selected. This option can be used multiple times on the command line. At least one field must be provided if the *-T fields* option is -selected. Column names may be used prefixed with "_ws.col." +selected. Column types may be used prefixed with "_ws.col." -Example: *tshark -e frame.number -e ip.addr -e udp -e _ws.col.Info* +Example: *tshark -e frame.number -e ip.addr -e udp -e _ws.col.info* Fields are separated by tab characters by default. *-E* controls the format of the printed fields. diff --git a/docbook/release-notes.adoc b/docbook/release-notes.adoc index 2941b82346..a3ce2737f0 100644 --- a/docbook/release-notes.adoc +++ b/docbook/release-notes.adoc @@ -143,7 +143,24 @@ Recognizes PCAP traces with the link type LINKTYPE_FIRA_UCI=299. the capture file via the GUI, similar to the options --inject-secrets and --discard-all-secrets in editcap. -// === Removed Features and Support +* The text of any configured column (displayed or hidden) can be filtered +anywhere that filters are used - in display filters, filters in taps, coloring +rules, Wireshark read filters, and the -Y, -R, and -e options to tshark, +the "Apply as Filter" GUI option, etc. +** The filter field names are prefixed by "_ws.col", followed by a lowercase +version of the COL_ name found in epan/column-utils.h, e.g. "_ws.col.info" +or "_ws.col.protocol" +** Using the column names as a filter is slower than other filter types +because the columns must be constructed, so when the same filtering +can be achieved via other fields, prefer that. + +=== Removed Features and Support + +* With the addition of the universal and consistent filtering support for +column text, the previous support in the -e option to tshark for displaying +column text via the column title, e.g. "_ws.col.Info", has been removed. +The previous implementation allowed names that are not legal filter names +and wsbuglink:16576[silently ignored columns that didn't exist.] // === Removed Dissectors diff --git a/epan/column-info.h b/epan/column-info.h index 85cccb16ae..0d26e65234 100644 --- a/epan/column-info.h +++ b/epan/column-info.h @@ -24,6 +24,10 @@ extern "C" { * Column info. */ +typedef struct _proto_node proto_tree; + +#define COLUMN_FIELD_FILTER "_ws.col." + /** Column expression */ typedef struct { const gchar **col_expr; /**< Filter expression */ @@ -43,6 +47,7 @@ typedef struct { gchar *col_buf; /**< Buffer into which to copy data for column */ int col_fence; /**< Stuff in column buffer before this index is immutable */ gboolean writable; /**< writable or not */ + int hf_id; } col_item_t; /** Column info */ @@ -111,6 +116,12 @@ gboolean col_has_time_fmt(column_info *cinfo, const gint col); WS_DLL_PUBLIC gboolean col_based_on_frame_data(column_info *cinfo, const gint col); +void +col_register_protocol(void); + +extern +void col_dissect(tvbuff_t *tvb, packet_info *pinfo, proto_tree *tree); + #ifdef __cplusplus } #endif /* __cplusplus */ diff --git a/epan/column-utils.c b/epan/column-utils.c index a07e8cced3..1d28140e51 100644 --- a/epan/column-utils.c +++ b/epan/column-utils.c @@ -26,6 +26,7 @@ #include "osi-utils.h" #include "value_string.h" #include "column-info.h" +#include "column.h" #include "proto.h" #include @@ -49,6 +50,9 @@ static char *col_decimal_point; /* Used to indicate updated column information, e.g. a new request/response. */ static gboolean col_data_changed_; +static int proto_cols = -1; +static gint ett_cols = -1; + /* Allocate all the data structures for constructing column data, given the number of columns. */ void @@ -363,6 +367,34 @@ void col_custom_set_edt(epan_dissect_t *edt, column_info *cinfo) } } +#if 0 +// Needed if we create _ws.col.custom +static void +col_custom_set(proto_tree *tree, column_info *cinfo) +{ + int i; + col_item_t* col_item; + + if (!HAVE_CUSTOM_COLS(cinfo)) + return; + + for (i = cinfo->col_first[COL_CUSTOM]; + i <= cinfo->col_last[COL_CUSTOM]; i++) { + col_item = &cinfo->columns[i]; + if (col_item->fmt_matx[COL_CUSTOM] && + col_item->col_custom_fields && + col_item->col_custom_fields_ids) { + col_item->col_data = col_item->col_buf; + cinfo->col_expr.col_expr[i] = proto_custom_set(tree, col_item->col_custom_fields_ids, + col_item->col_custom_occurrence, + col_item->col_buf, + cinfo->col_expr.col_expr_val[i], + COL_MAX_LEN); + } + } +} +#endif + void col_custom_prime_edt(epan_dissect_t *edt, column_info *cinfo) { @@ -2300,6 +2332,10 @@ col_fill_in(packet_info *pinfo, const gboolean fill_col_exprs, const gboolean fi col_set_port(pinfo, i, FALSE, FALSE, fill_col_exprs); break; + case COL_CUSTOM: + /* Formatting handled by col_custom_set_edt() / col_custom_get_filter() */ + break; + case NUM_COL_FMTS: /* keep compiler happy - shouldn't get here */ ws_assert_not_reached(); break; @@ -2308,9 +2344,13 @@ col_fill_in(packet_info *pinfo, const gboolean fill_col_exprs, const gboolean fi ws_assert_not_reached(); } /* - * Formatting handled by col_custom_set_edt() (COL_CUSTOM), expert.c - * (COL_EXPERT), or individual dissectors. + * Formatting handled by expert.c (COL_EXPERT), or individual + * dissectors. Fill in from the text using the internal hfid. */ + if (fill_col_exprs) { + pinfo->cinfo->col_expr.col_expr[i] = proto_registrar_get_nth(col_item->hf_id)->abbrev; + (void) g_strlcpy(pinfo->cinfo->col_expr.col_expr_val[i], pinfo->cinfo->columns[i].col_data, (col_item->col_fmt == COL_INFO) ? COL_MAX_INFO_LEN : COL_MAX_LEN); + } break; } } @@ -2358,6 +2398,57 @@ gboolean col_data_changed(void) { col_data_changed_ = FALSE; return cur_cdc; } + +void +col_register_protocol(void) +{ + /* This gets called by proto_init() before column_register_fields() + * gets called by the preference modules actually getting registered. + */ + if (proto_cols == -1) { + proto_cols = proto_get_id_by_filter_name("_ws.col"); + } + if (proto_cols == -1) { + proto_cols = proto_register_protocol("Wireshark Columns", "Columns", "_ws.col"); + } + static gint *ett[] = { + &ett_cols + }; + proto_register_subtree_array(ett, G_N_ELEMENTS(ett)); +} + +void +col_dissect(tvbuff_t *tvb, packet_info *pinfo, proto_tree *tree) +{ + proto_item *ti; + proto_tree *col_tree; + + column_info *cinfo = pinfo->cinfo; + + if (!cinfo) { + return; + } + + if (proto_field_is_referenced(tree, proto_cols)) { + // XXX: Needed if we also create _ws.col.custom + //col_custom_set(tree, cinfo); + col_fill_in(pinfo, FALSE, TRUE); + ti = proto_tree_add_item(tree, proto_cols, tvb, 0, 0, ENC_NA); + proto_item_set_hidden(ti); + col_tree = proto_item_add_subtree(ti, ett_cols); + for (int i = 0; i < cinfo->num_cols; ++i) { + if (cinfo->columns[i].hf_id != -1) { + if (cinfo->columns[i].col_fmt == COL_CUSTOM) { + ti = proto_tree_add_string_format(col_tree, cinfo->columns[i].hf_id, tvb, 0, 0, get_column_text(cinfo, i), "%s: %s", get_column_title(i), get_column_text(cinfo, i)); + } else { + ti = proto_tree_add_string(col_tree, cinfo->columns[i].hf_id, tvb, 0, 0, get_column_text(cinfo, i)); + } + proto_item_set_hidden(ti); + } + } + } +} + /* * Editor modelines * diff --git a/epan/column.c b/epan/column.c index 5cbc5c7833..66704ffa3d 100644 --- a/epan/column.c +++ b/epan/column.c @@ -26,6 +26,10 @@ #include #include +static int proto_cols = -1; +static hf_register_info *hf_cols = NULL; +static unsigned int hf_cols_cleanup = 0; + /* Given a format number (as defined in column-utils.h), returns its equivalent string */ const gchar * @@ -155,6 +159,67 @@ col_format_desc(const gint fmt_num) { return val_str; } +/* Given a format number (as defined in column-utils.h), returns its + filter abbreviation */ +const gchar * +col_format_abbrev(const gint fmt_num) { + + static const value_string alist_vals[] = { + + { COL_ABS_YMD_TIME, COLUMN_FIELD_FILTER"abs_ymd_time" }, + { COL_ABS_YDOY_TIME, COLUMN_FIELD_FILTER"abs_ydoy_time" }, + { COL_ABS_TIME, COLUMN_FIELD_FILTER"abs_time" }, + { COL_CUMULATIVE_BYTES, COLUMN_FIELD_FILTER"cumulative_bytes" }, + { COL_CUSTOM, COLUMN_FIELD_FILTER"custom" }, + { COL_DELTA_TIME_DIS, COLUMN_FIELD_FILTER"delta_time_dis" }, + { COL_DELTA_TIME, COLUMN_FIELD_FILTER"delta_time" }, + { COL_RES_DST, COLUMN_FIELD_FILTER"res_dst" }, + { COL_UNRES_DST, COLUMN_FIELD_FILTER"unres_dst" }, + { COL_RES_DST_PORT, COLUMN_FIELD_FILTER"res_dst_port" }, + { COL_UNRES_DST_PORT, COLUMN_FIELD_FILTER"unres_dst_port" }, + { COL_DEF_DST, COLUMN_FIELD_FILTER"def_dst" }, + { COL_DEF_DST_PORT, COLUMN_FIELD_FILTER"def_dst_port" }, + { COL_EXPERT, COLUMN_FIELD_FILTER"expert" }, + { COL_IF_DIR, COLUMN_FIELD_FILTER"if_dir" }, + { COL_FREQ_CHAN, COLUMN_FIELD_FILTER"freq_chan" }, + { COL_DEF_DL_DST, COLUMN_FIELD_FILTER"def_dl_dst" }, + { COL_DEF_DL_SRC, COLUMN_FIELD_FILTER"def_dl_src" }, + { COL_RES_DL_DST, COLUMN_FIELD_FILTER"res_dl_dst" }, + { COL_UNRES_DL_DST, COLUMN_FIELD_FILTER"unres_dl_dst" }, + { COL_RES_DL_SRC, COLUMN_FIELD_FILTER"res_dl_src" }, + { COL_UNRES_DL_SRC, COLUMN_FIELD_FILTER"unres_dl_src" }, + { COL_RSSI, COLUMN_FIELD_FILTER"rssi" }, + { COL_TX_RATE, COLUMN_FIELD_FILTER"tx_rate" }, + { COL_DSCP_VALUE, COLUMN_FIELD_FILTER"dscp" }, + { COL_INFO, COLUMN_FIELD_FILTER"info" }, + { COL_RES_NET_DST, COLUMN_FIELD_FILTER"res_net_dst" }, + { COL_UNRES_NET_DST, COLUMN_FIELD_FILTER"unres_net_dst" }, + { COL_RES_NET_SRC, COLUMN_FIELD_FILTER"res_net_src" }, + { COL_UNRES_NET_SRC, COLUMN_FIELD_FILTER"unres_net_src" }, + { COL_DEF_NET_DST, COLUMN_FIELD_FILTER"def_net_dst" }, + { COL_DEF_NET_SRC, COLUMN_FIELD_FILTER"def_net_src" }, + { COL_NUMBER, COLUMN_FIELD_FILTER"number" }, + { COL_PACKET_LENGTH, COLUMN_FIELD_FILTER"packet_length" }, + { COL_PROTOCOL, COLUMN_FIELD_FILTER"protocol" }, + { COL_REL_TIME, COLUMN_FIELD_FILTER"rel_time" }, + { COL_DEF_SRC, COLUMN_FIELD_FILTER"def_src" }, + { COL_DEF_SRC_PORT, COLUMN_FIELD_FILTER"def_src_port" }, + { COL_RES_SRC, COLUMN_FIELD_FILTER"res_src" }, + { COL_UNRES_SRC, COLUMN_FIELD_FILTER"unres_src" }, + { COL_RES_SRC_PORT, COLUMN_FIELD_FILTER"res_src_port" }, + { COL_UNRES_SRC_PORT, COLUMN_FIELD_FILTER"unres_src_port" }, + { COL_CLS_TIME, COLUMN_FIELD_FILTER"cls_time" }, + { COL_UTC_YMD_TIME, COLUMN_FIELD_FILTER"utc_ymc_time" }, + { COL_UTC_YDOY_TIME, COLUMN_FIELD_FILTER"utc_ydoy_time" }, + { COL_UTC_TIME, COLUMN_FIELD_FILTER"utc_time" }, + + { 0, NULL } + }; + + const gchar *val_str = try_val_to_str(fmt_num, alist_vals); + ws_assert(val_str != NULL); + return val_str; +} /* Array of columns that have been migrated to custom columns */ struct deprecated_columns { const gchar *col_fmt; @@ -931,13 +996,15 @@ col_finalize(column_info *cinfo) get_column_format_matches(col_item->fmt_matx, col_item->col_fmt); col_item->col_data = NULL; - if (col_item->col_fmt == COL_INFO) + if (col_item->col_fmt == COL_INFO) { col_item->col_buf = g_new(gchar, COL_MAX_INFO_LEN); - else + cinfo->col_expr.col_expr_val[i] = g_new(gchar, COL_MAX_INFO_LEN); + } else { col_item->col_buf = g_new(gchar, COL_MAX_LEN); + cinfo->col_expr.col_expr_val[i] = g_new(gchar, COL_MAX_LEN); + } cinfo->col_expr.col_expr[i] = ""; - cinfo->col_expr.col_expr_val[i] = g_new(gchar, COL_MAX_LEN); } cinfo->col_expr.col_expr[i] = NULL; @@ -975,6 +1042,7 @@ build_column_format_array(column_info *cinfo, const gint num_cols, const gboolea col_item->col_custom_fields = g_strdup(get_column_custom_fields(i)); col_item->col_custom_occurrence = get_column_custom_occurrence(i); } + col_item->hf_id = proto_registrar_get_id_byname(col_format_abbrev(col_item->col_fmt)); if(reset_fences) col_item->col_fence = 0; @@ -983,6 +1051,75 @@ build_column_format_array(column_info *cinfo, const gint num_cols, const gboolea col_finalize(cinfo); } +static void +column_deregister_fields(void) +{ + if (hf_cols) { + for (unsigned int i = 0; i < hf_cols_cleanup; ++i) { + proto_deregister_field(proto_cols, *(hf_cols[i].p_id)); + g_free(hf_cols[i].p_id); + } + proto_add_deregistered_data(hf_cols); + hf_cols = NULL; + hf_cols_cleanup = 0; + } +} + +void +column_register_fields(void) +{ + + int* hf_id; + GArray *hf_col_array; + hf_register_info new_hf; + fmt_data *cfmt; + gboolean *used_fmts; + if (proto_cols == -1) { + proto_cols = proto_get_id_by_filter_name("_ws.col"); + } + if (proto_cols == -1) { + proto_cols = proto_register_protocol("Wireshark Columns", "Columns", "_ws.col"); + } + column_deregister_fields(); + if (prefs.col_list != NULL) { + prefs.num_cols = g_list_length(prefs.col_list); + hf_col_array = g_array_new(FALSE, TRUE, sizeof(hf_register_info)); + used_fmts = g_new0(gboolean, NUM_COL_FMTS); + /* Only register a field for each format type once, but don't register + * these at all. The first two behave oddly (because they depend on + * whether the current field and previous fields are displayed). We + * might want to do custom columns in the future, though. + */ + used_fmts[COL_DELTA_TIME_DIS] = 1; + used_fmts[COL_CUMULATIVE_BYTES] = 1; + used_fmts[COL_CUSTOM] = 1; + + for (GList *elem = g_list_first(prefs.col_list); elem != NULL; elem = elem->next) { + cfmt = (fmt_data*)elem->data; + if (!used_fmts[cfmt->fmt]) { + used_fmts[cfmt->fmt] = TRUE; + hf_id = g_new(int, 1); + *hf_id = -1; + new_hf.p_id = hf_id; + new_hf.hfinfo.name = g_strdup(col_format_desc(cfmt->fmt)); + new_hf.hfinfo.abbrev = g_strdup(col_format_abbrev(cfmt->fmt)); + new_hf.hfinfo.type = FT_STRING; + new_hf.hfinfo.display = BASE_NONE; + new_hf.hfinfo.strings = NULL; + new_hf.hfinfo.bitmask = 0; + new_hf.hfinfo.blurb = NULL; + HFILL_INIT(new_hf); + g_array_append_vals(hf_col_array, &new_hf, 1); + } + } + g_free(used_fmts); + hf_cols_cleanup = hf_col_array->len; + + proto_register_field_array(proto_cols, (hf_register_info*)hf_col_array->data, hf_col_array->len); + hf_cols = (hf_register_info*)g_array_free(hf_col_array, FALSE); + } +} + /* * Editor modelines - https://www.wireshark.org/tools/modelines.html * diff --git a/epan/column.h b/epan/column.h index e44d02ef05..be83f1c31e 100644 --- a/epan/column.h +++ b/epan/column.h @@ -35,6 +35,8 @@ const gchar *col_format_to_string(const gint); WS_DLL_PUBLIC const gchar *col_format_desc(const gint); WS_DLL_PUBLIC +const gchar *col_format_abbrev(const gint); +WS_DLL_PUBLIC gint get_column_format(const gint); WS_DLL_PUBLIC void set_column_format(const gint, const gint); @@ -117,6 +119,9 @@ gboolean parse_column_format(fmt_data *cfmt, const char *fmt); */ WS_DLL_PUBLIC void try_convert_to_custom_column(char **fmt); + +WS_DLL_PUBLIC +void column_register_fields(void); #ifdef __cplusplus } #endif /* __cplusplus */ diff --git a/epan/dfilter/dfilter.c b/epan/dfilter/dfilter.c index 5b5ebf072f..8f80946723 100644 --- a/epan/dfilter/dfilter.c +++ b/epan/dfilter/dfilter.c @@ -710,6 +710,23 @@ dfilter_interested_in_proto(const dfilter_t *df, int proto_id) return FALSE; } +gboolean +dfilter_requires_columns(const dfilter_t *df) +{ + if (df == NULL) { + return FALSE; + } + + /* XXX: Could cache this like packet_cache_proto_handles */ + static int proto_cols = -1; + if (proto_cols == -1) { + proto_cols = proto_get_id_by_filter_name("_ws.col"); + } + ws_assert(proto_cols != -1); + + return dfilter_interested_in_proto(df, proto_cols); +} + GPtrArray * dfilter_deprecated_tokens(dfilter_t *df) { if (df->deprecated && df->deprecated->len > 0) { diff --git a/epan/dfilter/dfilter.h b/epan/dfilter/dfilter.h index 22c7c4b8f9..ac126729a6 100644 --- a/epan/dfilter/dfilter.h +++ b/epan/dfilter/dfilter.h @@ -151,6 +151,10 @@ dfilter_interested_in_field(const dfilter_t *df, int hfid); gboolean dfilter_interested_in_proto(const dfilter_t *df, int proto_id); +WS_DLL_PUBLIC +gboolean +dfilter_requires_columns(const dfilter_t *df); + WS_DLL_PUBLIC GPtrArray * dfilter_deprecated_tokens(dfilter_t *df); diff --git a/epan/dissectors/packet-frame.c b/epan/dissectors/packet-frame.c index 2c2b596fd1..08787e8555 100644 --- a/epan/dissectors/packet-frame.c +++ b/epan/dissectors/packet-frame.c @@ -33,6 +33,7 @@ #include #include #include +#include #include "packet-frame.h" #include "packet-bblog.h" @@ -1452,6 +1453,18 @@ dissect_frame(tvbuff_t *tvb, packet_info *pinfo, proto_tree *parent_tree, void* proto_item_set_generated(ti); } + /* Add the columns as fields. We have to do this here, so that + * they're available for postdissectors that want all the fields. + * + * Note the coloring rule names are set after this, which means + * that you can set a coloring rule based on the value of a column, + * like _ws.col.protocol or _ws.col.info. + * OTOH, if we created _ws.col.custom, and a custom column used + * frame.coloring_rule.name, filtering with it wouldn't work - + * but you can filter on that field directly, so that doesn't matter. + */ + col_dissect(tvb, pinfo, parent_tree); + /* Call postdissectors if we have any (while trying to avoid another * TRY/CATCH) */ diff --git a/epan/prefs.c b/epan/prefs.c index 1ac3ac04c3..4d0ba37234 100644 --- a/epan/prefs.c +++ b/epan/prefs.c @@ -2611,6 +2611,8 @@ column_format_init_cb(pref_t* pref, GList** value) dest_cfmt->resolved = src_cfmt->resolved; pref->default_val.list = g_list_append(pref->default_val.list, dest_cfmt); } + + column_register_fields(); } static void @@ -2717,6 +2719,7 @@ column_format_set_cb(pref_t* pref, const gchar* value, unsigned int* changed_fla prefs_clear_string_list(col_l); free_string_like_preference(hidden_pref); + column_register_fields(); return PREFS_SET_OK; } diff --git a/epan/print.c b/epan/print.c index 5c428bf69c..cb4a98a0fe 100644 --- a/epan/print.c +++ b/epan/print.c @@ -2056,8 +2056,6 @@ void output_fields_free(output_fields_t* fields) g_free(fields); } -#define COLUMN_FIELD_FILTER "_ws.col." - void output_fields_add(output_fields_t *fields, const gchar *field) { gchar *field_copy; @@ -2113,9 +2111,6 @@ output_field_check(void *data, void *user_data) gchar *field = (gchar *)data; GSList **invalid_fields = (GSList **)user_data; - if (!strncmp(field, COLUMN_FIELD_FILTER, strlen(COLUMN_FIELD_FILTER))) - return; - if (!proto_registrar_get_byname(field)) { *invalid_fields = g_slist_prepend(*invalid_fields, field); } @@ -2398,12 +2393,9 @@ static void proto_tree_get_node_field_values(proto_node *node, gpointer data) } } -static void write_specified_fields(fields_format format, output_fields_t *fields, epan_dissect_t *edt, column_info *cinfo, FILE *fh, json_dumper *dumper) +static void write_specified_fields(fields_format format, output_fields_t *fields, epan_dissect_t *edt, column_info *cinfo _U_, FILE *fh, json_dumper *dumper) { gsize i; - gint col; - gchar *col_name; - gpointer field_index; write_field_data_t data; @@ -2447,22 +2439,6 @@ static void write_specified_fields(fields_format format, output_fields_t *fields proto_tree_children_foreach(edt->tree, proto_tree_get_node_field_values, &data); - /* Add columns to fields */ - if (fields->includes_col_fields) { - for (col = 0; col < cinfo->num_cols; col++) { - if (!get_column_visible(col)) - continue; - /* Prepend COLUMN_FIELD_FILTER as the field name */ - col_name = ws_strdup_printf("%s%s", COLUMN_FIELD_FILTER, cinfo->columns[col].col_title); - field_index = g_hash_table_lookup(fields->field_indicies, col_name); - g_free(col_name); - - if (NULL != field_index) { - format_field_values(fields, field_index, g_strdup(get_column_text(cinfo, col))); - } - } - } - switch (format) { case FORMAT_CSV: for(i = 0; i < fields->fields->len; ++i) { diff --git a/epan/proto.c b/epan/proto.c index 5c849748ba..8fa8170718 100644 --- a/epan/proto.c +++ b/epan/proto.c @@ -41,7 +41,7 @@ #include "tvbuff.h" #include #include "charsets.h" -#include "column-utils.h" +#include "column-info.h" #include "to_str.h" #include "osi-utils.h" #include "expert.h" @@ -585,6 +585,7 @@ proto_init(GSList *register_all_plugin_protocols_list, register_number_string_decodinws_error(); register_string_errors(); ftypes_register_pseudofields(); + col_register_protocol(); /* Have each built-in dissector register its protocols, fields, dissector tables, and dissectors to be called through a diff --git a/epan/tap.c b/epan/tap.c index c547b633b7..c24367478b 100644 --- a/epan/tap.c +++ b/epan/tap.c @@ -708,6 +708,29 @@ tap_listeners_require_dissection(void) } +/* + * Return TRUE if we have one or more tap listeners that require the columns, + * FALSE otherwise. + */ +gboolean +tap_listeners_require_columns(void) +{ + tap_listener_t *tap_queue = tap_listener_queue; + + while(tap_queue) { + if(tap_queue->flags & TL_REQUIRES_COLUMNS) + return TRUE; + + if(dfilter_requires_columns(tap_queue->code)) + return TRUE; + + tap_queue = tap_queue->next; + } + + return FALSE; + +} + /* Returns TRUE there is an active tap listener for the specified tap id. */ gboolean have_tap_listener(int tap_id) diff --git a/epan/tap.h b/epan/tap.h index 08f6c92263..1d736a9f46 100644 --- a/epan/tap.h +++ b/epan/tap.h @@ -248,6 +248,12 @@ WS_DLL_PUBLIC void remove_tap_listener(void *tapdata); */ WS_DLL_PUBLIC gboolean tap_listeners_require_dissection(void); +/** + * Return TRUE if we have one or more tap listeners that require the columns, + * FALSE otherwise. + */ +WS_DLL_PUBLIC gboolean tap_listeners_require_columns(void); + /** Returns TRUE there is an active tap listener for the specified tap id. */ WS_DLL_PUBLIC gboolean have_tap_listener(int tap_id); diff --git a/file.c b/file.c index 978589058f..0990c83436 100644 --- a/file.c +++ b/file.c @@ -560,8 +560,10 @@ cf_read(capture_file *cf, gboolean reloading) epan_dissect_init(&edt, cf->epan, create_proto_tree, FALSE); - /* If any tap listeners require the columns, construct them. */ - cinfo = (tap_flags & TL_REQUIRES_COLUMNS) ? &cf->cinfo : NULL; + /* If the display filter or any tap listeners require the columns, + * construct them. */ + cinfo = (tap_listeners_require_columns() || + dfilter_requires_columns(dfcode)) ? &cf->cinfo : NULL; /* Find the size of the file. */ size = wtap_file_size(cf->provider.wth, NULL); @@ -828,8 +830,10 @@ cf_continue_tail(capture_file *cf, volatile int to_read, wtap_rec *rec, gint64 data_offset = 0; column_info *cinfo; - /* If any tap listeners require the columns, construct them. */ - cinfo = (tap_flags & TL_REQUIRES_COLUMNS) ? &cf->cinfo : NULL; + /* If the display filter or any tap listeners require the columns, + * construct them. */ + cinfo = (tap_listeners_require_columns() || + dfilter_requires_columns(dfcode)) ? &cf->cinfo : NULL; while (to_read != 0) { wtap_cleareof(cf->provider.wth); @@ -938,8 +942,10 @@ cf_finish_tail(capture_file *cf, wtap_rec *rec, Buffer *buf, int *err, /* Get the union of the flags for all tap listeners. */ tap_flags = union_of_tap_listener_flags(); - /* If any tap listeners require the columns, construct them. */ - cinfo = (tap_flags & TL_REQUIRES_COLUMNS) ? &cf->cinfo : NULL; + /* If the display filter or any tap listeners require the columns, + * construct them. */ + cinfo = (tap_listeners_require_columns() || + dfilter_requires_columns(dfcode)) ? &cf->cinfo : NULL; /* * Determine whether we need to create a protocol tree. @@ -1290,12 +1296,16 @@ read_record(capture_file *cf, wtap_rec *rec, Buffer *buf, dfilter_t *dfcode, if (cf->rfcode) { epan_dissect_t rf_edt; + column_info *rf_cinfo = NULL; epan_dissect_init(&rf_edt, cf->epan, TRUE, FALSE); epan_dissect_prime_with_dfilter(&rf_edt, cf->rfcode); + if (dfilter_requires_columns(cf->rfcode)) { + rf_cinfo = &cf->cinfo; + } epan_dissect_run(&rf_edt, cf->cd_t, rec, frame_tvbuff_new_buffer(&cf->provider, &fdlocal, buf), - &fdlocal, NULL); + &fdlocal, rf_cinfo); passed = dfilter_apply_edt(cf->rfcode, &rf_edt); epan_dissect_cleanup(&rf_edt); } @@ -1723,8 +1733,10 @@ rescan_packets(capture_file *cf, const char *action, const char *action_item, gb /* Get the union of the flags for all tap listeners. */ tap_flags = union_of_tap_listener_flags(); - /* If any tap listeners require the columns, construct them. */ - cinfo = (tap_flags & TL_REQUIRES_COLUMNS) ? &cf->cinfo : NULL; + /* If the display filter or any tap listeners require the columns, + * construct them. */ + cinfo = (tap_listeners_require_columns() || + dfilter_requires_columns(dfcode)) ? &cf->cinfo : NULL; /* * Determine whether we need to create a protocol tree. @@ -1784,6 +1796,9 @@ rescan_packets(capture_file *cf, const char *action, const char *action_item, gb if (!create_proto_tree && have_filtering_tap_listeners()) { create_proto_tree = TRUE; } + if (!cinfo && tap_listeners_require_columns()) { + cinfo = &cf->cinfo; + } /* We need to redissect the packets so we have to discard our old * packet list store. */ @@ -2337,7 +2352,7 @@ cf_retap_packets(capture_file *cf) tap_flags = union_of_tap_listener_flags(); /* If any tap listeners require the columns, construct them. */ - callback_args.cinfo = (tap_flags & TL_REQUIRES_COLUMNS) ? &cf->cinfo : NULL; + callback_args.cinfo = (tap_listeners_require_columns()) ? &cf->cinfo : NULL; /* * Determine whether we need to create a protocol tree. diff --git a/sharkd.c b/sharkd.c index 4aab823d8c..7dd2135243 100644 --- a/sharkd.c +++ b/sharkd.c @@ -584,7 +584,7 @@ sharkd_retap(void) tap_flags = union_of_tap_listener_flags(); /* If any tap listeners require the columns, construct them. */ - cinfo = (tap_flags & TL_REQUIRES_COLUMNS) ? &cfile.cinfo : NULL; + cinfo = (tap_listeners_require_columns()) ? &cfile.cinfo : NULL; /* * Determine whether we need to create a protocol tree. diff --git a/test/suite_dfilter/dfiltertest.py b/test/suite_dfilter/dfiltertest.py index 6a92c8f337..a28df64ce2 100644 --- a/test/suite_dfilter/dfiltertest.py +++ b/test/suite_dfilter/dfiltertest.py @@ -8,7 +8,7 @@ import pytest @pytest.fixture def dfilter_cmd(cmd_tshark, capture_file, request): - def wrapped(dfilter, frame_number=None, prefs=None): + def wrapped(dfilter, frame_number=None, prefs=None, read_filter=False): cmd = [ cmd_tshark, "-n", # No name resolution @@ -20,10 +20,17 @@ def dfilter_cmd(cmd_tshark, capture_file, request): "-2", # two-pass mode "--selected-frame={}".format(frame_number) ]) - cmd.extend([ - "-Y", # packet display filter (used to be -R) - dfilter - ]) + if read_filter: + cmd.extend([ + "-2", # two-pass mode + "-R", # read filter (requires two-pass mode) + dfilter + ]) + else: + cmd.extend([ + "-Y", # packet display filter (used to be -R) + dfilter + ]) if prefs: cmd.extend([ "-o", @@ -67,6 +74,20 @@ def checkDFilterCountWithSelectedFrame(dfilter_cmd, base_env): assert dfp_count == expected_count, msg return checkDFilterCount_real +@pytest.fixture +def checkDFilterCountReadFilter(dfilter_cmd, base_env): + def checkDFilterCount_real(dfilter, expected_count): + """Run a read filter in two pass mode and expect a certain number of packets.""" + output = subprocess.check_output(dfilter_cmd(dfilter, read_filter=True), + universal_newlines=True, + stderr=subprocess.STDOUT, + env=base_env) + + dfp_count = output.count("\n") + msg = "Expected %d, got: %s\noutput: %r" % \ + (expected_count, dfp_count, output) + assert dfp_count == expected_count, msg + return checkDFilterCount_real @pytest.fixture def checkDFilterFail(cmd_dftest, base_env): diff --git a/test/suite_dfilter/group_columns.py b/test/suite_dfilter/group_columns.py new file mode 100644 index 0000000000..5bbfbc90c9 --- /dev/null +++ b/test/suite_dfilter/group_columns.py @@ -0,0 +1,54 @@ +# Copyright (c) 2013 by Gilbert Ramirez +# +# SPDX-License-Identifier: GPL-2.0-or-later + +import pytest +from suite_dfilter.dfiltertest import * + +class TestDfilterColumns: + trace_file = "http.pcap" + + def test_exists_1(self, checkDFilterCount): + dfilter = "_ws.col.info" + checkDFilterCount(dfilter, 1) + + def test_exists_2(self, checkDFilterFail): + # Column not in the default configuration + dfilter = "_ws.col.expert" + error = f'"{dfilter}" is not a valid protocol or protocol field' + checkDFilterFail(dfilter, error) + + def test_exists_3(self, checkDFilterFail): + # Column not registered as field (it behaves unusally if filtered) + dfilter = "_ws.col.delta_time_dis" + error = f'"{dfilter}" is not a valid protocol or protocol field' + checkDFilterFail(dfilter, error) + + def test_func_1(self, checkDFilterCount): + dfilter = "len(_ws.col.protocol) == 4" + checkDFilterCount(dfilter, 1) + + def test_matches_1(self, checkDFilterSucceed): + dfilter = '_ws.col.info matches "^HEAD"' + checkDFilterSucceed(dfilter) + + def test_equal_1(self, checkDFilterCount): + dfilter = '_ws.col.protocol == "HTTP"' + checkDFilterCount(dfilter, 1) + + def test_equal_2(self, checkDFilterCount): + dfilter = '_ws.col.def_dst == "207.46.134.94"' + checkDFilterCount(dfilter, 1) + + def test_not_equal_1(self, checkDFilterCount): + dfilter = '_ws.col.def_src != "10.0.0.5"' + checkDFilterCount(dfilter, 0) + + def test_read_filter(self, checkDFilterCountReadFilter): + dfilter = '_ws.col.protocol == "HTTP"' + checkDFilterCountReadFilter(dfilter, 1) + + def test_add_column(self, checkDFilterCount): + # Add column to configuration + dfilter = '_ws.col.expert == "Chat"' + checkDFilterCount(dfilter, 1, 'gui.column.format:"Expert","%a"') diff --git a/test/suite_dissection.py b/test/suite_dissection.py index adc2f3353c..45bc01ff3d 100644 --- a/test/suite_dissection.py +++ b/test/suite_dissection.py @@ -628,7 +628,7 @@ class TestDissectTcp: '-r', capture_file('http-ooo2.pcap'), '-otcp.reassemble_out_of_order:TRUE', '-Tfields', - '-eframe.number', '-etcp.reassembled_in', '-e_ws.col.Info', + '-eframe.number', '-etcp.reassembled_in', '-e_ws.col.info', '-2', ), encoding='utf-8', env=test_env) lines = stdout.split('\n') diff --git a/tfshark.c b/tfshark.c index e6c5613a6b..fc8a637969 100644 --- a/tfshark.c +++ b/tfshark.c @@ -116,7 +116,7 @@ static const char *separator = ""; static gboolean process_file(capture_file *, int, gint64); static gboolean process_packet_single_pass(capture_file *cf, epan_dissect_t *edt, gint64 offset, wtap_rec *rec, - const guchar *pd, guint tap_flags); + const guchar *pd); static void show_print_file_io_error(int err); static gboolean write_preamble(capture_file *cf); static gboolean print_packet(capture_file *cf, epan_dissect_t *edt); @@ -182,7 +182,7 @@ print_usage(FILE *output) fprintf(output, " -T pdml|ps|psml|text|fields\n"); fprintf(output, " format of text output (def: text)\n"); fprintf(output, " -e field to print if -Tfields selected (e.g. tcp.port,\n"); - fprintf(output, " _ws.col.Info)\n"); + fprintf(output, " _ws.col.info)\n"); fprintf(output, " this option can be repeated to print multiple fields\n"); fprintf(output, " -E= set options for output when -Tfields selected:\n"); fprintf(output, " header=y|n switch headers on and off\n"); @@ -1091,7 +1091,7 @@ process_packet_first_pass(capture_file *cf, epan_dissect_t *edt, static gboolean process_packet_second_pass(capture_file *cf, epan_dissect_t *edt, frame_data *fdata, wtap_rec *rec, - Buffer *buf, guint tap_flags) + Buffer *buf) { column_info *cinfo; gboolean passed; @@ -1123,7 +1123,7 @@ process_packet_second_pass(capture_file *cf, epan_dissect_t *edt, 2) we're printing packet info but we're *not* verbose; in verbose mode, we print the protocol tree, not the protocol summary. */ - if ((tap_flags & TL_REQUIRES_COLUMNS) || (print_packet_info && print_summary)) + if ((tap_listeners_require_columns()) || (print_packet_info && print_summary)) cinfo = &cf->cinfo; else cinfo = NULL; @@ -1367,8 +1367,7 @@ process_file(capture_file *cf, int max_packet_count, gint64 max_byte_count) process_packet_second_pass(cf, edt, fdata, &cf->rec, &buf, tap_flags); } #else - if (!process_packet_second_pass(cf, edt, fdata, &cf->rec, &buf, - tap_flags)) + if (!process_packet_second_pass(cf, edt, fdata, &cf->rec, &buf)) return FALSE; #endif } @@ -1424,7 +1423,7 @@ process_file(capture_file *cf, int max_packet_count, gint64 max_byte_count) if (!process_packet_single_pass(cf, edt, data_offset, &file_rec/*wtap_get_rec(cf->provider.wth)*/, - raw_data, tap_flags)) + raw_data)) return FALSE; /* Stop reading if we have the maximum number of packets; @@ -1531,8 +1530,7 @@ out: static gboolean process_packet_single_pass(capture_file *cf, epan_dissect_t *edt, gint64 offset, - wtap_rec *rec, const guchar *pd, - guint tap_flags) + wtap_rec *rec, const guchar *pd) { frame_data fdata; column_info *cinfo; @@ -1566,7 +1564,7 @@ process_packet_single_pass(capture_file *cf, epan_dissect_t *edt, gint64 offset, mode, we print the protocol tree, not the protocol summary. or 3) there is a column mapped as an individual field */ - if ((tap_flags & TL_REQUIRES_COLUMNS) || (print_packet_info && print_summary) || output_fields_has_cols(output_fields)) + if ((tap_listeners_require_columns()) || (print_packet_info && print_summary) || output_fields_has_cols(output_fields)) cinfo = &cf->cinfo; else cinfo = NULL; diff --git a/tshark.c b/tshark.c index 6671284b3c..7679e973e2 100644 --- a/tshark.c +++ b/tshark.c @@ -453,7 +453,7 @@ print_usage(FILE *output) fprintf(output, " -J top level protocol filter if -T ek|pdml|json selected\n"); fprintf(output, " (e.g. \"http tcp\", filter which expands all child nodes)\n"); fprintf(output, " -e field to print if -Tfields selected (e.g. tcp.port,\n"); - fprintf(output, " _ws.col.Info)\n"); + fprintf(output, " _ws.col.info)\n"); fprintf(output, " this option can be repeated to print multiple fields\n"); fprintf(output, " -E= set options for output when -Tfields selected:\n"); fprintf(output, " bom=y|n print a UTF-8 BOM\n"); @@ -3096,6 +3096,8 @@ process_packet_first_pass(capture_file *cf, epan_dissect_t *edt, from the dissection or running taps on the packet; if we're doing any of that, we'll do it in the second pass.) */ if (edt) { + column_info *cinfo = NULL; + /* If we're running a read filter, prime the epan_dissect_t with that filter. */ if (cf->rfcode) @@ -3115,9 +3117,14 @@ process_packet_first_pass(capture_file *cf, epan_dissect_t *edt, cf->provider.ref = &ref_frame; } + /* If we're applying a filter that needs the columns, construct them. */ + if (dfilter_requires_columns(cf->rfcode) || dfilter_requires_columns(cf->dfcode)) { + cinfo = &cf->cinfo; + } + epan_dissect_run(edt, cf->cd_t, rec, frame_tvbuff_new_buffer(&cf->provider, &fdlocal, buf), - &fdlocal, NULL); + &fdlocal, cinfo); /* Run the read filter if we have one. */ if (cf->rfcode) @@ -3303,7 +3310,7 @@ process_cap_file_first_pass(capture_file *cf, int max_packet_count, static gboolean process_packet_second_pass(capture_file *cf, epan_dissect_t *edt, frame_data *fdata, wtap_rec *rec, - Buffer *buf, guint tap_flags) + Buffer *buf, guint tap_flags _U_) { column_info *cinfo; gboolean passed; @@ -3328,12 +3335,14 @@ process_packet_second_pass(capture_file *cf, epan_dissect_t *edt, col_custom_prime_edt(edt, &cf->cinfo); /* We only need the columns if either - 1) some tap needs the columns + 1) some tap or filter needs the columns or 2) we're printing packet info but we're *not* verbose; in verbose mode, we print the protocol tree, not the protocol summary. + or + 3) there is a column mapped to an individual field */ - if ((tap_flags & TL_REQUIRES_COLUMNS) || (print_packet_info && print_summary) || output_fields_has_cols(output_fields)) + if ((tap_listeners_require_columns()) || (print_packet_info && print_summary) || output_fields_has_cols(output_fields) || dfilter_requires_columns(cf->dfcode)) cinfo = &cf->cinfo; else cinfo = NULL; @@ -3959,7 +3968,7 @@ out: static gboolean process_packet_single_pass(capture_file *cf, epan_dissect_t *edt, gint64 offset, - wtap_rec *rec, Buffer *buf, guint tap_flags) + wtap_rec *rec, Buffer *buf, guint tap_flags _U_) { frame_data fdata; column_info *cinfo; @@ -3994,13 +4003,13 @@ process_packet_single_pass(capture_file *cf, epan_dissect_t *edt, gint64 offset, col_custom_prime_edt(edt, &cf->cinfo); /* We only need the columns if either - 1) some tap needs the columns + 1) some tap or filter needs the columns or 2) we're printing packet info but we're *not* verbose; in verbose mode, we print the protocol tree, not the protocol summary. or 3) there is a column mapped as an individual field */ - if ((tap_flags & TL_REQUIRES_COLUMNS) || (print_packet_info && print_summary) || output_fields_has_cols(output_fields)) + if ((tap_listeners_require_columns()) || (print_packet_info && print_summary) || output_fields_has_cols(output_fields) || dfilter_requires_columns(cf->dfcode)) cinfo = &cf->cinfo; else cinfo = NULL; diff --git a/ui/qt/packet_list.cpp b/ui/qt/packet_list.cpp index 03404e292b..ca23186a0d 100644 --- a/ui/qt/packet_list.cpp +++ b/ui/qt/packet_list.cpp @@ -1106,6 +1106,8 @@ frame_data *PacketList::getFDataForRow(int row) const void PacketList::columnsChanged() { columns_changed_ = true; + column_register_fields(); + mainApp->emitAppSignal(MainApplication::FieldsChanged); if (!cap_file_) { // Keep columns_changed_ = true until we load a capture file. return;