add compare-results.sh, call from start-testsuite.sh
Compare current test results to the expected results, and exit in error on
discrepancies.
Add compare-result.sh: (trivially) grep junit xml output to determine which
tests passed and which didn't, and compare against an expected-result.log,
another junit file from a previous run. Summarize and determine success.
Include an "xfail" feature: tests that are expected to fail are marked as
"xfail", unexpected failures as "FAIL".
In various subdirs, copy the current jenkins jobs' junit xml outputs as
expected-results.log, so that we will start getting useful output in both
jenkins runs and manual local runs.
In start-testsuite.sh, after running the tests, invoke the results comparison.
Due to the single-line parsing nature, the script so far does not distinguish
between error and failure. I doubt that we actually need to do that though.
Related: OS#3136
Change-Id: I87d62a8be73d73a5eeff61a842e7c27a0066079d
2018-04-05 14:56:38 +00:00
|
|
|
<?xml version="1.0"?>
|
2018-11-05 11:41:05 +00:00
|
|
|
<testsuite name='BSC_Tests' tests='95' failures='0' errors='0' skipped='0' inconc='0' time='MASKED'>
|
2018-04-11 13:56:41 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_ctrl_msc_connection_status' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_ctrl_msc0_connection_status' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_ctrl' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_act_noreply' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_act_counter' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_act_ack_noest' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_act_ack_est_ind_noreply' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_act_ack_est_ind_refused' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_act_nack' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_exhaustion' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_rel_rll_rel_ind' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_rel_conn_fail' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_rel_hard_clear' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_rel_hard_rlsd' time='MASKED'/>
|
2018-09-06 12:13:34 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_rel_hard_rlsd_ms_dead' time='MASKED'/>
|
2018-04-11 13:56:41 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_chan_rel_a_reset' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_outbound_connect' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_cic_only' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_csd' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_ctm' time='MASKED'/>
|
2018-06-07 13:47:26 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_sign' time='MASKED'/>
|
2018-04-11 13:56:41 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_fr_a5_0' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_fr_a5_1' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_fr_a5_1_codec_missing' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_fr_a5_3' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_fr_a5_4' time='MASKED'/>
|
2018-05-10 21:04:00 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_ciph_mode_a5_0' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_ciph_mode_a5_1' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_ciph_mode_a5_3' time='MASKED'/>
|
2018-04-11 13:56:41 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_codec_fr' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_codec_hr' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_codec_efr' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_codec_amr_f' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_assignment_codec_amr_h' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_rll_est_ind_inact_lchan' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_rll_est_ind_inval_sapi1' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_rll_est_ind_inval_sapi3' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_rll_est_ind_inval_sacch' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_tmsi_nochan' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_tmsi_any' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_tmsi_sdcch' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_tmsi_tch_f' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_tmsi_tch_hf' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_cgi' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lac_ci' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_ci' time='MASKED'/>
|
2018-05-29 15:14:02 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lai' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lac' time='MASKED'/>
|
2018-04-11 13:56:41 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_all' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_plmn_lac_rnc' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_rnc' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lac_rnc' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lacs' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_lacs_empty' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_nochan_cgi_unknown_cid' time='MASKED'/>
|
2018-04-11 13:54:07 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_a_reset' time='MASKED'/>
|
2018-04-11 13:56:41 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_imsi_load' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_counter' time='MASKED'/>
|
2018-11-05 11:41:05 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_paging_resp_unsol' time='MASKED'/>
|
2018-04-11 13:56:41 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_rsl_drop_counter' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_rsl_unknown_unit_id' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_oml_unknown_unit_id' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_classmark' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_unsol_ass_fail' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_unsol_ass_compl' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_unsol_ho_fail' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_err_82_short_msg' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_err_84_unknown_msg' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_ho_int' time='MASKED'/>
|
2018-03-19 13:58:46 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_ho_out_of_this_bsc' time='MASKED'/>
|
2018-10-09 16:28:06 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_ho_out_fail_no_msc_response' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_ho_out_fail_rr_ho_failure' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_ho_out_fail_no_ho_detect' time='MASKED'/>
|
2018-03-19 13:58:46 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_ho_into_this_bsc' time='MASKED'/>
|
2018-04-11 13:56:41 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_bssap_rlsd_does_not_cause_bssmap_reset' time='MASKED'/>
|
2018-09-06 12:13:34 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_bssmap_clear_does_not_cause_bssmap_reset' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_ms_rel_ind_does_not_cause_bssmap_reset' time='MASKED'/>
|
2018-04-11 13:54:07 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_dyn_pdch_ipa_act_deact' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_dyn_pdch_ipa_act_nack' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_dyn_pdch_osmo_act_deact' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_dyn_pdch_osmo_act_nack' time='MASKED'/>
|
2018-09-06 12:13:34 +00:00
|
|
|
<testcase classname='BSC_Tests' name='TC_early_conn_fail' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests' name='TC_late_conn_fail' time='MASKED'/>
|
2018-06-07 13:47:26 +00:00
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_gcr_only' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_gcr_bway_connect' time='MASKED'/>
|
2018-07-06 12:03:38 +00:00
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_gcr_bway_connect_hr' time='MASKED'/>
|
2018-07-10 07:15:09 +00:00
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_gcr_bway_codec_mismatch' time='MASKED'/>
|
2018-06-07 13:47:26 +00:00
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_gcr_nomatch_bway_connect' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_gcr_bway_dont_connect' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_gcr_unsuppported_cfg' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_gcr_unsuppported_csc' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_gcr_bway_dont_connect_csc' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_connect_break' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_connect_clear' time='MASKED'/>
|
2018-11-06 18:28:05 +00:00
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_bts_gcr_bway_connect' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_bts_gcr_bway_connect_hr' time='MASKED'/>
|
|
|
|
<testcase classname='BSC_Tests_LCLS' name='TC_lcls_bts_connect_break' time='MASKED'/>
|
add compare-results.sh, call from start-testsuite.sh
Compare current test results to the expected results, and exit in error on
discrepancies.
Add compare-result.sh: (trivially) grep junit xml output to determine which
tests passed and which didn't, and compare against an expected-result.log,
another junit file from a previous run. Summarize and determine success.
Include an "xfail" feature: tests that are expected to fail are marked as
"xfail", unexpected failures as "FAIL".
In various subdirs, copy the current jenkins jobs' junit xml outputs as
expected-results.log, so that we will start getting useful output in both
jenkins runs and manual local runs.
In start-testsuite.sh, after running the tests, invoke the results comparison.
Due to the single-line parsing nature, the script so far does not distinguish
between error and failure. I doubt that we actually need to do that though.
Related: OS#3136
Change-Id: I87d62a8be73d73a5eeff61a842e7c27a0066079d
2018-04-05 14:56:38 +00:00
|
|
|
</testsuite>
|