Categories
Wowza

BT.709 Wowza

Problem

Noticed that our lowest transcoded video streams had a different colors to the rest of the streams. We do multiple transcoding’s, both 720p, 480p, 360p, 270p and it was only 360p and 270p that had this disparity from the original. The difference is slight but annoying enough that you notice it if you are constantly switching between qualities.

  • Original
  • 360p

Solution

Setting the three values colour_primaries, transfer_characteristics, matrix_coefficients to be BT.709 for 360p and 270p.

You do this by setting the following encoder properties

NameValue
mainconcept.colour_primaries1
mainconcept.transfer_characteristics1
mainconcept.matrix_coefficients1

Background

There is multiple values that says what color space that is being used. using MPC-HC we can se that the color space we are using for most of the files is BT.709, as it should be for HD broadcasts.

Color primariesBT.709
Transfer characteristicsBT.709
Matrix coefficientsBT.709
For 1080p, 720p and 480p, MPC-HC reported the following color information

however then dropping the resolution to below 360p Wowza and Mainconcept switches the color space.

Color primariesBT.601 PAL
Transfer characteristicsBT.470 System B, BT.470 System G
Matrix coefficientsBT.601
For 360p and 270p different color information was reported compared to the original 1080p

Tests

Wowza allows you to set advanced settings for the Mainconcept encoder, however you must configure Wowza to output what those settings are. you can look at the post Wowza Transcoder options to se which settings that exists.

ParameterPossible Values
mainconcept.colour_primaries1: ITU-R Rec. BT.709-5,
4: ITU-R Rec. BT.470-6 System M,
5: ITU-R Rec. BT.601-6 625,
6: ITU-R Rec. BT.601-6 525,
7: SMPTE 240M, 8: Generic film (colour filters using Illuminant C),
9: Rec. ITU-R BT.2020
mainconcept.transfer_characteristics0:auto (default),
1: BT.709-5,
4: BT.470-6 System M,
5: BT.470-6 System B G,
6: BT.709-5,
7: SMPTE 240M,
8: Linear,
14: Rec. ITU-R BT.2020
mainconcept.matrix_coefficients0:auto (default),
1: BT.709-5 System 1125,
4..6: BT.709-5 System 1250,
7: SMPTE 240M,
9: Rec. ITU-R BT.2020 non-constant luminance system
Difference settings for colors

My educated guess would be that I should set the values for BT.709-5 System 1250 since I am in a pal country running 50fps. But doing so resultat in an outputted file that had Matrix Coeffcient as BT.601

The only combination of the above settings that resultat in BT.709 for all settings was the value 1 for all three properties.

Categories
Uncategorized

Move Git Repository

Short howto move an repository from one remote origin to another.

Step 1, make sure everything is synced before you begin.

git pull
git push

Step 2. Check what your current remote origin is

$ git remote -v
origin  https://old/repository (fetch)
origin  https://old/repository (push)

Step 3. Set your new remote origin

git remote set-url origin ssh://new/repository

Step 4. Verify

$ git remote -v
origin  ssh://new/repository (fetch)
origin  ssh://new/repository (push)
Categories
Uncategorized

Disable dns over https

For Chrome & Edge

chrome://flags/#dns-over-https
Categories
Uncategorized Wowza

Wowza Transcoder options

To view all the encoding parameters you need to add logVideoEncodingParameters and logAudioEncodingParameters in your transcoder configuration as well as enabling debug logging in log4j.properties.

To enable debug add the following to your transcoder profile

<Encode>
	<Name>270p</Name>
	<Enable>true</Enable>
	<Description></Description>
	<StreamName>mp4:${SourceStreamName}_270p</StreamName>
	<Video>
		...
	</Video>
	<Audio>
		...
	</Audio>
	<Properties>
		<Property>
			<Name>logVideoEncodingParameters</Name>
			<Value>true</Value>
			<Type>Boolean</Type>
		</Property>
		<Property>
			<Name>logAudioEncodingParameters</Name>
			<Value>true</Value>
			<Type>Boolean</Type>
		</Property>
	</Properties>
</Encode>

These are the options for your video codec (if using cpu)

# long: profile_id: 0=H264PROFILE_BASELINE, 1=H264PROFILE_MAIN, 3=H264PROFILE_HIGH
mainconcept.profile_id: 3
# long: level_id: = 10 -> 1.0, .. 51 -> 5.1
mainconcept.level_id: 32
# long: idr_interval: gop length
mainconcept.idr_interval: 0
# long: reordering_delay: max distance between two P frames
mainconcept.reordering_delay: 1
# long: use_b_slices
mainconcept.use_b_slices: 0
# long: interlace_mode: 0=H264_PROGRESSIVE, 1=H264_INTERLACED, 2=H264_MBAFF
mainconcept.interlace_mode: 0
# long: def_horizontal_size
mainconcept.def_horizontal_size: 852
# long: def_vertical_size
mainconcept.def_vertical_size: 480
# double: frame_rate
mainconcept.frame_rate: 50.0
# long: num_reference_frames
mainconcept.num_reference_frames: 4
# long: search_range: motion search: spatial search range
mainconcept.search_range: 255
# long: rd_optimization: motion search: rate-distortion optimization
mainconcept.rd_optimization: 1
# long: max_l0_active: motion search: max index of reference frames in list0
mainconcept.max_l0_active: 0
# long: max_l1_active: motion search: max index of reference frames in list1
mainconcept.max_l1_active: 0
# long: quant_pI: macro blocks: quantization parameter
mainconcept.quant_pI: 24
# long: quant_pP: macro blocks: quantization parameter
mainconcept.quant_pP: 25
# long: quant_pB: macro blocks: quantization parameter
mainconcept.quant_pB: 27
# long: bit_rate_mode: bit rate stuff: 0=H264_CBR, 1=H264_CQT, 2=H264_VBR or 3=H264_TQM
mainconcept.bit_rate_mode: 2
# long: bit_rate_buffer_size: bit rate stuff: vbv buffer size
mainconcept.bit_rate_buffer_size: 3740000
# long: bit_rate: bit rate stuff: avg bitrate; if 0, use above quantization parameters
mainconcept.bit_rate: 1700000
# long: max_bit_rate: bit rate stuff: max bitrate, used in VBR mode
mainconcept.max_bit_rate: 1869952
# long: inter_search_shape: prediction: 0=H264_INTERSEARCH_16x16 (use only 16x16 block size) or 1=H264_INTERSEARCH_8x8 (use block size down to 8x8)
mainconcept.inter_search_shape: 1
# long: entropy_coding_mode: coding mode: 0=H264_CAVLC or 1=H264_CABAC
mainconcept.entropy_coding_mode: 1
# long: use_hadamard_transform: coding mode: 0=SAD is used, 1=SATD is used
mainconcept.use_hadamard_transform: 0
# long: sar_width: vui parameters: sample aspect ratio: horizontal size in arbitrary units
mainconcept.sar_width: 1
# long: sar_height: vui parameters: sample aspect ratio: vertical size in arbitrary units
mainconcept.sar_height: 1
# long: video_format: vui parameters: 1=PAL, 2=NTSC
mainconcept.video_format: 1
# long: video_full_range: vui parameters: (currently) not used
mainconcept.video_full_range: 2
# long: num_units_in_tick: vui parameters: timing info use together with time_scale
mainconcept.num_units_in_tick: 1
# long: time_scale: vui parameters: timing info use together with num_units_in_tick (fps = time_scale/num_units_in_tick)
mainconcept.time_scale: 50
# long: vbv_buffer_fullness: advanced settings: initial vbv-fullness
mainconcept.vbv_buffer_fullness: 10
# long: vbv_buffer_fullness_trg: advanced settings: final vbv-fullness
mainconcept.vbv_buffer_fullness_trg: 100
# long: vbv_buffer_units: advanced settings: units of vbv-fullness and buffer size
mainconcept.vbv_buffer_units: 0
# long: cpb_removal_delay: advanced settings: CPB removal delay for the first picture (needed for segment merging)
mainconcept.cpb_removal_delay: 0
# long: bit_rate_scale: external setting of bit_rate_scale (avoids recalculation of bitrate)
mainconcept.bit_rate_scale: 1
# long: cpb_size_scale: external setting of cpb_size_scale (avoids recalculation of bitrate)
mainconcept.cpb_size_scale: 3
# long[4]: max_frame_size: max frames size for I, P, Bref, B frames
mainconcept.max_frame_size: 0,0,0,0
# long: hrd_maintain: 0=hrd model disabled, 1=hrd model enabled
mainconcept.hrd_maintain: 1
# long[4]: min_frame_size: min frames size for I, P, Bref, B frames
mainconcept.min_frame_size: 0,0,0,0
# long: hrd_low_delay: 0=low delay hrd disabled, 1=low delay hrd enabled
mainconcept.hrd_low_delay: 0
# long: smooth_factor: Quantizer curve compression smooth factor, 0 = disabled
mainconcept.smooth_factor: 0
# long: use_deblocking_filter: in-loop filter:
mainconcept.use_deblocking_filter: 1
# long: deblocking_alphaC0_offset: in-loop filter:
mainconcept.deblocking_alphaC0_offset: -1
# long: deblocking_beta_offset: in-loop filter:
mainconcept.deblocking_beta_offset: -1
# long: adaptive_deblocking: in-loop filter: adaptive deblocking filter
mainconcept.adaptive_deblocking: 0
# long: video_type: type issues: one of above defines
mainconcept.video_type: 5
# long: video_pulldown_flag: type issues: one of above flags
mainconcept.video_pulldown_flag: 0
# long: overscan_appropriate_flag: additional vui parameters: controls overscan_appropriate_flag of VUI
mainconcept.overscan_appropriate_flag: 0
# long: stream_type: file/stream issues: 0=H264_STREAM_TYPE_I, 1=H264_STREAM_TYPE_I_SEI, 3=H264_STREAM_TYPE_II, 4=H264_STREAM_TYPE_II_NO_SEI
mainconcept.stream_type: 2
# long: frame_mbs_mode: file/stream issues: 0=H264_FRAME_MBS_ON, 1=H264_FRAME_MBS_OFF
mainconcept.frame_mbs_mode: 0
# long: bit_depth_luma: bit_depth_chroma
mainconcept.bit_depth_luma: 8
# long: bit_depth_chroma: bit_depth_chroma
mainconcept.bit_depth_chroma: 8
# long: chroma_format: 1=H264_CHROMA_400, 2=H264_CHROMA_420 or 3=H264_CHROMA_422, 4=H264_CHROMA_444
mainconcept.chroma_format: 2
# long: vui_presentation: configuration of VUI header: 0 - old school / auto mode, 1 - customized configuration, depending on following flags/bits, 0x002 - aspect_ratio_info_present_flag, 0x004 - overscan_info_present_flag (not supported), 0x008 - video_signal_type_present_flag, 0x010 - colour_description_present_flag, 0x020 - chroma_loc_info_present_flag (not supported), 0x040 - timing_info_present_flag, 0x080 - nal_hrd_parameters_present_flag, 0x100 - vcl_hrd_parameters_present_flag, 0x200 - pic_struct_present_flag, 0x400 - bitstream_restriction_flag
mainconcept.vui_presentation: 0
# long: write_au_delimiters: write access unit delimiters
mainconcept.write_au_delimiters: 1
# long: write_seq_end_code: write seq end code
mainconcept.write_seq_end_code: 1
# long: write_timestamps: write picture timecode in PT SEI
mainconcept.write_timestamps: 1
# long: timestamp_offset: frame offset (in #frames) for timestamps (default = 0)
mainconcept.timestamp_offset: 0
# long: drop_frame_timecode: use NTSC drop frame timecode notation for 29.97 and 59.94 target frame rates
mainconcept.drop_frame_timecode: 0
# long: write_single_sei_per_nalu: writes single sei message per nal unit
mainconcept.write_single_sei_per_nalu: 0
# long: write_seq_par_set: behaviour of writing sequence parameter set (default = 0): 0 - old school (SPS once per IDR), 1 - SPS once per I-frame
mainconcept.write_seq_par_set: 0
# long: write_pic_par_set: behaviour of writing picture parameter set (default = 0): 0 - old school (PPS once per IDR), 1 - PPS once per I picture, 2 - PPS once per picture
mainconcept.write_pic_par_set: 0
# long: log2_max_poc: allows to specify custom log2_max_pic_order_cnt_lsb_minus4 value: valid range is 4..16, default - 8
mainconcept.log2_max_poc: 8
# long: log2_max_frame_num: allows to specify custom log2_max_frame_num_minus4 value: valid range is 4..16, default - 8
mainconcept.log2_max_frame_num: 8
# long: pic_order_cnt_type: allows to specify custom pic_order_cnt_type value: valid values are 0 (default) and 2
mainconcept.pic_order_cnt_type: 0
# long: pic_order_present_flag: controls pic_order_present_flag value in PPS (e.g. for SBTVD-T)
mainconcept.pic_order_present_flag: 0
# long: fixed_frame_rate: controls fixed_frame_rate_flag in VUI
mainconcept.fixed_frame_rate: 1
# long: frame_based_timing: controls time_scale / num_units_in_tick ration in VUI (field or frame rate)
mainconcept.frame_based_timing: 0
# long: vcsd_mode: scene detection: visual content scene detection, 0: OFF, 1: IDR (see vcsd_mode_flags)
mainconcept.vcsd_mode: 0
# long: vcsd_sensibility: describes sensibility of scene detector (0: high, 100: 0)
mainconcept.vcsd_sensibility: 0
# long: slice_mode: advanced settings: = 1, use multiple slices per picture
mainconcept.slice_mode: 1
# long: slice_arg: advanced settings: number of slices per picture
mainconcept.slice_arg: 1
# long: b_slice_reference: advanced settings: use b slices as reference too (b -> B)
mainconcept.b_slice_reference: 0
# long: b_slice_pyramid: advanced settings: pyramidal gop structure  (...bBb...)
mainconcept.b_slice_pyramid: 0
# long: cb_offset: advanced settings: chroma quality offset (-X -> increase quality, +X -> decrease quality)
mainconcept.cb_offset: 1
# long: cr_offset: advanced settings: chroma quality offset (-X -> increase quality, +X -> decrease quality)
mainconcept.cr_offset: 1
# long: me_subpel_mode: advanced settings: 0=H264_FULL_PEL, 1=H264_HALF_PEL, 2=H264_QUARTER_PEL
mainconcept.me_subpel_mode: 2
# long: me_weighted_p_mode: advanced settings: whether to use explicit WP for P-frames or not
mainconcept.me_weighted_p_mode: 1
# long: me_weighted_b_mode: advanced settings: (currently) not used
mainconcept.me_weighted_b_mode: 0
# long: enable_fast_intra_decisions: advanced settings: enables fast intra decisions
mainconcept.enable_fast_intra_decisions: 2
# long: enable_fast_inter_decisions: advanced settings: enables fast inter decisions
mainconcept.enable_fast_inter_decisions: 2
# long: pic_ar_x: advanced settings: picture aspect ratio: horizontal size
mainconcept.pic_ar_x: -1
# long: pic_ar_y: advanced settings: picture aspect ratio: vertical size
mainconcept.pic_ar_y: -1
# long: calc_quality: advanced settings: calculate PSNR or not
mainconcept.calc_quality: 0
# long: cpu_opt: advanced settings: cpu optimization
mainconcept.cpu_opt: 0
# long: num_threads: advanced settings: number of threads
mainconcept.num_threads: 0
# long: live_mode: advanced settings: online or offline mode (not yet implemented)
mainconcept.live_mode: 0
# long: buffering: advanced settings: maximum number of seconds to buffer
mainconcept.buffering: 0
# long: min_quant: advanced settings: minimum quantization parameter
mainconcept.min_quant: 0
# long: max_quant: advanced settings: maximum quantization parameter
mainconcept.max_quant: 51
# long: max_slice_size: advanced settings: maximum slice size in bits, set to 0 if you do not need it
mainconcept.max_slice_size: 0
# long: encoding_buffering: advanced settings: maximum number of seconds to buffer encoding queue
mainconcept.encoding_buffering: 0
# long: low_delay: advanced settings: low delay mode (not yet implemented)
mainconcept.low_delay: 0
# long: air_mode: advanced settings: adaptive intra refresh mode
mainconcept.air_mode: 0
# long: detach_thread: advanced settings: run core in a new thread or not
mainconcept.detach_thread: 1
# long: constrained_intra_pred: advanced settings: constrained intra prediction for improving error resilience
mainconcept.constrained_intra_pred: 0
# long: air_split_frequency: advanced settings: frequency of intra lines for adaptive air mode split
mainconcept.air_split_frequency: 0
# long: air_qp_offset: advanced settings: qp offset for adaptive intra mode lines
mainconcept.air_qp_offset: 0
# long: min_idr_interval: advanced gop settings: min gop length
mainconcept.min_idr_interval: 1
# long: adaptive_b_frames: advanced gop settings: use adaptive B-frames placement or not
mainconcept.adaptive_b_frames: 0
# long: idr_frequency: advanced gop settings: 0 means only first frame is IDR, 1 means every I-frame is IDR, etc
mainconcept.idr_frequency: 1
# long: field_order: advanced gop settings: 0=TFF, 1=BFF
mainconcept.field_order: 0
# long: fixed_i_position: advanced gop settings: constant I frame position
mainconcept.fixed_i_position: 0
# long: isolated_gops: advanced gop settings: allows to limit referencing to frames from the previous GOP, may be useful when idr_frequency != 1
mainconcept.isolated_gops: 0
# long: hierar_p_frames: advanced gop settings: GOP structure with hierarchically coded P frames (for temporal scalability with out using B frames) (not yet supported in enc_avc/trunk)
mainconcept.hierar_p_frames: 0
# long: fast_multi_ref_me: advanced me settings: enables fast decisions for multi-ref ME
mainconcept.fast_multi_ref_me: 1
# long: fast_sub_block_me: advanced me settings: enables fast decisions for sub-block ME
mainconcept.fast_sub_block_me: 1
# long: allow_out_of_pic_mvs: advanced me settings: enables mvs out of picture boundaries
mainconcept.allow_out_of_pic_mvs: 1
# long: constrained_ref_list: advanced me settings: use constrained reference picture list
mainconcept.constrained_ref_list: 0
# long: enable_intra_big: advanced intra settings: allows to use 16x16 intra mode in intra slices
mainconcept.enable_intra_big: 1
# long: enable_intra_8x8: advanced intra settings: same for 8x8
mainconcept.enable_intra_8x8: 1
# long: enable_intra_4x4: advanced intra settings: same for 4x4
mainconcept.enable_intra_4x4: 1
# long: enable_intra_pcm: advanced intra settings: same for PCM
mainconcept.enable_intra_pcm: 0
# long: enable_inter_big: advanced intra settings: allows to use 16x16 intra mode in inter slices
mainconcept.enable_inter_big: 1
# long: enable_inter_8x8: advanced intra settings: same for 8x8
mainconcept.enable_inter_8x8: 1
# long: enable_inter_4x4: advanced intra settings: same for 4x4
mainconcept.enable_inter_4x4: 1
# long: enable_inter_pcm: advanced intra settings: same for PCM
mainconcept.enable_inter_pcm: 0
# long: fast_rd_optimization: advanced rdo settings: allows fast RDO
mainconcept.fast_rd_optimization: 1
# long: quant_mode: advanced rdo settings: quantization optimization mode
mainconcept.quant_mode: 2
# long: grain_mode: advanced rdo settings: granular noise optimization mode, not used any more
mainconcept.grain_mode: 0
# long: grain_opt_strength: advanced rdo settings: scalable film grain optimization [0..100], use 0 to turn it off
mainconcept.grain_opt_strength: 0
# long[8]: adaptive_quant_strength: advanced rdo settings: adaptive quantization strength [-100..100] or every mode (from H264_AQUANT_MODE_BRIGHTNESS to COMPLEXITY), last 5 strengths are not used at the moment
mainconcept.adaptive_quant_strength: 0,0,0,0,0,0,0,0
# long: denoise_strength_y: Psycho-visual enhancement: denoise strength for luma [0..100]
mainconcept.denoise_strength_y: 0
# long: denoise_strength_c: Psycho-visual enhancement: denoise strength for chroma [0..100]
mainconcept.denoise_strength_c: 0
# long: black_norm_level: Psycho-visual enhancement: black normalization level, any luma less than or equal to black_norm_level will be set to 16
mainconcept.black_norm_level: 0
# long: pulse_reduction: Psycho-visual enhancement: key frame pulsing reduction
mainconcept.pulse_reduction: 0
# long: aux_format_idc: alpha plane: aux_format_idc
mainconcept.aux_format_idc: -1
# long: bit_depth_aux: alpha plane: bit_depth_aux
mainconcept.bit_depth_aux: 8
# long: alpha_incr_flag: alpha plane: alpha_incr_flag
mainconcept.alpha_incr_flag: 0
# long: alpha_opaque_value: alpha plane: alpha_opaque_value
mainconcept.alpha_opaque_value: 0
# long: alpha_transparent_value: alpha plane: alpha_transparent_value
mainconcept.alpha_transparent_value: 0
# long: seq_scaling_matrix_present_flag: seq_scaling_list_present_flag equal[i] to 1 specifies that the syntax structure
mainconcept.seq_scaling_matrix_present_flag: 0
# long[8]: seq_scaling_list_present_flag: for scaling list i is present in the sequence parameter set (and not in the picture parameter set)
mainconcept.seq_scaling_list_present_flag: 0,0,0,0,0,0,0,0
# long[16]: intra_y_4x4_scaling_list: scaling lists: intra_y_4x4_scaling_list
mainconcept.intra_y_4x4_scaling_list: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
# long[16]: intra_cb_4x4_scaling_list: scaling lists: intra_cb_4x4_scaling_list
mainconcept.intra_cb_4x4_scaling_list: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
# long[16]: intra_cr_4x4_scaling_list: scaling lists: intra_cr_4x4_scaling_list
mainconcept.intra_cr_4x4_scaling_list: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
# long[16]: inter_y_4x4_scaling_list: scaling lists: inter_y_4x4_scaling_list
mainconcept.inter_y_4x4_scaling_list: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
# long[16]: inter_cb_4x4_scaling_list: scaling lists: inter_cb_4x4_scaling_list
mainconcept.inter_cb_4x4_scaling_list: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
# long[16]: inter_cr_4x4_scaling_list: scaling lists: inter_cr_4x4_scaling_list
mainconcept.inter_cr_4x4_scaling_list: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
# long[64]: intra_y_8x8_scaling_list: scaling lists: intra_y_8x8_scaling_list
mainconcept.intra_y_8x8_scaling_list: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
# long[64]: inter_y_8x8_scaling_list: scaling lists: inter_y_8x8_scaling_list
mainconcept.inter_y_8x8_scaling_list: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
# long: performance_level: (0-15) 0:H264_PERF_FASTEST, 9:H264_PERF_BALANCED, 15:H264_PERF_BEST_Q
mainconcept.performance_level: -1
# long: keyframe_follow: 0:off, 1:on (default)
mainconcept.keyframe_follow: 1
# long: maintain_original_pts: 0:off, 1:on (default)
mainconcept.maintain_original_pts: 1
# long: colour_primaries: 0:auto (default), 1: ITU-R Rec. BT.709-5, 4: ITU-R Rec. BT.470-6 System M, 5: ITU-R Rec. BT.601-6 625, 6: ITU-R Rec. BT.601-6 525, 7: SMPTE 240M, 8: Generic film (colour filters using Illuminant C), 9: Rec. ITU-R BT.2020
mainconcept.colour_primaries: 0
# long: transfer_characteristics: 0:auto (default), 1: BT.709-5,4: BT.470-6 System M,5: BT.470-6 System B G,6: BT.709-5,7: SMPTE 240M,8: Linear,14: Rec. ITU-R BT.2020
mainconcept.transfer_characteristics: 0
# long: matrix_coefficients: 0:auto (default), 1: BT.709-5 System 1125, 4..6: BT.709-5 System 1250, 7: SMPTE 240M, 9: Rec. ITU-R BT.2020 non-constant luminance system
mainconcept.matrix_coefficients: 0
# long: extended_sar: Enable extended sample aspect ratio
mainconcept.extended_sar: 0

These are the parameters available for your audio.

# long: channel copy: 0=donothing, 1=copy-left-over-right, 2=copy-right-over-left
mainconcept.channel_copy: 0
# long: crc protection: 0=off, 1=on
mainconcept.protect_adts_stream: 0
# long: high frequency cut-off
mainconcept.hf_cutoff: 12000
# long: mpeg version: 6=MPEG2_AAC_AUDIO, 7=MPEG4_AAC_AUDIO, 16=MPEG4_AAC_AUDIO_PSP
mainconcept.mpeg_version: 7
# long: variable bitrate mode: 0 = off
mainconcept.vbr: 0
# long: temporal noise shaping (tns) algorithm: 0=off, 1=on
mainconcept.tns: 1
# long: aac object type
mainconcept.aac_object_type: 2
# long: high efficiency: 0=off, 1=HE-v1, 2=HE-v2
mainconcept.he: 0

I extracted this from the logfile using the following command

awk 'BEGIN { FS="\t" } { print $9 }'  wowzastreamingengine_access.log | grep 'mainconcept\|^#' > parameters.log

Update 26 April

As of Wowza version 4.8.11 Wowza has included a new transcoder engine called beamr4. here are the parameters available for beamr4

# long: profile_idc: H.264 profile (default=100/110): PROFILE_IDC_BASELINE=66, PROFILE_IDC_MAIN=77, PROFILE_IDC_HIGH=100, PROFILE_IDC_HIGH_10=110, PROFILE_IDC_HIGH_422=122-642757.407
beamr4.profile_idc: 100-642757.409
# long: level_idc: H.264 level: 0(default)=auto, 10=level 1, 9=level 1b, 11=level 1.1 etc.-642757.41
beamr4.level_idc: 31-642757.41
# long: sym_mode: Symbol mode: 0=CAVLC, 1(default)=CABAC-642757.41
beamr4.sym_mode: 1-642757.41
# long: interlace_mode: Interlace coding mode (default=0): INTERLACE_MODE_NONE=0, INTERLACE_MODE_ALL_FIELD_TOP_FIRST=1, INTERLACE_MODE_ALL_FIELD_BOTTOM_FIRST=2, INTERLACE_MODE_FRAME_MBAFF=3-642757.41
beamr4.interlace_mode: 0-642757.411
# long: constrained_intra_pred: Constrained intra prediction flag (default=0)-642757.411
beamr4.constrained_intra_pred: 0-642757.411
# long: poc_type: Picture Order Count (POC) type (default=0)-642757.411
beamr4.poc_type: 0-642757.411
# long: direct_mode: Direct mode type for B-frames: 0=temporal, 1=spatial-642757.412
beamr4.direct_mode: 1-642757.412
# long: chroma_qp_offset: Offset for chroma QP [-12..+12]-642757.412
beamr4.chroma_qp_offset: 0-642757.412
# long: interlace_flags: Interlace coding feature bit mask (default=0): 0x01=INT_PREDICT_BOTTOM_FROM_TOP, 0x02=INT_INTRA_FOR_BOTH_FIELD, 0x04=INT_BOTTOM_FIRST, 0x08=INT_FORCE_INTERLACE_PICTS, 0x10=INT_MBAFF_BOTTOM_ZERO_POC, 0x20=INT_MBAFF_USE_FULL_FF_DECIDE, 0x40=INT_DISABLE_BOTTOM_FIELD_PREPROC, 0x80=INT_ADD_TELECINE_PICT_STRUCT, 0x100=INT_TELECINE_START_WITH_3-642757.412
beamr4.interlace_flags: 0-642757.412
# long: weighted_pred_flag: Enables weighted prediction for P-frames: 0=off, 1(default)=on-642757.412
beamr4.weighted_pred_flag: 1-642757.413
# long: weighted_bipred_idc: Enables weighted prediction for B-frames: 0(default)=off, 2=on-642757.413
beamr4.weighted_bipred_idc: 2-642757.413
# long: enc_flags: Encoder feature bit mask (default=0x100): ENC_DISABLE_VUI = 0x1, ENC_SLICE_TYPE_012 = 0x2, ENC_SPS_ONLY_ONCE = 0x4, ENC_REC_POINT_IDR = 0x8, ENC_FRAME_PACKING = 0x10, ENC_MBS_DATA = 0x20, ENC_REDUCE_DEAD_ZONE = 0x40, ENC_HARD_SCENES = 0x80, ENC_PSY_OPT_DECISIONS = 0x100-642757.413
beamr4.enc_flags: 256-642757.413
# long: avc_intra_class: AVC-Intra class encoding (default=0): 0=off, 50=class 50, 100=class 100-642757.413
beamr4.avc_intra_class: 0-642757.414
# long: avc_intra_flags: AVC-intra encoding bit mask (default=0): 1=AVC_I_FORCE_PANASONIC-642757.414
beamr4.avc_intra_flags: 1-642757.414
# long: speed: Speed/quality mode: [0..16] 0=slowest (best quality), 2=default, 16=fastest (lowest quality)-642757.414
beamr4.speed: 6-642757.414
# long: preproc_enable: Preprocessing Enable: 0(default)=disabled, 1=enabled-642757.414
beamr4.preproc_enable: 0-642757.414
# long: crop_enable: Picture Cropping: 0(default)=disabled, 1=enabled-642757.415
beamr4.crop_enable: 0-642757.415
# long: crop_left: Left offset in pixels-642757.415
beamr4.crop_left: 0-642757.415
# long: crop_top: Top offset in pixels-642757.415
beamr4.crop_top: 0-642757.415
# long: crop_right: Right offset in pixels-642757.416
beamr4.crop_right: 0-642757.416
# long: crop_bottom: Bottom offset in pixels-642757.416
beamr4.crop_bottom: 0-642757.416
# long: source_type: Source Type: 0=progressive, 1=top field first, 2=bottom field first-642757.416
beamr4.source_type: 0-642757.416
# long: me_denoise_level: ME temporal denoise intensity (default=0): N_NONE=0, WEAK=1, MODERATE=2, MIDDLE=3, STRONG=4, VERY_STRONG=5, HARD=6-642757.416
beamr4.me_denoise_level: 0-642757.417
# long: me_denoise_skip_luma: ME denoise: 0(default)=process luma, 1=skip luma-642757.417
beamr4.me_denoise_skip_luma: 0-642757.417
# long: me_denoise_skip_chroma: ME denoise: 0(default)=process chroma, 1=skip chroma-642757.417
beamr4.me_denoise_skip_chroma: 0-642757.417
# long: intra_precision: Intra complexity analysis precision: [0..4] 0=disabled, 2=default, 4=maximum-642757.417
beamr4.intra_precision: 2-642757.417
# long: idr_period: Period of IDR frames: 0(default)=only first, N=every N-th I-frame-642757.418
beamr4.idr_period: 1-642757.418
# long: bframes: Number of B-frames between P's: [0..7] 0=no B-frames-642757.418
beamr4.bframes: 0-642757.418
# long: min_bframes: Minimum number of B-frames for adaptive mode: [0..bfames] 0=min, gop.bframes=disabled-642757.418
beamr4.min_bframes: 0-642757.418
# long: emulate_b: B-frame emulation with non-ref. P's (requires non-zero bframes): 0(default)=off, 1=on, 2=on, natural order-642757.418
beamr4.emulate_b: 0-642757.418
# long: aud: Enable Access Unit Delimiters: 0(default)=disable, 1=enable AUD+PPS, 2=enable AUD only-642757.419
beamr4.aud: 1-642757.419
# long: min_intra_period: Minimal distance between intra frames for continuous scene change (frames): 4=default-642757.419
beamr4.min_intra_period: 4-642757.419
# long: sps_period: How often SPS/PPS is included (frames): 0(default)=not set-642757.419
beamr4.sps_period: 0-642757.419
# long: gop_flags: GOP feature bit mask (default= 0x3): 0x1=GOP_USE_HIERARCHICAL_B, 0x2=GOP_NO_IDR_ON_SCENE_CHANGE, 0x4=GOP_FORCE_B_IN_ODD_POSITION, 0x10=GOP_VARIABLE_FPS-642757.419
beamr4.gop_flags: 2-642757.419
# long: rc_type: Type of the Rate Control (default=2): 0=RATE_CONTROL_QP, 1=RATE_CONTROL_VBR, 2=RATE_CONTROL_CBR, 3=RATE_CONTROL_CBR_WITH_FILLER, 7=RATE_CONTROL_CABR_VBR, 8=RATE_CONTROL_CABR_PASS_1, 10=RATE_CONTROL_CSQ, 11=RATE_CONTROL_CABR_CSQ-642757.42
beamr4.rc_type: 2-642757.42
# long: rc_flags: RC feature bit mask (default=0): 0x01=RC_IGNORE_VBR_BUFFER_OVERFLOW, 0x02=RC_BOUND_MAX_QP_DELTA_B, 0x04=RC_BOUND_MIN_QP_DELTA_B, 0x10=RC_ADD_FILLER, 0x20=RC_SET_CBR_FLAG, 0x40=RC_FIXED_GOP_SIZE, 0x100=RC_EXCLUDE_T1, 0x200=RC_EXCLUDE_T2, 0x400=RC_RESTRICT_FRAME_SIZE, 0x800=RC_ALL_T_LAYERS, 0x800000=RC_CABR_FAST-642757.42
beamr4.rc_flags: 0-642757.42
# long: kbps: Desired bitrate (kbits per sec)-642757.42
beamr4.kbps: 2800-642757.42
# long: qp_intra: Quant Parameter for I-frames (requires auto_qp=0): [0..51] 30=default-642757.42
beamr4.qp_intra: 30-642757.42
# long: csq_factor: CSQ quality factor: [0..51] (default=23)-642757.421
beamr4.csq_factor: 23-642757.421
# long: qp_modulation: QP modulation mode: 0= off, 1(default),2= on, 4=improved-642757.421
beamr4.qp_modulation: 4-642757.421
# long: qp_delta_p: Base QP delta between I and P: [0..51] (default=4)-642757.421
beamr4.qp_delta_p: 4-642757.422
# long: qp_delta_b: Base QP delta between P and non-reference B: [0..51] (default=4)-642757.422
beamr4.qp_delta_b: 4-642757.422
# long: qp_delta_b_ref: Base QP delta between P and reference B: [0.51] (default=auto)-642757.422
beamr4.qp_delta_b_ref: 32767-642757.422
# long: auto_qp: Enables automatic QP selection: 0=off, 1(default)=on-642757.423
beamr4.auto_qp: 1-642757.423
# long: qp_max: Maximum allowed QP for Rate Control: 51=default-642757.423
beamr4.qp_max: 51-642757.423
# long: qp_min: Minimum allowed QP for Rate Control: 8=default-642757.423
beamr4.qp_min: 12-642757.423
# long: vbv_length: Rate control buffer length in msecs: default=2000-642757.424
beamr4.vbv_length: 2000-642757.424
# long: initial_cpb_removal_delay: Initial fullness for VBV buffer (1/90000 sec): (-1)(default)=auto-642757.424
beamr4.initial_cpb_removal_delay: 135000-642757.424
# long: mb_update: Enables MB-level rate-control: 0= off, 1(default)=per-MB-row update; 2= per-MB update (CBR only)-642757.424
beamr4.mb_update: 1-642757.424
# long: look_ahead: Number of look-ahead frames: 0=off, 1=default-642757.424
beamr4.look_ahead: 1-642757.425
# long: max_kbps: Maximum allowed peak bitrate (Kbps) for VBR mode: 0(default)=not set-642757.425
beamr4.max_kbps: 0-642757.425
# long: max_intra_frame_bytes: Maximum size of intra (I) frames (bytes): 0(default)=no restriction-642757.425
beamr4.max_intra_frame_bytes: 0-642757.425
# long: min_intra_frame_bytes: Minimum size of intra (I) frames (bytes): 0(default)=no restriction-642757.425
beamr4.min_intra_frame_bytes: 0-642757.425
# long: max_inter_frame_bytes: Maximum size of inter (P or B) frames (bytes): 0(default)=no restriction-642757.426
beamr4.max_inter_frame_bytes: 0-642757.426
# long: gop_bytes: Size of GOP in bytes (effective only with 0x40 set in rc_flags): 0=auto (based on kbps or max_kbps)-642757.426
beamr4.gop_bytes: 0-642757.426
# long: max_refs: Number of pictures (frames/fields) used for motion search (reference pictures): [1..10] (default=1)-642757.426
beamr4.max_refs: 2-642757.426
# long: subdiv: Macroblock subdivision bit mask (default=7): [1..7] MODE_16X16 = 1, MODE_16X8 = 2, MODE_8X16 = 4-642757.426
beamr4.subdiv: 7-642757.426
# long: me_flags: ME feature bit mask (default=0x3010): 0x10=ME_LIMIT_B_REFS, 0x20=ME_DISABLE_PREPROC_ME, 0x40=ME_DISABLE_PREPROC_COMPLEXITY, 0x1000=ME_USE_FULL_LOG_SEARCH_P, 0x2000=ME_USE_FULL_LOG_SEARCH_B, 0x4000=ME_VPA_POSTPROC_OFF-642757.427
beamr4.me_flags: 12304-642757.427
# long: search_range: Maximum search range in full pels: [1..64] -1(default)= auto (dep. on picture size)-642757.427
beamr4.search_range: 32-642757.427
# long: slice_mode: Slice mode: 0(default)=none, 1=#mbs per slice, 2=#bytes per slice; 3=#slices-642757.427
beamr4.slice_mode: 0-642757.427
# long: slice_param: Number value for the slice.mode: 0=default-642757.427
beamr4.slice_param: 0-642757.427
# long: i_param: Number value for the slice.mode for I-slices: 0=default-642757.428
beamr4.i_param: 0-642757.428
# long: b_param: Number value for the slice.mode for B-slices: 0=default-642757.428
beamr4.b_param: 0-642757.428
# long: deblock_flag: Loop filtering: 0=params below ignored, 1=params below used (default=0)-642757.428
beamr4.deblock_flag: 0-642757.428
# long: deblock_disable: Loop filter in slice header: 0=on, 1=off (default=0)-642757.428
beamr4.deblock_disable: 0-642757.428
# long: alpha_c0: Alpha & C0 offset div. 2: [-6..+6] (default=0)-642757.429
beamr4.alpha_c0: 0-642757.429
# long: beta_c0: Beta offset div. 2: [-6..+6] (default=0)-642757.429
beamr4.beta_c0: 0-642757.429
# long: mt_disable: Multi-threading: 0(default)= MT on, 1= MT off-642757.429
beamr4.mt_disable: 0-642757.429
# long: num_threads: No. of worker threads to run: 0(default)=auto (CPU threads)-642757.429
beamr4.num_threads: 32-642757.429
# long: max_pict_tasks: Max no. of simultaneously coded pictures: [0, 1..5] 0(default)=auto-642757.43
beamr4.max_pict_tasks: 2-642757.43
# long: aff_masks: CPU affinity masks (default=0x0000000000000000)-642757.43
beamr4.aff_masks: 0-642757.43
# long: er_enable: ER tools: 0(default)=disabled, 1=enabled-642757.43
beamr4.er_enable: 0-642757.43
# long: initial_expected_loss_percent: Initial expected loss rate (percents): [0..99] 0(default)=no ER-642757.43
beamr4.initial_expected_loss_percent: 0-642757.43
# long: intra_update_method: Intra update method: 0=off, 1(default)=adaptive (motion tracking), 2=rolling intra MB rows, 3=random intra MBs-642757.431
beamr4.intra_update_method: 0-642757.431
# long: cir_start_offset: Offset from intra frame for starting cyclic intra refresh: 1=default-642757.431
beamr4.cir_start_offset: 1-642757.431
# long: cir_explicit_period: Explicit period for cyclic intra update: 0(default)=derived automatically from the loss percent-642757.431
beamr4.cir_explicit_period: 0-642757.431
# long: total_intra_update_period: Period for picture full intra update (frames) (valid only if er_enable=1 and initial_expected_loss_percent>0): 0=default-642757.431
beamr4.total_intra_update_period: 0-642757.431
# long: full_motion_update_period: Long update period (fast and slow motion) for adaptive method (frames): 0(default)=disabled, 5, 6..keyframes/2=recommended-642757.432
beamr4.full_motion_update_period: 0-642757.432
# long: fast_motion_update_period: Short update period (fast motion) for adaptive method(frames): 0(default)=disabled, 1, 2, 3=recommended-642757.432
beamr4.fast_motion_update_period: 0-642757.432
# long: aspect_ratio_info_present_flag: Aspect ratio info present flag (default=0)-642757.432
beamr4.aspect_ratio_info_present_flag: 0-642757.432
# long: aspect_ratio_idc: Aspect ratio IDC (default=0): [0..255] 0=not specified, 255=custom-642757.432
beamr4.aspect_ratio_idc: 0-642757.432
# long: sar_width: Custom aspect ratio sar_width (default=0)-642757.432
beamr4.sar_width: 0-642757.433
# long: sar_height: Custom aspect ratio sar_height (default=0)-642757.433
beamr4.sar_height: 0-642757.433
# long: video_signal_type_present_flag: Specifies if video_format, video_full_range_flag and colour_description_present_flag are present (default=0)-642757.433
beamr4.video_signal_type_present_flag: 0-642757.433
# long: video_format: Indicates the representation of the pictures: [0..5] (default=5)-642757.433
beamr4.video_format: 0-642757.433
# long: video_full_range_flag: Indicates the black level and range of the luma and chroma signals as derived from E′Y, E′PB, and E′PR or E′R, E′G and E′B real-valued component signals (default=0)-642757.433
beamr4.video_full_range_flag: 0-642757.434
# long: colour_description_present_flag: Specifies if colour_primaries, transfer_characteristics and matrix_coeffs are present (default=0)-642757.434
beamr4.colour_description_present_flag: 0-642757.434
# long: colour_primaries: Indicates the chromaticity coordinates of the source primaries as specified in Table E-3 of the standard in terms of the CIE 1931 definition of x and y as specified in ISO 11664-1 (default=2)-642757.434
beamr4.colour_primaries: 0-642757.434
# long: transfer_characteristics: Indicates the opto-electronic transfer characteristic of the source picture as specified in Table E-4 of the standard (default=2)-642757.434
beamr4.transfer_characteristics: 0-642757.434
# long: matrix_coefficients: Describes the matrix coefficients used in deriving luma and chroma signals from the green, blue and red or Y, Z and X primaries, as specified in Table E-5 of the standard (default=2)-642757.435
beamr4.matrix_coefficients: 0-642757.435
# long: chroma_loc_info_present_flag: Specified if chroma_sample_loc_top_field and chroma_sample_loc_type_bottom_field are present-642757.435
beamr4.chroma_loc_info_present_flag: 0-642757.435
# long: chroma_sample_loc_type_top_field: Specifies location-642757.435
beamr4.chroma_sample_loc_type_top_field: 0-642757.435
# long: chroma_sample_loc_type_bottom_field: Specifies location-642757.435
beamr4.chroma_sample_loc_type_bottom_field: 0-642757.435
# long: bitstream_restriction_flag-642757.436
beamr4.bitstream_restriction_flag: 0-642757.436
# long: scene_detect: scene change detection threshold (0..100), default 70-642757.436
beamr4.scene_detect: 0-642757.436
Categories
Uncategorized

WSL2 / Vmmem

There is an ongoing issue with vmmem eating up all your memory in windows 10 2004. There is a workaround.

Workaround: Create a %UserProfile%\.wslconfig file in Windows and use it to limit memory assigned to WSL2 VM.

Apostolos
https://github.com/microsoft/WSL/issues/4166#issuecomment-526725261

Below is an example of .wslconfig

[wsl2]
memory=6GB
swap=0
localhostForwarding=true
Categories
Linux

RabbitMQ Memory Limit

RabbitMQ has an memory limit, then RabbitMQ hits this limit weird things starts to happen. Example The .NET library for RabbitMQ Client hangs then you try to dispose the connection.

How do you know that RabbitMQ has run out of memory? you have to check the RabbitMQ node (picture below)

Then Memory is read you are out of memory

RabbitMQ will not allocate all available memory in the VM or container you have set up.

High memory watermark blocks publishers and prevents new messages from being enqueued. Since garbage collection can double the memory used by a queue, it is unsafe to set the high memory watermark above 0.5. The default high memory watermark is set to 0.4 since this is safer as not all memory is used by queues. This is entirely workload specific, which differs across RabbitMQ deployments.

https://www.rabbitmq.com/memory-use.html

A freshly started RabbitMQ uses roughly 100MB, that means that you cannot limit your container to less than 250MB of memory if you want your RabbitMQ to actually be useable

Categories
Linux Network

Test Unix socket

This is a small test/benchmark to se if the use of a unix domain socket in Nginx would affect performance in any significant way for an api

The api is hosted by .NET Core and containerized. Test is done by using apache benchmark. doing 100 concurrent request 50 times. The request that is done has a payload of 1,7 KB.

ab -n 5000 -c 100 "http://localhost/v3/api/"

Result

This is the average request per second after 10 tests against each configuration.

Requests per second
HTTP100
HTTP (Keepalive enabled)118
Unix Socket98
Unix Socket (Keepalive enabled)120

Keepalive

Nginx option keepalive has an impact on both http and unix sockets. It looks to be far more important than the actual connection type.

upstream backend {
        keepalive 100;
        server 10.265.44.3:8080;
}

Categories
.NET

Newtonsoft/System.Text.Json Testing

Microsoft has released NET Core 3 and with that included System.Text.Json a new json serializer to replace Newtonsoft. One of the claims is that This new serializer is faster. There is already multiple benchmarks proving that claim. However I wanted to see what the difference actually became in real life against one of my existing APIs.

End to end testing

I will do an E2E testing using ab (Apache Benchmark) to see if there will be a difference in response time and how many simultaneous request the api can handle. I will test everything 10 times and then average out the value.

To test the the response time i will do one single request with ab and se how long it takes.

ab -n 1 -c 1 "https://localhost:5001/v3/api/"

To test how many request per second i can handle i will use ab to send 100 simultaneous request to the server 5 times.

ab -n 500 -c 100 "https://localhost:5001/v3/"

Test 1

The first test is fetching a single item from the API. The payload is 1,7 KB, the request in is fetching the data IDistributedCache, most of the work is therefor deserialize the data from IDistributedCache and then serialize it again for the response.

Response time 1/reqRequest per second
Newtonsoft14,8 ms98,7 req/s
System.Text.Json 18,4 ms100,2 req/s

Test 2

The second test fetches 48 items from the api, the payload is 36,6 KB. Most of the time is spent on fetching data from the database


Response time 1/req
Request per second
Newtonsoft 129,8 ms26,11 req/s
System.Text.Json 123,5 ms
31,3 req/s

Test 2b

I found the result of test 2 interesting and decided to redo the test but increase the total amount of request from 500 to 5000.

Request per second
Newtonsoft 19,4 req/s
System.Text.Json 30,53 req/s

Conclusion

I am a bit amazed on how big the difference became, for test 2. looking into some other tests there actually might be something behind it. quoting from the comments at The Battle of C# to JSON Serializers in .NET Core 3

Up to the initial buffer size (8-16kb depending on the lib)? Nothing, they all pretty much behave the same, the buffer is filled and after the serialization is done, the buffer is flushed to the output pipe/stream.
After that size it gets interesting. System.Text.Json is capable of flushing the data and reusing the old small buffer, Utf8Json/Spanjson will rent a new buffer from the pool and copy the data and continue.

TORNHOOF (@TORN_HOOF)

As the payload in Test 2 is quit big (36,6 KB), it might explain the increase in throughput compared to test 1.

Categories
.NET

Entity Framework Core 3

I recently upgraded an .NET Core 2 project to .NET Core 3 and with that I also updated other packages, including EF Core. Now EF Core 3 has quite a list of breaking changes you can se them in the list below.

https://docs.microsoft.com/en-us/ef/core/what-is-new/ef-core-3.0/breaking-changes

The worst change for my application was one that is quit a bit down on the list and one that I did not see until I found the actual issue in my code.

Before 3.0, eagerly loading collection navigations via Include operators caused multiple queries to be generated on relational database, one for each related entity type.

Eager loading of related entities now happens in a single query

This does not sound to bad, nothing will stop working, but it will effect performance. If it effects performance depends on your query, some will go faster and some will go slower.

In my case I had an query that did not like this change at all. the query created an list of items. These items then had a bunch of data from other tables, created an 3d dimensional object. I Mocked up an example below on how the Linq Query looked like.

return context.Item
	.Select(i => new ResultItem()
	{
		ItemId   = i.ItemId,
		Siblings = i.parentNavigation.Items.Count(),
		Position = i.parentNavigation.Items.Count(p => p.Order < I.Order),
		tags = i.Tag.Select(t => new ResultTag {
			tag = t.tag
		})
	}).ToList();

In EF Core 2, this worked quite well. EF Core divided the code in two parts. First it did one query to fetch the items, count the siblings and position, and then it would do multiple queries to fetch the tags. So if I fetched 12 items. it would be 1 query to fetch the items and then 12 queries to fetch all the tags.

EF Core 3 however does one query, ONE BIG QUERY. In this case it resulted in some performance drop. In EF Core 2 it had a stable 250ms execution time, independent on how many tags each item had. In EF Core 3 it hovered around 650ms but could be slower if one item had more tags than usual.

Improvement 1

var items = context.Item
	.Select(i => new ResultItem()
	{
		ItemId   = i.ItemId,
                ParentId = i.ParentId,
                Order = i.Order
	}).ToList();

foreach (var item in items)
{
	item.Siblings = context.Item.Count(i => i.ParentId == item.ParentId);
	item.Position = context.Item.Count(i => i.ParentId == item.ParentId && i.Order < item.Order);
	item.tags = context.Tag.Where(t => t.ItemId = item.ItemId).Select(t => new ResultTag {
			tag = t.tag
		}).ToList();
}

First step was to separate the different queries. We simply fetched the item first and then looped over them to fetch the extra data needed. This resulted in quite some improvement, we decreased the time from around 650ms to 350ms. But it was still not fast enough.

The reason it still not as fast as before is because this code results in a lot of queries. If we fetch 12 items, it will result 1 query to fetch the items, then it will loop over all the items and do 3 separate queries for each item. Resulting in a total of 1+36 queries against the database.

Each query means another roundtrip to the database. And this is partly the reason why EF Core decided to make the change in EF core 3, to try to avoid as many roundtrips as possible.

Improvement 2

var items = context.Item
	.Select(i => new ResultItem()
	{
		ItemId   = i.ItemId,
                ParentId = i.ParentId,
                Order = i.Order
	}).ToList();

var itemIds = items.Select(c => c.ItemId);
var tags = context.Tag.Where(t => itemIds.Contains(t.ItemId)).Select(t => new ResultTag {
	ItemId = t.ItemId,
	Tag = t.Tag
}).ToList();

var parentIds = items.Select(i => i.ParentId);
var siblings = context.Item.Where(i => parentIds.Contains(i.parentId)).Select(i => new SiblingItem {
	ParentId = i.ParentId,
	Order = i.Order
}).ToList();

foreach (var item in items)
{
	item.Siblings = siblings.Count(i => i.ParentId == item.ParentItem);
	item.Position = siblings.Count(i => i.ParentId == item.ParentItem && i.Order < item.Order);
	item.tags = tags.Where(t => t.ItemId = item.ItemId);
}

Now we have complicated our code somewhat, but this was the fastest way of doing it. As previous example we first fetch all the items. Our next step is for all items we fetched, select their siblings in on query and select all their tags in another query. We can then loop over the items and filter out the result for each item. This resulted in a total of 3 queries being sent to the database and the execution time dropped to under 100ms.

Categories
Programming

VS Code: Wrap Attributes

Go to File > Settings > Preferences. Expand Extensions and click on HTML. Scroll down until you find the setting Wrap Attribute. set to force-aligned

Enabling this setting will make the formatter to but each html attributes on a new line. You activate the formatter by pressing Shift+Alt+ F