Dynamic conversion of HDR10+ SEI to DV P8 NAL

Unsupported/EOL HW/SW Community Projects
1

@quietvoid

Was thinking about dynamic HDR10+ conversion to DV P8 (or P7 MEL)
Was previously using your tools to do the same offline.

Currently CE and Android Kodi are using your lib to allow conversion of DV P7 → P8 (or P7 MEL) and there is code in the BitstreamConverter to identify and remove HDR10+ also.

It strikes me that the processing overhead to convert from one metadata type to the other would not be very high and could be done dynamically - though no expert so maybe not understanding something there.

Elements required as I see it:

  • Extract the HDR10+ SEI - already have the code basics.

  • Combine with Display Mastering Luminance etc. to convert to a DV P8 NAL etc (your code bases also I think has all the elements required - ideally just an additional method to the current lib to take the SEI/metadata and return an RPU! :slight_smile: )

  • Add back the new P8 NAL to the stream - maybe a little more involved to do dynamically?

  • Alter the hints from HDR10 to DV with appropriate stream level metadata.

I think this would be very powerful for the Android version and here in the CE version.

Any thoughts/blockers?, happy to try and do some of this on the dev side - I think though you would have a very strong idea and like to follow your guidance and anything you could help provide to make it straight forward.

thx.
cpm

Putting it out in the forum so others can also opine if think if this is great feature to add or not, if it’s a goer then can add a poll to gauge interest here without lots of replies pilling up.

4 Likes
2

I don’t have many HDR10+ Blu-rays but my Bravia TV only supports DV and HLG so this would be great for me.

3

I think this would be a great feature for users who have a DoVi TV that doesn’t support HDR10+. However, thinking about user impact, I think working to improve DoVi compatibility with PGS subtitles would have a greater impact.

4

Another vote for it being a good idea, particular given hdr10+ content is out there and many tv’s support dv but not hdr10+.

I think this could get interesting in terms of trying to come up with a better way of converting hdr10+ to DV. From this comment the dovi_tool seems the to only set L1 data, not sure if this info is outdated though. The more interesting part in the link seems to be the comments about a bezier curve to adjust mapping and being unable to translate that directly.

5

Yes, lots of LG Oleds out there that don’t support HDR10+. I’d certainly welcome this option.

6

Great idea! I would love to see that option and I will support wirh some testing if needed.

7

As an LG owner who manually converts HDR10+ content to DV before watching it, I think this would be an awesome feature :slight_smile:

8

There are some more options not sure on the processing overhead.

9

Isn’t that still just L1?

edit: quick look at the code the only calculation I’ve found is

let max_pq = (nits_to_pq(max_nits.round()) * 4095.0).round() as u16;
let avg_pq = (nits_to_pq(avg_nits.round()) * 4095.0).round() as u16;

where

pub fn nits_to_pq(nits: f64) -> f64 {
    let y = nits / ST2084_Y_MAX;

    ((ST2084_C1 + ST2084_C2 * y.powf(ST2084_M1)) / (1.0 + ST2084_C3 * y.powf(ST2084_M1)))
        .powf(ST2084_M2)
}
10

Not looked at the conversion code yet, just skim reading the readme lol - presumed if using some histogram distributions may help map into the DV L1 space better - was assuming that was connected to the bezier curve comment, though maybe not! just pointing the options out for now.

Ok, read it better now, just for the max brightness.
Anyway any DV L1 is worth while.

11

The issue with doing this on the fly is that you need HDR10+ parsing in libdovi.
It doesn’t really belong in there though, so I’m not sure how I feel about it.

12

Thanks for getting back :smiley:

As you unquestionable already know, so just talking out loud for others to follow, to separate out the concerns could have a 2nd lib for just conversion keeping the libdovi untouched - at some point something must know how to convert and thus knowledge of both sides, though that is a whole new project and overhead in its own right but could house all the conversion and be used across your tools when converting.

13

All the hdr10+ parsing is already available to use in the linux part of the code. Is it possible to get libdovi into the linux part of the code?

If so, modifiying the c api of libdovi to accept hdr10+ data in a format that isn’t a json file would probably be enough

14

And what does it provide exactly? In what form is the metadata then?

It’s not an option as I don’t want libdovi to depend on the other HDR10+ lib for parsing.
The simplest is creating a new library that builds off of both.

15

A struct with all the data.

Suppose you could also just pass in the raw sei data as well.

But by this point your clearly right to just make a new tool that just uses both your existing hdr10+ and dovi tools.

16

If you know that the struct contains the brightness histogram, we can just extend libdovi to be able to create a new RPU through the C API.
Would need to think of a simple way to expose the current generator options.

17

Not familiar with it enough to know what to look for when you are asking about a histogram. The struct is

hdr10+ struct

struct vframe_hdr_plus_sei {
u16 present_flag;
u16 itu_t_t35_country_code;
u16 itu_t_t35_terminal_provider_code;
u16 itu_t_t35_terminal_provider_oriented_code;
u16 application_identifier;
u16 application_version;
/num_windows max is 3/
u16 num_windows;
/windows xy/
u16 window_upper_left_corner_x[3];
u16 window_upper_left_corner_y[3];
u16 window_lower_right_corner_x[3];
u16 window_lower_right_corner_y[3];
u16 center_of_ellipse_x[3];
u16 center_of_ellipse_y[3];
u16 rotation_angle[3];
u16 semimajor_axis_internal_ellipse[3];
u16 semimajor_axis_external_ellipse[3];
u16 semiminor_axis_external_ellipse[3];
u16 overlap_process_option[3];
/target luminance/
u32 tgt_sys_disp_max_lumi;
u16 tgt_sys_disp_act_pk_lumi_flag;
u16 num_rows_tgt_sys_disp_act_pk_lumi;
u16 num_cols_tgt_sys_disp_act_pk_lumi;
u16 tgt_sys_disp_act_pk_lumi[25][25];

/*num_windows max is 3, e.g maxscl[num_windows][i];*/
u32 maxscl[3][3];
u32 average_maxrgb[3];
u16 num_distribution_maxrgb_percentiles[3];
u16 distribution_maxrgb_percentages[3][15];
u32 distribution_maxrgb_percentiles[3][15];
u16 fraction_bright_pixels[3];

u16 mast_disp_act_pk_lumi_flag;
u16 num_rows_mast_disp_act_pk_lumi;
u16 num_cols_mast_disp_act_pk_lumi;
u16 mast_disp_act_pk_lumi[25][25];
/*num_windows max is 3, e.g knee_point_x[num_windows]*/
u16 tone_mapping_flag[3];
u16 knee_point_x[3];
u16 knee_point_y[3];
u16 num_bezier_curve_anchors[3];
u16 bezier_curve_anchors[3][15];
u16 color_saturation_mapping_flag[3];
u16 color_saturation_weight[3];

};

18

FYI, had a blocker previously with passing HDR10+ into the DV Engine - throws errors unless removing the SEI, saying that existing combined P7 DV/HDR10+ streams seem fine (HDR10+ on the BL and RPU on the EL) so maybe just the presence of the RPU will be enough but seems likely needs to be on the EL for it to work ok (for HDR10+ only this maybe a major pain to create a EL stream for the RPU - so best bet will be to strip the SEI off when adding the RPU - code in place to do this on the Kodi side already).

19

Created a struct based representation of the hdr10+ sei in C++ (Kodi side)

structs based on the hdr10plus_tool rust representation 
struct ProcessingWindow {
  uint16_t window_upper_left_corner_x;
  uint16_t window_upper_left_corner_y;
  uint16_t window_lower_right_corner_x;
  uint16_t window_lower_right_corner_y;

  uint16_t center_of_ellipse_x;
  uint16_t center_of_ellipse_y;
  uint8_t rotation_angle;

  uint16_t semimajor_axis_internal_ellipse;
  uint16_t semimajor_axis_external_ellipse;
  uint16_t semiminor_axis_external_ellipse;

  bool overlap_process_option;
};

struct DistributionMaxRgb {
  uint8_t percentage;
  uint32_t percentile;
};

struct ActualTargetedSystemDisplay {
  uint8_t num_rows_targeted_system_display_actual_peak_luminance;
  uint8_t num_cols_targeted_system_display_actual_peak_luminance;
  std::vector<std::vector<uint8_t>> targeted_system_display_actual_peak_luminance;
};

struct ActualMasteringDisplay {
  uint8_t num_rows_mastering_display_actual_peak_luminance = 0;
  uint8_t num_cols_mastering_display_actual_peak_luminance = 0;
  std::vector<uint8_t> mastering_display_actual_peak_luminance;
};

struct BezierCurve {
  uint16_t knee_point_x = 0;
  uint16_t knee_point_y = 0;
  uint8_t num_bezier_curve_anchors = 0;
  std::vector<uint16_t> bezier_curve_anchors;
};

struct Hdr10PlusMetadata {
  uint8_t itu_t_t35_country_code;
  uint16_t itu_t_t35_terminal_provider_code;
  uint16_t itu_t_t35_terminal_provider_oriented_code;

  uint8_t application_identifier;
  uint8_t application_version;
  uint8_t num_windows;

  std::optional<std::vector<ProcessingWindow>> processing_windows;

  uint32_t targeted_system_display_maximum_luminance;
  bool targeted_system_display_actual_peak_luminance_flag;

  std::optional<ActualTargetedSystemDisplay> actual_targeted_system_display;

  uint32_t maxscl[3];
  uint32_t average_maxrgb;

  uint8_t num_distribution_maxrgb_percentiles;
  std::vector<DistributionMaxRgb> distribution_maxrgb;
  uint16_t fraction_bright_pixels;

  bool mastering_display_actual_peak_luminance_flag;
  std::optional<ActualMasteringDisplay> actual_mastering_display;

  bool tone_mapping_flag;
  std::optional<BezierCurve> bezier_curve;

  bool color_saturation_mapping_flag;
  uint8_t color_saturation_weight;
};
added parsing and logging for the first frame in kodi (CE) 
2024-07-01 14:20:59.130 T:4255     info <general>: CBitstreamConverter::BitstreamConvert First Metadata
2024-07-01 14:20:59.130 T:4255     info <general>: CBitstreamConverter::BitstreamConvert itu_t_t35_country_code [181]
2024-07-01 14:20:59.130 T:4255     info <general>: CBitstreamConverter::BitstreamConvert itu_t_t35_terminal_provider_code [60]
2024-07-01 14:20:59.130 T:4255     info <general>: CBitstreamConverter::BitstreamConvert itu_t_t35_terminal_provider_oriented_code [1]
2024-07-01 14:20:59.130 T:4255     info <general>: CBitstreamConverter::BitstreamConvert application_identifier [4]
2024-07-01 14:20:59.130 T:4255     info <general>: CBitstreamConverter::BitstreamConvert application_version [1]
2024-07-01 14:20:59.130 T:4255     info <general>: CBitstreamConverter::BitstreamConvert num_windows [1]
2024-07-01 14:20:59.130 T:4255     info <general>: CBitstreamConverter::BitstreamConvert targeted_system_display_maximum_luminance [400]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert targeted_system_display_actual_peak_luminance_flag [false]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert maxscl [3704] [3713] [4148]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert average_maxrgb [27]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert num_distribution_maxrgb_percentiles [9]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[0] percentage [1]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[0] percentile [6]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[1] percentage [1]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[1] percentile [6]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[2] percentage [1]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[2] percentile [6]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[3] percentage [1]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[3] percentile [6]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[4] percentage [1]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[4] percentile [6]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[5] percentage [1]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[5] percentile [6]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[6] percentage [1]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[6] percentile [6]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[7] percentage [1]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[7] percentile [6]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[8] percentage [1]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert distribution_maxrgb[8] percentile [6]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert fraction_bright_pixels [8]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert mastering_display_actual_peak_luminance_flag [false]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert tone_mapping_flag [false]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert color_saturation_mapping_flag [false]
2024-07-01 14:20:59.131 T:4255     info <general>: CBitstreamConverter::BitstreamConvert color_saturation_weight [122]

Looks reasonable so far.

20

Any reason you can’t just use the structs from the c api of the hdr10plus_tool from <libhdr10plus-rs/hdr10plus.h> rather than duplicating work?