BPG Specification version 0.9.5 Copyright (c) 2014-2015 Fabrice Bellard 1) Introduction --------------- BPG is a lossy and lossless picture compression format based on HEVC [1]. It supports grayscale, YCbCr, RGB, YCgCo color spaces with an optional alpha channel. CMYK is supported by reusing the alpha channel to encode an additional white component. The bit depth of each component is from 8 to 14 bits. The color values are stored either in full range (JPEG case) or limited range (video case). The YCbCr color space is either BT 601 (JPEG case), BT 709 or BT 2020. The chroma can be subsampled by a factor of two in horizontal or both in horizontal and vertical directions (4:4:4, 4:2:2 or 4:2:0 chroma formats are supported). In order to be able to transcode JPEG images or video frames without modification to the chroma, both JPEG and MPEG2 chroma sample positions are supported. Progressive decoding and display is supported by interleaving the alpha and color data. Arbitrary metadata (such as EXIF, ICC profile, XMP) are supported. Animations are supported as an optional feature. Decoders not supporting animation display the first frame of the animation. 2) Bitstream conventions ------------------------ The bit stream is byte aligned and bit fields are read from most significant to least signficant bit in each byte. - u(n) is an unsigned integer stored on n bits. - ue7(n) is an unsigned integer of at most n bits stored on a variable number of bytes. All the bytes except the last one have a '1' as their first bit. The unsigned integer is represented as the concatenation of the remaining 7 bit codewords. Only the shortest encoding for a given unsigned integer shall be accepted by the decoder (i.e. the first byte is never 0x80). Example: Encoded bytes Unsigned integer value 0x08 8 0x84 0x1e 542 0xac 0xbe 0x17 728855 - ue(v) : unsigned integer 0-th order Exp-Golomb-coded (see HEVC specification). - b(8) is an arbitrary byte. 3) File format -------------- 3.1) Syntax ----------- heic_file() { file_magic u(32) pixel_format u(3) alpha1_flag u(1) bit_depth_minus_8 u(4) color_space u(4) extension_present_flag u(1) alpha2_flag u(1) limited_range_flag u(1) animation_flag u(1) picture_width ue7(32) picture_height ue7(32) picture_data_length ue7(32) if (extension_present_flag) extension_data_length ue7(32) if (extension_present_flag) { extension_data() } hevc_header_and_data() } extension_data() { while (more_bytes()) { extension_tag ue7(32) extension_tag_length ue7(32) if (extension_tag == 5) { animation_control_extension(extension_tag_length) } else { for(j = 0; j < extension_tag_length; j++) { extension_tag_data_byte b(8) } } } } animation_control_extension(payload_length) { loop_count ue7(16) frame_period_num ue7(16) frame_period_den ue7(16) while (more_bytes()) { dummy_byte b(8) } } hevc_header_and_data() { if (alpha1_flag || alpha2_flag) { hevc_header() } hevc_header() hevc_data() } hevc_header() { hevc_header_length ue7(32) log2_min_luma_coding_block_size_minus3 ue(v) log2_diff_max_min_luma_coding_block_size ue(v) log2_min_transform_block_size_minus2 ue(v) log2_diff_max_min_transform_block_size ue(v) max_transform_hierarchy_depth_intra ue(v) sample_adaptive_offset_enabled_flag u(1) pcm_enabled_flag u(1) if (pcm_enabled_flag) { pcm_sample_bit_depth_luma_minus1 u(4) pcm_sample_bit_depth_chroma_minus1 u(4) log2_min_pcm_luma_coding_block_size_minus3 ue(v) log2_diff_max_min_pcm_luma_coding_block_size ue(v) pcm_loop_filter_disabled_flag u(1) } strong_intra_smoothing_enabled_flag u(1) sps_extension_present_flag u(1) if (sps_extension_present_flag) { sps_range_extension_flag u(1) sps_extension_7bits u(7) } if (sps_range_extension_flag) { transform_skip_rotation_enabled_flag u(1) transform_skip_context_enabled_flag u(1) implicit_rdpcm_enabled_flag u(1) explicit_rdpcm_enabled_flag u(1) extended_precision_processing_flag u(1) intra_smoothing_disabled_flag u(1) high_precision_offsets_enabled_flag u(1) persistent_rice_adaptation_enabled_flag u(1) cabac_bypass_alignment_enabled_flag u(1) } trailing_bits u(v) } hevc_data() { for(i = 0; i < v; i++) { hevc_data_byte b(8) } } frame_duration_sei(payloadSize) { frame_duration u(16) } 3.2) Semantics -------------- 'file_magic' is defined as 0x425047fb. 'pixel_format' indicates the chroma subsampling: 0 : Grayscale 1 : 4:2:0. Chroma at position (0.5, 0.5) (JPEG chroma position) 2 : 4:2:2. Chroma at position (0.5, 0) (JPEG chroma position) 3 : 4:4:4 4 : 4:2:0. Chroma at position (0, 0.5) (MPEG2 chroma position) 5 : 4:2:2. Chroma at position (0, 0) (MPEG2 chroma position) The other values are reserved. 'alpha1_flag' and 'alpha2_flag' give information about the alpha plane: alpha1_flag=0 alpha2_flag=0: no alpha plane. alpha1_flag=1 alpha2_flag=0: alpha present. The color is not premultiplied. alpha1_flag=1 alpha2_flag=1: alpha present. The color is premultiplied. The resulting non-premultiplied R', G', B' shall be recovered as: if A != 0 R' = min(R / A, 1), G' = min(G / A, 1), B' = min(B / A, 1) else R' = G' = B' = 1 . alpha1_flag=0 alpha2_flag=1: the alpha plane is present and contains the W color component (CMYK color). The resulting CMYK data can be recovered as follows: C = (1 - R), M = (1 - G), Y = (1 - B), K = (1 - W) . In case no color profile is specified, the sRGB color R'G'B' shall be computed as: R' = R * W, G' = G * W, B' = B * W . 'bit_depth_minus_8' is the number of bits used for each component minus 8. In this version of the specification, bit_depth_minus_8 <= 6. 'extension_present_flag' indicates that extension data are present. 'color_space' specifies how to convert the color planes to RGB. It must be 0 when pixel_format = 0 (grayscale): 0 : YCbCr (BT 601, same as JPEG and HEVC matrix_coeffs = 5) 1 : RGB (component order: G B R) 2 : YCgCo (same as HEVC matrix_coeffs = 8) 3 : YCbCr (BT 709, same as HEVC matrix_coeffs = 1) 4 : YCbCr (BT 2020 non constant luminance system, same as HEVC matrix_coeffs = 9) 5 : reserved for BT 2020 constant luminance system, not supported in this version of the specification. The other values are reserved. YCbCr is defined using the BT 601, BT 709 or BT 2020 conversion matrices. For RGB, G is stored as the Y plane. B in the Cb plane and R in the Cr plane. YCgCo is defined as HEVC matrix_coeffs = 8. Y is stored in the Y plane. Cg in the Cb plane and Co in the Cr plane. If no color profile is present, the RGB output data are assumed to be in the sRGB color space [6]. 'limited_range_flag': opposite of the HEVC video_full_range_flag. The value zero indicates that the full range of each color component is used. The value one indicates that a limited range is used: - (16 << (bit_depth - 8) to (235 << (bit_depth - 8)) for Y and G, B, R, - (16 << (bit_depth - 8) to (240 << (bit_depth - 8)) for Cb and Cr. For the YCgCo color space, the range limitation shall be done on the RGB data. The alpha (or W) plane always uses the full range. 'animation_flag'. The value '1' indicates that more than one frame are encoded in the hevc data. The animation control extension must be present. If the decoder does not support animations, it shall decode the first frame only and ignore the animation information. 'picture_width' is the picture width in pixels. The value 0 is not allowed. 'picture_height' is the picture height in pixels. The value 0 is not allowed. 'picture_data_length' is the picture data length in bytes. The special value of zero indicates that the picture data goes up to the end of the file. 'extension_data_length' is the extension data length in bytes. 'extension_data()' is the extension data. 'extension_tag' is the extension tag. The following values are defined: 1: EXIF data. 2: ICC profile (see [4]) 3: XMP (see [5]) 4: Thumbnail (the thumbnail shall be a lower resolution version of the image and stored in BPG format). 5: Animation control data. The decoder shall ignore the tags it does not support. 'extension_tag_length' is the length in bytes of the extension tag. 'loop_count' gives the number of times the animation shall be played. The value of 0 means infinite. 'frame_period_num' and 'frame_period_den' encode the default delay between each frame as frame_period_num/frame_period_den seconds. The value of 0 for 'frame_period_num' or 'frame_period_den' is forbidden. 'hevc_header_length' is the length in bytes of the following data up to and including 'trailing_bits'. 'log2_min_luma_coding_block_size_minus3', 'log2_diff_max_min_luma_coding_block_size', 'log2_min_transform_block_size_minus2', 'log2_diff_max_min_transform_block_size', 'max_transform_hierarchy_depth_intra', 'sample_adaptive_offset_enabled_flag', 'pcm_enabled_flag', 'pcm_sample_bit_depth_luma_minus1', 'pcm_sample_bit_depth_chroma_minus1', 'log2_min_pcm_luma_coding_block_size_minus3', 'log2_diff_max_min_pcm_luma_coding_block_size', 'pcm_loop_filter_disabled_flag', 'strong_intra_smoothing_enabled_flag', 'sps_extension_flag' 'sps_extension_present_flag', 'sps_range_extension_flag' 'transform_skip_rotation_enabled_flag', 'transform_skip_context_enabled_flag', 'implicit_rdpcm_enabled_flag', 'explicit_rdpcm_enabled_flag', 'extended_precision_processing_flag', 'intra_smoothing_disabled_flag', 'high_precision_offsets_enabled_flag', 'persistent_rice_adaptation_enabled_flag', 'cabac_bypass_alignment_enabled_flag' are the corresponding fields of the HEVC SPS syntax element. 'trailing_bits' has a value of 0 and has a length from 0 to 7 bits so that the next data is byte aligned. 'hevc_data()' contains the corresponding HEVC picture data, excluding the first NAL start code (i.e. the first 0x00 0x00 0x01 or 0x00 0x00 0x00 0x01 bytes). The VPS and SPS NALs shall not be included in the HEVC picture data. The decoder can recover the necessary fields from the header by doing the following assumptions: - vps_video_parameter_set_id = 0 - sps_video_parameter_set_id = 0 - sps_max_sub_layers = 1 - sps_seq_parameter_set_id = 0 - chroma_format_idc: for picture data: chroma_format_idc = pixel_format for alpha data: chroma_format_idc = 0. - separate_colour_plane_flag = 0 - pic_width_in_luma_samples = ceil(picture_width/cb_size) * cb_size - pic_height_in_luma_samples = ceil(picture_height/cb_size) * cb_size with cb_size = 1 << log2_min_luma_coding_block_size - bit_depth_luma_minus8 = bit_depth_minus_8 - bit_depth_chroma_minus8 = bit_depth_minus_8 - max_transform_hierarchy_depth_inter = max_transform_hierarchy_depth_intra - scaling_list_enabled_flag = 0 - log2_max_pic_order_cnt_lsb_minus4 = 4 - amp_enabled_flag = 1 - sps_temporal_mvp_enabled_flag = 1 Alpha data encoding: - If alpha data is present, all the corresponding NALs have nuh_layer_id = 1. NALs for color data shall have nuh_layer_id = 0. - Alpha data shall use the same tile sizes as color data and shall have the same entropy_coding_sync_enabled_flag value as color data. - Alpha slices shall use the same number of coding units as color slices and should be interleaved with color slices. alpha NALs shall come before the corresponding color NALs. Animation encoding: - The optional prefix SEI with payloadType = 257 (defined in frame_duration_sei()) specifies that the image must be repeated 'frame_duration' times. 'frame_duration' shall not be zero. If the frame duration SEI is not present for a given frame, frame_duration = 1 shall be assumed by the decoder. If alpha data is present, the frame duration SEI shall be present only for the color data. 3.3) HEVC Profile ----------------- Conforming HEVC bit streams shall conform to the Main 4:4:4 16 Still Picture, Level 8.5 of the HEVC specification with the following modifications. - separate_colour_plane_flag shall be 0 when present. - bit_depth_luma_minus8 <= 6 - bit_depth_chroma_minus8 = bit_depth_luma_minus8 - explicit_rdpcm_enabled_flag = 0 (does not matter for intra frames) - extended_precision_processing_flag = 0 - cabac_bypass_alignment_enabled_flag = 0 - high_precision_offsets_enabled_flag = 0 (does not matter for intra frames) - If the encoded image is larger than the size indicated by picture_width and picture_height, the lower right part of the decoded image shall be cropped. If a horizontal (resp. vertical) decimation by two is done for the chroma and that the width (resp. height) is n pixels, ceil(n/2) pixels must be kept as the resulting chroma information. When animations are present, the next frames shall be encoded with the following changes: - P slices are allowed (but B slices are not allowed). - Only the previous picture can be used as reference (hence a DPB size of 2 pictures). 4) Design choices ----------------- (This section is informative) - Our design principle was to keep the format as simple as possible while taking the HEVC codec as basis. Our main metric to evaluate the simplicity was the size of a software decoder which outputs 32 bit RGBA pixel data. - Pixel formats: we wanted to be able to convert JPEG images to BPG with as little loss as possible. So supporting the same color space (BT 601 YCbCr) with the same range (full range) and most of the allowed JPEG chroma formats (4:4:4, 4:2:2, 4:2:0 or grayscale) was mandatory to avoid going back to RGB or doing a subsampling or interpolation. - Alpha support: alpha support is mandatory. We chose to use a separate HEVC monochrome plane to handle it instead of another format to simplify the decoder. The color is either non-premultiplied or premultiplied. Premultiplied alpha usually gives a better compression. Non-premultiplied alpha is supported in case no loss is needed on the color components. In order to allow progressive display, the alpha and color data are interleaved (the nuh_layed_id NAL field is 0 for color data and 1 for alpha data). The alpha and color slices should contain the same number of coding units and each alpha slice should come before the corresponding color slice. Since alpha slices are usually smaller than color slices, it allows a progressive display even if there is a single slice. - Color spaces: In addition to YCbCr, RGB is supported for the high quality or lossless cases. YCgCo is supported because it may give slightly better results than YCbCr for high quality images. CMYK is supported so that JPEGs containing this color space can be converted. The alpha plane is used to store the W (1-K) plane. The data is stored with inverted components (1-X) so that the conversion to RGB is simplified. The support of the BT 709 and BT 2020 (non constant luminance) YCbCr encodings and of the limited range color values were added to reduce the losses when converting video frames. - Bit depth: we decided to support the HEVC bit depths 8 to 14. The added complexity is small and it allows to support high quality pictures from cameras. - Picture file format: keeping a completely standard HEVC stream would have meant a more difficult parsing for the picture header which is a problem for the various image utilities to get the basic picture information (pixel format, width, height). So we added a small header before the HEVC bit stream. The picture header is byte oriended so it is easy to parse. - HEVC bit stream: the standard HEVC headers (the VPS and SPS NALs) give an overhead of about 60 bytes for no added value in the case of picture compression. Since the alpha plane uses a different HEVC bit stream, it also adds the same overhead again. So we removed the VPS and SPS NALs and added a very small header with the equivalent information (typically 4 bytes). We also removed the first NAL start code which is not useful. It is still possible to reconstruct a standard HEVC stream to feed an unmodified hardware decoder if needed. - Extensions: the metadata are stored at the beginning of the file so that they can be read at the same time as the header. Since metadata tend to evolve faster than the image formats, we left room for extension by using a (tag, lengh) representation. The decoder can easily skip all the metadata because their length is explicitly stored in the image header. - Animations: they are interesting compared to WebM or MP4 short videos for the following reasons: * transparency is supported * lossless encoding is supported * the decoding resources are smaller than with a generic video player because only two frames need to be stored (DPB size = 2). * the animations are expected to be small so the decoder can cache all the decoded frames in memory. * the animation can be decoded as a still image if the decoder does not support animations. Compared to the other animated image formats (GIF, APNG, WebP), the compression ratio is usually much higher because of the HEVC inter frame prediction. 5) References ------------- [1] High efficiency video coding (HEVC) version 2 (ITU-T Recommendation H.265) [2] JPEG File Interchange Format version 1.02 ( http://www.w3.org/Graphics/JPEG/jfif3.pdf ) [3] EXIF version 2.2 (JEITA CP-3451) [4] The International Color Consortium ( http://www.color.org/ ) [5] Extensible Metadata Platform (XMP) http://www.adobe.com/devnet/xmp.html [6] sRGB color space, IEC 61966-2-1