|-js 6ddlZddlmZddlmZddlmZddlmZddl Z ddl m Z ddl mZmZdd lmZmZmZdd lmZmZmZmZmZmZdd lmZmZdd lmZm Z m!Z!m"Z"e!rdd lm#Z#e rddl$m%Z&e"j'e(Z)ddgZ*GddeZ+ddhdhddhdhddhfZ,dZ- d&de.ee.zdzde.dzde/de/de0e1e.ff d Z2 d'de.ee.ze0e1e.fzezdzde.dzde/de/de0f d!Z3d"e4d#e5de4fd$Z6d%Z7dS)(N)Iterable)deepcopy)partial)Any)validate_typed_dict) BatchFeatureImageProcessingMixin) center_crop normalizerescale)ChannelDimension ImageInputSizeDictget_image_sizemake_flat_list_of_imagesvalidate_preprocess_arguments) ImagesKwargsUnpack)auto_docstringis_torchvision_availableis_vision_availablelogging)PILImageResampling) functionalprocessor_classimage_processor_typeceZdZdZeZdZdZdgZde effd Z dZ de de ed e fd Zd Zd Z d.de ded e fdZd dde dede ed eefdZde de ed e fdZ d/deeezeeefzezdzdeeezeeefzezdzdeeezeeefzezdzdedzdeeezdzdeeezdzd efdZeZ d0dedzdedzdedzdeeezdzdeeezdzdedzdedzdedzdedzd d!fd"Z e!de de ed e fd#Z"d eeefffd$ Z# d1d%e$j%d&ed'ee&zdzd(ee&zdzd e$j%f d)Z' d1d%e$j%d*eeezd+eeezd'ee&zdzd(ee&zdzd e$j%f d,Z( d1d%e$j%deeefd'ee&zdzd(ee&zdzd e$j%f d-Z)xZ*S)2BaseImageProcessorue Base class for image processors with an inheritance-based backend architecture. This class defines the preprocessing pipeline: kwargs validation, input preparation, and dispatching to the backend's `_preprocess` method. Backend subclasses (`TorchvisionBackend`, `PilBackend`) inherit from this class and implement the actual image operations (resize, crop, rescale, normalize, etc.). Model-specific image processors then inherit from the appropriate backend class. Architecture Overview --------------------- The class hierarchy is: BaseImageProcessor (this class) ├── TorchvisionBackend (GPU-accelerated, torch.Tensor) │ └── ModelImageProcessor (e.g. LlavaNextImageProcessor) └── PilBackend (portable CPU, np.ndarray) └── ModelImageProcessorPil (e.g. CLIPImageProcessorPil) The preprocessing flow is: __call__() → preprocess() → _preprocess_image_like_inputs() → _prepare_image_like_inputs() (calls process_image per image) → _preprocess() (batch operations: resize, crop, etc.) - `process_image`: Implemented by backends. Converts a single raw input (PIL, NumPy, or Tensor) to the backend's working format (torch.Tensor or np.ndarray), handles RGB conversion and channel reordering. - `_preprocess`: Implemented by backends. Performs the actual batch processing (resize, center crop, rescale, normalize, pad) and returns a `BatchFeature`. Basic Implementation -------------------- For processors that only need standard operations (resize, center crop, rescale, normalize), inherit from a backend and define class attributes: from transformers.image_processing_backends import PilBackend class MyImageProcessorPil(PilBackend): resample = PILImageResampling.BILINEAR image_mean = IMAGENET_DEFAULT_MEAN image_std = IMAGENET_DEFAULT_STD size = {"height": 224, "width": 224} do_resize = True do_rescale = True do_normalize = True The backend's `_preprocess` method handles the standard pipeline automatically. Custom Processing ----------------- For processors that need custom logic (e.g., patch-based processing, multiple input types), override `_preprocess` in your model-specific processor. The `_preprocess` method receives already-prepared images (converted to the backend format with channels-first ordering) and performs the actual processing: class MyImageProcessor(TorchvisionBackend): def _preprocess(self, images, do_resize, size, do_normalize, image_mean, image_std, **kwargs): # Group images by shape for efficient batched operations grouped_images, grouped_images_index = group_images_by_shape(images) processed_groups = {} for shape, stacked_images in grouped_images.items(): if do_resize: stacked_images = self.resize(stacked_images, size=size) if do_normalize: stacked_images = self.normalize(stacked_images, mean=image_mean, std=image_std) processed_groups[shape] = stacked_images processed_images = reorder_images(processed_groups, grouped_images_index) return BatchFeature(data={"pixel_values": processed_images}) For processors handling multiple input types (e.g., images + segmentation maps), override `_preprocess_image_like_inputs`: def _preprocess_image_like_inputs( self, images: ImageInput, segmentation_maps: ImageInput | None = None, **kwargs, ) -> BatchFeature: images = self._prepare_image_like_inputs(images, **kwargs) batch_feature = self._preprocess(images, **kwargs) if segmentation_maps is not None: maps = self._prepare_image_like_inputs(segmentation_maps, **kwargs) batch_feature["labels"] = self._preprocess(maps, **kwargs).pixel_values return batch_feature Extending Backend Behavior -------------------------- To customize operations for a specific backend, subclass the backend and override its methods: from transformers.image_processing_backends import TorchvisionBackend, PilBackend class MyTorchvisionProcessor(TorchvisionBackend): def resize(self, image, size, **kwargs): # Custom resize logic for torchvision return super().resize(image, size, **kwargs) class MyPilProcessor(PilBackend): def resize(self, image, size, **kwargs): # Custom resize logic for PIL return super().resize(image, size, **kwargs) Custom Parameters ----------------- To add parameters beyond `ImagesKwargs`, create a custom kwargs class and set it as `valid_kwargs`: class MyImageProcessorKwargs(ImagesKwargs): custom_param: int | None = None class MyImageProcessor(TorchvisionBackend): valid_kwargs = MyImageProcessorKwargs custom_param = 10 # default value Key Notes --------- - Backend selection is done at the class level: inherit from `TorchvisionBackend` or `PilBackend` - Backends receive images as `torch.Tensor` (Torchvision) or `np.ndarray` (PIL), always channels-first - All images have channel dimension first during processing, regardless of backend - Arguments not provided by users default to class attribute values - Backend classes encapsulate backend-specific logic (resize, normalize, etc.) and can be overridden Tgp? pixel_valueskwargsc :tjdi|dS)N)super__init__)selfr! __class__s c/var/www/html/banglarbhumi/venv/lib/python3.11/site-packages/transformers/image_processing_utils.pyr%zBaseImageProcessor.__init__s&""6"""""c vi}|jjD]A}||d}||||< tt ||d||<B|jdi|}|D]\}}t|||t|jj |_ dS)z^Resolve and set instance attributes from kwargs and class-level defaults for all valid kwargs.Nr#) valid_kwargs__annotations__poprgetattr_standardize_kwargsitemssetattrlistkeys_valid_kwargs_names)r&r! attributeskeykwargvalues r(_set_attributesz"BaseImageProcessor._set_attributess $4 E ECJJsD))E "' 3"*74d+C+C"D"D 3-T-;; ;; $**,, & &JC D#u % % % %#'(9(I(N(N(P(P#Q#Q   r)imagesreturnc$|j|g|Ri|S)z)Preprocess an image or a batch of images.) preprocessr&r:argsr!s r(__call__zBaseImageProcessor.__call__s$tv7777777r)ct)a Process a single raw image into the backend's working format. Implemented by backend subclasses (`TorchvisionBackend`, `PilBackend`). Converts a raw input (PIL Image, NumPy array, or torch Tensor) to the backend's internal format (`torch.Tensor` for Torchvision, `np.ndarray` for PIL), handles RGB conversion and ensures channels-first ordering. NotImplementedErrorr&r?r!s r( process_imagez BaseImageProcessor.process_images "!r)ct)a- Perform the actual batch image preprocessing (resize, center crop, rescale, normalize, pad). Implemented by backend subclasses (`TorchvisionBackend`, `PilBackend`). Receives a list of already-prepared images (in the backend's format, channels-first) and applies the configured preprocessing operations. Returns a `BatchFeature` with the processed pixel values. Model-specific processors can override this method to implement custom preprocessing logic (e.g., patch-based processing in LLaVA-NeXT). rBrDs r( _preprocesszBaseImageProcessor._preprocesss "!r)expected_ndimscN||}t||S)z Prepare the images structure for processing. Args: images (`ImageInput`): The input images to process. Returns: `ImageInput`: The images with a valid nesting. rI) fetch_imagesr)r&r:rIs r(_prepare_images_structurez,BaseImageProcessor._prepare_images_structures*""6**'~NNNNr)rKc |||}t|jg|Ri|t|dko"t |dt t z}|rfd|D}nfd|D}|S)a Prepare image-like inputs for processing by converting each image via `process_image`. Flattens the input structure and applies `process_image` (implemented by the backend) to each individual image, converting raw inputs (PIL, NumPy, Tensor) into the backend's working format with channels-first ordering. Args: images (`ImageInput`): The image-like inputs to process. expected_ndims (`int`, *optional*, defaults to 3): The expected number of dimensions for the images. Returns: `list[torch.Tensor]` or `list[np.ndarray]`: The prepared images in the backend's format, with channels-first ordering. rKrc,g|]}fd|DS)c&g|] }|Sr#r#.0imgprocess_image_partials r( zLBaseImageProcessor._prepare_image_like_inputs...#s% S S S!6!6s!;!; S S Sr)r#)rR nested_listrTs r(rUzABaseImageProcessor._prepare_image_like_inputs..#s/nnnXc S S S S{ S S Snnnr)c&g|] }|Sr#r#rQs r(rUzABaseImageProcessor._prepare_image_like_inputs..%s%MMMs 5 5c : :MMMr))rMrrElen isinstancer2tuple)r&r:rIr?r!has_nested_structureprocessed_imagesrTs @r(_prepare_image_like_inputsz-BaseImageProcessor._prepare_image_like_inputss0//~/VV '(: LT L L LV L L"6{{QV:fQiPU3V3V  Nnnnngmnnn  MMMMfMMM r)c@|j|fi|}|j|g|Ri|S)a Preprocess image-like inputs by preparing them and dispatching to `_preprocess`. This method first calls `_prepare_image_like_inputs` to convert raw inputs into the backend's format, then calls `_preprocess` for the actual batch processing. Override this method in model-specific processors that need to handle multiple image-like input types (e.g., images and segmentation maps) or need custom orchestration of the preprocessing pipeline. )r]rGr>s r(_preprocess_image_like_inputsz0BaseImageProcessor._preprocess_image_like_inputs)sA10BB6BBt8888888r)Nsize crop_sizepad_sizedefault_to_square image_mean image_stdc |i}|0t|tstd it||}|0t|tstd it|d}|0t|tstd it|d}t|trt |}t|trt |}||d<||d<||d<||d<||d <|S) z Standardize kwargs to canonical format before validation. Can be overridden by subclasses to customize the processing of kwargs. N)r`rcra) param_namerb)r`rgr`rdrer#)rYr get_size_dictr2rZ)r&r`rarbrcrdrer!s r(r/z&BaseImageProcessor._standardize_kwargs:s" >F  JtX$>$> \\mIZ[[[\\D  Ix)H)H  TT={#S#S#STTI   8X(F(F VV-X*"U"U"UVVH j$ ' ' +z**J i & & )i((Iv'{%z)|'{ r) do_rescalerescale_factor do_normalize do_resizedo_center_cropresamplez7PILImageResampling | tvF.InterpolationMode | int | Nonec 8t||||||| |||  dS)z@ Validate the kwargs for the preprocess method. ) rirjrkrdrermrarlr`rnN)r) r&rirjrkrdrerlr`rmrarnr!s r(_validate_preprocess_kwargsz.BaseImageProcessor._validate_preprocess_kwargs`sC" &!)%!) r)c t|j||jD]'}||t ||d(|jdi|}|jdi||j|g|Ri|S)z; Preprocess an image or a batch of images. Nr#)rr+r4 setdefaultr.r/rpr_)r&r:r?r! kwarg_names r(r=zBaseImageProcessor.preprocess~s D-v6662 K KJ   j'$ D*I*I J J J J*)33F33 )(2262221t1&J4JJJ6JJJr)ct}i}|D]\\}}t|trt |}|,t t||d}|dkr||||<W|||<]|dd|dd|S)N NOT_FOUND_valid_processor_keysr4) r$to_dictr0rYrdictr.typer-)r&processor_dict filtered_dictr6r8 class_defaultr's r(rwzBaseImageProcessor.to_dicts** (..00 + +JC%** $U } 'T C E E  K//M4M).M#&%* c""14888/666r)imagescale data_formatinput_data_formatc $t|f|||d|S)a Rescale an image by a scale factor. image = image * scale. Args: image (`np.ndarray`): Image to rescale. scale (`float`): The scaling factor to rescale pixel values by. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. Returns: `np.ndarray`: The rescaled image. )r~rr)r )r&r}r~rrr!s r(r zBaseImageProcessor.rescales%<urE{Vgrrkqrrrr)meanstdc &t|f||||d|S)aZ Normalize an image. image = (image - image_mean) / image_std. Args: image (`np.ndarray`): Image to normalize. mean (`float` or `Iterable[float]`): Image mean to use for normalization. std (`float` or `Iterable[float]`): Image standard deviation to use for normalization. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. Returns: `np.ndarray`: The normalized image. )rrrr)r )r&r}rrrrr!s r(r zBaseImageProcessor.normalizes6B  #;Rc  gm   r)c t|}d|vsd|vr$td|t|f|d|df||d|S)a  Center crop an image to `(size["height"], size["width"])`. If the input size is smaller than `crop_size` along any edge, the image is padded with 0's and then center cropped. Args: image (`np.ndarray`): Image to center crop. size (`dict[str, int]`): Size of the output image. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. heightwidthz=The size dictionary must have keys 'height' and 'width'. Got )r`rr)rh ValueErrorr3r )r&r}r`rrr!s r(r zBaseImageProcessor.center_crops8T"" 4  7$#6#6j]a]f]f]h]hjjkk k  x.$w-0#/       r))rH)NNNNNN) NNNNNNNNNN)NN)+__name__ __module__ __qualname____doc__rr+rcrjmodel_input_namesrr%r9rr r@rErGintrMr2rr]r_rrxstrrboolfloatr/_further_process_kwargsrZrprr=rwnpndarrayrr r r __classcell__)r's@r(rr<sJ~~@ LN'(# !5###### R R R8z8F<RKWcKKK^K&c3h06:;? sszss++d2 s !11D8 s ssssL6:;? # # z# huo%# Xe_ $ # ++d2 # !11D8 #  # # # # R6:;? % % z% 38n% ++d2 % !11D8 %  % % % % % % % % r)rrr shortest_edge longest_edge max_height max_widthct|tsdSt|}tD] }||krdS dS)NFT)rYrxsetr3VALID_SIZE_DICT_KEYS) size_dictsize_dict_keys allowed_keyss r(is_valid_size_dictrs[ i & &u))**N, \ ) )44 * 5r)Tr`max_sizercheight_width_orderr;ct|tr|r|td||dSt|tr|s d|i}|||d<|St|ttfr|r|d|ddSt|ttfr|s|d|ddS|||rtdd|iStd|) NzLCannot specify both size as an int, with default_to_square=True and max_size)rrrrrrz7Cannot specify both default_to_square=True and max_sizez+Could not convert size input to size dict: )rYrrrZr2)r`rrcrrs r(convert_to_size_dictr(s)$*!2*  kll l... D#   *'8 *$d+  (0In % D5$- ( (*-?*q'DG444 D5$- ( (*1C*q'DG444 (.  XVWW W)) I4II J JJr)c t|ttzs>t||||}t|dt d|d|dn't|trt|}n|}t|s.t|dt d| |S)a Converts the old size parameter in the config into the new dict expected in the config. This is to ensure backwards compatibility with the old image processor configs and removes ambiguity over whether the tuple is in (height, width) or (width, height) format. - If `size` is tuple, it is converted to `{"height": size[0], "width": size[1]}` or `{"height": size[1], "width": size[0]}` if `height_width_order` is `False`. - If `size` is an int, and `default_to_square` is `True`, it is converted to `{"height": size, "width": size}`. - If `size` is an int and `default_to_square` is False, it is converted to `{"shortest_edge": size}`. If `max_size` is set, it is added to the dict as `{"longest_edge": max_size}`. - If `size` is `None` and `default_to_square` is False, the result is `{"longest_edge": max_size}` (requires `max_size` to be set). Tuple/list/SizeDict/dict `size` values do not use `max_size`. Args: size (`int | Iterable[int] | dict[str, int] | SizeDict`, *optional*): The `size` parameter to be cast into a size dictionary. max_size (`int | None`, *optional*): With `default_to_square=False`, sets `longest_edge` when `size` is an int or `None`; unused for dict, `SizeDict`, or tuple/list `size`. Raises if set with `default_to_square=True` when `size` is an int or `None`. height_width_order (`bool`, *optional*, defaults to `True`): If `size` is a tuple, whether it's in (height, width) or (width, height) order. default_to_square (`bool`, *optional*, defaults to `True`): If `size` is an int, whether to default to a square image or not. z@ should be a dictionary with one of the following sets of keys: z, got z. Converted to .z- must have one of the following set of keys: ) rYrxrrloggerinforrrr3)r`rrrcrgrs r(rhrhGs> dD8O , , (x9JL^__   * *[o * *w{ * *& * * *    D( # #JJ  i ( (  v vH\ v vdmdrdrdtdt v v    r) original_sizepossible_resolutionsc*|\}}d}d}td}|D]w\}}t||z ||z } t|| zt|| z} } t| | z||z} ||z| z } | |ks | |kr| |kr| }| }||f}x|S)a Selects the best resolution from a list of possible resolutions based on the original size. This is done by calculating the effective and wasted resolution for each possible resolution. The best fit resolution is the one that maximizes the effective resolution and minimizes the wasted resolution. Args: original_size (tuple): The original size of the image in the format (height, width). possible_resolutions (list): A list of possible resolutions in the format [(height1, width1), (height2, width2), ...]. Returns: tuple: The best fit resolution in the format (height, width). Nrinf)rminr)rroriginal_heightoriginal_widthbest_fitmax_effective_resolutionmin_wasted_resolutionrrr~downscaled_widthdownscaled_heighteffective_resolutionwasted_resolutions r(select_best_resolutionrys"'4#O^H !%LL- ' ' EN*F_,DEE.1.52H.I.I3afOfKgKg+"#36G#GZiIijj"V^/CC ": : : $< < rsn $$$$$$;;;;;;EEEEEEEE==========322222220//////<;;;;;;  H % % U U U U U -U U U rwn%; (,"# KK   $KDjKK K  #s(^ KKKK@DH#" //  S#X . 9D @/Dj// /  ////d#%#t#PU####L!!!!!r)