Use docling with spark UDF or pandas UDF but get no parallelization

[qian-yu-db]

Hi,
I wrap a pipeline in a pandas UDF to process a table where the column contains the path to pdf files.

@pandas_udf("string")
def extract_md_from_pdf_docling(pdf_files):
    from docling.document_converter import DocumentConverter, PdfFormatOption
    from docling.datamodel.pipeline_options import PdfPipelineOptions, TableFormerMode
    from docling.datamodel.base_models import InputFormat
    import re

    def extract_md(pdf_file):
        pipeline_options = PdfPipelineOptions()
        pipeline_options.do_table_structure = True
        pipeline_options.table_structure_options.mode = TableFormerMode.ACCURATE
        pipeline_options.images_scale = 2.0
        pipeline_options.generate_picture_images = True

        doc_converter = DocumentConverter(
            format_options={
                InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
            }
        )
        pdf_file = re.sub(r"^dbfs:", "", pdf_file)
        result = doc_converter.convert(pdf_file)
        return result.document.export_to_markdown()
    return pd.Series([extract_md(pdf_file) for pdf_file in pdf_files])

The job however only ran in sequence instead of running in parallel at work nodes. Any recommendations?

[qian-yu-db]

I read more on the technical report. Section 3 stated that "Docling implements a linear pipeline of operations, which execute sequentially on each given document" . Shall I conclude that parallel operations of multiple pipelines for multiple documents are not supported?

Thanks!

[cau-git]

Hi @qian-yu-db, this will entirely depend on how pandas-udf works. If your distribution scheme launches multiple python processes to parallelize processing of documents, then you should benefit. There is no parallelization within a single document, because the pipeline already exploits CPU/GPU parallelism where it can in the AI models, and since python is limited by the GIL otherwise, it makes no sense to parallelize the pipeline within a single document.