我有一个包含表格、文本和一些图片的PDF文件。我想要提取PDF中所有的表格。
当前，我需要手动在页面上寻找表格。然后，我会将该页面捕获并保存到另一个PDF中。
import pypdf import PdfReader, PdfWriter
filename = "Sammamish.pdf"
reader = PdfReader(filename)
page = reader.pages[126]
writer = PdfWriter()
writer.add_page(page)
new_filename = "allTables.pdf"
with open(new_filename, "wb") as output_stream:
writer.write(output_stream)
我的目标是从整个PDF文档中提取表格。
对于遇到包含图像的pdf且需要使用OCR的任何人，本答案提供解决方案。我没有找到可行的现成解决方案，没有给我所需的准确性。
以下是我发现有效的步骤。
使用https://poppler.freedesktop.org/中的pdfimages将pdf的页面转换为图像。
使用Tesseract进行旋转检测，并使用ImageMagick mogrify进行修复。
使用OpenCV查找并提取表格。
使用OpenCV从表格中查找并提取每个单元格。
使用OpenCV裁剪和清除每个单元格，以使OCR软件不会混淆。
使用Tesseract对每个单元格进行OCR。
将每个单元格的提取文本合并到您需要的格式中。
我编写了一个Python软件包，其中包含可以帮助完成这些步骤的模块。
Repo: https://github.com/eihli/image-table-ocr
Docs & Source: https://eihli.github.io/image-table-ocr/pdf_table_extraction_and_ocr.html
其中一些步骤不需要代码，它们利用外部工具如pdfimages和tesseract。我将为那些需要代码的步骤提供一些简要示例。
在找到表格的方法时，此链接是一个很好的参考。https://answers.opencv.org/question/63847/how-to-extract-tables-from-an-image/
import cv2
def find_tables(image):
BLUR_KERNEL_SIZE = (17, 17)
STD_DEV_X_DIRECTION = 0
STD_DEV_Y_DIRECTION = 0
blurred = cv2.GaussianBlur(image, BLUR_KERNEL_SIZE, STD_DEV_X_DIRECTION, STD_DEV_Y_DIRECTION)
MAX_COLOR_VAL = 255
BLOCK_SIZE = 15
SUBTRACT_FROM_MEAN = -2
img_bin = cv2.adaptiveThreshold(
~blurred,
MAX_COLOR_VAL,
cv2.ADAPTIVE_THRESH_MEAN_C,
cv2.THRESH_BINARY,
BLOCK_SIZE,
SUBTRACT_FROM_MEAN,
)
vertical = horizontal = img_bin.copy()
SCALE = 5
image_width, image_height = horizontal.shape
horizontal_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (int(image_width / SCALE), 1))
horizontally_opened = cv2.morphologyEx(img_bin, cv2.MORPH_OPEN, horizontal_kernel)
vertical_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1, int(image_height / SCALE)))
vertically_opened = cv2.morphologyEx(img_bin, cv2.MORPH_OPEN, vertical_kernel)
horizontally_dilated = cv2.dilate(horizontally_opened, cv2.getStructuringElement(cv2.MORPH_RECT, (40, 1)))
vertically_dilated = cv2.dilate(vertically_opened, cv2.getStructuringElement(cv2.MORPH_RECT, (1, 60)))
mask = horizontally_dilated + vertically_dilated
contours, hierarchy = cv2.findContours(
mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE,
)
MIN_TABLE_AREA = 1e5
contours = [c for c in contours if cv2.contourArea(c) > MIN_TABLE_AREA]
perimeter_lengths = [cv2.arcLength(c, True) for c in contours]
epsilons = [0.1 * p for p in perimeter_lengths]
approx_polys = [cv2.approxPolyDP(c, e, True) for c, e in zip(contours, epsilons)]
bounding_rects = [cv2.boundingRect(a) for a in approx_polys]
# The link where a lot of this code was borrowed from recommends an
# additional step to check the number of "joints" inside this bounding rectangle.
# A table should have a lot of intersections. We might have a rectangular image
# here though which would only have 4 intersections, 1 at each corner.
# Leaving that step as a future TODO if it is ever necessary.
images = [image[y:y+h, x:x+w] for x, y, w, h in bounding_rects]
return images
这与第2个问题非常相似，因此我不会包含所有的代码。我参考的部分将在单元格排序中。
我们想要从左到右、从上到下地识别单元格。
我们将找到具有最左上角的矩形。然后，我们将查找所有中心点位于该左上矩形的顶部y坐标和底部y坐标之间的矩形。然后，我们将按其中心点的x值对这些矩形进行排序。我们将从列表中删除这些矩形并重复这个过程。
def cell_in_same_row(c1, c2):
c1_center = c1[1] + c1[3] - c1[3] / 2
c2_bottom = c2[1] + c2[3]
c2_top = c2[1]
return c2_top < c1_center < c2_bottom
orig_cells = [c for c in cells]
rows = []
while cells:
first = cells[0]
rest = cells[1:]
cells_in_same_row = sorted(
[
c for c in rest
if cell_in_same_row(c, first)
],
key=lambda c: c[0]
)
row_cells = sorted([first] + cells_in_same_row, key=lambda c: c[0])
rows.append(row_cells)
cells = [
c for c in rest
if not cell_in_same_row(c, first)
]
# Sort rows by average height of their center.
def avg_height_of_center(row):
centers = [y + h - h / 2 for x, y, w, h in row]
return sum(centers) / len(centers)
rows.sort(key=avg_height_of_center)
这是我提取pdf的代码。
import pandas as pd
import tabula
file = "filename.pdf"
path = 'enter your directory path here' + file
df = tabula.read_pdf(path, pages = '1', multiple_tables = True)
print(df)
请参考我的repo获取更多详细信息。
amazon-textract-textractor包的代码片段：from textractor import Textractor
from textractor.data.constants import TextractFeatures
extractor = Textractor(profile_name="default")
document = extractor.analyze_document(
file_source="./0kWSg.png",
features=[TextractFeatures.TABLES],
)
document.visualize(with_words=False)
然后通过一些pandas操作
df = document.tables[0][1:,:].to_pandas()
df[0] = df[0]+' '+df[1]
df= df.drop(columns=1)
df.columns = df.iloc[0]
df = df[1:]
df
camelot-py。import camelot
tables = camelot.read_pdf('foo.pdf')
您可以选择以csv、json、excel、html或sqlite格式保存表格，并选择是否将输出压缩为ZIP存档。
tables.export('foo.csv', f='csv', compress=False)
编辑: tabula-py 看起来比 camelot-py 快大约6倍，因此应该使用它。
import camelot
import cProfile
import pstats
import tabula
cmd_tabula = "tabula.read_pdf('table.pdf', pages='1', lattice=True)"
prof_tabula = cProfile.Profile().run(cmd_tabula)
time_tabula = pstats.Stats(prof_tabula).total_tt
cmd_camelot = "camelot.read_pdf('table.pdf', pages='1', flavor='lattice')"
prof_camelot = cProfile.Profile().run(cmd_camelot)
time_camelot = pstats.Stats(prof_camelot).total_tt
print(time_tabula, time_camelot, time_camelot/time_tabula)
提供
1.8495559890000015 11.057014036000016 5.978199147125147
from pprint import pprint
from io import StringIO
import re
from pdfminer.high_level import extract_text_to_fp
from pdfminer.layout import LAParams
from lxml import html
ID_LEFT_BORDER = 56
ID_RIGHT_BORDER = 156
QTY_LEFT_BORDER = 355
QTY_RIGHT_BORDER = 455
# Read PDF file and convert it to HTML
output = StringIO()
with open('example.pdf', 'rb') as pdf_file:
extract_text_to_fp(pdf_file, output, laparams=LAParams(), output_type='html', codec=None)
raw_html = output.getvalue()
# Extract all DIV tags
tree = html.fromstring(raw_html)
divs = tree.xpath('.//div')
# Sort and filter DIV tags
filtered_divs = {'ID': [], 'Qty': []}
for div in divs:
# extract styles from a tag
div_style = div.get('style')
# print(div_style)
# position:absolute; border: textbox 1px solid; writing-mode:lr-tb; left:292px; top:1157px; width:27px; height:12px;
# get left position
try:
left = re.findall(r'left:([0-9]+)px', div_style)[0]
except IndexError:
continue
# div contains ID if div's left position between ID_LEFT_BORDER and ID_RIGHT_BORDER
if ID_LEFT_BORDER < int(left) < ID_RIGHT_BORDER:
filtered_divs['ID'].append(div.text_content().strip('\n'))
# div contains Quantity if div's left position between QTY_LEFT_BORDER and QTY_RIGHT_BORDER
if QTY_LEFT_BORDER < int(left) < QTY_RIGHT_BORDER:
filtered_divs['Qty'].append(div.text_content().strip('\n'))
# Merge and clear lists with data
data = []
for row in zip(filtered_divs['ID'], filtered_divs['Qty']):
if 'ID' in row[0]:
continue
data_row = {'ID': row[0].split(' ')[0], 'Quantity': row[1]}
data.append(data_row)
pprint(data)