Paramiko SFTP连接在上传大文件时中断

Question

我正在尝试将一个大文件（1GB）上传到SFTP，但是我一直得到以下变体：

File "/venv/lib/python2.7/site-packages/paramiko/file.py", line 339, in write
    self._write_all(data)
  File "/venv/lib/python2.7/site-packages/paramiko/file.py", line 456, in _write_all
    count = self._write(data)
  File "/venv/lib/python2.7/site-packages/paramiko/sftp_file.py", line 180, in _write
    t, msg = self.sftp._read_response(req)
  File "/venv/lib/python2.7/site-packages/paramiko/sftp_client.py", line 762, in _read_response
    raise SSHException('Server connection dropped: %s' % str(e))
SSHException: Server connection dropped: 

我注意到如果我将 sftp_file.py 中的 MAX_REQUEST_SIZE（在 sftp_file.py 中）更新为1024而不是32768，它可以工作。这是否意味着我的唯一选择是复制/粘贴一个具有 MAX_REQUEST_SIZE = 1024 的自定义版本的 sftp_file.py？还有其他人有不会减慢上传速度的建议吗？

更新：最后几次尝试更新 MAX_REQUEST_SIZE 时，它出现了 "OperationalError: SSL SYSCALL error: EOF detected" 错误。供参考，这是我目前正在做的事情：

    transport = paramiko.Transport((hostname, port))
    transport.connect(username, password)
    sftp = paramiko.SFTPClient.from_transport(transport)
    f = sftp.open(ftp_path, 'wb')
    f.write(file_obj.read())
    f.close()
    sftp.close()
    transport.close()

- bpow

Answer 1

我认为你需要减小块大小是相当奇怪的。SSH的RFC4253规定：

所有实现必须能够处理未压缩有效载荷长度不超过32768字节和总数据包大小不超过35000字节（包括“packet_length”，“padding_length”，“payload”，“随机填充”和“mac”）的数据包。

无论如何，你可以尝试使用以下方法而不是使用自定义版本的sftp_file.py：

sftpclient = sshclient.open_sftp()
with sftpclient.file(remote_file_path, mode='w') as rem_file:
    rem_file.MAX_REQUEST_SIZE = 1024
    rem_file.write(data)

并且尝试调整 MAX_REQUEST_SIZE（例如16384、24576等）

- fernandezcuesta

Answer 2

在使用get之前，我添加了打开/关闭操作，出于某些我不知道的原因，它可以处理大文件。

    file = self._connection.open(remote_path, 'rb')
    file.close()
    self._connection.get(remote_path, local_path,callback=None)

- Cristian Avendaño

Answer 3

我收到了一个错误信息：“ERROR_SFTP_PULL，AUTHENTICATION ERROR Server connection dropped”： -从我的本地服务器到远程服务器的普通无密码sftp（get）正常工作，但是通过Python脚本进行SFTP PULL失败。它在4天前正常运行。 -下面的代码部分看起来很糟糕：

1.  authReturnCode = "<class 'paramiko.ssh_exception.BadAuthenticationType'>", not sure please help
2. **errorKey = "INFO_SFTP_PULL, "+ eachFiles + " successfully copyied from SFX for " + custName**

Failure log:
===========
2021-04-26 15:00:02,459 INFO::ipndProxyApp.py:787: Processing data for: eDCP
2021-04-26 15:00:02,851 INFO::transport.py:1746: Connected (version 2.0, client SSHD)
2021-04-26 15:00:03,860 INFO::transport.py:1746: Authentication (publickey) successful!
2021-04-26 15:00:03,922 INFO::sftp.py:158: [chan 0] Opened sftp connection (server version 3)
2021-04-26 15:15:15,376 INFO::sftp.py:158: [chan 0] sftp session closed.
2021-04-26 15:15:15,491 ERROR::ipndProxyApp.py:119: ERROR_SFTP_PULL, AUTHENTICATION ERROR Server connection dropped:

Normal log is:
=============
2021-04-22 15:00:02,138 INFO::ipndProxyApp.py:782: Processing data for: eDCP
2021-04-22 15:00:02,256 INFO::transport.py:1746: Connected (version 2.0, client SSHD)
2021-04-22 15:00:02,563 INFO::transport.py:1746: Authentication (publickey) successful!
2021-04-22 15:00:02,586 INFO::sftp.py:158: [chan 0] Opened sftp connection (server version 3)
2021-04-22 15:00:09,999 INFO::sftp.py:158: [chan 0] sftp session closed.
2021-04-22 15:00:10,003 INFO::ipndProxyApp.py:122: INFO_SFTP_PULL, FILE_2021-04-21-18-00-35.txt successfully copied from SFX for eDCP

Python Code:
===========
import os, sys, logging, shutil
import pysftp, mysql.connector
import subprocess, csv, argparse
from datetime import datetime, timedelta
import time

# Importing Configuration settings or other supporting scripts
import settings
import SQLqueries
import fieldLengthVerification

# Static Configuration settings, logging settings, global variables
logging.basicConfig(filename='/var/log/ipnd.log', format='%(asctime)s %(levelname)s::%(filename)s:%(lineno)d: %(message)s', dateformat='%Y-%m-%d %I:%M:%S',level=logging.INFO)
currentDateTime = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
currentDate = datetime.now().strftime("%Y-%m-%d")
yesterdays_date = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
twoDaysBack = (datetime.now() - timedelta(days=2)).strftime("%Y-%m-%d")
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None


# Handling Authentication errors for the SFT authentications only
def errorHandling(errorFromProgram):
    """
    Function to handle authentication error for sftp only, it accepts exception errors and return string keyword.

    Parameters:
    errorFromProgram (class) : Error class generated by Exceptions

    Returns:
    str: str(errorFromProgram) or "Authentication Success" or "AUTHENTICATION ERROR " + str(errorFromProgram)
    """

    # This function to perform error handling and return string against the provided input  
    
    authReturnCode = "<class 'paramiko.ssh_exception.BadAuthenticationType'>"

    # Handling RHEL version of sftp (<class 'paramiko.ssh_exception.AuthenticationException'>)
    if str(errorFromProgram) == "Authentication failed.":
        errorKeyWord = str(errorFromProgram)
        return errorKeyWord

    # Handling SUSE version of sftp (<class 'paramiko.ssh_exception.BadAuthenticationType'>)
    elif str(type(errorFromProgram)) == authReturnCode:
        errorExcetption,a = errorFromProgram
        errorKeyWord = errorExcetption
        return errorKeyWord

    # Handling other situation then provious two
    elif (str(errorFromProgram) != "Authentication failed.") or (str(type(errorFromProgram)) != authReturnCode):    
        errorKeyWord = "AUTHENTICATION ERROR " + str(errorFromProgram)  
        return errorKeyWord

    # Handling other conditions if not handled in provious if and elif
    else:
                infoKeyWord = "Authentication Success"
        return infoKeyWord


# Fetch eDCP Data files from SFX sftp server

def fetchFilesFromSftpLocation(custName, fileSrcHost, remoteFileLocation, fileSrcUser, fileSrcPassPhrase, dateToProcessFiles):
    """
    Function to fetch files from remove servers and store in location directory, return nothing and accept following parameters

    Parameters:
    custName (str): Customer Name.
    fileSrcHost (str): Hostname (FQDN) or IP address of the source server from where files would be fetched.
    fileLocation (str): Remote file location.
    fileSrcUser (str): User to connect remote server to fetch files.
    fileSrcPassPhrase (str): Password to connect remote server to fetch files.
    dateToProcessFiles (str): Date to fetch the files for   
    """

    # try to connect SFX server over sftp
    try:
        sftp = pysftp.Connection(fileSrcHost, username = fileSrcUser)

        # change to remote directory
        sftp.chdir(remoteFileLocation)
        listOfFilesAtSFX = sftp.listdir()

        for eachFiles in listOfFilesAtSFX:
            
            if eachFiles.startswith("IPNDUP" + custName.upper() +"_" + dateToProcessFiles):
                
                # Fetch files from SFTP
                try:    
                    sftp.get(eachFiles)
                                        
                    **errorKey = "INFO_SFTP_PULL, "+ eachFiles + " successfully copyied from SFX for " + custName**

                except Exception as e:
                    errorKey = "ERROR_SFTP_PULL, "+ errorHandling(e)
                    #print(errorKey)

        sftp.close()

    # Capture Exception
    except Exception as e:
        errorKey = errorHandling(e)
        errorKey = "ERROR_SFTP_PULL, "+ errorKey


        # Checking conditions to generate ERROR or INFO
        if len(errorKey) > 0:
                listOfErrorKeys = errorKey.split('_')
                if "ERROR" in listOfErrorKeys:
                        logging.error("%s" % errorKey)
                        sys.exit(1)
                else:
                        logging.info("%s" % errorKey)

        else:
                logging.error("%s" % "No Error keywords found")

- Pramod Tomar