安装Scrapy在10.9 Mavericks系统上出现问题

Question

安装Scrapy在10.9 Mavericks系统上出现问题

pythonmacospython-2.7scrapyosx-mavericks

3

我想使用我的homebrew python版本安装Scrapy。尽管我已经更新了所有内容并且在路径中已经安装，但我无法使用它。

以下是我操作的步骤：

检查brew:

$ brew doctor
Your system is ready to brew.

检查 Python 和 Pip：

$ which python
/usr/local/bin/python
$ python --version
Python 2.7.6
$ which pip
/usr/local/bin/pip
$ pip -- version
pip 1.5.4 from /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg (python 2.7)

我是一名有用的助手，可以为您进行文本翻译。

(这正常吗？看起来好像有两份副本，可能会产生冲突)

我已经安装了Scrapy，但当我再次尝试时，出现以下情况：

$ pip install scrapy
Requirement already satisfied (use --upgrade to upgrade): scrapy in /usr/local/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->scrapy)
Cleaning up...

这对我来说很奇怪，因为我希望我的模块被安装在/usr/local/share/python而不是/usr/local/lib/python2.7/site-packages。果然，在/usr/local/lib/python2.7/site-packages中没有scrapy，但在/usr/local/share/python中有。

如讨论Scrapy installation on OSX Lion，我已将/usr/local/share/python/scrapy添加到我的路径中。

但是which scrapy返回空白。

最有趣的部分是当我尝试使用pip install --upgrade scrapy '更新' scrapy时：

~ ∮ pip install --upgrade scrapy
Requirement already up-to-date: scrapy in /usr/local/lib/python2.7/site-packages
Requirement already up-to-date: Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: queuelib in /usr/local/lib/python2.7/site-packages (from scrapy)
Downloading/unpacking lxml from https://pypi.python.org/packages/source/l/lxml/lxml-3.3.2.tar.gz#md5=a3ea7bf74b718ecb46d9fd5198eec92d (from scrapy)
  Downloading lxml-3.3.2.tar.gz (3.5MB): 3.5MB downloaded
  Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/lxml/setup.py) egg_info for package lxml
    /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    Building lxml version 3.3.2.
    Building without Cython.
    Using build configuration of libxslt 1.1.28

    warning: no previously-included files found matching '*.py'
Downloading/unpacking pyOpenSSL from https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.14.tar.gz#md5=8579ff3a1d858858acfba5f046a4ddf7 (from scrapy)
  Downloading pyOpenSSL-0.14.tar.gz (128kB): 128kB downloaded
  Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/pyOpenSSL/setup.py) egg_info for package pyOpenSSL

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    no previously-included directories found matching 'doc/_build'
Requirement already up-to-date: cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Downloading/unpacking cryptography>=0.2.1 (from pyOpenSSL->scrapy)
  Downloading cryptography-0.2.1.tar.gz (13.8MB): 13.8MB downloaded
  Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py) egg_info for package cryptography
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found

    Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/cffi-0.8.1-py2.7-macosx-10.9-x86_64.egg
    Searching for pycparser
    Reading https://pypi.python.org/simple/pycparser/
    Best match: pycparser 2.10
    Downloading https://pypi.python.org/packages/source/p/pycparser/pycparser-2.10.tar.gz#md5=d87aed98c8a9f386aa56d365fe4d515f
    Processing pycparser-2.10.tar.gz
    Writing /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/setup.cfg
    Running pycparser-2.10/setup.py -q bdist_egg --dist-dir /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/egg-dist-tmp-pniw4C
    zip_safe flag not set; analyzing archive contents...

    Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/pycparser-2.10-py2.7.egg

    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 113, in <module>
        "build": cffi_build,
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
        dist.run_commands()
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "<string>", line 15, in replacement_run
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 186, in find_sources
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 246, in run
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 282, in add_defaults
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/sdist.py", line 167, in add_defaults
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
        self.finalize_options()
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/build_py.py", line 26, in finalize_options
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options
        ('force', 'force'))
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options
        src_cmd_obj.ensure_finalized()
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
        self.finalize_options()
      File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 52, in finalize_options
        from cryptography.hazmat.primitives import constant_time, padding
      File "cryptography/hazmat/primitives/constant_time.py", line 18, in <module>
        import six
    ImportError: No module named six
    Complete output from command python setup.py egg_info:
    Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found



Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/cffi-0.8.1-py2.7-macosx-10.9-x86_64.egg

Searching for pycparser

Reading https://pypi.python.org/simple/pycparser/

Best match: pycparser 2.10

Downloading https://pypi.python.org/packages/source/p/pycparser/pycparser-2.10.tar.gz#md5=d87aed98c8a9f386aa56d365fe4d515f

Processing pycparser-2.10.tar.gz

Writing /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/setup.cfg

Running pycparser-2.10/setup.py -q bdist_egg --dist-dir /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/egg-dist-tmp-pniw4C

zip_safe flag not set; analyzing archive contents...



Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/pycparser-2.10-py2.7.egg

running egg_info

creating pip-egg-info/cryptography.egg-info

writing requirements to pip-egg-info/cryptography.egg-info/requires.txt

writing pip-egg-info/cryptography.egg-info/PKG-INFO

writing top-level names to pip-egg-info/cryptography.egg-info/top_level.txt

writing dependency_links to pip-egg-info/cryptography.egg-info/dependency_links.txt

writing manifest file 'pip-egg-info/cryptography.egg-info/SOURCES.txt'

warning: manifest_maker: standard file '-c' not found



Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 113, in <module>

    "build": cffi_build,

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup

    dist.run_commands()

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands

    self.run_command(cmd)

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command

    cmd_obj.run()

  File "<string>", line 15, in replacement_run

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 186, in find_sources

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 246, in run

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 282, in add_defaults

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/sdist.py", line 167, in add_defaults

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command

    cmd_obj.ensure_finalized()

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized

    self.finalize_options()

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/build_py.py", line 26, in finalize_options

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options

    ('force', 'force'))

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options

    src_cmd_obj.ensure_finalized()

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized

    self.finalize_options()

  File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 52, in finalize_options

    from cryptography.hazmat.primitives import constant_time, padding

  File "cryptography/hazmat/primitives/constant_time.py", line 18, in <module>

    import six

ImportError: No module named six

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_myAccount/cryptography
Storing debug log for failure in /Users/myAccount/.pip/pip.log
~ ∮

我已经尝试过解决一些问题，例如pip install six和尝试谷歌搜索egg_info failed with error code 1。大多数解决方案涉及未安装setuptools：

$ pip install setuptools
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg
Cleaning up...

看起来我的brew安装的Python和错误的链接/多个版本的pip导致模块被安装在一个地方，而我的Python在另一个地方（这解释了为什么>>> import scrapy失败，但不解释为什么$ scrapy失败）。

关于破损的Python模块的类似帖子建议运行xcode-select --install，我已确认更新。Mac OS X 10.9后无法安装PIL 也许现在是virtualenv的时候了？

提前感谢！

- santeko

是的，virtualenv 可能会解决很多这些问题。 - hd1

嗯...我已经做过这个了，而且没有使用homebrew...只需执行sudo pip install scrapy（sudo是因为通常mac权限会出问题），它应该可以工作...除非你的电脑有问题。 - A.J. Uppal

@aj8uppal 我使用 pip install scrapy 时得到了与您相同的结果。我有另一台没有 homebrew 的笔记本电脑，您的建议可行（我可以使用命令行函数但无法使用 Python 导入）。然而，在这台电脑上，scrapy 在命令行中无法识别，也无法被 Python 识别 >>> import scrapy。 - santeko

@hd1 我尝试安装 virtualenv，但效果与 scrapy 差不多。我按照以下步骤进行：https://dev59.com/fnLYa4cB1Zd3GeqPbsg7 按照第一个答案执行： `~ ∮ pip install virtualenvRequirement already satisfied (use --upgrade to upgrade): virtualenv in /usr/local/lib/python2.7/site-packages Cleaning up...~ ∮ virtualenv test-bash: virtualenv: command not found` - santeko

你在执行时是否使用了 Scrapy 的完整路径？如果没有，请尝试这样做。 - hd1

@hd1 scrapy不在pip所说的位置：`pip install scrapyRequirement already satisfied (use --upgrade to upgrade): scrapy in /usr/local/lib/python2.7/site-packages`但它并不在那里。 - santeko

5个回答

2

以上方法在我的OS X 10.10 Yosemite系统中均无效。

但我在这里找到了解决方案。

使用pkg-config即可：

brew install pkg-config libffi
export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.0.13/lib/pkgconfig/
pip install cffi

您需要根据您的libffi版本更改路径。

- Douglas Schmidt

这对我在10.9 Mavericks中也起作用了。使用“locate libffi.pc”查找我的位置：/usr/local/Cellar/libffi/3.0.13/lib/pkgconfig/libffi.pc - MikeiLL

2

我曾在安装virtualenv时遇到类似的问题：Can't pip install virtualenv on Mac 10.9 Mavericks with brewed python，那里提供的解决方案同样适用于我的情况，但我仍不理解为什么会出现这个问题以及为什么这个解决方案有效。但它确实有效。

卸载并重新安装scrapy：

$pip uninstall scrapy
$pip install scrapy

这将从/usr/local/lib/python2.7/site-packages中移除模块，并安装到/usr/local/bin/。问题在于，根据homebrew python文档，pip应该安装在/site-packages中并且应该正常工作。但是它们放在/site-packages中不仅不能正常工作，而且pip默认安装在/usr/local/bin/中。这个解决方法虽然可行，但我真的很想知道为什么它能起作用，以及为什么这是一个问题。也许这两个问题可以合并？

- santeko

0

其他帖子对我没用，但这个有用：

sudo ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install scrapy

- williaster

0

您需要让您的安装环境访问与OpenSSL捆绑的Crypto库。以下是对我有效的解决方案。

env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include"

请记得将此更改为您更新的OpenSSL软件包的位置。

- Nick Woodhams

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- jranzenbach · Accepted Answer

以下是我的建议：

这是对我有用的方法：

Since you're using Homebrew as your package manager, the the programs installed through it will be set to /usr/local/bin rather than the standard /usr/bin path setting for out of the box applications. You'll want to ensure that /usr/local/bin is at the top of the file in /etc/paths. The file should be locked, so you may need to save changes in the vi editor with the following:
```
:w !sudo tee %
```

$ which python should return /usr/local/bin/python, else:

`$ brew unlink python`
`$ brew install python`
`$ brew linkapps`

$ which pip should return /usr/local/bin/pip, else
```
`$ brew install pip`
```

Then...

`$ sudo pip uninstall Scrapy`
`$ pip install Scrapy`
`$ pip install --upgrade Scrapy`
`$ pip install --upgrade lxml`
`$ scrapy startproject tutorial`