06月11, 2017

Python 开发环境

Anaconda 是一个收集了超过720开放源码软件包与免费社区支持的 Python 发行版本,其中就有一个叫做 conda 的软件包管理、依赖管理和沙盒环境管理的神器,号称支持任何语言。对于 Python 这门编程语言来说,有一个叫做 virtualenv 的工具和一个叫做 virtualenvwrapper 的工具可以为之提供沙盒环境,区别是后者对前者进行了封装扩展,使用起来更简单方便些了。

安装 Anaconda3

照着文档步骤做就对了(MacOS command line install)。

Anaconda 添加到 PATH 环境变量,我的是 ~/.zshrc

# https://docs.continuum.io/anaconda/
export PATH="/Users/xlang/anaconda3/bin:$PATH"

验证安装结果(VERIFYING YOUR INSTALLATION):

$ conda list
# packages in environment at /Users/xlang/anaconda3:
#
_license                  1.1                      py36_1  
alabaster                 0.7.10                   py36_0  
anaconda                  4.4.0               np112py36_0  
...略

使用 Conda 管理沙盒环境

配置首选镜像站,清华大学开源软件镜像站(TUNA)中提供了 Anaconda 的仓库镜像(使用帮助)。

# Add the tuna channel
$ conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
# Add the conda-forge channel
$ conda config --add channels conda-forge
$ conda config --set show_channel_urls yes

使用 conda create -n|--name ENVIRONMENT [package_spec [package_spec ...]] 创建 scrapy 环境:

$ conda create -n scrapyenv scrapy

使用 conda info -e|--envsconda-env list 查看当前环境列表:

$ conda info --envs
# conda environments:
#
scrapyenv                /Users/xlang/anaconda3/envs/scrapyenv
root                  *  /Users/xlang/anaconda3

* 表示当前使用的名称为 root 的环境,启动 scrapyenv 环境:

$ source activate scrapyenv # 启动环境
(scrapyenv) $ conda list scrapy
# packages in environment at /Users/xlang/anaconda3/envs/scrapyenv:
#
scrapy                    1.4.0                    py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge

可以看到沙盒环境中已经有 scrapy 包了,创建了一个命名为 spider 项目:

(scrapyenv) $ pwd
/Users/xlang/pydir
(scrapyenv) $ scrapy startproject spider
New Scrapy project 'spider', using template directory '/Users/xlang/anaconda3/envs/scrapyenv/lib/python3.6/site-packages/scrapy/templates/project', created in:
    /Users/xlang/pydir/spider

You can start your first spider with:
    cd spider
    scrapy genspider example example.com
(scrapyenv) $ source deactivate # 停止运行

使用 Virtualenv 管理沙盒环境

# https://virtualenv.pypa.io/en/stable/installation/
# https://github.com/kennethreitz/python-guide/blob/master/docs/dev/virtualenvs.rst
$ [sudo] pip install virtualenv

安装好 virtualenv 后,看看 User Guide 怎么用,嗯 virtualenv ~/pydir/scrapyenv 就可以做一个沙盒环境了,如下:

$ virtualenv ~/pydir/scrapyenv
Running virtualenv with interpreter /Users/xlang/pydir/scrapyenv/bin/python
dyld: Library not loaded: @rpath/libpython3.6m.dylib
  Referenced from: /Users/xlang/pydir/scrapyenv/bin/python
  Reason: image not found

这是什么问题? 前面安装了 anaconda 发行版,系统里面有多个Python环境。 (Many Problems with python's virtualenv)

先卸载掉 pip 安装的,用 conda 包管理器来重新安装 virtualenv

$ pip uninstall virtualenv
$ conda install virtualenv

然后针对沙盒环境 virtualenv 设置环境变量,使其应用 anaconda 发行版中提供的 python解析器,以下方式都行:

$ export VIRTUALENV_PYTHON=/Users/xlang/anaconda3/bin/python3
$ virtualenv ~/pydir/scrapyenv
# 或者
$ virtualenv --python=/Users/xlang/anaconda3/bin/python3 ~/pydir/scrapyenv

或者增加配置文件 ~/.virtualenv/virtualenv.ini:

[virtualenv]
python = /Users/xlang/anaconda3/bin/python3

接下来使用 virtualenv 创建沙盒环境,并安装 scrapy 包:

$ virtualenv ~/pydir/scrapyenv
Using base prefix '/Users/xlang/anaconda3'
New python executable in /Users/xlang/pydir/scrapyenv/bin/python
copying /Users/xlang/anaconda3/bin/python => /Users/xlang/pydir/scrapyenv/bin/python
copying /Users/xlang/anaconda3/bin/../lib/libpython3.6m.dylib => /Users/xlang/pydir/scrapyenv/lib/libpython3.6m.dylib
Installing setuptools, pip, wheel...done.
$ pwd
/Users/xlang/pydir/scrapyenv
$ source bin/activate
(scrapyenv) $ pip install scrapy
(scrapyenv) $ scrapy startproject tutorial
New Scrapy project 'tutorial', using template directory '/Users/xlang/.virtualenvs/scrapyenv/lib/python3.6/site-packages/scrapy/templates/project', created in:
    /Users/xlang/pydir/scrapyenv/tutorial

You can start your first spider with:
    cd tutorial
    scrapy genspider example example.com
(scrapyenv) $ deactivate

使用 Virtualenvwrapper 管理沙盒环境

~/.zshrc 中加入下面这几行:

# https://virtualenvwrapper.readthedocs.io/en/latest/install.html
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/pydir
source /Users/xlang/anaconda3/bin/virtualenvwrapper.sh

查看帮助可以看到众多扩展 virtualenv 的便捷工具命令:

$ virtualenvwrapper -h

virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv
tool.  The extensions include wrappers for creating and deleting
virtual environments and otherwise managing your development workflow,
making it easier to work on more than one project at a time without
introducing conflicts in their dependencies.

For more information please refer to the documentation:

    http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html

Commands available:

  add2virtualenv: add directory to the import path

  allvirtualenv: run a command in all virtualenvs

  cdproject: change directory to the active project

  cdsitepackages: change to the site-packages directory

  cdvirtualenv: change to the $VIRTUAL_ENV directory

  cpvirtualenv: duplicate the named virtualenv to make a new one

  lssitepackages: list contents of the site-packages directory

  lsvirtualenv: list virtualenvs

  mkproject: create a new project directory and its associated virtualenv

  mktmpenv: create a temporary virtualenv

  mkvirtualenv: Create a new virtualenv in $WORKON_HOME

  rmvirtualenv: Remove a virtualenv

  setvirtualenvproject: associate a project directory with a virtualenv

  showvirtualenv: show details of a single virtualenv

  toggleglobalsitepackages: turn access to global site-packages on/off

  virtualenvwrapper: show this help message

  wipeenv: remove all packages installed in the current virtualenv

  workon: list or change working virtualenvs

其中 mkproject 命令可用来创建项目,并默认在 WORKON_HOME 环境变量指定的目录创建同名沙盒环境,比如:

$ mkproject scrapyenv2
Using base prefix '/Users/xlang/anaconda3'
New python executable in /Users/xlang/.virtualenvs/scrapyenv2/bin/python
copying /Users/xlang/anaconda3/bin/python => /Users/xlang/.virtualenvs/scrapyenv2/bin/python
copying /Users/xlang/anaconda3/bin/../lib/libpython3.6m.dylib => /Users/xlang/.virtualenvs/scrapyenv2/lib/libpython3.6m.dylib
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /Users/xlang/.virtualenvs/scrapyenv2/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/xlang/.virtualenvs/scrapyenv2/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/xlang/.virtualenvs/scrapyenv2/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/xlang/.virtualenvs/scrapyenv2/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/xlang/.virtualenvs/scrapyenv2/bin/get_env_details
Error: deactivate must be sourced. Run 'source deactivate'
instead of 'deactivate'.

Creating /Users/xlang/pydir/scrapyenv2
Setting project for scrapyenv2 to /Users/xlang/pydir/scrapyenv2
(scrapyenv2) $ pwd
/Users/xlang/pydir/scrapyenv2
(scrapyenv2) $ deactivate

可以看到目录 /Users/xlang/pydir/scrapyenv2 和同名沙盒环境 /Users/xlang/.virtualenvs/scrapyenv2 都创建好了,并且当前已经进入沙盒环境和相对应的项目目录中, 扩展工具默认沙盒环境被放到了前面配置的目录中了,类似 conda 提供的沙盒环境。

$ tree -d -L 1 scrapyenv2
scrapyenv2

0 directories
$ tree -d -L 1 scrapyenv
scrapyenv
├── bin
├── include
└── lib

3 directories

对比 virtualenv ~/pydir/scrapyenv 创建沙盒环境,mkproject 将沙盒环境设置默认配置的 /Users/xlang/.virtualenvs/scrapyenv2 目录了。

  1. Anaconda 文档#术语表
  2. 关于conda和anaconda不可不知的事实和误解原文#Conda: Myths and Misconceptions)。

本文链接:https://xlange.com/post/getting-started-python-from-scrapy

-- EOF --

Comments

?