在CentOS 7环境源码编译Python 3

我在 CentOS 7 的古老环境中部署 Python virtualenv 遇到异常，让我非常头疼。由于历史原因不得不维护一些旧OS系统，实在是非常浪费精力。

一狠心，我决定从 更新CentOS 7的Develop Toolset 开始，完整把所有开发、程序运行环境全部重新编译升级一遍，做出一个类似 Docker Atlas 容器化干净的系统:

编译

  • 在 CentOS 上准备一些必要工具:

在CentOS 7上准备工具
yum install yum-utils make wget

  • 安装python编译需要的所有软件包:

在CentOS 7上安装编译Python3所需的所有依赖
# yum安装
sudo yum install yum-utils
sudo yum builddep python3


# 对于使用了dnf的系统，则安装 DNF builddep Plugin https://dnf-plugins-core.readthedocs.io/en/latest/builddep.html
# https://devguide.python.org/getting-started/setup-building/
sudo dnf install dnf-plugins-core  # install this to use 'dnf builddep'
sudo dnf builddep python3

也可以直接安装:

在CentOS 7上安装编译Python3所需的所有依赖(手工)
sudo yum groupinstall "Development Tools" -y

# 需要激活epel才能安装 openssl11-devel
sudo yum install epel-release

# xz-devel 就是 Devel libraries & headers for liblzma
sudo yum install openssl11-devel libffi-devel ncurses-devel \
    gdbm-devel zlib-devel bzip2-devel xz-devel \
    sqlite-devel readline-devel tk-devel libuuid-devel -y

需要注意， EPEL 安装 openssl11-devel 不能被默认找到，需要执行以下方法来为后续 configure --with-openssl=DIR 做准备:

通过 EPEL 安装的 openssl11-devel 需要创建软链接以便后续 configure 传递路径
mkdir /usr/local/openssl11
cd /usr/local/openssl11
ln -s /usr/lib64/openssl11 lib
ln -s /usr/include/openssl11 include

对于 Ubuntu Linux 则直接安装

在Debian上安装编译Python3所需的所有依赖(手工)
sudo apt-get install build-essential gdb lcov pkg-config \
      libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \
      libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
      lzma lzma-dev tk-dev uuid-dev zlib1g-dev

备注

configure 参数可以参考 3.11.4 Documentation » Python Setup and Usage » 3. Configure Python

  • 编译Python，激活速度优化:

编译python3
wget https://www.python.org/ftp/python/3.11.4/Python-3.11.4.tgz
tar xzf Python-3.11.4.tgz
cd Python-3.11.4

#./configure --enable-optimizations --with-lto --with-computed-gotos --with-system-ffi
#make -j "$(nproc)"

# --with-ensurepip 确保同时安装pip
#./configure --enable-optimizations --with-ensurepip=install

# 我结合使用参数如下
./configure --enable-optimizations --with-lto --with-computed-gotos --with-system-ffi \
    --with-ensurepip=install --with-openssl=/usr/local/openssl11
make
# altinstall 避免覆盖系统python
sudo make altinstall

参数说明:

  • --enable-optimizations 激活 Profile Guided Optimization (PGO) ，Clang编译器需要 llvm-profdata 程序

  • --with-lto 激活 Link Time Optimization (LTO) `` 对于最佳性能，推荐采用 ``--enable-optimizations --with-lto (PGO + LTO)，Clang编译器需要 llvm-ar

  • --with-computed-gotos 在评估循环是激活计算gotos，对于支持的编译器默认激活

  • --with-system-ffi 使用已经安装的 ffi 库来编译 _ctypes 扩展模块，这个是默认系统依赖的

  • with-ensurepip=[upgrade|install|no] 默认是 upugrade ，当使用 install 参数时会运行 python -m ensurepip --altinstall

  • --with-openssl=DIR 传递非系统默认openssl目录，此时会自动检测 runtime library ；或者直接使用 --with-openssl-rpath=[no|auto|DIR] 指定 DIR

备注

make altinstall 之后，安装在 /usr/local/bin 目录下的 python3.11 和 pip3.11 就可以用来构建 Python virtualenv

一些异常及处理

初次报错

  • 系统需要 openssl-1.1.1 或更高版本才能支持 ssl 模块(CentOS 7使用的是openssl-1.0.2)，需要注意 EPEL 安装 openssl11 位置非默认，需要传递 configure 参数 --with-openssl=DIR

我的初次编译报错如下(上文已经逐个修正):

The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel
_dbm                  _gdbm                 _hashlib
_lzma                 _ssl                  _tkinter
_uuid                 readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py have not
been built, they are *disabled* by configure:
_sqlite3


Failed to build these modules:
_ctypes


Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer

再次报错(部分解决)

  • 虽然安装了 uuid-devel ，但是依然报错没有支持 _uuid 模块:

    The necessary bits to build these optional modules were not found:
_tkinter              _uuid
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

  • 对应解决方法: _uuid 需要安装 libuuid-devel 解决

  • 我还是没有解决 _tkinter : _tkinter 似乎需要 tk-devel (我最初以为是要安装 tkinter )，但是我安装了还是有 _tkinter 模块无法找到(虽然我也已经安装了 tkinter 软件包)，但是还是有这个报错( 奇怪了 )。我自检查了 configure 输出，确实发现:

configure 依然报错找不到 _tkinter ，虽然我已经安装了 tk-devel 软件包
...
checking for additional Modules/Setup files...
checking for stdlib extension module _multiprocessing... yes
checking for stdlib extension module _posixshmem... yes
checking for stdlib extension module fcntl... yes
checking for stdlib extension module mmap... yes
checking for stdlib extension module _socket... yes
checking for stdlib extension module grp... yes
checking for stdlib extension module ossaudiodev... yes
checking for stdlib extension module pwd... yes
checking for stdlib extension module resource... yes
checking for stdlib extension module _scproxy... n/a
checking for stdlib extension module spwd... yes
checking for stdlib extension module syslog... yes
checking for stdlib extension module termios... yes
checking for stdlib extension module pyexpat... yes
checking for stdlib extension module _elementtree... yes
checking for stdlib extension module _md5... yes
checking for stdlib extension module _sha1... yes
checking for stdlib extension module _sha256... yes
checking for stdlib extension module _sha512... yes
checking for stdlib extension module _sha3... yes
checking for stdlib extension module _blake2... yes
checking for stdlib extension module _crypt... yes
checking for stdlib extension module _decimal... yes
checking for stdlib extension module _gdbm... yes
checking for stdlib extension module nis... yes
checking for stdlib extension module _sqlite3... yes
checking for stdlib extension module _tkinter... missing
checking for stdlib extension module _uuid... yes
checking for stdlib extension module zlib... yes
checking for stdlib extension module _bz2... yes
checking for stdlib extension module _lzma... yes
checking for stdlib extension module _ssl... yes
checking for stdlib extension module _hashlib... yes
checking for stdlib extension module _testcapi... yes
checking for stdlib extension module _testclinic... yes
checking for stdlib extension module _testinternalcapi... yes
checking for stdlib extension module _testbuffer... yes
checking for stdlib extension module _testimportmultiple... yes
checking for stdlib extension module _testmultiphase... yes
checking for stdlib extension module _xxtestfuzz... yes
checking for stdlib extension module _ctypes_test... yes
checking for stdlib extension module xxlimited... yes
checking for stdlib extension module xxlimited_35... yes
...

备注

我还没有解决最后编译的Python 3不支持 _tkinter 模块问题，这个模块是做tk图形开发，在服务器端用不上。不过，比较奇怪，看起来我配置应该没有错才对…留待以后再尝试了(生产环境魔改的aliOS真是蛋疼)

参考