Error python is missing or unusable

I’ll start with the solution. I needed to:

sudo apt-get install python2.7-dev

(that apparently did not need to previously be installed, as I was able to compile prior to recent updates)

Now for the problem (in case others search for it).

I had the previous compile working… for quite some time… but this new one turned out to be a bit more challenging for me. I purged all of the old stuff — wiping out the esp-open-sdk completely. I then fetch & compile like this:

cd ~/workspace/esp-open-sdk
make clean
git pull
git submodule sync
git submodule update --init
make

all goes well, but then I would get an error like this when almost done:

[INFO ]  Installing C library
[INFO ]  Installing C library: done in 42.67s (at 12:09)
[INFO ]  =================================================================
[INFO ]  Installing final gcc compiler
[INFO ]  Installing final gcc compiler: done in 412.31s (at 19:01)
[INFO ]  =================================================================
[INFO ]  Installing cross-gdb
[ERROR]    configure: error: python is missing or unusable
[ERROR]    make[4]: *** [configure-gdb] Error 1
[ERROR]    make[3]: *** [all] Error 2
[ERROR]
[ERROR]  >>
[ERROR]  >>  Build failed in step 'Installing cross-gdb'
[ERROR]  >>        called in step '(top-level)'
[ERROR]  >>
[ERROR]  >>  Error happened in: CT_DoExecLog[scripts/functions@257]
[ERROR]  >>        called from: do_debug_gdb_build[scripts/build/debug/300-gdb.sh@120]
[ERROR]  >>        called from: do_debug[scripts/build/debug.sh@35]
[ERROR]  >>        called from: main[scripts/crosstool-NG.sh@646]
[ERROR]  >>
[ERROR]  >>  For more info on this error, look at the file: 'build.log'
[ERROR]  >>  There is a list of known issues, some with workarounds, in:
[ERROR]  >>      'share/doc/crosstool-ng/crosstool-ng-1.22.0-55-gecfc19a/B - Known issues.txt'
[ERROR]
[ERROR]  (elapsed: 20:14.68)
[20:15] / ct-ng:152: recipe for target 'build' failed

There’s nothing related in the known issues file. The build.log has the basically the same log file info displayed, but with an additional note just before the error that has a «no» for python2.7:

[ALL  ]    checking for XML_StopParser... yes
[ALL  ]    checking whether to use python... yes
[ALL  ]    checking for python... /usr/bin/python
[ALL  ]    checking for python2.7... no
[ERROR]    configure: error: python is missing or unusable
[ALL  ]    Makefile:8724: recipe for target 'configure-gdb' failed
[ERROR]    make[4]: *** [configure-gdb] Error 1

Here’s my python info:

me@mymachine:~$ apt-cache policy python
python:
  Installed: 2.7.9-1
  Candidate: 2.7.9-1
  Version table:
 *** 2.7.9-1 0
        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
me@mymachine:~$ apt-cache policy python3
python3:
  Installed: 3.4.2-2
  Candidate: 3.4.2-2
  Version table:
 *** 3.4.2-2 0
        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status

if I launch python, it seems to work, and the default is v2.7:

$ python
Python 2.7.9 (default, Mar  1 2015, 12:57:24)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print "hello world"
hello world

it is not python that was a problem, but the missing python2.7-dev !

and so I’ve updated my building script described here: http://forum.micropython.org/viewtopic. … 9769#p9769 or more specifically on github here https://github.com/gojimmypi/build_esp8266 — those are my own instructions & notes! the official build instructions are here: http://forum.micropython.org/viewtopic.php?f=16&t=1689

@pfalcon — perhaps the python2.7-dev could be added here: https://github.com/pfalcon/esp-open-sdk as well?

hopefully this helps someone…

btw the official build instructions are here: http://forum.micropython.org/viewtopic.php?f=16&t=1689

В данной заметке речь пойдёт о замечательном средстве автоматизации сборки кросс-тулчейнов crosstool-ng, практически незаменимого инструмента для любого уважающего себя embedded-разработчика. Если вам приходилось по-серьёзному собирать софт из

x86-linux под arm-linux, то вы наверняка слышали о нём.

В данном руководстве рассматривается не столько кросс-компиляция по архитектуре, сколько кросс-компиляция по системе — сборка под Linux в Darwin.

Disclaimer

В интернете есть несколько статей по сборке crosstool-ng под OS X, например на benmont.com и в официальном руководстве. Тем не менее, в некоторых статьях встречается множество ошибок и устаревших сведений, а в других описываются лишь общие черты. Здесь будет описан мой путь, по которому я успешно собрал toolchain в июле 2013.

Подготовка

Эта часть зависит от того, какой пакетный менеджер вы используете в OS X — MacPorts или Homebrew. Я для себя давно выбрал ports-way, так что буду писать исходя из этого.

1. Регистрозависимая файловая система

Тут всё просто, в OS X есть утилита Disk Utility, воспользуемся ей для создания нового раздела. Потребуется 5+ Гб.

2. Инструменты

Предполагается, что у вас установлен MacPorts. Установим следующие пакеты:

ncurses
lzma
libtool
binutils
gsed
gawk
grep
gcc48

Проверим, какие версии gcc есть в системе: sudo port select —list gcc. Нам нужна mp-gcc48 — выбираем по-умолчанию командой sudo port select —set gcc mp-gcc48.

3. Установка crosstool-ng

Собирать сам инструментарий достаточно просто, воспользуемся официальной инструкцией:

hg clone http://crosstool-ng.org/hg/crosstool-ng
cd crosstool-ng
./bootstrap

Вынужден прервать официальную инструкцию: в файле kconfig/zconf.hash.c не хватает строчки

#include <stddef.h>

./configure 
--with-objcopy=/opt/local/bin/gobjcopy 
--with-objdump=/opt/local/bin/gobjdump 
--with-readelf=/opt/local/bin/greadelf 
--with-sed=/opt/local/bin/gsed 
--with-libtool=/opt/local/bin/glibtool 
--with-grep=/opt/local/bin/grep 
--with-install=/opt/local/bin/ginstall
make
sudo make install

Это установит ct-ng в /usr/local/bin. Домашняя директория ct-ng: /usr/local/lib/ct-ng.hg+default-2685dfa9de14 в зависимости от ревизии. В этой директории отредактируем файл scripts/functions, заменив строчку для Darwin строчкой от Linux:

--- scripts/functions.orig  2013-07-09 22:15:12.000000000 +0400
+++ scripts/functions   2013-07-06 04:28:41.000000000 +0400
@@ -460,7 +460,7 @@
                 mode="$(stat -c '%a' "$(dirname "${dir}")")"
                 ;;
             Darwin|*BSD)
-                mode="$(stat -f '%Lp' "$(dirname "${dir}")")"
+                mode="$(stat -c '%a' "$(dirname "${dir}")")"
                 ;;
             *)
                 CT_Abort "Unhandled host OS $CT_SYS_OS"

Это потому, что ct-ng случайно хавает версию gstat из GNU-набора, вместо оригинального stat из OS X. Полюбуйтесь красотой и изящностью здешнего кода и закройте файл.

Можете также скопировать скрипт

ct-ng.comp для bash-completion, работает хорошо.
Теперь нужно выбрать временную папку, в которой будут коваться наш замечательный cross-toolchain и его sysroot. У меня это /Volumes/Unixen/ct-config, перейдите в вашу папку и начнём настройку.

Настройка crosstool-ng

Прежде чем начать настройку, унаследуем конфигурацию от шаблона. Нас интересует x86_64-unknown-linux-gnu:

cd /Volumes/Unixen/ct-config
ct-ng x86_64-unknown-linux-gnu
ct-ng menuconfig

После этого вы видите меню, в котором мы будем настраивать наш инструментарий.

1. Paths and misc options

Здесь важно указать опции Local tarballs directory (/Volumes/Unixen/src) и Prefix directory (/Volumes/Unixen/${CT_TARGET}), не забудьте создать /Volumes/Unixen/src.

2. C compiler

Здесь я отключил поддержку Java и Fortran, потому что не знаю, как поведёт себя GCC при сборке с включёнными фичами. Обязательно отключаем [ ] Link libstdc++ statically into the gcc binary, иначе будет ошибка

collect2: error: ld returned 1 exit status
ld: library not found for -lcrt0.o 

3. Debug facilities

Здесь придётся отключить поддержку dmalloc и ltrace, так как иначе будут нерешаемые проблемы. В разделе gdb следует отключить [ ] Native gdb и, если нет необходимости, отключить [*] Enable python scripting (проблема с python, но решение будет ниже). Я использую версию gdb version (7.3.1).

4. Companion libraries

Здесь строго следующие версии библиотек, иначе будут ошибки компиляции и autotools, эти версии подбирал методом тыка, зачастую помогал выбор более свежей.

      * GMP version (5.0.2)
      * MPFR version (3.1.2)
      * PPL version (0.11.2)
      * CLooG version (0.15.11)
      * MPC version (1.0.1)

Сборка toolchain

Практически всё готово. В процессе сборки может возникнуть следующая ошибка (версия ядра указана моя):

[ERROR]    /Volumes/Unixen/ct-config/.build/src/linux-3.8.11/arch/x86/tools/relocs.c:8:17: fatal error: elf.h: No such file or directory

Поэтому заранее позаботимся об этом, взяв elf.h из доверенного источника. Если под рукой нет, возьмите мой elf.h. Класть нужно в /usr/include.

Ещё нужно подправить лимит открытых файлов (RE: Libc iconvdata compilation problem):

ulimit -n 1024

Если во время сборки у вас возникнет ошибка в gdb, если вы не отключили [*] Enable python scripting раньше:

configure: error: python is missing or unusable

то отредактируйте файл .build/gdb-*/gdb/python/python-config.py, закомментировав строки

#            if getvar('LINKFORSHARED') is not None:
#                libs.extend(getvar('LINKFORSHARED').split())

Кажется, всё готово.

ct-ng build

На Macbook Air с i5 сборка идёт порядка 69 минут, причём у вас скорее всего в середине всплывут какие-нибудь ошибки. Так что отойти далеко от компьютера не получится.

Заключение

/Volumes/Unixen $ du -csh ct-config/

4,0G    ct-config/
4,0G    итого

/Volumes/Unixen $ x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu-gcc -v

Using built-in specs.
COLLECT_GCC=x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/Volumes/Unixen/x86_64-unknown-linux-gnu/libexec/gcc/x86_64-unknown-linux-gnu/4.7.3/lto-wrapper
Configured with: /Volumes/Unixen/ct-config/.build/src/gcc-4.7.3/configure --build=x86_64-build_apple-darwin12.4.0 --host=x86_64-build_apple-darwin12.4.0 --target=x86_64-unknown-linux-gnu --prefix=/Volumes/Unixen/x86_64-unknown-linux-gnu --with-sysroot=/Volumes/Unixen/x86_64-unknown-linux-gnu/x86_64-unknown-linux-gnu/sysroot --enable-languages=c,c++ .....
Target: x86_64-unknown-linux-gnu
Thread model: posix
gcc version 4.7.3 (crosstool-NG hg+default-2685dfa9de14) 

Можно попробовать что-нибудь собрать:

cd linux-source-3.9
export PATH=$PATH:/Volumes/Unixen/x86_64-unknown-linux-gnu/bin
export C_INCLUDE_PATH=/usr/include:/opt/local/include:/Volumes/Unixen/x86_64-unknown-linux-gnu/x86_64-unknown-linux-gnu/sysroot/usr/include
make ARCH=x86_64 CROSS_COMPILE=x86_64-unknown-linux-gnu- all -j4

Успехов,
Xlab was there.

< Gt r = “12”/>: brew maintainer < gt r = “10”/> needs to merge upstream, and from the end of the article, it is obviously impossible to merge. My PR will not be lost there, just download the patch from < gt r = “11”/> and apply it yourself.

Python is faith. Versions are also part of faith.

Just kidding. The reason why I have to compile it myself is because a gdb plugin does not support Py2.x. What’s more, 3 is more to my taste than 2 in some details. The brew formula only supports the optional link brew version of python2, and there is no 3, so I have to study it myself.

In fact, what surprised me the most is that some mainstream distributions (such as Ubuntu) have linked gdb to python3, but no one has researched it for several years, no one has mentioned it, and I can’t find it online — when will the mac be so diao?

The version used at the time of writing this article is < gt r = “13”/>.

Check wrong

If you don’t want to see it, you can skip to the next chapter on how to fix the error and compile successfully.

I first checked the formula with < gt r = “14”/>, and found that it specifies < gt r = “15”/> to implement the python linked to brew. I tried to add a 3 at the end. Report such an error.

 checking whether to use python... /usr/local/opt/python3
 checking for python2.7... no
 checking for python2.6... no
 checking for python2.5... no
 checking for python2.4... no
 configure: error: no usable python found at /usr/local/opt/python3
 make[1]: *** [configure-gdb] Error 1
 make: *** [all] Error 2

Tell me that py3 is not available. This error is generated during configure, and configure is generated with autoconf [^ 1]. So I searched for “no usable python” in configure.ac script and located it in the following position.

  if test "${have_libpython}" = no; then
    case "${with_python}" in
    yes)
      AC_MSG_ERROR([python is missing or unusable])
      ;;
    auto)
      AC_MSG_WARN([python is missing or unusable; some features may be unavailable.])
      ;;
    *)
      AC_MSG_ERROR([no usable python found at ${with_python}])
      ;;
    esac

Since < gt r = “16”/> is the parameter we just specified, then < gt r = “17”/> is the available python path detected by the system.

Scroll up a little further and find that the location where this variable is set is on line 895, as a test result to test whether the compilation parameters obtained from python-config are available,

    case "${python_version}" in
    python*)
      AC_TRY_LIBPYTHON(${python_version}, have_libpython,
                       ${python_includes}, ${python_libs})
      ;;

The source of the compile parameters is on line 838 of the configure.ac.

    python_includes=`${python_prog} ${srcdir}/python/python-config.py --includes`
    python_libs=`${python_prog} ${srcdir}/python/python-config.py --ldflags`
    python_prefix=`${python_prog} ${srcdir}/python/python-config.py --exec-prefix`

Comparing its results with the original flags, it is still a little different.

$ python3 python/python-config.py --ldflags
-L/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin -ldl -framework CoreFoundation -lpython3.6m -Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.6/Python

$ python3-config --ldflags
-L/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin -lpython3.6m -ldl -framework CoreFoundation
# 下面是ubuntu的输出
~$ python3-config --ldflags
-L/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -L/usr/lib -lpython3.5m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

Forcibly replace this python-config with python’s own, compile it, and just give gdb’s formula < gt r = “18”/>. Later, I compared the formula written by gdb itself with the official python-config of python. Except for the details, most of them are similar. There is only one sentence.

@@ -72,7 +72,7 @@ for opt in opt_flags:
                     libs.insert(0, '-L' + getvar('LIBPL'))
                 elif os.name == 'nt':
                     libs.insert(0, '-L' + sysconfig.PREFIX + '/libs')
-            if getvar('LINKFORSHARED') is not None:
+            if not getvar('PYTHONFRAMEWORK'):
                 libs.extend(getvar('LINKFORSHARED').split())
         print (to_unix_path(' '.join(libs)))

There is also a patch similar to [^ 2] on the Internet. Just delete these 10 and 11 lines. I don’t think it’s appropriate to not follow the official one.

Traceability

In fact, I am more interested in why two redundant codes are necessary to implement the same function? Why don’t you trust official tools? I have long heard that many distributions have already released gdb with python3 integration. The brew maintainer said: Wait for the upstream update. However, after so many years, the default is still python2.

So I came down from ppa and researched the source code of the ubuntu version of gdb, and found that in addition to src, there is also a folder patch. There is a python-config.patch in it, which is written in much the same way as my patch. (Surprise! The pioneer of history!) I went down to changelog [^ 3] to see which version added python3 support.

+  * debian/patches: Update for new upstream version
+    - Remove patches subsumed by new upstream version/tarball:
+      + ppc64le.diff
+      + hurd-new-RPC-reply-stub-functions.patch
+      + hurd-adapt-to-changed-MIG-output.patch
+      + hurd-adjust-to-startup-with-shell-changes.patch
+      + hurd-make-MIG-output-parsing-more-robust.patch
+      + kfreebsd_bug752390.diff
+      + restore-run.1.patch
+    - Adjust patches to work with new upstream version:
+      + gdb-fortran-main.patch: Update for upstream changes
+      + python-config.patch: Remove parts applied upstream
+
+  [ Héctor Orón Martínez ]
+  * Switch gdb to python3 (Closes: #752581)
+    + Add gdb-python2 package for use with python2-only scripts
+  * Enable babeltrace on supported arches
+
+  [ Samuel Bronson ]
+  * Make gdb-python2 provide gdb.
+  * Totally rewrite the NEWS entry about the Python changes.
+    + And split out most of the new text to README.python_switch.
+
+ -- Samuel Bronson <naesten@gmail.com>  Tue, 26 Aug 2014 15:34:03 -0400
+
 gdb (7.8-0ubuntu2) utopic; urgency=medium

Version 7.8-0ubuntu2. Remarks still: Modified the patch that was merged upstream. But why is it still .patch when merged upstream? Take a look at the beginning: debian/patches. 😯… Dare you have big eyebrows and big eyes, the code is still third-hand?

Let’s look at the debian code again, they have a dedicated < gt r = “19”/> service, from there you can find patches written by all debian contributors. So, the linux boss is not ubuntu, and in the end, don’t you still have to look at debian’s face? Here I only found the patches contained in a few release versions. The earliest version to add this patch is < gt r = “20”/>, you can see the time: 2013-08-14. It’s a year earlier than Ubuntu. It seems that Ubuntu is not very fast and bleeding-edge!

So what have gdb developers been doing all these years? In fact, they know faster than Hong Kong journalists. As early as December 2012, five days before Christmas, an ubuntu developer submitted a similar patch for gdb-7.5 on the mailing list. However! Why hasn’t this been merged? Check out the conversation on the mailing list [^ 4]:

From: Matthias Klose <doko at ubuntu dot com>
Date: Thu, 20 Dec 2012 17:13:14 +0100

Am 20.12.2012 16:59, schrieb Joel Brobecker:
> Can you tell us what problem you are trying to solve?

cross building in a multiarch environment, where the host libraries are
installed, and a <host>-python-config is shipped as part of the system python.
This <host>-python-config is a backport of http://bugs.python.org/issue16235
(not yet applied upstream).

==========
From: Joel Brobecker <brobecker at adacore dot com>
Date: Fri, 21 Dec 2012 11:57:21 +0400
I think that this was the plan, yes; but Doug Evans would probably
know more, as he is the main author of this code (I only fiddled
with it).

==========
From: Doug Evans <dje at google dot com>
Date: Fri, 21 Dec 2012 09:30:47 -0800

The patch is fine with me.
[The plan wasn't to shift and call the "real" python-config ... at the
time there was no real python-config for cross builds.]

==========
From: Joel Brobecker <brobecker at adacore dot com>
Date: Fri, 21 Dec 2012 21:50:59 +0400
Can we make this new behavior optional? The "real" python-config
is broken for several platforms, and having our own that we can
control was very convenient.

==========
From: Doug Evans <dje at google dot com>
Date: Sat, 22 Dec 2012 11:33:08 -0800
I certainly don't want to break your use-case.

I need to look at the patch again (but I probably won't get to it
until next week).

He shrank and waited for Lao Tzu to go to work on Monday. Note the time: Sat, 22 Dec. Well, Saturday. I’m counting, the 23rd, Sunday. 24! 25! 26! Christmas, three days, small, long, fake!

So in a happy and peaceful atmosphere, the bug was so forgotten (´･_･`)

Repair

By the way, there seems to be a sentence I didn’t say at the beginning… The bugs and fixes encountered in this article are only for the mac system (pro test 10.12.6). If you have been installing software with apt-get, it has already helped you Wipe your butt clean. What’s more, the python-config of mac is a python script, while the one under ubuntu is a shell script, and the code is generated, so even if you really want to change it, it is not a solution.

Put the following patch into a text file (eg < gt r = “21”/>), then put this file in the gdb source directory and run < gt r = “22”/>. Then compile normally. This file has not been changed in many versions, so it probably works.

diff --git a/gdb/python/python-config.py b/gdb/python/python-config.py
index c2b2969..554dac9 100644
--- a/gdb/python/python-config.py
+++ b/gdb/python/python-config.py
@@ -72,7 +72,7 @@ for opt in opt_flags:
                     libs.insert(0, '-L' + getvar('LIBPL'))
                 elif os.name == 'nt':
                     libs.insert(0, '-L' + sysconfig.PREFIX + '/libs')
-            if getvar('LINKFORSHARED') is not None:
+            if not getvar('PYTHONFRAMEWORK'):
                 libs.extend(getvar('LINKFORSHARED').split())
         print (to_unix_path(' '.join(libs)))

References