Re: What's the correct best method for setting up ONiE development?

Sriram Ramkrishna


Yep, did do the make distclean although I did hte make without the -j option to make sure there wasn't some race condition. I was reading about how you should run distclean in one of the issues that I saw - so I followed that guideline.

It's a clean clone - I setup DUE and then cloned the ONiE repository and then followed the instructions in DUE and in the instructions for my architecture.

I decided to follow your instructions precisely and so the end result of the build of using the ! make -j1 MACHINE-kvm_x86_64 all demo is this (note the breakage is still in the cross compiler):

[INFO ]  =================================================================
[INFO ]  Installing ISL for host
[ERROR]    make[4]: *** [Makefile:1397: isl_scheduler.lo] Error 1
[ERROR]    make[4]: *** Waiting for unfinished jobs....
[ERROR]    make[4]: *** [Makefile:1397: isl_sort.lo] Error 1
[ERROR]    make[4]: *** [Makefile:1397: isl_space.lo] Error 1
[ERROR]    make[3]: *** [Makefile:1505: all-recursive] Error 1
[ERROR]    make[2]: *** [Makefile:1110: all] Error 2
[ERROR]  >>
[ERROR]  >>  Build failed in step 'Installing ISL for host'
[ERROR]  >>        called in step '(top-level)'
[ERROR]  >>
[ERROR]  >>  Error happened in: CT_DoExecLog[scripts/functions@338]
[ERROR]  >>        called from: do_isl_backend[scripts/build/companion_libs/]
[ERROR]  >>        called from: do_isl_for_host[scripts/build/companion_libs/]
[ERROR]  >>        called from: do_companion_libs_for_host[scripts/build/]
[ERROR]  >>        called from: main[scripts/]
[ERROR]  >>
[ERROR]  >>  For more info on this error, look at the file: 'build.log'
[ERROR]  >>  There is a list of known issues, some with workarounds, in:
[ERROR]  >>      '/home/sri/src/onie/build/crosstool-ng/crosstool-ng-1.23.0/docs/B - Known issues.txt'
[ERROR]  >>
[ERROR]  >>  If you feel this is a bug in crosstool-NG, report it at:
[ERROR]  >>
[ERROR]  >>
[ERROR]  >>  Make sure your report includes all the information pertinent to this issue.
[ERROR]  >>  Read the bug reporting guidelines here:
[ERROR]  >>
[ERROR]  (elapsed: 4:25.78)
/home/sri/src/onie/build/crosstool-ng/crosstool-ng-1.23.0/ct-ng:147: recipe for target 'build' failed
make[1]: *** [build] Error 2
make[1]: Leaving directory '/home/sri/src/onie/build/x-tools/x86_64-g6.3.0-lnx4.9.95-uClibc-ng-1.0.22/build'
rm: cannot remove '/home/sri/src/onie/build/x-tools/x86_64-g6.3.0-lnx4.9.95-uClibc-ng-1.0.22/build/.config.2': No such file or directory
make/xtools.make:123: recipe for target '/home/sri/src/onie/build/x-tools/x86_64-g6.3.0-lnx4.9.95-uClibc-ng-1.0.22/stamp/xtools-build' failed
make: *** [/home/sri/src/onie/build/x-tools/x86_64-g6.3.0-lnx4.9.95-uClibc-ng-1.0.22/stamp/xtools-build] Error 1

From: <> on behalf of Alex Doyle via <adoyle@...>
Sent: Wednesday, February 10, 2021 8:00 AM
To: <>
Subject: Re: [OCP-ONIE] What's the correct best method for setting up ONiE development?
Hi Sri
 Hm. Usually it rolls right through there, so I'm thinking it could be an environmental configuration or a code modification issue (and the first one would be particularly irksome as that is the sort of problem DUE is supposed to fix...).

 Your invocation looks correct, so I'm seeing no obvious reason for a failure here so we'll have to poke around a bit. A couple of thoughts, which may seem obvious but it's helpful for establishing a baseline...

Try running:
 make distclean ; make -jMACHINE=kvm_x86_64 all demo

The make distclean should clear out everything and re download components. If you'd had a previous compile fail somewhere the makefile logic might not be catching it and think that it had completed successfully.  I've seen this happen if I interrupt a build, then try to build again.

Note the -j1 there - it'll take longer to build, but it should be harder for errors to be hidden in the log output.

Other things to consider:

I've occasionally seen what looked like race conditions in parallel make, but this happens rarely for me, and I've never been able to reproduce them, as a make distclean and a rebuild usually works.

Is it an unmodified check out of ONIE? i.e. "git reset --hard" changes nothing? (I'd expect that since it is a tools build problem, but I've been surprised before)

Does your $PATH in DUE have anything unusual in it? (I've seen PATH settings trip up builds by accessing different component versions)
 A working one for ONIE would be (lifted from my current ONIE contiainer):

Is DUE running on the Debian 9 system you'd mentioned,  or Red Hat? Red Hat support is new, and while it should​ be the same by the time the containers are running there were a couple of differences in how it got set up vs Debian.

If none of that helps, can you post the output of a:
dpkg -l
...which will let me check the container component versions against what I'm using, and the full output of the -j1 build I'd suggested above

Let me know how it goes  - I'll roll whatever we find into the documentation to keep the next person through from getting burned 🙂


Alex Doyle
ONIE Project Lead

From: <> on behalf of Sriram Ramkrishna <sriram.ramkrishna@...>
Sent: Wednesday, February 10, 2021 5:02 AM
To: <>
Subject: Re: [OCP-ONIE] What's the correct best method for setting up ONiE development?
External email: Use caution opening links or attachments

Hi Alex! Thanks for your quick response!

I am in fact using DUE and I followed all the instructions that you so kindly documented below. My real problem is what happens after: 

make -j4 MACHINE=kvm_x86_64 all demo

Inside the debian9 container that DUE created. (lovely work btw) The cross compiler fails to build which is what I had even when I didn't use DUE. So I'm trying to debug the cross compiler issues. Basically this is the last bit:

/home/sri/src/onie/build/crosstool-ng/crosstool-ng-1.23.0/ct-ng:147: recipe for target 'build' failed

I had this with and without DUE on a Debian 9 system. Having exhausted other means of debugging I thought I would ask on the mailing list. 🙂

Here is the pastebin for that error:

Would love to hear your thoughts.


From: <> on behalf of Alex Doyle via <adoyle@...>
Sent: Tuesday, February 9, 2021 4:04 PM
To: <>
Subject: Re: [OCP-ONIE] What's the correct best method for setting up ONiE development?
Hi Sri,
I really recommend DUE to set up the build environment using Docker containers for a couple of reasons:
  1. The standard configuration it applies is the same one I'm using for processing pull requests, development, and quarterly releases, so it should consistently provide the correct versions of everything needed to build.
  2. If there are problems, it provides a common frame of reference to debug them.
  3. And I'm using it for ONIE work, so any problems there get my attention pretty quick.
I'm getting the impression there are some weak points in the documentation for setting it up - yours isn't the first email asking setup questions - so if you can identify the things that are tripping you up, do let me know so I can make it clearer ( at this point I've been looking at things long enough to not see things that might be unclear to a set of fresh eyes. ) Hm. In writing out the rest of this, I'm thinking this 'quickstart' info needs to be more prominently featured in the documentation...

That aside, here's some actionable information:

So, DUE getting started:
  1. wget  <-get latest release
  2. sudo dpkg -i due_2.3.0-1.all.deb  <- install it. Will throw missing dependency errors, that's fine
  3. sudo apt-get install --fix-broken    <- use APT to fix the missing dependencies. Will want to install Docker if you don't have it.

Then follow:

To use DUE to create an ONIE container. I'd recommend using the Debian 9 version (10 only works with the component-upgrade-2020 branch, which is not heavily tested yet, and probably breaks existing platforms at this point.)

./due --create --from debian:9 --description "ONIE Build Debian 9" --name onie-build-debian-9 --prompt ONIE-9 --tag onie --use-template onie

due --run 
...will let you select a container, or if there's only one, it'll run it and mount your home directory, at which point you should be able to build ONIE.

I've also got a tutorial video that should be mostly correct here:

It's a number of DUE versions out of date, but I think the workflow still holds.

I'd also suggest starting by trying to build the kvm target
 (although you'll also need to install qemu if you want to run it. There is some help here:,
and if that doesn't pan out let me know. I've got an upgraded version of that script that I'm working on because I can never remember what arguments qemu needs....)
  1. cd ./build-config
  2.  make -j4 MACHINE=kvm_x86_64 all demo

Let me know how it goes, and what things aren't clear in the documentation so I can get them straightened out.

Alex Doyle
ONIE Project Lead

From: <> on behalf of Sriram Ramkrishna <sriram.ramkrishna@...>
Sent: Tuesday, February 9, 2021 2:50 PM
To: <>
Subject: [OCP-ONIE] What's the correct best method for setting up ONiE development?
External email: Use caution opening links or attachments

I'm working on enabling a platform that was enabled before (Facebook Wedge 100) but has some issues. I'm trying to build the environment, but I'm currently having a lot of issues getting the cross compiler to get compiled. I've tried solutions like DUE - but I'm still having the same issues.

If there is a blog post, documentation, or video that I can look at - that would be very appreciated.


Join to automatically receive all group messages.