The road to run everywhere: Native WebODM on Windows
One of the cool things about open source software is that it has a tendency to eventually run everywhere. If you have the source code available and certain constraints allow it, with sufficient time a port can be made to different platforms. This is not always the case with proprietary software, where the vendor decides which platforms to support.
In the beginning, ODM could only run on Ubuntu Linux. Later, we opted to leverage docker, which is an amazing (yet confusing) technology we use to bring many of the OpenDroneMap tools to other platforms such as MacOS and Windows. The docker abstraction allows us to focus on building the software without worrying about the platforms. If you can install docker, you can run OpenDroneMap software. Simple. But not without issues. Especially on Windows, docker has several quirks.
We initially delayed efforts to make a native port to Windows for two main reasons:
- To focus developer time on quality of outputs, speed and features.
- Previous attempts at making a Windows native port (2 of them across 3 years) failed to yield a working prototype (and lots of lessons learned).
Third time is the charm. I’m pleased to announce that starting today, you can run ODM, NodeODM and WebODM entirely natively on Windows. Even without docker. So where/how can you install it?
- ODM (command line): download the setup from the releases page: https://github.com/OpenDroneMap/ODM/releases
- NodeODM (API): download the Windows bundle from the releases page: https://github.com/OpenDroneMap/NodeODM/releases
- WebODM (user interface): see the downloads page. We currently ask that people make a contribution and purchase the pre-built native Windows installer for WebODM, to support the project. All changes required to run WebODM natively, including the recent Desktop Mode support, are already on GitHub (but you will need to do some work).
This was no easy feat. Let me tell you a bit about the challenges we encountered:
- ODM has a several primary dependencies (OpenCV, OpenSfM, mvs-texturing, PoissonRecon, PDAL, untwine, entwine, GRASS, OpenMVS, dem2mesh, etc.). These dependencies in turn have in some cases lots of other dependencies.
- Most of these dependencies were/are written with non-Windows systems in mind.
- Path handling requires special care on Windows. For example, you cannot create symbolic links on Windows, unless you have administrative permissions (that’s a no go). Paths have backward slashes instead of forward slashes. Single quotes are no good for wrapping paths with spaces in Windows shell commands and functions such as pipes.quote in Python will use single quotes. CTRL+C will prompt the user for a confirmation to terminate a process (instead of just terminating the process), which will hang forever in headless processes like NodeODM.
- Building Python packages from source code (which ODM requires) leaves several hard-coded paths in our packages, making deployment to other machines tricky.
- Python virtual environments weren’t meant to be used for deployment (they also have hard-coded paths). But ODM needs a dedicated Python environment to run.
On the plus side, we had already proven last year that the user interface could work natively on Windows:
So all that remained was to get the ODM and NodeODM components to work too.
- We set off to build each major dependency (and contributed several patches to other upstream FOSS projects in the process). We used a tool from Microsoft, VCPKG, to help us build several secondary dependencies for the projects that needed them. This was probably the hardest and most time consuming task. Libraries missing headers, architecture set instructions problems, pre-processor define issues, runtime dependencies not being found, path resolution issues, CMake variable propagation troubles. The list goes on. But divide and conquer, lots of coffee and one by one, we managed to build them all (the entire saga can be read here, thanks to all the community members that followed the process and gave encouragement along the way).
- We refactored/cleaned up the directory structure and eliminated the build directory in the process, allowing us to keep all of our dependencies in a single SuperBuild/install directory. This made it easier to package the software in subsequent steps.
- We fought the idiosyncrasies of Windows Batch scripting and the Windows shell with some clever tricks to enable a user to simply run: run d:\my\dataset from a command prompt (the ODM Console) and allow to cancel it via a CTRL+C.
- We hot-patch several hard-coded paths as part of the ODM startup process to make sure our Python modules work as expected.
- We rewrote the cutline computation algorithm in Python instead of using GRASS, which allowed us to shed some space requirements (and improve performance).
- We wrote an installer script using InnoSetup (another awesome FOSS) and we got pretty clever to automatically sign the final installer with a code signing certificate (otherwise Windows and other AV software would complain).
- Lots of testing, which yielded its own set of problems, like undefined behavior in MVE/libjpeg causing a file descriptor leak due to C/C++ exception throwing, which were resolved after some debugging.
- With ODM working, we turned the page to NodeODM. Luckily, NodeODM is a relatively simple piece of software and is written in portable NodeJS. Save a few minor and not so minor issues, we managed to use nexe to generate a portable executable for Windows. Here we found out that Windows exit codes do not propagate well unless you explicitly call exit on every Batch script (but that breaks console-based workflows), so we wrote some clever workarounds.
- With NodeODM and ODM both working, the path to full Windows native WebODM was finally possible. Using the same technique that we explored last year, we generated a single executable setup that installs everything that you need. We finally improved the Desktop Mode capability of WebODM to give a more native feel to the application, which remains a web-based application running under an Electron wrapper.
- We finally built an automatic updater system to allow people to check for the latest version of the software.
So there you have it: Windows native WebODM.
What does this mean for docker? Nothing. Docker is and will continue to remain a core part of our toolset and deployment mechanism. This is simply another way to run WebODM that we hope you will enjoy.