Checking a running process in WebODM

Edit: the dreaded “Reconstructing all views” message has been replaced with a progress monitor! But, how to dig into the back end and explore the machine that does the work is always a helpful skill to have… .

Learning objectives:

We’ll learn how to check how far along a process is when it is calculating depthmaps and giving no feedback
Along the way, we’ll also learn how to look at the docker images running for WebODM and
login to said docker images in order to
inspect that status of processing data.
Let’s go!

So, you threw a really big dataset at WebODM, and now you are waiting. It’s been hours, maybe days, and it’s stuck on the dreaded “Reconstructing all views”:

Did you make the depthmap resolution really high, because you wanted really detailed data? Did you make it too high? How long is this going to take?

I have had this dilemma too. Sometimes I just get disgusted with myself and my predilection for choosing ridiculously computationally expensive settings, kill the process, turn the settings down a bit, and restart. But this can waste hours or days of processing, which feels wrong.

The alternative

We could do the alternative. We could poke under the hood of WebODM and see how it’s progressing. For this project that’s been running for 146+ hours, this is what I decided to do.

The depthmaps that are running are being done in MVE, which can give us really detailed information about progress, but unfortunately, it makes logs a real mess, so we have it logging nothing. Let’s see how we can get around this and check in our status.

Finding the docker instance and logging in:

First, we log into the machine where WebODM is running. We need a list of the existing docker machines, as we need access to the correct machine to look at how things are progressing.

docker ps

The result should give us something like this:

CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                                         NAMES
4b0659fe6761        opendronemap/webodm_webapp   "/bin/bash -c 'chmod…"   38 hours ago        Up 38 hours         0.0.0.0:443->8000/tcp, 0.0.0.0:80->8080/tcp   webapp
0e26ebf918f2        opendronemap/webodm_webapp   "/bin/bash -c '/webo…"   38 hours ago        Up 38 hours                                                       worker
1954c5136d44        redis                        "docker-entrypoint.s…"   38 hours ago        Up 38 hours         6379/tcp                                      broker
bdc69502ca50        opendronemap/webodm_db       "docker-entrypoint.s…"   38 hours ago        Up 38 hours         0.0.0.0:32769->5432/tcp                       db
81f401a0e138        opendronemap/nodeodm         "/usr/bin/nodejs /va…"   38 hours ago        Up 38 hours         0.0.0.0:3000->3000/tcp                        webodm_node-odm_1

We want to access the webodm_node-odm_1 node, in most cases. To do this we use docker exec as follows:

docker exec -it webodm_node-odm_1 bash
root@81f401a0e138:/var/www#

Woah! We are now inside the node!

Checking the available data directories:

Typically, if we only have one process running, there will only be one dataset in the /var/www/data directory

cd /var/www/data/99002823-c48b-4af5-af1b-c0fef2ed8b56/

Checking our depthmap data from MVE:

For depthmaps nearly complete in MVE, there will be a file called depth-L1.mvei. We need to find out how many these are as compared with the number of images that we need depthmaps for. We’ll use a combination of the find command and wc (or word count):

find . -name depth-L1.mvei | wc -l

In my case, I have 2,485 images, or roughly 2/3s of my images processed. Looks like I am 6 days into a 9 day process before we get done with the MVE step.

I guess I will wait until Monday to check again… .