Vector tile conversion from Esri geodatabase with gdal, tippecanoe, and nodejs

Introduction

Background

My colleagues have a good contour line data in ArcGIS geodatabase (.gdb) format. I wanted to make vector tiles from contour lines stored in ArcGIS Geodatabase. The structure of the geodatabase is as shown in the following figure.

Considering an approach

At first, I was thinking to export it to shape files to be converted into GeoJSONs which is the input format of the Tippecanoe (mapbox's vector tile conversion tool). However, the source data is more than 1GB, and I wanted to avoid exporting any intermediate file.

I explored a way to export data as GeoJSON from geodatabase, and found that the gdal (ogr2ogr) provide such function to export the data into GeoJSON/GeoJSONs from geodatabase. And, fortunately, I know that there is a way to efficiently export data from the source to be converted into vector tile as an asynchronous process by using nodejs "spawn" and "pipe." (Ref: this article (in Japanese) by @hfu).

Therefore, my goal is to efficiently create vector tile from Esri geodatabase using gdal, nodejs (in particular, spawn and pipe), and tippecanoe.

My Working Environment

• nodejs: v16.15.0
• npm: 8.5.5
• tippecanoe: v1.36.0
• GDAL: 3.4.1, released 2021/12/27
• Platform: Ubuntu 22.04 LTS (built on Docker for windows)

Working Repository

Procedure

Step 1: Confirm gdal (ogr2ogr) command

The data is stored in a directory. At first, I have checked if gdal (ogr2ogr) command work well.

I can see the exported data as stdout with the following command. Fortunately, the data structure is simple and I did not have to specify any layer under the geodatabase.

ogr2ogr -f GeoJSONSeq -lco RS=YES /vsistdout/ test_area/test_area.gdb

Then, I was able to see the result.

Step 2: Making scripts for nodejs (downstream is shown as stdout)

Follwoing @hfu's script, I prepared the following scripts (index0.js and default/config.hjson).
For testing purpose, at first, gdal output is spawn and piped into stdout.

index0.js

const config = require('config')
const Parser = require('json-text-sequence').parser
const { spawn } = require('child_process')

const srcs = config.get('srcs')
const ogr2ogrPath = config.get('ogr2ogrPath')

const downstream = process.stdout

for (const src of srcs) {
    const parser = new Parser()
      .on('data', f => {
        f.tippecanoe = {
            layer: src.layer,
            minzoom: src.minzoom,
            maxizoom: src.maxzoom
        }
        delete f.properties.SHAPE_Length //SHAPE_Length is not necessary
        downstream.write(`\x1e${JSON.stringify(f)}\n`)
      })
    const ogr2ogr = spawn(ogr2ogrPath, [
      '-f', 'GeoJSONSeq',
      '-lco', 'RS=YES',
      '/vsistdout/',
      src.url
    ])
    ogr2ogr.stdout.pipe(parser)
  }

config/default.hjson

{
    minzoom: 10
    maxzoom: 12
    srcs: [
        {
        url: test_area.gdb
        layer: elev
        minzoom: 10
        maxzoom: 12
        }
    ]
    ogr2ogrPath: ogr2ogr
    tippecanoePath: tippecanoe
    dstDir: zxy
}

Once I ran "index0.js", I saw GeoJSON sequence was exported.

Step 3: preparing the scipt: Downstream into Tippecanoe

Now, gdal result should be piped into tippecanoe. index0.js was extended as below. A const "Tippecanoe" was added, and donwstream is now into tippecanoe.

In addition, it is needed to add "nOpenFiles" to end downstream for each.

index.js

const config = require('config')
const Parser = require('json-text-sequence').parser
const { spawn } = require('child_process')

const minzoom = config.get('minzoom')
const maxzoom = config.get('maxzoom')
const srcs = config.get('srcs')
const ogr2ogrPath = config.get('ogr2ogrPath')
const tippecanoePath = config.get('tippecanoePath')
const dstDir = config.get('dstDir')

const tippecanoe = spawn(tippecanoePath, [
    `--output-to-directory=${dstDir}`,
    `--no-tile-compression`,
    `--minimum-zoom=${minzoom}`,
    `--maximum-zoom=${maxzoom}`
  ], { stdio: ['pipe', 'inherit', 'inherit'] })

//const downstream = process.stdout
const downstream = tippecanoe.stdin

let nOpenFiles = 0

for (const src of srcs) {
    nOpenFiles++
    const parser = new Parser()
      .on('data', f => {
        f.tippecanoe = {
            layer: src.layer,
            minzoom: src.minzoom,
            maxizoom: src.maxzoom
        }
        delete f.properties.SHAPE_Length //SHAPE_Length is not necessary
        //console.log(JSON.stringify(f, null, 2))
        downstream.write(`\x1e${JSON.stringify(f)}\n`)
        //downstream.write(`\x1e${JSON.stringify(f.properties)}\n`)
      })
      .on('finish', () =>{
        nOpenFiles--
        if (nOpenFiles === 0){
            downstream.end()
        }
      })
    const ogr2ogr = spawn(ogr2ogrPath, [
      '-f', 'GeoJSONSeq',
      '-lco', 'RS=YES',
      '/vsistdout/',
      src.url
    ])
    ogr2ogr.stdout.pipe(parser)
  }

Then, I was able to run the script.

node index.js

It will take some time for the conversion. I am now waiting for the result.

Result

I got zxy vector tile in pbf format. I hosted them with my web server and it can be displayed in our web map.
(You can see contour lines which are from our contour data in Esri gdb file.)

Conclusion

I tried vector tile conversion from Esri geodatabase with open source software.

This time, my test was done with a gdb file that contains extracted features in some part from the original database.
For future, because our original database contains contour lines for whole globe, it would be important to think about exporting by regions.

Features will downstream into GeoJSON sequence from a GDB format, then they are piped into Tippecanoe. But, it would be wise to separate the region during exporting into GeoJSON sequence.
Postgres/PostGIS can do that with its query, but I do not know if Esri Geodatabase has such function (with enough efficiency). I will think more about these issues for the future..

My effort is a part of UN Vector Tile Toolkit activities under the UN Open GIS.

Acknowledgement

This work follows the existing method by @hfu. Although it was developed a few years ago, I think it is still relevant. I appreciate his great work.
(After all, I found that the difference of the source file did not affect his original script.)

Reference