Introduction
Background
My colleagues have a good contour line data in ArcGIS geodatabase (.gdb) format. I wanted to make vector tiles from contour lines stored in ArcGIS Geodatabase. The structure of the geodatabase is as shown in the following figure.
Considering an approach
At first, I was thinking to export it to shape files to be converted into GeoJSONs which is the input format of the Tippecanoe (mapbox's vector tile conversion tool). However, the source data is more than 1GB, and I wanted to avoid exporting any intermediate file.
I explored a way to export data as GeoJSON from geodatabase, and found that the gdal (ogr2ogr) provide such function to export the data into GeoJSON/GeoJSONs from geodatabase. And, fortunately, I know that there is a way to efficiently export data from the source to be converted into vector tile as an asynchronous process by using nodejs "spawn" and "pipe." (Ref: this article (in Japanese) by @hfu).
Therefore, my goal is to efficiently create vector tile from Esri geodatabase using gdal, nodejs (in particular, spawn and pipe), and tippecanoe.
My Working Environment
- nodejs: v16.15.0
- npm: 8.5.5
- tippecanoe: v1.36.0
- GDAL: 3.4.1, released 2021/12/27
- Platform: Ubuntu 22.04 LTS (built on Docker for windows)
Working Repository
Procedure
Step 1: Confirm gdal (ogr2ogr) command
The data is stored in a directory. At first, I have checked if gdal (ogr2ogr) command work well.
I can see the exported data as stdout with the following command. Fortunately, the data structure is simple and I did not have to specify any layer under the geodatabase.
ogr2ogr -f GeoJSONSeq -lco RS=YES /vsistdout/ test_area/test_area.gdb
Then, I was able to see the result.
Step 2: Making scripts for nodejs (downstream is shown as stdout)
Follwoing @hfu's script, I prepared the following scripts (index0.js and default/config.hjson).
For testing purpose, at first, gdal output is spawn and piped into stdout.
const config = require('config')
const Parser = require('json-text-sequence').parser
const { spawn } = require('child_process')
const srcs = config.get('srcs')
const ogr2ogrPath = config.get('ogr2ogrPath')
const downstream = process.stdout
for (const src of srcs) {
const parser = new Parser()
.on('data', f => {
f.tippecanoe = {
layer: src.layer,
minzoom: src.minzoom,
maxizoom: src.maxzoom
}
delete f.properties.SHAPE_Length //SHAPE_Length is not necessary
downstream.write(`\x1e${JSON.stringify(f)}\n`)
})
const ogr2ogr = spawn(ogr2ogrPath, [
'-f', 'GeoJSONSeq',
'-lco', 'RS=YES',
'/vsistdout/',
src.url
])
ogr2ogr.stdout.pipe(parser)
}
{
minzoom: 10
maxzoom: 12
srcs: [
{
url: test_area.gdb
layer: elev
minzoom: 10
maxzoom: 12
}
]
ogr2ogrPath: ogr2ogr
tippecanoePath: tippecanoe
dstDir: zxy
}
Once I ran "index0.js", I saw GeoJSON sequence was exported.
Step 3: preparing the scipt: Downstream into Tippecanoe
Now, gdal result should be piped into tippecanoe. index0.js was extended as below. A const "Tippecanoe" was added, and donwstream is now into tippecanoe.
In addition, it is needed to add "nOpenFiles" to end downstream for each.
const config = require('config')
const Parser = require('json-text-sequence').parser
const { spawn } = require('child_process')
const minzoom = config.get('minzoom')
const maxzoom = config.get('maxzoom')
const srcs = config.get('srcs')
const ogr2ogrPath = config.get('ogr2ogrPath')
const tippecanoePath = config.get('tippecanoePath')
const dstDir = config.get('dstDir')
const tippecanoe = spawn(tippecanoePath, [
`--output-to-directory=${dstDir}`,
`--no-tile-compression`,
`--minimum-zoom=${minzoom}`,
`--maximum-zoom=${maxzoom}`
], { stdio: ['pipe', 'inherit', 'inherit'] })
//const downstream = process.stdout
const downstream = tippecanoe.stdin
let nOpenFiles = 0
for (const src of srcs) {
nOpenFiles++
const parser = new Parser()
.on('data', f => {
f.tippecanoe = {
layer: src.layer,
minzoom: src.minzoom,
maxizoom: src.maxzoom
}
delete f.properties.SHAPE_Length //SHAPE_Length is not necessary
//console.log(JSON.stringify(f, null, 2))
downstream.write(`\x1e${JSON.stringify(f)}\n`)
//downstream.write(`\x1e${JSON.stringify(f.properties)}\n`)
})
.on('finish', () =>{
nOpenFiles--
if (nOpenFiles === 0){
downstream.end()
}
})
const ogr2ogr = spawn(ogr2ogrPath, [
'-f', 'GeoJSONSeq',
'-lco', 'RS=YES',
'/vsistdout/',
src.url
])
ogr2ogr.stdout.pipe(parser)
}
Then, I was able to run the script.
node index.js
It will take some time for the conversion. I am now waiting for the result.
Result
I got zxy vector tile in pbf format. I hosted them with my web server and it can be displayed in our web map.
(You can see contour lines which are from our contour data in Esri gdb file.)
Conclusion
I tried vector tile conversion from Esri geodatabase with open source software.
This time, my test was done with a gdb file that contains extracted features in some part from the original database.
For future, because our original database contains contour lines for whole globe, it would be important to think about exporting by regions.
Features will downstream into GeoJSON sequence from a GDB format, then they are piped into Tippecanoe. But, it would be wise to separate the region during exporting into GeoJSON sequence.
Postgres/PostGIS can do that with its query, but I do not know if Esri Geodatabase has such function (with enough efficiency). I will think more about these issues for the future..
My effort is a part of UN Vector Tile Toolkit activities under the UN Open GIS.
Acknowledgement
This work follows the existing method by @hfu. Although it was developed a few years ago, I think it is still relevant. I appreciate his great work.
(After all, I found that the difference of the source file did not affect his original script.)
Reference