Introduction

Are you interested in creating vector tile easily with an open source tool? Here, I would like to introduce how we can quickly convert vector tile from GeoJSON with our tool from UN Vector Tile Toolkit (UNVT).

At this demonstration, I will use GeoJSON served from Esri ArcGIS server as an item data, and the conversion is done in the Windows PC, with the support of Docker container (unvt/nanban).

What is UNVT?
UNVT, or United Nations Vector Tile Toolkit, is a collection of Open Source Software for various vector tile processing for better web mapping. UNVT project started in 2018 and it has been developed as a part of the UN Open GIS Initiative which aims to develop an Open Source GIS bundle that meets the requirements of UN operations.

For those who want to know more about UNVT, please refer to my talk at FOSS4G 2021 from here.

Workflow

This time, we will produce vector tile from existing GeoJSON file.

First, we start a Docker container, named unvt/nanban, where some of UNVT and other tools is installed.
Secondly, we will prepare a list of the sources in YAML files.
Then, we run unvt/kata on the Docker container to filter the source and forward the data into the data conversion tool Tippecanoe (note: tippecanoe is from mapbox).

Prerequisites

Source data

This time, we will use a GeoJSON served from ESRI ArcGIS server. There is an item (BNDA_SEN_2008-07-10_lastupdate) from here for testing purpose:
https://geoportal.un.org/arcgis/sharing/rest/content/items/3b628266c28345ecbcac152255eba4be

Then, its GeoJSON data is accessible from the following URL, and you will see a GeoJSON file:
https://geoportal.un.org/arcgis/sharing/rest/content/items/3b628266c28345ecbcac152255eba4be/data
This is the source URL we use.

At first, I tried to see if I can access GeoJSON file through FeatureService (see ArcGIS Rest API for detail) by an ArcGIS server, but I found it difficult to get the GeoJSON file from the FeatureService. Because 1) I could not get all features in a layer while I could get a JSON object of a certain feature, and 2) Esri's json format is not the same with the GeoJSON (Example: A feature from USA Census Tract Areas by Esri ).

Therefore, I concluded I should directly access to a GeoJSON file that is served as an item data from ArcGIS server (https://developers.arcgis.com/rest/users-groups-and-items/item.htm)

My working environment

Windows 10 (Enterprise)
Docker version 20.10.8
PowerShell version 5.1.19041.1237

Repository of this trial

For this work, I have created a repository as below.
https://github.com/ubukawa/kata-test2

Procedure

Step 1: Run a Docker container (unvt/nanban)

Because it is a little difficult to use the tool for vector tile conversion called tippecanoe (by mapbox), we used a Docker container. (I do not explain how we can use Docker here.)

In Powershell, go to the working directly and run unvt/nanban as below. (If necessary, please clone my github repository before you starting.)

Docker run -it --rm -v ${PWD}:/data unvt/nanban
cd /data

Please make sure that the working directly was connected as "/data" in the container. This container already includes various tools, we can type "kata --version" to see if we have the latest unvt/kata tool.

If you see something like above figure, we can go to the next step. (version 1.0.3 is coming soon. But, the npm package unvt/kata has still ver. 1.0.2. You may need to wait for a while if you see version 1.0.2, because we need version 1.0.3.)

Previously, unvt/nanban was developed based on nodejs version 12. If your Docker image is older, please pull it again to upgrade it. (As the nodejs version 12 will come to end of its life at the end of April, unvt/nanban is prepared with nodejs version 16.)

Step 2: Preparing the list of source layers

At the working directly, we create "kata.yml." I have prepared the following file. If you want to use another file, please feel free to do so. You can add more than one layer.

BNDA_SEN_2008:
  source: https://geoportal.un.org/arcgis/sharing/rest/content/items/3b628266c28345ecbcac152255eba4be/data
  minzoom: 1
  maxzoom: 5
## If you want
#layer 2: # this will be a layer name in the vector tile
#  source: directory/data.geojson #both location and URL are fine.
#  minzoom: 0
#  maxzoom: 6

Step 3: Running kata filter to obtain geojson file

Then, let's confirm if we can make a filtered GeoJSON, precisely it is GeoJSONSeq, from the source using kata filter. Try the following command.

kata filter kata.yml

Because you do not specified the output location nor next command, you will see output as a standard output.

After confirming the output, try following command to get it in txt format.

kata filter kata.yml > output.geojson

Step 4: Converting geojson into vector tile.

Use the following command to convert the output file into vector tile.

tippecanoe -e zxy --no-tile-compression --no-feature-limit --no-tile-size-limit output.geojson

"-e" option is for getting tile in folder structure (z/x/y.pbf instead of a single mbtiles file)
"--no-tile-complression", "--no-feature-limit" and other options may not be necessary, but I added it just in case.

The following figure shows the result of step 3 and step 4. (It is in "kata" development env, so I used "dist/cli.js" instead of "kata". But, do not worry about it. Kata v.1.0.3 will be available from npm soon (I hope).)

Now, you have vector tile in the zxy folder.

It is okay that you directly convert the tile from the source without using the intermediate geojson file. (Step 3 and Step 4 are done all at once)

kata filter kata.yml | tippecanoe -e zxy2 --no-tile-compression --no-feature-limit --no-tile-size-limit

Then, you have vector tiles. Congratulations!!

Step 5: Check the metadata (list of layers, their properties, and zoom level ranges)

Remember, we have "metadata.json" in the zxy folder. We can check the vector tile by kata command.

kata document zxy/metadata.json

Then, you will see the list of layer(s) and its(their) properties and zoom levels. From the following figure, I could see that there is a single layer with several properties, which is correct. However, I saw the different zoom level range from 0 to 14 while actual range is from 1 to 5. The "kata" may need to fix it.

Conclusion

I have confirmed that we can create vector tile from GeoJSON file from ESRI server (served as item), and demonstrated how to to it.

We can say that it is EASY to make vector tile with unvt/kata and unvt/uanban. We can also say that we can easily confirm the structure of the vector tile, but unvt/kata may need to fix a small bug on zoom level ranges as of 11 April 2022.

We used a JSON file from ESRI server, but we can use any geojson files from a webserver or your local directory. Or, if you have shapefile, you just need to convert it to geojson using gdal (ogr2ogr) installed in unvt/nanban.

Small tips

If your data is big, more than several GB in output mbtiles format, the converstion with this tool would be tough because unvt/kata doest not have any measure to separate output by spatial extent.

Acknowledgement

unvt/kata is developed by the Geospatial Information of Authority of Japan (GSI) and released as FOSS4G tool. Thank you GSI for its precious contribution. I also heard that colleagues from the Geolonia, Inc. developed the tool. Thank you also for their efforts.

Reference

UN Open GIS Initiative http://unopengis.org/unopengis/main/main.php
FOSS4G 2021 - Deployment of open source vector tile technology with UN Vector Tile Toolkit https://www.youtube.com/watch?v=OGdzdVhV0cA
ArcGIS REST API FeatureService https://developers.arcgis.com/rest/services-reference/enterprise/feature-service.htm
ArcGIS Blog "How to Access Item Data Stored in ArcGIS Online or Portal with ArcGIS REST API" https://community.esri.com/t5/arcgis-online-blog/how-to-access-item-data-stored-in-arcgis-online-or/ba-p/891392
unvt/kata https://github.com/unvt/kata