Converting PMtiles from pbf tiles using felt/tippecanoe (tile-join)

Introduction

PMtiles is a single-file archive format for tile data. I think it would be also useful for vector tiles. Therefore, with this articule, I will introduce my test of the data conversion from existing pbf vector tiles in the Internet to PMtiles.

PMTiles is a single-file archive format for tiled data. A PMTiles archive can be hosted on a commodity storage platform such as S3, and enables low-cost, zero-maintenance map applications that are "serverless" - free of a custom tile backend or third party provider. (from protomaps/pmtiles)

(I am working with the vector tile over 150GM in total, and I am not yet sure if PMtiles is good to deal with these large data base. I am currently deal with such vector tiles by separating them into several hundred mbtiles files and host it by using mabox/mbtiles module and nodejs/express server routing. )

My environment

In order to use tippecanoe, we use a linux server. However, with this open demonstration, I used docker container so that I can show it easily.

• Windows 10
• Docker: unvt/nanban (pulled in January 2023)
  • Ubuntu based container
  • felt/tippecanoe version 2.17.0

Procedure

I tried to access the vector tile from my colleagues' Esri server. This time, my wokring repository is here: https://github.com/ubukawa/109

If you would like to follow the process, make sure that you confirm your vector tile URL. If you already have your pbf tiles, you can just start from the step 2.

In the windows Powershell, I started a Docker container unvt/nanban.

docker run -it --rm -v ${PWD}:/data unvt/nanban
cd /data

Step 1: Downloading vector tiles (pbf tiles)

1-1. Making directory for pbf tiles

Under the 0_dl folder, I have created directories for pbf tiles. My target was from zoom level 0 to 6 with global coverage, so I used the following command. Then, there is a folder structure for pbf tiles.

mkdir.sh

rm -rf 0_dl; mkdir 0_dl;for z in {0..6}; do mkdir 0_dl/${z};for x in `seq $((2 ** z))`; do mkdir 0_dl/${z}/$((x-1));done;done

1-2. Downloading tiles

Then, using curl, I have downloaded the files. It took for a while. In my environment, it tool about one hour.

download.sh

for z in {0..6}; do echo $z;for x in `seq $((2 ** z))`; do echo $z/$((x-1));for y in `seq $((2 ** z))`; do echo $z/$((x-1))/$((y-1)).pbf; curl https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/Clearmap_Webplain/VectorTileServer/tile/$z/$((y-1))/$((x-1)).pbf --output 0_dl/$z/$((x-1))/$((y-1)).pbf;done;done;done

Please pay enough attention that the order of tile URL for ArcGIS REST API is z/y/x while other tile schema often uses z/x/y order. (I forgot about it and did the process again.)
If you would like to download pbf tiles from non-Esri vector tile server, you may need to change z/y/x order to z/x/y order. Sh script would be as follows.

download-nonEsri.sh

for z in {0..6}; do echo $z;for x in `seq $((2 ** z))`; do echo $z/$((x-1));for y in `seq $((2 ** z))`; do echo $z/$((x-1))/$((y-1)).pbf; curl https://pro-ags2.dfs.un.org/arcgis/rest/services/Hosted/Clearmap_Webplain/VectorTileServer/tile/$z/$((x-1))/$((y-1)).pbf --output 0_dl/$z/$((x-1))/$((y-1)).pbf;done;done;done

If you are good at bash, the above two sh scripts can be merged, and you can do it with one script.

1-3. Deleting empty tile

If there is no tile, it seems that Esri server returns the following response. I confirned that its size is 590 byte.

Let's check the files in the directory.

find ./0_dl -type f -size 590c | xargs ls -al

Then, I confirmed that the data size of all the found files was 590 byte. I thought all are the same file, and deleted them all. And, let's also delete empty direcroty as they will disturb the next process.
I realized that there is no tile in zoom level 6, so I did not need to download these 4,096 (64 * 64) files at all!!

delete-null.sh

find ./0_dl -type f -size 590c | xargs rm
find 0_dl -type d -empty -delete

Step 2: Making PM tiles from the downloaded pbf tiles

The vector tile conversion tool, felt/tippecanoe, now supprts the PMtiles format since its vversion 2.17.

I have confired the version of my tippecanoe and run tile-join, a tool included in tippecanoe, to obtain PMtiles fild from a series of pbf tiles in the 0_dl directory.

tippecanoe --version
tile-join -pk -o test123.pmtiles 0_dl

At first, I did not add "-pk (--no-tile-size-limit)" option and lost some tiles. So, it would be good to add -pk option.

If you have empty directory, you will fail data conversion as below.

If you would like to convert pmtiles from mbtiles, the following command would work.

tile-join -o output.pmtiles input.mbtiles

Step 3: Review of PMtiles

I heard that we can easily review PMtiles with online PMTiles Viewer at https://protomaps.github.io/PMTiles/

I draged and dropped a file and saw the following. It looked okay.

Conclusion

Exisitng pbf tiles can be easily downloaded into PMtiles with felt/tippecanoe. However, if we have a lot of pbf tiles, we may need some time to download them. I agree that PMtiles would help vector tile hosing, in particular for middle sized data set. I meant that I am not sure about the tiles over several hundred GB at this moment. I would like to keep watching the development of PMtiles.

Acknowledgement

I respect the great work by colleagues. I also respect my colleagues who developed data.

References