Control permissions to data catalog
Lake Formation allows fine-grained authority control in the data catalog.
For each user, you can prohibit adding tables to the database of the data catalog, hide certain columns, and limit the SQL that can be executed. I'll try.
S3 pass
s3://test-lf01/in00
Database name
lf01
table name
test1
Test IAM User
test_user1
Upload CSV file to S3
File name: person.csv
Upload the following CSV file to s3://test-lf01/in00
taro, 20190101, 101, 00001
jiro, 20190202, 102, 00001
ojiro, 20190303, 101, 00002
hanako, 20190404, 103, 00004
Schema assumes the following
name (string)
date (string)
storeid (string)
accountnumber (string)
Create data catalog
Create database
Click "Database" from the left menu on the Lake Formation screen, and click [Create database] in the upper right
Enter "lf01" in Name, uncheck "Grant All to Everyone for new tables in this database", and click [Create database] at the bottom right
※"Everyone" is an element of the Lake Formation user group. "Everyone" includes all principals with IAM permissions on Lake Formation, not the general public.
"Grant All to Everyone for new tables in this database" is a setting that grants all privileges to "Everyone" for new tables created in this database. The default is checked.
The database was ready.
Create Table
Click "Tables" from the menu on the left side of the Lake Formation screen, and click [Create table] in the upper right (this time manually create the table)
Enter the following (following)
Name: "test1"
Database: "lf01"
Include path to s3://lf01/in00/
* It seems that you cannot select a bucket in another region
Check CSV
Click [Add column] in the upper right.
Add columns. The column to add↓
name(string)
date(string)
storeid(string)
accountnumber(string)
Click [Submit] at the bottom right when you finish adding.
The table was ready.
Authority check
For the database, the administrator has full privileges and "Everyone" has full privileges (by default, full privileges are given to "Everyone" when creating a database)
For the table, the administrator has full authority and no "Everyone" (as described above, this time when creating a database, the newly created table is set not to give full authority to "Everyone"). So only the administrator can operate it.
Check with Athena
First of all, the administrator can confirm that the data catalog created by Lake Formation can be viewed on Athena.
Authority management by Lake Formation
Confirm authority control at the column level, which is one of the sales of Lake Formation
This time, I don't want to display the storeid and accountnumber columns for some users. I don't want to drop the table. Put the control.
Create IAM user
The procedure is omitted, but test_user1 with Administrator policy was created like this.
test_user1 is administrator at IAM
When you access Athena with test_user1, you can see lf01 of Database, but the table is not displayed because there is no authority to the table yet. Of course, queries are also rejected.
You could see it before Lake Formation appeared. You can think of this feature as a Lake Formation resource-based policy.
Set by Data Permissions of Lake Formation. Hide only the specific columns that you originally wanted to do
I don't want to display the storeid and accountnumber columns for the test_user1 user. I don't want to drop the table. Put the control.
Click "Data Permissions" in the left menu on the LakeFormation screen, then click [Grant] in the upper right.
Enter the following (following)
IAM user add roles : test_user1
Database: lf01
Table: test1
Enter the following and click [Grant] at the bottom right
Select "Include columns" for Column
Select "name" and "date" for Include columns
Check "select" for Table permissions
The result looks something like this
Check with Athena
For test_user1, only the name and date columns are visible.
DDL is rejected properly
You can see a message that looks like a permission error as follows.
MetaException(message:Insufficient Lake Formation permission(s):...
The Lake Access Recent Access Activity log looks something like this
* Since it is a CloudTrail log, it will be output after about 10 minutes. Also on this screen, the filter is quite weak ...
Click on View event in the upper right to check details
(Supplement) _DataCatalog default settings
You can enable or disable the following two default permissions for newly created databases and tables (default is enabled):
- Give All permission to "Everyone" for newly created database
- Grant ALL permission to "Everyone" on newly created table
If you don't need it, uncheck it and click [Save] at the bottom right. As a result, the newly created database or table is not automatically granted to "Everyone" (it can be operated with IAM authority), and operations that give explicit fine-grained permissions on Lake Formation can be performed. I can do it.
"Everyone" is an entity that exists for migration from GlueDataCatalog.
Please come here
How to use lake formation
https://qiita.com/pioho07/items/093d971dcb5656730ea3
How to use Glue(Japanese)
https://qiita.com/pioho07/items/32f76a16cbf49f9f712f