2022 Airflow on ec2
Overview 2022年になっても、まだ、airflowをec2上にinstallしていて、さらに、はまるので,いろいろメモ 設定としては dockerなし、 ec2にそのままairflow...
0
0
Comment0
119 search resultsShowing 1~20 results
Stocked
You need to log-in
simple BPE
import re, collections def get_stats(vocab): pairs = collections.defaultdict(int) for word, freq...
0
0
Comment0
bigquery ddl with hive partition on gcs
CREATE EXTERNAL TABLE dataset.AutoHivePartitionedTable WITH PARTITION COLUMNS OPTIONS ( uris=['gs...
0
0
Comment0
redshift data api example
# !/bin/sh ...
0
0
Comment0
how to drop user in redshift
check schema select nspname as schemaname , array_to_string(nspacl, ',') as ...
0
0
Comment0
Redshift spectrumで日付単位でadd partitionするやつ
# !/bin/sh ...
0
0
Comment0
athenaで日付単位でadd partitionするやつ
# !/bin/sh ...
0
0
Comment0
mmlspark on hdp spark env.
git clone git clone https://github.com/rikima/mmlspark.git build sbt package
0
0
Comment0
macにairflowをanacoda3でinstall するときにlimits.hがないーと怒られる場合
limits.hがないとおこられたら sudo open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_m...
0
0
Comment0
starting and stopping ambari services via rest api
stop all services curl -i -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":...
0
0
Comment0
HDPのzeppelinでphoenixを使うには
zeppelin spark interpreterで | spark.driver.extraClassPath | /usr/hdp/current/phoenix-client/ | |...
0
0
Comment0
mondrian schemaで memberのformatを変更するには
schema.xml <Dimension name="4. Department"> <Hierarchy hasAll="true"> <Level name=...
0
0
Comment0
sbtでstormを使うには
build.sbt name := "StormTest" version := "0.1" scalaVersion := "2.11.8" resolvers += "clojars" at...
0
0
Comment0
lua script to login facebook on Splash
function main(splash, args) assert(splash:go(args.url)) assert(splash:wait(0.5)) local form = spl...
0
0
Comment0
How to explode a column in pandas
def explode(self, df, explode_column, delimiter=","): assert explode_column[-1] is "s" df["id"] =...
0
0
Comment0
How to visualize a word2vec vector trained by fastText on a tensorboard
import os import tensorflow as tf import numpy as np import fastText from tensorflow.contrib.tens...
0
0
Comment0
replace a column name in dataframe
df.columns.foreach { case col => col.toLowerCase match { case "a_cookie_id" => df = df.with...
0
0
Comment0
save json via spark dataframe on AWS S3
spark.sparkContext.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "[access key id]") spark.spar...
0
0
Comment0
CD法メモ
Gibbs distribution \begin{eqnarray} p(\bf{x}; \bf{\xi}) &=& \frac{e^{-E(\bf{x}; \bf{\xi})...
0
0
Comment0
join hive table and phoenix table on Spark without hive-phoenix-handler
load and register a phoenix table as tempolary table val df = spark.read.format("jdbc") .option("...
0
0
Comment0
119 search resultsShowing 1~20 results
Qiita is a knowledge sharing service for engineers.
- You can follow users and tags
- You can stock useful information
- You can make edit suggestions for articles