DRYな備忘録

Don't Repeat Yourself.

Terraformってなんぞや

目的

  • Terraformのなんたるかを知る
  • Terraformの初歩的な使い方を知る
  • Terraformを使ってみる

調査

サンプルを眺める

より。HCL(HashiCorp Configuration Language) という文法で記述される。

resource "aws_elb" "frontend" {
  name = "frontend-load-balancer"
  listener {
    instance_port     = 8000
    instance_protocol = "http"
    lb_port           = 80
    lb_protocol       = "http"
  }
 
  instances = ["${aws_instance.app.*.id}"]
}
 
resource "aws_instance" "app" {
  count = 5

  ami           = "ami-408c7f28"
  instance_type = "t1.micro"
}

# aws_elb.frontendという名前のELBを建てて、80番ポートで受けて、
# その参照先が aws_instance.appという名前に帰属する
# 5つのインスタンスの8000番に向いているんだろうな、
# という雰囲気がなんとなくわかる。
# だけどこれ、VPCとかSecurityGroupとかどうなってるんだ?
# An AMI
variable "ami" {
  description = "the AMI to use"
}

/* A multi
   line comment. */
resource "aws_instance" "web" {
  ami               = "${var.ami}"
  count             = 2
  source_dest_check = false

  connection {
    user = "root"
  }
}

terraformのインストール

% brew search terraform
% brew info terraform
% brew install terraform
# 中略...
/usr/local/Homebrew/Library/Homebrew/brew.rb:12:in `<main>': Homebrew must be run under Ruby 2.3! (RuntimeError)

だっる。

% ruby --version
ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin16]

クソ古いやん。

% rbenv versions
* system (set by /Users/otiai10/.rbenv/version)
% rbenv install --list | grep 2.3
% rbenv install 2.3.5
% rbenv global 2.3.5
% rbenv rehash
% ruby --version
ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-darwin16]
% brew install terraform
% terraform --version
Terraform v0.10.7

はい。

Hello, Terraform!

% mkdir -p ~/tmp/terraform/hello
% cd ~/tmp/terraform/hello
% vi hello.tf
# プロバイダの設定。
# AWSを使うこと、API keys、リージョンなどを指定する。
provider "aws" {
    #  > Terraform will automatically search for saved API credentials (for example, in ~/.aws/credentials)
    # とのことなので、AWS Keysは割愛します
    /*
    access_key = "ACCESS_KEY"
    secret_key = "SECRET_KEY"
    */

    # ワケあってシドニー
    region = "ap-southeast-2"
}

# 具体的な構成の設定。
resource "aws_instance" "hello-terraform" {
    ami = "ami-2757f631"
    instance_type = "t2.micro"
}

この状況でplanしてみると、

Plugin reinitialization required. Please run "terraform init".
Reason: Could not satisfy plugin requirements.

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

1 error(s) occurred:
# プロバイダの設定。

* provider.aws: no suitable version installed
  version requirements: "(any version)"
  versions installed: none

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints from each module, run "terraform providers".

error satisfying plugin requirements

まずinitしろと叱られる。

% terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.1.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 1.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

aws pluginがインストールされた模様。あらためてplanする

% terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_instance.hello-terraform
      id:                           <computed>
      ami:                          "ami-2757f631"
      associate_public_ip_address:  <computed>
      availability_zone:            <computed>
      ebs_block_device.#:           <computed>
      ephemeral_block_device.#:     <computed>
      instance_state:               <computed>
      instance_type:                "t2.micro"
      ipv6_address_count:           <computed>
      ipv6_addresses.#:             <computed>
      key_name:                     <computed>
      network_interface.#:          <computed>
      network_interface_id:         <computed>
      placement_group:              <computed>
      primary_network_interface_id: <computed>
      private_dns:                  <computed>
      private_ip:                   <computed>
      public_dns:                   <computed>
      public_ip:                    <computed>
      root_block_device.#:          <computed>
      security_groups.#:            <computed>
      source_dest_check:            "true"
      subnet_id:                    <computed>
      tenancy:                      <computed>
      volume_tags.%:                <computed>
      vpc_security_group_ids.#:     <computed>


Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

当たり前だけど、もっと色々指定可能であることが分かる。前述の疑問(「だけどこれ、VPCとかSecurityGroupとかどうなってるんだ?」)は、vpc_security_group_idsあたりだろうか。いよいよ、applyしてローンチしてみる。

% terraform apply
# 中略
Error applying plan:

1 error(s) occurred:

* aws_instance.hello-terraform: 1 error(s) occurred:

* aws_instance.hello-terraform: Error launching source instance: InvalidAMIID.NotFound: The image id '[ami-2757f631]' does not exist
    status code: 400, request id: 00d6f091-b677-47ca-bc17-f8671eb07501

適当にドキュメントに書かれたAMI IDを使っていたので「AMIが存在しない」と叱られる。region変えたからかな。ちゃんと書いてやる。シドニーでいちばん簡単なAmazon Linuxってこれami-66fbe605でいいんだろうか。

resource "aws_instance" "hello-terraform" {
-    ami = "ami-2757f631"
+    ami = "ami-66fbe605"
    instance_type = "t2.micro"
}

あらためて。

% terraform apply
# 中略...
Apply complete! Resources:1 added, 0 changed, 0 destroyed.

f:id:otiai10:20171017143708p:plain

いいですね。

% terraform show

で色々確認できる。気づいたこと、

  • VPCを指定しなかったので、default VPCになってる
  • SecurityGroupも、defaultのものを使っている
    • 新規作成したい場合は、おそらくsgも.tfに書いて参照させればよいのだろう
    • VPC.tfでつくれるのかな?
  • key pairを指定しなかったので、おそらくsshはできない気がする
  • public IPも消せるのだろう
    • associate_public_ip_address = true これだ
  • Build Infrastructure - Terraform by HashiCorp
    • インスタンスつくった後のプロビジョニング(前述で「サーバ設定」と呼称したもの)もできるっぽい
    • その場合、key pairは自動でつくられるのかな?
  • このhello.tfによって定義されインスタンス化されクラスタへの参照は、terraform.tfstate, terraform.tfstate.backup に保存されているっぽい

クラスタインスタンスの変更・編集

The prefix "-/+" means that Terraform will destroy and recreate the resource, versus purely updating it in-place. While some attributes can do in-place updates (which are shown with a "~" prefix), AMI changing on EC2 instance requires a new resource. Terraform handles these details for you, and the execution plan makes it clear what Terraform will do.

resource "aws_instance" "hello-terraform" {
-    ami = "ami-66fbe605"
-    instance_type = "t2.micro"
+    ami = "ami-5bc2f938"
+    instance_type = "t2.nano"
}

AMIの変更・インスタンスタイプの変更は、インスタンスの作り直しが発生するので、かならず-/+となるはず。

% terraform plan
# 中略...
-/+ aws_instance.hello-terraform (new resource required)
      id:     "i-0f4c9ccc6ad36fb74" => <computed> (forces new resource)
      ami:    "ami-66fbe605" => "ami-5bc2f938" (forces new resource)
# 中略...

% terraform apply

クラスタインスタンスの削除

% terraform plan -destroy
% terraform destroy
# 中略
Destroy complete! Resources: 1 destroyed.

その他

DevOpsを支えるHashiCorpツール大全 ThinkIT Books

DevOpsを支えるHashiCorpツール大全 ThinkIT Books

雑感

  • 結局DSLを覚えるコストがあるけれど、覚えてしまえば便利そう
  • CloudFormationをポチポチやるよりはよっぽどよさそう
  • また肉離れした。全力で動いても壊れない身体が欲しい...

DRYな備忘録として

Tesseract-OCRをソースからコンパイルする(4.00.00dev)

背景

Tesseract-OCR 4.00.00devで動かない、というissueが来た。

github.com

前回記事

Tesseract-OCRのDockerコンテナ内でのビルド

otiai10.hatenablog.com

今回の成果物

FROM debian:stretch

ARG TESS="4.00.00dev"
ARG LEPTO="1.74.2"

# Prepare dependencies
RUN apt-get update
RUN apt-get install -y \
  wget \
  make \
  autoconf \
  automake \
  libtool \
  autoconf-archive \
  pkg-config \
  libpng-dev \
  libjpeg-dev \
  libtiff-dev \
  zlib1g-dev \
  libicu-dev \
  libpango1.0-dev \
  libcairo2-dev

ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib

# Compile Leptonica
WORKDIR /
RUN mkdir -p /tmp/leptonica \
  && wget https://github.com/DanBloomberg/leptonica/archive/${LEPTO}.tar.gz \
  && tar -xzvf ${LEPTO}.tar.gz -C /tmp/leptonica \
  && mv /tmp/leptonica/* /leptonica
WORKDIR /leptonica

RUN autoreconf -i \
  && ./autobuild \
  && ./configure \
  && make \
  && make install

# Compile Tesseract
WORKDIR /
RUN mkdir -p /tmp/tesseract \
  && wget https://github.com/tesseract-ocr/tesseract/archive/${TESS}.tar.gz \
  && tar -xzvf ${TESS}.tar.gz -C /tmp/tesseract \
  && mv /tmp/tesseract/* /tesseract
WORKDIR /tesseract

RUN ./autogen.sh \
  && ./configure \
  && make \
  && make install

# Recover location
WORKDIR /

# Load languages
RUN wget https://github.com/tesseract-ocr/tessdata/raw/master/eng.traineddata -P /usr/local/share/tessdata
RUN wget https://github.com/tesseract-ocr/tessdata/raw/master/deu.traineddata -P /usr/local/share/tessdata
RUN wget https://github.com/tesseract-ocr/tessdata/raw/master/fra.traineddata -P /usr/local/share/tessdata
RUN wget https://github.com/tesseract-ocr/tessdata/raw/master/spa.traineddata -P /usr/local/share/tessdata
RUN wget https://github.com/tesseract-ocr/tessdata/raw/master/jpn.traineddata -P /usr/local/share/tessdata

雑感

  • ブログのアウトプットが成果物だけという手抜き感がいなめない

DRYな備忘録として

atom command not found

Problem

# MacOS
% atom .
zsh: command not found: atom
% which atom
atom not found

Anything can’t exist without Atom

Research

Solution

% which atom
atom not found
% /Applications/Atom.app/Contents/Resources/app/atom.sh
# and close the window just opened
% which atom
/usr/local/bin/atom
% atom .

It works.

【Go言語】意地でもsliceをswitch-caseで比較したい

invalid case []byte literal in switch (can only compare slice foo to nil)

switch response[:4] {
case []byte{0,0,0,0}, []byte{1,0,0,0}, []byte{2,0,0,0}:
    fmt.Println("do something")
}
// これ動かないやつ

可変長であるsliceはnilとの比較だけが許されているようだ。固定長ならどうか?

やったこと

  1. 固定長arrayをつくる
  2. sliceとしてcopyにかける
  3. 固定長arrayと比較する
var header [4]byte
copy(header[:], response[:4])
switch header {
case [4]byte{0,0,0,0}, [4]byte{1,0,0,0,}, [4]byte{2,0,0,0}:
    fmt.Println("do something")
}

The Go Playground

DRYな備忘録として

EC2インスタンスにsshが接続できない原因のまとめ

目次(随時追加)

  1. InternetGatewayがVPCに未追加
  2. Pemファイルのパーミッションが不適切
  3. sshを試みるUserが違う

当該SubnetのRouteTableに当該InternetGatewayが追加されていない: Operation timed out

問題

  • sshコマンドを打ってもOperationTimeoutになる
    • TCP的に到達していない感じ

原因

解決

  1. EC2インスタンス一覧
  2. 当該インスタンスSubnet IDをメモる
  3. VPC→Subnets→当該SubnetのRoute TableのリンクをクリックしてRoute Table設定へ行く
  4. 当該RouteTableのRoutesでEdit
  5. (便宜上とりあえず)Destination0.0.0.0/0として当該VPCにattachされているInternetGatewayを追加

ダウンロードしたKeyPairファイル(.pem)のアクセス権限: Permission denied (publickey).

問題

  • sshがネットワーク的に到達しているが以下のようなエラーが出る
% ssh -i /tmp/my-project.pem ec2-user@X.X.X.X
Permissions 0644 for '/tmp/my-project.pem' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/tmp/my-project.pem": bad permissions
Permission denied (publickey).

原因

  • 最後の一文だけ見て焦るけど、おちついて全文読みましょう
  • pemファイルのファイルアクセス権限の問題なので、chmodの話

解決

% chmod 600 /tmp/my-project.pem

ローカルのデフォルトユーザでsshを試みている: Permission denied (publickey).

問題

Host my-test
    HostName 123.123.123.123
    IdentityFile ~/.ssh/tmp/my-test.pem

として、sshを試みるも、

% ssh my-test
Permission denied (publickey).

となる。

原因

  • 自動生成されるidentityは、ec2-user名義のpemであるので、Userを指定する必要がある。

解決

% ssh ec2-user@my-test

あるいは、~/.ssh/configにおいて、

Host my-test
    HostName 123.123.123.123
+    User ec2-user
    IdentityFile ~/.ssh/tmp/my-test.pem

とする。

雑感

  • 何事も、慣れだ

Amazon Web Services実践入門 (WEB+DB PRESS plus)

Amazon Web Services実践入門 (WEB+DB PRESS plus)

(メモ)Go言語のinvalid recursive typeエラー、再帰データ型の実装とイテレーションについて

jsonでいうとこんな感じの

{
    "id": 48,
    "name": "foo",
    "entry": {
        "id": 36,
        "name": "bar",
        "entry": {
            "id": 24,
            "name": "baz",
            "entry": {
                "id": 12,
                "name": "qux"
            }
        }
    }
}

問題

コンパイル時のinvalid recursive typeエラーは、ポインタ型にすれば解決 invalid recursive type in a struct in go - Stack Overflow

実装

package main

import (
    "encoding/json"
    "fmt"
    "strings"
)

const data = `{
  "id": 48,
  "name": "foo",
  "entry": {
      "id": 36,
      "name": "bar",
      "entry": {
          "id": 24,
          "name": "baz",
          "entry": {
              "id": 12,
              "name": "qux"
          }
      }
  }
}`

type Entry struct {
    ID    int    `json:"id"`
    Name  string `json:"name"`
    Child *Entry `json:"entry"`
}

func main() {

    root := new(Entry)
    if err := json.NewDecoder(strings.NewReader(data)).Decode(root); err != nil {
        panic(err)
    }

    for entry := root; entry.Child != nil; entry = entry.Child {
        fmt.Printf("%s has a child: %+v\n", entry.Name, entry.Child)
    }
}

https://play.golang.org/p/vBNfDuhLHB

参考

雑感

Go言語によるWebアプリケーション開発

Go言語によるWebアプリケーション開発

(メモ)docker/machineのテストが落ちる問題

github.com

bash-4.4$ DRIVER=virtualbox make test-integration test/integration/core/core-commands.bats
test/integration/run-bats.sh test/integration/core/core-commands.bats
INFO: Run the tests with B2D_CACHE=1 to avoid downloading the boot2docker iso each time.
=> test/integration/core/core-commands.bats
 ✓ virtualbox: machine should not exist
 ✓ virtualbox: appears with ls
 ✓ virtualbox: has status 'started' appearing in ls
 ✓ virtualbox: create with same name again fails
 ✓ virtualbox: run busybox container
 ✓ virtualbox: url
 ✓ virtualbox: ip
 ✓ virtualbox: ssh
 ✓ virtualbox: version
 ✓ virtualbox: docker commands with the socket should work
 ✓ virtualbox: stop
 ✓ virtualbox: machine should show stopped after stop
 ✓ virtualbox: url should show an error when machine is stopped
 ✓ virtualbox: env should show an error when machine is stopped
 ✓ virtualbox: version should show an error when machine is stopped
 ✗ virtualbox: machine should not allow upgrade when stopped
   (in test file test/integration/core/core-commands.bats, line 109)
     `[[ "$status" -eq 1 ]]` failed
   Starting machine so machine can be upgraded... Starting "bats-virtualbox-test-shared-1498198301"... (bats-virtualbox-test-shared-1498198301) Check network to re-create if needed... (bats-virtualbox-test-shared-1498198301) Waiting for an IP... Machine "bats-virtualbox-test-shared-1498198301" was started. Waiting for SSH to be available... Detecting the provisioner... Waiting for SSH to be available... Detecting the provisioner... Upgrading docker... Stopping machine to do the upgrade... Upgrading machine "bats-virtualbox-test-shared-1498198301"... Copying /tmp/machine-bats-test-virtualbox/cache/boot2docker.iso to /tmp/machine-bats-test-virtualbox/machines/bats-virtualbox-test-shared-1498198301/boot2docker.iso... Starting machine back up... (bats-virtualbox-test-shared-1498198301) Check network to re-create if needed... (bats-virtualbox-test-shared-1498198301) Waiting for an IP... Restarting docker...
 ✗ virtualbox: start
   (in test file test/integration/core/core-commands.bats, line 115)
     `[ "$status" -eq 0  ]` failed
   Starting "bats-virtualbox-test-shared-1498198301"... Machine "bats-virtualbox-test-shared-1498198301" is already running.
 ✓ virtualbox: machine should show running after start
 ✓ virtualbox: kill
 ✓ virtualbox: machine should show stopped after kill
 ✓ virtualbox: restart
 ✓ virtualbox: machine should show running after restart
 ✓ virtualbox: status

23 tests, 2 failures

make: *** [test-integration] Error 1
bash-4.4$

Code Reading

まず、(in test file test/integration/core/core-commands.bats, line 109) とあるので、その行へ行く

@test "$DRIVER: machine should not allow upgrade when stopped" {
  run machine upgrade $NAME
  echo ${output}
  [[ "$status" -eq 1 ]]
}

run machine upgradeがstatus 1で終わるべきだが、status 0で成功している。

次に、run machine upgradeの内容としてcommands/upgrade.goを見る

package commands

import "github.com/docker/machine/libmachine"

func cmdUpgrade(c CommandLine, api libmachine.API) error {
    return runAction("upgrade", c, api)
}

commands/commands.go

runAction抜粋

   if errs := runActionForeachMachine(actionName, hosts); len(errs) > 0 {
        return consolidateErrs(errs)
    }

runActionForeachMachine抜粋

   for _, machine := range machines {
        numConcurrentActions++
        go machineCommand(actionName, machine, errorChan)
    }

machineCommand

// machineCommand maps the command name to the corresponding machine command.
// We run commands concurrently and communicate back an error if there was one.
func machineCommand(actionName string, host *host.Host, errorChan chan<- error) {
    // TODO: These actions should have their own type.
    commands := map[string](func() error){
        "configureAuth": host.ConfigureAuth,
        "start":         host.Start,
        "stop":          host.Stop,
        "restart":       host.Restart,
        "kill":          host.Kill,
        "upgrade":       host.Upgrade,
        "ip":            printIP(host),
        "provision":     host.Provision,
    }

    log.Debugf("command=%s machine=%s", actionName, host.Name)

    errorChan <- commands[actionName]()
}

host.Upgradeが呼ばれている

libmachine/host/host.go

Upgrade冒頭抜粋

func (h *Host) Upgrade() error {
    machineState, err := h.Driver.GetState()
    if err != nil {
        return err
    }

    if machineState != state.Running {
        log.Info("Starting machine so machine can be upgraded...")
        if err := h.Start(); err != nil {
            return err
        }
    }

テストのエラー文言にもあったように、明らかにUpgradeするためにStartを試みている。この実装をした日時と最後にテストが編集された日時が欲しい。blameしてみる。

f:id:otiai10:20170623163828p:plain

このコミットは、Start machine if needed for upgrade · docker/machine@e55dfb2 · GitHub

2017/02/23。対して、test/integration/core/core-commands.batsは、History for test/integration/core/core-commands.bats - docker/machine · GitHub この時点で2016/01/08なので、明らかにテストのほうが追従していない。

試しに、host.Upgradeの中で、machineが動いて無くてもStartしないように変更を加えると、

func (h *Host) Upgrade() error {
-  machineState, err := h.Driver.GetState()
-  if err != nil {
-      return err
-  }
-
-  if machineState != state.Running {
-      log.Info("Starting machine so machine can be upgraded...")
-      if err := h.Start(); err != nil {
-          return err
-      }
-  }

    provisioner, err := provision.DetectProvisioner(h.Driver)
    if err != nil {
        return err
    }

testそのものは

bash-4.4$ DRIVER=virtualbox make test-integration test/integration/core/core-commands.bats
test/integration/run-bats.sh test/integration/core/core-commands.bats
INFO: Run the tests with B2D_CACHE=1 to avoid downloading the boot2docker iso each time.
=> test/integration/core/core-commands.bats
 ✓ virtualbox: machine should not exist
 ✓ virtualbox: appears with ls
 ✓ virtualbox: has status 'started' appearing in ls
 ✓ virtualbox: create with same name again fails
 ✓ virtualbox: run busybox container
 ✓ virtualbox: url
 ✓ virtualbox: ip
 ✓ virtualbox: ssh
 ✓ virtualbox: version
 ✓ virtualbox: docker commands with the socket should work
 ✓ virtualbox: stop
 ✓ virtualbox: machine should show stopped after stop
 ✓ virtualbox: url should show an error when machine is stopped
 ✓ virtualbox: env should show an error when machine is stopped
 ✓ virtualbox: version should show an error when machine is stopped
 ✓ virtualbox: machine should not allow upgrade when stopped
 ✓ virtualbox: start
 ✓ virtualbox: machine should show running after start
 ✓ virtualbox: kill
 ✓ virtualbox: machine should show stopped after kill
 ✓ virtualbox: restart
 ✓ virtualbox: machine should show running after restart
 ✓ virtualbox: status

23 tests, 0 failures

make: Nothing to be done for `test/integration/core/core-commands.bats`.
bash-4.4$

通る。

結論

以下のどちらか

  1. host.Upgradeの仕様を戻す
  2. core-commands.batsのシナリオをちゃんと更新する

まあ2かな。

DRY


追記 2017/06/23 17:40

issue立てた

github.com

追記 2017/08/22

pull-req送って、7月にマージされてた

github.com