My Shell Tips

I work on a remote terminal most of the time. This article collects some tips from my experience.

bash

multiple line editing

Sometimes a complex command consists of multiple lines and you need to edit them before executing it. The lines may be in any format, e.g. json.

• use quotes

1
2
3
4

curl http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance -d '
some_metric{label="val1"} 42
another_metric 2398.283
'

• Here Documents

1
2
3
4

cat <<EOF | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance
some_metric{label="val1"} 42
another_metric 2398.283
EOF

• edit via vim

Above ways are inconvenient, because you could not jump back and forth the lines to edit.

Now you need vim.

1

export EDITOR=vim

Just press C-x C-e to enter vim, and bash would create a temp file to edit, and when you save and exit vim, bash will execute the file content.

You could even paste an string in bash and then press C-x C-e, bash would pass that string to vim as initial content and let you continue editing.

history customization

1
2
3
4
5
6
7

# increase history size
export HISTSIZE=10000
export HISTFILESIZE=10000

# bash history is not automatically saved by default.
# below setting would save history after each command execution.
PROMPT_COMMAND='$PROMPT_COMMAND; history -a'

Process Substitution

Some command requires file arguments only, then process substitution is the solution.

For example, diff two curl responses:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

diff -aNur <(curl -s httpbin.org/get) <(curl -s httpbin.org/get)
--- /dev/fd/63  2022-10-18 16:02:18.633005560 +0800
+++ /dev/fd/62  2022-10-18 16:02:18.633005560 +0800
@@ -4,7 +4,7 @@
     "Accept": "*/*",
     "Host": "httpbin.org",
     "User-Agent": "curl/7.68.0",
-    "X-Amzn-Trace-Id": "Root=1-634e5d8b-1eb31b42468a62f11ca13683"
+    "X-Amzn-Trace-Id": "Root=1-634e5d8b-3d27e89e310ea9fb56e8c0a8"
   },

alias completion

Normally we prefer short alias instead of long command, but bash doesn’t support command completion for alias. Then you could use complete-alias plugin.

1

git clone https://github.com/cykerway/complete-alias ~/.complete-alias

For example, alias kubectl as k.

~/.bashrc

1
2
3
4

alias k=kubectl
source <(kubectl completion bash)
. ~/.complete-alias/complete_alias
complete -F _complete_alias k

long string literal

• Use concatenation

1
2
3

export FOOBAR="/foo/abc"
FOOBAR+=":/bar/xyz"
FOOBAR+=":/foobar"

• Use backward slash

Note that spaces from the line beginning will be included in the string.

1
2
3

export FOOBAR="/foo/abc\
:/bar/xyz\
:/foobar"

tmux

tmux is a brilliant terminal tool you have to use. With tmux, you could parallel your works in one terminal.

Most useful features:

• multiple tmux sessions, each tmux session contains multiple windows
• tmux session runs in background
  • no worry about ssh connection gets broken and you could resume where you were
  • run server programs in the foreground but left them continue running after detaching
  • shared with other user to watch or edit the screen together

attach last session

It’s useful when your ssh connection is broken, you login again and try to attach last session you were in.

1

tmux a

set session name

Set a meaningful session name: C-b $, e.g. dev.

You could use this name to attach later.

1

tmux att -t dev

session jump

Show all tmux session/window in tree view and jump to any arbitary window.

Setup shortcut:

~/.tmux.conf

unbind t
bind t choose-tree

Press C-b t to show the windows tree:

(0)   - dev: 8 windows (attached)
(1)   ├─> 0: bash* (1 panes) "foo"
(2)   ├─> 1: bash (1 panes) "foo"
(3)   ├─> 2: bash (1 panes) "foo"
(4)   ├─> 3: bash (1 panes) "foo"
(5)   ├─> 4: bash- (1 panes) "foo"
(6)   ├─> 5: vim (1 panes) "foo"
(7)   ├─> 6: bash (1 panes) "foo"
(8)   └─> 7: bash (1 panes) "foo"
(9)   - ops: 1 windows
(M-a) └─> 0: bash* (1 panes) "foo"

Each window shown in the tree has a key:

• for 0-9, press the number key will jump to the corresponding window
• for M-*, press Alt+* will jump to the corresponding window

jump between two last windows

It’s useful to swtich between your last two windows: C-b l

Tmux Resurrect

With this tmux plugin, you could resume everything after reboot or shutdown, e.g. power off.

https://github.com/tmux-plugins/tmux-resurrect

tmux-resurrect saves all the little details from your tmux environment so it can be completely restored after a system restart (or when you feel like it). No configuration is required. You should feel like you never quit tmux.

fzf

Command history search in bash has no fuzzy search and no history list. fzf could help you there.

Install fzf:

1
2
3

git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf
~/.fzf/install
git clone https://github.com/4z3/fzf-plugins ~/.fzf-plugins

Edit ~/.bashrc:

1
2
3
4
5
6

[ -f ~/.fzf.bash ] && source ~/.fzf.bash

# when you select one command, press `C-e` to edit it instead of executing it directly:
FZF_CTRL_R_EDIT_KEY=ctrl-e
FZF_CTRL_R_EXEC_KEY=enter
source ~/.fzf-plugins/history-exec.bash

Press C-r to search history interactively in fuzzy way.

zoxide

The builtin cd of bash is very inconvenient, it has no history and fuzzy jump.

I prefer the tool zoxide, which could improve your work productivity.

https://github.com/ajeetdsouza/zoxide

It remembers which directories you use most frequently, so you can “jump” to them in just a few keystrokes.

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

root@myhost:~# z /tmp/
root@myhost:~# mkdir -p foo/bar/{apple,orange/{abc-bi,xyz},food/{abc,xyz}}
root@myhost:/tmp# tree foo
foo
└── bar
    ├── apple
    ├── food
    │   ├── abc
    │   └── xyz
    └── orange
        ├── abc-bi
        └── xyz

8 directories, 0 files

root@myhost:/tmp# z foo/bar/apple/
root@myhost:/tmp/foo/bar/apple# z ../food/abc/
root@myhost:/tmp/foo/bar/food/abc# z /tmp/foo/bar/orange/abc-bi/

# input `abc` only would match `/tmp/foo/bar/food/abc`
root@myhost:/tmp/foo/bar/orange/abc-bi# z abc

# more specific word, with `-` suffix, then jump to `abc-bi`
root@myhost:/tmp/foo/bar/food/abc# z abc-

root@myhost:/tmp/foo/bar/orange/abc-bi# z /tmp/foo/bar/food/xyz/
root@myhost:/tmp/foo/bar/food/xyz# z /tmp/foo/bar/orange/xyz

# specify multiple path segments
root@myhost:/tmp/foo/bar/orange/xyz# z food xyz

# choose alternatives for `xyz` interactively
root@myhost:/tmp/foo/bar/food/xyz# zi xyz

ssh

socks5 proxy sever

1

ssh -o ServerAliveInterval=60 -N -D 127.0.0.1:10000 root@vpn -p 10022 &

Then use 127.0.0.1:10000 as your proxy address, and the traffic would go through the safe ssh connection.

port forwarding

Sometimes you need to let two machines access to each other, but for security, you could not export ports except ssh port. In this case, you could use port forwarding.

For example, you need to access foo web server from the same port 1984 but with different IP address bindings, because you need to simulate sending different cookies.

1
2

ssh -N -L 127.0.0.1:1984:127.0.0.1:1984 foo &
ssh -N -L 127.0.0.2:1984:127.0.0.2:1984 foo &

jump host

Some servers could only be accessed in internal network. You could use gateway server as jump host, then no need to login gateway first and run ssh from there. Another benefit is you don’t have to store credential, e.g. private keys in the gateway machine, and everything works just like you login the target machine directly.

~/.ssh/config

Host gateway
    ...

Host foobar
    HostName internal.machine
    User foobar
    IdentityFile ~/.ssh/foobar.pem
    ProxyJump gateway

git

Configure ~/.gitconfig to improve work productivity.

• commit history tree

git tree

1
2

[alias]
    tree = log --graph --decorate --oneline --all

• exclude some files changes temporarily

1
2
3
4

[alias]
    ignore = !git update-index --assume-unchanged
    unignore = !git update-index --no-assume-unchanged
    ignored = !git ls-files -v | grep ^[a-z]

• make diff fancy, e.g. hightlight different parts of two lines

1
2
3
4
5
6
7

[core]
    pager = /root/diff-so-fancy/diff-so-fancy | less --tabs=4 -RFX
[interactive]
    diffFilter = /root/diff-so-fancy/diff-so-fancy --patch
[diff-so-fancy]
    changeHunkIndicators = false
    stripLeadingSymbols = false

• credential cache, then no need to input access token for a period

1
2

[credential]
    helper = cache --timeout 7200

find

Sometimes you need to check which files you modified within some period.

For example, backup all json files modified in last hour:

1

find dir -mmin -60 -name '*.json' -type f -exec mv {} /tmp/backup/ \;

less

less is a necessary tool to view a file.

• tail -f

Press F to watch new changes of the file (e.g. log file) from the tail constantly.

• filter lines

Press &<pattern> to show lines only match the pattern.

Press &!<pattern> to show lines not matching the pattern.

jq

jq is a brilliant tool to analyze json in the terminal.

https://stedolan.github.io/jq/manual/

Prettify JSON

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

etcdctl get --print-value-only /apisix/routes/test | jq
{
  "status": 1,
  "plugins": {
    "proxy-rewrite": {
      "uri": "/get",
      "use_real_request_uri_unsafe": false
    }
  },
  "id": "test",
  "priority": 0,
  "update_time": 1666154686,
  "upstream": {
    "hash_on": "vars",
    "nodes": {
      "httpbin.org": 1
    },
    "scheme": "http",
    "pass_host": "pass",
    "type": "roundrobin"
  },
  "uri": "/foo",
  "methods": [
    "GET"
  ],
  "create_time": 1666154686
}

Analyze JSON

• Analyze file

jq '.[] | select(.color=="yellow" and .price>=0.5)' fruits.json

• Analyze command output

For example, we use kcadmin.sh to do keycloak provisioning. We need to determine one client’s id and use this id to configure its profile.

Output from kcadm.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

[ {
    "id" : "0895c867-8fc1-4817-9bf3-06dddb6eace8",
    "clientId" : "account",
    "name" : "${client_account}",
  }, {
    "id" : "3b51e2a9-1eec-483a-b62d-d056fb7b622b",
    "clientId" : "account-console",
    "name" : "${client_account-console}",
  },
  ...
]

Use jq to get the desired id:

1
2

kcadm.sh get clients -r test --fields id,clientId 2>/dev/null | \
     jq -r '.[] | select(.clientId=="account") | .id'

Merge JSON

Example1:

1

jq -s '.[0] * .[1] | {somekey: .somekey}' <file1> <file2>

Example2:

1
2
3
4

echo '[ {"a":1}, {"b":2} ]' | \
jq --argjson input1 '[ { "c":3 } ]' \
   --argjson input2 '[ { "d":4 }, { "e": 5} ]' \
   '. = $input1 + . +  $input2'