分类 Linux 下的文章

jq: shell 脚本中JSON创建、解析工具

0x01 背景

在Linux下开发监控、自动化shell脚本是运维的日常工作之一,JSON是通用且流行的数据交换格式,在Web API中使用很普遍

0x02 问题/诉求

如何在Linux shell脚本中创建与解析JSON呢?

0x03 土办法

1. 创建JSON

变量嵌入

#!/bin/bash
FILENAME="/tmp/xxxx"
curl -i -X POST -H "Content-Type: application/json" -d '{"type": "test", "params":{"item":{"file":"'"$FILENAME"'"}}}' http://192.168.0.1/api
#!/bin/bash
header="Content-Type: application/json"
FILENAME="/tmp/xxxx"
request_body=$(< <(cat <<EOF
{
  "type": "test",
  "params": {
    "item": {
      "file": "$FILENAME"
    }
  }
}
EOF
))
curl -i -X POST -H "$header" -d "$request_body" http://192.168.0.1/api

循环拼接

#!/bin/bash

arr=""
while read var; do
  arr="\"${var}\",${arr}"
done < list

json=`echo ${arr} | sed 's/,$//g'`
echo "[${json}]"

文本内容:
$ cat list
aa
bb
cc
dd
123
r4
t55
5t
gg
1
rgs
g
f
gs

脚本输出:
$ sh t.sh
["gs","f","g","rgs","1","gg","5t","t55","r4","123","dd","cc","bb","aa"]

2. 解析JSON

awk

没有具体尝试过,可参考stackoverflow上的一则回答

0x04 完美解法 —— jq

jq is a lightweight and flexible command-line JSON processor.

jq官网

$ jq -h
jq - commandline JSON processor [version 1.5-1-a5b5cbe]
Usage: jq [options] <jq filter> [file...]

    jq is a tool for processing JSON inputs, applying the
    given filter to its JSON text inputs and producing the
    filter's results as JSON on standard output.
    The simplest filter is ., which is the identity filter,
    copying jq's input to its output unmodified (except for
    formatting).
    For more advanced filters see the jq(1) manpage ("man jq")
    and/or https://stedolan.github.io/jq

    Some of the options include:
     -c        compact instead of pretty-printed output;
     -n        use `null` as the single input value;
     -e        set the exit status code based on the output;
     -s        read (slurp) all inputs into an array; apply filter to it;
     -r        output raw strings, not JSON texts;
     -R        read raw strings, not JSON texts;
     -C        colorize JSON;
     -M        monochrome (don't colorize JSON);
     -S        sort keys of objects on output;
     --tab    use tabs for indentation;
     --arg a v    set variable $a to value <v>;
     --argjson a v    set variable $a to JSON value <v>;
     --slurpfile a f    set variable $a to an array of JSON texts read from <f>;
    See the manpage for more options.

1. 创建JSON

变量嵌入

$ FILENAME="/tmp/xxxx"
$ jq -n -c -M --arg v "$FILENAME" '{"type":"test","params":{"item":{"file": $v}}}'
{"type":"test","params":{"item":{"file":"/tmp/xxxx"}}}

// 去掉 -c 参数可格式化输出json
$ jq -n -M --arg v "$FILENAME" '{"type":"test","params":{"item":{"file": $v}}}'
{
  "type": "test",
  "params": {
    "item": {
      "file": "/tmp/xxxx"
    }
  }
}

循环拼接

$ jq -R -M -c -s 'split("\n")' < list
["aa","bb","cc","dd","123","r4","t55","5t","gg","1","rgs","g","f","gs",""]

数组末尾的空字符串,可以另想办法处理下。

2. 解析JSON

这块可直接学习jq官方教程,由浅入深,讲的很详细。

        jq '.foo'
Input    {"foo": 42, "bar": "less interesting data"}
Output    42

        jq '.foo'
Input    {"notfoo": true, "alsonotfoo": false}
Output    null

        jq '.["foo"]'
Input    {"foo": 42}
Output    42

        jq '.[0]'
Input    [{"name":"JSON", "good":true}, {"name":"XML", "good":false}]
Output    {"name":"JSON", "good":true}

        jq '.[2]'
Input    [{"name":"JSON", "good":true}, {"name":"XML", "good":false}]
Output    null

        jq '.[2:4]'
Input    ["a","b","c","d","e"]
Output    ["c", "d"]

        jq '.[2:4]'
Input    "abcdefghi"
Output    "cd"

        jq '.[:3]'
Input    ["a","b","c","d","e"]
Output    ["a", "b", "c"]

        jq '.[-2:]'
Input    ["a","b","c","d","e"]
Output    ["d", "e"]

        jq '.[-2]'
Input    [1,2,3]
Output    2

Nginx配置SSL支持

贴下配置,方便下次查阅。

server
    {
        listen       443;
        server_name domain.com;
        index index.html index.htm index.php default.html default.htm default.php;
        root  /path/to/webroot;
        ssl on;
        ssl_certificate /path/to/site.crt;
        ssl_certificate_key /path/to/site.key;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; //设定支持的协议,禁用SSLv3
        ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"; //禁用RC4等
        ssl_prefer_server_ciphers on;



<h3>如果需要自动跳转http到https,可以在上面的server配置块上面再加一个server块,内容如下</h3>

server
    {
        listen       80;
        server_name domain.com;
        return 301 https://$server_name$request_uri;
    }

<h3>如果需要http和https(ssl)共存,可以参考下面的配置</h3>

server {
           listen 80 default backlog=2048;
           listen 443 ssl;
           server_name domain.com;
           root /var/www/html;

           ssl_certificate /usr/local/Tengine/sslcrt/domain.com.crt;
           ssl_certificate_key /usr/local/Tengine/sslcrt/domain.com.key;
       }

# 把ssl on;这行去掉,ssl写在443端口后面,这样http和https的链接都可以用

SSH Authentication Refused: Bad Ownership or Modes for Directory问题解决办法

配置密钥登录SSH的过程中遇到些问题,记录下方便后续工作。

1、SSH Authentication Refused: Bad Ownership or Modes for Directory /home/user

从字面上可以看出是目录的属主和权限配置不当,查找资料得知:SSH不希望home目录和~/.ssh目录对组有写权限,通过下面几条命令改下

chmod g-w /home/user
chmod 700 /home/user/.ssh
chmod 600 /home/user/.ssh/authorized_keys

还有另外一种方法是在sshd_config文件中配置

StrictModes off

,但是不推荐,因为这是全局配置,关闭了不安全。

2、fatal: bad ownership or modes for chroot directory component "/var/www/project"

在sftp配置中,

ChrootDirectory /var/www/project

参数限定用户登录sftp后的活动范围。
这个参数配置要求比较严格,需要限制目录的每个上级目录的属主都是root,例如/var/www/project,它就要求/ /var /var/www三个目录的owner都是root。如果不好更改,可以用软链接的形式实现,例如

ln -s /var/www/project /home/project

配置Apache mod_proxy模块反向代理tomcat,去除tomcat访问的端口号

首先检查apache根目录下的modules目录中是否有mod_proxy.so、mod_proxy_http.so文件,如果没有,说明编译安装apache时没有配置proxy模块。可以参照这篇文章的方法,给已经编译运行的Apache增加mod_proxy模块。

配置完后重启apache生效,此时就已经加载mod_proxy.so、mod_proxy_http.so这两个模块了,可以

grep 'proxy' conf/httpd.conf
检查下。如果有用到AJP或者tomcat负载均衡,则还需编译增加其他模块,这个再自行搜索即可。

然后在apache的conf/vhost/目录中新增一个conf配置文件,可以复制已存的拿来改。如下是我的内容

<VirtualHost *:80>
DocumentRoot /www/domain.com
ServerName www.domain.com
ServerAlias *.domain.com

ProxyRequests Off
ProxyPass /download.html !   #表示改URL不反代
ProxyPass /img !
ProxyPass / http://www.domain.com:8080/MyTomcatApp/
ProxyPassReverse / http://www.domain.com:8080/MyTomcatApp/
</VirtualHost>

参考文章:

http://freeloda.blog.51cto.com/2033581/1301382
https://rvdb.wordpress.com/2012/04/26/reverse-proxying-tomcat-webapps-behind-apache/

Ubuntu sftp配置及上传文件权限问题解决方案

编辑配置文件/etc/ssh/sshd_config

Subsystem sftp /usr/lib/openssh/sftp-server

改成

Subsystem sftp internal-sftp

增加如下内容

Match user ftpuser
ChrootDirectory %h
ForceCommand internal-sftp

多个用户则重复配置上面的三行。

然后修改用户,设定shell为/bin/false,就可以限制用户无法登录SSH只能使用sftp


配置默认上传文件权限

Subsystem sftp internal-sftp -u 0002

增加上面加粗的-u 0002

表示配置sftp中的umask为0002,这样在sftp里新建目录权限就是775(777-002),文件权限664(666-002)

参考文章:

Limiting Access with SFTP Jails on Debian and Ubuntu
How do I set default permissions for SFTP for an Ubuntu Server?
【系统管理】Linux Umask 介绍