Several ways to execute python scripts using shell scripts

created at 07-14-2021 views: 28

scenarios we use shell language

The shell language is a packaging script for a wide range of use cases for Unix-like systems
In general engineering practice, we will need to use it in the following scenarios:

  1. Execute a set of actions of the operating system in the OS, such as move, copy files, find files, replace strings in files, etc. We will combine the commands into a shell script
  2. Operation and maintenance of tools, such as start, stop, etc. of a program
  3. System operation and maintenance tools, such as regular file cleaning, resource utilization information reporting, etc.
  4. Run a set of tests to test our web service
  5. Other needs for processing small data volume and processing micro logic

If our code volume exceeds 100 lines, consider using python is a better choice.

When do we encounter the problem of embedded python script in shell language

The typical scenario is, parsing a returned json string, using shell processing is really a bit troublesome, but we want to quickly verify some ideas, of course we can change to a python script to do this, but the script has already written the header and At the end, there is only one data analysis left. At this time, avoid writing complicated sed and awk processing commands, and use python's json library to process it will be very light;

For another reason, maybe the use of this function has been fixed. The upstream program is a command that requires a shell, such as a bash plug-in in the CI link of the pipeline. There may be a large number of such bash scripts in the continuous integration built in the Jenkins suite. 

Below we consider such a typical scenario:
Obtain the return data of an interface from the http interface, and then process it.
This return is a more complicated json structure, how to parse it?

For example, we need to process a json string of this shape, which is returned from a remote web service, and we need to find the age named Zhangsan

{
    "status": 0,
    "time": "2020-02-08",
    "datas": [
        {
             "name": "Zhangsan",
             "age":18,
             "heigth" : 170,
             "address":"Shenzhen", 
            "email": "zhangsan@google.com"
        },
        {
            "name": "Lisi",
            "age": 20,
            "heigth": 185,
            "address": "Nanjing",
            "email": "lisi@google.com"
        }
    ]
}

Use sed and awk to solve

# !/bin/bash

But we know that it is more convenient to use python to parse the json string, and programming is also very natural, so a reasonable idea is to embed python logic in the shell script for json parsing

several ways to embed python script

Option 1:

Pile python statements directly in the shell statement, write the structure to the console, and the subsequent shell program can get the output from stdin from the pipeline

x='{"key1": "value1", "key2":"value2"}'
python -c "import sys,json;sys.stdout.write(json.dumps(json.loads('$x')));"

This method is obviously very effective for simple json structure, but for our example, we can use the following method to get the age of the first set of data, but it is actually hardcode, which is only suitable for returning only one set in the structure In the case of data, if the data returned by the http interface is more complicated, it is difficult to handle it here. If we try to write a loop, it will make the code string behind python -c very complicated.

#!/bin/bash

function GetPersonAgeFromResp() {
    local data=$1
    if which python;
    then
        local data=$1
        python -c "import sys,json;sys.stdout.write(json.dumps(json.loads('$data').get('datas')[0].get('age'));"
        return 0
   else
       return 1
}

Therefore, there is another way to use the echo command to echo the python statements line by line to stdin, and then use pipe redirection as the input of python, as follows:

The echo command displays the python commands, which is clear to read and easy to maintain. If there are any problems, you can check them intuitively.
Personally recommend the following two options:

Option 2

#!/bin/bash
d='{"status":0,"time":"2020-02-08","datas":[{"name":"Zhangsan","age":18,"heigth":170,"address":"Shenzhen","email":"zhangsan@google.com"},{"name":
"Lisi","age":20,"heigth":185,"address":"Nanjing","email":"lisi@google.com"}]}'
function GetPersonAgeFromResp() {
      (
       local name=$1
       echo "import sys,json"
       echo "x=0;datas=json.loads('$2').get('datas')"
       echo "for data in datas:"
       echo "    if data.get('name') == '$name':"
       echo "        x=data.get('age');break;"
       echo "print(x)" 
     ) | python
    }

GetPersonAgeFromResp 'Zhangsan' $d

Output:

$18

Option3

Directly use cat << EOF> filename.py [statements] EOF to put python statements together, organize them into a file, and then execute,

Except for an outer control, basically Python code is written.
This situation is suitable for the case where the python script is relatively long.

Of course, you can't write this in large quantities, otherwise why not write all tasks as python scripts?

cat << EOF > /tmp/parse_json.py

import json
import sys
import os

p = {
    "name": sys.argv[1],
    "key": sys.argv[2],
    "raw_data" : sys.argv[3]
}

data_object = json.loads(p["raw_data"]).get('datas')
for o in data_object:
    if p['key'] in o:
        age=o[p['key']]
        sys.stdout.write(str(age) + '\n')
        sys.exit(0)

sys.stdout.write('Could not find the key: %s\n' % p['key'])
sys.exit(1)

EOF

python /tmp/parse_json.py "zhangsan" "age" $d
python /tmp/parse_json.py "zhangsan" "haha" $d

Output:

$ 18
Could not find the key: haha

summary

If the scale of scripts and json strings is large, you should consider giving up shell scripts. Python scripts will be more natural and faster. Unless your context is really fixed, consider embedding python in shell scripts.

created at:07-14-2021
edited at: 07-14-2021: