How To Execute Shell Commands With Python?

Introduction

Sysadmin, a short-form for “System administrator”, plays an essential role in the company’s IT department.  They often cover a broad range of technical support. For example, sysadmin tasks may range from installing and deploying servers to troubleshooting and technical support for projects.

A regular thing to do, especially for a sysadmin, is to execute shell commands. Routine tasks like system health monitoring, backup & disaster recovery etc., are performed using shell commands.

Execution of shell commands typically ends up in a bash or batch file. Luckily, it can also be done with Python.

It’s reasonable to think: why use python for shell? 

Following are the few reasons which showcase the usefulness of executing shell commands with python:

• Python has an easy and interactive interface for importing required modules.
• If you want to run any external file like .exe or any application, you can utilise these methods.
• Python gives developers more power and control. It helps in automating the workflow. Disk scan, removing unwanted cache files, backup, recovery and many more repetitive tasks that lack creativity and productivity in developers life can be implemented using the python shell scripts. 

Since python is powerful and fast, this article will explain various ways to execute shell commands with python.

What is a shell?

Shell is a term, which is often used and misunderstood. In computing, a shell is software that provides an interface for accessing the operating system’s functionalities. 

Shell in the operating system can be either a GUI (Graphical User Interface) or a CLI (Command Line Interface) based on the functionality and essential operation of the device.

In windows, the shell is software that allows you to interact with the operating system via a “command line”, also known as a terminal in Linux and Mac.

You can open a command line in windows using Windows+R » type cmd » Enter.

Python offers convenient modules to interact with the operating system, such as

• os
• platform
• subprocess
• shutils
• glob
• sys

You’ll see here how to execute shell commands with python using the most used os and subprocess modules.

OS MODULE

The naive approach to run a shell command is by using os.system():

• Let’s first create a new Python file called shell_cmd.py or any name of your choice.
• Second, in the Python file,  import the os module, which contains the system function that executes shell commands.
• system() function takes an only string as an argument.
• Type whatever you want to see as an output or perform an action.

• Run this .py file, and you should see the corresponding output. 
• Executing the code will open the command prompt, as shown below.

Before proceeding further, let’s see one more example of performing the change directory (cd) command using python.

Code:

import os
#First, let's print the current working directory
print("Current Working Directory",os.getcwd())
#Now, let's change the directory
path="F:\Coding Ninjas"
os.chdir(path)
print("Changed to",os.getcwd())

• os.getcwd(): returns the current working directory of a process.
• path: Choose directory-path you want to open
• os.chdir(path): changes the current working directory to the path you pass as an argument.

Subprocess Module

• The subprocess is the most versatile approach and the recommended module to execute shell commands with Python.
• The subprocess module allows you to spawn new processes.
• It connects to the process’ input/output/error pipes and obtains their return codes. 
• This module replaces several older modules and functions like os.system and os.spawn*.

You can find how the subprocess module can replace the older modules and functions in the python official documentation subprocess-replacements section.

Let’s go through the functionalities in the subprocess module for executing shell commands:

subprocess.run()

1. The suggested approach to invoke subprocesses is to use the run() function. This method runs a command and returns a ‘CompletedProcess’ instance after completion.

Below syntax shows the full run() function signature, i.e. the number and types of arguments we can pass.

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None, **other_popen_kwargs)

args:  This may be a list or a string.

Sample python code to print the list of current working directory files in Linux:

import subprocess

file1 = subprocess.run(["ls", "-l"])
print("The exit code was: %d" % file1.returncode)

Note: The highlighted parts are explained separately below the codes to give you a more clear understanding.

• ([“ls”, “-l”]): This is a Linux command that ensures minimal information and one file per line to make processing file names easy. 
• returncode: This denotes the exit status of the child process. 

An exit status of 0 indicates that it ran successfully, and any other number means an error.

Run the command described by args. Wait for the command to complete, then return a CompletedProcess instance, i.e. the return value from run(), representing a finished process.

2. Now, if you want to provide input to a command, the subprocess.run() allows this by its input argument.

import subprocess

input_file1 = subprocess.run(["ls"], stdout=subprocess.PIPE, text=True, input="Hello How are you?")
print(input_file1.stdout)

• stdout=subprocess.PIPE: This indicates subprocess.run() to redirect the command’s output to an object to be manually read later.
• stdout: This captures output from the child process.
• text=True:  This returns stdout and stderr as strings. The default return type is bytes.
• input: This argument allows passing data that should be a string or None if no data is sent to the child.

The subprocess.run() function gives us extensive versatility that os.system() doesn’t while executing shell commands.

subprocess.Popen()

For more advanced functionalities, we can explore the subprocess.Popen() class. Below syntax shows the full Popen() function signature, i.e. the number and types of arguments we can pass.

class subprocess.Popen(args, bufsize=-1, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=None, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=(), *, group=None, extra_groups=None, user=None, umask=-1, encoding=None, errors=None, text=None)

args can be a list, a single string or a path-like object.

• Popen class is responsible for handling the underlying creation and management of the executed process.

The following python code will print the stdout,stderr part of the subprocess:

import subprocess

process = subprocess.Popen(['ls', 'Hello How are you?'],
                     stdout=subprocess.PIPE, 
                     stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
stdout, stderr


Output:
(b'Hello How are you?\n', b'')

• communicate(): It interacts with the process and primarily manages the input and output. 
  • Send data to stdin. 
  • Read data from stdout and stderr until end-of-file is reached. 
  • Then wait for the process to terminate and set the returncode attribute.
  • Finally, returns a tuple (stdout_data, stderr_data). The data will be strings if the file was opened in text mode; otherwise, bytes.

Now, suppose we don’t want to wait for the complete execution of the process, then 

Popen.poll() comes into the picture.

• If a command has finished running, it returns an exit code else None if it’s still executing.

Now, after exploring the core idea behind these modules, you must be wondering about the ideal conditions to use each one of them.

Don’t worry. The table below shows some differences to facilitate your decision depending upon usage:

	os.system	subprocess. run	subprocess.Popen
Requires parsed arguments	no	yes	yes
Waits for the command	yes	yes	no
Communicates with stdin and stdout	no	yes	yes
Returns	return value	object	object

You have understood the basics of executing shell commands with python. You can experiment with different other commands to automate more complex and run external tasks. For now, let’s look at some frequent questions regarding the shell commands with python.

Frequently Asked Questions

Key Takeaways

This article starts with a basic introduction to python shell commands and why one should use them. Further, it explains three main methods to execute shell commands with python:

• os.system()
• subprocess.run()
• subprocess.Popen() 

By Aanchal Tiwari