Watch Folders and Fire Custom Events with Zadara Container Services (Docker)
Ever wish your storage was just a little more intelligent and customizable? That day is here!
With the inclusion of Zadara Container Services (ZCS) in our 15.07 software release, Docker can be utilized by an administrator to perform nearly limitless operations on your share’s files. Running Docker containers on your Zadara storage comes with the unique ability to do data processing with the data locally accessible to the program – rather than needing to traverse the network layer for data access.
Here are but a few of the possibilities:
- Video Processing – Watch a folder for new raw video files, transcode them to one or many desired output formats, then store the output to a preferred folder.
- Aging – On a scheduled basis, find all files in a path older than X days, and move them to another folder, or copy them to another larger share.
- Anti-Virus – When a user writes a file, trigger an Anti-Virus scan.
Visually, this looks like the following diagram:
inotify
The Linux kernel and glibc support a facility called inotify which can very efficiently notify a watching program that a new file has been created, a file has been changed, a file has been deleted, or many other actions.
In this example, we will use a shell script that calls “inotifywait” to watch for written files, then take an action on those files.
Script
The script demonstrated here will take a single argument – a watched directory – and call “inotifywait” on it. Inside the Bash while loop working with “inotifywait”, we can take any action we choose.
In this example, we are simply echoing to standard output that a file has been written. This echo statement shows how the filename is captured into a variable, “$file”, and can be used for any purpose.
First, let’s validate that the script is receiving the correct number of arguments, and exit with a usage statement if not:
if [ $# -lt 1 ] ; then echo "Usage:" echo " $0 " exit 1 fi
Now, we’ll store the first argument ($1) as a variable called “$watchdir” for legibility:
watchdir=${1}
Finally, we’ll launch “inotifywait” – which will continuously run, and we’ll loop through any ongoing files it finds with a Bash while statement:
echo "Waiting for new files in ${watchdir}..."
inotifywait ${watchdir} -m -q -e close_write --format %f . | while IFS= read -r file; do
# Do some work here using ${file} variable... For example, to create
# a Redis key "newfile" with the contents being the filename:
#
# redis-cli -h 10.10.10.10 set newfile $(basename ${file})
#
# The "hostname" can be changed into an argument from the "Args"
# setting. e.g. "redis_server=${2}", then "-h ${redis_server}"
# Output what inotify saw to the console log. Log can be downloaded
# from the "Logs" tab for the running container.
echo "-------------------------------------------------------------"
echo "Saw new file ${file} in ${watchdir}"
done
Conclusion
As you can see, the ability to run Docker containers on top of your storage is a very powerful feature. The preceding code provides a framework for far more powerful and customizable software that integrates nicely with your storage.
For more information:
To see a sample Dockerfile and script with this code, please refer to our Zadara Container Services repository: https://github.com/zadarastorage/dockerfiles/tree/master/inotify
And as always, you can refer to our documentation on how to create your own Docker containers: https://github.com/zadarastorage/dockerfiles
If you have any questions, contact us at Zadara Support.
