[bash] Extract all distinct lines from file

Using a combination of cut and uniq, it is possible to extract all distinct lines from a given file:

cut -d\; -f1 file.txt | uniq

It is optionally possible to sort the file as well adding sort to the command:

cut -d\; -f1 file.txt | sort | uniq

[bash] Extract characters before marker from all lines in a file with sed

Using sed, it is possible to quickly extract all characters before a given marker from all lines in a file with:

sed 's/MARKER.*//' file.txt

[bash] Loop over all files in folder filtering by extension

In bash, it is possible to loop over all files with a specific extension in a folder with:

for file in folder/*.extension
do
    something with $file
done

Note that if directory is empty, one iteration will performed on the nonexistent *.extension file. To avoid that set the nullglob shell option for that run only as such:

shopt -s nullglob

your_script_here

shopt -u nullglob

[bash] Store content of file in a variable

In bash, it is possible to store the content of a file in a variable simply with:

file=/path/to/somefile.something
content="`cat $file`"

Note the special quotes used

[bash] String substitution with parameter expansion

In bash, it is possible to use parameter expansion with syntax ${expression} to perform a string replace. Here are some examples, the first value MUST be a variable.

To replace ONE occurrence of substring:

result=${string/substring/replace_with}

For example

string=grogblog
result=${string/blog/logs}

will return groglogs

To replace ALL occurrences of substring:

result=${string//substring//replace_with}

For example

string=grogbloggrogblog
result=${string//blog/logs}

will return groglogsgroglogs

Note the difference is simply the // instead of / at the beginning

[bash] Substring with parameter expansion

In bash it is possible to use parameter expansion with syntax ${expression}to perform a substring operation. Here are some examples, the first value MUST be a variable.

To cut out some text from the BEGINNING of the string:

result=${string#string_to_cut}

For example

string=groglogs
result=${string#grog}

will return logs

To cut out some text BEFORE the END of the string:

result=${string%cut_point*}

For example

string=grog.logs
result=${string%.*}

will return grog

They can also be chained to operate on the same variable in a sequence