[bash] Extract all distinct lines from file

Using a combination of cut and uniq, it is possible to extract all distinct lines from a given file:

cut -d\; -f1 file.txt | uniq

It is optionally possible to sort the file as well adding sort to the command:

cut -d\; -f1 file.txt | sort | uniq

[bash] Extract characters before marker from all lines in a file with sed

Using sed, it is possible to quickly extract all characters before a given marker from all lines in a file with:

sed 's/MARKER.*//' file.txt

[Docker] Move docker directory to different location on NTFS filesystem

The default location of docker files (images, volumes, networks, etc) on most distributions is /var/lib/docker

This can be easily changed with the daemon.json configuration file.

First stop the docker service and make sure it is completely stopped.

You can now edit or create the configuration file under /etc/docker folder. In there, you can specify a new location for the docker files with the graph attribute:

{
   "graph": "/path/to/new/docker_folder"
}


You can then copy ALL the content to the new location or let docker recreate everything from scratch there.

WARNING: if the new location is on a different filesystem you might need to change the storage driver AND existing data might become inaccessible (it will remain on the filesystem though)

To set a new storage driver, set the storage-driver attribute as well (example for NTFS):

{
   "graph": "/path/to/new/docker_folder",   "storage-driver": "vfs"
}

[GNOME] Create application shortcut that can be added to favorites

Some applications you install come as archives that you extract in a desired location. However, not all come with an installer that takes care of creating the proper application entries, leaving the task to the user.

You can add a shortcut for any application by creating a file APPLICATION_NAME.desktop under /usr/share/applications and adding this content:

[Desktop Entry]
Version=1.0
Encoding=UTF-8
Type=Application
Terminal=false
Categories=OPTIONAL_CATEGORY_LIST

GenericName=APPLICATION_NAME
Comment=OPTIONAL_COMMENT
Name=APPLICATION_NAME
Path=FULL_PATH_TO_APPLICATION_FOLDER
Exec=FULL_PATH_TO_APPLICATION_EXECUTABLE

Icon=OPTIONAL_FULL_PATH_TO_ICON

StartupWMClass=APPLICATION_WM_CLASS

[Linux] Find WM_CLASS value for an application

In order to group application windows together, the WMClass and WMInstance attributes have been specified in the Inter-Client Communication Conventions Manual for X server 11.

Long story short, they are two strings that are used to associate application windows to a specific application, they can be set by the application (WMClass) or the user (WMInstance).

They are very useful especially for those applications that are customizations or based on existing tools, which alone would have a different value (eg Java or Eclipse based apps).

Finding the value for a window is as simple as opening a terminal and running:

xprop

Then clicking on the desired window

[Linux] Custom keyboard layout QWERTZ to QWERTY

We already saw how to customize a keyboard layout on Windows, now it's the turn of Linux, using XKB for Xorg.

Remapping keys is as simple as editing a text file. You will find all available layouts under /usr/share/X11/xkb/symbols, you can use them as basis for a new layout or directly edit them.

Pick the layout you want to change (maybe make a backup first), then invert the VALUES ONLY of the two lines containing the lower and uppercase z and y, then save.

You will need to restart X server and the changes will be in effect.

[Windows] Share environment variables in Windows Subsystem for Linux

Now that you have your fancy WSL up and running, you want to use it and all your environment variables are NOT available (of course).

Luckily there is a workaround, using the WSLENV variable. Unfortunately, it is not a direct 1:1 translation of your existing variables, rather you will need to craft them manually and convert between the two environments before you run a command or switch environment.

For example, to pass your JAVA_HOME correctly to wsl:

set JAVA_HOME=%JAVA_HOME%
set WSLENV=JAVA_HOME/up
wsl
echo $JAVA_HOME

And the magic is behind that last /up flag which says: set the value when invoking WSL from Windows (u) and translate between Windows and Unix paths (p)

WARNING: spaces are NOT escaped
WARNING2: only ONE variable is shared, so you cannot share both JAVA_HOME and PATH for example at the same time

[Windows] Activate and configure Windows Subsystem for Linux

Windows 10 introduced an update that adds a very cool feature: Windows Subsystem for Linux. If you are a fan of Cygwin, or similar tools, you will greatly like this feature as well.

Setup could be a bit more straightforward, but still not a big issue:

1. Enable Windows Subsystem for Linux as a Windows feature from Control Panel > Programs and Features > Turn Windows features on or off
2. Reboot
3. Make sure LxssManager service is running
4. Install a Linux distro from Windows store, search for "WSL" and pick the one you prefer, for example Alpine or Ubuntu, if no distro is installed, you get an error instead:

Windows Subsystem for Linux has no installed distributions. Distributions can be installed by visiting the Windows Store: https://aka.ms/wslstore Press any key to continue...

And going to that link brings you nowhere.. so just install it yourself from the store.

After the installation is completed (might ask you to setup a user and password), you can open any command prompt and type wsl to launch it.

Be careful, that command was different in the past: lxrun or bash and there are apparently 2 versions of WSL as well, use whatever makes it run for you

[cURL] Use file content as payload

With curl, it is possible to specify a file content as payload simply with:

-d @$file

For example:

curl -X POST some_url -H some_headers -d @$file

[bash] Loop over all files in folder filtering by extension

In bash, it is possible to loop over all files with a specific extension in a folder with:

for file in folder/*.extension
do
    something with $file
done

Note that if directory is empty, one iteration will performed on the nonexistent *.extension file. To avoid that set the nullglob shell option for that run only as such:

shopt -s nullglob

your_script_here

shopt -u nullglob

[bash] Store content of file in a variable

In bash, it is possible to store the content of a file in a variable simply with:

file=/path/to/somefile.something
content="`cat $file`"

Note the special quotes used

[bash] String substitution with parameter expansion

In bash, it is possible to use parameter expansion with syntax ${expression} to perform a string replace. Here are some examples, the first value MUST be a variable.

To replace ONE occurrence of substring:

result=${string/substring/replace_with}

For example

string=grogblog
result=${string/blog/logs}

will return groglogs

To replace ALL occurrences of substring:

result=${string//substring//replace_with}

For example

string=grogbloggrogblog
result=${string//blog/logs}

will return groglogsgroglogs

Note the difference is simply the // instead of / at the beginning

[bash] Substring with parameter expansion

In bash it is possible to use parameter expansion with syntax ${expression}to perform a substring operation. Here are some examples, the first value MUST be a variable.

To cut out some text from the BEGINNING of the string:

result=${string#string_to_cut}

For example

string=groglogs
result=${string#grog}

will return logs

To cut out some text BEFORE the END of the string:

result=${string%cut_point*}

For example

string=grog.logs
result=${string%.*}

will return grog

They can also be chained to operate on the same variable in a sequence

[Linux] Text handling for counting and replacing words

Here is a list of handy commands for VI and SED to count and replace text in a file. For both editors the escape character is /:

VI:

- count total words:

 :%s/\i\+/&/gn    

- count occurrences of particular word:

 :%s/WORD/&/gn    

SED:

- copy all text from marker until end of file:

 sed -n -n '/MARKER/,$p' in >> out  

- replace all occurrences of SOURCE to TARGET with optional characters before SOURCE:

 sed -E 's/(OPTIONAL)?SOURCE/\1TARGET/g' in >> out  

[Ubuntu] Install Microsoft fonts

Most of the world uses Microsoft Windows and Office, therefore it might be sometimes necessary to generate documents using Microsoft fonts, which of course are only available on Windows.

For Ubuntu, the ttf-mscorefonts-installer package contains SOME of those fonts, simply install it and you can start using them AFTER regenerating the font cache with fc-cache

[Linux] Bind special key press to ALSA control action

If using ALSA, it's possible to bind special keyboard keys to action controls with a few configuration lines.

First of all, to find the available action controls, run in a terminal: amixer scontrols

Then, find the key event we need to bind to the desired action, and add it as an event,action pair in a new configuration file under /etc/acpi/events for example:

event=button/mute MUTE 00000080 00000000 K
action=/usr/bin/amixer sset 'Master',0 toggle

which in this case for my lapotop binds the mute button to the mute action. It is also possible to execute a script instead, for example:

event=ibm/hotkey HKEY 00000080 00001009
action=/etc/acpi/undock.sh

which runs some actions when the laptop is undocked.

[Linux] Find which event is generated on special keypress

To determine which ACPI event is generated when a special key is pressed, simply open a terminal and run: acpi_listen

Then press any special key (media keys and such, normal keys won't generate an event) and you will see the output eg:

button/volumedown VOLDN 00000080 00000000 K
button/mute MUTE 00000080 00000000 K
button/volumeup VOLUP 00000080 00000000 K

[Ubuntu] Boot USB after error mmx64.efi not found

UEFI, great leap into the future and also great pain.

There is a funny error that can happen after creating a UEFI bootable USB to install a linux distro for example. If the installation is not completed immediately and the system is restarted, some files are left in some area of the EFI partition (I guess) and subsequent attempts to boot the exact same device will fail with an error:

Failed to open \EFI\BOOT\mmx64.efi - Not Found
Failed to load image \EFI\BOOT\mmx64.efi: Not Found
Failed to start MokManager: Not Fond
Something has gone seriously wrong: import_mok_state() failed

The easiest fix, is to simply browse the USB drive and go into the efi/boot folder, then copy the grubx64.efi file and rename it to mmx64.efi

The next boot will work again as intended. Magic of the machines

[Ubuntu] Revert kernel back to older version

Intro story: So much luck in a single day.

I always keep only 2 kernel versions after every update, so that I can boot back if something goes wrong. This usually helps, except today I decided to upgrade my Ubuntu from 16.04 to 18.04 since I like spending my nights fixing stuff after unnecessary upgrades, but hey, the future is there.

So I first fully updated my current installation (1st kernel bump), then migrated to the new one (2nd kernel bump) and it turns out I got two 4.15 kernels, with only different minor version.

Double interesting, my wireless card:

02:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8821AE 802.11ac PCIe Wireless Network Adapter

apparently has issues (read: bug) with kernel 4.15 preventing it from working at all (not even connected to the router), and even more interesting, on one version it can actually only connect to websites under google.com domain..

Anyway, quick fix was to revert back my kernel, here is how for future me to remember:

1. go to http://kernel.ubuntu.com/~kernel-ppa/mainline/ and download in the same folder: linux-headers-VERSION-generic-XXX_SYSTEM.deb, linux-headers-VERSION_all.deb, linux-image-VERSION-generic-XXX_SYSTEM.deb for your system, eg I have 64 bit system and got amd64 files
2. install them by going into the folder where you downloaded them, make sure there are only those three .deb files and run: sudo dpkg -i *.deb
3. restart the system, then press SHIFT at boot to enter the GRUB choices, then check from the top line (number 0) to the line of your kernel version. If your installation has the "Advanced" menu entry and does not display all kernels, go there and do the same counting. We need this to set a default GRUB entry to always boot in this kernel
4. edit /etc/default/grub and change the value of GRUB_DEFAULT to the number you counted OR to (exactly as shown) "1>N" if you have the "Advanced" menu option. This means you want to boot into the Nth entry (starting from 0) under the "Advanced" menu. Also set GRUB_TIMEOUT to some value in seconds that's more than 1, to have some time to block the boot process in case of mistakes
5. run sudo update-grub and you are set

Note that this is a TEMPORARY fix, you would not want to have a fully updated system run on an older version of the kernel, mainly because some obscure dependency or feature might be missing/not working, therefore check back periodically to see if your issue has been addressed and try the latest kernel version out!

[xorg] Set display to full RGB output

Depending on the monitor being used, the RGB color model supported might be either "Full RGB" or "Limited 16:235"; usually the latter is the one used by TV monitors.

To test the output setting when using xorg, first find out the name of the video connection being used by running:

xrandr

which will list the names and statuses of the available connections. Then manually change the output setting with:

xrandr --output NAME --set "Broadcast RGB" "Full"

Or use "Limited 16:235" instead.

This will not survive a reboot, therefore use whatever method suits you best to force this at every boot, eg by editing/creating the ~/.xprofile file