Software packages also contain information about their abilities. Files such as info.plist must be scanned and added to the LaunchServices database. For example, if it grows, a right click on a file may take a little longer to make this noticeably enough.
Sometimes, a program installs a bucket full of fonts, which no one wants, mainly certain office applications.
Caches are nice if they are small. MacOS caches grow and are self-correct over time, requiring a deletion because there is no build-in mechanism to fix it.
LaunchDaemons and LaunchAgents (on / System / Library /, ~ / Library / and / Library /)  are also within application packages and at least: "Don't run on time" doesn't mean that They don't run "once", for example when you don't really look. These are for LaunchDaemons, LoginItems and XPCServices. Usually, these are moved on app launch to their "correct" locations, but they can also be active from within the app (or by a proxy copied to a nested container in ~ / Library).
̵1; really bothered: VOX.app/Contents/Library/LoginItems/Loop.app & VOX Agent
– Another example may be DefaultFolderX: every time a file dialog opens this app adds custom routines from 9 frames to a one-time process
– some audio software installs plug-ins in / Library / Audio / Plug-Ins which then hang in the audio system and are not visible from Activity Monitor, despite adding the system load when the sound is playing, the app destroys the plug at the back
– HP Printer Software has a device circuit running, which eats the CPU and is hard to find on disk
All the programs go by crude. Caches, receipts, preferences, database entries are no longer valid etc. Everything within an application package goes if you drag it to the trash. But I do not know of any single de-installer that really catches the all cruft that is being filled by most applications.
Therefore, several programs installed the slower the system gets. Compared to the 90s Windows, this should not come out of hand as easily.
Some of the best places to really care for other than / Programs:
/ Library / Extensions
/ Library / Audio / Plug-Ins
/ Library / Bundles
/ Library / Caches
/ Library / Frames
/ Library / Input Methods
/ Library / Internet Plug-Ins
] / Library / LoginHook
/ Library / QuickLook
/ Library / ScriptingAdditions
/ private / var / db
/ private / var / folders
This list is far from complete and also applies to the corresponding folders under /Users.
Uncontrolled – and often quite unnecessary – installers, write Files all over the place are one problem. Unless you can inspect them with eg. Pacifist, they should be considered "unsafe" and closely monitored.
There is no uninstall program in macOS. Once a program is installed, you need to rely on the developer that the program does not write files anywhere and adds logins, boot programs, extensions, frames, fonts, daemons, input methods or plugins – to the system (or user library)  The sheer number of applications / executables presents a slowdown factor. A rather small one, compared to the above, but it still adds to the problem.
Actual Malware or PUPA. By downloading and trying out many programs, the chance that a bad apple will enter the system increases and destroys the mark.
If the system is installed on HFSplus, of course, the good old inefficiency of the system comes to a certain extent. For example, directory structures become more inefficient over time, a reason why products like DiskWarrior exist. At APFS things are different, but there are not enough reports out there to say empirically whether it got better or worse overall. Usually, all SSDs are reduced when they become full and with age. The more files on it, the more decisions the controller has to make on where to write the next bits and in the background, all these pieces must be mixed around for TRIM, GC and wear (not to mention common bugs like in Samsung 840EVOs ).
After a new installation you can realize that it is not necessarily a very clean installation: Sometimes a problem or shutdown factor was the reason for installing the system, but then you used Migration Assistant, which is not always intelligent enough not to also migrate The problem for the new installation too.
Psychology – After installing a clean system it will surely feel great to be able to do things again, after sitting through long wait and restart and rebuilding things like caches and spotlight indexes, etc.
When your Mac goes slow, give it a tuneup
Speed up your Mac
7 Common Mistakes That Reduce Your Mac
Why Does My Mac Run Slow?
Why does my Mac go slow? 5 Quick Tips to Speed Up Your Mac!
26 Reasons why a Mac computer runs slowly [Infographic] Keep apps and extensions to a minimum
How to do it: Clean and increase your Mac with free, reliable downloads
How-To – Encrypt your hard drive
Like @jksoegaard with proper comments: A single 1KiB file that is never used will not lower the system. The above is merely addressing the general flare that poorly written and maintained software loads without benefit to any user on small but expensive SSDs, where these files and larger are numbering in thousands. A few kilobytes here a few megs there, after a while you realize that some gigabytes are laid up and are just plain waste.
Unfortunately, falls most of the macOS software now in that category (primarily example: /System/iOSSupport/System/Library/PrivateFrameworks/MobileIcons.framework/Versions/A/Resources/MessagesDefaultIcon-32.png -> 32×24 pixels, but occupy the modest amount of 10,265,058 bytes on the disk! Or the latest cruft by design.) It is not worth hunting these inflators daily, but you paid for this waste. So, once in a while, good scrubbing, when possible, is okay.