You should see File Explorer in the list of search results. Right-click it, then choose Pin to taskbar. Presto! When you return to the Desktop, you'll see your beloved File Explorer icon.

Right-click on the Taskbar icon and further click on ‘Pin to Taskbar’ That’s all you have to do to permanently base your File Explorer icon on the Windows 11 Taskbar.