I still read books mostly off my phone (samsung s8), and while most are surprised I'm not blind yet at 43, it works for me, including movies or whatever. Each their own.
If you want a wifi-based media server, they are out there, but I've not seen any I'd invest in that didn't reek of chinese crap, or who knows what backdoors built into some generic unmaintained linux os. A pi of some form would be your best option for control, a pi zero + wifi + battery, and ymmv how well you tune it for power savings. If keeping in a pocket of a backpack, add (very) large external usb battery, voila.
If including any networking, ethernet or particularly wireless, performance will suck on any mobile-ish chip I find. Minimal cpu + minimal wifi will probably not be kind for battery-based usage. I've looked into "portable nas's" before, and this was always the issue, regardless of storage mediums, ranging from spinner disks to ssd. Maybe different with modern nvme's or other if an efficient interface, not looked in a bit.
If not using with mobiles, why not just go usb-to-disk then? Most usb enclosures are fairly compatible across devices, even android with otg. and you can get nvme-based usb enclosures that are small, yet as capacious/fast as you wish to spend to make them. I've done this with old "still alive" ssd's of mine as a mule for media/music when I travel, or expendable steam game volumes. An amazon vendor screwed up and sent me a small nvme-to-usb enclosure in place of something else ordered recently, rather tiny and discrete with usb 3.1. I kept one to play with...
I've seen PoE daughterboards for a pi even that are intriguing, as I tend to keep poe available in my house where wired, but not practical for mobile. Most enterprises though...
There are also things like a usbarmory, full hackable arm on a usb-stick. I have one and tinkered with for a bit, just add usb power + as big a sd as needed, and offers some fair capabilities for diy embedded hacking with a normal debian distro. Again pi far more friendly, but make due with what you have/need.
I still say keeping a filer or two, (synology, qnap, freenas, other) at home and available via stable vpn is a good fallback as any. One can (I think?) always tether off a mobile these days as long as not carrying a flip phone still, works well for me when I just need to get at some data I forgot to replicate.
-mb