I asked ChatGPT to give me a solution in php8 for this. Here’s what it wrote:
<?php
$log_file = '/var/log/download.log'; // change this to your desired log file location
$download_folder = '/var/www/downloads'; // change this to your desired download folder location
function log_message($message) {
global $log_file;
$timestamp = date('Y-m-d H:i:s');
$log_message = $timestamp . ' - ' . $message . PHP_EOL;
file_put_contents($log_file, $log_message, FILE_APPEND);
}
if ($_SERVER['REQUEST_METHOD'] == 'POST') {
$urls = $_POST['urls'];
if (is_array($urls) && count($urls) > 0) {
foreach ($urls as $url) {
$filename = basename($url);
$download_path = $download_folder . '/' . $filename;
log_message('Downloading ' . $url . ' to ' . $download_path);
$result = file_put_contents($download_path, fopen($url, 'rb'));
if ($result === false) {
log_message('Error downloading ' . $url);
} else {
log_message('Successfully downloaded ' . $url . ' to ' . $download_path);
}
}
log_message('Downloaded ' . count($urls) . ' files');
} else {
log_message('No valid URLs provided');
}
} else {
log_message('Invalid request method');
}
?>
Does this look reasonable?
I’m not sure if this can be run on a shared hosting environment, however, b/c it would get triggered on ALL incoming requests, which is not needed.
It would need to listen on a specific port#. Is that possible without making it a daemon?
Can it be run only by requests coming into the CGI folder on a specific domain? ChatGPT is sort of going in circles at this point.
-David Schwartz
> On May 15, 2023, at 9:40 AM, David Schwartz via PLUG-discuss <plug-discuss@lists.phxlinux.org> wrote:
>
> Hmmm, kind of like a remote wget …
>
> Actually,they’d tend to be done in batches, so I’d send a list of names to be copied.
>
> Is there a super-simple way for a php script handle one single POST request that only does one thing, without a ton of overhead needed for an entire REST-based service with multiple endpoints?
>
> Wget would probably be overkill assuming the php script can just issue an HTTP file download request.
>
> -David Schwartz
>
>
>
>> On May 15, 2023, at 9:28 AM, Bob Elzer via PLUG-discuss <plug-discuss@lists.phxlinux.org <mailto:plug-discuss@lists.phxlinux.org>> wrote:
>>
>> What about setting up a CGI script on the Linux server that you pass the URL to, it could do a wget to retrieve the file to the directory you specify.
>>
>>
>> On Sun, May 14, 2023, 8:50 PM David Schwartz via PLUG-discuss <plug-discuss@lists.phxlinux.org <mailto:plug-discuss@lists.phxlinux.org>> wrote:
>> I’m building a web app that uses a 3rd-party text-to-speech (TTS) service; it's one of many things supported by a REST service I’ve created that runs on a Windows host somewhere. This service sends requests to the TTS service and gets back a URL to an MP3 file on their server. These files are only there for about an hour before they get deleted.
>>
>> My service sends back those URLs to the client, which is typically running on a mobile device. They can be consumed without any problem at the moment, telling me the TTS provider has disable CORS restrictions.
>>
>> Many of the requests that will be made are unique and will never be duplicated, so the fact that their vocalizations (the MP3 files) get deleted after an hour is not a problem.
>>
>> However, some of them (20-30%) are very likely to be duplicated, and it’s worth saving them somewhere so they can be re-used in the future. (The TTS service charges based on characters sent to them, and by reusing the MP3 files over time, a lot of cost savings can accrue.)
>>
>> In my mind, I need to set up a way to cache these files somewhere.
>>
>> I don’t want to save them on the server that’s hosting the REST service because of the bandwidth costs.
>>
>> I’ve tried a few different things and it turns out this brings up CORS issues.
>>
>> I have my own web host and found out I can add a line to an htaccess file that will allow the files to be accessed. I can’t do that with hosted services like FileStack (which has other limitations as well).
>>
>> It looks like there’s a way to do it with Dropbox by changing the URL from this form:
>>
>>
>> https://www.dropbox.com/s/x12nrtdi08ipo352/sample-abc.mp3?dl=0 <https://www.dropbox.com/s/x12nrtdi08ipo352/sample-abc.mp3?dl=0>
>>
>>
>> to this form:
>>
>>
>> https://dl.dropboxusercontent.com/s/x12nrtdi08ipo352/sample-abc.mp3 <https://dl.dropboxusercontent.com/s/x12nrtdi08ipo352/sample-abc.mp3>
>>
>> So, this brings up the question of HOW TO MOVE THE FILES INTO THE CACHE?
>>
>> Here’s my biggest constraint: I can access the files for up to an hour by using the TTS server’s URLs. Within that time-frame, they need to have been moved over to the host that’s doing the caching. After that, the code will quickly check to see if the requests have already been processed and are in the cache; if so, it will return a URL to the cached file, saving a needless encoding request.
>>
>> If I use Dropbox, I can simply set up the Dropbox app on the server hosting my REST service, and save the files to the Dropbox file tree. They’ll be copied into Dropbox automatically. But this means I’ll have a modest cost associated with maintaining a Dropbox account for this specific purpose ($130/yr).
>>
>>
>> Alternatively, I can copy the files from the REST server over to my own host.
>>
>> What I’d like to ask this group is … what’s the best way to accomplish that?
>>
>> My host is currently on a shared reseller hosting plan, but as this scales-up, I’ll move it to a dedicated host.
>>
>> It’s running cPanel on a Linux server, probably running CentOS.
>>
>> I can set up cron jobs, and I’m told I can get some limited access to a shell (rsh probably) if needed.
>>
>> This is the same host I tested with the htaccess file to allow the files to be accessed without CORS issues.
>>
>> I’m wondering if it’s best to have the REST server copy the files from the TTS site to the other host somehow (eg, with FTP)?
>>
>> Or use something like rsync on the host to sweep files from the TTS site into the host, driven by a list provided by the REST service?
>>
>> Can I run rsync on a Windows host that copies files from server-A to server-B?
>>
>> Or maybe you guys have some better ideas? I’d love to hear some pros and cons about any solutions that might work.
>>
>> Note that I have not considered a cloud service other than FileStack (which I’ve ruled-out using). Regardless, the files will still need to be copied from the TTS provider’s site to the cache host before they get deleted. THIS process is what I’m wanting to resolve.
>>
>> -David Schwartz
>>
>>
>>
>>
>> ---------------------------------------------------
>> PLUG-discuss mailing list: PLUG-discuss@lists.phxlinux.org <mailto:PLUG-discuss@lists.phxlinux.org>
>> To subscribe, unsubscribe, or to change your mail settings:
>> https://lists.phxlinux.org/mailman/listinfo/plug-discuss <https://lists.phxlinux.org/mailman/listinfo/plug-discuss>
>> ---------------------------------------------------
>> PLUG-discuss mailing list: PLUG-discuss@lists.phxlinux.org <mailto:PLUG-discuss@lists.phxlinux.org>
>> To subscribe, unsubscribe, or to change your mail settings:
>> https://lists.phxlinux.org/mailman/listinfo/plug-discuss <https://lists.phxlinux.org/mailman/listinfo/plug-discuss>
>
> ---------------------------------------------------
> PLUG-discuss mailing list: PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
---------------------------------------------------
PLUG-discuss mailing list:
PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss
(text/plain)