Threaded Mode | Linear Mode

GrumpyPatzer · 25-11-2023, 17:47

Having been prompted to investigate alternatives to the very old (but customisable) version of Twonky that I have been using for over a decade on a QNAP 459 NAS, I am now thinking about switching to using a Raspberry Pi instead of my newer QNAP NAS (which appears to have an intermittent power supply problem), and am even wondering whether it would be viable to use cloud-based storage.

Have any of you attempted to use cloud storage with Minimserver (or, indeed, any other music streaming application) and, if so, were you able to make it work satisfactorily?

Because of this NAS problem, I decided to try loading my music collection to Microsoft OneDrive cloud storage (we have 6TB, albeit split into 1TB chunks, as part of an MS 365 subscription). I had intended that as a secure (?), last-ditch backup, but then wondered if — particularly for use with a Pi (to avoid the need for a separate SSD) — I could actually use it as the primary location from which to stream using MinimServer.

As Simon rightly mentions in another thread, the cloud storage would have to be mounted in such a way as to appear part of the (local) file system. It does seem possible to do that, after a fashion, in MacOS but, based on my very limited experience of it, it seems that the way OneDrive works is to download a file when it is accessed. That isn’t the way I would want it to behave — and it would make the initial scan of the music library completely impractical (even with very high speed fibre broadband).

It may be that this downloading of files on demand is specific to the way OneDrive is implemented for MacOS (I do not know enough about cloud storage to comment on whether other providers’ cloud storage would work differently), but one possible solution may be a package called rclone, which allows a variety of cloud storage types to be locally mounted, and may circumvent this difficulty with the initial scan. Another problem, though (possibly specific to OneDrive), is that access is throttled once a certain number of requests have been made in a specified time period. This may make it impossible to do the scan in any reasonable length of time.

I will investigate more fully when I have time but, for now, have two questions (probably for Simon):

1. When MinimSever scans a music library, is it able to access an individual file’s tag information without loading the whole file?

(I’m assuming it must be, given how quickly the scan seems to be done even with spinning disks on my NAS although, for cloud storage to work, rclone or similar would also have to be capable of reading just the necessary part of a file without having to locally cache the whole thing.)

2. Can MinimServer persist the information it obtains from scanning the music library across reboots?

(I’m wondering if it would be possible, having done a scan, to continue using that information until a rescan is manually requested.)

Thanks ….
Andrew

simoncn · (This post was last modified: 25-11-2023 19:07 by simoncn.)

I will be interested to hear whether you are able to find a workable cloud-based solution. Ideally this would look like a network drive mounted via an SMB connection.

(25-11-2023 17:47)GrumpyPatzer Wrote: 1. When MinimSever scans a music library, is it able to access an individual file’s tag information without loading the whole file?

MinimServer reads only as much of the file as is needed to read the audio information and tag information. For most file formats, this information is at the start of the file. In a few cases, tag information can be at the end of the file. If so, MinimServer seeks to this point in the file rather than reading all the intervening data. How well this seek performs depends on the file system.

Quote:(I’m assuming it must be, given how quickly the scan seems to be done even with spinning disks on my NAS although, for cloud storage to work, rclone or similar would also have to be capable of reading just the necessary part of a file without having to locally cache the whole thing.)

Quite a lot of work has gone into fine-tuning this. Smile

Quote:2. Can MinimServer persist the information it obtains from scanning the music library across reboots?

When the first scan has completed, all the information that has been read is stored in a local cache file on disk. A subsequent rescan reads this cache file in tandem with scanning the library. For each file in the library, if the file size and last modified date match those in the cache entry for the file, the cached information in loaded and no data is read from the file. Any new library files not found in the cache are read (as described above) and their file data is added to the cache. Any files in the cache but no longer in the library have their cache data deleted. Rescanning in this way is typically about 10x faster than the initial scan. Again, quite a lot of work has gone into fine-tuning this.

There is no automatic rescanning in MinimServer. After any change to the library, the user must rescan manually. This is different from Twonky, which spends a lot of CPU resource watching for file changes and rescanning when it detects a change.

GrumpyPatzer · 26-11-2023, 19:16

Thanks for the quick and comprehensive reply.

(25-11-2023 18:44)simoncn Wrote: I will be interested to hear whether you are able to find a workable cloud-based solution. Ideally this would look like a network drive mounted via an SMB connection.

(25-11-2023 17:47)GrumpyPatzer Wrote: 1. When MinimSever scans a music library, is it able to access an individual file’s tag information without loading the whole file?

MinimServer reads only as much of the file as is needed to read the audio information and tag information. For most file formats, this information is at the start of the file. In a few cases, tag information can be at the end of the file. If so, MinimServer seeks to this point in the file rather than reading all the intervening data. How well this seek performs depends on the file system.

Quite a lot of work has gone into fine-tuning this.

Quote:2. Can MinimServer persist the information it obtains from scanning the music library across reboots?

When the first scan has completed, all the information that has been read is stored in a local cache file on disk. A subsequent rescan reads this cache file in tandem with scanning the library. For each file in the library, if the file size and last modified date match those in the cache entry for the file, the cached information in loaded and no data is read from the file. Any new library files not found in the cache are read (as described above) and their file data is added to the cache. Any files in the cache but no longer in the library have their cache data deleted. Rescanning in this way is typically about 10x faster than the initial scan. Again, quite a lot of work has gone into fine-tuning this.

There is no automatic rescanning in MinimServer. After any change to the library, the user must rescan manually. This is different from Twonky, which spends a lot of CPU resource watching for file changes and rescanning when it detects a change.

Rather neat (on both counts!). As it turns out, these optimisations may prove to be very relevant to streaming from cloud based storage. I could not resist having a shot at getting this working (having already uploaded about 25% of my music collection, so roughly 250GB, to MS OneDrive). In no way could I claim to have thought carefully about how best to do this; all I have done so far is to google ways to effect a local mount of OneDrive storage and to spend a few minutes experimenting with different options. There is undoubtedly scope to tune the performance (it is most unlikely that the configuration I am using is optimal). And I strongly suspect that there will be other cloud storage services that will be better suited to streaming media.

That said, I have got it working, to the point where I have (a) scanned just over 8,000 FLAC files; and (b) been able to play several pieces, including a 192/24 recording, with no drop-outs or undue delays. This is running MinimServer v2.2 on a MacBook Pro with a 100Mbps FTTP broadband connection, using an old Linn DS player with a wired LAN connection. The two packages I am using to make the OneDrive storage appear as a locally mounted part of the filesystem are called rclone and FUSE-T.

Some observations:

1. As I anticipated, the initial scan is slow. Having spent only a few short time tuning, the scanning rate I achieved was c.3 files per second (although I now realise that that was with the laptop connected using wifi, not an ethernet cable, which may have introduced some further latency).

2. When I first viewed the library on my iPad, loading the artwork for 274 albums was not particularly snappy (it took a second or two each time I scrolled down), but that was far from unusable. And I am wondering if there is some caching of artwork that will then come into play.

3. I have only played about a quarter of an hours’ worth of music, but it worked flawlessly. No drop-outs or problems of any kind. It is possible that, after playing music for longer, OneDrive might start throttling (if it thinks that too many API requests are being made), but that remains to be seen.

4. For this to be a good alternative to a NAS, I would want it to be easier to use than it currently is. With my NAS, I simply turn it on and off. But it may be possible to achieve a similar level of user friendliness with a Raspberry Pi (particularly if one is prepared to leave it turned on all the time; I certainly would not wish to do that with a laptop).

Now, it may be that there is scope to improve scanning performance somewhat. The first thing to look at, I think, is the set of rclone parameters that control things like buffer sizes and caching. Rather strangely, the best scan rate I have achieved so far has been with a FUSE-T rwsize of 2MB and rclone ifs-read-chunk-size and buffer-size set to 512kB.

In isolation, those parameters don't mean much (I certainly would not claim to know precisely how rclone and FUSE-T work). And I’m not expecting advice about what might work better. But, in order to have some chance of tuning it, I think I need to better understand how MinimServer’s scanning process works. In the MinimServer log, I have observed trace records like

FLAC block type-0 length=34
FLAC block type=4 length=967
Found Vorbis comment block
FLAC block type=3 length-216
FLAC block type=-127 length=7265

The length of the type 0 blocks always seem to be 34, the others vary a bit.

So, if you’ll forgive a couple more questions (sorry to be a nuisance …):

1. With FLAC files like this, how much data is MinimServer having to read, in order to obtain the tag information?

2. Is this reading of tag information being done using several (four?) distinct read operations?

(Depending on how exactly rclone and FUSE-T work, it may be that each distinct read operation has some associated latency, even if the data has been buffered.)

And one further question (which may seem unrelated but is relevant):

3. Having used it on one system, is there a way to transfer that instance of the MinimServer license to another system?

(The reason that is relevant is that I would like to try this cloud storage-based operation on a Raspberry Pi. I have ordered a new one, but might try it on a much older model if it’s possible to then switch the license to the new Pi when it arrives. But it’s absolutely no problem at all if the license is fixed to the system on which it is first used; if that’s the case, I will simply wait until the new Pi turns up.)

Thanks ….
Andrew

simoncn · 27-11-2023, 13:54

(26-11-2023 19:16)GrumpyPatzer Wrote: That said, I have got it working, to the point where I have (a) scanned just over 8,000 FLAC files; and (b) been able to play several pieces, including a 192/24 recording, with no drop-outs or undue delays. This is running MinimServer v2.2 on a MacBook Pro with a 100Mbps FTTP broadband connection, using an old Linn DS player with a wired LAN connection. The two packages I am using to make the OneDrive storage appear as a locally mounted part of the filesystem are called rclone and FUSE-T.

This sounds very encouraging.

Quote:1. As I anticipated, the initial scan is slow. Having spent only a few short time tuning, the scanning rate I achieved was c.3 files per second (although I now realise that that was with the laptop connected using wifi, not an ethernet cable, which may have introduced some further latency).

Changing to Ethernet is likely to make a significant difference. It does for me when I am doing a a large download from the internet or a bulk file copy across my local network.

Quote:2. When I first viewed the library on my iPad, loading the artwork for 274 albums was not particularly snappy (it took a second or two each time I scrolled down), but that was far from unusable. And I am wondering if there is some caching of artwork that will then come into play.

There is no artwork caching in MinimServer but some control points (including those from Linn) do this.

Quote:In the MinimServer log, I have observed trace records like

FLAC block type-0 length=34
FLAC block type=4 length=967
Found Vorbis comment block
FLAC block type=3 length-216
FLAC block type=-127 length=7265

The length of the type 0 blocks always seem to be 34, the others vary a bit.

So, if you’ll forgive a couple more questions (sorry to be a nuisance …):

1. With FLAC files like this, how much data is MinimServer having to read, in order to obtain the tag information?

2. Is this reading of tag information being done using several (four?) distinct read operations?

This means the file contains a STREAMINFO block (34 bytes, containing audio information), a VORBIS_COMMENT block (967 bytes, containing tag information), a SEEKTABLE block (length 216 bytes) and a PADDING block (7265 bytes) followed by the audio data. MinimServer reads the data in the STREAMINFO and VORBIS_COMMENT blocks and skips the data in the other blocks. MinimServer uses a buffered input stream to do all this in order to minimise the number of underlying I/O read and seek operations on the file.

Quote:3. Having used it on one system, is there a way to transfer that instance of the MinimServer license to another system?

The full license can be activated on up to 3 devices at the same time. To move an activation from device 1 to device 2, deactivate the full license from device 1 using the License tab on the MinimServer configuration web page for device 1 and then activate the license on device 2. To see which activations are in use on which devices, open the License details page and enter your license details.

GrumpyPatzer · 29-11-2023, 21:19

Thank you for another comprehensive and clear reply. That all makes good sense.

As I mentioned, the best scan performance I have managed to obtain (no more than around 3 files / sec) is with counter-intuitively large FUSE-T and rclone parameters (e.g. 512kB for rclone read chunk and buffer size), even though not that much data is being read. It is mildly irritating not to be able to work out why this is happening (although that is in no way an implied criticism of MinimServer; I strongly suspect that this is happening because of the way Mac OS, rclone and FUSE-T interact).

So, at the risk of spending too much time on this (given that 3 files / sec is probably adequate for an operation that, in effect, only needs to be done once), I have done two things: used WireShark to capture what is happening on the lo0 interface; and a simple test (on the command line) to obtain the first 32kB from files stored on OneDrive. The way I did the command line test was simply to loop round doing a ‘head -c 32768 …’ command for ten FLAC files on OneDrive. (It needed to be ten different files, because once a file has been read, a second read will be performed *much* more quickly. It is clearly being cached or buffered, probably by rclone, even though I have got VFS caching disabled.)

Using WireShark, I can see the NFS open operation for each file, a set of read operations and the data coming back from OneDrive. Main observations:

1. In both cases (command line test and MinimServer scan), it is typically taking between 200 and 350ms for each file operation, usually over 250ms but occasionally spiking to over 500ms. Unsurprisingly, the bulk of that time is that wait between the read(s) and the data being returned.

2. I can see the FLAC tag information in the first 32k chunk of data, so am fairly confident that I am observing the right things.

3. For each file, there are 16 individual read operations of 32kB each, so 512kB is being returned. (Based on what you say, it seems pretty likely that MinimServer is not asking for that much data. And the ‘head’ command only needs 32kB.)

4. If I reduce the rclone buffer and read-chunk parameters to 64k, the whole operation (to open the file and return 512kB) takes several times longer, typically well over 1s.

So it seems that — for both the ten ‘head’ commands and the MinimServer scan — a minimum of 512kB is being obtained, even though this *may* be more than is being requested.

I do not know why this should be the case. It is, I suppose, possible that this is Mac OS file handling behaviour. The fact that 512kB is still being obtained when I reduce the rclone parameters to 64kB suggests (although not conclusively) that the 512kB is not as a result of rclone behaviour. Also, if we accept that 512kB is being requested, it seems plausible (and is consistent with what I am observing) that reducing the rclone read-chunk size to 64kB would slow things down — if that is causing there to be more round trips to OneDrive.

It may simply be that the latency of opening and obtaining the first chunk of data from a file stored in OneDrive is typically going to average around 300ms, at best, irrespective of broadband speed (as it happens, our broadband service was upgraded from 150 to 500Mbps today).

Unless I can think of (or you can suggest) something else to try, I’ll probably leave it there, at least for now. When I get time to set up my new Pi (which has just arrived), I’ll set it up with MinimServer and rclone and will try playing music stored in OneDrive over a longer period of time. I suppose it’s possible that, if the 512kB is indeed a Mac OS characteristic, we may observe different behaviour with Raspbian or Ubuntu on a Pi.

Andrew

simoncn · 30-11-2023, 11:07

(29-11-2023 21:19)GrumpyPatzer Wrote: 1. In both cases (command line test and MinimServer scan), it is typically taking between 200 and 350ms for each file operation, usually over 250ms but occasionally spiking to over 500ms. Unsurprisingly, the bulk of that time is that wait between the read(s) and the data being returned.

From this it appears that about 3 files per second is the most that can be achieved, even if only one read operation is needed for each file.

Quote:3. For each file, there are 16 individual read operations of 32kB each, so 512kB is being returned. (Based on what you say, it seems pretty likely that MinimServer is not asking for that much data. And the ‘head’ command only needs 32kB.)

Are the 16 reads just for a MinimServer scan or does this also happen for the 32 kB HEAD request?

Quote:So it seems that — for both the ten ‘head’ commands and the MinimServer scan — a minimum of 512kB is being obtained, even though this *may* be more than is being requested.

I do not know why this should be the case. It is, I suppose, possible that this is Mac OS file handling behaviour. The fact that 512kB is still being obtained when I reduce the rclone parameters to 64kB suggests (although not conclusively) that the 512kB is not as a result of rclone behaviour. Also, if we accept that 512kB is being requested, it seems plausible (and is consistent with what I am observing) that reducing the rclone read-chunk size to 64kB would slow things down — if that is causing there to be more round trips to OneDrive.

This seems to be the most likely explanation of what you are seeing.

Quote:It may simply be that the latency of opening and obtaining the first chunk of data from a file stored in OneDrive is typically going to average around 300ms, at best, irrespective of broadband speed (as it happens, our broadband service was upgraded from 150 to 500Mbps today).

My broadband connection is about 70 Mbps and I see latency of a few ms for simple network requests. This suggests the 300 ms latency you are seeing is almost all caused by OneDrive and increasing network bandwidth will make very little difference to the overall latency.

Quote:Unless I can think of (or you can suggest) something else to try, I’ll probably leave it there, at least for now. When I get time to set up my new Pi (which has just arrived), I’ll set it up with MinimServer and rclone and will try playing music stored in OneDrive over a longer period of time. I suppose it’s possible that, if the 512kB is indeed a Mac OS characteristic, we may observe different behaviour with Raspbian or Ubuntu on a Pi.

I will be interested to hear whether this changes what you are seeing.

GrumpyPatzer · (This post was last modified: 07-12-2023 19:39 by GrumpyPatzer.)

With apologies for having taken so long to reply ....

I finally found time to install rclone and MinimServer on my new Pi, and have tried playing some music using the copy of my collection that I have uploaded to MS OneDrive.

In summary: so far, it seems to work fine! See further comments below.

(30-11-2023 11:07)simoncn Wrote:
(29-11-2023 21:19)GrumpyPatzer Wrote: 1. In both cases (command line test and MinimServer scan), it is typically taking between 200 and 350ms for each file operation, usually over 250ms but occasionally spiking to over 500ms. Unsurprisingly, the bulk of that time is that wait between the read(s) and the data being returned.

From this it appears that about 3 files per second is the most that can be achieved, even if only one read operation is needed for each file.

Correct.

I have now done a complete scan of my collection (just over 25,000 FLAC files) from the Pi. I did not do any precise measurements, but it seemed to be somewhat slower, closer to 2 files/sec.

That may be because the combination of the standard version of rclone (which is what I am using to effect a local mount of the OneDrive storage) and the LINUX filesystem does not offer the ability to use the two parameters that I was able to use on MacOS, namely (a) disabling the updating of file access time; and (b) specifying r/w size.

So it's possible that, with a bit of faffing about, I could improve the performance -- but, having now done the scan, I've no real reason to do so.

(30-11-2023 11:07)simoncn Wrote:
(29-11-2023 21:19)GrumpyPatzer Wrote: 3. For each file, there are 16 individual read operations of 32kB each, so 512kB is being returned. (Based on what you say, it seems pretty likely that MinimServer is not asking for that much data. And the ‘head’ command only needs 32kB.)

Are the 16 reads just for a MinimServer scan or does this also happen for the 32 kB HEAD request?

It happened for both. Which reinforces my view that this may well be file system behaviour.

(30-11-2023 11:07)simoncn Wrote:
(29-11-2023 21:19)GrumpyPatzer Wrote: So it seems that — for both the ten ‘head’ commands and the MinimServer scan — a minimum of 512kB is being obtained, even though this *may* be more than is being requested.

I do not know why this should be the case. It is, I suppose, possible that this is Mac OS file handling behaviour. The fact that 512kB is still being obtained when I reduce the rclone parameters to 64kB suggests (although not conclusively) that the 512kB is not as a result of rclone behaviour. Also, if we accept that 512kB is being requested, it seems plausible (and is consistent with what I am observing) that reducing the rclone read-chunk size to 64kB would slow things down — if that is causing there to be more round trips to OneDrive.

This seems to be the most likely explanation of what you are seeing.

Quote:It may simply be that the latency of opening and obtaining the first chunk of data from a file stored in OneDrive is typically going to average around 300ms, at best, irrespective of broadband speed (as it happens, our broadband service was upgraded from 150 to 500Mbps today).

My broadband connection is about 70 Mbps and I see latency of a few ms for simple network requests. This suggests the 300 ms latency you are seeing is almost all caused by OneDrive and increasing network bandwidth will make very little difference to the overall latency.

Exactly. It's quite possible that there is no real mileage in spending too much more time on this -- particularly give that, for me at least, it's working quite satisfactorily.

(30-11-2023 11:07)simoncn Wrote:
(29-11-2023 21:19)GrumpyPatzer Wrote: Unless I can think of (or you can suggest) something else to try, I’ll probably leave it there, at least for now. When I get time to set up my new Pi (which has just arrived), I’ll set it up with MinimServer and rclone and will try playing music stored in OneDrive over a longer period of time. I suppose it’s possible that, if the 512kB is indeed a Mac OS characteristic, we may observe different behaviour with Raspbian or Ubuntu on a Pi.

I will be interested to hear whether this changes what you are seeing.

So, as noted above, the scan performance on the Pi seems somewhat worse, but it's really not a big problem. It seems to me rather nice to be able to play music from my collection held on cloud storage using something as simply as a Pi, with no NAS fan or spinning disk involved!

Andrew

simoncn · 08-12-2023, 12:25

I am pleased to hear you have this working.

You will need to do a rescan whenever you add new files or update tags in files. This requires MinimServer to check the attributes of all existing files to see if they have been updated and read any files that were added or updated.

I would be interested to know whether checking a file's attributes takes about the same amount of time as reading the file or is a quicker operation. This should become clear when you do your next rescan. If this takes significantly less time than doing the initial scan, it means that reading a file's attributes is quicker than reading a portion of the file.

GrumpyPatzer · 08-12-2023, 21:05

(08-12-2023 12:25)simoncn Wrote: I am pleased to hear you have this working.

You will need to do a rescan whenever you add new files or update tags in files. This requires MinimServer to check the attributes of all existing files to see if they have been updated and read any files that were added or updated.

I would be interested to know whether checking a file's attributes takes about the same amount of time as reading the file or is a quicker operation. This should become clear when you do your next rescan. If this takes significantly less time than doing the initial scan, it means that reading a file's attributes is quicker than reading a portion of the file.

The scan done at startup (at which point it is presumably doing the attribute check you describe) is very much quicker, approx. 8 minutes vs. over 3 hours.

It seems to me quite plausible that OneDrive might cache directory information, including attributes such as last modification time, but that is pure speculation on my part. At some point, I suppose I could put Wireshark on the Pi to check what the round trips look like, but I’m afraid I don’t have time to do that right now.

I dare say when I upload some new albums it will be back to the c.300ms per file round trip, but that really doesn’t matter for a few tens of files at a time.

Thanks again for the various explanations and guidance.

Andrew

simoncn · 08-12-2023, 23:42

Thanks for confirming this. A startup/rescan time of 8 minutes is somewhat longer than would be expected for a local library of this size but doesn't seem unreasonable or unacceptable.