Support running multiple record engines as a pool
-
- Expert
- Posts: 11110
- Joined: Wed Jan 24, 2007 1:04 am
- Device ID: 10A05954 10802091 131B34B7 13231F92 1070A18E 1073ED6F 15300C36
- x 26
Re: NEW - support running multiple record engines as a pool
Personally for such, I would no longer have the share in the directory of the other... I used to do it this way without issue to archive things but it would also show "everything" twice when the other was running as an engine rather than just a share. (Note network shares in that manner are not supported file locations for the DVR, but I have had good luck when one was used as an archive location but changes made to such location may not be picked up automatically after a while)
Re: NEW - support running multiple record engines as a pool
FYI - it is ok to use a local path that is shared but we don't support using a remote network share.
The key problem is monitoring for file changes which is required for operation. It will work at first but once you hit about 62 folders you hit a resource limit when using a remote share.
Nick
The key problem is monitoring for file changes which is required for operation. It will work at first but once you hit about 62 folders you hit a resource limit when using a remote share.
Nick
Re: NEW - support running multiple record engines as a pool
Not sure how you are sharing, but neither NFS nor SMB reliably work due to their lack of complete filesystem emulation. While one can sort of get things to work some of the time, the recommendation is not to share for it will fail in weird ways eventually.
Re: NEW - support running multiple record engines as a pool
Prior to this new enhancement, I had both servers setup as a primary/backup where they would both determine if the recording engine was running on the other and if not the other would start.
Like most users, I would like to keep all of the recordings in the same location. But it isn't necessary.
It would be nice if we could configure the recording engine where to write the log file and what name it should have. But it isn't necessary, I would just need to determine if I really need to have more than one recording engine running.
Re: NEW - support running multiple record engines as a pool
Feature - the HDHomeRun support for running multiple record engines also provides support for hardware redundancy.
If a server fails or drops offline, all future recordings will continue with the remaining server(s). At most the recording-in-progress is truncated.
Nick
If a server fails or drops offline, all future recordings will continue with the remaining server(s). At most the recording-in-progress is truncated.
Nick
Re: Support running multiple record engines as a pool
I am testing this on 2 NAS'S everything is working great when recording only records to 1 NAS ... but I also noticed that when watching LIVETV on the same channel on 2 devices it writes the LIVETV buffer to both NAS's and uses 2 tuners.
-
- Posts: 271
- Joined: Sun Apr 09, 2017 11:12 am
- Device ID: 1326E235 1313788A
- Location: Olympia, WA
- x 1
- Contact:
Re: NEW - support running multiple record engines as a pool
I 3rd this --signcarver wrote: ↑Sat May 18, 2019 12:55 pmI believe nickk stated it would generally use the one with most space... I can see that I also would like to set a preference and perhaps another preference for LiveTV.
I'd like to set it up only as a failover in the event that my primary record engine is down for whatever reason, or runs out of space completely (or nearly completely)
I don't just want the secondary record engine to start recording if the primary record engine's free space gets lower than the secondary one's free space.
Hoping there's an affinity setting in the near future (e.g. 80/20 like my domain controllers), or an explicit integer for record order (e.g. 0,1,2,3)
-
- Posts: 271
- Joined: Sun Apr 09, 2017 11:12 am
- Device ID: 1326E235 1313788A
- Location: Olympia, WA
- x 1
- Contact:
Re: NEW - support running multiple record engines as a pool
I have kind of an unconventional setup in that my fileserver is an OmniOS VM (for windows domain compatibility, Samba is awful) but there's no HDHomeRun record engine for OmniOS, so I use an NFS share in fstab in a FreeBSD VM, and they share a dedicated subnet specifically for the NFS share using vmxnet3 (it's an ESXi host).
I set it all up before reading that shares weren't supported, but have been running it over a year now without issue. I record the news so I can watch on delay and skip the commercials while working from home - so I do a lot of transport ops, switching between recordings, etc.
I ran a straight FreeBSD setup w/ local storage for about 6-8 months before that, and I haven't noticed a difference.
To check on the directory limit, I just ran...:
Code: Select all
$ tree -L 1 | tail -1
129 directories, 32 files
Honestly, I'm not sure if the zfs makes a difference for large sequential reads, but I know it caches files to memory to be fed out as quickly as possible. The VM has 20GB ram allocated to an OS that probably runs in under 512MB.
The other thing is the "network connection" between the two VMs is virtual and contained within the single host. I've done iperf3 tests on it and got around 26GBps throughput, so it's not exactly slow.
I personally wouldn't want to run a share across the network for recording HDHR, even with a 10Gbps backbone, due to latency issues, and I'm not advocating anyone else do this with their hypervisor, as obviously it's unsupported. But thankfully it's been working OK so far, and is kind of interesting.
-
- Posts: 271
- Joined: Sun Apr 09, 2017 11:12 am
- Device ID: 1326E235 1313788A
- Location: Olympia, WA
- x 1
- Contact:
Re: Support running multiple record engines as a pool
I can confirm that I have two record engines working on two separate ESXi hosts in FreeBSD 12, it appears to be showing all recordings from both locations.
Thankfully it's only recording on the one I want it to, since it has more free space. Please let me know if there's an update that will allow me to set record engine priority, affinity, or something like that, so I can make it a proper failover setup.
Thanks for the great work!
Edit: I'm noticing the next morning using the two-DVR setup that my UI doesn't refresh new recordings unless I hit the "recordings" button. This used to happen occasionally on my single-DVR setup (maybe once every couple days?) but now happens nearly every time I view recordings.
It looks like the UI is reverting to an earlier state. I'm not sure if it has to do more with the test UI or the two-DVR setup, but it just started this morning and setting up the 2nd record engine is the last thing I've done to my setup.
Ohhh... I'm also noticing I'm on test UI 20190726b now, too - early access on (W10 1809). Last I checked I was on 0723. I'm not sure when it updated but may have coincided with this issue.
So basically enough has changed that I'm not sure what it is at this point but I'm generating logs if that helps you guys troubleshoot. It's nothing more than a minor nuisance though.
Thankfully it's only recording on the one I want it to, since it has more free space. Please let me know if there's an update that will allow me to set record engine priority, affinity, or something like that, so I can make it a proper failover setup.
Thanks for the great work!
Edit: I'm noticing the next morning using the two-DVR setup that my UI doesn't refresh new recordings unless I hit the "recordings" button. This used to happen occasionally on my single-DVR setup (maybe once every couple days?) but now happens nearly every time I view recordings.
It looks like the UI is reverting to an earlier state. I'm not sure if it has to do more with the test UI or the two-DVR setup, but it just started this morning and setting up the 2nd record engine is the last thing I've done to my setup.
Ohhh... I'm also noticing I'm on test UI 20190726b now, too - early access on (W10 1809). Last I checked I was on 0723. I'm not sure when it updated but may have coincided with this issue.
So basically enough has changed that I'm not sure what it is at this point but I'm generating logs if that helps you guys troubleshoot. It's nothing more than a minor nuisance though.
-
- Posts: 271
- Joined: Sun Apr 09, 2017 11:12 am
- Device ID: 1326E235 1313788A
- Location: Olympia, WA
- x 1
- Contact:
Re: NEW - support running multiple record engines as a pool
I tested this today:nickk wrote: ↑Sun Jun 02, 2019 9:07 am Feature - the HDHomeRun support for running multiple record engines also provides support for hardware redundancy.
If a server fails or drops offline, all future recordings will continue with the remaining server(s). At most the recording-in-progress is truncated.
Nick
Recording continuity didn't work for me, in that when I took my primary record server down, the recordings that were recording at the time did not continue recording
I did not wait to see if the next scheduled recordings at the top of the hour would have started on the secondary server, but I will have to test that at some point
Interestingly, the recordings that were recording on the primary record server started recording again (towards the end) when I brought the primary record server back up
The secondary record server was OK for live TV, and the recordings on that server were present - but the live TV (which I was watching when I took the primary record server down) stopped and I had to re-start the HDHR client
Still some kinks to work out? Or am I just expecting too much?
Thanks
-
- Expert
- Posts: 11110
- Joined: Wed Jan 24, 2007 1:04 am
- Device ID: 10A05954 10802091 131B34B7 13231F92 1070A18E 1073ED6F 15300C36
- x 26
Re: Support running multiple record engines as a pool
Your expecting too much... recordings "must" start on time as there is no late start (except at start of engine when it first comes up) and as such the recordings would be truncated (though an engine restarting may pick up in progress, or you might try poking another engine when it was down, though that might not work unless it was a new order). What you were supposed to do is wait for the next set of recordings and see that they did record.
-
- Posts: 271
- Joined: Sun Apr 09, 2017 11:12 am
- Device ID: 1326E235 1313788A
- Location: Olympia, WA
- x 1
- Contact:
Re: Support running multiple record engines as a pool
Was kind of on a whim, I was setting up a VMDK (migrating away from vnet NFS) and just wanted to see what would happensigncarver wrote: ↑Mon Jul 29, 2019 2:33 pm Your expecting too much... recordings "must" start on time as there is no late start (except at start of engine when it first comes up) and as such the recordings would be truncated (though an engine restarting may pick up in progress, or you might try poking another engine when it was down, though that might not work unless it was a new order). What you were supposed to do is wait for the next set of recordings and see that they did record.
oh well! Never hurts to try. Would love to see proper failover capabilities eventually. that would be super cool.
-
- Posts: 271
- Joined: Sun Apr 09, 2017 11:12 am
- Device ID: 1326E235 1313788A
- Location: Olympia, WA
- x 1
- Contact:
Re: Support running multiple record engines as a pool
I'm no longer using NFS for any recording storage. If I'm running two record engines, the list of my recordings still has to be "refreshed" by hitting the recordings button again on the top menu about 1 out of every 3 times I return to the recordings menu.
It looks like the client is only seeing one of the two about a third of the time until I hit the recordings button again. The problem goes away if I stop one of the recording services.
I'm on 20190715beta1
Edit: Tonight my client stopped showing the recordings from the primary record server at all. Had to turn off the secondary (backup) record engine in order to see my recordings again. Weird.
It looks like the client is only seeing one of the two about a third of the time until I hit the recordings button again. The problem goes away if I stop one of the recording services.
I'm on 20190715beta1
Edit: Tonight my client stopped showing the recordings from the primary record server at all. Had to turn off the secondary (backup) record engine in order to see my recordings again. Weird.
-
- Posts: 271
- Joined: Sun Apr 09, 2017 11:12 am
- Device ID: 1326E235 1313788A
- Location: Olympia, WA
- x 1
- Contact:
Re: Support running multiple record engines as a pool
I noticed there's a new version so I thought I'd try testing the dual recorders again
20190815beta1 ... will report back
20190815beta1 ... will report back
-
- Posts: 271
- Joined: Sun Apr 09, 2017 11:12 am
- Device ID: 1326E235 1313788A
- Location: Olympia, WA
- x 1
- Contact:
Re: Support running multiple record engines as a pool
OK tested out 20190815Beta1 and still having an issue where at first the two seem to peacefully co-exist. I might have to click "recordings" to "refresh" one of the servers because the "secondary" record server's recordings tend to appear first, but hitting "recordings" can make the "primary" record server's recordings appear without too much trouble
Well, once the recordings on the primary record server reached a certain level, clicking the "recordings" button wouldn't make them appear anymore. This is a problem because obviously that's where most my recordings are. So I had to turn off the "secondary" record server in order to see my "primary" record server again - once it is the only record server running on my network.
To re-cap, I'm running two FreeBSD 12-Release in VMs on 2 separate ESXi 6.7U1 hosts in a domain environment (2x2012R2 DCs). DCs providing AD/IDMU local dns, dhcp. Gateway/firewall OPNSense 19.2 (root hints from DCs, not domain forwarder)
Well, once the recordings on the primary record server reached a certain level, clicking the "recordings" button wouldn't make them appear anymore. This is a problem because obviously that's where most my recordings are. So I had to turn off the "secondary" record server in order to see my "primary" record server again - once it is the only record server running on my network.
To re-cap, I'm running two FreeBSD 12-Release in VMs on 2 separate ESXi 6.7U1 hosts in a domain environment (2x2012R2 DCs). DCs providing AD/IDMU local dns, dhcp. Gateway/firewall OPNSense 19.2 (root hints from DCs, not domain forwarder)